[GitHub] carbondata pull request #2073: [CARBONDATA-2260] CarbonThriftServer should s...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2073 ---
[GitHub] carbondata pull request #2073: [CARBONDATA-2260] CarbonThriftServer should s...
GitHub user xubo245 reopened a pull request: https://github.com/apache/carbondata/pull/2073 [CARBONDATA-2260] CarbonThriftServer should support store carbon table on S3 CarbonThriftServer should support store carbon table on S3 Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [x] Any interfaces changed? - [x] Any backward compatibility impacted? - [x] Document update required? - [x] Testing done Step 1: start carbonThriftServer by org.apache.carbondata.examples.CarbonThriftServerExample User should use themselves storePath, AK, SK and endPoint Step 2: start beeline like: beeline -u jdbc:hive2://IP:1 IP and port should provide Now it support running with local, HDFS and S3 mode Test example: ``` root@ecs-909c:~# beeline -u jdbc:hive2://192.168.0.206:1 Connecting to jdbc:hive2://192.168.0.206:1 2018-03-17 17:44:29 INFO Utils:310 - Supplied authorities: 192.168.0.206:1 2018-03-17 17:44:29 INFO Utils:397 - Resolved authority: 192.168.0.206:1 2018-03-17 17:44:29 INFO HiveConnection:203 - Will try to open client transport with JDBC Uri: jdbc:hive2://192.168.0.206:1 Connected to: Spark SQL (version 2.2.1) Driver: Hive JDBC (version 1.2.1.spark2) Transaction isolation: TRANSACTION_REPEATABLE_READ Beeline version 1.2.1.spark2 by Apache Hive 0: jdbc:hive2://192.168.0.206:1> Drop table if exists local_table2; +-+--+ | Result | +-+--+ +-+--+ No rows selected (21.205 seconds) 0: jdbc:hive2://192.168.0.206:1> 0: jdbc:hive2://192.168.0.206:1> CREATE TABLE if not exists local_table2( shortField SHORT, intField INT, bigintField LONG, doubleField DOUBLE, stringField STRING, timestampField TIMESTAMP, decimalField DECIMAL(18,2), dateField DATE, charField CHAR(5), floatField FLOAT ) STORED BY 'carbondata' LOCATION 's3a://carbonstore/local_table2' TBLPROPERTIES('SORT_COLUMNS'=''); +-+--+ | Result | +-+--+ +-+--+ No rows selected (1.296 seconds) 0: jdbc:hive2://192.168.0.206:1> 0: jdbc:hive2://192.168.0.206:1> 0: jdbc:hive2://192.168.0.206:1> LOAD DATA LOCAL INPATH '/huawei/xubo/git/carbondata2/examples/spark2/src/main/resources/data1.csv' INTO TABLE local_table2 OPTIONS('HEADER'='true'); +-+--+ | Result | +-+--+ +-+--+ No rows selected (3.339 seconds) 0: jdbc:hive2://192.168.0.206:1> 0: jdbc:hive2://192.168.0.206:1> select * from local_table2; +-+---+--+--+--++---+-++-+--+ | shortfield | intfield | bigintfield | doublefield | stringfield | timestampfield | decimalfield | datefield | charfield | floatfield | +-+---+--+--+--++---+-++-+--+ | 1 | 10| 1100 | 48.4 | spark| 2015-04-23 12:01:01.0 | 1.23 | 2015-04-23 | aaa| 2.5 | | 5 | 17| 1140 | 43.4 | spark| 2015-07-27 12:01:02.0 | 3.45 | 2015-07-27 | bbb| 2.5 | | 1 | 11| 1100 | 44.4 | flink| 2015-05-23 12:01:03.0 | 23.23 | 2015-05-23 | ccc| 2.5 | | 1 | 10| 1150 | 43.4 | spark| 2015-07-24 12:01:04.0 | 254.12| 2015-07-24 | ddd| 2.5 | | 1 | 10| 1100 | 47.4 | spark| 2015-07-23 12:01:05.0 | 876.14| 2015-07-23 | | 3.5 | | 3 | 14| 1160 | 43.4 | hive | 2015-07-26 12:01:06.0 | 3454.32 | 2015-07-26 | ff | 2.5 | | 2 | 10| 1100 | 43.4 | impala | 2015-07-23 12:01:07.0 | 456.98| 2015-07-23 | ggg| 2.5 | | 1 | 10| 1100 | 43.4 | spark| 2015-05-23 12:01:08.0 | 32.53 | 2015-05-23 | hhh| 2.5 | | 4 | 16| 1130 | 42.4 | impala | 2015-07-23 12:01:09.0 | 67.23 | 2015-07-23 | iii| 2.5 | | 1 | 10| 1100 | 43.4 | spark| 2015-07-23 12:01:10.0 | 832.23| 2015-07-23 | jjj| 2.5 | +-+---+--+--+--++---+--
[GitHub] carbondata pull request #2073: [CARBONDATA-2260] CarbonThriftServer should s...
Github user xubo245 closed the pull request at: https://github.com/apache/carbondata/pull/2073 ---
[GitHub] carbondata pull request #2073: [CARBONDATA-2260] CarbonThriftServer should s...
Github user xubo245 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2073#discussion_r176409638 --- Diff: integration/spark2/src/main/scala/org/apache/carbondata/spark/thriftserver/CarbonThriftServer.scala --- @@ -34,10 +36,23 @@ object CarbonThriftServer { import org.apache.spark.sql.CarbonSession._ val sparkConf = new SparkConf(loadDefaults = true) + +val logger: Logger = LoggerFactory.getLogger(this.getClass) +if (args.length != 1 && args.length != 4) { + logger.error("parameters: storePath [access-key] [secret-key] [s3-endpoint]") --- End diff -- Fix it, now it support local, HDFS and S3 mode ---
[GitHub] carbondata pull request #2073: [CARBONDATA-2260] CarbonThriftServer should s...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2073#discussion_r176074650 --- Diff: integration/spark2/src/main/scala/org/apache/carbondata/spark/thriftserver/CarbonThriftServer.scala --- @@ -34,10 +36,23 @@ object CarbonThriftServer { import org.apache.spark.sql.CarbonSession._ val sparkConf = new SparkConf(loadDefaults = true) + +val logger: Logger = LoggerFactory.getLogger(this.getClass) +if (args.length != 1 && args.length != 4) { + logger.error("parameters: storePath [access-key] [secret-key] [s3-endpoint]") --- End diff -- After this change, how user can use this class for HDFS cluster? ---
[GitHub] carbondata pull request #2073: [CARBONDATA-2260] CarbonThriftServer should s...
GitHub user xubo245 opened a pull request: https://github.com/apache/carbondata/pull/2073 [CARBONDATA-2260] CarbonThriftServer should support store carbon table on S3 CarbonThriftServer should support store carbon table on S3 Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Step 1: start carbonThriftServer by org.apache.carbondata.examples.CarbonThriftServerExample User should use themselves storePath, AK, SK and endPoint Step 2: start beeline like: beeline -u jdbc:hive2://192.168.0.206:1 IP and port should provide Test example: ``` root@ecs-909c:~# beeline -u jdbc:hive2://192.168.0.206:1 Connecting to jdbc:hive2://192.168.0.206:1 2018-03-17 17:44:29 INFO Utils:310 - Supplied authorities: 192.168.0.206:1 2018-03-17 17:44:29 INFO Utils:397 - Resolved authority: 192.168.0.206:1 2018-03-17 17:44:29 INFO HiveConnection:203 - Will try to open client transport with JDBC Uri: jdbc:hive2://192.168.0.206:1 Connected to: Spark SQL (version 2.2.1) Driver: Hive JDBC (version 1.2.1.spark2) Transaction isolation: TRANSACTION_REPEATABLE_READ Beeline version 1.2.1.spark2 by Apache Hive 0: jdbc:hive2://192.168.0.206:1> Drop table if exists local_table2; +-+--+ | Result | +-+--+ +-+--+ No rows selected (21.205 seconds) 0: jdbc:hive2://192.168.0.206:1> 0: jdbc:hive2://192.168.0.206:1> CREATE TABLE if not exists local_table2( shortField SHORT, intField INT, bigintField LONG, doubleField DOUBLE, stringField STRING, timestampField TIMESTAMP, decimalField DECIMAL(18,2), dateField DATE, charField CHAR(5), floatField FLOAT ) STORED BY 'carbondata' LOCATION 's3a://carbonstore/local_table2' TBLPROPERTIES('SORT_COLUMNS'=''); +-+--+ | Result | +-+--+ +-+--+ No rows selected (1.296 seconds) 0: jdbc:hive2://192.168.0.206:1> 0: jdbc:hive2://192.168.0.206:1> 0: jdbc:hive2://192.168.0.206:1> LOAD DATA LOCAL INPATH '/huawei/xubo/git/carbondata2/examples/spark2/src/main/resources/data1.csv' INTO TABLE local_table2 OPTIONS('HEADER'='true'); +-+--+ | Result | +-+--+ +-+--+ No rows selected (3.339 seconds) 0: jdbc:hive2://192.168.0.206:1> 0: jdbc:hive2://192.168.0.206:1> select * from local_table2; +-+---+--+--+--++---+-++-+--+ | shortfield | intfield | bigintfield | doublefield | stringfield | timestampfield | decimalfield | datefield | charfield | floatfield | +-+---+--+--+--++---+-++-+--+ | 1 | 10| 1100 | 48.4 | spark| 2015-04-23 12:01:01.0 | 1.23 | 2015-04-23 | aaa| 2.5 | | 5 | 17| 1140 | 43.4 | spark| 2015-07-27 12:01:02.0 | 3.45 | 2015-07-27 | bbb| 2.5 | | 1 | 11| 1100 | 44.4 | flink| 2015-05-23 12:01:03.0 | 23.23 | 2015-05-23 | ccc| 2.5 | | 1 | 10| 1150 | 43.4 | spark| 2015-07-24 12:01:04.0 | 254.12| 2015-07-24 | ddd| 2.5 | | 1 | 10| 1100 | 47.4 | spark| 2015-07-23 12:01:05.0 | 876.14| 2015-07-23 | | 3.5 | | 3 | 14| 1160 | 43.4 | hive | 2015-07-26 12:01:06.0 | 3454.32 | 2015-07-26 | ff | 2.5 | | 2 | 10| 1100 | 43.4 | impala | 2015-07-23 12:01:07.0 | 456.98| 2015-07-23 | ggg| 2.5 | | 1 | 10| 1100 | 43.4 | spark| 2015-05-23 12:01:08.0 | 32.53 | 2015-05-23 | hhh| 2.5 | | 4 | 16| 1130 | 42.4 | impala | 2015-07-23 12:01:09.0 | 67.23 | 2015-07-23 | iii| 2.5 | | 1 | 10| 1100 | 43.4 | spark| 2015-07-23 12:01:10.0 | 832.23| 2015-07-23 | jjj| 2.5 | +-+---+--+--+--++---+-++-+--+ 10