[GitHub] carbondata pull request #2073: [CARBONDATA-2260] CarbonThriftServer should s...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2073 ---
[GitHub] carbondata pull request #2073: [CARBONDATA-2260] CarbonThriftServer should s...
GitHub user xubo245 reopened a pull request:
https://github.com/apache/carbondata/pull/2073
[CARBONDATA-2260] CarbonThriftServer should support store carbon table on S3
CarbonThriftServer should support store carbon table on S3
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [x] Any interfaces changed?
- [x] Any backward compatibility impacted?
- [x] Document update required?
- [x] Testing done
Step 1:
start carbonThriftServer by
org.apache.carbondata.examples.CarbonThriftServerExample
User should use themselves storePath, AK, SK and endPoint
Step 2:
start beeline like: beeline -u jdbc:hive2://IP:1
IP and port should provide
Now it support running with local, HDFS and S3 mode
Test example:
```
root@ecs-909c:~# beeline -u jdbc:hive2://192.168.0.206:1
Connecting to jdbc:hive2://192.168.0.206:1
2018-03-17 17:44:29 INFO Utils:310 - Supplied authorities:
192.168.0.206:1
2018-03-17 17:44:29 INFO Utils:397 - Resolved authority:
192.168.0.206:1
2018-03-17 17:44:29 INFO HiveConnection:203 - Will try to open client
transport with JDBC Uri: jdbc:hive2://192.168.0.206:1
Connected to: Spark SQL (version 2.2.1)
Driver: Hive JDBC (version 1.2.1.spark2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.2.1.spark2 by Apache Hive
0: jdbc:hive2://192.168.0.206:1> Drop table if exists local_table2;
+-+--+
| Result |
+-+--+
+-+--+
No rows selected (21.205 seconds)
0: jdbc:hive2://192.168.0.206:1>
0: jdbc:hive2://192.168.0.206:1> CREATE TABLE if not exists
local_table2( shortField SHORT, intField INT, bigintField LONG, doubleField
DOUBLE, stringField STRING, timestampField TIMESTAMP, decimalField
DECIMAL(18,2), dateField DATE, charField CHAR(5), floatField FLOAT ) STORED BY
'carbondata' LOCATION 's3a://carbonstore/local_table2'
TBLPROPERTIES('SORT_COLUMNS'='');
+-+--+
| Result |
+-+--+
+-+--+
No rows selected (1.296 seconds)
0: jdbc:hive2://192.168.0.206:1>
0: jdbc:hive2://192.168.0.206:1>
0: jdbc:hive2://192.168.0.206:1> LOAD DATA LOCAL INPATH
'/huawei/xubo/git/carbondata2/examples/spark2/src/main/resources/data1.csv'
INTO TABLE local_table2 OPTIONS('HEADER'='true');
+-+--+
| Result |
+-+--+
+-+--+
No rows selected (3.339 seconds)
0: jdbc:hive2://192.168.0.206:1>
0: jdbc:hive2://192.168.0.206:1> select * from local_table2;
+-+---+--+--+--++---+-++-+--+
| shortfield | intfield | bigintfield | doublefield | stringfield |
timestampfield | decimalfield | datefield | charfield | floatfield |
+-+---+--+--+--++---+-++-+--+
| 1 | 10| 1100 | 48.4 | spark|
2015-04-23 12:01:01.0 | 1.23 | 2015-04-23 | aaa| 2.5
|
| 5 | 17| 1140 | 43.4 | spark|
2015-07-27 12:01:02.0 | 3.45 | 2015-07-27 | bbb| 2.5
|
| 1 | 11| 1100 | 44.4 | flink|
2015-05-23 12:01:03.0 | 23.23 | 2015-05-23 | ccc| 2.5
|
| 1 | 10| 1150 | 43.4 | spark|
2015-07-24 12:01:04.0 | 254.12| 2015-07-24 | ddd| 2.5
|
| 1 | 10| 1100 | 47.4 | spark|
2015-07-23 12:01:05.0 | 876.14| 2015-07-23 | | 3.5
|
| 3 | 14| 1160 | 43.4 | hive |
2015-07-26 12:01:06.0 | 3454.32 | 2015-07-26 | ff | 2.5
|
| 2 | 10| 1100 | 43.4 | impala |
2015-07-23 12:01:07.0 | 456.98| 2015-07-23 | ggg| 2.5
|
| 1 | 10| 1100 | 43.4 | spark|
2015-05-23 12:01:08.0 | 32.53 | 2015-05-23 | hhh| 2.5
|
| 4 | 16| 1130 | 42.4 | impala |
2015-07-23 12:01:09.0 | 67.23 | 2015-07-23 | iii| 2.5
|
| 1 | 10| 1100 | 43.4 | spark|
2015-07-23 12:01:10.0 | 832.23| 2015-07-23 | jjj| 2.5
|
[GitHub] carbondata pull request #2073: [CARBONDATA-2260] CarbonThriftServer should s...
Github user xubo245 closed the pull request at: https://github.com/apache/carbondata/pull/2073 ---
[GitHub] carbondata pull request #2073: [CARBONDATA-2260] CarbonThriftServer should s...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2073#discussion_r176409638
--- Diff:
integration/spark2/src/main/scala/org/apache/carbondata/spark/thriftserver/CarbonThriftServer.scala
---
@@ -34,10 +36,23 @@ object CarbonThriftServer {
import org.apache.spark.sql.CarbonSession._
val sparkConf = new SparkConf(loadDefaults = true)
+
+val logger: Logger = LoggerFactory.getLogger(this.getClass)
+if (args.length != 1 && args.length != 4) {
+ logger.error("parameters: storePath [access-key] [secret-key]
[s3-endpoint]")
--- End diff --
Fix it, now it support local, HDFS and S3 mode
---
[GitHub] carbondata pull request #2073: [CARBONDATA-2260] CarbonThriftServer should s...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2073#discussion_r176074650
--- Diff:
integration/spark2/src/main/scala/org/apache/carbondata/spark/thriftserver/CarbonThriftServer.scala
---
@@ -34,10 +36,23 @@ object CarbonThriftServer {
import org.apache.spark.sql.CarbonSession._
val sparkConf = new SparkConf(loadDefaults = true)
+
+val logger: Logger = LoggerFactory.getLogger(this.getClass)
+if (args.length != 1 && args.length != 4) {
+ logger.error("parameters: storePath [access-key] [secret-key]
[s3-endpoint]")
--- End diff --
After this change, how user can use this class for HDFS cluster?
---
[GitHub] carbondata pull request #2073: [CARBONDATA-2260] CarbonThriftServer should s...
GitHub user xubo245 opened a pull request:
https://github.com/apache/carbondata/pull/2073
[CARBONDATA-2260] CarbonThriftServer should support store carbon table on S3
CarbonThriftServer should support store carbon table on S3
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed?
- [ ] Any backward compatibility impacted?
- [ ] Document update required?
- [ ] Testing done
Step 1:
start carbonThriftServer by
org.apache.carbondata.examples.CarbonThriftServerExample
User should use themselves storePath, AK, SK and endPoint
Step 2:
start beeline like: beeline -u jdbc:hive2://192.168.0.206:1
IP and port should provide
Test example:
```
root@ecs-909c:~# beeline -u jdbc:hive2://192.168.0.206:1
Connecting to jdbc:hive2://192.168.0.206:1
2018-03-17 17:44:29 INFO Utils:310 - Supplied authorities:
192.168.0.206:1
2018-03-17 17:44:29 INFO Utils:397 - Resolved authority:
192.168.0.206:1
2018-03-17 17:44:29 INFO HiveConnection:203 - Will try to open client
transport with JDBC Uri: jdbc:hive2://192.168.0.206:1
Connected to: Spark SQL (version 2.2.1)
Driver: Hive JDBC (version 1.2.1.spark2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.2.1.spark2 by Apache Hive
0: jdbc:hive2://192.168.0.206:1> Drop table if exists local_table2;
+-+--+
| Result |
+-+--+
+-+--+
No rows selected (21.205 seconds)
0: jdbc:hive2://192.168.0.206:1>
0: jdbc:hive2://192.168.0.206:1> CREATE TABLE if not exists
local_table2( shortField SHORT, intField INT, bigintField LONG, doubleField
DOUBLE, stringField STRING, timestampField TIMESTAMP, decimalField
DECIMAL(18,2), dateField DATE, charField CHAR(5), floatField FLOAT ) STORED BY
'carbondata' LOCATION 's3a://carbonstore/local_table2'
TBLPROPERTIES('SORT_COLUMNS'='');
+-+--+
| Result |
+-+--+
+-+--+
No rows selected (1.296 seconds)
0: jdbc:hive2://192.168.0.206:1>
0: jdbc:hive2://192.168.0.206:1>
0: jdbc:hive2://192.168.0.206:1> LOAD DATA LOCAL INPATH
'/huawei/xubo/git/carbondata2/examples/spark2/src/main/resources/data1.csv'
INTO TABLE local_table2 OPTIONS('HEADER'='true');
+-+--+
| Result |
+-+--+
+-+--+
No rows selected (3.339 seconds)
0: jdbc:hive2://192.168.0.206:1>
0: jdbc:hive2://192.168.0.206:1> select * from local_table2;
+-+---+--+--+--++---+-++-+--+
| shortfield | intfield | bigintfield | doublefield | stringfield |
timestampfield | decimalfield | datefield | charfield | floatfield |
+-+---+--+--+--++---+-++-+--+
| 1 | 10| 1100 | 48.4 | spark|
2015-04-23 12:01:01.0 | 1.23 | 2015-04-23 | aaa| 2.5
|
| 5 | 17| 1140 | 43.4 | spark|
2015-07-27 12:01:02.0 | 3.45 | 2015-07-27 | bbb| 2.5
|
| 1 | 11| 1100 | 44.4 | flink|
2015-05-23 12:01:03.0 | 23.23 | 2015-05-23 | ccc| 2.5
|
| 1 | 10| 1150 | 43.4 | spark|
2015-07-24 12:01:04.0 | 254.12| 2015-07-24 | ddd| 2.5
|
| 1 | 10| 1100 | 47.4 | spark|
2015-07-23 12:01:05.0 | 876.14| 2015-07-23 | | 3.5
|
| 3 | 14| 1160 | 43.4 | hive |
2015-07-26 12:01:06.0 | 3454.32 | 2015-07-26 | ff | 2.5
|
| 2 | 10| 1100 | 43.4 | impala |
2015-07-23 12:01:07.0 | 456.98| 2015-07-23 | ggg| 2.5
|
| 1 | 10| 1100 | 43.4 | spark|
2015-05-23 12:01:08.0 | 32.53 | 2015-05-23 | hhh| 2.5
|
| 4 | 16| 1130 | 42.4 | impala |
2015-07-23 12:01:09.0 | 67.23 | 2015-07-23 | iii| 2.5
|
| 1 | 10| 1100 | 43.4 | spark|
2015-07-23 12:01:10.0 | 832.23| 2015-07-23 | jjj| 2.5
|
+-+---+--+--+--++---+-++-+--+
