[GitHub] incubator-carbondata pull request #417: fix conf issue for scanrdd

2016-12-10 Thread scwf 
Github user scwf commented on a diff in the pull request:

https://github.com/apache/incubator-carbondata/pull/417#discussion_r91848571
  
--- Diff: 
integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala ---
@@ -37,6 +36,9 @@ object CarbonEnv {
 
   @volatile private var carbonEnv: CarbonEnv = _
 
+  // set readsupport class global so that the executor can get it.
+  SparkReadSupport.readSupportClass = classOf[SparkRowReadSupportImpl]
--- End diff --

also the same


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata issue #417: fix conf issue for scanrdd

2016-12-10 Thread scwf 
Github user scwf commented on the issue:

https://github.com/apache/incubator-carbondata/pull/417
  
/cc @QiangCai @jackylk plz review this


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #417: fix conf issue for scanrdd

2016-12-10 Thread scwf 
GitHub user scwf opened a pull request:

https://github.com/apache/incubator-carbondata/pull/417

fix conf issue for scanrdd

we should sync the conf in executor side

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/KirinKing/incubator-carbondata fix-scan

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-carbondata/pull/417.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #417


commit 1c54cd4ef942be0f5e2eef40c34db98a2233a0dc
Author: wangfei <wangfei_he...@126.com>
Date:   2016-12-10T23:24:03Z

fix conf issue for scanrdd




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata issue #415: Fix integrate with spark 2.1

2016-12-09 Thread scwf 
Github user scwf commented on the issue:

https://github.com/apache/incubator-carbondata/pull/415
  
/cc @QiangCai  can you try this fix


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #415: Fix integrate with spark 2.1

2016-12-09 Thread scwf 
GitHub user scwf opened a pull request:

https://github.com/apache/incubator-carbondata/pull/415

Fix integrate with spark 2.1



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/KirinKing/incubator-carbondata fix-2.1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-carbondata/pull/415.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #415


commit 74e9b98d2596d5c52fa8ef43aeddeaf6ca689c01
Author: wangfei <wangfei_he...@126.com>
Date:   2016-12-09T09:28:59Z

use carbon property to get the store path

commit 906ab88ed4ebe178bf1a52e1bb26f3dc12d4f07c
Author: wangfei <wangfei_he...@126.com>
Date:   2016-12-09T09:51:32Z

remove nouse imports

commit 14cfe617f46e84a27f83da2fe2edd1e73298b7d1
Author: wangfei <wangfei_he...@126.com>
Date:   2016-12-09T13:14:04Z

fix bugs with spark 2.1




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #243: [CARBONDATA-323] load data not suppo...

2016-12-05 Thread scwf 
Github user scwf closed the pull request at:

https://github.com/apache/incubator-carbondata/pull/243


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata issue #243: [CARBONDATA-323] load data not support loca...

2016-12-05 Thread scwf 
Github user scwf commented on the issue:

https://github.com/apache/incubator-carbondata/pull/243
  
i will close this, and in spark2 integration we do not support load local 
for carbon, in spark1 i think there is no problem


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata issue #393: [SPARK2] exclude dependency

2016-12-04 Thread scwf 
Github user scwf commented on the issue:

https://github.com/apache/incubator-carbondata/pull/393
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata issue #389: [CARBONDATA-491] spark2 integration: Do not...

2016-12-04 Thread scwf 
Github user scwf commented on the issue:

https://github.com/apache/incubator-carbondata/pull/389
  
@jackylk i fix the test suite, plz review again


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata issue #390: [CARBONDATA-492]fix a bug of profile spark-...

2016-12-03 Thread scwf 
Github user scwf commented on the issue:

https://github.com/apache/incubator-carbondata/pull/390
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #389: Do not use runnable command in spark...

2016-12-03 Thread scwf 
GitHub user scwf opened a pull request:

https://github.com/apache/incubator-carbondata/pull/389

Do not use runnable command in spark2

Runnable command is a inner interface in spark, it may changed from 
versions, we'd better not use it.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/KirinKing/incubator-carbondata runcommand

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-carbondata/pull/389.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #389


commit 59d8cd5fd90c9cb952a487a9ca4592c3ed5823ee
Author: wangfei <wangfei_he...@126.com>
Date:   2016-12-03T12:40:39Z

do not reply on runnable command




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #386: Fix spark2 decimal issue

2016-12-02 Thread scwf 
GitHub user scwf opened a pull request:

https://github.com/apache/incubator-carbondata/pull/386

Fix spark2 decimal issue

also added a test suite for decimal

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/KirinKing/incubator-carbondata fix-decimal

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-carbondata/pull/386.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #386


commit 4da98d3c54021d8744809df0e779fac2dbc34a5a
Author: wangfei <wangfei_he...@126.com>
Date:   2016-12-03T04:44:07Z

fix spark2 decimal

commit f351c8d6e586019d13a087dddaddb759127ab948
Author: wangfei <wangfei_he...@126.com>
Date:   2016-12-03T04:52:30Z

code clean




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #385: Fix spark2 decimal

2016-12-02 Thread scwf 
Github user scwf closed the pull request at:

https://github.com/apache/incubator-carbondata/pull/385


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #385: Fix spark2 decimal

2016-12-02 Thread scwf 
GitHub user scwf opened a pull request:

https://github.com/apache/incubator-carbondata/pull/385

Fix spark2 decimal

also added test suite for decimal

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/KirinKing/incubator-carbondata SJS

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-carbondata/pull/385.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #385


commit cf11617c2e8c4480873356ce2e2333a4bc8a180f
Author: wangfei <wangfei_he...@126.com>
Date:   2016-12-03T00:34:53Z

add sjs testsuite

commit c1587b87d8de2bc9a95da1bd35f2dc59aa6376a6
Author: wangfei <wangfei_he...@126.com>
Date:   2016-12-03T01:26:01Z

decimal => double and date => string

commit 76c8f66486dabf51ae5cf184846d991e6affce6d
Author: wangfei <wangfei_he...@126.com>
Date:   2016-12-03T01:59:13Z

use original datatype to create table

commit 7b27e416fa8575a7f0abb0e01689f9bf1b7a7da5
Author: wangfei <wangfei_he...@126.com>
Date:   2016-12-03T01:59:50Z

date => string

commit 4b40a64051b2dfa13e17f53e88915e92d8b04182
Author: wangfei <wangfei_he...@126.com>
Date:   2016-12-03T02:25:42Z

added clean store

commit 34537e8ff9681c84102e006c1132709b0912ea60
Author: wangfei <wangfei_he...@126.com>
Date:   2016-12-03T02:26:28Z

fix 4.2.7 cartession join

commit e3280cf6647ee958f35b8ce623d17cac5195bcd2
Author: wangfei <wangfei_he...@126.com>
Date:   2016-12-03T02:29:45Z

fix 4.2.4




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata issue #372: [CARBONDATA-473] integrate with spark 2 sta...

2016-11-30 Thread scwf 
Github user scwf commented on the issue:

https://github.com/apache/incubator-carbondata/pull/372
  
the result has some issues for the carbon example:
```
+--++---+---+---+
|shortField|intField|bigintField|doubleField|stringField|
+--++---+---+---+
| 1|  11|   null|   44.4|  flink|
| 3|  14|   null|   43.4|   hive|
| 2|  10|   null|   43.4| impala|
| 4|  16|   null|   42.4| impala|
| 1|  10|   null|   48.4|  spark|
| 5|  17|   null|   43.4|  spark|
| 1|  10|   null|   43.4|  spark|
| 1|  10|   null|   47.4|  spark|
| 1|  10|   null|   43.4|  spark|
| 1|  10|   null|   43.4|  spark|
+--++---+---+---+
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #372: [CARBONDATA-473] integrate with spar...

2016-11-30 Thread scwf 
Github user scwf commented on a diff in the pull request:

https://github.com/apache/incubator-carbondata/pull/372#discussion_r90207315
  
--- Diff: integration/spark-common/pom.xml ---
@@ -84,6 +79,11 @@
   junit
 
 
+  com.databricks
+  spark-csv_${scala.binary.version}
+  1.2.0
--- End diff --

we can extract this as a property


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #372: [CARBONDATA-473] integrate with spar...

2016-11-30 Thread scwf 
GitHub user scwf opened a pull request:

https://github.com/apache/incubator-carbondata/pull/372

[CARBONDATA-473] integrate with spark 2 stable datasource api

1.  read path: based on spark2 datasource api
2.  load path: give a tool to do data load


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/KirinKing/incubator-carbondata spark2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-carbondata/pull/372.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #372


commit a78efaea951194b1e9a3273b817d5e1641ed0f2c
Author: jackylk <jacky.li...@huawei.com>
Date:   2016-11-30T08:28:34Z

add spark2 module

commit 8d536ced3c1c147bb734c769a28cebbf33f2f035
Author: jackylk <jacky.li...@huawei.com>
Date:   2016-11-30T08:49:03Z

rebase

commit 308fe24500a81373dac7d87d5f2a563a35ff547d
Author: jackylk <jacky.li...@huawei.com>
Date:   2016-11-30T08:49:57Z

rebase

commit d3416dc128c0d4c37d9f8191796e596d9ca7a77b
Author: jackylk <jacky.li...@huawei.com>
Date:   2016-11-29T12:12:22Z

rebase

commit 120914467ee93a6613911fcf25171fa952eccd83
Author: jackylk <jacky.li...@huawei.com>
Date:   2016-11-30T08:50:32Z

rebase

commit d4cee19fbd54aa008662e36479b0ee0dda0ef384
Author: QiangCai <qiang...@qq.com>
Date:   2016-11-29T13:25:58Z

carbon api for spark2

commit bdfa20346d2ce432f925c02c47a9d2dd7dd5d807
Author: QiangCai <qiang...@qq.com>
Date:   2016-11-29T16:38:53Z

fix late decoder

commit 9a96466209a5918b2fc4f55a728055bc0093
Author: jackylk <jacky.li...@huawei.com>
Date:   2016-11-30T09:16:16Z

fix

commit 68f61c195a90b027e9d0dbc5a87915ca68d90c00
Author: Fei Wang <wangfei_he...@126.com>
Date:   2016-11-30T09:31:56Z

Merge pull request #26 from jackylk/kingmaster

Kingmaster




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata issue #365: [CARBONDATA-463] Extract code to spark-comm...

2016-11-28 Thread scwf 
Github user scwf commented on the issue:

https://github.com/apache/incubator-carbondata/pull/365
  
i think we should merge this into our spark2 branch, otherwise it will 
leads to large conflicts


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---