[GitHub] carbondata issue #1367: [CARBONDATA-1398] [WIP] Support query from specified...

[rahulforallp]

Github user rahulforallp commented on the issue:

https://github.com/apache/carbondata/pull/1367
  
@xuchuanyin  thanks. I will attach the doc as well.


---

[jira] [Updated] (CARBONDATA-1485) Alter add partition on dictionary_exclude timestamp column is not working

[dhatchayani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

dhatchayani updated CARBONDATA-1485:

Description: 
Alter add range partition on dictionary_exclude timestamp column is not as 
expected. On show partitions, it is showing the partition count. But the 
carbondata files are not written as expected.
(1) Limitation on timestamp dictionary_exclude column. Partition is working 
fine with dictionary_include timestamp column.
(2) Check other partition type also.
(3) Check all the alter functionalites like add, split.

  was:Alter add range partition on dictionary_exclude timestamp column is not 
as expected. On show partitions, it is showing the partition count. But the 
carbondata files are not written as expected.


> Alter add partition on dictionary_exclude timestamp column is not working
> -
>
> Key: CARBONDATA-1485
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1485
> Project: CarbonData
>  Issue Type: Bug
>Reporter: dhatchayani
>
> Alter add range partition on dictionary_exclude timestamp column is not as 
> expected. On show partitions, it is showing the partition count. But the 
> carbondata files are not written as expected.
> (1) Limitation on timestamp dictionary_exclude column. Partition is working 
> fine with dictionary_include timestamp column.
> (2) Check other partition type also.
> (3) Check all the alter functionalites like add, split.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] carbondata issue #1361: [CARBONDATA-1481] Compaction support global sort

[xubo245]

Github user xubo245 commented on the issue:

https://github.com/apache/carbondata/pull/1361
  
@QiangCai OK， it has been changed.


---

[GitHub] carbondata issue #1361: [CARBONDATA-1481] Compaction support global sort

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1361
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/858/



---

[GitHub] carbondata issue #1361: [CARBONDATA-1481] Compaction support global sort

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1361
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/227/



---

[GitHub] carbondata issue #1361: [CARBONDATA-1481] Compaction support global sort

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1361
  
Build Success with Spark 1.6, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/103/



---

[GitHub] carbondata issue #1361: [CARBONDATA-1481] Compaction support global sort

[QiangCai]

Github user QiangCai commented on the issue:

https://github.com/apache/carbondata/pull/1361
  
@xubo245  better to use the input files in 
integration/spark-common-test/src/test/resources/compaction


---

[GitHub] carbondata pull request #1364: [CARBONDATA-1483] Open auto merge when loadin...

[xubo245]

Github user xubo245 closed the pull request at:

https://github.com/apache/carbondata/pull/1364


---

[GitHub] carbondata issue #1364: [CARBONDATA-1483] Open auto merge when loading and i...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1364
  
Build Success with Spark 1.6, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/102/



---

[GitHub] carbondata issue #1364: [CARBONDATA-1483] Open auto merge when loading and i...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1364
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/226/



---

[GitHub] carbondata issue #1367: [CARBONDATA-1398] [WIP] Support query from specified...

[xuchuanyin]

Github user xuchuanyin commented on the issue:

https://github.com/apache/carbondata/pull/1367
  
:+1: I came across the same idea yesterday: query from the specified 
segment. Happy to see you have already worked on this.

Need to provide the related usage in document.


---

[jira] [Resolved] (CARBONDATA-1316) 15. alter table drop partition

[Liang Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liang Chen resolved CARBONDATA-1316.

   Resolution: Fixed
Fix Version/s: 1.2.0

> 15. alter table drop partition
> --
>
> Key: CARBONDATA-1316
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1316
> Project: CarbonData
>  Issue Type: Sub-task
>  Components: core, spark-integration, sql
>Reporter: Cao, Lionel
>Assignee: Cao, Lionel
> Fix For: 1.2.0
>
>  Time Spent: 5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (CARBONDATA-1450) Support timestamp more than 68 years, Enhance NoDictionary Datatypes - int , long

[Venkata Ramana G (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Venkata Ramana G resolved CARBONDATA-1450.
--
   Resolution: Fixed
Fix Version/s: 1.2.0

> Support timestamp more than 68 years, Enhance NoDictionary Datatypes - int , 
> long
> -
>
> Key: CARBONDATA-1450
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1450
> Project: CarbonData
>  Issue Type: New Feature
>Reporter: dhatchayani
>Assignee: dhatchayani
> Fix For: 1.2.0
>
>  Time Spent: 16.5h
>  Remaining Estimate: 0h
>
> Problem:
> Current implementation supports timestamp as direct dictionary only. As 
> dictionary is always integer only 68 years of range is supported.
> Solution:
> So this issue will support timestamp as default DICTIONARY_EXCLUDE. Allowing 
> to store internally as unix long timestamp.
> Problem:
> Int and Bigint(long) types are supported only as Dictionary include or 
> measure, they are not allowed in dictionary exclude.
> Solution:
> Support Int and Bigint(long) as Dictionary exclude and also Sort columns 
> support for int,long, bigint.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CARBONDATA-1450) Support timestamp more than 68 years, Enhance NoDictionary Datatypes - int , long

[Venkata Ramana G (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Venkata Ramana G updated CARBONDATA-1450:
-
Description: 
Problem:
Current implementation supports timestamp as direct dictionary only. As 
dictionary is always integer only 68 years of range is supported.
Solution:
So this issue will support timestamp as default DICTIONARY_EXCLUDE. Allowing to 
store internally as unix long timestamp.

Problem:
Int and Bigint(long) types are supported only as Dictionary include or measure, 
they are not allowed in dictionary exclude.
Solution:
Support Int and Bigint(long) as Dictionary exclude and also Sort columns 
support for int,long, bigint.

> Support timestamp more than 68 years, Enhance NoDictionary Datatypes - int , 
> long
> -
>
> Key: CARBONDATA-1450
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1450
> Project: CarbonData
>  Issue Type: New Feature
>Reporter: dhatchayani
>Assignee: dhatchayani
>  Time Spent: 16.5h
>  Remaining Estimate: 0h
>
> Problem:
> Current implementation supports timestamp as direct dictionary only. As 
> dictionary is always integer only 68 years of range is supported.
> Solution:
> So this issue will support timestamp as default DICTIONARY_EXCLUDE. Allowing 
> to store internally as unix long timestamp.
> Problem:
> Int and Bigint(long) types are supported only as Dictionary include or 
> measure, they are not allowed in dictionary exclude.
> Solution:
> Support Int and Bigint(long) as Dictionary exclude and also Sort columns 
> support for int,long, bigint.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1322
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/856/



---

[GitHub] carbondata pull request #1322: [CARBONDATA-1450] Support timestamp more than...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/carbondata/pull/1322


---

[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1322
  
Build Success with Spark 1.6, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/101/



---

[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1322
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/225/



---

[jira] [Updated] (CARBONDATA-1173) Streaming Ingest: Write path framework implementation

[Aniket Adnaik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aniket Adnaik updated CARBONDATA-1173:
--
Affects Version/s: (was: NONE)
   1.2.0

> Streaming Ingest: Write path framework implementation
> -
>
> Key: CARBONDATA-1173
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1173
> Project: CarbonData
>  Issue Type: Sub-task
>  Components: core, data-load, hadoop-integration, spark-integration
>Affects Versions: 1.2.0
>Reporter: Aniket Adnaik
>Assignee: Aniket Adnaik
> Fix For: 1.3.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Carbondata with Spark Structured streaming write path framework  
>  - Carbondata StreamingOutputWriter, StreamingRecordWriter, metadata writer  
>classes, etc
>  - initial framework for streaming ingest feature



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CARBONDATA-1173) Streaming Ingest: Write path framework implementation

[Aniket Adnaik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aniket Adnaik updated CARBONDATA-1173:
--
Affects Version/s: (was: 1.3.0)
   NONE

> Streaming Ingest: Write path framework implementation
> -
>
> Key: CARBONDATA-1173
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1173
> Project: CarbonData
>  Issue Type: Sub-task
>  Components: core, data-load, hadoop-integration, spark-integration
>Affects Versions: NONE
>Reporter: Aniket Adnaik
>Assignee: Aniket Adnaik
> Fix For: 1.3.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Carbondata with Spark Structured streaming write path framework  
>  - Carbondata StreamingOutputWriter, StreamingRecordWriter, metadata writer  
>classes, etc
>  - initial framework for streaming ingest feature



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CARBONDATA-1173) Streaming Ingest: Write path framework implementation

[Aniket Adnaik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aniket Adnaik updated CARBONDATA-1173:
--
Fix Version/s: (was: NONE)
   1.3.0

> Streaming Ingest: Write path framework implementation
> -
>
> Key: CARBONDATA-1173
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1173
> Project: CarbonData
>  Issue Type: Sub-task
>  Components: core, data-load, hadoop-integration, spark-integration
>Affects Versions: NONE
>Reporter: Aniket Adnaik
>Assignee: Aniket Adnaik
> Fix For: 1.3.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Carbondata with Spark Structured streaming write path framework  
>  - Carbondata StreamingOutputWriter, StreamingRecordWriter, metadata writer  
>classes, etc
>  - initial framework for streaming ingest feature



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CARBONDATA-1173) Streaming Ingest: Write path framework implementation

[Aniket Adnaik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aniket Adnaik updated CARBONDATA-1173:
--
Affects Version/s: 1.3.0

> Streaming Ingest: Write path framework implementation
> -
>
> Key: CARBONDATA-1173
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1173
> Project: CarbonData
>  Issue Type: Sub-task
>  Components: core, data-load, hadoop-integration, spark-integration
>Affects Versions: NONE
>Reporter: Aniket Adnaik
>Assignee: Aniket Adnaik
> Fix For: 1.3.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Carbondata with Spark Structured streaming write path framework  
>  - Carbondata StreamingOutputWriter, StreamingRecordWriter, metadata writer  
>classes, etc
>  - initial framework for streaming ingest feature



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...

[dhatchayani]

Github user dhatchayani commented on the issue:

https://github.com/apache/carbondata/pull/1322
  
retest this please


---

[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1322
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/855/



---

[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1322
  
Build Success with Spark 1.6, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/100/



---

[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1322
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/224/



---

[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...

[dhatchayani]

Github user dhatchayani commented on the issue:

https://github.com/apache/carbondata/pull/1322
  
retest this please


---

[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1322
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/854/



---

[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1322
  
Build Success with Spark 1.6, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/99/



---

[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1322
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/223/



---

[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...

[dhatchayani]

Github user dhatchayani commented on the issue:

https://github.com/apache/carbondata/pull/1322
  
retest this please


---

[GitHub] carbondata issue #1367: [CARBONDATA-1398] [WIP] Support query from specified...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1367
  
Build Failed with Spark 1.6, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/98/



---

[GitHub] carbondata issue #1367: [CARBONDATA-1398] [WIP] Support query from specified...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1367
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/222/



---

[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1322
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/852/



---

[GitHub] carbondata pull request #1367: [CARBONDATA-1398] [WIP] Support query from sp...

[rahulforallp]

GitHub user rahulforallp opened a pull request:

https://github.com/apache/carbondata/pull/1367

[CARBONDATA-1398] [WIP] Support query from specified segments



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rahulforallp/incubator-carbondata 
CARBONDATA-1398

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1367.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1367


commit c747c8fe57061aa12540825fa8533a81c6fabb68
Author: rahulforallp 
Date:   2017-09-14T13:14:09Z

[CARBONDATA-1398] support query from specified segments




---

[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1322
  
Build Success with Spark 1.6, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/97/



---

[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1322
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/221/



---

[GitHub] carbondata pull request #1366: [Release]Add dependency of spark-sql-2.1 to p...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/carbondata/pull/1366


---

[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1322
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/851/



---

[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...

[dhatchayani]

Github user dhatchayani commented on the issue:

https://github.com/apache/carbondata/pull/1322
  
retest this please


---

[GitHub] carbondata issue #1366: [Release]Add dependency of spark-sql-2.1 to presto a...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1366
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/850/



---

[GitHub] carbondata issue #1366: [Release]Add dependency of spark-sql-2.1 to presto a...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1366
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/219/



---

[GitHub] carbondata issue #1366: [Release]Add dependency of spark-sql-2.1 to presto a...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1366
  
Build Success with Spark 1.6, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/94/



---

[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1322
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/218/



---

[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1322
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/849/



---

[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1322
  
Build Failed with Spark 1.6, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/93/



---

[GitHub] carbondata issue #985: [CARBONDATA-1090] added integration test cases for al...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/985
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/848/



---

[GitHub] carbondata issue #985: [CARBONDATA-1090] added integration test cases for al...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/985
  
Build Failed with Spark 1.6, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/92/



---

[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1322
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/216/



---

[GitHub] carbondata pull request #1366: [Release]Add dependency of spark-sql-2.1 to p...

[ravipesala]

GitHub user ravipesala opened a pull request:

https://github.com/apache/carbondata/pull/1366

[Release]Add dependency of spark-sql-2.1 to presto as it fails when 
building using spark-1.6 version.

Spark 2.1 dependency got added to presto in the PR 
https://github.com/apache/carbondata/pull/1307 .
So when using build-all profile using 1.6 profile to build carbon, presto 
fails as it only depends on spark 2.1 package.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ravipesala/incubator-carbondata 
presto-build-all

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1366.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1366


commit 5aea2b331144af4e4cd0833ecdb5f66aa4ac9b34
Author: Ravindra Pesala 
Date:   2017-09-18T11:36:21Z

Add dependency of spark-sql-2.1 to presto as it fails when building using 
spark-1.6 version.




---

[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1322
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/847/



---

[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1322
  
Build Success with Spark 1.6, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/91/



---

[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1322
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/846/



---

[GitHub] carbondata pull request #1322: [CARBONDATA-1450] Support timestamp more than...

[dhatchayani]

Github user dhatchayani commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1322#discussion_r139395991
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/scan/executor/util/RestructureUtil.java
 ---
@@ -211,10 +213,21 @@ private static Object 
getDirectDictionaryDefaultValue(DataType dataType, byte[]
* @param defaultValue
* @return
*/
-  private static Object getNoDictionaryDefaultValue(byte[] defaultValue) {
+  private static Object getNoDictionaryDefaultValue(DataType datatype, 
byte[] defaultValue) {
 Object noDictionaryDefaultValue = null;
 if (!isDefaultValueNull(defaultValue)) {
-  noDictionaryDefaultValue = UTF8String.fromBytes(defaultValue);
+  switch (datatype) {
+case INT:
+  noDictionaryDefaultValue = ByteUtil.toInt(defaultValue, 0, 
defaultValue.length);
+  break;
+case LONG:
+case TIMESTAMP:
+  long timestampValue = ByteUtil.toLong(defaultValue, 0, 
defaultValue.length);
--- End diff --

handled


---

[GitHub] carbondata pull request #1322: [CARBONDATA-1450] Support timestamp more than...

[dhatchayani]

Github user dhatchayani commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1322#discussion_r139395822
  
--- Diff: 
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/sortcolumns/TestSortColumns.scala
 ---
@@ -31,6 +31,42 @@ class TestSortColumns extends QueryTest with 
BeforeAndAfterAll {
 sql(s"""LOAD DATA local inpath '$resourcesPath/data.csv' INTO TABLE 
origintable1 OPTIONS('DELIMITER'= ',', 'QUOTECHAR'= '\"')""")
   }
 
+  test("create table sort columns dictionary include - int") {
--- End diff --

added


---

[GitHub] carbondata issue #985: [CARBONDATA-1090] added integration test cases for al...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/985
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/845/



---

[GitHub] carbondata issue #985: [CARBONDATA-1090] added integration test cases for al...

[anubhav100]

Github user anubhav100 commented on the issue:

https://github.com/apache/carbondata/pull/985
  
retest this please


---

[GitHub] carbondata issue #985: [CARBONDATA-1090] added integration test cases for al...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/985
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/214/



---

[GitHub] carbondata issue #985: [CARBONDATA-1090] added integration test cases for al...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/985
  
Build Success with Spark 1.6, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/90/



---

[GitHub] carbondata pull request #1322: [CARBONDATA-1450] Support timestamp more than...

[gvramana]

Github user gvramana commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1322#discussion_r139389587
  
--- Diff: 
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/sortcolumns/TestSortColumns.scala
 ---
@@ -31,6 +31,42 @@ class TestSortColumns extends QueryTest with 
BeforeAndAfterAll {
 sql(s"""LOAD DATA local inpath '$resourcesPath/data.csv' INTO TABLE 
origintable1 OPTIONS('DELIMITER'= ',', 'QUOTECHAR'= '\"')""")
   }
 
+  test("create table sort columns dictionary include - int") {
--- End diff --

Add test cases for Int and Long no-Dictionary cases


---

[GitHub] carbondata issue #985: [CARBONDATA-1090] added integration test cases for al...

[anubhav100]

Github user anubhav100 commented on the issue:

https://github.com/apache/carbondata/pull/985
  
retest this please


---

[GitHub] carbondata pull request #1322: [CARBONDATA-1450] Support timestamp more than...

[gvramana]

Github user gvramana commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1322#discussion_r139386365
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/scan/executor/util/RestructureUtil.java
 ---
@@ -211,10 +213,21 @@ private static Object 
getDirectDictionaryDefaultValue(DataType dataType, byte[]
* @param defaultValue
* @return
*/
-  private static Object getNoDictionaryDefaultValue(byte[] defaultValue) {
+  private static Object getNoDictionaryDefaultValue(DataType datatype, 
byte[] defaultValue) {
 Object noDictionaryDefaultValue = null;
 if (!isDefaultValueNull(defaultValue)) {
-  noDictionaryDefaultValue = UTF8String.fromBytes(defaultValue);
+  switch (datatype) {
+case INT:
+  noDictionaryDefaultValue = ByteUtil.toInt(defaultValue, 0, 
defaultValue.length);
+  break;
+case LONG:
+case TIMESTAMP:
+  long timestampValue = ByteUtil.toLong(defaultValue, 0, 
defaultValue.length);
--- End diff --

Here Long case to be separately handled.


---

[GitHub] carbondata issue #1365: [WIP] move Load command to seperate file

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1365
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/844/



---

[GitHub] carbondata issue #1365: [WIP] move Load command to seperate file

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1365
  
Build Success with Spark 1.6, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/89/



---

[jira] [Updated] (CARBONDATA-1484) carbonthriftserver cache is not refreshed

[dylan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

dylan updated CARBONDATA-1484:
--
External issue URL: 
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/Fw-carbonthriftserver-can-not-be-load-many-times-td21823.html

> carbonthriftserver cache is not refreshed
> -
>
> Key: CARBONDATA-1484
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1484
> Project: CarbonData
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.1.0
> Environment: spark:1.6.0,mysql hive metastore
>Reporter: dylan
> Fix For: NONE
>
>
>   i am use carbondata version is 1.1.0 and spark version is 1.6.0. 
> and I reproduce in accordance with the official quick-start-guide case 
> again, 
> 1.Creating a Table 
> cc.sql("create table IF NOT EXISTS  carbondb.test_table(id string,name 
> String,city String,age int) stored by 'carbondata' ") 
> 2.load data into table 
>   cc.sql("load data inpath 'hdfs://nameservice1/user/zz/sample.csv' into 
> table carbondb.test_table") 
> 3.start carbonthriftserver 
>/home/zz/spark-1.6.0-bin-hadoop2.6/bin/spark-submit  --master local[*] 
> --driver-java-options="-Dcarbon.properties.filepath=/home/zz/spark-1.6.0-bin-hadoop2.6/conf/carbon.properties"
>  
> --executor-memory 4G  --driver-memory 2g  --conf 
> spark.serializer=org.apache.spark.serializer.KryoSerializer   --conf 
> "spark.sql.shuffle.partitions=3" --conf spark.speculation=true   --class 
> org.apache.carbondata.spark.thriftserver.CarbonThriftServer 
> /home/zz/spark-1.6.0-bin-hadoop2.6/carbonlib/carbondata_2.10-1.1.0-shade-hadoop2.2.0.jar
>  
> hdfs://nameservice1/user/zz/rp_carbon_store 
>4.Connecting to CarbonData Thrift Server Using Beeline. 
>   
>5.drop table 
>cc.sql("drop table carbondb.test_table") 
>6.recreate table and load data 
> cc.sql("create table IF NOT EXISTS  carbondb.test_table(id string,name 
> String,city String,age int) stored by 'carbondata' ") 
> cc.sql("load data inpath 'hdfs://nameservice1/user/zz/sample.csv' into 
> table carbondb.test_table") 
>7.select data use beeline 
>  
>Like the above error, the cache is not updated 
>  in my case,1,2,5 and 6 step is one session on spark-shell ,4 and 7 is 
> one session on beeline, 
> and i test this case on the current carbondata master branch(1.2.0), 
>  when i use beeline there is no Btree load failed info,but in my table there 
> is no data,All the data is null,but in spark-shell is ok. 
> spark-shell: 
> +---+-++---+ 
> | id| name|city|age| 
> +---+-++---+ 
> |  1|david|shenzhen| 31| 
> |  2|eason|shenzhen| 27| 
> |  3|jarry|   wuhan| 35| 
> +---+-++---+ 
> beeline: 
> 0: jdbc:hive2://localhost:1> select * from  carbondb.test_table; 
> +---+---+---+---+--+ 
> |  id   | name  | city  |  age  | 
> +---+---+---+---+--+ 
> | NULL  | NULL  | NULL  | NULL  | 
> | NULL  | NULL  | NULL  | NULL  | 
> | NULL  | NULL  | NULL  | NULL  | 
> +---+---+---+---+--+ 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (CARBONDATA-1484) carbonthriftserver cache is not refreshed

[dylan (JIRA)]

dylan created CARBONDATA-1484:
-

 Summary: carbonthriftserver cache is not refreshed
 Key: CARBONDATA-1484
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1484
 Project: CarbonData
  Issue Type: Bug
  Components: core
Affects Versions: 1.1.0
 Environment: spark:1.6.0,mysql hive metastore
Reporter: dylan
 Fix For: NONE


  i am use carbondata version is 1.1.0 and spark version is 1.6.0. 
and I reproduce in accordance with the official quick-start-guide case 
again, 
1.Creating a Table 
cc.sql("create table IF NOT EXISTS  carbondb.test_table(id string,name 
String,city String,age int) stored by 'carbondata' ") 

2.load data into table 
  cc.sql("load data inpath 'hdfs://nameservice1/user/zz/sample.csv' into 
table carbondb.test_table") 

3.start carbonthriftserver 
   /home/zz/spark-1.6.0-bin-hadoop2.6/bin/spark-submit  --master local[*] 
--driver-java-options="-Dcarbon.properties.filepath=/home/zz/spark-1.6.0-bin-hadoop2.6/conf/carbon.properties"
 
--executor-memory 4G  --driver-memory 2g  --conf 
spark.serializer=org.apache.spark.serializer.KryoSerializer   --conf 
"spark.sql.shuffle.partitions=3" --conf spark.speculation=true   --class 
org.apache.carbondata.spark.thriftserver.CarbonThriftServer 
/home/zz/spark-1.6.0-bin-hadoop2.6/carbonlib/carbondata_2.10-1.1.0-shade-hadoop2.2.0.jar
 
hdfs://nameservice1/user/zz/rp_carbon_store 

   4.Connecting to CarbonData Thrift Server Using Beeline. 

  

   5.drop table 
   cc.sql("drop table carbondb.test_table") 

   6.recreate table and load data 
cc.sql("create table IF NOT EXISTS  carbondb.test_table(id string,name 
String,city String,age int) stored by 'carbondata' ") 
cc.sql("load data inpath 'hdfs://nameservice1/user/zz/sample.csv' into 
table carbondb.test_table") 

   7.select data use beeline 
 
   Like the above error, the cache is not updated 

 in my case,1,2,5 and 6 step is one session on spark-shell ,4 and 7 is 
one session on beeline, 

and i test this case on the current carbondata master branch(1.2.0), 

 when i use beeline there is no Btree load failed info,but in my table there 
is no data,All the data is null,but in spark-shell is ok. 
spark-shell: 
+---+-++---+ 
| id| name|city|age| 
+---+-++---+ 
|  1|david|shenzhen| 31| 
|  2|eason|shenzhen| 27| 
|  3|jarry|   wuhan| 35| 
+---+-++---+ 

beeline: 
0: jdbc:hive2://localhost:1> select * from  carbondb.test_table; 
+---+---+---+---+--+ 
|  id   | name  | city  |  age  | 
+---+---+---+---+--+ 
| NULL  | NULL  | NULL  | NULL  | 
| NULL  | NULL  | NULL  | NULL  | 
| NULL  | NULL  | NULL  | NULL  | 
+---+---+---+---+--+ 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] carbondata issue #1365: [WIP] move Load command to seperate file

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1365
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/213/



---

[GitHub] carbondata issue #1365: [WIP] move Load command to seperate file

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1365
  
Build Success with Spark 1.6, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/88/



---

[GitHub] carbondata issue #1365: [WIP] move Load command to seperate file

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1365
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/212/



---

[GitHub] carbondata issue #1365: [WIP] move Load command to seperate file

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1365
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/843/



---

[GitHub] carbondata pull request #1365: [WIP] move Load command to seperate file

[kunal642]

GitHub user kunal642 opened a pull request:

https://github.com/apache/carbondata/pull/1365

[WIP] move Load command to seperate file



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kunal642/carbondata dataload_refactor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1365.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1365


commit b886fd0b14271a6303e0213f8fb4576d9e767a47
Author: kunal642 
Date:   2017-09-15T05:25:21Z

move Load command to seperate file




---

[GitHub] carbondata issue #1364: [CARBONDATA-1483] Open auto merge when loading and i...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1364
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/842/



---

[GitHub] carbondata pull request #1317: [CARBONDATA-1316] drop partition

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/carbondata/pull/1317


---

[jira] [Resolved] (CARBONDATA-1478) Update compaction documentation

[Ravindra Pesala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala resolved CARBONDATA-1478.
-
Resolution: Fixed

> Update compaction documentation
> ---
>
> Key: CARBONDATA-1478
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1478
> Project: CarbonData
>  Issue Type: Improvement
>  Components: docs
>Affects Versions: 1.2.0
>Reporter: xubo245
>Assignee: xubo245
> Fix For: 1.2.0
>
>   Original Estimate: 240h
>  Time Spent: 1.5h
>  Remaining Estimate: 238.5h
>
> Some examples about compaction in documentation should be updated.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] carbondata pull request #1357: [CARBONDATA-1478] Update compaction documenta...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/carbondata/pull/1357


---

[jira] [Updated] (CARBONDATA-889) Optimize pom dependency with exclusion to remove unnecessary dependency jar

[Ravindra Pesala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala updated CARBONDATA-889:
---
Fix Version/s: (was: 1.2.0)
   1.3.0

> Optimize pom dependency with exclusion to remove unnecessary dependency jar 
> 
>
> Key: CARBONDATA-889
> URL: https://issues.apache.org/jira/browse/CARBONDATA-889
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: Liang Chen
>Assignee: Ravindra Pesala
> Fix For: 1.3.0
>
>
> For example: 
> The below's spark dependency will introduce around 90 dependency jar, but 
> some of them are unnecessary jar for CarbonData.
> 
>   org.apache.spark
>   spark-sql_${scala.binary.version}
> 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (CARBONDATA-882) Add SORT_COLUMNS option support in dataframe writer

[Ravindra Pesala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala resolved CARBONDATA-882.

Resolution: Fixed
  Assignee: Jacky Li

> Add SORT_COLUMNS option support in dataframe writer
> ---
>
> Key: CARBONDATA-882
> URL: https://issues.apache.org/jira/browse/CARBONDATA-882
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: Jacky Li
>Assignee: Jacky Li
> Fix For: 1.2.0
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> User can should be able to specify SORT_COLUMNS option when using 
> dataframe.write



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (CARBONDATA-888) Dictionary include / exclude option in dataframe writer

[Ravindra Pesala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala resolved CARBONDATA-888.

Resolution: Fixed

> Dictionary include / exclude option in dataframe writer
> ---
>
> Key: CARBONDATA-888
> URL: https://issues.apache.org/jira/browse/CARBONDATA-888
> Project: CarbonData
>  Issue Type: Improvement
>  Components: spark-integration
>Affects Versions: 1.2.0
> Environment: HDP 2.5, Spark 1.6
>Reporter: Sanoj MG
>Assignee: Sanoj MG
>Priority: Minor
> Fix For: 1.2.0
>
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> While creating a Carbondata table from dataframe, currently it is not 
> possible to specify columns that needs to be included in or excluded from the 
> dictionary. An option is required to specify it as below : 
> df.write.format("carbondata")
>   .option("tableName", "test")
>   .option("compress","true")
>   .option("dictionary_include","incol1,intcol2")
>   .option("dictionary_exclude","stringcol1,stringcol2")
>   .mode(SaveMode.Overwrite)
> .save()
> We have lot of integer columns that are dimensions, dataframe.save is used to 
> quickly create tables instead of writing ddls, and it would be nice to have 
> this feature to execute POCs.  
>  
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (CARBONDATA-1046) Single_pass_loading is throwing an error in Spark1.6 in automation

[Ravindra Pesala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala resolved CARBONDATA-1046.
-
   Resolution: Cannot Reproduce
Fix Version/s: (was: 1.2.0)
   NONE

> Single_pass_loading is throwing an error in Spark1.6 in automation
> --
>
> Key: CARBONDATA-1046
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1046
> Project: CarbonData
>  Issue Type: Bug
> Environment: Spark1.6
>Reporter: SWATI RAO
>Assignee: anubhav tarar
> Fix For: NONE
>
> Attachments: 7000_UniqData.csv
>
>
> Steps to Reproduce :
> Create Table :
> CREATE TABLE uniqdata_INCLUDEDICTIONARY (CUST_ID int,CUST_NAME 
> String,ACTIVE_EMUI_VERSION string, DOB timestamp, DOJ timestamp, 
> BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 bigint,DECIMAL_COLUMN1 decimal(30,10), 
> DECIMAL_COLUMN2 decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 
> double,INTEGER_COLUMN1 int) STORED BY 'org.apache.carbondata.format' 
> TBLPROPERTIES('DICTIONARY_INCLUDE'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (1.709 seconds)
> Load Query :
> LOAD DATA INPATH 
> 'hdfs://hadoop-master:54310/BabuStore/Data/uniqdata/7000_UniqData.csv' into 
> table uniqdata_INCLUDEDICTIONARY OPTIONS('DELIMITER'=',' , 
> 'QUOTECHAR'='"','BAD_RECORDS_LOGGER_ENABLE'='TRUE', 
> 'BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','SINGLE_Pass'='true');
> Stack Trace :
> INFO  11-05 13:54:45,047 - Running query 'LOAD DATA INPATH 
> 'hdfs://hadoop-master:54310/BabuStore/Data/uniqdata/7000_UniqData.csv' into 
> table uniqdata_INCLUDEDICTIONARY OPTIONS('DELIMITER'=',' , 
> 'QUOTECHAR'='"','BAD_RECORDS_LOGGER_ENABLE'='TRUE', 
> 'BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','SINGLE_Pass'='true')'
>  with 44e92bcb-f9e1-4b2e-835e-a82eae525fe4
> INFO  11-05 13:54:45,047 - pool-31-thread-3 Query [LOAD DATA INPATH 
> 'HDFS://HADOOP-MASTER:54310/BABUSTORE/DATA/UNIQDATA/7000_UNIQDATA.CSV' INTO 
> TABLE UNIQDATA_INCLUDEDICTIONARY OPTIONS('DELIMITER'=',' , 
> 'QUOTECHAR'='"','BAD_RECORDS_LOGGER_ENABLE'='TRUE', 
> 'BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,DOUBLE_COLUMN1,DOUBLE_COLUMN2,INTEGER_COLUMN1','SINGLE_PASS'='TRUE')]
> INFO  11-05 13:54:45,065 - pool-31-thread-3 HDFS lock 
> path:hdfs://192.168.2.145:54310/opt/olapcontent/default/uniqdata_includedictionary/meta.lock
> INFO  11-05 13:54:45,097 - Successfully able to get the table metadata file 
> lock
> INFO  11-05 13:54:45,099 - pool-31-thread-3 Initiating Direct Load for the 
> Table : (default.uniqdata_includedictionary)
> AUDIT 11-05 13:54:45,100 - [hadoop-master][hduser][Thread-150]Data load 
> request has been received for table default.uniqdata_includedictionary
> AUDIT 11-05 13:54:45,100 - [hadoop-master][hduser][Thread-150]Data is loading 
> with New Data Flow for table default.uniqdata_includedictionary
> ERROR 11-05 13:54:45,104 - Dictionary server Dictionary Server Start Failed
> java.net.BindException: Address already in use
>   at sun.nio.ch.Net.bind0(Native Method)
>   at sun.nio.ch.Net.bind(Net.java:433)
>   at sun.nio.ch.Net.bind(Net.java:425)
>   at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
>   at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
>   at 
> io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:125)
>   at 
> io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:485)
>   at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1089)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:430)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:415)
>   at 
> io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:903)
>   at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:198)
>   at 
> io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:348)
>   at 
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
>   at 

[jira] [Updated] (CARBONDATA-1014) Refactor on data loading and encoding override

[Ravindra Pesala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala updated CARBONDATA-1014:

Fix Version/s: (was: 1.2.0)
   1.3.0

> Refactor on data loading and encoding override
> --
>
> Key: CARBONDATA-1014
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1014
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: Jacky Li
> Fix For: 1.3.0
>
>
> Refactor on current data loading flow to make it:
> 1. Use vectorized processing as early as possible
> 2. Make index build (sorting) CPU cache efficient, by using rowId and key 
> column vector to sort
> 3. Open interface for format extension, including column encoding, 
> compression, statistics.
> Design doc will be posted in this JIRA soon.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CARBONDATA-1016) Make sort step output ColumnPage

[Ravindra Pesala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala updated CARBONDATA-1016:

Fix Version/s: (was: 1.2.0)
   1.3.0

> Make sort step output ColumnPage
> 
>
> Key: CARBONDATA-1016
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1016
> Project: CarbonData
>  Issue Type: Sub-task
>Reporter: Jacky Li
> Fix For: 1.3.0
>
>
> Currently in UnsafeInMemoryIntermediateDataMerger, it stores data using 
> UnsafeCarbonRowPage[] unsafeCarbonRowPages. This will be more efficient if we 
> change it to use ColumnPage which is the main data structure used in write 
> step.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (CARBONDATA-1065) Implement set command in carbon to update carbon properties dynamically

[Ravindra Pesala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala resolved CARBONDATA-1065.
-
Resolution: Fixed

> Implement set command in carbon to update carbon properties dynamically
> ---
>
> Key: CARBONDATA-1065
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1065
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: Ravindra Pesala
>Assignee: Manohar Vanam
> Fix For: 1.2.0
>
>  Time Spent: 13h 50m
>  Remaining Estimate: 0h
>
> Currently there is no way to set the carbon properties dynamically, always 
> need to update the carbon.properties file and restart the driver.
> So it will be easy for users to give option to add/update the properties 
> dynamically through set command.
> The below sql command updates the carbon properties in driver and as well as 
> in executor
> {code}
> set key1=value1
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (CARBONDATA-1094) Wrong results returned by the query in case inverted index is not created on a column

[Ravindra Pesala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala resolved CARBONDATA-1094.
-
Resolution: Fixed

> Wrong results returned by the query in case inverted index is not created on 
> a column
> -
>
> Key: CARBONDATA-1094
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1094
> Project: CarbonData
>  Issue Type: Bug
>Reporter: Manish Gupta
>Assignee: Manish Gupta
> Fix For: 1.2.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> While creating a table if a column is not specified as sort column or 
> included as no inverted index then the column will not be sorted during data 
> load. Unsorted data will have incorrect min/max values and inverted index 
> will not be created for that column.
> During query, if filter exists for that column it gives incorrect results as 
> binary search cannot be applied on the unsorted data.
> Commands to reproduce
> -
> CREATE TABLE IF NOT EXISTS index1 (id Int, name String, city String) STORED 
> BY 'org.apache.carbondata.format' 
> TBLPROPERTIES('NO_INVERTED_INDEX'='name,city', 'DICTIONARY_EXCLUDE'='city')
> LOAD DATA LOCAL INPATH '' into table index1
> SELECT * FROM index1 WHERE city >= 'Shanghai'
> +---+--+--+
> | id|  name|  city|
> +---+--+--+
> | 11| James|Washington|
> |  5|  John|   Beijing|
> | 20| Kevin| Singapore|
> | 17|  Lisa|  Hangzhou|
> | 12| Maria|Berlin|
> |  2|  Mark| Paris|
> |  9|  Mary| Tokyo|
> |  6|Michel|   Chicago|
> | 16|  Paul|  Shanghai|
> | 14| Peter|Boston|
> |  7|Robert|   Houston|
> |  4|  Sara| Tokyo|
> |  8| Sunny|Boston|
> +---+--+--+



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (CARBONDATA-1104) Query failure while using unsafe for query execution numeric data type column specified as sort column

[Ravindra Pesala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala resolved CARBONDATA-1104.
-
Resolution: Fixed

> Query failure while using unsafe for query execution numeric data type column 
> specified as sort column
> --
>
> Key: CARBONDATA-1104
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1104
> Project: CarbonData
>  Issue Type: Bug
>Reporter: Manish Gupta
>Assignee: Manish Gupta
> Fix For: 1.2.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Steps to reproduce
> -
> 1. Set the parameter enable.unsafe.in.query.processing = true
> 2. CREATE TABLE sorttable1 (empno int, empname String, designation String, 
> doj Timestamp, workgroupcategory int, workgroupcategoryname String, deptno 
> int, deptname String, projectcode int, projectjoindate Timestamp, 
> projectenddate Timestamp,attendance int,utilization int,salary int) STORED BY 
> 'org.apache.carbondata.format' tblproperties('sort_columns'='empno')
> 3. LOAD DATA local inpath '' INTO TABLE sorttable1 
> OPTIONS('DELIMITER'= ',', 'QUOTECHAR'= '"')
> 4. select empno from sorttable1
> Exception thrown
> 
> 17/05/29 08:43:20 ERROR Executor: Exception in task 0.0 in stage 6.0 (TID 12)
> java.lang.NullPointerException
>   at 
> org.apache.spark.sql.execution.vectorized.ColumnVector.arrayData(ColumnVector.java:858)
>   at 
> org.apache.spark.sql.execution.vectorized.OffHeapColumnVector.putByteArray(OffHeapColumnVector.java:421)
>   at 
> org.apache.spark.sql.execution.vectorized.ColumnVector.putByteArray(ColumnVector.java:569)
>   at 
> org.apache.carbondata.spark.vectorreader.ColumnarVectorWrapper.putBytes(ColumnarVectorWrapper.java:85)
>   at 
> org.apache.carbondata.core.datastore.chunk.store.impl.unsafe.UnsafeVariableLengthDimesionDataChunkStore.fillRow(UnsafeVariableLengthDimesionDataChunkStore.java:167)
>   at 
> org.apache.carbondata.core.datastore.chunk.impl.VariableLengthDimensionDataChunk.fillConvertedChunkData(VariableLengthDimensionDataChunk.java:112)
>   at 
> org.apache.carbondata.core.scan.result.AbstractScannedResult.fillColumnarNoDictionaryBatch(AbstractScannedResult.java:228)
>   at 
> org.apache.carbondata.core.scan.collector.impl.DictionaryBasedVectorResultCollector.scanAndFillResult(DictionaryBasedVectorResultCollector.java:154)
>   at 
> org.apache.carbondata.core.scan.collector.impl.DictionaryBasedVectorResultCollector.collectVectorBatch(DictionaryBasedVectorResultCollector.java:147)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CARBONDATA-1115) load csv data fail

[Ravindra Pesala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala updated CARBONDATA-1115:

Fix Version/s: (was: 1.2.0)
   1.3.0

> load csv data fail
> --
>
> Key: CARBONDATA-1115
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1115
> Project: CarbonData
>  Issue Type: Bug
>  Components: examples
>Affects Versions: 1.2.0
> Environment: centos 7, spark2.1.0, hadoop 2.7
>Reporter: hyd
> Fix For: 1.3.0
>
>
> is it a bug, or my environment has problem, can anyone help me.
> [root@localhost spark-2.1.0-bin-hadoop2.7]# ls /home/carbondata/sample.csv 
> /home/carbondata/sample.csv
> [root@localhost spark-2.1.0-bin-hadoop2.7]# ./bin/spark-shell --master 
> spark://192.168.32.114:7077 --total-executor-cores 2 --executor-memory 2G
> Using Spark's default log4j profile: 
> org/apache/spark/log4j-defaults.properties
> Setting default log level to "WARN".
> To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use 
> setLogLevel(newLevel).
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/opt/spark-2.1.0-bin-hadoop2.7/carbonlib/carbondata_2.11-1.1.0-shade-hadoop2.2.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/opt/spark-2.1.0-bin-hadoop2.7/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 17/06/01 14:44:54 WARN NativeCodeLoader: Unable to load native-hadoop library 
> for your platform... using builtin-java classes where applicable
> 17/06/01 14:44:54 WARN SparkConf: 
> SPARK_CLASSPATH was detected (set to './carbonlib/*').
> This is deprecated in Spark 1.0+.
> Please instead use:
>  - ./spark-submit with --driver-class-path to augment the driver classpath
>  - spark.executor.extraClassPath to augment the executor classpath
> 
> 17/06/01 14:44:54 WARN SparkConf: Setting 'spark.executor.extraClassPath' to 
> './carbonlib/*' as a work-around.
> 17/06/01 14:44:54 WARN SparkConf: Setting 'spark.driver.extraClassPath' to 
> './carbonlib/*' as a work-around.
> 17/06/01 14:44:54 WARN Utils: Your hostname, localhost.localdomain resolves 
> to a loopback address: 127.0.0.1; using 192.168.32.114 instead (on interface 
> em1)
> 17/06/01 14:44:54 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to 
> another address
> 17/06/01 14:44:59 WARN ObjectStore: Failed to get database global_temp, 
> returning NoSuchObjectException
> Spark context Web UI available at http://192.168.32.114:4040
> Spark context available as 'sc' (master = spark://192.168.32.114:7077, app id 
> = app-20170601144454-0001).
> Spark session available as 'spark'.
> Welcome to
>     __
>  / __/__  ___ _/ /__
> _\ \/ _ \/ _ `/ __/  '_/
>/___/ .__/\_,_/_/ /_/\_\   version 2.1.0
>   /_/
>  
> Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_121)
> Type in expressions to have them evaluated.
> Type :help for more information.
> scala> import org.apache.spark.sql.SparkSession
> import org.apache.spark.sql.SparkSession
> scala> import org.apache.spark.sql.CarbonSession._
> import org.apache.spark.sql.CarbonSession._
> scala> val carbon = 
> SparkSession.builder().config(sc.getConf).getOrCreateCarbonSession("hdfs://192.168.32.114/test")
> 17/06/01 14:45:35 WARN SparkContext: Using an existing SparkContext; some 
> configuration may not take effect.
> 17/06/01 14:45:38 WARN ObjectStore: Failed to get database global_temp, 
> returning NoSuchObjectException
> carbon: org.apache.spark.sql.SparkSession = 
> org.apache.spark.sql.CarbonSession@2165b170
> scala> carbon.sql("CREATE TABLE IF NOT EXISTS test_table(id string, name 
> string, city string, age Int) STORED BY 'carbondata'")
> 17/06/01 14:45:45 AUDIT CreateTable: 
> [localhost.localdomain][root][Thread-1]Creating Table with Database name 
> [default] and Table name [test_table]
> res0: org.apache.spark.sql.DataFrame = []
> scala> carbon.sql("LOAD DATA LOCAL INPATH '/home/carbondata/sample.csv' INTO 
> TABLE test_table")
> 17/06/01 14:45:54 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, 
> 192.168.32.114, executor 0): java.lang.ClassCastException: cannot assign 
> instance of scala.collection.immutable.List$SerializationProxy to field 
> org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type 
> scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD
>   at 
> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2133)
>   at 
> java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1305)
>   at 
> 

[jira] [Resolved] (CARBONDATA-1123) Rename interface and variable for RLE encoding

[Ravindra Pesala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala resolved CARBONDATA-1123.
-
Resolution: Fixed

> Rename interface and variable for RLE encoding
> --
>
> Key: CARBONDATA-1123
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1123
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: Jacky Li
>Assignee: Jacky Li
> Fix For: 1.2.0
>
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> Currently inside RLE encoding implementation, interface and variable is not 
> very readable. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CARBONDATA-1127) Add fixed length encoding for timestamp/date data type

[Ravindra Pesala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala updated CARBONDATA-1127:

Fix Version/s: (was: 1.2.0)
   1.3.0

> Add fixed length encoding for timestamp/date data type
> --
>
> Key: CARBONDATA-1127
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1127
> Project: CarbonData
>  Issue Type: Sub-task
>Reporter: Jacky Li
> Fix For: 1.3.0
>
>
> After this is done, we can move timestamp/date from noDictionary to normal 
> ColumnPage (stored as int[] internally)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CARBONDATA-1125) Add SQL and dataframe option for encoding override

[Ravindra Pesala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala updated CARBONDATA-1125:

Fix Version/s: (was: 1.2.0)
   1.3.0

> Add SQL and dataframe option for encoding override
> --
>
> Key: CARBONDATA-1125
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1125
> Project: CarbonData
>  Issue Type: Sub-task
>Reporter: Jacky Li
> Fix For: 1.3.0
>
>
> User should be able to specify the encoding type for particular field in the 
> table



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CARBONDATA-1141) Data load is partially successful but delete error

[Ravindra Pesala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala updated CARBONDATA-1141:

Fix Version/s: (was: 1.2.0)
   1.3.0

> Data load is partially successful  but delete error
> ---
>
> Key: CARBONDATA-1141
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1141
> Project: CarbonData
>  Issue Type: Bug
>  Components: spark-integration, sql
>Affects Versions: 1.2.0
> Environment: spark on 
> yarn，carbondata1.2.0，hadoop2.7，spark2.1.0，hive2.1.0
>Reporter: zhuzhibin
> Fix For: 1.3.0
>
> Attachments: error1.png, error.png
>
>
> when I tried to load data into table (data size is about 300 million)，the log 
> showed me that “Data load is partially successful for table",
> but when I executed delete table operation,some errors appeared,the error 
> message is "java.lang.ArrayIndexOutOfBoundsException: 1
> at 
> org.apache.carbondata.core.mutate.CarbonUpdateUtil.getRequiredFieldFromTID(CarbonUpdateUtil.java:67)".
> when I executed another delete table operation with where condition,it was 
> succeeful,but executed select operation then appeared 
> "java.lang.ArrayIndexOutOfBoundsException Driver stacktrace:
>   at 
> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1435)"
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CARBONDATA-1174) Streaming Ingest: Write path schema validation/inference

[Ravindra Pesala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala updated CARBONDATA-1174:

Fix Version/s: (was: 1.2.0)
   1.3.0

> Streaming Ingest: Write path schema validation/inference
> 
>
> Key: CARBONDATA-1174
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1174
> Project: CarbonData
>  Issue Type: Sub-task
>  Components: core, spark-integration
>Affects Versions: 1.2.0
>Reporter: Aniket Adnaik
>Assignee: Aniket Adnaik
>  Labels: features
> Fix For: 1.3.0
>
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> Streaming Ingest: Write path 
> - schema validation / schema inference from existing carbondata table
> - streaming ingest allowed to existing tables only



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CARBONDATA-1175) Streaming Ingest: Write path data conversion/transformation

[Ravindra Pesala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala updated CARBONDATA-1175:

Fix Version/s: (was: 1.2.0)
   1.3.0

> Streaming Ingest: Write path data conversion/transformation
> ---
>
> Key: CARBONDATA-1175
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1175
> Project: CarbonData
>  Issue Type: Sub-task
>  Components: core, data-load, data-query, examples, file-format, 
> spark-integration, sql
>Affects Versions: 1.2.0
>Reporter: Aniket Adnaik
> Fix For: 1.3.0
>
>
> Streaming Ingest: Write path Data conversion/Transformation
> - input data is a byte stream in catalyst InternalRow format (row major), 
> which needs to be converted to column format
> - column converter and corresponding iterators needs to be created before 
> invoking carbon layer load path
> - various carbon properties needs to be set, for example SORT_SCOPE (to skip 
> sorting), blocket size, skip global dictionary creation etc 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CARBONDATA-1176) Streaming Ingest: Write path streaming segment/file creation

[Ravindra Pesala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala updated CARBONDATA-1176:

Fix Version/s: (was: 1.2.0)
   1.3.0

> Streaming Ingest: Write path streaming segment/file creation
> 
>
> Key: CARBONDATA-1176
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1176
> Project: CarbonData
>  Issue Type: Sub-task
>  Components: core, data-load, hadoop-integration
>Affects Versions: 1.2.0
>Reporter: Aniket Adnaik
> Fix For: 1.3.0
>
>
> Streaming Ingest: Write path : segment /streaming file 
> - Streaming segment creation and streaming file creation
>  - resolve conflict with spark structured streaming file names. Spark 
> structured streaming names streaming files with unique batch id to avoid 
> overwriting
>  - maintain spark structured streaming recover-ability as streaming file 
> names generated by spark structured streaming are unique and recorded in 
> spark structured streaming metadata. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (CARBONDATA-1186) Class Cast Exception In SortStepRowutil when loading data

[Ravindra Pesala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala resolved CARBONDATA-1186.
-
Resolution: Duplicate

> Class Cast Exception In SortStepRowutil when loading data
> -
>
> Key: CARBONDATA-1186
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1186
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-load
>Affects Versions: 1.2.0
>Reporter: anubhav tarar
>Assignee: anubhav tarar
>Priority: Trivial
> Fix For: 1.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CARBONDATA-1186) Class Cast Exception In SortStepRowutil when loading data

[Ravindra Pesala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala updated CARBONDATA-1186:

Fix Version/s: (was: 1.2.0)
   NONE

> Class Cast Exception In SortStepRowutil when loading data
> -
>
> Key: CARBONDATA-1186
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1186
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-load
>Affects Versions: 1.2.0
>Reporter: anubhav tarar
>Assignee: anubhav tarar
>Priority: Trivial
> Fix For: NONE
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CARBONDATA-1192) Unable to Select Data From more than one table in hive

[Ravindra Pesala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala updated CARBONDATA-1192:

Fix Version/s: (was: 1.2.0)
   1.3.0

> Unable to Select Data From more than one table in hive
> --
>
> Key: CARBONDATA-1192
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1192
> Project: CarbonData
>  Issue Type: Bug
>  Components: hive-integration
>Affects Versions: 1.2.0
> Environment: hive 1.2,spark 2.1
>Reporter: anubhav tarar
>Assignee: anubhav tarar
> Fix For: 1.3.0
>
>
> inside spark shell
> carbon.sql("DROP TABLE IF EXISTS CUSTOMER ")
> carbon.sql("CREATE TABLE CUSTOMER ( C_CUSTKEY INT ,\n C_NAME STRING ,\n 
> C_ADDRESS STRING ,\n " +
>"C_NATIONKEY INT ,\n C_PHONE STRING ,\n C_ACCTBAL DECIMAL(15,2) 
> ,\n C_MKTSEGMENT " +
>"STRING ,\n C_COMMENT STRING ) STORED BY 'carbondata' ")
> carbon.sql("LOAD DATA INPATH 
> \"hdfs://localhost:54310/user1/customer.csv\" INTO TABLE customer " +
>   "OPTIONS('DELIMITER'='|' , 'QUOTECHAR'='\"' , 
> 'FILEHEADER'='C_CUSTKEY,C_NAME," +
>   
> "C_ADDRESS,C_NATIONKEY,C_PHONE,C_ACCTBAL,C_MKTSEGMENT,C_COMMENT')")
>  carbon.sql("DROP TABLE IF EXISTS ORDERS ")
> carbon.sql("CREATE TABLE ORDERS ( O_ORDERKEY INT ,O_CUSTKEY INT 
> ,O_ORDERSTATUS STRING ,O_TOTALPRICE DECIMAL(15,2) , O_ORDERDATE TIMESTAMP , 
> O_ORDERPRIORITY STRING , O_CLERK STRING , O_SHIPPRIORITY INT , O_COMMENT 
> STRING ) STORED BY 'carbondata' ")
> carbon.sql("LOAD DATA INPATH 'hdfs://localhost:54310/user1/orders.csv' 
> INTO TABLE orders " +
>   "OPTIONS('DELIMITER'='|' , 
> 'QUOTECHAR'='\"','FILEHEADER'='O_ORDERKEY,O_CUSTKEY," +
>   
> "O_ORDERSTATUS,O_TOTALPRICE,O_ORDERDATE,O_ORDERPRIORITY,O_CLERK,O_SHIPPRIORITY,"
>  +
>   "O_COMMENT')")
> read data from hive shell
> hive> select o_custkey,c_custkey from orders,customer limit 2;
> Warning: Shuffle Join JOIN[4][tables = [orders, customer]] in Stage 
> 'Stage-1:MAPRED' is a cross product
> Query ID = hduser_20170619125257_d889efa9-261f-436e-9489-fd15d6b76beb
> Total jobs = 1
> Stage-1 is selected by condition resolver.
> Launching Job 1 out of 1
> Number of reduce tasks determined at compile time: 1
> In order to change the average load for a reducer (in bytes):
>   set hive.exec.reducers.bytes.per.reducer=
> In order to limit the maximum number of reducers:
>   set hive.exec.reducers.max=
> In order to set a constant number of reducers:
>   set mapreduce.job.reduces=
> Job running in-process (local Hadoop)
> 2017-06-19 12:53:01,987 Stage-1 map = 0%,  reduce = 0%
> 2017-06-19 12:53:49,113 Stage-1 map = 38%,  reduce = 0%
> 2017-06-19 12:53:51,127 Stage-1 map = 100%,  reduce = 0%
> Ended Job = job_local1708233203_0001 with errors
> Error during job, obtaining debugging information...
> Job Tracking URL: http://localhost:8080/
> FAILED: Execution Error, return code 2 from 
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> MapReduce Jobs Launched: 
> Stage-Stage-1:  HDFS Read: 12033731 HDFS Write: 0 FAIL
> Total MapReduce CPU Time Spent: 0 msec
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (CARBONDATA-1198) Change Unsafe configuration to dynamic

[Ravindra Pesala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala resolved CARBONDATA-1198.
-
   Resolution: Duplicate
Fix Version/s: (was: 1.2.0)
   NONE

Duplicate to CARBONDATA-1199

> Change Unsafe configuration to dynamic
> --
>
> Key: CARBONDATA-1198
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1198
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: Jacky Li
> Fix For: NONE
>
>
> Currently unsafe column page configuration is a static config, it should 
> support dynamic config



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (CARBONDATA-1196) Add 3 Bytes data type support in value compression

[Ravindra Pesala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala resolved CARBONDATA-1196.
-
Resolution: Fixed
  Assignee: Jacky Li

> Add 3 Bytes data type support in value compression
> --
>
> Key: CARBONDATA-1196
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1196
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: Jacky Li
>Assignee: Jacky Li
> Fix For: 1.2.0
>
>  Time Spent: 6h 40m
>  Remaining Estimate: 0h
>
> If there is an integer column and its max value is less than Math.pow(2, 23), 
> and min value bigger than -Math.pow(2, 23), we can apply value compression to 
> 3 bytes



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CARBONDATA-1199) Change Unsafe configuration to dynamic

[Ravindra Pesala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala updated CARBONDATA-1199:

Fix Version/s: (was: 1.2.0)
   1.3.0

> Change Unsafe configuration to dynamic
> --
>
> Key: CARBONDATA-1199
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1199
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: Jacky Li
> Fix For: 1.3.0
>
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Currently unsafe column page configuration is a static config, it should 
> support dynamic config



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (CARBONDATA-1248) LazyColumnPage should extend from ColumnPage

[Ravindra Pesala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala resolved CARBONDATA-1248.
-
Resolution: Fixed
  Assignee: Jacky Li

> LazyColumnPage should extend from ColumnPage
> 
>
> Key: CARBONDATA-1248
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1248
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: Jacky Li
>Assignee: Jacky Li
> Fix For: 1.2.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> LazyColumnPage supports getLong, getDouble only, other function should throw 
> Exceptions



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (CARBONDATA-1252) Add BAD_RECORD_PATH option in Load options section in the Carbon Help doc

[Ravindra Pesala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala resolved CARBONDATA-1252.
-
Resolution: Fixed

> Add BAD_RECORD_PATH option in Load options section in the Carbon Help doc
> -
>
> Key: CARBONDATA-1252
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1252
> Project: CarbonData
>  Issue Type: Sub-task
>Reporter: Mohammad Shahid Khan
>Assignee: Gururaj Shetty
>Priority: Minor
> Fix For: 1.2.0
>
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)