[jira] [Commented] (CARBONDATA-906) Always OOM error when import large dataset (100milion rows)
[ https://issues.apache.org/jira/browse/CARBONDATA-906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15974275#comment-15974275 ] Liang Chen commented on CARBONDATA-906: --- [~crabo] Be pleased to hearing you got good progress! 1.Can you share what optimization actions be taken by you ? 2.Can you close this JIRA if your issues have been solved. Regards Liang > Always OOM error when import large dataset (100milion rows) > --- > > Key: CARBONDATA-906 > URL: https://issues.apache.org/jira/browse/CARBONDATA-906 > Project: CarbonData > Issue Type: Bug > Components: data-load >Affects Versions: 1.0.0-incubating >Reporter: Crabo Yang > Attachments: carbon.properties > > > java.lang.OutOfMemoryError: GC overhead limit exceeded > at > java.util.concurrent.ConcurrentHashMap$Segment.put(ConcurrentHashMap.java:457) > at > java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1130) > at > org.apache.carbondata.core.cache.dictionary.ColumnReverseDictionaryInfo.addDataToDictionaryMap(ColumnReverseDictionaryInfo.java:101) > at > org.apache.carbondata.core.cache.dictionary.ColumnReverseDictionaryInfo.addDictionaryChunk(ColumnReverseDictionaryInfo.java:88) > at > org.apache.carbondata.core.cache.dictionary.DictionaryCacheLoaderImpl.fillDictionaryValuesAndAddToDictionaryChunks(DictionaryCacheLoaderImpl.java:113) > at > org.apache.carbondata.core.cache.dictionary.DictionaryCacheLoaderImpl.load(DictionaryCacheLoaderImpl.java:81) > at > org.apache.carbondata.core.cache.dictionary.AbstractDictionaryCache.loadDictionaryData(AbstractDictionaryCache.java:236) > at > org.apache.carbondata.core.cache.dictionary.AbstractDictionaryCache.checkAndLoadDictionaryData(AbstractDictionaryCache.java:186) > at > org.apache.carbondata.core.cache.dictionary.ReverseDictionaryCache.getDictionary(ReverseDictionaryCache.java:174) > at > org.apache.carbondata.core.cache.dictionary.ReverseDictionaryCache.get(ReverseDictionaryCache.java:67) > at > org.apache.carbondata.core.cache.dictionary.ReverseDictionaryCache.get(ReverseDictionaryCache.java:38) > at > org.apache.carbondata.processing.newflow.converter.impl.DictionaryFieldConverterImpl.(DictionaryFieldConverterImpl.java:92) > at > org.apache.carbondata.processing.newflow.converter.impl.FieldEncoderFactory.createFieldEncoder(FieldEncoderFactory.java:77) > at > org.apache.carbondata.processing.newflow.converter.impl.RowConverterImpl.initialize(RowConverterImpl.java:102) > at > org.apache.carbondata.processing.newflow.steps.DataConverterProcessorStepImpl.initialize(DataConverterProcessorStepImpl.java:69) > at > org.apache.carbondata.processing.newflow.steps.SortProcessorStepImpl.initialize(SortProcessorStepImpl.java:57) > at > org.apache.carbondata.processing.newflow.steps.DataWriterProcessorStepImpl.initialize(DataWriterProcessorStepImpl.java:79) > at > org.apache.carbondata.processing.newflow.DataLoadExecutor.execute(DataLoadExecutor.java:45) > at > org.apache.carbondata.spark.rdd.NewDataFrameLoaderRDD$$anon$2.(NewCarbonDataLoadRDD.scala:425) > at > org.apache.carbondata.spark.rdd.NewDataFrameLoaderRDD.compute(NewCarbonDataLoadRDD.scala:383) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) > at org.apache.spark.scheduler.Task.run(Task.scala:89) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-857) [Documentation] Alter Table - ADD COLUMNS
[ https://issues.apache.org/jira/browse/CARBONDATA-857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-857. --- Resolution: Fixed Fix Version/s: 1.1.0-incubating > [Documentation] Alter Table - ADD COLUMNS > - > > Key: CARBONDATA-857 > URL: https://issues.apache.org/jira/browse/CARBONDATA-857 > Project: CarbonData > Issue Type: Sub-task >Reporter: Gururaj Shetty >Assignee: Pallavi Singh > Fix For: 1.1.0-incubating > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Documentation for ADD COLUMNS > Should include the following: > Function/Description > Syntax > Parameter Description > Usage Guidelines > Example(s) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-894) Add license header for vectorreader/AddColumnTestCases,ChangeDataTypeTestCases,DropColumnTestCases
[ https://issues.apache.org/jira/browse/CARBONDATA-894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-894. --- Resolution: Fixed Fix Version/s: 1.1.0-incubating > Add license header for > vectorreader/AddColumnTestCases,ChangeDataTypeTestCases,DropColumnTestCases > -- > > Key: CARBONDATA-894 > URL: https://issues.apache.org/jira/browse/CARBONDATA-894 > Project: CarbonData > Issue Type: Sub-task > Components: spark-integration >Reporter: Liang Chen >Assignee: Gururaj Shetty >Priority: Trivial > Fix For: 1.1.0-incubating > > Time Spent: 40m > Remaining Estimate: 0h > > Add license header for the three files : > vectorreader/AddColumnTestCases, > vectorreader/ChangeDataTypeTestCases, > vectorreader/DropColumnTestCases -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CARBONDATA-906) Always OOM error when import large dataset (100milion rows)
[ https://issues.apache.org/jira/browse/CARBONDATA-906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970356#comment-15970356 ] Liang Chen commented on CARBONDATA-906: --- [~crabo] Totally ,did you use 20 nodes to load data ? Please do the below two actions: 1. please configure the bellow 3 parameters in carbon.properties(note: please copy the latest carbon.properties to all nodes) carbon.graph.rowset.size=1 (by default is 10, please set to 1/10 for reducing Rowset size exchanged between data load graph) carbon.number.of.cores.while.loading=5 (if your machine has 5 cores, then set to 5, if your each machine has 6 cores,then set to 6) carbon.sort.size=5 ( by default is 50, please set to 1/10 for reducing temp intermediate files) 2.For high cardinality String column, please use DICTIONARY_EXCLUDE, you can refer to : https://github.com/apache/incubator-carbondata/blob/master/docs/useful-tips-on-carbondata.md > Always OOM error when import large dataset (100milion rows) > --- > > Key: CARBONDATA-906 > URL: https://issues.apache.org/jira/browse/CARBONDATA-906 > Project: CarbonData > Issue Type: Bug > Components: data-load >Affects Versions: 1.0.0-incubating >Reporter: Crabo Yang > Attachments: carbon.properties > > > java.lang.OutOfMemoryError: GC overhead limit exceeded > at > java.util.concurrent.ConcurrentHashMap$Segment.put(ConcurrentHashMap.java:457) > at > java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1130) > at > org.apache.carbondata.core.cache.dictionary.ColumnReverseDictionaryInfo.addDataToDictionaryMap(ColumnReverseDictionaryInfo.java:101) > at > org.apache.carbondata.core.cache.dictionary.ColumnReverseDictionaryInfo.addDictionaryChunk(ColumnReverseDictionaryInfo.java:88) > at > org.apache.carbondata.core.cache.dictionary.DictionaryCacheLoaderImpl.fillDictionaryValuesAndAddToDictionaryChunks(DictionaryCacheLoaderImpl.java:113) > at > org.apache.carbondata.core.cache.dictionary.DictionaryCacheLoaderImpl.load(DictionaryCacheLoaderImpl.java:81) > at > org.apache.carbondata.core.cache.dictionary.AbstractDictionaryCache.loadDictionaryData(AbstractDictionaryCache.java:236) > at > org.apache.carbondata.core.cache.dictionary.AbstractDictionaryCache.checkAndLoadDictionaryData(AbstractDictionaryCache.java:186) > at > org.apache.carbondata.core.cache.dictionary.ReverseDictionaryCache.getDictionary(ReverseDictionaryCache.java:174) > at > org.apache.carbondata.core.cache.dictionary.ReverseDictionaryCache.get(ReverseDictionaryCache.java:67) > at > org.apache.carbondata.core.cache.dictionary.ReverseDictionaryCache.get(ReverseDictionaryCache.java:38) > at > org.apache.carbondata.processing.newflow.converter.impl.DictionaryFieldConverterImpl.(DictionaryFieldConverterImpl.java:92) > at > org.apache.carbondata.processing.newflow.converter.impl.FieldEncoderFactory.createFieldEncoder(FieldEncoderFactory.java:77) > at > org.apache.carbondata.processing.newflow.converter.impl.RowConverterImpl.initialize(RowConverterImpl.java:102) > at > org.apache.carbondata.processing.newflow.steps.DataConverterProcessorStepImpl.initialize(DataConverterProcessorStepImpl.java:69) > at > org.apache.carbondata.processing.newflow.steps.SortProcessorStepImpl.initialize(SortProcessorStepImpl.java:57) > at > org.apache.carbondata.processing.newflow.steps.DataWriterProcessorStepImpl.initialize(DataWriterProcessorStepImpl.java:79) > at > org.apache.carbondata.processing.newflow.DataLoadExecutor.execute(DataLoadExecutor.java:45) > at > org.apache.carbondata.spark.rdd.NewDataFrameLoaderRDD$$anon$2.(NewCarbonDataLoadRDD.scala:425) > at > org.apache.carbondata.spark.rdd.NewDataFrameLoaderRDD.compute(NewCarbonDataLoadRDD.scala:383) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) > at org.apache.spark.scheduler.Task.run(Task.scala:89) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-928) Add link to configuration parameters in docs
[ https://issues.apache.org/jira/browse/CARBONDATA-928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-928. --- Resolution: Fixed > Add link to configuration parameters in docs > > > Key: CARBONDATA-928 > URL: https://issues.apache.org/jira/browse/CARBONDATA-928 > Project: CarbonData > Issue Type: Improvement > Components: docs >Affects Versions: 1.0.0-incubating >Reporter: Sanoj MG >Assignee: Sanoj MG >Priority: Minor > Fix For: 1.1.0-incubating > > Time Spent: 0.5h > Remaining Estimate: 0h > > Add a link to configuration parameters in the main page of doc -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CARBONDATA-899) Added Support for Decimal data type and Fixed the timestamp and date issues for Spark-2.1
[ https://issues.apache.org/jira/browse/CARBONDATA-899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-899: -- Issue Type: Sub-task (was: Improvement) Parent: CARBONDATA-728 > Added Support for Decimal data type and Fixed the timestamp and date issues > for Spark-2.1 > - > > Key: CARBONDATA-899 > URL: https://issues.apache.org/jira/browse/CARBONDATA-899 > Project: CarbonData > Issue Type: Sub-task > Components: presto-integration >Reporter: Bhavya Aggarwal >Assignee: Bhavya Aggarwal >Priority: Minor > Time Spent: 40m > Remaining Estimate: 0h > > The Decimal Type correct support is added as well as issues related to > timestamp and date are resolved. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-816) Add examples for hive integration under /Examples
[ https://issues.apache.org/jira/browse/CARBONDATA-816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-816. --- Resolution: Fixed Fix Version/s: 1.2.0-incubating > Add examples for hive integration under /Examples > - > > Key: CARBONDATA-816 > URL: https://issues.apache.org/jira/browse/CARBONDATA-816 > Project: CarbonData > Issue Type: Sub-task > Components: examples >Reporter: Liang Chen >Assignee: anubhav tarar >Priority: Minor > Fix For: 1.2.0-incubating > > Time Spent: 5h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CARBONDATA-907) The grammar for DELETE SEGMENT FOR DATE in website is not correct
[ https://issues.apache.org/jira/browse/CARBONDATA-907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-907: -- Fix Version/s: (was: 1.1.1-incubating) 1.1.0-incubating > The grammar for DELETE SEGMENT FOR DATE in website is not correct > -- > > Key: CARBONDATA-907 > URL: https://issues.apache.org/jira/browse/CARBONDATA-907 > Project: CarbonData > Issue Type: Bug >Reporter: chenerlu >Assignee: chenerlu > Fix For: 1.1.0-incubating > > Time Spent: 1h 10m > Remaining Estimate: 0h > > The grammar for DELETE SEGMENT FOR DATE in website is not correct -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-907) The grammar for DELETE SEGMENT FOR DATE in website is not correct
[ https://issues.apache.org/jira/browse/CARBONDATA-907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-907. --- Resolution: Fixed Fix Version/s: 1.1.1-incubating > The grammar for DELETE SEGMENT FOR DATE in website is not correct > -- > > Key: CARBONDATA-907 > URL: https://issues.apache.org/jira/browse/CARBONDATA-907 > Project: CarbonData > Issue Type: Bug >Reporter: chenerlu >Assignee: chenerlu > Fix For: 1.1.1-incubating > > Time Spent: 1h 10m > Remaining Estimate: 0h > > The grammar for DELETE SEGMENT FOR DATE in website is not correct -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (CARBONDATA-907) The grammar for DELETE SEGMENT FOR DATE in website is not correct
[ https://issues.apache.org/jira/browse/CARBONDATA-907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen reassigned CARBONDATA-907: - Assignee: chenerlu > The grammar for DELETE SEGMENT FOR DATE in website is not correct > -- > > Key: CARBONDATA-907 > URL: https://issues.apache.org/jira/browse/CARBONDATA-907 > Project: CarbonData > Issue Type: Bug >Reporter: chenerlu >Assignee: chenerlu > Time Spent: 1h 10m > Remaining Estimate: 0h > > The grammar for DELETE SEGMENT FOR DATE in website is not correct -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (CARBONDATA-894) Add license header for vectorreader/AddColumnTestCases,ChangeDataTypeTestCases,DropColumnTestCases
[ https://issues.apache.org/jira/browse/CARBONDATA-894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen reassigned CARBONDATA-894: - Assignee: Gururaj Shetty > Add license header for > vectorreader/AddColumnTestCases,ChangeDataTypeTestCases,DropColumnTestCases > -- > > Key: CARBONDATA-894 > URL: https://issues.apache.org/jira/browse/CARBONDATA-894 > Project: CarbonData > Issue Type: Sub-task > Components: spark-integration >Reporter: Liang Chen >Assignee: Gururaj Shetty >Priority: Trivial > > Add license header for the three files : > vectorreader/AddColumnTestCases, > vectorreader/ChangeDataTypeTestCases, > vectorreader/DropColumnTestCases -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CARBONDATA-910) Implement Partition feature
[
https://issues.apache.org/jira/browse/CARBONDATA-910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965516#comment-15965516
]
Liang Chen commented on CARBONDATA-910:
---
Two comments:
1.Partition is a big feature, please consider separating into multiple
tasks(pull requests) to implement.
2.Please put "partition feature" to mailing list for adequate discussion
> Implement Partition feature
> ---
>
> Key: CARBONDATA-910
> URL: https://issues.apache.org/jira/browse/CARBONDATA-910
> Project: CarbonData
> Issue Type: New Feature
> Components: core, data-load, data-query
>Reporter: Cao, Lionel
>Assignee: Cao, Lionel
>
> Why need partition table
> Partition table provide an option to divide table into some smaller pieces.
> With partition table:
> 1. Data could be better managed, organized and stored.
> 2. We can avoid full table scan in some scenario and improve query
> performance. (partition column in filter,
> multiple partition tables join in the same partition column etc.)
> Partitioning design
> Range Partitioning
>range partitioning maps data to partitions according to the range of
> partition column values, operator '<' defines non-inclusive upper bound of
> current partition.
> List Partitioning
>list partitioning allows you map data to partitions with specific
> value list
> Hash Partitioning
>hash partitioning maps data to partitions with hash algorithm and put
> them to the given number of partitions
> Composite Partitioning(2 levels at most for now)
>Range-Range, Range-List, Range-Hash, List-Range, List-List, List-Hash,
> Hash-Range, Hash-List, Hash-Hash
> DDL-Create
> Create table sales(
> itemid long,
> logdate datetime,
> customerid int
> ...
> ...)
> [partition by range logdate(...)]
> [subpartition by list area(...)]
> Stored By 'carbondata'
> [tblproperties(...)];
> range partition:
> partition by range logdate(< '2016-01-01', < '2017-01-01', <
> '2017-02-01', < '2017-03-01', < '2099-01-01')
> list partition:
> partition by list area('Asia', 'Europe', 'North America', 'Africa',
> 'Oceania')
> hash partition:
> partition by hash(itemid, 9)
> composite partition:
> partition by range logdate(< '2016- -01', < '2017-01-01', <
> '2017-02-01', < '2017-03-01', < '2099-01-01')
> subpartition by list area('Asia', 'Europe', 'North America', 'Africa',
> 'Oceania')
> DDL-Rebuild, Add
> Alter table sales rebuild partition by (range|list|hash)(...);
> Alter table salse add partition (< '2018-01-01');#only support range
> partitioning, list partitioning
> Alter table salse add partition ('South America');
> #Note: No delete operation for partition, please use rebuild.
> If need delete data, use delete statement, but the definition of partition
> will not be deleted.
> Partition Table Data Store
> [Option One]
> Use the current design, keep partition folder out of segments
> Fact
>|___Part0
>| |___Segment_0
>| |___ ***-[bucketId]-.carbondata
>| |___ ***-[bucketId]-.carbondata
>| |___Segment_1
>| ...
>|___Part1
>| |___Segment_0
>| |___Segment_1
>|...
> [Option Two]
> remove partition folder, add partition id into file name and build btree in
> driver side.
> Fact
>|___Segment_0
>| |___ ***-[bucketId]-[partitionId].carbondata
>| |___ ***-[bucketId]-[partitionId].carbondata
>|___Segment_1
>|___Segment_2
>...
> Pros & Cons:
> Option one would be faster to locate target files
> Option two need to store more metadata of folders
> Partition Table MetaData Store
> partitioni info should be stored in file footer/index file and load into
> memory before user query.
> Relationship with Bucket
> Bucket should be lower level of partition.
> Partition Table Query
> Example:
> Select * from sales
> where logdate <= date '2016-12-01';
> User should remember to add a partition filter when write SQL on a partition
> table.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (CARBONDATA-906) Always OOM error when import large dataset (100milion rows)
[ https://issues.apache.org/jira/browse/CARBONDATA-906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965508#comment-15965508 ] Liang Chen commented on CARBONDATA-906: --- Hi Can you provide these info : 1. Do you use 20 nodes cluster(32GB ram) for data loading ? or only use a few of nodes to load data. 2. Please provide "CREATE TABLE" script and 100 columns' cardinality(distinct value) 3. Please provide carbon.properties file and start spark cluster script. > Always OOM error when import large dataset (100milion rows) > --- > > Key: CARBONDATA-906 > URL: https://issues.apache.org/jira/browse/CARBONDATA-906 > Project: CarbonData > Issue Type: Bug > Components: data-load >Affects Versions: 1.0.0-incubating >Reporter: Crabo Yang > > java.lang.OutOfMemoryError: GC overhead limit exceeded > at > java.util.concurrent.ConcurrentHashMap$Segment.put(ConcurrentHashMap.java:457) > at > java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1130) > at > org.apache.carbondata.core.cache.dictionary.ColumnReverseDictionaryInfo.addDataToDictionaryMap(ColumnReverseDictionaryInfo.java:101) > at > org.apache.carbondata.core.cache.dictionary.ColumnReverseDictionaryInfo.addDictionaryChunk(ColumnReverseDictionaryInfo.java:88) > at > org.apache.carbondata.core.cache.dictionary.DictionaryCacheLoaderImpl.fillDictionaryValuesAndAddToDictionaryChunks(DictionaryCacheLoaderImpl.java:113) > at > org.apache.carbondata.core.cache.dictionary.DictionaryCacheLoaderImpl.load(DictionaryCacheLoaderImpl.java:81) > at > org.apache.carbondata.core.cache.dictionary.AbstractDictionaryCache.loadDictionaryData(AbstractDictionaryCache.java:236) > at > org.apache.carbondata.core.cache.dictionary.AbstractDictionaryCache.checkAndLoadDictionaryData(AbstractDictionaryCache.java:186) > at > org.apache.carbondata.core.cache.dictionary.ReverseDictionaryCache.getDictionary(ReverseDictionaryCache.java:174) > at > org.apache.carbondata.core.cache.dictionary.ReverseDictionaryCache.get(ReverseDictionaryCache.java:67) > at > org.apache.carbondata.core.cache.dictionary.ReverseDictionaryCache.get(ReverseDictionaryCache.java:38) > at > org.apache.carbondata.processing.newflow.converter.impl.DictionaryFieldConverterImpl.(DictionaryFieldConverterImpl.java:92) > at > org.apache.carbondata.processing.newflow.converter.impl.FieldEncoderFactory.createFieldEncoder(FieldEncoderFactory.java:77) > at > org.apache.carbondata.processing.newflow.converter.impl.RowConverterImpl.initialize(RowConverterImpl.java:102) > at > org.apache.carbondata.processing.newflow.steps.DataConverterProcessorStepImpl.initialize(DataConverterProcessorStepImpl.java:69) > at > org.apache.carbondata.processing.newflow.steps.SortProcessorStepImpl.initialize(SortProcessorStepImpl.java:57) > at > org.apache.carbondata.processing.newflow.steps.DataWriterProcessorStepImpl.initialize(DataWriterProcessorStepImpl.java:79) > at > org.apache.carbondata.processing.newflow.DataLoadExecutor.execute(DataLoadExecutor.java:45) > at > org.apache.carbondata.spark.rdd.NewDataFrameLoaderRDD$$anon$2.(NewCarbonDataLoadRDD.scala:425) > at > org.apache.carbondata.spark.rdd.NewDataFrameLoaderRDD.compute(NewCarbonDataLoadRDD.scala:383) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) > at org.apache.spark.scheduler.Task.run(Task.scala:89) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CARBONDATA-901) Fix some spelling mistakes
[ https://issues.apache.org/jira/browse/CARBONDATA-901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964830#comment-15964830 ] Liang Chen commented on CARBONDATA-901: --- Hi Cao Gaofei Already gave the contributor right to you , so now you can assign issue to yourself. > Fix some spelling mistakes > -- > > Key: CARBONDATA-901 > URL: https://issues.apache.org/jira/browse/CARBONDATA-901 > Project: CarbonData > Issue Type: Improvement > Components: core >Reporter: Cao Gaofei >Assignee: Cao Gaofei >Priority: Trivial > Time Spent: 0.5h > Remaining Estimate: 0h > > https://github.com/apache/incubator-carbondata/pull/782 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-893) MR testcase hangs in Hadoop 2.7.2 version profile
[ https://issues.apache.org/jira/browse/CARBONDATA-893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-893. --- Resolution: Fixed Assignee: Ravindra Pesala Fix Version/s: 1.1.0-incubating > MR testcase hangs in Hadoop 2.7.2 version profile > - > > Key: CARBONDATA-893 > URL: https://issues.apache.org/jira/browse/CARBONDATA-893 > Project: CarbonData > Issue Type: Bug >Reporter: Ravindra Pesala >Assignee: Ravindra Pesala > Fix For: 1.1.0-incubating > > Time Spent: 40m > Remaining Estimate: 0h > > MR testcase hangs in Hadoop 2.7.2 version profile -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CARBONDATA-894) Add license header for vectorreader/AddColumnTestCases,ChangeDataTypeTestCases,DropColumnTestCases
[ https://issues.apache.org/jira/browse/CARBONDATA-894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-894: -- Issue Type: Sub-task (was: Bug) Parent: CARBONDATA-895 > Add license header for > vectorreader/AddColumnTestCases,ChangeDataTypeTestCases,DropColumnTestCases > -- > > Key: CARBONDATA-894 > URL: https://issues.apache.org/jira/browse/CARBONDATA-894 > Project: CarbonData > Issue Type: Sub-task > Components: spark-integration >Reporter: Liang Chen >Priority: Trivial > > Add license header for the three files : > vectorreader/AddColumnTestCases, > vectorreader/ChangeDataTypeTestCases, > vectorreader/DropColumnTestCases -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CARBONDATA-895) Fix license header checking issues
Liang Chen created CARBONDATA-895: - Summary: Fix license header checking issues Key: CARBONDATA-895 URL: https://issues.apache.org/jira/browse/CARBONDATA-895 Project: CarbonData Issue Type: Bug Components: build Reporter: Liang Chen Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CARBONDATA-894) Add license header for vectorreader/AddColumnTestCases,ChangeDataTypeTestCases,DropColumnTestCases
Liang Chen created CARBONDATA-894: - Summary: Add license header for vectorreader/AddColumnTestCases,ChangeDataTypeTestCases,DropColumnTestCases Key: CARBONDATA-894 URL: https://issues.apache.org/jira/browse/CARBONDATA-894 Project: CarbonData Issue Type: Bug Components: spark-integration Reporter: Liang Chen Priority: Trivial Add license header for the three files : vectorreader/AddColumnTestCases, vectorreader/ChangeDataTypeTestCases, vectorreader/DropColumnTestCases -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CARBONDATA-891) Fix compilation issue of LocalFileLockTest generate new folder "carbon.store"
[ https://issues.apache.org/jira/browse/CARBONDATA-891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-891: -- Description: Fix compilation issue of LocalFileLockTest generate new folder "carbon.store" (was: Fix compilation issue of AlterTableValidationTestCase generate new folder "carbon.store") > Fix compilation issue of LocalFileLockTest generate new folder "carbon.store" > - > > Key: CARBONDATA-891 > URL: https://issues.apache.org/jira/browse/CARBONDATA-891 > Project: CarbonData > Issue Type: Bug > Components: build, core >Reporter: Liang Chen >Assignee: Liang Chen > Fix For: 1.1.0-incubating > > > Fix compilation issue of LocalFileLockTest generate new folder "carbon.store" -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CARBONDATA-891) Fix compilation issue of LocalFileLockTest generate new folder "carbon.store"
[ https://issues.apache.org/jira/browse/CARBONDATA-891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-891: -- Summary: Fix compilation issue of LocalFileLockTest generate new folder "carbon.store" (was: Fix compilation issue of AlterTableValidationTestCase generate new folder "carbon.store") > Fix compilation issue of LocalFileLockTest generate new folder "carbon.store" > - > > Key: CARBONDATA-891 > URL: https://issues.apache.org/jira/browse/CARBONDATA-891 > Project: CarbonData > Issue Type: Bug > Components: build, core >Reporter: Liang Chen >Assignee: Liang Chen > Fix For: 1.1.0-incubating > > > Fix compilation issue of AlterTableValidationTestCase generate new folder > "carbon.store" -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CARBONDATA-891) Fix compilation issue of AlterTableValidationTestCase generate new folder "carbon.store"
Liang Chen created CARBONDATA-891: - Summary: Fix compilation issue of AlterTableValidationTestCase generate new folder "carbon.store" Key: CARBONDATA-891 URL: https://issues.apache.org/jira/browse/CARBONDATA-891 Project: CarbonData Issue Type: Bug Components: build, core Reporter: Liang Chen Assignee: Liang Chen Fix For: 1.1.0-incubating Fix compilation issue of AlterTableValidationTestCase generate new folder "carbon.store" -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-884) [Documentation] information on assembly jar to be provided in Quick Start
[ https://issues.apache.org/jira/browse/CARBONDATA-884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-884. --- Resolution: Fixed Fix Version/s: 1.1.0-incubating > [Documentation] information on assembly jar to be provided in Quick Start > - > > Key: CARBONDATA-884 > URL: https://issues.apache.org/jira/browse/CARBONDATA-884 > Project: CarbonData > Issue Type: Improvement >Reporter: Gururaj Shetty >Assignee: Sanoj MG >Priority: Minor > Fix For: 1.1.0-incubating > > Time Spent: 40m > Remaining Estimate: 0h > > In Quick start we have mentioned the below command: > Start Spark shell by running the following command in the Spark directory: > ./bin/spark-shell --jars > It is better to mention for the user from where to take the assembly jar. > For example: the assembly jar will be present in the target folder when you > build the project. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CARBONDATA-888) Dictionary include / exclude option in dataframe writer
[
https://issues.apache.org/jira/browse/CARBONDATA-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962030#comment-15962030
]
Liang Chen commented on CARBONDATA-888:
---
Sure, please let me know your jira account email id, i will give your right.
> Dictionary include / exclude option in dataframe writer
> ---
>
> Key: CARBONDATA-888
> URL: https://issues.apache.org/jira/browse/CARBONDATA-888
> Project: CarbonData
> Issue Type: Improvement
> Components: spark-integration
>Affects Versions: 1.2.0-incubating
> Environment: HDP 2.5, Spark 1.6
>Reporter: Sanoj MG
>Priority: Minor
> Fix For: 1.2.0-incubating
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> While creating a Carbondata table from dataframe, currently it is not
> possible to specify columns that needs to be included in or excluded from the
> dictionary. An option is required to specify it as below :
> df.write.format("carbondata")
> .option("tableName", "test")
> .option("compress","true")
> .option("dictionary_include","incol1,intcol2")
> .option("dictionary_exclude","stringcol1,stringcol2")
> .mode(SaveMode.Overwrite)
> .save()
> We have lot of integer columns that are dimensions, dataframe.save is used to
> quickly create tables instead of writing ddls, and it would be nice to have
> this feature to execute POCs.
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-885) Inconsistent usage of " " in queries in ddl operations on Carbondata
[ https://issues.apache.org/jira/browse/CARBONDATA-885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-885. --- Resolution: Fixed > Inconsistent usage of " " in queries in ddl operations on Carbondata > > > Key: CARBONDATA-885 > URL: https://issues.apache.org/jira/browse/CARBONDATA-885 > Project: CarbonData > Issue Type: Bug > Components: docs >Reporter: Pallavi Singh >Assignee: Ramandeep30 >Priority: Trivial > Fix For: 1.1.0-incubating > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (CARBONDATA-885) Inconsistent usage of " " in queries in ddl operations on Carbondata
[ https://issues.apache.org/jira/browse/CARBONDATA-885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen reassigned CARBONDATA-885: - Assignee: Ramandeep30 > Inconsistent usage of " " in queries in ddl operations on Carbondata > > > Key: CARBONDATA-885 > URL: https://issues.apache.org/jira/browse/CARBONDATA-885 > Project: CarbonData > Issue Type: Bug > Components: docs >Reporter: Pallavi Singh >Assignee: Ramandeep30 >Priority: Trivial > Fix For: 1.1.0-incubating > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-856) [Documentation] Alter Table - TABLE RENAME
[ https://issues.apache.org/jira/browse/CARBONDATA-856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-856. --- Resolution: Fixed Fix Version/s: 1.1.0-incubating > [Documentation] Alter Table - TABLE RENAME > --- > > Key: CARBONDATA-856 > URL: https://issues.apache.org/jira/browse/CARBONDATA-856 > Project: CarbonData > Issue Type: Sub-task >Reporter: Gururaj Shetty >Assignee: Srigopal Mohanty > Fix For: 1.1.0-incubating > > Time Spent: 1h > Remaining Estimate: 0h > > Documentation for TABLE RENAME > Should include the following: > Function/Description > Syntax > Parameter Description > Usage Guidelines > Example(s) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-878) Inconsistent stylin in quick-start.md file
[ https://issues.apache.org/jira/browse/CARBONDATA-878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-878. --- Resolution: Fixed Fix Version/s: 1.1.0-incubating > Inconsistent stylin in quick-start.md file > -- > > Key: CARBONDATA-878 > URL: https://issues.apache.org/jira/browse/CARBONDATA-878 > Project: CarbonData > Issue Type: Improvement > Components: docs >Reporter: Pallavi Singh >Assignee: Gururaj Shetty >Priority: Trivial > Fix For: 1.1.0-incubating > > > https://github.com/apache/incubator-carbondata/pull/747 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CARBONDATA-878) Inconsistent stylin in quick-start.md file
[ https://issues.apache.org/jira/browse/CARBONDATA-878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-878: -- Description: https://github.com/apache/incubator-carbondata/pull/747 > Inconsistent stylin in quick-start.md file > -- > > Key: CARBONDATA-878 > URL: https://issues.apache.org/jira/browse/CARBONDATA-878 > Project: CarbonData > Issue Type: Improvement > Components: docs >Reporter: Pallavi Singh >Assignee: Gururaj Shetty >Priority: Trivial > > https://github.com/apache/incubator-carbondata/pull/747 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-872) Fix comment issues of integration/presto for easier reading
[ https://issues.apache.org/jira/browse/CARBONDATA-872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-872. --- Resolution: Fixed Fix Version/s: 1.2.0-incubating > Fix comment issues of integration/presto for easier reading > --- > > Key: CARBONDATA-872 > URL: https://issues.apache.org/jira/browse/CARBONDATA-872 > Project: CarbonData > Issue Type: Bug > Components: presto-integration >Reporter: Liang Chen >Assignee: ffpeng > Fix For: 1.2.0-incubating > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Fix comment issues of integration/presto for easier reading -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-862) USE_KETTLE option described in dml-operation-on-carbondata.md document doesn't work
[ https://issues.apache.org/jira/browse/CARBONDATA-862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-862. --- Resolution: Fixed Fix Version/s: 1.1.0-incubating > USE_KETTLE option described in dml-operation-on-carbondata.md document > doesn't work > --- > > Key: CARBONDATA-862 > URL: https://issues.apache.org/jira/browse/CARBONDATA-862 > Project: CarbonData > Issue Type: Bug > Components: docs >Affects Versions: 1.0.0-incubating >Reporter: Divya Gupta >Assignee: Divya Gupta >Priority: Minor > Fix For: 1.1.0-incubating > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-866) remove kettle configuration from master/docs/configuration-parameters.md
[ https://issues.apache.org/jira/browse/CARBONDATA-866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-866. --- Resolution: Fixed Assignee: Vinod Rohilla Fix Version/s: 1.1.0-incubating > remove kettle configuration from master/docs/configuration-parameters.md > > > Key: CARBONDATA-866 > URL: https://issues.apache.org/jira/browse/CARBONDATA-866 > Project: CarbonData > Issue Type: Bug >Reporter: manoj mathpal >Assignee: Vinod Rohilla >Priority: Minor > Fix For: 1.1.0-incubating > > Time Spent: 50m > Remaining Estimate: 0h > > remove kettle configuration from master/docs/configuration-parameters.md > files. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-865) Remove configurations for Kettle from master/docs/installation-guide.md
[ https://issues.apache.org/jira/browse/CARBONDATA-865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-865. --- Resolution: Fixed Fix Version/s: 1.1.0-incubating > Remove configurations for Kettle from master/docs/installation-guide.md > --- > > Key: CARBONDATA-865 > URL: https://issues.apache.org/jira/browse/CARBONDATA-865 > Project: CarbonData > Issue Type: Bug >Affects Versions: 1.0.0-incubating >Reporter: Vinod Rohilla >Assignee: manoj mathpal >Priority: Minor > Fix For: 1.1.0-incubating > > Time Spent: 1h > Remaining Estimate: 0h > > Actual Result:Configurations displays for Kettle under > installation-guide.md file. > Expected Result: Remove configurations for Kettle from > master/docs/installation-guide.md file. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-815) Add basic hive integration code
[ https://issues.apache.org/jira/browse/CARBONDATA-815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-815. --- Resolution: Fixed > Add basic hive integration code > --- > > Key: CARBONDATA-815 > URL: https://issues.apache.org/jira/browse/CARBONDATA-815 > Project: CarbonData > Issue Type: Sub-task > Components: hive-integration >Reporter: Liang Chen >Assignee: cen yuhai > Fix For: 1.2.0-incubating > > Time Spent: 2h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CARBONDATA-872) Fix comment issues of integration/presto for easier reading
Liang Chen created CARBONDATA-872: - Summary: Fix comment issues of integration/presto for easier reading Key: CARBONDATA-872 URL: https://issues.apache.org/jira/browse/CARBONDATA-872 Project: CarbonData Issue Type: Bug Components: presto-integration Reporter: Liang Chen Assignee: ffpeng Fix comment issues of integration/presto for easier reading -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CARBONDATA-803) Incorrect results returned by not equal to filter on dictionary column with numeric data type
[
https://issues.apache.org/jira/browse/CARBONDATA-803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liang Chen updated CARBONDATA-803:
--
Fix Version/s: (was: 1.0.1-incubating)
> Incorrect results returned by not equal to filter on dictionary column with
> numeric data type
> -
>
> Key: CARBONDATA-803
> URL: https://issues.apache.org/jira/browse/CARBONDATA-803
> Project: CarbonData
> Issue Type: Bug
>Reporter: Manish Gupta
>Assignee: Manish Gupta
> Fix For: 1.1.0-incubating
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Whenever a not equal to filter is applied on dictionary column with numeric
> datatype, the cast added by spark plan is removed while creating carbon
> filters from spark filter. Due to this plan modification incorrect results
> are returned by spark.
> Steps to reproduce the issue:
> 1. CREATE TABLE IF NOT EXISTS carbon(ID Int, date Timestamp, country String,
> name String, phonetype String, serialname String, salary Int) STORED BY
> 'org.apache.carbondata.format' TBLPROPERTIES('dictionary_include'='id')
> 2. LOAD DATA LOCAL INPATH '$csvFilePath' into table carbon
> 3. select Id from test_not_equal_to_carbon where id != '7'
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (CARBONDATA-811) Refactor dictionary based result collector class
[ https://issues.apache.org/jira/browse/CARBONDATA-811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-811: -- Fix Version/s: (was: 1.0.1-incubating) 1.1.0-incubating > Refactor dictionary based result collector class > > > Key: CARBONDATA-811 > URL: https://issues.apache.org/jira/browse/CARBONDATA-811 > Project: CarbonData > Issue Type: Bug >Reporter: Srinath Thota >Priority: Minor > Fix For: 1.1.0-incubating > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Problem: For each batch result collector class is filling all the class level > variable this may hit the performance > Solution: fill it in constructor, so only once it will be initialize. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (CARBONDATA-755) Confusing comment about default kettle_use
[ https://issues.apache.org/jira/browse/CARBONDATA-755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen closed CARBONDATA-755. - Resolution: Invalid > Confusing comment about default kettle_use > -- > > Key: CARBONDATA-755 > URL: https://issues.apache.org/jira/browse/CARBONDATA-755 > Project: CarbonData > Issue Type: Bug > Components: examples >Reporter: sehriff >Assignee: sehriff >Priority: Minor > Labels: patch > Time Spent: 1h 20m > Remaining Estimate: 0h > > Fix comment issues of CarbonExample.scala about default use_kettle > https://github.com/apache/incubator-carbondata/pull/640/commits -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CARBONDATA-755) Confusing comment about default kettle_use
[ https://issues.apache.org/jira/browse/CARBONDATA-755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-755: -- Fix Version/s: (was: 1.0.1-incubating) (was: 1.1.0-incubating) > Confusing comment about default kettle_use > -- > > Key: CARBONDATA-755 > URL: https://issues.apache.org/jira/browse/CARBONDATA-755 > Project: CarbonData > Issue Type: Bug > Components: examples >Reporter: sehriff >Assignee: sehriff >Priority: Minor > Labels: patch > Time Spent: 1h 20m > Remaining Estimate: 0h > > Fix comment issues of CarbonExample.scala about default use_kettle > https://github.com/apache/incubator-carbondata/pull/640/commits -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-848) Select count(*) from table gives an exception in Presto
[ https://issues.apache.org/jira/browse/CARBONDATA-848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-848. --- Resolution: Fixed Fix Version/s: 1.2.0 > Select count(*) from table gives an exception in Presto > --- > > Key: CARBONDATA-848 > URL: https://issues.apache.org/jira/browse/CARBONDATA-848 > Project: CarbonData > Issue Type: Bug > Components: presto-integration >Reporter: Bhavya Aggarwal >Assignee: Bhavya Aggarwal > Fix For: 1.2.0 > > Time Spent: 1h > Remaining Estimate: 0h > > The select count(*) is giving an ArrayIndexOutOfException in Presto connector. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CARBONDATA-850) Fix the comment definition issues of CarbonData thrift files
[ https://issues.apache.org/jira/browse/CARBONDATA-850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-850: -- Description: Fix the comment definition issues of CarbonData thrift files, for helping users to easier understand CarbonData file format (was: Fix the comment definition issues of CarbonData thrift files, for help users to easier understand CarbonData file format) > Fix the comment definition issues of CarbonData thrift files > > > Key: CARBONDATA-850 > URL: https://issues.apache.org/jira/browse/CARBONDATA-850 > Project: CarbonData > Issue Type: Bug > Components: file-format >Reporter: Liang Chen >Assignee: Liang Chen >Priority: Minor > Fix For: 1.1.0-incubating > > > Fix the comment definition issues of CarbonData thrift files, for helping > users to easier understand CarbonData file format -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CARBONDATA-850) Fix the comment definition issues of CarbonData thrift files
Liang Chen created CARBONDATA-850: - Summary: Fix the comment definition issues of CarbonData thrift files Key: CARBONDATA-850 URL: https://issues.apache.org/jira/browse/CARBONDATA-850 Project: CarbonData Issue Type: Bug Components: file-format Reporter: Liang Chen Assignee: Liang Chen Priority: Minor Fix For: 1.1.0-incubating Fix the comment definition issues of CarbonData thrift files, for help users to easier understand CarbonData file format -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-835) Null values in carbon table gives a NullPointerException when querying from Presto
[ https://issues.apache.org/jira/browse/CARBONDATA-835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-835. --- Resolution: Fixed Fix Version/s: 1.2.0 > Null values in carbon table gives a NullPointerException when querying from > Presto > -- > > Key: CARBONDATA-835 > URL: https://issues.apache.org/jira/browse/CARBONDATA-835 > Project: CarbonData > Issue Type: Bug > Components: presto-integration > Environment: Presto >Reporter: Bhavya Aggarwal >Assignee: Bhavya Aggarwal >Priority: Minor > Fix For: 1.2.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Null values in carbon table gives a NullPointerException when querying from > Presto -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-834) Describe Table in Presto gives incorrect order of columns
[ https://issues.apache.org/jira/browse/CARBONDATA-834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-834. --- Resolution: Fixed Fix Version/s: 1.2.0 > Describe Table in Presto gives incorrect order of columns > - > > Key: CARBONDATA-834 > URL: https://issues.apache.org/jira/browse/CARBONDATA-834 > Project: CarbonData > Issue Type: Bug > Components: presto-integration > Environment: Presto >Reporter: Bhavya Aggarwal >Assignee: Bhavya Aggarwal >Priority: Minor > Fix For: 1.2.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Describe Table in Presto gives incorrect order of columns -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (CARBONDATA-827) Query statistics log format is incorrect
[ https://issues.apache.org/jira/browse/CARBONDATA-827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen reassigned CARBONDATA-827: - Assignee: Jacky Li Request participants: (was: ) Priority: Minor (was: Major) Fix Version/s: 1.1.0-incubating > Query statistics log format is incorrect > > > Key: CARBONDATA-827 > URL: https://issues.apache.org/jira/browse/CARBONDATA-827 > Project: CarbonData > Issue Type: Bug >Reporter: Jacky Li >Assignee: Jacky Li >Priority: Minor > Fix For: 1.1.0-incubating > > Time Spent: 50m > Remaining Estimate: 0h > > The output log for query statistics has repeated numbers which is incorrect -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-827) Query statistics log format is incorrect
[ https://issues.apache.org/jira/browse/CARBONDATA-827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-827. --- Resolution: Fixed > Query statistics log format is incorrect > > > Key: CARBONDATA-827 > URL: https://issues.apache.org/jira/browse/CARBONDATA-827 > Project: CarbonData > Issue Type: Bug >Reporter: Jacky Li >Assignee: Jacky Li >Priority: Minor > Fix For: 1.1.0-incubating > > Time Spent: 50m > Remaining Estimate: 0h > > The output log for query statistics has repeated numbers which is incorrect -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-708) Between operator does not work properly in carbondata.
[
https://issues.apache.org/jira/browse/CARBONDATA-708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liang Chen resolved CARBONDATA-708.
---
Resolution: Fixed
Assignee: Bhavya Aggarwal
Fix Version/s: 1.1.0-incubating
> Between operator does not work properly in carbondata.
> --
>
> Key: CARBONDATA-708
> URL: https://issues.apache.org/jira/browse/CARBONDATA-708
> Project: CarbonData
> Issue Type: Bug
> Components: sql
>Affects Versions: 1.1.0-incubating
> Environment: Spark 2.1
>Reporter: SWATI RAO
>Assignee: Bhavya Aggarwal
>Priority: Minor
> Fix For: 1.1.0-incubating
>
> Attachments: 100_hive_test.csv
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Incorrect result displays.
> Note:Issue exist when you take Latest_HOUR as a String but if you take
> Latest_HOUR as a int then it's working fine.
> Steps to reproduce:
> 1:Create table using following Command
> " create table Carbon_automation (imei string,deviceInformationId int,MAC
> string,deviceColor string,device_backColor string,modelId string,marketName
> string,AMSize string,ROMSize string,CUPAudit string,CPIClocked string,series
> string,productionDate timestamp,bomCode string,internalModels string,
> deliveryTime string, channelsId string, channelsName string , deliveryAreaId
> string, deliveryCountry string, deliveryProvince string, deliveryCity
> string,deliveryDistrict string, deliveryStreet string, oxSingleNumber string,
> ActiveCheckTime string, ActiveAreaId string, ActiveCountry string,
> ActiveProvince string, Activecity string, ActiveDistrict string, ActiveStreet
> string, ActiveOperatorId string, Active_releaseId string, Active_EMUIVersion
> string, Active_operaSysVersion string, Active_BacVerNumber string,
> Active_BacFlashVer string, Active_webUIVersion string,
> Active_webUITypeCarrVer string,Active_webTypeDataVerNumber string,
> Active_operatorsVersion string, Active_phonePADPartitionedVersions string,
> Latest_YEAR int, Latest_MONTH int, Latest_DAY int, Latest_HOUR string,
> Latest_areaId string, Latest_country string, Latest_province string,
> Latest_city string, Latest_district string, Latest_street string,
> Latest_releaseId string, Latest_EMUIVersion string, Latest_operaSysVersion
> string, Latest_BacVerNumber string, Latest_BacFlashVer string,
> Latest_webUIVersion string, Latest_webUITypeCarrVer string,
> Latest_webTypeDataVerNumber string, Latest_operatorsVersion string,
> Latest_phonePADPartitionedVersions string, Latest_operatorId string,
> gamePointDescription string,gamePointId double,contractNumber
> double,imei_count int) STORED BY 'org.apache.carbondata.format' TBLPROPERTIES
> ('DICTIONARY_INCLUDE'='deviceInformationId,Latest_YEAR,Latest_MONTH,Latest_DAY')"
> 2:Load Data with following command
> " LOAD DATA INPATH 'HDFS_URL/BabuStore/Data/HiveData' INTO TABLE
> Carbon_automation
> OPTIONS('DELIMITER'=',','QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='imei,deviceInformationId,MAC,deviceColor,device_backColor,modelId,marketName,AMSize,ROMSize,CUPAudit,CPIClocked,series,productionDate,bomCode,internalModels,deliveryTime,channelsId,channelsName,deliveryAreaId,deliveryCountry,deliveryProvince,deliveryCity,deliveryDistrict,deliveryStreet,oxSingleNumber,contractNumber,ActiveCheckTime,ActiveAreaId,ActiveCountry,ActiveProvince,Activecity,ActiveDistrict,ActiveStreet,ActiveOperatorId,Active_releaseId,Active_EMUIVersion,Active_operaSysVersion,Active_BacVerNumber,Active_BacFlashVer,Active_webUIVersion,Active_webUITypeCarrVer,Active_webTypeDataVerNumber,Active_operatorsVersion,Active_phonePADPartitionedVersions,Latest_YEAR,Latest_MONTH,Latest_DAY,Latest_HOUR,Latest_areaId,Latest_country,Latest_province,Latest_city,Latest_district,Latest_street,Latest_releaseId,Latest_EMUIVersion,Latest_operaSysVersion,Latest_BacVerNumber,Latest_BacFlashVer,Latest_webUIVersion,Latest_webUITypeCarrVer,Latest_webTypeDataVerNumber,Latest_operatorsVersion,Latest_phonePADPartitionedVersions,Latest_operatorId,gamePointId,gamePointDescription,imei_count')"
> 3:Run the Query
> select Latest_DAY,Latest_HOUR,count(distinct AMSize) as
> AMSize_number,sum(gamePointId+contractNumber) as total from Carbon_automation
> where Latest_HOUR between 12 and 15 group by Latest_DAY,Latest_HOUR order by
> total desc
> 4:No Result display:
> +-+--+++--+
> | Latest_DAY | Latest_HOUR | AMSize_number | total |
> +-+--+++--+
> +-+--+++--+
> No rows selected (2.133 seconds).
> 5:CSV Attached "100_hive_test.csv"
> Expected Result:Correct Result should be display.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-826) Create carbondata-connector for query carbon data in presto
[ https://issues.apache.org/jira/browse/CARBONDATA-826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-826. --- Resolution: Fixed Fix Version/s: 1.2.0 > Create carbondata-connector for query carbon data in presto > --- > > Key: CARBONDATA-826 > URL: https://issues.apache.org/jira/browse/CARBONDATA-826 > Project: CarbonData > Issue Type: Sub-task > Components: presto-integration >Reporter: Liang Chen >Assignee: Liang Chen >Priority: Minor > Fix For: 1.2.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > 1.In CarbonData project, generate carbondata-connector of presto > 2.Copy carbondata-connector to presto/plugin/ > 3.Run query in presto to read carbon data. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-828) Fix length issue of model.dimensions in CarbonGlobalDictionaryGenerateRDD
[
https://issues.apache.org/jira/browse/CARBONDATA-828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liang Chen resolved CARBONDATA-828.
---
Resolution: Fixed
> Fix length issue of model.dimensions in CarbonGlobalDictionaryGenerateRDD
> -
>
> Key: CARBONDATA-828
> URL: https://issues.apache.org/jira/browse/CARBONDATA-828
> Project: CarbonData
> Issue Type: Bug
> Components: spark-integration
>Affects Versions: 1.0.0-incubating
>Reporter: tianli
>Assignee: tianli
>Priority: Minor
> Labels: patch
> Fix For: 1.1.0-incubating
>
> Attachments: CarbonGlobalDictionaryRDD.scala
>
> Original Estimate: 1h
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> org.apache.carbondata.spark.rdd.CarbonGlobalDictionaryGenerateRDD find a bug:
> code line 378-380
> if (model.isFirstLoad && model.highCardIdentifyEnable
> && !model.isComplexes(split.index)
> && model.dimensions(split.index).isColumnar) {
> model.dimensions(split.index).isColumnar must change to
> model.primDimensions(split.index).isColumnar because model.isComplexes.length
> may be != model.dimensions.length when create table use DICTIONARY_EXCLUDE
> or column datatype is complex
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (CARBONDATA-828) Fix length issue of model.dimensions in CarbonGlobalDictionaryGenerateRDD
[
https://issues.apache.org/jira/browse/CARBONDATA-828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liang Chen updated CARBONDATA-828:
--
Affects Version/s: (was: 1.1.0-incubating)
Request participants: (was: )
Priority: Minor (was: Major)
Fix Version/s: (was: 1.0.0-incubating)
> Fix length issue of model.dimensions in CarbonGlobalDictionaryGenerateRDD
> -
>
> Key: CARBONDATA-828
> URL: https://issues.apache.org/jira/browse/CARBONDATA-828
> Project: CarbonData
> Issue Type: Bug
> Components: spark-integration
>Affects Versions: 1.0.0-incubating
>Reporter: tianli
>Assignee: tianli
>Priority: Minor
> Labels: patch
> Fix For: 1.1.0-incubating
>
> Attachments: CarbonGlobalDictionaryRDD.scala
>
> Original Estimate: 1h
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> org.apache.carbondata.spark.rdd.CarbonGlobalDictionaryGenerateRDD find a bug:
> code line 378-380
> if (model.isFirstLoad && model.highCardIdentifyEnable
> && !model.isComplexes(split.index)
> && model.dimensions(split.index).isColumnar) {
> model.dimensions(split.index).isColumnar must change to
> model.primDimensions(split.index).isColumnar because model.isComplexes.length
> may be != model.dimensions.length when create table use DICTIONARY_EXCLUDE
> or column datatype is complex
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (CARBONDATA-828) Fix length issue of model.dimensions in CarbonGlobalDictionaryGenerateRDD
[
https://issues.apache.org/jira/browse/CARBONDATA-828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liang Chen updated CARBONDATA-828:
--
Summary: Fix length issue of model.dimensions in
CarbonGlobalDictionaryGenerateRDD (was: CarbonGlobalDictionaryGenerateRDD)
> Fix length issue of model.dimensions in CarbonGlobalDictionaryGenerateRDD
> -
>
> Key: CARBONDATA-828
> URL: https://issues.apache.org/jira/browse/CARBONDATA-828
> Project: CarbonData
> Issue Type: Bug
> Components: spark-integration
>Affects Versions: 1.0.0-incubating, 1.1.0-incubating
>Reporter: tianli
>Assignee: tianli
> Labels: patch
> Fix For: 1.0.0-incubating, 1.1.0-incubating
>
> Attachments: CarbonGlobalDictionaryRDD.scala
>
> Original Estimate: 1h
> Time Spent: 1h
> Remaining Estimate: 0h
>
> org.apache.carbondata.spark.rdd.CarbonGlobalDictionaryGenerateRDD find a bug:
> code line 378-380
> if (model.isFirstLoad && model.highCardIdentifyEnable
> && !model.isComplexes(split.index)
> && model.dimensions(split.index).isColumnar) {
> model.dimensions(split.index).isColumnar must change to
> model.primDimensions(split.index).isColumnar because model.isComplexes.length
> may be != model.dimensions.length when create table use DICTIONARY_EXCLUDE
> or column datatype is complex
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (CARBONDATA-828) CarbonGlobalDictionaryGenerateRDD
[
https://issues.apache.org/jira/browse/CARBONDATA-828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liang Chen reassigned CARBONDATA-828:
-
Assignee: tianli
> CarbonGlobalDictionaryGenerateRDD
> -
>
> Key: CARBONDATA-828
> URL: https://issues.apache.org/jira/browse/CARBONDATA-828
> Project: CarbonData
> Issue Type: Bug
> Components: spark-integration
>Affects Versions: 1.0.0-incubating, 1.1.0-incubating
>Reporter: tianli
>Assignee: tianli
> Labels: patch
> Fix For: 1.0.0-incubating, 1.1.0-incubating
>
> Attachments: CarbonGlobalDictionaryRDD.scala
>
> Original Estimate: 1h
> Time Spent: 1h
> Remaining Estimate: 0h
>
> org.apache.carbondata.spark.rdd.CarbonGlobalDictionaryGenerateRDD find a bug:
> code line 378-380
> if (model.isFirstLoad && model.highCardIdentifyEnable
> && !model.isComplexes(split.index)
> && model.dimensions(split.index).isColumnar) {
> model.dimensions(split.index).isColumnar must change to
> model.primDimensions(split.index).isColumnar because model.isComplexes.length
> may be != model.dimensions.length when create table use DICTIONARY_EXCLUDE
> or column datatype is complex
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (CARBONDATA-826) Create carbondata-connector for query carbon data in presto
[ https://issues.apache.org/jira/browse/CARBONDATA-826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-826: -- Summary: Create carbondata-connector for query carbon data in presto (was: Create carbondata-connector of presto for supporting presto query carbon data) > Create carbondata-connector for query carbon data in presto > --- > > Key: CARBONDATA-826 > URL: https://issues.apache.org/jira/browse/CARBONDATA-826 > Project: CarbonData > Issue Type: Sub-task > Components: presto-integration >Reporter: Liang Chen >Assignee: Liang Chen >Priority: Minor > > 1.In CarbonData project, generate carbondata-connector of presto > 2.Copy carbondata-connector to presto/plugin/ > 3.Run query in presto to read carbon data. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CARBONDATA-826) Create carbondata-connector of presto for supporting presto query carbon data
Liang Chen created CARBONDATA-826: - Summary: Create carbondata-connector of presto for supporting presto query carbon data Key: CARBONDATA-826 URL: https://issues.apache.org/jira/browse/CARBONDATA-826 Project: CarbonData Issue Type: Sub-task Components: presto-integration Reporter: Liang Chen Assignee: Liang Chen Priority: Minor 1.In CarbonData project, generate carbondata-connector of presto 2.Copy carbondata-connector to presto/plugin/ 3.Run query in presto to read carbon data. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-801) [Documentation] Examples format to be fixed
[ https://issues.apache.org/jira/browse/CARBONDATA-801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-801. --- Resolution: Fixed Fix Version/s: 1.1.0-incubating > [Documentation] Examples format to be fixed > --- > > Key: CARBONDATA-801 > URL: https://issues.apache.org/jira/browse/CARBONDATA-801 > Project: CarbonData > Issue Type: Bug >Reporter: Gururaj Shetty >Assignee: Srinath Thota >Priority: Minor > Fix For: 1.1.0-incubating > > Time Spent: 1h > Remaining Estimate: 0h > > Some examples provided in DDL are enclosed in “” which might not work in some > scenarios. Need to replace the “” in the examples to ‘’. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (CARBONDATA-801) [Documentation] Examples format to be fixed
[ https://issues.apache.org/jira/browse/CARBONDATA-801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen reassigned CARBONDATA-801: - Assignee: Srinath Thota (was: Gururaj Shetty) > [Documentation] Examples format to be fixed > --- > > Key: CARBONDATA-801 > URL: https://issues.apache.org/jira/browse/CARBONDATA-801 > Project: CarbonData > Issue Type: Bug >Reporter: Gururaj Shetty >Assignee: Srinath Thota >Priority: Minor > Time Spent: 1h > Remaining Estimate: 0h > > Some examples provided in DDL are enclosed in “” which might not work in some > scenarios. Need to replace the “” in the examples to ‘’. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-813) Fix pom issues and add the correct dependency jar to build success for integration/presto
[ https://issues.apache.org/jira/browse/CARBONDATA-813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-813. --- Resolution: Fixed Fix Version/s: 1.2.0 > Fix pom issues and add the correct dependency jar to build success for > integration/presto > - > > Key: CARBONDATA-813 > URL: https://issues.apache.org/jira/browse/CARBONDATA-813 > Project: CarbonData > Issue Type: Sub-task > Components: presto-integration >Reporter: Liang Chen >Assignee: Liang Chen >Priority: Minor > Fix For: 1.2.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Fix all issues to build success for integration/presto -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CARBONDATA-727) Hive integration
[ https://issues.apache.org/jira/browse/CARBONDATA-727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-727: -- Component/s: (was: hadoop-integration) hive-integration > Hive integration > > > Key: CARBONDATA-727 > URL: https://issues.apache.org/jira/browse/CARBONDATA-727 > Project: CarbonData > Issue Type: New Feature > Components: hive-integration >Affects Versions: NONE >Reporter: cen yuhai >Assignee: cen yuhai > Time Spent: 4.5h > Remaining Estimate: 0h > > Now hive is widely used in warehouse. I think we should support hive -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CARBONDATA-817) Optimize performance by leveraging CarbonData's unique features
Liang Chen created CARBONDATA-817: - Summary: Optimize performance by leveraging CarbonData's unique features Key: CARBONDATA-817 URL: https://issues.apache.org/jira/browse/CARBONDATA-817 Project: CarbonData Issue Type: Sub-task Reporter: Liang Chen Optimize performance by leveraging CarbonData's unique features, like: index and dictionary encoding etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CARBONDATA-815) Add basic hive integration code
[ https://issues.apache.org/jira/browse/CARBONDATA-815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-815: -- Request participants: (was: ) Fix Version/s: 1.2.0 > Add basic hive integration code > --- > > Key: CARBONDATA-815 > URL: https://issues.apache.org/jira/browse/CARBONDATA-815 > Project: CarbonData > Issue Type: Sub-task > Components: hive-integration >Reporter: Liang Chen >Assignee: cen yuhai > Fix For: 1.2.0 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CARBONDATA-816) Add examples for hive integration under /Examples
Liang Chen created CARBONDATA-816: - Summary: Add examples for hive integration under /Examples Key: CARBONDATA-816 URL: https://issues.apache.org/jira/browse/CARBONDATA-816 Project: CarbonData Issue Type: Sub-task Components: examples Reporter: Liang Chen Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CARBONDATA-815) Add basic hive integration code
Liang Chen created CARBONDATA-815: - Summary: Add basic hive integration code Key: CARBONDATA-815 URL: https://issues.apache.org/jira/browse/CARBONDATA-815 Project: CarbonData Issue Type: Sub-task Components: hive-integration Reporter: Liang Chen Assignee: cen yuhai -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-706) Mulitiple OR operators does not work properly in carbondata
[
https://issues.apache.org/jira/browse/CARBONDATA-706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liang Chen resolved CARBONDATA-706.
---
Resolution: Fixed
Assignee: Ravindra Pesala
Fix Version/s: 1.1.0-incubating
> Mulitiple OR operators does not work properly in carbondata
> ---
>
> Key: CARBONDATA-706
> URL: https://issues.apache.org/jira/browse/CARBONDATA-706
> Project: CarbonData
> Issue Type: Bug
> Components: sql
>Affects Versions: 1.1.0-incubating
> Environment: Spark 2.1
>Reporter: SWATI RAO
>Assignee: Ravindra Pesala
>Priority: Minor
> Fix For: 1.1.0-incubating
>
> Attachments: 100_hive_test.csv
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Incorrect result displays with multiple OR operators.
> Steps to Reproduces:
> 1:Create table using following Command
> " create table Carbon_automation (imei string,deviceInformationId int,MAC
> string,deviceColor string,device_backColor string,modelId string,marketName
> string,AMSize string,ROMSize string,CUPAudit string,CPIClocked string,series
> string,productionDate timestamp,bomCode string,internalModels string,
> deliveryTime string, channelsId string, channelsName string , deliveryAreaId
> string, deliveryCountry string, deliveryProvince string, deliveryCity
> string,deliveryDistrict string, deliveryStreet string, oxSingleNumber string,
> ActiveCheckTime string, ActiveAreaId string, ActiveCountry string,
> ActiveProvince string, Activecity string, ActiveDistrict string, ActiveStreet
> string, ActiveOperatorId string, Active_releaseId string, Active_EMUIVersion
> string, Active_operaSysVersion string, Active_BacVerNumber string,
> Active_BacFlashVer string, Active_webUIVersion string,
> Active_webUITypeCarrVer string,Active_webTypeDataVerNumber string,
> Active_operatorsVersion string, Active_phonePADPartitionedVersions string,
> Latest_YEAR int, Latest_MONTH int, Latest_DAY int, Latest_HOUR string,
> Latest_areaId string, Latest_country string, Latest_province string,
> Latest_city string, Latest_district string, Latest_street string,
> Latest_releaseId string, Latest_EMUIVersion string, Latest_operaSysVersion
> string, Latest_BacVerNumber string, Latest_BacFlashVer string,
> Latest_webUIVersion string, Latest_webUITypeCarrVer string,
> Latest_webTypeDataVerNumber string, Latest_operatorsVersion string,
> Latest_phonePADPartitionedVersions string, Latest_operatorId string,
> gamePointDescription string,gamePointId double,contractNumber
> double,imei_count int) STORED BY 'org.apache.carbondata.format' TBLPROPERTIES
> ('DICTIONARY_INCLUDE'='deviceInformationId,Latest_YEAR,Latest_MONTH,Latest_DAY')"
> 2:Load Data with following command
> " LOAD DATA INPATH 'HDFS_URL/BabuStore/Data/HiveData' INTO TABLE
> Carbon_automation
> OPTIONS('DELIMITER'=',','QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='imei,deviceInformationId,MAC,deviceColor,device_backColor,modelId,marketName,AMSize,ROMSize,CUPAudit,CPIClocked,series,productionDate,bomCode,internalModels,deliveryTime,channelsId,channelsName,deliveryAreaId,deliveryCountry,deliveryProvince,deliveryCity,deliveryDistrict,deliveryStreet,oxSingleNumber,contractNumber,ActiveCheckTime,ActiveAreaId,ActiveCountry,ActiveProvince,Activecity,ActiveDistrict,ActiveStreet,ActiveOperatorId,Active_releaseId,Active_EMUIVersion,Active_operaSysVersion,Active_BacVerNumber,Active_BacFlashVer,Active_webUIVersion,Active_webUITypeCarrVer,Active_webTypeDataVerNumber,Active_operatorsVersion,Active_phonePADPartitionedVersions,Latest_YEAR,Latest_MONTH,Latest_DAY,Latest_HOUR,Latest_areaId,Latest_country,Latest_province,Latest_city,Latest_district,Latest_street,Latest_releaseId,Latest_EMUIVersion,Latest_operaSysVersion,Latest_BacVerNumber,Latest_BacFlashVer,Latest_webUIVersion,Latest_webUITypeCarrVer,Latest_webTypeDataVerNumber,Latest_operatorsVersion,Latest_phonePADPartitionedVersions,Latest_operatorId,gamePointId,gamePointDescription,imei_count')"
> 3:Now run the Select Query:
> " select imei,gamePointId, channelsId,series from Carbon_automation where
> channelsId >=10 OR channelsId <=1 or series='7Series' "
> 4:Result Displays
> " 0: jdbc:hive2://localhost:1> select imei,gamePointId, channelsId,series
> from Carbon_automation where channelsId >=10 OR channelsId <=1 or
> series='7Series';
> +-+--+-+--+--+
> |imei | gamePointId | channelsId | series |
> +-+--+-+--+--+
> | 1AA1| 2738.562 | 4 | 7Series |
> | 1AA10 | 1714.635 | 4 | 7Series |
> | 1AA100 | 1271.0 | 6 | 5Series |
> | 1AA1000 | 692.0| 3 | 5Series |
> | 1AA1|
[jira] [Created] (CARBONDATA-813) Fix pom issues and add the correct dependency jar to build success for integration/presto
Liang Chen created CARBONDATA-813: - Summary: Fix pom issues and add the correct dependency jar to build success for integration/presto Key: CARBONDATA-813 URL: https://issues.apache.org/jira/browse/CARBONDATA-813 Project: CarbonData Issue Type: Sub-task Components: presto-integration Reporter: Liang Chen Assignee: Liang Chen Priority: Minor Fix all issues to build success for integration/presto -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CARBONDATA-808) Create PrestoExample
Liang Chen created CARBONDATA-808: - Summary: Create PrestoExample Key: CARBONDATA-808 URL: https://issues.apache.org/jira/browse/CARBONDATA-808 Project: CarbonData Issue Type: Sub-task Reporter: Liang Chen Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-807) Add the basic presto integration code
[ https://issues.apache.org/jira/browse/CARBONDATA-807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-807. --- Resolution: Fixed > Add the basic presto integration code > - > > Key: CARBONDATA-807 > URL: https://issues.apache.org/jira/browse/CARBONDATA-807 > Project: CarbonData > Issue Type: Sub-task > Components: hadoop-integration >Reporter: Liang Chen >Assignee: ffpeng > Fix For: 1.2.0 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CARBONDATA-807) Add the basic presto integration code
[ https://issues.apache.org/jira/browse/CARBONDATA-807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-807: -- Fix Version/s: 1.2.0 > Add the basic presto integration code > - > > Key: CARBONDATA-807 > URL: https://issues.apache.org/jira/browse/CARBONDATA-807 > Project: CarbonData > Issue Type: Sub-task > Components: hadoop-integration >Reporter: Liang Chen >Assignee: ffpeng > Fix For: 1.2.0 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CARBONDATA-807) Add the basic presto integration code
Liang Chen created CARBONDATA-807: - Summary: Add the basic presto integration code Key: CARBONDATA-807 URL: https://issues.apache.org/jira/browse/CARBONDATA-807 Project: CarbonData Issue Type: Sub-task Reporter: Liang Chen Assignee: ffpeng -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CARBONDATA-806) Aggregation query optimization
Liang Chen created CARBONDATA-806: - Summary: Aggregation query optimization Key: CARBONDATA-806 URL: https://issues.apache.org/jira/browse/CARBONDATA-806 Project: CarbonData Issue Type: Sub-task Components: presto-integration Reporter: Liang Chen Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CARBONDATA-805) Fix groupid,package name,Class name issues
Liang Chen created CARBONDATA-805: - Summary: Fix groupid,package name,Class name issues Key: CARBONDATA-805 URL: https://issues.apache.org/jira/browse/CARBONDATA-805 Project: CarbonData Issue Type: Sub-task Components: presto-integration Reporter: Liang Chen Assignee: Liang Chen Priority: Minor Fix groupid,package name,Class name issues -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-799) change word from currenr to current
[ https://issues.apache.org/jira/browse/CARBONDATA-799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-799. --- Resolution: Fixed Fix Version/s: 1.0.1-incubating 1.1.0-incubating > change word from currenr to current > --- > > Key: CARBONDATA-799 > URL: https://issues.apache.org/jira/browse/CARBONDATA-799 > Project: CarbonData > Issue Type: Improvement >Reporter: Jarck >Assignee: Jarck >Priority: Minor > Fix For: 1.1.0-incubating, 1.0.1-incubating > > Time Spent: 40m > Remaining Estimate: 0h > > change word from currenr to current -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-793) Count with null values is giving wrong result.
[ https://issues.apache.org/jira/browse/CARBONDATA-793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-793. --- Resolution: Fixed Assignee: Ravindra Pesala Fix Version/s: 1.0.1-incubating 1.1.0-incubating > Count with null values is giving wrong result. > -- > > Key: CARBONDATA-793 > URL: https://issues.apache.org/jira/browse/CARBONDATA-793 > Project: CarbonData > Issue Type: Bug >Reporter: Ravindra Pesala >Assignee: Ravindra Pesala >Priority: Minor > Fix For: 1.1.0-incubating, 1.0.1-incubating > > Time Spent: 50m > Remaining Estimate: 0h > > if the data has null values then it should not count the data. But it is > counting now. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-758) remove kettle related code in CarbonExample.scala
[ https://issues.apache.org/jira/browse/CARBONDATA-758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-758. --- Resolution: Fixed Fix Version/s: 1.1.0-incubating > remove kettle related code in CarbonExample.scala > - > > Key: CARBONDATA-758 > URL: https://issues.apache.org/jira/browse/CARBONDATA-758 > Project: CarbonData > Issue Type: Improvement >Reporter: sehriff >Assignee: sehriff >Priority: Minor > Fix For: 1.1.0-incubating > > Time Spent: 50m > Remaining Estimate: 0h > > https://github.com/apache/incubator-carbondata/pull/648 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CARBONDATA-763) Add L5 loading support, global sorting like HBase
[ https://issues.apache.org/jira/browse/CARBONDATA-763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15906934#comment-15906934 ] Liang Chen commented on CARBONDATA-763: --- Can you provide more information about "L5 loading support" ? > Add L5 loading support, global sorting like HBase > - > > Key: CARBONDATA-763 > URL: https://issues.apache.org/jira/browse/CARBONDATA-763 > Project: CarbonData > Issue Type: New Feature >Reporter: QiangCai > > Add L5 loading support, global sorting like HBase -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (CARBONDATA-755) Confusing comment about default kettle_use
[ https://issues.apache.org/jira/browse/CARBONDATA-755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen reassigned CARBONDATA-755: - Assignee: sehriff > Confusing comment about default kettle_use > -- > > Key: CARBONDATA-755 > URL: https://issues.apache.org/jira/browse/CARBONDATA-755 > Project: CarbonData > Issue Type: Bug > Components: examples >Reporter: sehriff >Assignee: sehriff >Priority: Minor > Labels: patch > Fix For: 1.1.0-incubating, 1.0.1-incubating > > Time Spent: 10m > Remaining Estimate: 0h > > Fix comment issues of CarbonExample.scala about default use_kettle > https://github.com/apache/incubator-carbondata/pull/640/commits -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CARBONDATA-755) Confusing comment about default kettle_use
[ https://issues.apache.org/jira/browse/CARBONDATA-755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-755: -- Request participants: (was: ) Fix Version/s: 1.0.1-incubating 1.1.0-incubating Description: Fix comment issues of CarbonExample.scala about default use_kettle https://github.com/apache/incubator-carbondata/pull/640/commits was:https://github.com/apache/incubator-carbondata/pull/640/commits Issue Type: Bug (was: Test) > Confusing comment about default kettle_use > -- > > Key: CARBONDATA-755 > URL: https://issues.apache.org/jira/browse/CARBONDATA-755 > Project: CarbonData > Issue Type: Bug > Components: examples >Reporter: sehriff >Priority: Minor > Labels: patch > Fix For: 1.1.0-incubating, 1.0.1-incubating > > Time Spent: 10m > Remaining Estimate: 0h > > Fix comment issues of CarbonExample.scala about default use_kettle > https://github.com/apache/incubator-carbondata/pull/640/commits -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CARBONDATA-755) Confusing comment about default kettle_use
[ https://issues.apache.org/jira/browse/CARBONDATA-755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904619#comment-15904619 ] Liang Chen commented on CARBONDATA-755: --- Hi sehriff Thank you started to fix this issue. Please let me know your jira account's email id, i will give you contributor right, then you can assign this issue to your self. > Confusing comment about default kettle_use > -- > > Key: CARBONDATA-755 > URL: https://issues.apache.org/jira/browse/CARBONDATA-755 > Project: CarbonData > Issue Type: Test > Components: examples >Reporter: sehriff >Priority: Minor > Labels: patch > Time Spent: 10m > Remaining Estimate: 0h > > https://github.com/apache/incubator-carbondata/pull/640/commits -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-714) DOCUMENTATION - How to handle the bad records
[ https://issues.apache.org/jira/browse/CARBONDATA-714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-714. --- Resolution: Fixed Fix Version/s: 1.0.1-incubating > DOCUMENTATION - How to handle the bad records > - > > Key: CARBONDATA-714 > URL: https://issues.apache.org/jira/browse/CARBONDATA-714 > Project: CarbonData > Issue Type: Improvement >Reporter: Gururaj Shetty >Assignee: Pallavi Singh >Priority: Minor > Fix For: 1.0.1-incubating > > Time Spent: 1h 40m > Remaining Estimate: 0h > > A TroubleShooting topic can be added on how to handle the bad records: > Some of the solution which can be captures are: > 1. Writing to CSV. What are the properties user need to set > 2. Null > 3. Fail when there is a bad records > Etc -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CARBONDATA-714) DOCUMENTATION - How to handle the bad records
[ https://issues.apache.org/jira/browse/CARBONDATA-714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-714: -- Summary: DOCUMENTATION - How to handle the bad records (was: DOCUMENTATION - How to handle the bad records can be documented) > DOCUMENTATION - How to handle the bad records > - > > Key: CARBONDATA-714 > URL: https://issues.apache.org/jira/browse/CARBONDATA-714 > Project: CarbonData > Issue Type: Improvement >Reporter: Gururaj Shetty >Assignee: Pallavi Singh >Priority: Minor > Fix For: 1.0.1-incubating > > Time Spent: 1h 40m > Remaining Estimate: 0h > > A TroubleShooting topic can be added on how to handle the bad records: > Some of the solution which can be captures are: > 1. Writing to CSV. What are the properties user need to set > 2. Null > 3. Fail when there is a bad records > Etc -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CARBONDATA-753) Fix Date and Timestamp format issues
[ https://issues.apache.org/jira/browse/CARBONDATA-753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-753: -- Description: Fix Date and Timestamp format issues: 1.Optimize the description of CARBON_TIMESTAMP_FORMAT_DATE_FORMAT in CarbonCommonConstants.java 2.Correct fields' definition of Date and Timestamp in examples. 3.Add example script how to show raw data's timestamp format. currently spark.sql.show() by default using "-mm-dd hh:mm:ss.f" as Timestamp.toString() format, users always wanting the show data same as raw data format. was: Fix Date and Timestamp format issues: 1.Optimize the description of CARBON_TIMESTAMP_FORMAT_DATE_FORMAT in CarbonCommonConstants.java 2.Correct fields' definition of Date and Timestamp in examples. 3.Add example, how to show raw data's timestamp format. currently spark.sql.show() by default using "-mm-dd hh:mm:ss.f" as Timestamp.toString() format, users always wanting the raw data format. > Fix Date and Timestamp format issues > > > Key: CARBONDATA-753 > URL: https://issues.apache.org/jira/browse/CARBONDATA-753 > Project: CarbonData > Issue Type: Bug > Components: core, examples >Affects Versions: 1.0.0-incubating >Reporter: Liang Chen >Assignee: Liang Chen >Priority: Minor > Fix For: 1.1.0-incubating, 1.0.1-incubating > > > Fix Date and Timestamp format issues: > 1.Optimize the description of CARBON_TIMESTAMP_FORMAT_DATE_FORMAT in > CarbonCommonConstants.java > 2.Correct fields' definition of Date and Timestamp in examples. > 3.Add example script how to show raw data's timestamp format. currently > spark.sql.show() by default using "-mm-dd hh:mm:ss.f" as > Timestamp.toString() format, users always wanting the show data same as raw > data format. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CARBONDATA-753) Fix Date and Timestamp format issues
[ https://issues.apache.org/jira/browse/CARBONDATA-753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-753: -- Description: Fix Date and Timestamp format issues: 1.Optimize the description of CARBON_TIMESTAMP_FORMAT_DATE_FORMAT in CarbonCommonConstants.java 2.Correct fields' definition of Date and Timestamp in examples. 3.Add example, how to show raw data's timestamp format. currently spark.sql.show() by default using "-mm-dd hh:mm:ss.f" as Timestamp.toString() format, users always wanting the raw data format. was: Fix Date and Timestamp format issues: 1.Optimize the description of CARBON_TIMESTAMP_FORMAT_DATE_FORMAT in CarbonCommonConstants.java 2.Correct filed definition of Date and Timestamp in examples. 3.Add example, how to show raw data's timestamp format. currently spark.sql.show() by default using "-mm-dd hh:mm:ss.f" as Timestamp.toString() format, users always wanting the raw data format. > Fix Date and Timestamp format issues > > > Key: CARBONDATA-753 > URL: https://issues.apache.org/jira/browse/CARBONDATA-753 > Project: CarbonData > Issue Type: Bug > Components: core, examples >Affects Versions: 1.0.0-incubating >Reporter: Liang Chen >Assignee: Liang Chen >Priority: Minor > Fix For: 1.1.0-incubating, 1.0.1-incubating > > > Fix Date and Timestamp format issues: > 1.Optimize the description of CARBON_TIMESTAMP_FORMAT_DATE_FORMAT in > CarbonCommonConstants.java > 2.Correct fields' definition of Date and Timestamp in examples. > 3.Add example, how to show raw data's timestamp format. currently > spark.sql.show() by default using "-mm-dd hh:mm:ss.f" as > Timestamp.toString() format, users always wanting the raw data format. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CARBONDATA-753) Fix Date and Timestamp format issues
Liang Chen created CARBONDATA-753: - Summary: Fix Date and Timestamp format issues Key: CARBONDATA-753 URL: https://issues.apache.org/jira/browse/CARBONDATA-753 Project: CarbonData Issue Type: Bug Components: core, examples Affects Versions: 1.0.0-incubating Reporter: Liang Chen Assignee: Liang Chen Priority: Minor Fix For: 1.1.0-incubating, 1.0.1-incubating Fix Date and Timestamp format issues: 1.Optimize the description of CARBON_TIMESTAMP_FORMAT_DATE_FORMAT in CarbonCommonConstants.java 2.Correct filed definition of Date and Timestamp in examples. 3.Add example, how to show raw data's timestamp format. currently spark.sql.show() by default using "-mm-dd hh:mm:ss.f" as Timestamp.toString() format, users always wanting the raw data format. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CARBONDATA-745) Does carbondata apply to scenes that need to sort historical and current data?
[ https://issues.apache.org/jira/browse/CARBONDATA-745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899315#comment-15899315 ] Liang Chen commented on CARBONDATA-745: --- First, see if i understood it: There is a sorted table, and will generate a new table with real-time data. the two tables have one same column(eg: name is key), you want to do join by column "key" to merge two tables into one table, complete sort operation along with joining operation , is it correct ? During loading data, can't do like that. For two different table data, you must manually do join operation as per your requires. Regards Liang > Does carbondata apply to scenes that need to sort historical and current data? > -- > > Key: CARBONDATA-745 > URL: https://issues.apache.org/jira/browse/CARBONDATA-745 > Project: CarbonData > Issue Type: Wish > Components: data-load >Affects Versions: 1.0.0-incubating >Reporter: ke xu >Priority: Minor > > Does carbondata apply to scenes that need to sort historical and current data? > Now there is a new scene that needs to sort the current data and historical > data in real time according to certain rules,and returns the sorted data when > querying. > It's just like hbase put data sorted by rowkey. > We want to sort the data when we load the data without having to spend extra > time to sort。 > Is it suitable for this scene? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CARBONDATA-750) Improve exception information description while user input wrong creation table script
Liang Chen created CARBONDATA-750:
-
Summary: Improve exception information description while user
input wrong creation table script
Key: CARBONDATA-750
URL: https://issues.apache.org/jira/browse/CARBONDATA-750
Project: CarbonData
Issue Type: Improvement
Components: sql
Reporter: Liang Chen
Priority: Minor
1. Use wrong creation table script:
scala> carbon.sql("CREATE TABLE carbontable1 (id,int,age string,year,int)
STORED BY 'carbondata'")
java.lang.RuntimeException: [1.1] failure: identifier matching regex (?i)ALTER
expected
CREATE TABLE carbontable1 (id,int,age string,year,int) STORED BY 'carbondata'
^
at scala.sys.package$.error(package.scala:27)
at
org.apache.spark.sql.parser.CarbonSpark2SqlParser.parse(CarbonSpark2SqlParser.scala:45)
at
org.apache.spark.sql.parser.CarbonSparkSqlParser.parsePlan(CarbonSparkSqlParser.scala:51)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592)
2.Need improve the exception information description, like : unexpected ","
found
CREATE TABLE carbontable1 (id,int,age string,year,int) STORED BY
^
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (CARBONDATA-749) Unexpected error log message while dropping carbon table
Liang Chen created CARBONDATA-749:
-
Summary: Unexpected error log message while dropping carbon table
Key: CARBONDATA-749
URL: https://issues.apache.org/jira/browse/CARBONDATA-749
Project: CarbonData
Issue Type: Bug
Components: sql
Affects Versions: 1.0.0-incubating
Reporter: Liang Chen
Priority: Minor
1.Create a table with the below script:
carbon.sql("CREATE TABLE carbontable1 (id int, age string, year string) STORED
BY 'carbondata'")
2.Drop table "carbontable1" with the below script:
carbon.sql("drop table carbontable1")
Unexpected error log message as below:
AUDIT 07-03 07:50:11,944 - [AppledeMacBook-Pro.local][apple][Thread-1]Deleting
table [carbontable1] under database [default]
AUDIT 07-03 07:50:12,086 - [AppledeMacBook-Pro.local][apple][Thread-1]Creating
Table with Database name [default] and Table name [carbontable1]
AUDIT 07-03 07:50:12,095 - [AppledeMacBook-Pro.local][apple][Thread-1]Table
creation with Database name [default] and Table name [carbontable1] failed.
Table [carbontable1] already exists under database [default]
WARN 07-03 07:50:12,095 -
org.spark_project.guava.util.concurrent.UncheckedExecutionException:
java.lang.RuntimeException: Table [carbontable1] already exists under database
[default]
org.spark_project.guava.util.concurrent.UncheckedExecutionException:
java.lang.RuntimeException: Table [carbontable1] already exists under database
[default]
at
org.spark_project.guava.cache.LocalCache$Segment.get(LocalCache.java:2263)
at org.spark_project.guava.cache.LocalCache.get(LocalCache.java:4000)
at
org.spark_project.guava.cache.LocalCache.getOrLoad(LocalCache.java:4004)
at
org.spark_project.guava.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874)
at
org.spark_project.guava.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4880)
at
org.spark_project.guava.cache.LocalCache$LocalLoadingCache.apply(LocalCache.java:4898)
at
org.apache.spark.sql.hive.HiveMetastoreCatalog.lookupRelation(HiveMetastoreCatalog.scala:110)
at
org.apache.spark.sql.hive.HiveSessionCatalog.lookupRelation(HiveSessionCatalog.scala:69)
at org.apache.spark.sql.SparkSession.table(SparkSession.scala:578)
at org.apache.spark.sql.SparkSession.table(SparkSession.scala:574)
at
org.apache.spark.sql.execution.command.DropTableCommand.run(ddl.scala:203)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at
org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)
at
org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:87)
at
org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:87)
at org.apache.spark.sql.Dataset.(Dataset.scala:185)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592)
at
org.apache.spark.sql.hive.CarbonHiveMetadataUtil$.invalidateAndDropTable(CarbonHiveMetadataUtil.scala:44)
at
org.apache.spark.sql.hive.CarbonMetastore.dropTable(CarbonMetastore.scala:435)
at
org.apache.spark.sql.execution.command.CarbonDropTableCommand.run(carbonTableSchema.scala:665)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at
org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132)
at
[jira] [Resolved] (CARBONDATA-719) Add a release guide in documentation
[ https://issues.apache.org/jira/browse/CARBONDATA-719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-719. --- Resolution: Fixed Fix Version/s: 1.0.1-incubating 1.1.0-incubating > Add a release guide in documentation > > > Key: CARBONDATA-719 > URL: https://issues.apache.org/jira/browse/CARBONDATA-719 > Project: CarbonData > Issue Type: Task > Components: docs >Reporter: Jean-Baptiste Onofré >Assignee: Jean-Baptiste Onofré > Fix For: 1.1.0-incubating, 1.0.1-incubating > > Time Spent: 1h 40m > Remaining Estimate: 0h > > We have to provide a release guide explaining the steps to perform a > CarbonData release. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (CARBONDATA-722) Add direct links to contribution guide, mailing lists, source repositories and Jira
[ https://issues.apache.org/jira/browse/CARBONDATA-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15888263#comment-15888263 ] Liang Chen edited comment on CARBONDATA-722 at 2/28/17 3:42 PM: Already added apache jira link to website was (Author: chenliang613): Already add apache jira link to website > Add direct links to contribution guide, mailing lists, source repositories > and Jira > --- > > Key: CARBONDATA-722 > URL: https://issues.apache.org/jira/browse/CARBONDATA-722 > Project: CarbonData > Issue Type: Task > Components: website >Reporter: Jean-Baptiste Onofré >Assignee: Jean-Baptiste Onofré > > In order to facilitate contribution, we should provide direct links and > visibility to the contribution guide, mailing lists, source repositories and > Jira directly on the website. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CARBONDATA-722) Add direct links to contribution guide, mailing lists, source repositories and Jira
[ https://issues.apache.org/jira/browse/CARBONDATA-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15888263#comment-15888263 ] Liang Chen commented on CARBONDATA-722: --- Already add apache jira link to website > Add direct links to contribution guide, mailing lists, source repositories > and Jira > --- > > Key: CARBONDATA-722 > URL: https://issues.apache.org/jira/browse/CARBONDATA-722 > Project: CarbonData > Issue Type: Task > Components: website >Reporter: Jean-Baptiste Onofré >Assignee: Jean-Baptiste Onofré > > In order to facilitate contribution, we should provide direct links and > visibility to the contribution guide, mailing lists, source repositories and > Jira directly on the website. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-733) Fixed testcase failure issue
[ https://issues.apache.org/jira/browse/CARBONDATA-733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-733. --- Resolution: Fixed Fix Version/s: 1.0.1-incubating > Fixed testcase failure issue > - > > Key: CARBONDATA-733 > URL: https://issues.apache.org/jira/browse/CARBONDATA-733 > Project: CarbonData > Issue Type: Bug >Reporter: kumar vishal >Assignee: kumar vishal > Fix For: 1.0.1-incubating > > Time Spent: 1h > Remaining Estimate: 0h > > Fixed testcase failure issue -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-700) invalid example of no_inverted_index in carbondata ddl docs
[
https://issues.apache.org/jira/browse/CARBONDATA-700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liang Chen resolved CARBONDATA-700.
---
Resolution: Fixed
Fix Version/s: 1.0.1-incubating
> invalid example of no_inverted_index in carbondata ddl docs
> ---
>
> Key: CARBONDATA-700
> URL: https://issues.apache.org/jira/browse/CARBONDATA-700
> Project: CarbonData
> Issue Type: Bug
>Reporter: anubhav tarar
>Assignee: anubhav tarar
> Fix For: 1.0.1-incubating
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> example given in carbon docs ddl is wrong
> 0: jdbc:hive2://localhost:1> CREATE TABLE IF NOT EXISTS
> productSchema.productSalesTable ( productNumber Int, productName String,
> storeCity String, storeProvince String, productCategory String, productBatch
> String, saleQuantity Int, revenue Int) STORED BY 'carbondata' TBLPROPERTIES
> ('COLUMN_GROUPS'='(productName,productCategory)',
> 'DICTIONARY_EXCLUDE'='productName', 'DICTIONARY_INCLUDE'='productNumber',
> 'NO_INVERTED_INDEX'='productBatch');
> Error: org.apache.carbondata.spark.exception.MalformedCarbonCommandException:
> Column group is not supported for no dictionary columns:productname
> (state=,code=0) and further more column group should be contigous as well,so
> there are two issues
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-718) All files have to contain Apache license header
[
https://issues.apache.org/jira/browse/CARBONDATA-718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liang Chen resolved CARBONDATA-718.
---
Resolution: Fixed
Fix Version/s: 1.0.1-incubating
> All files have to contain Apache license header
> ---
>
> Key: CARBONDATA-718
> URL: https://issues.apache.org/jira/browse/CARBONDATA-718
> Project: CarbonData
> Issue Type: Bug
> Components: core
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
> Fix For: 1.0.1-incubating
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Executing rat plugin shows several files without ASF headers, especially
> {{integration/spark2}} scala files, {{README.md}} file, ...
> I gonna fix that and include rat in our build (with some exclusion).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (CARBONDATA-728) Support integration with presto
Liang Chen created CARBONDATA-728: - Summary: Support integration with presto Key: CARBONDATA-728 URL: https://issues.apache.org/jira/browse/CARBONDATA-728 Project: CarbonData Issue Type: New Feature Components: hadoop-integration Reporter: Liang Chen CarbonData as data store, support integration with presto -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CARBONDATA-727) Hive integration
[ https://issues.apache.org/jira/browse/CARBONDATA-727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-727: -- Issue Type: New Feature (was: Bug) > Hive integration > > > Key: CARBONDATA-727 > URL: https://issues.apache.org/jira/browse/CARBONDATA-727 > Project: CarbonData > Issue Type: New Feature > Components: hadoop-integration >Affects Versions: NONE >Reporter: cen yuhai > > Now hive is widely used in warehouse. I think we should support hive -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (CARBONDATA-709) Incorrect documentation for bucketing in ddl section
[
https://issues.apache.org/jira/browse/CARBONDATA-709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liang Chen reassigned CARBONDATA-709:
-
Assignee: anubhav tarar
> Incorrect documentation for bucketing in ddl section
>
>
> Key: CARBONDATA-709
> URL: https://issues.apache.org/jira/browse/CARBONDATA-709
> Project: CarbonData
> Issue Type: Bug
> Components: docs
>Affects Versions: 1.0.0-incubating
>Reporter: anubhav tarar
>Assignee: anubhav tarar
> Fix For: 1.0.1-incubating
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> in docs ddl bucketing section Columns in the BUCKETCOLUMN parameter must be
> either a dimension or a measure but combination of both is not supported.
> this line is incorrect here is the example
> 0: jdbc:hive2://localhost:1> CREATE TABLE uniqData_t11(ID Int,name
> string)stored by 'carbondata'
> TBLPROPERTIES("DICTIONARY_EXCLUDE"="name","bucketnumber"="1",
> "bucketcolumns"="ID");
> Error: java.lang.RuntimeException: Bucket field must be dimension column and
> should not be measure or complex column: ID (state=,code=0)
> so bucketing coloumn should be dimension only
> plus in parameter description tableName is not required and the example that
> added for bucketing is also wrong
> 0: jdbc:hive2://localhost:1> CREATE TABLE IF NOT EXISTS
> productSchema.productSalesTable ( productNumber Int, productName String,
> storeCity String, storeProvince String, productCategory String, productBatch
> String, saleQuantity Int, revenue Int) STORED BY 'carbondata' TBLPROPERTIES
> ('COLUMN_GROUPS'='(productName,productCategory)',
> 'DICTIONARY_EXCLUDE'='productName', 'DICTIONARY_INCLUDE'='productNumber',
> 'NO_INVERTED_INDEX'='productBatch', 'BUCKETNUMBER'='4',
> 'BUCKETCOLUMNS'='productNumber,saleQuantity');
> Error: org.apache.carbondata.spark.exception.MalformedCarbonCommandException:
> Invalid column group,column in group should be contiguous as per schema.
> (state=,code=0)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (CARBONDATA-713) Use store location in properties when user didn't pass the location as the parameter
[ https://issues.apache.org/jira/browse/CARBONDATA-713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15875266#comment-15875266 ] Liang Chen edited comment on CARBONDATA-713 at 2/21/17 2:47 AM: Hi Can you explain more, why this is a bug? System uses "/carbon.store" as default store location if users don't specify the parameter. was (Author: chenliang613): Hi Can you explain more, why this is a bug? System uses "/carbon.store" as default store location if users don't specify the parameter. > Use store location in properties when user didn't pass the location as the > parameter > > > Key: CARBONDATA-713 > URL: https://issues.apache.org/jira/browse/CARBONDATA-713 > Project: CarbonData > Issue Type: Bug > Components: spark-integration >Affects Versions: 1.1.0-incubating >Reporter: Yadong Qi > > The store location of carbon comes from three places: > 1. default location path in code(../carbon.store) > 2. configurate "carbon.storelocation" in carbon.properties > 3. pass the location as the parameter > The priority is low to high. > But when I create a CarbonContext or CarbonSession without any parameters and > configurate "carbon.storelocation" in carbon.properties, the final value of > location is defalut(../carbon.store) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CARBONDATA-713) Use store location in properties when user didn't pass the location as the parameter
[ https://issues.apache.org/jira/browse/CARBONDATA-713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15875266#comment-15875266 ] Liang Chen commented on CARBONDATA-713: --- Hi Can you explain more, why this is a bug? System uses "/carbon.store" as default store location if users don't specify the parameter. > Use store location in properties when user didn't pass the location as the > parameter > > > Key: CARBONDATA-713 > URL: https://issues.apache.org/jira/browse/CARBONDATA-713 > Project: CarbonData > Issue Type: Bug > Components: spark-integration >Affects Versions: 1.1.0-incubating >Reporter: Yadong Qi > > The store location of carbon comes from three places: > 1. default location path in code(../carbon.store) > 2. configurate "carbon.storelocation" in carbon.properties > 3. pass the location as the parameter > The priority is low to high. > But when I create a CarbonContext or CarbonSession without any parameters and > configurate "carbon.storelocation" in carbon.properties, the final value of > location is defalut(../carbon.store) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-687) Updated Documentation for New Features in Release 1.0.0
[ https://issues.apache.org/jira/browse/CARBONDATA-687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-687. --- Resolution: Fixed Assignee: Pallavi Singh Fix Version/s: 1.0.1-incubating 1.1.0-incubating > Updated Documentation for New Features in Release 1.0.0 > --- > > Key: CARBONDATA-687 > URL: https://issues.apache.org/jira/browse/CARBONDATA-687 > Project: CarbonData > Issue Type: Improvement > Components: docs >Affects Versions: 1.0.0-incubating >Reporter: Pallavi Singh >Assignee: Pallavi Singh >Priority: Minor > Fix For: 1.1.0-incubating, 1.0.1-incubating > > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CARBONDATA-703) Update build command after optimizing thrift compile issues
Liang Chen created CARBONDATA-703: - Summary: Update build command after optimizing thrift compile issues Key: CARBONDATA-703 URL: https://issues.apache.org/jira/browse/CARBONDATA-703 Project: CarbonData Issue Type: Bug Reporter: Liang Chen Assignee: Liang Chen Priority: Minor Fix For: 1.0.1-incubating Update build command after optimizing thrift compile issues -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-702) Created carbondata repository with adding format jar for facilitating compile
[ https://issues.apache.org/jira/browse/CARBONDATA-702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-702. --- Resolution: Fixed > Created carbondata repository with adding format jar for facilitating compile > - > > Key: CARBONDATA-702 > URL: https://issues.apache.org/jira/browse/CARBONDATA-702 > Project: CarbonData > Issue Type: Bug > Components: file-format >Reporter: Ravindra Pesala >Assignee: Ravindra Pesala >Priority: Minor > Fix For: 1.0.1-incubating > > Time Spent: 50m > Remaining Estimate: 0h > > Create carbondata repository to keep format jar for facilitating compile. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CARBONDATA-702) Created carbondata repository with adding format jar for facilitating compile
[ https://issues.apache.org/jira/browse/CARBONDATA-702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-702: -- Description: Create carbondata repository to keep format jar for facilitating compile. (was: Create carbondata repository to keep format jar. At the time of IPMC voting, the format jar will be downloaded from this reoository, so IPMC no need to install thrift to build the carbondata.) > Created carbondata repository with adding format jar for facilitating compile > - > > Key: CARBONDATA-702 > URL: https://issues.apache.org/jira/browse/CARBONDATA-702 > Project: CarbonData > Issue Type: Bug > Components: file-format >Reporter: Ravindra Pesala >Assignee: Ravindra Pesala >Priority: Minor > Fix For: 1.0.1-incubating > > Time Spent: 50m > Remaining Estimate: 0h > > Create carbondata repository to keep format jar for facilitating compile. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (CARBONDATA-702) Created carbondata repository with adding format jar for facilitating compile
[ https://issues.apache.org/jira/browse/CARBONDATA-702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen reassigned CARBONDATA-702: - Assignee: Ravindra Pesala Request participants: (was: ) Fix Version/s: 1.0.1-incubating Component/s: file-format Summary: Created carbondata repository with adding format jar for facilitating compile (was: Create carbondata repository to keep format jar) > Created carbondata repository with adding format jar for facilitating compile > - > > Key: CARBONDATA-702 > URL: https://issues.apache.org/jira/browse/CARBONDATA-702 > Project: CarbonData > Issue Type: Bug > Components: file-format >Reporter: Ravindra Pesala >Assignee: Ravindra Pesala >Priority: Minor > Fix For: 1.0.1-incubating > > Time Spent: 50m > Remaining Estimate: 0h > > Create carbondata repository to keep format jar. At the time of IPMC voting, > the format jar will be downloaded from this reoository, so IPMC no need to > install thrift to build the carbondata. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
