[jira] [Created] (CARBONDATA-976) Wrong entry getting deleted from schemaEvolution during alter revert
Kunal Kapoor created CARBONDATA-976: --- Summary: Wrong entry getting deleted from schemaEvolution during alter revert Key: CARBONDATA-976 URL: https://issues.apache.org/jira/browse/CARBONDATA-976 Project: CarbonData Issue Type: Bug Reporter: Kunal Kapoor Assignee: Kunal Kapoor -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CARBONDATA-1383) Handle findbugs related to ExecutorService.submit(Callable) unhandled future value
Kunal Kapoor created CARBONDATA-1383: Summary: Handle findbugs related to ExecutorService.submit(Callable) unhandled future value Key: CARBONDATA-1383 URL: https://issues.apache.org/jira/browse/CARBONDATA-1383 Project: CarbonData Issue Type: Sub-task Reporter: Kunal Kapoor Priority: Trivial Future values needs to be handled for exception or interrupted scenarios. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1305) On creating the dictinary with large dictionary csv NegativeArraySizeException is thrown
Kunal Kapoor created CARBONDATA-1305:
Summary: On creating the dictinary with large dictionary csv
NegativeArraySizeException is thrown
Key: CARBONDATA-1305
URL: https://issues.apache.org/jira/browse/CARBONDATA-1305
Project: CarbonData
Issue Type: Bug
Reporter: Kunal Kapoor
Assignee: Kunal Kapoor
Step to reproduce:
1. create table table1 (a string, b bigint) stored by 'carbondata';
2. LOAD DATA inpath 'hdfs://hacluster/user/xyz/datafile_0.csv' into table
table1 options('DELIMITER'=',',
'QUOTECHAR'='"','COLUMNDICT'='a:hdfs://hacluster/user/xyz/dict.csv','FILEHEADER'='a,b','SINGLE_PASS'='TRUE');
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (CARBONDATA-1229) Restrict Drop table if load is in progress
[ https://issues.apache.org/jira/browse/CARBONDATA-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor updated CARBONDATA-1229: - Description: (was: Steps to reproduce:- 1. Create table 2. Load data using single_pass 3. Delete table 4. recreate table with the same name) Issue Type: Improvement (was: Bug) Summary: Restrict Drop table if load is in progress (was: null being populated in table if user deletes table during single pass load) > Restrict Drop table if load is in progress > -- > > Key: CARBONDATA-1229 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1229 > Project: CarbonData > Issue Type: Improvement >Reporter: Kunal Kapoor >Assignee: Kunal Kapoor > Time Spent: 6h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1317) Multiple dictionary files being created in single_pass
Kunal Kapoor created CARBONDATA-1317: Summary: Multiple dictionary files being created in single_pass Key: CARBONDATA-1317 URL: https://issues.apache.org/jira/browse/CARBONDATA-1317 Project: CarbonData Issue Type: Bug Reporter: Kunal Kapoor Assignee: Kunal Kapoor Steps to reproduce:- 1. Create table 2. Load 2 times 3. Drop table 4. Create table with same name 5. Load 2 times -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (CARBONDATA-1243) Data Load Failure while loading the data from CSV
[
https://issues.apache.org/jira/browse/CARBONDATA-1243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16066403#comment-16066403
]
Kunal Kapoor edited comment on CARBONDATA-1243 at 6/28/17 12:21 PM:
This is fixed with https://github.com/apache/carbondata/pull/1107
Please verify again
was (Author: kunal):
This is fixed with https://github.com/apache/carbondata/pull/1107
> Data Load Failure while loading the data from CSV
> -
>
> Key: CARBONDATA-1243
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1243
> Project: CarbonData
> Issue Type: Bug
> Components: data-load
> Environment: Spark 1.6
> Spark 2.1
>Reporter: Prabhat Kashyap
> Attachments: 2000_UniqData.csv
>
>
> Data Load failure while loading the Data from CSV.
> Steps to reproduces:
> 1: Create a table
> CREATE TABLE uniqdata (CUST_ID int,CUST_NAME String,ACTIVE_EMUI_VERSION
> string, DOB timestamp, DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2
> bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2
> decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,INTEGER_COLUMN1
> int) STORED BY 'org.apache.carbondata.format' TBLPROPERTIES
> ("TABLE_BLOCKSIZE"= "256 MB")
> 0: jdbc:hive2://localhost:1> ;
> +-+--+
> | Result |
> +-+--+
> +-+--+
> No rows selected (3.823 seconds).
> 2:Load the data into table:
> LOAD DATA INPATH 'hdfs://localhost:54310/2000_UniqData_tabdelm.csv' into
> table uniqdata OPTIONS('DELIMITER'='/' ,
> 'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');
> 3:Error While Data Load:
> Error: java.lang.IllegalArgumentException: Pathname
> /hdfs:/localhost:54310/2000_UniqData_tabdelm.csv from
> hdfs://localhost:54310/hdfs:/localhost:54310/2000_UniqData_tabdelm.csv is not
> a valid DFS filename. (state=,code=0)
> Expected Result: Data should be loaded successfully
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CARBONDATA-1243) Data Load Failure while loading the data from CSV
[
https://issues.apache.org/jira/browse/CARBONDATA-1243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16066403#comment-16066403
]
Kunal Kapoor commented on CARBONDATA-1243:
--
This is fixed with https://github.com/apache/carbondata/pull/1107
> Data Load Failure while loading the data from CSV
> -
>
> Key: CARBONDATA-1243
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1243
> Project: CarbonData
> Issue Type: Bug
> Components: data-load
> Environment: Spark 1.6
> Spark 2.1
>Reporter: Prabhat Kashyap
> Attachments: 2000_UniqData.csv
>
>
> Data Load failure while loading the Data from CSV.
> Steps to reproduces:
> 1: Create a table
> CREATE TABLE uniqdata (CUST_ID int,CUST_NAME String,ACTIVE_EMUI_VERSION
> string, DOB timestamp, DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2
> bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2
> decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,INTEGER_COLUMN1
> int) STORED BY 'org.apache.carbondata.format' TBLPROPERTIES
> ("TABLE_BLOCKSIZE"= "256 MB")
> 0: jdbc:hive2://localhost:1> ;
> +-+--+
> | Result |
> +-+--+
> +-+--+
> No rows selected (3.823 seconds).
> 2:Load the data into table:
> LOAD DATA INPATH 'hdfs://localhost:54310/2000_UniqData_tabdelm.csv' into
> table uniqdata OPTIONS('DELIMITER'='/' ,
> 'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');
> 3:Error While Data Load:
> Error: java.lang.IllegalArgumentException: Pathname
> /hdfs:/localhost:54310/2000_UniqData_tabdelm.csv from
> hdfs://localhost:54310/hdfs:/localhost:54310/2000_UniqData_tabdelm.csv is not
> a valid DFS filename. (state=,code=0)
> Expected Result: Data should be loaded successfully
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (CARBONDATA-1243) Data Load Failure while loading the data from CSV
[
https://issues.apache.org/jira/browse/CARBONDATA-1243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16066403#comment-16066403
]
Kunal Kapoor edited comment on CARBONDATA-1243 at 6/28/17 12:21 PM:
This is fixed with https://github.com/apache/carbondata/pull/1107
was (Author: kunal):
This is fixed with https://github.com/apache/carbondata/pull/1107
> Data Load Failure while loading the data from CSV
> -
>
> Key: CARBONDATA-1243
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1243
> Project: CarbonData
> Issue Type: Bug
> Components: data-load
> Environment: Spark 1.6
> Spark 2.1
>Reporter: Prabhat Kashyap
> Attachments: 2000_UniqData.csv
>
>
> Data Load failure while loading the Data from CSV.
> Steps to reproduces:
> 1: Create a table
> CREATE TABLE uniqdata (CUST_ID int,CUST_NAME String,ACTIVE_EMUI_VERSION
> string, DOB timestamp, DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2
> bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2
> decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,INTEGER_COLUMN1
> int) STORED BY 'org.apache.carbondata.format' TBLPROPERTIES
> ("TABLE_BLOCKSIZE"= "256 MB")
> 0: jdbc:hive2://localhost:1> ;
> +-+--+
> | Result |
> +-+--+
> +-+--+
> No rows selected (3.823 seconds).
> 2:Load the data into table:
> LOAD DATA INPATH 'hdfs://localhost:54310/2000_UniqData_tabdelm.csv' into
> table uniqdata OPTIONS('DELIMITER'='/' ,
> 'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');
> 3:Error While Data Load:
> Error: java.lang.IllegalArgumentException: Pathname
> /hdfs:/localhost:54310/2000_UniqData_tabdelm.csv from
> hdfs://localhost:54310/hdfs:/localhost:54310/2000_UniqData_tabdelm.csv is not
> a valid DFS filename. (state=,code=0)
> Expected Result: Data should be loaded successfully
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (CARBONDATA-1001) Data type change should support int to long conversion
Kunal Kapoor created CARBONDATA-1001: Summary: Data type change should support int to long conversion Key: CARBONDATA-1001 URL: https://issues.apache.org/jira/browse/CARBONDATA-1001 Project: CarbonData Issue Type: Bug Reporter: Kunal Kapoor Assignee: Kunal Kapoor Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CARBONDATA-773) During parallel load multiple instances of DictionaryServer are being created.
[ https://issues.apache.org/jira/browse/CARBONDATA-773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor updated CARBONDATA-773: Environment: spark-1.6 spark-2.1 Request participants: (was: ) > During parallel load multiple instances of DictionaryServer are being created. > -- > > Key: CARBONDATA-773 > URL: https://issues.apache.org/jira/browse/CARBONDATA-773 > Project: CarbonData > Issue Type: Improvement > Environment: spark-1.6 > spark-2.1 >Reporter: Kunal Kapoor >Assignee: Kunal Kapoor >Priority: Minor > Time Spent: 2h 50m > Remaining Estimate: 0h > > During parallel load multiple instances of DictionaryServer are being created. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CARBONDATA-1064) Fix NullPointerException in Not InExpression
[ https://issues.apache.org/jira/browse/CARBONDATA-1064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor updated CARBONDATA-1064: - Description: select * from expression_test where id not in (1,2,'', NULL, ' ') In the above query Right expression value is null in NotInExpression which cased throws NullPointerException > Fix NullPointerException in Not InExpression > > > Key: CARBONDATA-1064 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1064 > Project: CarbonData > Issue Type: Bug >Reporter: Kunal Kapoor >Assignee: Kunal Kapoor >Priority: Minor > Time Spent: 50m > Remaining Estimate: 0h > > select * from expression_test where id not in (1,2,'', NULL, ' ') > In the above query Right expression value is null in NotInExpression which > cased throws NullPointerException -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CARBONDATA-1064) Fix NullPointerException in Not InExpression
[ https://issues.apache.org/jira/browse/CARBONDATA-1064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor updated CARBONDATA-1064: - Summary: Fix NullPointerException in Not InExpression (was: restrict measures in no_inverted_index option) > Fix NullPointerException in Not InExpression > > > Key: CARBONDATA-1064 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1064 > Project: CarbonData > Issue Type: Bug >Reporter: Kunal Kapoor >Assignee: Kunal Kapoor >Priority: Minor > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (CARBONDATA-1056) Data_load failure using single_pass true with spark 2.1
[
https://issues.apache.org/jira/browse/CARBONDATA-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Kapoor reassigned CARBONDATA-1056:
Assignee: Kunal Kapoor
> Data_load failure using single_pass true with spark 2.1
> ---
>
> Key: CARBONDATA-1056
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1056
> Project: CarbonData
> Issue Type: Bug
> Components: data-load
>Affects Versions: 1.1.0
> Environment: spark 2.1
>Reporter: Vandana Yadav
>Assignee: Kunal Kapoor
>Priority: Minor
> Attachments: 2000_UniqData.csv
>
>
> Data_load failure using single_pass true with spark 2.1
> Steps to reproduce:
> 1)Create Table:
> CREATE TABLE uniq_exclude_sp1 (CUST_ID int,CUST_NAME
> String,ACTIVE_EMUI_VERSION string, DOB timestamp, DOJ timestamp,
> BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 bigint,DECIMAL_COLUMN1 decimal(30,10),
> DECIMAL_COLUMN2 decimal(36,10),Double_COLUMN1 double, Double_COLUMN2
> double,INTEGER_COLUMN1 int) STORED BY 'org.apache.carbondata.format'
> TBLPROPERTIES('DICTIONARY_EXCLUDE'='CUST_NAME,ACTIVE_EMUI_VERSION');
> 2) Load Data:
> LOAD DATA INPATH 'hdfs://localhost:54310/2000_UniqData.csv' into table
> uniq_exclude_sp1 OPTIONS('DELIMITER'=',' ,
> 'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','SINGLE_Pass'='true');
> 3)Result:
> Actual result on beeline:
> Error: java.lang.Exception: Dataload failed due to error while writing
> dictionary file! (state=,code=0)
> Expected Result: data should be load successfully
> 4)Thriftserver logs:
> 17/05/16 16:07:20 INFO SparkExecuteStatementOperation: Running query 'LOAD
> DATA INPATH 'hdfs://localhost:54310/2000_UniqData.csv' into table
> uniq_exclude_sp1 OPTIONS('DELIMITER'=',' ,
> 'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','SINGLE_Pass'='true')'
> with 34eb7e9e-bd49-495c-af68-8f0b5e36b786
> 17/05/16 16:07:20 INFO CarbonSparkSqlParser: Parsing command: LOAD DATA
> INPATH 'hdfs://localhost:54310/2000_UniqData.csv' into table uniq_exclude_sp1
> OPTIONS('DELIMITER'=',' ,
> 'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','SINGLE_Pass'='true')
> 17/05/16 16:07:20 INFO CarbonLateDecodeRule: pool-23-thread-4 Skip
> CarbonOptimizer
> 17/05/16 16:07:20 INFO HdfsFileLock: pool-23-thread-4 HDFS lock
> path:hdfs://localhost:54310/opt/prestocarbonStore/default/uniq_exclude_sp1/meta.lock
> 17/05/16 16:07:20 INFO LoadTable: pool-23-thread-4 Successfully able to get
> the table metadata file lock
> 17/05/16 16:07:20 INFO LoadTable: pool-23-thread-4 Initiating Direct Load for
> the Table : (default.uniq_exclude_sp1)
> 17/05/16 16:07:20 AUDIT CarbonDataRDDFactory$:
> [knoldus][hduser][Thread-137]Data load request has been received for table
> default.uniq_exclude_sp1
> 17/05/16 16:07:20 INFO CommonUtil$: pool-23-thread-4 [Block Distribution]
> 17/05/16 16:07:20 INFO CommonUtil$: pool-23-thread-4 totalInputSpaceConsumed:
> 376223 , defaultParallelism: 4
> 17/05/16 16:07:20 INFO CommonUtil$: pool-23-thread-4
> mapreduce.input.fileinputformat.split.maxsize: 16777216
> 17/05/16 16:07:20 INFO FileInputFormat: Total input paths to process : 1
> 17/05/16 16:07:20 INFO DistributionUtil$: pool-23-thread-4 Executors
> configured : 1
> 17/05/16 16:07:20 INFO DistributionUtil$: pool-23-thread-4 Total Time taken
> to ensure the required executors : 0
> 17/05/16 16:07:20 INFO DistributionUtil$: pool-23-thread-4 Time elapsed to
> allocate the required executors: 0
> 17/05/16 16:07:20 INFO CarbonDataRDDFactory$: pool-23-thread-4 Total Time
> taken in block allocation: 1
> 17/05/16 16:07:20 INFO CarbonDataRDDFactory$: pool-23-thread-4 Total no of
> blocks: 1, No.of Nodes: 1
> 17/05/16 16:07:20 INFO CarbonDataRDDFactory$: pool-23-thread-4 #Node: knoldus
> no.of.blocks: 1
> 17/05/16 16:07:20 INFO MemoryStore: Block broadcast_2 stored as values in
> memory (estimated size 53.7 MB, free 291.4 MB)
> 17/05/16 16:07:20 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes
> in memory (estimated size 23.2 KB, free 291.4 MB)
> 17/05/16 16:07:20 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory
> on 192.168.1.10:42046 (size: 23.2 KB, free: 366.2 MB)
> 17/05/16 16:07:20 INFO SparkContext: Created broadcast 2 from broadcast at
> NewCarbonDataLoadRDD.scala:185
> 17/05/16 16:07:20 INFO SparkContext: Starting job:
[jira] [Created] (CARBONDATA-1229) null being populated in table if user deletes table during single pass load
Kunal Kapoor created CARBONDATA-1229: Summary: null being populated in table if user deletes table during single pass load Key: CARBONDATA-1229 URL: https://issues.apache.org/jira/browse/CARBONDATA-1229 Project: CarbonData Issue Type: Bug Reporter: Kunal Kapoor Assignee: Kunal Kapoor Steps to reproduce:- 1. Create table 2. Load data using single_pass 3. Delete table 4. recreate table with the same name -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1093) User data is getting printed in logs if the server fails to respond to client
Kunal Kapoor created CARBONDATA-1093:
Summary: User data is getting printed in logs if the server fails
to respond to client
Key: CARBONDATA-1093
URL: https://issues.apache.org/jira/browse/CARBONDATA-1093
Project: CarbonData
Issue Type: Bug
Reporter: Kunal Kapoor
Assignee: Kunal Kapoor
Priority: Critical
java.lang.RuntimeException: Request timed out for key : DictionaryKey{
columnName='p_cap_time', data='2010301', dictionaryValue=-1,
type=DICT_GENERATION}
at
org.apache.carbondata.core.dictionary.client.DictionaryClientHandler.getDictionary(DictionaryClientHandler.java:94)
at
org.apache.carbondata.core.dictionary.client.DictionaryClient.getDictionary(DictionaryClient.java:81)
at
org.apache.carbondata.processing.newflow.dictionary.DictionaryServerClientDictionary.getOrGenerateKey(DictionaryServerClientDictionary.java:57)
at
org.apache.carbondata.processing.newflow.dictionary.DictionaryServerClientDictionary.getOrGenerateKey(DictionaryServerClientDictionary.java:32)
at
org.apache.carbondata.processing.newflow.converter.impl.DictionaryFieldConverterImpl.convert(DictionaryFieldConverterImpl.java:117)
at
org.apache.carbondata.processing.newflow.converter.impl.RowConverterImpl.convert(RowConverterImpl.java:155)
at
org.apache.carbondata.processing.newflow.steps.DataConverterProcessorStepImpl.processRowBatch(DataConverterProcessorStepImpl.java:107)
at
org.apache.carbondata.processing.newflow.steps.DataConverterProcessorStepImpl$1.next(DataConverterProcessorStepImpl.java:93)
at
org.apache.carbondata.processing.newflow.steps.DataConverterProcessorStepImpl$1.next(DataConverterProcessorStepImpl.java:81)
at
org.apache.carbondata.processing.newflow.sort.impl.UnsafeParallelReadMergeSorterImpl$SortIteratorThread.call(UnsafeParallelReadMergeSorterImpl.java:206)
at
org.apache.carbondata.processing.newflow.sort.impl.UnsafeParallelReadMergeSorterImpl$SortIteratorThread.call(UnsafeParallelReadMergeSorterImpl.java:181)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Solution: Remove the data field while printing logs
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (CARBONDATA-1113) Add validation for partition column feature
[ https://issues.apache.org/jira/browse/CARBONDATA-1113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor reassigned CARBONDATA-1113: Assignee: Kunal Kapoor > Add validation for partition column feature > --- > > Key: CARBONDATA-1113 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1113 > Project: CarbonData > Issue Type: Bug >Reporter: Kunal Kapoor >Assignee: Kunal Kapoor > Time Spent: 1h 40m > Remaining Estimate: 0h > > Add various validation to partition columns like:- > 1. Partition columns cannot be drop using alter command > 2. Range partition info should have values in ascending order -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (CARBONDATA-1144) Drop column operation failed in Alter table.
[
https://issues.apache.org/jira/browse/CARBONDATA-1144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Kapoor reassigned CARBONDATA-1144:
Assignee: Kunal Kapoor
> Drop column operation failed in Alter table.
> -
>
> Key: CARBONDATA-1144
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1144
> Project: CarbonData
> Issue Type: Bug
> Components: data-query
>Affects Versions: 1.2.0
> Environment: Spark 2.1
>Reporter: Vinod Rohilla
>Assignee: Kunal Kapoor
>Priority: Minor
> Attachments: 2000_UniqData.csv
>
>
> Drop column does not work in Alter table.
> Steps to reproduce:
> 1: Create a table in Carbon:
> CREATE TABLE uniqdata (CUST_ID int,CUST_NAME String,ACTIVE_EMUI_VERSION
> string, DOB timestamp, DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2
> bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2
> decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,INTEGER_COLUMN1
> int) STORED BY 'org.apache.carbondata.format' TBLPROPERTIES
> ("TABLE_BLOCKSIZE"= "256 MB");
> 2: Load Data in a table:
> LOAD DATA INPATH 'hdfs://localhost:54310/2000_UniqData.csv' into table
> uniqdata OPTIONS('DELIMITER'=',' ,
> 'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');
> 3: Run the following query: alter table uniqdata drop columns(CUST_NAME);
> 4: Result:
> 0: jdbc:hive2://localhost:1> alter table uniqdata drop columns(CUST_NAME);
> Error: java.lang.RuntimeException: Alter table drop column operation failed:
> null (state=,code=0)
> Expected Result: Column should be dropped.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (CARBONDATA-1135) Show partition column information in describe formatted command
Kunal Kapoor created CARBONDATA-1135: Summary: Show partition column information in describe formatted command Key: CARBONDATA-1135 URL: https://issues.apache.org/jira/browse/CARBONDATA-1135 Project: CarbonData Issue Type: Improvement Reporter: Kunal Kapoor Assignee: Kunal Kapoor Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CARBONDATA-1188) Incorrect data is displayed for double data type
Kunal Kapoor created CARBONDATA-1188:
Summary: Incorrect data is displayed for double data type
Key: CARBONDATA-1188
URL: https://issues.apache.org/jira/browse/CARBONDATA-1188
Project: CarbonData
Issue Type: Bug
Reporter: Kunal Kapoor
Assignee: Kunal Kapoor
Attachments: 100.csv
create table Comp_VMALL_DICTIONARY_EXCLUDE (imei string,gamePointId double)
STORED BY 'org.apache.carbondata.format'
TBLPROPERTIES('DICTIONARY_EXCLUDE'='imei')
LOAD DATA INPATH '/home/kunal/Downloads/100.csv' INTO table
Comp_VMALL_DICTIONARY_EXCLUDE options ('DELIMITER'=',', 'QUOTECHAR'='"',
'BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='imei,gamePointId')
select * from Comp_VMALL_DICTIONARY_EXCLUDE
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (CARBONDATA-1188) Incorrect data is displayed for double data type
[
https://issues.apache.org/jira/browse/CARBONDATA-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Kapoor updated CARBONDATA-1188:
-
Labels: duplicate (was: )
> Incorrect data is displayed for double data type
>
>
> Key: CARBONDATA-1188
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1188
> Project: CarbonData
> Issue Type: Bug
>Reporter: Kunal Kapoor
>Assignee: Kunal Kapoor
> Labels: duplicate
> Attachments: 100.csv
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> create table Comp_VMALL_DICTIONARY_EXCLUDE (imei string,gamePointId double)
> STORED BY 'org.apache.carbondata.format'
> TBLPROPERTIES('DICTIONARY_EXCLUDE'='imei')
> LOAD DATA INPATH '/home/kunal/Downloads/100.csv' INTO table
> Comp_VMALL_DICTIONARY_EXCLUDE options ('DELIMITER'=',', 'QUOTECHAR'='"',
> 'BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='imei,gamePointId')
> select * from Comp_VMALL_DICTIONARY_EXCLUDE
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (CARBONDATA-1188) Incorrect data is displayed for double data type
[
https://issues.apache.org/jira/browse/CARBONDATA-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16053860#comment-16053860
]
Kunal Kapoor edited comment on CARBONDATA-1188 at 6/19/17 11:51 AM:
duplicate to
https://issues.apache.org/jira/projects/CARBONDATA/issues/CARBONDATA-1184?filter=allopenissues
was (Author: kunal):
duplicate to
[#https://issues.apache.org/jira/projects/CARBONDATA/issues/CARBONDATA-1184?filter=allopenissues]
> Incorrect data is displayed for double data type
>
>
> Key: CARBONDATA-1188
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1188
> Project: CarbonData
> Issue Type: Bug
>Reporter: Kunal Kapoor
>Assignee: Kunal Kapoor
> Labels: duplicate
> Attachments: 100.csv
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> create table Comp_VMALL_DICTIONARY_EXCLUDE (imei string,gamePointId double)
> STORED BY 'org.apache.carbondata.format'
> TBLPROPERTIES('DICTIONARY_EXCLUDE'='imei')
> LOAD DATA INPATH '/home/kunal/Downloads/100.csv' INTO table
> Comp_VMALL_DICTIONARY_EXCLUDE options ('DELIMITER'=',', 'QUOTECHAR'='"',
> 'BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='imei,gamePointId')
> select * from Comp_VMALL_DICTIONARY_EXCLUDE
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CARBONDATA-1188) Incorrect data is displayed for double data type
[
https://issues.apache.org/jira/browse/CARBONDATA-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16053860#comment-16053860
]
Kunal Kapoor commented on CARBONDATA-1188:
--
duplicate to
https://issues.apache.org/jira/projects/CARBONDATA/issues/CARBONDATA-1184?filter=allopenissues
> Incorrect data is displayed for double data type
>
>
> Key: CARBONDATA-1188
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1188
> Project: CarbonData
> Issue Type: Bug
>Reporter: Kunal Kapoor
>Assignee: Kunal Kapoor
> Labels: duplicate
> Attachments: 100.csv
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> create table Comp_VMALL_DICTIONARY_EXCLUDE (imei string,gamePointId double)
> STORED BY 'org.apache.carbondata.format'
> TBLPROPERTIES('DICTIONARY_EXCLUDE'='imei')
> LOAD DATA INPATH '/home/kunal/Downloads/100.csv' INTO table
> Comp_VMALL_DICTIONARY_EXCLUDE options ('DELIMITER'=',', 'QUOTECHAR'='"',
> 'BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='imei,gamePointId')
> select * from Comp_VMALL_DICTIONARY_EXCLUDE
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (CARBONDATA-1188) Incorrect data is displayed for double data type
[
https://issues.apache.org/jira/browse/CARBONDATA-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16053860#comment-16053860
]
Kunal Kapoor edited comment on CARBONDATA-1188 at 6/19/17 11:47 AM:
duplicate to
[#https://issues.apache.org/jira/projects/CARBONDATA/issues/CARBONDATA-1184?filter=allopenissues]
was (Author: kunal):
duplicate to
https://issues.apache.org/jira/projects/CARBONDATA/issues/CARBONDATA-1184?filter=allopenissues
> Incorrect data is displayed for double data type
>
>
> Key: CARBONDATA-1188
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1188
> Project: CarbonData
> Issue Type: Bug
>Reporter: Kunal Kapoor
>Assignee: Kunal Kapoor
> Labels: duplicate
> Attachments: 100.csv
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> create table Comp_VMALL_DICTIONARY_EXCLUDE (imei string,gamePointId double)
> STORED BY 'org.apache.carbondata.format'
> TBLPROPERTIES('DICTIONARY_EXCLUDE'='imei')
> LOAD DATA INPATH '/home/kunal/Downloads/100.csv' INTO table
> Comp_VMALL_DICTIONARY_EXCLUDE options ('DELIMITER'=',', 'QUOTECHAR'='"',
> 'BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='imei,gamePointId')
> select * from Comp_VMALL_DICTIONARY_EXCLUDE
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (CARBONDATA-1037) Select query is not returning any data when we query on New Table after Alter Table rename operation
[
https://issues.apache.org/jira/browse/CARBONDATA-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Kapoor reassigned CARBONDATA-1037:
Assignee: Kunal Kapoor
> Select query is not returning any data when we query on New Table after Alter
> Table rename operation
>
>
> Key: CARBONDATA-1037
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1037
> Project: CarbonData
> Issue Type: Bug
> Components: data-query
>Affects Versions: 1.1.0
> Environment: 3 node cluster SUSE 11 SP4
>Reporter: Priyal Sachdeva
>Assignee: Kunal Kapoor
> Fix For: NONE
>
> Attachments: show_tables.JPG
>
>
> create database Priyal;
> Use Priyal;
> Create Table
> CREATE TABLE uniqdata111785 (CUST_ID int,CUST_NAME String,ACTIVE_EMUI_VERSION
> string, DOB timestamp, DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2
> bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2
> decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,INTEGER_COLUMN1
> int) STORED BY 'org.apache.carbondata.format'
> TBLPROPERTIES('DICTIONARY_INCLUDE'='INTEGER_COLUMN1,CUST_ID');
> Load Data into Table
> LOAD DATA INPATH 'hdfs://hacluster/user/Priyal/2000_UniqData.csv' into table
> uniqdata111785 OPTIONS('DELIMITER'=',' ,
> 'QUOTECHAR'='"','BAD_RECORDS_LOGGER_ENABLE'='TRUE',
> 'BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');
> LOAD DATA INPATH 'hdfs://hacluster/user/Priyal/2000_UniqData.csv' into table
> uniqdata111785 OPTIONS('DELIMITER'=',' ,
> 'QUOTECHAR'='"','BAD_RECORDS_LOGGER_ENABLE'='TRUE',
> 'BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');
> Select query on Old Table
> 0: jdbc:hive2://172.168.100.199:23040> select * from uniqdata111785 limit 10;
> +--+--++++-+-+-+-+--+---+--+--+
> | CUST_ID |CUST_NAME |ACTIVE_EMUI_VERSION | DOB
> | DOJ | BIGINT_COLUMN1 | BIGINT_COLUMN2 |
> DECIMAL_COLUMN1 | DECIMAL_COLUMN2 |Double_COLUMN1|
> Double_COLUMN2 | INTEGER_COLUMN1 |
> +--+--++++-+-+-+-+--+---+--+--+
> | 1| CUST_NAME_01000 | ACTIVE_EMUI_VERSION_01000 | 1972-09-27
> 00:00:00.0 | 1972-09-27 00:00:00.0 | 123372037854| -223372035854 |
> 12345679901.123400 | 22345679901.123400 | 1.12345674897976E10 |
> -1.12345674897976E10 | 1001 |
> | 10001| CUST_NAME_01001 | ACTIVE_EMUI_VERSION_01001 | 1972-09-28
> 00:00:00.0 | 1972-09-28 00:00:00.0 | 123372037855| -223372035853 |
> 12345679902.123400 | 22345679902.123400 | 1.12345674897976E10 |
> -1.12345674897976E10 | 1002 |
> | 10002| CUST_NAME_01002 | ACTIVE_EMUI_VERSION_01002 | 1972-09-29
> 00:00:00.0 | 1972-09-29 00:00:00.0 | 123372037856| -223372035852 |
> 12345679903.123400 | 22345679903.123400 | 1.12345674897976E10 |
> -1.12345674897976E10 | 1003 |
> | 10003| CUST_NAME_01003 | ACTIVE_EMUI_VERSION_01003 | 1972-09-30
> 00:00:00.0 | 1972-09-30 00:00:00.0 | 123372037857| -223372035851 |
> 12345679904.123400 | 22345679904.123400 | 1.12345674897976E10 |
> -1.12345674897976E10 | 1004 |
> | 10004| CUST_NAME_01004 | ACTIVE_EMUI_VERSION_01004 | 1972-10-01
> 00:00:00.0 | 1972-10-01 00:00:00.0 | 123372037858| -223372035850 |
> 12345679905.123400 | 22345679905.123400 | 1.12345674897976E10 |
> -1.12345674897976E10 | 1005 |
> | 10005| CUST_NAME_01005 | ACTIVE_EMUI_VERSION_01005 | 1972-10-02
> 00:00:00.0 | 1972-10-02 00:00:00.0 | 123372037859| -223372035849 |
> 12345679906.123400 | 22345679906.123400 | 1.12345674897976E10 |
> -1.12345674897976E10 | 1006 |
> | 10006| CUST_NAME_01006 | ACTIVE_EMUI_VERSION_01006 | 1972-10-03
> 00:00:00.0 | 1972-10-03 00:00:00.0 | 123372037860| -223372035848 |
> 12345679907.123400 | 22345679907.123400 | 1.12345674897976E10 |
> -1.12345674897976E10 | 1007 |
[jira] [Created] (CARBONDATA-1075) Close Dictionary Server when application ends
Kunal Kapoor created CARBONDATA-1075: Summary: Close Dictionary Server when application ends Key: CARBONDATA-1075 URL: https://issues.apache.org/jira/browse/CARBONDATA-1075 Project: CarbonData Issue Type: Bug Reporter: Kunal Kapoor Assignee: Kunal Kapoor Analysis: The dictionary server which is started when a load is done with "single_pass=true" is never closed which did not allow the carbon application to exit because it is running on the main thread. Solution: Add a listener to the sparkContext which closes the server when application ends -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (CARBONDATA-1188) Incorrect data is displayed for double data type
[
https://issues.apache.org/jira/browse/CARBONDATA-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Kapoor closed CARBONDATA-1188.
Resolution: Duplicate
> Incorrect data is displayed for double data type
>
>
> Key: CARBONDATA-1188
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1188
> Project: CarbonData
> Issue Type: Bug
>Reporter: Kunal Kapoor
>Assignee: Kunal Kapoor
> Labels: duplicate
> Attachments: 100.csv
>
> Time Spent: 7h
> Remaining Estimate: 0h
>
> create table Comp_VMALL_DICTIONARY_EXCLUDE (imei string,gamePointId double)
> STORED BY 'org.apache.carbondata.format'
> TBLPROPERTIES('DICTIONARY_EXCLUDE'='imei')
> LOAD DATA INPATH '/home/kunal/Downloads/100.csv' INTO table
> Comp_VMALL_DICTIONARY_EXCLUDE options ('DELIMITER'=',', 'QUOTECHAR'='"',
> 'BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='imei,gamePointId')
> select * from Comp_VMALL_DICTIONARY_EXCLUDE
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (CARBONDATA-1490) Unnecessary space is being allocated for measures in carbon row
Kunal Kapoor created CARBONDATA-1490:
Summary: Unnecessary space is being allocated for measures in
carbon row
Key: CARBONDATA-1490
URL: https://issues.apache.org/jira/browse/CARBONDATA-1490
Project: CarbonData
Issue Type: Bug
Reporter: Kunal Kapoor
Assignee: Kunal Kapoor
Priority: Minor
Consider the following query which has only one column of complex type. While
loading when we create carbon row for each of the records, unnecessary space in
the carbon row is being allocated for measures which *do not exist*.
spark.sql("CREATE TABLE carbon_table( complexData ARRAY) STORED BY
'carbondata'
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Closed] (CARBONDATA-1171) Add support for show partition DDL
[ https://issues.apache.org/jira/browse/CARBONDATA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor closed CARBONDATA-1171. Resolution: Invalid > Add support for show partition DDL > -- > > Key: CARBONDATA-1171 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1171 > Project: CarbonData > Issue Type: Bug > Components: sql >Reporter: Kunal Kapoor >Assignee: Kunal Kapoor >Priority: Minor > Time Spent: 5.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1493) Basic Maptype support
Kunal Kapoor created CARBONDATA-1493: Summary: Basic Maptype support Key: CARBONDATA-1493 URL: https://issues.apache.org/jira/browse/CARBONDATA-1493 Project: CarbonData Issue Type: Sub-task Reporter: Kunal Kapoor Assignee: Kunal Kapoor -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1596) ClassCastException is thrown by IntermediateFileMerger for decimal columns
Kunal Kapoor created CARBONDATA-1596: Summary: ClassCastException is thrown by IntermediateFileMerger for decimal columns Key: CARBONDATA-1596 URL: https://issues.apache.org/jira/browse/CARBONDATA-1596 Project: CarbonData Issue Type: Bug Reporter: Kunal Kapoor Assignee: Kunal Kapoor Priority: Minor When intermediate file merger tries to merge the sort files it converts the row data to their appropriate datatypes. While converting decimal types ClassCastException was being thrown. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CARBONDATA-1711) Carbon1.3.0-Pre-AggregateTable - Show datamap on table does not work
[ https://issues.apache.org/jira/browse/CARBONDATA-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253095#comment-16253095 ] Kunal Kapoor commented on CARBONDATA-1711: -- [~Ram@huawei] show datamap is not yet supported > Carbon1.3.0-Pre-AggregateTable - Show datamap on table does not > work > - > > Key: CARBONDATA-1711 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1711 > Project: CarbonData > Issue Type: Bug > Components: core >Affects Versions: 1.3.0 > Environment: Test >Reporter: Ramakrishna S >Priority: Minor > Fix For: 1.3.0 > > > 0: jdbc:hive2://10.18.98.34:23040> create datamap agr_lineitem ON TABLE > lineitem USING "org.apache.carbondata.datamap.AggregateDataMapHandler" as > select L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from > lineitem group by L_RETURNFLAG, L_LINESTATUS; > Error: java.lang.RuntimeException: Table [lineitem_agr_lineitem] already > exists under database [default] (state=,code=0) > 0: jdbc:hive2://10.18.98.34:23040> show tables; > +---+---+--+--+ > | database | tableName | isTemporary | > +---+---+--+--+ > | default | flow_carbon_test4 | false| > | default | jl_r3 | false| > | default | lineitem | false| > | default | lineitem_agr_lineitem | false| > | default | sensor_reading_blockblank_false | false| > | default | sensor_reading_blockblank_false1 | false| > | default | sensor_reading_blockblank_false2 | false| > | default | sensor_reading_false | false| > | default | sensor_reading_true | false| > | default | t1| false| > | default | t1_agg_t1 | false| > | default | tc4 | false| > | default | uniqdata | false| > +---+---+--+--+ > 13 rows selected (0.04 seconds) > 0: jdbc:hive2://10.18.98.34:23040> show datamap on table lineitem; > Error: java.lang.RuntimeException: > BaseSqlParser > missing 'FUNCTIONS' at 'on'(line 1, pos 13) > == SQL == > show datamap on table lineitem > -^^^ > CarbonSqlParser [1.6] failure: identifier matching regex (?i)SEGMENTS > expected > show datamap on table lineitem -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1841) Data is not being loaded into pre-aggregation table after creation
Kunal Kapoor created CARBONDATA-1841: Summary: Data is not being loaded into pre-aggregation table after creation Key: CARBONDATA-1841 URL: https://issues.apache.org/jira/browse/CARBONDATA-1841 Project: CarbonData Issue Type: Bug Reporter: Kunal Kapoor Assignee: Kunal Kapoor -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (CARBONDATA-1841) Data is not being loaded into pre-aggregation table after creation
[ https://issues.apache.org/jira/browse/CARBONDATA-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor closed CARBONDATA-1841. Resolution: Implemented > Data is not being loaded into pre-aggregation table after creation > -- > > Key: CARBONDATA-1841 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1841 > Project: CarbonData > Issue Type: Bug >Reporter: Kunal Kapoor >Assignee: Kunal Kapoor > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (CARBONDATA-1841) Data is not being loaded into pre-aggregation table after creation
[ https://issues.apache.org/jira/browse/CARBONDATA-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor resolved CARBONDATA-1841. -- Resolution: Fixed > Data is not being loaded into pre-aggregation table after creation > -- > > Key: CARBONDATA-1841 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1841 > Project: CarbonData > Issue Type: Bug >Reporter: Kunal Kapoor >Assignee: Kunal Kapoor > Time Spent: 1.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (CARBONDATA-1743) Carbon1.3.0-Pre-AggregateTable - Query returns no value if run at the time of pre-aggregate table creation
[
https://issues.apache.org/jira/browse/CARBONDATA-1743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Kapoor reassigned CARBONDATA-1743:
Assignee: Kunal Kapoor
> Carbon1.3.0-Pre-AggregateTable - Query returns no value if run at the time of
> pre-aggregate table creation
> --
>
> Key: CARBONDATA-1743
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1743
> Project: CarbonData
> Issue Type: Bug
> Components: data-load
>Affects Versions: 1.3.0
> Environment: Test - 3 node ant cluster
>Reporter: Ramakrishna S
>Assignee: Kunal Kapoor
> Labels: DFX
> Fix For: 1.3.0
>
>
> Steps:
> 1. Create table and load with large data
> create table if not exists lineitem4(L_SHIPDATE string,L_SHIPMODE
> string,L_SHIPINSTRUCT string,L_RETURNFLAG string,L_RECEIPTDATE
> string,L_ORDERKEY string,L_PARTKEY string,L_SUPPKEY string,L_LINENUMBER
> int,L_QUANTITY double,L_EXTENDEDPRICE double,L_DISCOUNT double,L_TAX
> double,L_LINESTATUS string,L_COMMITDATE string,L_COMMENT string) STORED BY
> 'org.apache.carbondata.format' TBLPROPERTIES
> ('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
> load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table
> lineitem4
> options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');
> 2. Create a pre-aggregate table
> create datamap agr_lineitem4 ON TABLE lineitem4 USING
> "org.apache.carbondata.datamap.AggregateDataMapHandler" as select
> L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from lineitem4
> group by L_RETURNFLAG, L_LINESTATUS;
> 3. Run aggregate query at the same time
> select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from
> lineitem4 group by l_returnflag, l_linestatus;
> *+Expected:+*: aggregate query should fetch data either from main table or
> pre-aggregate table.
> *+Actual:+* aggregate query does not return data until the pre-aggregate
> table is created
> 0: jdbc:hive2://10.18.98.48:23040> select
> l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from lineitem4
> group by l_returnflag, l_linestatus;
> +---+---+--+---+--+
> | l_returnflag | l_linestatus | sum(l_quantity) | sum(l_extendedprice) |
> +---+---+--+---+--+
> +---+---+--+---+--+
> No rows selected (1.74 seconds)
> 0: jdbc:hive2://10.18.98.48:23040> select
> l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from lineitem4
> group by l_returnflag, l_linestatus;
> +---+---+--+---+--+
> | l_returnflag | l_linestatus | sum(l_quantity) | sum(l_extendedprice) |
> +---+---+--+---+--+
> +---+---+--+---+--+
> No rows selected (0.746 seconds)
> 0: jdbc:hive2://10.18.98.48:23040> select
> l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from lineitem4
> group by l_returnflag, l_linestatus;
> +---+---+--++--+
> | l_returnflag | l_linestatus | sum(l_quantity) | sum(l_extendedprice) |
> +---+---+--++--+
> | N | F | 2.9808092E7 | 4.471079473931997E10 |
> | A | F | 1.145546488E9| 1.717580824169429E12 |
> | N | O | 2.31980219E9 | 3.4789002701143467E12 |
> | R | F | 1.146403932E9| 1.7190627928317903E12 |
> +---+---+--++--+
> 4 rows selected (0.8 seconds)
> 0: jdbc:hive2://10.18.98.48:23040> select
> l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from lineitem4
> group by l_returnflag, l_linestatus;
> +---+---+--++--+
> | l_returnflag | l_linestatus | sum(l_quantity) | sum(l_extendedprice) |
> +---+---+--++--+
> | N | F | 2.9808092E7 | 4.471079473931997E10 |
> | A | F | 1.145546488E9| 1.717580824169429E12 |
> | N | O | 2.31980219E9
[jira] [Created] (CARBONDATA-1738) Block direct load on pre-aggregate table
Kunal Kapoor created CARBONDATA-1738: Summary: Block direct load on pre-aggregate table Key: CARBONDATA-1738 URL: https://issues.apache.org/jira/browse/CARBONDATA-1738 Project: CarbonData Issue Type: Improvement Reporter: Kunal Kapoor Assignee: Kunal Kapoor Priority: Minor The user should not be able to execute load directly on the pre-aggregate table therefore load and insert have to be blocked. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1705) NullPointerException is thrown while loading data into table
Kunal Kapoor created CARBONDATA-1705:
Summary: NullPointerException is thrown while loading data into
table
Key: CARBONDATA-1705
URL: https://issues.apache.org/jira/browse/CARBONDATA-1705
Project: CarbonData
Issue Type: Bug
Reporter: Kunal Kapoor
Assignee: Kunal Kapoor
*Steps:*
1. CREATE TABLE maintable(id int, name string, city string, age int) STORED BY
'org.apache.carbondata.format' TBLPROPERTIES('dictionary_include'='id')
2. create tablemaintable_preagg_avg stored BY 'carbondata' tblproperties
('parent'='maintable') as select id,avg(age) from maintable group by id
3. LOAD DATA LOCAL INPATH 'sample.csv' into table maintable
*Exception:*
17/11/12 00:04:17 ERROR TaskContextImpl: Error in TaskCompletionListener
java.lang.NullPointerException
at
org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:97)
at
org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:95)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at
org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:95)
at org.apache.spark.scheduler.Task.run(Task.scala:112)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
17/11/12 00:04:17 ERROR Executor: Exception in task 0.0 in stage 5163.0 (TID
65678)
org.apache.spark.util.TaskCompletionListenerException
at
org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:105)
at org.apache.spark.scheduler.Task.run(Task.scala:112)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
17/11/12 00:04:17 ERROR TaskSetManager: Task 0 in stage 5163.0 failed 1 times;
aborting job
17/11/12 00:04:17 ERROR CarbonDataRDDFactory$:
ScalaTest-main-running-TestPreAggregateLoad load data frame failed
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in
stage 5163.0 failed 1 times, most recent failure: Lost task 0.0 in stage 5163.0
(TID 65678, localhost, executor driver):
org.apache.spark.util.TaskCompletionListenerException
at
org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:105)
at org.apache.spark.scheduler.Task.run(Task.scala:112)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Driver stacktrace:
at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1435)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1423)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1422)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1422)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
at scala.Option.foreach(Option.scala:257)
at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:802)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1650)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at
org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:628)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1918)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1931)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1944)
at
[jira] [Updated] (CARBONDATA-1705) NullPointerException is thrown while loading data into table with pre-aggregate table
[
https://issues.apache.org/jira/browse/CARBONDATA-1705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Kapoor updated CARBONDATA-1705:
-
Summary: NullPointerException is thrown while loading data into table with
pre-aggregate table (was: NullPointerException is thrown while loading data
into table)
> NullPointerException is thrown while loading data into table with
> pre-aggregate table
> -
>
> Key: CARBONDATA-1705
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1705
> Project: CarbonData
> Issue Type: Bug
>Reporter: Kunal Kapoor
>Assignee: Kunal Kapoor
>
> *Steps:*
> 1. CREATE TABLE maintable(id int, name string, city string, age int) STORED
> BY 'org.apache.carbondata.format' TBLPROPERTIES('dictionary_include'='id')
> 2. create tablemaintable_preagg_avg stored BY 'carbondata' tblproperties
> ('parent'='maintable') as select id,avg(age) from maintable group by id
> 3. LOAD DATA LOCAL INPATH 'sample.csv' into table maintable
> *Exception:*
> 17/11/12 00:04:17 ERROR TaskContextImpl: Error in TaskCompletionListener
> java.lang.NullPointerException
> at
> org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:97)
> at
> org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:95)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:95)
> at org.apache.spark.scheduler.Task.run(Task.scala:112)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 17/11/12 00:04:17 ERROR Executor: Exception in task 0.0 in stage 5163.0 (TID
> 65678)
> org.apache.spark.util.TaskCompletionListenerException
> at
> org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:105)
> at org.apache.spark.scheduler.Task.run(Task.scala:112)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 17/11/12 00:04:17 ERROR TaskSetManager: Task 0 in stage 5163.0 failed 1
> times; aborting job
> 17/11/12 00:04:17 ERROR CarbonDataRDDFactory$:
> ScalaTest-main-running-TestPreAggregateLoad load data frame failed
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in
> stage 5163.0 failed 1 times, most recent failure: Lost task 0.0 in stage
> 5163.0 (TID 65678, localhost, executor driver):
> org.apache.spark.util.TaskCompletionListenerException
> at
> org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:105)
> at org.apache.spark.scheduler.Task.run(Task.scala:112)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Driver stacktrace:
> at
> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1435)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1423)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1422)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1422)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
> at scala.Option.foreach(Option.scala:257)
> at
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:802)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1650)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
> at
>
[jira] [Assigned] (CARBONDATA-1757) Carbon 1.3.0- Pre_aggregate: After creating datamap on parent table, avg is not correct.
[
https://issues.apache.org/jira/browse/CARBONDATA-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Kapoor reassigned CARBONDATA-1757:
Assignee: Kunal Kapoor
> Carbon 1.3.0- Pre_aggregate: After creating datamap on parent table, avg is
> not correct.
>
>
> Key: CARBONDATA-1757
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1757
> Project: CarbonData
> Issue Type: Bug
> Components: data-query
>Affects Versions: 1.3.0
>Reporter: Ayushi Sharma
>Assignee: Kunal Kapoor
> Labels: functional
>
> Steps:
> 1. create table cust_2 (c_custkey int, c_name string, c_address string,
> c_nationkey bigint, c_phone string,c_acctbal decimal, c_mktsegment string,
> c_comment string) STORED BY 'org.apache.carbondata.format';
> 2. load data inpath 'hdfs://hacluster/customer/customer3.csv' into table
> cust_2
> options('DELIMITER'='|','QUOTECHAR'='"','FILEHEADER'='c_custkey,c_name,c_address,c_nationkey,c_phone,c_acctbal,c_mktsegment,c_comment');
> load data inpath 'hdfs://hacluster/customer/customer3.csv' into table cust_2
> options('DELIMITER'='|','QUOTECHAR'='"','FILEHEADER'='c_custkey,c_name,c_address,c_nationkey,c_phone,c_acctbal,c_mktsegment,c_comment');
> load data inpath 'hdfs://hacluster/customer/customer4.csv' into table cust_2
> options('DELIMITER'='|','QUOTECHAR'='"','FILEHEADER'='c_custkey,c_name,c_address,c_nationkey,c_phone,c_acctbal,c_mktsegment,c_comment');
> load data inpath 'hdfs://hacluster/customer/customer5.csv' into table cust_2
> options('DELIMITER'='|','QUOTECHAR'='"','FILEHEADER'='c_custkey,c_name,c_address,c_nationkey,c_phone,c_acctbal,c_mktsegment,c_comment');
> load data inpath 'hdfs://hacluster/customer/customer6.csv' into table cust_2
> options('DELIMITER'='|','QUOTECHAR'='"','FILEHEADER'='c_custkey,c_name,c_address,c_nationkey,c_phone,c_acctbal,c_mktsegment,c_comment');
> load data inpath 'hdfs://hacluster/customer/customer7.csv' into table cust_2
> options('DELIMITER'='|','QUOTECHAR'='"','FILEHEADER'='c_custkey,c_name,c_address,c_nationkey,c_phone,c_acctbal,c_mktsegment,c_comment');
> load data inpath 'hdfs://hacluster/customer/customer8.csv' into table cust_2
> options('DELIMITER'='|','QUOTECHAR'='"','FILEHEADER'='c_custkey,c_name,c_address,c_nationkey,c_phone,c_acctbal,c_mktsegment,c_comment');
> load data inpath 'hdfs://hacluster/customer/customer9.csv' into table cust_2
> options('DELIMITER'='|','QUOTECHAR'='"','FILEHEADER'='c_custkey,c_name,c_address,c_nationkey,c_phone,c_acctbal,c_mktsegment,c_comment');
> load data inpath 'hdfs://hacluster/customer/customer10.csv' into table
> cust_2
> options('DELIMITER'='|','QUOTECHAR'='"','FILEHEADER'='c_custkey,c_name,c_address,c_nationkey,c_phone,c_acctbal,c_mktsegment,c_comment');
> load data inpath 'hdfs://hacluster/customer/customer11.csv' into table
> cust_2
> options('DELIMITER'='|','QUOTECHAR'='"','FILEHEADER'='c_custkey,c_name,c_address,c_nationkey,c_phone,c_acctbal,c_mktsegment,c_comment');
> load data inpath 'hdfs://hacluster/customer/customer12.csv' into table
> cust_2
> options('DELIMITER'='|','QUOTECHAR'='"','FILEHEADER'='c_custkey,c_name,c_address,c_nationkey,c_phone,c_acctbal,c_mktsegment,c_comment');
> load data inpath 'hdfs://hacluster/customer/customer13.csv' into table
> cust_2
> options('DELIMITER'='|','QUOTECHAR'='"','FILEHEADER'='c_custkey,c_name,c_address,c_nationkey,c_phone,c_acctbal,c_mktsegment,c_comment');
> load data inpath 'hdfs://hacluster/customer/customer14.csv' into table
> cust_2
> options('DELIMITER'='|','QUOTECHAR'='"','FILEHEADER'='c_custkey,c_name,c_address,c_nationkey,c_phone,c_acctbal,c_mktsegment,c_comment');
> 3. SELECT c_custkey, c_name, sum(c_acctbal), avg(c_acctbal) FROM cust_2 GROUP
> BY c_custkey, c_name;
> 4. set carbon.input.segments.default.cust_2=0,1;
> 5. SELECT c_custkey, c_name, sum(c_acctbal), avg(c_acctbal) FROM cust_2 GROUP
> BY c_custkey, c_name;
> 6. CREATE DATAMAP tt1 ON TABLE cust_2 USING
> "org.apache.carbondata.datamap.AggregateDataMapHandler" AS SELECT c_custkey,
> c_name, sum(c_acctbal), avg(c_acctbal) FROM cust_2 GROUP BY c_custkey, c_name;
> 7. SELECT c_custkey, c_name, sum(c_acctbal), avg(c_acctbal) FROM cust_2
> GROUP BY c_custkey, c_name;
> 8. set carbon.input.segments.default.cust_2=*;
> 9. SELECT c_custkey, c_name, sum(c_acctbal), avg(c_acctbal) FROM cust_2 GROUP
> BY c_custkey, c_name;
> Issue:
> After creating datamap, avg is not correct
> Expected Output:
> Avg should have been displayed correctly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CARBONDATA-1849) n Create pre aggregate table failled
[
https://issues.apache.org/jira/browse/CARBONDATA-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16283200#comment-16283200
]
Kunal Kapoor commented on CARBONDATA-1849:
--
resolved with https://github.com/apache/carbondata/pull/1613
> n Create pre aggregate table failled
>
>
> Key: CARBONDATA-1849
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1849
> Project: CarbonData
> Issue Type: Bug
> Components: spark-integration
>Reporter: Mohammad Shahid Khan
>Assignee: Kunal Kapoor
>
> Create pre aggregate table is failing
>spark.sql("""create table carbontable (c1 string,c2 int,c3 string,c5
> string) STORED BY 'org.apache.carbondata.format'""")
> spark.sql("insert into carbontable select 'a',1,'aa','aaa'")
> spark. sql("insert into carbontable select 'b',1,'aa','aaa'")
> spark.sql("insert into carbontable select 'a',10,'aa','aaa'")
> spark.sql("create datamap preagg1 on table carbontable using
> 'preaggregate' as select c1,sum(c2) from carbontable group by c1")
> Exception:
> {code}
> Exception in thread "main"
> org.apache.carbondata.processing.loading.exception.CarbonDataLoadingException:
> CSV header in DDL is not proper. Column names in schema and CSV header are
> not the same.
> at
> org.apache.carbondata.spark.util.CommonUtil$.getCsvHeaderColumns(CommonUtil.scala:637)
> at
> org.apache.carbondata.spark.util.DataLoadingUtil$.buildCarbonLoadModel(DataLoadingUtil.scala:312)
> at
> org.apache.spark.sql.execution.command.management.CarbonLoadDataCommand.processData(CarbonLoadDataCommand.scala:133)
> at
> org.apache.spark.sql.execution.command.DataCommand.run(package.scala:71)
> at
> org.apache.spark.sql.execution.command.preaaggregate.CreatePreAggregateTableCommand.processData(CreatePreAggregateTableCommand.scala:139)
> at
> org.apache.spark.sql.execution.command.datamap.CarbonCreateDataMapCommand.processData(CarbonCreateDataMapCommand.scala:80)
> at
> org.apache.spark.sql.execution.command.AtomicRunnableCommand.run(package.scala:86)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132)
> at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:87)
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:87)
> at org.apache.spark.sql.Dataset.(Dataset.scala:185)
> at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)
> at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592)
> at
> org.apache.carbondata.examples.CarbonSessionExample$.main(CarbonSessionExample.scala:35)
> at
> org.apache.carbondata.examples.CarbonSessionExample.main(CarbonSessionExample.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (CARBONDATA-1064) Fix NullPointerException in Not InExpression
[ https://issues.apache.org/jira/browse/CARBONDATA-1064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor resolved CARBONDATA-1064. -- Resolution: Fixed > Fix NullPointerException in Not InExpression > > > Key: CARBONDATA-1064 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1064 > Project: CarbonData > Issue Type: Bug >Reporter: Kunal Kapoor >Assignee: Kunal Kapoor >Priority: Minor > Time Spent: 2h > Remaining Estimate: 0h > > select * from expression_test where id not in (1,2,'', NULL, ' ') > In the above query Right expression value is null in NotInExpression which > cased throws NullPointerException -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (CARBONDATA-1849) n Create pre aggregate table failled
[
https://issues.apache.org/jira/browse/CARBONDATA-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Kapoor reassigned CARBONDATA-1849:
Assignee: Kunal Kapoor
> n Create pre aggregate table failled
>
>
> Key: CARBONDATA-1849
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1849
> Project: CarbonData
> Issue Type: Bug
> Components: spark-integration
>Reporter: Mohammad Shahid Khan
>Assignee: Kunal Kapoor
>
> Create pre aggregate table is failing
>spark.sql("""create table carbontable (c1 string,c2 int,c3 string,c5
> string) STORED BY 'org.apache.carbondata.format'""")
> spark.sql("insert into carbontable select 'a',1,'aa','aaa'")
> spark. sql("insert into carbontable select 'b',1,'aa','aaa'")
> spark.sql("insert into carbontable select 'a',10,'aa','aaa'")
> spark.sql("create datamap preagg1 on table carbontable using
> 'preaggregate' as select c1,sum(c2) from carbontable group by c1")
> Exception:
> {code}
> Exception in thread "main"
> org.apache.carbondata.processing.loading.exception.CarbonDataLoadingException:
> CSV header in DDL is not proper. Column names in schema and CSV header are
> not the same.
> at
> org.apache.carbondata.spark.util.CommonUtil$.getCsvHeaderColumns(CommonUtil.scala:637)
> at
> org.apache.carbondata.spark.util.DataLoadingUtil$.buildCarbonLoadModel(DataLoadingUtil.scala:312)
> at
> org.apache.spark.sql.execution.command.management.CarbonLoadDataCommand.processData(CarbonLoadDataCommand.scala:133)
> at
> org.apache.spark.sql.execution.command.DataCommand.run(package.scala:71)
> at
> org.apache.spark.sql.execution.command.preaaggregate.CreatePreAggregateTableCommand.processData(CreatePreAggregateTableCommand.scala:139)
> at
> org.apache.spark.sql.execution.command.datamap.CarbonCreateDataMapCommand.processData(CarbonCreateDataMapCommand.scala:80)
> at
> org.apache.spark.sql.execution.command.AtomicRunnableCommand.run(package.scala:86)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132)
> at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:87)
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:87)
> at org.apache.spark.sql.Dataset.(Dataset.scala:185)
> at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)
> at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592)
> at
> org.apache.carbondata.examples.CarbonSessionExample$.main(CarbonSessionExample.scala:35)
> at
> org.apache.carbondata.examples.CarbonSessionExample.main(CarbonSessionExample.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (CARBONDATA-1881) insert overwrite not working properly for pre-aggregate tables
Kunal Kapoor created CARBONDATA-1881: Summary: insert overwrite not working properly for pre-aggregate tables Key: CARBONDATA-1881 URL: https://issues.apache.org/jira/browse/CARBONDATA-1881 Project: CarbonData Issue Type: Bug Reporter: Kunal Kapoor Assignee: Kunal Kapoor Priority: Minor when insert overwrite if fired on the main table then the pre-aggregate tables are not overwritten with the new values instead the values are appended to the table like a normal insert -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1886) Stale folders are not getting deleted on deletion on table status file
Kunal Kapoor created CARBONDATA-1886: Summary: Stale folders are not getting deleted on deletion on table status file Key: CARBONDATA-1886 URL: https://issues.apache.org/jira/browse/CARBONDATA-1886 Project: CarbonData Issue Type: Bug Reporter: Kunal Kapoor Assignee: Kunal Kapoor Priority: Minor Stale segment folder is not getting deleted when its entry is not present in table status file. Therefore wrong results are retrieved from the table Steps to reproduce 1. Create table 2. insert into table 3. delete table status file 4. insert into table 5. select * from table. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (CARBONDATA-1527) 11. Don't allow update/delete and delete segment operations on main table which has aggregation tables.
[ https://issues.apache.org/jira/browse/CARBONDATA-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor reassigned CARBONDATA-1527: Assignee: Kunal Kapoor > 11. Don't allow update/delete and delete segment operations on main table > which has aggregation tables. > > > Key: CARBONDATA-1527 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1527 > Project: CarbonData > Issue Type: Sub-task >Reporter: Ravindra Pesala >Assignee: Kunal Kapoor > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (CARBONDATA-1525) 9. Drop all related aggregation tables upon deleting of main table.
[ https://issues.apache.org/jira/browse/CARBONDATA-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor reassigned CARBONDATA-1525: Assignee: Kunal Kapoor > 9. Drop all related aggregation tables upon deleting of main table. > --- > > Key: CARBONDATA-1525 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1525 > Project: CarbonData > Issue Type: Sub-task >Reporter: Ravindra Pesala >Assignee: Kunal Kapoor > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (CARBONDATA-1524) 8. Refresh the cache of main table after droping of aggregation table.
[ https://issues.apache.org/jira/browse/CARBONDATA-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor reassigned CARBONDATA-1524: Assignee: Kunal Kapoor > 8. Refresh the cache of main table after droping of aggregation table. > -- > > Key: CARBONDATA-1524 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1524 > Project: CarbonData > Issue Type: Sub-task >Reporter: Ravindra Pesala >Assignee: Kunal Kapoor > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-2452) DataLoading failure when Struct with timesamp/date
Kunal Kapoor created CARBONDATA-2452:
Summary: DataLoading failure when Struct with timesamp/date
Key: CARBONDATA-2452
URL: https://issues.apache.org/jira/browse/CARBONDATA-2452
Project: CarbonData
Issue Type: Bug
Reporter: Kunal Kapoor
Assignee: Kunal Kapoor
val myaSchema=
"""
| {
| "namespace": "com.apache.schema",
| "type": "record",
| "name": "StudentActivity",
| "fields": [
| {
| "name": "id",
| "type": "int"
| },
| {
| "name": "course_details",
| "type": {
| "name": "course_details",
| "type": "record",
| "fields": [
| {
| "name": "course_struct_course_time",
| "type": "string"
| }
| ]
| }
| }
| ]
|}
""".stripMargin
var json = """{
| "id": 101,
| "course_details": {
| "course_struct_course_time":"2014-01-05"
| }
|}""".stripMargin
val nn = new org.apache.avro.Schema.Parser().parse(myaSchema)
val converter = new JsonAvroConverter
var record = converter
.convertToGenericDataRecord(json.getBytes(CharEncoding.UTF_8), nn)
val fields = new Array[Field](2)
fields(0)=new Field("id", DataTypes.INT)
val fld_s = new java.util.ArrayList[StructField]
fld_s.add(new StructField("course_struct_course_time", DataTypes.DATE))
fields(1)=new Field("course_details", "struct",fld_s)
try {
val writer = CarbonWriter.builder.withSchema(new
Schema(fields)).sortBy(Array("id"))
.outputPath("D:/Hadoop/carbon_1_3/sdk/carbon_bln_2").isTransactionalTable(false).buildWriterForAvroInput
writer.write(record)
writer.close()
}
catch {
case e: Exception => {
e.printStackTrace()
}
}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CARBONDATA-2450) Bad Records are not workng for Complex data type
Kunal Kapoor created CARBONDATA-2450:
Summary: Bad Records are not workng for Complex data type
Key: CARBONDATA-2450
URL: https://issues.apache.org/jira/browse/CARBONDATA-2450
Project: CarbonData
Issue Type: Bug
Reporter: Kunal Kapoor
Assignee: Kunal Kapoor
val myaSchema=
"""
| {
| "namespace": "com.apache.schema",
| "type": "record",
| "name": "StudentActivity",
| "fields": [
| {
| "name": "id",
| "type": "string"
| },
| {
| "name": "course_details",
| "type": {
| "name": "course_details",
| "type": "record",
| "fields": [
| {
| "name": "course_struct_course_string",
| "type": "string"
| }
| ]
| }
| },
| {
| "name":"salary_string",
| "type":
| {
| "type":"array",
| "items": "string"
| }
| }
| ]
|}
""".stripMargin
var json = """{
| "id": "cust_1",
| "course_details": {
| "course_struct_course_string":"babu"
| },
| "salary_string":["babu","lal"]
|}""".stripMargin
val nn = new org.apache.avro.Schema.Parser().parse(myaSchema)
val converter = new JsonAvroConverter
var record = converter
.convertToGenericDataRecord(json.getBytes(CharEncoding.UTF_8), nn)
val fields = new Array[Field](3)
fields(0)=new Field("id", DataTypes.STRING)
val fld_s = new java.util.ArrayList[StructField]
fld_s.add(new StructField("carbon_int", DataTypes.INT))
fields(1)=new Field("course_details", "struct",fld_s)
val fld_a = new java.util.ArrayList[StructField]
fld_a.add(new StructField("carbon_array", DataTypes.INT))
fields(2)=new Field("salary_string", "array",fld_a)
try {
val writer = CarbonWriter.builder.withSchema(new
Schema(fields)).sortBy(Array("id"))
.outputPath("D:/Hadoop/carbon_1_3/sdk/carbon_bln_4").isTransactionalTable(false).buildWriterForAvroInput
writer.write(record)
writer.close()
}
catch {
case e: Exception => {
e.printStackTrace()
}
}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CARBONDATA-2451) NullPointerException when sort has column which does not exists
Kunal Kapoor created CARBONDATA-2451:
Summary: NullPointerException when sort has column which does not
exists
Key: CARBONDATA-2451
URL: https://issues.apache.org/jira/browse/CARBONDATA-2451
Project: CarbonData
Issue Type: Bug
Reporter: Kunal Kapoor
Assignee: Kunal Kapoor
val myaSchema=
"""
| {
| "namespace": "com.apache.schema",
| "type": "record",
| "name": "StudentActivity",
| "fields": [
| {
| "name": "id",
| "type": "int"
| },
| {
| "name": "course_details",
| "type": {
| "name": "course_details",
| "type": "record",
| "fields": [
| {
| "name": "course_struct_course_time",
| "type": "string"
| }
| ]
| }
| }
| ]
|}
""".stripMargin
var json = """{
| "id": 101,
| "course_details": {
| "course_struct_course_time":"2014-01-05"
| }
|}""".stripMargin
val nn = new org.apache.avro.Schema.Parser().parse(myaSchema)
val converter = new JsonAvroConverter
var record = converter
.convertToGenericDataRecord(json.getBytes(CharEncoding.UTF_8), nn)
val fields = new Array[Field](2)
fields(0)=new Field("id", DataTypes.INT)
val fld_s = new java.util.ArrayList[StructField]
fld_s.add(new StructField("course_struct_course_time", DataTypes.STRING))
fields(1)=new Field("course_details", "struct",fld_s)
try {
val writer = CarbonWriter.builder.withSchema(new
Schema(fields)).sortBy(Array("name,id"))
.outputPath("D:/Hadoop/carbon_1_3/sdk/carbon_bln_2").isTransactionalTable(false).buildWriterForAvroInput
writer.write(record)
writer.close()
}
catch {
case e: Exception => {
e.printStackTrace()
}
}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CARBONDATA-2453) Data write fails if complex data type contains a field with same name as any other field(not of same complex data type)
Kunal Kapoor created CARBONDATA-2453:
Summary: Data write fails if complex data type contains a field
with same name as any other field(not of same complex data type)
Key: CARBONDATA-2453
URL: https://issues.apache.org/jira/browse/CARBONDATA-2453
Project: CarbonData
Issue Type: Bug
Reporter: Kunal Kapoor
val myaSchema=
"""{
| "type": "record",
| "name": "Order",
| "namespace": "com.apache.schema",
| "fields": [
| {
| "name": "id",
| "type": "long"
| },
| {
| "name": "date",
| "type": "long"
| },
| {
| "name": "entries",
| "type": {
| "type": "array",
| "items": {
| "type": "record",
| "name": "Entry",
| "fields": [
| {
| "name": "id",
| "type": "long"
| },
| {
| "name": "description",
| "type": "string"
| },
| {
| "name": "quantity",
| "type": "int"
| }
| ]
| }
|
| }
| }
| ]
|}""".stripMargin
var json = """{"id": 1525693807954, "date": 1419058662391170, "entries": [{"id":
1525693807954, "description": "Appliances", "quantity": 125}, {"id":
1525693807954,
"description": "Electronics", "quantity": 250}]}
""".stripMargin
val nn = new org.apache.avro.Schema.Parser().parse(myaSchema)
val converter = new JsonAvroConverter
var record = converter
.convertToGenericDataRecord(json.getBytes(CharEncoding.UTF_8), nn)
val fields = new Array[Field](3)
fields(0)=new Field("id", DataTypes.LONG)
fields(1)=new Field("date",DataTypes.DATE)
val fld_a = new java.util.ArrayList[StructField]
fld_a.add(new StructField("id", DataTypes.LONG))
fld_a.add(new StructField("desc", DataTypes.STRING))
fld_a.add(new StructField("QTY", DataTypes.INT))
fields(2)=new Field("entries", DataTypes.createStructType(fld_a))
try {
val writer = CarbonWriter.builder.withSchema(new Schema(fields))
.outputPath("D:/Hadoop/carbon_1_3/sdk/carbon_bln_4").isTransactionalTable(false).buildWriterForAvroInput
writer.write(record)
writer.close()
}
catch {
case e: Exception => {
e.printStackTrace()
}
}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (CARBONDATA-2424) Add documentation for properties of Pre-Aggregate tables
[ https://issues.apache.org/jira/browse/CARBONDATA-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor resolved CARBONDATA-2424. -- Resolution: Fixed Fix Version/s: 1.4.0 > Add documentation for properties of Pre-Aggregate tables > > > Key: CARBONDATA-2424 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2424 > Project: CarbonData > Issue Type: Task >Reporter: Praveen M P >Priority: Minor > Fix For: 1.4.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (CARBONDATA-2463) if two insert operations are running concurrently 1 task fails and causes wrong no of records in select
[ https://issues.apache.org/jira/browse/CARBONDATA-2463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor resolved CARBONDATA-2463. -- Resolution: Fixed Fix Version/s: 1.4.0 > if two insert operations are running concurrently 1 task fails and causes > wrong no of records in select > --- > > Key: CARBONDATA-2463 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2463 > Project: CarbonData > Issue Type: Bug >Reporter: Rahul Kumar >Assignee: Rahul Kumar >Priority: Major > Fix For: 1.4.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > If two insert operations are running concurrently then 1 task fails for one > of the job. However both jobs are successs. > > Below is the exception: > org.apache.carbondata.processing.loading.exception.CarbonDataLoadingException: > Error while initializing data handler : -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (CARBONDATA-2461) null as avro type is throwing Unsuuported data type exception
[
https://issues.apache.org/jira/browse/CARBONDATA-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Kapoor reassigned CARBONDATA-2461:
Assignee: Kunal Kapoor
Description: (was: val avroschema="""{
| "type": "record",
| "name": "RecordName",
| "fields": [
| {
| "name": "timeStamp",
| "type": "long",
| "doc": "Type inferred from '1500277699000'"
| },
| {
| "name": "vin",
| "type": "string",
| "doc": "Type inferred from '\"LFV0A24G5G3028152\"'"
| },
| {
| "name": "event",
| "type": "string",
| "doc": "Type inferred from '\"upload\"'"
| },
| {
| "name": "signals",
| "type": {
| "type": "array",
| "items": {
| "type": "record",
| "name": "signals",
| "fields": [
| {
| "name": "valueType",
| "type": "string",
| "doc": "Type inferred from '\"int\"'"
| },
| {
| "name": "status",
| "type": "string",
| "doc": "Type inferred from '\"VALID\"'"
| },
| {
| "name": "code",
| "type": "string",
| "doc": "Type inferred from '\"V048\"'"
| },
| {
| "name": "name_en",
| "type": "string",
| "doc": "Type inferred from '\"VehicleStatus\"'"
| },
| {
| "name": "name",
| "type": "string",
| "doc": "Type inferred from '\"车辆状态\"'"
| },
| {
| "name": "value",
| "type": "string",
| "doc": "Type inferred from '\"3\"'"
| }
| ]
| }
| }
| },
| {
| "name": "metaData",
| "type": "string",
| "doc": "Type inferred from '\"\{\\\"reissueFlag\\\":0}\"'"
| },
| {
| "name": "orginalReport",
| "type": null,
| "doc": "Type inferred from 'null'"
| },
| {
| "name": "uploadType",
| "type": "string",
| "doc": "Type inferred from '\"upload\"'"
| },
| {
| "name": "uploadTime",
| "type": "long",
| "doc": "Type inferred from '1500277699000'"
| }
| ]
|}""".stripMargin
val
jvalue=SupportSDK_1.readFileAndConvertoStringObject("D:/Hadoop/carbon_1_3/sdk/JVData.json");
val fields = new Array[Field](8)
fields(0)=new Field("timeStamp", DataTypes.LONG)
fields(1)=new Field("vin", DataTypes.STRING)
fields(2)=new Field("event", DataTypes.STRING)
val fld_s = new java.util.ArrayList[StructField]
fld_s.add(new StructField("valueType", DataTypes.STRING))
fld_s.add(new StructField("status", DataTypes.STRING))
fld_s.add(new StructField("code", DataTypes.STRING))
fld_s.add(new StructField("name_en", DataTypes.STRING))
fld_s.add(new StructField("name", DataTypes.STRING))
fld_s.add(new StructField("value", DataTypes.STRING))
val fld_s_n = new java.util.ArrayList[StructField]
fld_s_n.add(new StructField("signals", DataTypes.createStructType(fld_s)))
fields(3)=new Field("signals", "array",fld_s_n)
fields(4)=new Field("metaData", DataTypes.STRING)
fields(5)=new Field("orginalReport", DataTypes.STRING)
fields(6)=new Field("uploadType", DataTypes.STRING)
fields(7)=new Field("uploadTime", DataTypes.LONG)
val converter = new JsonAvroConverter();
val parsedAschema=new org.apache.avro.Schema.Parser().parse(avroschema)
val record = converter.convertToGenericDataRecord(
jvalue.getBytes(CharEncoding.UTF_8),parsedAschema );
try {
val writer = CarbonWriter.builder.withSchema(new
Schema(fields)).sortBy(Array("vin"))
.outputPath("D:/Hadoop/carbon_1_3/sdk/carbon_cust_1").isTransactionalTable(false).buildWriterForAvroInput
writer.write(record)
writer.close()
}
catch {
case e: Exception => {
e.printStackTrace()
}
})
Summary: null as avro type is throwing Unsuuported data type exception
(was: DIS customer Schema is not supported,Even Mapped to String Column )
> null as avro type is throwing Unsuuported data type exception
> -
>
> Key: CARBONDATA-2461
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2461
> Project: CarbonData
> Issue Type: Bug
>Reporter: Praveen M P
>Assignee: Kunal Kapoor
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (CARBONDATA-2352) Add SDV Test Cases for Partition with Pre-Aggregate table
[ https://issues.apache.org/jira/browse/CARBONDATA-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor resolved CARBONDATA-2352. -- Resolution: Fixed Fix Version/s: 1.4.0 > Add SDV Test Cases for Partition with Pre-Aggregate table > - > > Key: CARBONDATA-2352 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2352 > Project: CarbonData > Issue Type: Test >Reporter: Praveen M P >Assignee: Praveen M P >Priority: Minor > Fix For: 1.4.0 > > Time Spent: 6h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (CARBONDATA-2442) Reading two sdk writer output with differnt schema should prompt exception
[ https://issues.apache.org/jira/browse/CARBONDATA-2442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor resolved CARBONDATA-2442. -- Resolution: Fixed Fix Version/s: 1.4.0 > Reading two sdk writer output with differnt schema should prompt exception > -- > > Key: CARBONDATA-2442 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2442 > Project: CarbonData > Issue Type: Bug >Reporter: Ajantha Bhat >Assignee: Ajantha Bhat >Priority: Major > Fix For: 1.4.0 > > Time Spent: 6h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (CARBONDATA-2442) Reading two sdk writer output with differnt schema should prompt exception
[ https://issues.apache.org/jira/browse/CARBONDATA-2442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor reassigned CARBONDATA-2442: Assignee: Ajantha Bhat > Reading two sdk writer output with differnt schema should prompt exception > -- > > Key: CARBONDATA-2442 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2442 > Project: CarbonData > Issue Type: Bug >Reporter: Ajantha Bhat >Assignee: Ajantha Bhat >Priority: Major > Fix For: 1.4.0 > > Time Spent: 6h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (CARBONDATA-2469) External Table must show its location instead of default store path in describe formatted
[ https://issues.apache.org/jira/browse/CARBONDATA-2469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor reassigned CARBONDATA-2469: Assignee: Ajantha Bhat > External Table must show its location instead of default store path in > describe formatted > - > > Key: CARBONDATA-2469 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2469 > Project: CarbonData > Issue Type: Bug >Reporter: Ajantha Bhat >Assignee: Ajantha Bhat >Priority: Major > Fix For: 1.4.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (CARBONDATA-2469) External Table must show its location instead of default store path in describe formatted
[ https://issues.apache.org/jira/browse/CARBONDATA-2469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor resolved CARBONDATA-2469. -- Resolution: Fixed Fix Version/s: 1.4.0 > External Table must show its location instead of default store path in > describe formatted > - > > Key: CARBONDATA-2469 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2469 > Project: CarbonData > Issue Type: Bug >Reporter: Ajantha Bhat >Assignee: Ajantha Bhat >Priority: Major > Fix For: 1.4.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (CARBONDATA-2426) IOException after compaction on Pre-Aggregate table on Partition table
[ https://issues.apache.org/jira/browse/CARBONDATA-2426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor resolved CARBONDATA-2426. -- Resolution: Fixed Fix Version/s: 1.4.0 > IOException after compaction on Pre-Aggregate table on Partition table > -- > > Key: CARBONDATA-2426 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2426 > Project: CarbonData > Issue Type: Bug >Reporter: Praveen M P >Priority: Major > Fix For: 1.4.0 > > Time Spent: 4h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (CARBONDATA-2470) Refactor AlterTableCompactionPostStatusUpdateEvent usage in compaction flow
[ https://issues.apache.org/jira/browse/CARBONDATA-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor resolved CARBONDATA-2470. -- Resolution: Fixed Fix Version/s: 1.4.0 > Refactor AlterTableCompactionPostStatusUpdateEvent usage in compaction flow > --- > > Key: CARBONDATA-2470 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2470 > Project: CarbonData > Issue Type: Improvement >Reporter: dhatchayani >Assignee: dhatchayani >Priority: Minor > Fix For: 1.4.0 > > Time Spent: 4h 20m > Remaining Estimate: 0h > > AlterTableCompactionPostStatusUpdateEvent in compaction flow is controlled > only by the preaggregate listener. If the CommitPreAggregateListener sets the > commitComplete property to true, this event will not be fired for the next > iteration -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (CARBONDATA-2465) Improve the carbondata file reliability in data load when direct hdfs write is enabled
[ https://issues.apache.org/jira/browse/CARBONDATA-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor resolved CARBONDATA-2465. -- Resolution: Fixed Fix Version/s: 1.4.0 > Improve the carbondata file reliability in data load when direct hdfs write > is enabled > -- > > Key: CARBONDATA-2465 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2465 > Project: CarbonData > Issue Type: Bug >Reporter: Kanaka Kumar Avvaru >Assignee: Kanaka Kumar Avvaru >Priority: Major > Fix For: 1.4.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > At present if we enable direct write on HDFS, file is written with > replication of 1. As the setReplication is asyn operation, it can cause data > loss if any DN is down before NN finishes the replication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CARBONDATA-2514) Duplicate columns in CarbonWriter is throwing NullPointerException
Kunal Kapoor created CARBONDATA-2514: Summary: Duplicate columns in CarbonWriter is throwing NullPointerException Key: CARBONDATA-2514 URL: https://issues.apache.org/jira/browse/CARBONDATA-2514 Project: CarbonData Issue Type: Bug Reporter: Kunal Kapoor -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (CARBONDATA-2198) Streaming data to a table with bad_records_action as IGNORE throws ClassCastException
[
https://issues.apache.org/jira/browse/CARBONDATA-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Kapoor resolved CARBONDATA-2198.
--
Resolution: Fixed
Fix Version/s: 1.4.0
> Streaming data to a table with bad_records_action as IGNORE throws
> ClassCastException
> -
>
> Key: CARBONDATA-2198
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2198
> Project: CarbonData
> Issue Type: Bug
> Components: data-load
>Affects Versions: 1.4.0
>Reporter: Geetika Gupta
>Assignee: anubhav tarar
>Priority: Minor
> Fix For: 1.4.0
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> Steps to reproduce:
> /*
> * Licensed to the Apache Software Foundation (ASF) under one or more
> * contributor license agreements. See the NOTICE file distributed with
> * this work for additional information regarding copyright ownership.
> * The ASF licenses this file to You under the Apache License, Version 2.0
> * (the "License"); you may not use this file except in compliance with
> * the License. You may obtain a copy of the License at
> *
> * [http://www.apache.org/licenses/LICENSE-2.0]
> *
> * Unless required by applicable law or agreed to in writing, software
> * distributed under the License is distributed on an "AS IS" BASIS,
> * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> * See the License for the specific language governing permissions and
> * limitations under the License.
> */
> package org.apache.carbondata.examples
> import java.io.\{File, PrintWriter}
> import java.net.ServerSocket
> import org.apache.spark.sql.\{CarbonEnv, SparkSession}
> import org.apache.spark.sql.streaming.\{ProcessingTime, StreamingQuery}
> import org.apache.carbondata.core.constants.CarbonCommonConstants
> import org.apache.carbondata.core.util.CarbonProperties
> import org.apache.carbondata.core.util.path.\{CarbonStorePath,
> CarbonTablePath}
> // scalastyle:off println
> object CarbonStructuredStreamingExample {
> def main(args: Array[String]) {
> // setup paths
> val rootPath = new File(this.getClass.getResource("/").getPath
> + "../../../..").getCanonicalPath
> val storeLocation = s"$rootPath/examples/spark2/target/store"
> val warehouse = s"$rootPath/examples/spark2/target/warehouse"
> val metastoredb = s"$rootPath/examples/spark2/target"
> val streamTableName = s"stream_table"
> CarbonProperties.getInstance()
> .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT, "/MM/dd")
> import org.apache.spark.sql.CarbonSession._
> val spark = SparkSession
> .builder()
> .master("local")
> .appName("CarbonStructuredStreamingExample")
> .config("spark.sql.warehouse.dir", warehouse)
> .getOrCreateCarbonSession(storeLocation, metastoredb)
> spark.sparkContext.setLogLevel("ERROR")
> val requireCreateTable = true
> val useComplexDataType = false
> if (requireCreateTable) {
> // drop table if exists previously
> spark.sql(s"DROP TABLE IF EXISTS ${ streamTableName }")
> // Create target carbon table and populate with initial data
> if (useComplexDataType) {
> spark.sql(
> s"""
> |CREATE TABLE ${ streamTableName }(|
> |id INT,|
> |name STRING,|
> |city STRING,|
> |salary FLOAT,|
> |file struct|
> |)|
> |STORED BY 'carbondata'|
> |TBLPROPERTIES(|
> |'streaming'='true', 'sort_columns'='name', 'dictionary_include'='city')|
> |""".stripMargin)
> } else {
> spark.sql(
> s"""|
> |CREATE TABLE ${ streamTableName }(|
> |id INT,|
> |name STRING,|
> |city STRING,|
> |salary FLOAT|
> |)|
> |STORED BY 'carbondata'|
> |TBLPROPERTIES(|
> |'streaming'='true', 'sort_columns'='name')|
> |""".stripMargin)
> }|
> val carbonTable = CarbonEnv.getCarbonTable(Some("default"),
> streamTableName)(spark)
> val tablePath =
> CarbonStorePath.getCarbonTablePath(carbonTable.getAbsoluteTableIdentifier)
> // streaming ingest
> val serverSocket = new ServerSocket(7071)
> val thread1 = startStreaming(spark, tablePath)
> val thread2 = writeSocket(serverSocket)
> System.out.println("type enter to interrupt streaming")
> System.in.read()
> thread1.interrupt()
> thread2.interrupt()
> serverSocket.close()
> }
> spark.sql(s"select * from $streamTableName").show
> spark.stop()
> System.out.println("streaming finished")
> }
> def showTableCount(spark: SparkSession, tableName: String): Thread = {
> val thread = new Thread() {
> override def run(): Unit = {
> for (_ <- 0 to 1000)
> { spark.sql(s"select count(*) from $tableName").show(truncate = false)
> Thread.sleep(1000 * 3) }
> }
> }
> thread.start()
> thread
> }
> def startStreaming(spark: SparkSession, tablePath: CarbonTablePath): Thread =
> {
> val thread = new Thread() {
> override def run(): Unit = {
> var qry:
[jira] [Created] (CARBONDATA-2538) No exception is thrown if writer path has only lock files
Kunal Kapoor created CARBONDATA-2538: Summary: No exception is thrown if writer path has only lock files Key: CARBONDATA-2538 URL: https://issues.apache.org/jira/browse/CARBONDATA-2538 Project: CarbonData Issue Type: Bug Reporter: Kunal Kapoor Assignee: Kunal Kapoor Steps to reproduce: # Create external table # Manually delete the index and carbon files # Describe table (lock files would be created) # Select from table -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (CARBONDATA-2491) There are some error when reader twice with SDK carbonReader
[
https://issues.apache.org/jira/browse/CARBONDATA-2491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Kapoor resolved CARBONDATA-2491.
--
Resolution: Fixed
Fix Version/s: 1.4.1
> There are some error when reader twice with SDK carbonReader
>
>
> Key: CARBONDATA-2491
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2491
> Project: CarbonData
> Issue Type: Bug
>Reporter: xubo245
>Assignee: xubo245
>Priority: Major
> Fix For: 1.4.1
>
> Time Spent: 11h 10m
> Remaining Estimate: 0h
>
> Test code in org.apache.carbondata.sdk.file.CarbonReaderTest:
> {code:java}
> @Test
> public void testWriteAndReadFiles() throws IOException,
> InterruptedException {
> String path = "./testWriteFiles";
> FileUtils.deleteDirectory(new File(path));
> Field[] fields = new Field[2];
> fields[0] = new Field("name", DataTypes.STRING);
> fields[1] = new Field("age", DataTypes.INT);
> TestUtil.writeFilesAndVerify(new Schema(fields), path, true);
> CarbonReader reader = CarbonReader.builder(path, "_temp")
> .projection(new String[]{"name", "age"}).build();
> // expected output after sorting
> String[] name = new String[100];
> int[] age = new int[100];
> for (int i = 0; i < 100; i++) {
> name[i] = "robot" + (i / 10);
> age[i] = (i % 10) * 10 + i / 10;
> }
> int i = 0;
> while (reader.hasNext()) {
> Object[] row = (Object[]) reader.readNextRow();
> // Default sort column is applied for dimensions. So, need to validate
> accordingly
> Assert.assertEquals(name[i], row[0]);
> Assert.assertEquals(age[i], row[1]);
> i++;
> }
> Assert.assertEquals(i, 100);
> reader.close();
> // Read again
> CarbonReader reader2 = CarbonReader
> .builder(path, "_temp")
> .projection(new String[]{"name", "age"})
> .build();
> i = 0;
> while (reader2.hasNext()) {
> Object[] row = (Object[]) reader2.readNextRow();
> // Default sort column is applied for dimensions. So, need to validate
> accordingly
> Assert.assertEquals(name[i], row[0]);
> Assert.assertEquals(age[i], row[1]);
> i++;
> }
> Assert.assertEquals(i, 100);
> reader2.close();
> FileUtils.deleteDirectory(new File(path));
> }
> {code}
> There are some error when build reader in the second tie with SDK
> carbonReader:
> When run this case first time:
> Error1
> {code:java}
> java.lang.StringIndexOutOfBoundsException: String index out of range: -1
> at java.lang.String.substring(String.java:1967)
> at
> org.apache.carbondata.core.util.path.CarbonTablePath$DataFileUtil.getTaskNo(CarbonTablePath.java:510)
> at
> org.apache.carbondata.hadoop.api.CarbonInputFormat.getDataBlocksOfSegment(CarbonInputFormat.java:372)
> at
> org.apache.carbondata.hadoop.api.CarbonFileInputFormat.getSplits(CarbonFileInputFormat.java:197)
> at
> org.apache.carbondata.hadoop.api.CarbonFileInputFormat.getSplits(CarbonFileInputFormat.java:166)
> at
> org.apache.carbondata.sdk.file.CarbonReaderBuilder.build(CarbonReaderBuilder.java:160)
> at
> org.apache.carbondata.sdk.file.CarbonReaderTest.testWriteAndReadFiles(CarbonReaderTest.java:64)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at junit.framework.TestCase.runTest(TestCase.java:176)
> at junit.framework.TestCase.runBare(TestCase.java:141)
> at junit.framework.TestResult$1.protect(TestResult.java:122)
> at junit.framework.TestResult.runProtected(TestResult.java:142)
> at junit.framework.TestResult.run(TestResult.java:125)
> at junit.framework.TestCase.run(TestCase.java:129)
> at junit.framework.TestSuite.runTest(TestSuite.java:255)
> at junit.framework.TestSuite.run(TestSuite.java:250)
> at
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
> at
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> {code}
> sometimes has the second exception:
> Error2:
> {code:java}
>
[jira] [Resolved] (CARBONDATA-2481) Adding SDV testcases for SDK Writer
[ https://issues.apache.org/jira/browse/CARBONDATA-2481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor resolved CARBONDATA-2481. -- Resolution: Fixed Fix Version/s: 1.4.1 > Adding SDV testcases for SDK Writer > --- > > Key: CARBONDATA-2481 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2481 > Project: CarbonData > Issue Type: Test >Reporter: Indhumathi Muthumurugesh >Assignee: Indhumathi Muthumurugesh >Priority: Minor > Fix For: 1.4.1 > > Time Spent: 5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CARBONDATA-2498) Change CarbonWriterBuilder interface to take schema while creating writer
Kunal Kapoor created CARBONDATA-2498: Summary: Change CarbonWriterBuilder interface to take schema while creating writer Key: CARBONDATA-2498 URL: https://issues.apache.org/jira/browse/CARBONDATA-2498 Project: CarbonData Issue Type: Bug Reporter: Kunal Kapoor -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CARBONDATA-2505) Update documentation for describe formatted on partition tables
Kunal Kapoor created CARBONDATA-2505: Summary: Update documentation for describe formatted on partition tables Key: CARBONDATA-2505 URL: https://issues.apache.org/jira/browse/CARBONDATA-2505 Project: CarbonData Issue Type: Bug Reporter: Kunal Kapoor -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (CARBONDATA-2555) SDK Reader should have isTransactionalTable = false by default, to be inline with SDK writer
[ https://issues.apache.org/jira/browse/CARBONDATA-2555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor resolved CARBONDATA-2555. -- Resolution: Fixed Fix Version/s: 1.4.0 > SDK Reader should have isTransactionalTable = false by default, to be inline > with SDK writer > > > Key: CARBONDATA-2555 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2555 > Project: CarbonData > Issue Type: Bug >Reporter: Ajantha Bhat >Assignee: Ajantha Bhat >Priority: Major > Fix For: 1.4.0 > > Time Spent: 3h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (CARBONDATA-2519) Add document for CarbonReader
[ https://issues.apache.org/jira/browse/CARBONDATA-2519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor resolved CARBONDATA-2519. -- Resolution: Fixed Fix Version/s: 1.4.1 > Add document for CarbonReader > - > > Key: CARBONDATA-2519 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2519 > Project: CarbonData > Issue Type: Improvement >Reporter: xubo245 >Assignee: xubo245 >Priority: Major > Fix For: 1.4.1 > > Time Spent: 5h 40m > Remaining Estimate: 0h > > Add document for CarbonReader, and change the carbon writer guide document -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CARBONDATA-2554) Support Logical types(date and timestamp) for AvroCarbonWriter.
Kunal Kapoor created CARBONDATA-2554: Summary: Support Logical types(date and timestamp) for AvroCarbonWriter. Key: CARBONDATA-2554 URL: https://issues.apache.org/jira/browse/CARBONDATA-2554 Project: CarbonData Issue Type: Improvement Reporter: Kunal Kapoor Assignee: Kunal Kapoor -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (CARBONDATA-2640) Support In Memory Locking mechanism
[ https://issues.apache.org/jira/browse/CARBONDATA-2640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor closed CARBONDATA-2640. Resolution: Not A Problem > Support In Memory Locking mechanism > --- > > Key: CARBONDATA-2640 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2640 > Project: CarbonData > Issue Type: Sub-task >Reporter: Kunal Kapoor >Assignee: Kunal Kapoor >Priority: Minor > Time Spent: 3h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CARBONDATA-2641) Refactor code to create S3CarbonFile
Kunal Kapoor created CARBONDATA-2641: Summary: Refactor code to create S3CarbonFile Key: CARBONDATA-2641 URL: https://issues.apache.org/jira/browse/CARBONDATA-2641 Project: CarbonData Issue Type: Sub-task Reporter: Kunal Kapoor -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CARBONDATA-2640) Support In Memory Locking mechanism
Kunal Kapoor created CARBONDATA-2640: Summary: Support In Memory Locking mechanism Key: CARBONDATA-2640 URL: https://issues.apache.org/jira/browse/CARBONDATA-2640 Project: CarbonData Issue Type: Sub-task Reporter: Kunal Kapoor -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CARBONDATA-2639) Support S3 as storage
Kunal Kapoor created CARBONDATA-2639: Summary: Support S3 as storage Key: CARBONDATA-2639 URL: https://issues.apache.org/jira/browse/CARBONDATA-2639 Project: CarbonData Issue Type: Improvement Reporter: Kunal Kapoor -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (CARBONDATA-2641) Refactor code to create S3CarbonFile
[ https://issues.apache.org/jira/browse/CARBONDATA-2641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor closed CARBONDATA-2641. Resolution: Invalid > Refactor code to create S3CarbonFile > > > Key: CARBONDATA-2641 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2641 > Project: CarbonData > Issue Type: Sub-task >Reporter: Kunal Kapoor >Assignee: Kunal Kapoor >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (CARBONDATA-2641) Refactor code to create S3CarbonFile
[ https://issues.apache.org/jira/browse/CARBONDATA-2641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor reassigned CARBONDATA-2641: Assignee: Kunal Kapoor > Refactor code to create S3CarbonFile > > > Key: CARBONDATA-2641 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2641 > Project: CarbonData > Issue Type: Sub-task >Reporter: Kunal Kapoor >Assignee: Kunal Kapoor >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (CARBONDATA-2640) Support In Memory Locking mechanism
[ https://issues.apache.org/jira/browse/CARBONDATA-2640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor reassigned CARBONDATA-2640: Assignee: Kunal Kapoor > Support In Memory Locking mechanism > --- > > Key: CARBONDATA-2640 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2640 > Project: CarbonData > Issue Type: Sub-task >Reporter: Kunal Kapoor >Assignee: Kunal Kapoor >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CARBONDATA-2642) Introduce configurable Lock Path
Kunal Kapoor created CARBONDATA-2642: Summary: Introduce configurable Lock Path Key: CARBONDATA-2642 URL: https://issues.apache.org/jira/browse/CARBONDATA-2642 Project: CarbonData Issue Type: Bug Reporter: Kunal Kapoor Assignee: Kunal Kapoor -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-2642) Introduce configurable Lock Path
[ https://issues.apache.org/jira/browse/CARBONDATA-2642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor updated CARBONDATA-2642: - Issue Type: Improvement (was: Bug) > Introduce configurable Lock Path > > > Key: CARBONDATA-2642 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2642 > Project: CarbonData > Issue Type: Improvement >Reporter: Kunal Kapoor >Assignee: Kunal Kapoor >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (CARBONDATA-2419) sortColumns Order we are getting wrong as we set for external table is fixed
[ https://issues.apache.org/jira/browse/CARBONDATA-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor resolved CARBONDATA-2419. -- Resolution: Fixed Fix Version/s: 1.4.0 > sortColumns Order we are getting wrong as we set for external table is fixed > > > Key: CARBONDATA-2419 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2419 > Project: CarbonData > Issue Type: Bug >Reporter: Akash R Nilugal >Priority: Major > Fix For: 1.4.0 > > Time Spent: 2h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (CARBONDATA-2419) sortColumns Order we are getting wrong as we set for external table is fixed
[ https://issues.apache.org/jira/browse/CARBONDATA-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor reassigned CARBONDATA-2419: Assignee: Rahul Kumar > sortColumns Order we are getting wrong as we set for external table is fixed > > > Key: CARBONDATA-2419 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2419 > Project: CarbonData > Issue Type: Bug >Reporter: Akash R Nilugal >Assignee: Rahul Kumar >Priority: Major > Fix For: 1.4.0 > > Time Spent: 2h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (CARBONDATA-2394) Setting segments in thread local space but not getting reflected in the driver
[ https://issues.apache.org/jira/browse/CARBONDATA-2394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor resolved CARBONDATA-2394. -- Resolution: Fixed Fix Version/s: 1.3.0 1.4.0 > Setting segments in thread local space but not getting reflected in the driver > -- > > Key: CARBONDATA-2394 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2394 > Project: CarbonData > Issue Type: Bug >Reporter: Rahul Kumar >Assignee: Rahul Kumar >Priority: Major > Fix For: 1.4.0, 1.3.0 > > Time Spent: 3h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CARBONDATA-1928) Separate the lock property for concurrent load and others
Kunal Kapoor created CARBONDATA-1928: Summary: Separate the lock property for concurrent load and others Key: CARBONDATA-1928 URL: https://issues.apache.org/jira/browse/CARBONDATA-1928 Project: CarbonData Issue Type: Improvement Reporter: Kunal Kapoor Assignee: Kunal Kapoor Priority: Minor Currently the property that is used to configure the lock retry count and the interval between retries is common for all the locks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1930) Dictionary not found exception is thrown when filter expression is given in aggergate table query
Kunal Kapoor created CARBONDATA-1930:
Summary: Dictionary not found exception is thrown when filter
expression is given in aggergate table query
Key: CARBONDATA-1930
URL: https://issues.apache.org/jira/browse/CARBONDATA-1930
Project: CarbonData
Issue Type: Bug
Reporter: Kunal Kapoor
Assignee: Kunal Kapoor
Priority: Minor
Steps to reproduce;
1. CREATE TABLE filtertable(id int, name string, city string, age string)
STORED BY 'org.apache.carbondata.format'
TBLPROPERTIES('dictionary_include'='name,age')
2. LOAD DATA LOCAL INPATH
3. create datamap agg9 on table filtertable using 'preaggregate' as select
name, age, sum(age) from filtertable group by name, age
4. select name, sum(age) from filtertable where age = '29' group by name, age
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (CARBONDATA-1978) Preaggregate table loading failed when using HiveMetastore
Kunal Kapoor created CARBONDATA-1978: Summary: Preaggregate table loading failed when using HiveMetastore Key: CARBONDATA-1978 URL: https://issues.apache.org/jira/browse/CARBONDATA-1978 Project: CarbonData Issue Type: Bug Reporter: Kunal Kapoor Assignee: Kunal Kapoor spark.carbon.hive.schema.store 1. set spark.carbon.hive.schema.store=true in carbon properties 2. create main table 3. load data into maintable 4. create pre-aggregate table *Exception:* Caused by: java.lang.IllegalArgumentException: unsupported data type:INT at org.apache.carbondata.processing.sort.sortdata.SortDataRows.writeData(SortDataRows.java:314) at org.apache.carbondata.processing.sort.sortdata.SortDataRows.writeDataTofile(SortDataRows.java:235) at org.apache.carbondata.processing.sort.sortdata.SortDataRows.startSorting(SortDataRows.java:215) at org.apache.carbondata.processing.loading.sort.impl.ParallelReadMergeSorterImpl.processRowToNextStep(ParallelReadMergeSorterImpl.java:174) at org.apache.carbondata.processing.loading.sort.impl.ParallelReadMergeSorterImpl.sort(ParallelReadMergeSorterImpl.java:112) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-2012) Add Transaction support for pre-aggregation table load
Kunal Kapoor created CARBONDATA-2012: Summary: Add Transaction support for pre-aggregation table load Key: CARBONDATA-2012 URL: https://issues.apache.org/jira/browse/CARBONDATA-2012 Project: CarbonData Issue Type: Improvement Reporter: Kunal Kapoor Assignee: Kunal Kapoor Currently the load process is like this: 1. load main table 2. load preagg1 and write table status 3. load preagg2 and write table status 4. write table status for maintable Improved process: 1. load main table 2. load preagg1 3. load preagg2 4. write table status for preagg2 5. write table status for preagg1 6. write table status for maintable -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-2069) Data is not loaded into preaggregate table when table is created when data load is in progress for main table
Kunal Kapoor created CARBONDATA-2069: Summary: Data is not loaded into preaggregate table when table is created when data load is in progress for main table Key: CARBONDATA-2069 URL: https://issues.apache.org/jira/browse/CARBONDATA-2069 Project: CarbonData Issue Type: Bug Reporter: Kunal Kapoor Assignee: Kunal Kapoor Problem: # Load data into maintable # create datamap parallelly preaggregate table will not have any data while data load is successful for main table. This will make the pre-aggregate table inconsistent -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (CARBONDATA-2107) Average query is failing when data map has both sum(column) and avg(column) of big int, int type
[ https://issues.apache.org/jira/browse/CARBONDATA-2107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor resolved CARBONDATA-2107. -- Resolution: Fixed Fix Version/s: 1.3.0 > Average query is failing when data map has both sum(column) and avg(column) > of big int, int type > > > Key: CARBONDATA-2107 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2107 > Project: CarbonData > Issue Type: Bug >Reporter: kumar vishal >Assignee: kumar vishal >Priority: Major > Fix For: 1.3.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > Average query is failing when data map has both sum(column) and avg(column) > of big int, int type -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (CARBONDATA-2112) Data getting garbled after datamap creation when table is created with GLOBAL SORT
[
https://issues.apache.org/jira/browse/CARBONDATA-2112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Kapoor resolved CARBONDATA-2112.
--
Resolution: Fixed
Fix Version/s: 1.3.0
> Data getting garbled after datamap creation when table is created with GLOBAL
> SORT
> --
>
> Key: CARBONDATA-2112
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2112
> Project: CarbonData
> Issue Type: Bug
> Components: data-query
> Environment: spark-2.1
>Reporter: Sangeeta Gulia
>Assignee: anubhav tarar
>Priority: Minor
> Fix For: 1.3.0
>
> Attachments: 2000_UniqData.csv
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Data is getting garbled after datamap creation when table is created with
> BATCH_SORT/GLOBAL_SORT.
>
> Steps to reproduce :
> spark.sql("drop table if exists uniqdata_batchsort_compact3")
> spark.sql("CREATE TABLE uniqdata_batchsort_compact3 (CUST_ID int,CUST_NAME
> String,ACTIVE_EMUI_VERSION string, DOB timestamp, DOJ timestamp,
> BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 bigint,DECIMAL_COLUMN1 decimal(30,10),
> DECIMAL_COLUMN2 decimal(36,10),Double_COLUMN1 double, Double_COLUMN2
> double,INTEGER_COLUMN1 int) STORED BY 'carbondata'
> TBLPROPERTIES('SORT_SCOPE'='GLOBAL_SORT')").show()
> spark.sql("LOAD DATA INPATH '/home/sangeeta/Desktop/2000_UniqData.csv' into
> table " +
> "uniqdata_batchsort_compact3 OPTIONS('DELIMITER'=',' , 'QUOTECHAR'='\"'," +
>
> "'BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,"
> +
> "DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2," +
> "Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','batch_sort_size_inmb'='1')")
> spark.sql("LOAD DATA INPATH '/home/sangeeta/Desktop/2000_UniqData.csv' into
> table " +
> "uniqdata_batchsort_compact3 OPTIONS('DELIMITER'=',' , 'QUOTECHAR'='\"'," +
>
> "'BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,"
> +
> "DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2," +
> "Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','batch_sort_size_inmb'='1')")
> spark.sql("LOAD DATA INPATH '/home/sangeeta/Desktop/2000_UniqData.csv' into
> table " +
> "uniqdata_batchsort_compact3 OPTIONS('DELIMITER'=',' , 'QUOTECHAR'='\"'," +
>
> "'BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,"
> +
> "DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2," +
> "Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','batch_sort_size_inmb'='1')")
> spark.sql("select cust_id, avg(cust_id) from uniqdata_batchsort_compact3
> group by cust_id ").show(50)
> +---++
> |cust_id|avg(cust_id)|
> +---++
> | 9376| 9376.0|
> | 9427| 9427.0|
> | 9465| 9465.0|
> | 9852| 9852.0|
> | 9900| 9900.0|
> | 10206| 10206.0|
> | 10362| 10362.0|
> | 10623| 10623.0|
> | 10817| 10817.0|
> | 9182| 9182.0|
> | 9564| 9564.0|
> | 9879| 9879.0|
> | 10081| 10081.0|
> | 10121| 10121.0|
> | 10230| 10230.0|
> | 10462| 10462.0|
> | 10703| 10703.0|
> | 10914| 10914.0|
> | 9162| 9162.0|
> | 9383| 9383.0|
> | 9454| 9454.0|
> | 9517| 9517.0|
> | 9558| 9558.0|
> | 10708| 10708.0|
> | 10798| 10798.0|
> | 10862| 10862.0|
> | 9071| 9071.0|
> | 9169| 9169.0|
> | 9946| 9946.0|
> | 10468| 10468.0|
> | 10745| 10745.0|
> | 10768| 10768.0|
> | 9153| 9153.0|
> | 9206| 9206.0|
> | 9403| 9403.0|
> | 9597| 9597.0|
> | 9647| 9647.0|
> | 9775| 9775.0|
> | 10032| 10032.0|
> | 10395| 10395.0|
> | 10527| 10527.0|
> | 10567| 10567.0|
> | 10632| 10632.0|
> | 10788| 10788.0|
> | 10815| 10815.0|
> | 10840| 10840.0|
> | 9181| 9181.0|
> | 9344| 9344.0|
> | 9575| 9575.0|
> | 9675| 9675.0|
> +---++
> only showing top 50 rows
> Note: Here the cust_id is coming correct .
> spark.sql("create datamap uniqdata_agg on table uniqdata_batchsort_compact3
> using " +
> "'preaggregate' as select avg(cust_id) from uniqdata_batchsort_compact3
> group by cust_id")
> spark.sql("select cust_id, avg(cust_id) from uniqdata_batchsort_compact3
> group by cust_id ").show(50)
> +---++
> |cust_id|avg(cust_id)|
> +---++
> | 27651| 9217.0|
> | 31944| 10648.0|
> | 32667| 10889.0|
> | 28242| 9414.0|
> | 29841| 9947.0|
> | 28728| 9576.0|
> | 27255| 9085.0|
> | 32571| 10857.0|
> | 30276| 10092.0|
> | 27276| 9092.0|
> | 31503| 10501.0|
> | 27687| 9229.0|
> | 27183| 9061.0|
> | 29334| 9778.0|
> | 29913| 9971.0|
> | 28683| 9561.0|
> | 31545| 10515.0|
> | 30405| 10135.0|
> | 27693| 9231.0|
> | 29649| 9883.0|
> | 30537| 10179.0|
> | 32709| 10903.0|
> | 29586| 9862.0|
> | 32895| 10965.0|
> | 32415| 10805.0|
> | 31644| 10548.0|
> | 30030| 10010.0|
> | 31713| 10571.0|
> | 28083| 9361.0|
> | 27813| 9271.0|
> | 27171| 9057.0|
> | 27189|
[jira] [Created] (CARBONDATA-2119) CarbonDataWriterException thrown when loading using global_sort
Kunal Kapoor created CARBONDATA-2119:
Summary: CarbonDataWriterException thrown when loading using
global_sort
Key: CARBONDATA-2119
URL: https://issues.apache.org/jira/browse/CARBONDATA-2119
Project: CarbonData
Issue Type: Bug
Reporter: Kunal Kapoor
Assignee: Kunal Kapoor
CREATE TABLE uniqdata_globalsort1 (CUST_ID int,CUST_NAME
String,ACTIVE_EMUI_VERSION string, DOB timestamp, DOJ timestamp, BIGINT_COLUMN1
bigint,BIGINT_COLUMN2 bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2
decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,INTEGER_COLUMN1
int) STORED BY 'carbondata' TBLPROPERTIES('SORT_SCOPE'='GLOBAL_SORT')
LOAD DATA INPATH 'hdfs://hacluster/chetan/2000_UniqData.csv' into table
uniqdata_globalsort1 OPTIONS('DELIMITER'=',' ,
'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');
*EXCEPTION*
There is an unexpected error: unable to generate the mdkey
org.apache.carbondata.spark.load.DataLoadProcessorStepOnSpark$.writeFunc(DataLoadProcessorStepOnSpark.scala:222)org.apache.carbondata.spark.load.DataLoadProcessBuilderOnSpark$$anonfun$loadDataUsingGlobalSort$1.apply(DataLoadProcessBuilderOnSpark.scala:136)
org.apache.carbondata.spark.load.DataLoadProcessBuilderOnSpark$$anonfun$loadDataUsingGlobalSort$1.apply(DataLoadProcessBuilderOnSpark.scala:135)
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
org.apache.spark.scheduler.Task.run(Task.scala:99)
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
at org.apache.spark.TaskContextImpl.invokeListeners(TaskContextImpl.scala:138)
at org.apache.spark.TaskContextImpl.markTaskFailed(TaskContextImpl.scala:106)
at org.apache.spark.scheduler.Task.run(Task.scala:104)
... 4 more
Caused by:
org.apache.carbondata.processing.loading.exception.CarbonDataLoadingException:
unable to generate the mdkey
at
org.apache.carbondata.processing.loading.steps.DataWriterProcessorStepImpl.processRow(DataWriterProcessorStepImpl.java:189)
at
org.apache.carbondata.spark.load.DataLoadProcessorStepOnSpark$.writeFunc(DataLoadProcessorStepOnSpark.scala:207)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (CARBONDATA-2061) Check for only valid IN_PROGRESS segments
[ https://issues.apache.org/jira/browse/CARBONDATA-2061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor resolved CARBONDATA-2061. -- Resolution: Fixed Fix Version/s: 1.3.0 > Check for only valid IN_PROGRESS segments > - > > Key: CARBONDATA-2061 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2061 > Project: CarbonData > Issue Type: Bug >Reporter: dhatchayani >Assignee: dhatchayani >Priority: Major > Fix For: 1.3.0 > > Time Spent: 4h > Remaining Estimate: 0h > > While checking for IN_PROGRESS segments of a table during other operations, > we should check only for valid IN_PROGRESS segments. Some segments may be > invalid like cancelled and may still in IN_PROGRESS state,those segments > should be considered as stale segments. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (CARBONDATA-2078) Add “IF NOT EXISTS” feature for create datamap
[ https://issues.apache.org/jira/browse/CARBONDATA-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor resolved CARBONDATA-2078. -- Resolution: Fixed > Add “IF NOT EXISTS” feature for create datamap > -- > > Key: CARBONDATA-2078 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2078 > Project: CarbonData > Issue Type: Improvement > Components: core, spark-integration >Affects Versions: 1.3.0 >Reporter: xubo245 >Assignee: xubo245 >Priority: Minor > Fix For: 1.3.0 > > Time Spent: 9h > Remaining Estimate: 0h > > Add “IF NOT EXISTS” feature for create datamap -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (CARBONDATA-2146) Preaggregate table is not dropped from metastore if creation fails
[ https://issues.apache.org/jira/browse/CARBONDATA-2146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor closed CARBONDATA-2146. Resolution: Duplicate > Preaggregate table is not dropped from metastore if creation fails > -- > > Key: CARBONDATA-2146 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2146 > Project: CarbonData > Issue Type: Bug >Affects Versions: 1.3.0 >Reporter: Kunal Kapoor >Assignee: Kunal Kapoor >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CARBONDATA-2146) Preaggregate table is not dropped from metastore if creation fails
Kunal Kapoor created CARBONDATA-2146: Summary: Preaggregate table is not dropped from metastore if creation fails Key: CARBONDATA-2146 URL: https://issues.apache.org/jira/browse/CARBONDATA-2146 Project: CarbonData Issue Type: Bug Affects Versions: 1.3.0 Reporter: Kunal Kapoor Assignee: Kunal Kapoor -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (CARBONDATA-2122) Redirect Bad Record Path Should Throw Exception on Empty Location
[ https://issues.apache.org/jira/browse/CARBONDATA-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor resolved CARBONDATA-2122. -- Resolution: Fixed > Redirect Bad Record Path Should Throw Exception on Empty Location > - > > Key: CARBONDATA-2122 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2122 > Project: CarbonData > Issue Type: Bug > Components: data-load >Affects Versions: 1.3.0 > Environment: Spark-2.1 >Reporter: Jatin >Assignee: Jatin >Priority: Minor > Fix For: 1.3.0 > > Time Spent: 7h 10m > Remaining Estimate: 0h > > Data Load having bad record redirect with empty location should throw > exception of Invalid Path. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-1952) Incorrect Result displays while applying delete query on table
[
https://issues.apache.org/jira/browse/CARBONDATA-1952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Kapoor updated CARBONDATA-1952:
-
Component/s: (was: data-query)
> Incorrect Result displays while applying delete query on table
> --
>
> Key: CARBONDATA-1952
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1952
> Project: CarbonData
> Issue Type: Bug
>Affects Versions: 1.3.0
> Environment: spark 2.1
>Reporter: Vandana Yadav
>Priority: Minor
> Attachments: 2000_UniqData.csv
>
>
> Incorrect Result while applying delete query on the table.
> Steps to reproduce:
> 1)Create table:
> CREATE TABLE uniqdata_delete (CUST_ID int,CUST_NAME
> String,ACTIVE_EMUI_VERSION string, DOB timestamp, DOJ timestamp,
> BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 bigint,DECIMAL_COLUMN1 decimal(30,10),
> DECIMAL_COLUMN2 decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,
> INTEGER_COLUMN1 int) STORED BY 'org.apache.carbondata.format' TBLPROPERTIES
> ("TABLE_BLOCKSIZE"= "256 MB")
> 2)Load Data Into table:
> LOAD DATA INPATH 'HDFS_URL/BabuStore/Data/uniqdata/2000_UniqData.csv' into
> table uniqdata_delete
> OPTIONS('FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1')
> 3)Execute Query:
> a) select count(*) from uniqdata_delete;
> Output:
> +---+--+
> | count(1) |
> +---+--+
> | 2013 |
> +---+--+
> 1 row selected (0.203 seconds)
> b)delete from uniqdata_delete where CUST_ID NOT IN(9996,);
> c) select count(*) from uniqdata_delete;
> output:
> +---+--+
> | count(1) |
> +---+--+
> | 14|
> +---+--+
> 1 row selected (0.22 seconds)
> d) select * from uniqdata_delete;
> output:
> +--+--++++-+-+-+-+--+---+--+--+
> | cust_id |cust_name |active_emui_version | dob
> | doj | bigint_column1 | bigint_column2 |
> decimal_column1 | decimal_column2 |double_column1|
> double_column2 | integer_column1 |
> +--+--++++-+-+-+-+--+---+--+--+
> | NULL | || NULL
> | NULL | NULL| NULL| NULL
>| NULL| NULL | NULL
> | NULL |
> | NULL | || NULL
> | NULL | 1233720368578 | NULL| NULL
>| NULL| NULL | NULL
> | NULL |
> | NULL | || NULL
> | NULL | NULL| -223372036854 | NULL
>| NULL| NULL | NULL
> | NULL |
> | NULL | || NULL
> | NULL | NULL| NULL|
> 12345678901.123400 | NULL| NULL |
> NULL | NULL |
> | NULL | || NULL
> | NULL | NULL| NULL| NULL
>| 22345678901.123400 | NULL | NULL
> | NULL |
> | NULL | || NULL
> | NULL | NULL| NULL| NULL
>| NULL| 1.12345674897976E10 | NULL
> | NULL |
> | NULL | || NULL
> | NULL | NULL| NULL| NULL
>| NULL| NULL |
> -1.12345674897976E10 | NULL |
> | NULL | || NULL
> | NULL | NULL| NULL| NULL
>| NULL| NULL | NULL
[jira] [Updated] (CARBONDATA-1952) Incorrect Result displays while applying delete query on table
[
https://issues.apache.org/jira/browse/CARBONDATA-1952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Kapoor updated CARBONDATA-1952:
-
Component/s: hive-integration
> Incorrect Result displays while applying delete query on table
> --
>
> Key: CARBONDATA-1952
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1952
> Project: CarbonData
> Issue Type: Bug
> Components: hive-integration
>Affects Versions: 1.3.0
> Environment: spark 2.1
>Reporter: Vandana Yadav
>Priority: Minor
> Attachments: 2000_UniqData.csv
>
>
> Incorrect Result while applying delete query on the table.
> Steps to reproduce:
> 1)Create table:
> CREATE TABLE uniqdata_delete (CUST_ID int,CUST_NAME
> String,ACTIVE_EMUI_VERSION string, DOB timestamp, DOJ timestamp,
> BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 bigint,DECIMAL_COLUMN1 decimal(30,10),
> DECIMAL_COLUMN2 decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,
> INTEGER_COLUMN1 int) STORED BY 'org.apache.carbondata.format' TBLPROPERTIES
> ("TABLE_BLOCKSIZE"= "256 MB")
> 2)Load Data Into table:
> LOAD DATA INPATH 'HDFS_URL/BabuStore/Data/uniqdata/2000_UniqData.csv' into
> table uniqdata_delete
> OPTIONS('FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1')
> 3)Execute Query:
> a) select count(*) from uniqdata_delete;
> Output:
> +---+--+
> | count(1) |
> +---+--+
> | 2013 |
> +---+--+
> 1 row selected (0.203 seconds)
> b)delete from uniqdata_delete where CUST_ID NOT IN(9996,);
> c) select count(*) from uniqdata_delete;
> output:
> +---+--+
> | count(1) |
> +---+--+
> | 14|
> +---+--+
> 1 row selected (0.22 seconds)
> d) select * from uniqdata_delete;
> output:
> +--+--++++-+-+-+-+--+---+--+--+
> | cust_id |cust_name |active_emui_version | dob
> | doj | bigint_column1 | bigint_column2 |
> decimal_column1 | decimal_column2 |double_column1|
> double_column2 | integer_column1 |
> +--+--++++-+-+-+-+--+---+--+--+
> | NULL | || NULL
> | NULL | NULL| NULL| NULL
>| NULL| NULL | NULL
> | NULL |
> | NULL | || NULL
> | NULL | 1233720368578 | NULL| NULL
>| NULL| NULL | NULL
> | NULL |
> | NULL | || NULL
> | NULL | NULL| -223372036854 | NULL
>| NULL| NULL | NULL
> | NULL |
> | NULL | || NULL
> | NULL | NULL| NULL|
> 12345678901.123400 | NULL| NULL |
> NULL | NULL |
> | NULL | || NULL
> | NULL | NULL| NULL| NULL
>| 22345678901.123400 | NULL | NULL
> | NULL |
> | NULL | || NULL
> | NULL | NULL| NULL| NULL
>| NULL| 1.12345674897976E10 | NULL
> | NULL |
> | NULL | || NULL
> | NULL | NULL| NULL| NULL
>| NULL| NULL |
> -1.12345674897976E10 | NULL |
> | NULL | || NULL
> | NULL | NULL| NULL| NULL
>| NULL
[jira] [Updated] (CARBONDATA-1928) Separate the lock property for concurrent load and others
[ https://issues.apache.org/jira/browse/CARBONDATA-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor updated CARBONDATA-1928: - Description: Currently the property that is used to configure the lock retry count and the interval between retries is common for all the locks. This will be problematic when the user has configured the retries to 10/20 for concurrent loading. This property will be affecting other lock behaviours also, all other locks would have to retry for 10 times too. was:Currently the property that is used to configure the lock retry count and the interval between retries is common for all the locks. > Separate the lock property for concurrent load and others > - > > Key: CARBONDATA-1928 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1928 > Project: CarbonData > Issue Type: Improvement >Reporter: Kunal Kapoor >Assignee: Kunal Kapoor >Priority: Minor > > Currently the property that is used to configure the lock retry count and the > interval between retries is common for all the locks. > This will be problematic when the user has configured the retries to 10/20 > for concurrent loading. This property will be affecting other lock behaviours > also, all other locks would have to retry for 10 times too. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
