Chetan Bhat created CARBONDATA-2787:
---------------------------------------
Summary: Data load on table with single pass true having
preaggregate datamaps without precreated dictionary throws incorrect error
message
Key: CARBONDATA-2787
URL: https://issues.apache.org/jira/browse/CARBONDATA-2787
Project: CarbonData
Issue Type: Bug
Components: data-load
Affects Versions: 1.4.1
Environment: Spark 2.2
Reporter: Chetan Bhat
Steps :
Create table .
Create 5 datamaps.
load data in table with single pass true without precreated dictionary
configuration.
0: jdbc:hive2://10.18.98.136:23040/default> CREATE TABLE
uniqdata_decimal(CUST_ID int,CUST_NAME String,ACTIVE_EMUI_VERSION string, DOB
timestamp, DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2
bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2
decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,INTEGER_COLUMN1
int) STORED BY 'org.apache.carbondata.format'
TBLPROPERTIES('DICTIONARY_INCLUDE'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1',
'SORT_SCOPE'='GLOBAL_SORT');
+---------+--+
| Result |
+---------+--+
+---------+--+
No rows selected (0.21 seconds)
0: jdbc:hive2://10.18.98.136:23040/default> create datamap uniqdata_agg_decimal
on table uniqdata_decimal using 'preaggregate' as select cust_name,
avg(DECIMAL_COLUMN1) from uniqdata_decimal group by DECIMAL_COLUMN1, cust_name;
+---------+--+
| Result |
+---------+--+
+---------+--+
No rows selected (0.557 seconds)
0: jdbc:hive2://10.18.98.136:23040/default> create datamap
uniqdata_agg_sum_decimal on table uniqdata_decimal using 'preaggregate' as
select cust_name, sum(DECIMAL_COLUMN1) from uniqdata_decimal group by
DECIMAL_COLUMN1, cust_name;
+---------+--+
| Result |
+---------+--+
+---------+--+
No rows selected (0.515 seconds)
0: jdbc:hive2://10.18.98.136:23040/default> create datamap
uniqdata_agg_count_decimal on table uniqdata_decimal using 'preaggregate' as
select cust_name, count(DECIMAL_COLUMN1) from uniqdata_decimal group by
DECIMAL_COLUMN1, cust_name;
+---------+--+
| Result |
+---------+--+
+---------+--+
No rows selected (0.492 seconds)
0: jdbc:hive2://10.18.98.136:23040/default> create datamap
uniqdata_agg_min_decimal on table uniqdata_decimal using 'preaggregate' as
select cust_name, min(DECIMAL_COLUMN1) from uniqdata_decimal group by
DECIMAL_COLUMN1, cust_name;
+---------+--+
| Result |
+---------+--+
+---------+--+
No rows selected (0.571 seconds)
0: jdbc:hive2://10.18.98.136:23040/default> create datamap
uniqdata_agg_max_decimal on table uniqdata_decimal using 'preaggregate' as
select cust_name, max(DECIMAL_COLUMN1) from uniqdata_decimal group by
DECIMAL_COLUMN1, cust_name;
+---------+--+
| Result |
+---------+--+
+---------+--+
No rows selected (0.471 seconds)
0: jdbc:hive2://10.18.98.136:23040/default> show datamap on table
uniqdata_decimal;
+-----------------------------+---------------+---------------------------------------------------+---------------------+--+
| DataMapName | ClassName | Associated Table | DataMap Properties |
+-----------------------------+---------------+---------------------------------------------------+---------------------+--+
| uniqdata_agg_decimal | preaggregate |
test.uniqdata_decimal_uniqdata_agg_decimal | |
| uniqdata_agg_sum_decimal | preaggregate |
test.uniqdata_decimal_uniqdata_agg_sum_decimal | |
| uniqdata_agg_count_decimal | preaggregate |
test.uniqdata_decimal_uniqdata_agg_count_decimal | |
| uniqdata_agg_min_decimal | preaggregate |
test.uniqdata_decimal_uniqdata_agg_min_decimal | |
| uniqdata_agg_max_decimal | preaggregate |
test.uniqdata_decimal_uniqdata_agg_max_decimal | |
+-----------------------------+---------------+---------------------------------------------------+---------------------+--+
5 rows selected (0.035 seconds)
0: jdbc:hive2://10.18.98.136:23040/default> LOAD DATA INPATH
'hdfs://hacluster/chetan/2000_UniqData.csv' into table uniqdata_decimal
OPTIONS('DELIMITER'=',',
'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1',
*'SINGLE_PASS'='true')*;
Issue : Load fails with incorrect error message as shown below.
0: jdbc:hive2://10.18.98.136:23040/default> LOAD DATA INPATH
'hdfs://hacluster/chetan/2000_UniqData.csv' into table uniqdata_decimal
OPTIONS('DELIMITER'=',',
'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1',
*'SINGLE_PASS'='true'*);
*Error: java.lang.Exception: DataLoad failure (state=,code=0)*
Expected : The error message should be displayed on load failure as single pass
should not be configured without precreated dictionary.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
