Vinod Rohilla created CARBONDATA-1029:
-----------------------------------------
Summary: Load data time difference with Single-pass load.
Key: CARBONDATA-1029
URL: https://issues.apache.org/jira/browse/CARBONDATA-1029
Project: CarbonData
Issue Type: Bug
Components: data-load
Affects Versions: 1.1.0
Environment: Spark 2.1, AWS Cluster
Reporter: Vinod Rohilla
Priority: Minor
Load data without single pass takes less time as compare to Single-pass load.
Note :CSV Size is 10.21 GB.
Result:
A) Data Load without Single Pass:
0: jdbc:hive2://localhost:10000> LOAD DATA INPATH
'hdfs://hadoop-master:54310/data/uniqdata_bench14.csv' into table uniqdata
OPTIONS('DELIMITER'=',' ,
'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');
+---------+--+
| Result |
+---------+--+
+---------+--+
No rows selected (114.641 seconds)
B) Load Data with Single Pass:
0: jdbc:hive2://localhost:10000> LOAD DATA INPATH
'hdfs://hadoop-master:54310/data/uniqdata_bench14.csv' into table uniqdata
OPTIONS('DELIMITER'=',' ,
'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','SINGLE_Pass'='true');
+---------+--+
| Result |
+---------+--+
+---------+--+
No rows selected (123.858 seconds)
Expected Result: If user load data with Single-pass then it should take less
time as compare to without single pass load.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
