Prasanna Ravichandran created CARBONDATA-3908:
-------------------------------------------------
Summary: When a carbon segment is added through the alter add
segments query, then it is not accounting the added carbon segment values.
Key: CARBONDATA-3908
URL: https://issues.apache.org/jira/browse/CARBONDATA-3908
Project: CarbonData
Issue Type: Bug
Affects Versions: 2.0.0
Environment: FI cluster and opensource cluster.
Reporter: Prasanna Ravichandran
When a carbon segment is added through the alter add segments query, then it is
not accounting the added carbon segment values. If we do count(*) on the added
segment, then it is always showing as 0.
Test queries:
drop table if exists uniqdata;
CREATE TABLE uniqdata (cust_id int,cust_name String,active_emui_version string,
dob timestamp, doj timestamp, bigint_column1 bigint,bigint_column2
bigint,decimal_column1 decimal(30,10), decimal_column2
decimal(36,36),double_column1 double, double_column2 double,integer_column1
int) stored as carbondata;
load data inpath 'hdfs://hacluster/BabuStore/Data/2000_UniqData.csv' into table
uniqdata
options('fileheader'='cust_id,cust_name,active_emui_version,dob,doj,bigint_column1,bigint_column2,decimal_column1,decimal_column2,double_column1,double_column2,integer_column1','bad_records_action'='force');
--hdfs dfs -mkdir /uniqdata-carbon-segment;
--hdfs dfs -cp /user/hive/warehouse/uniqdata/Fact/Part0/Segment_0/*
/uniqdata-carbon-segment/
Alter table uniqdata add segment options
('path'='hdfs://hacluster/uniqdata-carbon-segment/','format'='carbon');
select count(*) from uniqdata;--4000 expected as one load of 2000 records
happened and same segment is added again;
set carbon.input.segments.default.uniqdata=1;
select count(*) from uniqdata;--2000 expected - it should just show the records
count of added segments;
CONSOLE:
/> set carbon.input.segments.default.uniqdata=1;
+-----------------------------------------+--------+
| key | value |
+-----------------------------------------+--------+
| carbon.input.segments.default.uniqdata | 1 |
+-----------------------------------------+--------+
1 row selected (0.192 seconds)
/> select count(*) from uniqdata;
INFO : Execution ID: 1734
+-----------+
| count(1) |
+-----------+
| 2000 |
+-----------+
1 row selected (4.036 seconds)
/> set carbon.input.segments.default.uniqdata=2;
+-----------------------------------------+--------+
| key | value |
+-----------------------------------------+--------+
| carbon.input.segments.default.uniqdata | 2 |
+-----------------------------------------+--------+
1 row selected (0.088 seconds)
/> select count(*) from uniqdata;
INFO : Execution ID: 1745
+-----------+
| count(1) |
+-----------+
| 2000 |
+-----------+
1 row selected (6.056 seconds)
/> set carbon.input.segments.default.uniqdata=3;
+-----------------------------------------+--------+
| key | value |
+-----------------------------------------+--------+
| carbon.input.segments.default.uniqdata | 3 |
+-----------------------------------------+--------+
1 row selected (0.161 seconds)
/> select count(*) from uniqdata;
INFO : Execution ID: 1753
+-----------+
| count(1) |
+-----------+
| 0 |
+-----------+
1 row selected (4.875 seconds)
/> show segments for table uniqdata;
+-----+----------+--------------------------+------------------+------------+------------+-------------+--------------+
| ID | Status | Load Start Time | Load Time Taken | Partition | Data Size |
Index Size | File Format |
+-----+----------+--------------------------+------------------+------------+------------+-------------+--------------+
| 4 | Success | 2020-07-17 16:01:53.673 | 5.579S | {} | 269.10KB | 7.21KB |
columnar_v3 |
| 3 | Success | 2020-07-17 16:00:24.866 | 0.578S | {} | 88.55KB | 1.81KB |
columnar_v3 |
| 2 | Success | 2020-07-17 15:07:54.273 | 0.642S | {} | 36.72KB | NA | orc |
| 1 | Success | 2020-07-17 15:03:59.767 | 0.564S | {} | 89.26KB | NA | parquet |
| 0 | Success | 2020-07-16 12:44:32.095 | 4.484S | {} | 88.55KB | 1.81KB |
columnar_v3 |
+-----+----------+--------------------------+------------------+------------+------------+-------------+--------------+
Expected result: Records added by adding carbon segment should be considered.
Actual result: Records added by adding carbon segment is not considered.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
