[jira] [Updated] (CARBONDATA-4029) After delete in the table which has Alter-added SDK segments, then the count(*) is 0.

2021-03-16 Thread Ajantha Bhat (Jira) 


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-4029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ajantha Bhat updated CARBONDATA-4029:
-
Fix Version/s: (was: 2.2.0)
   2.1.1

> After delete in the table which has Alter-added SDK segments, then the 
> count(*) is 0.
> -
>
> Key: CARBONDATA-4029
> URL: https://issues.apache.org/jira/browse/CARBONDATA-4029
> Project: CarbonData
>  Issue Type: Bug
>Affects Versions: 2.0.0
> Environment: 3 node FI cluster
>Reporter: Prasanna Ravichandran
>Priority: Minor
> Fix For: 2.1.1
>
> Attachments: Primitive.rar
>
>  Time Spent: 7h 10m
>  Remaining Estimate: 0h
>
> Do delete on a table which has alter added SDK segments. then the count* is 
> 0. Even count* will be 0 even any number of SDK segments are added after it.
> Test queries:
> drop table if exists external_primitive;
> create table external_primitive (id int, name string, rank smallint, salary 
> double, active boolean, dob date, doj timestamp, city string, dept string) 
> stored as carbondata;
> --before executing the below alter add segment-place the attached SDK files 
> in hdfs at /sdkfiles/primitive2 folder;
> alter table external_primitive add segment 
> options('path'='hdfs://hacluster/sdkfiles/primitive2','format'='carbon');select
>  * from external_primitive;
> delete from external_primitive where id =2;select * from external_primitive;
> Console output:
> /> drop table if exists external_primitive;
> +-+
> | Result |
> +-+
> +-+
> No rows selected (1.586 seconds)
> /> create table external_primitive (id int, name string, rank smallint, 
> salary double, active boolean, dob date, doj timestamp, city string, dept 
> string) stored as carbondata;
> +-+
> | Result |
> +-+
> +-+
> No rows selected (0.774 seconds)
> /> alter table external_primitive add segment 
> options('path'='hdfs://hacluster/sdkfiles/primitive2','format'='carbon');select
>  * from external_primitive;
> +-+
> | Result |
> +-+
> +-+
> No rows selected (1.077 seconds)
> INFO : Execution ID: 320
> +-+---+---+--+-+-++++
> | id | name | rank | salary | active | dob | doj | city | dept |
> +-+---+---+--+-+-++++
> | 1 | AAA | 3 | 3444345.66 | true | 1979-12-09 | 2011-02-10 01:00:20.0 | Pune 
> | IT |
> | 2 | BBB | 2 | 543124.66 | false | 1987-02-19 | 2017-01-01 12:00:20.0 | 
> Bangalore | DATA |
> | 3 | CCC | 1 | 787878.888 | false | 1982-05-12 | 2015-12-01 02:20:20.0 | 
> Pune | DATA |
> | 4 | DDD | 1 | 9.24 | true | 1981-04-09 | 2000-01-15 07:00:20.0 | Delhi 
> | MAINS |
> | 5 | EEE | 3 | 545656.99 | true | 1987-12-09 | 2017-11-25 04:00:20.0 | Delhi 
> | IT |
> | 6 | FFF | 2 | 768678.0 | false | 1987-12-20 | 2017-01-10 05:00:20.0 | 
> Bangalore | DATA |
> | 7 | GGG | 3 | 765665.0 | true | 1983-06-12 | 2017-01-01 02:00:20.0 | Pune | 
> IT |
> | 8 | HHH | 2 | 567567.66 | false | 1979-01-12 | 1995-01-01 12:00:20.0 | 
> Bangalore | DATA |
> | 9 | III | 2 | 787878.767 | true | 1985-02-19 | 2005-08-15 01:00:20.0 | Pune 
> | DATA |
> | 10 | JJJ | 3 | 887877.14 | true | 2000-05-19 | 2016-10-10 12:00:20.0 | 
> Bangalore | MAINS |
> | 18 | | 3 | 7.86786786787E9 | true | 1980-10-05 | 1995-10-07 22:00:20.0 | 
> Bangalore | IT |
> | 19 | | 2 | 5464545.33 | true | 1986-06-06 | 2008-08-15 01:00:20.0 | Delhi | 
> DATA |
> | 20 | NULL | 3 | 7867867.34 | true | 2000-05-01 | 2014-01-18 12:00:20.0 | 
> Bangalore | MAINS |
> +-+---+---+--+-+-++++
> 13 rows selected (2.458 seconds)
> /> delete from external_primitive where id =2;select * from 
> external_primitive;
> INFO : Execution ID: 322
> ++
> | Deleted Row Count |
> ++
> | 1 |
> ++
> 1 row selected (3.723 seconds)
> +-+---+---+-+-+--+--+---+---+
> | id | name | rank | salary | active | dob | doj | city | dept |
> +-+---+---+-+-+--+--+---+---+
> +-+---+---+-+-+--+--+---+---+
> No rows selected (1.531 seconds)
> /> alter table external_primitive add segment 
> options('path'='hdfs://hacluster/sdkfiles/primitive3','format'='carbon');select
>  * from external_primitive;
> +-+
> | Result |
> +-+
> +-+
> No rows selected (0.766 seconds)
> +-+---+---+-+-+--+--+---+---+
> | id | name | rank | salary | active | dob | doj | cit

[jira] [Updated] (CARBONDATA-4029) After delete in the table which has Alter-added SDK segments, then the count(*) is 0.

2020-10-30 Thread Prasanna Ravichandran (Jira) 


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-4029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanna Ravichandran updated CARBONDATA-4029:
--
Description: 
Do delete on a table which has alter added SDK segments. then the count* is 0. 
Even count* will be 0 even any number of SDK segments are added after it.

Test queries:

drop table if exists external_primitive;
create table external_primitive (id int, name string, rank smallint, salary 
double, active boolean, dob date, doj timestamp, city string, dept string) 
stored as carbondata;

--before executing the below alter add segment-place the attached SDK files in 
hdfs at /sdkfiles/primitive2 folder;

alter table external_primitive add segment 
options('path'='hdfs://hacluster/sdkfiles/primitive2','format'='carbon');select 
* from external_primitive;
delete from external_primitive where id =2;select * from external_primitive;

Console output:

/> drop table if exists external_primitive;
+-+
| Result |
+-+
+-+
No rows selected (1.586 seconds)
/> create table external_primitive (id int, name string, rank smallint, salary 
double, active boolean, dob date, doj timestamp, city string, dept string) 
stored as carbondata;
+-+
| Result |
+-+
+-+
No rows selected (0.774 seconds)

/> alter table external_primitive add segment 
options('path'='hdfs://hacluster/sdkfiles/primitive2','format'='carbon');select 
* from external_primitive;
+-+
| Result |
+-+
+-+
No rows selected (1.077 seconds)
INFO : Execution ID: 320
+-+---+---+--+-+-++++
| id | name | rank | salary | active | dob | doj | city | dept |
+-+---+---+--+-+-++++
| 1 | AAA | 3 | 3444345.66 | true | 1979-12-09 | 2011-02-10 01:00:20.0 | Pune | 
IT |
| 2 | BBB | 2 | 543124.66 | false | 1987-02-19 | 2017-01-01 12:00:20.0 | 
Bangalore | DATA |
| 3 | CCC | 1 | 787878.888 | false | 1982-05-12 | 2015-12-01 02:20:20.0 | Pune 
| DATA |
| 4 | DDD | 1 | 9.24 | true | 1981-04-09 | 2000-01-15 07:00:20.0 | Delhi | 
MAINS |
| 5 | EEE | 3 | 545656.99 | true | 1987-12-09 | 2017-11-25 04:00:20.0 | Delhi | 
IT |
| 6 | FFF | 2 | 768678.0 | false | 1987-12-20 | 2017-01-10 05:00:20.0 | 
Bangalore | DATA |
| 7 | GGG | 3 | 765665.0 | true | 1983-06-12 | 2017-01-01 02:00:20.0 | Pune | 
IT |
| 8 | HHH | 2 | 567567.66 | false | 1979-01-12 | 1995-01-01 12:00:20.0 | 
Bangalore | DATA |
| 9 | III | 2 | 787878.767 | true | 1985-02-19 | 2005-08-15 01:00:20.0 | Pune | 
DATA |
| 10 | JJJ | 3 | 887877.14 | true | 2000-05-19 | 2016-10-10 12:00:20.0 | 
Bangalore | MAINS |
| 18 | | 3 | 7.86786786787E9 | true | 1980-10-05 | 1995-10-07 22:00:20.0 | 
Bangalore | IT |
| 19 | | 2 | 5464545.33 | true | 1986-06-06 | 2008-08-15 01:00:20.0 | Delhi | 
DATA |
| 20 | NULL | 3 | 7867867.34 | true | 2000-05-01 | 2014-01-18 12:00:20.0 | 
Bangalore | MAINS |
+-+---+---+--+-+-++++
13 rows selected (2.458 seconds)
/> delete from external_primitive where id =2;select * from external_primitive;
INFO : Execution ID: 322
++
| Deleted Row Count |
++
| 1 |
++
1 row selected (3.723 seconds)
+-+---+---+-+-+--+--+---+---+
| id | name | rank | salary | active | dob | doj | city | dept |
+-+---+---+-+-+--+--+---+---+
+-+---+---+-+-+--+--+---+---+
No rows selected (1.531 seconds)
/> alter table external_primitive add segment 
options('path'='hdfs://hacluster/sdkfiles/primitive3','format'='carbon');select 
* from external_primitive;
+-+
| Result |
+-+
+-+
No rows selected (0.766 seconds)
+-+---+---+-+-+--+--+---+---+
| id | name | rank | salary | active | dob | doj | city | dept |
+-+---+---+-+-+--+--+---+---+
+-+---+---+-+-+--+--+---+---+
No rows selected (1.439 seconds)
/> select count(*) from external_primitive;
INFO : Execution ID: 335
+---+
| count(1) |
+---+
| 0 |
+---+
1 row selected (1.278 seconds)
/>

> After delete in the table which has Alter-added SDK segments, then the 
> count(*) is 0.
> -
>
> Key: CARBONDATA-4029
> URL: https://issues.apache.org/jira/browse/CARBONDATA-4029
> Project: CarbonData
>  Issue Type: Bug
>Affects Versions: 2.0.0
> Environment: 3 node FI cluster
>Reporter: Prasanna Ravichandran
>

[jira] [Updated] (CARBONDATA-4029) After delete in the table which has Alter-added SDK segments, then the count(*) is 0.

2020-10-30 Thread Prasanna Ravichandran (Jira) 


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-4029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanna Ravichandran updated CARBONDATA-4029:
--
Description: (was: We are getting Number format exception while 
querying on the date columns. Attached the SDK files also.

Test queries:

--SDK compaction;
 drop table if exists external_primitive;
 create table external_primitive (id int, name string, rank smallint, salary 
double, active boolean, dob date, doj timestamp, city string, dept string) 
stored as carbondata;
 alter table external_primitive add segment 
options('path'='hdfs://hacluster/sdkfiles/primitive','format'='carbon');
 alter table external_primitive add segment 
options('path'='hdfs://hacluster/sdkfiles/primitive2','format'='carbon');
 alter table external_primitive add segment 
options('path'='hdfs://hacluster/sdkfiles/primitive3','format'='carbon');
 alter table external_primitive add segment 
options('path'='hdfs://hacluster/sdkfiles/primitive4','format'='carbon');
 alter table external_primitive add segment 
options('path'='hdfs://hacluster/sdkfiles/primitive5','format'='carbon');
 
 alter table external_primitive compact 'minor'; --working fine pass;
 select count(*) from external_primitive;--working fine pass;

show segments for table external_primitive;
 select * from external_primitive limit 13; --working fine pass;
 select * from external_primitive limit 14; --failed getting number format 
exception;
select min(dob) from external_primitive; --failed getting number format 
exception;
select max(dob) from external_primitive; --working;
select dob from external_primitive; --failed getting number format exception;

Console:

*0: /> show segments for table external_primitive;*
+--++--+--+++-+--+
| ID | Status | Load Start Time | Load Time Taken | Partition | Data Size | 
Index Size | File Format |
+--++--+--+++-+--+
| 4 | Success | 2020-10-13 11:52:04.012 | 0.511S | {} | 1.88KB | 655.0B | 
columnar_v3 |
| 3 | Compacted | 2020-10-13 11:52:00.587 | 0.828S | {} | 1.88KB | 655.0B | 
columnar_v3 |
| 2 | Compacted | 2020-10-13 11:51:57.767 | 0.775S | {} | 1.88KB | 655.0B | 
columnar_v3 |
| 1 | Compacted | 2020-10-13 11:51:54.678 | 1.024S | {} | 1.88KB | 655.0B | 
columnar_v3 |
| 0.1 | Success | 2020-10-13 11:52:05.986 | 5.785S | {} | 9.62KB | 5.01KB | 
columnar_v3 |
| 0 | Compacted | 2020-10-13 11:51:51.072 | 1.125S | {} | 8.55KB | 4.25KB | 
columnar_v3 |
+--++--+--+++-+--+
6 rows selected (0.45 seconds)
*0: /> select * from external_primitive limit 13;* --working fine pass;
INFO : Execution ID: 95
+-+---+---+--+-+-++++
| id | name | rank | salary | active | dob | doj | city | dept |
+-+---+---+--+-+-++++
| 1 | AAA | 3 | 3444345.66 | true | 1979-12-09 | 2011-02-09 22:30:20.0 | Pune | 
IT |
| 2 | BBB | 2 | 543124.66 | false | 1987-02-19 | 2017-01-01 09:30:20.0 | 
Bangalore | DATA |
| 3 | CCC | 1 | 787878.888 | false | 1982-05-12 | 2015-11-30 23:50:20.0 | Pune 
| DATA |
| 4 | DDD | 1 | 9.24 | true | 1981-04-09 | 2000-01-15 04:30:20.0 | Delhi | 
MAINS |
| 5 | EEE | 3 | 545656.99 | true | 1987-12-09 | 2017-11-25 01:30:20.0 | Delhi | 
IT |
| 6 | FFF | 2 | 768678.0 | false | 1987-12-20 | 2017-01-10 02:30:20.0 | 
Bangalore | DATA |
| 7 | GGG | 3 | 765665.0 | true | 1983-06-12 | 2016-12-31 23:30:20.0 | Pune | 
IT |
| 8 | HHH | 2 | 567567.66 | false | 1979-01-12 | 1995-01-01 09:30:20.0 | 
Bangalore | DATA |
| 9 | III | 2 | 787878.767 | true | 1985-02-19 | 2005-08-14 22:30:20.0 | Pune | 
DATA |
| 10 | JJJ | 3 | 887877.14 | true | 2000-05-19 | 2016-10-10 09:30:20.0 | 
Bangalore | MAINS |
| 18 | | 3 | 7.86786786787E9 | true | 1980-10-05 | 1995-10-07 19:30:20.0 | 
Bangalore | IT |
| 19 | | 2 | 5464545.33 | true | 1986-06-06 | 2008-08-14 22:30:20.0 | Delhi | 
DATA |
| 20 | NULL | 3 | 7867867.34 | true | 2000-05-01 | 2014-01-18 09:30:20.0 | 
Bangalore | MAINS |
+-+---+---+--+-+-++++
13 rows selected (1.775 seconds)
*0: /> select * from external_primitive limit 14;* --failed getting number 
format exception;
INFO : Execution ID: 97
*java.lang.NumberFormatException: For input string: "776"*
 at 
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
 at java.lang.Integer.parseInt(Integer.java:569)
 at java.lang.Integer.parseInt(Integer.java:615)
 at java.sql.Date.valueOf(Date.java:133)
 at org.apache.hive.jdbc.HiveBaseR

[jira] [Updated] (CARBONDATA-4029) After delete in the table which has Alter-added SDK segments, then the count(*) is 0.

2020-10-30 Thread Prasanna Ravichandran (Jira) 


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-4029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanna Ravichandran updated CARBONDATA-4029:
--
Summary: After delete in the table which has Alter-added SDK segments, then 
the count(*) is 0.  (was: Getting Number format exception while querying on 
date columns in SDK carbon table.)

> After delete in the table which has Alter-added SDK segments, then the 
> count(*) is 0.
> -
>
> Key: CARBONDATA-4029
> URL: https://issues.apache.org/jira/browse/CARBONDATA-4029
> Project: CarbonData
>  Issue Type: Bug
>Affects Versions: 2.0.0
> Environment: 3 node FI cluster
>Reporter: Prasanna Ravichandran
>Priority: Minor
> Attachments: Primitive.rar
>
>
> We are getting Number format exception while querying on the date columns. 
> Attached the SDK files also.
> Test queries:
> --SDK compaction;
>  drop table if exists external_primitive;
>  create table external_primitive (id int, name string, rank smallint, salary 
> double, active boolean, dob date, doj timestamp, city string, dept string) 
> stored as carbondata;
>  alter table external_primitive add segment 
> options('path'='hdfs://hacluster/sdkfiles/primitive','format'='carbon');
>  alter table external_primitive add segment 
> options('path'='hdfs://hacluster/sdkfiles/primitive2','format'='carbon');
>  alter table external_primitive add segment 
> options('path'='hdfs://hacluster/sdkfiles/primitive3','format'='carbon');
>  alter table external_primitive add segment 
> options('path'='hdfs://hacluster/sdkfiles/primitive4','format'='carbon');
>  alter table external_primitive add segment 
> options('path'='hdfs://hacluster/sdkfiles/primitive5','format'='carbon');
>  
>  alter table external_primitive compact 'minor'; --working fine pass;
>  select count(*) from external_primitive;--working fine pass;
> show segments for table external_primitive;
>  select * from external_primitive limit 13; --working fine pass;
>  select * from external_primitive limit 14; --failed getting number format 
> exception;
> select min(dob) from external_primitive; --failed getting number format 
> exception;
> select max(dob) from external_primitive; --working;
> select dob from external_primitive; --failed getting number format exception;
> Console:
> *0: /> show segments for table external_primitive;*
> +--++--+--+++-+--+
> | ID | Status | Load Start Time | Load Time Taken | Partition | Data Size | 
> Index Size | File Format |
> +--++--+--+++-+--+
> | 4 | Success | 2020-10-13 11:52:04.012 | 0.511S | {} | 1.88KB | 655.0B | 
> columnar_v3 |
> | 3 | Compacted | 2020-10-13 11:52:00.587 | 0.828S | {} | 1.88KB | 655.0B | 
> columnar_v3 |
> | 2 | Compacted | 2020-10-13 11:51:57.767 | 0.775S | {} | 1.88KB | 655.0B | 
> columnar_v3 |
> | 1 | Compacted | 2020-10-13 11:51:54.678 | 1.024S | {} | 1.88KB | 655.0B | 
> columnar_v3 |
> | 0.1 | Success | 2020-10-13 11:52:05.986 | 5.785S | {} | 9.62KB | 5.01KB | 
> columnar_v3 |
> | 0 | Compacted | 2020-10-13 11:51:51.072 | 1.125S | {} | 8.55KB | 4.25KB | 
> columnar_v3 |
> +--++--+--+++-+--+
> 6 rows selected (0.45 seconds)
> *0: /> select * from external_primitive limit 13;* --working fine pass;
> INFO : Execution ID: 95
> +-+---+---+--+-+-++++
> | id | name | rank | salary | active | dob | doj | city | dept |
> +-+---+---+--+-+-++++
> | 1 | AAA | 3 | 3444345.66 | true | 1979-12-09 | 2011-02-09 22:30:20.0 | Pune 
> | IT |
> | 2 | BBB | 2 | 543124.66 | false | 1987-02-19 | 2017-01-01 09:30:20.0 | 
> Bangalore | DATA |
> | 3 | CCC | 1 | 787878.888 | false | 1982-05-12 | 2015-11-30 23:50:20.0 | 
> Pune | DATA |
> | 4 | DDD | 1 | 9.24 | true | 1981-04-09 | 2000-01-15 04:30:20.0 | Delhi 
> | MAINS |
> | 5 | EEE | 3 | 545656.99 | true | 1987-12-09 | 2017-11-25 01:30:20.0 | Delhi 
> | IT |
> | 6 | FFF | 2 | 768678.0 | false | 1987-12-20 | 2017-01-10 02:30:20.0 | 
> Bangalore | DATA |
> | 7 | GGG | 3 | 765665.0 | true | 1983-06-12 | 2016-12-31 23:30:20.0 | Pune | 
> IT |
> | 8 | HHH | 2 | 567567.66 | false | 1979-01-12 | 1995-01-01 09:30:20.0 | 
> Bangalore | DATA |
> | 9 | III | 2 | 787878.767 | true | 1985-02-19 | 2005-08-14 22:30:20.0 | Pune 
> | DATA |
> | 10 | JJJ | 3 | 887877.14 | true | 2000-05-19 | 2016-10-10 09:30:20.0 | 
> Bangalore | MAINS |
>