[jira] [Created] (CARBONDATA-3036) Carbon 1.5.0 B010 - Select query fails when min/max exceeds and index tree cached

[Ramakrishna S (JIRA)]

Ramakrishna S created CARBONDATA-3036:
-

 Summary: Carbon 1.5.0 B010 - Select query fails when min/max 
exceeds and index tree cached
 Key: CARBONDATA-3036
 URL: https://issues.apache.org/jira/browse/CARBONDATA-3036
 Project: CarbonData
  Issue Type: Bug
  Components: data-query
Affects Versions: 1.5.0
 Environment: 17 node cluster 
0: jdbc:hive2://192.168.137.14:22550/default> show segments for table 
jinling_localsort_3;
++--+--+--++--++-+--+
| SegmentSequenceId  |  Status  | Load Start Time  |  Load End Time 
  | Merged To  | File Format  | Data Size  | Index Size  |
++--+--+--++--++-+--+
| 8  | Success  | 2018-10-17 08:30:23.103  | 2018-10-17 
08:30:28.476  | NA | COLUMNAR_V3  | 185.68MB   | 243.36KB|
| 7  | Success  | 2018-10-17 08:30:16.359  | 2018-10-17 
08:30:21.667  | NA | COLUMNAR_V3  | 185.68MB   | 243.36KB|
| 6  | Success  | 2018-10-17 08:30:09.439  | 2018-10-17 
08:30:14.97   | NA | COLUMNAR_V3  | 185.68MB   | 243.36KB|
| 5  | Success  | 2018-10-17 08:30:02.688  | 2018-10-17 
08:30:08.036  | NA | COLUMNAR_V3  | 186.02MB   | 243.35KB|
| 4  | Success  | 2018-10-17 08:27:09.09   | 2018-10-17 
08:27:14.553  | NA | COLUMNAR_V3  | 186.34MB   | 243.35KB|
| 3  | Success  | 2018-10-17 08:27:00.659  | 2018-10-17 
08:27:06.096  | NA | COLUMNAR_V3  | 185.68MB   | 243.36KB|
| 2  | Success  | 2018-10-17 08:26:52.047  | 2018-10-17 
08:26:57.47   | NA | COLUMNAR_V3  | 186.02MB   | 243.35KB|
| 1  | Success  | 2018-10-17 08:26:43.462  | 2018-10-17 
08:26:48.899  | NA | COLUMNAR_V3  | 186.34MB   | 243.35KB|
| 0  | Success  | 2018-10-17 08:20:46.654  | 2018-10-17 
08:20:52.681  | NA | COLUMNAR_V3  | 186.02MB   | 243.35KB|

Reporter: Ramakrishna S


If configured cache columns and min, max present flag columns is different*, it 
is possible to get wrong data* due to wrong index is mapped.

results inconsistent when cache is set but min/max exceeds. Column is 
dictionary excluded.

set carbon.minmax.allowed.byte.count=50;

 

0: jdbc:hive2://192.168.137.14:22550/default> use db01;
+-+--+
| Result |
+-+--+
+-+--+
No rows selected (0.06 seconds)
0: jdbc:hive2://192.168.137.14:22550/default> show tables;
+---+--+--+--+
| database | tableName | isTemporary |
+---+--+--+--+
| db01 | jinling_localsort_3 | false |
+---+--+--+--+
1 row selected (0.068 seconds)
0: jdbc:hive2://192.168.137.14:22550/default> alter table jinling_localsort_3 
set TBLPROPERTIES('COLUMN_META_CACHE'='user_imsi,user_num,dim24');
+-+--+
| Result |
+-+--+
+-+--+
No rows selected (3.495 seconds)
0: jdbc:hive2://192.168.137.14:22550/default> select count(*) from 
jinling_localsort_3;
+---+--+
| count(1) |
+---+--+
| 900 |
+---+--+
1 row selected (2.606 seconds)
0: jdbc:hive2://192.168.137.14:22550/default> select dim24 from 
jinling_localsort_3 limit 2;
++--+
| dim24 |
++--+
| E6Phek Hobbit from the Shire and eight companions set out on a journey to 
destroy the powerful One Ring and save Middle-earth from the Dark Lord 
Sauron.A |
| Q4Plek Hobbit from the Shire and eight companions set out on a journey to 
destroy the powerful One Ring and save Middle-earth from the Dark Lord 
Sauron.A |
++--+
2 rows selected (457.462 seconds)
0: jdbc:hive2://192.168.137.14:22550/default> select count(*) from 
jinling_localsort_3 where dim24 = 'E6Phek Hobbit from the Shire and eight 
companions set out on a journey to destroy the powerful One Ring and save 
Middle-earth from the Dark Lord 

[jira] [Resolved] (CARBONDATA-1824) Carbon 1.3.0 - Spark 2.2-Residual segment files left over when load failure happens

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S resolved CARBONDATA-1824.
---
Resolution: Fixed

Resolved under Carbondata-1759

> Carbon 1.3.0 - Spark 2.2-Residual segment files left over when load failure 
> happens
> ---
>
> Key: CARBONDATA-1824
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1824
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-load
>Affects Versions: 1.3.0
> Environment: Test - 3 node ant cluster
>Reporter: Ramakrishna S
>Assignee: dhatchayani
>Priority: Minor
>  Labels: DFX
> Fix For: 1.3.0
>
>
> Steps:
> Beeline:
> 1. Create a table with batch sort as sort type, keep block size small
> 2. Run Load/Insert/Compaction the table
> 3. Bring down thrift server when carbon data is being written to the segment
> 4. Do show segments on the table
> *+Expected:+* It should not show the residual segments  
> *+Actual:+* The segment intended for load is shown as marked for delete and 
> it does not get deleted with clean file. No impact on the table as such.
> *+Query:+*
> create table if not exists lineitem1(L_SHIPDATE string,L_SHIPMODE 
> string,L_SHIPINSTRUCT string,L_RETURNFLAG string,L_RECEIPTDATE 
> string,L_ORDERKEY string,L_PARTKEY string,L_SUPPKEY   string,L_LINENUMBER 
> int,L_QUANTITY double,L_EXTENDEDPRICE double,L_DISCOUNT double,L_TAX 
> double,L_LINESTATUS string,L_COMMITDATE string,L_COMMENT  string) STORED BY 
> 'org.apache.carbondata.format' TBLPROPERTIES 
> ('table_blocksize'='1','sort_scope'='BATCH_SORT','batch_sort_size_inmb'='5000');
> load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table 
> lineitem 
> options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');
> 0: jdbc:hive2://10.18.98.34:23040> select count(*) from t_carbn0161;
> +---+--+
> | count(1)  |
> +---+--+
> | 0 |
> +---+--+
> 1 row selected (13.011 seconds)
> 0: jdbc:hive2://10.18.98.34:23040> show segments for table lineitem1;
> +++--+--++--+--+
> | SegmentSequenceId  |   Status   | Load Start Time  |  
> Load End Time   | Merged To  | File Format  |
> +++--+--++--+--+
> | 1  | Marked for Delete  | 2017-11-28 19:14:46.265  | 
> 2017-11-28 19:15:28.396  | NA | COLUMNAR_V3  |
> | 0  | Marked for Delete  | 2017-11-28 19:12:58.269  | 
> 2017-11-28 19:13:37.26   | NA | COLUMNAR_V3  |
> +++--+--++--+--+
> 0: jdbc:hive2://10.18.98.34:23040> clean files for table t_carbn0161;
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (7.473 seconds)
> 0: jdbc:hive2://10.18.98.34:23040> show segments for table lineitem1;
> +++--+--++--+--+
> | SegmentSequenceId  |   Status   | Load Start Time  |  
> Load End Time   | Merged To  | File Format  |
> +++--+--++--+--+
> | 1  | Marked for Delete  | 2017-11-28 19:14:46.265  | 
> 2017-11-28 19:15:28.396  | NA | COLUMNAR_V3  |
> | 0  | Marked for Delete  | 2017-11-28 19:12:58.269  | 
> 2017-11-28 19:13:37.26   | NA | COLUMNAR_V3  |
> +++--+--++--+--+



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (CARBONDATA-1807) Carbon1.3.0-Pre-AggregateTable - Pre-aggregate creation not throwing error for wrong syntax and results in further query failures

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S closed CARBONDATA-1807.
-
Resolution: Fixed

This issue has been fixed now.

> Carbon1.3.0-Pre-AggregateTable - Pre-aggregate creation not throwing error 
> for wrong syntax and results in further query failures
> -
>
> Key: CARBONDATA-1807
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1807
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-load
>Affects Versions: 1.3.0
> Environment: Test - 3 node ant cluster
>Reporter: Ramakrishna S
>Assignee: kumar vishal
>Priority: Minor
>  Labels: DFX
> Fix For: 1.3.0
>
>
> Steps:
> Beeline:
> 1. Create table and load with  data
> 2. create a pre-aggregate table with incorrect syntax.
> 3. Run select count(*) on aggregate table
> 4. Run aggregate select query on main table
> *+Expected:+* Pre-aggregate table creation should have trhown syntax error 
> *+Actual:+* Pre-aggregate table is shown successful, but aggregate query 
> fails.
> *+Query:+*
> create table if not exists JL_r31
> (
> p_cap_time String,
> city String,
> product_code String,
> user_base_station String,
> user_belong_area_code String,
> user_num String,
> user_imsi String,
> user_id String,
> user_msisdn String,
> dim1 String,
> dim2 String,
> dim3 String,
> dim4 String,
> dim5 String,
> dim6 String,
> dim7 String,
> dim8 String,
> dim9 String,
> dim10 String,
> dim11 String,
> dim12 String,
> dim13 String,
> dim14 String,
> dim15 String,
> dim16 String,
> dim17 String,
> dim18 String,
> dim19 String,
> dim20 String,
> dim21 String,
> dim22 String,
> dim23 String,
> dim24 String,
> dim25 String,
> dim26 String,
> dim27 String,
> dim28 String,
> dim29 String,
> dim30 String,
> dim31 String,
> dim32 String,
> dim33 String,
> dim34 String,
> dim35 String,
> dim36 String,
> dim37 String,
> dim38 String,
> dim39 String,
> dim40 String,
> dim41 String,
> dim42 String,
> dim43 String,
> dim44 String,
> dim45 String,
> dim46 String,
> dim47 String,
> dim48 String,
> dim49 String,
> dim50 String,
> dim51 String,
> dim52 String,
> dim53 String,
> dim54 String,
> dim55 String,
> dim56 String,
> dim57 String,
> dim58 String,
> dim59 String,
> dim60 String,
> dim61 String,
> dim62 String,
> dim63 String,
> dim64 String,
> dim65 String,
> dim66 String,
> dim67 String,
> dim68 String,
> dim69 String,
> dim70 String,
> dim71 String,
> dim72 String,
> dim73 String,
> dim74 String,
> dim75 String,
> dim76 String,
> dim77 String,
> dim78 String,
> dim79 String,
> dim80 String,
> dim81 String,
> M1 double,
> M2 double,
> M3 double,
> M4 double,
> M5 double,
> M6 double,
> M7 double,
> M8 double,
> M9 double,
> M10 double )
> stored by 'org.apache.carbondata.format' 
> TBLPROPERTIES('DICTIONARY_EXCLUDE'='user_num,user_imsi,user_ID,user_msisdn,user_base_station,user_belong_area_code','table_blocksize'='512');
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (0.55 seconds)
> LOAD DATA inpath 'hdfs://hacluster/user/test/jin_test2.csv' into table JL_r31 
> options('DELIMITER'=',', 
> 'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','IS_EMPTY_DATA_BAD_RECORD'='TRUE','FILEHEADER'='p_cap_time,city,product_code,user_base_station,user_belong_area_code,user_num,user_imsi,user_id,user_msisdn,dim1,dim2,dim3,dim4,dim5,dim6,dim7,dim8,dim9,dim10,dim11,dim12,dim13,dim14,dim15,dim16,dim17,dim18,dim19,dim20,dim21,dim22,dim23,dim24,dim25,dim26,dim27,dim28,dim29,dim30,dim31,dim32,dim33,dim34,dim35,dim36,dim37,dim38,dim39,dim40,dim41,dim42,dim43,dim44,dim45,dim46,dim47,dim48,dim49,dim50,dim51,dim52,dim53,dim54,dim55,dim56,dim57,dim58,dim59,dim60,dim61,dim62,dim63,dim64,dim65,dim66,dim67,dim68,dim69,dim70,dim71,dim72,dim73,dim74,dim75,dim76,dim77,dim78,dim79,dim80,dim81,M1,M2,M3,M4,M5,M6,M7,M8,M9,M10');
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (14.049 seconds)
> 0: jdbc:hive2://10.18.98.136:23040> create datamap agr_JL_r31 ON TABLE JL_r31 
> USING 'org.apache.carbondta.datamap.AggregateDataMapHandler' as select 
> user_num,user_imsi,sum(user_id),count(user_id) from JL_r31 group by  
> user_num, user_imsi;
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (0.397 seconds)
> 0: jdbc:hive2://10.18.98.136:23040> select count(*) from JL_r31_agr_JL_r31;
> Error: org.apache.spark.sql.AnalysisException: Table or view not found: 
> JL_r31_agr_JL_r31; line 1 pos 21 (state=,code=0)
> 0: jdbc:hive2://10.18.98.136:23040> select 
> user_num,user_imsi,sum(user_id),count(user_id) from JL_r31 group by  
> user_num, user_imsi;
> Error: java.lang.ClassCastException: 
> 

[jira] [Closed] (CARBONDATA-1790) (Carbon1.3.0 - Streaming) Data load in Stream Segment fails if batch load is performed in between the streaming

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S closed CARBONDATA-1790.
-
Resolution: Fixed

This is is resolved in the B007 release.

> (Carbon1.3.0 - Streaming) Data load in Stream Segment fails if batch load is 
> performed in between the streaming
> ---
>
> Key: CARBONDATA-1790
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1790
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-query
>Affects Versions: 1.3.0
> Environment: 3 node ant cluster
>Reporter: Ramakrishna S
>Assignee: Bhavya Aggarwal
>  Labels: DFX
>
> Steps :
> 1. Create a streaming table and do a batch load
> 2. Set up the Streaming , so that it does streaming in chunk of 1000 records 
> 20 times
> 3. Do another batch load on the table
> 4. Do one more time streaming
> +-++--+--+--++--+
> | Segment Id  |   Status   | Load Start Time  |  Load End Time
>| File Format  | Merged To  |
> +-++--+--+--++--+
> | 2   | Success| 2017-11-21 21:42:36.77   | 2017-11-21 
> 21:42:40.396  | COLUMNAR_V3  | NA |
> | 1   | Streaming  | 2017-11-21 21:40:46.2| NULL  
>| ROW_V1   | NA |
> | 0   | Success| 2017-11-21 21:40:39.782  | 2017-11-21 
> 21:40:43.168  | COLUMNAR_V3  | NA |
> +-++--+--+--++--+
> *+Expected:+* Data should be loaded
> *+Actual+* : Data load fiails
> 1. One addition offset file is created(marked in bold)
> -rw-r--r--   2 root users 62 2017-11-21 21:40 
> /user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/0
> -rw-r--r--   2 root users 63 2017-11-21 21:40 
> /user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/1
> -rw-r--r--   2 root users 63 2017-11-21 21:42 
> /user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/10
> -rw-r--r--   2 root users 63 2017-11-21 21:40 
> /user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/2
> -rw-r--r--   2 root users 63 2017-11-21 21:41 
> /user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/3
> -rw-r--r--   2 root users 64 2017-11-21 21:41 
> /user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/4
> -rw-r--r--   2 root users 64 2017-11-21 21:41 
> /user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/5
> -rw-r--r--   2 root users 64 2017-11-21 21:41 
> /user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/6
> -rw-r--r--   2 root users 64 2017-11-21 21:41 
> /user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/7
> -rw-r--r--   2 root users 64 2017-11-21 21:41 
> /user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/8
> *-rw-r--r--   2 root users 63 2017-11-21 21:42 
> /user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/9*
> 2. Following error thrown:
> === Streaming Query ===
> Identifier: [id = 3a5334bc-d471-4676-b6ce-f21105d491d1, runId = 
> b2be9f97-8141-46be-89db-9a0f98d13369]
> Current Offsets: 
> {org.apache.spark.sql.execution.streaming.TextSocketSource@14c45193: 1000}
> Current State: ACTIVE
> Thread State: RUNNABLE
> Logical Plan:
> org.apache.spark.sql.execution.streaming.TextSocketSource@14c45193
> at 
> org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runBatches(StreamExecution.scala:284)
> at 
> org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:177)
> Caused by: java.lang.RuntimeException: Offsets committed out of order: 20019 
> followed by 1000
> at scala.sys.package$.error(package.scala:27)
> at 
> org.apache.spark.sql.execution.streaming.TextSocketSource.commit(socket.scala:151)
> at 
> org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$org$apache$spark$sql$execution$streaming$StreamExecution$$constructNextBatch$2$$anonfun$apply$mcV$sp$4.apply(StreamExecution.scala:421)
> at 
> org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$org$apache$spark$sql$execution$streaming$StreamExecution$$constructNextBatch$2$$anonfun$apply$mcV$sp$4.apply(StreamExecution.scala:420)
> at 

[jira] [Closed] (CARBONDATA-1711) Carbon1.3.0-DataMap - Show datamap on table does not work

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S closed CARBONDATA-1711.
-
Resolution: Fixed

The issue is closed as its not reproducible in the latest build

> Carbon1.3.0-DataMap - Show datamap  on table  does not work
> --
>
> Key: CARBONDATA-1711
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1711
> Project: CarbonData
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.3.0
> Environment: Test
>Reporter: Ramakrishna S
>Assignee: anubhav tarar
>Priority: Minor
>  Labels: Functional
> Fix For: 1.3.0
>
>
> 0: jdbc:hive2://10.18.98.34:23040> create datamap agr_lineitem ON TABLE 
> lineitem USING "org.apache.carbondata.datamap.AggregateDataMapHandler" as 
> select L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from 
> lineitem group by  L_RETURNFLAG, L_LINESTATUS;
> Error: java.lang.RuntimeException: Table [lineitem_agr_lineitem] already 
> exists under database [default] (state=,code=0)
> 0: jdbc:hive2://10.18.98.34:23040> show tables;
> +---+---+--+--+
> | database  | tableName | isTemporary  |
> +---+---+--+--+
> | default   | flow_carbon_test4 | false|
> | default   | jl_r3 | false|
> | default   | lineitem  | false|
> | default   | lineitem_agr_lineitem | false|
> | default   | sensor_reading_blockblank_false   | false|
> | default   | sensor_reading_blockblank_false1  | false|
> | default   | sensor_reading_blockblank_false2  | false|
> | default   | sensor_reading_false  | false|
> | default   | sensor_reading_true   | false|
> | default   | t1| false|
> | default   | t1_agg_t1 | false|
> | default   | tc4   | false|
> | default   | uniqdata  | false|
> +---+---+--+--+
> 13 rows selected (0.04 seconds)
> 0: jdbc:hive2://10.18.98.34:23040> show datamap on table lineitem;
> Error: java.lang.RuntimeException:
> BaseSqlParser
> missing 'FUNCTIONS' at 'on'(line 1, pos 13)
> == SQL ==
> show datamap on table lineitem
> -^^^
> CarbonSqlParser [1.6] failure: identifier matching regex (?i)SEGMENTS 
> expected
> show datamap on table lineitem



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (CARBONDATA-1743) Carbon1.3.0-Pre-AggregateTable - Query returns no value if run at the time of pre-aggregate table creation

[Ramakrishna S (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-1743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16285692#comment-16285692
 ] 

Ramakrishna S edited comment on CARBONDATA-1743 at 12/11/17 9:54 AM:
-

Same behaviour found if pre-agg table is created along with the parent table 
load, NULL values will be inserted to pre-agg table.

0: jdbc:hive2://10.18.98.34:23040> select * from lineitem1_agr_line limit 2;
+---+-++--+
| lineitem1_l_shipdate  | lineitem1_l_returnflag  | lineitem1_l_partkey_count  |
+---+-++--+
| NULL  | NULL| NULL   |
| NULL  | NULL| NULL   |
+---+-+


was (Author: ram@huawei):
Same behaviour found if pre-agg table is created along with the parent table 
load, NULL values will be inserted to pre-agg table.

> Carbon1.3.0-Pre-AggregateTable - Query returns no value if run at the time of 
> pre-aggregate table creation
> --
>
> Key: CARBONDATA-1743
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1743
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-load
>Affects Versions: 1.3.0
> Environment: Test - 3 node ant cluster
>Reporter: Ramakrishna S
>Assignee: Kunal Kapoor
>  Labels: DFX
> Fix For: 1.3.0
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Steps:
> 1. Create table and load with large data
> create table if not exists lineitem4(L_SHIPDATE string,L_SHIPMODE 
> string,L_SHIPINSTRUCT string,L_RETURNFLAG string,L_RECEIPTDATE 
> string,L_ORDERKEY string,L_PARTKEY string,L_SUPPKEY   string,L_LINENUMBER 
> int,L_QUANTITY double,L_EXTENDEDPRICE double,L_DISCOUNT double,L_TAX 
> double,L_LINESTATUS string,L_COMMITDATE string,L_COMMENT  string) STORED BY 
> 'org.apache.carbondata.format' TBLPROPERTIES 
> ('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
> load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table 
> lineitem4 
> options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');
> 2. Create a pre-aggregate table 
> create datamap agr_lineitem4 ON TABLE lineitem4 USING 
> "org.apache.carbondata.datamap.AggregateDataMapHandler" as select 
> L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from lineitem4 
> group by  L_RETURNFLAG, L_LINESTATUS;
> 3. Run aggregate query at the same time
>  select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
> lineitem4 group by l_returnflag, l_linestatus;
> *+Expected:+*: aggregate query should fetch data either from main table or 
> pre-aggregate table.
> *+Actual:+* aggregate query does not return data until the pre-aggregate 
> table is created
> 0: jdbc:hive2://10.18.98.48:23040> select 
> l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from lineitem4 
> group by l_returnflag, l_linestatus;
> +---+---+--+---+--+
> | l_returnflag  | l_linestatus  | sum(l_quantity)  | sum(l_extendedprice)  |
> +---+---+--+---+--+
> +---+---+--+---+--+
> No rows selected (1.74 seconds)
> 0: jdbc:hive2://10.18.98.48:23040> select 
> l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from lineitem4 
> group by l_returnflag, l_linestatus;
> +---+---+--+---+--+
> | l_returnflag  | l_linestatus  | sum(l_quantity)  | sum(l_extendedprice)  |
> +---+---+--+---+--+
> +---+---+--+---+--+
> No rows selected (0.746 seconds)
> 0: jdbc:hive2://10.18.98.48:23040> select 
> l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from lineitem4 
> group by l_returnflag, l_linestatus;
> +---+---+--++--+
> | l_returnflag  | l_linestatus  | sum(l_quantity)  |  sum(l_extendedprice)  |
> +---+---+--++--+
> | N | F | 2.9808092E7  | 4.471079473931997E10   |
> 

[jira] [Commented] (CARBONDATA-1743) Carbon1.3.0-Pre-AggregateTable - Query returns no value if run at the time of pre-aggregate table creation

[Ramakrishna S (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-1743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16285692#comment-16285692
 ] 

Ramakrishna S commented on CARBONDATA-1743:
---

Same behaviour found if pre-agg table is created along with the parent table 
load, NULL values will be inserted to pre-agg table.

> Carbon1.3.0-Pre-AggregateTable - Query returns no value if run at the time of 
> pre-aggregate table creation
> --
>
> Key: CARBONDATA-1743
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1743
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-load
>Affects Versions: 1.3.0
> Environment: Test - 3 node ant cluster
>Reporter: Ramakrishna S
>Assignee: Kunal Kapoor
>  Labels: DFX
> Fix For: 1.3.0
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Steps:
> 1. Create table and load with large data
> create table if not exists lineitem4(L_SHIPDATE string,L_SHIPMODE 
> string,L_SHIPINSTRUCT string,L_RETURNFLAG string,L_RECEIPTDATE 
> string,L_ORDERKEY string,L_PARTKEY string,L_SUPPKEY   string,L_LINENUMBER 
> int,L_QUANTITY double,L_EXTENDEDPRICE double,L_DISCOUNT double,L_TAX 
> double,L_LINESTATUS string,L_COMMITDATE string,L_COMMENT  string) STORED BY 
> 'org.apache.carbondata.format' TBLPROPERTIES 
> ('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
> load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table 
> lineitem4 
> options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');
> 2. Create a pre-aggregate table 
> create datamap agr_lineitem4 ON TABLE lineitem4 USING 
> "org.apache.carbondata.datamap.AggregateDataMapHandler" as select 
> L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from lineitem4 
> group by  L_RETURNFLAG, L_LINESTATUS;
> 3. Run aggregate query at the same time
>  select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
> lineitem4 group by l_returnflag, l_linestatus;
> *+Expected:+*: aggregate query should fetch data either from main table or 
> pre-aggregate table.
> *+Actual:+* aggregate query does not return data until the pre-aggregate 
> table is created
> 0: jdbc:hive2://10.18.98.48:23040> select 
> l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from lineitem4 
> group by l_returnflag, l_linestatus;
> +---+---+--+---+--+
> | l_returnflag  | l_linestatus  | sum(l_quantity)  | sum(l_extendedprice)  |
> +---+---+--+---+--+
> +---+---+--+---+--+
> No rows selected (1.74 seconds)
> 0: jdbc:hive2://10.18.98.48:23040> select 
> l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from lineitem4 
> group by l_returnflag, l_linestatus;
> +---+---+--+---+--+
> | l_returnflag  | l_linestatus  | sum(l_quantity)  | sum(l_extendedprice)  |
> +---+---+--+---+--+
> +---+---+--+---+--+
> No rows selected (0.746 seconds)
> 0: jdbc:hive2://10.18.98.48:23040> select 
> l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from lineitem4 
> group by l_returnflag, l_linestatus;
> +---+---+--++--+
> | l_returnflag  | l_linestatus  | sum(l_quantity)  |  sum(l_extendedprice)  |
> +---+---+--++--+
> | N | F | 2.9808092E7  | 4.471079473931997E10   |
> | A | F | 1.145546488E9| 1.717580824169429E12   |
> | N | O | 2.31980219E9 | 3.4789002701143467E12  |
> | R | F | 1.146403932E9| 1.7190627928317903E12  |
> +---+---+--++--+
> 4 rows selected (0.8 seconds)
> 0: jdbc:hive2://10.18.98.48:23040> select 
> l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from lineitem4 
> group by l_returnflag, l_linestatus;
> +---+---+--++--+
> | l_returnflag  | l_linestatus  | sum(l_quantity)  |  sum(l_extendedprice)  |
> +---+---+--++--+
> | N  

[jira] [Updated] (CARBONDATA-1825) Carbon 1.3.0 - Spark 2.2- Data load fails on 20k columns carbon table with CarbonDataWriterException

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1825:
--
Summary: Carbon 1.3.0 - Spark 2.2- Data load fails on 20k columns carbon 
table with CarbonDataWriterException  (was: Carbon 1.3.0 - Spark 2.2- Data load 
fails on carbon table with 20k columns with CarbonDataWriterException)

> Carbon 1.3.0 - Spark 2.2- Data load fails on 20k columns carbon table with 
> CarbonDataWriterException
> 
>
> Key: CARBONDATA-1825
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1825
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-load
>Affects Versions: 1.3.0
> Environment: Test - 3 node ant cluster
>Reporter: Ramakrishna S
>Assignee: kumar vishal
>Priority: Minor
>  Labels: DFX
> Fix For: 1.3.0
>
>
> Steps:
> Beeline:
> 1. Create carbon table with 20k columns
> 2. Run table load
> *+Expected:+* Table load should be success
> *+Actual:+*  table load fails



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CARBONDATA-1825) Carbon 1.3.0 - Spark 2.2- Data load fails on carbon table with 20k columns with CarbonDataWriterException

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1825:
--
Description: 
Steps:
Beeline:
1. Create carbon table with 20k columns
2. Run table load

*+Expected:+* Table load should be success
*+Actual:+*  table load fails





  was:
Steps:
Beeline:
1. Create a table with batch sort as sort type, keep block size small
2. Run Load/Insert/Compaction the table
3. Bring down thrift server when carbon data is being written to the segment
4. Do show segments on the table

*+Expected:+* It should not show the residual segments  
*+Actual:+* The segment intended for load is shown as marked for delete and it 
does not get deleted with clean file. No impact on the table as such.


*+Query:+*
create table if not exists lineitem1(L_SHIPDATE string,L_SHIPMODE 
string,L_SHIPINSTRUCT string,L_RETURNFLAG string,L_RECEIPTDATE 
string,L_ORDERKEY string,L_PARTKEY string,L_SUPPKEY   string,L_LINENUMBER 
int,L_QUANTITY double,L_EXTENDEDPRICE double,L_DISCOUNT double,L_TAX 
double,L_LINESTATUS string,L_COMMITDATE string,L_COMMENT  string) STORED BY 
'org.apache.carbondata.format' TBLPROPERTIES 
('table_blocksize'='1','sort_scope'='BATCH_SORT','batch_sort_size_inmb'='5000');

load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table 
lineitem 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');

0: jdbc:hive2://10.18.98.34:23040> select count(*) from t_carbn0161;
+---+--+
| count(1)  |
+---+--+
| 0 |
+---+--+
1 row selected (13.011 seconds)
0: jdbc:hive2://10.18.98.34:23040> show segments for table lineitem1;
+++--+--++--+--+
| SegmentSequenceId  |   Status   | Load Start Time  |  
Load End Time   | Merged To  | File Format  |
+++--+--++--+--+
| 1  | Marked for Delete  | 2017-11-28 19:14:46.265  | 
2017-11-28 19:15:28.396  | NA | COLUMNAR_V3  |
| 0  | Marked for Delete  | 2017-11-28 19:12:58.269  | 
2017-11-28 19:13:37.26   | NA | COLUMNAR_V3  |
+++--+--++--+--+
0: jdbc:hive2://10.18.98.34:23040> clean files for table t_carbn0161;
+-+--+
| Result  |
+-+--+
+-+--+
No rows selected (7.473 seconds)
0: jdbc:hive2://10.18.98.34:23040> show segments for table lineitem1;
+++--+--++--+--+
| SegmentSequenceId  |   Status   | Load Start Time  |  
Load End Time   | Merged To  | File Format  |
+++--+--++--+--+
| 1  | Marked for Delete  | 2017-11-28 19:14:46.265  | 
2017-11-28 19:15:28.396  | NA | COLUMNAR_V3  |
| 0  | Marked for Delete  | 2017-11-28 19:12:58.269  | 
2017-11-28 19:13:37.26   | NA | COLUMNAR_V3  |
+++--+--++--+--+



> Carbon 1.3.0 - Spark 2.2- Data load fails on carbon table with 20k columns 
> with CarbonDataWriterException
> -
>
> Key: CARBONDATA-1825
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1825
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-load
>Affects Versions: 1.3.0
> Environment: Test - 3 node ant cluster
>Reporter: Ramakrishna S
>Assignee: kumar vishal
>Priority: Minor
>  Labels: DFX
> Fix For: 1.3.0
>
>
> Steps:
> Beeline:
> 1. Create carbon table with 20k columns
> 2. Run table load
> *+Expected:+* Table load should be success
> *+Actual:+*  table load fails



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CARBONDATA-1825) Carbon 1.3.0 - Spark 2.2- Data load fails on carbon table with 20k columns with CarbonDataWriterException

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1825:
--
Summary: Carbon 1.3.0 - Spark 2.2- Data load fails on carbon table with 20k 
columns with CarbonDataWriterException  (was: Carbon 1.3.0 - Spark 2.2-Residual 
segment files left over when load failure happens)

> Carbon 1.3.0 - Spark 2.2- Data load fails on carbon table with 20k columns 
> with CarbonDataWriterException
> -
>
> Key: CARBONDATA-1825
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1825
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-load
>Affects Versions: 1.3.0
> Environment: Test - 3 node ant cluster
>Reporter: Ramakrishna S
>Assignee: kumar vishal
>Priority: Minor
>  Labels: DFX
> Fix For: 1.3.0
>
>
> Steps:
> Beeline:
> 1. Create a table with batch sort as sort type, keep block size small
> 2. Run Load/Insert/Compaction the table
> 3. Bring down thrift server when carbon data is being written to the segment
> 4. Do show segments on the table
> *+Expected:+* It should not show the residual segments  
> *+Actual:+* The segment intended for load is shown as marked for delete and 
> it does not get deleted with clean file. No impact on the table as such.
> *+Query:+*
> create table if not exists lineitem1(L_SHIPDATE string,L_SHIPMODE 
> string,L_SHIPINSTRUCT string,L_RETURNFLAG string,L_RECEIPTDATE 
> string,L_ORDERKEY string,L_PARTKEY string,L_SUPPKEY   string,L_LINENUMBER 
> int,L_QUANTITY double,L_EXTENDEDPRICE double,L_DISCOUNT double,L_TAX 
> double,L_LINESTATUS string,L_COMMITDATE string,L_COMMENT  string) STORED BY 
> 'org.apache.carbondata.format' TBLPROPERTIES 
> ('table_blocksize'='1','sort_scope'='BATCH_SORT','batch_sort_size_inmb'='5000');
> load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table 
> lineitem 
> options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');
> 0: jdbc:hive2://10.18.98.34:23040> select count(*) from t_carbn0161;
> +---+--+
> | count(1)  |
> +---+--+
> | 0 |
> +---+--+
> 1 row selected (13.011 seconds)
> 0: jdbc:hive2://10.18.98.34:23040> show segments for table lineitem1;
> +++--+--++--+--+
> | SegmentSequenceId  |   Status   | Load Start Time  |  
> Load End Time   | Merged To  | File Format  |
> +++--+--++--+--+
> | 1  | Marked for Delete  | 2017-11-28 19:14:46.265  | 
> 2017-11-28 19:15:28.396  | NA | COLUMNAR_V3  |
> | 0  | Marked for Delete  | 2017-11-28 19:12:58.269  | 
> 2017-11-28 19:13:37.26   | NA | COLUMNAR_V3  |
> +++--+--++--+--+
> 0: jdbc:hive2://10.18.98.34:23040> clean files for table t_carbn0161;
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (7.473 seconds)
> 0: jdbc:hive2://10.18.98.34:23040> show segments for table lineitem1;
> +++--+--++--+--+
> | SegmentSequenceId  |   Status   | Load Start Time  |  
> Load End Time   | Merged To  | File Format  |
> +++--+--++--+--+
> | 1  | Marked for Delete  | 2017-11-28 19:14:46.265  | 
> 2017-11-28 19:15:28.396  | NA | COLUMNAR_V3  |
> | 0  | Marked for Delete  | 2017-11-28 19:12:58.269  | 
> 2017-11-28 19:13:37.26   | NA | COLUMNAR_V3  |
> +++--+--++--+--+



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (CARBONDATA-1825) Carbon 1.3.0 - Spark 2.2-Residual segment files left over when load failure happens

[Ramakrishna S (JIRA)]

Ramakrishna S created CARBONDATA-1825:
-

 Summary: Carbon 1.3.0 - Spark 2.2-Residual segment files left over 
when load failure happens
 Key: CARBONDATA-1825
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1825
 Project: CarbonData
  Issue Type: Bug
  Components: data-load
Affects Versions: 1.3.0
 Environment: Test - 3 node ant cluster
Reporter: Ramakrishna S
Assignee: kumar vishal
Priority: Minor
 Fix For: 1.3.0


Steps:
Beeline:
1. Create a table with batch sort as sort type, keep block size small
2. Run Load/Insert/Compaction the table
3. Bring down thrift server when carbon data is being written to the segment
4. Do show segments on the table

*+Expected:+* It should not show the residual segments  
*+Actual:+* The segment intended for load is shown as marked for delete and it 
does not get deleted with clean file. No impact on the table as such.


*+Query:+*
create table if not exists lineitem1(L_SHIPDATE string,L_SHIPMODE 
string,L_SHIPINSTRUCT string,L_RETURNFLAG string,L_RECEIPTDATE 
string,L_ORDERKEY string,L_PARTKEY string,L_SUPPKEY   string,L_LINENUMBER 
int,L_QUANTITY double,L_EXTENDEDPRICE double,L_DISCOUNT double,L_TAX 
double,L_LINESTATUS string,L_COMMITDATE string,L_COMMENT  string) STORED BY 
'org.apache.carbondata.format' TBLPROPERTIES 
('table_blocksize'='1','sort_scope'='BATCH_SORT','batch_sort_size_inmb'='5000');

load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table 
lineitem 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');

0: jdbc:hive2://10.18.98.34:23040> select count(*) from t_carbn0161;
+---+--+
| count(1)  |
+---+--+
| 0 |
+---+--+
1 row selected (13.011 seconds)
0: jdbc:hive2://10.18.98.34:23040> show segments for table lineitem1;
+++--+--++--+--+
| SegmentSequenceId  |   Status   | Load Start Time  |  
Load End Time   | Merged To  | File Format  |
+++--+--++--+--+
| 1  | Marked for Delete  | 2017-11-28 19:14:46.265  | 
2017-11-28 19:15:28.396  | NA | COLUMNAR_V3  |
| 0  | Marked for Delete  | 2017-11-28 19:12:58.269  | 
2017-11-28 19:13:37.26   | NA | COLUMNAR_V3  |
+++--+--++--+--+
0: jdbc:hive2://10.18.98.34:23040> clean files for table t_carbn0161;
+-+--+
| Result  |
+-+--+
+-+--+
No rows selected (7.473 seconds)
0: jdbc:hive2://10.18.98.34:23040> show segments for table lineitem1;
+++--+--++--+--+
| SegmentSequenceId  |   Status   | Load Start Time  |  
Load End Time   | Merged To  | File Format  |
+++--+--++--+--+
| 1  | Marked for Delete  | 2017-11-28 19:14:46.265  | 
2017-11-28 19:15:28.396  | NA | COLUMNAR_V3  |
| 0  | Marked for Delete  | 2017-11-28 19:12:58.269  | 
2017-11-28 19:13:37.26   | NA | COLUMNAR_V3  |
+++--+--++--+--+




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CARBONDATA-1824) Spark 2.2-Residual segment files left over when load failure happens

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1824:
--
Description: 
Steps:
Beeline:
1. Create a table with batch sort as sort type, keep block size small
2. Run Load/Insert/Compaction the table
3. Bring down thrift server when carbon data is being written to the segment
4. Do show segments on the table

*+Expected:+* It should not show the residual segments  
*+Actual:+* The segment intended for load is shown as marked for delete and it 
does not get deleted with clean file. No impact on the table as such.


*+Query:+*
create table if not exists lineitem1(L_SHIPDATE string,L_SHIPMODE 
string,L_SHIPINSTRUCT string,L_RETURNFLAG string,L_RECEIPTDATE 
string,L_ORDERKEY string,L_PARTKEY string,L_SUPPKEY   string,L_LINENUMBER 
int,L_QUANTITY double,L_EXTENDEDPRICE double,L_DISCOUNT double,L_TAX 
double,L_LINESTATUS string,L_COMMITDATE string,L_COMMENT  string) STORED BY 
'org.apache.carbondata.format' TBLPROPERTIES 
('table_blocksize'='1','sort_scope'='BATCH_SORT','batch_sort_size_inmb'='5000');

load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table 
lineitem 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');

0: jdbc:hive2://10.18.98.34:23040> select count(*) from t_carbn0161;
+---+--+
| count(1)  |
+---+--+
| 0 |
+---+--+
1 row selected (13.011 seconds)
0: jdbc:hive2://10.18.98.34:23040> show segments for table lineitem1;
+++--+--++--+--+
| SegmentSequenceId  |   Status   | Load Start Time  |  
Load End Time   | Merged To  | File Format  |
+++--+--++--+--+
| 1  | Marked for Delete  | 2017-11-28 19:14:46.265  | 
2017-11-28 19:15:28.396  | NA | COLUMNAR_V3  |
| 0  | Marked for Delete  | 2017-11-28 19:12:58.269  | 
2017-11-28 19:13:37.26   | NA | COLUMNAR_V3  |
+++--+--++--+--+
0: jdbc:hive2://10.18.98.34:23040> clean files for table t_carbn0161;
+-+--+
| Result  |
+-+--+
+-+--+
No rows selected (7.473 seconds)
0: jdbc:hive2://10.18.98.34:23040> show segments for table lineitem1;
+++--+--++--+--+
| SegmentSequenceId  |   Status   | Load Start Time  |  
Load End Time   | Merged To  | File Format  |
+++--+--++--+--+
| 1  | Marked for Delete  | 2017-11-28 19:14:46.265  | 
2017-11-28 19:15:28.396  | NA | COLUMNAR_V3  |
| 0  | Marked for Delete  | 2017-11-28 19:12:58.269  | 
2017-11-28 19:13:37.26   | NA | COLUMNAR_V3  |
+++--+--++--+--+


  was:
Steps:
Beeline:
1. Create table and load with  data
2. create a pre-aggregate table with incorrect syntax.
3. Run select count(*) on aggregate table
4. Run aggregate select query on main table

*+Expected:+* Pre-aggregate table creation should have trhown syntax error 
*+Actual:+* Pre-aggregate table is shown successful, but aggregate query fails.


*+Query:+*
create table if not exists JL_r31
(
p_cap_time String,
city String,
product_code String,
user_base_station String,
user_belong_area_code String,
user_num String,
user_imsi String,
user_id String,
user_msisdn String,
dim1 String,
dim2 String,
dim3 String,
dim4 String,
dim5 String,
dim6 String,
dim7 String,
dim8 String,
dim9 String,
dim10 String,
dim11 String,
dim12 String,
dim13 String,
dim14 String,
dim15 String,
dim16 String,
dim17 String,
dim18 String,
dim19 String,
dim20 String,
dim21 String,
dim22 String,
dim23 String,
dim24 String,
dim25 String,
dim26 String,
dim27 String,
dim28 String,
dim29 String,
dim30 String,
dim31 String,
dim32 String,
dim33 String,
dim34 String,
dim35 String,
dim36 String,
dim37 String,
dim38 String,
dim39 String,
dim40 String,
dim41 String,
dim42 String,
dim43 String,
dim44 String,
dim45 String,
dim46 String,
dim47 String,
dim48 String,
dim49 String,
dim50 String,
dim51 String,
dim52 String,
dim53 String,
dim54 String,
dim55 String,
dim56 String,
dim57 String,
dim58 String,
dim59 String,
dim60 String,
dim61 String,
dim62 String,
dim63 String,
dim64 String,
dim65 String,
dim66 String,

[jira] [Created] (CARBONDATA-1824) Spark 2.2-Residual segment files left over when load failure happens

[Ramakrishna S (JIRA)]

Ramakrishna S created CARBONDATA-1824:
-

 Summary: Spark 2.2-Residual segment files left over when load 
failure happens
 Key: CARBONDATA-1824
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1824
 Project: CarbonData
  Issue Type: Bug
  Components: data-load
Affects Versions: 1.3.0
 Environment: Test - 3 node ant cluster
Reporter: Ramakrishna S
Assignee: kumar vishal
Priority: Minor
 Fix For: 1.3.0


Steps:
Beeline:
1. Create table and load with  data
2. create a pre-aggregate table with incorrect syntax.
3. Run select count(*) on aggregate table
4. Run aggregate select query on main table

*+Expected:+* Pre-aggregate table creation should have trhown syntax error 
*+Actual:+* Pre-aggregate table is shown successful, but aggregate query fails.


*+Query:+*
create table if not exists JL_r31
(
p_cap_time String,
city String,
product_code String,
user_base_station String,
user_belong_area_code String,
user_num String,
user_imsi String,
user_id String,
user_msisdn String,
dim1 String,
dim2 String,
dim3 String,
dim4 String,
dim5 String,
dim6 String,
dim7 String,
dim8 String,
dim9 String,
dim10 String,
dim11 String,
dim12 String,
dim13 String,
dim14 String,
dim15 String,
dim16 String,
dim17 String,
dim18 String,
dim19 String,
dim20 String,
dim21 String,
dim22 String,
dim23 String,
dim24 String,
dim25 String,
dim26 String,
dim27 String,
dim28 String,
dim29 String,
dim30 String,
dim31 String,
dim32 String,
dim33 String,
dim34 String,
dim35 String,
dim36 String,
dim37 String,
dim38 String,
dim39 String,
dim40 String,
dim41 String,
dim42 String,
dim43 String,
dim44 String,
dim45 String,
dim46 String,
dim47 String,
dim48 String,
dim49 String,
dim50 String,
dim51 String,
dim52 String,
dim53 String,
dim54 String,
dim55 String,
dim56 String,
dim57 String,
dim58 String,
dim59 String,
dim60 String,
dim61 String,
dim62 String,
dim63 String,
dim64 String,
dim65 String,
dim66 String,
dim67 String,
dim68 String,
dim69 String,
dim70 String,
dim71 String,
dim72 String,
dim73 String,
dim74 String,
dim75 String,
dim76 String,
dim77 String,
dim78 String,
dim79 String,
dim80 String,
dim81 String,
M1 double,
M2 double,
M3 double,
M4 double,
M5 double,
M6 double,
M7 double,
M8 double,
M9 double,
M10 double )
stored by 'org.apache.carbondata.format' 
TBLPROPERTIES('DICTIONARY_EXCLUDE'='user_num,user_imsi,user_ID,user_msisdn,user_base_station,user_belong_area_code','table_blocksize'='512');
+-+--+
| Result  |
+-+--+
+-+--+
No rows selected (0.55 seconds)

LOAD DATA inpath 'hdfs://hacluster/user/test/jin_test2.csv' into table JL_r31 
options('DELIMITER'=',', 
'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','IS_EMPTY_DATA_BAD_RECORD'='TRUE','FILEHEADER'='p_cap_time,city,product_code,user_base_station,user_belong_area_code,user_num,user_imsi,user_id,user_msisdn,dim1,dim2,dim3,dim4,dim5,dim6,dim7,dim8,dim9,dim10,dim11,dim12,dim13,dim14,dim15,dim16,dim17,dim18,dim19,dim20,dim21,dim22,dim23,dim24,dim25,dim26,dim27,dim28,dim29,dim30,dim31,dim32,dim33,dim34,dim35,dim36,dim37,dim38,dim39,dim40,dim41,dim42,dim43,dim44,dim45,dim46,dim47,dim48,dim49,dim50,dim51,dim52,dim53,dim54,dim55,dim56,dim57,dim58,dim59,dim60,dim61,dim62,dim63,dim64,dim65,dim66,dim67,dim68,dim69,dim70,dim71,dim72,dim73,dim74,dim75,dim76,dim77,dim78,dim79,dim80,dim81,M1,M2,M3,M4,M5,M6,M7,M8,M9,M10');
+-+--+
| Result  |
+-+--+
+-+--+
No rows selected (14.049 seconds)
0: jdbc:hive2://10.18.98.136:23040> create datamap agr_JL_r31 ON TABLE JL_r31 
USING 'org.apache.carbondta.datamap.AggregateDataMapHandler' as select 
user_num,user_imsi,sum(user_id),count(user_id) from JL_r31 group by  user_num, 
user_imsi;
+-+--+
| Result  |
+-+--+
+-+--+
No rows selected (0.397 seconds)
0: jdbc:hive2://10.18.98.136:23040> select count(*) from JL_r31_agr_JL_r31;
Error: org.apache.spark.sql.AnalysisException: Table or view not found: 
JL_r31_agr_JL_r31; line 1 pos 21 (state=,code=0)
0: jdbc:hive2://10.18.98.136:23040> select 
user_num,user_imsi,sum(user_id),count(user_id) from JL_r31 group by  user_num, 
user_imsi;
Error: java.lang.ClassCastException: 
org.apache.carbondata.core.metadata.schema.table.DataMapSchema cannot be cast 
to org.apache.carbondata.core.metadata.schema.table.AggregationDataMapSchema 
(state=,code=0)


Driver Logs:
2017-11-24 21:45:10,997 | INFO  | [pool-23-thread-4] | Parsing command: create 
datamap agr_JL_r31 ON TABLE JL_r31 USING 
'org.apache.carbondta.datamap.AggregateDataMapHandler' as select 
user_num,user_imsi,sum(user_id),count(user_id) from JL_r31 group by  user_num, 
user_imsi | org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)
2017-11-24 21:45:11,046 | INFO  | [pool-23-thread-4] | pool-23-thread-4 Skip 
CarbonOptimizer | 

[jira] [Commented] (CARBONDATA-1807) Carbon1.3.0-Pre-AggregateTable - Pre-aggregate creation not throwing error for wrong syntax and results in further query failures

[Ramakrishna S (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-1807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266386#comment-16266386
 ] 

Ramakrishna S commented on CARBONDATA-1807:
---

issue was with the create command - 
'org.apache.carbondta.datamap.AggregateDataMapHandler' , should have thrown the 
error for wrong syntax.
i have changed the description accordingly.

> Carbon1.3.0-Pre-AggregateTable - Pre-aggregate creation not throwing error 
> for wrong syntax and results in further query failures
> -
>
> Key: CARBONDATA-1807
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1807
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-load
>Affects Versions: 1.3.0
> Environment: Test - 3 node ant cluster
>Reporter: Ramakrishna S
>Assignee: kumar vishal
>Priority: Minor
>  Labels: DFX
> Fix For: 1.3.0
>
>
> Steps:
> Beeline:
> 1. Create table and load with  data
> 2. create a pre-aggregate table with incorrect syntax.
> 3. Run select count(*) on aggregate table
> 4. Run aggregate select query on main table
> *+Expected:+* Pre-aggregate table creation should have trhown syntax error 
> *+Actual:+* Pre-aggregate table is shown successful, but aggregate query 
> fails.
> *+Query:+*
> create table if not exists JL_r31
> (
> p_cap_time String,
> city String,
> product_code String,
> user_base_station String,
> user_belong_area_code String,
> user_num String,
> user_imsi String,
> user_id String,
> user_msisdn String,
> dim1 String,
> dim2 String,
> dim3 String,
> dim4 String,
> dim5 String,
> dim6 String,
> dim7 String,
> dim8 String,
> dim9 String,
> dim10 String,
> dim11 String,
> dim12 String,
> dim13 String,
> dim14 String,
> dim15 String,
> dim16 String,
> dim17 String,
> dim18 String,
> dim19 String,
> dim20 String,
> dim21 String,
> dim22 String,
> dim23 String,
> dim24 String,
> dim25 String,
> dim26 String,
> dim27 String,
> dim28 String,
> dim29 String,
> dim30 String,
> dim31 String,
> dim32 String,
> dim33 String,
> dim34 String,
> dim35 String,
> dim36 String,
> dim37 String,
> dim38 String,
> dim39 String,
> dim40 String,
> dim41 String,
> dim42 String,
> dim43 String,
> dim44 String,
> dim45 String,
> dim46 String,
> dim47 String,
> dim48 String,
> dim49 String,
> dim50 String,
> dim51 String,
> dim52 String,
> dim53 String,
> dim54 String,
> dim55 String,
> dim56 String,
> dim57 String,
> dim58 String,
> dim59 String,
> dim60 String,
> dim61 String,
> dim62 String,
> dim63 String,
> dim64 String,
> dim65 String,
> dim66 String,
> dim67 String,
> dim68 String,
> dim69 String,
> dim70 String,
> dim71 String,
> dim72 String,
> dim73 String,
> dim74 String,
> dim75 String,
> dim76 String,
> dim77 String,
> dim78 String,
> dim79 String,
> dim80 String,
> dim81 String,
> M1 double,
> M2 double,
> M3 double,
> M4 double,
> M5 double,
> M6 double,
> M7 double,
> M8 double,
> M9 double,
> M10 double )
> stored by 'org.apache.carbondata.format' 
> TBLPROPERTIES('DICTIONARY_EXCLUDE'='user_num,user_imsi,user_ID,user_msisdn,user_base_station,user_belong_area_code','table_blocksize'='512');
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (0.55 seconds)
> LOAD DATA inpath 'hdfs://hacluster/user/test/jin_test2.csv' into table JL_r31 
> options('DELIMITER'=',', 
> 'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','IS_EMPTY_DATA_BAD_RECORD'='TRUE','FILEHEADER'='p_cap_time,city,product_code,user_base_station,user_belong_area_code,user_num,user_imsi,user_id,user_msisdn,dim1,dim2,dim3,dim4,dim5,dim6,dim7,dim8,dim9,dim10,dim11,dim12,dim13,dim14,dim15,dim16,dim17,dim18,dim19,dim20,dim21,dim22,dim23,dim24,dim25,dim26,dim27,dim28,dim29,dim30,dim31,dim32,dim33,dim34,dim35,dim36,dim37,dim38,dim39,dim40,dim41,dim42,dim43,dim44,dim45,dim46,dim47,dim48,dim49,dim50,dim51,dim52,dim53,dim54,dim55,dim56,dim57,dim58,dim59,dim60,dim61,dim62,dim63,dim64,dim65,dim66,dim67,dim68,dim69,dim70,dim71,dim72,dim73,dim74,dim75,dim76,dim77,dim78,dim79,dim80,dim81,M1,M2,M3,M4,M5,M6,M7,M8,M9,M10');
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (14.049 seconds)
> 0: jdbc:hive2://10.18.98.136:23040> create datamap agr_JL_r31 ON TABLE JL_r31 
> USING 'org.apache.carbondta.datamap.AggregateDataMapHandler' as select 
> user_num,user_imsi,sum(user_id),count(user_id) from JL_r31 group by  
> user_num, user_imsi;
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (0.397 seconds)
> 0: jdbc:hive2://10.18.98.136:23040> select count(*) from JL_r31_agr_JL_r31;
> Error: org.apache.spark.sql.AnalysisException: Table or view not found: 
> JL_r31_agr_JL_r31; line 1 pos 21 (state=,code=0)
> 0: jdbc:hive2://10.18.98.136:23040> 

[jira] [Updated] (CARBONDATA-1807) Carbon1.3.0-Pre-AggregateTable - Pre-aggregate creation not throwing error for wrong syntax and results in further query failures

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1807:
--
Description: 
Steps:
Beeline:
1. Create table and load with  data
2. create a pre-aggregate table with incorrect syntax.
3. Run select count(*) on aggregate table
4. Run aggregate select query on main table

*+Expected:+* Pre-aggregate table creation should have trhown syntax error 
*+Actual:+* Pre-aggregate table is shown successful, but aggregate query fails.


*+Query:+*
create table if not exists JL_r31
(
p_cap_time String,
city String,
product_code String,
user_base_station String,
user_belong_area_code String,
user_num String,
user_imsi String,
user_id String,
user_msisdn String,
dim1 String,
dim2 String,
dim3 String,
dim4 String,
dim5 String,
dim6 String,
dim7 String,
dim8 String,
dim9 String,
dim10 String,
dim11 String,
dim12 String,
dim13 String,
dim14 String,
dim15 String,
dim16 String,
dim17 String,
dim18 String,
dim19 String,
dim20 String,
dim21 String,
dim22 String,
dim23 String,
dim24 String,
dim25 String,
dim26 String,
dim27 String,
dim28 String,
dim29 String,
dim30 String,
dim31 String,
dim32 String,
dim33 String,
dim34 String,
dim35 String,
dim36 String,
dim37 String,
dim38 String,
dim39 String,
dim40 String,
dim41 String,
dim42 String,
dim43 String,
dim44 String,
dim45 String,
dim46 String,
dim47 String,
dim48 String,
dim49 String,
dim50 String,
dim51 String,
dim52 String,
dim53 String,
dim54 String,
dim55 String,
dim56 String,
dim57 String,
dim58 String,
dim59 String,
dim60 String,
dim61 String,
dim62 String,
dim63 String,
dim64 String,
dim65 String,
dim66 String,
dim67 String,
dim68 String,
dim69 String,
dim70 String,
dim71 String,
dim72 String,
dim73 String,
dim74 String,
dim75 String,
dim76 String,
dim77 String,
dim78 String,
dim79 String,
dim80 String,
dim81 String,
M1 double,
M2 double,
M3 double,
M4 double,
M5 double,
M6 double,
M7 double,
M8 double,
M9 double,
M10 double )
stored by 'org.apache.carbondata.format' 
TBLPROPERTIES('DICTIONARY_EXCLUDE'='user_num,user_imsi,user_ID,user_msisdn,user_base_station,user_belong_area_code','table_blocksize'='512');
+-+--+
| Result  |
+-+--+
+-+--+
No rows selected (0.55 seconds)

LOAD DATA inpath 'hdfs://hacluster/user/test/jin_test2.csv' into table JL_r31 
options('DELIMITER'=',', 
'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','IS_EMPTY_DATA_BAD_RECORD'='TRUE','FILEHEADER'='p_cap_time,city,product_code,user_base_station,user_belong_area_code,user_num,user_imsi,user_id,user_msisdn,dim1,dim2,dim3,dim4,dim5,dim6,dim7,dim8,dim9,dim10,dim11,dim12,dim13,dim14,dim15,dim16,dim17,dim18,dim19,dim20,dim21,dim22,dim23,dim24,dim25,dim26,dim27,dim28,dim29,dim30,dim31,dim32,dim33,dim34,dim35,dim36,dim37,dim38,dim39,dim40,dim41,dim42,dim43,dim44,dim45,dim46,dim47,dim48,dim49,dim50,dim51,dim52,dim53,dim54,dim55,dim56,dim57,dim58,dim59,dim60,dim61,dim62,dim63,dim64,dim65,dim66,dim67,dim68,dim69,dim70,dim71,dim72,dim73,dim74,dim75,dim76,dim77,dim78,dim79,dim80,dim81,M1,M2,M3,M4,M5,M6,M7,M8,M9,M10');
+-+--+
| Result  |
+-+--+
+-+--+
No rows selected (14.049 seconds)
0: jdbc:hive2://10.18.98.136:23040> create datamap agr_JL_r31 ON TABLE JL_r31 
USING 'org.apache.carbondta.datamap.AggregateDataMapHandler' as select 
user_num,user_imsi,sum(user_id),count(user_id) from JL_r31 group by  user_num, 
user_imsi;
+-+--+
| Result  |
+-+--+
+-+--+
No rows selected (0.397 seconds)
0: jdbc:hive2://10.18.98.136:23040> select count(*) from JL_r31_agr_JL_r31;
Error: org.apache.spark.sql.AnalysisException: Table or view not found: 
JL_r31_agr_JL_r31; line 1 pos 21 (state=,code=0)
0: jdbc:hive2://10.18.98.136:23040> select 
user_num,user_imsi,sum(user_id),count(user_id) from JL_r31 group by  user_num, 
user_imsi;
Error: java.lang.ClassCastException: 
org.apache.carbondata.core.metadata.schema.table.DataMapSchema cannot be cast 
to org.apache.carbondata.core.metadata.schema.table.AggregationDataMapSchema 
(state=,code=0)


Driver Logs:
2017-11-24 21:45:10,997 | INFO  | [pool-23-thread-4] | Parsing command: create 
datamap agr_JL_r31 ON TABLE JL_r31 USING 
'org.apache.carbondta.datamap.AggregateDataMapHandler' as select 
user_num,user_imsi,sum(user_id),count(user_id) from JL_r31 group by  user_num, 
user_imsi | org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)
2017-11-24 21:45:11,046 | INFO  | [pool-23-thread-4] | pool-23-thread-4 Skip 
CarbonOptimizer | 
org.apache.carbondata.common.logging.impl.StandardLogService.logInfoMessage(StandardLogService.java:150)
2017-11-24 21:45:11,051 | INFO  | [pool-23-thread-4] | 5: get_table : 
db=default tbl=jl_r31 | 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.logInfo(HiveMetaStore.java:746)
2017-11-24 21:45:11,051 | INFO  | [pool-23-thread-4] | ugi=anonymous
ip=unknown-ip-addr  cmd=get_table : db=default tbl=jl_r31| 

[jira] [Updated] (CARBONDATA-1807) Carbon1.3.0-Pre-AggregateTable - Pre-aggregate creation not throwing error for wrong syntax and results in further query failures

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1807:
--
Summary: Carbon1.3.0-Pre-AggregateTable - Pre-aggregate creation not 
throwing error for wrong syntax and results in further query failures  (was: 
Carbon1.3.0-Pre-AggregateTable - Pre-aggregate not created when main table has 
more columns)

> Carbon1.3.0-Pre-AggregateTable - Pre-aggregate creation not throwing error 
> for wrong syntax and results in further query failures
> -
>
> Key: CARBONDATA-1807
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1807
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-load
>Affects Versions: 1.3.0
> Environment: Test - 3 node ant cluster
>Reporter: Ramakrishna S
>Assignee: kumar vishal
>Priority: Minor
>  Labels: DFX
> Fix For: 1.3.0
>
>
> Steps:
> Beeline:
> 1. Create table and load with  data
> 2. create a pre-aggregate table
> 3. Run select count(*) on aggregate table
> 4. Run select query on main table
> *+Expected:+* Pre-aggregate table should be used in the aggregate query 
> *+Actual:+* Pre-aggregate table is not used
> *+Query:+*
> create table if not exists JL_r31
> (
> p_cap_time String,
> city String,
> product_code String,
> user_base_station String,
> user_belong_area_code String,
> user_num String,
> user_imsi String,
> user_id String,
> user_msisdn String,
> dim1 String,
> dim2 String,
> dim3 String,
> dim4 String,
> dim5 String,
> dim6 String,
> dim7 String,
> dim8 String,
> dim9 String,
> dim10 String,
> dim11 String,
> dim12 String,
> dim13 String,
> dim14 String,
> dim15 String,
> dim16 String,
> dim17 String,
> dim18 String,
> dim19 String,
> dim20 String,
> dim21 String,
> dim22 String,
> dim23 String,
> dim24 String,
> dim25 String,
> dim26 String,
> dim27 String,
> dim28 String,
> dim29 String,
> dim30 String,
> dim31 String,
> dim32 String,
> dim33 String,
> dim34 String,
> dim35 String,
> dim36 String,
> dim37 String,
> dim38 String,
> dim39 String,
> dim40 String,
> dim41 String,
> dim42 String,
> dim43 String,
> dim44 String,
> dim45 String,
> dim46 String,
> dim47 String,
> dim48 String,
> dim49 String,
> dim50 String,
> dim51 String,
> dim52 String,
> dim53 String,
> dim54 String,
> dim55 String,
> dim56 String,
> dim57 String,
> dim58 String,
> dim59 String,
> dim60 String,
> dim61 String,
> dim62 String,
> dim63 String,
> dim64 String,
> dim65 String,
> dim66 String,
> dim67 String,
> dim68 String,
> dim69 String,
> dim70 String,
> dim71 String,
> dim72 String,
> dim73 String,
> dim74 String,
> dim75 String,
> dim76 String,
> dim77 String,
> dim78 String,
> dim79 String,
> dim80 String,
> dim81 String,
> M1 double,
> M2 double,
> M3 double,
> M4 double,
> M5 double,
> M6 double,
> M7 double,
> M8 double,
> M9 double,
> M10 double )
> stored by 'org.apache.carbondata.format' 
> TBLPROPERTIES('DICTIONARY_EXCLUDE'='user_num,user_imsi,user_ID,user_msisdn,user_base_station,user_belong_area_code','table_blocksize'='512');
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (0.55 seconds)
> LOAD DATA inpath 'hdfs://hacluster/user/test/jin_test2.csv' into table JL_r31 
> options('DELIMITER'=',', 
> 'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','IS_EMPTY_DATA_BAD_RECORD'='TRUE','FILEHEADER'='p_cap_time,city,product_code,user_base_station,user_belong_area_code,user_num,user_imsi,user_id,user_msisdn,dim1,dim2,dim3,dim4,dim5,dim6,dim7,dim8,dim9,dim10,dim11,dim12,dim13,dim14,dim15,dim16,dim17,dim18,dim19,dim20,dim21,dim22,dim23,dim24,dim25,dim26,dim27,dim28,dim29,dim30,dim31,dim32,dim33,dim34,dim35,dim36,dim37,dim38,dim39,dim40,dim41,dim42,dim43,dim44,dim45,dim46,dim47,dim48,dim49,dim50,dim51,dim52,dim53,dim54,dim55,dim56,dim57,dim58,dim59,dim60,dim61,dim62,dim63,dim64,dim65,dim66,dim67,dim68,dim69,dim70,dim71,dim72,dim73,dim74,dim75,dim76,dim77,dim78,dim79,dim80,dim81,M1,M2,M3,M4,M5,M6,M7,M8,M9,M10');
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (14.049 seconds)
> 0: jdbc:hive2://10.18.98.136:23040> create datamap agr_JL_r31 ON TABLE JL_r31 
> USING 'org.apache.carbondta.datamap.AggregateDataMapHandler' as select 
> user_num,user_imsi,sum(user_id),count(user_id) from JL_r31 group by  
> user_num, user_imsi;
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (0.397 seconds)
> 0: jdbc:hive2://10.18.98.136:23040> select count(*) from JL_r31_agr_JL_r31;
> Error: org.apache.spark.sql.AnalysisException: Table or view not found: 
> JL_r31_agr_JL_r31; line 1 pos 21 (state=,code=0)
> 0: jdbc:hive2://10.18.98.136:23040> select 
> 

[jira] [Updated] (CARBONDATA-1807) Carbon1.3.0-Pre-AggregateTable - Pre-aggregate not created when main table has more columns

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1807:
--
Description: 
Steps:
Beeline:
1. Create table and load with  data
2. create a pre-aggregate table
3. Run select count(*) on aggregate table
4. Run select query on main table

*+Expected:+* Pre-aggregate table should be used in the aggregate query 
*+Actual:+* Pre-aggregate table is not used


*+Query:+*
create table if not exists JL_r31
(
p_cap_time String,
city String,
product_code String,
user_base_station String,
user_belong_area_code String,
user_num String,
user_imsi String,
user_id String,
user_msisdn String,
dim1 String,
dim2 String,
dim3 String,
dim4 String,
dim5 String,
dim6 String,
dim7 String,
dim8 String,
dim9 String,
dim10 String,
dim11 String,
dim12 String,
dim13 String,
dim14 String,
dim15 String,
dim16 String,
dim17 String,
dim18 String,
dim19 String,
dim20 String,
dim21 String,
dim22 String,
dim23 String,
dim24 String,
dim25 String,
dim26 String,
dim27 String,
dim28 String,
dim29 String,
dim30 String,
dim31 String,
dim32 String,
dim33 String,
dim34 String,
dim35 String,
dim36 String,
dim37 String,
dim38 String,
dim39 String,
dim40 String,
dim41 String,
dim42 String,
dim43 String,
dim44 String,
dim45 String,
dim46 String,
dim47 String,
dim48 String,
dim49 String,
dim50 String,
dim51 String,
dim52 String,
dim53 String,
dim54 String,
dim55 String,
dim56 String,
dim57 String,
dim58 String,
dim59 String,
dim60 String,
dim61 String,
dim62 String,
dim63 String,
dim64 String,
dim65 String,
dim66 String,
dim67 String,
dim68 String,
dim69 String,
dim70 String,
dim71 String,
dim72 String,
dim73 String,
dim74 String,
dim75 String,
dim76 String,
dim77 String,
dim78 String,
dim79 String,
dim80 String,
dim81 String,
M1 double,
M2 double,
M3 double,
M4 double,
M5 double,
M6 double,
M7 double,
M8 double,
M9 double,
M10 double )
stored by 'org.apache.carbondata.format' 
TBLPROPERTIES('DICTIONARY_EXCLUDE'='user_num,user_imsi,user_ID,user_msisdn,user_base_station,user_belong_area_code','table_blocksize'='512');
+-+--+
| Result  |
+-+--+
+-+--+
No rows selected (0.55 seconds)

LOAD DATA inpath 'hdfs://hacluster/user/test/jin_test2.csv' into table JL_r31 
options('DELIMITER'=',', 
'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','IS_EMPTY_DATA_BAD_RECORD'='TRUE','FILEHEADER'='p_cap_time,city,product_code,user_base_station,user_belong_area_code,user_num,user_imsi,user_id,user_msisdn,dim1,dim2,dim3,dim4,dim5,dim6,dim7,dim8,dim9,dim10,dim11,dim12,dim13,dim14,dim15,dim16,dim17,dim18,dim19,dim20,dim21,dim22,dim23,dim24,dim25,dim26,dim27,dim28,dim29,dim30,dim31,dim32,dim33,dim34,dim35,dim36,dim37,dim38,dim39,dim40,dim41,dim42,dim43,dim44,dim45,dim46,dim47,dim48,dim49,dim50,dim51,dim52,dim53,dim54,dim55,dim56,dim57,dim58,dim59,dim60,dim61,dim62,dim63,dim64,dim65,dim66,dim67,dim68,dim69,dim70,dim71,dim72,dim73,dim74,dim75,dim76,dim77,dim78,dim79,dim80,dim81,M1,M2,M3,M4,M5,M6,M7,M8,M9,M10');
+-+--+
| Result  |
+-+--+
+-+--+
No rows selected (14.049 seconds)
0: jdbc:hive2://10.18.98.136:23040> create datamap agr_JL_r31 ON TABLE JL_r31 
USING 'org.apache.carbondta.datamap.AggregateDataMapHandler' as select 
user_num,user_imsi,sum(user_id),count(user_id) from JL_r31 group by  user_num, 
user_imsi;
+-+--+
| Result  |
+-+--+
+-+--+
No rows selected (0.397 seconds)
0: jdbc:hive2://10.18.98.136:23040> select count(*) from JL_r31_agr_JL_r31;
Error: org.apache.spark.sql.AnalysisException: Table or view not found: 
JL_r31_agr_JL_r31; line 1 pos 21 (state=,code=0)
0: jdbc:hive2://10.18.98.136:23040> select 
user_num,user_imsi,sum(user_id),count(user_id) from JL_r31 group by  user_num, 
user_imsi;
Error: java.lang.ClassCastException: 
org.apache.carbondata.core.metadata.schema.table.DataMapSchema cannot be cast 
to org.apache.carbondata.core.metadata.schema.table.AggregationDataMapSchema 
(state=,code=0)


Driver Logs:
2017-11-24 21:45:10,997 | INFO  | [pool-23-thread-4] | Parsing command: create 
datamap agr_JL_r31 ON TABLE JL_r31 USING 
'org.apache.carbondta.datamap.AggregateDataMapHandler' as select 
user_num,user_imsi,sum(user_id),count(user_id) from JL_r31 group by  user_num, 
user_imsi | org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)
2017-11-24 21:45:11,046 | INFO  | [pool-23-thread-4] | pool-23-thread-4 Skip 
CarbonOptimizer | 
org.apache.carbondata.common.logging.impl.StandardLogService.logInfoMessage(StandardLogService.java:150)
2017-11-24 21:45:11,051 | INFO  | [pool-23-thread-4] | 5: get_table : 
db=default tbl=jl_r31 | 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.logInfo(HiveMetaStore.java:746)
2017-11-24 21:45:11,051 | INFO  | [pool-23-thread-4] | ugi=anonymous
ip=unknown-ip-addr  cmd=get_table : db=default tbl=jl_r31| 

[jira] [Closed] (CARBONDATA-1794) Table delete in Beeline does not work on stream table

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S closed CARBONDATA-1794.
-
Resolution: Invalid

> Table delete in Beeline does not work on stream table
> -
>
> Key: CARBONDATA-1794
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1794
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-query
>Affects Versions: 1.3.0
> Environment: 3 node ant cluster
>Reporter: Ramakrishna S
>  Labels: DFX
>
> Steps :
> 1. Create a streaming table and do a batch load
> 2. Set up the Streaming , so that it does streaming in chunk of 1000 records 
> 20 times



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CARBONDATA-1794) Table delete in Beeline does not work on stream table

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1794:
--
Summary: Table delete in Beeline does not work on stream table  (was: Data 
load in Stream Segment fails if batch load is performed in between the 
streaming)

> Table delete in Beeline does not work on stream table
> -
>
> Key: CARBONDATA-1794
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1794
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-query
>Affects Versions: 1.3.0
> Environment: 3 node ant cluster
>Reporter: Ramakrishna S
>  Labels: DFX
>
> Steps :
> 1. Create a streaming table and do a batch load
> 2. Set up the Streaming , so that it does streaming in chunk of 1000 records 
> 20 times



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CARBONDATA-1794) Data load in Stream Segment fails if batch load is performed in between the streaming

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1794:
--
Summary: Data load in Stream Segment fails if batch load is performed in 
between the streaming  (was: (Carbon1.3.0 - Streaming) Data load in Stream 
Segment fails if batch load is performed in between the streaming)

> Data load in Stream Segment fails if batch load is performed in between the 
> streaming
> -
>
> Key: CARBONDATA-1794
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1794
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-query
>Affects Versions: 1.3.0
> Environment: 3 node ant cluster
>Reporter: Ramakrishna S
>  Labels: DFX
>
> Steps :
> 1. Create a streaming table and do a batch load
> 2. Set up the Streaming , so that it does streaming in chunk of 1000 records 
> 20 times



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CARBONDATA-1794) (Carbon1.3.0 - Streaming) Data load in Stream Segment fails if batch load is performed in between the streaming

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1794:
--
Description: 
Steps :
1. Create a streaming table and do a batch load
2. Set up the Streaming , so that it does streaming in chunk of 1000 records 20 
times


  was:
Steps :
1. Create a streaming table and do a batch load
2. Set up the Streaming , so that it does streaming in chunk of 1000 records 20 
times
3. Do another batch load on the table
4. Do one more time streaming
+-++--+--+--++--+
| Segment Id  |   Status   | Load Start Time  |  Load End Time  
 | File Format  | Merged To  |
+-++--+--+--++--+
| 2   | Success| 2017-11-21 21:42:36.77   | 2017-11-21 21:42:40.396 
 | COLUMNAR_V3  | NA |
| 1   | Streaming  | 2017-11-21 21:40:46.2| NULL
 | ROW_V1   | NA |
| 0   | Success| 2017-11-21 21:40:39.782  | 2017-11-21 21:40:43.168 
 | COLUMNAR_V3  | NA |
+-++--+--+--++--+


*+Expected:+* Data should be loaded
*+Actual+* : Data load fiails
1. One addition offset file is created(marked in bold)
-rw-r--r--   2 root users 62 2017-11-21 21:40 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/0
-rw-r--r--   2 root users 63 2017-11-21 21:40 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/1
-rw-r--r--   2 root users 63 2017-11-21 21:42 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/10
-rw-r--r--   2 root users 63 2017-11-21 21:40 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/2
-rw-r--r--   2 root users 63 2017-11-21 21:41 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/3
-rw-r--r--   2 root users 64 2017-11-21 21:41 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/4
-rw-r--r--   2 root users 64 2017-11-21 21:41 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/5
-rw-r--r--   2 root users 64 2017-11-21 21:41 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/6
-rw-r--r--   2 root users 64 2017-11-21 21:41 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/7
-rw-r--r--   2 root users 64 2017-11-21 21:41 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/8
*-rw-r--r--   2 root users 63 2017-11-21 21:42 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/9*
2. Following error thrown:
=== Streaming Query ===
Identifier: [id = 3a5334bc-d471-4676-b6ce-f21105d491d1, runId = 
b2be9f97-8141-46be-89db-9a0f98d13369]
Current Offsets: 
{org.apache.spark.sql.execution.streaming.TextSocketSource@14c45193: 1000}

Current State: ACTIVE
Thread State: RUNNABLE

Logical Plan:
org.apache.spark.sql.execution.streaming.TextSocketSource@14c45193




at 
org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runBatches(StreamExecution.scala:284)
at 
org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:177)
Caused by: java.lang.RuntimeException: Offsets committed out of order: 20019 
followed by 1000
at scala.sys.package$.error(package.scala:27)
at 
org.apache.spark.sql.execution.streaming.TextSocketSource.commit(socket.scala:151)
at 
org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$org$apache$spark$sql$execution$streaming$StreamExecution$$constructNextBatch$2$$anonfun$apply$mcV$sp$4.apply(StreamExecution.scala:421)
at 
org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$org$apache$spark$sql$execution$streaming$StreamExecution$$constructNextBatch$2$$anonfun$apply$mcV$sp$4.apply(StreamExecution.scala:420)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at 
org.apache.spark.sql.execution.streaming.StreamProgress.foreach(StreamProgress.scala:25)
at 
org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$org$apache$spark$sql$execution$streaming$StreamExecution$$constructNextBatch$2.apply$mcV$sp(StreamExecution.scala:420)
at 

[jira] [Created] (CARBONDATA-1794) (Carbon1.3.0 - Streaming) Data load in Stream Segment fails if batch load is performed in between the streaming

[Ramakrishna S (JIRA)]

Ramakrishna S created CARBONDATA-1794:
-

 Summary: (Carbon1.3.0 - Streaming) Data load in Stream Segment 
fails if batch load is performed in between the streaming
 Key: CARBONDATA-1794
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1794
 Project: CarbonData
  Issue Type: Bug
  Components: data-query
Affects Versions: 1.3.0
 Environment: 3 node ant cluster
Reporter: Ramakrishna S


Steps :
1. Create a streaming table and do a batch load
2. Set up the Streaming , so that it does streaming in chunk of 1000 records 20 
times
3. Do another batch load on the table
4. Do one more time streaming
+-++--+--+--++--+
| Segment Id  |   Status   | Load Start Time  |  Load End Time  
 | File Format  | Merged To  |
+-++--+--+--++--+
| 2   | Success| 2017-11-21 21:42:36.77   | 2017-11-21 21:42:40.396 
 | COLUMNAR_V3  | NA |
| 1   | Streaming  | 2017-11-21 21:40:46.2| NULL
 | ROW_V1   | NA |
| 0   | Success| 2017-11-21 21:40:39.782  | 2017-11-21 21:40:43.168 
 | COLUMNAR_V3  | NA |
+-++--+--+--++--+


*+Expected:+* Data should be loaded
*+Actual+* : Data load fiails
1. One addition offset file is created(marked in bold)
-rw-r--r--   2 root users 62 2017-11-21 21:40 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/0
-rw-r--r--   2 root users 63 2017-11-21 21:40 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/1
-rw-r--r--   2 root users 63 2017-11-21 21:42 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/10
-rw-r--r--   2 root users 63 2017-11-21 21:40 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/2
-rw-r--r--   2 root users 63 2017-11-21 21:41 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/3
-rw-r--r--   2 root users 64 2017-11-21 21:41 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/4
-rw-r--r--   2 root users 64 2017-11-21 21:41 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/5
-rw-r--r--   2 root users 64 2017-11-21 21:41 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/6
-rw-r--r--   2 root users 64 2017-11-21 21:41 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/7
-rw-r--r--   2 root users 64 2017-11-21 21:41 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/8
*-rw-r--r--   2 root users 63 2017-11-21 21:42 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/9*
2. Following error thrown:
=== Streaming Query ===
Identifier: [id = 3a5334bc-d471-4676-b6ce-f21105d491d1, runId = 
b2be9f97-8141-46be-89db-9a0f98d13369]
Current Offsets: 
{org.apache.spark.sql.execution.streaming.TextSocketSource@14c45193: 1000}

Current State: ACTIVE
Thread State: RUNNABLE

Logical Plan:
org.apache.spark.sql.execution.streaming.TextSocketSource@14c45193




at 
org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runBatches(StreamExecution.scala:284)
at 
org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:177)
Caused by: java.lang.RuntimeException: Offsets committed out of order: 20019 
followed by 1000
at scala.sys.package$.error(package.scala:27)
at 
org.apache.spark.sql.execution.streaming.TextSocketSource.commit(socket.scala:151)
at 
org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$org$apache$spark$sql$execution$streaming$StreamExecution$$constructNextBatch$2$$anonfun$apply$mcV$sp$4.apply(StreamExecution.scala:421)
at 
org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$org$apache$spark$sql$execution$streaming$StreamExecution$$constructNextBatch$2$$anonfun$apply$mcV$sp$4.apply(StreamExecution.scala:420)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at 
org.apache.spark.sql.execution.streaming.StreamProgress.foreach(StreamProgress.scala:25)
at 
org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$org$apache$spark$sql$execution$streaming$StreamExecution$$constructNextBatch$2.apply$mcV$sp(StreamExecution.scala:420)
at 

[jira] [Updated] (CARBONDATA-1790) (Carbon1.3.0 - Streaming) Data load in Stream Segment fails if batch load is performed in between the streaming

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1790:
--
Description: 
Steps :
1. Create a streaming table and do a batch load
2. Set up the Streaming , so that it does streaming in chunk of 1000 records 20 
times
3. Do another batch load on the table
4. Do one more time streaming
+-++--+--+--++--+
| Segment Id  |   Status   | Load Start Time  |  Load End Time  
 | File Format  | Merged To  |
+-++--+--+--++--+
| 2   | Success| 2017-11-21 21:42:36.77   | 2017-11-21 21:42:40.396 
 | COLUMNAR_V3  | NA |
| 1   | Streaming  | 2017-11-21 21:40:46.2| NULL
 | ROW_V1   | NA |
| 0   | Success| 2017-11-21 21:40:39.782  | 2017-11-21 21:40:43.168 
 | COLUMNAR_V3  | NA |
+-++--+--+--++--+


*+Expected:+* Data should be loaded
*+Actual+* : Data load fiails
1. One addition offset file is created(marked in bold)
-rw-r--r--   2 root users 62 2017-11-21 21:40 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/0
-rw-r--r--   2 root users 63 2017-11-21 21:40 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/1
-rw-r--r--   2 root users 63 2017-11-21 21:42 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/10
-rw-r--r--   2 root users 63 2017-11-21 21:40 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/2
-rw-r--r--   2 root users 63 2017-11-21 21:41 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/3
-rw-r--r--   2 root users 64 2017-11-21 21:41 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/4
-rw-r--r--   2 root users 64 2017-11-21 21:41 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/5
-rw-r--r--   2 root users 64 2017-11-21 21:41 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/6
-rw-r--r--   2 root users 64 2017-11-21 21:41 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/7
-rw-r--r--   2 root users 64 2017-11-21 21:41 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/8
*-rw-r--r--   2 root users 63 2017-11-21 21:42 
/user/hive/warehouse/Ram/default/stream_table5/.streaming/checkpoint/offsets/9*
2. Following error thrown:
=== Streaming Query ===
Identifier: [id = 3a5334bc-d471-4676-b6ce-f21105d491d1, runId = 
b2be9f97-8141-46be-89db-9a0f98d13369]
Current Offsets: 
{org.apache.spark.sql.execution.streaming.TextSocketSource@14c45193: 1000}

Current State: ACTIVE
Thread State: RUNNABLE

Logical Plan:
org.apache.spark.sql.execution.streaming.TextSocketSource@14c45193




at 
org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runBatches(StreamExecution.scala:284)
at 
org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:177)
Caused by: java.lang.RuntimeException: Offsets committed out of order: 20019 
followed by 1000
at scala.sys.package$.error(package.scala:27)
at 
org.apache.spark.sql.execution.streaming.TextSocketSource.commit(socket.scala:151)
at 
org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$org$apache$spark$sql$execution$streaming$StreamExecution$$constructNextBatch$2$$anonfun$apply$mcV$sp$4.apply(StreamExecution.scala:421)
at 
org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$org$apache$spark$sql$execution$streaming$StreamExecution$$constructNextBatch$2$$anonfun$apply$mcV$sp$4.apply(StreamExecution.scala:420)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at 
org.apache.spark.sql.execution.streaming.StreamProgress.foreach(StreamProgress.scala:25)
at 
org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$org$apache$spark$sql$execution$streaming$StreamExecution$$constructNextBatch$2.apply$mcV$sp(StreamExecution.scala:420)
at 
org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$org$apache$spark$sql$execution$streaming$StreamExecution$$constructNextBatch$2.apply(StreamExecution.scala:404)
at 

[jira] [Created] (CARBONDATA-1790) (Carbon1.3.0 - Streaming) Data load in Stream Segment fails if batch load is performed in between the streaming

[Ramakrishna S (JIRA)]

Ramakrishna S created CARBONDATA-1790:
-

 Summary: (Carbon1.3.0 - Streaming) Data load in Stream Segment 
fails if batch load is performed in between the streaming
 Key: CARBONDATA-1790
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1790
 Project: CarbonData
  Issue Type: Bug
  Components: data-query
Affects Versions: 1.3.0
 Environment: 3 node ant cluster
Reporter: Ramakrishna S


Steps :
User starts the thrift server using the command - bin/spark-submit --master 
yarn-client --executor-memory 10G --executor-cores 5 --driver-memory 5G 
--num-executors 3 --class 
org.apache.carbondata.spark.thriftserver.CarbonThriftServer 
/srv/spark2.2Bigdata/install/spark/sparkJdbc/carbonlib/carbondata_2.11-1.3.0-SNAPSHOT-shade-hadoop2.7.2.jar
 "hdfs://hacluster/user/hive/warehouse/carbon.store"
User connects to spark shell using the command - bin/spark-shell --master 
yarn-client --executor-memory 10G --executor-cores 5 --driver-memory 5G 
--num-executors 3 --jars 
/srv/spark2.2Bigdata/install/spark/sparkJdbc/carbonlib/carbondata_2.11-1.3.0-SNAPSHOT-shade-hadoop2.7.2.jar

In spark shell User creates a table and does streaming load in the table as per 
the below socket streaming script.
import java.io.{File, PrintWriter}
import java.net.ServerSocket

import org.apache.spark.sql.{CarbonEnv, SparkSession}
import org.apache.spark.sql.hive.CarbonRelation
import org.apache.spark.sql.streaming.{ProcessingTime, StreamingQuery}

import org.apache.carbondata.core.constants.CarbonCommonConstants
import org.apache.carbondata.core.util.CarbonProperties
import org.apache.carbondata.core.util.path.{CarbonStorePath, CarbonTablePath}

CarbonProperties.getInstance().addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
 "/MM/dd")

import org.apache.spark.sql.CarbonSession._

val carbonSession = SparkSession.
  builder().
  appName("StreamExample").
  getOrCreateCarbonSession("hdfs://hacluster/user/hive/warehouse/david")
   
carbonSession.sparkContext.setLogLevel("INFO")

def sql(sql: String) = carbonSession.sql(sql)

def writeSocket(serverSocket: ServerSocket): Thread = {
  val thread = new Thread() {
override def run(): Unit = {
  // wait for client to connection request and accept
  val clientSocket = serverSocket.accept()
  val socketWriter = new PrintWriter(clientSocket.getOutputStream())
  var index = 0
  for (_ <- 1 to 1000) {
// write 5 records per iteration
for (_ <- 0 to 100) {
  index = index + 1
  socketWriter.println(index.toString + ",name_" + index
   + ",city_" + index + "," + (index * 
1.00).toString +
   ",school_" + index + ":school_" + index + index 
+ "$" + index)
}
socketWriter.flush()
Thread.sleep(2000)
  }
  socketWriter.close()
  System.out.println("Socket closed")
}
  }
  thread.start()
  thread
}
  
def startStreaming(spark: SparkSession, tablePath: CarbonTablePath, tableName: 
String, port: Int): Thread = {
val thread = new Thread() {
  override def run(): Unit = {
var qry: StreamingQuery = null
try {
  val readSocketDF = spark.readStream
.format("socket")
.option("host", "10.18.98.34")
.option("port", port)
.load()

  qry = readSocketDF.writeStream
.format("carbondata")
.trigger(ProcessingTime("5 seconds"))
.option("checkpointLocation", tablePath.getStreamingCheckpointDir)
.option("tablePath", tablePath.getPath).option("tableName", 
tableName)
.start()

  qry.awaitTermination()
} catch {
  case ex: Throwable =>
ex.printStackTrace()
println("Done reading and writing streaming data")
} finally {
  qry.stop()
}
  }
}
thread.start()
thread
}

val streamTableName = "stream_table"

sql(s"CREATE TABLE $streamTableName (id INT,name STRING,city STRING,salary 
FLOAT) STORED BY 'carbondata' TBLPROPERTIES('streaming'='true', 
'sort_columns'='name')")

sql(s"LOAD DATA LOCAL INPATH 'hdfs://hacluster/tmp/streamSample.csv' INTO TABLE 
$streamTableName OPTIONS('HEADER'='true')")

sql(s"select * from $streamTableName").show

val carbonTable = CarbonEnv.getInstance(carbonSession).carbonMetastore.
  lookupRelation(Some("default"), 
streamTableName)(carbonSession).asInstanceOf[CarbonRelation].carbonTable

val tablePath = 
CarbonStorePath.getCarbonTablePath(carbonTable.getAbsoluteTableIdentifier)

val port = 7995
val serverSocket = new ServerSocket(port)
val socketThread = writeSocket(serverSocket)
val streamingThread = startStreaming(carbonSession, tablePath, streamTableName, 
port)

While load is in progress user executes select query on the streaming table 
from beeline.
0: 

[jira] [Commented] (CARBONDATA-1777) Carbon1.3.0-Pre-AggregateTable - Pre-aggregate tables created in Spark-shell sessions are not used in the beeline session

[Ramakrishna S (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-1777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259086#comment-16259086
 ] 

Ramakrishna S commented on CARBONDATA-1777:
---

[~kumarvishal], this happens when pre-aggregate table is created in a different 
session (spark-shell). but select * on aggregate table is working fine.


> Carbon1.3.0-Pre-AggregateTable - Pre-aggregate tables created in Spark-shell 
> sessions are not used in the beeline session
> -
>
> Key: CARBONDATA-1777
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1777
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-load
>Affects Versions: 1.3.0
> Environment: Test - 3 node ant cluster
>Reporter: Ramakrishna S
>Assignee: Kunal Kapoor
>  Labels: DFX
> Fix For: 1.3.0
>
>
> Steps:
> Beeline:
> 1. Create table and load with  data
> Spark-shell:
> 1. create a pre-aggregate table
> Beeline:
> 1. Run aggregate query
> *+Expected:+* Pre-aggregate table should be used in the aggregate query 
> *+Actual:+* Pre-aggregate table is not used
> 1.
> create table if not exists lineitem1(L_SHIPDATE string,L_SHIPMODE 
> string,L_SHIPINSTRUCT string,L_RETURNFLAG string,L_RECEIPTDATE 
> string,L_ORDERKEY string,L_PARTKEY string,L_SUPPKEY   string,L_LINENUMBER 
> int,L_QUANTITY double,L_EXTENDEDPRICE double,L_DISCOUNT double,L_TAX 
> double,L_LINESTATUS string,L_COMMITDATE string,L_COMMENT  string) STORED BY 
> 'org.apache.carbondata.format' TBLPROPERTIES 
> ('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
> load data inpath "hdfs://hacluster/user/test/lineitem.tbl.5" into table 
> lineitem1 
> options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');
> 2. 
>  carbon.sql("create datamap agr1_lineitem1 ON TABLE lineitem1 USING 
> 'org.apache.carbondata.datamap.AggregateDataMapHandler' as select 
> l_returnflag,l_linestatus,sum(l_quantity),avg(l_quantity),count(l_quantity) 
> from lineitem1 group by l_returnflag, l_linestatus").show();
> 3. 
> select 
> l_returnflag,l_linestatus,sum(l_quantity),avg(l_quantity),count(l_quantity) 
> from lineitem1 where l_returnflag = 'R' group by l_returnflag, l_linestatus;
> Actual:
> 0: jdbc:hive2://10.18.98.136:23040> show tables;
> +---+---+--+--+
> | database  | tableName | isTemporary  |
> +---+---+--+--+
> | test_db2  | lineitem1 | false|
> | test_db2  | lineitem1_agr1_lineitem1  | false|
> +---+---+--+--+
> 2 rows selected (0.047 seconds)
> Logs:
> 2017-11-20 15:46:48,314 | INFO  | [pool-23-thread-53] | Running query 'select 
> l_returnflag,l_linestatus,sum(l_quantity),avg(l_quantity),count(l_quantity) 
> from lineitem1 where l_returnflag = 'R' group by l_returnflag, l_linestatus' 
> with 7f3091a8-4d7b-40ac-840f-9db6f564c9cf | 
> org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)
> 2017-11-20 15:46:48,314 | INFO  | [pool-23-thread-53] | Parsing command: 
> select 
> l_returnflag,l_linestatus,sum(l_quantity),avg(l_quantity),count(l_quantity) 
> from lineitem1 where l_returnflag = 'R' group by l_returnflag, l_linestatus | 
> org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)
> 2017-11-20 15:46:48,353 | INFO  | [pool-23-thread-53] | 55: get_table : 
> db=test_db2 tbl=lineitem1 | 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.logInfo(HiveMetaStore.java:746)
> 2017-11-20 15:46:48,353 | INFO  | [pool-23-thread-53] | ugi=anonymous 
> ip=unknown-ip-addr  cmd=get_table : db=test_db2 tbl=lineitem1| 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.logAuditEvent(HiveMetaStore.java:371)
> 2017-11-20 15:46:48,354 | INFO  | [pool-23-thread-53] | 55: Opening raw store 
> with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore | 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:589)
> 2017-11-20 15:46:48,355 | INFO  | [pool-23-thread-53] | ObjectStore, 
> initialize called | 
> org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:289)
> 2017-11-20 15:46:48,360 | INFO  | [pool-23-thread-53] | Reading in results 
> for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection 
> used is closing | org.datanucleus.util.Log4JLogger.info(Log4JLogger.java:77)
> 2017-11-20 15:46:48,362 | INFO  | 

[jira] [Updated] (CARBONDATA-1711) Carbon1.3.0-DataMap - Show datamap on table does not work

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1711:
--
Summary: Carbon1.3.0-DataMap - Show datamap  on table  does not 
work  (was: Carbon1.3.0-Pre-AggregateTable - Show datamap  on table  
does not work)

> Carbon1.3.0-DataMap - Show datamap  on table  does not work
> --
>
> Key: CARBONDATA-1711
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1711
> Project: CarbonData
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.3.0
> Environment: Test
>Reporter: Ramakrishna S
>Priority: Minor
>  Labels: Functional
> Fix For: 1.3.0
>
>
> 0: jdbc:hive2://10.18.98.34:23040> create datamap agr_lineitem ON TABLE 
> lineitem USING "org.apache.carbondata.datamap.AggregateDataMapHandler" as 
> select L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from 
> lineitem group by  L_RETURNFLAG, L_LINESTATUS;
> Error: java.lang.RuntimeException: Table [lineitem_agr_lineitem] already 
> exists under database [default] (state=,code=0)
> 0: jdbc:hive2://10.18.98.34:23040> show tables;
> +---+---+--+--+
> | database  | tableName | isTemporary  |
> +---+---+--+--+
> | default   | flow_carbon_test4 | false|
> | default   | jl_r3 | false|
> | default   | lineitem  | false|
> | default   | lineitem_agr_lineitem | false|
> | default   | sensor_reading_blockblank_false   | false|
> | default   | sensor_reading_blockblank_false1  | false|
> | default   | sensor_reading_blockblank_false2  | false|
> | default   | sensor_reading_false  | false|
> | default   | sensor_reading_true   | false|
> | default   | t1| false|
> | default   | t1_agg_t1 | false|
> | default   | tc4   | false|
> | default   | uniqdata  | false|
> +---+---+--+--+
> 13 rows selected (0.04 seconds)
> 0: jdbc:hive2://10.18.98.34:23040> show datamap on table lineitem;
> Error: java.lang.RuntimeException:
> BaseSqlParser
> missing 'FUNCTIONS' at 'on'(line 1, pos 13)
> == SQL ==
> show datamap on table lineitem
> -^^^
> CarbonSqlParser [1.6] failure: identifier matching regex (?i)SEGMENTS 
> expected
> show datamap on table lineitem



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CARBONDATA-1777) Carbon1.3.0-Pre-AggregateTable - Pre-aggregate tables creation in Spark-shell sessions are not used in the beeline session

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1777:
--
Description: 
Steps:
Beeline:
1. Create table and load with  data
Spark-shell:
1. create a pre-aggregate table
Beeline:
1. Run aggregate query

*+Expected:+* Pre-aggregate table should be used in the aggregate query 
*+Actual:+* Pre-aggregate table is not used


1.
create table if not exists lineitem1(L_SHIPDATE string,L_SHIPMODE 
string,L_SHIPINSTRUCT string,L_RETURNFLAG string,L_RECEIPTDATE 
string,L_ORDERKEY string,L_PARTKEY string,L_SUPPKEY   string,L_LINENUMBER 
int,L_QUANTITY double,L_EXTENDEDPRICE double,L_DISCOUNT double,L_TAX 
double,L_LINESTATUS string,L_COMMITDATE string,L_COMMENT  string) STORED BY 
'org.apache.carbondata.format' TBLPROPERTIES 
('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
load data inpath "hdfs://hacluster/user/test/lineitem.tbl.5" into table 
lineitem1 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');

2. 

 carbon.sql("create datamap agr1_lineitem1 ON TABLE lineitem1 USING 
'org.apache.carbondata.datamap.AggregateDataMapHandler' as select 
l_returnflag,l_linestatus,sum(l_quantity),avg(l_quantity),count(l_quantity) 
from lineitem1 group by l_returnflag, l_linestatus").show();

3. 
select 
l_returnflag,l_linestatus,sum(l_quantity),avg(l_quantity),count(l_quantity) 
from lineitem1 where l_returnflag = 'R' group by l_returnflag, l_linestatus;

Actual:
0: jdbc:hive2://10.18.98.136:23040> show tables;
+---+---+--+--+
| database  | tableName | isTemporary  |
+---+---+--+--+
| test_db2  | lineitem1 | false|
| test_db2  | lineitem1_agr1_lineitem1  | false|
+---+---+--+--+
2 rows selected (0.047 seconds)

Logs:
2017-11-20 15:46:48,314 | INFO  | [pool-23-thread-53] | Running query 'select 
l_returnflag,l_linestatus,sum(l_quantity),avg(l_quantity),count(l_quantity) 
from lineitem1 where l_returnflag = 'R' group by l_returnflag, l_linestatus' 
with 7f3091a8-4d7b-40ac-840f-9db6f564c9cf | 
org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)
2017-11-20 15:46:48,314 | INFO  | [pool-23-thread-53] | Parsing command: select 
l_returnflag,l_linestatus,sum(l_quantity),avg(l_quantity),count(l_quantity) 
from lineitem1 where l_returnflag = 'R' group by l_returnflag, l_linestatus | 
org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)
2017-11-20 15:46:48,353 | INFO  | [pool-23-thread-53] | 55: get_table : 
db=test_db2 tbl=lineitem1 | 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.logInfo(HiveMetaStore.java:746)
2017-11-20 15:46:48,353 | INFO  | [pool-23-thread-53] | ugi=anonymous   
ip=unknown-ip-addr  cmd=get_table : db=test_db2 tbl=lineitem1| 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.logAuditEvent(HiveMetaStore.java:371)
2017-11-20 15:46:48,354 | INFO  | [pool-23-thread-53] | 55: Opening raw store 
with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore | 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:589)
2017-11-20 15:46:48,355 | INFO  | [pool-23-thread-53] | ObjectStore, initialize 
called | 
org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:289)
2017-11-20 15:46:48,360 | INFO  | [pool-23-thread-53] | Reading in results for 
query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used 
is closing | org.datanucleus.util.Log4JLogger.info(Log4JLogger.java:77)
2017-11-20 15:46:48,362 | INFO  | [pool-23-thread-53] | Using direct SQL, 
underlying DB is MYSQL | 
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.(MetaStoreDirectSql.java:139)
2017-11-20 15:46:48,362 | INFO  | [pool-23-thread-53] | Initialized ObjectStore 
| org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:272)
2017-11-20 15:46:48,376 | INFO  | [pool-23-thread-53] | Parsing command: 
array | 
org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)
2017-11-20 15:46:48,399 | INFO  | [pool-23-thread-53] | Schema changes have 
been detected for table: `lineitem1` | 
org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)
2017-11-20 15:46:48,399 | INFO  | [pool-23-thread-53] | 55: get_table : 
db=test_db2 tbl=lineitem1 | 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.logInfo(HiveMetaStore.java:746)
2017-11-20 15:46:48,400 | INFO  | [pool-23-thread-53] | ugi=anonymous   
ip=unknown-ip-addr  cmd=get_table : db=test_db2 tbl=lineitem1| 

[jira] [Updated] (CARBONDATA-1777) Carbon1.3.0-Pre-AggregateTable - Pre-aggregate tables created in Spark-shell sessions are not used in the beeline session

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1777:
--
Summary: Carbon1.3.0-Pre-AggregateTable - Pre-aggregate tables created in 
Spark-shell sessions are not used in the beeline session  (was: 
Carbon1.3.0-Pre-AggregateTable - Pre-aggregate tables creation in Spark-shell 
sessions are not used in the beeline session)

> Carbon1.3.0-Pre-AggregateTable - Pre-aggregate tables created in Spark-shell 
> sessions are not used in the beeline session
> -
>
> Key: CARBONDATA-1777
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1777
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-load
>Affects Versions: 1.3.0
> Environment: Test - 3 node ant cluster
>Reporter: Ramakrishna S
>Assignee: Kunal Kapoor
>  Labels: DFX
> Fix For: 1.3.0
>
>
> Steps:
> Beeline:
> 1. Create table and load with  data
> Spark-shell:
> 1. create a pre-aggregate table
> Beeline:
> 1. Run aggregate query
> *+Expected:+* Pre-aggregate table should be used in the aggregate query 
> *+Actual:+* Pre-aggregate table is not used
> 1.
> create table if not exists lineitem1(L_SHIPDATE string,L_SHIPMODE 
> string,L_SHIPINSTRUCT string,L_RETURNFLAG string,L_RECEIPTDATE 
> string,L_ORDERKEY string,L_PARTKEY string,L_SUPPKEY   string,L_LINENUMBER 
> int,L_QUANTITY double,L_EXTENDEDPRICE double,L_DISCOUNT double,L_TAX 
> double,L_LINESTATUS string,L_COMMITDATE string,L_COMMENT  string) STORED BY 
> 'org.apache.carbondata.format' TBLPROPERTIES 
> ('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
> load data inpath "hdfs://hacluster/user/test/lineitem.tbl.5" into table 
> lineitem1 
> options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');
> 2. 
>  carbon.sql("create datamap agr1_lineitem1 ON TABLE lineitem1 USING 
> 'org.apache.carbondata.datamap.AggregateDataMapHandler' as select 
> l_returnflag,l_linestatus,sum(l_quantity),avg(l_quantity),count(l_quantity) 
> from lineitem1 group by l_returnflag, l_linestatus").show();
> 3. 
> select 
> l_returnflag,l_linestatus,sum(l_quantity),avg(l_quantity),count(l_quantity) 
> from lineitem1 where l_returnflag = 'R' group by l_returnflag, l_linestatus;
> Actual:
> 0: jdbc:hive2://10.18.98.136:23040> show tables;
> +---+---+--+--+
> | database  | tableName | isTemporary  |
> +---+---+--+--+
> | test_db2  | lineitem1 | false|
> | test_db2  | lineitem1_agr1_lineitem1  | false|
> +---+---+--+--+
> 2 rows selected (0.047 seconds)
> Logs:
> 2017-11-20 15:46:48,314 | INFO  | [pool-23-thread-53] | Running query 'select 
> l_returnflag,l_linestatus,sum(l_quantity),avg(l_quantity),count(l_quantity) 
> from lineitem1 where l_returnflag = 'R' group by l_returnflag, l_linestatus' 
> with 7f3091a8-4d7b-40ac-840f-9db6f564c9cf | 
> org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)
> 2017-11-20 15:46:48,314 | INFO  | [pool-23-thread-53] | Parsing command: 
> select 
> l_returnflag,l_linestatus,sum(l_quantity),avg(l_quantity),count(l_quantity) 
> from lineitem1 where l_returnflag = 'R' group by l_returnflag, l_linestatus | 
> org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)
> 2017-11-20 15:46:48,353 | INFO  | [pool-23-thread-53] | 55: get_table : 
> db=test_db2 tbl=lineitem1 | 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.logInfo(HiveMetaStore.java:746)
> 2017-11-20 15:46:48,353 | INFO  | [pool-23-thread-53] | ugi=anonymous 
> ip=unknown-ip-addr  cmd=get_table : db=test_db2 tbl=lineitem1| 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.logAuditEvent(HiveMetaStore.java:371)
> 2017-11-20 15:46:48,354 | INFO  | [pool-23-thread-53] | 55: Opening raw store 
> with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore | 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:589)
> 2017-11-20 15:46:48,355 | INFO  | [pool-23-thread-53] | ObjectStore, 
> initialize called | 
> org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:289)
> 2017-11-20 15:46:48,360 | INFO  | [pool-23-thread-53] | Reading in results 
> for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection 
> used is closing | 

[jira] [Updated] (CARBONDATA-1763) Carbon1.3.0-Pre-AggregateTable - Recreating a failed pre-aggregate table fails due to table exists

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1763:
--
Description: 
Steps:
1. Create table and load with  data
2. Run update query on the table - this will take table metalock
3. In parallel run the pre-aggregate table create step - this will not be 
allowed due to table lock
4. Rerun pre-aggegate table create step

*+Expected:+* Pre-aggregate table should be created 
*+Actual:+* Pre-aggregate table creation fails

+Create, Load & Update+:
0: jdbc:hive2://10.18.98.136:23040> create table if not exists 
lineitem4(L_SHIPDATE string,L_SHIPMODE string,L_SHIPINSTRUCT 
string,L_RETURNFLAG string,L_RECEIPTDATE string,L_ORDERKEY string,L_PARTKEY 
string,L_SUPPKEY   string,L_LINENUMBER int,L_QUANTITY double,L_EXTENDEDPRICE 
double,L_DISCOUNT double,L_TAX double,L_LINESTATUS string,L_COMMITDATE 
string,L_COMMENT  string) STORED BY 'org.apache.carbondata.format' 
TBLPROPERTIES 
('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
+-+--+
| Result  |
+-+--+
+-+--+
No rows selected (0.266 seconds)
0: jdbc:hive2://10.18.98.136:23040> load data inpath 
"hdfs://hacluster/user/test/lineitem.tbl.5" into table lineitem4 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');
+-+--+
| Result  |
+-+--+
+-+--+
No rows selected (6.331 seconds)
0: jdbc:hive2://10.18.98.136:23040> update lineitem4 set (l_linestatus) = 
('xx');

+Create Datamap:+
0: jdbc:hive2://10.18.98.136:23040> create datamap agr_lineitem4 ON TABLE 
lineitem4 USING "org.apache.carbondata.datamap.AggregateDataMapHandler" as 
select 
l_returnflag,l_linestatus,sum(l_quantity),avg(l_quantity),count(l_quantity) 
from lineitem4  group by l_returnflag, l_linestatus;
Error: java.lang.RuntimeException: Acquire table lock failed after retry, 
please try after some time (state=,code=0)
0: jdbc:hive2://10.18.98.136:23040> select 
l_returnflag,l_linestatus,sum(l_quantity),avg(l_quantity),count(l_quantity) 
from lineitem4 group by l_returnflag, l_linestatus;
+---+---+--+-++--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  |   avg(l_quantity)   | 
count(l_quantity)  |
+---+---+--+-++--+
| N | xx| 1.2863213E7  | 25.48745561614304   | 
504688 |
| A | xx| 6318125.0| 25.506342144783375  | 
247708 |
| R | xx| 6321939.0| 25.532459087898417  | 
247604 |
+---+---+--+-++--+
3 rows selected (1.033 seconds)
0: jdbc:hive2://10.18.98.136:23040> create datamap agr_lineitem4 ON TABLE 
lineitem4 USING "org.apache.carbondata.datamap.AggregateDataMapHandler" as 
select 
l_returnflag,l_linestatus,sum(l_quantity),avg(l_quantity),count(l_quantity) 
from lineitem4  group by l_returnflag, l_linestatus;
Error: java.lang.RuntimeException: Table [lineitem4_agr_lineitem4] already 
exists under database [test_db1] (state=,code=0)


  was:
Steps:
1. Create table and load with large data
create table if not exists lineitem4(L_SHIPDATE string,L_SHIPMODE 
string,L_SHIPINSTRUCT string,L_RETURNFLAG string,L_RECEIPTDATE 
string,L_ORDERKEY string,L_PARTKEY string,L_SUPPKEY   string,L_LINENUMBER 
int,L_QUANTITY double,L_EXTENDEDPRICE double,L_DISCOUNT double,L_TAX 
double,L_LINESTATUS string,L_COMMITDATE string,L_COMMENT  string) STORED BY 
'org.apache.carbondata.format' TBLPROPERTIES 
('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table 
lineitem4 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');

2. Create a pre-aggregate table 
create datamap agr_lineitem4 ON TABLE lineitem4 USING 
"org.apache.carbondata.datamap.AggregateDataMapHandler" as select 
L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from lineitem4 
group by  L_RETURNFLAG, L_LINESTATUS;

3. Run aggregate query at the same time
 select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem4 group by l_returnflag, l_linestatus;

*+Expected:+*: aggregate query should fetch data either from main table or 

[jira] [Created] (CARBONDATA-1763) Carbon1.3.0-Pre-AggregateTable - Recreating a failed pre-aggregate table fails due to table exists

[Ramakrishna S (JIRA)]

Ramakrishna S created CARBONDATA-1763:
-

 Summary: Carbon1.3.0-Pre-AggregateTable - Recreating a failed 
pre-aggregate table fails due to table exists
 Key: CARBONDATA-1763
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1763
 Project: CarbonData
  Issue Type: Bug
  Components: data-load
Affects Versions: 1.3.0
 Environment: Test - 3 node ant cluster
Reporter: Ramakrishna S
Assignee: Kunal Kapoor
 Fix For: 1.3.0


Steps:
1. Create table and load with large data
create table if not exists lineitem4(L_SHIPDATE string,L_SHIPMODE 
string,L_SHIPINSTRUCT string,L_RETURNFLAG string,L_RECEIPTDATE 
string,L_ORDERKEY string,L_PARTKEY string,L_SUPPKEY   string,L_LINENUMBER 
int,L_QUANTITY double,L_EXTENDEDPRICE double,L_DISCOUNT double,L_TAX 
double,L_LINESTATUS string,L_COMMITDATE string,L_COMMENT  string) STORED BY 
'org.apache.carbondata.format' TBLPROPERTIES 
('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table 
lineitem4 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');

2. Create a pre-aggregate table 
create datamap agr_lineitem4 ON TABLE lineitem4 USING 
"org.apache.carbondata.datamap.AggregateDataMapHandler" as select 
L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from lineitem4 
group by  L_RETURNFLAG, L_LINESTATUS;

3. Run aggregate query at the same time
 select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem4 group by l_returnflag, l_linestatus;

*+Expected:+*: aggregate query should fetch data either from main table or 
pre-aggregate table.
*+Actual:+* aggregate query does not return data until the pre-aggregate table 
is created


0: jdbc:hive2://10.18.98.48:23040> select 
l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from lineitem4 
group by l_returnflag, l_linestatus;
+---+---+--+---+--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  | sum(l_extendedprice)  |
+---+---+--+---+--+
+---+---+--+---+--+
No rows selected (1.74 seconds)
0: jdbc:hive2://10.18.98.48:23040> select 
l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from lineitem4 
group by l_returnflag, l_linestatus;
+---+---+--+---+--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  | sum(l_extendedprice)  |
+---+---+--+---+--+
+---+---+--+---+--+
No rows selected (0.746 seconds)
0: jdbc:hive2://10.18.98.48:23040> select 
l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from lineitem4 
group by l_returnflag, l_linestatus;
+---+---+--++--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  |  sum(l_extendedprice)  |
+---+---+--++--+
| N | F | 2.9808092E7  | 4.471079473931997E10   |
| A | F | 1.145546488E9| 1.717580824169429E12   |
| N | O | 2.31980219E9 | 3.4789002701143467E12  |
| R | F | 1.146403932E9| 1.7190627928317903E12  |
+---+---+--++--+
4 rows selected (0.8 seconds)
0: jdbc:hive2://10.18.98.48:23040> select 
l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from lineitem4 
group by l_returnflag, l_linestatus;
+---+---+--++--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  |  sum(l_extendedprice)  |
+---+---+--++--+
| N | F | 2.9808092E7  | 4.471079473931997E10   |
| A | F | 1.145546488E9| 1.717580824169429E12   |
| N | O | 2.31980219E9 | 3.4789002701143467E12  |
| R | F | 1.146403932E9| 1.7190627928317903E12  |
+---+---+--++--+




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CARBONDATA-1713) Carbon1.3.0-Pre-AggregateTable - Aggregate query on main table fails after creating pre-aggregate table when upper case used for column name

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1713:
--
Summary: Carbon1.3.0-Pre-AggregateTable - Aggregate query on main table 
fails after creating pre-aggregate table when upper case used for column name  
(was: Carbon1.3.0-Pre-AggregateTable - Aggregate query on main table fails 
after creating pre-aggregate table)

> Carbon1.3.0-Pre-AggregateTable - Aggregate query on main table fails after 
> creating pre-aggregate table when upper case used for column name
> 
>
> Key: CARBONDATA-1713
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1713
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-query
>Affects Versions: 1.3.0
> Environment: ANT Test cluster - 3 node
>Reporter: Ramakrishna S
>Assignee: kumar vishal
>Priority: Minor
>  Labels: Functional, sanity
> Fix For: 1.3.0
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> 0: jdbc:hive2://10.18.98.34:23040> load data inpath 
> "hdfs://hacluster/user/test/lineitem.tbl.1" into table lineitem 
> options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');
> Error: org.apache.spark.sql.catalyst.analysis.NoSuchTableException: Table or 
> view 'lineitem' not found in database 'default'; (state=,code=0)
> 0: jdbc:hive2://10.18.98.34:23040> create table if not exists lineitem(
> 0: jdbc:hive2://10.18.98.34:23040> L_SHIPDATE string,
> 0: jdbc:hive2://10.18.98.34:23040> L_SHIPMODE string,
> 0: jdbc:hive2://10.18.98.34:23040> L_SHIPINSTRUCT string,
> 0: jdbc:hive2://10.18.98.34:23040> L_RETURNFLAG string,
> 0: jdbc:hive2://10.18.98.34:23040> L_RECEIPTDATE string,
> 0: jdbc:hive2://10.18.98.34:23040> L_ORDERKEY string,
> 0: jdbc:hive2://10.18.98.34:23040> L_PARTKEY string,
> 0: jdbc:hive2://10.18.98.34:23040> L_SUPPKEY   string,
> 0: jdbc:hive2://10.18.98.34:23040> L_LINENUMBER int,
> 0: jdbc:hive2://10.18.98.34:23040> L_QUANTITY double,
> 0: jdbc:hive2://10.18.98.34:23040> L_EXTENDEDPRICE double,
> 0: jdbc:hive2://10.18.98.34:23040> L_DISCOUNT double,
> 0: jdbc:hive2://10.18.98.34:23040> L_TAX double,
> 0: jdbc:hive2://10.18.98.34:23040> L_LINESTATUS string,
> 0: jdbc:hive2://10.18.98.34:23040> L_COMMITDATE string,
> 0: jdbc:hive2://10.18.98.34:23040> L_COMMENT  string
> 0: jdbc:hive2://10.18.98.34:23040> ) STORED BY 'org.apache.carbondata.format'
> 0: jdbc:hive2://10.18.98.34:23040> TBLPROPERTIES 
> ('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (0.338 seconds)
> 0: jdbc:hive2://10.18.98.34:23040> load data inpath 
> "hdfs://hacluster/user/test/lineitem.tbl.1" into table lineitem 
> options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (48.634 seconds)
> 0: jdbc:hive2://10.18.98.34:23040> create datamap agr_lineitem ON TABLE 
> lineitem USING "org.apache.carbondata.datamap.AggregateDataMapHandler" as 
> select L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from 
> lineitem group by  L_RETURNFLAG, L_LINESTATUS;
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (16.552 seconds)
> 0: jdbc:hive2://10.18.98.34:23040> select 
> L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from lineitem 
> group by  L_RETURNFLAG, L_LINESTATUS;
> Error: org.apache.spark.sql.AnalysisException: Column doesnot exists in Pre 
> Aggregate table; (state=,code=0)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (CARBONDATA-1713) Carbon1.3.0-Pre-AggregateTable - Aggregate query on main table fails after creating pre-aggregate table

[Ramakrishna S (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253213#comment-16253213
 ] 

Ramakrishna S edited comment on CARBONDATA-1713 at 11/17/17 5:05 AM:
-

Changing severity based on the clarification provided, will use lower case for 
query till this issue is fixed.


was (Author: ram@huawei):
Changing severity based on the clarification given.

> Carbon1.3.0-Pre-AggregateTable - Aggregate query on main table fails after 
> creating pre-aggregate table
> ---
>
> Key: CARBONDATA-1713
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1713
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-query
>Affects Versions: 1.3.0
> Environment: ANT Test cluster - 3 node
>Reporter: Ramakrishna S
>Assignee: kumar vishal
>Priority: Minor
>  Labels: Functional, sanity
> Fix For: 1.3.0
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> 0: jdbc:hive2://10.18.98.34:23040> load data inpath 
> "hdfs://hacluster/user/test/lineitem.tbl.1" into table lineitem 
> options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');
> Error: org.apache.spark.sql.catalyst.analysis.NoSuchTableException: Table or 
> view 'lineitem' not found in database 'default'; (state=,code=0)
> 0: jdbc:hive2://10.18.98.34:23040> create table if not exists lineitem(
> 0: jdbc:hive2://10.18.98.34:23040> L_SHIPDATE string,
> 0: jdbc:hive2://10.18.98.34:23040> L_SHIPMODE string,
> 0: jdbc:hive2://10.18.98.34:23040> L_SHIPINSTRUCT string,
> 0: jdbc:hive2://10.18.98.34:23040> L_RETURNFLAG string,
> 0: jdbc:hive2://10.18.98.34:23040> L_RECEIPTDATE string,
> 0: jdbc:hive2://10.18.98.34:23040> L_ORDERKEY string,
> 0: jdbc:hive2://10.18.98.34:23040> L_PARTKEY string,
> 0: jdbc:hive2://10.18.98.34:23040> L_SUPPKEY   string,
> 0: jdbc:hive2://10.18.98.34:23040> L_LINENUMBER int,
> 0: jdbc:hive2://10.18.98.34:23040> L_QUANTITY double,
> 0: jdbc:hive2://10.18.98.34:23040> L_EXTENDEDPRICE double,
> 0: jdbc:hive2://10.18.98.34:23040> L_DISCOUNT double,
> 0: jdbc:hive2://10.18.98.34:23040> L_TAX double,
> 0: jdbc:hive2://10.18.98.34:23040> L_LINESTATUS string,
> 0: jdbc:hive2://10.18.98.34:23040> L_COMMITDATE string,
> 0: jdbc:hive2://10.18.98.34:23040> L_COMMENT  string
> 0: jdbc:hive2://10.18.98.34:23040> ) STORED BY 'org.apache.carbondata.format'
> 0: jdbc:hive2://10.18.98.34:23040> TBLPROPERTIES 
> ('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (0.338 seconds)
> 0: jdbc:hive2://10.18.98.34:23040> load data inpath 
> "hdfs://hacluster/user/test/lineitem.tbl.1" into table lineitem 
> options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (48.634 seconds)
> 0: jdbc:hive2://10.18.98.34:23040> create datamap agr_lineitem ON TABLE 
> lineitem USING "org.apache.carbondata.datamap.AggregateDataMapHandler" as 
> select L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from 
> lineitem group by  L_RETURNFLAG, L_LINESTATUS;
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (16.552 seconds)
> 0: jdbc:hive2://10.18.98.34:23040> select 
> L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from lineitem 
> group by  L_RETURNFLAG, L_LINESTATUS;
> Error: org.apache.spark.sql.AnalysisException: Column doesnot exists in Pre 
> Aggregate table; (state=,code=0)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CARBONDATA-1743) Carbon1.3.0-Pre-AggregateTable - Query returns no value if run at the time of pre-aggregate table creation

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1743:
--
Description: 
Steps:
1. Create table and load with large data
create table if not exists lineitem4(L_SHIPDATE string,L_SHIPMODE 
string,L_SHIPINSTRUCT string,L_RETURNFLAG string,L_RECEIPTDATE 
string,L_ORDERKEY string,L_PARTKEY string,L_SUPPKEY   string,L_LINENUMBER 
int,L_QUANTITY double,L_EXTENDEDPRICE double,L_DISCOUNT double,L_TAX 
double,L_LINESTATUS string,L_COMMITDATE string,L_COMMENT  string) STORED BY 
'org.apache.carbondata.format' TBLPROPERTIES 
('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table 
lineitem4 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');

2. Create a pre-aggregate table 
create datamap agr_lineitem4 ON TABLE lineitem4 USING 
"org.apache.carbondata.datamap.AggregateDataMapHandler" as select 
L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from lineitem4 
group by  L_RETURNFLAG, L_LINESTATUS;

3. Run aggregate query at the same time
 select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem4 group by l_returnflag, l_linestatus;

*+Expected:+*: aggregate query should fetch data either from main table or 
pre-aggregate table.
*+Actual:+* aggregate query does not return data until the pre-aggregate table 
is created


0: jdbc:hive2://10.18.98.48:23040> select 
l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from lineitem4 
group by l_returnflag, l_linestatus;
+---+---+--+---+--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  | sum(l_extendedprice)  |
+---+---+--+---+--+
+---+---+--+---+--+
No rows selected (1.74 seconds)
0: jdbc:hive2://10.18.98.48:23040> select 
l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from lineitem4 
group by l_returnflag, l_linestatus;
+---+---+--+---+--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  | sum(l_extendedprice)  |
+---+---+--+---+--+
+---+---+--+---+--+
No rows selected (0.746 seconds)
0: jdbc:hive2://10.18.98.48:23040> select 
l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from lineitem4 
group by l_returnflag, l_linestatus;
+---+---+--++--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  |  sum(l_extendedprice)  |
+---+---+--++--+
| N | F | 2.9808092E7  | 4.471079473931997E10   |
| A | F | 1.145546488E9| 1.717580824169429E12   |
| N | O | 2.31980219E9 | 3.4789002701143467E12  |
| R | F | 1.146403932E9| 1.7190627928317903E12  |
+---+---+--++--+
4 rows selected (0.8 seconds)
0: jdbc:hive2://10.18.98.48:23040> select 
l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from lineitem4 
group by l_returnflag, l_linestatus;
+---+---+--++--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  |  sum(l_extendedprice)  |
+---+---+--++--+
| N | F | 2.9808092E7  | 4.471079473931997E10   |
| A | F | 1.145546488E9| 1.717580824169429E12   |
| N | O | 2.31980219E9 | 3.4789002701143467E12  |
| R | F | 1.146403932E9| 1.7190627928317903E12  |
+---+---+--++--+


  was:
lineitem3: has a pre-aggregate table 
select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem3 group by l_returnflag, l_linestatus order by l_returnflag, 
l_linestatus;
Error: org.apache.spark.sql.AnalysisException: expression 
'`lineitem3_l_returnflag`' is neither present in the group by, nor is it an 
aggregate function. Add to group by or wrap in first() (or first_value) if you 
don't care which value you get.;;
Project [l_returnflag#2356, l_linestatus#2366, sum(l_quantity)#2791, 
sum(l_extendedprice)#2792]
+- Sort [aggOrder#2795 ASC NULLS 

[jira] [Created] (CARBONDATA-1743) Carbon1.3.0-Pre-AggregateTable - Query returns no value if run at the time of pre-aggregate table creation

[Ramakrishna S (JIRA)]

Ramakrishna S created CARBONDATA-1743:
-

 Summary: Carbon1.3.0-Pre-AggregateTable - Query returns no value 
if run at the time of pre-aggregate table creation
 Key: CARBONDATA-1743
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1743
 Project: CarbonData
  Issue Type: Bug
  Components: data-load
Affects Versions: 1.3.0
 Environment: Test - 3 node ant cluster
Reporter: Ramakrishna S
 Fix For: 1.3.0


lineitem3: has a pre-aggregate table 
select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem3 group by l_returnflag, l_linestatus order by l_returnflag, 
l_linestatus;
Error: org.apache.spark.sql.AnalysisException: expression 
'`lineitem3_l_returnflag`' is neither present in the group by, nor is it an 
aggregate function. Add to group by or wrap in first() (or first_value) if you 
don't care which value you get.;;
Project [l_returnflag#2356, l_linestatus#2366, sum(l_quantity)#2791, 
sum(l_extendedprice)#2792]
+- Sort [aggOrder#2795 ASC NULLS FIRST, aggOrder#2796 ASC NULLS FIRST], true
   +- !Aggregate [l_returnflag#2356, l_linestatus#2366], [l_returnflag#2356, 
l_linestatus#2366, sum(l_quantity#2362) AS sum(l_quantity)#2791, 
sum(l_extendedprice#2363) AS sum(l_extendedprice)#2792, 
lineitem3_l_returnflag#2341 AS aggOrder#2795, lineitem3_l_linestatus#2342 AS 
aggOrder#2796]
  +- SubqueryAlias lineitem3
 +- 
Relation[L_SHIPDATE#2353,L_SHIPMODE#2354,L_SHIPINSTRUCT#2355,L_RETURNFLAG#2356,L_RECEIPTDATE#2357,L_ORDERKEY#2358,L_PARTKEY#2359,L_SUPPKEY#2360,L_LINENUMBER#2361,L_QUANTITY#2362,L_EXTENDEDPRICE#2363,L_DISCOUNT#2364,L_TAX#2365,L_LINESTATUS#2366,L_COMMITDATE#2367,L_COMMENT#2368]
 CarbonDatasourceHadoopRelation [ Database name :test_db1, Table name 
:lineitem3, Schema :Some(StructType(StructField(L_SHIPDATE,StringType,true), 
StructField(L_SHIPMODE,StringType,true), 
StructField(L_SHIPINSTRUCT,StringType,true), 
StructField(L_RETURNFLAG,StringType,true), 
StructField(L_RECEIPTDATE,StringType,true), 
StructField(L_ORDERKEY,StringType,true), 
StructField(L_PARTKEY,StringType,true), StructField(L_SUPPKEY,StringType,true), 
StructField(L_LINENUMBER,IntegerType,true), 
StructField(L_QUANTITY,DoubleType,true), 
StructField(L_EXTENDEDPRICE,DoubleType,true), 
StructField(L_DISCOUNT,DoubleType,true), StructField(L_TAX,DoubleType,true), 
StructField(L_LINESTATUS,StringType,true), 
StructField(L_COMMITDATE,StringType,true), 
StructField(L_COMMENT,StringType,true))) ] (state=,code=0)

lineitem4: no pre-aggregate table created
select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem4 group by l_returnflag, l_linestatus order by l_returnflag, 
l_linestatus;
+---+---+--++--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  |  sum(l_extendedprice)  |
+---+---+--++--+
| A | F | 1.263625E7   | 1.8938515425239815E10  |
| N | F | 327800.0 | 4.91387677622E8|
| N | O | 2.5398626E7  | 3.810981608977963E10   |
| R | F | 1.2643878E7  | 1.8948524305619884E10  |
+---+---+--++--+

*+Expected:+*: aggregate query with order by should run fine
*+Actual:+* aggregate query with order failed 





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CARBONDATA-1740) Carbon1.3.0-Pre-AggregateTable - Query plan exception for aggregate query with order by when main table is having pre-aggregate table

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1740:
--
Description: 
lineitem3: has a pre-aggregate table 
select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem3 group by l_returnflag, l_linestatus order by l_returnflag, 
l_linestatus;
Error: org.apache.spark.sql.AnalysisException: expression 
'`lineitem3_l_returnflag`' is neither present in the group by, nor is it an 
aggregate function. Add to group by or wrap in first() (or first_value) if you 
don't care which value you get.;;
Project [l_returnflag#2356, l_linestatus#2366, sum(l_quantity)#2791, 
sum(l_extendedprice)#2792]
+- Sort [aggOrder#2795 ASC NULLS FIRST, aggOrder#2796 ASC NULLS FIRST], true
   +- !Aggregate [l_returnflag#2356, l_linestatus#2366], [l_returnflag#2356, 
l_linestatus#2366, sum(l_quantity#2362) AS sum(l_quantity)#2791, 
sum(l_extendedprice#2363) AS sum(l_extendedprice)#2792, 
lineitem3_l_returnflag#2341 AS aggOrder#2795, lineitem3_l_linestatus#2342 AS 
aggOrder#2796]
  +- SubqueryAlias lineitem3
 +- 
Relation[L_SHIPDATE#2353,L_SHIPMODE#2354,L_SHIPINSTRUCT#2355,L_RETURNFLAG#2356,L_RECEIPTDATE#2357,L_ORDERKEY#2358,L_PARTKEY#2359,L_SUPPKEY#2360,L_LINENUMBER#2361,L_QUANTITY#2362,L_EXTENDEDPRICE#2363,L_DISCOUNT#2364,L_TAX#2365,L_LINESTATUS#2366,L_COMMITDATE#2367,L_COMMENT#2368]
 CarbonDatasourceHadoopRelation [ Database name :test_db1, Table name 
:lineitem3, Schema :Some(StructType(StructField(L_SHIPDATE,StringType,true), 
StructField(L_SHIPMODE,StringType,true), 
StructField(L_SHIPINSTRUCT,StringType,true), 
StructField(L_RETURNFLAG,StringType,true), 
StructField(L_RECEIPTDATE,StringType,true), 
StructField(L_ORDERKEY,StringType,true), 
StructField(L_PARTKEY,StringType,true), StructField(L_SUPPKEY,StringType,true), 
StructField(L_LINENUMBER,IntegerType,true), 
StructField(L_QUANTITY,DoubleType,true), 
StructField(L_EXTENDEDPRICE,DoubleType,true), 
StructField(L_DISCOUNT,DoubleType,true), StructField(L_TAX,DoubleType,true), 
StructField(L_LINESTATUS,StringType,true), 
StructField(L_COMMITDATE,StringType,true), 
StructField(L_COMMENT,StringType,true))) ] (state=,code=0)

lineitem4: no pre-aggregate table created
select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem4 group by l_returnflag, l_linestatus order by l_returnflag, 
l_linestatus;
+---+---+--++--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  |  sum(l_extendedprice)  |
+---+---+--++--+
| A | F | 1.263625E7   | 1.8938515425239815E10  |
| N | F | 327800.0 | 4.91387677622E8|
| N | O | 2.5398626E7  | 3.810981608977963E10   |
| R | F | 1.2643878E7  | 1.8948524305619884E10  |
+---+---+--++--+

*+Expected:+*: aggregate query with order by should run fine
*+Actual:+* aggregate query with order failed 



  was:

0: jdbc:hive2://10.18.98.48:23040> select 
l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from lineitem3 
group by l_returnflag, l_linestatus order by l_returnflag, l_linestatus;
Error: org.apache.spark.sql.AnalysisException: expression 
'`lineitem3_l_returnflag`' is neither present in the group by, nor is it an 
aggregate function. Add to group by or wrap in first() (or first_value) if you 
don't care which value you get.;;
Project [l_returnflag#2356, l_linestatus#2366, sum(l_quantity)#2791, 
sum(l_extendedprice)#2792]
+- Sort [aggOrder#2795 ASC NULLS FIRST, aggOrder#2796 ASC NULLS FIRST], true
   +- !Aggregate [l_returnflag#2356, l_linestatus#2366], [l_returnflag#2356, 
l_linestatus#2366, sum(l_quantity#2362) AS sum(l_quantity)#2791, 
sum(l_extendedprice#2363) AS sum(l_extendedprice)#2792, 
lineitem3_l_returnflag#2341 AS aggOrder#2795, lineitem3_l_linestatus#2342 AS 
aggOrder#2796]
  +- SubqueryAlias lineitem3
 +- 
Relation[L_SHIPDATE#2353,L_SHIPMODE#2354,L_SHIPINSTRUCT#2355,L_RETURNFLAG#2356,L_RECEIPTDATE#2357,L_ORDERKEY#2358,L_PARTKEY#2359,L_SUPPKEY#2360,L_LINENUMBER#2361,L_QUANTITY#2362,L_EXTENDEDPRICE#2363,L_DISCOUNT#2364,L_TAX#2365,L_LINESTATUS#2366,L_COMMITDATE#2367,L_COMMENT#2368]
 CarbonDatasourceHadoopRelation [ Database name :test_db1, Table name 
:lineitem3, Schema :Some(StructType(StructField(L_SHIPDATE,StringType,true), 
StructField(L_SHIPMODE,StringType,true), 
StructField(L_SHIPINSTRUCT,StringType,true), 
StructField(L_RETURNFLAG,StringType,true), 
StructField(L_RECEIPTDATE,StringType,true), 
StructField(L_ORDERKEY,StringType,true), 
StructField(L_PARTKEY,StringType,true), StructField(L_SUPPKEY,StringType,true), 
StructField(L_LINENUMBER,IntegerType,true), 

[jira] [Updated] (CARBONDATA-1740) Carbon1.3.0-Pre-AggregateTable - Query plan exception for aggregate query with order by when main table is having pre-aggregate table

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1740:
--
Summary: Carbon1.3.0-Pre-AggregateTable - Query plan exception for 
aggregate query with order by when main table is having pre-aggregate table  
(was: Carbon1.3.0-Pre-AggregateTable - Aggregate query with order by fails when 
main table is having pre-aggregate table)

> Carbon1.3.0-Pre-AggregateTable - Query plan exception for aggregate query 
> with order by when main table is having pre-aggregate table
> -
>
> Key: CARBONDATA-1740
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1740
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-load
>Affects Versions: 1.3.0
> Environment: Test - 3 node ant cluster
>Reporter: Ramakrishna S
>  Labels: DFX
> Fix For: 1.3.0
>
>
> 0: jdbc:hive2://10.18.98.48:23040> select 
> l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from lineitem3 
> group by l_returnflag, l_linestatus order by l_returnflag, l_linestatus;
> Error: org.apache.spark.sql.AnalysisException: expression 
> '`lineitem3_l_returnflag`' is neither present in the group by, nor is it an 
> aggregate function. Add to group by or wrap in first() (or first_value) if 
> you don't care which value you get.;;
> Project [l_returnflag#2356, l_linestatus#2366, sum(l_quantity)#2791, 
> sum(l_extendedprice)#2792]
> +- Sort [aggOrder#2795 ASC NULLS FIRST, aggOrder#2796 ASC NULLS FIRST], true
>+- !Aggregate [l_returnflag#2356, l_linestatus#2366], [l_returnflag#2356, 
> l_linestatus#2366, sum(l_quantity#2362) AS sum(l_quantity)#2791, 
> sum(l_extendedprice#2363) AS sum(l_extendedprice)#2792, 
> lineitem3_l_returnflag#2341 AS aggOrder#2795, lineitem3_l_linestatus#2342 AS 
> aggOrder#2796]
>   +- SubqueryAlias lineitem3
>  +- 
> Relation[L_SHIPDATE#2353,L_SHIPMODE#2354,L_SHIPINSTRUCT#2355,L_RETURNFLAG#2356,L_RECEIPTDATE#2357,L_ORDERKEY#2358,L_PARTKEY#2359,L_SUPPKEY#2360,L_LINENUMBER#2361,L_QUANTITY#2362,L_EXTENDEDPRICE#2363,L_DISCOUNT#2364,L_TAX#2365,L_LINESTATUS#2366,L_COMMITDATE#2367,L_COMMENT#2368]
>  CarbonDatasourceHadoopRelation [ Database name :test_db1, Table name 
> :lineitem3, Schema :Some(StructType(StructField(L_SHIPDATE,StringType,true), 
> StructField(L_SHIPMODE,StringType,true), 
> StructField(L_SHIPINSTRUCT,StringType,true), 
> StructField(L_RETURNFLAG,StringType,true), 
> StructField(L_RECEIPTDATE,StringType,true), 
> StructField(L_ORDERKEY,StringType,true), 
> StructField(L_PARTKEY,StringType,true), 
> StructField(L_SUPPKEY,StringType,true), 
> StructField(L_LINENUMBER,IntegerType,true), 
> StructField(L_QUANTITY,DoubleType,true), 
> StructField(L_EXTENDEDPRICE,DoubleType,true), 
> StructField(L_DISCOUNT,DoubleType,true), StructField(L_TAX,DoubleType,true), 
> StructField(L_LINESTATUS,StringType,true), 
> StructField(L_COMMITDATE,StringType,true), 
> StructField(L_COMMENT,StringType,true))) ] (state=,code=0)
> 0: jdbc:hive2://10.18.98.48:23040> select 
> l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from lineitem4 
> group by l_returnflag, l_linestatus order by l_returnflag, l_linestatus;
> +---+---+--++--+
> | l_returnflag  | l_linestatus  | sum(l_quantity)  |  sum(l_extendedprice)  |
> +---+---+--++--+
> | A | F | 1.263625E7   | 1.8938515425239815E10  |
> | N | F | 327800.0 | 4.91387677622E8|
> | N | O | 2.5398626E7  | 3.810981608977963E10   |
> | R | F | 1.2643878E7  | 1.8948524305619884E10  |
> +---+---+--++--+
> *+Expected:+*: one of these should have been the behavour:
> 1.Ignore segment filter and use all segments for pre-aggregate load. At the 
> time of query run, if segment filter is set then ignore the pre-aggr table 
> and fetch data from main table. (*Preferred*)
> Or
> 2. Reject pre-aggregate creation when segment filter is set or vis-a-versa.
> *+Actual:+* Partial data returned 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CARBONDATA-1740) Carbon1.3.0-Pre-AggregateTable - Aggregate query with order by fails when main table is having pre-aggregate table

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1740:
--
Description: 

0: jdbc:hive2://10.18.98.48:23040> select 
l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from lineitem3 
group by l_returnflag, l_linestatus order by l_returnflag, l_linestatus;
Error: org.apache.spark.sql.AnalysisException: expression 
'`lineitem3_l_returnflag`' is neither present in the group by, nor is it an 
aggregate function. Add to group by or wrap in first() (or first_value) if you 
don't care which value you get.;;
Project [l_returnflag#2356, l_linestatus#2366, sum(l_quantity)#2791, 
sum(l_extendedprice)#2792]
+- Sort [aggOrder#2795 ASC NULLS FIRST, aggOrder#2796 ASC NULLS FIRST], true
   +- !Aggregate [l_returnflag#2356, l_linestatus#2366], [l_returnflag#2356, 
l_linestatus#2366, sum(l_quantity#2362) AS sum(l_quantity)#2791, 
sum(l_extendedprice#2363) AS sum(l_extendedprice)#2792, 
lineitem3_l_returnflag#2341 AS aggOrder#2795, lineitem3_l_linestatus#2342 AS 
aggOrder#2796]
  +- SubqueryAlias lineitem3
 +- 
Relation[L_SHIPDATE#2353,L_SHIPMODE#2354,L_SHIPINSTRUCT#2355,L_RETURNFLAG#2356,L_RECEIPTDATE#2357,L_ORDERKEY#2358,L_PARTKEY#2359,L_SUPPKEY#2360,L_LINENUMBER#2361,L_QUANTITY#2362,L_EXTENDEDPRICE#2363,L_DISCOUNT#2364,L_TAX#2365,L_LINESTATUS#2366,L_COMMITDATE#2367,L_COMMENT#2368]
 CarbonDatasourceHadoopRelation [ Database name :test_db1, Table name 
:lineitem3, Schema :Some(StructType(StructField(L_SHIPDATE,StringType,true), 
StructField(L_SHIPMODE,StringType,true), 
StructField(L_SHIPINSTRUCT,StringType,true), 
StructField(L_RETURNFLAG,StringType,true), 
StructField(L_RECEIPTDATE,StringType,true), 
StructField(L_ORDERKEY,StringType,true), 
StructField(L_PARTKEY,StringType,true), StructField(L_SUPPKEY,StringType,true), 
StructField(L_LINENUMBER,IntegerType,true), 
StructField(L_QUANTITY,DoubleType,true), 
StructField(L_EXTENDEDPRICE,DoubleType,true), 
StructField(L_DISCOUNT,DoubleType,true), StructField(L_TAX,DoubleType,true), 
StructField(L_LINESTATUS,StringType,true), 
StructField(L_COMMITDATE,StringType,true), 
StructField(L_COMMENT,StringType,true))) ] (state=,code=0)

0: jdbc:hive2://10.18.98.48:23040> select 
l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from lineitem4 
group by l_returnflag, l_linestatus order by l_returnflag, l_linestatus;
+---+---+--++--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  |  sum(l_extendedprice)  |
+---+---+--++--+
| A | F | 1.263625E7   | 1.8938515425239815E10  |
| N | F | 327800.0 | 4.91387677622E8|
| N | O | 2.5398626E7  | 3.810981608977963E10   |
| R | F | 1.2643878E7  | 1.8948524305619884E10  |
+---+---+--++--+

*+Expected:+*: one of these should have been the behavour:
1.Ignore segment filter and use all segments for pre-aggregate load. At the 
time of query run, if segment filter is set then ignore the pre-aggr table and 
fetch data from main table. (*Preferred*)
Or
2. Reject pre-aggregate creation when segment filter is set or vis-a-versa.

*+Actual:+* Partial data returned 



  was:
1. Create a table
create table if not exists lineitem2(L_SHIPDATE string,L_SHIPMODE 
string,L_SHIPINSTRUCT string,L_RETURNFLAG string,L_RECEIPTDATE 
string,L_ORDERKEY string,L_PARTKEY string,L_SUPPKEY   string,L_LINENUMBER 
int,L_QUANTITY double,L_EXTENDEDPRICE double,L_DISCOUNT double,L_TAX 
double,L_LINESTATUS string,L_COMMITDATE string,L_COMMENT  string) STORED BY 
'org.apache.carbondata.format' TBLPROPERTIES 
('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
2. Load 2 times to create 2 segments
 load data inpath "hdfs://hacluster/user/test/lineitem.tbl.5" into table 
lineitem2 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');
3. Check the table content without setting any filter:
select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem2 group by l_returnflag, l_linestatus;
+---+---+--++--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  |  sum(l_extendedprice)  |
+---+---+--++--+
| N | F | 327800.0 | 4.91387677624E8|
| A | F | 1.263625E7   | 1.893851542524009E10   |
| N

[jira] [Created] (CARBONDATA-1740) Carbon1.3.0-Pre-AggregateTable - Aggregate query with order by fails when main table is having pre-aggregate table

[Ramakrishna S (JIRA)]

Ramakrishna S created CARBONDATA-1740:
-

 Summary: Carbon1.3.0-Pre-AggregateTable - Aggregate query with 
order by fails when main table is having pre-aggregate table
 Key: CARBONDATA-1740
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1740
 Project: CarbonData
  Issue Type: Bug
  Components: data-load
Affects Versions: 1.3.0
 Environment: Test - 3 node ant cluster
Reporter: Ramakrishna S
 Fix For: 1.3.0


1. Create a table
create table if not exists lineitem2(L_SHIPDATE string,L_SHIPMODE 
string,L_SHIPINSTRUCT string,L_RETURNFLAG string,L_RECEIPTDATE 
string,L_ORDERKEY string,L_PARTKEY string,L_SUPPKEY   string,L_LINENUMBER 
int,L_QUANTITY double,L_EXTENDEDPRICE double,L_DISCOUNT double,L_TAX 
double,L_LINESTATUS string,L_COMMITDATE string,L_COMMENT  string) STORED BY 
'org.apache.carbondata.format' TBLPROPERTIES 
('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
2. Load 2 times to create 2 segments
 load data inpath "hdfs://hacluster/user/test/lineitem.tbl.5" into table 
lineitem2 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');
3. Check the table content without setting any filter:
select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem2 group by l_returnflag, l_linestatus;
+---+---+--++--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  |  sum(l_extendedprice)  |
+---+---+--++--+
| N | F | 327800.0 | 4.91387677624E8|
| A | F | 1.263625E7   | 1.893851542524009E10   |
| N | O | 2.5398626E7  | 3.810981608977967E10   |
| R | F | 1.2643878E7  | 1.8948524305619976E10  |
+---+---+--++--+
4. Set segment filter on the main table:
set carbon.input.segments.test_db1.lineitem2=1;
+---++--+
|key| value  |
+---++--+
| carbon.input.segments.test_db1.lineitem2  | 1  |
+---++--+
5. Create pre-aggregate table 
create datamap agr_lineitem2 ON TABLE lineitem2 USING 
"org.apache.carbondata.datamap.AggregateDataMapHandler" as select 
L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from lineitem2 
group by  L_RETURNFLAG, L_LINESTATUS;
6. Check table content:
 select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem2 group by l_returnflag, l_linestatus;
+---+---+--++--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  |  sum(l_extendedprice)  |
+---+---+--++--+
| N | F | 163900.0 | 2.456938388124E8   |
| A | F | 6318125.0| 9.469257712620043E9|
| N | O | 1.2699313E7  | 1.9054908044889835E10  |
| R | F | 6321939.0| 9.474262152809986E9|
+---+---+--++--+

7. remove the filter on segment
0: jdbc:hive2://10.18.98.48:23040> reset;

8. Check the table conent:
 select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem2 group by l_returnflag, l_linestatus;
+---+---+--++--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  |  sum(l_extendedprice)  |
+---+---+--++--+
| N | F | 163900.0 | 2.456938388124E8   |
| A | F | 6318125.0| 9.469257712620043E9|
| N | O | 1.2699313E7  | 1.9054908044889835E10  |
| R | F | 6321939.0| 9.474262152809986E9|
+---+---+--++--+
4 rows selected (2.341 seconds)

9. Load one more time:
10. Check table content
select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem2 group by l_returnflag, l_linestatus;
+---+---+--++--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  |  sum(l_extendedprice)  |

[jira] [Updated] (CARBONDATA-1737) Carbon1.3.0-Pre-AggregateTable - Pre-aggregate table loads partially when segment filter is set on the main table

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1737:
--
Description: 
1. Create a table
create table if not exists lineitem2(L_SHIPDATE string,L_SHIPMODE 
string,L_SHIPINSTRUCT string,L_RETURNFLAG string,L_RECEIPTDATE 
string,L_ORDERKEY string,L_PARTKEY string,L_SUPPKEY   string,L_LINENUMBER 
int,L_QUANTITY double,L_EXTENDEDPRICE double,L_DISCOUNT double,L_TAX 
double,L_LINESTATUS string,L_COMMITDATE string,L_COMMENT  string) STORED BY 
'org.apache.carbondata.format' TBLPROPERTIES 
('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
2. Load 2 times to create 2 segments
 load data inpath "hdfs://hacluster/user/test/lineitem.tbl.5" into table 
lineitem2 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');
3. Check the table content without setting any filter:
select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem2 group by l_returnflag, l_linestatus;
+---+---+--++--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  |  sum(l_extendedprice)  |
+---+---+--++--+
| N | F | 327800.0 | 4.91387677624E8|
| A | F | 1.263625E7   | 1.893851542524009E10   |
| N | O | 2.5398626E7  | 3.810981608977967E10   |
| R | F | 1.2643878E7  | 1.8948524305619976E10  |
+---+---+--++--+
4. Set segment filter on the main table:
set carbon.input.segments.test_db1.lineitem2=1;
+---++--+
|key| value  |
+---++--+
| carbon.input.segments.test_db1.lineitem2  | 1  |
+---++--+
5. Create pre-aggregate table 
create datamap agr_lineitem2 ON TABLE lineitem2 USING 
"org.apache.carbondata.datamap.AggregateDataMapHandler" as select 
L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from lineitem2 
group by  L_RETURNFLAG, L_LINESTATUS;
6. Check table content:
 select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem2 group by l_returnflag, l_linestatus;
+---+---+--++--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  |  sum(l_extendedprice)  |
+---+---+--++--+
| N | F | 163900.0 | 2.456938388124E8   |
| A | F | 6318125.0| 9.469257712620043E9|
| N | O | 1.2699313E7  | 1.9054908044889835E10  |
| R | F | 6321939.0| 9.474262152809986E9|
+---+---+--++--+

7. remove the filter on segment
0: jdbc:hive2://10.18.98.48:23040> reset;

8. Check the table conent:
 select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem2 group by l_returnflag, l_linestatus;
+---+---+--++--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  |  sum(l_extendedprice)  |
+---+---+--++--+
| N | F | 163900.0 | 2.456938388124E8   |
| A | F | 6318125.0| 9.469257712620043E9|
| N | O | 1.2699313E7  | 1.9054908044889835E10  |
| R | F | 6321939.0| 9.474262152809986E9|
+---+---+--++--+
4 rows selected (2.341 seconds)

9. Load one more time:
10. Check table content
select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem2 group by l_returnflag, l_linestatus;
+---+---+--++--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  |  sum(l_extendedprice)  |
+---+---+--++--+
| N | F | 327800.0 | 4.91387677625E8|
| A | F | 1.263625E7   | 1.8938515425240086E10  |
| N | O | 2.5398626E7  | 3.810981608977967E10   |
| R | F | 1.2643878E7  | 

[jira] [Updated] (CARBONDATA-1737) Carbon1.3.0-Pre-AggregateTable - Pre-aggregate table loads partially when segment filter is set on the main table

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1737:
--
Description: 
1. Create a table
create table if not exists lineitem2(L_SHIPDATE string,L_SHIPMODE 
string,L_SHIPINSTRUCT string,L_RETURNFLAG string,L_RECEIPTDATE 
string,L_ORDERKEY string,L_PARTKEY string,L_SUPPKEY   string,L_LINENUMBER 
int,L_QUANTITY double,L_EXTENDEDPRICE double,L_DISCOUNT double,L_TAX 
double,L_LINESTATUS string,L_COMMITDATE string,L_COMMENT  string) STORED BY 
'org.apache.carbondata.format' TBLPROPERTIES 
('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
2. Load 2 times to create 2 segments
 load data inpath "hdfs://hacluster/user/test/lineitem.tbl.5" into table 
lineitem2 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');
3. Check the table content without setting any filter:
select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem2 group by l_returnflag, l_linestatus;
+---+---+--++--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  |  sum(l_extendedprice)  |
+---+---+--++--+
| N | F | 327800.0 | 4.91387677624E8|
| A | F | 1.263625E7   | 1.893851542524009E10   |
| N | O | 2.5398626E7  | 3.810981608977967E10   |
| R | F | 1.2643878E7  | 1.8948524305619976E10  |
+---+---+--++--+
4. Set segment filter on the main table:
set carbon.input.segments.test_db1.lineitem2=1;
+---++--+
|key| value  |
+---++--+
| carbon.input.segments.test_db1.lineitem2  | 1  |
+---++--+
5. Create pre-aggregate table 
create datamap agr_lineitem2 ON TABLE lineitem2 USING 
"org.apache.carbondata.datamap.AggregateDataMapHandler" as select 
L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from lineitem2 
group by  L_RETURNFLAG, L_LINESTATUS;
6. Check table content:
 select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem2 group by l_returnflag, l_linestatus;
+---+---+--++--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  |  sum(l_extendedprice)  |
+---+---+--++--+
| N | F | 163900.0 | 2.456938388124E8   |
| A | F | 6318125.0| 9.469257712620043E9|
| N | O | 1.2699313E7  | 1.9054908044889835E10  |
| R | F | 6321939.0| 9.474262152809986E9|
+---+---+--++--+

7. remove the filter on segment
0: jdbc:hive2://10.18.98.48:23040> reset;

8. Check the table conent:
 select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem2 group by l_returnflag, l_linestatus;
+---+---+--++--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  |  sum(l_extendedprice)  |
+---+---+--++--+
| N | F | 163900.0 | 2.456938388124E8   |
| A | F | 6318125.0| 9.469257712620043E9|
| N | O | 1.2699313E7  | 1.9054908044889835E10  |
| R | F | 6321939.0| 9.474262152809986E9|
+---+---+--++--+
4 rows selected (2.341 seconds)

9. Load one more time:
10. Check table content
select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem2 group by l_returnflag, l_linestatus;
+---+---+--++--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  |  sum(l_extendedprice)  |
+---+---+--++--+
| N | F | 327800.0 | 4.91387677625E8|
| A | F | 1.263625E7   | 1.8938515425240086E10  |
| N | O | 2.5398626E7  | 3.810981608977967E10   |
| R | F | 1.2643878E7  | 

[jira] [Created] (CARBONDATA-1737) Carbon1.3.0-Pre-AggregateTable - Pre-aggregate table loads partially when segment filter is set on the main table

[Ramakrishna S (JIRA)]

Ramakrishna S created CARBONDATA-1737:
-

 Summary: Carbon1.3.0-Pre-AggregateTable - Pre-aggregate table 
loads partially when segment filter is set on the main table
 Key: CARBONDATA-1737
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1737
 Project: CarbonData
  Issue Type: Bug
  Components: data-load
Affects Versions: 1.3.0
 Environment: Test - 3 node ant cluster
Reporter: Ramakrishna S
 Fix For: 1.3.0


1. Create a table
create table if not exists lineitem1(L_SHIPDATE string,L_SHIPMODE 
string,L_SHIPINSTRUCT string,L_RETURNFLAG string,L_RECEIPTDATE 
string,L_ORDERKEY string,L_PARTKEY string,L_SUPPKEY   string,L_LINENUMBER 
int,L_QUANTITY double,L_EXTENDEDPRICE double,L_DISCOUNT double,L_TAX 
double,L_LINESTATUS string,L_COMMITDATE string,L_COMMENT  string) STORED BY 
'org.apache.carbondata.format' TBLPROPERTIES 
('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
2. Run load :
load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table 
lineitem1 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');

3. create pre-agg table 
create datamap agr_lineitem3 ON TABLE lineitem3 USING 
"org.apache.carbondata.datamap.AggregateDataMapHandler" as select 
L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from lineitem3 
group by  L_RETURNFLAG, L_LINESTATUS;

3.  Check table content using aggregate query:
select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem3 group by l_returnflag, l_linestatus;

+---+---+--++--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  |  sum(l_extendedprice)  |
+---+---+--++--+
| N | F | 4913382.0| 7.369901176949993E9|
| A | F | 1.88818373E8 | 2.8310705145736383E11  |
| N | O | 3.82400594E8 | 5.734650756707479E11   |
| R | F | 1.88960009E8 | 2.833523780876951E11   |
+---+---+--++--+
4 rows selected (1.568 seconds)

4. Load one more time:
load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table 
lineitem1 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');

5.  Check table content using aggregate query:
select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem3 group by l_returnflag, l_linestatus;

+---+---+--++--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  |  sum(l_extendedprice)  |
+---+---+--++--+
| N | F | 9826764.0| 1.4739802353899986E10  |
| A | F | 3.77636746E8 | 5.662141029147278E11   |
| N | O | 7.64801188E8 | 1.1469301513414958E12  |
| R | F | 3.77920018E8 | 5.667047561753901E11   |
+---+---+--++--+

6. Set query from segment 1:

0: jdbc:hive2://10.18.98.48:23040> set 
carbon.input.segments.test_db1.lilneitem1=1;
+++--+
|key | value  |
+++--+
| carbon.input.segments.test_db1.lilneitem1  | 1  |
+++--+

7. Check table content using aggregate query:
select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem3 group by l_returnflag, l_linestatus;

*+Expected+*: It should return the values from segment 1 alone.
*+Actual :+* : It returns values from both segments
+---+---+--++--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  |  sum(l_extendedprice)  |
+---+---+--++--+
| N | F | 9826764.0| 1.4739802353899986E10  |
| A | F | 3.77636746E8 | 5.662141029147278E11   |
| N | O | 7.64801188E8 | 1.1469301513414958E12  |
| R | F | 3.77920018E8 | 5.667047561753901E11   |

[jira] [Updated] (CARBONDATA-1736) Carbon1.3.0-Pre-AggregateTable -Query from segment set is not effective when pre-aggregate table is present

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1736:
--
Description: 
1. Create a table
create table if not exists lineitem1(L_SHIPDATE string,L_SHIPMODE 
string,L_SHIPINSTRUCT string,L_RETURNFLAG string,L_RECEIPTDATE 
string,L_ORDERKEY string,L_PARTKEY string,L_SUPPKEY   string,L_LINENUMBER 
int,L_QUANTITY double,L_EXTENDEDPRICE double,L_DISCOUNT double,L_TAX 
double,L_LINESTATUS string,L_COMMITDATE string,L_COMMENT  string) STORED BY 
'org.apache.carbondata.format' TBLPROPERTIES 
('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
2. Run load :
load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table 
lineitem1 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');

3. create pre-agg table 
create datamap agr_lineitem3 ON TABLE lineitem3 USING 
"org.apache.carbondata.datamap.AggregateDataMapHandler" as select 
L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from lineitem3 
group by  L_RETURNFLAG, L_LINESTATUS;

3.  Check table content using aggregate query:
select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem3 group by l_returnflag, l_linestatus;

+---+---+--++--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  |  sum(l_extendedprice)  |
+---+---+--++--+
| N | F | 4913382.0| 7.369901176949993E9|
| A | F | 1.88818373E8 | 2.8310705145736383E11  |
| N | O | 3.82400594E8 | 5.734650756707479E11   |
| R | F | 1.88960009E8 | 2.833523780876951E11   |
+---+---+--++--+
4 rows selected (1.568 seconds)

4. Load one more time:
load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table 
lineitem1 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');

5.  Check table content using aggregate query:
select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem3 group by l_returnflag, l_linestatus;

+---+---+--++--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  |  sum(l_extendedprice)  |
+---+---+--++--+
| N | F | 9826764.0| 1.4739802353899986E10  |
| A | F | 3.77636746E8 | 5.662141029147278E11   |
| N | O | 7.64801188E8 | 1.1469301513414958E12  |
| R | F | 3.77920018E8 | 5.667047561753901E11   |
+---+---+--++--+

6. Set query from segment 1:

0: jdbc:hive2://10.18.98.48:23040> set 
carbon.input.segments.test_db1.lilneitem1=1;
+++--+
|key | value  |
+++--+
| carbon.input.segments.test_db1.lilneitem1  | 1  |
+++--+

7. Check table content using aggregate query:
select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem3 group by l_returnflag, l_linestatus;

*+Expected+*: It should return the values from segment 1 alone.
*+Actual :+* : It returns values from both segments
+---+---+--++--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  |  sum(l_extendedprice)  |
+---+---+--++--+
| N | F | 9826764.0| 1.4739802353899986E10  |
| A | F | 3.77636746E8 | 5.662141029147278E11   |
| N | O | 7.64801188E8 | 1.1469301513414958E12  |
| R | F | 3.77920018E8 | 5.667047561753901E11   |
+---+---+--++--+



  was:
1. Create a table
create table if not exists lineitem3(L_SHIPDATE string,L_SHIPMODE 
string,L_SHIPINSTRUCT string,L_RETURNFLAG string,L_RECEIPTDATE 
string,L_ORDERKEY string,L_PARTKEY string,L_SUPPKEY   string,L_LINENUMBER 
int,L_QUANTITY double,L_EXTENDEDPRICE double,L_DISCOUNT 

[jira] [Created] (CARBONDATA-1736) Carbon1.3.0-Pre-AggregateTable -Query from segment set is not effective when pre-aggregate table is present

[Ramakrishna S (JIRA)]

Ramakrishna S created CARBONDATA-1736:
-

 Summary:  Carbon1.3.0-Pre-AggregateTable -Query from segment set 
is not effective when pre-aggregate table is present 
 Key: CARBONDATA-1736
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1736
 Project: CarbonData
  Issue Type: Bug
  Components: data-load
Affects Versions: 1.3.0
 Environment: Test - 3 node ant cluster
Reporter: Ramakrishna S
 Fix For: 1.3.0


1. Create a table
create table if not exists lineitem3(L_SHIPDATE string,L_SHIPMODE 
string,L_SHIPINSTRUCT string,L_RETURNFLAG string,L_RECEIPTDATE 
string,L_ORDERKEY string,L_PARTKEY string,L_SUPPKEY   string,L_LINENUMBER 
int,L_QUANTITY double,L_EXTENDEDPRICE double,L_DISCOUNT double,L_TAX 
double,L_LINESTATUS string,L_COMMITDATE string,L_COMMENT  string) STORED BY 
'org.apache.carbondata.format' TBLPROPERTIES 
('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
2. Run load queries and create pre-agg table queries in diff console:
load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table 
lineitem3 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');

create datamap agr_lineitem3 ON TABLE lineitem3 USING 
"org.apache.carbondata.datamap.AggregateDataMapHandler" as select 
L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from lineitem3 
group by  L_RETURNFLAG, L_LINESTATUS;

3.  Check table content using aggregate query:
select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem3 group by l_returnflag, l_linestatus;

0: jdbc:hive2://10.18.98.34:23040> select 
l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from lineitem3 
group by l_returnflag, l_linestatus;
+---+---+--+---+--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  | sum(l_extendedprice)  |
+---+---+--+---+--+
+---+---+--+---+--+
No rows selected (1.258 seconds)


HDFS data:
BLR114307:/srv/spark2.2Bigdata/install/hadoop/datanode # bin/hadoop fs -ls 
/carbonstore/default/lineitem3_agr_lineitem3/Fact/Part0/Segment_0
BLR114307:/srv/spark2.2Bigdata/install/hadoop/datanode # bin/hadoop fs -ls 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0
Found 27 items
-rw-r--r--   2 root users  22148 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/1510740293106.carbonindexmerge
-rw-r--r--   2 root users   58353052 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-0-0_batchno0-0-1510740300247.carbondata
-rw-r--r--   2 root users   58351680 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-0-0_batchno1-0-1510740300247.carbondata
-rw-r--r--   2 root users   58364823 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-0-1_batchno0-0-1510740300247.carbondata
-rw-r--r--   2 root users   58356303 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-0-2_batchno0-0-1510740300247.carbondata
-rw-r--r--   2 root users   58342246 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-1-0_batchno0-0-1510740300247.carbondata
-rw-r--r--   2 root users   58353186 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-1-0_batchno1-0-1510740300247.carbondata
-rw-r--r--   2 root users   58352964 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-1-1_batchno0-0-1510740300247.carbondata
-rw-r--r--   2 root users   58357183 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-1-2_batchno0-0-1510740300247.carbondata
-rw-r--r--   2 root users   58345739 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-2-0_batchno0-0-1510740300247.carbondata

Yarn job stages:
29  
load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table 
lineitem3 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT')
collect at CommonUtil.scala:858 +details2017/11/15 18:10:51 0.1 s   
1/1 
28  
load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table 
lineitem3 

[jira] [Updated] (CARBONDATA-1719) Carbon1.3.0-Pre-AggregateTable - Empty segment is created when pre-aggr table created in parallel with table load, aggregate query returns no data

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1719:
--
Summary: Carbon1.3.0-Pre-AggregateTable - Empty segment is created when 
pre-aggr table created in parallel with table load, aggregate query returns no 
data  (was: Carbon1.3.0-Pre-AggregateTable - Empty segment is created if 
pre-aggr table created in parallel with table load, aggregate query returns no 
data)

> Carbon1.3.0-Pre-AggregateTable - Empty segment is created when pre-aggr table 
> created in parallel with table load, aggregate query returns no data
> --
>
> Key: CARBONDATA-1719
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1719
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-load
>Affects Versions: 1.3.0
> Environment: Test - 3 node ant cluster
>Reporter: Ramakrishna S
>  Labels: DFX
> Fix For: 1.3.0
>
>
> 1. Create a table
> create table if not exists lineitem3(L_SHIPDATE string,L_SHIPMODE 
> string,L_SHIPINSTRUCT string,L_RETURNFLAG string,L_RECEIPTDATE 
> string,L_ORDERKEY string,L_PARTKEY string,L_SUPPKEY   string,L_LINENUMBER 
> int,L_QUANTITY double,L_EXTENDEDPRICE double,L_DISCOUNT double,L_TAX 
> double,L_LINESTATUS string,L_COMMITDATE string,L_COMMENT  string) STORED BY 
> 'org.apache.carbondata.format' TBLPROPERTIES 
> ('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
> 2. Run load queries and create pre-agg table queries in diff console:
> load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table 
> lineitem3 
> options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');
> create datamap agr_lineitem3 ON TABLE lineitem3 USING 
> "org.apache.carbondata.datamap.AggregateDataMapHandler" as select 
> L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from lineitem3 
> group by  L_RETURNFLAG, L_LINESTATUS;
> 3.  Check table content using aggregate query:
> select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
> lineitem3 group by l_returnflag, l_linestatus;
> 0: jdbc:hive2://10.18.98.34:23040> select 
> l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from lineitem3 
> group by l_returnflag, l_linestatus;
> +---+---+--+---+--+
> | l_returnflag  | l_linestatus  | sum(l_quantity)  | sum(l_extendedprice)  |
> +---+---+--+---+--+
> +---+---+--+---+--+
> No rows selected (1.258 seconds)
> HDFS data:
> BLR114307:/srv/spark2.2Bigdata/install/hadoop/datanode # bin/hadoop fs 
> -ls /carbonstore/default/lineitem3_agr_lineitem3/Fact/Part0/Segment_0
> BLR114307:/srv/spark2.2Bigdata/install/hadoop/datanode # bin/hadoop fs 
> -ls /carbonstore/default/lineitem3/Fact/Part0/Segment_0
> Found 27 items
> -rw-r--r--   2 root users  22148 2017-11-15 18:05 
> /carbonstore/default/lineitem3/Fact/Part0/Segment_0/1510740293106.carbonindexmerge
> -rw-r--r--   2 root users   58353052 2017-11-15 18:05 
> /carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-0-0_batchno0-0-1510740300247.carbondata
> -rw-r--r--   2 root users   58351680 2017-11-15 18:05 
> /carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-0-0_batchno1-0-1510740300247.carbondata
> -rw-r--r--   2 root users   58364823 2017-11-15 18:05 
> /carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-0-1_batchno0-0-1510740300247.carbondata
> -rw-r--r--   2 root users   58356303 2017-11-15 18:05 
> /carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-0-2_batchno0-0-1510740300247.carbondata
> -rw-r--r--   2 root users   58342246 2017-11-15 18:05 
> /carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-1-0_batchno0-0-1510740300247.carbondata
> -rw-r--r--   2 root users   58353186 2017-11-15 18:05 
> /carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-1-0_batchno1-0-1510740300247.carbondata
> -rw-r--r--   2 root users   58352964 2017-11-15 18:05 
> /carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-1-1_batchno0-0-1510740300247.carbondata
> -rw-r--r--   2 root users   58357183 2017-11-15 18:05 
> /carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-1-2_batchno0-0-1510740300247.carbondata
> -rw-r--r--   2 root users   58345739 2017-11-15 18:05 
> 

[jira] [Updated] (CARBONDATA-1719) Carbon1.3.0-Pre-AggregateTable - Empty segment is created if pre-aggr table created in parallel with table load, aggregate query returns no data

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1719:
--
Request participants: Kanaka Kumar Avvaru  (was: )

> Carbon1.3.0-Pre-AggregateTable - Empty segment is created if pre-aggr table 
> created in parallel with table load, aggregate query returns no data
> 
>
> Key: CARBONDATA-1719
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1719
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-load
>Affects Versions: 1.3.0
> Environment: Test - 3 node ant cluster
>Reporter: Ramakrishna S
>  Labels: DFX
> Fix For: 1.3.0
>
>
> 1. Create a table
> create table if not exists lineitem3(L_SHIPDATE string,L_SHIPMODE 
> string,L_SHIPINSTRUCT string,L_RETURNFLAG string,L_RECEIPTDATE 
> string,L_ORDERKEY string,L_PARTKEY string,L_SUPPKEY   string,L_LINENUMBER 
> int,L_QUANTITY double,L_EXTENDEDPRICE double,L_DISCOUNT double,L_TAX 
> double,L_LINESTATUS string,L_COMMITDATE string,L_COMMENT  string) STORED BY 
> 'org.apache.carbondata.format' TBLPROPERTIES 
> ('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
> 2. Run load queries and create pre-agg table queries in diff console:
> load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table 
> lineitem3 
> options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');
> create datamap agr_lineitem3 ON TABLE lineitem3 USING 
> "org.apache.carbondata.datamap.AggregateDataMapHandler" as select 
> L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from lineitem3 
> group by  L_RETURNFLAG, L_LINESTATUS;
> 3.  Check table content using aggregate query:
> select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
> lineitem3 group by l_returnflag, l_linestatus;
> 0: jdbc:hive2://10.18.98.34:23040> select 
> l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from lineitem3 
> group by l_returnflag, l_linestatus;
> +---+---+--+---+--+
> | l_returnflag  | l_linestatus  | sum(l_quantity)  | sum(l_extendedprice)  |
> +---+---+--+---+--+
> +---+---+--+---+--+
> No rows selected (1.258 seconds)
> HDFS data:
> BLR114307:/srv/spark2.2Bigdata/install/hadoop/datanode # bin/hadoop fs 
> -ls /carbonstore/default/lineitem3_agr_lineitem3/Fact/Part0/Segment_0
> BLR114307:/srv/spark2.2Bigdata/install/hadoop/datanode # bin/hadoop fs 
> -ls /carbonstore/default/lineitem3/Fact/Part0/Segment_0
> Found 27 items
> -rw-r--r--   2 root users  22148 2017-11-15 18:05 
> /carbonstore/default/lineitem3/Fact/Part0/Segment_0/1510740293106.carbonindexmerge
> -rw-r--r--   2 root users   58353052 2017-11-15 18:05 
> /carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-0-0_batchno0-0-1510740300247.carbondata
> -rw-r--r--   2 root users   58351680 2017-11-15 18:05 
> /carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-0-0_batchno1-0-1510740300247.carbondata
> -rw-r--r--   2 root users   58364823 2017-11-15 18:05 
> /carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-0-1_batchno0-0-1510740300247.carbondata
> -rw-r--r--   2 root users   58356303 2017-11-15 18:05 
> /carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-0-2_batchno0-0-1510740300247.carbondata
> -rw-r--r--   2 root users   58342246 2017-11-15 18:05 
> /carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-1-0_batchno0-0-1510740300247.carbondata
> -rw-r--r--   2 root users   58353186 2017-11-15 18:05 
> /carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-1-0_batchno1-0-1510740300247.carbondata
> -rw-r--r--   2 root users   58352964 2017-11-15 18:05 
> /carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-1-1_batchno0-0-1510740300247.carbondata
> -rw-r--r--   2 root users   58357183 2017-11-15 18:05 
> /carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-1-2_batchno0-0-1510740300247.carbondata
> -rw-r--r--   2 root users   58345739 2017-11-15 18:05 
> /carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-2-0_batchno0-0-1510740300247.carbondata
> Yarn job stages:
> 29
> load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table 
> lineitem3 
> 

[jira] [Updated] (CARBONDATA-1719) Carbon1.3.0-Pre-AggregateTable - Empty segment is created if pre-aggr table created in parallel with table load, aggregate query returns no data

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1719:
--
Description: 
1. Create a table
create table if not exists lineitem3(L_SHIPDATE string,L_SHIPMODE 
string,L_SHIPINSTRUCT string,L_RETURNFLAG string,L_RECEIPTDATE 
string,L_ORDERKEY string,L_PARTKEY string,L_SUPPKEY   string,L_LINENUMBER 
int,L_QUANTITY double,L_EXTENDEDPRICE double,L_DISCOUNT double,L_TAX 
double,L_LINESTATUS string,L_COMMITDATE string,L_COMMENT  string) STORED BY 
'org.apache.carbondata.format' TBLPROPERTIES 
('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
2. Run load queries and create pre-agg table queries in diff console:
load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table 
lineitem3 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');

create datamap agr_lineitem3 ON TABLE lineitem3 USING 
"org.apache.carbondata.datamap.AggregateDataMapHandler" as select 
L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from lineitem3 
group by  L_RETURNFLAG, L_LINESTATUS;

3.  Check table content using aggregate query:
select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem3 group by l_returnflag, l_linestatus;

0: jdbc:hive2://10.18.98.34:23040> select 
l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from lineitem3 
group by l_returnflag, l_linestatus;
+---+---+--+---+--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  | sum(l_extendedprice)  |
+---+---+--+---+--+
+---+---+--+---+--+
No rows selected (1.258 seconds)


HDFS data:
BLR114307:/srv/spark2.2Bigdata/install/hadoop/datanode # bin/hadoop fs -ls 
/carbonstore/default/lineitem3_agr_lineitem3/Fact/Part0/Segment_0
BLR114307:/srv/spark2.2Bigdata/install/hadoop/datanode # bin/hadoop fs -ls 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0
Found 27 items
-rw-r--r--   2 root users  22148 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/1510740293106.carbonindexmerge
-rw-r--r--   2 root users   58353052 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-0-0_batchno0-0-1510740300247.carbondata
-rw-r--r--   2 root users   58351680 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-0-0_batchno1-0-1510740300247.carbondata
-rw-r--r--   2 root users   58364823 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-0-1_batchno0-0-1510740300247.carbondata
-rw-r--r--   2 root users   58356303 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-0-2_batchno0-0-1510740300247.carbondata
-rw-r--r--   2 root users   58342246 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-1-0_batchno0-0-1510740300247.carbondata
-rw-r--r--   2 root users   58353186 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-1-0_batchno1-0-1510740300247.carbondata
-rw-r--r--   2 root users   58352964 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-1-1_batchno0-0-1510740300247.carbondata
-rw-r--r--   2 root users   58357183 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-1-2_batchno0-0-1510740300247.carbondata
-rw-r--r--   2 root users   58345739 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-2-0_batchno0-0-1510740300247.carbondata

Yarn job stages:
29  
load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table 
lineitem3 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT')
collect at CommonUtil.scala:858 +details2017/11/15 18:10:51 0.1 s   
1/1 
28  
load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table 
lineitem3 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT')
collect at CarbonDataRDDFactory.scala:918 +details  2017/11/15 18:10:50 
1 s 
3/3 10.8 KB 
27  
load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table 
lineitem3 

[jira] [Updated] (CARBONDATA-1719) Carbon1.3.0-Pre-AggregateTable - Empty segment is created if pre-aggr table created in parallel with table load, aggregate query returns no data

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1719:
--
Description: 
1. Create a table
create table if not exists lineitem3(L_SHIPDATE string,L_SHIPMODE 
string,L_SHIPINSTRUCT string,L_RETURNFLAG string,L_RECEIPTDATE 
string,L_ORDERKEY string,L_PARTKEY string,L_SUPPKEY   string,L_LINENUMBER 
int,L_QUANTITY double,L_EXTENDEDPRICE double,L_DISCOUNT double,L_TAX 
double,L_LINESTATUS string,L_COMMITDATE string,L_COMMENT  string) STORED BY 
'org.apache.carbondata.format' TBLPROPERTIES 
('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
2. Run load queries and create pre-agg table queries in diff console:
load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table 
lineitem3 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');

create datamap agr_lineitem3 ON TABLE lineitem3 USING 
"org.apache.carbondata.datamap.AggregateDataMapHandler" as select 
L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from lineitem3 
group by  L_RETURNFLAG, L_LINESTATUS;

3.  Check table content using aggregate query:
select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem3 group by l_returnflag, l_linestatus;

0: jdbc:hive2://10.18.98.34:23040> select 
l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from lineitem3 
group by l_returnflag, l_linestatus;
+---+---+--+---+--+
| l_returnflag  | l_linestatus  | sum(l_quantity)  | sum(l_extendedprice)  |
+---+---+--+---+--+
+---+---+--+---+--+
No rows selected (1.258 seconds)


HDFS data:
BLR114307:/srv/spark2.2Bigdata/install/hadoop/datanode # bin/hadoop fs -ls 
/carbonstore/default/lineitem3_agr_lineitem3/Fact/Part0/Segment_0
17/11/15 18:15:18 WARN util.NativeCodeLoader: Unable to load native-hadoop 
library for your platform... using builtin-java classes where applicable
BLR114307:/srv/spark2.2Bigdata/install/hadoop/datanode # bin/hadoop fs -ls 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0
17/11/15 18:15:34 WARN util.NativeCodeLoader: Unable to load native-hadoop 
library for your platform... using builtin-java classes where applicable
Found 27 items
-rw-r--r--   2 root users  22148 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/1510740293106.carbonindexmerge
-rw-r--r--   2 root users   58353052 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-0-0_batchno0-0-1510740300247.carbondata
-rw-r--r--   2 root users   58351680 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-0-0_batchno1-0-1510740300247.carbondata
-rw-r--r--   2 root users   58364823 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-0-1_batchno0-0-1510740300247.carbondata
-rw-r--r--   2 root users   58356303 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-0-2_batchno0-0-1510740300247.carbondata
-rw-r--r--   2 root users   58342246 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-1-0_batchno0-0-1510740300247.carbondata
-rw-r--r--   2 root users   58353186 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-1-0_batchno1-0-1510740300247.carbondata
-rw-r--r--   2 root users   58352964 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-1-1_batchno0-0-1510740300247.carbondata
-rw-r--r--   2 root users   58357183 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-1-2_batchno0-0-1510740300247.carbondata
-rw-r--r--   2 root users   58345739 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-2-0_batchno0-0-1510740300247.carbondata

Yarn job stages:
29  
load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table 
lineitem3 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT')
collect at CommonUtil.scala:858 +details2017/11/15 18:10:51 0.1 s   
1/1 
28  
load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table 
lineitem3 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT')

[jira] [Created] (CARBONDATA-1719) Carbon1.3.0-Pre-AggregateTable - Empty segment is created if pre-aggr table created in parallel with table load, aggregate query returns no data

[Ramakrishna S (JIRA)]

Ramakrishna S created CARBONDATA-1719:
-

 Summary: Carbon1.3.0-Pre-AggregateTable - Empty segment is created 
if pre-aggr table created in parallel with table load, aggregate query returns 
no data
 Key: CARBONDATA-1719
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1719
 Project: CarbonData
  Issue Type: Bug
  Components: data-load
Affects Versions: 1.3.0
 Environment: Test - 3 node ant cluster
Reporter: Ramakrishna S
 Fix For: 1.3.0


1. Create a table
create table if not exists lineitem3(L_SHIPDATE string,L_SHIPMODE 
string,L_SHIPINSTRUCT string,L_RETURNFLAG string,L_RECEIPTDATE 
string,L_ORDERKEY string,L_PARTKEY string,L_SUPPKEY   string,L_LINENUMBER 
int,L_QUANTITY double,L_EXTENDEDPRICE double,L_DISCOUNT double,L_TAX 
double,L_LINESTATUS string,L_COMMITDATE string,L_COMMENT  string) STORED BY 
'org.apache.carbondata.format' TBLPROPERTIES 
('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
2. Run load queries and create pre-agg table queries in diff console:
load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table 
lineitem3 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');

create datamap agr_lineitem3 ON TABLE lineitem3 USING 
"org.apache.carbondata.datamap.AggregateDataMapHandler" as select 
L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from lineitem3 
group by  L_RETURNFLAG, L_LINESTATUS;

3.  Check table content using aggregate query:
select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem3 group by l_returnflag, l_linestatus;

0: jdbc:hive2://10.18.98.34:23040> show segments for table lineitem2;
++--+--+--++--+
| SegmentSequenceId  |  Status  | Load Start Time  |  Load End Time 
  | Merged To  |
++--+--+--++--+
| 0  | Success  | 2017-11-15 17:56:54.554  | 2017-11-15 
17:57:56.605  ||
++--+--+--++--+
1 row selected (0.179 seconds)
0: jdbc:hive2://10.18.98.34:23040> show segments for table lineitem1;

HDFS data:
BLR114307:/srv/spark2.2Bigdata/install/hadoop/datanode # bin/hadoop fs -ls 
/carbonstore/default/lineitem3_agr_lineitem3/Fact/Part0/Segment_0
17/11/15 18:15:18 WARN util.NativeCodeLoader: Unable to load native-hadoop 
library for your platform... using builtin-java classes where applicable
BLR114307:/srv/spark2.2Bigdata/install/hadoop/datanode # bin/hadoop fs -ls 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0
17/11/15 18:15:34 WARN util.NativeCodeLoader: Unable to load native-hadoop 
library for your platform... using builtin-java classes where applicable
Found 27 items
-rw-r--r--   2 root users  22148 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/1510740293106.carbonindexmerge
-rw-r--r--   2 root users   58353052 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-0-0_batchno0-0-1510740300247.carbondata
-rw-r--r--   2 root users   58351680 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-0-0_batchno1-0-1510740300247.carbondata
-rw-r--r--   2 root users   58364823 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-0-1_batchno0-0-1510740300247.carbondata
-rw-r--r--   2 root users   58356303 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-0-2_batchno0-0-1510740300247.carbondata
-rw-r--r--   2 root users   58342246 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-1-0_batchno0-0-1510740300247.carbondata
-rw-r--r--   2 root users   58353186 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-1-0_batchno1-0-1510740300247.carbondata
-rw-r--r--   2 root users   58352964 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-1-1_batchno0-0-1510740300247.carbondata
-rw-r--r--   2 root users   58357183 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-1-2_batchno0-0-1510740300247.carbondata
-rw-r--r--   2 root users   58345739 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-2-0_batchno0-0-1510740300247.carbondata

Yarn job stages:




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CARBONDATA-1719) Carbon1.3.0-Pre-AggregateTable - Empty segment is created if pre-aggr table created in parallel with table load, aggregate query returns no data

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1719:
--
Description: 
1. Create a table
create table if not exists lineitem3(L_SHIPDATE string,L_SHIPMODE 
string,L_SHIPINSTRUCT string,L_RETURNFLAG string,L_RECEIPTDATE 
string,L_ORDERKEY string,L_PARTKEY string,L_SUPPKEY   string,L_LINENUMBER 
int,L_QUANTITY double,L_EXTENDEDPRICE double,L_DISCOUNT double,L_TAX 
double,L_LINESTATUS string,L_COMMITDATE string,L_COMMENT  string) STORED BY 
'org.apache.carbondata.format' TBLPROPERTIES 
('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
2. Run load queries and create pre-agg table queries in diff console:
load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table 
lineitem3 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');

create datamap agr_lineitem3 ON TABLE lineitem3 USING 
"org.apache.carbondata.datamap.AggregateDataMapHandler" as select 
L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from lineitem3 
group by  L_RETURNFLAG, L_LINESTATUS;

3.  Check table content using aggregate query:
select l_returnflag,l_linestatus,sum(l_quantity),sum(l_extendedprice) from 
lineitem3 group by l_returnflag, l_linestatus;

0: jdbc:hive2://10.18.98.34:23040> show segments for table lineitem2;
++--+--+--++--+
| SegmentSequenceId  |  Status  | Load Start Time  |  Load End Time 
  | Merged To  |
++--+--+--++--+
| 0  | Success  | 2017-11-15 17:56:54.554  | 2017-11-15 
17:57:56.605  ||
++--+--+--++--+
1 row selected (0.179 seconds)
0: jdbc:hive2://10.18.98.34:23040> show segments for table lineitem1;

HDFS data:
BLR114307:/srv/spark2.2Bigdata/install/hadoop/datanode # bin/hadoop fs -ls 
/carbonstore/default/lineitem3_agr_lineitem3/Fact/Part0/Segment_0
17/11/15 18:15:18 WARN util.NativeCodeLoader: Unable to load native-hadoop 
library for your platform... using builtin-java classes where applicable
BLR114307:/srv/spark2.2Bigdata/install/hadoop/datanode # bin/hadoop fs -ls 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0
17/11/15 18:15:34 WARN util.NativeCodeLoader: Unable to load native-hadoop 
library for your platform... using builtin-java classes where applicable
Found 27 items
-rw-r--r--   2 root users  22148 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/1510740293106.carbonindexmerge
-rw-r--r--   2 root users   58353052 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-0-0_batchno0-0-1510740300247.carbondata
-rw-r--r--   2 root users   58351680 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-0-0_batchno1-0-1510740300247.carbondata
-rw-r--r--   2 root users   58364823 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-0-1_batchno0-0-1510740300247.carbondata
-rw-r--r--   2 root users   58356303 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-0-2_batchno0-0-1510740300247.carbondata
-rw-r--r--   2 root users   58342246 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-1-0_batchno0-0-1510740300247.carbondata
-rw-r--r--   2 root users   58353186 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-1-0_batchno1-0-1510740300247.carbondata
-rw-r--r--   2 root users   58352964 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-1-1_batchno0-0-1510740300247.carbondata
-rw-r--r--   2 root users   58357183 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-1-2_batchno0-0-1510740300247.carbondata
-rw-r--r--   2 root users   58345739 2017-11-15 18:05 
/carbonstore/default/lineitem3/Fact/Part0/Segment_0/part-2-0_batchno0-0-1510740300247.carbondata

Yarn job stages:
29  
load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table 
lineitem3 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT')
collect at CommonUtil.scala:858 +details2017/11/15 18:10:51 0.1 s   
1/1 
28  
load data inpath "hdfs://hacluster/user/test/lineitem.tbl.1" into table 
lineitem3 

[jira] [Commented] (CARBONDATA-1713) Carbon1.3.0-Pre-AggregateTable - Aggregate query on main table fails after creating pre-aggregate table

[Ramakrishna S (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253213#comment-16253213
 ] 

Ramakrishna S commented on CARBONDATA-1713:
---

Changing severity based on the clarification given.

> Carbon1.3.0-Pre-AggregateTable - Aggregate query on main table fails after 
> creating pre-aggregate table
> ---
>
> Key: CARBONDATA-1713
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1713
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-query
>Affects Versions: 1.3.0
> Environment: ANT Test cluster - 3 node
>Reporter: Ramakrishna S
>Assignee: kumar vishal
>Priority: Minor
>  Labels: sanity
> Fix For: 1.3.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> 0: jdbc:hive2://10.18.98.34:23040> load data inpath 
> "hdfs://hacluster/user/test/lineitem.tbl.1" into table lineitem 
> options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');
> Error: org.apache.spark.sql.catalyst.analysis.NoSuchTableException: Table or 
> view 'lineitem' not found in database 'default'; (state=,code=0)
> 0: jdbc:hive2://10.18.98.34:23040> create table if not exists lineitem(
> 0: jdbc:hive2://10.18.98.34:23040> L_SHIPDATE string,
> 0: jdbc:hive2://10.18.98.34:23040> L_SHIPMODE string,
> 0: jdbc:hive2://10.18.98.34:23040> L_SHIPINSTRUCT string,
> 0: jdbc:hive2://10.18.98.34:23040> L_RETURNFLAG string,
> 0: jdbc:hive2://10.18.98.34:23040> L_RECEIPTDATE string,
> 0: jdbc:hive2://10.18.98.34:23040> L_ORDERKEY string,
> 0: jdbc:hive2://10.18.98.34:23040> L_PARTKEY string,
> 0: jdbc:hive2://10.18.98.34:23040> L_SUPPKEY   string,
> 0: jdbc:hive2://10.18.98.34:23040> L_LINENUMBER int,
> 0: jdbc:hive2://10.18.98.34:23040> L_QUANTITY double,
> 0: jdbc:hive2://10.18.98.34:23040> L_EXTENDEDPRICE double,
> 0: jdbc:hive2://10.18.98.34:23040> L_DISCOUNT double,
> 0: jdbc:hive2://10.18.98.34:23040> L_TAX double,
> 0: jdbc:hive2://10.18.98.34:23040> L_LINESTATUS string,
> 0: jdbc:hive2://10.18.98.34:23040> L_COMMITDATE string,
> 0: jdbc:hive2://10.18.98.34:23040> L_COMMENT  string
> 0: jdbc:hive2://10.18.98.34:23040> ) STORED BY 'org.apache.carbondata.format'
> 0: jdbc:hive2://10.18.98.34:23040> TBLPROPERTIES 
> ('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (0.338 seconds)
> 0: jdbc:hive2://10.18.98.34:23040> load data inpath 
> "hdfs://hacluster/user/test/lineitem.tbl.1" into table lineitem 
> options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (48.634 seconds)
> 0: jdbc:hive2://10.18.98.34:23040> create datamap agr_lineitem ON TABLE 
> lineitem USING "org.apache.carbondata.datamap.AggregateDataMapHandler" as 
> select L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from 
> lineitem group by  L_RETURNFLAG, L_LINESTATUS;
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (16.552 seconds)
> 0: jdbc:hive2://10.18.98.34:23040> select 
> L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from lineitem 
> group by  L_RETURNFLAG, L_LINESTATUS;
> Error: org.apache.spark.sql.AnalysisException: Column doesnot exists in Pre 
> Aggregate table; (state=,code=0)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CARBONDATA-1713) Carbon1.3.0-Pre-AggregateTable - Aggregate query on main table fails after creating pre-aggregate table

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1713:
--
Priority: Minor  (was: Major)

> Carbon1.3.0-Pre-AggregateTable - Aggregate query on main table fails after 
> creating pre-aggregate table
> ---
>
> Key: CARBONDATA-1713
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1713
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-query
>Affects Versions: 1.3.0
> Environment: ANT Test cluster - 3 node
>Reporter: Ramakrishna S
>Assignee: kumar vishal
>Priority: Minor
>  Labels: sanity
> Fix For: 1.3.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> 0: jdbc:hive2://10.18.98.34:23040> load data inpath 
> "hdfs://hacluster/user/test/lineitem.tbl.1" into table lineitem 
> options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');
> Error: org.apache.spark.sql.catalyst.analysis.NoSuchTableException: Table or 
> view 'lineitem' not found in database 'default'; (state=,code=0)
> 0: jdbc:hive2://10.18.98.34:23040> create table if not exists lineitem(
> 0: jdbc:hive2://10.18.98.34:23040> L_SHIPDATE string,
> 0: jdbc:hive2://10.18.98.34:23040> L_SHIPMODE string,
> 0: jdbc:hive2://10.18.98.34:23040> L_SHIPINSTRUCT string,
> 0: jdbc:hive2://10.18.98.34:23040> L_RETURNFLAG string,
> 0: jdbc:hive2://10.18.98.34:23040> L_RECEIPTDATE string,
> 0: jdbc:hive2://10.18.98.34:23040> L_ORDERKEY string,
> 0: jdbc:hive2://10.18.98.34:23040> L_PARTKEY string,
> 0: jdbc:hive2://10.18.98.34:23040> L_SUPPKEY   string,
> 0: jdbc:hive2://10.18.98.34:23040> L_LINENUMBER int,
> 0: jdbc:hive2://10.18.98.34:23040> L_QUANTITY double,
> 0: jdbc:hive2://10.18.98.34:23040> L_EXTENDEDPRICE double,
> 0: jdbc:hive2://10.18.98.34:23040> L_DISCOUNT double,
> 0: jdbc:hive2://10.18.98.34:23040> L_TAX double,
> 0: jdbc:hive2://10.18.98.34:23040> L_LINESTATUS string,
> 0: jdbc:hive2://10.18.98.34:23040> L_COMMITDATE string,
> 0: jdbc:hive2://10.18.98.34:23040> L_COMMENT  string
> 0: jdbc:hive2://10.18.98.34:23040> ) STORED BY 'org.apache.carbondata.format'
> 0: jdbc:hive2://10.18.98.34:23040> TBLPROPERTIES 
> ('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (0.338 seconds)
> 0: jdbc:hive2://10.18.98.34:23040> load data inpath 
> "hdfs://hacluster/user/test/lineitem.tbl.1" into table lineitem 
> options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (48.634 seconds)
> 0: jdbc:hive2://10.18.98.34:23040> create datamap agr_lineitem ON TABLE 
> lineitem USING "org.apache.carbondata.datamap.AggregateDataMapHandler" as 
> select L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from 
> lineitem group by  L_RETURNFLAG, L_LINESTATUS;
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (16.552 seconds)
> 0: jdbc:hive2://10.18.98.34:23040> select 
> L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from lineitem 
> group by  L_RETURNFLAG, L_LINESTATUS;
> Error: org.apache.spark.sql.AnalysisException: Column doesnot exists in Pre 
> Aggregate table; (state=,code=0)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CARBONDATA-1713) Carbon1.3.0-Pre-AggregateTable - Aggregate query on main table fails after creating pre-aggregate table

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1713:
--
Priority: Major  (was: Blocker)

> Carbon1.3.0-Pre-AggregateTable - Aggregate query on main table fails after 
> creating pre-aggregate table
> ---
>
> Key: CARBONDATA-1713
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1713
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-query
>Affects Versions: 1.3.0
> Environment: ANT Test cluster - 3 node
>Reporter: Ramakrishna S
>Assignee: kumar vishal
>  Labels: sanity
> Fix For: 1.3.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> 0: jdbc:hive2://10.18.98.34:23040> load data inpath 
> "hdfs://hacluster/user/test/lineitem.tbl.1" into table lineitem 
> options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');
> Error: org.apache.spark.sql.catalyst.analysis.NoSuchTableException: Table or 
> view 'lineitem' not found in database 'default'; (state=,code=0)
> 0: jdbc:hive2://10.18.98.34:23040> create table if not exists lineitem(
> 0: jdbc:hive2://10.18.98.34:23040> L_SHIPDATE string,
> 0: jdbc:hive2://10.18.98.34:23040> L_SHIPMODE string,
> 0: jdbc:hive2://10.18.98.34:23040> L_SHIPINSTRUCT string,
> 0: jdbc:hive2://10.18.98.34:23040> L_RETURNFLAG string,
> 0: jdbc:hive2://10.18.98.34:23040> L_RECEIPTDATE string,
> 0: jdbc:hive2://10.18.98.34:23040> L_ORDERKEY string,
> 0: jdbc:hive2://10.18.98.34:23040> L_PARTKEY string,
> 0: jdbc:hive2://10.18.98.34:23040> L_SUPPKEY   string,
> 0: jdbc:hive2://10.18.98.34:23040> L_LINENUMBER int,
> 0: jdbc:hive2://10.18.98.34:23040> L_QUANTITY double,
> 0: jdbc:hive2://10.18.98.34:23040> L_EXTENDEDPRICE double,
> 0: jdbc:hive2://10.18.98.34:23040> L_DISCOUNT double,
> 0: jdbc:hive2://10.18.98.34:23040> L_TAX double,
> 0: jdbc:hive2://10.18.98.34:23040> L_LINESTATUS string,
> 0: jdbc:hive2://10.18.98.34:23040> L_COMMITDATE string,
> 0: jdbc:hive2://10.18.98.34:23040> L_COMMENT  string
> 0: jdbc:hive2://10.18.98.34:23040> ) STORED BY 'org.apache.carbondata.format'
> 0: jdbc:hive2://10.18.98.34:23040> TBLPROPERTIES 
> ('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (0.338 seconds)
> 0: jdbc:hive2://10.18.98.34:23040> load data inpath 
> "hdfs://hacluster/user/test/lineitem.tbl.1" into table lineitem 
> options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (48.634 seconds)
> 0: jdbc:hive2://10.18.98.34:23040> create datamap agr_lineitem ON TABLE 
> lineitem USING "org.apache.carbondata.datamap.AggregateDataMapHandler" as 
> select L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from 
> lineitem group by  L_RETURNFLAG, L_LINESTATUS;
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (16.552 seconds)
> 0: jdbc:hive2://10.18.98.34:23040> select 
> L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from lineitem 
> group by  L_RETURNFLAG, L_LINESTATUS;
> Error: org.apache.spark.sql.AnalysisException: Column doesnot exists in Pre 
> Aggregate table; (state=,code=0)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CARBONDATA-1713) Carbon1.3.0-Pre-AggregateTable - Aggregate query on main table fails after creating pre-aggregate table

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1713:
--
   Priority: Blocker  (was: Major)
Description: 
0: jdbc:hive2://10.18.98.34:23040> load data inpath 
"hdfs://hacluster/user/test/lineitem.tbl.1" into table lineitem 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');
Error: org.apache.spark.sql.catalyst.analysis.NoSuchTableException: Table or 
view 'lineitem' not found in database 'default'; (state=,code=0)
0: jdbc:hive2://10.18.98.34:23040> create table if not exists lineitem(
0: jdbc:hive2://10.18.98.34:23040> L_SHIPDATE string,
0: jdbc:hive2://10.18.98.34:23040> L_SHIPMODE string,
0: jdbc:hive2://10.18.98.34:23040> L_SHIPINSTRUCT string,
0: jdbc:hive2://10.18.98.34:23040> L_RETURNFLAG string,
0: jdbc:hive2://10.18.98.34:23040> L_RECEIPTDATE string,
0: jdbc:hive2://10.18.98.34:23040> L_ORDERKEY string,
0: jdbc:hive2://10.18.98.34:23040> L_PARTKEY string,
0: jdbc:hive2://10.18.98.34:23040> L_SUPPKEY   string,
0: jdbc:hive2://10.18.98.34:23040> L_LINENUMBER int,
0: jdbc:hive2://10.18.98.34:23040> L_QUANTITY double,
0: jdbc:hive2://10.18.98.34:23040> L_EXTENDEDPRICE double,
0: jdbc:hive2://10.18.98.34:23040> L_DISCOUNT double,
0: jdbc:hive2://10.18.98.34:23040> L_TAX double,
0: jdbc:hive2://10.18.98.34:23040> L_LINESTATUS string,
0: jdbc:hive2://10.18.98.34:23040> L_COMMITDATE string,
0: jdbc:hive2://10.18.98.34:23040> L_COMMENT  string
0: jdbc:hive2://10.18.98.34:23040> ) STORED BY 'org.apache.carbondata.format'
0: jdbc:hive2://10.18.98.34:23040> TBLPROPERTIES 
('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
+-+--+
| Result  |
+-+--+
+-+--+
No rows selected (0.338 seconds)
0: jdbc:hive2://10.18.98.34:23040> load data inpath 
"hdfs://hacluster/user/test/lineitem.tbl.1" into table lineitem 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');
+-+--+
| Result  |
+-+--+
+-+--+
No rows selected (48.634 seconds)
0: jdbc:hive2://10.18.98.34:23040> create datamap agr_lineitem ON TABLE 
lineitem USING "org.apache.carbondata.datamap.AggregateDataMapHandler" as 
select L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from 
lineitem group by  L_RETURNFLAG, L_LINESTATUS;
+-+--+
| Result  |
+-+--+
+-+--+
No rows selected (16.552 seconds)
0: jdbc:hive2://10.18.98.34:23040> select 
L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from lineitem 
group by  L_RETURNFLAG, L_LINESTATUS;
Error: org.apache.spark.sql.AnalysisException: Column doesnot exists in Pre 
Aggregate table; (state=,code=0)


  was:

0: jdbc:hive2://10.18.98.34:23040> load data inpath 
"hdfs://hacluster/user/test/lineitem.tbl.1" into table lineitem 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');
Error: org.apache.spark.sql.catalyst.analysis.NoSuchTableException: Table or 
view 'lineitem' not found in database 'default'; (state=,code=0)
0: jdbc:hive2://10.18.98.34:23040> create table if not exists lineitem(
0: jdbc:hive2://10.18.98.34:23040> L_SHIPDATE string,
0: jdbc:hive2://10.18.98.34:23040> L_SHIPMODE string,
0: jdbc:hive2://10.18.98.34:23040> L_SHIPINSTRUCT string,
0: jdbc:hive2://10.18.98.34:23040> L_RETURNFLAG string,
0: jdbc:hive2://10.18.98.34:23040> L_RECEIPTDATE string,
0: jdbc:hive2://10.18.98.34:23040> L_ORDERKEY string,
0: jdbc:hive2://10.18.98.34:23040> L_PARTKEY string,
0: jdbc:hive2://10.18.98.34:23040> L_SUPPKEY   string,
0: jdbc:hive2://10.18.98.34:23040> L_LINENUMBER int,
0: jdbc:hive2://10.18.98.34:23040> L_QUANTITY double,
0: jdbc:hive2://10.18.98.34:23040> L_EXTENDEDPRICE double,
0: jdbc:hive2://10.18.98.34:23040> L_DISCOUNT double,
0: jdbc:hive2://10.18.98.34:23040> L_TAX double,
0: jdbc:hive2://10.18.98.34:23040> L_LINESTATUS string,
0: jdbc:hive2://10.18.98.34:23040> L_COMMITDATE string,
0: jdbc:hive2://10.18.98.34:23040> L_COMMENT  string
0: jdbc:hive2://10.18.98.34:23040> ) STORED BY 'org.apache.carbondata.format'
0: jdbc:hive2://10.18.98.34:23040> TBLPROPERTIES 
('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
+-+--+
| Result  |
+-+--+
+-+--+
No rows selected (0.338 seconds)
0: 

[jira] [Updated] (CARBONDATA-1713) Carbon1.3.0-Pre-AggregateTable - Aggregate query on main table fails after creating pre-aggregate table

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1713:
--
Request participants: Kanaka Kumar Avvaru  (was: )

> Carbon1.3.0-Pre-AggregateTable - Aggregate query on main table fails after 
> creating pre-aggregate table
> ---
>
> Key: CARBONDATA-1713
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1713
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-query
>Affects Versions: 1.3.0
> Environment: ANT Test cluster - 3 node
>Reporter: Ramakrishna S
>  Labels: sanity
> Fix For: 1.3.0
>
>
> 0: jdbc:hive2://10.18.98.34:23040> load data inpath 
> "hdfs://hacluster/user/test/lineitem.tbl.1" into table lineitem 
> options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');
> Error: org.apache.spark.sql.catalyst.analysis.NoSuchTableException: Table or 
> view 'lineitem' not found in database 'default'; (state=,code=0)
> 0: jdbc:hive2://10.18.98.34:23040> create table if not exists lineitem(
> 0: jdbc:hive2://10.18.98.34:23040> L_SHIPDATE string,
> 0: jdbc:hive2://10.18.98.34:23040> L_SHIPMODE string,
> 0: jdbc:hive2://10.18.98.34:23040> L_SHIPINSTRUCT string,
> 0: jdbc:hive2://10.18.98.34:23040> L_RETURNFLAG string,
> 0: jdbc:hive2://10.18.98.34:23040> L_RECEIPTDATE string,
> 0: jdbc:hive2://10.18.98.34:23040> L_ORDERKEY string,
> 0: jdbc:hive2://10.18.98.34:23040> L_PARTKEY string,
> 0: jdbc:hive2://10.18.98.34:23040> L_SUPPKEY   string,
> 0: jdbc:hive2://10.18.98.34:23040> L_LINENUMBER int,
> 0: jdbc:hive2://10.18.98.34:23040> L_QUANTITY double,
> 0: jdbc:hive2://10.18.98.34:23040> L_EXTENDEDPRICE double,
> 0: jdbc:hive2://10.18.98.34:23040> L_DISCOUNT double,
> 0: jdbc:hive2://10.18.98.34:23040> L_TAX double,
> 0: jdbc:hive2://10.18.98.34:23040> L_LINESTATUS string,
> 0: jdbc:hive2://10.18.98.34:23040> L_COMMITDATE string,
> 0: jdbc:hive2://10.18.98.34:23040> L_COMMENT  string
> 0: jdbc:hive2://10.18.98.34:23040> ) STORED BY 'org.apache.carbondata.format'
> 0: jdbc:hive2://10.18.98.34:23040> TBLPROPERTIES 
> ('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (0.338 seconds)
> 0: jdbc:hive2://10.18.98.34:23040> load data inpath 
> "hdfs://hacluster/user/test/lineitem.tbl.1" into table lineitem 
> options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (48.634 seconds)
> 0: jdbc:hive2://10.18.98.34:23040> create datamap agr_lineitem ON TABLE 
> lineitem USING "org.apache.carbondata.datamap.AggregateDataMapHandler" as 
> select L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from 
> lineitem group by  L_RETURNFLAG, L_LINESTATUS;
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (16.552 seconds)
> 0: jdbc:hive2://10.18.98.34:23040> select 
> L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from lineitem 
> group by  L_RETURNFLAG, L_LINESTATUS;
> Error: org.apache.spark.sql.AnalysisException: Column doesnot exists in Pre 
> Aggregate table; (state=,code=0)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (CARBONDATA-1713) Carbon1.3.0-Pre-AggregateTable - Aggregate query on main table fails after creating pre-aggregate table

[Ramakrishna S (JIRA)]

Ramakrishna S created CARBONDATA-1713:
-

 Summary: Carbon1.3.0-Pre-AggregateTable - Aggregate query on main 
table fails after creating pre-aggregate table
 Key: CARBONDATA-1713
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1713
 Project: CarbonData
  Issue Type: Bug
  Components: data-query
Affects Versions: 1.3.0
 Environment: ANT Test cluster - 3 node
Reporter: Ramakrishna S
 Fix For: 1.3.0



0: jdbc:hive2://10.18.98.34:23040> load data inpath 
"hdfs://hacluster/user/test/lineitem.tbl.1" into table lineitem 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');
Error: org.apache.spark.sql.catalyst.analysis.NoSuchTableException: Table or 
view 'lineitem' not found in database 'default'; (state=,code=0)
0: jdbc:hive2://10.18.98.34:23040> create table if not exists lineitem(
0: jdbc:hive2://10.18.98.34:23040> L_SHIPDATE string,
0: jdbc:hive2://10.18.98.34:23040> L_SHIPMODE string,
0: jdbc:hive2://10.18.98.34:23040> L_SHIPINSTRUCT string,
0: jdbc:hive2://10.18.98.34:23040> L_RETURNFLAG string,
0: jdbc:hive2://10.18.98.34:23040> L_RECEIPTDATE string,
0: jdbc:hive2://10.18.98.34:23040> L_ORDERKEY string,
0: jdbc:hive2://10.18.98.34:23040> L_PARTKEY string,
0: jdbc:hive2://10.18.98.34:23040> L_SUPPKEY   string,
0: jdbc:hive2://10.18.98.34:23040> L_LINENUMBER int,
0: jdbc:hive2://10.18.98.34:23040> L_QUANTITY double,
0: jdbc:hive2://10.18.98.34:23040> L_EXTENDEDPRICE double,
0: jdbc:hive2://10.18.98.34:23040> L_DISCOUNT double,
0: jdbc:hive2://10.18.98.34:23040> L_TAX double,
0: jdbc:hive2://10.18.98.34:23040> L_LINESTATUS string,
0: jdbc:hive2://10.18.98.34:23040> L_COMMITDATE string,
0: jdbc:hive2://10.18.98.34:23040> L_COMMENT  string
0: jdbc:hive2://10.18.98.34:23040> ) STORED BY 'org.apache.carbondata.format'
0: jdbc:hive2://10.18.98.34:23040> TBLPROPERTIES 
('table_blocksize'='128','NO_INVERTED_INDEX'='L_SHIPDATE,L_SHIPMODE,L_SHIPINSTRUCT,L_RETURNFLAG,L_RECEIPTDATE,L_ORDERKEY,L_PARTKEY,L_SUPPKEY','sort_columns'='');
+-+--+
| Result  |
+-+--+
+-+--+
No rows selected (0.338 seconds)
0: jdbc:hive2://10.18.98.34:23040> load data inpath 
"hdfs://hacluster/user/test/lineitem.tbl.1" into table lineitem 
options('DELIMITER'='|','FILEHEADER'='L_ORDERKEY,L_PARTKEY,L_SUPPKEY,L_LINENUMBER,L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT,L_TAX,L_RETURNFLAG,L_LINESTATUS,L_SHIPDATE,L_COMMITDATE,L_RECEIPTDATE,L_SHIPINSTRUCT,L_SHIPMODE,L_COMMENT');
+-+--+
| Result  |
+-+--+
+-+--+
No rows selected (48.634 seconds)
0: jdbc:hive2://10.18.98.34:23040> create datamap agr_lineitem ON TABLE 
lineitem USING "org.apache.carbondata.datamap.AggregateDataMapHandler" as 
select L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from 
lineitem group by  L_RETURNFLAG, L_LINESTATUS;
+-+--+
| Result  |
+-+--+
+-+--+
No rows selected (16.552 seconds)
0: jdbc:hive2://10.18.98.34:23040> select 
L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from lineitem 
group by  L_RETURNFLAG, L_LINESTATUS;
Error: org.apache.spark.sql.AnalysisException: Column doesnot exists in Pre 
Aggregate table; (state=,code=0)




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CARBONDATA-1711) Carbon1.3.0-Pre-AggregateTable - Show datamap on table does not work

[Ramakrishna S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramakrishna S updated CARBONDATA-1711:
--
Request participants: Kanaka Kumar Avvaru, Chetan Bhat  (was: )

> Carbon1.3.0-Pre-AggregateTable - Show datamap  on table  does not 
> work
> -
>
> Key: CARBONDATA-1711
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1711
> Project: CarbonData
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.3.0
> Environment: Test
>Reporter: Ramakrishna S
>Priority: Minor
> Fix For: 1.3.0
>
>
> 0: jdbc:hive2://10.18.98.34:23040> create datamap agr_lineitem ON TABLE 
> lineitem USING "org.apache.carbondata.datamap.AggregateDataMapHandler" as 
> select L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from 
> lineitem group by  L_RETURNFLAG, L_LINESTATUS;
> Error: java.lang.RuntimeException: Table [lineitem_agr_lineitem] already 
> exists under database [default] (state=,code=0)
> 0: jdbc:hive2://10.18.98.34:23040> show tables;
> +---+---+--+--+
> | database  | tableName | isTemporary  |
> +---+---+--+--+
> | default   | flow_carbon_test4 | false|
> | default   | jl_r3 | false|
> | default   | lineitem  | false|
> | default   | lineitem_agr_lineitem | false|
> | default   | sensor_reading_blockblank_false   | false|
> | default   | sensor_reading_blockblank_false1  | false|
> | default   | sensor_reading_blockblank_false2  | false|
> | default   | sensor_reading_false  | false|
> | default   | sensor_reading_true   | false|
> | default   | t1| false|
> | default   | t1_agg_t1 | false|
> | default   | tc4   | false|
> | default   | uniqdata  | false|
> +---+---+--+--+
> 13 rows selected (0.04 seconds)
> 0: jdbc:hive2://10.18.98.34:23040> show datamap on table lineitem;
> Error: java.lang.RuntimeException:
> BaseSqlParser
> missing 'FUNCTIONS' at 'on'(line 1, pos 13)
> == SQL ==
> show datamap on table lineitem
> -^^^
> CarbonSqlParser [1.6] failure: identifier matching regex (?i)SEGMENTS 
> expected
> show datamap on table lineitem



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (CARBONDATA-1711) Carbon1.3.0-Pre-AggregateTable - Show datamap on table does not work

[Ramakrishna S (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252943#comment-16252943
 ] 

Ramakrishna S edited comment on CARBONDATA-1711 at 11/15/17 4:59 AM:
-

Similar issue found in these table operations as well:
As per doc:
l User can drop the pre-aggregate table by
DROP DATAMAP agg_sales
l User can show pre-aggregate table by
SHOW DATAMAP ON TABLE sale
l User can get the information of the pre-aggregate table by
DESC DATAMAP agg_sales ON TABLE sales
But they dont seem to work:
0: jdbc:hive2://10.18.98.34:23040> desc datamap agr_lineitem on table lineitems;
Error: java.lang.RuntimeException:
BaseSqlParser
mismatched input 'on' expecting {, '.'}(line 1, pos 26)

== SQL ==
desc datamap agr_lineitem on table lineitems
--^^^

CarbonSqlParser [1.1] failure: identifier matching regex (?i)CREATE expected

desc datamap agr_lineitem on table lineitems
^ (state=,code=0)
0: jdbc:hive2://10.18.98.34:23040> SHOW DATAMAP ON TABLE lineitem;
Error: java.lang.RuntimeException:
BaseSqlParser
missing 'FUNCTIONS' at 'ON'(line 1, pos 13)

== SQL ==
SHOW DATAMAP ON TABLE lineitem
-^^^

CarbonSqlParser [1.6] failure: identifier matching regex (?i)SEGMENTS 
expected

SHOW DATAMAP ON TABLE lineitem
 ^ (state=,code=0)
0: jdbc:hive2://10.18.98.34:23040> desc datamap agr_lineitem on table lineitem;
Error: java.lang.RuntimeException:
BaseSqlParser
mismatched input 'on' expecting {, '.'}(line 1, pos 26)

== SQL ==
desc datamap agr_lineitem on table lineitem
--^^^

CarbonSqlParser [1.1] failure: identifier matching regex (?i)CREATE expected

desc datamap agr_lineitem on table lineitem
^ (state=,code=0)
0: jdbc:hive2://10.18.98.34:23040> DROP DATAMAP agr_lineitem;
Error: java.lang.RuntimeException:
BaseSqlParser
no viable alternative at input 'DROP DATAMAP'(line 1, pos 5)

== SQL ==
DROP DATAMAP agr_lineitem
-^^^

CarbonSqlParser [1.1] failure: identifier matching regex (?i)CREATE expected

DROP DATAMAP agr_lineitem
^ (state=,code=0)



was (Author: ram@huawei):
Similar issue found in these table operations as well:
As per doc:

> Carbon1.3.0-Pre-AggregateTable - Show datamap  on table  does not 
> work
> -
>
> Key: CARBONDATA-1711
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1711
> Project: CarbonData
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.3.0
> Environment: Test
>Reporter: Ramakrishna S
>Priority: Minor
> Fix For: 1.3.0
>
>
> 0: jdbc:hive2://10.18.98.34:23040> create datamap agr_lineitem ON TABLE 
> lineitem USING "org.apache.carbondata.datamap.AggregateDataMapHandler" as 
> select L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from 
> lineitem group by  L_RETURNFLAG, L_LINESTATUS;
> Error: java.lang.RuntimeException: Table [lineitem_agr_lineitem] already 
> exists under database [default] (state=,code=0)
> 0: jdbc:hive2://10.18.98.34:23040> show tables;
> +---+---+--+--+
> | database  | tableName | isTemporary  |
> +---+---+--+--+
> | default   | flow_carbon_test4 | false|
> | default   | jl_r3 | false|
> | default   | lineitem  | false|
> | default   | lineitem_agr_lineitem | false|
> | default   | sensor_reading_blockblank_false   | false|
> | default   | sensor_reading_blockblank_false1  | false|
> | default   | sensor_reading_blockblank_false2  | false|
> | default   | sensor_reading_false  | false|
> | default   | sensor_reading_true   | false|
> | default   | t1| false|
> | default   | t1_agg_t1 | false|
> | default   | tc4   | false|
> | default   | uniqdata  | false|
> +---+---+--+--+
> 13 rows selected (0.04 seconds)
> 0: jdbc:hive2://10.18.98.34:23040> show datamap on table lineitem;
> Error: java.lang.RuntimeException:
> BaseSqlParser
> missing 'FUNCTIONS' at 'on'(line 1, pos 13)
> == SQL ==
> show datamap on table lineitem
> -^^^
> CarbonSqlParser [1.6] failure: identifier matching regex (?i)SEGMENTS 
> expected
> show datamap on table lineitem



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (CARBONDATA-1711) Carbon1.3.0-Pre-AggregateTable - Show datamap on table does not work

[Ramakrishna S (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252943#comment-16252943
 ] 

Ramakrishna S edited comment on CARBONDATA-1711 at 11/15/17 4:58 AM:
-

Similar issue found in these table operations as well:
As per doc:


was (Author: ram@huawei):
Similar issue found in other table operations as well:

> Carbon1.3.0-Pre-AggregateTable - Show datamap  on table  does not 
> work
> -
>
> Key: CARBONDATA-1711
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1711
> Project: CarbonData
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.3.0
> Environment: Test
>Reporter: Ramakrishna S
>Priority: Minor
> Fix For: 1.3.0
>
>
> 0: jdbc:hive2://10.18.98.34:23040> create datamap agr_lineitem ON TABLE 
> lineitem USING "org.apache.carbondata.datamap.AggregateDataMapHandler" as 
> select L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from 
> lineitem group by  L_RETURNFLAG, L_LINESTATUS;
> Error: java.lang.RuntimeException: Table [lineitem_agr_lineitem] already 
> exists under database [default] (state=,code=0)
> 0: jdbc:hive2://10.18.98.34:23040> show tables;
> +---+---+--+--+
> | database  | tableName | isTemporary  |
> +---+---+--+--+
> | default   | flow_carbon_test4 | false|
> | default   | jl_r3 | false|
> | default   | lineitem  | false|
> | default   | lineitem_agr_lineitem | false|
> | default   | sensor_reading_blockblank_false   | false|
> | default   | sensor_reading_blockblank_false1  | false|
> | default   | sensor_reading_blockblank_false2  | false|
> | default   | sensor_reading_false  | false|
> | default   | sensor_reading_true   | false|
> | default   | t1| false|
> | default   | t1_agg_t1 | false|
> | default   | tc4   | false|
> | default   | uniqdata  | false|
> +---+---+--+--+
> 13 rows selected (0.04 seconds)
> 0: jdbc:hive2://10.18.98.34:23040> show datamap on table lineitem;
> Error: java.lang.RuntimeException:
> BaseSqlParser
> missing 'FUNCTIONS' at 'on'(line 1, pos 13)
> == SQL ==
> show datamap on table lineitem
> -^^^
> CarbonSqlParser [1.6] failure: identifier matching regex (?i)SEGMENTS 
> expected
> show datamap on table lineitem



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (CARBONDATA-1711) Carbon1.3.0-Pre-AggregateTable - Show datamap on table does not work

[Ramakrishna S (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252943#comment-16252943
 ] 

Ramakrishna S commented on CARBONDATA-1711:
---

Similar issue found in other table operations as well:

> Carbon1.3.0-Pre-AggregateTable - Show datamap  on table  does not 
> work
> -
>
> Key: CARBONDATA-1711
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1711
> Project: CarbonData
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.3.0
> Environment: Test
>Reporter: Ramakrishna S
>Priority: Minor
> Fix For: 1.3.0
>
>
> 0: jdbc:hive2://10.18.98.34:23040> create datamap agr_lineitem ON TABLE 
> lineitem USING "org.apache.carbondata.datamap.AggregateDataMapHandler" as 
> select L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from 
> lineitem group by  L_RETURNFLAG, L_LINESTATUS;
> Error: java.lang.RuntimeException: Table [lineitem_agr_lineitem] already 
> exists under database [default] (state=,code=0)
> 0: jdbc:hive2://10.18.98.34:23040> show tables;
> +---+---+--+--+
> | database  | tableName | isTemporary  |
> +---+---+--+--+
> | default   | flow_carbon_test4 | false|
> | default   | jl_r3 | false|
> | default   | lineitem  | false|
> | default   | lineitem_agr_lineitem | false|
> | default   | sensor_reading_blockblank_false   | false|
> | default   | sensor_reading_blockblank_false1  | false|
> | default   | sensor_reading_blockblank_false2  | false|
> | default   | sensor_reading_false  | false|
> | default   | sensor_reading_true   | false|
> | default   | t1| false|
> | default   | t1_agg_t1 | false|
> | default   | tc4   | false|
> | default   | uniqdata  | false|
> +---+---+--+--+
> 13 rows selected (0.04 seconds)
> 0: jdbc:hive2://10.18.98.34:23040> show datamap on table lineitem;
> Error: java.lang.RuntimeException:
> BaseSqlParser
> missing 'FUNCTIONS' at 'on'(line 1, pos 13)
> == SQL ==
> show datamap on table lineitem
> -^^^
> CarbonSqlParser [1.6] failure: identifier matching regex (?i)SEGMENTS 
> expected
> show datamap on table lineitem



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (CARBONDATA-1711) Carbon1.3.0-Pre-AggregateTable - Show datamap on table does not work

[Ramakrishna S (JIRA)]

Ramakrishna S created CARBONDATA-1711:
-

 Summary: Carbon1.3.0-Pre-AggregateTable - Show datamap  on table 
 does not work
 Key: CARBONDATA-1711
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1711
 Project: CarbonData
  Issue Type: Bug
  Components: core
Affects Versions: 1.3.0
 Environment: Test
Reporter: Ramakrishna S
Priority: Minor
 Fix For: 1.3.0




0: jdbc:hive2://10.18.98.34:23040> create datamap agr_lineitem ON TABLE 
lineitem USING "org.apache.carbondata.datamap.AggregateDataMapHandler" as 
select L_RETURNFLAG,L_LINESTATUS,sum(L_QUANTITY),sum(L_EXTENDEDPRICE) from 
lineitem group by  L_RETURNFLAG, L_LINESTATUS;
Error: java.lang.RuntimeException: Table [lineitem_agr_lineitem] already exists 
under database [default] (state=,code=0)
0: jdbc:hive2://10.18.98.34:23040> show tables;
+---+---+--+--+
| database  | tableName | isTemporary  |
+---+---+--+--+
| default   | flow_carbon_test4 | false|
| default   | jl_r3 | false|
| default   | lineitem  | false|
| default   | lineitem_agr_lineitem | false|
| default   | sensor_reading_blockblank_false   | false|
| default   | sensor_reading_blockblank_false1  | false|
| default   | sensor_reading_blockblank_false2  | false|
| default   | sensor_reading_false  | false|
| default   | sensor_reading_true   | false|
| default   | t1| false|
| default   | t1_agg_t1 | false|
| default   | tc4   | false|
| default   | uniqdata  | false|
+---+---+--+--+
13 rows selected (0.04 seconds)
0: jdbc:hive2://10.18.98.34:23040> show datamap on table lineitem;
Error: java.lang.RuntimeException:
BaseSqlParser
missing 'FUNCTIONS' at 'on'(line 1, pos 13)

== SQL ==
show datamap on table lineitem
-^^^

CarbonSqlParser [1.6] failure: identifier matching regex (?i)SEGMENTS 
expected

show datamap on table lineitem




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)