[GitHub] carbondata issue #1939: [CARBONDATA-2139] Optimize CTAS documentation and te...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1939
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3390/



---

[GitHub] carbondata issue #1939: [CARBONDATA-2139] Optimize CTAS documentation and te...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1939
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3549/



---

[GitHub] carbondata issue #1940: [CARBONDATA-2140 ] Refactoring code to improve perfo...

[bhavya411]

Github user bhavya411 commented on the issue:

https://github.com/apache/carbondata/pull/1940
  
retest sdv please.


---

[GitHub] carbondata issue #1939: [CARBONDATA-2139] Optimize CTAS documentation and te...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1939
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2312/



---

[GitHub] carbondata issue #1904: [CARBONDATA-2059] - Changes to support compaction fo...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1904
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3548/



---

[GitHub] carbondata issue #1904: [CARBONDATA-2059] - Changes to support compaction fo...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1904
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2311/



---

[GitHub] carbondata issue #1940: [CARBONDATA-2140 ] Refactoring code to improve perfo...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1940
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3389/



---

[GitHub] carbondata issue #1938: [CARBONDATA-2138] Added documentation for HEADER opt...

[sgururajshetty]

Github user sgururajshetty commented on the issue:

https://github.com/apache/carbondata/pull/1938
  
@QiangCai fixed the review comment. Kindly review and merge.


---

[GitHub] carbondata issue #1904: [CARBONDATA-2059] - Changes to support compaction fo...

[ashwini-krishnakumar]

Github user ashwini-krishnakumar commented on the issue:

https://github.com/apache/carbondata/pull/1904
  
retest this please


---

[GitHub] carbondata issue #1940: [CARBONDATA-2140 ] Refactoring code to improve perfo...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1940
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2310/



---

[jira] [Updated] (CARBONDATA-1516) Support pre-aggregate tables and timeseries in carbondata

[Liang Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liang Chen updated CARBONDATA-1516:
---
Attachment: CarbonData Pre-aggregation Table_v1.3.pdf

> Support pre-aggregate tables and timeseries in carbondata
> -
>
> Key: CARBONDATA-1516
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1516
> Project: CarbonData
>  Issue Type: New Feature
>Reporter: Ravindra Pesala
>Priority: Major
> Attachments: CarbonData Pre-aggregation Table.pdf, CarbonData 
> Pre-aggregation Table_v1.1.pdf, CarbonData Pre-aggregation Table_v1.2.pdf, 
> CarbonData Pre-aggregation Table_v1.3.pdf
>
>
> Currently Carbondata has standard SQL capability on distributed data 
> sets.Carbondata should support pre-aggregating tables for timeseries and 
> improve query performance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata issue #1940: [CARBONDATA-2140 ] Refactoring code to improve perfo...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1940
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3547/



---

[GitHub] carbondata issue #1937: [Carbondata 2137] Delete query performance improved

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1937
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3388/



---

[GitHub] carbondata issue #1937: [Carbondata 2137] Delete query performance improved

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1937
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2309/



---

[GitHub] carbondata issue #1937: [Carbondata 2137] Delete query performance improved

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1937
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3546/



---

[jira] [Closed] (CARBONDATA-1685) Upgrade Presto Version in CarbonData to 0.186

[Bhavya Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhavya Aggarwal closed CARBONDATA-1685.
---
Resolution: Invalid

Accidentally created

> Upgrade Presto Version in CarbonData to 0.186
> -
>
> Key: CARBONDATA-1685
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1685
> Project: CarbonData
>  Issue Type: Improvement
>  Components: presto-integration
>Affects Versions: 1.3.0
>Reporter: Bhavya Aggarwal
>Assignee: Bhavya Aggarwal
>Priority: Major
>
> Upgrade the Presto Version in carbondata to 0.186



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (CARBONDATA-1683) Upgrade Presto Version in CarbonData to 0.186

[Bhavya Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhavya Aggarwal closed CARBONDATA-1683.
---
Resolution: Fixed

Accidentally created

> Upgrade Presto Version in CarbonData to 0.186
> -
>
> Key: CARBONDATA-1683
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1683
> Project: CarbonData
>  Issue Type: Improvement
>  Components: presto-integration
>Affects Versions: 1.3.0
>Reporter: Bhavya Aggarwal
>Assignee: Bhavya Aggarwal
>Priority: Major
>
> Upgrade the Presto Version in carbondata to 0.186



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (CARBONDATA-1684) Upgrade Presto Version in CarbonData to 0.186

[Bhavya Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhavya Aggarwal closed CARBONDATA-1684.
---
Resolution: Fixed

> Upgrade Presto Version in CarbonData to 0.186
> -
>
> Key: CARBONDATA-1684
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1684
> Project: CarbonData
>  Issue Type: Improvement
>  Components: presto-integration
>Affects Versions: 1.3.0
>Reporter: Bhavya Aggarwal
>Assignee: Bhavya Aggarwal
>Priority: Major
>
> Upgrade the Presto Version in carbondata to 0.186



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (CARBONDATA-1759) (Carbon1.3.0 - Clean Files) Clean command is not working correctly for segments marked for delete due to insert overwrite job

[Ajeet Rai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ajeet Rai closed CARBONDATA-1759.
-

Issue is fixed.   Clean command is  working correctly for segments marked for 
delete due to insert overwrite job.

> (Carbon1.3.0 - Clean Files) Clean command is not working correctly for  
> segments marked for delete due to insert overwrite job
> --
>
> Key: CARBONDATA-1759
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1759
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-query
>Affects Versions: 1.3.0
> Environment: 3 Node ant cluster
>Reporter: Ajeet Rai
>Assignee: Akash R Nilugal
>Priority: Major
>  Labels: dfx
> Fix For: 1.3.0
>
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> Carbon1.3.0  Clean command is not working correctly for  segments marked for 
> delete due to insert overwrite job.
> 1: Create a table
> CREATE TABLE IF NOT EXISTS flow_carbon_new999(txn_dte String,dt String,txn_bk 
> String,txn_br String,own_bk String,own_br String,opp_bk String,bus_opr_cde 
> String,opt_prd_cde String,cus_no String,cus_ac String,opp_ac_nme  
> String,opp_ac String,bv_no  String,aco_ac String,ac_dte String,txn_cnt 
> int,jrn_par int,mfm_jrn_no String,cbn_jrn_no String,ibs_jrn_no String,vch_no 
> String,vch_seq String,srv_cde String,bus_cd_no  String,id_flg String,bv_cde 
> String,txn_time  String,txn_tlr String,ety_tlr String,ety_bk String,ety_br 
> String,bus_pss_no String,chk_flg String,chk_tlr String,chk_jrn_no String,  
> bus_sys_no String,txn_sub_cde String,fin_bus_cde String,fin_bus_sub_cde 
> String,chl  String,tml_id String,sus_no String,sus_seq String,  cho_seq 
> String,  itm_itm String,itm_sub String,itm_sss String,dc_flg String,amt  
> decimal(15,2),bal  decimal(15,2),ccy  String,spv_flg String,vch_vld_dte 
> String,pst_bk String,pst_br String,ec_flg String,aco_tlr String,gen_flg 
> String,his_rec_sum_flg String,his_flg String,vch_typ String,val_dte 
> String,opp_ac_flg String,cmb_flg String,ass_vch_flg String,cus_pps_flg 
> String,bus_rmk_cde String,vch_bus_rmk String,tec_rmk_cde String,vch_tec_rmk 
> String,gems_last_upd_d String,maps_date String,maps_job String)STORED BY 
> 'org.apache.carbondata.format' 
> TBLPROPERTIES('DICTIONARY_INCLUDE'='txn_cnt,jrn_par,amt,bal','No_Inverted_Index'=
>  'txn_dte,dt,txn_bk,txn_br,own_bk ,own_br ,opp_bk ,bus_opr_cde ,opt_prd_cde 
> ,cus_no ,cus_ac ,opp_ac_nme  ,opp_ac ,bv_no  ,aco_ac ,ac_dte ,txn_cnt  
> ,jrn_par  ,mfm_jrn_no ,cbn_jrn_no ,ibs_jrn_no ,vch_no ,vch_seq ,srv_cde 
> ,bus_cd_no  ,id_flg ,bv_cde ,txn_time  ,txn_tlr ,ety_tlr ,ety_bk ,ety_br 
> ,bus_pss_no ,chk_flg ,chk_tlr ,chk_jrn_no , bus_sys_no ,txn_sub_cde 
> ,fin_bus_cde ,fin_bus_sub_cde ,chl  ,tml_id ,sus_no ,sus_seq , cho_seq , 
> itm_itm ,itm_sub ,itm_sss ,dc_flg ,amt,bal,ccy  ,spv_flg ,vch_vld_dte ,pst_bk 
> ,pst_br ,ec_flg ,aco_tlr ,gen_flg ,his_rec_sum_flg ,his_flg ,vch_typ ,val_dte 
> ,opp_ac_flg ,cmb_flg ,ass_vch_flg ,cus_pps_flg ,bus_rmk_cde ,vch_bus_rmk 
> ,tec_rmk_cde ,vch_tec_rmk ,gems_last_upd_d ,maps_date ,maps_job' );
> 2: start a data load.
> LOAD DATA inpath 'hdfs://hacluster/user/test/20140101_1_1.csv' into 
> table flow_carbon_new999 options('DELIMITER'=',', 
> 'QUOTECHAR'='"','header'='false');
> 3: run a insert overwrite job 
> insert into table  flow_carbon_new999 select * from flow_carbon_new666;
> 4: run show segment query:
> show segments for table ajeet.flow_carbon_new999
> 5: Observe that all previous segments are marked for delete
> 6: run clean query
> CLEAN FILES FOR TABLE ajeet.flow_carbon_new999;
> 7: again run show segment query
> 8: Observe that still all previous segments which are marked for delete are 
> shown as result.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata pull request #1940: [CARBONDATA-2140 ] Refactoring code to improv...

[bhavya411]

GitHub user bhavya411 opened a pull request:

https://github.com/apache/carbondata/pull/1940

[CARBONDATA-2140 ] Refactoring code to improve performance and removing 
unnecessary code in Presto Integration

This PR is for optimizing Presto performance and refactoring the code to 
remove unnecessary classes and making it simpler.

Be sure to do all of the following checklist to help us incorporate 
your contribution quickly and easily:

 - [X] Any interfaces changed?  No
 
 - [X] Any backward compatibility impacted?   No 
 
 - [X] Documentation Remains Same

 - [X] Testing done
 - No new test cases needed as the functionality remain same
 - All existing test cases are passing.
 - Ran the TPCH queries to check the performance .
   
 - [X] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bhavya411/incubator-carbondata CARBONDATA-2140

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1940.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1940


commit 533c1d2db0d4608081ed65bb98ea3bbe94e189ea
Author: Bhavya 
Date:   2018-02-06T13:27:42Z

Reafactored Code to remove Cursor and RecordSetProvider




---

[jira] [Created] (CARBONDATA-2140) Presto Integration - Code Refactoring

[Bhavya Aggarwal (JIRA)]

Bhavya Aggarwal created CARBONDATA-2140:
---

 Summary: Presto Integration - Code Refactoring
 Key: CARBONDATA-2140
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2140
 Project: CarbonData
  Issue Type: Improvement
  Components: presto-integration
Reporter: Bhavya Aggarwal
Assignee: Bhavya Aggarwal


Presto Integration - Code Refactoring to remove unnecessary class and improve 
the performance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata issue #1939: [CARBONDATA-2139] Optimize CTAS documentation and te...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1939
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3387/



---

[GitHub] carbondata issue #1939: [CARBONDATA-2139] Optimize CTAS documentation and te...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1939
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2308/



---

[GitHub] carbondata issue #1939: [CARBONDATA-2139] Optimize CTAS documentation and te...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1939
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3545/



---

[GitHub] carbondata pull request #1939: [CARBONDATA-2139] Optimize CTAS documentation...

[xubo245]

GitHub user xubo245 opened a pull request:

https://github.com/apache/carbondata/pull/1939

[CARBONDATA-2139] Optimize CTAS documentation and test case

Optimize CTAS:

- optimize documentation
- add test case
- drop table after finishing run test acse, remove the file of table from 
disk

Be sure to do all of the following checklist to help us incorporate 
your contribution quickly and easily:

 - [ ] Any interfaces changed?
 
 - [ ] Any backward compatibility impacted?
 
 - [ ] Document update required?

 - [ ] Testing done
Please provide details on 
- Whether new unit test cases have been added or why no new tests 
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance 
test report.
- Any additional information to help reviewers in testing this 
change.
   
 - [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xubo245/carbondata 
CARBONDATA-3129_CTASOptimize

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1939.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1939


commit 28f61381a0a5bc030838fbed9a1a04348a73f546
Author: xubo245 <601450868@...>
Date:   2018-02-07T04:16:41Z

[CARBONDATA-2139] Optimize CTAS documentation and test case




---

[jira] [Created] (CARBONDATA-2139) Optimize CTAS documentation and test case

[xubo245 (JIRA)]

xubo245 created CARBONDATA-2139:
---

 Summary: Optimize CTAS documentation and test case
 Key: CARBONDATA-2139
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2139
 Project: CarbonData
  Issue Type: Improvement
  Components: docs, test
Affects Versions: 1.3.0
Reporter: xubo245
Assignee: xubo245
 Fix For: 1.3.0


Optimize CTAS:
* optimize documentation 
* add test case
* drop table  after finishing run test acse, remove the file of table from disk



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata pull request #1938: [CARBONDATA-2138] Added documentation for HEA...

[QiangCai]

Github user QiangCai commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1938#discussion_r166509548
  
--- Diff: docs/data-management-on-carbondata.md ---
@@ -330,6 +330,14 @@ This tutorial is going to introduce all commands and 
data operations on CarbonDa
 OPTIONS('COMMENTCHAR'='#')
 ```
 
+  - **HEADER:** When you load the CSV file without the file header and the 
file header is the same with the table schema, then add 'HEADER'='false' to 
load data SQL as user need not provide the file header. By default the value is 
'true'.
+  false: CSV file is without file header.
+  true: CSV file is with file header.
+  
+```
+OPTIONS('HEADER'='false') 
+```
+
   - **FILEHEADER:** Headers can be provided in the LOAD DATA command if 
headers are missing in the source files.
 
--- End diff --

suggest mentioning: If HEADER option exists and the value is true, not 
require FILEHEADER option.


---

[GitHub] carbondata pull request #1938: [CARBONDATA-2138] Added documentation for HEA...

[QiangCai]

Github user QiangCai commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1938#discussion_r166505141
  
--- Diff: docs/data-management-on-carbondata.md ---
@@ -407,6 +415,7 @@ This tutorial is going to introduce all commands and 
data operations on CarbonDa
projectjoindate,projectenddate,attendance,utilization,salary',
'MULTILINE'='true','ESCAPECHAR'='\','COMPLEX_DELIMITER_LEVEL_1'='$',
'COMPLEX_DELIMITER_LEVEL_2'=':',
+   'HEADER'='false',
--- End diff --

suggest moving this line to 413 line.


---

[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1856
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3386/



---

[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1856
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2307/



---

[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1856
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3544/



---

[GitHub] carbondata issue #1857: [WIP][CARBONDATA-2073][CARBONDATA-1516][Tests] Add t...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1857
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3385/



---

[GitHub] carbondata issue #1857: [WIP][CARBONDATA-2073][CARBONDATA-1516][Tests] Add t...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1857
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2306/



---

[GitHub] carbondata issue #1857: [WIP][CARBONDATA-2073][CARBONDATA-1516][Tests] Add t...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1857
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3543/



---

[GitHub] carbondata issue #1937: [Carbondata 2137] Delete query performance improved

[sraghunandan]

Github user sraghunandan commented on the issue:

https://github.com/apache/carbondata/pull/1937
  
@rahulforallp pls add the performance test report


---

[GitHub] carbondata issue #1772: [CARBONDATA-1995] Unify all writer steps and make te...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1772
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3542/



---

[GitHub] carbondata issue #1772: [CARBONDATA-1995] Unify all writer steps and make te...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1772
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2305/



---

[GitHub] carbondata issue #1772: [CARBONDATA-1995] Unify all writer steps and make te...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1772
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3384/



---

[GitHub] carbondata issue #1857: [WIP][CARBONDATA-2073][CARBONDATA-1516][Tests] Add t...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1857
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3383/



---

[GitHub] carbondata issue #1938: [CARBONDATA-2138] Added documentation for HEADER opt...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1938
  
Can one of the admins verify this patch?


---

[GitHub] carbondata issue #1938: [CARBONDATA-2138] Added documentation for HEADER opt...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1938
  
Can one of the admins verify this patch?


---

[GitHub] carbondata issue #1938: [CARBONDATA-2138] Added documentation for HEADER opt...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1938
  
Can one of the admins verify this patch?


---

[GitHub] carbondata pull request #1938: [CARBONDATA-2138] Added documentation for HEA...

[sgururajshetty]

GitHub user sgururajshetty opened a pull request:

https://github.com/apache/carbondata/pull/1938

[CARBONDATA-2138] Added documentation for HEADER option while loading data

Added documentation for HEADER option in load data

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sgururajshetty/carbondata 2138

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1938.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1938


commit 20ec4b85d4bf41d1ee99402a8e59aa4ad5d1f08f
Author: sgururajshetty 
Date:   2018-02-06T15:28:59Z

Added documentation for HEADER option while loading data




---

[jira] [Created] (CARBONDATA-2138) Documentation for HEADER option

[Gururaj Shetty (JIRA)]

Gururaj Shetty created CARBONDATA-2138:
--

 Summary: Documentation for HEADER option
 Key: CARBONDATA-2138
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2138
 Project: CarbonData
  Issue Type: Task
Reporter: Gururaj Shetty
Assignee: Gururaj Shetty


Add documentation for HEADER option as per the discussion in the below mailing 
list.

http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/Discussion-Add-HEADER-option-to-load-data-sql-td17080.html#a17138



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata issue #1937: [Carbondata 2137] Delete query performance improved

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1937
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3382/



---

[GitHub] carbondata issue #1937: [Carbondata 2137] Delete query performance improved

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1937
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3381/



---

[GitHub] carbondata issue #1857: [WIP][CARBONDATA-2073][CARBONDATA-1516][Tests] Add t...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1857
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2304/



---

[GitHub] carbondata issue #1857: [WIP][CARBONDATA-2073][CARBONDATA-1516][Tests] Add t...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1857
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3541/



---

[GitHub] carbondata issue #1937: [Carbondata 2137] Delete query performance improved

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1937
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3380/



---

[GitHub] carbondata issue #1937: [Carbondata 2137] Delete query performance improved

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1937
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3540/



---

[GitHub] carbondata issue #1937: [Carbondata 2137] Delete query performance improved

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1937
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2303/



---

[GitHub] carbondata issue #1937: [Carbondata 2137] Delete query performance improved

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1937
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3539/



---

[jira] [Closed] (CARBONDATA-1761) (Carbon1.3.0 - DELETE SEGMENT BY ID) In Progress Segment is marked for delete if respective id is given in delete segment by id query

[Ajeet Rai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ajeet Rai closed CARBONDATA-1761.
-

Issue is fixed. A segment is not deleted if load is still in progress for that 
segment id.

> (Carbon1.3.0 - DELETE SEGMENT BY ID) In Progress Segment is marked for delete 
> if respective id is given in delete segment by id query
> -
>
> Key: CARBONDATA-1761
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1761
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-load
>Affects Versions: 1.3.0
> Environment: 3 Node ant cluster
> Description
>Reporter: Ajeet Rai
>Assignee: Akash R Nilugal
>Priority: Major
>  Labels: dfx
> Fix For: 1.3.0
>
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> (Carbon1.3.0 - DELETE SEGMENT BY ID) In Progress Segment is marked for delete 
> if respective id is given in delete segment by id query.
> 1: Create a table
> CREATE TABLE IF NOT EXISTS flow_carbon_new999(txn_dte String,dt String,txn_bk 
> String,txn_br String,own_bk String,own_br String,opp_bk String,bus_opr_cde 
> String,opt_prd_cde String,cus_no String,cus_ac String,opp_ac_nme 
> String,opp_ac String,bv_no String,aco_ac String,ac_dte String,txn_cnt 
> int,jrn_par int,mfm_jrn_no String,cbn_jrn_no String,ibs_jrn_no String,vch_no 
> String,vch_seq String,srv_cde String,bus_cd_no String,id_flg String,bv_cde 
> String,txn_time String,txn_tlr String,ety_tlr String,ety_bk String,ety_br 
> String,bus_pss_no String,chk_flg String,chk_tlr String,chk_jrn_no String, 
> bus_sys_no String,txn_sub_cde String,fin_bus_cde String,fin_bus_sub_cde 
> String,chl String,tml_id String,sus_no String,sus_seq String, cho_seq String, 
> itm_itm String,itm_sub String,itm_sss String,dc_flg String,amt 
> decimal(15,2),bal decimal(15,2),ccy String,spv_flg String,vch_vld_dte 
> String,pst_bk String,pst_br String,ec_flg String,aco_tlr String,gen_flg 
> String,his_rec_sum_flg String,his_flg String,vch_typ String,val_dte 
> String,opp_ac_flg String,cmb_flg String,ass_vch_flg String,cus_pps_flg 
> String,bus_rmk_cde String,vch_bus_rmk String,tec_rmk_cde String,vch_tec_rmk 
> String,gems_last_upd_d String,maps_date String,maps_job String)STORED BY 
> 'org.apache.carbondata.format' 
> TBLPROPERTIES('DICTIONARY_INCLUDE'='txn_cnt,jrn_par,amt,bal','No_Inverted_Index'=
>  'txn_dte,dt,txn_bk,txn_br,own_bk ,own_br ,opp_bk ,bus_opr_cde ,opt_prd_cde 
> ,cus_no ,cus_ac ,opp_ac_nme ,opp_ac ,bv_no ,aco_ac ,ac_dte ,txn_cnt ,jrn_par 
> ,mfm_jrn_no ,cbn_jrn_no ,ibs_jrn_no ,vch_no ,vch_seq ,srv_cde ,bus_cd_no 
> ,id_flg ,bv_cde ,txn_time ,txn_tlr ,ety_tlr ,ety_bk ,ety_br ,bus_pss_no 
> ,chk_flg ,chk_tlr ,chk_jrn_no , bus_sys_no ,txn_sub_cde ,fin_bus_cde 
> ,fin_bus_sub_cde ,chl ,tml_id ,sus_no ,sus_seq , cho_seq , itm_itm ,itm_sub 
> ,itm_sss ,dc_flg ,amt,bal,ccy ,spv_flg ,vch_vld_dte ,pst_bk ,pst_br ,ec_flg 
> ,aco_tlr ,gen_flg ,his_rec_sum_flg ,his_flg ,vch_typ ,val_dte ,opp_ac_flg 
> ,cmb_flg ,ass_vch_flg ,cus_pps_flg ,bus_rmk_cde ,vch_bus_rmk ,tec_rmk_cde 
> ,vch_tec_rmk ,gems_last_upd_d ,maps_date ,maps_job' );
> 2: start a data load.
> LOAD DATA inpath 'hdfs://hacluster/user/test/20140101_1_1.csv' into 
> table flow_carbon_new999 options('DELIMITER'=',', 
> 'QUOTECHAR'='"','header'='false');
> 3: run a insert into/overwrite job
> insert into table flow_carbon_new999 select * from flow_carbon_new666;
> 4: show segments for table flow_carbon_new999;
> 5: Observe that load/insert/overwrite job is started with new segment id
> 6: now run a delete segment by id query with this id.
> DELETE FROM TABLE ajeet.flow_carbon_new999 WHERE SEGMENT.ID IN (34)
> 7: again run show segment and see this segment which is still in progress is 
> marked for delete.
> 8: Observe that insert/load job is still running and after some time(in next 
> job of load/insert/overwrite), this job fails with below error:
> Error: java.lang.RuntimeException: It seems insert overwrite has been issued 
> during load (state=,code=0)
> This is not correct behaviour and it should be handled.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata pull request #1937: [Carbondata 2137] Delete query performance im...

[rahulforallp]

GitHub user rahulforallp opened a pull request:

https://github.com/apache/carbondata/pull/1937

[Carbondata 2137] Delete query performance improved

Be sure to do all of the following checklist to help us incorporate 
your contribution quickly and easily:

 - [X] Any interfaces changed? No
 
 - [X] Any backward compatibility impacted? No
 
 - [X] Document update required? No

 - [X] Testing done
   Manual testing done
 - [X] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA.  NA



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rahulforallp/incubator-carbondata 
CARBONDATA-2137

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1937.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1937


commit 74881471e018cd5a3ff099ea3fd90456ef7c716b
Author: dhatchayani 
Date:   2018-02-02T05:25:19Z

[CARBONDATA-1410] Thread leak issue in case of data loading failure

commit 9ddd5fe5f9796f0d871ae272fdd2fccdf6a8b130
Author: rahulforallp 
Date:   2018-02-06T12:31:34Z

[CARBONDATA-2137] Delete query performance improved




---

[jira] [Closed] (CARBONDATA-1789) Carbon1.3.0 Concurrent Load-Drop: user is able to drop table even if insert/load job is running

[Ajeet Rai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ajeet Rai closed CARBONDATA-1789.
-

Issue is fixed.When user tries to drop a table while load is still in progress, 
a error message is shown to user:

Error: org.apache.carbondata.spark.exception.ConcurrentOperationException: 
loading is in progress for table db1.t1, drop table operation is not allowed 
(state=,code=0)                                                                 
                                                                            

> Carbon1.3.0 Concurrent Load-Drop: user is able to drop table even if 
> insert/load job is running
> ---
>
> Key: CARBONDATA-1789
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1789
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-load
> Environment: 3 Node ant cluster
>Reporter: Ajeet Rai
>Assignee: Akash R Nilugal
>Priority: Major
>  Labels: dfx
> Fix For: 1.3.0
>
>  Time Spent: 6h 50m
>  Remaining Estimate: 0h
>
> Carbon1.3.0 Concurrent Load-Drop: user is able to drop table even if 
> insert/load job is running
> Steps:
> 1:  Create a table
> 2: Start a insert job
> 3: Concurrently drop the table
> 4: Observe that drop is success
> 5: Observe that insert job is running and after some times job fails
> Expected behvaiour: drop job should wait for insert job to complete



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (CARBONDATA-1802) Carbon1.3.0 Alter:Alter query fails if a column is dropped and there is no key column

[Ajeet Rai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ajeet Rai closed CARBONDATA-1802.
-

Issue is fixed.Alter query is success after dropping the column in a table 
where there is no key column.

> Carbon1.3.0  Alter:Alter query fails if a column is dropped and there is no 
> key column
> --
>
> Key: CARBONDATA-1802
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1802
> Project: CarbonData
>  Issue Type: Bug
>Affects Versions: 1.3.0
> Environment:   3 Node ant cluster
>Reporter: Ajeet Rai
>Assignee: dhatchayani
>Priority: Major
>  Labels: functional
> Fix For: 1.3.0
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Carbon1.3.0  Alter:Alter query fails if a column is dropped and there is no 
> key column.
> Steps:
> 1: create table ttt(c int,d int,e int) stored by 'carbondata';
> 2: Alter table ttt drop columns(c);
> 3: observe that below error is coming:
> Error: java.lang.RuntimeException: Alter table drop column operation failed: 
> Alter drop operation failed. AtLeast one key column should exist after drop.
> Expected: Since user is able to create a table with all numeric columns, Same 
> should be supported in Alter feature.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (CARBONDATA-2137) Delete query is taking more time while processing the carbondata.

[Rahul Kumar (JIRA)]

Rahul Kumar created CARBONDATA-2137:
---

 Summary: Delete query is taking more time while processing the 
carbondata.
 Key: CARBONDATA-2137
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2137
 Project: CarbonData
  Issue Type: Improvement
Reporter: Rahul Kumar
Assignee: Rahul Kumar


*Expected Output* : Delete query should take less time

*Actual Output* : Delete Query is taking 20min

*Following the steps to reproduce :* 
 * create table and load 500 million records
 * create hive table with 10% of data
 * delete the data in main-table using hive table
 * check the performance

*Following is the configuration used :*
 * SPARK_EXECUTOR_MEMORY : 200G
 * SPARK_DRIVER_MEMORY : 20G
 * SPARK_EXECUTOR_CORES : 32
 * SPARK_EXECUTOR_INSTANCEs : 3



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1856
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3379/



---

[jira] [Created] (CARBONDATA-2136) Exception displays while loading data with BAD_RECORDS_ACTION = REDIRECT

[Vandana Yadav (JIRA)]

Vandana Yadav created CARBONDATA-2136:
-

 Summary: Exception displays while loading data with 
BAD_RECORDS_ACTION = REDIRECT
 Key: CARBONDATA-2136
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2136
 Project: CarbonData
  Issue Type: Bug
  Components: data-load
Affects Versions: 1.3.0
 Environment: spark 2.1
Reporter: Vandana Yadav
 Attachments: 2000_UniqData.csv

Exception displays while loading data with BAD_RECORDS_ACTION = REDIRECT

Steps to reproduce:

1) create the table:

CREATE TABLE uniqdata(CUST_ID int,CUST_NAME String,ACTIVE_EMUI_VERSION string, 
DOB timestamp, DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 
bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2 
decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,INTEGER_COLUMN1 
int) STORED BY 'org.apache.carbondata.format' 
TBLPROPERTIES('DICTIONARY_INCLUDE'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1',"TABLE_BLOCKSIZE"=
 "256 
MB",'SORT_SCOPE'='NO_SORT','NO_INVERTED_INDEX'='CUST_ID,CUST_NAME,Double_COLUMN1,DECIMAL_COLUMN2');

2) Load Data:

LOAD DATA INPATH 'hdfs://localhost:54310/Data/uniqdata/2000_UniqData.csv' into 
table uniqdata OPTIONS('DELIMITER'=',', 
'QUOTECHAR'='"','BAD_RECORDS_ACTION'='REDIRECT','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');

Expected Result: data should be loaded successfully.

Actual Result:

Error: java.lang.Exception: DataLoad failure: There is an unexpected error: 
unable to generate the mdkey (state=,code=0)

 

3) ThriftServer logs: 

18/02/06 16:38:11 INFO SparkExecuteStatementOperation: Running query 'LOAD DATA 
INPATH 'hdfs://localhost:54310/Data/uniqdata/2000_UniqData.csv' into table 
uniqdata OPTIONS('DELIMITER'=',', 
'QUOTECHAR'='"','BAD_RECORDS_ACTION'='REDIRECT','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1')'
 with 87eb4af5-e485-4a0b-bcae-6589f1252291
18/02/06 16:38:11 INFO CarbonSparkSqlParser: Parsing command: LOAD DATA INPATH 
'hdfs://localhost:54310/Data/uniqdata/2000_UniqData.csv' into table uniqdata 
OPTIONS('DELIMITER'=',', 
'QUOTECHAR'='"','BAD_RECORDS_ACTION'='REDIRECT','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1')
18/02/06 16:38:11 INFO CarbonLateDecodeRule: pool-23-thread-41 skip 
CarbonOptimizer
18/02/06 16:38:11 INFO CarbonLateDecodeRule: pool-23-thread-41 Skip 
CarbonOptimizer
18/02/06 16:38:11 INFO HiveMetaStore: 42: get_table : db=bug tbl=uniqdata
18/02/06 16:38:11 INFO audit: ugi=hduser ip=unknown-ip-addr cmd=get_table : 
db=bug tbl=uniqdata 
18/02/06 16:38:11 INFO HiveMetaStore: 42: Opening raw store with implemenation 
class:org.apache.hadoop.hive.metastore.ObjectStore
18/02/06 16:38:11 INFO ObjectStore: ObjectStore, initialize called
18/02/06 16:38:11 INFO Query: Reading in results for query 
"org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is 
closing
18/02/06 16:38:11 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is 
DERBY
18/02/06 16:38:11 INFO ObjectStore: Initialized ObjectStore
18/02/06 16:38:11 INFO CatalystSqlParser: Parsing command: array
18/02/06 16:38:11 INFO CarbonLoadDataCommand: pool-23-thread-41 Deleting stale 
folders if present for table bug.uniqdata
18/02/06 16:38:11 INFO CarbonLoadDataCommand: pool-23-thread-41 Initiating 
Direct Load for the Table : (bug.uniqdata)
18/02/06 16:38:12 INFO HdfsFileLock: pool-23-thread-41 HDFS lock 
path:hdfs://localhost:54310/opt/prestocarbonStore/bug/uniqdata/tablestatus.lock
18/02/06 16:38:12 INFO HdfsFileLock: pool-23-thread-41 HDFS lock 
path:hdfs://localhost:54310/opt/prestocarbonStore/bug/uniqdata/Segment_1.lock
18/02/06 16:38:12 INFO DeleteLoadFolders: pool-23-thread-41 Info: Deleted the 
load 1
18/02/06 16:38:12 INFO DeleteLoadFolders: pool-23-thread-41 Info: Segment lock 
on segment:1 is released
18/02/06 16:38:12 INFO DataLoadingUtil$: pool-23-thread-41 Table status lock 
has been successfully acquired.
18/02/06 16:38:12 INFO HdfsFileLock: pool-23-thread-41 Deleted the lock file 
hdfs://localhost:54310/opt/prestocarbonStore/bug/uniqdata/tablestatus.lock
18/02/06 16:38:12 INFO CarbonLockUtil: pool-23-thread-41 Table status lock has 
been successfully released
18/02/06 16:38:12 WARN DeleteLoadFolders: pool-23-thread-41 Files are not found 
in segment 
hdfs://localhost:54310/opt/prestocarbonStore/bug/uniqdata/Fact/Part0/Segment_0 
it seems, files are already being deleted
18/02/06 16:38:12 WARN DeleteLoadFolders: pool-23-thread-41 Files are not found 

[GitHub] carbondata issue #1857: [WIP][CARBONDATA-2073][CARBONDATA-1516][Tests] Add t...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1857
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3378/



---

[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1856
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3537/



---

[GitHub] carbondata issue #1936: [CARBONDATA-2135] Documentation for Table comment an...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1936
  
Can one of the admins verify this patch?


---

[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1856
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2301/



---

[GitHub] carbondata issue #1936: [CARBONDATA-2135] Documentation for Table comment an...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1936
  
Can one of the admins verify this patch?


---

[GitHub] carbondata issue #1857: [WIP][CARBONDATA-2073][CARBONDATA-1516][Tests] Add t...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1857
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3536/



---

[GitHub] carbondata issue #1857: [WIP][CARBONDATA-2073][CARBONDATA-1516][Tests] Add t...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1857
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2300/



---

[GitHub] carbondata issue #1936: [CARBONDATA-2135] Documentation for Table comment an...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1936
  
Can one of the admins verify this patch?


---

[GitHub] carbondata pull request #1936: [CARBONDATA-2135] Documentation for Table com...

[sgururajshetty]

GitHub user sgururajshetty opened a pull request:

https://github.com/apache/carbondata/pull/1936

[CARBONDATA-2135] Documentation for Table comment and Column Comment

Documentation for table comment and column comment

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sgururajshetty/carbondata 2135

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1936.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1936


commit c78d3a56f209dea3259e0af147a96641de4e9c0f
Author: sgururajshetty 
Date:   2018-02-06T10:36:42Z

Documentation for Table comment and Column Comment




---

[jira] [Commented] (CARBONDATA-1671) Support set/unset table comment for ALTER table

[Gururaj Shetty (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-1671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353692#comment-16353692
 ] 

Gururaj Shetty commented on CARBONDATA-1671:


Documentation Update:

https://issues.apache.org/jira/browse/CARBONDATA-2135

> Support set/unset table comment for ALTER table
> ---
>
> Key: CARBONDATA-1671
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1671
> Project: CarbonData
>  Issue Type: Sub-task
>Reporter: Pawan Malwal
>Assignee: Pawan Malwal
>Priority: Major
>
> Table comment set/unset using **ALTER TABLE  SET/UNSET TBLPROPERTIES** 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CARBONDATA-1607) Support Column comment for carbon table

[Gururaj Shetty (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-1607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353691#comment-16353691
 ] 

Gururaj Shetty commented on CARBONDATA-1607:


Documentation update: 

https://issues.apache.org/jira/browse/CARBONDATA-2135

> Support Column comment for carbon table
> ---
>
> Key: CARBONDATA-1607
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1607
> Project: CarbonData
>  Issue Type: New Feature
>Reporter: Akash R Nilugal
>Assignee: Akash R Nilugal
>Priority: Minor
>
> support column comment for table, so when table is described, we can show the 
> comment for that specific column, if comment is not mentioned, comment 
> default value is null



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (CARBONDATA-2135) Documentation for Table Comment and Column Comment

[Gururaj Shetty (JIRA)]

Gururaj Shetty created CARBONDATA-2135:
--

 Summary: Documentation for Table Comment and Column Comment
 Key: CARBONDATA-2135
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2135
 Project: CarbonData
  Issue Type: Task
Reporter: Gururaj Shetty
Assignee: Gururaj Shetty


Add documentation for Table Comment and Column Comment



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata issue #1935: [CARBONDATA-2134] Prevent implicit column filter lis...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1935
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3377/



---

[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1856
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3376/



---

[GitHub] carbondata issue #1935: [CARBONDATA-2134] Prevent implicit column filter lis...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1935
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2299/



---

[GitHub] carbondata issue #1935: [CARBONDATA-2134] Prevent implicit column filter lis...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1935
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3535/



---

[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1856
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3534/



---

[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1856
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2298/



---

[GitHub] carbondata issue #1934: [CARBONDATA-2133] Fixed Exception displays after per...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1934
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3375/



---

[GitHub] carbondata issue #1934: [CARBONDATA-2133] Fixed Exception displays after per...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1934
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2297/



---

[GitHub] carbondata issue #1934: [CARBONDATA-2133] Fixed Exception displays after per...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1934
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3533/



---

[GitHub] carbondata pull request #1935: [CARBONDATA-2134] Prevent implicit column fil...

[manishgupta88]

GitHub user manishgupta88 opened a pull request:

https://github.com/apache/carbondata/pull/1935

[CARBONDATA-2134] Prevent implicit column filter list from getting 
serialized while submitting task to executor

**Problem**
In the current store blocklet pruning in driver and no further pruning 
takes place in the executor side. But still the implicit column filter list 
being sent to executor. As the size of list grows the cost of serializing and 
deserializing the list is increasing which can impact the query performance.

**Solution**
Remove the list from the filter expression before submitting the task to 
executor.

Be sure to do all of the following checklist to help us incorporate 
your contribution quickly and easily:

 - [ ] Any interfaces changed?
 No
 - [ ] Any backward compatibility impacted?
 No
 - [ ] Document update required?
No
 - [ ] Testing done
UT added
 - [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 
NA


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manishgupta88/carbondata 
executor_filter_list_serialization

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1935.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1935


commit 4630dbf30f26adff8b59ae57af12a2b183e8195f
Author: m00258959 
Date:   2018-02-05T11:40:18Z

Modified code to prevent implicit column array list from serializing and 
deserializing to executor to improve query performance




---

[jira] [Created] (CARBONDATA-2134) Prevent implicit column filter list from getting serialized while submitting task to executor

[Manish Gupta (JIRA)]

Manish Gupta created CARBONDATA-2134:


 Summary: Prevent implicit column filter list from getting 
serialized while submitting task to executor
 Key: CARBONDATA-2134
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2134
 Project: CarbonData
  Issue Type: Bug
Reporter: Manish Gupta
Assignee: Manish Gupta


**Problem**
In the current store blocklet pruning in driver and no further pruning takes 
place in the executor side. But still the implicit column filter list being 
sent to executor. As the size of list grows the cost of serializing and 
deserializing the list is increasing which can impact the query performance.

**Solution**
Remove the list from the filter expression before submitting the task to 
executor.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata pull request #1934: [CARBONDATA-2133] Fixed Exception displays af...

[anubhav100]

GitHub user anubhav100 opened a pull request:

https://github.com/apache/carbondata/pull/1934

[CARBONDATA-2133] Fixed Exception displays after performing select query on 
newly added Boolean Type




**Problem** : In Restructure util and  
RestructureBasedVectorResultCollector to get the default value of a measure 
type the case for boolean data type was missing,and in DataTypeUtil to store 
default value in bytes case of boolean data type was missing

**Solution**:Add the Required Cases

**Complete Description**:

**Classes changed**:

1.**RestructureBasedVectorResultCollector**:

**in the below method case for boolean data type was missing**

private void fillDataForNonExistingMeasures() {
for (int i = 0; i < 
tableBlockExecutionInfos.getActualQueryMeasures().length; i++) {
  if (!measureInfo.getMeasureExists()[i]) {
int queryOrder = 
tableBlockExecutionInfos.getActualQueryMeasures()[i].getQueryOrder();
CarbonMeasure measure = 
tableBlockExecutionInfos.getActualQueryMeasures()[i].getMeasure();
ColumnVectorInfo columnVectorInfo = allColumnInfo[queryOrder];
CarbonColumnVector vector = columnVectorInfo.vector;
Object defaultValue = measureDefaultValues[i];
if (null == defaultValue) {
  vector.putNulls(columnVectorInfo.vectorOffset, 
columnVectorInfo.size);
} else {
  DataType dataType = measureInfo.getMeasureDataTypes()[i];
  if (dataType == DataTypes.SHORT) {
vector.putShorts(columnVectorInfo.vectorOffset, 
columnVectorInfo.size,
(short) defaultValue);
  } else if (dataType == DataTypes.INT) {
vector
.putInts(columnVectorInfo.vectorOffset, 
columnVectorInfo.size, (int) defaultValue);
  } else if (dataType == DataTypes.LONG) {
vector.putLongs(columnVectorInfo.vectorOffset, 
columnVectorInfo.size,
(long) defaultValue);
  } else if (DataTypes.isDecimal(dataType)) {
vector.putDecimals(columnVectorInfo.vectorOffset, 
columnVectorInfo.size,
((Decimal) defaultValue).toJavaBigDecimal(), 
measure.getPrecision());
  } else {
vector.putDoubles(columnVectorInfo.vectorOffset, 
columnVectorInfo.size,
(double) defaultValue);
  }
}
  }
}
  }

2.**RestructureUtil**:

**in below methods getMeasureDefaultValue,getMeasureDefaultValueByType the 
if statement  for boolean data type was missing**

public static Object getMeasureDefaultValue(ColumnSchema columnSchema, 
byte[] defaultValue) {
Object measureDefaultValue = null;
if (!isDefaultValueNull(defaultValue)) {
  String value;
  DataType dataType = columnSchema.getDataType();
  if (dataType == DataTypes.SHORT) {
value = new String(defaultValue, 
Charset.forName(CarbonCommonConstants.DEFAULT_CHARSET));
measureDefaultValue = Short.valueOf(value);
  } else if (dataType == DataTypes.LONG) {
value = new String(defaultValue, 
Charset.forName(CarbonCommonConstants.DEFAULT_CHARSET));
measureDefaultValue = Long.parseLong(value);
  } else if (dataType == DataTypes.INT) {
value = new String(defaultValue, 
Charset.forName(CarbonCommonConstants.DEFAULT_CHARSET));
measureDefaultValue = Integer.parseInt(value);
  } else if (DataTypes.isDecimal(dataType)) {
BigDecimal decimal = DataTypeUtil.byteToBigDecimal(defaultValue);
if (columnSchema.getScale() > decimal.scale()) {
  decimal = decimal.setScale(columnSchema.getScale(), 
RoundingMode.HALF_UP);
}
measureDefaultValue = decimal;
  } else {
value = new String(defaultValue, 
Charset.forName(CarbonCommonConstants.DEFAULT_CHARSET));
Double parsedValue = Double.valueOf(value);
if (!Double.isInfinite(parsedValue) && !Double.isNaN(parsedValue)) {
  measureDefaultValue = parsedValue;
}
  }
}
return measureDefaultValue;
  }

3.**DatatypeUtil**: To store default value on bytes case of boolean data 
type was missing

public static byte[] convertDataToBytesBasedOnDataType(String data, 
ColumnSchema columnSchema) {
if (null == data) {
  return null;
} else if (CarbonCommonConstants.MEMBER_DEFAULT_VAL.equals(data)) {
  LOGGER.error("Default value should not be carbon specific null value 
: " + data);
  return null;
}
try {
  long parsedIntVal = 0;
  DataType dataType = columnSchema.getDataType();
  if (dataType == DataTypes.INT) {
parsedIntVal = (long)