[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3988: [WIP] Clean index files when clean files command executed

2020-10-16 Thread GitBox


CarbonDataQA1 commented on pull request #3988:
URL: https://github.com/apache/carbondata/pull/3988#issuecomment-710648705


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4494/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3988: [WIP] Clean index files when clean files command executed

2020-10-16 Thread GitBox


CarbonDataQA1 commented on pull request #3988:
URL: https://github.com/apache/carbondata/pull/3988#issuecomment-710589053


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2740/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ShreelekhyaG opened a new pull request #3988: [WIP] Clean index files when clean files command executed

2020-10-16 Thread GitBox


ShreelekhyaG opened a new pull request #3988:
URL: https://github.com/apache/carbondata/pull/3988


### Why is this PR needed?
Clean index files when clean files command executed- avoid auto clean up of 
index files
   Avoid multiple segment file and status file updates.

### What changes were proposed in this PR?
   When clean files command is executed, clean only those that have existed for 
more than 1 hour.
   
### Does this PR introduce any user interface change?
- No
- Yes. (please explain the change and update document)
   
### Is any new testcase added?
- Yes
   
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3914: [CARBONDATA-3979] Added Hive local dictionary support example

2020-10-16 Thread GitBox


CarbonDataQA1 commented on pull request #3914:
URL: https://github.com/apache/carbondata/pull/3914#issuecomment-710132890


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2739/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3985: [CARBONDATA-3965]Fixed float variable target datatype in case of adaptive encoding

2020-10-16 Thread GitBox


CarbonDataQA1 commented on pull request #3985:
URL: https://github.com/apache/carbondata/pull/3985#issuecomment-710113383


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2736/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3948: [HOTFIX] Fix random 11 testcase failure in CI

2020-10-16 Thread GitBox


CarbonDataQA1 commented on pull request #3948:
URL: https://github.com/apache/carbondata/pull/3948#issuecomment-710111981


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2738/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3948: [HOTFIX] Fix random 11 testcase failure in CI

2020-10-16 Thread GitBox


CarbonDataQA1 commented on pull request #3948:
URL: https://github.com/apache/carbondata/pull/3948#issuecomment-710110125


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4492/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3914: [CARBONDATA-3979] Added Hive local dictionary support example

2020-10-16 Thread GitBox


CarbonDataQA1 commented on pull request #3914:
URL: https://github.com/apache/carbondata/pull/3914#issuecomment-710090390


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4493/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3985: [CARBONDATA-3965]Fixed float variable target datatype in case of adaptive encoding

2020-10-16 Thread GitBox


CarbonDataQA1 commented on pull request #3985:
URL: https://github.com/apache/carbondata/pull/3985#issuecomment-710085979


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4490/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ShreelekhyaG commented on pull request #3914: [CARBONDATA-3979] Added Hive local dictionary support example

2020-10-16 Thread GitBox


ShreelekhyaG commented on pull request #3914:
URL: https://github.com/apache/carbondata/pull/3914#issuecomment-710064999


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3980: [CARBONDATA-3901] [CARBONDATA-3903] [CARBONDATA-3824] SI creation on unkbown table and doc changes.

2020-10-16 Thread GitBox


CarbonDataQA1 commented on pull request #3980:
URL: https://github.com/apache/carbondata/pull/3980#issuecomment-710056435


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2735/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3980: [CARBONDATA-3901] [CARBONDATA-3903] [CARBONDATA-3824] SI creation on unkbown table and doc changes.

2020-10-16 Thread GitBox


CarbonDataQA1 commented on pull request #3980:
URL: https://github.com/apache/carbondata/pull/3980#issuecomment-710047594


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4489/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ajantha-bhat commented on pull request #3948: [HOTFIX] Fix random 11 testcase failure in CI

2020-10-16 Thread GitBox


ajantha-bhat commented on pull request #3948:
URL: https://github.com/apache/carbondata/pull/3948#issuecomment-710046428


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3948: [HOTFIX] Fix random 11 testcase failure in CI

2020-10-16 Thread GitBox


CarbonDataQA1 commented on pull request #3948:
URL: https://github.com/apache/carbondata/pull/3948#issuecomment-710042635


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4491/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ajantha-bhat commented on pull request #3948: [HOTFIX] Fix random 11 testcase failure in CI

2020-10-16 Thread GitBox


ajantha-bhat commented on pull request #3948:
URL: https://github.com/apache/carbondata/pull/3948#issuecomment-710039056


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3983: [CARBONDATA-4036]Fix special char(`) issue in create table, when column name contains ` character

2020-10-16 Thread GitBox


CarbonDataQA1 commented on pull request #3983:
URL: https://github.com/apache/carbondata/pull/3983#issuecomment-710034096


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2734/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3983: [CARBONDATA-4036]Fix special char(`) issue in create table, when column name contains ` character

2020-10-16 Thread GitBox


CarbonDataQA1 commented on pull request #3983:
URL: https://github.com/apache/carbondata/pull/3983#issuecomment-710025847


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4488/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3914: [CARBONDATA-3979] Added Hive local dictionary support example

2020-10-16 Thread GitBox


CarbonDataQA1 commented on pull request #3914:
URL: https://github.com/apache/carbondata/pull/3914#issuecomment-710024822


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2732/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] nihal0107 commented on pull request #3985: [CARBONDATA-3965]Fixed float variable target datatype in case of adaptive encoding

2020-10-16 Thread GitBox


nihal0107 commented on pull request #3985:
URL: https://github.com/apache/carbondata/pull/3985#issuecomment-710024408


   retest this please.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3914: [CARBONDATA-3979] Added Hive local dictionary support example

2020-10-16 Thread GitBox


CarbonDataQA1 commented on pull request #3914:
URL: https://github.com/apache/carbondata/pull/3914#issuecomment-710017108


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4486/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3985: [CARBONDATA-3965]Fixed float variable target datatype in case of adaptive encoding

2020-10-16 Thread GitBox


CarbonDataQA1 commented on pull request #3985:
URL: https://github.com/apache/carbondata/pull/3985#issuecomment-710015410


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4485/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3985: [CARBONDATA-3965]Fixed float variable target datatype in case of adaptive encoding

2020-10-16 Thread GitBox


CarbonDataQA1 commented on pull request #3985:
URL: https://github.com/apache/carbondata/pull/3985#issuecomment-710001432


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2731/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3986: [CARBONDATA-4034] Improve the time-consuming of Horizontal Compaction for update

2020-10-16 Thread GitBox


CarbonDataQA1 commented on pull request #3986:
URL: https://github.com/apache/carbondata/pull/3986#issuecomment-709984578


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4484/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] akashrn5 commented on pull request #3983: [CARBONDATA-4036]Fix special char(`) issue in create table, when column name contains ` character

2020-10-16 Thread GitBox


akashrn5 commented on pull request #3983:
URL: https://github.com/apache/carbondata/pull/3983#issuecomment-709978439


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3986: [CARBONDATA-4034] Improve the time-consuming of Horizontal Compaction for update

2020-10-16 Thread GitBox


CarbonDataQA1 commented on pull request #3986:
URL: https://github.com/apache/carbondata/pull/3986#issuecomment-709976082


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2730/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3987: [WIP ]local-dictionary

2020-10-16 Thread GitBox


CarbonDataQA1 commented on pull request #3987:
URL: https://github.com/apache/carbondata/pull/3987#issuecomment-709975490


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4487/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3987: [WIP ]local-dictionary

2020-10-16 Thread GitBox


CarbonDataQA1 commented on pull request #3987:
URL: https://github.com/apache/carbondata/pull/3987#issuecomment-709975025


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2733/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] akkio-97 opened a new pull request #3987: [WIP ]local-dictionary

2020-10-16 Thread GitBox


akkio-97 opened a new pull request #3987:
URL: https://github.com/apache/carbondata/pull/3987


### Why is this PR needed?


### What changes were proposed in this PR?
   
   
### Does this PR introduce any user interface change?
- No
- Yes. (please explain the change and update document)
   
### Is any new testcase added?
- No
- Yes
   
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[jira] [Updated] (CARBONDATA-4036) When the ` character is present in column name, the table creation fails

2020-10-16 Thread Akash R Nilugal (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-4036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akash R Nilugal updated CARBONDATA-4036:

Description: 
When the ` character is present in column name, the table creation fails

sql("create table special_char(`i#d` string, `nam(e` 

[jira] [Issue Comment Deleted] (CARBONDATA-4035) MV table is not hit when sum() is applied on decimal column.

2020-10-16 Thread Akash R Nilugal (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akash R Nilugal updated CARBONDATA-4035:

Comment: was deleted

(was:  sql("drop table if exists special_char")
sql("create table special_char(`i#d` string, `nam(e` 

[jira] [Created] (CARBONDATA-4036) When the ` character is present in column name, the table creation fails

2020-10-16 Thread Akash R Nilugal (Jira)
Akash R Nilugal created CARBONDATA-4036:
---

 Summary: When the ` character is present in column name, the table 
creation fails
 Key: CARBONDATA-4036
 URL: https://issues.apache.org/jira/browse/CARBONDATA-4036
 Project: CarbonData
  Issue Type: Bug
Reporter: Akash R Nilugal
Assignee: Akash R Nilugal


When the ` character is present in column name, the table creation fails



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (CARBONDATA-4035) MV table is not hit when sum() is applied on decimal column.

2020-10-16 Thread Akash R Nilugal (Jira)


[ 
https://issues.apache.org/jira/browse/CARBONDATA-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17215334#comment-17215334
 ] 

Akash R Nilugal commented on CARBONDATA-4035:
-

 sql("drop table if exists special_char")
sql("create table special_char(`i#d` string, `nam(e` 

[GitHub] [carbondata] kunal642 commented on pull request #3914: [CARBONDATA-3979] Added Hive local dictionary support example

2020-10-16 Thread GitBox


kunal642 commented on pull request #3914:
URL: https://github.com/apache/carbondata/pull/3914#issuecomment-709969962


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[jira] [Created] (CARBONDATA-4035) MV table is not hit when sum() is applied on decimal column.

2020-10-16 Thread Akash R Nilugal (Jira)
Akash R Nilugal created CARBONDATA-4035:
---

 Summary: MV table is not hit when sum() is applied on decimal 
column.
 Key: CARBONDATA-4035
 URL: https://issues.apache.org/jira/browse/CARBONDATA-4035
 Project: CarbonData
  Issue Type: Bug
Reporter: Akash R Nilugal
Assignee: Akash R Nilugal


MV table is not hit when sum() is applied on decimal column.

sql("drop table if exists sum_agg_decimal")
sql("create table sum_agg_decimal(salary1 decimal(7,2),salary2 
decimal(7,2),salary3 decimal(7,2),salary4 decimal(7,2),empname string) stored 
as carbondata")
sql("drop materialized view if exists decimal_mv")
sql("create materialized view decimal_mv as select empname, sum(salary1 - 
salary2) from sum_agg_decimal group by empname")
sql("explain select empname, sum( salary1 - salary2) from sum_agg_decimal 
group by empname").show(false)





--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[GitHub] [carbondata] nihal0107 removed a comment on pull request #3985: [CARBONDATA-3965]Fixed float variable target datatype in case of adaptive encoding

2020-10-16 Thread GitBox


nihal0107 removed a comment on pull request #3985:
URL: https://github.com/apache/carbondata/pull/3985#issuecomment-709897532


   retest this please.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] marchpure commented on a change in pull request #3986: [CARBONDATA-4034] Improve the time-consuming of Horizontal Compaction for update

2020-10-16 Thread GitBox


marchpure commented on a change in pull request #3986:
URL: https://github.com/apache/carbondata/pull/3986#discussion_r506261693



##
File path: 
processing/src/main/java/org/apache/carbondata/processing/merger/CarbonDataMergerUtil.java
##
@@ -1246,8 +1209,22 @@ public static boolean isHorizontalCompactionEnabled() {
 // set the update status.
 segmentUpdateStatusManager.setUpdateStatusDetails(segmentUpdateDetails);
 
-CarbonFile[] deleteDeltaFiles =
-segmentUpdateStatusManager.getDeleteDeltaFilesList(new Segment(seg), 
blockName);
+// only when SegmentUpdateDetails contain the specified block
+// will the method getDeleteDeltaFilesList be executed
+List blockNameList = 
segmentUpdateStatusManager.getBlockNameFromSegment(seg);
+Map> blockAndDeleteDeltaFilesMap = new 
HashMap<>();
+CarbonFile[] deleteDeltaFiles = null;
+if (blockNameList.contains(blockName)) {
+  blockAndDeleteDeltaFilesMap =
+  segmentUpdateStatusManager.getDeleteDeltaFilesList(new Segment(seg));
+}
+if (blockAndDeleteDeltaFilesMap.containsKey(blockName)) {
+  List deleteDeltaFileList = 
blockAndDeleteDeltaFilesMap.get(blockName);
+  deleteDeltaFiles = deleteDeltaFileList.toArray(new 
CarbonFile[deleteDeltaFileList.size()]);
+}
+
+// CarbonFile[] deleteDeltaFiles =

Review comment:
   delete these 2 lines

##
File path: 
processing/src/main/java/org/apache/carbondata/processing/merger/CarbonDataMergerUtil.java
##
@@ -1246,8 +1209,22 @@ public static boolean isHorizontalCompactionEnabled() {
 // set the update status.
 segmentUpdateStatusManager.setUpdateStatusDetails(segmentUpdateDetails);
 
-CarbonFile[] deleteDeltaFiles =
-segmentUpdateStatusManager.getDeleteDeltaFilesList(new Segment(seg), 
blockName);
+// only when SegmentUpdateDetails contain the specified block
+// will the method getDeleteDeltaFilesList be executed
+List blockNameList = 
segmentUpdateStatusManager.getBlockNameFromSegment(seg);
+Map> blockAndDeleteDeltaFilesMap = new 
HashMap<>();
+CarbonFile[] deleteDeltaFiles = null;

Review comment:
   = new CarbonFile[0]

##
File path: 
processing/src/main/java/org/apache/carbondata/processing/merger/CarbonDataMergerUtil.java
##
@@ -1246,8 +1209,22 @@ public static boolean isHorizontalCompactionEnabled() {
 // set the update status.
 segmentUpdateStatusManager.setUpdateStatusDetails(segmentUpdateDetails);
 
-CarbonFile[] deleteDeltaFiles =
-segmentUpdateStatusManager.getDeleteDeltaFilesList(new Segment(seg), 
blockName);
+// only when SegmentUpdateDetails contain the specified block
+// will the method getDeleteDeltaFilesList be executed
+List blockNameList = 
segmentUpdateStatusManager.getBlockNameFromSegment(seg);
+Map> blockAndDeleteDeltaFilesMap = new 
HashMap<>();
+CarbonFile[] deleteDeltaFiles = null;
+if (blockNameList.contains(blockName)) {

Review comment:
   if (blockNameList.contains(blockName)) { 
 blockAndDeleteDeltaFilesMap =
 if (blockAndDeleteDeltaFilesMap.containsKey(blockName)) {
 }
   }

##
File path: 
processing/src/main/java/org/apache/carbondata/processing/merger/CarbonDataMergerUtil.java
##
@@ -1246,8 +1209,22 @@ public static boolean isHorizontalCompactionEnabled() {
 // set the update status.
 segmentUpdateStatusManager.setUpdateStatusDetails(segmentUpdateDetails);
 
-CarbonFile[] deleteDeltaFiles =
-segmentUpdateStatusManager.getDeleteDeltaFilesList(new Segment(seg), 
blockName);
+// only when SegmentUpdateDetails contain the specified block
+// will the method getDeleteDeltaFilesList be executed
+List blockNameList = 
segmentUpdateStatusManager.getBlockNameFromSegment(seg);
+Map> blockAndDeleteDeltaFilesMap = new 
HashMap<>();
+CarbonFile[] deleteDeltaFiles = null;
+if (blockNameList.contains(blockName)) {
+  blockAndDeleteDeltaFilesMap =
+  segmentUpdateStatusManager.getDeleteDeltaFilesList(new Segment(seg));
+}
+if (blockAndDeleteDeltaFilesMap.containsKey(blockName)) {

Review comment:
   no need to covert to array





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] nihal0107 commented on pull request #3985: [CARBONDATA-3965]Fixed float variable target datatype in case of adaptive encoding

2020-10-16 Thread GitBox


nihal0107 commented on pull request #3985:
URL: https://github.com/apache/carbondata/pull/3985#issuecomment-709957980


   retest this please.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] nihal0107 removed a comment on pull request #3985: [CARBONDATA-3965]Fixed float variable target datatype in case of adaptive encoding

2020-10-16 Thread GitBox


nihal0107 removed a comment on pull request #3985:
URL: https://github.com/apache/carbondata/pull/3985#issuecomment-709745588


   retest this please.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3695: [WIP] partition optimization

2020-10-16 Thread GitBox


CarbonDataQA1 commented on pull request #3695:
URL: https://github.com/apache/carbondata/pull/3695#issuecomment-709940050


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2728/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3695: [WIP] partition optimization

2020-10-16 Thread GitBox


CarbonDataQA1 commented on pull request #3695:
URL: https://github.com/apache/carbondata/pull/3695#issuecomment-709938508


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4482/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] shenjiayu17 commented on a change in pull request #3986: [CARBONDATA-4034] Improve the time-consuming of Horizontal Compaction for update

2020-10-16 Thread GitBox


shenjiayu17 commented on a change in pull request #3986:
URL: https://github.com/apache/carbondata/pull/3986#discussion_r506203367



##
File path: 
core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentUpdateStatusManager.java
##
@@ -455,6 +455,51 @@ public boolean accept(CarbonFile pathName) {
 return null;
   }
 
+  public Map> getDeleteDeltaFilesForSegment(final 
Segment seg) {
+String segmentPath = CarbonTablePath.getSegmentPath(
+  identifier.getTablePath(), seg.getSegmentNo());

Review comment:
   Done.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] shenjiayu17 commented on a change in pull request #3986: [CARBONDATA-4034] Improve the time-consuming of Horizontal Compaction for update

2020-10-16 Thread GitBox


shenjiayu17 commented on a change in pull request #3986:
URL: https://github.com/apache/carbondata/pull/3986#discussion_r506202461



##
File path: 
core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentUpdateStatusManager.java
##
@@ -455,6 +455,51 @@ public boolean accept(CarbonFile pathName) {
 return null;
   }
 
+  public Map> getDeleteDeltaFilesForSegment(final 
Segment seg) {

Review comment:
   Done.
   Keep function name getDeleteDeltaFilesList and change the return type to 
Map>





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] shenjiayu17 commented on a change in pull request #3986: [CARBONDATA-4034] Improve the time-consuming of Horizontal Compaction for update

2020-10-16 Thread GitBox


shenjiayu17 commented on a change in pull request #3986:
URL: https://github.com/apache/carbondata/pull/3986#discussion_r506020436



##
File path: 
processing/src/main/java/org/apache/carbondata/processing/merger/CarbonDataMergerUtil.java
##
@@ -1039,22 +1039,10 @@ private static boolean 
isSegmentValid(LoadMetadataDetails seg) {
 if (CompactionType.IUD_DELETE_DELTA == compactionTypeIUD) {
   int numberDeleteDeltaFilesThreshold =
   
CarbonProperties.getInstance().getNoDeleteDeltaFilesThresholdForIUDCompaction();
-  List deleteSegments = new ArrayList<>();
   for (Segment seg : segments) {
-if (checkDeleteDeltaFilesInSeg(seg, segmentUpdateStatusManager,

Review comment:
   Done.
   Combined function checkDeleteDeltaFilesInSeg and function 
getDeleteDeltaFilesInSeg to new function checkAndGetDeleteDeltaFilesInSeg





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] shenjiayu17 commented on a change in pull request #3986: [CARBONDATA-4034] Improve the time-consuming of Horizontal Compaction for update

2020-10-16 Thread GitBox


shenjiayu17 commented on a change in pull request #3986:
URL: https://github.com/apache/carbondata/pull/3986#discussion_r506020781



##
File path: 
processing/src/main/java/org/apache/carbondata/processing/merger/CarbonDataMergerUtil.java
##
@@ -1210,6 +1198,39 @@ private static boolean 
checkDeleteDeltaFilesInSeg(Segment seg,
 return blockLists;
   }
 
+  private static List checkAndGetDeleteDeltaFilesInSeg(Segment seg,
+  SegmentUpdateStatusManager segmentUpdateStatusManager, int 
numberDeltaFilesThreshold) {
+
+List blockLists = new ArrayList<>();
+
+Map> blockAndDeleteDeltaFilesMap =
+  segmentUpdateStatusManager.getDeleteDeltaFilesForSegment(seg);
+
+List blockNameList =
+  segmentUpdateStatusManager.getBlockNameFromSegment(seg.getSegmentNo());
+
+Set uniqueBlocks = new HashSet();
+for (final String blockName : blockNameList) {
+
+  List deleteDeltaFiles = 
blockAndDeleteDeltaFilesMap.get(blockName);
+
+  if (null != deleteDeltaFiles) {
+for (CarbonFile blocks : deleteDeltaFiles) {

Review comment:
   Done. 
   Added judgement: 
   if (deleteDeltaFiles.size() <= numberDeltaFilesThreshold) continue





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3914: [CARBONDATA-3979] Added Hive local dictionary support example

2020-10-16 Thread GitBox


CarbonDataQA1 commented on pull request #3914:
URL: https://github.com/apache/carbondata/pull/3914#issuecomment-709918196


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4481/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3914: [CARBONDATA-3979] Added Hive local dictionary support example

2020-10-16 Thread GitBox


CarbonDataQA1 commented on pull request #3914:
URL: https://github.com/apache/carbondata/pull/3914#issuecomment-709903460


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2727/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] nihal0107 commented on pull request #3985: [CARBONDATA-3965]Fixed float variable target datatype in case of adaptive encoding

2020-10-16 Thread GitBox


nihal0107 commented on pull request #3985:
URL: https://github.com/apache/carbondata/pull/3985#issuecomment-709897532


   retest this please.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3985: [CARBONDATA-3965]Fixed float variable target datatype in case of adaptive encoding

2020-10-16 Thread GitBox


CarbonDataQA1 commented on pull request #3985:
URL: https://github.com/apache/carbondata/pull/3985#issuecomment-709896788


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2726/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3948: [HOTFIX] Fix random 11 testcase failure in CI

2020-10-16 Thread GitBox


CarbonDataQA1 commented on pull request #3948:
URL: https://github.com/apache/carbondata/pull/3948#issuecomment-709870946


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4479/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3948: [HOTFIX] Fix random 11 testcase failure in CI

2020-10-16 Thread GitBox


CarbonDataQA1 commented on pull request #3948:
URL: https://github.com/apache/carbondata/pull/3948#issuecomment-709865455


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2725/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3789: [CARBONDATA-3864] Store Size Optimization

2020-10-16 Thread GitBox


CarbonDataQA1 commented on pull request #3789:
URL: https://github.com/apache/carbondata/pull/3789#issuecomment-709862439


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2724/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3789: [CARBONDATA-3864] Store Size Optimization

2020-10-16 Thread GitBox


CarbonDataQA1 commented on pull request #3789:
URL: https://github.com/apache/carbondata/pull/3789#issuecomment-709861587


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4478/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] kunal642 commented on a change in pull request #3932: [CARBONDATA-3994] Skip Order by for map task if it is a first sort column and use limit pushdown for array_contains filter

2020-10-16 Thread GitBox


kunal642 commented on a change in pull request #3932:
URL: https://github.com/apache/carbondata/pull/3932#discussion_r506086546



##
File path: 
integration/spark/src/main/scala/org/apache/spark/sql/execution/strategy/CarbonLateDecodeStrategy.scala
##
@@ -984,6 +988,80 @@ private[sql] class CarbonLateDecodeStrategy extends 
SparkStrategy {
   null)(sparkSession)
 }
   }
+
+  object ExtractTakeOrderedAndProjectExec {
+
+def unapply(plan: LogicalPlan): Option[CarbonTakeOrderedAndProjectExec] = {
+  val allRelations = plan.collect { case logicalRelation: LogicalRelation 
=> logicalRelation }
+  // push down order by limit to carbon map task,
+  // only when there are only one CarbonDatasourceHadoopRelation
+  if (allRelations.size != 1 ||
+  allRelations.exists(x => 
!x.relation.isInstanceOf[CarbonDatasourceHadoopRelation])) {
+return None
+  }
+  //  check and Replace TakeOrderedAndProject with 
CarbonTakeOrderedAndProjectExec.
+  val relation = 
allRelations.head.relation.asInstanceOf[CarbonDatasourceHadoopRelation]
+  val sparkPlan = plan match {
+case ReturnAnswer(rootPlan) => rootPlan match {
+  case Limit(IntegerLiteral(limit), Sort(order, true, child)) =>
+TakeOrderedAndProjectExec(limit,
+  order,
+  child.output,
+  planLater(pushLimit(limit, child)))
+  case Limit(IntegerLiteral(limit), Project(projectList, Sort(order, 
true, child))) =>
+TakeOrderedAndProjectExec(limit, order, projectList, 
planLater(pushLimit(limit, child)))

Review comment:
   instead of TakeOrderedAndProjectExec, we should directly prepare 
CarbonTakeOrderedAndProjectExec instance based on the checks below. I feel this 
TakeOrderedAndProjectExec creation is unnecessary





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] kunal642 commented on a change in pull request #3932: [CARBONDATA-3994] Skip Order by for map task if it is a first sort column and use limit pushdown for array_contains filter

2020-10-16 Thread GitBox


kunal642 commented on a change in pull request #3932:
URL: https://github.com/apache/carbondata/pull/3932#discussion_r506085998



##
File path: 
integration/spark/src/main/scala/org/apache/spark/sql/execution/CarbonTakeOrderedAndProjectExec.scala
##
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.serializer.Serializer
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, NamedExpression, 
SortOrder, UnsafeProjection}
+import 
org.apache.spark.sql.catalyst.expressions.codegen.LazilyGeneratedOrdering
+import org.apache.spark.sql.catalyst.plans.physical.{Partitioning, 
SinglePartition}
+import org.apache.spark.sql.execution.exchange.ShuffleExchangeExec
+import org.apache.spark.util.Utils
+
+// To skip the order at map task
+case class CarbonTakeOrderedAndProjectExec(
+limit: Int,
+sortOrder: Seq[SortOrder],
+projectList: Seq[NamedExpression],
+child: SparkPlan,
+skipMapOrder: Boolean = false,
+readFromHead: Boolean = true) extends UnaryExecNode {

Review comment:
   CarbonTakeOrderedAndProjectExec should extend TakeOrderedAndProjectExec, 
and unmodified methods should not be overridden 





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] kunal642 commented on pull request #3914: [CARBONDATA-3979] Added Hive local dictionary support example

2020-10-16 Thread GitBox


kunal642 commented on pull request #3914:
URL: https://github.com/apache/carbondata/pull/3914#issuecomment-709835508


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org