[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3988: [WIP] Clean index files when clean files command executed
CarbonDataQA1 commented on pull request #3988: URL: https://github.com/apache/carbondata/pull/3988#issuecomment-710648705 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4494/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3988: [WIP] Clean index files when clean files command executed
CarbonDataQA1 commented on pull request #3988: URL: https://github.com/apache/carbondata/pull/3988#issuecomment-710589053 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2740/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG opened a new pull request #3988: [WIP] Clean index files when clean files command executed
ShreelekhyaG opened a new pull request #3988: URL: https://github.com/apache/carbondata/pull/3988 ### Why is this PR needed? Clean index files when clean files command executed- avoid auto clean up of index files Avoid multiple segment file and status file updates. ### What changes were proposed in this PR? When clean files command is executed, clean only those that have existed for more than 1 hour. ### Does this PR introduce any user interface change? - No - Yes. (please explain the change and update document) ### Is any new testcase added? - Yes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3914: [CARBONDATA-3979] Added Hive local dictionary support example
CarbonDataQA1 commented on pull request #3914: URL: https://github.com/apache/carbondata/pull/3914#issuecomment-710132890 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2739/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3985: [CARBONDATA-3965]Fixed float variable target datatype in case of adaptive encoding
CarbonDataQA1 commented on pull request #3985: URL: https://github.com/apache/carbondata/pull/3985#issuecomment-710113383 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2736/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3948: [HOTFIX] Fix random 11 testcase failure in CI
CarbonDataQA1 commented on pull request #3948: URL: https://github.com/apache/carbondata/pull/3948#issuecomment-710111981 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2738/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3948: [HOTFIX] Fix random 11 testcase failure in CI
CarbonDataQA1 commented on pull request #3948: URL: https://github.com/apache/carbondata/pull/3948#issuecomment-710110125 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4492/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3914: [CARBONDATA-3979] Added Hive local dictionary support example
CarbonDataQA1 commented on pull request #3914: URL: https://github.com/apache/carbondata/pull/3914#issuecomment-710090390 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4493/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3985: [CARBONDATA-3965]Fixed float variable target datatype in case of adaptive encoding
CarbonDataQA1 commented on pull request #3985: URL: https://github.com/apache/carbondata/pull/3985#issuecomment-710085979 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4490/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on pull request #3914: [CARBONDATA-3979] Added Hive local dictionary support example
ShreelekhyaG commented on pull request #3914: URL: https://github.com/apache/carbondata/pull/3914#issuecomment-710064999 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3980: [CARBONDATA-3901] [CARBONDATA-3903] [CARBONDATA-3824] SI creation on unkbown table and doc changes.
CarbonDataQA1 commented on pull request #3980: URL: https://github.com/apache/carbondata/pull/3980#issuecomment-710056435 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2735/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3980: [CARBONDATA-3901] [CARBONDATA-3903] [CARBONDATA-3824] SI creation on unkbown table and doc changes.
CarbonDataQA1 commented on pull request #3980: URL: https://github.com/apache/carbondata/pull/3980#issuecomment-710047594 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4489/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on pull request #3948: [HOTFIX] Fix random 11 testcase failure in CI
ajantha-bhat commented on pull request #3948: URL: https://github.com/apache/carbondata/pull/3948#issuecomment-710046428 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3948: [HOTFIX] Fix random 11 testcase failure in CI
CarbonDataQA1 commented on pull request #3948: URL: https://github.com/apache/carbondata/pull/3948#issuecomment-710042635 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4491/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on pull request #3948: [HOTFIX] Fix random 11 testcase failure in CI
ajantha-bhat commented on pull request #3948: URL: https://github.com/apache/carbondata/pull/3948#issuecomment-710039056 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3983: [CARBONDATA-4036]Fix special char(`) issue in create table, when column name contains ` character
CarbonDataQA1 commented on pull request #3983: URL: https://github.com/apache/carbondata/pull/3983#issuecomment-710034096 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2734/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3983: [CARBONDATA-4036]Fix special char(`) issue in create table, when column name contains ` character
CarbonDataQA1 commented on pull request #3983: URL: https://github.com/apache/carbondata/pull/3983#issuecomment-710025847 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4488/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3914: [CARBONDATA-3979] Added Hive local dictionary support example
CarbonDataQA1 commented on pull request #3914: URL: https://github.com/apache/carbondata/pull/3914#issuecomment-710024822 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2732/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] nihal0107 commented on pull request #3985: [CARBONDATA-3965]Fixed float variable target datatype in case of adaptive encoding
nihal0107 commented on pull request #3985: URL: https://github.com/apache/carbondata/pull/3985#issuecomment-710024408 retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3914: [CARBONDATA-3979] Added Hive local dictionary support example
CarbonDataQA1 commented on pull request #3914: URL: https://github.com/apache/carbondata/pull/3914#issuecomment-710017108 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4486/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3985: [CARBONDATA-3965]Fixed float variable target datatype in case of adaptive encoding
CarbonDataQA1 commented on pull request #3985: URL: https://github.com/apache/carbondata/pull/3985#issuecomment-710015410 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4485/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3985: [CARBONDATA-3965]Fixed float variable target datatype in case of adaptive encoding
CarbonDataQA1 commented on pull request #3985: URL: https://github.com/apache/carbondata/pull/3985#issuecomment-710001432 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2731/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3986: [CARBONDATA-4034] Improve the time-consuming of Horizontal Compaction for update
CarbonDataQA1 commented on pull request #3986: URL: https://github.com/apache/carbondata/pull/3986#issuecomment-709984578 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4484/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on pull request #3983: [CARBONDATA-4036]Fix special char(`) issue in create table, when column name contains ` character
akashrn5 commented on pull request #3983: URL: https://github.com/apache/carbondata/pull/3983#issuecomment-709978439 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3986: [CARBONDATA-4034] Improve the time-consuming of Horizontal Compaction for update
CarbonDataQA1 commented on pull request #3986: URL: https://github.com/apache/carbondata/pull/3986#issuecomment-709976082 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2730/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3987: [WIP ]local-dictionary
CarbonDataQA1 commented on pull request #3987: URL: https://github.com/apache/carbondata/pull/3987#issuecomment-709975490 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4487/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3987: [WIP ]local-dictionary
CarbonDataQA1 commented on pull request #3987: URL: https://github.com/apache/carbondata/pull/3987#issuecomment-709975025 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2733/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akkio-97 opened a new pull request #3987: [WIP ]local-dictionary
akkio-97 opened a new pull request #3987: URL: https://github.com/apache/carbondata/pull/3987 ### Why is this PR needed? ### What changes were proposed in this PR? ### Does this PR introduce any user interface change? - No - Yes. (please explain the change and update document) ### Is any new testcase added? - No - Yes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (CARBONDATA-4036) When the ` character is present in column name, the table creation fails
[
https://issues.apache.org/jira/browse/CARBONDATA-4036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akash R Nilugal updated CARBONDATA-4036:
Description:
When the ` character is present in column name, the table creation fails
sql("create table special_char(`i#d` string, `nam(e` string,`ci)&#@!ty`
string,`a\be` int, `ag!e` float, `na^me1` Decimal(8,4), ```a``bc``!!d``` int)
stored as carbondata" +
" tblproperties('INVERTED_INDEX'='`a`bc`!!d`',
'SORT_COLUMNS'='`a`bc`!!d`')")
was:When the ` character is present in column name, the table creation fails
> When the ` character is present in column name, the table creation fails
>
>
> Key: CARBONDATA-4036
> URL: https://issues.apache.org/jira/browse/CARBONDATA-4036
> Project: CarbonData
> Issue Type: Bug
>Reporter: Akash R Nilugal
>Assignee: Akash R Nilugal
>Priority: Minor
>
> When the ` character is present in column name, the table creation fails
> sql("create table special_char(`i#d` string, `nam(e` string,`ci)&#@!ty`
> string,`a\be` int, `ag!e` float, `na^me1` Decimal(8,4), ```a``bc``!!d``` int)
> stored as carbondata" +
> " tblproperties('INVERTED_INDEX'='`a`bc`!!d`',
> 'SORT_COLUMNS'='`a`bc`!!d`')")
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CARBONDATA-4036) When the ` character is present in column name, the table creation fails
Akash R Nilugal created CARBONDATA-4036: --- Summary: When the ` character is present in column name, the table creation fails Key: CARBONDATA-4036 URL: https://issues.apache.org/jira/browse/CARBONDATA-4036 Project: CarbonData Issue Type: Bug Reporter: Akash R Nilugal Assignee: Akash R Nilugal When the ` character is present in column name, the table creation fails -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CARBONDATA-4035) MV table is not hit when sum() is applied on decimal column.
[
https://issues.apache.org/jira/browse/CARBONDATA-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215334#comment-17215334
]
Akash R Nilugal commented on CARBONDATA-4035:
-
sql("drop table if exists special_char")
sql("create table special_char(`i#d` string, `nam(e` string,`ci)&#@!ty`
string,`a\be` int, `ag!e` float, `na^me1` Decimal(8,4), ```a``bc``!!d``` int)
stored as carbondata" +
" tblproperties('INVERTED_INDEX'='`a`bc`!!d`',
'SORT_COLUMNS'='`a`bc`!!d`')")
> MV table is not hit when sum() is applied on decimal column.
>
>
> Key: CARBONDATA-4035
> URL: https://issues.apache.org/jira/browse/CARBONDATA-4035
> Project: CarbonData
> Issue Type: Bug
>Reporter: Akash R Nilugal
>Assignee: Akash R Nilugal
>Priority: Minor
>
> MV table is not hit when sum() is applied on decimal column.
> sql("drop table if exists sum_agg_decimal")
> sql("create table sum_agg_decimal(salary1 decimal(7,2),salary2
> decimal(7,2),salary3 decimal(7,2),salary4 decimal(7,2),empname string) stored
> as carbondata")
> sql("drop materialized view if exists decimal_mv")
> sql("create materialized view decimal_mv as select empname, sum(salary1 -
> salary2) from sum_agg_decimal group by empname")
> sql("explain select empname, sum( salary1 - salary2) from sum_agg_decimal
> group by empname").show(false)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Issue Comment Deleted] (CARBONDATA-4035) MV table is not hit when sum() is applied on decimal column.
[
https://issues.apache.org/jira/browse/CARBONDATA-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akash R Nilugal updated CARBONDATA-4035:
Comment: was deleted
(was: sql("drop table if exists special_char")
sql("create table special_char(`i#d` string, `nam(e` string,`ci)&#@!ty`
string,`a\be` int, `ag!e` float, `na^me1` Decimal(8,4), ```a``bc``!!d``` int)
stored as carbondata" +
" tblproperties('INVERTED_INDEX'='`a`bc`!!d`',
'SORT_COLUMNS'='`a`bc`!!d`')"))
> MV table is not hit when sum() is applied on decimal column.
>
>
> Key: CARBONDATA-4035
> URL: https://issues.apache.org/jira/browse/CARBONDATA-4035
> Project: CarbonData
> Issue Type: Bug
>Reporter: Akash R Nilugal
>Assignee: Akash R Nilugal
>Priority: Minor
>
> MV table is not hit when sum() is applied on decimal column.
> sql("drop table if exists sum_agg_decimal")
> sql("create table sum_agg_decimal(salary1 decimal(7,2),salary2
> decimal(7,2),salary3 decimal(7,2),salary4 decimal(7,2),empname string) stored
> as carbondata")
> sql("drop materialized view if exists decimal_mv")
> sql("create materialized view decimal_mv as select empname, sum(salary1 -
> salary2) from sum_agg_decimal group by empname")
> sql("explain select empname, sum( salary1 - salary2) from sum_agg_decimal
> group by empname").show(false)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [carbondata] kunal642 commented on pull request #3914: [CARBONDATA-3979] Added Hive local dictionary support example
kunal642 commented on pull request #3914: URL: https://github.com/apache/carbondata/pull/3914#issuecomment-709969962 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (CARBONDATA-4035) MV table is not hit when sum() is applied on decimal column.
Akash R Nilugal created CARBONDATA-4035:
---
Summary: MV table is not hit when sum() is applied on decimal
column.
Key: CARBONDATA-4035
URL: https://issues.apache.org/jira/browse/CARBONDATA-4035
Project: CarbonData
Issue Type: Bug
Reporter: Akash R Nilugal
Assignee: Akash R Nilugal
MV table is not hit when sum() is applied on decimal column.
sql("drop table if exists sum_agg_decimal")
sql("create table sum_agg_decimal(salary1 decimal(7,2),salary2
decimal(7,2),salary3 decimal(7,2),salary4 decimal(7,2),empname string) stored
as carbondata")
sql("drop materialized view if exists decimal_mv")
sql("create materialized view decimal_mv as select empname, sum(salary1 -
salary2) from sum_agg_decimal group by empname")
sql("explain select empname, sum( salary1 - salary2) from sum_agg_decimal
group by empname").show(false)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [carbondata] nihal0107 removed a comment on pull request #3985: [CARBONDATA-3965]Fixed float variable target datatype in case of adaptive encoding
nihal0107 removed a comment on pull request #3985: URL: https://github.com/apache/carbondata/pull/3985#issuecomment-709897532 retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] marchpure commented on a change in pull request #3986: [CARBONDATA-4034] Improve the time-consuming of Horizontal Compaction for update
marchpure commented on a change in pull request #3986:
URL: https://github.com/apache/carbondata/pull/3986#discussion_r506261693
##
File path:
processing/src/main/java/org/apache/carbondata/processing/merger/CarbonDataMergerUtil.java
##
@@ -1246,8 +1209,22 @@ public static boolean isHorizontalCompactionEnabled() {
// set the update status.
segmentUpdateStatusManager.setUpdateStatusDetails(segmentUpdateDetails);
-CarbonFile[] deleteDeltaFiles =
-segmentUpdateStatusManager.getDeleteDeltaFilesList(new Segment(seg),
blockName);
+// only when SegmentUpdateDetails contain the specified block
+// will the method getDeleteDeltaFilesList be executed
+List blockNameList =
segmentUpdateStatusManager.getBlockNameFromSegment(seg);
+Map> blockAndDeleteDeltaFilesMap = new
HashMap<>();
+CarbonFile[] deleteDeltaFiles = null;
+if (blockNameList.contains(blockName)) {
+ blockAndDeleteDeltaFilesMap =
+ segmentUpdateStatusManager.getDeleteDeltaFilesList(new Segment(seg));
+}
+if (blockAndDeleteDeltaFilesMap.containsKey(blockName)) {
+ List deleteDeltaFileList =
blockAndDeleteDeltaFilesMap.get(blockName);
+ deleteDeltaFiles = deleteDeltaFileList.toArray(new
CarbonFile[deleteDeltaFileList.size()]);
+}
+
+// CarbonFile[] deleteDeltaFiles =
Review comment:
delete these 2 lines
##
File path:
processing/src/main/java/org/apache/carbondata/processing/merger/CarbonDataMergerUtil.java
##
@@ -1246,8 +1209,22 @@ public static boolean isHorizontalCompactionEnabled() {
// set the update status.
segmentUpdateStatusManager.setUpdateStatusDetails(segmentUpdateDetails);
-CarbonFile[] deleteDeltaFiles =
-segmentUpdateStatusManager.getDeleteDeltaFilesList(new Segment(seg),
blockName);
+// only when SegmentUpdateDetails contain the specified block
+// will the method getDeleteDeltaFilesList be executed
+List blockNameList =
segmentUpdateStatusManager.getBlockNameFromSegment(seg);
+Map> blockAndDeleteDeltaFilesMap = new
HashMap<>();
+CarbonFile[] deleteDeltaFiles = null;
Review comment:
= new CarbonFile[0]
##
File path:
processing/src/main/java/org/apache/carbondata/processing/merger/CarbonDataMergerUtil.java
##
@@ -1246,8 +1209,22 @@ public static boolean isHorizontalCompactionEnabled() {
// set the update status.
segmentUpdateStatusManager.setUpdateStatusDetails(segmentUpdateDetails);
-CarbonFile[] deleteDeltaFiles =
-segmentUpdateStatusManager.getDeleteDeltaFilesList(new Segment(seg),
blockName);
+// only when SegmentUpdateDetails contain the specified block
+// will the method getDeleteDeltaFilesList be executed
+List blockNameList =
segmentUpdateStatusManager.getBlockNameFromSegment(seg);
+Map> blockAndDeleteDeltaFilesMap = new
HashMap<>();
+CarbonFile[] deleteDeltaFiles = null;
+if (blockNameList.contains(blockName)) {
Review comment:
if (blockNameList.contains(blockName)) {
blockAndDeleteDeltaFilesMap =
if (blockAndDeleteDeltaFilesMap.containsKey(blockName)) {
}
}
##
File path:
processing/src/main/java/org/apache/carbondata/processing/merger/CarbonDataMergerUtil.java
##
@@ -1246,8 +1209,22 @@ public static boolean isHorizontalCompactionEnabled() {
// set the update status.
segmentUpdateStatusManager.setUpdateStatusDetails(segmentUpdateDetails);
-CarbonFile[] deleteDeltaFiles =
-segmentUpdateStatusManager.getDeleteDeltaFilesList(new Segment(seg),
blockName);
+// only when SegmentUpdateDetails contain the specified block
+// will the method getDeleteDeltaFilesList be executed
+List blockNameList =
segmentUpdateStatusManager.getBlockNameFromSegment(seg);
+Map> blockAndDeleteDeltaFilesMap = new
HashMap<>();
+CarbonFile[] deleteDeltaFiles = null;
+if (blockNameList.contains(blockName)) {
+ blockAndDeleteDeltaFilesMap =
+ segmentUpdateStatusManager.getDeleteDeltaFilesList(new Segment(seg));
+}
+if (blockAndDeleteDeltaFilesMap.containsKey(blockName)) {
Review comment:
no need to covert to array
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] nihal0107 commented on pull request #3985: [CARBONDATA-3965]Fixed float variable target datatype in case of adaptive encoding
nihal0107 commented on pull request #3985: URL: https://github.com/apache/carbondata/pull/3985#issuecomment-709957980 retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] nihal0107 removed a comment on pull request #3985: [CARBONDATA-3965]Fixed float variable target datatype in case of adaptive encoding
nihal0107 removed a comment on pull request #3985: URL: https://github.com/apache/carbondata/pull/3985#issuecomment-709745588 retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3695: [WIP] partition optimization
CarbonDataQA1 commented on pull request #3695: URL: https://github.com/apache/carbondata/pull/3695#issuecomment-709940050 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2728/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3695: [WIP] partition optimization
CarbonDataQA1 commented on pull request #3695: URL: https://github.com/apache/carbondata/pull/3695#issuecomment-709938508 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4482/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] shenjiayu17 commented on a change in pull request #3986: [CARBONDATA-4034] Improve the time-consuming of Horizontal Compaction for update
shenjiayu17 commented on a change in pull request #3986:
URL: https://github.com/apache/carbondata/pull/3986#discussion_r506203367
##
File path:
core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentUpdateStatusManager.java
##
@@ -455,6 +455,51 @@ public boolean accept(CarbonFile pathName) {
return null;
}
+ public Map> getDeleteDeltaFilesForSegment(final
Segment seg) {
+String segmentPath = CarbonTablePath.getSegmentPath(
+ identifier.getTablePath(), seg.getSegmentNo());
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] shenjiayu17 commented on a change in pull request #3986: [CARBONDATA-4034] Improve the time-consuming of Horizontal Compaction for update
shenjiayu17 commented on a change in pull request #3986:
URL: https://github.com/apache/carbondata/pull/3986#discussion_r506202461
##
File path:
core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentUpdateStatusManager.java
##
@@ -455,6 +455,51 @@ public boolean accept(CarbonFile pathName) {
return null;
}
+ public Map> getDeleteDeltaFilesForSegment(final
Segment seg) {
Review comment:
Done.
Keep function name getDeleteDeltaFilesList and change the return type to
Map>
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] shenjiayu17 commented on a change in pull request #3986: [CARBONDATA-4034] Improve the time-consuming of Horizontal Compaction for update
shenjiayu17 commented on a change in pull request #3986:
URL: https://github.com/apache/carbondata/pull/3986#discussion_r506020436
##
File path:
processing/src/main/java/org/apache/carbondata/processing/merger/CarbonDataMergerUtil.java
##
@@ -1039,22 +1039,10 @@ private static boolean
isSegmentValid(LoadMetadataDetails seg) {
if (CompactionType.IUD_DELETE_DELTA == compactionTypeIUD) {
int numberDeleteDeltaFilesThreshold =
CarbonProperties.getInstance().getNoDeleteDeltaFilesThresholdForIUDCompaction();
- List deleteSegments = new ArrayList<>();
for (Segment seg : segments) {
-if (checkDeleteDeltaFilesInSeg(seg, segmentUpdateStatusManager,
Review comment:
Done.
Combined function checkDeleteDeltaFilesInSeg and function
getDeleteDeltaFilesInSeg to new function checkAndGetDeleteDeltaFilesInSeg
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] shenjiayu17 commented on a change in pull request #3986: [CARBONDATA-4034] Improve the time-consuming of Horizontal Compaction for update
shenjiayu17 commented on a change in pull request #3986:
URL: https://github.com/apache/carbondata/pull/3986#discussion_r506020781
##
File path:
processing/src/main/java/org/apache/carbondata/processing/merger/CarbonDataMergerUtil.java
##
@@ -1210,6 +1198,39 @@ private static boolean
checkDeleteDeltaFilesInSeg(Segment seg,
return blockLists;
}
+ private static List checkAndGetDeleteDeltaFilesInSeg(Segment seg,
+ SegmentUpdateStatusManager segmentUpdateStatusManager, int
numberDeltaFilesThreshold) {
+
+List blockLists = new ArrayList<>();
+
+Map> blockAndDeleteDeltaFilesMap =
+ segmentUpdateStatusManager.getDeleteDeltaFilesForSegment(seg);
+
+List blockNameList =
+ segmentUpdateStatusManager.getBlockNameFromSegment(seg.getSegmentNo());
+
+Set uniqueBlocks = new HashSet();
+for (final String blockName : blockNameList) {
+
+ List deleteDeltaFiles =
blockAndDeleteDeltaFilesMap.get(blockName);
+
+ if (null != deleteDeltaFiles) {
+for (CarbonFile blocks : deleteDeltaFiles) {
Review comment:
Done.
Added judgement:
if (deleteDeltaFiles.size() <= numberDeltaFilesThreshold) continue
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3914: [CARBONDATA-3979] Added Hive local dictionary support example
CarbonDataQA1 commented on pull request #3914: URL: https://github.com/apache/carbondata/pull/3914#issuecomment-709918196 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4481/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3914: [CARBONDATA-3979] Added Hive local dictionary support example
CarbonDataQA1 commented on pull request #3914: URL: https://github.com/apache/carbondata/pull/3914#issuecomment-709903460 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2727/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] nihal0107 commented on pull request #3985: [CARBONDATA-3965]Fixed float variable target datatype in case of adaptive encoding
nihal0107 commented on pull request #3985: URL: https://github.com/apache/carbondata/pull/3985#issuecomment-709897532 retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3985: [CARBONDATA-3965]Fixed float variable target datatype in case of adaptive encoding
CarbonDataQA1 commented on pull request #3985: URL: https://github.com/apache/carbondata/pull/3985#issuecomment-709896788 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2726/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3948: [HOTFIX] Fix random 11 testcase failure in CI
CarbonDataQA1 commented on pull request #3948: URL: https://github.com/apache/carbondata/pull/3948#issuecomment-709870946 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4479/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3948: [HOTFIX] Fix random 11 testcase failure in CI
CarbonDataQA1 commented on pull request #3948: URL: https://github.com/apache/carbondata/pull/3948#issuecomment-709865455 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2725/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3789: [CARBONDATA-3864] Store Size Optimization
CarbonDataQA1 commented on pull request #3789: URL: https://github.com/apache/carbondata/pull/3789#issuecomment-709862439 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2724/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3789: [CARBONDATA-3864] Store Size Optimization
CarbonDataQA1 commented on pull request #3789: URL: https://github.com/apache/carbondata/pull/3789#issuecomment-709861587 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4478/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
