[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3848: [CARBONDATA-3891] Fix loading data will update all segments updateDeltaEndTimestamp
ajantha-bhat commented on a change in pull request #3848:
URL: https://github.com/apache/carbondata/pull/3848#discussion_r489988970
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/segment/ShowSegmentTestCase.scala
##
@@ -224,6 +224,37 @@ class ShowSegmentTestCase extends QueryTest with
BeforeAndAfterAll {
sql("drop table if exists a")
}
+ test("test loading data into partitioned table with segment's
updateDeltaEndTimestamp not change") {
Review comment:
Thanks for working on it. Just a small suggestion. This showSegment
testcase is meant for only segment display related test case.
can you move this test case to existing `InsertIntoCarbonTableTestCase` file
?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3935: [CARBONDATA-3993] Remove deletePartialLoadData in data loading process
CarbonDataQA1 commented on pull request #3935: URL: https://github.com/apache/carbondata/pull/3935#issuecomment-693794609 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2370/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3935: [CARBONDATA-3993] Remove deletePartialLoadData in data loading process
CarbonDataQA1 commented on pull request #3935: URL: https://github.com/apache/carbondata/pull/3935#issuecomment-693793107 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4112/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3928: [CARBONDATA-3990] Fix DropCache log error when indexmap is null on SI table
CarbonDataQA1 commented on pull request #3928: URL: https://github.com/apache/carbondata/pull/3928#issuecomment-693634493 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2367/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3934: [WIP] Support Global Unique Id for SegmentNo
CarbonDataQA1 commented on pull request #3934: URL: https://github.com/apache/carbondata/pull/3934#issuecomment-693598913 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4111/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3934: [WIP] Support Global Unique Id for SegmentNo
CarbonDataQA1 commented on pull request #3934: URL: https://github.com/apache/carbondata/pull/3934#issuecomment-693592854 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2369/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3928: [CARBONDATA-3990] Fix DropCache log error when indexmap is null on SI table
CarbonDataQA1 commented on pull request #3928: URL: https://github.com/apache/carbondata/pull/3928#issuecomment-693571957 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4109/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on pull request #3928: [CARBONDATA-3990] Fix DropCache log error when indexmap is null on SI table
Indhumathi27 commented on pull request #3928: URL: https://github.com/apache/carbondata/pull/3928#issuecomment-693567094 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3932: [CARBONDATA-3994] Skip Order by for map task if it is sort column and use limit pushdown for array_contains filter
CarbonDataQA1 commented on pull request #3932: URL: https://github.com/apache/carbondata/pull/3932#issuecomment-693522190 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2366/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3932: [CARBONDATA-3994] Skip Order by for map task if it is sort column and use limit pushdown for array_contains filter
CarbonDataQA1 commented on pull request #3932: URL: https://github.com/apache/carbondata/pull/3932#issuecomment-693520481 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4108/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3930: [CARBONDATA-3991]Fix the set modified time function on S3 and Alluxio…
CarbonDataQA1 commented on pull request #3930: URL: https://github.com/apache/carbondata/pull/3930#issuecomment-693448164 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2361/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] asfgit closed pull request #3927: [CARBONDATA-3989]Fix unnecessary segment files creation when segment is neither updated nor deleted.
asfgit closed pull request #3927: URL: https://github.com/apache/carbondata/pull/3927 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (CARBONDATA-3989) Unnecessary segment files are created even when the segments are neither updated nor deleted
[ https://issues.apache.org/jira/browse/CARBONDATA-3989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajantha Bhat resolved CARBONDATA-3989. -- Fix Version/s: 2.1.0 Resolution: Fixed > Unnecessary segment files are created even when the segments are neither > updated nor deleted > > > Key: CARBONDATA-3989 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3989 > Project: CarbonData > Issue Type: Bug >Reporter: Akash R Nilugal >Assignee: Akash R Nilugal >Priority: Minor > Fix For: 2.1.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Unnecessary segment files are created even when the segments are neither > updated nor deleted -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3930: [CARBONDATA-3991]Fix the set modified time function on S3 and Alluxio…
CarbonDataQA1 commented on pull request #3930: URL: https://github.com/apache/carbondata/pull/3930#issuecomment-693446768 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4103/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on pull request #3932: [CARBONDATA-3994] Skip Order by for map task if it is sort column and use limit pushdown for array_contains filter
ajantha-bhat commented on pull request #3932: URL: https://github.com/apache/carbondata/pull/3932#issuecomment-693445107 @QiangCai , @li36909 please check This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (CARBONDATA-3994) Skip Order by for map task if it is sort column and use limit pushdown for array_contains filter
Ajantha Bhat created CARBONDATA-3994:
Summary: Skip Order by for map task if it is sort column and use
limit pushdown for array_contains filter
Key: CARBONDATA-3994
URL: https://issues.apache.org/jira/browse/CARBONDATA-3994
Project: CarbonData
Issue Type: Bug
Reporter: Ajantha Bhat
Assignee: Ajantha Bhat
When the order by column is in sort column, every map task output will be
already sorted. No need to sort the data again.
Hence skipping the order at map task by changing plan node from
{{TakeOrderedAndProject}} --> {{CarbonTakeOrderedAndProjectExec}}
Also in this scenario collecting the limit at map task and Array_contains()
will use this limit value for row scan filtering to break scan once limit value
is reached.
Also added a carbon property to control this .
{{carbon.mapOrderPushDown._.column}}
Note: later we can improve this for other filters also to use the limit value.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3934: [WIP] Support Global Unique Id for SegmentNo
CarbonDataQA1 commented on pull request #3934: URL: https://github.com/apache/carbondata/pull/3934#issuecomment-693425427 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4106/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3934: [WIP] Support Global Unique Id for SegmentNo
CarbonDataQA1 commented on pull request #3934: URL: https://github.com/apache/carbondata/pull/3934#issuecomment-693423909 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2364/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3934: [WIP] Support Global Unique Id for SegmentNo
CarbonDataQA1 commented on pull request #3934: URL: https://github.com/apache/carbondata/pull/3934#issuecomment-693402190 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4105/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3934: [WIP] Support Global Unique Id for SegmentNo
CarbonDataQA1 commented on pull request #3934: URL: https://github.com/apache/carbondata/pull/3934#issuecomment-693400949 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2363/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3935: [CARBONDATA-3993] Remove deletePartialLoadData in data loading process
CarbonDataQA1 commented on pull request #3935: URL: https://github.com/apache/carbondata/pull/3935#issuecomment-693391877 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4104/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3935: [CARBONDATA-3993] Remove deletePartialLoadData in data loading process
CarbonDataQA1 commented on pull request #3935: URL: https://github.com/apache/carbondata/pull/3935#issuecomment-693391021 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2362/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3928: [CARBONDATA-3990] Fix DropCache log error when indexmap is null on SI table
CarbonDataQA1 commented on pull request #3928: URL: https://github.com/apache/carbondata/pull/3928#issuecomment-693388210 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2359/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] jackylk commented on pull request #3935: [CARBONDATA-3993] Remove deletePartialLoadData in data loading process
jackylk commented on pull request #3935: URL: https://github.com/apache/carbondata/pull/3935#issuecomment-693387784 add to whitelist This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3932: [WIP] order by limit pushdown for array_contains filter
CarbonDataQA1 commented on pull request #3932: URL: https://github.com/apache/carbondata/pull/3932#issuecomment-693381520 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3927: [CARBONDATA-3989]Fix unnecessary segment files creation when segment is neither updated nor deleted.
CarbonDataQA1 commented on pull request #3927: URL: https://github.com/apache/carbondata/pull/3927#issuecomment-693363425 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2356/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3927: [CARBONDATA-3989]Fix unnecessary segment files creation when segment is neither updated nor deleted.
CarbonDataQA1 commented on pull request #3927: URL: https://github.com/apache/carbondata/pull/3927#issuecomment-693362694 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4097/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3934: [WIP] Support Global Unique Id for SegmentNo
CarbonDataQA1 commented on pull request #3934: URL: https://github.com/apache/carbondata/pull/3934#issuecomment-693353041 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4102/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3934: [WIP] Support Global Unique Id for SegmentNo
CarbonDataQA1 commented on pull request #3934: URL: https://github.com/apache/carbondata/pull/3934#issuecomment-693352561 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2360/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3935: [CARBONDATA-3993] Remove deletePartialLoadData in data loading process
CarbonDataQA1 commented on pull request #3935: URL: https://github.com/apache/carbondata/pull/3935#issuecomment-693346444 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Pickupolddriver opened a new pull request #3935: [CARBONDATA-3993] Remove deletePartialLoadData in data loading process
Pickupolddriver opened a new pull request #3935: URL: https://github.com/apache/carbondata/pull/3935 ### Why is this PR needed? Currently, in data management scenarios(Data Loading, Segments Compaction.etc), there exist some data deletion actions. And these actions are dangerous because they are written in different places and some corner cases will cause data deletion accidentally. ### What changes were proposed in this PR? This PR remove deleting INSERT_IN_PROGRESS and INSERT_OVERWRITE_IN_PROGRESS status segments during loading and compaction flow. ### Does this PR introduce any user interface change? - No ### Is any new testcase added? - No This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (CARBONDATA-3993) Remove data deletion action during loading and compaction flow
[ https://issues.apache.org/jira/browse/CARBONDATA-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianxi Li updated CARBONDATA-3993: -- Description: Currently, in data management scenarios(Data Loading, Segments Compaction.etc), there exist some data deletion actions. And these actions are dangerous because they are written in different places and some corner cases will cause data deletion accidentally. In this PR, it removes the stale data deletion in loading and compaction flow. And will ignore the in-process segments during updating the metadata process. Summary: Remove data deletion action during loading and compaction flow (was: HotFix) > Remove data deletion action during loading and compaction flow > -- > > Key: CARBONDATA-3993 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3993 > Project: CarbonData > Issue Type: Bug >Reporter: Jianxi Li >Priority: Major > > Currently, in data management scenarios(Data Loading, Segments > Compaction.etc), there exist some data deletion actions. And these actions > are dangerous because they are written in different places and some corner > cases will cause data deletion accidentally. > > In this PR, it removes the stale data deletion in loading and compaction > flow. And will ignore the in-process segments during updating the metadata > process. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] Pickupolddriver closed pull request #3933: Remove deletePartialLoadData command in loading and compaction process
Pickupolddriver closed pull request #3933: URL: https://github.com/apache/carbondata/pull/3933 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (CARBONDATA-3993) HotFix
Jianxi Li created CARBONDATA-3993: - Summary: HotFix Key: CARBONDATA-3993 URL: https://issues.apache.org/jira/browse/CARBONDATA-3993 Project: CarbonData Issue Type: Bug Reporter: Jianxi Li -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] marchpure opened a new pull request #3934: [WIP] Support Global Unique Id for SegmentNo
marchpure opened a new pull request #3934: URL: https://github.com/apache/carbondata/pull/3934 Why is this PR needed? XXX What changes were proposed in this PR? XXX Does this PR introduce any user interface change? No Is any new testcase added? Yes ### Why is this PR needed? ### What changes were proposed in this PR? ### Does this PR introduce any user interface change? - No - Yes. (please explain the change and update document) ### Is any new testcase added? - No - Yes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3933: Remove deletePartialLoadData command in loading and compaction process
CarbonDataQA1 commented on pull request #3933: URL: https://github.com/apache/carbondata/pull/3933#issuecomment-693337455 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Pickupolddriver opened a new pull request #3933: Remove deletePartialLoadData command in loading and compaction process
Pickupolddriver opened a new pull request #3933: URL: https://github.com/apache/carbondata/pull/3933 ### Why is this PR needed? Currently, in data management scenarios(Data Loading, Segments Compaction.etc), there exist some data deletion actions. And these actions are dangerous because they are written in different places and some corner cases will cause data deletion accidentally. ### What changes were proposed in this PR? This PR remove deleting INSERT_IN_PROGRESS and INSERT_OVERWRITE_IN_PROGRESS status segments during loading and compaction flow. ### Does this PR introduce any user interface change? - No - Yes. (please explain the change and update document) ### Is any new testcase added? - No - Yes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3701: [CARBONDATA-3770] Improve partition count star query performance
CarbonDataQA1 commented on pull request #3701: URL: https://github.com/apache/carbondata/pull/3701#issuecomment-693335920 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2355/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3701: [CARBONDATA-3770] Improve partition count star query performance
CarbonDataQA1 commented on pull request #3701: URL: https://github.com/apache/carbondata/pull/3701#issuecomment-693335184 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4096/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3928: [CARBONDATA-3990] Fix DropCache log error when indexmap is null on SI table
CarbonDataQA1 commented on pull request #3928: URL: https://github.com/apache/carbondata/pull/3928#issuecomment-693334191 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4101/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat opened a new pull request #3932: [WIP] order by limit pushdown for array_contains filter
ajantha-bhat opened a new pull request #3932: URL: https://github.com/apache/carbondata/pull/3932 ### Why is this PR needed? ### What changes were proposed in this PR? ### Does this PR introduce any user interface change? - No - Yes. (please explain the change and update document) ### Is any new testcase added? - No - Yes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on pull request #3924: [CARBONDATA-3988] Allow SI creation on first dimension column
ajantha-bhat commented on pull request #3924: URL: https://github.com/apache/carbondata/pull/3924#issuecomment-693325597 @QiangCai : please check and merge this This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3930: [CARBONDATA-3991]Fix the set modified time function on S3 and Alluxio…
CarbonDataQA1 commented on pull request #3930: URL: https://github.com/apache/carbondata/pull/3930#issuecomment-693314372 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4094/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3930: [CARBONDATA-3991]Fix the set modified time function on S3 and Alluxio…
CarbonDataQA1 commented on pull request #3930: URL: https://github.com/apache/carbondata/pull/3930#issuecomment-693313428 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2353/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on pull request #3927: [CARBONDATA-3989]Fix unnecessary segment files creation when segment is neither updated nor deleted.
ajantha-bhat commented on pull request #3927: URL: https://github.com/apache/carbondata/pull/3927#issuecomment-693310888 LGTM. will merge once the build passes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3927: [CARBONDATA-3989]Fix unnecessary segment files creation when segment is neither updated nor deleted.
akashrn5 commented on a change in pull request #3927:
URL: https://github.com/apache/carbondata/pull/3927#discussion_r489324392
##
File path:
integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
##
@@ -725,43 +725,53 @@ object CarbonDataRDDFactory {
val metadataDetails =
SegmentStatusManager.readTableStatusFile(
CarbonTablePath.getTableStatusFilePath(carbonTable.getTablePath))
+val updateTableStatusFile =
CarbonUpdateUtil.getUpdateStatusFileName(updateModel
+ .updatedTimeStamp.toString)
+val updatedSegments =
SegmentUpdateStatusManager.readLoadMetadata(updateTableStatusFile,
+ carbonTable.getTablePath).map(_.getSegmentName).toSet
val segmentFiles = segmentDetails.asScala.map { seg =>
- val load =
-metadataDetails.find(_.getLoadName.equals(seg.getSegmentNo)).get
- val segmentFile = load.getSegmentFile
- var segmentFiles: Seq[CarbonFile] = Seq.empty[CarbonFile]
+ // create new segment files and merge for only updated segments
+ if (updatedSegments.contains(seg.getSegmentNo)) {
+val load =
+ metadataDetails.find(_.getLoadName.equals(seg.getSegmentNo)).get
+val segmentFile = load.getSegmentFile
+var segmentFiles: Seq[CarbonFile] = Seq.empty[CarbonFile]
- val segmentMetaDataInfo = segmentMetaDataInfoMap.get(seg.getSegmentNo)
- val file = SegmentFileStore.writeSegmentFile(
-carbonTable,
-seg.getSegmentNo,
-String.valueOf(System.currentTimeMillis()),
-load.getPath,
-segmentMetaDataInfo)
+val segmentMetaDataInfo = segmentMetaDataInfoMap.get(seg.getSegmentNo)
+val file = SegmentFileStore.writeSegmentFile(
Review comment:
here method level comment says for new updates on the same segment we
need to write new segment file with new segmentMetaDataInfo and merge with old
segment file , i think that comment is pretty clear. It should be ok i think.
Please check once.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3927: [CARBONDATA-3989]Fix unnecessary segment files creation when segment is neither updated nor deleted.
akashrn5 commented on a change in pull request #3927:
URL: https://github.com/apache/carbondata/pull/3927#discussion_r489323476
##
File path:
integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
##
@@ -725,43 +725,53 @@ object CarbonDataRDDFactory {
val metadataDetails =
SegmentStatusManager.readTableStatusFile(
CarbonTablePath.getTableStatusFilePath(carbonTable.getTablePath))
+val updateTableStatusFile =
CarbonUpdateUtil.getUpdateStatusFileName(updateModel
+ .updatedTimeStamp.toString)
+val updatedSegments =
SegmentUpdateStatusManager.readLoadMetadata(updateTableStatusFile,
+ carbonTable.getTablePath).map(_.getSegmentName).toSet
val segmentFiles = segmentDetails.asScala.map { seg =>
- val load =
-metadataDetails.find(_.getLoadName.equals(seg.getSegmentNo)).get
- val segmentFile = load.getSegmentFile
- var segmentFiles: Seq[CarbonFile] = Seq.empty[CarbonFile]
+ // create new segment files and merge for only updated segments
Review comment:
changed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3931: [CARBONDATA-3992] Fixed null pointer exception for drop index.
CarbonDataQA1 commented on pull request #3931: URL: https://github.com/apache/carbondata/pull/3931#issuecomment-693303174 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4093/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3931: [CARBONDATA-3992] Fixed null pointer exception for drop index.
CarbonDataQA1 commented on pull request #3931: URL: https://github.com/apache/carbondata/pull/3931#issuecomment-693301736 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2352/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3927: [CARBONDATA-3989]Fix unnecessary segment files creation when segment is neither updated nor deleted.
ajantha-bhat commented on a change in pull request #3927:
URL: https://github.com/apache/carbondata/pull/3927#discussion_r489299596
##
File path:
integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
##
@@ -725,43 +725,53 @@ object CarbonDataRDDFactory {
val metadataDetails =
SegmentStatusManager.readTableStatusFile(
CarbonTablePath.getTableStatusFilePath(carbonTable.getTablePath))
+val updateTableStatusFile =
CarbonUpdateUtil.getUpdateStatusFileName(updateModel
+ .updatedTimeStamp.toString)
+val updatedSegments =
SegmentUpdateStatusManager.readLoadMetadata(updateTableStatusFile,
+ carbonTable.getTablePath).map(_.getSegmentName).toSet
val segmentFiles = segmentDetails.asScala.map { seg =>
- val load =
-metadataDetails.find(_.getLoadName.equals(seg.getSegmentNo)).get
- val segmentFile = load.getSegmentFile
- var segmentFiles: Seq[CarbonFile] = Seq.empty[CarbonFile]
+ // create new segment files and merge for only updated segments
Review comment:
here `seg`, `file`, `carbonFile` are not good variable names, you can
check and refactor if you want
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3927: [CARBONDATA-3989]Fix unnecessary segment files creation when segment is neither updated nor deleted.
ajantha-bhat commented on a change in pull request #3927:
URL: https://github.com/apache/carbondata/pull/3927#discussion_r489300387
##
File path:
integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
##
@@ -725,43 +725,53 @@ object CarbonDataRDDFactory {
val metadataDetails =
SegmentStatusManager.readTableStatusFile(
CarbonTablePath.getTableStatusFilePath(carbonTable.getTablePath))
+val updateTableStatusFile =
CarbonUpdateUtil.getUpdateStatusFileName(updateModel
+ .updatedTimeStamp.toString)
+val updatedSegments =
SegmentUpdateStatusManager.readLoadMetadata(updateTableStatusFile,
+ carbonTable.getTablePath).map(_.getSegmentName).toSet
val segmentFiles = segmentDetails.asScala.map { seg =>
- val load =
-metadataDetails.find(_.getLoadName.equals(seg.getSegmentNo)).get
- val segmentFile = load.getSegmentFile
- var segmentFiles: Seq[CarbonFile] = Seq.empty[CarbonFile]
+ // create new segment files and merge for only updated segments
+ if (updatedSegments.contains(seg.getSegmentNo)) {
+val load =
+ metadataDetails.find(_.getLoadName.equals(seg.getSegmentNo)).get
+val segmentFile = load.getSegmentFile
+var segmentFiles: Seq[CarbonFile] = Seq.empty[CarbonFile]
- val segmentMetaDataInfo = segmentMetaDataInfoMap.get(seg.getSegmentNo)
- val file = SegmentFileStore.writeSegmentFile(
-carbonTable,
-seg.getSegmentNo,
-String.valueOf(System.currentTimeMillis()),
-load.getPath,
-segmentMetaDataInfo)
+val segmentMetaDataInfo = segmentMetaDataInfoMap.get(seg.getSegmentNo)
+val file = SegmentFileStore.writeSegmentFile(
Review comment:
here writing segment file again to just update the current timestamp ?
you can add comment If possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3927: [CARBONDATA-3989]Fix unnecessary segment files creation when segment is neither updated nor deleted.
ajantha-bhat commented on a change in pull request #3927:
URL: https://github.com/apache/carbondata/pull/3927#discussion_r489299596
##
File path:
integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
##
@@ -725,43 +725,53 @@ object CarbonDataRDDFactory {
val metadataDetails =
SegmentStatusManager.readTableStatusFile(
CarbonTablePath.getTableStatusFilePath(carbonTable.getTablePath))
+val updateTableStatusFile =
CarbonUpdateUtil.getUpdateStatusFileName(updateModel
+ .updatedTimeStamp.toString)
+val updatedSegments =
SegmentUpdateStatusManager.readLoadMetadata(updateTableStatusFile,
+ carbonTable.getTablePath).map(_.getSegmentName).toSet
val segmentFiles = segmentDetails.asScala.map { seg =>
- val load =
-metadataDetails.find(_.getLoadName.equals(seg.getSegmentNo)).get
- val segmentFile = load.getSegmentFile
- var segmentFiles: Seq[CarbonFile] = Seq.empty[CarbonFile]
+ // create new segment files and merge for only updated segments
Review comment:
I thought you will strictly follow to clean up the base code in the area
of modification to make it more readable and maintainable!
here `seg`, `file`, `carbonFile` are not good variable names, you can check
and refactor if you want
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3701: [CARBONDATA-3770] Improve partition count star query performance
CarbonDataQA1 commented on pull request #3701: URL: https://github.com/apache/carbondata/pull/3701#issuecomment-693271356 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4095/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3701: [CARBONDATA-3770] Improve partition count star query performance
CarbonDataQA1 commented on pull request #3701: URL: https://github.com/apache/carbondata/pull/3701#issuecomment-693270961 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2354/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] nihal0107 closed pull request #3931: [CARBONDATA-3992] Fixed null pointer exception for drop index.
nihal0107 closed pull request #3931: URL: https://github.com/apache/carbondata/pull/3931 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (CARBONDATA-3984) compaction on table having range column after altering data type from string to long string fails.
[ https://issues.apache.org/jira/browse/CARBONDATA-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akash R Nilugal resolved CARBONDATA-3984. - Fix Version/s: 2.1.0 Resolution: Fixed > compaction on table having range column after altering data type from string > to long string fails. > -- > > Key: CARBONDATA-3984 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3984 > Project: CarbonData > Issue Type: Bug > Components: core, spark-integration >Affects Versions: 2.0.0 >Reporter: Karan >Priority: Major > Fix For: 2.1.0 > > Time Spent: 3h 50m > Remaining Estimate: 0h > > When dataType of a String column which is also provided as range column in > table properties is altered to longStringColumn. It shows following error > while performing compaction on the table. > > VARCHAR not supported for the filter expression; at > org.apache.spark.sql.util.CarbonException$.analysisException > (CarbonException.scala: 23) at > org.apache.carbondata.spark.rdd.CarbonMergerRDD $$ anon $ 1. > (CarbonMergerRDD.scala: 227) at > org.apache.carbondata.spark.rdd.CarbonMergerRDD.internalCompute ( > CarbonMergerRDD.scala: 104) at > org.apache.carbondata.spark.rdd.CarbonRDD.compute -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] asfgit closed pull request #3923: [CARBONDATA-3984] Fix compaction and longStringColumn validation issue.
asfgit closed pull request #3923: URL: https://github.com/apache/carbondata/pull/3923 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] nihal0107 opened a new pull request #3931: [CARBONDATA-3992] Fixed null pointer exception for drop index.
nihal0107 opened a new pull request #3931: URL: https://github.com/apache/carbondata/pull/3931 ### Why is this PR needed? Set the index server as true but index server is not running. In this case drop index is throwing null pointer. ### What changes were proposed in this PR? Added a null check for index map. ### Does this PR introduce any user interface change? - No ### Is any new testcase added? - No This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on pull request #3923: [CARBONDATA-3984] Fix compaction and longStringColumn validation issue.
akashrn5 commented on pull request #3923: URL: https://github.com/apache/carbondata/pull/3923#issuecomment-693239175 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3923: [CARBONDATA-3984] Fix compaction and longStringColumn validation issue.
CarbonDataQA1 commented on pull request #3923: URL: https://github.com/apache/carbondata/pull/3923#issuecomment-693238183 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2351/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3923: [CARBONDATA-3984] Fix compaction and longStringColumn validation issue.
CarbonDataQA1 commented on pull request #3923: URL: https://github.com/apache/carbondata/pull/3923#issuecomment-693236971 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4092/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (CARBONDATA-3992) Drop Index is throwing null pointer exception.
Nihal kumar ojha created CARBONDATA-3992: Summary: Drop Index is throwing null pointer exception. Key: CARBONDATA-3992 URL: https://issues.apache.org/jira/browse/CARBONDATA-3992 Project: CarbonData Issue Type: Bug Reporter: Nihal kumar ojha Index server set to true but index server is not running. Create an index as 'carbondata' and try to drop the index -> throwing null pointer exception. IndexStoreMandaer.Java -> line 98 -- This message was sent by Atlassian Jira (v8.3.4#803005)
