[jira] [Created] (CARBONDATA-3861) Support show stage information
Zhi Liu created CARBONDATA-3861: --- Summary: Support show stage information Key: CARBONDATA-3861 URL: https://issues.apache.org/jira/browse/CARBONDATA-3861 Project: CarbonData Issue Type: New Feature Reporter: Zhi Liu Sometimes, user need to know information about table stages, so that they can make stage load plan better. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3796: [WIP] Lock and retry to read tablestatus before throwing …
CarbonDataQA1 commented on pull request #3796: URL: https://github.com/apache/carbondata/pull/3796#issuecomment-647097497 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3181/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3796: [WIP] Lock and retry to read tablestatus before throwing …
CarbonDataQA1 commented on pull request #3796: URL: https://github.com/apache/carbondata/pull/3796#issuecomment-647097624 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1455/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3798: [WIP] Support 'show segment include stage'
CarbonDataQA1 commented on pull request #3798: URL: https://github.com/apache/carbondata/pull/3798#issuecomment-647168625 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1458/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3797: [WIP] Support show segment information
CarbonDataQA1 commented on pull request #3797: URL: https://github.com/apache/carbondata/pull/3797#issuecomment-647146761 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1457/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (CARBONDATA-3862) Insert stage performance optimazation
Xingjun Hao created CARBONDATA-3862: --- Summary: Insert stage performance optimazation Key: CARBONDATA-3862 URL: https://issues.apache.org/jira/browse/CARBONDATA-3862 Project: CarbonData Issue Type: New Feature Reporter: Xingjun Hao There are two major performance bottlenecks of insert stage. 1) Get LastModifyTime of stagefiles requires a lot of access to OBS 2) Parallelism is not supported Which shall be optimazed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] marchpure opened a new pull request #3800: [WIP] Lock to read tablestatus
marchpure opened a new pull request #3800: URL: https://github.com/apache/carbondata/pull/3800 Why is this PR needed? when storing table status file in object store, reading of table status file mayfail (receive EOFException or JsonSyntaxException) when table status file is being modifying we shall retry add the lock to read tablestatus before throwing EOFException or JsonSyntaxException What changes were proposed in this PR? Add lock to read tablestatus Does this PR introduce any user interface change? NO Is any new testcase added? NO ### Why is this PR needed? ### What changes were proposed in this PR? ### Does this PR introduce any user interface change? - No - Yes. (please explain the change and update document) ### Is any new testcase added? - No - Yes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3798: [WIP] Support 'show segment include stage'
CarbonDataQA1 commented on pull request #3798: URL: https://github.com/apache/carbondata/pull/3798#issuecomment-647168876 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3184/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3797: [WIP] Support show segment information
CarbonDataQA1 commented on pull request #3797: URL: https://github.com/apache/carbondata/pull/3797#issuecomment-647146911 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3183/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3796: [CARBONDATA-3859] Retry to read tablestatus before throwing EOFException or JsonSyntaxException or FileNotFoundException
CarbonDataQA1 commented on pull request #3796: URL: https://github.com/apache/carbondata/pull/3796#issuecomment-647156457 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3182/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] marchpure opened a new pull request #3799: [CARBONDATA-3862] Insert stage performance optimazation
marchpure opened a new pull request #3799: URL: https://github.com/apache/carbondata/pull/3799 ### Why is this PR needed? There are two major performance bottlenecks of 'insert stage'. 1) Get LastModifyTime of stagefiles requires a lot of access to OBS. 2) Parallelism is not supported ### What changes were proposed in this PR? 1) Cache the lastmodifytime info when list stage files. 2) support insert stage in parallel. we add a tag 'loading' to the stages in process. different insertstage processes can load different data separately by choose the stages without 'loading' tag or stages loaded timeout. which avoid loading the same data between concurrent insertstage processes. The 'loading' tag is actually an empty file with '.loading' suffix filename. ### Does this PR introduce any user interface change? NO ### Is any new testcase added? YES This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] niuge01 opened a new pull request #3797: [WIP] Support show segment information
niuge01 opened a new pull request #3797: URL: https://github.com/apache/carbondata/pull/3797 ### Why is this PR needed? Sometimes, user need to know information about table stages, so that they can make stage load plan better. ### What changes were proposed in this PR? Add "INCLUDE STAGE" keywords in "SHOW SEGMENTS" command, show stage information as segment information ### Does this PR introduce any user interface change? - No - Yes. (please explain the change and update document) ### Is any new testcase added? - No - Yes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3796: [CARBONDATA-3859] Retry to read tablestatus before throwing EOFException or JsonSyntaxException
CarbonDataQA1 commented on pull request #3796: URL: https://github.com/apache/carbondata/pull/3796#issuecomment-647171957 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3185/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3796: [CARBONDATA-3859] Retry to read tablestatus before throwing EOFException or JsonSyntaxException
CarbonDataQA1 commented on pull request #3796: URL: https://github.com/apache/carbondata/pull/3796#issuecomment-647172066 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1459/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3799: [CARBONDATA-3862] Insert stage performance optimazation
CarbonDataQA1 commented on pull request #3799: URL: https://github.com/apache/carbondata/pull/3799#issuecomment-647196597 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3186/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (CARBONDATA-3863) index service go back to emmbedded mode
Taoli created CARBONDATA-3863: - Summary: index service go back to emmbedded mode Key: CARBONDATA-3863 URL: https://issues.apache.org/jira/browse/CARBONDATA-3863 Project: CarbonData Issue Type: Bug Affects Versions: 2.0.0 Reporter: Taoli when use index service,some way may cause the floder "/tmp/indexservertmp" get max-directory-item exception. in that case the index service go back to emmbedded mode. the error is like above: Exception occured: The directory item limit of /tmp/indexservertmp is exceeded: limit=1048576 items=1048576. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3799: [CARBONDATA-3862] Insert stage performance optimazation
CarbonDataQA1 commented on pull request #3799: URL: https://github.com/apache/carbondata/pull/3799#issuecomment-647197331 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1460/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3800: [WIP] Lock to read tablestatus
CarbonDataQA1 commented on pull request #3800: URL: https://github.com/apache/carbondata/pull/3800#issuecomment-647203937 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1461/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3800: [WIP] Lock to read tablestatus
CarbonDataQA1 commented on pull request #3800: URL: https://github.com/apache/carbondata/pull/3800#issuecomment-647204213 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3187/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] marchpure opened a new pull request #3798: [WIP] Support 'show segment include stage'
marchpure opened a new pull request #3798: URL: https://github.com/apache/carbondata/pull/3798 ### Why is this PR needed? ### What changes were proposed in this PR? ### Does this PR introduce any user interface change? - No - Yes. (please explain the change and update document) ### Is any new testcase added? - No - Yes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3796: [CARBONDATA-3859] Retry to read tablestatus before throwing EOFException or JsonSyntaxException
CarbonDataQA1 commented on pull request #3796: URL: https://github.com/apache/carbondata/pull/3796#issuecomment-647156922 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1456/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
