Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

danny0405 merged PR #9776: URL: https://github.com/apache/hudi/pull/9776 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

hudi-bot commented on PR #9776: URL: https://github.com/apache/hudi/pull/9776#issuecomment-1754574396 ## CI report: * 6b730068fa6ca60dfdd04f720334a49fa19a8b31 UNKNOWN * a8359503c149a51440d51243d117e50f88761935 Azure:

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

hudi-bot commented on PR #9776: URL: https://github.com/apache/hudi/pull/9776#issuecomment-1754481715 ## CI report: * 6b730068fa6ca60dfdd04f720334a49fa19a8b31 UNKNOWN * 3ded7c8973d785e4f7cec8e77393be3bed0699b7 Azure:

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

hudi-bot commented on PR #9776: URL: https://github.com/apache/hudi/pull/9776#issuecomment-1754405201 ## CI report: * 6b730068fa6ca60dfdd04f720334a49fa19a8b31 UNKNOWN * de2560dfe639845337ad6dd861e8cf980c2a3c3b Azure:

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

hudi-bot commented on PR #9776: URL: https://github.com/apache/hudi/pull/9776#issuecomment-1754354388 ## CI report: * 6b730068fa6ca60dfdd04f720334a49fa19a8b31 UNKNOWN * de2560dfe639845337ad6dd861e8cf980c2a3c3b Azure:

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

hudi-bot commented on PR #9776: URL: https://github.com/apache/hudi/pull/9776#issuecomment-1754306053 ## CI report: * 6b730068fa6ca60dfdd04f720334a49fa19a8b31 UNKNOWN * de2560dfe639845337ad6dd861e8cf980c2a3c3b Azure:

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

hudi-bot commented on PR #9776: URL: https://github.com/apache/hudi/pull/9776#issuecomment-1754300514 ## CI report: * 6b730068fa6ca60dfdd04f720334a49fa19a8b31 UNKNOWN * de2560dfe639845337ad6dd861e8cf980c2a3c3b Azure:

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

hudi-bot commented on PR #9776: URL: https://github.com/apache/hudi/pull/9776#issuecomment-1754233082 ## CI report: * 6b730068fa6ca60dfdd04f720334a49fa19a8b31 UNKNOWN * 2b284abe79624c633d78e9f16e66dab74bc45329 Azure:

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

hudi-bot commented on PR #9776: URL: https://github.com/apache/hudi/pull/9776#issuecomment-1754226799 ## CI report: * 6b730068fa6ca60dfdd04f720334a49fa19a8b31 UNKNOWN * 2b284abe79624c633d78e9f16e66dab74bc45329 Azure:

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1351187636 ## hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java: ## @@ -216,6 +233,9 @@ protected List buildFileGroups(Stream baseFileS

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1351187636 ## hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java: ## @@ -216,6 +233,9 @@ protected List buildFileGroups(Stream baseFileS

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1351183761 ## hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/rollback/TestMergeOnReadRollbackActionExecutor.java: ## @@ -126,21 +126,19 @@ public void

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1351135624 ## hudi-common/src/main/java/org/apache/hudi/common/table/timeline/CompletionTimeQueryView.java: ## @@ -16,27 +16,33 @@ * limitations under the License. */

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1351133865 ## hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java: ## @@ -544,15 +543,40 @@ public static Option> getLatestLogVersion(FileSystem fs, P } /**

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

codope commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1350025100 ## hudi-common/src/main/java/org/apache/hudi/common/table/timeline/CompletionTimeQueryView.java: ## @@ -16,27 +16,33 @@ * limitations under the License. */ -package

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

codope commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1350024714 ## hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/upgrade/TestUpgradeDowngrade.java: ## @@ -844,6 +847,47 @@ private Pair, List>

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

codope commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1350023322 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/plan/generators/BaseHoodieCompactionPlanGenerator.java: ## @@ -118,8 +120,21 @@ public

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

codope commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1350022791 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java: ## @@ -726,19 +727,24 @@ protected void

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

codope commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349974150 ## hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java: ## @@ -216,6 +233,9 @@ protected List buildFileGroups(Stream baseFileS

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

codope commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349972699 ## hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/rollback/TestMergeOnReadRollbackActionExecutor.java: ## @@ -126,21 +126,19 @@ public void

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

hudi-bot commented on PR #9776: URL: https://github.com/apache/hudi/pull/9776#issuecomment-1751662286 ## CI report: * 6b730068fa6ca60dfdd04f720334a49fa19a8b31 UNKNOWN * 2b284abe79624c633d78e9f16e66dab74bc45329 Azure:

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

hudi-bot commented on PR #9776: URL: https://github.com/apache/hudi/pull/9776#issuecomment-1751634049 ## CI report: * 6b730068fa6ca60dfdd04f720334a49fa19a8b31 UNKNOWN * 268d48b2f47310fd490b80052eff3a5d01aea5c9 Azure:

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

hudi-bot commented on PR #9776: URL: https://github.com/apache/hudi/pull/9776#issuecomment-1751631996 ## CI report: * 6b730068fa6ca60dfdd04f720334a49fa19a8b31 UNKNOWN * 268d48b2f47310fd490b80052eff3a5d01aea5c9 Azure:

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349456496 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/plan/generators/BaseHoodieCompactionPlanGenerator.java: ## @@ -118,8 +120,21 @@

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349449472 ## hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/timeline/TestCompletionTimeQueryView.java: ## Review Comment: We can not do that, some

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349447360 ## hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java: ## @@ -203,11 +221,10 @@ protected List buildFileGroups(Stream

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349445871 ## hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java: ## @@ -203,11 +221,10 @@ protected List buildFileGroups(Stream

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349444563 ## hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestHoodieClientMultiWriter.java: ## @@ -516,10 +516,13 @@ public void

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349444521 ## hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/rollback/TestMergeOnReadRollbackActionExecutor.java: ## @@ -126,21 +126,19 @@ public void

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349444353 ## hudi-common/src/main/java/org/apache/hudi/common/util/HoodieRecordUtils.java: ## @@ -109,4 +110,11 @@ public static boolean

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349444096 ## hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java: ## @@ -216,6 +233,9 @@ protected List buildFileGroups(Stream baseFileS

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349443600 ## hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java: ## @@ -138,6 +142,20 @@ protected void refreshTimeline(HoodieTimeline

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349442571 ## hudi-common/src/main/java/org/apache/hudi/common/table/timeline/CompletionTimeQueryView.java: ## @@ -69,6 +84,64 @@ public

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349442374 ## hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFormatWriter.java: ## @@ -90,15 +101,25 @@ public void withOutputStream(FSDataOutputStream

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349442256 ## hudi-common/src/main/java/org/apache/hudi/common/table/timeline/CompletionTimeQueryView.java: ## @@ -16,27 +16,33 @@ * limitations under the License. */

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349442027 ## hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/upgrade/TestUpgradeDowngrade.java: ## @@ -844,6 +847,47 @@ private Pair, List>

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349441899 ## hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java: ## @@ -544,15 +543,27 @@ public static Option> getLatestLogVersion(FileSystem fs, P } /**

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349441742 ## hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java: ## @@ -544,15 +543,27 @@ public static Option> getLatestLogVersion(FileSystem fs, P } /**

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349441335 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java: ## @@ -164,66 +163,68 @@ public HoodieAppendHandle(HoodieWriteConfig config,

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349440980 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java: ## @@ -726,19 +727,24 @@ protected void

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349440950 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieWriteHandle.java: ## @@ -233,41 +234,45 @@ private static Schema

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349440717 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java: ## @@ -164,66 +163,68 @@ public HoodieAppendHandle(HoodieWriteConfig config,

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349440642 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java: ## @@ -951,12 +951,6 @@ private void startCommit(String instantTime,

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

codope commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1348351265 ## hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java: ## @@ -203,11 +221,10 @@ protected List buildFileGroups(Stream baseFileS

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

codope commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1348276299 ## hudi-common/src/main/java/org/apache/hudi/common/model/HoodieFileGroup.java: ## @@ -101,12 +103,36 @@ public void addBaseFile(HoodieBaseFile dataFile) { /** *

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

codope commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1348240543 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java: ## @@ -726,19 +727,24 @@ protected void

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

hudi-bot commented on PR #9776: URL: https://github.com/apache/hudi/pull/9776#issuecomment-1744851626 ## CI report: * 6b730068fa6ca60dfdd04f720334a49fa19a8b31 UNKNOWN * 268d48b2f47310fd490b80052eff3a5d01aea5c9 Azure:

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

hudi-bot commented on PR #9776: URL: https://github.com/apache/hudi/pull/9776#issuecomment-1744658477 ## CI report: * 6b730068fa6ca60dfdd04f720334a49fa19a8b31 UNKNOWN * 06eb344da3e6c4ce270a2b63e34908a507aac786 Azure:

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

hudi-bot commented on PR #9776: URL: https://github.com/apache/hudi/pull/9776#issuecomment-1744589641 ## CI report: * 6b730068fa6ca60dfdd04f720334a49fa19a8b31 UNKNOWN * 06eb344da3e6c4ce270a2b63e34908a507aac786 Azure:

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

hudi-bot commented on PR #9776: URL: https://github.com/apache/hudi/pull/9776#issuecomment-1744273065 ## CI report: * 6b730068fa6ca60dfdd04f720334a49fa19a8b31 UNKNOWN * 06eb344da3e6c4ce270a2b63e34908a507aac786 Azure:

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

hudi-bot commented on PR #9776: URL: https://github.com/apache/hudi/pull/9776#issuecomment-1744236147 ## CI report: * 6b730068fa6ca60dfdd04f720334a49fa19a8b31 UNKNOWN * a66ddcefb23b24aa73d95cabbdb2fe9b79a8aa79 Azure:

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1343426405 ## hudi-common/src/main/java/org/apache/hudi/common/model/HoodieFileGroup.java: ## @@ -101,12 +103,36 @@ public void addBaseFile(HoodieBaseFile dataFile) { /**

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1343421100 ## hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java: ## @@ -544,15 +543,40 @@ public static Option> getLatestLogVersion(FileSystem fs, P } /**

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1343421100 ## hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java: ## @@ -544,15 +543,40 @@ public static Option> getLatestLogVersion(FileSystem fs, P } /**

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1343415901 ## hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java: ## @@ -544,15 +543,40 @@ public static Option> getLatestLogVersion(FileSystem fs, P } /**

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

vinothchandar commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1342607441 ## hudi-common/src/main/java/org/apache/hudi/common/model/HoodieFileGroup.java: ## @@ -101,12 +103,36 @@ public void addBaseFile(HoodieBaseFile dataFile) { /**