linliu-code commented on PR #17847:
URL: https://github.com/apache/hudi/pull/17847#issuecomment-3774040685
> hi @linliu-code for the failed the flink tests, the cause is: this pr will
update `getAllFileSlicesBeforeOn` method, which now would filters out file
slices based on committed instants in the timeline.
>
> ```java
> private boolean isFileSliceCommitted(FileSlice slice) {
> if (!compareTimestamps(slice.getBaseInstantTime(),
LESSER_THAN_OR_EQUALS, lastInstant.get().getTimestamp())) {
> return false;
> }
>
> return
timeline.containsOrBeforeTimelineStarts(slice.getBaseInstantTime());
> }
> ```
>
> When skip compaction is enabled for flink `IncrementalInputSplits`, the
compaction instants is not included in the timeline, so the file slice for
second commit is not included in the result, because its base instant time is
not in the timeline.
>
> ### Solution
> Modify `AbstractTableFileSystemView#fetchAllLogsMergedFileSlice` as:
>
> ```java
> private Option<FileSlice> fetchAllLogsMergedFileSlice(HoodieFileGroup
fileGroup, String maxInstantTime) {
> List<FileSlice> fileSlices =
fileGroup.getAllRawFileSlices().collect(Collectors.toList());
> fileSlices = fileSlices.stream().filter(slice ->
compareTimestamps(slice.getBaseInstantTime(), LESSER_THAN_OR_EQUALS,
maxInstantTime)).collect(Collectors.toList());
> ...
> }
> ```
@cshuo Thanks for pointing out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
