cshuo commented on PR #17847:
URL: https://github.com/apache/hudi/pull/17847#issuecomment-3766080000
hi @linliu-code for the failed the flink tests, the cause is: this pr will
update `getAllFileSlicesBeforeOn` method by filtering out file slices based on
committed instants in time.
```java
private boolean isFileSliceCommitted(FileSlice slice) {
if (!compareTimestamps(slice.getBaseInstantTime(),
LESSER_THAN_OR_EQUALS, lastInstant.get().getTimestamp())) {
return false;
}
return
timeline.containsOrBeforeTimelineStarts(slice.getBaseInstantTime());
}
```
When skip compaction is enabled for flink `IncrementalInputSplits`, the
compaction instants is not included in the timeline, so the file slice for
second commit is not included in the result, because its base instant time is
not in the timeline.
### Solution
Modify `AbstractTableFileSystemView#fetchAllLogsMergedFileSlice` as:
```java
private Option<FileSlice> fetchAllLogsMergedFileSlice(HoodieFileGroup
fileGroup, String maxInstantTime) {
List<FileSlice> fileSlices =
fileGroup.getAllRawFileSlices().collect(Collectors.toList());
fileSlices = fileSlices.stream().filter(slice ->
compareTimestamps(slice.getBaseInstantTime(), LESSER_THAN_OR_EQUALS,
maxInstantTime)).collect(Collectors.toList());
...
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
