danny0405 commented on issue #19774:
URL: https://github.com/apache/hudi/issues/19774#issuecomment-5451608691
An alternative—and currently my preferred fix—is to make the minimal change
in `isFileSliceCommitted`, without changing file-group construction.
A file slice's base instant acts as a slicing barrier; it does not
necessarily have to be committed itself. This is analogous to a pending
compaction instant. If the first physical file is a pending log at `t2`, the
existing attribution creates a slice at `t2`, and later completed logs
`t3`/`t4` correctly remain in that slice. The actual bug is that slice
visibility currently checks only the barrier instant and discards the entire
slice before V8+ file-level filtering can run.
The slice should be considered committed/visible when either its base
instant is on the timeline or at least one contained log belongs to a committed
delta instant:
```java
private boolean isFileSliceCommitted(FileSlice slice) {
if (!compareTimestamps(
slice.getBaseInstantTime(),
LESSER_THAN_OR_EQUALS,
lastInstant.get().requestedTime())) {
return false;
}
return timeline.containsOrBeforeTimelineStarts(slice.getBaseInstantTime())
|| slice.getLogFiles()
.anyMatch(logFile ->
timeline.containsOrBeforeTimelineStarts(logFile.getDeltaCommitTime()));
}
```
For the reported sequence:
```text
raw slice t2: [t2 (pending), t3, t4]
visible slice t2: [t3, t4]
```
Once the slice passes this check, the existing V8+ `filterUncommittedFiles`
removes the pending `t2` log and retains the committed `t3`/`t4` logs. A slice
containing only pending files remains hidden from committed APIs and remains
accessible through the explicit including-inflight APIs.
This also keeps pre-V8/0.x compatibility: legacy writers query the latest
valid base instant before writing and encode the selected base instant in the
log filename. No new batch API, sorting relocation, or visibility change to
`addLogFile` is needed.
### Alternative minimal patch
[Download
`hudi-19774.patch`](https://gist.githubusercontent.com/danny0405/c864bd3e40802725b910667072e4a6d5/raw/ab802130e4d14ee72177e2ee0c7cc50d4bd5e7e8/hudi-19774.patch)
· [View this patch revision on
GitHub](https://gist.github.com/danny0405/c864bd3e40802725b910667072e4a6d5/fea80f9778e1677dd0464e98b4e2cd49e81cf1b0)
Regression coverage should verify that a slice anchored by an uncommitted
first log is retained because it contains later committed logs, while the
reader view filters only the uncommitted physical log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
