xiarixiaoyao commented on PR #10898:
URL: https://github.com/apache/hudi/pull/10898#issuecomment-2029573829
@xuzifu666 @danny0405 @beyond1920
i think we should solve the root cause of bucket duplication.
There are currently three situations where bucket file duplication occurs
1. Spark Speculation execution . Turn
off speculative execution , we can solve this problem
2. hoodier archiver Parallel deletet complete timeline . 1.0
has solved this problem.
3. Concurrent into ovewrite of multiple spark writer . this is a bug
need to fixed.
now focus on the scence3: Concurrent into ovewrite of multiple spark writer
when hudi build fileslice, hudi will call isFileSliceCommitted to Determine
if the current file is committed.
```
/**
* A FileSlice is considered committed, if one of the following is true -
There is a committed data file - There are
* some log files, that are based off a commit or delta commit.
*/
private boolean isFileSliceCommitted(FileSlice slice) {
if (!compareTimestamps(slice.getBaseInstantTime(),
LESSER_THAN_OR_EQUALS, lastInstant.get().getTimestamp())) {
return false;
}
return
timeline.containsOrBeforeTimelineStarts(slice.getBaseInstantTime());
}
```
this is ok for single concurrent write scenario, but for mutil write the
logical of isFileSliceCommitted has some bugs.
If a file has a smaller commit time then smallest complete commit, Hudi will
directly determine that the file is committed, even
if it is a Garbage file or (File generated by write failure)
eg: two spark app insert overwrite hudi BUCKET table with same partition.
app1: start write commit at 00000001 write files: 000000000--uuid1.parquet
app2: start write commit at 00000002 write files: 000000000--uuid2.parquet
app1 maybe failed to write due to OCC /cancel/OOM, but
000000000--uuid1.parquet is already written.
when hudi build fileslice, 000000000--uuid1.parquet is considered as
committed. since it‘s committime 00000001 < smallest complete commit 00000002.
this is wrong, committime 00000001 is not committed
maybe we can modify isFileSliceCommitted like this
```
private boolean isFileSliceCommitted(FileSlice slice) {
if (!compareTimestamps(slice.getBaseInstantTime(),
LESSER_THAN_OR_EQUALS, lastInstant.get().getTimestamp())) {
return false;
}
return
timeline.containsOrBeforeTimelineStarts(slice.getBaseInstantTime()) &&
UncompleteTimelineNotContains(slice.getBaseInstantTime());
}
```
finally, I think Hudi's fileslices should be managed uniformly, just like
iceberg/delta lakes, rather than being obtained through list operation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
