rangareddy commented on issue #16860:
URL: https://github.com/apache/hudi/issues/16860#issuecomment-5493297632
Closing as fixed: the reported defect no longer reproduces on `master`
(checked at `1e59c9eb6d9e`).
### Only one of the three methods was ever affected
At the time this was filed (`358ecc77a74c`, 2025-02-21),
`TestData::checkWrittenData` selected a single base file per partition and read
only that one:
```java
File latestDataFile = Arrays.stream(dataFiles)
.max(Comparator.comparing(f -> FSUtils.getCommitTime(f.getName())))
.orElse(dataFiles[0]);
```
So with more than one bucket in a partition, every other base file went
unchecked. That matches the report exactly.
On current `master` it collects every latest file slice that has a base
file, reads all of them into one buffer, sorts, and compares:
https://github.com/apache/hudi/blob/1e59c9eb6d9e341d5fd10e1418c0c3deb34b6579/hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/utils/TestData.java#L837-L883
`TestData::checkWrittenDataCOW` and `TestData::checkWrittenDataMOR` were
never affected. Both already iterated over all latest base files / all latest
merged file slices per partition when this ticket was opened, so no change was
needed there.
### What fixed it
Commit `8a653836db55c1490cd3e3964d22f3e7abc780fe` (#19118, "feat(writer):
Switch default to native log format for table version >= 10") replaced the
single-file selection with the file-slice loop. It was a side effect of the
log-format work rather than a deliberate fix here, which is why no commit
references HUDI-9056.
Multi-bucket coverage also landed independently:
`ITTestBucketStreamWrite::testBucketWriteIntoMultipleBuckets` (`bucketNum = 2`,
both COW and MOR) was added in `f5de6628b88c927ef2784d916014891c2feaa468`
(#12796, HUDI-8920), five days after this ticket was created.
I also swept `TestData` for any remaining single-latest-file selection and
found none.
### One small leftover, not worth keeping this open
`ITTestBucketStreamWrite::testBucketStreamWriteAfterRollbackFirstFileGroupCreation`
still calls `doWrite(tablePath, isCow, 1)`. The reason that mattered is gone,
since `checkWrittenData` now aggregates across base files, so raising the
bucket count there is safe and would give the rollback path genuine
multi-bucket coverage. That is a good-first-issue-sized follow-up rather than
remaining scope on this task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
