leaves12138 opened a new pull request, #10072:
URL: https://github.com/apache/paimon/pull/10072
## What is the purpose of the change?
`PreAssignSplitAssigner.ofDynamicPartitionPruning()` rebuilds FAIR
assignments from the collection supplied to its constructor, rather than the
live pending queues. If a dynamic-filtering event arrives after a matching
split has been handed out, that split becomes pending again. Repeated events
can replay it again. The same reconstruction can discard progress supplied
through `addSplitsBack()` or lose splits added after construction.
This is a pre-existing state-management issue, reproduced on `master` at
`b026557f6d8aa4bab27bfeb1ea1e0a00224fa845`; it is not a claim that file-size
weighting introduced the bug. The existing late-filtering regression happens to
filter out the already-assigned split and therefore does not exercise this case.
## Brief change log
- Use live pending splits instead of retaining the initial collection.
- Before assignment starts, retain weighted/grouped FAIR planning over the
filtered pending splits.
- After assignment starts, prune each reader's pending queue in place. Do
not move work to a reader whose queue has already been drained.
- Preserve returned split offsets, authorization wrappers, newly added
splits, and pending counts.
- Add regressions for late/repeated filtering, reader ownership, returned
progress, and splits added before the first assignment. Keep PREEMPTIVE as a
passing control.
## Validation
Java 8, Flink 1.20.1:
- Unpatched baseline with the new regressions: 4 assertion failures, 1
passing PREEMPTIVE control, no test errors.
- Patched targeted suite: 44 tests passed, no failures, errors, or skips.
This includes `BinPackingTest`, `FairAssignModeTest`,
`PreemptiveAssignModeTest`, `PreAssignSplitAssignerTest`,
`FlinkSourceBuilderTest`, and the two file-size/system-table tests in
`DataTableSourceTest`.
- Spotless formatting and reactor Checkstyle checks passed.
- A wider 53-test run had 51 passes and 2 `DataTableSourceTest` errors
involving `NoSuchMethodError: DataFileWriter.setEncoder(...)` from the Avro
runtime classpath. That run is **not** reported as green; this patch does not
change dependencies.
- An additional isolated Java runner exercised the actual `BinPacking`
implementation with 5,000 deterministic nonnegative-weight cases covering empty
inputs, more readers than splits, zero/equal/skewed weights, and weights above
the 32-bit range.
- No production job or full Flink integration suite was run.
Package the reactor first so the code-generation plugin resources are
available, then run the focused tests:
```bash
mvn -pl paimon-flink/paimon-flink-common -am -Pfast-build,flink1 -DskipTests
package
mvn -pl paimon-flink/paimon-flink-common -am -Pfast-build,flink1 \
-DfailIfNoTests=false \
'-Dtest=FairAssignModeTest,PreemptiveAssignModeTest,PreAssignSplitAssignerTest,FlinkSourceBuilderTest,DataTableSourceTest#testBoundedSystemTableUsesFileSizeWeightModeAfterCopy+testBoundedSystemTableRejectsFileSizeWithPreemptiveMode,BinPackingTest'
test
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
