JingsongLi commented on code in PR #9370:
URL: https://github.com/apache/paimon/pull/9370#discussion_r4183205827
##########
paimon-core/src/main/java/org/apache/paimon/operation/AbstractFileStoreWrite.java:
##########
@@ -674,8 +697,33 @@ private RestoreFiles scanExistingFileMetas(
partInfo.get(), bucket),
e);
}
- if (restored.totalBuckets() != null && validateNumBuckets) {
- checkNumBuckets(partInfo.get(), expectedTotalBuckets,
restored.totalBuckets());
+ Integer restoredTotalBuckets = restored.totalBuckets();
+ if (restoredTotalBuckets != null
+ && validateNumBuckets
+ && expectedTotalBuckets != restoredTotalBuckets) {
+ if (partitionType.getFieldCount() > 0
+ && options.bucketPerPartitionCountEnabled()
+ && !strictBucketCount) {
+ if (bucket >= restoredTotalBuckets) {
+ throw new RuntimeException(
+ String.format(
Review Comment:
[P2] Ignore removed active buckets when restoring lookup state
The new per-partition range check also runs when LookupSinkWrite restores
ACTIVE_BUCKETS_STATE by calling write.compact(savedPartition, savedBucket,
false). After a valid partition downscale, that state can still contain a
bucket which no longer exists. I reproduced the real constructor path with a
catalog/table using changelog-producer=lookup: save active (pt=1,bucket=3) from
the original 4-bucket layout, perform a correct wrapper-based partition
overwrite to 2 buckets, and retain schema default=4 for the table. The durable
mapping is pt=1->2 with one readable row, but restoring the saved state throws
here (Trying to write bucket 3 ... partition only has 2 buckets) before any new
input. An in-range saved bucket is a passing control, and the original
empty-bucket behavior also permits startup. StateValueFilter only distributes
state ownership; it does not remove buckets outside the new partition layout.
This contradicts the documented suspend-with-savepoint -> rescale -> restart-f
rom-savepoint workflow for lookup jobs. Filter obsolete active compaction
bucket hints against the current partition layout before invoking compact,
while retaining the guard for actual data writes; add a lookup
savepoint/downscale restart regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
