dwangatt commented on code in PR #9370:
URL: https://github.com/apache/paimon/pull/9370#discussion_r3947183707
##########
paimon-core/src/main/java/org/apache/paimon/operation/AbstractFileStoreWrite.java:
##########
@@ -674,8 +674,31 @@ private RestoreFiles scanExistingFileMetas(
partInfo.get(), bucket),
e);
}
- if (restored.totalBuckets() != null && validateNumBuckets) {
- checkNumBuckets(partInfo.get(), expectedTotalBuckets,
restored.totalBuckets());
+ Integer restoredTotalBuckets = restored.totalBuckets();
+ if (restoredTotalBuckets != null
+ && validateNumBuckets
+ && expectedTotalBuckets != restoredTotalBuckets) {
+ if (partitionType.getFieldCount() > 0 &&
options.bucketPerPartitionCountEnabled()) {
+ if (bucket >= restoredTotalBuckets) {
+ throw new RuntimeException(
+ String.format(
+ "Trying to write bucket %d to %s, but the
partition only has %d "
+ + "buckets (table default: %d).
Recompute the bucket using the "
+ + "partition's bucket count, or
rescale the partition via "
+ + "INSERT OVERWRITE.",
+ bucket,
+ partInfo.get(),
+ restoredTotalBuckets,
+ expectedTotalBuckets));
+ }
+ LOG.info(
+ "{} uses {} buckets (expected: {}). Accepting
per-partition bucket count.",
Review Comment:
Thanks @zhuxiangyi that's a good point. I updated PR to fail job when bucket
does not match.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
