JingsongLi commented on PR #7865: URL: https://github.com/apache/paimon/pull/7865#issuecomment-4988444759
`bucket.per-partition-count-enabled` is set to `false` by default, and the documentation explicitly states that when this setting is disabled, all partitions must share the table-level bucket count; however, when `AbstractFileStoreWrite.scanExistingFileMetas` detects a discrepancy between the old and new bucket counts in a partitioned table, it still unconditionally allows writes where `bucket < restoredTotalBuckets` without checking this setting. - For example, suppose an existing partition uses 4 buckets, the table-level bucket count is changed to 2, and the switch remains disabled. New writes are routed to buckets 0 and 1 using modulo 2, while old data is distributed using modulo 4; the current check would allow this, potentially causing the same primary key to land in different buckets, resulting in duplicate keys and data errors. - It is recommended to allow partition-level bucket mismatches only when `options.bucketPerPartitionCountEnabled()` is set to `true`; when disabled, maintain the original mismatch rejection behavior and add a regression test that defaults to the switch being off. - The existing `PartitionBucketMappingTest` and `FileSystemWriteRestoreTest`—a total of 9 tests—pass, but the tests involving different numbers of buckets per partition explicitly enable the switch, thus missing this specific scenario. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
