litiliu opened a new issue, #3638: URL: https://github.com/apache/fluss/issues/3638
### Search before asking - [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar. ### Fluss version main (development) ### Please describe the bug 🐞 ### Problem The Iceberg lake tiering writer does not validate that the current Iceberg partition spec is still compatible with the partitioning definition used by Fluss. An external Iceberg client can evolve the partition spec after lake tiering has been enabled. A newly created or restarted tiering writer may then load the evolved Iceberg spec but continue using partition values calculated according to the Fluss table's original bucket definition. This problem already applies to the existing explicit bucket partitioning and does not depend on implicit/hidden partition support. ### Reproduction example 1. Create a Fluss primary-key table distributed by `id` into 8 buckets and enable Iceberg lake tiering. 2. Wait for the Iceberg table to be created and verify that its current partition spec contains `bucket[8](id)`. 3. Stop or pause the tiering job so that its lake writer will be recreated later. 4. Using an external Iceberg client, evolve the Iceberg table's current partition spec from `bucket[8](id)` to `bucket[16](id)`. The Fluss table remains configured with 8 buckets. 5. Restart the tiering job, causing a new Iceberg lake writer to load the current `bucket[16](id)` spec. 6. Write a record whose `bucket[8](id)` value differs from its `bucket[16](id)` value, and wait for it to be tiered. 7. Inspect the committed data file's partition metadata or query the table with a predicate on that `id`. ### Current behavior The writer creates a `PartitionKey` using the current Iceberg spec, but directly sets the Fluss table bucket ID into its bucket field. It does not apply the current Iceberg bucket transform to the original `id` value. For example, if `bucket[8](id) = 3` while `bucket[16](id) = 11`, the writer records `3` as the value of the `bucket[16](id)` partition field. Because both values are integers, this mismatch may not cause a type or commit error. The file can therefore be committed with incorrect partition metadata. Iceberg readers may project the predicate using `bucket[16](id)` and prune that file, causing the record to be missing from filtered queries even though the data file was committed successfully. ### Expected behavior When creating the Iceberg lake writer, Fluss should verify that the current Iceberg partition spec is semantically equivalent to the partitioning definition expected from the Fluss table. If it is incompatible, writer initialization should fail with a clear error before any incorrectly partitioned file can be committed. This is different from supporting an independently configured lake partition spec in #3309; this issue is about validating the currently supported Fluss-to-Iceberg partition mapping. ### Solution Add an Iceberg partition-spec compatibility validator to the lake writer. The validation should compare the expected Fluss-to-Iceberg mapping with the current Iceberg spec, including: - source columns; - transform types; - transform parameters such as the bucket count; - partition field order where it affects the produced partition row. Run the validation when the Iceberg writer is initialized. If table metadata can be refreshed while the writer remains active, cache the validated Iceberg spec ID and validate again when the current spec ID changes. Reject incompatible specs with a non-retriable, descriptive exception that includes both the expected and current partition definitions. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
