OIiveirra opened a new issue, #69544: URL: https://github.com/apache/doris/issues/69544
### Problem Iceberg REST DROP COLUMN cannot currently atomically verify the complete retained partition-spec set with the standard Iceberg 1.11.0 update requirements. A concurrent writer can restore a previously expired partition-source reference without advancing the partition counter, allowing a schema DROP to leave a retained nondefault spec referring to a missing current-schema source field. This is tracked separately from #69408, which is being narrowed to direct catalogs with complete metadata CAS. REST retains branch-4.2's ordinary SDK commit behavior; this follow-up is not fixed by narrowing that PR. ### Reproduction boundary 1. Construct/register valid format-v2 metadata with historical source `x` using partition field ID 10, current source `y` using ID 11, and `last-partition-id=999`. 2. While x's historical spec is retained, prepare an SDK nondefault spec that adds x to the y spec. The SDK reuses field ID 10. 3. Expire unused partition metadata so a DROP refresh sees only y. A currently unpartitioned table after complete unused-history cleanup needs coverage too. 4. Stage DROP x against that refreshed base, then publish the previously prepared spec before applying the DROP request. 5. UUID/schema/column and partition-counter requirements can all pass; the counter remains 999 while a retained nondefault spec refers to deleted x. The 1000 partition-ID starting convention is not a v2 validity requirement. A numeric counter threshold cannot prove that no partition history existed. ### Evidence The current-head AI review describes the case: https://github.com/apache/doris/pull/69408#pullrequestreview-5624948910 A local metadata-only probe using the official `iceberg-core-1.11.0.jar` confirmed low-ID construction/JSON parsing, SDK field reuse, successful validation of the DROP requirements, and the resulting missing schema field with a retained spec source reference. REST requirement application was simulated in memory; this is not an end-to-end Doris HTTP or downstream-engine read reproduction. The SDK partitionType call itself did not throw in that probe. Direct full-metadata CAS rejected the concurrent metadata change. ### Expected work Add a real public-DROP HTTP reproduction, including current-partitioned and now-unpartitioned variants. Provide a server-enforced full-metadata generation check or equivalent atomic validation of all retained spec source references. Preserve ordinary legal REST DROP and normal SDK authentication/cache behavior where possible; any temporary compatibility restriction must be explicit. Do not rely only on the partition-ID counter, current spec, client-side locks, or an unguarded fallback. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
