xiaoxuandev opened a new pull request, #17814: URL: https://github.com/apache/iceberg/pull/17814
## Problem `rewriteDataFiles` fails on a table that declared a sort order at creation when the output is unsorted relative to that order — always for z-order, and for sort whenever the requested order matches none the table declares ``` java.lang.IllegalArgumentException: Cannot use output sort order id 0 because the table does not contain a sort order with that id ``` ## Root cause `SparkShufflingFileRewriteRunner` sets `output-sort-order-id` from `SortOrderUtil.findTableSortOrder`, which returns `SortOrder.unsorted()` (id 0) when no table sort order matches. `SparkWriteConf.outputSortOrderId` then rejected that id unless the table contained it: ```java Preconditions.checkArgument(table.sortOrders().containsKey(explicitId), ...); ``` Id 0 is reserved for unsorted order and is valid for any table, but `sortOrders()` contains it only when the table was created unsorted. A table that declares a sort order at creation (only possible through the Iceberg API, REST, or another engine — `SparkCatalog` never sets one) gets `INITIAL_SORT_ORDER_ID = 1` and never lists 0, so the check rejected an id the rewrite legitimately produced. The implicit branch of the same method already returns that id without consulting the table. ## Fix Accept the reserved unsorted order id without requiring the table to list it. Unknown ids are still rejected. Downstream, `table.sortOrders().get(0)` yields `null`, `DataFiles.Builder.withSortOrder` ignores it, and the file is written with `sort_order_id = 0`, which is the correct value. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
