jiwen624 opened a new pull request, #18860: URL: https://github.com/apache/iceberg/pull/18860
After a partition field is added and then dropped (e.g., when a partition change gets rolled back), the table goes back to its old spec ID. Streaming reads can then silently skip the first append after the revert. With async planning enabled, the stream can instead hang permanently, even across restarts. The cause: Manifest merging writes the commit's manifest for the dropped spec first, and it only holds existing files. Micro-batch positions count those files, but the planners end the snapshot at its added-file count. This change fixes the end position in each planner while keeping their numbering, so existing checkpoints still resume correctly. Repro: add a partition field, append 1 and 2, drop the field, append 3 and then 4, then stream from the beginning. Before: 1, 2, 4. After: 1, 2, 3, 4. Added `readStreamAfterPartitionSpecRevert` to `TestStructuredStreamingRead3`; it fails on main with both planners. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
