pallevam opened a new pull request, #18551: URL: https://github.com/apache/iceberg/pull/18551
`DeleteOrphanFilesSparkAction` parallelizes the files listed on the driver into a single partition. With prefix listing on tables with 100M+ files, serializing that one partition exceeds the JVM array size limit and fails with `OutOfMemoryError`. The partition count is now based on the number of listed files, at most 100,000 files per partition, so small listings still use one partition. The Hadoop listing path had the same single partition and is fixed too. This is Spark 4.2 only; I will backport to 3.5, 4.0 and 4.1 once this is agreed. Fixes #17387 Test plan: `TestRemoveOrphanFilesAction3` passes, which runs every scenario with and without prefix listing. No new test: reproducing the failure needs more than 100,000 listed files, which is too slow for a unit test. --- **AI Disclosure** - Model: Claude Opus 5.5 - Platform/Tool: Claude Code - Human Oversight: [unknown - human to fill in] - Prompt Summary: Fix the OutOfMemoryError in DeleteOrphanFilesSparkAction with prefix listing reported in #17387. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
