foosekis4j opened a new pull request, #18897: URL: https://github.com/apache/iceberg/pull/18897
Iceberg's Spark integration has no way for an accelerated reader or compaction runner to take over the per-file work while Iceberg keeps planning, delete resolution and commits. This adds two extension points to Spark 4.2, both inactive unless a provider is named: `PartitionReaderFactoryProvider` (session conf `spark.sql.iceberg.partition-reader-factory-provider`, read option `partition-reader-factory-provider`), which wraps the batch scan's reader factory and delegates task groups it cannot serve, with a public `IcebergInputPartition` exposing the task group, table, IO, projection and case sensitivity; and `FileRewriteRunnerProvider` (`spark.sql.iceberg.rewrite-runner-provider`, action option `rewrite-runner-provider`), which supplies the bin-pack `rewrite_data_files` runner. Providers are found with `ServiceLoader`; an unknown name fails listing the providers found. Test plan: CPU-only fake providers that wrap Iceberg's own reader and bin-pack runner check selection by session conf and by option, delegation of unsupported task groups, fallback to bin-pack when a provider returns null, rejection with sort and z-order, and the error for an unknown name (`TestPartitionReaderFactoryProvider` 5, `TestRewriteDataFilesRunnerProvider` 7). `TestSparkReaderDeletes` (192) and `TestRewriteDataFilesAction` (207) pass unchanged. --- **AI Disclosure** - Model: Claude Opus 5.5 - Platform/Tool: Claude Code - Human Oversight: partially reviewed - Prompt Summary: Port two engine-neutral Spark extension points (partition reader factory provider, rewrite runner provider) to Spark 4.2 with CPU-only tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
