anuragmantri opened a new pull request, #17868:
URL: https://github.com/apache/iceberg/pull/17868
Related to #11648.
`RewriteDataFilesSparkAction` has disabled the executor cache for delete
files unconditionally since #13820. As noted in the discussion on #11648, that
is the right default for position deletes but costly for equality deletes,
which cannot be narrowed to a single data file while reading and are therefore
re-read in full for every data file whose bounds overlap.
This adds a rewrite option so users can opt back in, where a Spark executor
cache setting is exposed as a rewrite option rather than a session property.
Existing Default behavior is unchanged.
Usage from SQL:
```sql
CALL catalog_name.system.rewrite_data_files(
table => 'db.sample',
options => map('executor-cache.delete-files.enabled', 'true')
);
```
Usage from the action API:
```java
SparkActions.get(spark)
.rewriteDataFiles(table)
.option(RewriteDataFilesSparkAction.EXECUTOR_CACHE_DELETE_FILES_ENABLED,
"true")
.execute();
```
Tests added to `TestSparkExecutorCache` assert the number of times each
delete file is opened during a rewrite: 2 per delete file with the option at
its default, once per data file, and 1 with the option enabled.
---
**AI Disclosure**
- Model: Claude Opus 5
- Platform/Tool: Claude Code
- Human Oversight: fully reviewed by me after the changes were made.
- Prompt Summary: Make the executor cache optional during `RewriteDataFiles`
via a new rewrite option, keeping the existing default, then review the change
against AGENTS.md and add tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
