Baunsgaard opened a new pull request, #16937:
URL: https://github.com/apache/iceberg/pull/16937
## What
`TestRewriteDataFilesAction` generated its input data files through Spark on
every `writeRecords(...)` call: `spark.createDataFrame()` bean-reflection over
the records, a `repartition(files)` shuffle, a `sortWithinPartitions`, and a
separate Spark job per write. This change writes the inputs directly with the
Iceberg API instead.
The new path builds the records in memory, groups them by partition via
`PartitionKey`, splits each group into the requested number of files, writes
each file with `FileHelpers.writeDataFile`, and commits them in a single
`newAppend()`.
## Why
The suite is the single heaviest Spark core test class, and on the `SCALE =
400000` cases the Spark write overhead (DataFrame conversion + shuffle + sort +
per-write job scheduling) dominates the actual rewrite under test. The suite
asserts on file counts and row content, not on the row-to-file assignment, so a
deterministic contiguous split is equivalent.
A measured A/B (identical compiled binary, runtime toggle, interleaved reps
at low load) showed **−26% wall-clock on the full v4.1 suite** (701.9 s → 520.3
s).
## What stays on Spark
Two tests keep the Spark writer because they depend on the exact on-disk
layout it produces:
- `testBinPackCombineMixedFiles` — calibrated to Spark's on-disk byte sizes.
- `testRemoveDangledPositionDeletesPartitionEvolution` — depends on
data-file ordering
and the row position its position delete targets.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
