adriangb opened a new pull request, #10917:
URL: https://github.com/apache/arrow-rs/pull/10917
### Motivation
#10565 removed `get_column_writers`, which was the only public constructor
that took explicit `WriterProperties`. With it gone there is no supported way
to build `ArrowColumnWriter`s at properties other than the ones the file
writer was created with, and no way to build writers for only some of a row
group's columns.
This restores that capability on `ArrowRowGroupWriterFactory`, with the page
store and, under the `encryption` feature, the file encryptor correctly
wired, and adds subset selection on top:
- `create_selected_column_writers(row_group_index, props, select)` builds
column writers with the given properties for only the leaf columns `select`
accepts. It returns one entry per leaf column in leaf order, holding `None`
for the unselected ones so indices stay aligned with the schema. An
unselected column allocates nothing, including no page store, which matters
when the page store factory allocates something more expensive than a heap
buffer.
- `page_store_factory()` returns the configured `PageStoreFactory`, so a
caller that produces some of a row group's column chunks by another route
can buffer them through the same store.
`create_column_writers` now delegates to the new accessor with every column
selected, so there is one implementation rather than two and the default
write path is unchanged.
### What it enables
The writer's encoding choices are made ahead of time from the schema and a
few size limits. A caller that would rather own that decision can now measure
it: encode the same rows under several candidate property sets, compare the
compressed sizes the resulting `ColumnCloseResult`s report, and keep the
smallest. Subset selection is what makes this affordable, because a column
that has already decided need not be built at all.
The included example is the demonstration of that. It is a self-contained
adaptive writer that probes a prefix of each row group per undecided column,
settles a column once a candidate wins clearly, and re-races periodically.
### Measured results
An adaptive writer built on these two accessors alone, against an unmodified
`ArrowWriter`, ZSTD:
| input | bytes |
| --- | ---: |
| TPC-H `orders` | -26.6% |
| TPC-H `lineitem` | -23.4% |
| ClickBench `hits_0` | -7.7% |
| ClickBench `hits_1` | -5.0% |
| ClickBench `hits_2` | -6.5% |
| synthetic timestamps (uncompressed) | -84.8% |
| synthetic f64 measurements | -20.5% |
Both arms used identical writer properties and reproduced the input's own row
group boundaries, and every output file was verified to read back to the
source rows exactly. On the real files the probing writer's wall clock came
out at 0.79x to 0.85x of the stock writer measured in the same runs, and the
synthetic timestamp case at roughly 0.3x; those particular runs shared the
machine with other work, so treat the timings as indicative rather than
precise. Byte counts are deterministic and reproduced identically across
repeated runs.
### The example
`parquet/examples/chunk_probe_writer.rs`, behind the `arrow` feature, is
included and self-verifying. It generates a deterministic dataset whose best
encoding differs per column, writes it both ways at identical properties,
asserts both files read back exactly, and prints the byte totals and the
encoding each column ended on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
