rabenhorst opened a new pull request, #12073:
URL: https://github.com/apache/arrow-rs/pull/12073
# Which issue does this PR close?
- Closes #12072.
# Rationale for this change
`SerializedFileWriter` keeps one `Vec<Option<Sbbf>>`,
`Vec<Option<ColumnIndexMetaData>>`
and `Vec<Option<OffsetIndexMetaData>>` per row group until close, with one
entry per
leaf column even when every entry is `None`. On 64-bit targets that is 24 +
232 + 48 =
304 bytes per column per row group. This happens when column chunks are
copied with
`SerializedRowGroupWriter::append_column` without indexes, when page
statistics are
disabled, and for bloom filters already written with
`BloomFilterPosition::AfterRowGroup`.
When many small files are concatenated into a large one with
`append_column`, these
all-`None` vectors can be a large part of the writer's heap.
# What changes are included in this PR?
- `SerializedFileWriter` stores `None` instead of a vector for a row group
that has no
bloom filter, column index or offset index entry.
- `write_bloom_filters` skips such row groups, and takes the row group's
vector after
writing it, so `AfterRowGroup` no longer retains a vector of `None`s.
- At close, absent row groups are expanded to the existing dense layout when
building
the `PageIndex`, so the metadata writer and the returned `ParquetMetaData`
see exactly
what they saw before.
`SerializedRowGroupWriter` and the public `OnCloseRowGroup` callback are
unchanged
(the callback still receives one entry per column).
# Are these changes tested?
Yes. Two new tests in `file::writer`, using `append_column`:
- no bloom filters or indexes, with the offset index enabled and disabled:
nothing is
retained per row group, and the returned page index is unchanged;
- some row groups with and some without bloom filters and page indexes, for
some
columns, for `BloomFilterPosition::End` and `AfterRowGroup`, with the
offset index
enabled and disabled: bloom filter and index locations, bloom filter
contents, and the
returned `PageIndex` compared with what is read back from the file.
I also compared this branch with `main` outside the crate: 36 files
(appended chunks
with no, all, mixed, offset-only and bloom-only indexes; `next_column` with
no, chunk and
mixed per-column statistics and bloom filters; both bloom filter positions;
offset index
enabled and disabled) are byte-identical, and so is the `Debug` output of
the returned
`ParquetMetaData`.
Heap held by the writer (counting allocator), 200 nullable INT64 leaves,
chunks
appended with `append_column`, no indexes, default `WriterProperties`:
| row groups | before close, `main` | before close, this PR | peak during
close, `main` | peak during close, this PR |
|---:|---:|---:|---:|---:|
| 1,000 | 152.2 MB | 91.4 MB | 152.3 MB | 101.1 MB |
| 10,000 | 1,522.8 MB | 914.8 MB | 1,523.8 MB | 1,011.8 MB |
That is 304 bytes less per column per row group before close. The returned
`ParquetMetaData` is the same size (it still contains the dense offset index
of
`None`s, 48 bytes per column per row group), and the number of allocations is
unchanged.
# Are there any user-facing changes?
No. Written files, the returned `ParquetMetaData` and the public API are
unchanged.
# AI usage
The implementation, the tests, the comparison harness and this description
were
written with Claude Code (Claude Opus 5.5).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
