SteNicholas opened a new pull request, #418:
URL: https://github.com/apache/paimon-cpp/pull/418
<!-- PR titles must follow Conventional Commits: <type>(<optional-scope>):
<description> -->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close #417
<!-- What is the purpose of the change -->
After a Parquet data file was closed, `DataFileWriter` and
`KeyValueDataFileWriter` built the column
stats of its `DataFileMeta` by reopening the file just written through
`ParquetStatsExtractor`: an
open, a length lookup, ranged reads of the footer and a full Thrift decode
of `FileMetaData`. This
happens for every data file produced by writes and compactions, on both
append and primary-key
tables, and on remote storage it adds round trips to every rolled file.
The writer already holds this metadata: after
`parquet::arrow::FileWriter::Close()`,
`FileWriter::metadata()` returns the `FileMetaData` serialized into the
footer. This PR derives the
stats from it and keeps reading the file back as the fallback.
- **Keep the metadata**: `ParquetFormatWriter::Finish()` retains
`writer_->metadata()` once
`Close()` has succeeded.
- **Share the conversion**: the part of
`ParquetStatsExtractor::ExtractWithFileInfo()` that runs
after the footer is decoded moves into `ExtractFromMetadata()`, which
takes a
`parquet::FileMetaData` and covers the per-row-group merge, the conversion
to `ColumnStats`,
nested fields and `FileInfo`. Stats read back from the footer and stats
taken from the writer go
through the same code. The row group metadata is now created once per row
group instead of once
per column chunk.
- **Use it from the data file writers**: `ParquetFormatWriter` implements
the new internal
interface `WrittenFileStatsProvider`
(`src/paimon/common/format/written_file_stats_provider.h`).
`DataFileWriterBase::ExtractFileStats()` takes the stats from a format
writer implementing it and
otherwise reads them back through `FormatStatsExtractor`, as the other
formats still do.
`SingleFileWriter::GetFormatWriter()` exposes the format writer, which
outlives `Close()`.
- **Out of scope**: callers that extract stats from files they did not
write, e.g. migration in
`src/paimon/core/migrate/file_meta_utils.cpp`, keep reading the footer.
As noted in #417, the in-memory `FileMetaData` is not built the same way as
a decoded footer (for
example, its writer version is not parsed from `created_by`), so the tests
check that both paths
produce the same stats instead of assuming it.
### Tests
<!-- List UT and IT cases to verify this change -->
UT, `src/paimon/format/parquet/parquet_stats_extractor_test.cpp`
(`parquet_format_test`):
- `CheckStats` and `TestNullForAllType` also check that
`ParquetFormatWriter::ExtractWrittenFileStats()`
returns the same `ColumnStats` and `SimpleStats` as reading the footer
back. Together they cover
every primitive type, DECIMAL stored as INT32, INT64 and
FIXED_LEN_BYTE_ARRAY, TIMESTAMP
including INT96, nested and vector types, and all-null columns.
- `TestWrittenFileStatsAcrossRowGroups`: a file of three row groups with
NaN, null, all-null and
nested values, plus the error returned before `Finish()`.
- `TestWrittenFileStatsOfEmptyFile`: a file without row groups.
UT, `src/paimon/core/io/data_file_writer_base_test.cpp` (`core_test`),
`DataFileWriterStatsTest`
parameterized on `parquet` and `orc`:
- `AppendWriter`, `KeyValueWriter`: with a file system that counts `Open()`
calls, closing a Parquet
data file opens no file while ORC still reads it back once, and the
`value_stats` (plus
`key_stats` for the primary-key writer) stored in `DataFileMeta` equal the
stats read back from
the file.
### API and Format
<!-- Does this change affect API in include dir or storage format or
protocol -->
No. `FormatWriter` and `FormatStatsExtractor` under `include/paimon/format/`
are unchanged, so
their ABI is kept; the new interface is internal. The storage format and the
`SimpleStats` stored in
`DataFileMeta` are unchanged.
### Documentation
<!-- Does this change introduce a new feature -->
No. This is a performance improvement without user-facing behavior changes.
### Generative AI tooling
Generated-by: Claude Code 2.1.289 (Claude Opus 5.5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
