SteNicholas opened a new pull request, #232: URL: https://github.com/apache/paimon-cpp/pull/232
### Purpose Linked issue: close #228 `benchmark/` only had table-level cases, so a change confined to the format layer — writer properties, encoding passthrough, per-batch overhead in the read loop — could not be measured without the noise of catalog lookup, split planning, merge/sort and commit. This adds `paimon-parquet-format-benchmark`, driving `ParquetWriterBuilder` and `ParquetFileBatchReader` directly. **Writer cases** — flat primitives; `VARCHAR` by cardinality with and without file-level dictionary encoding; dictionary-encoded `VARCHAR` / `INTEGER` input arrays against flat controls that hold value, cardinality and width fixed; `DECIMAL` at precision 9 / 18 / 38; nested `STRUCT` / `LIST` / `VECTOR` / `MAP`; null density 0–100%; rows per `AddBatch`; column count; row group size; the byte-triggered writer memory threshold; and every codec Parquet accepts. **Reader cases** — full scan; projection; predicate-filtered reads with page-index filtering on and off; skip-heavy reads driven by a strided selection bitmap whose strides straddle the coalesce hole limit; null density; `DECIMAL`; `DOUBLE`; dictionary against plain files; rows per `NextBatch`; nested column reads. **Metrics** — `ns_per_row` / `bytes_per_row` alongside `ns_per_input_row` / `bytes_per_input_row`, so settings that prune by different amounts stay comparable (pruning shrinks the per-materialized-row numerator and denominator together), plus the reader's own `row_groups` / `row_groups_after_filter` / `batches` counters. Read cases assert on the rows they materialized, and the process exits non-zero on any `SkipWithError`, so a case that stops measuring what it claims fails instead of looking fast. Coverage is **adapted from** Velox's Parquet benchmarks ([velox#18052](https://github.com/facebookincubator/velox/pull/18052) for the writer, `ParquetReaderBenchmark` for the reader), not reproduced from them. The differences — local FileSystem vs in-memory sink, timing boundaries, per-axis sweeps vs a combined matrix — are documented in the source header and in the user docs, so results are not presented as cross-comparable. Velox's dict-of-dict / selective-flattening sub-axes have no analogue here: that logic lives in Velox's own writer, whereas Arrow alone makes the decision in paimon-cpp. ### Tests New `paimon-parquet-format-benchmark-test` (7 cases). The benchmark only builds under `PAIMON_BUILD_BENCHMARKS`, which CI does not set, so this test is what keeps the format-layer assumptions it rests on covered: - `RegisteredCodecsWrite` — every registered codec writes and reads back with matching values; asserts `"lz4"` is still rejected (it resolves to Arrow's `LZ4_FRAME`, which `parquet::IsCodecSupported` rejects — this is why the sweep registers `lz4_raw` and `lz4_hadoop` instead). - `DictionaryInputRoundTrip` — dictionary-encoded `VARCHAR` / `INT32` input over multiple `AddBatch` calls, compared value by value against the flat equivalent. - `DecimalPrecisionRoundTrip` — precision 9 / 18 / 38, checking type and values (38 is the only one reaching `FIXED_LEN_BYTE_ARRAY`). - `NestedRoundTrip` — `LIST` / `MAP` round trip, full row and per-column projection. - `FilteredAndBitmapRowCounts` — predicate row counts bounded at the row-group grain, and selection-bitmap lower bound. - `PlainAndDictionaryFilesAgree` — the two files the encoding case compares must decode identically. - `MemoryThresholdFlushesRowGroups` — a small `parquet.writer.max.memory.use` cuts extra row groups on both plain and dictionary input, and row-group boundaries do not change the data. > **Validation status:** at the requester's instruction, no build, unit test, benchmark or CTest run was performed locally. `pre-commit` (clang-format, cpplint, codespell, sphinx-lint, cmake-format) passes on all changed files, and `git diff --check` is clean. **CI is the first compile of this code** — please treat a red build as expected-to-be-possible rather than surprising, particularly around `arrow::Concatenate` over struct arrays containing `MAP`/dictionary children, and the 8 KiB threshold in `MemoryThresholdFlushesRowGroups` actually triggering a flush. ### API and Format No. Nothing under `include/` changes, and no storage format or protocol is affected. The new files are a benchmark binary and its test; the only existing files touched are `benchmark/CMakeLists.txt` and the benchmark docs. ### Documentation Yes — `docs/source/examples/benchmark.rst` gains a `paimon-parquet-format-benchmark` section covering the case inventory, how to read each counter (including why filtered cases must be compared on `ns_per_input_row` rather than `ns_per_row`, and that `row_groups_after_filter` is row-group level and does not show page-level pruning), and the explicit statement that results are not comparable with Velox's. ### Generative AI tooling Generated-by: Claude Opus 5 (Claude Code) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
