lucasfang opened a new pull request, #226: URL: https://github.com/apache/paimon-cpp/pull/226
# PR Description > Template: `.github/PULL_REQUEST_TEMPLATE.md` ## PR Title feat(parquet): support data page row count limit for parquet writes ### Purpose Linked issue: close #225 Support splitting Parquet data pages by row count, aligned with parquet-mr's `parquet.page.row.count.limit` (PARQUET-1414). Previously the Parquet writer split pages only by byte size (`parquet.page.size`), so a single page could contain a very large number of rows, leading to coarse page-index filtering granularity, read amplification, and page layouts inconsistent with files written by the Java side. This PR introduces a row count limit (default 20000, same as parquet-mr); a page is finished when either the row count or the byte size limit is reached first. Changes: - `cmake_modules/arrow.diff`: patch arrow's `parquet::WriterProperties` to add the `data_page_row_count_limit` builder method, getter, and the constant `DEFAULT_DATA_PAGE_ROW_COUNT_LIMIT = 20000`; extend the page-split condition in `column_writer.cc` with a `num_buffered_rows_ >= properties_->data_page_row_count_limit()` check in addition to the byte-size check. - `src/paimon/format/parquet/parquet_format_defs.h`: add the table option key `PARQUET_PAGE_ROW_COUNT_LIMIT = "parquet.page.row.count.limit"` (the key name is identical to parquet-mr's). - `src/paimon/format/parquet/parquet_writer_builder.cpp`: `PrepareWriterProperties` reads the option and applies it to the arrow `WriterProperties`, falling back to the default 20000 when unset. - `src/paimon/format/parquet/parquet_format_writer_test.cpp`: update the `TestMemoryControl` assertions — with the default row-count page split in effect, all-null pages are RLE-encoded and finish early, so the writer peak memory stays far below the budget; for the non-null case the budget upper bound is relaxed to 2.5x (the budget is mostly held as finished page buffers, which BufferedPageWriter copies into its in-memory sink when the row group is flushed, transiently doubling the footprint; ~2.2x measured). Note: the limit is checked at write batch granularity, so a page may exceed the limit by up to one write batch (consistent with parquet-mr semantics). ### Tests - UT: `ParquetWriterBuilderTest.DefaultPrepareWriterProperties` adds a default-value assertion (`data_page_row_count_limit == 20000`); `ParquetWriterBuilderTest.PrepareWriterProperties` adds an assertion that the option `parquet.page.row.count.limit=40000` is propagated. - UT: `ParquetFormatWriterTest.TestMemoryControl` adapts the memory assertions to the new default page-split behavior. - Actual results: `paimon-parquet-format-test` 187/187 passed, `paimon-write-inte-test` 117/117 passed, `paimon-read-inte-test` 278/278 passed. ### API and Format No change to the public API under `include/`. Adds the table option `parquet.page.row.count.limit` (default 20000). The storage format stays compatible: only the data page layout of newly written files changes (more pages, bounded rows per page); no new format feature is introduced and older readers can read the files normally. ### Documentation New table option `parquet.page.row.count.limit`: the maximum number of rows in a single data page, aligned with parquet-mr's option of the same name (PARQUET-1414), default 20000; a page is finished when either this row count or the byte size (`parquet.page.size`) is reached first. ### Generative AI tooling Generated-by: Qoder -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
