lucasfang opened a new issue, #225: URL: https://github.com/apache/paimon-cpp/issues/225
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar. ### Motivation # Issue > Template: `.github/ISSUE_TEMPLATE/feature.yaml`, label: `enhancement` ## Title [Feature] Support parquet page row count limit ## Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar. ## Motivation The Paimon-cpp Parquet writer currently splits data pages only by byte size (`parquet.page.size`, default 1MB), so a single page may contain hundreds of thousands or even millions of rows. This causes two problems: 1. **Read amplification**: the filtering granularity of the page index (ColumnIndex/OffsetIndex) is entirely determined by page size. When a predicate selects only a few rows, the reader still has to read and decode the whole large page, which dilutes the benefit of page-level filtering. 2. **Inconsistency with parquet-mr**: the Java side limits rows per page via `parquet.page.row.count.limit` (PARQUET-1414, default 20000). The C++ writer lacks the corresponding capability, so files written by C++ and Java differ significantly in page layout, making read-path behavior unpredictable in mixed read/write scenarios. ## Solution Following PARQUET-1414 of parquet-mr, add a row count limit for Parquet data pages; a page is finished when either the row count or the byte size limit is reached first: 1. **Arrow patch (`cmake_modules/arrow.diff`)**: add a `data_page_row_count_limit` builder method, getter, and the constant `DEFAULT_DATA_PAGE_ROW_COUNT_LIMIT = 20000` (aligned with the parquet-mr default) to `parquet::WriterProperties`; in `column_writer.cc`, extend the page-split condition with a `num_buffered_rows_ >= properties_->data_page_row_count_limit()` check in addition to the byte-size check. 2. **Paimon-side wiring**: add the table option `parquet.page.row.count.limit` in `parquet_format_defs.h` (the key name is identical to parquet-mr's); `ParquetWriterBuilder::PrepareWriterProperties` reads the option and applies it to the arrow `WriterProperties`, falling back to the default 20000 when unset. 3. **Semantics**: the limit is checked at write batch granularity, so a page may exceed the limit by up to one write batch, consistent with parquet-mr; existing files are unaffected, only the page layout of newly written files changes. ## Anything else? - This feature aligns with PR #459 (feat: support parquet page row count limit) of alibaba/paimon-cpp; its arrow patch parts (properties.h, column_writer.cc) are byte-for-byte identical. - The default row-count page split changes writer memory behavior: pages of highly compressible data (e.g., all-null columns) finish early due to the row limit, so the writer peak memory stays well below the budget; the assertions in `ParquetFormatWriterTest.TestMemoryControl` are adjusted accordingly. - Smaller pages bring slightly more page header / offset index metadata overhead, a known trade-off controllable via the row limit. ## Are you willing to submit a PR? - [x] I'm willing to submit a PR! ### Solution _No response_ ### Anything else? _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
