Gabriel39 opened a new pull request, #69971:
URL: https://github.com/apache/doris/pull/69971
### What problem does this PR solve?
Parquet output relies on implicit page-index and data-page-size defaults, so
users cannot explicitly omit indexes or tune the target page size.
Add two validated session variables shared by OUTFILE, EXPORT, Hive and
Iceberg Parquet writers:
- `enable_parquet_write_page_index`: write ColumnIndex/OffsetIndex metadata;
defaults to `true`, preserving the current master behavior.
- `parquet_write_data_page_size`: target uncompressed data-page size in
bytes; defaults to `1048576`, accepts positive 32-bit integers, and applies
with indexes enabled or disabled.
Keep the existing 128 KiB target cap for Hadoop-framed LZ4. Other codecs use
the requested target. Row-group size, dictionary-page size, and
data-page/format versions are unchanged. Optional query fields preserve
existing defaults when omitted by older FEs.
### Release note
Add session controls for Parquet page-index output and target data-page size.
### Validation
- 16 BE ASAN tests passed, including real file metadata/readback, index
on/off, dictionary on/off, null/all-null columns, Parquet versions, Data Page
V1/V2, page offset bounds, Hadoop-framed LZ4 limits, and Hive/Iceberg partition
writers.
- 9 FE tests passed for defaults, variable forwarding, Thrift serialization,
valid bounds, and invalid input.
- The new tests failed for the missing settings before implementation.
- FE Checkstyle, clang-format 16, build hygiene, and clang-tidy passed.
- No cluster SQL run or performance benchmark was performed locally; CI
validation is pending.
### Check List (For Author)
- Test:
- [x] Unit Test
- Behavior changed:
- [x] Yes. Explicit controls are available; current defaults are preserved.
- Does this need documentation?
- [x] Yes. Paired English and Chinese documentation will be linked here.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
