JingsongLi commented on PR #9976: URL: https://github.com/apache/paimon/pull/9976#issuecomment-5746971265
I think the default behavior should align with the Java writer: write the Parquet Page Index whenever the installed PyArrow version supports it, instead of defaulting to `false`. Paimon Java, through parquet-java, already writes `ColumnIndex` and `OffsetIndex` by default. Keeping Python disabled by default means that files written to the same table have different metadata depending on the writer implementation. A plain `default_value(True)` would break the PyArrow 6/7 versions that PyPaimon still supports, so I suggest capability-aware, three-state semantics: - unset: enable automatically on PyArrow >= 13; omit the argument on older versions; - explicit `true`: enable it, and fail before creating output when PyArrow < 13; - explicit `false`: always disable it. This could be implemented by making the option have no configured default and resolving the default in `create_parquet_writer_options` based on the installed PyArrow version. The documentation could describe the default as: "enabled when supported by the installed PyArrow version (PyArrow >= 13); otherwise disabled." The tests should then expect `setting=None` to produce Page Indexes on PyArrow >= 13, while a mocked PyArrow 12 should still write successfully without passing `write_page_index`. The option name itself looks reasonable to me: "Page Index" is the Parquet specification term, and `write-page-index` is consistent with PyArrow `write_page_index` and Impala `PARQUET_WRITE_PAGE_INDEX`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
