SteNicholas commented on issue #229: URL: https://github.com/apache/paimon-cpp/issues/229#issuecomment-5353820971
Closing this as invalid — the premise is wrong, and I should have checked before filing. `ParquetWriterBuilder::PrepareWriterProperties` already calls `builder.enable_store_decimal_as_integer()` unconditionally (`src/paimon/format/parquet/parquet_writer_builder.cpp:68`), and has done so since the format layer was introduced in #48. The option lives on `::parquet::WriterProperties`, not on `::parquet::ArrowWriterProperties` — I checked only the latter (`parquet_format_writer.cpp:58`, which sets just `enable_deprecated_int96_timestamps`) and wrongly concluded it was unset. So there is no divergence to fix. Arrow's mapping under `store_decimal_as_integer` (`cpp/src/parquet/arrow/schema.cc:362`): - `1 <= precision <= 9` → `INT32` - `9 < precision <= 18` → `INT64` - otherwise → `FIXED_LEN_BYTE_ARRAY` is exactly Java Paimon's `ParquetSchemaConverter.is32BitDecimal` (`precision <= 9`) / `is64BitDecimal` (`precision <= 18 && precision > 9`) split. paimon-cpp and Java Paimon already produce identical physical types for every decimal precision, short decimals already take arrow's `DecimalIntegerTransfer` fast path on read, and the size trade-off table in the description is moot. Existing tests already depend on this: `parquet_stats_extractor_test.cpp:78,291` and `file_reader_wrapper_test.cpp:224` set the same property when constructing fixtures. The one thing from [velox#17992](https://github.com/facebookincubator/velox/pull/17992) that survives is vectorizing arrow's `DecimalIntegerTransfer` (`cpp/src/parquet/arrow/reader_internal.cc:645`), which converts one value at a time. That stays where it belongs — a measurement candidate under #228, not a change to make on speculation. Sorry for the noise. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
