Joob1n opened a new pull request, #346: URL: https://github.com/apache/paimon-cpp/pull/346
### Purpose Linked issue: close #345 `ColumnIndexFilter::CompareEncodedWithLiteral` inferred the physical type of DECIMAL page statistics from the length of the encoded min/max — 4 bytes meant INT32, 8 bytes meant INT64 (both decoded with a native-endian `memcpy`), anything else was decoded as big-endian two's complement. Parquet allows DECIMAL as FIXED_LEN_BYTE_ARRAY for any precision, with length `ceil((precision * log2(10) + 1) / 8)`. Precision 7-9 yields exactly 4 bytes and precision 17-18 exactly 8, so those columns took the INT32/INT64 branch and had their big-endian bounds read little-endian. Page min `1.00` (unscaled 100, stored as `00 00 00 64`) read back as 1,677,721,600, so `col < 5.00` answered "no" and dropped a page that does match. Page-index filtering is pruning, not filtering, so nothing downstream brings those rows back. This passes the `::parquet::SchemaDescriptor` down from `FileReaderWrapper::CalculateFilteredRowRanges` through `ColumnIndexFilter` and dispatches on the column's physical type, the way `parquet_stats_extractor.cpp` already does for row-group statistics. INT32/INT64 keep the plain little-endian decode; FIXED_LEN_BYTE_ARRAY and BYTE_ARRAY go through `Decimal::FromUnscaledBytes`. When no schema is available the comparison is skipped, so no page is pruned. Paimon's own writers use INT32 for precision <= 9 and INT64 for precision <= 18, so this only reaches users through files written elsewhere: Arrow with the default `store_decimal_as_integer = false`, Spark with `spark.sql.parquet.writeLegacyFormat = true`, or Hive. ### Tests New cases in `src/paimon/format/parquet/column_index_filter_test.cpp`: - `ColumnIndexFilterTest.DecimalFixedLenByteArrayPages` — writes DECIMAL(9,2) and DECIMAL(18,2) columns as FIXED_LEN_BYTE_ARRAY with the page index enabled, asserts the writer really produced FLBA of length 4 and 8, then checks that `<`, `>` and `=` keep exactly the pages holding matching rows and prune the rest. Both cases fail on the previous decoding logic and pass with this change. - `ColumnIndexFilterTest.DecimalWithoutSchemaKeepsAllPages` — with no schema descriptor, no page is pruned. Run locally (macOS arm64, Debug, `-Dfmt_SOURCE=BUNDLED`): - `./build/debug/paimon-parquet-format-test --gtest_filter='ColumnIndexFilterTest.*:RowRangesTest.*'` — 36/36 pass. - Full `paimon-parquet-format-test` (225 cases, run one per process): 147 pass, 78 fail. Those 78 fail identically on unmodified `main` — they abort in fixture setup at `arrow/ipc/json_simple.cc:156 DCHECK(*out)` inside `ArrayFromJSON`, which is unrelated to this change and Debug-only. ### API and Format No. `ColumnIndexFilter::CalculateRowRanges` gains a `const ::parquet::SchemaDescriptor*` parameter, but the class is internal to `src/paimon/format/parquet/` and is not part of the public headers in `include/`. No storage format or protocol change. ### Documentation No. ### Generative AI tooling Generated-by: Claude Code (Claude Opus 5) https://claude.ai/code/session_01NnX2Q5dAwrmeWYmf9n1mCC -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
