wgtmac commented on issue #42368: URL: https://github.com/apache/arrow/issues/42368#issuecomment-6593582584
I asked AI to investigate this and unfortunately it is not resolved yet. Below is its response: -------- **Root cause** Parquet’s deprecated `BIT_PACKED` level encoding is packed **most-significant-bit first**. Arrow’s current decoder reuses the logic for the newer RLE/bit-packed hybrid encoding, which is **least-significant-bit first**. For levels with a bit width greater than 1, this produces incorrect repetition/definition levels. With a bit width of 1, both orders look identical, which is why many existing tests still pass. A minimal check against the current Arrow code: ```text Bytes: 05 39 77, width=3 Expected: 0,1,2,3,4,5,6,7 Arrow returns: 5,0,4,4,3,6,5,3 ``` **Test coverage is missing** `parquet-testing` contains many files whose metadata mentions `BIT_PACKED`, including: - `nullable.impala.parquet` - `alltypes_tiny_pages.parquet` - `nested_lists.snappy.parquet` - `fixed_length_decimal_legacy.parquet` But looking at the actual data-page headers: - Nested columns generally use `RLE` for definition levels. - Some files report `rep=BIT_PACKED`, but the maximum repetition level is 0, so no repetition levels are actually decoded. - In files such as `fixed_length_decimal_legacy.parquet`, the definition level needs only 1 bit, so MSB-first and LSB-first produce identical results. - Files that genuinely use `def=BIT_PACKED`, such as `floating_orders_nan_count.parquet`, contain only `REQUIRED` columns, so there are no levels to exercise. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
