1fanwang opened a new pull request, #3798: URL: https://github.com/apache/parquet-java/pull/3798
### Rationale for this change `parquet pages` overstates dictionary sizes in compressed files. For the GZIP file below, it reports 248 B even though the stored dictionary is 33 B. That makes the output misleading when investigating compression. The data-page fix in https://github.com/apache/parquet-java/pull/3326 left the dictionary row using the decompressed size. Closes https://github.com/apache/parquet-java/issues/2870. ### What changes are included in this PR? Use the existing page-header lookup for dictionary size and average size, as the data-page rows already do. Files and the core page API are unchanged. ### Are these changes tested? #### Testing Done Built the standalone CLI with JDK 17 and Thrift 0.24.0, then ran both versions against the same file: | Scenario | Command | Before | After | | --- | --- | --- | --- | | GZIP dictionary | `java -jar <cli.jar> pages -c color example.parquet` | 248 B; 124.00 B per entry | 33 B; 16.50 B per entry | | Stored page header | `java -jar <cli.jar> pages --raw -c color example.parquet` | 33 compressed bytes | Unchanged | The baseline is https://github.com/apache/parquet-java/commit/2df8d02678dab4bb8b926a0d3221cc652984c7ab. The fixed jar comes from this branch. Build commands, with the resulting runtime jars saved as `before-cli.jar` and `after-cli.jar`: ```sh ./mvnw -B -ntp -pl parquet-cli -am -Plocal -DskipTests package ./mvnw -B -ntp -pl parquet-cli -am -Plocal -Dtest=ShowPagesCommandTest,ConvertCSVCommandTest -Dsurefire.failIfNoSpecifiedTests=false package ``` Commands run with those jars: ```sh python3 -c 'from pathlib import Path; Path("input.csv").write_text("color\n" + ("a" * 120 + "\n" + "b" * 120 + "\n") * 100)' java -jar before-cli.jar convert-csv input.csv --require color --compression-codec GZIP -o example.parquet java -jar before-cli.jar pages -c color example.parquet java -jar after-cli.jar pages -c color example.parquet java -jar after-cli.jar pages --raw -c color example.parquet ``` <details> <summary>Raw logs</summary> ```text Before Column: color -------------------------------------------------------------------------------- page type enc count avg size size rows nulls min / max 0-D dict G _ 2 124.00 B 248 B 0-1 data G R 200 0.13 B 25 B After Column: color -------------------------------------------------------------------------------- page type enc count avg size size rows nulls min / max 0-D dict G _ 2 16.50 B 33 B 0-1 data G R 200 0.13 B 25 B Raw headers Start of chunk (rowGroup: 0, columnName: color, dictPageOffset: 4, dataPageOffset: 57, numValues: 200, totalSize: 102) Page 0. (offset: 4, headerSize: 20) { "compressed_page_size" : 33, "crc" : -205207594, "dictionary_page_header" : { "encoding" : 2, "num_values" : 2 }, "type" : 2, "uncompressed_page_size" : 248 } Page 1. (offset: 57, headerSize: 24) { "compressed_page_size" : 25, "crc" : 332693654, "data_page_header" : { "definition_level_encoding" : 4, "encoding" : 2, "num_values" : 200, "repetition_level_encoding" : 4 }, "type" : 0, "uncompressed_page_size" : 27 } End of chunk (offset: 105) ``` </details> Repeated the CLI comparison with V2 pages and an uncompressed file. The GZIP dictionary reports 33 B in both versions; data-page output and raw headers are unchanged. Uncompressed output is identical before and after. The regression writes real files with both page versions, GZIP or no compression, and dictionary encoding on or off. It compares the command's size and average-size columns with the original page headers. The GZIP dictionary cases fail before the fix. ### Are there any user-facing changes? Dictionary rows now report on-disk compressed bytes rather than decompressed bytes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
