divjotarora commented on code in PR #3798:
URL: https://github.com/apache/parquet-java/pull/3798#discussion_r4047831279
##########
parquet-cli/src/main/java/org/apache/parquet/cli/commands/ShowPagesCommand.java:
##########
@@ -229,9 +229,7 @@ private long getPageCompressedSize() {
}
private String printDictionaryPage(DictionaryPage dict) {
- // TODO: the compressed size of a dictionary page is lost in Parquet
- dict.getUncompressedSize();
- long totalSize = dict.getCompressedSize();
+ long totalSize = getPageCompressedSize();
Review Comment:
`getPageCompressedSize` calls `columnChunk.hasDictionaryPage()` to determine
the page offset. This function does:
```
public boolean hasDictionaryPage() {
EncodingStats stats = getEncodingStats();
if (stats != null) {
// ensure there is a dictionary page and that it is used to encode
data pages
return stats.hasDictionaryPages() && stats.hasDictionaryEncodedPages();
}
Set<Encoding> encodings = getEncodings();
return (encodings.contains(PLAIN_DICTIONARY) ||
encodings.contains(RLE_DICTIONARY));
}
```
In an edge case where a writer emits a dictionary page followed by no
`PLAIN_DICTIONARY` or `RLE_DICTIONARY` data pages, we would get wrong results
because `hasDictionaryPage()` returns false.
Realistically, writers wouldn't do this, but I didn't find any wording in
the spec explicitly disallowing it. The `TestParquetFileWriter`
[code](https://github.com/apache/parquet-java/blob/3f1261ce08dc50d02dc8d93d19413b2e0cf7d985/parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestParquetFileWriter.java#L229-L235)
in this repo seems to do exactly this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
