ggershinsky commented on a change in pull request #925:
URL: https://github.com/apache/parquet-mr/pull/925#discussion_r700112078
##########
File path:
parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java
##########
@@ -1254,14 +1276,41 @@ static FileMetaData
filterFileMetaDataByMidpoint(FileMetaData metaData, RangeMet
return metaData;
}
+ private static long tryUseFileOffset(RowGroup rowGroup, long preStartIndex,
long preCompressedSize) {
+ long startIndex = 0;
+ startIndex = rowGroup.getFile_offset();
+ // skip checking the first rowGroup
+ // (in case of summary file, there are multiple first groups from
different footers)
+ if (preStartIndex != 0 && preStartIndex <= startIndex) {
+
+ //calculate start index for other blocks
+ long minStartIndex = preStartIndex + preCompressedSize;
+ if (startIndex < minStartIndex) {
+ // a bad offset detected, try first column's offset
+ // can not use minStartIndex in case of padding
+ startIndex = getOffset(rowGroup.getColumns().get(0));
Review comment:
@loudongfeng @gszadovszky @shangxinli @sunchao . Looks like the problem
is only in one mode of encryption ("encrypted footer", where the ColumnMetaData
is not set). Let me propose the following:
long minStartIndex = preStartIndex + preCompressedSize;
if (startIndex < minStartIndex) {
// a bad offset detected, try first column's offset if available
ColumnChunk columnChunk = rowGroup.getColumns().get(0);
if (columnChunk.isSetMeta_data()) {
startIndex = getOffset(columnChunk);
} else { // EncryptedFooter mode. Plaintext ColumnMetaData is not
available.
// use minStartIndex (imprecise in case of padding, but good enough
for filtering)
startIndex = minStartIndex;
}
what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
