loudongfeng commented on a change in pull request #925:
URL: https://github.com/apache/parquet-mr/pull/925#discussion_r699976767
##########
File path:
parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java
##########
@@ -201,9 +201,23 @@ public FileMetaData toParquetMetadata(int currentVersion,
ParquetMetadata parque
List<BlockMetaData> blocks = parquetMetadata.getBlocks();
List<RowGroup> rowGroups = new ArrayList<RowGroup>();
long numRows = 0;
+ long preBlockStartPos = 0;
+ long preBlockCompressedSize = 0;
for (BlockMetaData block : blocks) {
numRows += block.getRowCount();
+ long blockStartPos = block.getStartingPos();
+ // first block
+ if (blockStartPos == 4) {
+ preBlockStartPos = 0;
+ preBlockCompressedSize = 0;
+ }
+ if (preBlockStartPos != 0) {
+ assert blockStartPos >= preBlockStartPos + preBlockCompressedSize;
Review comment:
There are alignments between row groups. see PARQUET-306 and
https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileWriter.java#L411
##########
File path:
parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java
##########
@@ -201,9 +201,23 @@ public FileMetaData toParquetMetadata(int currentVersion,
ParquetMetadata parque
List<BlockMetaData> blocks = parquetMetadata.getBlocks();
List<RowGroup> rowGroups = new ArrayList<RowGroup>();
long numRows = 0;
+ long preBlockStartPos = 0;
+ long preBlockCompressedSize = 0;
for (BlockMetaData block : blocks) {
numRows += block.getRowCount();
+ long blockStartPos = block.getStartingPos();
+ // first block
+ if (blockStartPos == 4) {
Review comment:
To address _common_metadata file case, which will merge multiple file
footers into a single meta file.
##########
File path:
parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java
##########
@@ -1254,14 +1276,41 @@ static FileMetaData
filterFileMetaDataByMidpoint(FileMetaData metaData, RangeMet
return metaData;
}
+ private static long tryUseFileOffset(RowGroup rowGroup, long preStartIndex,
long preCompressedSize) {
+ long startIndex = 0;
+ startIndex = rowGroup.getFile_offset();
+ // skip checking the first rowGroup
+ // (in case of summary file, there are multiple first groups from
different footers)
+ if (preStartIndex != 0 && preStartIndex <= startIndex) {
+
+ //calculate start index for other blocks
+ long minStartIndex = preStartIndex + preCompressedSize;
+ if (startIndex < minStartIndex) {
+ // a bad offset detected, try first column's offset
+ // can not use minStartIndex in case of padding
+ startIndex = getOffset(rowGroup.getColumns().get(0));
Review comment:
In case of encrypted files with wongly setted file_offset, i have no
idea how to fix it when alignment padding take place.
If no padding, then we can just use the calculated index.
I didn't find any footer meta about padding position or something else
indicating padding or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
