wgtmac commented on code in PR #1174:
URL: https://github.com/apache/parquet-mr/pull/1174#discussion_r1363512338
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/rewrite/ParquetRewriter.java:
##########
@@ -265,6 +265,10 @@ private void processBlocksFromReader() throws IOException {
BlockMetaData blockMetaData = meta.getBlocks().get(blockId);
List<ColumnChunkMetaData> columnsInOrder = blockMetaData.getColumns();
+ List<ColumnIndex> columnIndexes = readAllColumnIndexes(reader,
columnsInOrder, descriptorsMap);
Review Comment:
> Do you mean to read the indexes column by column to reduce memory
footprint?
No, my suggested interface does not restrict any implementation detail, at
least they should be ready at the `readXXX()` call. You can still read all
indexes at once (controlled by a config). We can configurated it to release any
consumed index object to reduce memory footprint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
