[
https://issues.apache.org/jira/browse/PARQUET-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17823759#comment-17823759
]
Claire McGinty commented on PARQUET-2443:
-----------------------------------------
I pushed a branch implementing the DataPage-as-Iterator idea, here:
[https://github.com/apache/parquet-mr/compare/master...clairemcginty:parquet-mr:lazy-chunkreader]
However... it looks like ColumnReaderBase#checkRead works by continually
invoking `readPage` [until the row group is fully
consumed|https://github.com/apache/parquet-mr/blob/master/parquet-column/src/main/java/org/apache/parquet/column/impl/ColumnReaderBase.java#L653-L676],
so the row group effectively gets materialized there. Any pointers on how I
should modify that code block?
> Support lazy materialization of row groups in ParquetFileReader
> ---------------------------------------------------------------
>
> Key: PARQUET-2443
> URL: https://issues.apache.org/jira/browse/PARQUET-2443
> Project: Parquet
> Issue Type: Improvement
> Reporter: Claire McGinty
> Priority: Major
>
> Motivation: The current behavior of ParquetFilterReader#readNextRowGroup is
> to eagerly enumerate all chunks in the row group, then read all pages in the
> chunk. For distributed data workloads, this can cause significant memory
> pressure, particularly for use cases that require the colocation of multiple
> Parquet files on a single worker.
>
> Proposal: A Parquet Configuration option that enables lazy row group reading,
> i.e., only a page at a time (plus whatever header is necessary to read that
> header). The Configuration option could be either a flag, or an int value for
> how many pages/page bytes to buffer at a time.
>
> I think this could be accomplished by modifying
> [ParquetFileReader#readAllPages|https://github.com/apache/parquet-mr/blob/cf294a35ca4b635f44357fe4c170f3b66870dc6e/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java#L1727]
> to re-implement pagesInChunk as an Iterator<DataPage>, rather than a
> List<DataPage>. Then, ColumnChunkPageReader could parse the Configuration
> option above and decide whether to fully materialize the iterator or not.
>
> I'm happy to try to create a draft/branch for this to get some early feedback
> on the idea!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
