Hi everyone,

I would like to start a discussion on “Lazy Materialization for Parquet Read 
Performance Improvement"

Chao and I propose a Parquet reader with lazy materialization. For Spark-SQL 
filter operations, evaluating the filters first and lazily materializing only 
the used values can save computation wastes and improve the read performance.
The current implementation of Spark requires the read values to materialize 
(i.e. decompress, de-code, etc...) onto memory first before applying the 
filters even though the filters may eventually throw away many values.

We made our design doc as follows.
SPIP Jira: https://issues.apache.org/jira/browse/SPARK-42256 
SPIP Doc: 

Liang-Chi was kind enough to shepherd this effort. 

Thank you

Reply via email to