[
https://issues.apache.org/jira/browse/SPARK-36529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated SPARK-36529:
-----------------------------
Attachment: (was: image.png)
> Decouple CPU with IO work in vectorized Parquet reader
> ------------------------------------------------------
>
> Key: SPARK-36529
> URL: https://issues.apache.org/jira/browse/SPARK-36529
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 3.3.0
> Reporter: Chao Sun
> Priority: Major
>
> Currently it seems the vectorized Parquet reader does almost everything in a
> sequential manner:
> 1. read the row group using file system API (perhaps from remote storage like
> S3)
> 2. allocate buffers and store those row group bytes into them
> 3. decompress the data pages
> 4. in Spark, decode all the read columns one by one
> 5. read the next row group and repeat from 1.
> A lot of improvements can be done to decouple the IO and CPU intensive work.
> In addition, we could parallelize the row group loading and column decoding,
> and utilizing all the cores available for a Spark task.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
