[
https://issues.apache.org/jira/browse/SPARK-36528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated SPARK-36528:
-----------------------------
Description: Currently Spark first decode (e.g., RLE/bit-packed, PLAIN)
into column vector and then operate on the decoded data. However, it may be
more efficient to directly operate on encoded data (e.g., when the data is
using RLE encoding). This can also potentially work with encodings in Parquet
v2 format, such as DELTA_BYTE_ARRAY. (was: Currently Spark first decode (e.g.,
RLE/bit-packed, PLAIN) into column vector and then operate on the decoded data.
However, it may be more efficient to directly operate on encoded data (e.g.,
when the data is using RLE encoding).)
> Implement lazy decoding for the vectorized Parquet reader
> ---------------------------------------------------------
>
> Key: SPARK-36528
> URL: https://issues.apache.org/jira/browse/SPARK-36528
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 3.3.0
> Reporter: Chao Sun
> Priority: Major
>
> Currently Spark first decode (e.g., RLE/bit-packed, PLAIN) into column vector
> and then operate on the decoded data. However, it may be more efficient to
> directly operate on encoded data (e.g., when the data is using RLE encoding).
> This can also potentially work with encodings in Parquet v2 format, such as
> DELTA_BYTE_ARRAY.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
