[
https://issues.apache.org/jira/browse/SPARK-36528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated SPARK-36528:
-----------------------------
Description: Currently Spark first decode (e.g., RLE/bit-packed, PLAIN)
into column vector and then operate on the decoded data. However, it may be
more efficient to directly operate on encoded data, for instance, performing
filter or aggregation on RLE-encoded data, or performing comparison over
dictionary-encoded string data. This can also potentially work with encodings
in Parquet v2 format, such as DELTA_BYTE_ARRAY. (was: Currently Spark first
decode (e.g., RLE/bit-packed, PLAIN) into column vector and then operate on the
decoded data. However, it may be more efficient to directly operate on encoded
data (e.g., when the data is using RLE encoding). This can also potentially
work with encodings in Parquet v2 format, such as DELTA_BYTE_ARRAY.)
> Implement lazy decoding for the vectorized Parquet reader
> ---------------------------------------------------------
>
> Key: SPARK-36528
> URL: https://issues.apache.org/jira/browse/SPARK-36528
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 3.3.0
> Reporter: Chao Sun
> Priority: Major
>
> Currently Spark first decode (e.g., RLE/bit-packed, PLAIN) into column vector
> and then operate on the decoded data. However, it may be more efficient to
> directly operate on encoded data, for instance, performing filter or
> aggregation on RLE-encoded data, or performing comparison over
> dictionary-encoded string data. This can also potentially work with encodings
> in Parquet v2 format, such as DELTA_BYTE_ARRAY.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
