[
https://issues.apache.org/jira/browse/SPARK-26345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757209#comment-16757209
]
Zoltan Ivanfi commented on SPARK-26345:
---------------------------------------
Please note that column indexes will automatically get utilized if
[spark.sql.parquet.enableVectorizedReader|https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-vectorized-parquet-reader.html#spark.sql.parquet.enableVectorizedReader]
= false. If spark.sql.parquet.enableVectorizedReader = true, on the other hand
(which is the default), then column indexes could only be utilized by
duplicating the internal logic of parquet-mr, which would be disproportonate
effort. We, the developers of the column index feature did not expect Spark to
make this huge investment, and we would like to provide a vectorized API
instead in a future release.
> Parquet support Column indexes
> ------------------------------
>
> Key: SPARK-26345
> URL: https://issues.apache.org/jira/browse/SPARK-26345
> Project: Spark
> Issue Type: Umbrella
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Yuming Wang
> Priority: Major
>
> Parquet 1.11.0 supports column indexing. Spark can supports this feature for
> good read performance.
> More details:
> https://issues.apache.org/jira/browse/PARQUET-1201
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
