Yang Jie created SPARK-61174:
--------------------------------
Summary: Built-in V2 file scans should report
columnarSupportMode() instead of packing every file at planning
Key: SPARK-61174
URL: https://issues.apache.org/jira/browse/SPARK-61174
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Yang Jie
`FileScan` keeps the default `columnarSupportMode()`, `PARTITION_DEFINED`, so
`DataSourceV2Strategy` asking `supportsColumnar` plans `inputPartitions`, which
splits and packs every listed file at planning. With SPARK-30628, a scan that a
runtime filter prunes packs the surviving files again at execution, so with DPP
keeping 10 of 10,000 partitions of a table with a million files, the driver
packs a million files at planning only to learn one boolean. V1 packs only the
partitions it selects.
The reader factories decide columnar reads from the schema and the conf, not
from the partition: Parquet and ORC from their vectorized reader settings and
the result schema, and CSV, JSON, text and Avro never read columnar. So
`ParquetScan` and `OrcScan` could report `SUPPORTED` or `UNSUPPORTED` from the
same check their factories use, and the other four `UNSUPPORTED`, leaving the
default of `FileScan` for other subclasses.
Found in the review of #55579 (SPARK-30628).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
