[ 
https://issues.apache.org/jira/browse/SPARK-61174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18145217#comment-18145217
 ] 

Hemanth Boyina commented on SPARK-61174:
----------------------------------------

Hi [~LuciferYang] , are you planning to work on this ? If not I would be 
interested to pick this up. Please let me know if that would be okay.

> Built-in V2 file scans should report columnarSupportMode() instead of packing 
> every file at planning
> ----------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61174
>                 URL: https://issues.apache.org/jira/browse/SPARK-61174
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Yang Jie
>            Priority: Major
>
> `FileScan` keeps the default `columnarSupportMode()`, `PARTITION_DEFINED`, so 
> `DataSourceV2Strategy` asking `supportsColumnar` plans `inputPartitions`, 
> which splits and packs every listed file at planning. With SPARK-30628, a 
> scan that a runtime filter prunes packs the surviving files again at 
> execution, so with DPP keeping 10 of 10,000 partitions of a table with a 
> million files, the driver packs a million files at planning only to learn one 
> boolean. V1 packs only the partitions it selects.
> The reader factories decide columnar reads from the schema and the conf, not 
> from the partition: Parquet and ORC from their vectorized reader settings and 
> the result schema, and CSV, JSON, text and Avro never read columnar. So 
> `ParquetScan` and `OrcScan` could report `SUPPORTED` or `UNSUPPORTED` from 
> the same check their factories use, and the other four `UNSUPPORTED`, leaving 
> the default of `FileScan` for other subclasses.
> Found in the review of #55579 (SPARK-30628).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to