[ https://issues.apache.org/jira/browse/SPARK-27789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun updated SPARK-27789: ---------------------------------- Affects Version/s: (was: 2.4.3) (was: 2.4.2) (was: 2.4.1) (was: 2.4.0) 3.0.0 > Use stopEarly in codegen of ColumnarBatchScan > --------------------------------------------- > > Key: SPARK-27789 > URL: https://issues.apache.org/jira/browse/SPARK-27789 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.0.0 > Reporter: EdisonWang > Priority: Minor > > Suppose that we have a hive table like this > ```sql("create table parquet_test (id int) using parquet")```, and our query > sql is `select id from parquet_test limit 10`. > With `spark.sql.hive.convertMetastoreParquet` set to false, the sql > execution will go into `InputAdapter`, in its codegen, it can use `stopEarly` > to accelerate local limit. > But if we set `spark.sql.hive.convertMetastoreParquet` to true, the sql > exectuion will go into `ColumnarBatchScan`, which didn't optimize local limit. > In this patch, We use `stopEarly` in `ColumnarBatchScan` as well -- This message was sent by Atlassian JIRA (v7.6.14#76016) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org