Chao Sun created SPARK-32694: -------------------------------- Summary: Pushdown cast to data sources Key: SPARK-32694 URL: https://issues.apache.org/jira/browse/SPARK-32694 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.0.0 Reporter: Chao Sun
Currently we don't support pushing down cast to data source (see [link|http://apache-spark-developers-list.1001551.n3.nabble.com/SparkSql-Casting-of-Predicate-Literals-tp29956p30035.html] for a discussion). For instance, in the following code snippet: {code} case class Person(name: String, age: Short) Seq(Person("John", 32), Person("David", 25), Person("Mike", 18)).toDS().write.parquet("/tmp/person.parquet") val personDS = spark.read.parquet("/tmp/person.parquet") personDS.createOrReplaceTempView("person") spark.sql("SELECT * FROM person where age < 30") {code} The predicate won't be pushed down to Parquet data source because in {{DataSourceStrategy}}, {{PushableColumnBase}} only handles a few limited cases such as {{Attribute}} and {{GetStructField}}. Potentially we can handle {{Cast}} here as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org