maytasm opened a new pull request, #8412: URL: https://github.com/apache/iceberg/pull/8412
Spark 3 fail to plan query with filter on short type literal i.e. `select * from some_hive_table_with_smallint_column where small_int_column = 1` where `small_int_column` is a smallint type column. So even if the 1 is cast as int or the `small_int_column` is cast as int, Spark will convert the 1 literal to short type. When this filter (with short type literal) is pushed down to Iceberg, Iceberg throws an Exception saying that Cannot create expression literal from java.lang.Short This problem is also reproducible when the column is tinyint. In this case, Iceberg throws an Exception saying that Cannot create expression literal from java.lang.Byte Note that this query works in Spark 2.4 as Spark 2.4 doesn't push down the query to Iceberg. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
