beliefer commented on code in PR #44006:
URL: https://github.com/apache/spark/pull/44006#discussion_r1410609979
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala:
##########
@@ -613,7 +613,13 @@ class ParquetFilters(
value == null || (nameToParquetField(name).fieldType match {
case ParquetBooleanType => value.isInstanceOf[JBoolean]
case ParquetIntegerType if value.isInstanceOf[Period] => true
- case ParquetByteType | ParquetShortType | ParquetIntegerType =>
value.isInstanceOf[Number]
+ case ParquetByteType | ParquetShortType | ParquetIntegerType => value
match {
+ // Byte/Short/Int are all stored as INT32 in Parquet so filters are
built using type Int.
+ // We don't create a filter if the value would overflow.
+ case _: JByte | _: JShort | _: Integer => true
+ case v: JLong => v.longValue() >= Int.MinValue && v.longValue() <=
Int.MaxValue
Review Comment:
Thank you for the explanation. I got the two use case.
But users may not notice the detail and confused by the behavior. I think we
can delay the support until Delta table supports filter on long value.
Of course, skip creating row group filters if value exceeds the long range
looks good too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
