beliefer commented on code in PR #44006:
URL: https://github.com/apache/spark/pull/44006#discussion_r1410191710
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala:
##########
@@ -613,7 +613,13 @@ class ParquetFilters(
value == null || (nameToParquetField(name).fieldType match {
case ParquetBooleanType => value.isInstanceOf[JBoolean]
case ParquetIntegerType if value.isInstanceOf[Period] => true
- case ParquetByteType | ParquetShortType | ParquetIntegerType =>
value.isInstanceOf[Number]
+ case ParquetByteType | ParquetShortType | ParquetIntegerType => value
match {
+ // Byte/Short/Int are all stored as INT32 in Parquet so filters are
built using type Int.
+ // We don't create a filter if the value would overflow.
+ case _: JByte | _: JShort | _: Integer => true
+ case v: JLong => v.longValue() >= Int.MinValue && v.longValue() <=
Int.MaxValue
Review Comment:
For simply, how about forbid the Long value direclty ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
