manuzhang commented on code in PR #16665:
URL: https://github.com/apache/iceberg/pull/16665#discussion_r3613696970
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SparkBatch.java:
##########
@@ -189,6 +191,15 @@ private boolean supportsParquetBatchReads(ScanTask task) {
}
private boolean supportsParquetBatchReads(Types.NestedField field) {
+ // The vectorized Parquet reader exposes time values through Arrow's
TimeMicroVector, which
+ // returns microseconds, while Spark's TimeType expects nanoseconds. Until
the vectorized path
+ // performs that conversion, fall back to row-based reads when a time
column is projected. This
+ // check must run before the metadata column check to catch a time field
nested in a metadata
+ // column like _partition.
+ if (TypeUtil.find(field.type(), type -> type.typeId() == Type.TypeID.TIME)
!= null) {
Review Comment:
The fallback checks only the query projection, but batch execution reads
`deleteFilter.requiredSchema()`, which adds equality-delete fields. If a query
selects id while equality deletes use a TIME column, vectorization remains
enabled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
