uros-b commented on code in PR #56965:
URL: https://github.com/apache/spark/pull/56965#discussion_r3524912304
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/types/ops/ParquetTypeOps.scala:
##########
@@ -249,4 +252,25 @@ private[parquet] object ParquetTypeOps {
private[parquet] def getVectorUpdaterOrNull(
dt: DataType, descriptor: ColumnDescriptor): ParquetVectorUpdater =
apply(dt).flatMap(_.getVectorUpdater(descriptor)).orNull
+
+ /**
+ * Reverse lookup for filter pushdown: given a Parquet field's logical
annotation and
+ * primitive type (from the file schema), returns the framework filter ops
that owns that
+ * encoding, if any. Used by `ParquetFilters` (via its FrameworkFilterOps
extractor) so
+ * framework types participate in predicate pushdown with no per-type
changes there.
+ */
+ private[parquet] def filterOpsFor(
+ logicalTypeAnnotation: LogicalTypeAnnotation,
+ primitiveTypeName: PrimitiveTypeName): Option[ParquetFilterOps] =
+ filterOpsList.find { ops =>
+ ops.primitiveTypeName == primitiveTypeName &&
+ ops.logicalTypeAnnotation == logicalTypeAnnotation
+ }
Review Comment:
Nit: The reverse lookup matches on primitiveTypeName + logicalTypeAnnotation
only, dropping the length field the old case ParquetTimeMicrosType case-object
compared. Verified equivalent for this PR: getTypeLength() is 0 for any
non-FIXED_LEN_BYTE_ARRAY primitive, so the single INT64 entry exactly
reproduces the old length-0 match. Latent trap only if a future framework type
registers a FIXED_LEN_BYTE_ARRAY-backed ops; a one-line comment would suffice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
