viirya commented on code in PR #57088:
URL: https://github.com/apache/spark/pull/57088#discussion_r3541717167
##########
sql/api/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala:
##########
@@ -437,6 +478,23 @@ private[sql] object ArrowUtils {
}
}
+ /**
+ * Whether the Arrow struct field is the lossless representation of a
CalendarInterval built by
+ * `toArrowField` with `losslessInternalTypes = true`. Also callable from
Java
+ * (ArrowColumnVector) to select the interval accessor for such structs.
+ */
+ def isCalendarIntervalStructField(field: Field): Boolean = {
+ field.getType.isInstanceOf[ArrowType.Struct] &&
+ field.getChildren.asScala
+ .map(_.getName)
+ .asJava
+ .containsAll(Seq("months", "days", "microseconds").asJava) &&
+ field.getChildren.asScala.exists { child =>
+ child.getName == "months" &&
+ child.getMetadata.getOrDefault(calendarIntervalStructKey, "false") ==
"true"
Review Comment:
Hardened, and good catch on the positional-write/by-name-read asymmetry.
Both recognizers (`isCalendarIntervalStructField` and, since it had the
identical looseness, `isTimestampNanosStructField` from SPARK-57975) now accept
only the exact canonical shape: child count, order, names, integer widths, and
non-nullability, via a shared helper. Anything tagged but non-canonical falls
back to the generic struct handling, which is order-faithful. Added negative
tests for reordered, wrong-width, extra-child, and missing-child tagged schemas
for both types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
