xkrogen commented on a change in pull request #31201:
URL: https://github.com/apache/spark/pull/31201#discussion_r562087894
##########
File path:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -201,4 +201,25 @@ private[sql] object AvroUtils extends Logging {
}
}
}
+
+ /**
+ * Extract a single field from `avroFields` which has the same name has
`catalystField`,
+ * performing the matching with proper case sensitivity according to
[[SQLConf.CASE_SENSITIVE]].
+ *
+ * @param avroFields The fields within which to search for a match.
+ * @param catalystField The catalyst field for which to search for a match.
+ * @return `Some(match)` if a matching Avro field is found, otherwise `None`.
+ */
+ private[avro] def getAvroFieldForCatalystField(
Review comment:
Completely right, good call. Also thanks for the pointer on
`SQLConf.resolver`, that is much nicer!
##########
File path:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -201,4 +201,25 @@ private[sql] object AvroUtils extends Logging {
}
}
}
+
+ /**
+ * Extract a single field from `avroFields` which has the same name has
`catalystField`,
+ * performing the matching with proper case sensitivity according to
[[SQLConf.CASE_SENSITIVE]].
+ *
+ * @param avroFields The fields within which to search for a match.
+ * @param catalystField The catalyst field for which to search for a match.
+ * @return `Some(match)` if a matching Avro field is found, otherwise `None`.
+ */
+ private[avro] def getAvroFieldForCatalystField(
+ avroFields: TraversableOnce[Schema.Field],
+ catalystField: StructField): Option[Schema.Field] = {
+ val isEqual: (String, String) => Boolean =
+ if (SQLConf.get.caseSensitiveAnalysis) { _.equals(_) } else {
_.equalsIgnoreCase(_) }
+ avroFields.filter(f => isEqual(f.name(), catalystField.name)).toSeq match {
+ case Seq(avroField) => Some(avroField)
+ case Seq() => None
+ case s => throw new IncompatibleSchemaException(
+ s"Searching for ${catalystField.name} in Avro schema gave ${s.size}
matches")
Review comment:
Good idea on printing Avro field names! I added a test case as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
