srowen commented on a change in pull request #31809:
URL: https://github.com/apache/spark/pull/31809#discussion_r602793873
##########
File path:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroDataToCatalyst.scala
##########
@@ -91,11 +91,31 @@ private[avro] case class AvroDataToCatalyst(
null
}
+ /**
+ * This computes the offset for avro binary decoding based on the presence
of a magic number
+ * defined in avro BinaryMessageEncoder.java as of now we are assuming that
the binary
+ * provided was encoded via avro BinaryEncoder.java. However some
serializers use
Review comment:
I wonder how safe it is to assume that if the two bytes match the magic
number, they're the magic number? would a regular Avro file without it ever
start with these bytes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
