Github user dbtsai commented on a diff in the pull request:
https://github.com/apache/spark/pull/21847#discussion_r206356380
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala ---
@@ -146,9 +159,13 @@ class AvroSerializer(rootCatalystType: DataType,
rootAvroType: Schema, nullable:
private def newStructConverter(
catalystStruct: StructType, avroStruct: Schema): InternalRow =>
Record = {
val avroFields = avroStruct.getFields
- assert(avroFields.size() == catalystStruct.length)
+ if (avroFields.size != catalystStruct.length) {
+ throw new IncompatibleSchemaException(
+ s"Field list length of ${catalystStruct} does not correspond to
length of ${avroStruct}")
--- End diff --
`avroStruct` will print the schema in JSON which will be big and hard to
read. `catalystStruct.toString` is not well defined. How about we just print
`avroStruct.size` and `catalystStruct.length` with proper message?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
