Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21952#discussion_r207443421
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala ---
@@ -151,11 +155,12 @@ class AvroSerializer(rootCatalystType: DataType,
rootAvroType: Schema, nullable:
case (f1, f2) => newConverter(f1.dataType,
resolveNullableType(f2.schema(), f1.nullable))
}
val numFields = catalystStruct.length
+ val containsNull = catalystStruct.exists(_.nullable)
--- End diff --
this only works when all the fields are not nullable, I don't think it's
very useful.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
