Github user lindblombr commented on a diff in the pull request:
https://github.com/apache/spark/pull/21847#discussion_r205592450
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala ---
@@ -165,16 +183,100 @@ class AvroSerializer(rootCatalystType: DataType,
rootAvroType: Schema, nullable:
result
}
- private def resolveNullableType(avroType: Schema, nullable: Boolean):
Schema = {
- if (nullable) {
+ private def resolveNullableType(avroType: Schema, catalystType: DataType,
+ nullable: Boolean): Schema = {
+ if (nullable && avroType.getType == Type.UNION) {
--- End diff --
I've converted this into a match statement that covers the four cases:
* nullable == false and Type.UNION => should "resolve" the union to the
appropriate type
* nullable == true and Type.UNION => should "resolve" the union to the
appropriate type
* nullable == Any and Any (Type) => just return the Type
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
