[ https://issues.apache.org/jira/browse/SPARK-31074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17054697#comment-17054697 ]
L. C. Hsieh commented on SPARK-31074: ------------------------------------- Based on the description, is this the same issue as you created at SPARK-31071? > Avro serializer should not fail when a nullable Spark field is written to a > non-null Avro column > ------------------------------------------------------------------------------------------------ > > Key: SPARK-31074 > URL: https://issues.apache.org/jira/browse/SPARK-31074 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 2.4.4 > Reporter: Kyrill Alyoshin > Priority: Major > > Spark StructType schema are strongly biased towards having _nullable_ fields. > In fact, this is what _Encoders.bean()_ does - any non-primitive field is > automatically _nullable_. When we attempt to serialize dataframes into > *user-supplied* Avro schemas where such corresponding fields are marked as > _non-null_ (i.e., they are not of _union_ type) any such attempt will fail > with the following exception > > {code:java} > Caused by: org.apache.avro.AvroRuntimeException: Not a union: "string" > at org.apache.avro.Schema.getTypes(Schema.java:299) > at > org.apache.spark.sql.avro.AvroSerializer.org$apache$spark$sql$avro$AvroSerializer$$resolveNullableType(AvroSerializer.scala:229) > at > org.apache.spark.sql.avro.AvroSerializer$$anonfun$3.apply(AvroSerializer.scala:209) > {code} > This seems as rather draconian. We certainly should be able to write a field > of the same type and with the same name if it is not a null into a > non-nullable Avro column. In fact, the problem is so *severe* that it is not > clear what should be done in such situations when Avro schema is given to you > as part of API communication contract (i.e., it is non-changeable). > This is an important issue. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org