Hi Giri, Can you share a simplified implementation code and the model you are trying to emit.
Thanks. - Maulik On Sat, Jun 4, 2016 at 3:04 PM, Giri P <[email protected]> wrote: > I need the other way either null or array of records. I was able to insert > not null records but when I try inserting nulls it throws me error > > I'm using spark to write avro files > > org.apache.spark.SparkException: Job aborted due to stage failure: Failed > to serialize task 16, not attempting to retry it. Exception during > serialization: java.io.NotSerializableException: > org.apache.avro.Schema$UnionSchema > Serialization stack: > - object not serializable (class: > org.apache.avro.Schema$UnionSchema, value: > ["null",{"type":"array","items":{"type":"record","name":"MatchedAttrTest","namespace":"com.conversantmedia.data.cp.avro","fields":[{"name":"id","type":"long","doc":"Attribute > id"},{"name":"updateDate","type":"long","doc":"Update date of the > attribute"},{"name":"value","type":"string","doc":"Value of the > attribute"}]}}]) > - writeObject data (class: java.lang.Throwable) > - object (class org.apache.avro.UnresolvedUnionException, > org.apache.avro.UnresolvedUnionException: Not in union > ["null",{"type":"array","items":{"type":"record","name":"MatchedAttrTest","namespace":"com.conversantmedia.data.cp.avro","fields":[{"name":"id","type":"long","doc":"Attribute > id"},{"name":"updateDate","type":"long","doc":"Update date of the > attribute"},{"name":"value","type":"string","doc":"Value of the > attribute"}]}}]: ) > - writeObject data (class: java.lang.Throwable) > - object (class java.io.IOException, java.io.IOException: > org.apache.avro.UnresolvedUnionException: Not in union > ["null",{"type":"array","items":{"type":"record","name":"MatchedAttrTest","namespace":"com.conversantmedia.data.cp.avro","fields":[{"name":"id","type":"long","doc":"Attribute > id"},{"name":"updateDate","type":"long","doc":"Update date of the > attribute"},{"name":"value","type":"string","doc":"Value of the > attribute"}]}}]: ) > - writeObject data (class: > org.apache.spark.rdd.ParallelCollectionPartition) > - object (class org.apache.spark.rdd.ParallelCollectionPartition, > org.apache.spark.rdd.ParallelCollectionPartition@7b0) > - element of array (index: 0) > - array (class [Ljava.lang.Object;, size 10) > - field (class: scala.collection.mutable.ArraySeq, name: array, > type: class [Ljava.lang.Object;) > - object (class scala.collection.mutable.ArraySeq, > ArraySeq(org.apache.spark.rdd.ParallelCollectionPartition@7b0, > org.apache.spark.rdd.ParallelCollectionPartition@7b1, > org.apache.spark.rdd.ParallelCollectionPartition@7b2, > org.apache.spark.rdd.ParallelCollectionPartition@7b3, > org.apache.spark.rdd.ParallelCollectionPartition@7b4, > org.apache.spark.rdd.ParallelCollectionPartition@7b5, > org.apache.spark.rdd.ParallelCollectionPartition@7b6, > org.apache.spark.rdd.ParallelCollectionPartition@7b7, > org.apache.spark.rdd.ParallelCollectionPartition@7b8, > org.apache.spark.rdd.ParallelCollectionPartition@7b9)) > - writeObject data (class: > org.apache.spark.rdd.CoalescedRDDPartition) > - object (class org.apache.spark.rdd.CoalescedRDDPartition, > CoalescedRDDPartition(0,MapPartitionsRDD[8] at map at > <console>:81,[I@19cc3c95,None)) > - field (class: org.apache.spark.scheduler.ResultTask, name: > partition, type: interface org.apache.spark.Partition) > - object (class org.apache.spark.scheduler.ResultTask, > ResultTask(4, 0)) > > > On Fri, Jun 3, 2016 at 10:53 PM, Doug Cutting <[email protected]> wrote: > >> Your schema permits null or an array of records. I suspect you want an >> array containing nulls or records, e.g., >> >> {"type":"array","items":["null",{"type":"record"... >> On Jun 3, 2016 5:54 PM, "Giri P" <[email protected]> wrote: >> >>> Hi, >>> >>> I'm getting below error when I try to insert null into union of array >>> >>> Caused by: org.apache.avro.UnresolvedUnionException: Not in union >>> ["null",{"type":"array","items":{"type":"record","name":"DeMatchedAttr","namespace":"cp","fields":[{"name":"id","type":"long","doc":"Attribute >>> id"},{"name":"updateDate","type":"long","doc":"Update date of the >>> attribute"},{"name":"value","type":"string","doc":"Value of the >>> attribute"}]}}]: null >>> >>> Is there any issue with the schema ? >>> >>> Thanks for the help >>> >>> -Giri >>> >> >
