+1 for trimming the schema down and iterating On Thu, Dec 19, 2019 at 10:07 PM Kabeer Ahmed <kab...@linuxmail.org> wrote:
> Hi Nishith, > > I do not want to diverge this thread. I looked into the jira link that you > have sent which is - ( > https://issues.apache.org/jira/projects/PARQUET/issues/PARQUET-1656?filter=allopenissues > ( > https://link.getmailspring.com/link/f14acfa3-1f88-4290-94f9-e4c3ed928...@getmailspring.com/0?redirect=https%3A%2F%2Fissues.apache.org%2Fjira%2Fprojects%2FPARQUET%2Fissues%2FPARQUET-1656%3Ffilter%3Dallopenissues&recipient=ZGV2QGh1ZGkuYXBhY2hlLm9yZw%3D%3D)). > In the example of this jira, I see that the schema has changed especially > wrt addition of a default value. This has never worked for me in the past > and the rules of avro schema evolution are as below (I cant put an > authoritative link but these are derived from experience): > Always provide a default value for the field to start with. > > Never change a field's data type. > > If a new few field is added in the future, always provide a default value. > (back to rule 1). > > Never rename an existing field but instead use aliases. > > In the example given in the jira, we see that there is a default value > being added later when it was missed in the first step. I have highlighted > one field in the ps section of the email below. This has not worked for me > or atleast I can say that this has given issues in the future. > > Coming back to this thread, I think the issue Ethan is having is not (or > should not) be directly related to the jira. Ethan seems to be using delta > streamer from a source but encountering issues in the 2nd load whilst the > 1st one goes without issues. So he is not necessarily changing the > definition of the schemas manually. > If you have other thoughts, please feel free. We can discuss in another > thread. > Ethan - please see if you can replicate the issue with a subset of the > schema. This involves effort but from past experience, I can always vouch > for this practice as it enables others to help you faster or atleast we > understand, file and fix a bug faster. > Thanks, > Kabeer. > > ____________________________ > First version of schema: > > { > "default": null, > "name": "master_cluster", > "type": [ > "null", > { > "fields": [ > { "name": "uuid", "type": "string" } > , > { "name": "namespace", "type": "string" } > , > > { "name": "version", "type": "long" } > ], > "name": "master_cluster", > "type": "record" > } > ] > }, > > ___________________________________ > Second version of the schema: > > { > "default": null, > "name": "master_cluster", > "type": [ > "null", > { > "fields": [ > { "default": null, "name": "uuid", "type": [ "null", "string" ] } > , > { "default": null, "name": "namespace", "type": [ "null", "string" ] } > , > > { "default": null, "name": "version", "type": [ "null", "long" ] } > ], > "name": "VORGmaster_cluster", > "type": "record" > } > ] > }, > > On Dec 19 2019, at 10:59 pm, Y Ethan Guo <ethan.guoyi...@gmail.com> wrote: > > Got it. Thanks for the clarification. > > > > On Thu, Dec 19, 2019 at 2:54 PM nishith agarwal <n3.nas...@gmail.com> > wrote: > > > Ethan, > > > There isn't one available in the open-source, it's an internal build we > > > have. > > > > > > -Nishith > > > On Thu, Dec 19, 2019 at 2:50 PM Y Ethan Guo <ethan.guoyi...@gmail.com> > > > wrote: > > > > > > > Thanks Kabeer and Nishith for the responses. > > > > The schema hasn't been changed. I'm now trying to reproduce the > problem > > > > locally with some synthetic data, given that it's expensive to > iterate in > > > > my cluster testing setup. > > > > > > > > @Nishith, thanks for the pointer. Is there an existing build of > > > > parquet-avro v1.8.1 with the fix? I don't see the fix attached to the > > > > ticket. I suppose that I also need to rebuild Hudi utilities bundle > to > > > > pick that up. > > > > > > > > On Thu, Dec 19, 2019 at 1:51 PM nishith agarwal <n3.nas...@gmail.com > > > > > > wrote: > > > > > > > > > Ethan, > > > > > Unless this is a backwards incompatible schema change, this seems > > > > > related to a parquet-avro reader bug we've seen before, find more > > > > > > > > > > details > > > > > here : > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/projects/PARQUET/issues/PARQUET-1656?filter=allopenissues > > > > > . > > > > > There's a fix for the parquet-avro reader for 1.8.1 which we > patched > > > > > > > > > > and > > > > > works for us in production. There isn't a proper fix available for > > > > > > > > > > later > > > > > parquet versions since the code has changed quite a bit. Which is > also > > > > > > > > the > > > > > reason why the patch for 1.8.1 is not being upstreamed since fewer > > > > > > > > > > people > > > > > are using it. > > > > > > > > > > -Nishith > > > > > On Thu, Dec 19, 2019 at 12:34 PM Kabeer Ahmed < > kab...@linuxmail.org> > > > > > wrote: > > > > > > > > > > > Hi Ethan, > > > > > > It is often tricky to debug or help with issues when I do not > have an > > > > > idea > > > > > > of the data. My "guess" is that your schema is changing. This > could > > > > > > > > > > > > > > > be > > > > > > related to: https://stackoverflow.com/a/42946528/4517001 ( > > > > > > > > > > > > > > > > > > > https://link.getmailspring.com/link/966cdc69-f6ce-4745-88bd-0e5553efa...@getmailspring.com/0?redirect=https%3A%2F%2Fstackoverflow.com%2Fa%2F42946528%2F4517001&recipient=ZGV2QGh1ZGkuYXBhY2hlLm9yZw%3D%3D > > > > > > ). > > > > > > It may help if you could send sample data of the first pass that > went > > > > > > without issues and the second set of data that caused an issue. > > > > > > > > > > > > > > > Please > > > > > > check that you are allowed to share the data. > > > > > > Thanks > > > > > > Kabeer. > > > > > > > > > > > > On Dec 19 2019, at 7:33 pm, Y Ethan Guo < > ethan.guoyi...@gmail.com> > > > > > wrote: > > > > > > > Hi folks, > > > > > > > > > > > > > > I'm testing a new Deltastreamer job in cluster which > incrementally > > > > > pulls > > > > > > > data from an upstream Hudi table and upserts the dataset into > > > > > > > > > > > > > > > > > > > > > another > > > > > > > table. The first run of Deltastreamer job which involves only > > > > > > > > > > > > > > > > > > > > > inserts > > > > > > > succeeded. The second run of the job which involves updates > throws > > > > > > > > > > > > > > > > > > > the > > > > > > > following exception. I'm using a snapshot build of Hudi: > > > > > > > 0.4.8-SNAPSHOT[1]. I believe this is related to schema, but I > don't > > > > > > > > > > > > > > > > know > > > > > > > how I should debug and fix this. Any suggestion is appreciated. > > > > > > > > > > > > > > org.apache.spark.SparkException: Job aborted due to stage > failure: > > > > > > > Task 33 in stage 78.0 failed 4 times, most recent failure: Lost > > > > > > > > > > > > > > > > > > > > > task > > > > > > > 33.3 in stage 78.0 (TID 13283, executor 5): > > > > > > > com.uber.hoodie.exception.HoodieUpsertException: Error > upserting > > > > > > > bucketType UPDATE for partition :33 > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpsertPartition(HoodieCopyOnWriteTable.java:271) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.HoodieWriteClient.lambda$upsertRecordsInternal$7ef77fd$1(HoodieWriteClient.java:442) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:844) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:844) > > > > > > > at > > > > > > > > > > > > > > > > > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > > > > > > > at > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) > > > > > > > at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) > > > > > > > at > > > > > > > > > > > > > > > > > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > > > > > > > at > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) > > > > > > > at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:336) > > > > > > > at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:334) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1055) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1029) > > > > > > > at > > > > > > > > > > > > > > > > > > > org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:969) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1029) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:760) > > > > > > > at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334) > > > > > > > at org.apache.spark.rdd.RDD.iterator(RDD.scala:285) > > > > > > > at > > > > > > > > > > > > > > > > > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > > > > > > > at > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) > > > > > > > at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) > > > > > > > at org.apache.spark.scheduler.Task.run(Task.scala:108) > > > > > > > at > > > > > > > > > > > > > > > > > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > > > > at java.lang.Thread.run(Thread.java:748) > > > > > > > Caused by: com.uber.hoodie.exception.HoodieException: > > > > > > > com.uber.hoodie.exception.HoodieException: > > > > > > > java.util.concurrent.ExecutionException: > > > > > > > com.uber.hoodie.exception.HoodieException: operation has failed > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpdateInternal(HoodieCopyOnWriteTable.java:206) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpdate(HoodieCopyOnWriteTable.java:181) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpsertPartition(HoodieCopyOnWriteTable.java:263) > > > > > > > ... 28 more > > > > > > > Caused by: com.uber.hoodie.exception.HoodieException: > > > > > > > java.util.concurrent.ExecutionException: > > > > > > > com.uber.hoodie.exception.HoodieException: operation has failed > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:146) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpdateInternal(HoodieCopyOnWriteTable.java:204) > > > > > > > ... 30 more > > > > > > > Caused by: java.util.concurrent.ExecutionException: > > > > > > > com.uber.hoodie.exception.HoodieException: operation has failed > > > > > > > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > > > > > > > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:144) > > > > > > > ... 31 more > > > > > > > Caused by: com.uber.hoodie.exception.HoodieException: > operation has > > > > > > > > > > > > failed > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.common.util.queue.BoundedInMemoryQueue.throwExceptionIfFailed(BoundedInMemoryQueue.java:232) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.common.util.queue.BoundedInMemoryQueue.readNextRecord(BoundedInMemoryQueue.java:211) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.common.util.queue.BoundedInMemoryQueue.access$100(BoundedInMemoryQueue.java:49) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.common.util.queue.BoundedInMemoryQueue$QueueIterator.hasNext(BoundedInMemoryQueue.java:262) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:37) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:124) > > > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > > > > > ... 3 more > > > > > > > Caused by: java.lang.ClassCastException: required binary key > (UTF8) > > > > > > > > > > > > > > > > > > > is > > > > > > > not a group > > > > > > > at org.apache.parquet.schema.Type.asGroupType(Type.java:202) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroRecordConverter$MapConverter.<init>(AvroRecordConverter.java:821) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:210) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroRecordConverter.access$200(AvroRecordConverter.java:63) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.<init>(AvroRecordConverter.java:435) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.<init>(AvroRecordConverter.java:385) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:206) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroRecordConverter.access$200(AvroRecordConverter.java:63) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroRecordConverter$MapConverter$MapKeyValueConverter.<init>(AvroRecordConverter.java:872) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroRecordConverter$MapConverter.<init>(AvroRecordConverter.java:822) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:210) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroRecordConverter.<init>(AvroRecordConverter.java:112) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroRecordConverter.<init>(AvroRecordConverter.java:79) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroRecordMaterializer.<init>(AvroRecordMaterializer.java:33) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:132) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:175) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:149) > > > > > > > at > > > > > > > > > > > > > > > > > > > org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:125) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.func.ParquetReaderIterator.hasNext(ParquetReaderIterator.java:45) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.common.util.queue.IteratorBasedQueueProducer.produce(IteratorBasedQueueProducer.java:44) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.lambda$null$0(BoundedInMemoryExecutor.java:94) > > > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > > > > > at > > > > > > > > > > > > > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > > > > > > ... 4 more > > > > > > > > > > > > > > Driver stacktrace: > > > > > > > at org.apache.spark.scheduler.DAGScheduler.org > > > > > > > > > > > > > > > > > > > > > > > > > $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1517) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1505) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1504) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1504) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:814) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:814) > > > > > > > at scala.Option.foreach(Option.scala:257) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:814) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1732) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1687) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1676) > > > > > > > at > org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) > > > > > > > at > > > > > > > > > > > > > > > > > org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:630) > > > > > > > at > org.apache.spark.SparkContext.runJob(SparkContext.scala:2029) > > > > > > > at > org.apache.spark.SparkContext.runJob(SparkContext.scala:2126) > > > > > > > at > org.apache.spark.rdd.RDD$$anonfun$fold$1.apply(RDD.scala:1089) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) > > > > > > > at org.apache.spark.rdd.RDD.withScope(RDD.scala:362) > > > > > > > at org.apache.spark.rdd.RDD.fold(RDD.scala:1083) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.rdd.DoubleRDDFunctions$$anonfun$sum$1.apply$mcD$sp(DoubleRDDFunctions.scala:35) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.rdd.DoubleRDDFunctions$$anonfun$sum$1.apply(DoubleRDDFunctions.scala:35) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.rdd.DoubleRDDFunctions$$anonfun$sum$1.apply(DoubleRDDFunctions.scala:35) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) > > > > > > > at org.apache.spark.rdd.RDD.withScope(RDD.scala:362) > > > > > > > at > > > > > > > > > > > > > > > > > org.apache.spark.rdd.DoubleRDDFunctions.sum(DoubleRDDFunctions.scala:34) > > > > > > > at > > > > > > > > > > > > > > > > > > > org.apache.spark.api.java.JavaDoubleRDD.sum(JavaDoubleRDD.scala:165) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:279) > > > > > > > ... 47 elided > > > > > > > Caused by: com.uber.hoodie.exception.HoodieUpsertException: > Error > > > > > > > upserting bucketType UPDATE for partition :33 > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpsertPartition(HoodieCopyOnWriteTable.java:271) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.HoodieWriteClient.lambda$upsertRecordsInternal$7ef77fd$1(HoodieWriteClient.java:442) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:844) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:844) > > > > > > > at > > > > > > > > > > > > > > > > > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > > > > > > > at > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) > > > > > > > at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) > > > > > > > at > > > > > > > > > > > > > > > > > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > > > > > > > at > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) > > > > > > > at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:336) > > > > > > > at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:334) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1055) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1029) > > > > > > > at > > > > > > > > > > > > > > > > > > > org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:969) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1029) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:760) > > > > > > > at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334) > > > > > > > at org.apache.spark.rdd.RDD.iterator(RDD.scala:285) > > > > > > > at > > > > > > > > > > > > > > > > > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > > > > > > > at > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) > > > > > > > at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) > > > > > > > at org.apache.spark.scheduler.Task.run(Task.scala:108) > > > > > > > at > > > > > > > > > > > > > > > > > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338) > > > > > > > ... 3 more > > > > > > > Caused by: com.uber.hoodie.exception.HoodieException: > > > > > > > com.uber.hoodie.exception.HoodieException: > > > > > > > java.util.concurrent.ExecutionException: > > > > > > > com.uber.hoodie.exception.HoodieException: operation has failed > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpdateInternal(HoodieCopyOnWriteTable.java:206) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpdate(HoodieCopyOnWriteTable.java:181) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpsertPartition(HoodieCopyOnWriteTable.java:263) > > > > > > > ... 28 more > > > > > > > Caused by: com.uber.hoodie.exception.HoodieException: > > > > > > > java.util.concurrent.ExecutionException: > > > > > > > com.uber.hoodie.exception.HoodieException: operation has failed > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:146) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpdateInternal(HoodieCopyOnWriteTable.java:204) > > > > > > > ... 30 more > > > > > > > Caused by: java.util.concurrent.ExecutionException: > > > > > > > com.uber.hoodie.exception.HoodieException: operation has failed > > > > > > > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > > > > > > > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:144) > > > > > > > ... 31 more > > > > > > > Caused by: com.uber.hoodie.exception.HoodieException: > operation has > > > > > > > > > > > > failed > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.common.util.queue.BoundedInMemoryQueue.throwExceptionIfFailed(BoundedInMemoryQueue.java:232) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.common.util.queue.BoundedInMemoryQueue.readNextRecord(BoundedInMemoryQueue.java:211) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.common.util.queue.BoundedInMemoryQueue.access$100(BoundedInMemoryQueue.java:49) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.common.util.queue.BoundedInMemoryQueue$QueueIterator.hasNext(BoundedInMemoryQueue.java:262) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:37) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:124) > > > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > > > > > ... 3 more > > > > > > > Caused by: java.lang.ClassCastException: required binary key > (UTF8) > > > > > > > > > > > > > > > > > > > is > > > > > > > not a group > > > > > > > at org.apache.parquet.schema.Type.asGroupType(Type.java:202) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroRecordConverter$MapConverter.<init>(AvroRecordConverter.java:821) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:210) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroRecordConverter.access$200(AvroRecordConverter.java:63) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.<init>(AvroRecordConverter.java:435) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.<init>(AvroRecordConverter.java:385) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:206) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroRecordConverter.access$200(AvroRecordConverter.java:63) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroRecordConverter$MapConverter$MapKeyValueConverter.<init>(AvroRecordConverter.java:872) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroRecordConverter$MapConverter.<init>(AvroRecordConverter.java:822) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:210) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroRecordConverter.<init>(AvroRecordConverter.java:112) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroRecordConverter.<init>(AvroRecordConverter.java:79) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroRecordMaterializer.<init>(AvroRecordMaterializer.java:33) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:132) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:175) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:149) > > > > > > > at > > > > > > > > > > > > > > > > > > > org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:125) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.func.ParquetReaderIterator.hasNext(ParquetReaderIterator.java:45) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.common.util.queue.IteratorBasedQueueProducer.produce(IteratorBasedQueueProducer.java:44) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.lambda$null$0(BoundedInMemoryExecutor.java:94) > > > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > > > > > at > > > > > > > > > > > > > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > > > > > > ... 4 more > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > https://oss.sonatype.org/content/repositories/snapshots/com/uber/hoodie/hoodie-utilities-bundle/0.4.8-SNAPSHOT/hoodie-utilities-bundle-0.4.8-20190531.060546-1.jar > > > > > > > > > > > > > > Thanks, > > > > > > > - Ethan > > > > > > > > > > > > > > > > > > > > > > > >
