Hi Umesh,
I suspect you are including both hoodie-common and hoodie-spark-bundle jars in
your runtime package dependencies. There will be a version of
HoodieAvroWriteSupport constructor with a proper shaded signature in Hoodie
Spark bundle but this may not be picked if hoodie-common jar is also included.
It should be sufficient to include only bundle packages (e:g
hoodie-spark-bundle) at runtime. If you need to use other hoodie packages (for
your implementation), you can mark them with "provided dependency scope"
Let us know if this is the case ? If not, please provide more context about the
dependencies getting added (pom.xml/spark logs showing packages getting added).
Thanks,Balaji.V
On Saturday, March 23, 2019, 1:27:52 AM PDT, Umesh Kacha
<[email protected]> wrote:
Hi I filtered out nulls in dataframe for review_date field and it went
ahead but failed with the following exception. It looks like some run time
libs are missing I thought com.uber.hoodie:hoodie-spark-bundle:0.4.5 is
uber jar it has all the transitive dependencies it need. No?
org.apache.spark.SparkException: Job aborted due to stage failure: Task 7
in stage 22.0 failed 1 times, most recent failure: Lost task 7.0 in stage
22.0 (TID 3216, localhost, executor driver): java.lang.RuntimeException:
com.uber.hoodie.exception.HoodieException:
com.uber.hoodie.exception.HoodieException:
java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError:
com.uber.hoodie.avro.HoodieAvroWriteSupport.<init>(Lcom/uber/hoodie/org/apache/parquet/schema/MessageType;Lorg/apache/avro/Schema;Lcom/uber/hoodie/common/BloomFilter;)V
at
com.uber.hoodie.func.LazyIterableIterator.next(LazyIterableIterator.java:121)
at
scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) at
scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) at
org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
at
org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:349)
at
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1187)
at
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1161)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1096) at
org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1161)
at
org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:883)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:351) at
org.apache.spark.rdd.RDD.iterator(RDD.scala:302) at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:60) at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:340) at
org.apache.spark.rdd.RDD.iterator(RDD.scala:304) at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at
org.apache.spark.scheduler.Task.doRunTask(Task.scala:139) at
org.apache.spark.scheduler.Task.run(Task.scala:112) at
org.apache.spark.executor.Executor$TaskRunner$$anonfun$13.apply(Executor.scala:497)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1432)
Caused by: com.uber.hoodie.exception.HoodieException:
com.uber.hoodie.exception.HoodieException:
java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError:
com.uber.hoodie.avro.HoodieAvroWriteSupport.<init>(Lcom/uber/hoodie/org/apache/parquet/schema/MessageType;Lorg/apache/avro/Schema;Lcom/uber/hoodie/common/BloomFilter;)V
at
com.uber.hoodie.func.CopyOnWriteLazyInsertIterable.computeNext(CopyOnWriteLazyInsertIterable.java:106)
at
com.uber.hoodie.func.CopyOnWriteLazyInsertIterable.computeNext(CopyOnWriteLazyInsertIterable.java:45)
at
com.uber.hoodie.func.LazyIterableIterator.next(LazyIterableIterator.java:119)
... 24 more Caused by: com.uber.hoodie.exception.HoodieException:
java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError:
com.uber.hoodie.avro.HoodieAvroWriteSupport.<init>(Lcom/uber/hoodie/org/apache/parquet/schema/MessageType;Lorg/apache/avro/Schema;Lcom/uber/hoodie/common/BloomFilter;)V
at
com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:146)
at
com.uber.hoodie.func.CopyOnWriteLazyInsertIterable.computeNext(CopyOnWriteLazyInsertIterable.java:102)
... 26 more Caused by: java.util.concurrent.ExecutionException:
java.lang.NoSuchMethodError:
com.uber.hoodie.avro.HoodieAvroWriteSupport.<init>(Lcom/uber/hoodie/org/apache/parquet/schema/MessageType;Lorg/apache/avro/Schema;Lcom/uber/hoodie/common/BloomFilter;)V
at java.util.concurrent.FutureTask.report(FutureTask.java:122) at
java.util.concurrent.FutureTask.get(FutureTask.java:192) at
com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:144)
... 27 more Caused by: java.lang.NoSuchMethodError:
com.uber.hoodie.avro.HoodieAvroWriteSupport.<init>(Lcom/uber/hoodie/org/apache/parquet/schema/MessageType;Lorg/apache/avro/Schema;Lcom/uber/hoodie/common/BloomFilter;)V
at
com.uber.hoodie.io.storage.HoodieStorageWriterFactory.newParquetStorageWriter(HoodieStorageWriterFactory.java:47)
at
com.uber.hoodie.io.storage.HoodieStorageWriterFactory.getStorageWriter(HoodieStorageWriterFactory.java:38)
at com.uber.hoodie.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:71)
at
com.uber.hoodie.func.CopyOnWriteLazyInsertIterable$CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteLazyInsertIterable.java:149)
at
com.uber.hoodie.func.CopyOnWriteLazyInsertIterable$CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteLazyInsertIterable.java:127)
at
com.uber.hoodie.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:38)
at
com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:124)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 more
Driver stacktrace: at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:2100)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:2088)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:2087)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2087)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:1076)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:1076)
at scala.Option.foreach(Option.scala:257) at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1076)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2319)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2267)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2255)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49) at
org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:873) at
org.apache.spark.SparkContext.runJob(SparkContext.scala:2252) at
org.apache.spark.SparkContext.runJob(SparkContext.scala:2274) at
org.apache.spark.SparkContext.runJob(SparkContext.scala:2293) at
org.apache.spark.SparkContext.runJob(SparkContext.scala:2318) at
org.apache.spark.rdd.RDD.count(RDD.scala:1184) at
com.uber.hoodie.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:151)
at com.uber.hoodie.DefaultSource.createRelation(DefaultSource.scala:91) at
org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:72)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:70)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:88)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:143)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$5.apply(SparkPlan.scala:183)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at
org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:180)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:131) at
org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:114)
at
org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:114)
at
org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:690)
at
org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:690)
at
org.apache.spark.sql.execution.SQLExecution$$anonfun$withCustomExecutionEnv$1.apply(SQLExecution.scala:99)
at
org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:228)
at
org.apache.spark.sql.execution.SQLExecution$.withCustomExecutionEnv(SQLExecution.scala:85)
at
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:158)
at
org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:690)
at
org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:290)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:284) at
org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:232) at
line2873d1c3d6574f6ca1d33c50e3f9f73548.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(
command-1085008513297384:10) at
line2873d1c3d6574f6ca1d33c50e3f9f73548.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(
command-1085008513297384:61) at
line2873d1c3d6574f6ca1d33c50e3f9f73548.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(
command-1085008513297384:63) at
line2873d1c3d6574f6ca1d33c50e3f9f73548.$read$$iw$$iw$$iw$$iw$$iw.<init>(
command-1085008513297384:65) at
line2873d1c3d6574f6ca1d33c50e3f9f73548.$read$$iw$$iw$$iw$$iw.<init>(
command-1085008513297384:67) at
line2873d1c3d6574f6ca1d33c50e3f9f73548.$read$$iw$$iw$$iw.<init>(
command-1085008513297384:69) at
line2873d1c3d6574f6ca1d33c50e3f9f73548.$read$$iw$$iw.<init>(
command-1085008513297384:71) at
line2873d1c3d6574f6ca1d33c50e3f9f73548.$read$$iw.<init>(
command-1085008513297384:73) at
line2873d1c3d6574f6ca1d33c50e3f9f73548.$read.<init>(
command-1085008513297384:75) at
line2873d1c3d6574f6ca1d33c50e3f9f73548.$read$.<init>(
command-1085008513297384:79) at
line2873d1c3d6574f6ca1d33c50e3f9f73548.$read$.<clinit>(command-1085008513297384)
at
line2873d1c3d6574f6ca1d33c50e3f9f73548.$eval$.$print$lzycompute(<notebook>:7)
at line2873d1c3d6574f6ca1d33c50e3f9f73548.$eval$.$print(<notebook>:6) at
line2873d1c3d6574f6ca1d33c50e3f9f73548.$eval.$print(<notebook>) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786) at
scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047) at
scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:638)
at
scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637)
at
scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
at
scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
at
scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:637)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569) at
scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565) at
com.databricks.backend.daemon.driver.DriverILoop.execute(DriverILoop.scala:199)
at
com.databricks.backend.daemon.driver.ScalaDriverLocal$$anonfun$repl$1.apply$mcV$sp(ScalaDriverLocal.scala:189)
at
com.databricks.backend.daemon.driver.ScalaDriverLocal$$anonfun$repl$1.apply(ScalaDriverLocal.scala:189)
at
com.databricks.backend.daemon.driver.ScalaDriverLocal$$anonfun$repl$1.apply(ScalaDriverLocal.scala:189)
at
com.databricks.backend.daemon.driver.DriverLocal$TrapExitInternal$.trapExit(DriverLocal.scala:587)
at
com.databricks.backend.daemon.driver.DriverLocal$TrapExit$.apply(DriverLocal.scala:542)
at
com.databricks.backend.daemon.driver.ScalaDriverLocal.repl(ScalaDriverLocal.scala:189)
at
com.databricks.backend.daemon.driver.DriverLocal$$anonfun$execute$7.apply(DriverLocal.scala:324)
at
com.databricks.backend.daemon.driver.DriverLocal$$anonfun$execute$7.apply(DriverLocal.scala:304)
at
com.databricks.logging.UsageLogging$$anonfun$withAttributionContext$1.apply(UsageLogging.scala:235)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) at
com.databricks.logging.UsageLogging$class.withAttributionContext(UsageLogging.scala:230)
at
com.databricks.backend.daemon.driver.DriverLocal.withAttributionContext(DriverLocal.scala:45)
at
com.databricks.logging.UsageLogging$class.withAttributionTags(UsageLogging.scala:268)
at
com.databricks.backend.daemon.driver.DriverLocal.withAttributionTags(DriverLocal.scala:45)
at
com.databricks.backend.daemon.driver.DriverLocal.execute(DriverLocal.scala:304)
at
com.databricks.backend.daemon.driver.DriverWrapper$$anonfun$tryExecutingCommand$2.apply(DriverWrapper.scala:589)
at
com.databricks.backend.daemon.driver.DriverWrapper$$anonfun$tryExecutingCommand$2.apply(DriverWrapper.scala:589)
at scala.util.Try$.apply(Try.scala:192) at
com.databricks.backend.daemon.driver.DriverWrapper.tryExecutingCommand(DriverWrapper.scala:584)
at
com.databricks.backend.daemon.driver.DriverWrapper.getCommandOutputAndError(DriverWrapper.scala:475)
at
com.databricks.backend.daemon.driver.DriverWrapper.executeCommand(DriverWrapper.scala:542)
at
com.databricks.backend.daemon.driver.DriverWrapper.runInnerLoop(DriverWrapper.scala:381)
at
com.databricks.backend.daemon.driver.DriverWrapper.runInner(DriverWrapper.scala:328)
at
com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:215)
at java.lang.Thread.run(Thread.java:748) Caused by:
java.lang.RuntimeException: com.uber.hoodie.exception.HoodieException:
com.uber.hoodie.exception.HoodieException:
java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError:
com.uber.hoodie.avro.HoodieAvroWriteSupport.<init>(Lcom/uber/hoodie/org/apache/parquet/schema/MessageType;Lorg/apache/avro/Schema;Lcom/uber/hoodie/common/BloomFilter;)V
at
com.uber.hoodie.func.LazyIterableIterator.next(LazyIterableIterator.java:121)
at
scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) at
scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) at
org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
at
org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:349)
at
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1187)
at
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1161)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1096) at
org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1161)
at
org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:883)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:351) at
org.apache.spark.rdd.RDD.iterator(RDD.scala:302) at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:60) at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:340) at
org.apache.spark.rdd.RDD.iterator(RDD.scala:304) at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at
org.apache.spark.scheduler.Task.doRunTask(Task.scala:139) at
org.apache.spark.scheduler.Task.run(Task.scala:112) at
org.apache.spark.executor.Executor$TaskRunner$$anonfun$13.apply(Executor.scala:497)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1432) at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:503) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) Caused by:
com.uber.hoodie.exception.HoodieException:
com.uber.hoodie.exception.HoodieException:
java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError:
com.uber.hoodie.avro.HoodieAvroWriteSupport.<init>(Lcom/uber/hoodie/org/apache/parquet/schema/MessageType;Lorg/apache/avro/Schema;Lcom/uber/hoodie/common/BloomFilter;)V
at
com.uber.hoodie.func.CopyOnWriteLazyInsertIterable.computeNext(CopyOnWriteLazyInsertIterable.java:106)
at
com.uber.hoodie.func.CopyOnWriteLazyInsertIterable.computeNext(CopyOnWriteLazyInsertIterable.java:45)
at
com.uber.hoodie.func.LazyIterableIterator.next(LazyIterableIterator.java:119)
at
scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) at
scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) at
org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
at
org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:349)
at
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1187)
at
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1161)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1096) at
org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1161)
at
org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:883)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:351) at
org.apache.spark.rdd.RDD.iterator(RDD.scala:302) at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:60) at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:340) at
org.apache.spark.rdd.RDD.iterator(RDD.scala:304) at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at
org.apache.spark.scheduler.Task.doRunTask(Task.scala:139) at
org.apache.spark.scheduler.Task.run(Task.scala:112) at
org.apache.spark.executor.Executor$TaskRunner$$anonfun$13.apply(Executor.scala:497)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1432) at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:503) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) Caused by:
com.uber.hoodie.exception.HoodieException:
java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError:
com.uber.hoodie.avro.HoodieAvroWriteSupport.<init>(Lcom/uber/hoodie/org/apache/parquet/schema/MessageType;Lorg/apache/avro/Schema;Lcom/uber/hoodie/common/BloomFilter;)V
at
com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:146)
at
com.uber.hoodie.func.CopyOnWriteLazyInsertIterable.computeNext(CopyOnWriteLazyInsertIterable.java:102)
at
com.uber.hoodie.func.CopyOnWriteLazyInsertIterable.computeNext(CopyOnWriteLazyInsertIterable.java:45)
at
com.uber.hoodie.func.LazyIterableIterator.next(LazyIterableIterator.java:119)
at
scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) at
scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) at
org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
at
org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:349)
at
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1187)
at
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1161)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1096) at
org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1161)
at
org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:883)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:351) at
org.apache.spark.rdd.RDD.iterator(RDD.scala:302) at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:60) at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:340) at
org.apache.spark.rdd.RDD.iterator(RDD.scala:304) at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at
org.apache.spark.scheduler.Task.doRunTask(Task.scala:139) at
org.apache.spark.scheduler.Task.run(Task.scala:112) at
org.apache.spark.executor.Executor$TaskRunner$$anonfun$13.apply(Executor.scala:497)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1432) at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:503) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) Caused by:
java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError:
com.uber.hoodie.avro.HoodieAvroWriteSupport.<init>(Lcom/uber/hoodie/org/apache/parquet/schema/MessageType;Lorg/apache/avro/Schema;Lcom/uber/hoodie/common/BloomFilter;)V
at java.util.concurrent.FutureTask.report(FutureTask.java:122) at
java.util.concurrent.FutureTask.get(FutureTask.java:192) at
com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:144)
at
com.uber.hoodie.func.CopyOnWriteLazyInsertIterable.computeNext(CopyOnWriteLazyInsertIterable.java:102)
at
com.uber.hoodie.func.CopyOnWriteLazyInsertIterable.computeNext(CopyOnWriteLazyInsertIterable.java:45)
at
com.uber.hoodie.func.LazyIterableIterator.next(LazyIterableIterator.java:119)
at
scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) at
scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) at
org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
at
org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:349)
at
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1187)
at
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1161)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1096) at
org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1161)
at
org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:883)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:351) at
org.apache.spark.rdd.RDD.iterator(RDD.scala:302) at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:60) at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:340) at
org.apache.spark.rdd.RDD.iterator(RDD.scala:304) at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at
org.apache.spark.scheduler.Task.doRunTask(Task.scala:139) at
org.apache.spark.scheduler.Task.run(Task.scala:112) at
org.apache.spark.executor.Executor$TaskRunner$$anonfun$13.apply(Executor.scala:497)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1432) at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:503) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) Caused by:
java.lang.NoSuchMethodError:
com.uber.hoodie.avro.HoodieAvroWriteSupport.<init>(Lcom/uber/hoodie/org/apache/parquet/schema/MessageType;Lorg/apache/avro/Schema;Lcom/uber/hoodie/common/BloomFilter;)V
at
com.uber.hoodie.io.storage.HoodieStorageWriterFactory.newParquetStorageWriter(HoodieStorageWriterFactory.java:47)
at
com.uber.hoodie.io.storage.HoodieStorageWriterFactory.getStorageWriter(HoodieStorageWriterFactory.java:38)
at com.uber.hoodie.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:71)
at
com.uber.hoodie.func.CopyOnWriteLazyInsertIterable$CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteLazyInsertIterable.java:149)
at
com.uber.hoodie.func.CopyOnWriteLazyInsertIterable$CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteLazyInsertIterable.java:127)
at
com.uber.hoodie.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:38)
at
com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:124)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.uber.hoodie.exception.HoodieException:
com.uber.hoodie.exception.HoodieException:
java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError:
com.uber.hoodie.avro.HoodieAvroWriteSupport.<init>(Lcom/uber/hoodie/org/apache/parquet/schema/MessageType;Lorg/apache/avro/Schema;Lcom/uber/hoodie/common/BloomFilter;)V
at
com.uber.hoodie.func.CopyOnWriteLazyInsertIterable.computeNext(CopyOnWriteLazyInsertIterable.java:106)
at
com.uber.hoodie.func.CopyOnWriteLazyInsertIterable.computeNext(CopyOnWriteLazyInsertIterable.java:45)
at
com.uber.hoodie.func.LazyIterableIterator.next(LazyIterableIterator.java:119)
... 24 more Caused by: com.uber.hoodie.exception.HoodieException:
java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError:
com.uber.hoodie.avro.HoodieAvroWriteSupport.<init>(Lcom/uber/hoodie/org/apache/parquet/schema/MessageType;Lorg/apache/avro/Schema;Lcom/uber/hoodie/common/BloomFilter;)V
at
com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:146)
at
com.uber.hoodie.func.CopyOnWriteLazyInsertIterable.computeNext(CopyOnWriteLazyInsertIterable.java:102)
... 26 more Caused by: java.util.concurrent.ExecutionException:
java.lang.NoSuchMethodError:
com.uber.hoodie.avro.HoodieAvroWriteSupport.<init>(Lcom/uber/hoodie/org/apache/parquet/schema/MessageType;Lorg/apache/avro/Schema;Lcom/uber/hoodie/common/BloomFilter;)V
at java.util.concurrent.FutureTask.report(FutureTask.java:122) at
java.util.concurrent.FutureTask.get(FutureTask.java:192) at
com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:144)
... 27 more Caused by: java.lang.NoSuchMethodError:
com.uber.hoodie.avro.HoodieAvroWriteSupport.<init>(Lcom/uber/hoodie/org/apache/parquet/schema/MessageType;Lorg/apache/avro/Schema;Lcom/uber/hoodie/common/BloomFilter;)V
at
com.uber.hoodie.io.storage.HoodieStorageWriterFactory.newParquetStorageWriter(HoodieStorageWriterFactory.java:47)
at
com.uber.hoodie.io.storage.HoodieStorageWriterFactory.getStorageWriter(HoodieStorageWriterFactory.java:38)
at com.uber.hoodie.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:71)
at
com.uber.hoodie.func.CopyOnWriteLazyInsertIterable$CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteLazyInsertIterable.java:149)
at
com.uber.hoodie.func.CopyOnWriteLazyInsertIterable$CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteLazyInsertIterable.java:127)
at
com.uber.hoodie.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:38)
at
com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:124)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 more
Driver stacktrace: at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:2100)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:2088)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:2087)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2087)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:1076)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:1076)
at scala.Option.foreach(Option.scala:257) at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1076)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2319)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2267)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2255)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49) at
org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:873) at
org.apache.spark.SparkContext.runJob(SparkContext.scala:2252) at
org.apache.spark.SparkContext.runJob(SparkContext.scala:2274) at
org.apache.spark.SparkContext.runJob(SparkContext.scala:2293) at
org.apache.spark.SparkContext.runJob(SparkContext.scala:2318) at
org.apache.spark.rdd.RDD.count(RDD.scala:1184) at
com.uber.hoodie.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:151)
at com.uber.hoodie.DefaultSource.createRelation(DefaultSource.scala:91) at
org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:72)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:70)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:88)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:143)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$5.apply(SparkPlan.scala:183)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at
org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:180)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:131) at
org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:114)
at
org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:114)
at
org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:690)
at
org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:690)
at
org.apache.spark.sql.execution.SQLExecution$$anonfun$withCustomExecutionEnv$1.apply(SQLExecution.scala:99)
at
org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:228)
at
org.apache.spark.sql.execution.SQLExecution$.withCustomExecutionEnv(SQLExecution.scala:85)
at
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:158)
at
org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:690)
at
org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:290)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:284) at
org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:232) at
line2873d1c3d6574f6ca1d33c50e3f9f73548.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(
command-1085008513297384:10) at
line2873d1c3d6574f6ca1d33c50e3f9f73548.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(
command-1085008513297384:61) at
line2873d1c3d6574f6ca1d33c50e3f9f73548.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(
command-1085008513297384:63) at
line2873d1c3d6574f6ca1d33c50e3f9f73548.$read$$iw$$iw$$iw$$iw$$iw.<init>(
command-1085008513297384:65) at
line2873d1c3d6574f6ca1d33c50e3f9f73548.$read$$iw$$iw$$iw$$iw.<init>(
command-1085008513297384:67) at
line2873d1c3d6574f6ca1d33c50e3f9f73548.$read$$iw$$iw$$iw.<init>(
command-1085008513297384:69) at
line2873d1c3d6574f6ca1d33c50e3f9f73548.$read$$iw$$iw.<init>(
command-1085008513297384:71) at
line2873d1c3d6574f6ca1d33c50e3f9f73548.$read$$iw.<init>(
command-1085008513297384:73) at
line2873d1c3d6574f6ca1d33c50e3f9f73548.$read.<init>(
command-1085008513297384:75) at
line2873d1c3d6574f6ca1d33c50e3f9f73548.$read$.<init>(
command-1085008513297384:79) at
line2873d1c3d6574f6ca1d33c50e3f9f73548.$read$.<clinit>(command-1085008513297384)
at
line2873d1c3d6574f6ca1d33c50e3f9f73548.$eval$.$print$lzycompute(<notebook>:7)
at line2873d1c3d6574f6ca1d33c50e3f9f73548.$eval$.$print(<notebook>:6) at
line2873d1c3d6574f6ca1d33c50e3f9f73548.$eval.$print(<notebook>) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786) at
scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047) at
scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:638)
at
scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637)
at
scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
at
scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
at
scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:637)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569) at
scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565) at
com.databricks.backend.daemon.driver.DriverILoop.execute(DriverILoop.scala:199)
at
com.databricks.backend.daemon.driver.ScalaDriverLocal$$anonfun$repl$1.apply$mcV$sp(ScalaDriverLocal.scala:189)
at
com.databricks.backend.daemon.driver.ScalaDriverLocal$$anonfun$repl$1.apply(ScalaDriverLocal.scala:189)
at
com.databricks.backend.daemon.driver.ScalaDriverLocal$$anonfun$repl$1.apply(ScalaDriverLocal.scala:189)
at
com.databricks.backend.daemon.driver.DriverLocal$TrapExitInternal$.trapExit(DriverLocal.scala:587)
at
com.databricks.backend.daemon.driver.DriverLocal$TrapExit$.apply(DriverLocal.scala:542)
at
com.databricks.backend.daemon.driver.ScalaDriverLocal.repl(ScalaDriverLocal.scala:189)
at
com.databricks.backend.daemon.driver.DriverLocal$$anonfun$execute$7.apply(DriverLocal.scala:324)
at
com.databricks.backend.daemon.driver.DriverLocal$$anonfun$execute$7.apply(DriverLocal.scala:304)
at
com.databricks.logging.UsageLogging$$anonfun$withAttributionContext$1.apply(UsageLogging.scala:235)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) at
com.databricks.logging.UsageLogging$class.withAttributionContext(UsageLogging.scala:230)
at
com.databricks.backend.daemon.driver.DriverLocal.withAttributionContext(DriverLocal.scala:45)
at
com.databricks.logging.UsageLogging$class.withAttributionTags(UsageLogging.scala:268)
at
com.databricks.backend.daemon.driver.DriverLocal.withAttributionTags(DriverLocal.scala:45)
at
com.databricks.backend.daemon.driver.DriverLocal.execute(DriverLocal.scala:304)
at
com.databricks.backend.daemon.driver.DriverWrapper$$anonfun$tryExecutingCommand$2.apply(DriverWrapper.scala:589)
at
com.databricks.backend.daemon.driver.DriverWrapper$$anonfun$tryExecutingCommand$2.apply(DriverWrapper.scala:589)
at scala.util.Try$.apply(Try.scala:192) at
com.databricks.backend.daemon.driver.DriverWrapper.tryExecutingCommand(DriverWrapper.scala:584)
at
com.databricks.backend.daemon.driver.DriverWrapper.getCommandOutputAndError(DriverWrapper.scala:475)
at
com.databricks.backend.daemon.driver.DriverWrapper.executeCommand(DriverWrapper.scala:542)
at
com.databricks.backend.daemon.driver.DriverWrapper.runInnerLoop(DriverWrapper.scala:381)
at
com.databricks.backend.daemon.driver.DriverWrapper.runInner(DriverWrapper.scala:328)
at
com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:215)
at java.lang.Thread.run(Thread.java:748) Caused by:
java.lang.RuntimeException: com.uber.hoodie.exception.HoodieException:
com.uber.hoodie.exception.HoodieException:
java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError:
com.uber.hoodie.avro.HoodieAvroWriteSupport.<init>(Lcom/uber/hoodie/org/apache/parquet/schema/MessageType;Lorg/apache/avro/Schema;Lcom/uber/hoodie/common/BloomFilter;)V
at
com.uber.hoodie.func.LazyIterableIterator.next(LazyIterableIterator.java:121)
at
scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) at
scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) at
org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
at
org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:349)
at
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1187)
at
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1161)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1096) at
org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1161)
at
org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:883)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:351) at
org.apache.spark.rdd.RDD.iterator(RDD.scala:302) at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:60) at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:340) at
org.apache.spark.rdd.RDD.iterator(RDD.scala:304) at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at
org.apache.spark.scheduler.Task.doRunTask(Task.scala:139) at
org.apache.spark.scheduler.Task.run(Task.scala:112) at
org.apache.spark.executor.Executor$TaskRunner$$anonfun$13.apply(Executor.scala:497)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1432) at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:503) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) Caused by:
com.uber.hoodie.exception.HoodieException:
com.uber.hoodie.exception.HoodieException:
java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError:
com.uber.hoodie.avro.HoodieAvroWriteSupport.<init>(Lcom/uber/hoodie/org/apache/parquet/schema/MessageType;Lorg/apache/avro/Schema;Lcom/uber/hoodie/common/BloomFilter;)V
at
com.uber.hoodie.func.CopyOnWriteLazyInsertIterable.computeNext(CopyOnWriteLazyInsertIterable.java:106)
at
com.uber.hoodie.func.CopyOnWriteLazyInsertIterable.computeNext(CopyOnWriteLazyInsertIterable.java:45)
at
com.uber.hoodie.func.LazyIterableIterator.next(LazyIterableIterator.java:119)
at
scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) at
scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) at
org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
at
org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:349)
at
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1187)
at
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1161)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1096) at
org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1161)
at
org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:883)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:351) at
org.apache.spark.rdd.RDD.iterator(RDD.scala:302) at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:60) at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:340) at
org.apache.spark.rdd.RDD.iterator(RDD.scala:304) at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at
org.apache.spark.scheduler.Task.doRunTask(Task.scala:139) at
org.apache.spark.scheduler.Task.run(Task.scala:112) at
org.apache.spark.executor.Executor$TaskRunner$$anonfun$13.apply(Executor.scala:497)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1432) at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:503) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) Caused by:
com.uber.hoodie.exception.HoodieException:
java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError:
com.uber.hoodie.avro.HoodieAvroWriteSupport.<init>(Lcom/uber/hoodie/org/apache/parquet/schema/MessageType;Lorg/apache/avro/Schema;Lcom/uber/hoodie/common/BloomFilter;)V
at
com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:146)
at
com.uber.hoodie.func.CopyOnWriteLazyInsertIterable.computeNext(CopyOnWriteLazyInsertIterable.java:102)
at
com.uber.hoodie.func.CopyOnWriteLazyInsertIterable.computeNext(CopyOnWriteLazyInsertIterable.java:45)
at
com.uber.hoodie.func.LazyIterableIterator.next(LazyIterableIterator.java:119)
at
scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) at
scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) at
org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
at
org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:349)
at
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1187)
at
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1161)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1096) at
org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1161)
at
org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:883)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:351) at
org.apache.spark.rdd.RDD.iterator(RDD.scala:302) at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:60) at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:340) at
org.apache.spark.rdd.RDD.iterator(RDD.scala:304) at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at
org.apache.spark.scheduler.Task.doRunTask(Task.scala:139) at
org.apache.spark.scheduler.Task.run(Task.scala:112) at
org.apache.spark.executor.Executor$TaskRunner$$anonfun$13.apply(Executor.scala:497)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1432) at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:503) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) Caused by:
java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError:
com.uber.hoodie.avro.HoodieAvroWriteSupport.<init>(Lcom/uber/hoodie/org/apache/parquet/schema/MessageType;Lorg/apache/avro/Schema;Lcom/uber/hoodie/common/BloomFilter;)V
at java.util.concurrent.FutureTask.report(FutureTask.java:122) at
java.util.concurrent.FutureTask.get(FutureTask.java:192) at
com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:144)
at
com.uber.hoodie.func.CopyOnWriteLazyInsertIterable.computeNext(CopyOnWriteLazyInsertIterable.java:102)
at
com.uber.hoodie.func.CopyOnWriteLazyInsertIterable.computeNext(CopyOnWriteLazyInsertIterable.java:45)
at
com.uber.hoodie.func.LazyIterableIterator.next(LazyIterableIterator.java:119)
at
scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) at
scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) at
org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
at
org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:349)
at
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1187)
at
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1161)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1096) at
org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1161)
at
org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:883)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:351) at
org.apache.spark.rdd.RDD.iterator(RDD.scala:302) at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:60) at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:340) at
org.apache.spark.rdd.RDD.iterator(RDD.scala:304) at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at
org.apache.spark.scheduler.Task.doRunTask(Task.scala:139) at
org.apache.spark.scheduler.Task.run(Task.scala:112) at
org.apache.spark.executor.Executor$TaskRunner$$anonfun$13.apply(Executor.scala:497)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1432) at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:503) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) Caused by:
java.lang.NoSuchMethodError:
com.uber.hoodie.avro.HoodieAvroWriteSupport.<init>(Lcom/uber/hoodie/org/apache/parquet/schema/MessageType;Lorg/apache/avro/Schema;Lcom/uber/hoodie/common/BloomFilter;)V
at
com.uber.hoodie.io.storage.HoodieStorageWriterFactory.newParquetStorageWriter(HoodieStorageWriterFactory.java:47)
at
com.uber.hoodie.io.storage.HoodieStorageWriterFactory.getStorageWriter(HoodieStorageWriterFactory.java:38)
at com.uber.hoodie.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:71)
at
com.uber.hoodie.func.CopyOnWriteLazyInsertIterable$CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteLazyInsertIterable.java:149)
at
com.uber.hoodie.func.CopyOnWriteLazyInsertIterable$CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteLazyInsertIterable.java:127)
at
com.uber.hoodie.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:38)
at
com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:124)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
On Mon, Mar 11, 2019 at 8:43 AM Vinoth Chandar <[email protected]> wrote:
> +1 yes. if its actually null.
>
> Good catch, Frank! :)
>
> On Sun, Mar 10, 2019 at 7:23 PM kaka chen <[email protected]> wrote:
>
> > A possible root cause is the filed of record is null.
> >
> > public static String getNestedFieldValAsString(GenericRecord record,
> > String fieldName) {
> > String[] parts = fieldName.split("\\.");
> > GenericRecord valueNode = record;
> > int i = 0;
> > for (;i < parts.length; i++) {
> > String part = parts[i];
> > Object val = valueNode.get(part);
> > if (val == null) {
> > break;
> > }
> >
> > // return, if last part of name
> > if (i == parts.length - 1) {
> > return val.toString();
> > } else {
> > // VC: Need a test here
> > if (!(val instanceof GenericRecord)) {
> > throw new HoodieException("Cannot find a record at part value
> > :" + part);
> > }
> > valueNode = (GenericRecord) val;
> > }
> > }
> > throw new HoodieException(fieldName + "(Part -" + parts[i] + ")
> > field not found in record. "
> > + "Acceptable fields were :" + valueNode.getSchema().getFields()
> > .stream().map(Field::name).collect(Collectors.toList()));
> > }
> >
> >
> > Vinoth Chandar <[email protected]> 于2019年3月10日周日 下午2:11写道：
> >
> > > Hmmm. Thats interesting. I can see that the parsing works, since the
> > > exception said "Part - review_date". There are definitely users who
> have
> > > done this before.
> > > So not sure what's going on.
> > >
> > > Can you paste the generated Avro schema? following is the corresponding
> > > code line
> > > log.info(s"Registered avro schema : ${schema.toString(true)}")
> > >
> > > May be create a gist (gist.github.com), for easier sharing of
> > > code/stacktrace?
> > > Thanks
> > > Vinoth
> > >
> > > On Sat, Mar 9, 2019 at 1:33 PM Umesh Kacha <[email protected]>
> > wrote:
> > >
> > > > Hi Vinoth thanks I have already did and checked that please see red
> > > column
> > > > highlighted below.
> > > >
> > > > root |-- marketplace: string (nullable = true) |-- customer_id:
> string
> > > > (nullable = true) |-- review_id: string (nullable = true) |--
> > product_id:
> > > > string (nullable = true) |-- product_parent: string (nullable = true)
> > |--
> > > > product_title: string (nullable = true) |-- product_category: string
> > > > (nullable = true) |-- star_rating: string (nullable = true) |--
> > > > helpful_votes: string (nullable = true) |-- total_votes: string
> > > (nullable =
> > > > true) |-- vine: string (nullable = true) |-- verified_purchase:
> string
> > > > (nullable = true) |-- review_headline: string (nullable = true) |--
> > > > review_body: string (nullable = true) |-- review_date: string
> > (nullable =
> > > > true) |-- year: integer (nullable = true)
> > > >
> > > > On Sun, Mar 10, 2019 at 2:27 AM Vinoth Chandar <[email protected]>
> > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > >>review_date(Part
> > > > > -review_date) field not found in record
> > > > >
> > > > > Seems like the precombine field is not in the input DF? Can you try
> > > doing
> > > > > df1.printSchema and check that once?
> > > > >
> > > > > On Sat, Mar 9, 2019 at 11:52 AM Umesh Kacha <[email protected]
> >
> > > > wrote:
> > > > >
> > > > > > Hi I have the following code using which I am trying to bulk
> insert
> > > > huge
> > > > > > csv file loaded into Spark DataFrame but it fails saying column
> > > > > review_date
> > > > > > not found but that column is definitely there in dataframe.
> Please
> > > > guide.
> > > > > >
> > > > > > df1.write
> > > > > > .format("com.uber.hoodie")
> > > > > > .option(DataSourceWriteOptions.STORAGE_TYPE_OPT_KEY,
> > > > > > HoodieTableType.COPY_ON_WRITE.name())
> > > > > > .option(DataSourceWriteOptions.OPERATION_OPT_KEY,
> > > > > > DataSourceWriteOptions.BULK_INSERT_OPERATION_OPT_VAL) // insert
> > > > > > .option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY,
> > > > > > "customer_id")
> > > > > > .option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY,
> > > > "year")
> > > > > > .option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY,
> > > > > > "review_date")
> > > > > > .option(HoodieWriteConfig.TABLE_NAME, "hoodie_test_table")
> > > > > > .mode(SaveMode.Overwrite)
> > > > > > .save("/tmp/hoodie/test_hoodie")
> > > > > >
> > > > > >
> > > > > > Caused by: com.uber.hoodie.exception.HoodieException:
> > > review_date(Part
> > > > > > -review_date) field not found in record. Acceptable fields were
> > > > > > :[marketplace, customer_id, review_id, product_id,
> product_parent,
> > > > > > product_title, product_category, star_rating, helpful_votes,
> > > > total_votes,
> > > > > > vine, verified_purchase, review_headline, review_body,
> review_date,
> > > > year]
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> com.uber.hoodie.DataSourceUtils.getNestedFieldValAsString(DataSourceUtils.java:79)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> com.uber.hoodie.HoodieSparkSqlWriter$$anonfun$1.apply(HoodieSparkSqlWriter.scala:93)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> com.uber.hoodie.HoodieSparkSqlWriter$$anonfun$1.apply(HoodieSparkSqlWriter.scala:92)
> > > > > > at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) at
> > > > > > scala.collection.Iterator$$anon$11.next(Iterator.scala:409) at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:193)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:62)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
> > > > > > at org.apache.spark.scheduler.Task.doRunTask(Task.scala:139) at
> > > > > > org.apache.spark.scheduler.Task.run(Task.scala:112) at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.spark.executor.Executor$TaskRunner$$anonfun$13.apply(Executor.scala:497)
> > > > > > at
> > org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1432)
> > > at
> > > > > >
> > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:503)
> > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > > Command took 4.45 seconds -- by [email protected] at
> > 3/10/2019,
> > > > > > 1:17:42
> > > > > > AM on Spark_Hudi
> > > > > > Shift+Enter to run shortcuts
> > > > > >
> > > > >
> > > >
> > >
> >
>
