Dear All
I am unable to run hudi with hadoop 3.0.1. The spark injection job is failing
saying method not found.
19/03/14 13:30:12 INFO IteratorBasedQueueProducer: starting to buffer records
error
------
19/03/14 13:30:12 INFO BoundedInMemoryExecutor: starting consumer thread
19/03/14 13:30:12 INFO FSUtils: Hadoop Configuration: fs.defaultFS:
[hdfs://x.x.x.x:8020], Config:[Configuration: ], FileSystem:
[DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1685428217_1, ugi=hudi
(auth:SIMPLE)]]]
19/03/14 13:30:12 INFO HoodieIOHandle: Deleting 0 files generated by previous
failed attempts.
19/03/14 13:30:12 INFO FSUtils: Hadoop Configuration: fs.defaultFS:
[hdfs://x.x.x.x:8020], Config:[Configuration: ], FileSystem:
[DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1685428217_1, ugi=hudi
(auth:SIMPLE)]]]
19/03/14 13:30:12 INFO FSUtils: Hadoop Configuration: fs.defaultFS:
[hdfs://x.x.x.:8020], Config:[Configuration: ], FileSystem:
[DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1685428217_1, ugi=hudi
(auth:SIMPLE)]]]
19/03/14 13:30:12 INFO IteratorBasedQueueProducer: finished buffering records
19/03/14 13:30:12 INFO FSUtils: Hadoop Configuration: fs.defaultFS:
[hdfs://10.0.0.28.local:8020], Config:[Configuration: ], FileSystem:
[DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1685428217_1, ugi=hudi
(auth:SIMPLE)]]]
19/03/14 13:30:12 WARN BlockManager: Putting block rdd_16_0 failed due to
exception java.lang.RuntimeException:
com.uber.hoodie.exception.HoodieException:
com.uber.hoodie.exception.HoodieException:
java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError:
org.apache.hadoop.fs.FSDataOutputStream: method <init>(Ljava/io/OutputStream;)V
not found.
19/03/14 13:30:12 WARN BlockManager: Block rdd_16_0 could not be removed as it
was not found on disk or in memory
19/03/14 13:30:12 ERROR Executor: Exception in task 0.0 in stage 7.0 (TID 6)
java.lang.RuntimeException: com.uber.hoodie.exception.HoodieException:
com.uber.hoodie.exception.HoodieException:
java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError:
org.apache.hadoop.fs.FSDataOutputStream: method <init>(Ljava/io/OutputStream;)V
not found
at
com.uber.hoodie.func.LazyIterableIterator.next(LazyIterableIterator.java:121)
at
scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
at
org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:378)
at
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1109)
at
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1083)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1018)
at
org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1083)
at
org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:809)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.uber.hoodie.exception.HoodieException:
com.uber.hoodie.exception.HoodieException:
java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError:
org.apache.hadoop.fs.FSDataOutputStream: method <init>(Ljava/io/OutputStream;)V
not found
Is there any hope for hudi to work with hadoop3?. Please assist on this.
Thanks & Regards
Rahul
