Hi Rahul, Plan sounds good. Thanks! If you can share your JIRA id, I will add you as a contributor and make a JIRA assigned to you as well.
Thanks Vinoth On Fri, Mar 15, 2019 at 1:24 AM [email protected] < [email protected]> wrote: > > > On 2019/03/15 05:46:51, [email protected] <[email protected]> > wrote: > > > > > > On 2019/03/15 00:56:00, Vinoth Chandar <[email protected]> wrote: > > > Hi Rahul, > > > > > > We have not tested Hudi with Hadoop 3 yet. Seems like the > > > FSDataOutputStream constructor got changed in 3.x. > > > Could you paste the entire stack trace here or preferably in a gist? > > > > > > Also, are you able to drive this, if we provide help? > > > > > > Thanks > > > Vinoth > > > > > > On Thu, Mar 14, 2019 at 1:19 AM [email protected] < > > > [email protected]> wrote: > > > > > > > Dear All > > > > > > > > I am unable to run hudi with hadoop 3.0.1. The spark injection job > is > > > > failing saying method not found. > > > > 19/03/14 13:30:12 INFO IteratorBasedQueueProducer: starting to buffer > > > > records > > > > > > > > error > > > > ------ > > > > 19/03/14 13:30:12 INFO BoundedInMemoryExecutor: starting consumer > thread > > > > 19/03/14 13:30:12 INFO FSUtils: Hadoop Configuration: fs.defaultFS: > > > > [hdfs://x.x.x.x:8020], Config:[Configuration: ], FileSystem: > > > > [DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1685428217_1, > ugi=hudi > > > > (auth:SIMPLE)]]] > > > > 19/03/14 13:30:12 INFO HoodieIOHandle: Deleting 0 files generated by > > > > previous failed attempts. > > > > 19/03/14 13:30:12 INFO FSUtils: Hadoop Configuration: fs.defaultFS: > > > > [hdfs://x.x.x.x:8020], Config:[Configuration: ], FileSystem: > > > > [DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1685428217_1, > ugi=hudi > > > > (auth:SIMPLE)]]] > > > > 19/03/14 13:30:12 INFO FSUtils: Hadoop Configuration: fs.defaultFS: > > > > [hdfs://x.x.x.:8020], Config:[Configuration: ], FileSystem: > > > > [DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1685428217_1, > ugi=hudi > > > > (auth:SIMPLE)]]] > > > > 19/03/14 13:30:12 INFO IteratorBasedQueueProducer: finished buffering > > > > records > > > > 19/03/14 13:30:12 INFO FSUtils: Hadoop Configuration: fs.defaultFS: > > > > [hdfs://10.0.0.28.local:8020], Config:[Configuration: ], FileSystem: > > > > [DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1685428217_1, > ugi=hudi > > > > (auth:SIMPLE)]]] > > > > 19/03/14 13:30:12 WARN BlockManager: Putting block rdd_16_0 failed > due to > > > > exception java.lang.RuntimeException: > > > > com.uber.hoodie.exception.HoodieException: > > > > com.uber.hoodie.exception.HoodieException: > > > > java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: > > > > org.apache.hadoop.fs.FSDataOutputStream: method > > > > <init>(Ljava/io/OutputStream;)V not found. > > > > 19/03/14 13:30:12 WARN BlockManager: Block rdd_16_0 could not be > removed > > > > as it was not found on disk or in memory > > > > 19/03/14 13:30:12 ERROR Executor: Exception in task 0.0 in stage 7.0 > (TID > > > > 6) > > > > java.lang.RuntimeException: > com.uber.hoodie.exception.HoodieException: > > > > com.uber.hoodie.exception.HoodieException: > > > > java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: > > > > org.apache.hadoop.fs.FSDataOutputStream: method > > > > <init>(Ljava/io/OutputStream;)V not found > > > > at > > > > > com.uber.hoodie.func.LazyIterableIterator.next(LazyIterableIterator.java:121) > > > > at > > > > > scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43) > > > > at > scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) > > > > at > scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) > > > > at > > > > > org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:378) > > > > at > > > > > org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1109) > > > > at > > > > > org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1083) > > > > at > > > > org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1018) > > > > at > > > > > org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1083) > > > > at > > > > > org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:809) > > > > at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335) > > > > at org.apache.spark.rdd.RDD.iterator(RDD.scala:286) > > > > at > > > > > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > > > > at > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324) > > > > at org.apache.spark.rdd.RDD.iterator(RDD.scala:288) > > > > at > > > > org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) > > > > at org.apache.spark.scheduler.Task.run(Task.scala:109) > > > > at > > > > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345) > > > > at > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > > > > at java.lang.Thread.run(Thread.java:745) > > > > Caused by: com.uber.hoodie.exception.HoodieException: > > > > com.uber.hoodie.exception.HoodieException: > > > > java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: > > > > org.apache.hadoop.fs.FSDataOutputStream: method > > > > <init>(Ljava/io/OutputStream;)V not found > > > > > > > > > > > > Is there any hope for hudi to work with hadoop3?. Please assist on > this. > > > > > > > > Thanks & Regards > > > > Rahul > > > > > > > > > > > > > > > > > > > > > > > Dear Vinoth > > > > I checked this case & I found that the Deprecated constructor of > FSDataOutputStream class > > FSDataOutputStream(OutputStream out) > > has been removed in later releases. So in hadoop 3 only 2 constructors > are there in this class > > 1. FSDataOutputStream(OutputStream out, > org.apache.hadoop.fs.FileSystem.Statistics stats) > > 2. FSDataOutputStream(OutputStream out, > org.apache.hadoop.fs.FileSystem.Statistics stats, long startPosition) > > > > So as a quick fix i changed the constructor which is calling from > SizeAwareFSDataOutputStream.java of hudi. > > > > public SizeAwareFSDataOutputStream(FSDataOutputStream out, Runnable > closeCallback) > > throws IOException { > > super(out); > > this.closeCallback = closeCallback; > > } > > > > After changing the super(out) to super(out,null) it's working fine. > > > > Thanks & Regards > > Rahul P > > > > > > > > > > > > > > > > > > > > > Dear Vinoth > > Only DeltaStreamer insertion i checked with this. Please note still the > rest of the code is compiled with older version of hadoop only. I am facing > issues while building the project with hadoop3 & hive3. I will get back to > you on this. > > Thanks & Regards > Rahul P > >
