Great. added you as contributor. On Mon, Mar 18, 2019 at 3:16 AM [email protected] < [email protected]> wrote:
> > > On 2019/03/15 18:01:43, Vinoth Chandar <[email protected]> wrote: > > Hi Rahul, > > > > Plan sounds good. Thanks! > > If you can share your JIRA id, I will add you as a contributor and make a > > JIRA assigned to you as well. > > > > Thanks > > Vinoth > > > > On Fri, Mar 15, 2019 at 1:24 AM [email protected] < > > [email protected]> wrote: > > > > > > > > > > > On 2019/03/15 05:46:51, [email protected] < > [email protected]> > > > wrote: > > > > > > > > > > > > On 2019/03/15 00:56:00, Vinoth Chandar <[email protected]> wrote: > > > > > Hi Rahul, > > > > > > > > > > We have not tested Hudi with Hadoop 3 yet. Seems like the > > > > > FSDataOutputStream constructor got changed in 3.x. > > > > > Could you paste the entire stack trace here or preferably in a > gist? > > > > > > > > > > Also, are you able to drive this, if we provide help? > > > > > > > > > > Thanks > > > > > Vinoth > > > > > > > > > > On Thu, Mar 14, 2019 at 1:19 AM [email protected] < > > > > > [email protected]> wrote: > > > > > > > > > > > Dear All > > > > > > > > > > > > I am unable to run hudi with hadoop 3.0.1. The spark injection > job > > > is > > > > > > failing saying method not found. > > > > > > 19/03/14 13:30:12 INFO IteratorBasedQueueProducer: starting to > buffer > > > > > > records > > > > > > > > > > > > error > > > > > > ------ > > > > > > 19/03/14 13:30:12 INFO BoundedInMemoryExecutor: starting consumer > > > thread > > > > > > 19/03/14 13:30:12 INFO FSUtils: Hadoop Configuration: > fs.defaultFS: > > > > > > [hdfs://x.x.x.x:8020], Config:[Configuration: ], FileSystem: > > > > > > [DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1685428217_1, > > > ugi=hudi > > > > > > (auth:SIMPLE)]]] > > > > > > 19/03/14 13:30:12 INFO HoodieIOHandle: Deleting 0 files > generated by > > > > > > previous failed attempts. > > > > > > 19/03/14 13:30:12 INFO FSUtils: Hadoop Configuration: > fs.defaultFS: > > > > > > [hdfs://x.x.x.x:8020], Config:[Configuration: ], FileSystem: > > > > > > [DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1685428217_1, > > > ugi=hudi > > > > > > (auth:SIMPLE)]]] > > > > > > 19/03/14 13:30:12 INFO FSUtils: Hadoop Configuration: > fs.defaultFS: > > > > > > [hdfs://x.x.x.:8020], Config:[Configuration: ], FileSystem: > > > > > > [DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1685428217_1, > > > ugi=hudi > > > > > > (auth:SIMPLE)]]] > > > > > > 19/03/14 13:30:12 INFO IteratorBasedQueueProducer: finished > buffering > > > > > > records > > > > > > 19/03/14 13:30:12 INFO FSUtils: Hadoop Configuration: > fs.defaultFS: > > > > > > [hdfs://10.0.0.28.local:8020], Config:[Configuration: ], > FileSystem: > > > > > > [DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1685428217_1, > > > ugi=hudi > > > > > > (auth:SIMPLE)]]] > > > > > > 19/03/14 13:30:12 WARN BlockManager: Putting block rdd_16_0 > failed > > > due to > > > > > > exception java.lang.RuntimeException: > > > > > > com.uber.hoodie.exception.HoodieException: > > > > > > com.uber.hoodie.exception.HoodieException: > > > > > > java.util.concurrent.ExecutionException: > java.lang.NoSuchMethodError: > > > > > > org.apache.hadoop.fs.FSDataOutputStream: method > > > > > > <init>(Ljava/io/OutputStream;)V not found. > > > > > > 19/03/14 13:30:12 WARN BlockManager: Block rdd_16_0 could not be > > > removed > > > > > > as it was not found on disk or in memory > > > > > > 19/03/14 13:30:12 ERROR Executor: Exception in task 0.0 in stage > 7.0 > > > (TID > > > > > > 6) > > > > > > java.lang.RuntimeException: > > > com.uber.hoodie.exception.HoodieException: > > > > > > com.uber.hoodie.exception.HoodieException: > > > > > > java.util.concurrent.ExecutionException: > java.lang.NoSuchMethodError: > > > > > > org.apache.hadoop.fs.FSDataOutputStream: method > > > > > > <init>(Ljava/io/OutputStream;)V not found > > > > > > at > > > > > > > > > > com.uber.hoodie.func.LazyIterableIterator.next(LazyIterableIterator.java:121) > > > > > > at > > > > > > > > > > scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43) > > > > > > at > > > scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) > > > > > > at > > > scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) > > > > > > at > > > > > > > > > > org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:378) > > > > > > at > > > > > > > > > > org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1109) > > > > > > at > > > > > > > > > > org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1083) > > > > > > at > > > > > > > org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1018) > > > > > > at > > > > > > > > > > org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1083) > > > > > > at > > > > > > > > > > org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:809) > > > > > > at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335) > > > > > > at org.apache.spark.rdd.RDD.iterator(RDD.scala:286) > > > > > > at > > > > > > > > > > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > > > > > > at > > > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324) > > > > > > at org.apache.spark.rdd.RDD.iterator(RDD.scala:288) > > > > > > at > > > > > > > org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) > > > > > > at org.apache.spark.scheduler.Task.run(Task.scala:109) > > > > > > at > > > > > > > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345) > > > > > > at > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > > > > > at > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > > > > > > at java.lang.Thread.run(Thread.java:745) > > > > > > Caused by: com.uber.hoodie.exception.HoodieException: > > > > > > com.uber.hoodie.exception.HoodieException: > > > > > > java.util.concurrent.ExecutionException: > java.lang.NoSuchMethodError: > > > > > > org.apache.hadoop.fs.FSDataOutputStream: method > > > > > > <init>(Ljava/io/OutputStream;)V not found > > > > > > > > > > > > > > > > > > Is there any hope for hudi to work with hadoop3?. Please assist > on > > > this. > > > > > > > > > > > > Thanks & Regards > > > > > > Rahul > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Dear Vinoth > > > > > > > > I checked this case & I found that the Deprecated constructor of > > > FSDataOutputStream class > > > > FSDataOutputStream(OutputStream out) > > > > has been removed in later releases. So in hadoop 3 only 2 > constructors > > > are there in this class > > > > 1. FSDataOutputStream(OutputStream out, > > > org.apache.hadoop.fs.FileSystem.Statistics stats) > > > > 2. FSDataOutputStream(OutputStream out, > > > org.apache.hadoop.fs.FileSystem.Statistics stats, long startPosition) > > > > > > > > So as a quick fix i changed the constructor which is calling from > > > SizeAwareFSDataOutputStream.java of hudi. > > > > > > > > public SizeAwareFSDataOutputStream(FSDataOutputStream out, Runnable > > > closeCallback) > > > > throws IOException { > > > > super(out); > > > > this.closeCallback = closeCallback; > > > > } > > > > > > > > After changing the super(out) to super(out,null) it's working fine. > > > > > > > > Thanks & Regards > > > > Rahul P > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Dear Vinoth > > > > > > Only DeltaStreamer insertion i checked with this. Please note still the > > > rest of the code is compiled with older version of hadoop only. I am > facing > > > issues while building the project with hadoop3 & hive3. I will get > back to > > > you on this. > > > > > > Thanks & Regards > > > Rahul P > > > > > > > > Dear Vinoth > > My jira id is rahuledavalath > > Thanks & Regards > Rahul >
