amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException URL: https://github.com/apache/incubator-hudi/issues/741#issuecomment-504144982 I am getting below error. Caused by: com.uber.hoodie.exception.HoodieException: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: BlockAlignedAvroParquetWriter does not support scheme maprfs 19/06/20 14:00:32 WARN Utils: Service 'SparkUI' could not bind on port 4240. Attempting port 4241. 19/06/20 14:00:32 WARN Utils: Service 'SparkUI' could not bind on port 4241. Attempting port 4242. 19/06/20 14:00:35 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. 19/06/20 14:01:03 WARN SparkContext: Using an existing SparkContext; some configuration may not take effect. 19/06/20 14:01:24 WARN TaskSetManager: Lost task 0.0 in stage 21.0 (TID 23, dbsls0327.uhc.com, executor 2): java.lang.RuntimeException: com.uber.hoodie.exception.HoodieException: com.uber.hoodie.exception.HoodieException: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: BlockAlignedAvroParquetWriter does not support scheme maprfs at com.uber.hoodie.func.LazyIterableIterator.next(LazyIterableIterator.java:123) at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43) at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) at org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:371) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1055) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1029) at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:969) at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1029) at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:760) at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334) at org.apache.spark.rdd.RDD.iterator(RDD.scala:285) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:108) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: com.uber.hoodie.exception.HoodieException: com.uber.hoodie.exception.HoodieException: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: BlockAlignedAvroParquetWriter does not support scheme maprfs at com.uber.hoodie.func.CopyOnWriteLazyInsertIterable.computeNext(CopyOnWriteLazyInsertIterable.java:106) at com.uber.hoodie.func.CopyOnWriteLazyInsertIterable.computeNext(CopyOnWriteLazyInsertIterable.java:43) at com.uber.hoodie.func.LazyIterableIterator.next(LazyIterableIterator.java:121) ... 20 more Caused by: com.uber.hoodie.exception.HoodieException: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: BlockAlignedAvroParquetWriter does not support scheme maprfs at com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:148) at com.uber.hoodie.func.CopyOnWriteLazyInsertIterable.computeNext(CopyOnWriteLazyInsertIterable.java:102) ... 22 more Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: BlockAlignedAvroParquetWriter does not support scheme maprfs at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:146) ... 23 more Caused by: java.lang.IllegalArgumentException: BlockAlignedAvroParquetWriter does not support scheme maprfs at com.uber.hoodie.common.io.storage.HoodieWrapperFileSystem.getHoodieScheme(HoodieWrapperFileSystem.java:110) at com.uber.hoodie.common.io.storage.HoodieWrapperFileSystem.convertToHoodiePath(HoodieWrapperFileSystem.java:86) at com.uber.hoodie.io.storage.HoodieParquetWriter.<init>(HoodieParquetWriter.java:57) at com.uber.hoodie.io.storage.HoodieStorageWriterFactory.newParquetStorageWriter(HoodieStorageWriterFactory.java:56) at com.uber.hoodie.io.storage.HoodieStorageWriterFactory.getStorageWriter(HoodieStorageWriterFactory.java:39) at com.uber.hoodie.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:70) at com.uber.hoodie.func.CopyOnWriteLazyInsertIterable$CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteLazyInsertIterable.java:142) at com.uber.hoodie.func.CopyOnWriteLazyInsertIterable$CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteLazyInsertIterable.java:130) at com.uber.hoodie.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:38) at com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:126) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 more 19/06/20 14:01:25 ERROR TaskSetManager: Task 0 in stage 21.0 failed 4 times; aborting job Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 21.0 failed 4 times, most recent failure: Lost task 0.3 in stage 21.0 (TID 26, dbsls0327.uhc.com, executor 2): java.lang.RuntimeException: com.uber.hoodie.exception.HoodieException: com.uber.hoodie.exception.HoodieException: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: BlockAlignedAvroParquetWriter does not support scheme maprfs at com.uber.hoodie.func.LazyIterableIterator.next(LazyIterableIterator.java:123) at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43) at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) at org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:371) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1055) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1029) at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:969) at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1029) at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:760) at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334) at org.apache.spark.rdd.RDD.iterator(RDD.scala:285) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:108) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: com.uber.hoodie.exception.HoodieException: com.uber.hoodie.exception.HoodieException: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: BlockAlignedAvroParquetWriter does not support scheme maprfs at com.uber.hoodie.func.CopyOnWriteLazyInsertIterable.computeNext(CopyOnWriteLazyInsertIterable.java:106) at com.uber.hoodie.func.CopyOnWriteLazyInsertIterable.computeNext(CopyOnWriteLazyInsertIterable.java:43) at com.uber.hoodie.func.LazyIterableIterator.next(LazyIterableIterator.java:121) ... 20 more Caused by: com.uber.hoodie.exception.HoodieException: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: BlockAlignedAvroParquetWriter does not support scheme maprfs at com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:148) at com.uber.hoodie.func.CopyOnWriteLazyInsertIterable.computeNext(CopyOnWriteLazyInsertIterable.java:102) ... 22 more Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: BlockAlignedAvroParquetWriter does not support scheme maprfs at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:146) ... 23 more Caused by: java.lang.IllegalArgumentException: BlockAlignedAvroParquetWriter does not support scheme maprfs at com.uber.hoodie.common.io.storage.HoodieWrapperFileSystem.getHoodieScheme(HoodieWrapperFileSystem.java:110) at com.uber.hoodie.common.io.storage.HoodieWrapperFileSystem.convertToHoodiePath(HoodieWrapperFileSystem.java:86) at com.uber.hoodie.io.storage.HoodieParquetWriter.<init>(HoodieParquetWriter.java:57) at com.uber.hoodie.io.storage.HoodieStorageWriterFactory.newParquetStorageWriter(HoodieStorageWriterFactory.java:56) at com.uber.hoodie.io.storage.HoodieStorageWriterFactory.getStorageWriter(HoodieStorageWriterFactory.java:39) at com.uber.hoodie.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:70) at com.uber.hoodie.func.CopyOnWriteLazyInsertIterable$CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteLazyInsertIterable.java:142) at com.uber.hoodie.func.CopyOnWriteLazyInsertIterable$CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteLazyInsertIterable.java:130) at com.uber.hoodie.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:38) at com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:126) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 more Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1517) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1505) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1504) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1504) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:814) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:814) at scala.Option.foreach(Option.scala:257) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:814) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1732) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1687) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1676) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:630) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2029) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2126) at org.apache.spark.rdd.RDD$$anonfun$fold$1.apply(RDD.scala:1089) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:362) at org.apache.spark.rdd.RDD.fold(RDD.scala:1083) at org.apache.spark.rdd.DoubleRDDFunctions$$anonfun$sum$1.apply$mcD$sp(DoubleRDDFunctions.scala:35) at org.apache.spark.rdd.DoubleRDDFunctions$$anonfun$sum$1.apply(DoubleRDDFunctions.scala:35) at org.apache.spark.rdd.DoubleRDDFunctions$$anonfun$sum$1.apply(DoubleRDDFunctions.scala:35) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:362) at org.apache.spark.rdd.DoubleRDDFunctions.sum(DoubleRDDFunctions.scala:34) at org.apache.spark.api.java.JavaDoubleRDD.sum(JavaDoubleRDD.scala:165) at com.uber.hoodie.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:279) at com.uber.hoodie.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:469) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:780) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.RuntimeException: com.uber.hoodie.exception.HoodieException: com.uber.hoodie.exception.HoodieException: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: BlockAlignedAvroParquetWriter does not support scheme maprfs at com.uber.hoodie.func.LazyIterableIterator.next(LazyIterableIterator.java:123) at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43) at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) at org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:371) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1055) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1029) at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:969) at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1029) at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:760) at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334) at org.apache.spark.rdd.RDD.iterator(RDD.scala:285) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:108) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: com.uber.hoodie.exception.HoodieException: com.uber.hoodie.exception.HoodieException: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: BlockAlignedAvroParquetWriter does not support scheme maprfs at com.uber.hoodie.func.CopyOnWriteLazyInsertIterable.computeNext(CopyOnWriteLazyInsertIterable.java:106) at com.uber.hoodie.func.CopyOnWriteLazyInsertIterable.computeNext(CopyOnWriteLazyInsertIterable.java:43) at com.uber.hoodie.func.LazyIterableIterator.next(LazyIterableIterator.java:121) ... 20 more Caused by: com.uber.hoodie.exception.HoodieException: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: BlockAlignedAvroParquetWriter does not support scheme maprfs at com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:148) at com.uber.hoodie.func.CopyOnWriteLazyInsertIterable.computeNext(CopyOnWriteLazyInsertIterable.java:102) ... 22 more Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: BlockAlignedAvroParquetWriter does not support scheme maprfs at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:146) ... 23 more Caused by: java.lang.IllegalArgumentException: BlockAlignedAvroParquetWriter does not support scheme maprfs at com.uber.hoodie.common.io.storage.HoodieWrapperFileSystem.getHoodieScheme(HoodieWrapperFileSystem.java:110) at com.uber.hoodie.common.io.storage.HoodieWrapperFileSystem.convertToHoodiePath(HoodieWrapperFileSystem.java:86) at com.uber.hoodie.io.storage.HoodieParquetWriter.<init>(HoodieParquetWriter.java:57) at com.uber.hoodie.io.storage.HoodieStorageWriterFactory.newParquetStorageWriter(HoodieStorageWriterFactory.java:56) at com.uber.hoodie.io.storage.HoodieStorageWriterFactory.getStorageWriter(HoodieStorageWriterFactory.java:39) at com.uber.hoodie.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:70) at com.uber.hoodie.func.CopyOnWriteLazyInsertIterable$CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteLazyInsertIterable.java:142) at com.uber.hoodie.func.CopyOnWriteLazyInsertIterable$CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteLazyInsertIterable.java:130) at com.uber.hoodie.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:38) at com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:126) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 more
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
