[GitHub] [hudi] lyogev opened a new issue #1751: [SUPPORT] Hudi not working with Spark 3.0.0

[GitBox]

lyogev opened a new issue #1751:
URL: https://github.com/apache/hudi/issues/1751


   **Describe the problem you faced**
   
   Trying to run hudi with spark 3.0.0, and getting an error
   
   **To Reproduce**
   
   Steps to reproduce the behavior:
   
   **Expected behavior**
   
   **Environment Description**
   
   * Hudi version :
   
   
   0.5.3
   
   * Spark version :
   
   3.0.0
   
   * Hive version :
   
   2.3.7
   
   * Hadoop version :
   
   3.2.0
   
   * Storage (HDFS/S3/GCS..) :
   
   * Running on Docker? (yes/no) :
   
   yes
   
   **Additional context**
   
   Add any other context about the problem here.
   
   **Stacktrace**
   
   ```
   Caused by: java.lang.NoSuchMethodError: 'java.lang.Object 
org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.fromRow(org.apache.spark.sql.catalyst.InternalRow)'
   at 
org.apache.hudi.AvroConversionUtils$.$anonfun$createRdd$1(AvroConversionUtils.scala:42)
   at scala.collection.Iterator$$anon$10.next(Iterator.scala:459)
   at scala.collection.Iterator$$anon$10.next(Iterator.scala:459)
   at scala.collection.Iterator$$anon$10.next(Iterator.scala:459)
   at scala.collection.Iterator$SliceIterator.next(Iterator.scala:271)
   at scala.collection.Iterator.foreach(Iterator.scala:941)
   at scala.collection.Iterator.foreach$(Iterator.scala:941)
   at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
   at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
   at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
   at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
   at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
   at scala.collection.TraversableOnce.to(TraversableOnce.scala:315)
   at scala.collection.TraversableOnce.to$(TraversableOnce.scala:313)
   at scala.collection.AbstractIterator.to(Iterator.scala:1429)
   at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:307)
   at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:307)
   at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1429)
   at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:294)
   at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:288)
   at scala.collection.AbstractIterator.toArray(Iterator.scala:1429)
   at org.apache.spark.rdd.RDD.$anonfun$take$2(RDD.scala:1423)
   at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2133)
   at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
   at org.apache.spark.scheduler.Task.run(Task.scala:127)
   at 
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
   at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown 
Source)
   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown 
Source)
   at java.base/java.lang.Thread.run(Unknown Source)```
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org