[
https://issues.apache.org/jira/browse/HUDI-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sagar Sumit closed HUDI-3857.
-----------------------------
Resolution: Cannot Reproduce
This happens with EMR spark. It does not reproduce with OSS spark. Either use
OSS spark or replace the emr specific spark-sql-amzn*.jar from with OSS
spark-sql jar :
https://repo1.maven.org/maven2/org/apache/spark/spark-sql_2.12/3.1.2/spark-sql_2.12-3.1.2.jar
> NoSuchMethodError: Continuous deltastreamer test with async compaction fails
> on EMR spark
> ------------------------------------------------------------------------------------------
>
> Key: HUDI-3857
> URL: https://issues.apache.org/jira/browse/HUDI-3857
> Project: Apache Hudi
> Issue Type: Test
> Reporter: Sagar Sumit
> Assignee: Sagar Sumit
> Priority: Blocker
> Fix For: 0.11.0
>
>
> EMR 6.5, Spark 3.1.2
> While running continuous deltastreamer with async compaction enabled, I hit
> this exception
> {code:java}
> Caused by: java.lang.NoSuchMethodError:
> org.apache.spark.sql.execution.datasources.PartitionedFile.<init>(Lorg/apache/spark/sql/catalyst/InternalRow;Ljava/lang/String;JJ[Ljava/lang/String;)V
> at
> org.apache.hudi.MergeOnReadSnapshotRelation.$anonfun$buildSplits$2(MergeOnReadSnapshotRelation.scala:130)
> at scala.Option.map(Option.scala:230)
> at
> org.apache.hudi.MergeOnReadSnapshotRelation.$anonfun$buildSplits$1(MergeOnReadSnapshotRelation.scala:128)
> at
> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at scala.collection.TraversableLike.map(TraversableLike.scala:238)
> at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
> at scala.collection.immutable.List.map(List.scala:298)
> at
> org.apache.hudi.MergeOnReadSnapshotRelation.buildSplits(MergeOnReadSnapshotRelation.scala:124)
> at
> org.apache.hudi.MergeOnReadSnapshotRelation.collectFileSplits(MergeOnReadSnapshotRelation.scala:108)
> at
> org.apache.hudi.MergeOnReadSnapshotRelation.collectFileSplits(MergeOnReadSnapshotRelation.scala:44)
> at
> org.apache.hudi.HoodieBaseRelation.buildScan(HoodieBaseRelation.scala:221)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
