rahil-c commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3662961897
@the-other-tim-brown @yihua For the integration test was failing for
```
Error: testAvroFileSinkWriter Time elapsed: 3.125 s <<< ERROR!
org.apache.spark.SparkException:
Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most
recent failure: Lost task 0.0 in stage 0.0 (TID 0)
(runnervm6qbrg.zfkur4ax2nlu5onhaw0pux0jqe.ex.internal.cloudapp.net executor
driver): java.lang.NoSuchMethodError:
org.apache.spark.sql.avro.AvroSerializer.<init>(Lorg/apache/spark/sql/types/DataType;Lorg/apache/hudi/common/schema/HoodieSchema;Z)V
at
org.apache.spark.sql.avro.HoodieSpark3_5AvroSerializer.<init>(HoodieSpark3_5AvroSerializer.scala:28)
at
org.apache.spark.sql.adapter.Spark3_5Adapter.createAvroSerializer(Spark3_5Adapter.scala:92)
at
org.apache.hudi.AvroConversionUtils$.$anonfun$createInternalRowToAvroConverter$1(AvroConversionUtils.scala:70)
at
java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at
org.apache.hudi.AvroConversionUtils$.createInternalRowToAvroConverter(AvroConversionUtils.scala:82)
at
org.apache.hudi.HoodieSparkUtils$.$anonfun$createRdd$2(HoodieSparkUtils.scala:119)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:858)
```
I think this is due to the fact that Hudi has its own class called
`AvroSerializer` which has the same naming as the spark avro `AvroSerializer`
class and likely runtime was getting confused by this and loading the wrong
class which was not using the constructor with HoodieSchema.
<img width="2038" height="476" alt="Screenshot 2025-12-16 at 3 57 45 PM"
src="https://github.com/user-attachments/assets/78126af3-2ff9-427b-9630-5bd9aafddf93";
/>
Im gonna push a change to see if the rename the class works since it worked
locally
<img width="2433" height="478" alt="Screenshot 2025-12-16 at 3 59 22 PM"
src="https://github.com/user-attachments/assets/642b52f0-6cca-45be-a9c8-b96461446f25";
/>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
