voonhous opened a new issue, #19234: URL: https://github.com/apache/hudi/issues/19234
**Describe the problem you faced** On a Spark 4.1 runtime, any Hudi read-modify write — `UPDATE`, `MERGE INTO`, `DELETE` (the upsert path that reads existing base files) — aborts with: ``` java.lang.NoClassDefFoundError: org/apache/parquet/variant/VariantConverters$ParentConverter Caused by: java.lang.ClassNotFoundException: org.apache.parquet.variant.VariantConverters$ParentConverter ``` A fresh `INSERT` / bulk write succeeds; only the read-modify operations fail. This was surfaced by the custom-type hive-sync E2E (`ITTestCustomTypeHiveSync`, added for the Spark 4.1 matrix in #19216). Because the failed writes never commit, the new partition the `MERGE` should have created is missing, so the test trips on a downstream Hive `SHOW PARTITIONS` assertion — but the real failure is the `NoClassDefFoundError` above (the fixtures' unconditional `*_SUCCESS` prints mask it). **Root cause** `org.apache.parquet.variant.VariantConverters` is a Parquet **1.15+** class that Spark 4.1's Parquet read path requires. The Hudi Spark bundle ships **Parquet 1.13.1** (root `pom.xml` `<parquet.version>1.13.1</parquet.version>`); the built `hoodie-spark-bundle` contains **zero** `org/apache/parquet/variant/*` classes. At runtime the read-modify path resolves the bundled 1.13.1 Parquet, which lacks `VariantConverters`, and fails. **To Reproduce** 1. Spark 4.1 runtime with the Hudi Spark 4.1 bundle on the classpath. 2. Create a partitioned COW table and `INSERT` rows — succeeds. 3. Run any `UPDATE` / `MERGE INTO` / `DELETE` — aborts with the `NoClassDefFoundError`. **Expected behavior** UPDATE/MERGE/DELETE work on Spark 4.1, as they do on Spark 3.5 / 4.0. **Environment Description** - Hudi version: 1.3.0-SNAPSHOT (master) - Spark version: 4.1.1 - Parquet: bundle ships 1.13.1; Spark 4.1 requires 1.15+ - Table type: COPY_ON_WRITE - Running on Docker?: yes (integ E2E) **Stacktrace (Spark 4.1 E2E, `ITTestCustomTypeHiveSync`)** ``` ERROR BaseSparkCommitActionExecutor: Error upserting bucketType UPDATE for partition :0 java.lang.NoClassDefFoundError: org/apache/parquet/variant/VariantConverters$ParentConverter Caused by: java.lang.ClassNotFoundException: org.apache.parquet.variant.VariantConverters$ParentConverter ... ERROR TaskSetManager: Task 0 in stage 41.0 failed 1 times; aborting job org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 41.0 failed 1 times, most recent failure: Lost task 0.0 in stage 41.0 (TID 54) (adhoc-1 executor driver): org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType UPDATE for partition :0 ... org.apache.hudi.exception.HoodieUpsertException: Failed to upsert for commit time 20260709102914322 ``` Downstream symptom — hive-sync then correctly finds nothing new (the write never committed): ``` INFO HiveSyncTool: Last commit time synced was found to be 20260709102854659, last commit completion time is found to be 20260709102908002 INFO HoodieSyncClient: Last commit time synced is 20260709102854659, Getting commits since then INFO HiveSyncTool: Storage partitions scan complete. Found 0 INFO HiveSyncTool: Sync complete for vector_test ``` **Proposed fix** Use a Spark-4.1-compatible Parquet (1.15.x, which includes `parquet-variant` / `VariantConverters`) for the Spark 4.1 profile/bundle, or stop shading an older Parquet that shadows Spark 4.1's. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
