salimTutuncu opened a new issue, #941: URL: https://github.com/apache/incubator-xtable/issues/941
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/incubator-xtable/issues?q=is%3Aissue) and found no similar issues. ### Please describe the bug 🐞 The published `xtable-spark-runtime_2.12:0.4.0-incubating` artifact fails against every Hudi version in the 0.15+/1.x generation with a binary linkage error when reading a Hudi source: ```text java.lang.NoSuchMethodError: 'org.apache.hudi.common.table.HoodieTableMetaClient$Builder org.apache.hudi.common.table.HoodieTableMetaClient$Builder.setConf(org.apache.hadoop.conf.Configuration)' ``` The Spark Runtime documentation states the thin bundle "reuses the Hudi, Iceberg and Delta libraries your cluster already has" (https://xtable.apache.org/docs/how-to-spark-runtime). For Hudi this only holds for 0.14.x: per vendor documentation, current managed Spark runtimes bundle newer generations (e.g. AWS Glue 5.1 lists Hudi 1.0.2, Amazon EMR 7.9 lists Hudi 0.15.0-amzn-6), so on those platforms the Hudi source path of the published artifact is unusable as-is. Measured results with upstream OSS bundles: Hudi 0.14.1 works; 0.15.0 and 1.0.2 fail with the error above. ### Why this is not a duplicate of #762 #762 / PR #772 upgraded `main` to Hudi 1.x (current `main` pom pins 1.2.0). However, the spark-runtime module does not exist on `main`; it lives on `branch-0.4`, which was deliberately kept on the Hudi 0.x line for the 0.4.0 release (#843). So the merged upgrade fixes future releases cut from `main`, but no released or currently releasable spark-runtime artifact benefits from it. This issue is about the released 0.4.x line and its documented compatibility contract. (Context also posted as a comment on #921.) ### Root cause The 0.4.0 bundle is compiled against Hudi 0.14.0 (https://github.com/apache/incubator-xtable/blob/e8f22867a6cb297f58b248150220fce71188d16a/pom.xml#L73). Its bytecode calls `HoodieTableMetaClient$Builder.setConf(org.apache.hadoop.conf.Configuration)` (https://github.com/apache/incubator-xtable/blob/e8f22867a6cb297f58b248150220fce71188d16a/xtable-core/src/main/java/org/apache/xtable/hudi/HudiConversionSourceProvider.java#L34-L50). Hudi 0.15.0 removed that overload in favor of `setConf(org.apache.hudi.storage.StorageConfiguration<?>)`; confirmed via `javap` against the 0.14.1, 0.15.0, and 1.0.2 bundles. Because the dependency is `provided`, the incompatibility surfaces only at runtime, on the user's cluster. A secondary consequence for inline use on Spark 3.5: writing Hudi requires the 0.15+ bundle (the 0.14.x spark3.4 bundle cannot run on Spark 3.5), while syncing requires 0.14.x, so no single classpath supports the documented write-then-sync pattern for Hudi at all with the published artifact. ### To Reproduce Environment: JDK 17, `pyspark==3.5.5`; jars from Maven Central: xtable-spark-runtime 0.4.0-incubating, iceberg-spark-runtime-3.5 1.9.2, hudi-spark3.5-bundle 0.15.0 (and optionally hudi-spark3.4-bundle 0.14.1 for the contrast run). Generator script included below. ```bash # 1. write a partitioned Hudi COW table with Hudi 0.15.0 (the generation current runtimes ship) spark-submit --jars hudi-spark3.5-bundle_2.12-0.15.0.jar \ --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \ gen_hudi.py /tmp/orders_hudi # 2. sync with the same Hudi on the classpath spark-submit --master 'local[2]' --class org.apache.xtable.spark.XTableSparkSync \ --jars hudi-spark3.5-bundle_2.12-0.15.0.jar,iceberg-spark-runtime-3.5_2.12-1.9.2.jar \ xtable-spark-runtime_2.12-0.4.0-incubating.jar \ --basepath /tmp/orders_hudi --sourceformat HUDI --targets ICEBERG \ --partitionspec order_date:VALUE ``` Observed: the `NoSuchMethodError` above, nonzero exit. Same failure with hudi-spark3.5-bundle 1.0.2. Contrast: replacing the Hudi jar with `hudi-spark3.4-bundle_2.12-0.14.1.jar` makes the same sync succeed with a fully correct target (rows, partitions, stats verified). ### Expected behavior One of, in order of preference: (a) a 0.4.x patch release of the spark-runtime compatible with the Hudi 0.15+/1.x generation, or built with the `main` Hudi upgrade; (b) failing that, the documented compatibility contract for 0.4.0 (Hudi 0.14.x only) stated on the Spark Runtime page, plus a fail-fast version check that reports "unsupported Hudi version X, 0.4.0 requires 0.14.x" instead of a raw `NoSuchMethodError`. ### Affected versions `0.4.0-incubating` published artifact; `branch-0.4` HEAD carries the same 0.14 pin. Not applicable to `main` (module absent there; `main` pom is on Hudi 1.2.0 as of 2026-09-04). #### Reproduction script gen_hudi.py ```python import sys from pyspark.sql import SparkSession, functions as F path = sys.argv[1] spark = SparkSession.builder.getOrCreate() df = (spark.range(0, 1000).withColumn("order_id", F.col("id")) .withColumn("ts", F.lit(0).cast("long")) .withColumn("order_date", F.lit("2026-08-01")).drop("id").repartition(2)) df.write.format("hudi").options(**{ "hoodie.table.name": "orders_hudi", "hoodie.datasource.write.table.type": "COPY_ON_WRITE", "hoodie.datasource.write.recordkey.field": "order_id", "hoodie.datasource.write.precombine.field": "ts", "hoodie.datasource.write.partitionpath.field": "order_date", "hoodie.datasource.write.hive_style_partitioning": "true", }).mode("overwrite").save(path) print("wrote 1000 rows ->", path) ``` ### Are you willing to submit PR? - [ ] I am willing to submit a PR! - [ ] I am willing to submit a PR but need help getting started! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
