rangareddy opened a new issue, #19297:
URL: https://github.com/apache/hudi/issues/19297
### Bug Description
**What happened:**
When using a locally built hudi-spark3.4-bundle (version 1.3.0-SNAPSHOT)
built with the -Dspark3.4 profile, executing a DataFrame write with
.format("hudi") fails immediately. The driver throws a
java.util.ServiceConfigurationError stating that the provider class
org.apache.hudi.Spark32PlusDefaultSource cannot be found on the classpath,
despite the SPI configuration tracking it.
**What you expected:**
The write operation should succeed, resolving
org.apache.hudi.Spark32PlusDefaultSource transparently via the internal
DataSourceRegister mappings packaged inside the bundle.
**Steps to reproduce:**
1. Build the project: Clone the source repository and compile using Spark
3.4 and Flink 1.20 profiles (`mvn -T 2C install -DskipTests -Dspark3.4
-Dflink1.20`)
2. Launch PySpark shell: Point to the locally generated bundle jar:
`pyspark \
--jars
packaging/hudi-spark-bundle/target/hudi-spark3.4-bundle_2.12-1.3.0-SNAPSHOT.jar
\
--conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' \
--conf
'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog'
\
--conf
'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension' \
--conf 'spark.kryo.registrator=org.apache.spark.HoodieSparkKryoRegistrar'`
3. Execute a DataFrame write operation:
```python
from pyspark.sql.types import StructType, StructField, StringType,
IntegerType
schema = StructType([
StructField("id", StringType(), nullable=False, metadata={"comment":
"Unique identifier"}),
StructField("name", StringType(), nullable=True, metadata={"comment":
"Name of the person"}),
StructField("age", IntegerType(), nullable=True, metadata={"comment":
"Age of the person"}),
StructField("date", StringType(), nullable=True, metadata={"comment":
"Partition field, date of entry"})
])
data = [("1", "John", 30, "2023-01-01"), ("2", "Jane", 25, "2023-01-01"),
("3", "Bob", 35, "2023-01-02")]
df = spark.createDataFrame(data, schema=schema)
hudi_options = {
'hoodie.table.name': 'Hudi_Table_With_Comments_130',
'hoodie.datasource.write.operation': 'insert',
'hoodie.datasource.write.partitionpath.field': 'date',
'hoodie.datasource.write.recordkey.field': 'id',
'hoodie.datasource.write.precombine.field': 'id',
'hoodie.schema.on.read.enable': 'true'
}
df.write.format("hudi").options(**hudi_options).mode("overwrite").save("/tmp/test/Hudi_Table_With_Comments_130")
```
### Environment
**Hudi version:** 1.3.0 (master)
**Query engine:** (Spark/Flink/Trino etc) Spark 3.4
**Relevant configs:** None
### Logs and Stack Trace
```python
>>>
df.write.format("hudi").options(**hudi_options).mode("overwrite").save(tablePath)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File
"/Users/rangareddy/ranga_work/apache/spark/spark-3.4.1/python/pyspark/sql/readwriter.py",
line 1398, in save
self._jwrite.save(path)
File
"/Users/rangareddy/ranga_work/apache/spark/spark-3.4.1/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py",
line 1322, in __call__
File
"/Users/rangareddy/ranga_work/apache/spark/spark-3.4.1/python/pyspark/errors/exceptions/captured.py",
line 169, in deco
return f(*a, **kw)
File
"/Users/rangareddy/ranga_work/apache/spark/spark-3.4.1/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py",
line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o79.save.
: java.util.ServiceConfigurationError:
org.apache.spark.sql.sources.DataSourceRegister: Provider
org.apache.hudi.Spark32PlusDefaultSource not found
at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:593)
at
java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.nextProviderClass(ServiceLoader.java:1219)
at
java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1228)
at
java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1273)
at
java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1309)
at
java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1393)
at
scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at
scala.collection.TraversableLike.filterImpl(TraversableLike.scala:303)
at
scala.collection.TraversableLike.filterImpl$(TraversableLike.scala:297)
at
scala.collection.AbstractTraversable.filterImpl(Traversable.scala:108)
at scala.collection.TraversableLike.filter(TraversableLike.scala:395)
at
scala.collection.TraversableLike.filter$(TraversableLike.scala:395)
at scala.collection.AbstractTraversable.filter(Traversable.scala:108)
at
org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:629)
at
org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSourceV2(DataSource.scala:697)
at
org.apache.spark.sql.DataFrameWriter.lookupV2Provider(DataFrameWriter.scala:860)
at
org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:256)
at
org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
