remeajayi2022 commented on issue #12301:
URL: https://github.com/apache/hudi/issues/12301#issuecomment-2529438153
Does your deployment utilize Dataproc? I can't really modify the spark
runtime since it's a managed service. The highest supported Spark version on
Dataproc is v3.5.1 which uses v3.23.4 of Protobuf. I've tested the job with
this Spark version , and as expected from your explanation there are still
Protobuf compatibility issues.
I did try to force the job to use hudi's protobuf version with this:
`--properties
spark.driver.userClassPathFirst=true,spark.executor.userClassPathFirst=true \
--jars gs://dp-data-eng-dev-storage/jars/protobuf-java-3.25.5.jar,`
But this resulted in a different `ClassNotFoundException`
Do you mind providing more details about the setup that worked for you so I
can replicate that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
