stathismar commented on issue #8311: URL: https://github.com/apache/hudi/issues/8311#issuecomment-1490438916
Hello again. I managed to overcome the issue. I started the SparkSQL console using the official spark image from dockerhub. I.e.: ```bash docker run --env-file aws-env.txt -it apache/spark:v3.3.1 /opt/spark/bin/spark-sql \ --packages org.apache.hudi:hudi-spark3.3-bundle_2.12:0.13.0,org.apache.hadoop:hadoop-aws:3.3.4,com.amazonaws:aws-java-sdk-bundle:1.12.398,org.apache.hudi:hudi-aws-bundle:0.13.0 \ --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' \ --conf 'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog' \ --conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension' \ --conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \ --conf spark.driver.extraJavaOptions="-Divy.cache.dir=/tmp -Divy.home=/tmp" ``` and ran exactly the same commands. **Everything worked as expected**! Locally I have (almost) the same Java/Scala versions as the one present in the [official Spark docker image](https://hub.docker.com/layers/apache/spark/v3.3.1/images/sha256-7b38e8ec4f04c8dbcba38ac74212a92aeff9a55d0068631df6fe65b1397b400d?context=explore) (my system has a newer point release for both Java and Scala, but I do not believe this should be a problem) The other thing I suspected is that in the Docker image we use the `user` with `UUID 185`. I tried to set `export SPARK_USER=185` locally but it also didn't help. Anyway my issue is solved, and I do not think there is any reason to reproduce and spend more time on this since most probably it has to do with some version incompatibility. cc. @yihua -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
