Lakshmi-Holla12 commented on issue #10376:
URL: https://github.com/apache/hudi/issues/10376#issuecomment-1875098138
Hi @ad1happy2go, I'm still experiencing the issue with EMR 6.15. By default,
Spark jar is located at /usr/lib/hudi, which I am currently using. During the
spark-submit, I provide the following parameters `--jars
/usr/lib/hudi/hudi-spark3-bundle_2.12-0.14.0-amzn-0.jar` and `--py-files
/usr/lib/hudi/<my src code path>`. However, I'm not sure why Spark is trying to
delete the files. I've tried a few steps to resolve the issue but to no luck:
1. I attempted putting the Hudi Spark jar in a different location and ran
into the same error.
2. I removed the `--jars` parameter during Spark submit, this time it looks
for `--py-files` and throws the same error with not able to delete
/usr/lib/hudi/./<my src code path>.
3. I tried setting this configuration `spark.hadoop.fs.s3a.deletion.enable`,
`false`. This was meant to disable the deletion of temporary files by the S3A
filesystem. However, I'm still facing the same issue.
4. I also tried setting `spark.hadoop.io.native.lib.available`, `false`.
This was to disable the use of native Unix OS commands for deletion. However,
I'm still facing the same issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
