deepakpanda93 commented on issue #10376: URL: https://github.com/apache/hudi/issues/10376#issuecomment-4855934017
@Lakshmi-Holla12 This isn't a Hudi bug — the error comes entirely from Spark's submit path, not Hudi. The stack is SparkSubmit.prepareSubmitEnvironment → downloadResourcesToCurrentDirectory → Utils.deleteRecursively: Spark 3.4 localizes the --jars/--py-files resources into the working directory and tries to delete the source, and since the source lives under /usr/lib/hudi/ (root-owned), the delete fails with Failed to delete: /usr/lib/hudi/./…jar. This is a Spark 3.4 + EMR behavior, not tied to any Hudi version — which matches your own finding that it works on EMR 6.11 (Spark 3.3.2) and fails on EMR 6.13/6.14/6.15 (all Spark 3.4.1), across Hudi 0.13.1 and 0.14.0. It also reproduces with only --py-files. Nothing in Hudi controls SparkSubmit's resource download/delete. Recommendations: - On EMR, the Hudi bundle in /usr/lib/hudi is already on the Spark classpath — don't re-pass it via --jars. - Put any custom jars and --py-files scripts in a writable, non-root location (e.g. your own S3 path or a user-owned dir), not /usr/lib/hudi. - Since this is EMR-on-EKS + Spark 3.4 runtime behavior on an -amzn build, AWS EMR support is the right channel if the above doesn't resolve it. This is an environment/usage issue rather than a Hudi defect. If you can reproduce a Failed to delete with a stack trace that includes Hudi code (on OSS Hudi + OSS Spark), please share all details. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
