Tyler-Rendina commented on issue #10590: URL: https://github.com/apache/hudi/issues/10590#issuecomment-2163683218
EMR on EKS gave me issues and I switched to EMR on EC2 about a year ago, probably needed to do the same thing done here. Planning to use something like kubeflow in the future. When you spin up a job and look at the environment vars in the spark UI you can ctrl + f for 'classpath' and see what the key value options are. If the location of the jar is not on the classpath then it must be specified in addition to the initial classpath specified in that var of the spark UI. In this case I was using --packages to get them directly from Maven, then I tried --jar and ran into version issues, hence the custom build above. This led to using the custom jars from s3. I imagine writing a more robust bootstrap script may have mitigated the issue. I also wanted to make sure the jar selected for hudi was not the preinstalled version in the particular case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
