vanzin commented on issue #23793: [SPARK-24736][k8s] Let spark-submit handle dependency resolution. URL: https://github.com/apache/spark/pull/23793#issuecomment-465260161 > if we look at the YARN cluster manager we also see similar logic around setting the PYTHONPATH YARN is not a good example to follow. It builds its own custom commands instead of using spark-submit / spark-class like the others. e.g. the k8s driver is run using spark-submit inside the cluster, and YARN's driver in cluster mode is not run that way. So the YARN code needs to handle a lot of things that are otherwise handled by spark-submit. YARN has also special handling for its distributed cache functionality, which others don't have. The goal here is basically to avoid making in the k8s backend the same mistakes YARN made. > Have you tested this with a zipfile or egg as a dependency I'll give it a shot. But like the `local:` dependency issue in executors, if that's not working, it's probably some issue in how the shared code deals with those, and I'd consider it a separate issue. (Which doesn't mean it shouldn't be fixed.)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
