Github user mccheah commented on the issue: https://github.com/apache/spark/pull/20669 @vanzin I have a question regarding how this will interact with files that need to exist before the JVM starts. When we used the init-container approach, we were specifically fetching the user's files before the user's JVM launches. This meant that all the user's dependencies were present at JVM boot time. Now, however, we are having spark-submit do the localization. But the spark-submit JVM is the same JVM that runs the user's code here. Let's take a concrete hypothetical example where the user would like to load a YourKit agent binary into the driver container for debugging. The user may not want to build an entirely separate docker image for this, or perhaps they're porting over a debugging workflow from YARN mode where they used `--files` to distribute this binary that they hosted in some remote location. In YARN mode, adding the yourkit agent to `spark.files` works because SparkSubmit distributes the files via `spark.yarn.dist.files` and using HDFS-backed localization. The files are localized before the driver JVM starts, and the yourkit agent is loaded correctly. However, in Kubernetes mode, without an init-container, the yourkit agent binary will always be localized after the driver JVM starts, which is too late. In other words, without an init-container, it's impossible for applications to depend on localizing files that must be present _before_ the JVM even launches. This is mitigated by docker being essentially a built-in localization mechanism in and of itself, and the fact that one can use secrets/config maps as an alternate mounting mechanism. But I'm curious as to our thoughts for those porting over YARN applications with this specific use case. I'm also curious as to how Mesos handles this, since they use a similar scheme too.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org