rmannibucau commented on PR #53351: URL: https://github.com/apache/spark/pull/53351#issuecomment-3677934639
@dongjoon-hyun > It doesn't make sense to me because Apache Spark Standalone Cluster has the full distribution already to launch Spark Master and Spark Worker JVM. What you need is to deploy only your application, not a custom Apache Spark again. This is exactly the point to have a custom distribution, kind of include the application inside it. Why do you have spark shell in spark distribution? It is an application so must not be there from your statement, this is exactly the same. Ultimately I do not want to have to rely on the internet or a random network config (think enterprise) to download the application so I want to bundle it upfront. > FYI, Apache Spark provides official Apache K8s Operator This is what i'm using for most applicative use cases but I need the custom distro case for human interaction (spark sql) - once again just to make it simple. `--packages` and all friends relying on ivy are not stable enough and too poorly configurable (excludes are a pain for ex) and having to download spark and then the libraries is pointless. Look at apache iceberg case, you just need the jars to use it with Apache Spark SQL Shell, why can't we make a distribution with the application prebundled to make it easier. Will also enable to customize some library version (parquet) which are conflicting and enforces you to use userClassPathFirst which has other side effects. So yes, this is needed to cover all usages even if most automated usages are, as you say, done through other ways. Side note: just out of curiosity, why do you fight that much for something trivial to do? Is there a blocker in the release to deploy the zip/targz? We do it in plenty of apache projects and every is happy. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
