Github user nishkamravi2 commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-84109293
Are we never loading the uber jar in this implementation? Can you quantify
the gains you think you get by loading a separate launcher jar as compared to
the uber jar? Are we convinced that those are significant enough to take away
the simplicity of a single uber jar and potentially breaking backward
compatibility? An uber jar allows a customer to copy it anywhere and point
their classpath to it.
With regards to the assembly jar name issue: I'm not sure the above example
is a breakage if it can be fixed by simply deleting one of those two jar files
once the Spark launcher prints an explicit message stating so. In general, I'm
not convinced that a strict assembly name check should be a solution for
anything. But, if you are, a good compromise would be to do that check in
spark-submit (or an outer shell script) than hard-coding it in the Spark code.
Sean, I think from a customer's point of view, the launcher is the
application and the launchee is Spark library. That simplicity is desired, as
compared to separating out stub code within Spark into a different library (for
very little in return) and increasing complexity?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
