RE: Official Stance on Not Using Spark Submit

:14 PM To: Mendelson, Assaf Subject: Re: Official Stance on Not Using Spark Submit Just folks who don't want to use spark-submit, no real use-cases I've seen yet. I didn't know about SparkLauncher myself and I don't think there are any official docs on that or launching spark as an embedded

Re: Official Stance on Not Using Spark Submit

Funny, someone from my team talked to me about that idea yesterday. We use SparkLauncher, but it just calls spark-submit that calls other scripts that starts a new Java program that tries to submit (in our case in cluster mode - driver is started in the Spark cluster) and exit. That make it a

Re: Official Stance on Not Using Spark Submit

I actually had not seen SparkLauncher before, that looks pretty great :) On Mon, Oct 10, 2016 at 10:17 AM Russell Spitzer wrote: > I'm definitely only talking about non-embedded uses here as I also use > embedded Spark (cassandra, and kafka) to run tests. This is

Re: Official Stance on Not Using Spark Submit

I'm definitely only talking about non-embedded uses here as I also use embedded Spark (cassandra, and kafka) to run tests. This is almost always safe since everything is in the same JVM. It's only once we get to launching against a real distributed env do we end up with issues. Since Pyspark uses

Re: Official Stance on Not Using Spark Submit

I have also 'embedded' a Spark driver without much trouble. It isn't that it can't work. The Launcher API is ptobably the recommended way to do that though. spark-submit is the way to go for non programmatic access. If you're not doing one of those things and it is not working, yeah I think

Re: Official Stance on Not Using Spark Submit

I've done this for some pyspark stuff. I didn't find it especially problematic. On Mon, Oct 10, 2016 at 12:58 PM, Reynold Xin wrote: > How are they using it? Calling some main function directly? > > > On Monday, October 10, 2016, Russell Spitzer