Spark users, I am testing different cluster spinup and batch submission jobs. Using the sequenceiq/spark docker package, I have succeeded in submitting "fat egg" (analogous to "fat jar") style python code remotely over YARN. spark-submit --py-files is able to transmit the packaged code to the cluster and run it.
I had to read some source code to ascertain how to get the py-files feature working with remote submission. I noticed that the code path for py-files over YARN protocol is considerably different from py-files over standalone protocol. Is the py-files behavior the same over standalone submission protocol, compared with YARN submission protocol? Regards, Jeff Henrikson --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
