I am having problems submitting an app through spark-submit when the master
is not "local". However the pi.py example which comes with Spark works with
any master. I believe my script has the same structure as pi.py, but for
some reason my script is not as flexible. Specifically, the failure occurs
when count() is called. Count is the first action in the script. Also,
Spark complains that is is losing executors however, interactively in
Jupyter, everything works perfectly with any master passed to spark conf.
Does anyone know what might be happening? Is there anywhere I can look up
the requirements for spark-submit scripts?