Github user NiharS commented on the issue:
https://github.com/apache/spark/pull/22192
You're right, I ran in local-cluster and it exited very quickly citing
executors shutting down after not being able to find my test plugin. Although,
the logs say that it does use a CoarseGrainedExecutorBackend:
`18/09/05 12:03:20 INFO CoarseGrainedExecutorBackend: Connecting to driver:
spark://CoarseGrainedScheduler@nihar-xxx:45767`
unless you mean Yarn only uses that one command line option. I'm looking
into how it reacts in regular standalone mode with different thread counts.
Thanks for pointing this out!
I'm looking up other spark-submit options that can be used to provide the
jar to other nodes, but I'm not super hopeful it's going to work out. If it
indeed doesn't, I'll start exploring other options once I figure out why the
pyspark tests are failing
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
