Neelesh Srinivas Salian created SPARK-13216:
-----------------------------------------------
Summary: Spark streaming application not honoring --num-executors
in restarting of an application from a checkpoint
Key: SPARK-13216
URL: https://issues.apache.org/jira/browse/SPARK-13216
Project: Spark
Issue Type: Bug
Components: Spark Submit, Streaming
Affects Versions: 1.5.0
Reporter: Neelesh Srinivas Salian
Scenario to help understand:
1) The Spark streaming job with 12 executors was initiated with checkpointing
enabled.
2) In version 1.3, the user was able to append the number of executors to 20
using --num-executors but was unable to do so in version 1.5.
In 1.5, the spark application still runs with 13 executors (1 for driver and 12
executors).
There is a need to start from the checkpoint itself and not restart the
application to avoid the loss of information.
3) Checked the code in 1.3 and 1.5, which shows the command ''--num-executors"
has been deprecated.
Any thoughts on this? Not sure if anyone hit this one specifically before.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
