[ https://issues.apache.org/jira/browse/SPARK-13216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15347067#comment-15347067 ]
Neelesh Srinivas Salian commented on SPARK-13216: ------------------------------------------------- [~srowen] are we moving forward with this? Based on [~hshreedharan]'s comment, it could be done. CC'ing [~tdas] here to get more insight. > Spark streaming application not honoring --num-executors in restarting of an > application from a checkpoint > ---------------------------------------------------------------------------------------------------------- > > Key: SPARK-13216 > URL: https://issues.apache.org/jira/browse/SPARK-13216 > Project: Spark > Issue Type: Bug > Components: Spark Submit, Streaming > Affects Versions: 1.5.0 > Reporter: Neelesh Srinivas Salian > Priority: Minor > Labels: Streaming > > Scenario to help understand: > 1) The Spark streaming job with 12 executors was initiated with checkpointing > enabled. > 2) In version 1.3, the user was able to append the number of executors to 20 > using --num-executors but was unable to do so in version 1.5. > In 1.5, the spark application still runs with 13 executors (1 for driver and > 12 executors). > There is a need to start from the checkpoint itself and not restart the > application to avoid the loss of information. > 3) Checked the code in 1.3 and 1.5, which shows the command > ''--num-executors" has been deprecated. > Any thoughts on this? Not sure if anyone hit this one specifically before. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org