GitHub user GraceH opened a pull request:
https://github.com/apache/spark/pull/9796
Return "false" is nothing to kill in killExecutors
In discussion (SPARK-9552), we proposed a force kill in `killExecutors`.
But if there is nothing to kill, it will return back with true
(acknowledgement). And then, it causes the certain executor(s) (which is not
eligible to kill) adding to pendingToRemove list for further actions.
In this patch, we'd like to change the return semantics. If there is
nothing to kill, we will return "false". and therefore all those non-eligible
executors won't be added to the pendingToRemove list.
@vanzin @andrewor14 As the follow up of PR#7888, please let me know your
comments.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/GraceH/spark emptyPendingToRemove
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9796.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9796
----
commit c23f887b62a75415bab74036e78d03b92b1a5541
Author: Grace <[email protected]>
Date: 2015-10-28T15:04:37Z
rebase to master branch
commit dc660f63c416c300bd3da48c6a0b9442633313fc
Author: Grace <[email protected]>
Date: 2015-11-10T07:10:09Z
change the task number to count; change the WithLoads to WithTaskCount
commit 27faa6b70e8332f6f70359739cb417be5be7e31e
Author: Grace <[email protected]>
Date: 2015-11-10T07:14:39Z
keep sparkcontext public API un-changed
commit 8774124e07666d337501d0e79bc27bbb35d78b74
Author: Grace <[email protected]>
Date: 2015-11-10T12:50:46Z
fix compile issue
commit 946ed7e0966b10a706372200551345304d2c8089
Author: Grace <[email protected]>
Date: 2015-11-10T13:04:08Z
fix compile issue
commit feefbfef79e396df0187ae947e1b39c80e810082
Author: Grace <[email protected]>
Date: 2015-11-11T07:10:41Z
keep public API
commit fa3c88ea1033b569b853e7a758587bf84323bd6c
Author: Grace <[email protected]>
Date: 2015-11-11T07:12:32Z
keep public API
commit cb78e5605679bad667e5a93ed24a1d280827d121
Author: Grace <[email protected]>
Date: 2015-11-11T07:14:05Z
remove unnecessary comments
commit 5bcfd8148af20c43a993dfaa5a90597070a7c343
Author: Grace <[email protected]>
Date: 2015-11-11T07:15:35Z
clean code
commit 01c236ad3cb435c8b63f8be59c3f5d099b797cf3
Author: Grace <[email protected]>
Date: 2015-11-11T07:16:24Z
clean code
commit 2108dbfdfa962b40ffabf8874a364e8bf1009f93
Author: Grace <[email protected]>
Date: 2015-11-12T13:53:54Z
refine code
commit c0a1d549e84b110e13e06810a9b64da1f5b3230d
Author: Grace <[email protected]>
Date: 2015-11-12T14:00:12Z
remove unnecessary imports
commit 4b1959f6c7d393f34e145aed5eca3ae28e7b7a83
Author: Grace <[email protected]>
Date: 2015-11-12T14:05:24Z
set sc.killExecutor as force = true
commit 0293d8241df79b9c60ebefb6e18c89456be30547
Author: Grace <[email protected]>
Date: 2015-11-12T14:20:44Z
use force = false to do the unittest
commit 342a59d34e76c55f2fc306b45d8510c74cf73af8
Author: Grace <[email protected]>
Date: 2015-11-13T03:32:06Z
refine the unit test & change semantics for force == true only
commit 806a64d8e360e4589f4ad1569cc8c6f379e5987b
Author: Grace <[email protected]>
Date: 2015-11-13T03:43:49Z
refine the unit test & change semantics for force == true only
commit 4ce0ec06d79d2d6a5c68d0415c25c41d19f34736
Author: Grace <[email protected]>
Date: 2015-11-13T03:51:48Z
remove unnecessary imports
commit 0000551cd2860a85a6895b8c0a3b1815a9a7f108
Author: Grace <[email protected]>
Date: 2015-11-13T04:55:42Z
fix checkstyle issue
commit c44ef8714c6cde70404043e065e81330746f7881
Author: Andrew Or <[email protected]>
Date: 2015-11-13T18:51:48Z
Suggestions
commit d3f51dbfa4d0b9e416c56a69383d19da2699d478
Author: Andrew Or <[email protected]>
Date: 2015-11-13T23:02:51Z
Clean up state in ExecutorAllocationManager
commit 0daeb5a70cea3aa3d0a34189fcaee6fef59d578e
Author: Jie Huang <[email protected]>
Date: 2015-11-17T00:55:20Z
Merge pull request #1 from andrewor14/pr-7888-suggestions
Suggestions for PR 7888
commit 1938e6158838bb1f21884c3d3eba9f0b8ffabe05
Author: Grace <[email protected]>
Date: 2015-11-17T08:29:05Z
fix unittest issue
commit 589083b8d9208cb1044556d0571f4dfd71284263
Author: Grace <[email protected]>
Date: 2015-11-18T07:05:00Z
return false is nothing to kill in killExecutors
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
