GitHub user squito opened a pull request:
https://github.com/apache/spark/pull/13951
[WIP][SPARK-8425]
just running tests, not ready for review just yet. Still need to do some
cleanup.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/squito/spark blacklist-SPARK-8426
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13951.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13951
----
commit 975a2a3c2b810f6b462eb46813075aac4928c0ae
Author: mwws <[email protected]>
Date: 2015-12-29T06:01:17Z
enhance blacklist mechanism
1. create new BlacklistTracker and BlacklistStrategy interface to
support
complex use case for blacklist mechanism.
2. make Yarn allocator aware of node blacklist information
3. three strategies implemented for convenience, also user can define
his own strategy
SingleTaskStrategy: remain default behavior before this change.
AdvanceSingleTaskStrategy: enhance SingleTaskStrategy by supporting
stage level node blacklist
ExecutorAndNodeStrategy: different taskSet can share blacklist
information.
commit 51d3c88720faffd6a1fb6910b999cdce0d446bcf
Author: mwws <[email protected]>
Date: 2016-01-13T05:43:46Z
change import order to meet new scala style check rule
commit 7e52311bcf4b5528d127d1d0a16bade7c039517e
Author: mwws <[email protected]>
Date: 2016-02-23T05:28:56Z
simplify code and fix typo
1. fix compile error after rebase to latest codebas.
2. simplify configuration.
3. fix typo.
4. enhance comment and unit text.
5. remove unused import.
6. remove ExecutorAndNode strategy.
commit b600604a0920054cf3b33bff047d84cbd302fb3c
Author: Imran Rashid <[email protected]>
Date: 2016-05-10T17:49:05Z
style
commit 45525a118db078f80b3e0e74abe7d7f2e04a7883
Author: Imran Rashid <[email protected]>
Date: 2016-05-10T19:27:39Z
small refactoring
commit a6e94d7dac23431462b576b059de701eac548643
Author: Imran Rashid <[email protected]>
Date: 2016-05-10T20:31:57Z
basic test framework for entire spark scheduler
commit 20fb3e98ea08cd3cb777227d0b08e35ea3408de3
Author: Imran Rashid <[email protected]>
Date: 2016-05-10T21:19:55Z
TaskResultGetter now expects there to always be non-null accum updates
commit 0ca981547832780de4405278fce8314f8be73a84
Author: Imran Rashid <[email protected]>
Date: 2016-05-13T18:13:59Z
switch to making backend run in another thread
commit 421c2a18c1a5799c75884f51c73205a7b92a6166
Author: Imran Rashid <[email protected]>
Date: 2016-05-13T22:13:07Z
remove MultiExecutorBackend for now
commit c0911874783bdfadd7749a66a935cba2669f4ffa
Author: Imran Rashid <[email protected]>
Date: 2016-05-17T15:58:50Z
remove uncertain comment about messageScheduler
commit 3b67b2a950d75bc7f532c7a1151aaee864dc541f
Author: Imran Rashid <[email protected]>
Date: 2016-05-17T18:19:17Z
cleanup
commit 79bc38416a914cebb33beaef3ec3179528848bca
Author: Imran Rashid <[email protected]>
Date: 2016-05-18T14:10:46Z
add BlacklistIntegrationSuite and corresponding refactoring
commit 8349b76ada807a9ce351cebbf4eddb88f67ca138
Author: Imran Rashid <[email protected]>
Date: 2016-05-18T15:37:19Z
cleanup
commit 7050b49b4dfd6e7b4e3e966e9e15e3558a343d67
Author: Imran Rashid <[email protected]>
Date: 2016-05-18T15:51:13Z
comments
commit f400741235a92c1491bfda7c1c7f5118c2915ce7
Author: Imran Rashid <[email protected]>
Date: 2016-05-18T15:52:45Z
Merge branch 'SPARK-10372-scheduler-integs' into blacklist_w_performance
commit 00953764f8d6a3807329ddff028c64a913cba662
Author: Imran Rashid <[email protected]>
Date: 2016-05-18T15:58:04Z
move dummy killTask to MockBackend, otherwise occasional problems even in
SingleCoreMockBackend when killTask is unsupported
commit cb5860ffe8a995ac7566e80a774cf57c498f6182
Author: Imran Rashid <[email protected]>
Date: 2016-05-18T15:58:04Z
move dummy killTask to MockBackend, otherwise occasional problems even in
SingleCoreMockBackend when killTask is unsupported
commit 8034995249e16e80fa7db5e709c38d1444b98f08
Author: Imran Rashid <[email protected]>
Date: 2016-05-18T17:30:09Z
take advantage of ExternalClusteManager extension
commit 360c7cdf7731fd173bbfaf5e66d413f743604e60
Author: Imran Rashid <[email protected]>
Date: 2016-05-18T17:39:52Z
cleanup
commit 08b28c671c363925afda2aebd592911be0838e5c
Author: Imran Rashid <[email protected]>
Date: 2016-05-18T17:56:16Z
Merge branch 'SPARK-10372-scheduler-integs' into blacklist_w_performance
commit c7a78b0df04e86ab942617f84106085c3e750e00
Author: Imran Rashid <[email protected]>
Date: 2016-05-19T17:58:01Z
performance updates to mock backend + some utils
commit ee59913ac216f44b9e0b46a2b662f42b26f48da8
Author: Imran Rashid <[email protected]>
Date: 2016-05-19T17:58:40Z
add performance tests
commit 22705dc39f1a54002fb2e5c0b7ac1587d3339f25
Author: Imran Rashid <[email protected]>
Date: 2016-05-19T18:01:39Z
Merge branch 'master' into blacklist_w_performance
commit 72d87ce0654c34d3fa22d6c3d43afdbef8f3ebd5
Author: Imran Rashid <[email protected]>
Date: 2016-05-19T18:05:09Z
Merge branch 'scheduler_performance_tests' into blacklist_w_performance
commit 4fcbc1da351d050d88357f47b98c85e0cbc0eefc
Author: Imran Rashid <[email protected]>
Date: 2016-05-19T18:29:38Z
bug fix in mock scheduler
commit 6ed19aeec4cccfcca8de7848dd9ff8a0160d920a
Author: Imran Rashid <[email protected]>
Date: 2016-05-20T16:45:43Z
style
commit 67acce9a56a8acc24c0f2cf2ca76378277ec24d3
Author: Imran Rashid <[email protected]>
Date: 2016-05-20T17:19:25Z
simplification and comments
commit 0530a9410d94d2fef01b96e597cdc6c5dd733d27
Author: Imran Rashid <[email protected]>
Date: 2016-05-20T17:20:52Z
Merge branch 'SPARK-10372-scheduler-integs' into scheduler_performance_tests
commit 17fcc9ec89ef9f4f32905fd33dcbe8f84c05eed7
Author: Imran Rashid <[email protected]>
Date: 2016-05-20T17:21:58Z
fix merge
commit b12b563d4890766d0d3ea31ccdf14d3da8fc8f82
Author: Imran Rashid <[email protected]>
Date: 2016-05-20T17:33:31Z
comments
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
