GitHub user mwws opened a pull request:
https://github.com/apache/spark/pull/8760
Spark 8426
1. add a global BlacklistTracker which is able to share blacklisted
executors information across TaskSetManager. Different BlacklistStrategy could
be applied to determine which executors should be put in blacklist. And it's
convenient to add new strategy later according to current design.
2. remove duplicate "failedExecutors" list in TaskSetManager and make
BlacklistTracker being the only interface for blacklist mechanism in codebase
which is logically cleaner.
3. for backward compatibility, implement DefaultStrategy to keep the same
semantics as current by default
4. apply BlacklistTracker in YARN to avoid allocating new executors on
blacklisted node, the blacklisted node information will be passed to YARN
ResourceManager. it affects in dynamic allocation mode.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mwws/spark SPARK-8426
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/8760.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #8760
----
commit 8519cb5ca4bbe86885fd7cc09fc9aeaf6fab1436
Author: unknown <[email protected]>
Date: 2015-07-28T05:19:29Z
change package name to test.org.apache.spark.sql.hive.execution
commit 7c91bb55b9424657409c7e5131cf9b791b9e4a66
Author: unknown <[email protected]>
Date: 2015-09-02T07:54:31Z
Merge remote-tracking branch 'apache-spark/master'
commit 9a25d71025a411e1fad4190e914465218a1a2d66
Author: mwws <[email protected]>
Date: 2015-09-10T05:29:44Z
add executor blacklist feature
1. Add blackListTracker and blackListStrategy class
2. remove deplicated failedExecutor in TaskSetManager
3. enhance YARN allocator to pass node blacklist info to ResourceManager
commit f648b4c185042ee9d38e11b7f213fe68868c8d72
Author: mwws <[email protected]>
Date: 2015-09-11T05:25:31Z
some enhancement
1. Add taskId to FailureInfo to be able to contain enough informantion
for backward compatibility
2. replace Seq with Set to speedup contains method and remove duplicate
element by natual
3. remove executors from failureExecutors when
(1)any task succeed on the executor
(2)the executor is removed by ExecutorAllocateClient
3. Move IsBlacklist() method into BlacklistTracker to make logic cleaner
4. fit spark code style, eg: lineend, comments start, max length of line
commit 954886f18b3811958b1c2a588f00652d678cf871
Author: mwws <[email protected]>
Date: 2015-09-11T07:47:40Z
fix compile error
fix compile error
commit 78feafbe23729f2102effec9abc90d6a865ba211
Author: mwws <[email protected]>
Date: 2015-09-14T04:29:08Z
farther change according to comment
1. rename some varilables and methods
2. correct FailureExecutors logic in CoarseGrainedSchedulerBackend
commit c84f362b4002eddde22f43c421434f413b04a5ee
Author: mwws <[email protected]>
Date: 2015-09-14T04:36:53Z
some change to fit scala style
fit scala style
commit 90ee814686e3bf7906fa447b7594aa6b3643de80
Author: mwws <[email protected]>
Date: 2015-09-14T05:22:00Z
revert incorrect package name change
revert incorrect package name change
commit 413b403ccc577d062b6eae10d995077631d78e97
Author: mwws <[email protected]>
Date: 2015-09-15T05:06:01Z
change strategy to keep semanetic as original
The original blacklist logic is taskId related, so I change interface
and add a default strategy to keep same semanetic as it.
commit 7c6eb632fb4132dc053683db617454ce7b335a31
Author: unknown <[email protected]>
Date: 2015-07-28T05:19:29Z
change package name to test.org.apache.spark.sql.hive.execution
commit b306caecf6d5112c44029ee016654119d3c5185d
Author: mwws <[email protected]>
Date: 2015-09-10T05:29:44Z
add executor blacklist feature
1. Add blackListTracker and blackListStrategy class
2. remove deplicated failedExecutor in TaskSetManager
3. enhance YARN allocator to pass node blacklist info to ResourceManager
commit 395773f1a91d13c4e21bd19bcc56e5a02c9c2784
Author: mwws <[email protected]>
Date: 2015-09-11T05:25:31Z
some enhancement
1. Add taskId to FailureInfo to be able to contain enough informantion
for backward compatibility
2. replace Seq with Set to speedup contains method and remove duplicate
element by natual
3. remove executors from failureExecutors when
(1)any task succeed on the executor
(2)the executor is removed by ExecutorAllocateClient
3. Move IsBlacklist() method into BlacklistTracker to make logic cleaner
4. fit spark code style, eg: lineend, comments start, max length of line
commit a7dfa20fec17de80755edde3b967f9345d8c2834
Author: mwws <[email protected]>
Date: 2015-09-11T07:47:40Z
fix compile error
fix compile error
commit 94d3a5d90e126d8c95580f2e7f3d5fff6dd7bc4c
Author: mwws <[email protected]>
Date: 2015-09-14T04:29:08Z
farther change according to comment
1. rename some varilables and methods
2. correct FailureExecutors logic in CoarseGrainedSchedulerBackend
commit 26fb5a5c41bbbdba793d8ea0c1d40cad299ffedb
Author: mwws <[email protected]>
Date: 2015-09-14T04:36:53Z
some change to fit scala style
fit scala style
commit f7ecc784788e7b3f0f75a975a781f1af7040bbc4
Author: mwws <[email protected]>
Date: 2015-09-14T05:22:00Z
revert incorrect package name change
revert incorrect package name change
commit f44d4e134f725dc42b3017bebe20f459b625df8c
Author: mwws <[email protected]>
Date: 2015-09-15T05:06:01Z
change strategy to keep semanetic as original
The original blacklist logic is taskId related, so I change interface
and add a default strategy to keep same semanetic as it.
commit c36ea5b44e9a3c4a91bb3625abc3863e2df2aa66
Author: mwws <[email protected]>
Date: 2015-09-15T06:03:46Z
Merge branch 'SPARK-8426' of https://github.com/mwws/spark into SPARK-8426
Conflicts:
core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
commit 24cdf30d6144d084b5c8ff23984e037bab18b4dd
Author: mwws <[email protected]>
Date: 2015-09-15T06:19:45Z
fix compile error
fix compile error casued by rebase
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
