GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/15036
[SPARK-17483] Minor refactoring in BlockManager status reporting and block
removal
This patch makes three minor refactorings to the BlockManager:
- Move the `if (info.tellMaster)` check out of `reportBlockStatus`; this
fixes an issue where a debug logging message would incorrectly claim to have
reported a block status to the master even though no message had been sent (in
case `info.tellMaster == false`). This also makes it easier to write code which
unconditionally sends block statuses to the master (which is necessary in
another patch of mine).
- Split `removeBlock()` into two methods, the existing method and an
internal `removeBlockInternal()` method which is designed to be called by
internal code that already holds a write lock on the block. This is also needed
by a followup patch.
- Instead of calling `getCurrentBlockStatus()` in `removeBlock()`, just
pass `BlockStatus.empty`; the block status should always be empty following
complete removal of a block.
These changes were originally authored as part of a bug fix patch which is
targeted at branch-2.0 and master; I've split them out here into their own
separate PR in order to make them easier to review and so that the
behavior-changing parts of my other patch can be isolated to their own PR.
/cc @ericl for review.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark
cache-failure-race-conditions-refactorings-only
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15036.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15036
----
commit a5936ba700e0efef90b14e817817fc2fdc382bf5
Author: Josh Rosen <[email protected]>
Date: 2016-09-09T01:38:59Z
Add internal version of removeBlock that can be called while holding lock.
commit f3890a53d969b09b8a9260c739c17d5610f895aa
Author: Josh Rosen <[email protected]>
Date: 2016-09-09T17:57:08Z
BlockStatus should always be empty following removal.
commit dca6a8a5dd88b25ba764431af6514287ad2dbca4
Author: Josh Rosen <[email protected]>
Date: 2016-09-09T18:19:03Z
Move if(info.tellMaster) check out of reportBlockStatus.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
