Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/791#issuecomment-43611475
As we know, memory store is used for add, read, remove blocks. Reading and
removing is quite simple, so let's focus on adding.
Adding may trigger dropping action, as I said before, dropping flag can
make each thread select different to-be-dropped blocks, so it's safe to do
dropping in parallel.
When dropping and reading are processing at the same time, the entry is
still there before dropping finished, so it's safe.
When dropping and removing are processing at the same time, if dropping
finished first, then remove will fail and it's OK. If removing finished first,
dropping thread will try to remove entry after write block into disk store, and
this remove action will fail because the entry has been removed, then dropping
thread will remove the block from disk store to cancel the dropping. You can
find this logic in BlockManage#dropFromMemory.
So I think my solution is MT safe already.
And about task termination, one reason is self exception(like disk error,
etc), another is killed by executor. Now I put dropping code in a try-catch and
if catch any exception, reset the dropping flag of to-be-dropped blocks and
throw that exception. I think this should work for handling corner cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
