Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3574#issuecomment-65910884
Ah, I think I see your concern: let's say that we a block and there are two
threads that are racing to perform operations on it: to use your example,
thread A wants to call `removeBlock()` and thread B wants to call
`dropFromMemory()`. For this code to work correctly, we want it to operate
correctly for all possible interleavings of those threads
If we consider the case where _all_ of thread A's steps execute before
_any_ of thread B's, then things work okay: thread A will have removed the
entry from `blockInfo` before thread B runs, so `B` will see that `info ==
null` and log a warning that the block has already been removed. The same is
true for B before A.
In another execution, though, both threads could find the `BlockInfo`
instance in the `blockInfo` map but race on acquiring its lock
(`info.synchronized`), so `info != null` will be true for both threads. I
agree that this could be a problem, but it might not be if the operations
performed in those threads are idempotent. Let's take a look and see if that's
the case:
- `removeBlock`: all of the operations here are idempotent, so at worst we
get a warning if we run this on a block that's removed by another racing thread.
- `dropOldBlocks`: similarly, this just consists of calls to
`*Store.remove()`, which are idempotent.
- `dropFromMemory`: this case might actually be problematic, since I think
that this method calls data store operations that don't handle missing blocks.
I'm going to look at this case in a little more detail, but I think that your
fix for this might be a good idea.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
