Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3582#issuecomment-65911726
To clarify a bit further: I think that `BlockInfo.waitForReady()` is
designed to allow callers to block until a block write has completed. If we
have many threads (readers) waiting for a block to be written, then think we
should be okay because `notifyAll()` will wake all of those threads when the
block becomes ready.
From your description, it sounds like you're worried about a multiple
writer-threads case, where we have many threads attempting to write the same
block and a failed write attempt from _one_ of them wakes up the waiting
threads and notifies them of a failure even though there's another write in
progress which might succeed. Is your goal to wait until _all_ of the pending
writes have failed before notifying a reader that the write has failed and to
wait for _one_ of them to succeed before notifying the reader that the write
succeeded?
I'll have to dig into the BlockManager internals to see whether we can ever
have multiple in-progress writes for the same block. Do you have an example of
when this can happen?
I'd be happy to look over the code and provide more feedback / suggestions,
but I want to make sure that I understand the motivation and confirm that this
is fixing an actual bug, since it seems like this adds a moderate amount of
complexity.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
