hvanhovell commented on code in PR #35991:
URL: https://github.com/apache/spark/pull/35991#discussion_r887524913
##########
core/src/main/scala/org/apache/spark/storage/BlockInfoManager.scala:
##########
@@ -360,12 +360,17 @@ private[storage] class BlockInfoManager extends Logging {
info.writerTask = BlockInfo.NO_WRITER
writeLocksByTask.get(taskAttemptId).remove(blockId)
Review Comment:
TBH in that case we are in a world of hurt. You don't know the state of the
block (might not be there yet, might be partially materialized, might be ok, or
might be deleted) when we release its write lock in `releaseAllLocksForTask`.
The problem with releasing is that after the release the block looks fine, and
other tasks might try to read it (and fail).
The good news is that the code that takes out the write locks is only in the
BlockManager and that it properly cleans up after itself (release locks, remove
data if needed). This means we could fix this issue by removing the release of
write locks from `releaseAllLocksForTask`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
