mridulm commented on code in PR #39459:
URL: https://github.com/apache/spark/pull/39459#discussion_r1071815096
##########
core/src/main/scala/org/apache/spark/storage/BlockManager.scala:
##########
@@ -1325,14 +1325,47 @@ private[spark] class BlockManager(
blockInfoManager.releaseAllLocksForTask(taskAttemptId)
}
+ /**
+ * Retrieve the given rdd block if it exists and is visible, otherwise call
the provided
+ * `makeIterator` method to compute the block, persist it, and return its
values.
+ *
+ * @return either a BlockResult if the block was successfully cached, or an
iterator if the block
+ * could not be cached.
+ */
+ def getOrElseUpdateRDDBlock[T](
+ taskId: Long,
+ blockId: RDDBlockId,
+ level: StorageLevel,
+ classTag: ClassTag[T],
+ makeIterator: () => Iterator[T]): Either[BlockResult, Iterator[T]] = {
+ val isCacheVisible = isRDDBlockVisible(blockId)
+ var computed: Boolean = false
+ val getIterator = () => {
+ computed = true
+ makeIterator()
+ }
+
+ val res = getOrElseUpdate(blockId, level, classTag, getIterator)
+ if (res.isLeft && !isCacheVisible) {
+ if (!computed) {
+ // Loaded from cache, re-compute to update accumulators.
+ makeIterator()
+ }
Review Comment:
If a block is not visible, and it was not computed in this task, we should
not be using that block's cached data at all - even if cached here or elsewhere.
Particularly for unordered or indeterminate computation, this can impact the
accumulators generated - and it wont match the result iterator returned.
So we cant simply call `makeIterator` here - as @Ngone51 suggested, we
should pass the flag to `getOrElseUpdate` and decide there based on visibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
