Github user ksakellis commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r22822258
--- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala ---
@@ -44,7 +44,14 @@ private[spark] class CacheManager(blockManager:
BlockManager) extends Logging {
blockManager.get(key) match {
case Some(blockResult) =>
// Partition is already materialized, so just return its values
+ val existingMetrics = context.taskMetrics.inputMetrics
+ val prevBytesRead = existingMetrics
+ .filter(_.readMethod == blockResult.inputMetrics.readMethod)
+ .map(_.bytesRead)
+ .getOrElse(0L)
--- End diff --
Yes, when the cache is being used then yes there will be clobbering. There
are a few solutions
1) we can not filter on readMethod and just append blindly. That way we
don't override any metrics but the eventual read method will not be correct
(either first wins or last wins - whatever we choose)
2) we model input metrics like we do with shuffle metrics where we collect
an array of them and then finally we sum them up. - this is a bigger change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
