Github user sujithjay commented on a diff in the pull request:
https://github.com/apache/spark/pull/20956#discussion_r178518257
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/tree/impl/NodeIdCache.scala ---
@@ -166,9 +166,13 @@ private[spark] class NodeIdCache(
}
}
}
+ if (nodeIdsForInstances != null) {
+ // Unpersist current one if one exists.
+ nodeIdsForInstances.unpersist(false)
+ }
if (prevNodeIdsForInstances != null) {
// Unpersist the previous one if one exists.
- prevNodeIdsForInstances.unpersist()
+ prevNodeIdsForInstances.unpersist(false)
--- End diff --
Same question as above. `deleteAllCheckpoints` is blocking because it
involves calls to `FileSystem.delete`. So, does it make sense to make the call
to `unpersist` non-blocking? Am I missing something here?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
