Github user MrBago commented on a diff in the pull request:
https://github.com/apache/spark/pull/19904#discussion_r156511049
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala ---
@@ -146,25 +147,18 @@ class CrossValidator @Since("1.2.0") (@Since("1.4.0")
override val uid: String)
val validationDataset = sparkSession.createDataFrame(validation,
schema).cache()
logDebug(s"Train split $splitIndex with multiple sets of
parameters.")
+ val completeFitCount = new AtomicInteger(0)
--- End diff --
You can use futures to do this, you need to use a var for `modelFutures`
then map on those futures to `Unit` collect those into a sequence and map on
that to unpersist, but why go to the trouble. What's the concern with doing it
in the final training thread. Why is this a change in behavior?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
