Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/19208#discussion_r148925715
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala ---
@@ -117,6 +123,12 @@ class CrossValidator @Since("1.2.0") (@Since("1.4.0")
override val uid: String)
instr.logParams(numFolds, seed, parallelism)
logTuningParams(instr)
+ val collectSubModelsParam = $(collectSubModels)
+
+ var subModels: Option[Array[Array[Model[_]]]] = if
(collectSubModelsParam) {
--- End diff --
I don't follow with #1, if we keep all the models (e.g. set
`collectSubModelsParam`) then the maximum memory cost will be
`$(estimatorParamMaps).length * sizeof(model)` in either case? If we don't keep
the models (e.g. set `collectSubModelsParam` to false) then you don't have to
collect the future back at the end and there is no additional overhead.
For #2, It's not that mutation impacts performance, its that it makes the
code less easy to reason about for no gain (unless I've misunderstood something
about part 1).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
