Github user MLnick commented on a diff in the pull request:
https://github.com/apache/spark/pull/16774#discussion_r110605449
--- Diff: docs/ml-tuning.md ---
@@ -55,6 +55,9 @@ for multiclass problems. The default metric used to
choose the best `ParamMap` c
method in each of these evaluators.
To help construct the parameter grid, users can use the
[`ParamGridBuilder`](api/scala/index.html#org.apache.spark.ml.tuning.ParamGridBuilder)
utility.
+Sets of parameters from the parameter grid can be evaluated in parallel by
setting `numParallelEval` with a value of 2 or more (a value of 1 will evaluate
in serial) before running model selection with `CrossValidator` or
`TrainValidationSplit`.
+The value of `numParallelEval` should be chosen carefully to maximize
parallelism without exceeding cluster resources, and will be capped at the
number of cores in the driver system. Generally speaking, a value up to 10
should be sufficient for most clusters.
+
--- End diff --
Also will need to mention that custom `ExecutorService` can be specified,
and some detail on the default thread pool it creates (and that it is a new
separate pool to avoid blocking any of the default Scala pools).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
