[GitHub] spark pull request #19350: [SPARK-22126][ML][WIP] Fix model-specific optimiz...
Github user WeichenXu123 closed the pull request at: https://github.com/apache/spark/pull/19350 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19350: [SPARK-22126][ML][WIP] Fix model-specific optimiz...
Github user WeichenXu123 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19350#discussion_r157118663
--- Diff: mllib/src/main/scala/org/apache/spark/ml/Estimator.scala ---
@@ -82,5 +86,49 @@ abstract class Estimator[M <: Model[M]] extends
PipelineStage {
paramMaps.map(fit(dataset, _))
}
+ /**
+ * (Java-specific)
+ */
+ @Since("2.3.0")
+ def fit(dataset: Dataset[_], paramMaps: Array[ParamMap],
+unpersistDatasetAfterFitting: Boolean, executionContext:
ExecutionContext,
+modelCallback: VoidFunction2[Model[_], Int]): Unit = {
+// Fit models in a Future for training in parallel
+val modelFutures = paramMaps.map { paramMap =>
+ Future[Model[_]] {
+fit(dataset, paramMap).asInstanceOf[Model[_]]
--- End diff --
@MLnick I dicussed with @jkbradley @MrBago offline and here is the newest
proposal
https://docs.google.com/document/d/1xw5M4sp1e0eQie75yIt-r6-GTuD5vpFf_I6v-AFBM3M/edit?usp=sharing
Thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19350: [SPARK-22126][ML][WIP] Fix model-specific optimiz...
Github user WeichenXu123 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19350#discussion_r156603289
--- Diff: mllib/src/main/scala/org/apache/spark/ml/Estimator.scala ---
@@ -82,5 +86,49 @@ abstract class Estimator[M <: Model[M]] extends
PipelineStage {
paramMaps.map(fit(dataset, _))
}
+ /**
+ * (Java-specific)
+ */
+ @Since("2.3.0")
+ def fit(dataset: Dataset[_], paramMaps: Array[ParamMap],
+unpersistDatasetAfterFitting: Boolean, executionContext:
ExecutionContext,
+modelCallback: VoidFunction2[Model[_], Int]): Unit = {
+// Fit models in a Future for training in parallel
+val modelFutures = paramMaps.map { paramMap =>
+ Future[Model[_]] {
+fit(dataset, paramMap).asInstanceOf[Model[_]]
--- End diff --
@MLnick Oh, the design is still under discussion on JIRA and will be
changed I think. I should mark this WIP. thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
