[GitHub] spark pull request: [SPARK-11846] Add save/load for AFTSurvivalReg...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9836#issuecomment-157985379
  
**[Test build #46321 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46321/consoleFull)**
 for PR 9836 at commit 
[`c784fab`](https://github.com/apache/spark/commit/c784fab77d658f04950b333f4ddb5302fa6e5ac6).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11846] Add save/load for AFTSurvivalReg...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9836#issuecomment-157993127
  
**[Test build #46321 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46321/consoleFull)**
 for PR 9836 at commit 
[`c784fab`](https://github.com/apache/spark/commit/c784fab77d658f04950b333f4ddb5302fa6e5ac6).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11846] Add save/load for AFTSurvivalReg...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9836#issuecomment-157993279
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46321/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11846] Add save/load for AFTSurvivalReg...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9836#issuecomment-157993278
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11846] Add save/load for AFTSurvivalReg...

[mengxr]

Github user mengxr commented on a diff in the pull request:

https://github.com/apache/spark/pull/9836#discussion_r45384357
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/ml/regression/IsotonicRegression.scala ---
@@ -240,4 +254,61 @@ class IsotonicRegressionModel private[ml] (
   override def transformSchema(schema: StructType): StructType = {
 validateAndTransformSchema(schema, fitting = false)
   }
+
+  @Since("1.6.0")
+  override def write: MLWriter =
+new IsotonicRegressionModelWriter(this)
+}
+
+@Since("1.6.0")
+object IsotonicRegressionModel extends MLReadable[IsotonicRegressionModel] 
{
+
+  @Since("1.6.0")
+  override def read: MLReader[IsotonicRegressionModel] = new 
IsotonicRegressionModelReader
+
+  @Since("1.6.0")
+  override def load(path: String): IsotonicRegressionModel = 
super.load(path)
+
+  /** [[MLWriter]] instance for [[IsotonicRegressionModel]] */
+  private[IsotonicRegressionModel] class IsotonicRegressionModelWriter (
+  instance: IsotonicRegressionModel
+) extends MLWriter with Logging {
+
+private case class Data(
+boundaries: Array[Double],
+predictions: Array[Double],
+isotonic: Boolean)
+
+override protected def saveImpl(path: String): Unit = {
+  // Save metadata and Params
+  DefaultParamsWriter.saveMetadata(instance, path, sc)
+  // Save model data: boundaries, predictions, isotonic
+  val data = Data(
+instance.oldModel.boundaries, instance.oldModel.predictions, 
instance.oldModel.isotonic)
+  val dataPath = new Path(path, "data").toString
+  
sqlContext.createDataFrame(Seq(data)).repartition(1).write.parquet(dataPath)
+}
+  }
+
+  private class IsotonicRegressionModelReader extends 
MLReader[IsotonicRegressionModel] {
+
+/** Checked against metadata when loading model */
+private val className = classOf[IsotonicRegressionModel].getName
+
+override def load(path: String): IsotonicRegressionModel = {
+  val metadata = DefaultParamsReader.loadMetadata(path, sc, className)
+
+  val dataPath = new Path(path, "data").toString
+  val data = sqlContext.read.format("parquet").load(dataPath)
+.select("boundaries", "predictions", "isotonic").head()
+  val boundaries = data.getAs[mutable.WrappedArray[Double]](0).toArray
--- End diff --

data.getAs[Seq[Double]](0).toArray


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11846] Add save/load for AFTSurvivalReg...

[mengxr]

Github user mengxr commented on a diff in the pull request:

https://github.com/apache/spark/pull/9836#discussion_r45384012
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/ml/regression/IsotonicRegression.scala ---
@@ -240,4 +254,61 @@ class IsotonicRegressionModel private[ml] (
   override def transformSchema(schema: StructType): StructType = {
 validateAndTransformSchema(schema, fitting = false)
   }
+
+  @Since("1.6.0")
+  override def write: MLWriter =
+new IsotonicRegressionModelWriter(this)
+}
+
+@Since("1.6.0")
+object IsotonicRegressionModel extends MLReadable[IsotonicRegressionModel] 
{
+
+  @Since("1.6.0")
+  override def read: MLReader[IsotonicRegressionModel] = new 
IsotonicRegressionModelReader
+
+  @Since("1.6.0")
+  override def load(path: String): IsotonicRegressionModel = 
super.load(path)
+
+  /** [[MLWriter]] instance for [[IsotonicRegressionModel]] */
+  private[IsotonicRegressionModel] class IsotonicRegressionModelWriter (
+  instance: IsotonicRegressionModel
+) extends MLWriter with Logging {
+
+private case class Data(
+boundaries: Array[Double],
+predictions: Array[Double],
+isotonic: Boolean)
+
+override protected def saveImpl(path: String): Unit = {
+  // Save metadata and Params
+  DefaultParamsWriter.saveMetadata(instance, path, sc)
+  // Save model data: boundaries, predictions, isotonic
+  val data = Data(
+instance.oldModel.boundaries, instance.oldModel.predictions, 
instance.oldModel.isotonic)
+  val dataPath = new Path(path, "data").toString
+  
sqlContext.createDataFrame(Seq(data)).repartition(1).write.parquet(dataPath)
+}
+  }
+
+  private class IsotonicRegressionModelReader extends 
MLReader[IsotonicRegressionModel] {
+
+/** Checked against metadata when loading model */
+private val className = classOf[IsotonicRegressionModel].getName
+
+override def load(path: String): IsotonicRegressionModel = {
+  val metadata = DefaultParamsReader.loadMetadata(path, sc, className)
+
+  val dataPath = new Path(path, "data").toString
+  val data = sqlContext.read.format("parquet").load(dataPath)
--- End diff --

minor: `sqlContext.read.parquet(dataPath)`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11846] Add save/load for AFTSurvivalReg...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9836#issuecomment-158235753
  
**[Test build #46370 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46370/consoleFull)**
 for PR 9836 at commit 
[`a9dcb8a`](https://github.com/apache/spark/commit/a9dcb8a487dec027bb14eaaad8ee8141dea0c5dd).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11846] Add save/load for AFTSurvivalReg...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9836#issuecomment-158243929
  
**[Test build #46370 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46370/consoleFull)**
 for PR 9836 at commit 
[`a9dcb8a`](https://github.com/apache/spark/commit/a9dcb8a487dec027bb14eaaad8ee8141dea0c5dd).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11846] Add save/load for AFTSurvivalReg...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9836#issuecomment-158244036
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46370/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11846] Add save/load for AFTSurvivalReg...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9836#issuecomment-158244035
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11846] Add save/load for AFTSurvivalReg...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9836#issuecomment-158204004
  
**[Test build #46352 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46352/consoleFull)**
 for PR 9836 at commit 
[`c53ddb2`](https://github.com/apache/spark/commit/c53ddb203c5d37e07fdf0391fc559ca82c557b95).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11846] Add save/load for AFTSurvivalReg...

[mengxr]

Github user mengxr commented on the pull request:

https://github.com/apache/spark/pull/9836#issuecomment-158187297
  
LGTM except minor style issues.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11846] Add save/load for AFTSurvivalReg...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9836#issuecomment-158218854
  
**[Test build #46352 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46352/consoleFull)**
 for PR 9836 at commit 
[`c53ddb2`](https://github.com/apache/spark/commit/c53ddb203c5d37e07fdf0391fc559ca82c557b95).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11846] Add save/load for AFTSurvivalReg...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9836#issuecomment-158219154
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11846] Add save/load for AFTSurvivalReg...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9836#issuecomment-158219155
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46352/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11846] Add save/load for AFTSurvivalReg...

[mengxr]

Github user mengxr commented on the pull request:

https://github.com/apache/spark/pull/9836#issuecomment-158291677
  
Merged into master and branch-1.6. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11846] Add save/load for AFTSurvivalReg...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/9836


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11846] Add save/load for AFTSurvivalReg...

[yinxusen]

GitHub user yinxusen opened a pull request:

https://github.com/apache/spark/pull/9836

[SPARK-11846] Add save/load for AFTSurvivalRegression and IsotonicRegression

https://issues.apache.org/jira/browse/SPARK-11846

@mengxr 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yinxusen/spark SPARK-11846

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/9836.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #9836


commit 4f387700f84c3050ef47162330f5a9f67ee58fae
Author: Xusen Yin 
Date:   2015-11-19T06:52:27Z

add AFTSurvivialRegression save/load

commit 9a2c72c24dccd6019e7e3948f43ea2b5ef9cf417
Author: Xusen Yin 
Date:   2015-11-19T07:19:41Z

add Isotonic regression save/load

commit fa1d84beb8c2d1f61046b35328f03f34cfc1f6cb
Author: Xusen Yin 
Date:   2015-11-19T07:47:46Z

add params

commit c784fab77d658f04950b333f4ddb5302fa6e5ac6
Author: Xusen Yin 
Date:   2015-11-19T07:56:47Z

fix misspell




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org