[
https://issues.apache.org/jira/browse/SPARK-4722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14232920#comment-14232920
]
Xiangrui Meng commented on SPARK-4722:
--------------------------------------
[~Arthur][ You can use `StreamingLinearRegression.model` to get the latest
model. It may be expensive and unnecessary to make predictOn return a DStream
of model weights. If you want to re-use the previously trained model, you save
the last model coefficients in the first run and then set initial weights in
the second run.
> StreamingLinearRegression should return a DStream of weights when calling
> trainOn
> ---------------------------------------------------------------------------------
>
> Key: SPARK-4722
> URL: https://issues.apache.org/jira/browse/SPARK-4722
> Project: Spark
> Issue Type: Improvement
> Components: MLlib, Streaming
> Reporter: Arthur Andres
> Priority: Minor
> Labels: mllib, regression, streaming
>
> When training a model with a stream of new data (Spark Streaming + Spark
> Mlllib), the weights (and the other part of the regression model) update at
> every iterations.
> At the moment the only output we can get is the prediction when calling
> predictOn (class StreamingLinearRegression)
> It would be a nice improvement if trainOn would return a Dstream of weights
> (and any other underlying model data) so we can access it and see it evolve.
> At the moment they are only outputted in the log
> For example this could then be saved so when reloading the application we can
> access this information without having to train the model again.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
