Incremental model update

2016-09-27 Thread debasishg 
Hello -

I have a question on how to handle incremental model updation in Spark ML ..

We have a time series where we predict the future conditioned on the past.
We can train a model offline based on historical data and then use that
model during prediction. 

But say, if the underlying process is non-stationary, the probability
distribution changes with time. In such cases we need to update the model so
as to reflect the current change in distribution.

We have 2 options -

a. retrain periodically
b. update the model incrementally

a. is expensive. 

What about b. ? I think in Spark we have StreamingKMeans that takes care of
this incremental model update for this classifier. Is this true ?

But what about other classifiers that don't have these Streaming
counterparts ? How do we handle such incremental model changes with those
classifiers where the underlying distribution changes ?

regards.




--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Incremental-model-update-tp27800.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Incremental model update

2016-09-27 Thread Debasish Ghosh 
Hello -

I have a question on how to handle incremental model updation in Spark ..

We have a time series where we predict the future conditioned on the past.
We can train a model offline based on historical data and then use that
model during prediction.

But say, if the underlying process is non-stationary, the probability
distribution changes with time. In such cases we need to update the model
so as to reflect the current change in distribution.

We have 2 options -

a. retrain periodically
b. update the model incrementally

a. is expensive.

What about b. ? I think in Spark we have StreamingKMeans that takes care of
this incremental model update for this classifier. Is this true ?

But what about other classifiers that don't have these Streaming
counterparts ? How do we handle such incremental model changes with those
classifiers where the underlying distribution changes ?

regards.

-- 
Debasish Ghosh
http://manning.com/ghosh2
http://manning.com/ghosh

Twttr: @debasishg
Blog: http://debasishg.blogspot.com
Code: http://github.com/debasishg