Hello - I have a question on how to handle incremental model updation in Spark ..
We have a time series where we predict the future conditioned on the past. We can train a model offline based on historical data and then use that model during prediction. But say, if the underlying process is non-stationary, the probability distribution changes with time. In such cases we need to update the model so as to reflect the current change in distribution. We have 2 options - a. retrain periodically b. update the model incrementally a. is expensive. What about b. ? I think in Spark we have StreamingKMeans that takes care of this incremental model update for this classifier. Is this true ? But what about other classifiers that don't have these Streaming counterparts ? How do we handle such incremental model changes with those classifiers where the underlying distribution changes ? regards. -- Debasish Ghosh http://manning.com/ghosh2 http://manning.com/ghosh Twttr: @debasishg Blog: http://debasishg.blogspot.com Code: http://github.com/debasishg
