Re: How to reuse a ML trained model?
errr...do you have any suggestions for me before 1.3 release? I can't believe there's no ML model serialize method in Spark. I think training the models are quite expensive, isn't it? Thanks, David On Sun, Mar 8, 2015 at 5:14 AM Burak Yavuz brk...@gmail.com wrote: Hi, There is model import/export for some of the ML algorithms on the current master (and they'll be shipped with the 1.3 release). Burak On Mar 7, 2015 4:17 AM, Xi Shen davidshe...@gmail.com wrote: Wait...it seem SparkContext does not provide a way to save/load object files. It can only save/load RDD. What do I missed here? Thanks, David On Sat, Mar 7, 2015 at 11:05 PM Xi Shen davidshe...@gmail.com wrote: Ah~it is serializable. Thanks! On Sat, Mar 7, 2015 at 10:59 PM Ekrem Aksoy ekremak...@gmail.com wrote: You can serialize your trained model to persist somewhere. Ekrem Aksoy On Sat, Mar 7, 2015 at 12:10 PM, Xi Shen davidshe...@gmail.com wrote: Hi, I checked a few ML algorithms in MLLib. https://spark.apache.org/docs/0.8.1/api/mllib/index.html# org.apache.spark.mllib.classification.LogisticRegressionModel I could not find a way to save the trained model. Does this means I have to train my model every time? Is there a more economic way to do this? I am thinking about something like: model.run(...) model.save(hdfs://path/to/hdfs) Then, next I can do: val model = Model.createFrom(hdfs://...) model.predict(vector) I am new to spark, maybe there are other ways to persistent the model? Thanks, David
Re: How to reuse a ML trained model?
You dont need SparkContext to simply serialize and deserialize objects. It is Java mechanism. On Mar 8, 2015 10:29 AM, Xi Shen davidshe...@gmail.com wrote: errr...do you have any suggestions for me before 1.3 release? I can't believe there's no ML model serialize method in Spark. I think training the models are quite expensive, isn't it? Thanks, David On Sun, Mar 8, 2015 at 5:14 AM Burak Yavuz brk...@gmail.com wrote: Hi, There is model import/export for some of the ML algorithms on the current master (and they'll be shipped with the 1.3 release). Burak On Mar 7, 2015 4:17 AM, Xi Shen davidshe...@gmail.com wrote: Wait...it seem SparkContext does not provide a way to save/load object files. It can only save/load RDD. What do I missed here? Thanks, David On Sat, Mar 7, 2015 at 11:05 PM Xi Shen davidshe...@gmail.com wrote: Ah~it is serializable. Thanks! On Sat, Mar 7, 2015 at 10:59 PM Ekrem Aksoy ekremak...@gmail.com wrote: You can serialize your trained model to persist somewhere. Ekrem Aksoy On Sat, Mar 7, 2015 at 12:10 PM, Xi Shen davidshe...@gmail.com wrote: Hi, I checked a few ML algorithms in MLLib. https://spark.apache.org/docs/0.8.1/api/mllib/index.html# org.apache.spark.mllib.classification.LogisticRegressionModel I could not find a way to save the trained model. Does this means I have to train my model every time? Is there a more economic way to do this? I am thinking about something like: model.run(...) model.save(hdfs://path/to/hdfs) Then, next I can do: val model = Model.createFrom(hdfs://...) model.predict(vector) I am new to spark, maybe there are other ways to persistent the model? Thanks, David
Re: How to reuse a ML trained model?
You may also take a look at PredictionIO, which can persist and then deploy MLlib models as web services. Simon On Sunday, March 8, 2015, Sean Owen so...@cloudera.com wrote: You dont need SparkContext to simply serialize and deserialize objects. It is Java mechanism. On Mar 8, 2015 10:29 AM, Xi Shen davidshe...@gmail.com javascript:_e(%7B%7D,'cvml','davidshe...@gmail.com'); wrote: errr...do you have any suggestions for me before 1.3 release? I can't believe there's no ML model serialize method in Spark. I think training the models are quite expensive, isn't it? Thanks, David On Sun, Mar 8, 2015 at 5:14 AM Burak Yavuz brk...@gmail.com javascript:_e(%7B%7D,'cvml','brk...@gmail.com'); wrote: Hi, There is model import/export for some of the ML algorithms on the current master (and they'll be shipped with the 1.3 release). Burak On Mar 7, 2015 4:17 AM, Xi Shen davidshe...@gmail.com javascript:_e(%7B%7D,'cvml','davidshe...@gmail.com'); wrote: Wait...it seem SparkContext does not provide a way to save/load object files. It can only save/load RDD. What do I missed here? Thanks, David On Sat, Mar 7, 2015 at 11:05 PM Xi Shen davidshe...@gmail.com javascript:_e(%7B%7D,'cvml','davidshe...@gmail.com'); wrote: Ah~it is serializable. Thanks! On Sat, Mar 7, 2015 at 10:59 PM Ekrem Aksoy ekremak...@gmail.com javascript:_e(%7B%7D,'cvml','ekremak...@gmail.com'); wrote: You can serialize your trained model to persist somewhere. Ekrem Aksoy On Sat, Mar 7, 2015 at 12:10 PM, Xi Shen davidshe...@gmail.com javascript:_e(%7B%7D,'cvml','davidshe...@gmail.com'); wrote: Hi, I checked a few ML algorithms in MLLib. https://spark.apache.org/docs/0.8.1/api/mllib/index.html# org.apache.spark.mllib.classification.LogisticRegressionModel I could not find a way to save the trained model. Does this means I have to train my model every time? Is there a more economic way to do this? I am thinking about something like: model.run(...) model.save(hdfs://path/to/hdfs) Then, next I can do: val model = Model.createFrom(hdfs://...) model.predict(vector) I am new to spark, maybe there are other ways to persistent the model? Thanks, David
Re: How to reuse a ML trained model?
Ah~it is serializable. Thanks! On Sat, Mar 7, 2015 at 10:59 PM Ekrem Aksoy ekremak...@gmail.com wrote: You can serialize your trained model to persist somewhere. Ekrem Aksoy On Sat, Mar 7, 2015 at 12:10 PM, Xi Shen davidshe...@gmail.com wrote: Hi, I checked a few ML algorithms in MLLib. https://spark.apache.org/docs/0.8.1/api/mllib/index.html#org.apache.spark.mllib.classification.LogisticRegressionModel I could not find a way to save the trained model. Does this means I have to train my model every time? Is there a more economic way to do this? I am thinking about something like: model.run(...) model.save(hdfs://path/to/hdfs) Then, next I can do: val model = Model.createFrom(hdfs://...) model.predict(vector) I am new to spark, maybe there are other ways to persistent the model? Thanks, David
Re: How to reuse a ML trained model?
Hi, There is model import/export for some of the ML algorithms on the current master (and they'll be shipped with the 1.3 release). Burak On Mar 7, 2015 4:17 AM, Xi Shen davidshe...@gmail.com wrote: Wait...it seem SparkContext does not provide a way to save/load object files. It can only save/load RDD. What do I missed here? Thanks, David On Sat, Mar 7, 2015 at 11:05 PM Xi Shen davidshe...@gmail.com wrote: Ah~it is serializable. Thanks! On Sat, Mar 7, 2015 at 10:59 PM Ekrem Aksoy ekremak...@gmail.com wrote: You can serialize your trained model to persist somewhere. Ekrem Aksoy On Sat, Mar 7, 2015 at 12:10 PM, Xi Shen davidshe...@gmail.com wrote: Hi, I checked a few ML algorithms in MLLib. https://spark.apache.org/docs/0.8.1/api/mllib/index.html# org.apache.spark.mllib.classification.LogisticRegressionModel I could not find a way to save the trained model. Does this means I have to train my model every time? Is there a more economic way to do this? I am thinking about something like: model.run(...) model.save(hdfs://path/to/hdfs) Then, next I can do: val model = Model.createFrom(hdfs://...) model.predict(vector) I am new to spark, maybe there are other ways to persistent the model? Thanks, David
Re: How to reuse a ML trained model?
You can serialize your trained model to persist somewhere. Ekrem Aksoy On Sat, Mar 7, 2015 at 12:10 PM, Xi Shen davidshe...@gmail.com wrote: Hi, I checked a few ML algorithms in MLLib. https://spark.apache.org/docs/0.8.1/api/mllib/index.html#org.apache.spark.mllib.classification.LogisticRegressionModel I could not find a way to save the trained model. Does this means I have to train my model every time? Is there a more economic way to do this? I am thinking about something like: model.run(...) model.save(hdfs://path/to/hdfs) Then, next I can do: val model = Model.createFrom(hdfs://...) model.predict(vector) I am new to spark, maybe there are other ways to persistent the model? Thanks, David
Re: How to reuse a ML trained model?
Wait...it seem SparkContext does not provide a way to save/load object files. It can only save/load RDD. What do I missed here? Thanks, David On Sat, Mar 7, 2015 at 11:05 PM Xi Shen davidshe...@gmail.com wrote: Ah~it is serializable. Thanks! On Sat, Mar 7, 2015 at 10:59 PM Ekrem Aksoy ekremak...@gmail.com wrote: You can serialize your trained model to persist somewhere. Ekrem Aksoy On Sat, Mar 7, 2015 at 12:10 PM, Xi Shen davidshe...@gmail.com wrote: Hi, I checked a few ML algorithms in MLLib. https://spark.apache.org/docs/0.8.1/api/mllib/index.html# org.apache.spark.mllib.classification.LogisticRegressionModel I could not find a way to save the trained model. Does this means I have to train my model every time? Is there a more economic way to do this? I am thinking about something like: model.run(...) model.save(hdfs://path/to/hdfs) Then, next I can do: val model = Model.createFrom(hdfs://...) model.predict(vector) I am new to spark, maybe there are other ways to persistent the model? Thanks, David
