Re: Saving a pyspark.ml.feature.PCA model

[Yanbo Liang]

Sorry for the wrong link, what you should refer is jpmml-sparkml (
https://github.com/jpmml/jpmml-sparkml).

Thanks
Yanbo

2016-07-24 4:46 GMT-07:00 Yanbo Liang :

> Spark does not support exporting ML models to PMML currently. You can try
> the third party jpmml-spark (https://github.com/jpmml/jpmml-spark)
> package which supports a part of ML models.
>
> Thanks
> Yanbo
>
> 2016-07-20 11:14 GMT-07:00 Ajinkya Kale :
>
>> Just found Google dataproc has a preview of spark 2.0. Tried it and
>> save/load works! Thanks Shuai.
>> Followup question - is there a way to export the pyspark.ml models to
>> PMML ? If not, what is the best way to integrate the model for inference in
>> a production service ?
>>
>> On Tue, Jul 19, 2016 at 8:22 PM Ajinkya Kale 
>> wrote:
>>
>>> I am using google cloud dataproc which comes with spark 1.6.1. So
>>> upgrade is not really an option.
>>> No way / hack to save the models in spark 1.6.1 ?
>>>
>>> On Tue, Jul 19, 2016 at 8:13 PM Shuai Lin 
>>> wrote:
>>>
 It's added in not-released-yet 2.0.0 version.

 https://issues.apache.org/jira/browse/SPARK-13036
 https://github.com/apache/spark/commit/83302c3b

 so i guess you need to wait for 2.0 release (or use the current rc4).

 On Wed, Jul 20, 2016 at 6:54 AM, Ajinkya Kale 
 wrote:

> Is there a way to save a pyspark.ml.feature.PCA model ? I know mllib
> has that but mllib does not have PCA afaik. How do people do model
> persistence for inference using the pyspark ml models ? Did not find any
> documentation on model persistency for ml.
>
> --ajinkya
>


>

Re: Saving a pyspark.ml.feature.PCA model

[Yanbo Liang]

Spark does not support exporting ML models to PMML currently. You can try
the third party jpmml-spark (https://github.com/jpmml/jpmml-spark) package
which supports a part of ML models.

Thanks
Yanbo

2016-07-20 11:14 GMT-07:00 Ajinkya Kale :

> Just found Google dataproc has a preview of spark 2.0. Tried it and
> save/load works! Thanks Shuai.
> Followup question - is there a way to export the pyspark.ml models to
> PMML ? If not, what is the best way to integrate the model for inference in
> a production service ?
>
> On Tue, Jul 19, 2016 at 8:22 PM Ajinkya Kale 
> wrote:
>
>> I am using google cloud dataproc which comes with spark 1.6.1. So upgrade
>> is not really an option.
>> No way / hack to save the models in spark 1.6.1 ?
>>
>> On Tue, Jul 19, 2016 at 8:13 PM Shuai Lin  wrote:
>>
>>> It's added in not-released-yet 2.0.0 version.
>>>
>>> https://issues.apache.org/jira/browse/SPARK-13036
>>> https://github.com/apache/spark/commit/83302c3b
>>>
>>> so i guess you need to wait for 2.0 release (or use the current rc4).
>>>
>>> On Wed, Jul 20, 2016 at 6:54 AM, Ajinkya Kale 
>>> wrote:
>>>
 Is there a way to save a pyspark.ml.feature.PCA model ? I know mllib
 has that but mllib does not have PCA afaik. How do people do model
 persistence for inference using the pyspark ml models ? Did not find any
 documentation on model persistency for ml.

 --ajinkya

>>>
>>>

Re: Saving a pyspark.ml.feature.PCA model

[Ajinkya Kale]

Just found Google dataproc has a preview of spark 2.0. Tried it and
save/load works! Thanks Shuai.
Followup question - is there a way to export the pyspark.ml models to PMML
? If not, what is the best way to integrate the model for inference in a
production service ?

On Tue, Jul 19, 2016 at 8:22 PM Ajinkya Kale  wrote:

> I am using google cloud dataproc which comes with spark 1.6.1. So upgrade
> is not really an option.
> No way / hack to save the models in spark 1.6.1 ?
>
> On Tue, Jul 19, 2016 at 8:13 PM Shuai Lin  wrote:
>
>> It's added in not-released-yet 2.0.0 version.
>>
>> https://issues.apache.org/jira/browse/SPARK-13036
>> https://github.com/apache/spark/commit/83302c3b
>>
>> so i guess you need to wait for 2.0 release (or use the current rc4).
>>
>> On Wed, Jul 20, 2016 at 6:54 AM, Ajinkya Kale 
>> wrote:
>>
>>> Is there a way to save a pyspark.ml.feature.PCA model ? I know mllib has
>>> that but mllib does not have PCA afaik. How do people do model persistence
>>> for inference using the pyspark ml models ? Did not find any documentation
>>> on model persistency for ml.
>>>
>>> --ajinkya
>>>
>>
>>

Re: Saving a pyspark.ml.feature.PCA model

[Ajinkya Kale]

I am using google cloud dataproc which comes with spark 1.6.1. So upgrade
is not really an option.
No way / hack to save the models in spark 1.6.1 ?

On Tue, Jul 19, 2016 at 8:13 PM Shuai Lin  wrote:

> It's added in not-released-yet 2.0.0 version.
>
> https://issues.apache.org/jira/browse/SPARK-13036
> https://github.com/apache/spark/commit/83302c3b
>
> so i guess you need to wait for 2.0 release (or use the current rc4).
>
> On Wed, Jul 20, 2016 at 6:54 AM, Ajinkya Kale 
> wrote:
>
>> Is there a way to save a pyspark.ml.feature.PCA model ? I know mllib has
>> that but mllib does not have PCA afaik. How do people do model persistence
>> for inference using the pyspark ml models ? Did not find any documentation
>> on model persistency for ml.
>>
>> --ajinkya
>>
>
>

Re: Saving a pyspark.ml.feature.PCA model

[Shuai Lin]

It's added in not-released-yet 2.0.0 version.

https://issues.apache.org/jira/browse/SPARK-13036
https://github.com/apache/spark/commit/83302c3b

so i guess you need to wait for 2.0 release (or use the current rc4).

On Wed, Jul 20, 2016 at 6:54 AM, Ajinkya Kale  wrote:

> Is there a way to save a pyspark.ml.feature.PCA model ? I know mllib has
> that but mllib does not have PCA afaik. How do people do model persistence
> for inference using the pyspark ml models ? Did not find any documentation
> on model persistency for ml.
>
> --ajinkya
>

Saving a pyspark.ml.feature.PCA model

[Ajinkya Kale]

Is there a way to save a pyspark.ml.feature.PCA model ? I know mllib has
that but mllib does not have PCA afaik. How do people do model persistence
for inference using the pyspark ml models ? Did not find any documentation
on model persistency for ml.

--ajinkya