[
https://issues.apache.org/jira/browse/SPARK-21542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16609495#comment-16609495
]
Peter Knight commented on SPARK-21542:
--------------------------------------
It would be really helpful to have some example code on how to use these.
I have tried:
{code}
from pyspark.ml import Transformer
from pyspark.ml.util import DefaultParamsReadable, DefaultParamsWritable
class MedianTrend(Transformer, DefaultParamsReadable, DefaultParamsWritable):
# code here to define Params and transform
# instantiate it
mt1 = MedianTrend(inputColList = ["v1"], outputColList = ["v1_trend_no_reset"],
sortCol = "date")
# then save andit
path1 = "test_MedianTrend"
mt1.write().overwrite().save(path1)
# then load it
mt1_loaded = mt1.load(path1)
df2 = mt1_loaded.transform(df)
df2.show()
{code}
This gives the following error:
{noformat}
'module' object has no attribute 'MedianTrend'{noformat}
> Helper functions for custom Python Persistence
> ----------------------------------------------
>
> Key: SPARK-21542
> URL: https://issues.apache.org/jira/browse/SPARK-21542
> Project: Spark
> Issue Type: New Feature
> Components: ML, PySpark
> Affects Versions: 2.2.0
> Reporter: Ajay Saini
> Assignee: Ajay Saini
> Priority: Major
> Fix For: 2.3.0
>
>
> Currently, there is no way to easily persist Json-serializable parameters in
> Python only. All parameters in Python are persisted by converting them to
> Java objects and using the Java persistence implementation. In order to
> facilitate the creation of custom Python-only pipeline stages, it would be
> good to have a Python-only persistence framework so that these stages do not
> need to be implemented in Scala for persistence.
> This task involves:
> - Adding implementations for DefaultParamsReadable, DefaultParamsWriteable,
> DefaultParamsReader, and DefaultParamsWriter in pyspark.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
