Github user ajaysaini725 commented on a diff in the pull request:
https://github.com/apache/spark/pull/18742#discussion_r130741050
--- Diff: python/pyspark/ml/util.py ---
@@ -283,3 +289,124 @@ def numFeatures(self):
Returns the number of features the model was trained on. If
unknown, returns -1
"""
return self._call_java("numFeatures")
+
+
+@inherit_doc
+class DefaultParamsWritable(MLWritable):
+
+ # overrides the write() function in MLWriteable
+ # users call .save() in MLWriteable which calls this write() function
and then calls
+ # the .save() in DefaultParamsWriter
+ # this can be overridden to return a different Writer (ex.
OneVsRestWriter as seen in Scala)
+ def write(self):
+ # instance of check for params?
+ return DefaultParamsWriter(self)
+
+
+@inherit_doc
+class DefaultParamsWriter(MLWriter):
+
+ def __init__(self, instance):
+ super(DefaultParamsWriter, self).__init__()
+ self.instance = instance
+ self.sc = SparkContext._active_spark_context
+
+ # if a model extends DefaultParamsWriteable this save() function is
called
+ def save(self, path):
+ if self.shouldOverwrite:
+ # This command removes a file. Is this enough?
+ os.remove(path)
+ DefaultParamsWriter.save_metadata(self.instance, path, self.sc)
+
--- End diff --
This implementation exists on the Scala side. However, in Python MLWriter
does not even define the saveImpl() function and instead says that the user
should override the save() function. I'm not entirely sure which is best to do
in Python but as of now everything in pyspark overrides save() instead of
saveImpl() so it is probably best to just stick with that pattern.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
