becketqin commented on a change in pull request #11420:
[FLINK-16565][python][ml] Make Pipeline Json compitable between Java and Python
if all PipelineStages are Java ones
URL: https://github.com/apache/flink/pull/11420#discussion_r394119038
##########
File path: flink-python/pyflink/ml/api/base.py
##########
@@ -265,11 +266,57 @@ def transform(self, t_env: TableEnvironment, input:
Table) -> Table:
return input
def to_json(self) -> str:
- import jsonpickle
- return str(jsonpickle.encode(self, keys=True))
+ """
+ If all PipelineStages in this Pipeline are Java ones, this method will
return a
+ Java json string, which can be loaded either from a Python Pipeline or
a Java Pipeline,
+ otherwise, it returns a Python json string which can only be loaded
from a Python Pipeline.
+ """
+ # if all PipelineStages are Java ones, we use Java toJson() to
generate Json string
+ # so that the string can also be loaded from Java side.
+ if all([type(stage) in [JavaTransformer, JavaEstimator, JavaModel]
+ for stage in self.get_stages()]):
+ j_pipeline =
get_gateway().jvm.org.apache.flink.ml.api.core.Pipeline()
+ for stage in self.get_stages():
+ stage._convert_params_to_java(stage._j_obj)
+ j_pipeline.appendStage(stage._j_obj)
+ return j_pipeline.toJson()
+ else:
+ import jsonpickle
+ return str(jsonpickle.encode(self, keys=True))
def load_json(self, json: str) -> None:
- import jsonpickle
- pipeline = jsonpickle.decode(json, keys=True)
- for stage in pipeline.get_stages():
- self.append_stage(stage)
+ """
+ This method can either load from a Java Pipeline json or a Python
Pipeline json.
+ """
+ # noinspection PyBroadException
+ try:
+ # try to load json with Python method
+ import jsonpickle
+ pipeline = jsonpickle.decode(json, keys=True)
+ for stage in pipeline.get_stages():
+ self.append_stage(stage)
+ except Exception:
Review comment:
Do we know the exact exception type here? It would good to not naively catch
all the exception if possible. If we don't know the exact exception, we should
probably remember the exception and raise it when the Java Pipeline load also
failed. The error message would then be something like:
"Cannot load the JSON as either a Java Pipeline or a Python Pipeline. Python
Pipeline load failed due to THIS_EXCEPTION. Java Pipeline load failed due to
ANOTHER_EXCEPTION."
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
