Cyanny created SPARK-22872:
------------------------------
Summary: Spark ML Pipeline Model Persistent Support Save Schema
Info
Key: SPARK-22872
URL: https://issues.apache.org/jira/browse/SPARK-22872
Project: Spark
Issue Type: IT Help
Components: ML
Affects Versions: 2.2.0
Reporter: Cyanny
Priority: Minor
Hi all,
I recently did a research about pmml, and my project need to transform many
model types to pmml files.
Moreover, JPMML(https://github.com/jpmml/jpmml-sparkml) provides many tools to
do that. I need to provide a uniform API for user, the API arguments are the
less the better.
I came with a issue that, sklearn, tensorflow, and lightgbm can produce only
one model file, including schema info and model data info.
but Spark PipelineModel only export a model file in parquet, there is no schema
info in the model file. And JPMML-SPARK needs two arguments: Schema and
PipelineModel
*Can spark PipelineModel include input data schema when export to a file? *
I found a solution, use dataframe API to export schema:
dataframe.limit(1).write.format("parquet").save("./model.schema")
*Are there any solutions to get the PipelineModel input schema?*
The situations about machine learning libraries to jpmml are as follows:
!http://i65.tinypic.com/2q1dxtd.jpg!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
