[
https://issues.apache.org/jira/browse/SPARK-20099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15942082#comment-15942082
]
Joseph K. Bradley commented on SPARK-20099:
-------------------------------------------
Linking [SPARK-15574] since it brought up a need for transformSchema in
pyspark.ml as well
> Add transformSchema to pyspark.ml
> ---------------------------------
>
> Key: SPARK-20099
> URL: https://issues.apache.org/jira/browse/SPARK-20099
> Project: Spark
> Issue Type: Improvement
> Components: ML, PySpark
> Affects Versions: 2.1.0
> Reporter: Joseph K. Bradley
>
> Python's ML API currently lacks the PipelineStage abstraction. This
> abstraction's main purpose is to provide transformSchema() for checking for
> early failures in a Pipeline.
> As mentioned in https://github.com/apache/spark/pull/17218 it would also be
> useful in Python for checking Params in Python wrapper for Scala
> implementations; in these, transformSchema would involve passing Params in
> Python to Scala, which would then be able to validate the Param values. This
> could prevent late failures from bad Param settings in Pipeline execution,
> while still allowing us to check Param values on only the Scala side.
> This issue is for adding transformSchema() to pyspark.ml. If it's
> reasonable, we could create a PipelineStage abstraction. But it'd probably
> be fine to add transformSchema() directly to Transformer and Estimator,
> rather than creating PipelineStage.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
