[
https://issues.apache.org/jira/browse/SPARK-16319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-16319.
-------------------------------
Resolution: Not A Problem
I am also not sure if/where the DAG condition is checked. In any event it's
required, and I guess it's possible to violate that by setting a circular
dependence of input/output cols and estimators, in a way it's not possible with
RDDs (?) I don't think the text should be removed. I also don't see why you say
in/out columns are ignored. Of course that's not true or else how would
anything work? nor is it true that stages are executed serially; stages may
execute in parallel.
> Non-linear (DAG) pipelines need better explanation
> --------------------------------------------------
>
> Key: SPARK-16319
> URL: https://issues.apache.org/jira/browse/SPARK-16319
> Project: Spark
> Issue Type: Documentation
> Components: ML
> Affects Versions: 2.0.0
> Reporter: Max Moroz
> Priority: Minor
>
> There's a
> [paragraph|http://spark.apache.org/docs/2.0.0-preview/ml-guide.html#details]
> about non-linear pipeline in the ML docs, but it's not clear how DAG pipeline
> differs from a linear pipeline, and in fact, it seems that a "DAG Pipeline"
> results in the behavior identical to that of a regular linear pipeline (the
> stages are simply applied in the order provided when the pipeline is
> created). In addition, no checks of input and output columns seem to occur
> when the pipeline.fit() or pipeline.transform() is called.
> It would be better to clarify in the docs and/or remove that paragraph.
> I'd be happy to write it up, but I have no idea what the intention of this
> concept is at this point.
> [Additional reference on
> SO|http://stackoverflow.com/questions/37541668/non-linear-dag-ml-pipelines-in-apache-spark]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
