[
https://issues.apache.org/jira/browse/SPARK-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14959978#comment-14959978
]
Yanbo Liang edited comment on SPARK-8418 at 10/16/15 1:26 AM:
--------------------------------------------------------------
[~josephkb] I don't think RFormula is the best way to resolve this issue
because it still use the pipeline chained transformers one by one to encode
multiple columns which is low performance.
I vote for strategy 2 of [~nburoojy] proposed. But I think we don't need to
reimplement all transformers to support a multi-value implementation because of
some feature transformers not needed.
Brief design doc:
* How input and output columns will be specified
/** @group setParam */
def setInputCols(value: Array[String]): this.type = set(inputCols, value)
/** @group setParam */
def setOutputCols(value: Array[String]): this.type = set(outputCols, value)
* Schema validation
Make transformSchema adaptive to multiple input and output columns.
* Code sharing to reduce duplication
For backwards compatibility, we must not modify current Params, we add a new
one for multiple inputs (and check for conflicting settings when running).
Reimplement transformers to support multi-value implementation and make the
single-value interface a trivial invocation of the multi-value code. I think we
should maximum reuse the transform function of a single-value to implement the
multi-value one, but it can not completely shared code depends on different
transformers.
I will firstly try to start sub-tasks with StringIndexer and OneHotEncoder
which is mostly common used.
was (Author: yanboliang):
[~josephkb] I don't think RFormula is the best way to resolve this issue
because it still use the pipeline chained transformers one by one to encode
multiple columns which is low performance.
I vote for strategy 2 of [~nburoojy] proposed. But I think we don't need to
reimplement all transformers to support a multi-value implementation because of
some feature transformers not needed.
Brief design doc:
* How input and output columns will be specified
/** @group setParam */
def setInputCols(value: Array[String]): this.type = set(inputCols, value)
/** @group setParam */
def setOutputCols(value: Array[String]): this.type = set(outputCols, value)
* Schema validation
Make transformSchema adaptive to multiple input and output columns.
* Code sharing to reduce duplication
For backwards compatibility, we must not modify current Params, we add a new
one for multiple inputs (and check for conflicting settings when running).
Reimplement transformers to support multi-value implementation and make the
single-value interface a trivial invocation of the multi-value code. I think we
should maximum reuse the transform function of a single-value to implement the
multi-value one.
I will firstly try to start sub-tasks with StringIndexer and OneHotEncoder
which is mostly common used.
> Add single- and multi-value support to ML Transformers
> ------------------------------------------------------
>
> Key: SPARK-8418
> URL: https://issues.apache.org/jira/browse/SPARK-8418
> Project: Spark
> Issue Type: Sub-task
> Components: ML
> Reporter: Joseph K. Bradley
>
> It would be convenient if all feature transformers supported transforming
> columns of single values and multiple values, specifically:
> * one column with one value (e.g., type {{Double}})
> * one column with multiple values (e.g., {{Array[Double]}} or {{Vector}})
> We could go as far as supporting multiple columns, but that may not be
> necessary since VectorAssembler could be used to handle that.
> Estimators under {{ml.feature}} should also support this.
> This will likely require a short design doc to describe:
> * how input and output columns will be specified
> * schema validation
> * code sharing to reduce duplication
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
