[
https://issues.apache.org/jira/browse/SPARK-8515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206111#comment-16206111
]
Timothy Hunter commented on SPARK-8515:
---------------------------------------
Before we commit to an implementation, we should think about the goal of adding
metadata in ML, because it comes with its own costs. For instance, there has
been a number of bug reports around them. See for example SPARK-2008,
SPARK-14862.
I see a couple of use cases for metadata:
- feature indexing -> that case should require just longs (or strings) for
each dimension of a feature vector
- expressing categorical info -> the Estimator -> Model -> Transformer pattern
is more appropriate, I believe
- vector dimensions -> I think that in all cases, the underlying code should
be able to proceed without this information, although this is debatable
> Improve ML attribute API
> ------------------------
>
> Key: SPARK-8515
> URL: https://issues.apache.org/jira/browse/SPARK-8515
> Project: Spark
> Issue Type: Improvement
> Components: ML
> Affects Versions: 1.4.0
> Reporter: Xiangrui Meng
> Labels: advanced
> Attachments: SPARK-8515.pdf
>
>
> In 1.4.0, we introduced ML attribute API to embed feature/label attribute
> info inside DataFrame's schema. However, the API is not very friendly to use.
> We should re-visit this API and see how we can improve it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
