Github user MLnick commented on the issue:
https://github.com/apache/spark/pull/11601
The reason we didn't support mode was partly due to time and mostly due to
not being certain about the performance (e.g. if mode was called on a
non-categorical double column it could become quite inefficient). I think the
main idea was to try to either detect categorical features with metadata, or
limit categorical features to int/long type. The latter though is not ideal
since most transformers may output categorical features as double
(`StringIndexer` for example).
The reasoning behind not supporting vector was mostly performance and some
complexity issues (IIRC it would require some internal amendments to colstats
and approx quantile e.g.
https://github.com/apache/spark/pull/11601#issuecomment-207279502).
There is not time pressure on this now, so we could work multi-column
and/or Vector support into this PR. Multi-column should be fairly
straightforward I think.
I'd be fine with either (a) supporting multiple numeric columns now (doing
vector later); (b) sticking with what we have but making it future-proof if
possible.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
