Github user yanboliang commented on the pull request:
https://github.com/apache/spark/pull/9183#issuecomment-150253984
Because of the multiple columns ```StringIndexer``` use ```Aggregate```
rather than ```countByValue``` to compute distinct value count, if two or more
values has the same count, it will has indeterminate order.
So 1) binary classification label column may be indexed to different
result(0, 1 or 1, 0); 2) ```OneHotEncoder``` will drop the last category in the
encoded vector by default, if there are more than one value can be drop, it
will indeterminate drop which one in this proposal.
I don't think we need to keep the order restriction produced by
```countByValue``` which may lead poor performance in the ```Aggregate```
implementation, so I disabled some test cases.
If my proposal work well, I can enable and update these test cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
