Github user yanboliang commented on the pull request:
https://github.com/apache/spark/pull/9183#issuecomment-150577787
Because of the multiple columns ```StringIndexer``` use ```Aggregate```
rather than ```countByValue``` to compute distinct value count, if two or more
values has the same count, it will has indeterminate order.
So
1) binary classification label column may be indexed to different result(0,
1 or 1, 0);
2) ```OneHotEncoder``` will drop the last category in the encoded vector by
default, if there are more than one value can be drop, it will indeterminate
drop which one in this proposal.
I don't think we need to keep the order restriction produced by
```countByValue``` which may lead poor performance in the ```Aggregate```
implementation, so I disabled some test cases. If my proposal work well, I can
enable and update these test cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
