Github user MLnick commented on the pull request:
https://github.com/apache/spark/pull/1671#issuecomment-51371929
Fair enough.
Standard approach for same token appearing multiple times in a training
example is additive, so it would act the same as HashingTF computation for
text (and as a one-hot style encoder for categorical features, and just
normally for real features).
To deal with collisions, typically take signed features by either (1) apply
two hashes, one of which determines sign (cf VW), or (2) use just one
signed hash function and take sign of hash value as sign of feature (cf
scikit-learn). The idea being that collisions cancel each other out.
On Wed, Aug 6, 2014 at 7:01 PM, Xiangrui Meng <[email protected]>
wrote:
> @MLnick <https://github.com/MLnick> FeatureHasher might be too general.
> One thing it is not clear from its name is how to resolve conflicts: same
> word appears more than once, or different words having the same hash
value.
> We can add FeatureHasher (or called HashingVectorizer) later. I think it
> might be useful to keep HashingTF as a special case.
>
> â
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/1671#issuecomment-51364477>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
