GitHub user VinceShieh opened a pull request:
https://github.com/apache/spark/pull/16883
[SPARK-17498][ML] enchance StringIndexer to handle unseen labels
## What changes were proposed in this pull request?
This PR is an enhancement to ML StringIndexer.
Before this PR, String Indexer only supports "skip"/"error" options to deal
with unseen records.
But those unseen records might still be useful and user would like to keep
the unseen labels in
certain use cases, This PR enables StringIndexer to support keeping unseen
labels as
indices [numLabels].
'''Before
StringIndexer().setHandleInvalid("skip")
StringIndexer().setHandleInvalid("error")
'''After
support the third option "keep"
StringIndexer().setHandleInvalid("keep")
## How was this patch tested?
Test added in StringIndexerSuite
Signed-off-by: VinceShieh <[email protected]>
(Please fill in changes proposed in this fix)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/VinceShieh/spark spark-17498
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16883.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16883
----
commit 30f3ba3e97d6ccbff9b3d37f7b5294e86a6ce68e
Author: VinceShieh <[email protected]>
Date: 2017-02-10T05:50:30Z
[SPARK-17498][ML] StringIndexer handles unseen labels
This PR is an enhancement to ML StringIndexer. Before this PR, String
Indexer only supports "skip"/"error" options
to deal with unseen records. But sometimes those unseen records might still
be useful in certain use cases, so user
would like to keep the unseen labels. This PR enables StringIndexer to
support keeping unseen labels as indices
[numLabels].
'''Before
StringIndexer().setHandleInvalid("skip")
StringIndexer().setHandleInvalid("error")
'''After
support the third option "keep"
StringIndexer().setHandleInvalid("keep")
Signed-off-by: VinceShieh <[email protected]>
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
