GitHub user VinceShieh opened a pull request: https://github.com/apache/spark/pull/16883
[SPARK-17498][ML] enchance StringIndexer to handle unseen labels ## What changes were proposed in this pull request? This PR is an enhancement to ML StringIndexer. Before this PR, String Indexer only supports "skip"/"error" options to deal with unseen records. But those unseen records might still be useful and user would like to keep the unseen labels in certain use cases, This PR enables StringIndexer to support keeping unseen labels as indices [numLabels]. '''Before StringIndexer().setHandleInvalid("skip") StringIndexer().setHandleInvalid("error") '''After support the third option "keep" StringIndexer().setHandleInvalid("keep") ## How was this patch tested? Test added in StringIndexerSuite Signed-off-by: VinceShieh <vincent....@intel.com> (Please fill in changes proposed in this fix) You can merge this pull request into a Git repository by running: $ git pull https://github.com/VinceShieh/spark spark-17498 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/16883.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16883 ---- commit 30f3ba3e97d6ccbff9b3d37f7b5294e86a6ce68e Author: VinceShieh <vincent....@intel.com> Date: 2017-02-10T05:50:30Z [SPARK-17498][ML] StringIndexer handles unseen labels This PR is an enhancement to ML StringIndexer. Before this PR, String Indexer only supports "skip"/"error" options to deal with unseen records. But sometimes those unseen records might still be useful in certain use cases, so user would like to keep the unseen labels. This PR enables StringIndexer to support keeping unseen labels as indices [numLabels]. '''Before StringIndexer().setHandleInvalid("skip") StringIndexer().setHandleInvalid("error") '''After support the third option "keep" StringIndexer().setHandleInvalid("keep") Signed-off-by: VinceShieh <vincent....@intel.com> ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org