Varun Bhandary created SPARK-59376:
--------------------------------------
Summary: Add frequency encoding to ml.feature
Key: SPARK-59376
URL: https://issues.apache.org/jira/browse/SPARK-59376
Project: Spark
Issue Type: New Feature
Components: ML
Affects Versions: 5.0.0
Reporter: Varun Bhandary
MLlib has no unsupervised encoder for high cardinality categorical features.
StringIndexer produces ordinals whose magnitude is meaningless to any model that
treats its features as numbers. OneHotEncoder is correct but adds a column per
category, which stops being practical after a few hundred. FeatureHasher avoids
the width at the cost of collisions and interpretability. TargetEncoder, added
in
4.0.0 by SPARK-37178, needs a label column, so it is not available for
unsupervised work at all.
That leaves clustering, anomaly detection and dimensionality reduction with no
good option for a column of, say, 50,000 merchant ids.
Frequency encoding fills that gap. Each category is replaced by how often it
occurs in the training data, as a proportion or as a raw count. It is one of the
standard treatments for high cardinality categoricals and it needs no label.
Proposed API, mirroring TargetEncoder so the two stay consistent:
FrequencyEncoder, an Estimator, producing FrequencyEncoderModel
inputCol / outputCol and inputCols / outputCols, pairs treated independently
handleInvalid, error or keep, where keep maps unseen categories to 0
normalize, default true, giving a proportion, or false for the raw count
numeric already indexed input, as TargetEncoder requires, so StringIndexer
feeds it
Fit is a single pass. The input columns are packed into an array and posexploded
so that one groupBy aggregates every column at once, which is the shape
SPARK-50267 gave TargetEncoder.fit. The fitted map is small, one entry per
category, and transform applies it with try_element_at against a literal, so
there is no join and no shuffle at transform time.
It is fair to point out that a user can write groupBy.count and a join for
themselves. The same is true of TargetEncoder, and in both cases the value is
not
the arithmetic. It is the fitted model semantics: the mapping is learned once
and
applied identically at train and at serving time, it survives save and load, it
composes inside a Pipeline, and unseen categories get defined behaviour instead
of silently becoming null.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
