[
https://issues.apache.org/jira/browse/SPARK-59376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Bhandary updated SPARK-59376:
-----------------------------------
Description:
MLlib has no unsupervised encoder for high cardinality categorical features.
StringIndexer produces ordinals whose magnitude means nothing to a model that
reads
its features as numbers. OneHotEncoder is correct but adds a column per
category,
which stops being practical after a few hundred. FeatureHasher avoids the width
by
accepting collisions and losing interpretability. TargetEncoder, added in 4.0.0
by
SPARK-37178, needs a label column, so it is unavailable for unsupervised work
entirely.
That leaves clustering, anomaly detection and dimensionality reduction with no
good
option for a column of fifty thousand merchant ids. That is a gap in the basics
rather than a missing exotic algorithm.
Frequency encoding fills it. Each category is replaced by how often it occurs
in the
training data, as a proportion of the training rows or as a raw count. No label
is
needed.
Why this belongs in Spark rather than in a local library
Fitting a frequency encoder is a full-data aggregation. A category's encoding
is its
count over every row, so unlike a sampling estimator there is no useful answer
to be
had from a subset. That is the work a groupBy does well: counting combines on
the map
side, so only per-partition per-category counts cross the network, and the
result is
one row per category however many rows went in. The physical plan for the shape
fit
uses, measured on a 100,000 row check that reduces to 5,000 result rows:
HashAggregate(keys=[index, value], functions=[partial_count(1)])
Exchange hashpartitioning(index, value, 200)
HashAggregate(keys=[index, value], functions=[count(1)])
The reduction happens before the shuffle rather than after it. Computing the
same
encoding outside Spark means bringing the whole column to a single machine
first,
which is the cost this avoids, and which grows with the data while the answer
does
not.
This is the same shape as TargetEncoder, whose fit is also a per-category
aggregation
over every row. It is worth contrasting with an estimator that genuinely does
not
need distributing: a sampling method that trains on a few hundred rows gains
nothing
from a cluster, whereas here the full pass is the computation.
As one concrete instance of the gap, DQX (https://github.com/databrickslabs/dqx)
implements frequency encoding itself for the categorical features feeding its
anomaly
detection, because ml.feature has no equivalent to call.
Proposed API, mirroring TargetEncoder so the two stay consistent
FrequencyEncoder, an Estimator, producing FrequencyEncoderModel
inputCol / outputCol and inputCols / outputCols, pairs treated independently
handleInvalid, error or keep, where keep encodes unseen categories as zero
normalize, default true for a proportion, false for the raw count
numeric already-indexed input, as TargetEncoder requires, so StringIndexer
feeds it
transform applies a map holding one entry per category with try_element_at
against a
literal, so encoding is a per-row projection with no join and no shuffle.
It is fair to say a user can write groupBy.count and a join themselves. The
same is
true of TargetEncoder, and in both cases the value is not the arithmetic. It is
the
fitted model semantics: the mapping is learned once and applied identically at
training and serving time, it survives save and load, it composes in a
Pipeline, and
unseen categories get defined behaviour instead of silently becoming null.
One property is worth stating up front rather than leaving to be discovered.
Categories occurring equally often receive the same encoding. That is inherent
to the
technique, so it is documented on the class, in ml-features.md, and asserted in
the
suite so it is not later mistaken for a bug.
was:
MLlib has no unsupervised encoder for high cardinality categorical features.
StringIndexer produces ordinals whose magnitude is meaningless to any model that
treats its features as numbers. OneHotEncoder is correct but adds a column per
category, which stops being practical after a few hundred. FeatureHasher avoids
the width at the cost of collisions and interpretability. TargetEncoder, added
in
4.0.0 by SPARK-37178, needs a label column, so it is not available for
unsupervised work at all.
That leaves clustering, anomaly detection and dimensionality reduction with no
good option for a column of, say, 50,000 merchant ids.
Frequency encoding fills that gap. Each category is replaced by how often it
occurs in the training data, as a proportion or as a raw count. It is one of the
standard treatments for high cardinality categoricals and it needs no label.
Proposed API, mirroring TargetEncoder so the two stay consistent:
FrequencyEncoder, an Estimator, producing FrequencyEncoderModel
inputCol / outputCol and inputCols / outputCols, pairs treated independently
handleInvalid, error or keep, where keep maps unseen categories to 0
normalize, default true, giving a proportion, or false for the raw count
numeric already indexed input, as TargetEncoder requires, so StringIndexer
feeds it
Fit is a single pass. The input columns are packed into an array and posexploded
so that one groupBy aggregates every column at once, which is the shape
SPARK-50267 gave TargetEncoder.fit. The fitted map is small, one entry per
category, and transform applies it with try_element_at against a literal, so
there is no join and no shuffle at transform time.
It is fair to point out that a user can write groupBy.count and a join for
themselves. The same is true of TargetEncoder, and in both cases the value is
not
the arithmetic. It is the fitted model semantics: the mapping is learned once
and
applied identically at train and at serving time, it survives save and load, it
composes inside a Pipeline, and unseen categories get defined behaviour instead
of silently becoming null.
> Add frequency encoding to ml.feature
> ------------------------------------
>
> Key: SPARK-59376
> URL: https://issues.apache.org/jira/browse/SPARK-59376
> Project: Spark
> Issue Type: New Feature
> Components: ML
> Affects Versions: 5.0.0
> Reporter: Varun Bhandary
> Priority: Major
> Labels: pull-request-available
>
> MLlib has no unsupervised encoder for high cardinality categorical features.
> StringIndexer produces ordinals whose magnitude means nothing to a model that
> reads
> its features as numbers. OneHotEncoder is correct but adds a column per
> category,
> which stops being practical after a few hundred. FeatureHasher avoids the
> width by
> accepting collisions and losing interpretability. TargetEncoder, added in
> 4.0.0 by
> SPARK-37178, needs a label column, so it is unavailable for unsupervised work
> entirely.
> That leaves clustering, anomaly detection and dimensionality reduction with
> no good
> option for a column of fifty thousand merchant ids. That is a gap in the
> basics
> rather than a missing exotic algorithm.
> Frequency encoding fills it. Each category is replaced by how often it occurs
> in the
> training data, as a proportion of the training rows or as a raw count. No
> label is
> needed.
> Why this belongs in Spark rather than in a local library
> Fitting a frequency encoder is a full-data aggregation. A category's encoding
> is its
> count over every row, so unlike a sampling estimator there is no useful
> answer to be
> had from a subset. That is the work a groupBy does well: counting combines on
> the map
> side, so only per-partition per-category counts cross the network, and the
> result is
> one row per category however many rows went in. The physical plan for the
> shape fit
> uses, measured on a 100,000 row check that reduces to 5,000 result rows:
> HashAggregate(keys=[index, value], functions=[partial_count(1)])
> Exchange hashpartitioning(index, value, 200)
> HashAggregate(keys=[index, value], functions=[count(1)])
> The reduction happens before the shuffle rather than after it. Computing the
> same
> encoding outside Spark means bringing the whole column to a single machine
> first,
> which is the cost this avoids, and which grows with the data while the answer
> does
> not.
> This is the same shape as TargetEncoder, whose fit is also a per-category
> aggregation
> over every row. It is worth contrasting with an estimator that genuinely does
> not
> need distributing: a sampling method that trains on a few hundred rows gains
> nothing
> from a cluster, whereas here the full pass is the computation.
> As one concrete instance of the gap, DQX
> (https://github.com/databrickslabs/dqx)
> implements frequency encoding itself for the categorical features feeding its
> anomaly
> detection, because ml.feature has no equivalent to call.
> Proposed API, mirroring TargetEncoder so the two stay consistent
> FrequencyEncoder, an Estimator, producing FrequencyEncoderModel
> inputCol / outputCol and inputCols / outputCols, pairs treated independently
> handleInvalid, error or keep, where keep encodes unseen categories as zero
> normalize, default true for a proportion, false for the raw count
> numeric already-indexed input, as TargetEncoder requires, so StringIndexer
> feeds it
> transform applies a map holding one entry per category with try_element_at
> against a
> literal, so encoding is a per-row projection with no join and no shuffle.
> It is fair to say a user can write groupBy.count and a join themselves. The
> same is
> true of TargetEncoder, and in both cases the value is not the arithmetic. It
> is the
> fitted model semantics: the mapping is learned once and applied identically at
> training and serving time, it survives save and load, it composes in a
> Pipeline, and
> unseen categories get defined behaviour instead of silently becoming null.
> One property is worth stating up front rather than leaving to be discovered.
> Categories occurring equally often receive the same encoding. That is
> inherent to the
> technique, so it is documented on the class, in ml-features.md, and asserted
> in the
> suite so it is not later mistaken for a bug.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
