vb-dbrks opened a new pull request, #58664:
URL: https://github.com/apache/spark/pull/58664
### What changes were proposed in this pull request?
This adds `FrequencyEncoder` and `FrequencyEncoderModel` to `ml.feature`.
Each category is
replaced by how often it occurs in the training data, as a proportion of the
training rows by
default or as a raw count when `normalize` is false.
The API mirrors `TargetEncoder` so the two stay consistent: an `Estimator`
and a `Model`,
`inputCol`/`outputCol` and `inputCols`/`outputCols` with pairs treated
independently,
`handleInvalid` of `error` or `keep`, and numeric already-indexed input so
`StringIndexer` feeds
it.
- `mllib/.../ml/feature/FrequencyEncoder.scala`, the estimator, model and
save/load
- `mllib/.../ml/feature/FrequencyEncoderSuite.scala`, 16 tests
- `mllib/.../ml/feature/JavaFrequencyEncoderSuite.java`, Java API coverage
- `python/pyspark/ml/feature.py`, the wrappers, plus `pyspark.ml.rst` and a
test in
`python/pyspark/ml/tests/test_feature.py`
- `docs/ml-features.md` and one example each in Scala, Java and Python
`fit` packs the input columns into an array and `posexplode`s it, so one
`groupBy` aggregates
every column rather than one shuffle per column, which is the shape
SPARK-50267 gave
`TargetEncoder.fit`. The fitted map holds one entry per category, and
`transform` applies it with
`try_element_at` against a literal, so encoding does no join and no shuffle.
Four places where mirroring `TargetEncoder` would have been wrong, called
out because a reviewer
comparing the two files will notice them:
- The output carries a `NumericAttribute` rather than a `NominalAttribute`.
An encoded frequency
is a continuous quantity and its ordering is the point of it.
- There is no unseen-category sentinel. Unseen categories encode to zero,
which is the frequency
actually observed for them, and the normalising denominator falls out of
summing the
per-category counts.
- `transform` short-circuits when a feature has no categories at all, which
is what a feature that
trained on nothing but nulls leaves behind. Indexing a map literal built
from an empty `Map` is a
needless analysis-time hazard.
- The model has no `setNormalize`. `normalize` only affects `fit`, so a
setter on a fitted model
would imply it could change encodings that are already computed.
`estimatedSize` accounts for the encodings rather than metadata alone, so
Spark Connect sizes the
model by what it actually carries, as SPARK-58279 did for the neighbouring
transformers.
### Why are the changes needed?
MLlib has no unsupervised encoder for high cardinality categorical features.
`StringIndexer` gives
ordinals whose magnitude means nothing to a model that reads its features as
numbers,
`OneHotEncoder` adds a column per category, `FeatureHasher` trades width for
collisions, and
`TargetEncoder` needs a label. That leaves clustering, anomaly detection and
dimensionality
reduction with no good option for a column of fifty thousand merchant ids.
Frequency encoding is one of the standard treatments for that case and it
needs no label.
It is fair to point out that a user can write `groupBy.count` and a join for
themselves. The same
is true of `TargetEncoder`, and in both cases the value is not the
arithmetic. It is the fitted
model semantics: the mapping is learned once and applied identically at
train and serving time, it
survives save and load, it composes inside a `Pipeline`, and unseen
categories get defined
behaviour instead of silently becoming null.
One property is worth stating plainly rather than leaving to be discovered:
categories that occur
equally often receive the same encoding. That is inherent to the technique,
not a limitation of
this implementation, so it is documented on the class, in `ml-features.md`,
and asserted in the
suite so it is not mistaken for a bug later.
### Does this PR introduce _any_ user-facing change?
Yes, a new feature. `FrequencyEncoder` and `FrequencyEncoderModel` are added
to `ml.feature` and to
the PySpark API. Nothing existing changes behaviour.
### How was this patch tested?
New tests, run locally against a full `mllib` build. `FrequencyEncoderSuite`
reports
`Tests: succeeded 16, failed 0`:
- `FrequencyEncoderSuite`, 16 tests, covering proportions and raw counts,
equally common categories
collapsing to one encoding, unseen values under both `keep` and `error`,
seen and unseen null
categories, missing and non-numeric input columns, non-indexed input,
default output column
names, a feature-count mismatch between model and params, and save/load
round trips for both the
estimator and the model in single and multi column configurations.
- `JavaFrequencyEncoderSuite`, exercising the Java API for proportions,
counts and unseen values.
- `test_frequency_encoder` in `python/pyspark/ml/tests/test_feature.py`,
plus doctests on the class.
Expected values are written as `count / total` rather than as reduced
fractions, deliberately:
`fit` computes `count / total`, and `1.0 / 3.0` is not guaranteed to be the
same double as
`3.0 / 9.0`.
`ruff check` and `ruff format --check` pass on every touched Python file
with the 0.14.8 pinned in
`dev/requirements.txt`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code 2.1.266 (Claude Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
