[
https://issues.apache.org/jira/browse/OPENNLP-1845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kristian Rickert resolved OPENNLP-1845.
---------------------------------------
Resolution: Fixed
> DocumentCategorizerDL softmax is numerically unstable (overflows to NaN for
> large logits)
> -----------------------------------------------------------------------------------------
>
> Key: OPENNLP-1845
> URL: https://issues.apache.org/jira/browse/OPENNLP-1845
> Project: OpenNLP
> Issue Type: Bug
> Components: dl
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> h2. Summary
> {{DocumentCategorizerDL.softmax}} exponentiates the raw logits directly,
> without first subtracting the maximum. For a sufficiently large logit
> {{Math.exp(logit)}}
> overflows to {{{}+Infinity{}}}, and the subsequent {{Infinity / Infinity}}
> produces {{NaN}} category scores.
> It also truncates each result to {{float}} before widening it back to
> {{{}double{}}}, discarding precision.
> h2. Details
> Current implementation:
> {code:java}
> private double[] softmax(final float[] input) {
> final double[] t = new double[input.length];
> double sum = 0.0;
> for (int x = 0; x < input.length; x++) {
> double val = Math.exp(input[x]); // no max subtraction -> overflows for
> large logits
> sum += val;
> t[x] = val;
> }
> final double[] output = new double[input.length];
> for (int x = 0; x < output.length; x++) {
> output[x] = (float) (t[x] / sum); // gratuitous float truncation
> }
> return output;
> }
> {code}
> For a model whose pre-softmax logits are small (e.g. the sentiment model used
> by the eval tests) no overflow occurs, which is why the existing results look
> correct. Models with larger logits return {{NaN}} scores instead of a valid
> distribution.
> h2. Fix
> Use the standard numerically stable softmax — subtract the maximum logit
> before exponentiating (mathematically identical to the naive form, but
> overflow-safe) — and keep {{double}} precision throughout:
> {code:java}
> static double[] softmax(final float[] input) {
> double max = Double.NEGATIVE_INFINITY;
> for (final float value : input) {
> max = Math.max(max, value);
> }
> final double[] t = new double[input.length];
> double sum = 0.0;
> for (int x = 0; x < input.length; x++) {
> final double val = Math.exp(input[x] - max);
> sum += val;
> t[x] = val;
> }
> final double[] output = new double[input.length];
> for (int x = 0; x < output.length; x++) {
> output[x] = t[x] / sum;
> }
> return output;
> }
> {code}
> h2. Minor cleanups (same class)
> While in {{{}DocumentCategorizerDL{}}}, two small same-class cleanups are
> included:
> * Rewrite {{tokenize()}} to advance an explicit index in a {{while}} loop
> instead of mutating the {{{}for{}}}-loop counter inside the body (identical
> chunking/overlap behavior, just clearer).
> * Fix the {{"Unload to perform..."}} -> {{"Unable to perform..."}} log
> message typo and add a Javadoc note that {{categorize()}} returns an empty
> array (and logs) when inference fails, documenting the retained historical
> behavior.
> h2. Backward compatibility
> Public API is unchanged. For models that did not overflow (small logits),
> scores change only at the float-vs-double rounding level (~1e-7), which is
> within the existing {{DocumentCategorizerDLEval}} tolerance of {{{}1e-6{}}};
> the pinned eval values should still hold (confirm against the data-dir model,
> or regenerate if needed). For models that previously overflowed, scores
> change from {{NaN}} to a valid distribution.
> h2. Testing
> Added unit tests for {{softmax}} ({{{}DocumentCategorizerDLTest{}}}):
> * uniform distribution for equal logits and sum-to-one,
> * numerical stability for large logits (the previous code returned {{NaN}}
> here),
> * a reference distribution ({{{}softmax([1,2,3]){}}}).
> {{mvn -pl opennlp-core/opennlp-ml/opennlp-dl test}} passes (28 tests).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
