Kristian Rickert created OPENNLP-1845:
-----------------------------------------
Summary: DocumentCategorizerDL softmax is numerically unstable
(overflows to NaN for large logits)
Key: OPENNLP-1845
URL: https://issues.apache.org/jira/browse/OPENNLP-1845
Project: OpenNLP
Issue Type: Bug
Components: dl
Reporter: Kristian Rickert
Assignee: Kristian Rickert
h2. Summary
{{DocumentCategorizerDL.softmax}} exponentiates the raw logits directly,
without first subtracting the maximum. For a sufficiently large logit
{{Math.exp(logit)}}
overflows to {{{}+Infinity{}}}, and the subsequent {{Infinity / Infinity}}
produces {{NaN}} category scores.
It also truncates each result to {{float}} before widening it back to
{{{}double{}}}, discarding precision.
h2. Details
Current implementation:
{code:java}
private double[] softmax(final float[] input) {
final double[] t = new double[input.length];
double sum = 0.0;
for (int x = 0; x < input.length; x++) {
double val = Math.exp(input[x]); // no max subtraction -> overflows for
large logits
sum += val;
t[x] = val;
}
final double[] output = new double[input.length];
for (int x = 0; x < output.length; x++) {
output[x] = (float) (t[x] / sum); // gratuitous float truncation
}
return output;
}
{code}
For a model whose pre-softmax logits are small (e.g. the sentiment model used
by the eval tests) no overflow occurs, which is why the existing results look
correct. Models with larger logits return {{NaN}} scores instead of a valid
distribution.
h2. Fix
Use the standard numerically stable softmax — subtract the maximum logit before
exponentiating (mathematically identical to the naive form, but overflow-safe)
— and keep {{double}} precision throughout:
{code:java}
static double[] softmax(final float[] input) {
double max = Double.NEGATIVE_INFINITY;
for (final float value : input) {
max = Math.max(max, value);
}
final double[] t = new double[input.length];
double sum = 0.0;
for (int x = 0; x < input.length; x++) {
final double val = Math.exp(input[x] - max);
sum += val;
t[x] = val;
}
final double[] output = new double[input.length];
for (int x = 0; x < output.length; x++) {
output[x] = t[x] / sum;
}
return output;
}
{code}
h2. Minor cleanups (same class)
While in {{{}DocumentCategorizerDL{}}}, two small same-class cleanups are
included:
* Rewrite {{tokenize()}} to advance an explicit index in a {{while}} loop
instead of mutating the {{{}for{}}}-loop counter inside the body (identical
chunking/overlap behavior, just clearer).
* Fix the {{"Unload to perform..."}} -> {{"Unable to perform..."}} log message
typo and add a Javadoc note that {{categorize()}} returns an empty array (and
logs) when inference fails, documenting the retained historical behavior.
h2. Backward compatibility
Public API is unchanged. For models that did not overflow (small logits),
scores change only at the float-vs-double rounding level (~1e-7), which is
within the existing {{DocumentCategorizerDLEval}} tolerance of {{{}1e-6{}}};
the pinned eval values should still hold (confirm against the data-dir model,
or regenerate if needed). For models that previously overflowed, scores change
from {{NaN}} to a valid distribution.
h2. Testing
Added unit tests for {{softmax}} ({{{}DocumentCategorizerDLTest{}}}):
* uniform distribution for equal logits and sum-to-one,
* numerical stability for large logits (the previous code returned {{NaN}}
here),
* a reference distribution ({{{}softmax([1,2,3]){}}}).
{{mvn -pl opennlp-core/opennlp-ml/opennlp-dl test}} passes (28 tests).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
