Kristian Rickert created OPENNLP-1839:
-----------------------------------------
Summary: DocumentCategorizerDL: close ONNX resources and fail
clearly on vocabulary misses
Key: OPENNLP-1839
URL: https://issues.apache.org/jira/browse/OPENNLP-1839
Project: OpenNLP
Issue Type: Bug
Reporter: Kristian Rickert
Assignee: Kristian Rickert
Follow-up to OPENNLP-1836 / PR #1072.
DocumentCategorizerDL already uses the correct single-segment BERT encoding
(attention_mask=1, token_type_ids=0), but it still shares two problems fixed
in SentenceVectorsDL:
h3. 1. Native-memory leak
In {{categorize()}}, each inference iteration creates {{OnnxTensor}} inputs and
calls {{session.run(inputs)}} without closing either the tensors or the
{{OrtSession.Result}}. SentenceVectorsDL (OPENNLP-1836) now closes both in a
try/finally block; the same pattern should be applied here.
Relevant code: {{DocumentCategorizerDL.categorize()}} (~lines 146–163).
h3. 2. NPE on vocabulary miss
In {{tokenize()}}, token ids are resolved with:
{code}
ids[x] = vocab.get(tokens[x]);
{code}
A missing token yields {{null}} unboxing to {{int}} → {{NullPointerException}}.
SentenceVectorsDL now throws {{IllegalArgumentException}} with a descriptive
message; apply the same handling here.
Relevant code: {{DocumentCategorizerDL.tokenize()}} (~lines 305–307).
h3. Proposal
* Close all {{OnnxTensor}} instances and {{OrtSession.Result}} after each
inference call (mirror SentenceVectorsDL).
* Replace the NPE with {{IllegalArgumentException}} naming the missing token.
* Add unit tests where feasible (mock or minimal ONNX session not required for
the vocab-miss path).
h3. Backport
Consider a 2.x maintenance backport of the leak + error-handling fixes only
(no vector-output change — doccat encoding was already correct).
h3. Links
* OPENNLP-1836: https://issues.apache.org/jira/browse/OPENNLP-1836
* PR #1072: https://github.com/apache/opennlp/pull/1072
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
