[
https://issues.apache.org/jira/browse/OPENNLP-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Wiesner updated OPENNLP-1839:
------------------------------------
Fix Version/s: 2.5.10
3.0.0-M4
> Fix native memory leak and vocabulary NPE in DocumentCategorizerDL
> ------------------------------------------------------------------
>
> Key: OPENNLP-1839
> URL: https://issues.apache.org/jira/browse/OPENNLP-1839
> Project: OpenNLP
> Issue Type: Bug
> Affects Versions: 2.5.9, 3.0.0-M3
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 2.5.10, 3.0.0-M4
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Follow-up to OPENNLP-1836 / PR #1072.
> DocumentCategorizerDL already uses the correct single-segment BERT encoding
> (attention_mask=1, token_type_ids=0), but it still shares two problems fixed
> in SentenceVectorsDL:
> h3. 1. Native-memory leak
> In {{categorize()}}, each inference iteration creates {{OnnxTensor}} inputs
> and
> calls {{session.run(inputs)}} without closing either the tensors or the
> {{OrtSession.Result}}. SentenceVectorsDL (OPENNLP-1836) now closes both in a
> try/finally block; the same pattern should be applied here.
> Relevant code: {{DocumentCategorizerDL.categorize()}} (~lines 146–163).
> h3. 2. NPE on vocabulary miss
> In {{tokenize()}}, token ids are resolved with:
> {code}
> ids[x] = vocab.get(tokens[x]);
> {code}
> A missing token yields {{null}} unboxing to {{int}} →
> {{NullPointerException}}.
> SentenceVectorsDL now throws {{IllegalArgumentException}} with a descriptive
> message; apply the same handling here.
> Relevant code: {{DocumentCategorizerDL.tokenize()}} (~lines 305–307).
> h3. Proposal
> * Close all {{OnnxTensor}} instances and {{OrtSession.Result}} after each
> inference call (mirror SentenceVectorsDL).
> * Replace the NPE with {{IllegalArgumentException}} naming the missing token.
> * Add unit tests where feasible (mock or minimal ONNX session not required for
> the vocab-miss path).
> h3. Backport
> Consider a 2.x maintenance backport of the leak + error-handling fixes only
> (no vector-output change — doccat encoding was already correct).
> h3. Links
> * OPENNLP-1836: https://issues.apache.org/jira/browse/OPENNLP-1836
> * PR #1072: https://github.com/apache/opennlp/pull/1072
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
