Kristian Rickert created OPENNLP-1846:
-----------------------------------------
Summary: Generalize NameFinderDL to recognize all entity types
(not only person)
Key: OPENNLP-1846
URL: https://issues.apache.org/jira/browse/OPENNLP-1846
Project: OpenNLP
Issue Type: Improvement
Components: dl
Reporter: Kristian Rickert
Assignee: Kristian Rickert
h2. Summary
{{NameFinderDL}} is hardcoded to recognize only the {{person}} entity type.
Although the caller supplies a full {{ids2Labels}} map, {{find()}} only acts on
the {{B-PER}}/{{I-PER}} labels and ignores every other
{{B-<TYPE>}}/{{I-<TYPE>}} the model emits (e.g. {{ORG}}, {{LOC}}, {{MISC}}). It
should decode all entity types the model was trained for.
A second, related defect: the resulting {{Span}} is constructed with the
matched *text* in the type slot ({{new Span(start, end, spanText,
confidence)}}), so {{Span.getType()}} returns the entity text rather than its
label.
h2. Current behavior
{code:java}
public static final String I_PER = "I-PER";
public static final String B_PER = "B-PER";
...
final String label = ids2Labels.get(maxIndex);
if (B_PER.equals(label)) { // only person is decoded
final SpanEnd spanEnd = findSpanEnd(...); // looks for I-PER only
...
spans.add(new Span(characterStart, characterEnd, spanText, confidence)); //
type = matched text
}
{code}
So a 4-class NER model such as {{dslim/bert-base-NER}} (PER/ORG/LOC/MISC)
returns only the person spans, each labelled with the covered text instead of
{{"PER"}}.
h2. Proposed change
Decode the BIO sequence generically:
* Begin a span on any label starting with {{B-}}; the entity type is the label
minus the {{B-}} prefix (e.g. {{B-ORG}} -> {{ORG}}).
* Extend the span while the following labels are {{I-<same type>}} (generalize
{{findSpanEnd}} from {{I-PER}} to {{I-<type>}}).
* Set {{Span.getType()}} to the decoded entity type (e.g. {{"PER"}},
{{"ORG"}}), not the matched text.
The {{B_PER}}/{{I_PER}} constants become unnecessary for the decode logic (may
be retained as documented examples). This makes {{ids2Labels}} fully drive
recognition for any BIO-tagged token-classification model.
h2. Backward compatibility
OpenNLP 3.0.0 is pre-release, so the behavioral change is acceptable. Notes:
* Models that emit only person labels keep working, now correctly labelled
{{"PER"}}.
* {{Span.getType()}} changes from the matched text to the entity label — this
is the intended/correct value and fixes the defect above.
* Multi-type models now return additional spans (ORG/LOC/MISC) that were
previously dropped. {{NameFinderDLEval}} currently pins person-only
expectations (e.g. exactly one span for "George Washington was president of the
United States."); those assertions must be updated to reflect the additional
entities and the corrected span types.
h2. Testing
* Update {{NameFinderDLEval}} to assert the full multi-type output (PER plus
ORG/LOC/MISC as applicable) and that {{Span.getType()}} holds the entity label.
* Add coverage for multi-token spans of non-person types and for adjacent spans
of different types.
h2. Downstream
This unblocks multi-type ONNX NER in the opennlp-grpc server: its DL name
finder wrapper already routes by entity type and only needs {{NameFinderDL}} to
emit them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
