[
https://issues.apache.org/jira/browse/OPENNLP-515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18082413#comment-18082413
]
Richard Zowalla commented on OPENNLP-515:
-----------------------------------------
Might picking this up and want to align on the design before implementing,
since the ticket
leaves a key ambiguity that affects the serialization formats.
h3. Current state
The native POS format ({{{}word_TAG{}}}) hardcodes the underscore in two roles
at once:
* As the word/tag delimiter: {{POSSample.parse(...)}} splits each whitespace
token on
{{lastIndexOf("_")}} (token = before, tag = after).
* As the de-facto MWE glue, by the convention this ticket describes: a
multi-word
expression like "traffic light" is written {{traffic_light}} so it becomes a
single
token, i.e. {{{}traffic_light_NN{}}}.
{{POSSample.toString()}} mirrors this, always emitting {{{}token + "_" +
tag{}}}.
Because {{_}} plays both roles, {{traffic_light_NN}} is ambiguous in principle
(one MWE
token vs. "traffic" + "light_NN"); {{lastIndexOf}} only resolves the trailing
tag. That
overload is exactly what makes "specify the MWE separator" non-trivial.
h3. Design question
What should a configurable MWE separator actually do on read/write? I see three
options:
# *Decode to spaces* – treat the separator purely as a serialization encoding
of
intra-token whitespace. Read {{traffic_light_NN}} -> token {{{}"traffic
light"{}}}, tag
{{{}NN{}}}; write the reverse. Most faithful to "MWE = whitespace-separated
words", but it
_changes today's behavior_ (currently the token stays {{{}traffic_light{}}}).
# *Keep the token literal, make the word/tag delimiter configurable* – token
stays
{{{}traffic_light{}}}; you'd instead pick a non-underscore delimiter (e.g.
{{{}traffic_light/NN{}}}) so underscores inside an MWE no longer collide. Fully
backward compatible (default {{{}_{}}}).
# *Both* – a configurable MWE separator (space-decoding, opt-in) plus a
configurable
word/tag delimiter; defaults reproduce current behavior exactly.
h3. Open points for discussion
* {*}Backward compatibility{*}: should the no-flags default stay byte-for-byte
identical to
today's output, or are we OK changing the default serialization? We could for
3.0.0 but that would break 2.x users, so might not what we want.
* {*}Scope{*}: the ticket lists Chunker, Doccat, Name Finder, Parser and the
CLI in addition
to the POS tagger. Do we want one cross-cutting change, or land the POS tagger
first as
the reference implementation and have the others follow the same pattern? The
separator
is most clearly meaningful for the token-based formats (POS / Chunker / Name
Finder);
it's less obvious what it means for Doccat and Parser (imho)
* {*}Where the parameter lives{*}: as a CLI/format-stream parameter only, or
also surfaced on
the API (e.g. on {{POSSample}} / the sample streams) so API- and CLI-trained
models are
reproducible from the same data – which is one of the motivations called out in
the
description. From my POV, this should also be in the API, so it can be used
programmatically.
My current preference is option 2 (or 3 with an opt-in default) to preserve
backward
compatibility, but keen to hear what others think before we start here.
> Request for multi-words expressions (MWE) support in serialization formats
> --------------------------------------------------------------------------
>
> Key: OPENNLP-515
> URL: https://issues.apache.org/jira/browse/OPENNLP-515
> Project: OpenNLP
> Issue Type: Improvement
> Components: Chunker, Command Line Interface, Doccat, Name Finder,
> Parser, POS Tagger
> Affects Versions: tools-1.5.3
> Reporter: Nicolas Hernandez
> Priority: Major
>
> Multi-words expressions (MWE) are expressions with whitespace-separated words
> like "traffic light", "in order to", "two thousand and one", "Jules Verne"...
> So far, by using the CLI to train a model (in particular a POS model), there
> was no way to specify what is a simple or a multi-word expressions.
> By convention, users use the underscore character to concat the words of MWE
> and make MWE a token.
> Consequently a model trained by the API on the same data can be distinct
> since this preprocessing is not required.
> We need to offer to the users the possibility to set by parameter in the CLI
> what is the MWE separator char sequence.
> This concerns both trainers and labelers.
> A default MWE separator should be specified which will be used when
> serializing data with MWEs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
