[
https://issues.apache.org/jira/browse/OPENNLP-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18082414#comment-18082414
]
Richard Zowalla commented on OPENNLP-1316:
------------------------------------------
Some thoughts and questions before we decide on scope.
First, a clarification of where this sits in the pipeline. Today's tokenizer
already
*splits* contractions PTB-style ({{{}don't{}}} -> {{do}} + {{{}n't{}}}) but
does not *expand*
them ({{{}n't{}}} stays {{{}n't{}}}, not {{{}not{}}}). So this is a new
normalization step, not a
tokenizer change. OpenNLP has a {{CharSequenceNormalizer}} SPI and a
{{{}DictionaryDetokenizer{}}}, but nothing that rewrites contractions.
h3. Discussion points
* {*}Use case / motivation{*}: what is the downstream goal –
negation/sentiment, search
normalization, training-data cleanup? This drives how much the ambiguous cases
matter
and whether a lossy rewrite is acceptable.
* {*}Ambiguity{*}: {{{}n't{}}}/{{{}'ve{}}}/{{{}'re{}}}/{{{}'ll{}}}/{{{}'m{}}}
are 1:1 and safe. {{'s}} (is / has /
possessive) and {{'d}} (had / would / did) are not. The NLTK-style answer is
"only
expand the unambiguous set"; I'd suggest we start there.
* {*}POS-based disambiguation is circular and often insufficient{*}: our POS
models are
trained on PTB-style tokens that are not expanded, and POS alone cannot
separate "he's
gone" (has) from "he's here" (is), nor reliably flag the possessive {{{}'s{}}}.
So
POS-driven expansion would be best-effort at most.
* {*}Reversibility / offsets{*}: expansion changes the surface form and
character spans, which
breaks alignment with the original text for NER, parsing and UIMA. Do we need to
preserve original offsets (e.g. a Span mapping), or is this purely a text->text
utility?
* {*}Language scope{*}: contraction rules are language-specific. Should this
be a pluggable,
resource-driven dictionary (consistent with our per-language model story)
rather than
hardcoded English?
* {*}Placement & API{*}: a new {{{}CharSequenceNormalizer{}}}, a token-list
transformer, or a
pre-tokenization text rewriter? Note expansion can change the token count,
which a
char-level normalizer does not normally do.
* {*}Licensing{*}: if we borrow a contraction list, it must be ASF-compatible
since we cannot
copy NLTK's data verbatim.
> Expand common contractions in the english language
> --------------------------------------------------
>
> Key: OPENNLP-1316
> URL: https://issues.apache.org/jira/browse/OPENNLP-1316
> Project: OpenNLP
> Issue Type: Question
> Reporter: Alan Wang
> Priority: Minor
>
> Hi, I want to know if OPENNLP needs to expand contractions.
> i.g. +_*n't*_+ -> _*not*_, +_*'ve*_+ -> _*have*_, +_*'m*_+ -> _*am*_, but
> +_*'s*_+ can be extended to _*is*_ or *_has_*, +_*'d*_+ can be extended to
> *_had_* or _*would*_, depending on the context.
> 1、Use POSTag to mark contractions to determine which extension is to be used.
> 2、Like nltk, extend only some contractions that are not ambiguous.
> Thanks!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
