[
https://issues.apache.org/jira/browse/OPENNLP-1189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Zemerick updated OPENNLP-1189:
-----------------------------------
Description:
The tokenizer training documentation for 1.8.4 states that "Tokens are either
separated by a whitespace or by a special <SPLIT> tag." However, it appears
that training files if the training data does not contain at least one <SPLIT>
tag. To reproduce:
Training on the sample data works fine:
{quote}Pierre Vinken<SPLIT>, 61 years old<SPLIT>, will join the board as a
nonexecutive director Nov. 29<SPLIT>.
Mr. Vinken is chairman of Elsevier N.V.<SPLIT>, the Dutch publishing
group<SPLIT>.
Rudolph Agnew<SPLIT>, 55 years old and former chairman of Consolidated Gold
Fields PLC<SPLIT>,
was named a nonexecutive director of this British industrial
conglomerate<SPLIT>.
{quote}
Replacing the <SPLIT> tags with whitespace causes the training to fail with
InsufficientTrainingDataException:
{quote}Pierre Vinken , 61 years old , will join the board as a nonexecutive
director Nov. 29 .
Mr. Vinken is chairman of Elsevier N.V. , the Dutch publishing group .
Rudolph Agnew , 55 years old and former chairman of Consolidated Gold Fields
PLC ,
was named a nonexecutive director of this British industrial conglomerate .
{quote}
Modifying the training data to contain a single <SPLIT> tag allows model
training to complete successfully:
{quote}Pierre Vinken<SPLIT>, 61 years old , will join the board as a
nonexecutive director Nov. 29 .
Mr. Vinken is chairman of Elsevier N.V. , the Dutch publishing group .
Rudolph Agnew , 55 years old and former chairman of Consolidated Gold Fields
PLC ,
was named a nonexecutive director of this British industrial conglomerate .
{quote}
was:
The tokenizer training documentation for 1.8.4 states that "Tokens are either
separated by a whitespace or by a special <SPLIT> tag." However, it appears
that training files if the training data does not contain at least one <SPLIT>
tag. To reproduce:
Training on the sample data works fine:
Pierre Vinken<SPLIT>, 61 years old<SPLIT>, will join the board as a
nonexecutive director Nov. 29<SPLIT>.
Mr. Vinken is chairman of Elsevier N.V.<SPLIT>, the Dutch publishing
group<SPLIT>.
Rudolph Agnew<SPLIT>, 55 years old and former chairman of Consolidated Gold
Fields PLC<SPLIT>,
was named a nonexecutive director of this British industrial
conglomerate<SPLIT>.
Replacing the <SPLIT> tags with whitespace causes the training to fail with
InsufficientTrainingDataException:
Pierre Vinken , 61 years old , will join the board as a nonexecutive director
Nov. 29 .
Mr. Vinken is chairman of Elsevier N.V. , the Dutch publishing group .
Rudolph Agnew , 55 years old and former chairman of Consolidated Gold Fields
PLC ,
was named a nonexecutive director of this British industrial conglomerate .
Modifying the training data to contain a single <SPLIT> tag causes training to
complete successfully:
Pierre Vinken<SPLIT>, 61 years old , will join the board as a nonexecutive
director Nov. 29 .
Mr. Vinken is chairman of Elsevier N.V. , the Dutch publishing group .
Rudolph Agnew , 55 years old and former chairman of Consolidated Gold Fields
PLC ,
was named a nonexecutive director of this British industrial conglomerate .
> Token model creation fails without at least one <SPLIT> tag
> -----------------------------------------------------------
>
> Key: OPENNLP-1189
> URL: https://issues.apache.org/jira/browse/OPENNLP-1189
> Project: OpenNLP
> Issue Type: Bug
> Components: Tokenizer
> Affects Versions: 1.8.4
> Reporter: Jeff Zemerick
> Priority: Major
>
> The tokenizer training documentation for 1.8.4 states that "Tokens are either
> separated by a whitespace or by a special <SPLIT> tag." However, it appears
> that training files if the training data does not contain at least one
> <SPLIT> tag. To reproduce:
> Training on the sample data works fine:
> {quote}Pierre Vinken<SPLIT>, 61 years old<SPLIT>, will join the board as a
> nonexecutive director Nov. 29<SPLIT>.
> Mr. Vinken is chairman of Elsevier N.V.<SPLIT>, the Dutch publishing
> group<SPLIT>.
> Rudolph Agnew<SPLIT>, 55 years old and former chairman of Consolidated Gold
> Fields PLC<SPLIT>,
> was named a nonexecutive director of this British industrial
> conglomerate<SPLIT>.
> {quote}
> Replacing the <SPLIT> tags with whitespace causes the training to fail with
> InsufficientTrainingDataException:
> {quote}Pierre Vinken , 61 years old , will join the board as a nonexecutive
> director Nov. 29 .
> Mr. Vinken is chairman of Elsevier N.V. , the Dutch publishing group .
> Rudolph Agnew , 55 years old and former chairman of Consolidated Gold Fields
> PLC ,
> was named a nonexecutive director of this British industrial conglomerate .
> {quote}
> Modifying the training data to contain a single <SPLIT> tag allows model
> training to complete successfully:
> {quote}Pierre Vinken<SPLIT>, 61 years old , will join the board as a
> nonexecutive director Nov. 29 .
> Mr. Vinken is chairman of Elsevier N.V. , the Dutch publishing group .
> Rudolph Agnew , 55 years old and former chairman of Consolidated Gold Fields
> PLC ,
> was named a nonexecutive director of this British industrial conglomerate .
> {quote}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
