[jira] [Commented] (OPENNLP-1387) Fix alphaNumOpt in tokenizer example

[ASF GitHub Bot (Jira)]

    [ 
https://issues.apache.org/jira/browse/OPENNLP-1387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17623728#comment-17623728
 ] 

ASF GitHub Bot commented on OPENNLP-1387:
-----------------------------------------

atarora commented on code in PR #426:
URL: https://github.com/apache/opennlp/pull/426#discussion_r1004275029


##########
opennlp-docs/src/docbkx/tokenizer.xml:
##########
@@ -258,37 +258,39 @@ Arguments description:
                                To train the english tokenizer use the 
following command:
                                <screen>
                            <![CDATA[
-$ opennlp TokenizerTrainer -model en-token.bin -alphaNumOpt -lang en -data 
en-token.train -encoding UTF-8
+$ opennlp TokenizerTrainer -model en-token.bin -alphaNumOpt isAlphaNumOpt 
-lang en -data en-token.train -encoding UTF-8

Review Comment:
   Thank you for taking a notice to this @jzonthemtn , I certainly see the 
difference :
   
   `$opennlp TokenizerTrainer -model en-token-test.bin -alphaNumOpt true -lang 
en -data en-token.train -encoding UTF-8 -cutoff 5
   Indexing events with TwoPass using cutoff of 5
   
        Computing event counts...  done. 45 events
        Indexing...  done.
   Sorting and merging events... done. Reduced 45 events to 25.
   Done indexing in 0,09 s.
   Incorporating indexed data for training...
   done.
        Number of Event Tokens: 25
            Number of Outcomes: 2
          Number of Predicates: 18
   ...done.
   
   
   
   $opennlp TokenizerTrainer -model en-token-test.bin -alphaNumOpt false -lang 
en -data en-token.train -encoding UTF-8 -cutoff 5
   Indexing events with TwoPass using cutoff of 5
   
        Computing event counts...  done. 212 events
        Indexing...  done.
   Sorting and merging events... done. Reduced 212 events to 171.
   Done indexing in 0,12 s.
   Incorporating indexed data for training...
   done.
        Number of Event Tokens: 171
            Number of Outcomes: 2
          Number of Predicates: 75
   ...done.`
   
   Worth updating the doc.





> Fix alphaNumOpt in tokenizer example
> ------------------------------------
>
>                 Key: OPENNLP-1387
>                 URL: https://issues.apache.org/jira/browse/OPENNLP-1387
>             Project: OpenNLP
>          Issue Type: Task
>          Components: Documentation, Tokenizer
>    Affects Versions: 1.9.4, 2.0.0
>            Reporter: Jeff Zemerick
>            Assignee: Atita Arora
>            Priority: Major
>             Fix For: 2.0.1
>
>
> The example command for the tokenizer in the documentation is:
> {{opennlp TokenizerTrainer -model en-token.bin -alphaNumOpt -lang en -data 
> en-token.train -encoding UTF-8}}
> This is an invalid command because the alphaNumOpt parameter must have a 
> value. Running this command as-is will give the error "Number of parameters 
> must be always be even." 
> The documentation for alphaNumOpt specifies it must have a value:
> {quote}
> -alphaNumOpt isAlphaNumOpt
> Optimization flag to skip alpha numeric tokens for further tokenization
> {quote}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)