No, we don't have any. It should be possible to take the tokenizer training data as it is andevaluate on it.
+1 to add detokenizer evaluation Jörn On Sun, 2017-01-15 at 02:42 +0100, Richard Eckart de Castilho wrote: > On 14.01.2017, at 20:54, Joern Kottmann <kottm...@gmail.com> wrote: > > > > You can do that, we have a rule based detokeizer which can be used > > to > > produce training data from tokenized input. > > > > Have a look at the detokenizer in the tokenizer package. > > However, do you have any evaluation of the detokenizer? > > Cheers, > > -- Richard
