[
https://issues.apache.org/jira/browse/LUCENE-9914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17318517#comment-17318517
]
Robert Muir commented on LUCENE-9914:
-------------------------------------
I don't think we should do this at all. We are generating a .jflex file so that
tokens get assigned appropriate type from the tokenizer. It isn't similar to
the domain name tokenizer at all, hence regenerating the standardtokenizer is
fast: around 600msec for me.
The UnicodeData really isn't a very efficient structure for storing this kind
of stuff: its just a sparsefixedbitset. It is better than a naive approach but
not as efficient as ICU datastructures (also in the JDK). I think we should try
to keep our usage of it limited.
> Modernize Emoji regeneration scripts
> ------------------------------------
>
> Key: LUCENE-9914
> URL: https://issues.apache.org/jira/browse/LUCENE-9914
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Dawid Weiss
> Assignee: Dawid Weiss
> Priority: Minor
>
> These are perl scripts... I don't think they had ant tasks in 8x and they
> haven't been used in a while. They don't seem too scary (for perl) - just
> fetch emoji unicode descriptions and parse them into a jflex macro and a test
> case.
> It'd be good to convert them to use python, groovy or even java so that they
> fit better in the build system. Alternatively - perhaps there is a way to get
> these codepoint properties from Java directly?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
