uschindler commented on PR #16680: URL: https://github.com/apache/lucene/pull/16680#issuecomment-5715804091
I think both of you have good opinions. The problem of the type attribute is that it only allows one type. In current code of analyzers the original type is copied over. As you see in the test, the tokenizer adds "word" as type (maybe different for other tokenizers, e.g. Japanese). The old code cloned all attributes, so the sub words also get "word" as type. The good idea here is that additional (generated tokens) get a new type assigned, e.g "subword". That helps with generic filters added later like those for payloads. It is not ideal, but it helps to classify later token additions. So in my opinion, this is fine with type attribute. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
