howardhuanghua commented on issue #15142:
URL: https://github.com/apache/lucene/issues/15142#issuecomment-3322794033
Hi @jpountz , thanks for the checking, sorry for the delay.
> Apparently a tokenizer created a 15MB term. But nobody is ever going to
run a query on a 15MB term, so this term should not have been indexed in the
first place?
>
I think there might be a situation in the Elasticsearch log data scenario.
Term might be 5MB, 10MB, it's not reasonable to grow the buffer continusuly. I
think we need a mechanism to shrink the term buffer.
> StandardTokenizer protects against it via the maxTokenLength option, so
this likely isn't a `StandardTokenizer` at play. What is your tokenizer?
Here is the customed tokenizer:
```
"analyzer": {
"tinymsg_analyzer": {
"filter": [
"10k_char_length"
],
"type": "custom",
"tokenizer": "tinymsg_tokenizer"
}
},
"tokenizer": {
"tinymsg_tokenizer": {
"pattern": "[@&?|#()='\\\",;:<>\\[\\]{}/ \\n\\t\\r\\\\]",
"type": "pattern"
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
