deemoliu commented on PR #16364: URL: https://github.com/apache/pinot/pull/16364#issuecomment-3110193826
> Do you have a sample query? I had the same concern as @ankitsultana - this can explode the index. I thought its better to have FST on the tokens/dictionary and use that to find matching dictionary ids I think it depends on which N we planned to use and what's the size of the column. Based on use cases, usually we used 2grams, 3grams. Last year we did a benchmark on Tag matching using a derived column of 3-grams with inverted index to simulate this behavior. For example. a tag value "dca1001" can generate 3-grams values like `dca`, `ca1`, `a10`, `100`, `001` which stored in a multi-value inverted index column called my_column_ngram. For Query, if we match `REGEXP_LIKE(my_column, '.*dca.*')` the regex query can be to `SELECT * FROM my_table WHERE my_column_ngram = 'dca'. The inverted indexes on my_column_ngram multi-value field is sufficient to return the query result. if we match REGEXP_LIKE(my_column, '.*dca1.*'), we can do `SELECT * FROM my_table WHERE my_column_ngram = 'dca' and my_column_ngram = 'ca1' and REGEXP_LIKE(my_column, '.*dca1.*'). The ngrams match using inverted index executed first, pre-filtered rows to accelerate the query performance for regex matching. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
