[
https://issues.apache.org/jira/browse/CASSANDRA-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16379662#comment-16379662
]
mck commented on CASSANDRA-14247:
---------------------------------
[~mkjellman], agree with all your points. I did do one profile run against an
infinite loop on the cities test csv file which is 1.3M, so it'll be easy to do
it again to validate your improvement in (2).
> SASI tokenizer for simple delimiter based entries
> -------------------------------------------------
>
> Key: CASSANDRA-14247
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14247
> Project: Cassandra
> Issue Type: Improvement
> Components: sasi
> Reporter: mck
> Assignee: mck
> Priority: Major
> Fix For: 4.0, 3.11.x
>
>
> Currently SASI offers only two tokenizer options:
> - NonTokenizerAnalyser
> - StandardAnalyzer
> The latter is built upon Snowball, powerful for human languages but overkill
> for simple tokenization.
> A simple tokenizer is proposed here. The need for this arose as a workaround
> of CASSANDRA-11182, and to avoid the disk usage explosion when having to
> resort to {{CONTAINS}}. See https://github.com/openzipkin/zipkin/issues/1861
> Example use of this would be:
> {code}
> CREATE CUSTOM INDEX span_annotation_query_idx
> ON zipkin2.span (annotation_query) USING
> 'org.apache.cassandra.index.sasi.SASIIndex'
> WITH OPTIONS = {
> 'analyzer_class':
> 'org.apache.cassandra.index.sasi.analyzer.DelimiterAnalyzer',
> 'delimiter': '░',
> 'case_sensitive': 'true',
> 'mode': 'prefix',
> 'analyzed': 'true'};
> {code}
> Original credit for this work goes to https://github.com/zuochangan
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
