Re: Twitter analyser
Hi,
This is what I've tried:
https://gist.github.com/anonymous/7383104
So far so good except that something is definitely wrong in my code as the
synonym is not emitted as a valid token it seems. This is how my indexing
analyzer is built:
private static final class MyIndexAnalyzer extends ReusableAnalyzerBase {
@Override
protected TokenStreamComponents createComponents(String fieldName,
Reader reader) {
final Tokenizer tokenizer = new
WhitespaceTokenizer(Version.LUCENE_36, reader);
final TwitterFilter twitterFilter = new
TwitterFilter(tokenizer);
final LowerCaseFilter filter = new
LowerCaseFilter(Version.LUCENE_36, twitterFilter);
return new TokenStreamComponents(tokenizer, filter);
}
}
I am expecting the lower filter to pick up the synonym exactly the same way
as the original token but it does not. If I have the following tweet Bla
Bla #SomeTAG, #sometag matches but sometag does not. All other use
cases not involving a case mismatch work as expected.
Does anyone knows what's wrong in my code?
Thanks for the support!
S.
On Tue, Nov 5, 2013 at 2:17 PM, Erick Erickson erickerick...@gmail.comwrote:
If your universe of items you want to match this way is small,
consider something akin to synonyms. Your indexing process
emits two tokens, with and without the @ or # which should
cover your situation.
FWIW,
Erick
On Tue, Nov 5, 2013 at 2:40 AM, Stéphane Nicoll
stephane.nic...@gmail.comwrote:
Hi,
I am building an application that indexes tweet and offer some basic
search facilities on them.
I am trying to find a combination where the following would work:
* foo matches the foo word, a mention (@foo) or the hashtag (#foo)
* @foo only matches the mention
* #foo matches only the hashtag
It should matches complete word so I used the WhiteSpaceAnalyzer for
indexing.
Any recommendation for this use case?
Thanks !
S.
Sent from my iPhone
-
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
Re: Twitter analyser
Replying to self: silly me. I am obviously creating the array with the
wrong length.
final String term = new String(buffer, 1, length);
should be replaced by
final String term = new String(buffer, 1, length -1);
and the silly trim can go away. I guess I need more coffee.
S.
On Sat, Nov 9, 2013 at 9:45 AM, Stephane Nicoll
stephane.nic...@gmail.comwrote:
Hi,
This is what I've tried:
https://gist.github.com/anonymous/7383104
So far so good except that something is definitely wrong in my code as the
synonym is not emitted as a valid token it seems. This is how my indexing
analyzer is built:
private static final class MyIndexAnalyzer extends ReusableAnalyzerBase {
@Override
protected TokenStreamComponents createComponents(String fieldName,
Reader reader) {
final Tokenizer tokenizer = new
WhitespaceTokenizer(Version.LUCENE_36, reader);
final TwitterFilter twitterFilter = new
TwitterFilter(tokenizer);
final LowerCaseFilter filter = new
LowerCaseFilter(Version.LUCENE_36, twitterFilter);
return new TokenStreamComponents(tokenizer, filter);
}
}
I am expecting the lower filter to pick up the synonym exactly the same
way as the original token but it does not. If I have the following tweet
Bla Bla #SomeTAG, #sometag matches but sometag does not. All other
use cases not involving a case mismatch work as expected.
Does anyone knows what's wrong in my code?
Thanks for the support!
S.
On Tue, Nov 5, 2013 at 2:17 PM, Erick Erickson erickerick...@gmail.comwrote:
If your universe of items you want to match this way is small,
consider something akin to synonyms. Your indexing process
emits two tokens, with and without the @ or # which should
cover your situation.
FWIW,
Erick
On Tue, Nov 5, 2013 at 2:40 AM, Stéphane Nicoll
stephane.nic...@gmail.comwrote:
Hi,
I am building an application that indexes tweet and offer some basic
search facilities on them.
I am trying to find a combination where the following would work:
* foo matches the foo word, a mention (@foo) or the hashtag (#foo)
* @foo only matches the mention
* #foo matches only the hashtag
It should matches complete word so I used the WhiteSpaceAnalyzer for
indexing.
Any recommendation for this use case?
Thanks !
S.
Sent from my iPhone
-
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
Re: Twitter analyser
This is a parts-of-speech analyzer for tweets. It would make your index far more useful. http://www.ark.cs.cmu.edu/TweetNLP/ On 11/04/2013 11:40 PM, Stéphane Nicoll wrote: Hi, I am building an application that indexes tweet and offer some basic search facilities on them. I am trying to find a combination where the following would work: * foo matches the foo word, a mention (@foo) or the hashtag (#foo) * @foo only matches the mention * #foo matches only the hashtag It should matches complete word so I used the WhiteSpaceAnalyzer for indexing. Any recommendation for this use case? Thanks ! S. Sent from my iPhone - To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
Re: Twitter analyser
If your universe of items you want to match this way is small, consider something akin to synonyms. Your indexing process emits two tokens, with and without the @ or # which should cover your situation. FWIW, Erick On Tue, Nov 5, 2013 at 2:40 AM, Stéphane Nicoll stephane.nic...@gmail.comwrote: Hi, I am building an application that indexes tweet and offer some basic search facilities on them. I am trying to find a combination where the following would work: * foo matches the foo word, a mention (@foo) or the hashtag (#foo) * @foo only matches the mention * #foo matches only the hashtag It should matches complete word so I used the WhiteSpaceAnalyzer for indexing. Any recommendation for this use case? Thanks ! S. Sent from my iPhone - To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
Re: Twitter analyser
Hi, Thanks for the reply. It's an index with tweets so any word really is a target for this. This would mean a significant increase of the index. My volumes are really small so that shouldn't be a problem (but performance/scalability is a concern). I have the control over the query. Another solution would be to translate a query on foo to foo or #foo or @foo WDYT? Thanks! S. On Tue, Nov 5, 2013 at 2:17 PM, Erick Erickson erickerick...@gmail.comwrote: If your universe of items you want to match this way is small, consider something akin to synonyms. Your indexing process emits two tokens, with and without the @ or # which should cover your situation. FWIW, Erick On Tue, Nov 5, 2013 at 2:40 AM, Stéphane Nicoll stephane.nic...@gmail.comwrote: Hi, I am building an application that indexes tweet and offer some basic search facilities on them. I am trying to find a combination where the following would work: * foo matches the foo word, a mention (@foo) or the hashtag (#foo) * @foo only matches the mention * #foo matches only the hashtag It should matches complete word so I used the WhiteSpaceAnalyzer for indexing. Any recommendation for this use case? Thanks ! S. Sent from my iPhone - To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
Re: Twitter analyser
You have to get the values _into_ the index with the special characters, that's where the issue is. Depending on your analysis chain special characters may or may not be even in your index to search in the first place. So it's not how many different words are after the special characters as much as how many special characters there are. So what I'm thinking is that as you index documents, you detect #foo, #blah, #whatever and index #foo, foo, #blah, blah etc. If all you have to do is specially handle tokens that start with just a few different chars it's not very hard. FWIW, Erick On Tue, Nov 5, 2013 at 8:33 AM, Stephane Nicoll stephane.nic...@gmail.comwrote: Hi, Thanks for the reply. It's an index with tweets so any word really is a target for this. This would mean a significant increase of the index. My volumes are really small so that shouldn't be a problem (but performance/scalability is a concern). I have the control over the query. Another solution would be to translate a query on foo to foo or #foo or @foo WDYT? Thanks! S. On Tue, Nov 5, 2013 at 2:17 PM, Erick Erickson erickerick...@gmail.com wrote: If your universe of items you want to match this way is small, consider something akin to synonyms. Your indexing process emits two tokens, with and without the @ or # which should cover your situation. FWIW, Erick On Tue, Nov 5, 2013 at 2:40 AM, Stéphane Nicoll stephane.nic...@gmail.comwrote: Hi, I am building an application that indexes tweet and offer some basic search facilities on them. I am trying to find a combination where the following would work: * foo matches the foo word, a mention (@foo) or the hashtag (#foo) * @foo only matches the mention * #foo matches only the hashtag It should matches complete word so I used the WhiteSpaceAnalyzer for indexing. Any recommendation for this use case? Thanks ! S. Sent from my iPhone - To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
Re: Twitter analyser
You can specify custom character types with the word delimiter filter, so you could define @ and # as digit and set SPLIT_ON_NUMERICS. This would cause @foo to tokenize as two adjacent terms, ditto for #foo. Unfortunately, A user name or tag that starts with a digit would not tokenize as desired, but that seems uncommon. foo would match all three since the @ or # would tokenize as a separate term. Use: public WordDelimiterFilter(TokenStream in, byte[] charTypeTable, int configurationFlags, CharArraySet protWords) See: http://lucene.apache.org/core/4_5_0/analyzers-common/org/apache/lucene/analysis/miscellaneous/WordDelimiterFilter.html -- Jack Krupansky -Original Message- From: Stéphane Nicoll Sent: Tuesday, November 05, 2013 2:40 AM To: java-user@lucene.apache.org Subject: Twitter analyser Hi, I am building an application that indexes tweet and offer some basic search facilities on them. I am trying to find a combination where the following would work: * foo matches the foo word, a mention (@foo) or the hashtag (#foo) * @foo only matches the mention * #foo matches only the hashtag It should matches complete word so I used the WhiteSpaceAnalyzer for indexing. Any recommendation for this use case? Thanks ! S. Sent from my iPhone - To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org - To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
