This could only make sense if the Streaming API supported "search engine
logic". Currently Streaming only supports keyword matching -- you have to
post-process to add additional predicate operators beyond OR. You can
reproduce the keyword match in a few lines of code, and the rest is
(currently) all up to you anyway. Just remember that a given tweet could
have triggered multiple predicates.

Beyond being a low priority feature, rendering and delivering custom
responses per user would be a performance risk. We currently can support a
very large number of filter clients per server, and we want to preserve this
performance.

-John Kalucki
http://twitter.com/jkalucki
Services, Twitter Inc.



On Sat, Dec 5, 2009 at 3:18 AM, Julien <julien.genest...@gmail.com> wrote:

> Thanks Dave,
>
> I think I get it from your example... yet, in our case, we have
> several thousands of keywords, and many many complex searches (with
> filter:, "and", "or", :near ... an so on).
>
> I keep thinking that instead of re-implementing on my side the search
> engine logic that Twitter has, it would be simpler for them to also
> send the macthing keywords. And even more elegant solution (yet
> slightly more complex) would be to be able to parse parameters along
> with the search I give, such as a unique search_id (that I can store
> on my side) and then, instead of giving me the matched keywords/search
> terms, they could just give me back that search_id. That would be
> something like this :
>
> Right now it is :
> POST  http://stream.twitter.com/1/statuses/filter.json
> track=paris,twitter+superfeedr,<http://stream.twitter.com/1/statuses/filter.json%0Atrack=paris,twitter+superfeedr,>"julien
> near:france"
>
> It would be awesome if I could do :
> POST  http://stream.twitter.com/1/statuses/filter.json
> track={"paris":"my_search_1","twitter
> +superfeedr":"my_search_2","julien near:france":"my_search_3"}
>
> And then, upon notifications, they would just pass me this search key
> my_search_xx
>
> I know and understand and implies a little bit of work for Twitter,
> but it also removes the pain from each susbcriber to this streaming
> API who has to re-implement again and again the "search engine" from
> Twitter.
>
>
>
>
>
>
> On Dec 4, 11:33 am, Dave Sherohman <d...@fishtwits.com> wrote:
> > On Thu, Dec 03, 2009 at 03:12:05PM -0800, Julien wrote:
> > > Well, then I'd need some help with that...
> >
> > > Again, it's easy with single search keywords, but I haven't found a
> > > solution for combined searches like twitter+stream or photo+Paris...
> > > because I would have to compare each combination of tokens in the
> > > tweet...
> >
> > > Can someone give more details.
> >
> > I don't mean to be flogging my site today, but take a look
> athttp://fishtwits.comfor the results I'm producing (just click the logo
> > at the top of the page to view the full site without logging in):  Any
> > tweets from users followed by FishTwits are scanned for fishing-related
> > terms and all such terms found in the tweet are displayed below it.  At
> > this moment, for instance, the first displayed tweet shows matches for
> > both "Fly Fishing" and "Sole".
> >
> > This is accomplished with the following Perl code (edited to remove
> > parts which aren't directly relevant):
> >
> > sub load_from_text {
> >   my ($class, $text) = @_;
> >
> >   unless($topic_regex) {
> >     require Regexp::Assemble;
> >     my $ra = Regexp::Assemble->new(
> >                chomp => 0,
> >                anchor_word_begin => 1,
> >                anchor_word_end => 1,
> >              );
> >     for my $topic (@topic_list) {
> >       $ra->add(lc $topic);
> >     }
> >     $topic_regex = $ra->re;
> >   }
> >
> >   $text = lc $text;
> >   my @topics = $text =~ /$topic_regex/g;
> >
> >   return sort @topics;
> >
> > }
> >
> > It first uses Regexp::Assemble to build a $topic_regex[1] which will
> > match any of the words/phrases found in the topic table, then does a
> > global match of $text (the body of the tweet being examined) against
> > $topic_regex, capturing all matches into the array @topics, which is
> > then sorted and returned to the caller.
> >
> > After the match is performed, @topics contains every search term which
> > is matched, no matter how many there may be, which should fill your
> > requirement for "combined searches", unless I'm misunderstanding it.
> >
> > If you mean you would want that "Fly Fishing", "Sole" tweet to return
> > three hits rather than two ("Fly Fishing", "Sole", "Fly Fishing+Sole"),
> > that's easy enough to create from @topics, just generate every
> > permutation of the terms which the individual tweet matched.
> >
> > [1]  If you're only dealing with 10 or so keywords, you'd probably be
> > just as well off building the regex by hand.  The main reason I'm using
> > Regexp::Assemble to do it on the fly is because manually creating and
> > then maintaining a regex that will efficiently match any of 1300 terms
> > would be a nightmare.
> >
> > --
> > Dave Sherohman
>

Reply via email to