[twitter-dev] Re: from:user and since_id breaking Search API
I'm *not* using since_id, but I'm sporadically getting warning
messages about it. Most of the time, my queries work.
d:\curl http://search.twitter.com/search.json?q=
%23jquerylang=enpage=1show_user=truerpp=15
{
results: [{
profile_image_url: http://a1.twimg.com/profile_images/353780044/
avatar_normal.png,
created_at: Thu, 29 Oct 2009 00:05:02 +,
from_user: 2twittdotcom,
to_user_id: null,
text: create a simple infinite carousel with #jquery
http://2twitt.com/3u7;,
id: 5244850622,
from_user_id: 33556519,
geo: null,
iso_language_code: en,
source: lt;a href=quot;http://apiwiki.twitter.com/quot;
rel=quot;nofollowquot;gt;APIlt;/agt;
}],
max_id: 5244850622,
since_id
: 5056020580,
refresh_url: ?since_id=5244850622q=%23jquery,
results_per_page: 15,
page: 1,
completed_in: 0.02854,
warning: adjusted since_id to 5056020580 (2009-10-22 00:00:00
UTC), requested since_id was older than allowed,
query: %23jquery
}
It's very annoying. Any ideas?
[twitter-dev] Re: from:user and since_id breaking Search API
I am seeing issues with twitter search using since_id. The search results the query returns is not correct for atleast few hours. I think, some synchronization happens every certain number of hours. Also, if since_id is old, it expires. I don't find it to be that useful. Thanks, Arjun. The synchronization is very On Oct 23, 8:06 am, Marc W marcwanchipm...@gmail.com wrote: Hello Damon! I'm not 100% sure I buy this explanation: 1. This problem wasn't happening a day or two ago. 2. I tried executing the query on the command line, and incremented the since_id by 1 maybe 8-10x ... it just doesn't return any results. Even weirder is that if I wait 20 minutes, and execute the same query with the same original since_id, then I might get some of the results, but not all of them. Currently, the only solution I can see is to simply never use since_id and just filter out - on the client - those tweetids I've seen already. Seems like a horrible waste of bandwidth and computing power, and especially strange given that this largely worked a few days ago, right before some other changes were rolled out that also seemed to cause a sudden increase in 500-series errors being returned from the servers and other weirdnesses. Thanks, Marc. On Oct 23, 9:31 pm, Damon Clinkscales sca...@pobox.com wrote: Christopher To my recollection, for search with since_id to work properly, the tweet id must be in the search index. In this case: http://search.twitter.com/search?q=+from%3Asilent_tester02 does not yield the Dinner, movie, drinks. tweet in the index. As an aside, I did an exact match search on that phrase above and it returned many results that are not exact matches. But that's a separate issue. You could file an issue about the fact that the results coming back are not always consistent, but the first thing I would do is make sure that I am using a since_id that actually exists in the search index. Granted this can be a bit of a pain to verify this 100% of the time because sometimes tweets do not end up in the search index (which appears to be the case here). But in my experience, most of the time, they do. So as a test, pick a tweet you know is in the index and make some calls with it over a period of time. See if the results are consistent. Best, -damon --http://twitter.com/damon On Tue, Oct 20, 2009 at 9:06 AM, Christopher Warren christopher.war...@gmail.com wrote: We have an app that runs searches regularly, and recently stopped receiving new tweets. After investigating we found a search combination that seems to break the search API. Instead of getting a response with no tweets, an .atom request errors and a .json request 404s. http://search.twitter.com/search.json?q=from:silent_tester02since_id... http://search.twitter.com/search.atom?q=from:silent_tester02since_id... Changing the query to not use from:username works as expect, but I've put several usernames in and they all respond the same way. I haven't managed to narrow down the cause of the problem much further than that, but we're handling it in our code by rescuing any failed searches and appending since: with the date of the most recent tweet to the q. Any thoughts on what might be causing this would be appreciated.
[twitter-dev] Re: from:user and since_id breaking Search API
We are experiencing the same thing as Marc. We are not using from: though. But intermittently it gets tweets in significant number. If I try to get the same set again using the since_id for this particular crawl, I again get no tweets. Any Idea on this issue? Thanks. On Oct 23, 10:48 am, Marc W marcwanchipm...@gmail.com wrote: We've started seeing this too. If we specify a since_id, then periodic refetches will frequently return no new tweets (but sometimes they will return new ones). If we simply drop the since_id, then all new tweets are fetched. Some new server optimization thing gone wrong? Thanks! On Oct 20, 10:06 pm, Christopher Warren christopher.war...@gmail.com wrote: We have an app that runssearchesregularly, and recently stopped receiving newtweets. After investigating we found a search combination that seems to break the search API. Instead of getting a response withnotweets, an .atom request errors and a .json request 404s. http://search.twitter.com/search.json?q=from:silent_tester02since_id.. Changing the query to not use from:username works as expect, but I've put several usernames in and they all respond the same way. I haven't managed to narrow down the cause of the problem much further than that, but we're handling it in our code by rescuing any failedsearchesand appending since: with the date of the most recent tweet to the q. Any thoughts on what might be causing this would be appreciated.
[twitter-dev] Re: from:user and since_id breaking Search API
Christopher To my recollection, for search with since_id to work properly, the tweet id must be in the search index. In this case: http://search.twitter.com/search?q=+from%3Asilent_tester02 does not yield the Dinner, movie, drinks. tweet in the index. As an aside, I did an exact match search on that phrase above and it returned many results that are not exact matches. But that's a separate issue. You could file an issue about the fact that the results coming back are not always consistent, but the first thing I would do is make sure that I am using a since_id that actually exists in the search index. Granted this can be a bit of a pain to verify this 100% of the time because sometimes tweets do not end up in the search index (which appears to be the case here). But in my experience, most of the time, they do. So as a test, pick a tweet you know is in the index and make some calls with it over a period of time. See if the results are consistent. Best, -damon -- http://twitter.com/damon On Tue, Oct 20, 2009 at 9:06 AM, Christopher Warren christopher.war...@gmail.com wrote: We have an app that runs searches regularly, and recently stopped receiving new tweets. After investigating we found a search combination that seems to break the search API. Instead of getting a response with no tweets, an .atom request errors and a .json request 404s. http://search.twitter.com/search.json?q=from:silent_tester02since_id=4979161317 http://search.twitter.com/search.atom?q=from:silent_tester02since_id=4979161317 Changing the query to not use from:username works as expect, but I've put several usernames in and they all respond the same way. I haven't managed to narrow down the cause of the problem much further than that, but we're handling it in our code by rescuing any failed searches and appending since: with the date of the most recent tweet to the q. Any thoughts on what might be causing this would be appreciated.
[twitter-dev] Re: from:user and since_id breaking Search API
I'm getting the same issue as Marc and harshavs too. I'm using since_id to start a search from the last set of resuts but generally get 0 results most of the time, even though the same search without since_id yields lots of results. I could start logging tweet IDs against user IDs, and then work backwards down the list of past results until I get a since_id that yields results, but it does seem like a bit of a pain. This is my first time using the search API so I can't even tell you when this started happening. It just happens! For info: search is using negation operators, OR and from: Dave On Oct 23, 10:29 am, harshavs harsh...@gmail.com wrote: We are experiencing the same thing as Marc. We are not using from: though. But intermittently it gets tweets in significant number. If I try to get the same set again using the since_id for this particular crawl, I again get no tweets. Any Idea on this issue? Thanks. On Oct 23, 10:48 am, Marc W marcwanchipm...@gmail.com wrote: We've started seeing this too. If we specify a since_id, then periodic refetches will frequently return no new tweets (but sometimes they will return new ones). If we simply drop the since_id, then all new tweets are fetched. Some new server optimization thing gone wrong? Thanks! On Oct 20, 10:06 pm, Christopher Warren christopher.war...@gmail.com wrote: We have an app that runssearchesregularly, and recently stopped receiving newtweets. After investigating we found a search combination that seems to break the search API. Instead of getting a response withnotweets, an .atom request errors and a .json request 404s. http://search.twitter.com/search.json?q=from:silent_tester02since_id.. Changing the query to not use from:username works as expect, but I've put several usernames in and they all respond the same way. I haven't managed to narrow down the cause of the problem much further than that, but we're handling it in our code by rescuing any failedsearchesand appending since: with the date of the most recent tweet to the q. Any thoughts on what might be causing this would be appreciated.
