Hi All,

We have been noticing gaps appearing in search results at times when doing geocoded searches in particular. For example with this search over South Eastern Australia :


Occasionally produces results with large gaps in the created_at for the tweets. For example, I just got these created_at for tweets returned :

Tue, 17 Nov 2009 22:59:50 +0000
Tue, 17 Nov 2009 22:59:49 +0000
Tue, 17 Nov 2009 22:59:48 +0000
Tue, 17 Nov 2009 22:59:43 +0000
Tue, 17 Nov 2009 22:52:04 +0000
Tue, 17 Nov 2009 22:52:04 +0000
Tue, 17 Nov 2009 22:51:34 +0000
Tue, 17 Nov 2009 22:50:21 +0000
Tue, 17 Nov 2009 22:50:20 +0000
Tue, 17 Nov 2009 22:43:37 +0000

This area is producing multiple tweets per second, but there are some gaps there many minutes long. A subsequent search 10's of seconds, to a few minutes later 'fills in these gaps' with many many more tweets from the periods in between these minutes-long gaps, confirming that the initial search was in fact sparse.

The same effect exists via the normal web search interface also.

It has previously been possible to just use the maximum id of the tweets from the previous search result set, and then only page through results until you saw that id again. But having the search results appear out of order means this method is not possible. It means you would have to search across all 15 pages x 100 rpp continuously in order to ensure something approaching a complete result set. Even then it will not always be possible if ~ 1500 of results 'appear at once'. This is not sustainable for both the app doing the searching, or from the point of view of the many additional requests that would have to hit Twitter's servers.

Is this a problem to be resolved, or moving forward are we going to continue to get results appearing via search out of order from their creation date like this?



Reply via email to