Hi all, I have a search by location capture program running and have recently noticed some strange behavior. The program essentially calls the search api roughly every minute, adjusting the since_id parameter to only retrieve new tweets. This was working fine, retrieving a fairly constant number of tweets until sometime around the 19th of December, when I started to notice some strange bursts. Upon investigating this further, it appears that the bursts occur regularly at 15 and 45 minutes past the hour. Looking at the created_at timestamps on the tweets returned each minute, there appears to be an increasing lag between the current time and the most recent tweet returned. The search index then seems to refresh itself at 15 and 45 minutes past the hour and brings itself up to date, only then to slowly fall behind again. I have also been able to replicate this behavior by using the advanced search form and clicking the refresh link every so often. Another thing that I've noticed is that this problem only occurs when the search radius is greater than 35km. I'm not sure what the exact cut off is.
Here are some stats I gathered by searching with the following url: http://search.twitter.com/search.json?geocode=-33.867138%2C151.2071%2C100.0km&lang=all&rpp=100 (searching around Sydney, NSW with radius 100km) Mon Jan 31 14:02:11 EST 2011> queued 76 messages, timestamp range: 13:55:25 to 13:56:07, lag = 364 secs Mon Jan 31 14:03:12 EST 2011> queued 59 messages, timestamp range: 13:56:09 to 13:56:46, lag = 386 secs Mon Jan 31 14:04:13 EST 2011> queued 71 messages, timestamp range: 13:56:47 to 13:57:21, lag = 412 secs Mon Jan 31 14:05:15 EST 2011> queued 58 messages, timestamp range: 13:57:22 to 13:57:55, lag = 440 secs Mon Jan 31 14:06:15 EST 2011> queued 58 messages, timestamp range: 13:57:55 to 13:58:27, lag = 468 secs Mon Jan 31 14:07:16 EST 2011> queued 53 messages, timestamp range: 13:58:28 to 13:59:01, lag = 495 secs Mon Jan 31 14:08:17 EST 2011> queued 75 messages, timestamp range: 13:59:03 to 13:59:35, lag = 522 secs Mon Jan 31 14:09:18 EST 2011> queued 80 messages, timestamp range: 13:59:35 to 14:00:07, lag = 551 secs Mon Jan 31 14:10:19 EST 2011> queued 85 messages, timestamp range: 14:00:08 to 14:00:38, lag = 581 secs Mon Jan 31 14:11:19 EST 2011> queued 64 messages, timestamp range: 14:00:38 to 14:01:14, lag = 605 secs Mon Jan 31 14:12:20 EST 2011> queued 82 messages, timestamp range: 14:01:16 to 14:01:51, lag = 629 secs Mon Jan 31 14:13:21 EST 2011> queued 68 messages, timestamp range: 14:01:51 to 14:02:28, lag = 653 secs Mon Jan 31 14:14:22 EST 2011> queued 73 messages, timestamp range: 14:02:29 to 14:03:05, lag = 677 secs Mon Jan 31 14:15:38 EST 2011> queued 1500 messages, timestamp range: 14:03:39 to 14:15:38, lag = 0 secs Mon Jan 31 14:16:39 EST 2011> queued 74 messages, timestamp range: 14:15:38 to 14:16:10, lag = 29 secs Mon Jan 31 14:17:40 EST 2011> queued 79 messages, timestamp range: 14:16:11 to 14:16:54, lag = 46 secs Mon Jan 31 14:18:41 EST 2011> queued 81 messages, timestamp range: 14:16:56 to 14:17:36, lag = 65 secs Mon Jan 31 14:19:41 EST 2011> queued 82 messages, timestamp range: 14:17:36 to 14:18:18, lag = 83 secs Mon Jan 31 14:20:42 EST 2011> queued 71 messages, timestamp range: 14:18:18 to 14:18:58, lag = 104 secs Mon Jan 31 14:21:43 EST 2011> queued 73 messages, timestamp range: 14:18:59 to 14:19:38, lag = 125 secs Mon Jan 31 14:22:44 EST 2011> queued 63 messages, timestamp range: 14:19:38 to 14:20:15, lag = 149 secs Mon Jan 31 14:23:44 EST 2011> queued 55 messages, timestamp range: 14:20:16 to 14:20:54, lag = 170 secs Mon Jan 31 14:24:45 EST 2011> queued 64 messages, timestamp range: 14:20:55 to 14:21:34, lag = 191 secs Mon Jan 31 14:25:46 EST 2011> queued 77 messages, timestamp range: 14:21:36 to 14:22:15, lag = 211 secs Mon Jan 31 14:26:47 EST 2011> queued 85 messages, timestamp range: 14:22:16 to 14:22:58, lag = 229 secs Mon Jan 31 14:27:48 EST 2011> queued 89 messages, timestamp range: 14:22:59 to 14:23:40, lag = 248 secs Mon Jan 31 14:28:50 EST 2011> queued 75 messages, timestamp range: 14:23:42 to 14:24:21, lag = 269 secs Mon Jan 31 14:29:50 EST 2011> queued 83 messages, timestamp range: 14:24:22 to 14:25:03, lag = 287 secs Mon Jan 31 14:30:51 EST 2011> queued 77 messages, timestamp range: 14:25:03 to 14:25:42, lag = 309 secs Mon Jan 31 14:31:52 EST 2011> queued 59 messages, timestamp range: 14:25:44 to 14:26:19, lag = 333 secs Mon Jan 31 14:32:53 EST 2011> queued 62 messages, timestamp range: 14:26:19 to 14:26:57, lag = 356 secs Mon Jan 31 14:33:53 EST 2011> queued 74 messages, timestamp range: 14:26:58 to 14:27:35, lag = 378 secs Mon Jan 31 14:34:54 EST 2011> queued 42 messages, timestamp range: 14:27:37 to 14:28:08, lag = 406 secs Mon Jan 31 14:35:55 EST 2011> queued 62 messages, timestamp range: 14:28:10 to 14:28:44, lag = 431 secs Mon Jan 31 14:36:56 EST 2011> queued 72 messages, timestamp range: 14:28:46 to 14:29:18, lag = 458 secs Mon Jan 31 14:37:57 EST 2011> queued 64 messages, timestamp range: 14:29:20 to 14:29:54, lag = 483 secs Mon Jan 31 14:38:58 EST 2011> queued 82 messages, timestamp range: 14:29:55 to 14:30:29, lag = 509 secs Mon Jan 31 14:39:59 EST 2011> queued 62 messages, timestamp range: 14:30:30 to 14:31:05, lag = 534 secs Mon Jan 31 14:41:00 EST 2011> queued 71 messages, timestamp range: 14:31:06 to 14:31:42, lag = 558 secs Mon Jan 31 14:42:01 EST 2011> queued 55 messages, timestamp range: 14:31:42 to 14:32:21, lag = 580 secs Mon Jan 31 14:43:01 EST 2011> queued 78 messages, timestamp range: 14:32:22 to 14:33:01, lag = 600 secs Mon Jan 31 14:44:02 EST 2011> queued 64 messages, timestamp range: 14:33:02 to 14:33:39, lag = 623 secs Mon Jan 31 14:45:16 EST 2011> queued 1387 messages, timestamp range: 14:33:39 to 14:45:17, lag = 0 secs Mon Jan 31 14:46:16 EST 2011> queued 45 messages, timestamp range: 14:45:18 to 14:45:47, lag = 29 secs Mon Jan 31 14:47:17 EST 2011> queued 78 messages, timestamp range: 14:45:48 to 14:46:32, lag = 45 secs Mon Jan 31 14:48:17 EST 2011> queued 83 messages, timestamp range: 14:46:33 to 14:47:16, lag = 61 secs Mon Jan 31 14:49:18 EST 2011> queued 78 messages, timestamp range: 14:47:16 to 14:47:58, lag = 80 secs Mon Jan 31 14:50:19 EST 2011> queued 72 messages, timestamp range: 14:47:58 to 14:48:36, lag = 103 secs Mon Jan 31 14:51:20 EST 2011> queued 67 messages, timestamp range: 14:48:36 to 14:49:13, lag = 127 secs Mon Jan 31 14:52:21 EST 2011> queued 76 messages, timestamp range: 14:49:14 to 14:49:51, lag = 150 secs Mon Jan 31 14:53:22 EST 2011> queued 63 messages, timestamp range: 14:49:52 to 14:50:28, lag = 174 secs Mon Jan 31 14:54:23 EST 2011> queued 71 messages, timestamp range: 14:50:28 to 14:51:06, lag = 197 secs Mon Jan 31 14:55:24 EST 2011> queued 80 messages, timestamp range: 14:51:08 to 14:51:47, lag = 217 secs Mon Jan 31 14:56:25 EST 2011> queued 71 messages, timestamp range: 14:51:49 to 14:52:29, lag = 236 secs Mon Jan 31 14:57:25 EST 2011> queued 100 messages, timestamp range: 14:52:35 to 14:53:13, lag = 252 secs Mon Jan 31 14:58:26 EST 2011> queued 83 messages, timestamp range: 14:53:13 to 14:53:53, lag = 273 secs Mon Jan 31 14:59:27 EST 2011> queued 79 messages, timestamp range: 14:53:55 to 14:54:33, lag = 294 secs Mon Jan 31 15:00:28 EST 2011> queued 61 messages, timestamp range: 14:54:33 to 14:55:11, lag = 317 secs Mon Jan 31 15:01:29 EST 2011> queued 60 messages, timestamp range: 14:55:13 to 14:55:49, lag = 340 secs Mon Jan 31 15:02:33 EST 2011> queued 78 messages, timestamp range: 14:53:55 to 14:56:24, lag = 369 secs Mon Jan 31 15:03:34 EST 2011> queued 74 messages, timestamp range: 14:56:26 to 14:57:08, lag = 386 secs Mon Jan 31 15:04:34 EST 2011> queued 60 messages, timestamp range: 14:57:09 to 14:57:43, lag = 411 secs Mon Jan 31 15:05:35 EST 2011> queued 65 messages, timestamp range: 14:57:46 to 14:58:18, lag = 437 secs Mon Jan 31 15:06:36 EST 2011> queued 64 messages, timestamp range: 14:58:19 to 14:58:52, lag = 464 secs Mon Jan 31 15:07:37 EST 2011> queued 60 messages, timestamp range: 14:58:52 to 14:59:25, lag = 492 secs Mon Jan 31 15:08:37 EST 2011> queued 70 messages, timestamp range: 14:59:25 to 15:00:00, lag = 517 secs Mon Jan 31 15:09:39 EST 2011> queued 84 messages, timestamp range: 15:00:01 to 15:00:34, lag = 545 secs Mon Jan 31 15:10:39 EST 2011> queued 83 messages, timestamp range: 15:00:35 to 15:01:14, lag = 565 secs Mon Jan 31 15:11:40 EST 2011> queued 71 messages, timestamp range: 15:01:15 to 15:01:51, lag = 589 secs Mon Jan 31 15:12:41 EST 2011> queued 61 messages, timestamp range: 15:01:52 to 15:02:29, lag = 612 secs Mon Jan 31 15:13:42 EST 2011> queued 61 messages, timestamp range: 15:02:30 to 15:03:08, lag = 634 secs Mon Jan 31 15:14:53 EST 2011> queued 1295 messages, timestamp range: 15:03:08 to 15:14:56, lag = -2 secs Mon Jan 31 15:15:54 EST 2011> queued 143 messages, timestamp range: 15:14:57 to 15:16:08, lag = -13 secs Mon Jan 31 15:16:55 EST 2011> queued 38 messages, timestamp range: 15:16:09 to 15:16:27, lag = 28 secs Mon Jan 31 15:17:56 EST 2011> queued 81 messages, timestamp range: 15:16:28 to 15:17:23, lag = 33 secs Mon Jan 31 15:18:56 EST 2011> queued 69 messages, timestamp range: 15:17:25 to 15:18:07, lag = 49 secs Mon Jan 31 15:19:57 EST 2011> queued 78 messages, timestamp range: 15:18:08 to 15:18:50, lag = 67 secs Mon Jan 31 15:20:58 EST 2011> queued 65 messages, timestamp range: 15:18:51 to 15:19:33, lag = 85 secs Mon Jan 31 15:21:58 EST 2011> queued 75 messages, timestamp range: 15:19:34 to 15:20:15, lag = 103 secs Mon Jan 31 15:22:59 EST 2011> queued 65 messages, timestamp range: 15:20:16 to 15:20:55, lag = 124 secs Mon Jan 31 15:24:00 EST 2011> queued 64 messages, timestamp range: 15:20:56 to 15:21:38, lag = 142 secs Mon Jan 31 15:25:01 EST 2011> queued 86 messages, timestamp range: 15:21:39 to 15:22:23, lag = 158 secs Mon Jan 31 15:26:02 EST 2011> queued 75 messages, timestamp range: 15:22:24 to 15:23:08, lag = 174 secs Mon Jan 31 15:27:03 EST 2011> queued 89 messages, timestamp range: 15:23:09 to 15:23:56, lag = 187 secs Mon Jan 31 15:28:04 EST 2011> queued 75 messages, timestamp range: 15:23:57 to 15:24:41, lag = 203 secs Mon Jan 31 15:29:05 EST 2011> queued 72 messages, timestamp range: 15:24:43 to 15:25:27, lag = 218 secs Mon Jan 31 15:30:06 EST 2011> queued 81 messages, timestamp range: 15:25:27 to 15:26:14, lag = 232 secs Mon Jan 31 15:31:07 EST 2011> queued 84 messages, timestamp range: 15:26:15 to 15:27:01, lag = 246 secs As you can see, the lag reaches about 10 minutes before fixing itself. Is this the expected behavior now for location searches with large radius? Or just a temporary problem? Thanks, Bella -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
