Hi Thom,
This is generally referred to as the 'moving window problem', and
our next_page URL handles it. This is covered by using the max_id
parameter so we are paginating from a fixed point and new matches
don't throw off the pagination. To find matching tweets after your
original page=1 query you need to re-query again from the beginning …
and we include a refresh_url element (or rel="refresh" in Atom) for
reference.
Thanks;
– Matt Sanford / @mzsanford
Twitter Dev
On May 1, 2009, at 12:47 PM, Thom Allen wrote:
Matt, this is exactly what I was getting ready to ask a question
about. I hope I can implement the correct procedure with your
information. Thank you.
I do have a question though. If I paginate using next_page, is the
next page only relevant to my search query? Meaning, if new posts
find there way into the stream that meet my criteria, and I am
asking for page two, will I only get items that were originally on
page two from my original search? Or will page two now contain items
that were originally on page one before. Not sure how the API
handles this. Thanks for any clarification.
--Thom
On Fri, May 1, 2009 at 9:38 AM, Matt Sanford <[email protected]> wrote:
Hi all,
I've noticed an increasing number of search API questions whose
root cause has been the fact that since_id and max_id do not work
together. I dug into our code and found the reason for this and it
turns out to provide a pretty solid performance gain. I started to
look at ways around this and there are none that I can implement at
the moment. I did launch a few small changes yesterday to make this
easier to spot:
1. The next_url (rel="next" element in Atom results) should never
contain since_id. This was working correctly in Atom results, but JSON
was incorrectly including it. That erroneous since_id has been
removed.
2. In both Atom and JSON we should warn you if we do something
unexpected. If you search with since_id and there are more than one
page of results you'll now get a warning ("warning" in JSON,
"twitter:warning" in Atom) telling you since_id was removed for
pagination.
Everything is functionally the same, I've just added a warning to
let you know when things like this happen. The main issue here is that
since you cannot paginate with since_id you cannot set since_id and
then paginate until there are no results. You need to search with
since_id and then paginate if needed (via next_page or rel="next")
until you reach a status <= since_id.
Thanks;
— Matt Sanford / @mzsanford
Twitter Dev
