On Mon, Apr 13, 2009 at 7:43 AM, Nick Arnett <[email protected]> wrote:
Ack.... chrome sent the reply before I was ready... trying to fix the white
space to make it proper Python, I hit tab and return, which sent the
message!
Anyway, you wouldn't need the lines that have to do with next_page or
results, those are for other kinds of requests. The idea is to increment a
page (&page=1)parameter in your request URL in the loop, continuing until
you get no data (or a result code other than 200), which indicates that
you've gotten all the statuses Twitter is storing.
Let me try again to show example code here... There's some stuff here that
lets me other url parameters, which you might not need. self.http is an
instance of httplib2.Http(), self. max_pages is a variable for an upper
limit on the number of pages I get and it sets self.last_status_id to the id
of the last status it received. There's some simple error handling and
retries when the error seems to be something temporary.
I haven't tested this code, but I adapted it from something that is working
fairly reliably. The white space is screwed up
i = 0
concat_content = []
url_parameters = ["count=200"]
trys = 0
paged = True
page_url = "http://twitter.com/statuses/user_timeline/%s.json"; %
(twitter_id)
while 1:
i += 1
params = "&".join(url_parameters)
if paged:
if len(params):
page_url = url + "?%s&page=%s" % (params, i)
else:
page_url = url + "?page=%s" % (i)
elif len(params):
page_url = url + "?%s" % (params)
else:
page_url = url
response, content = self.http.request(page_url, method)
if response.status == 200:
if response['content-type'].startswith('application/json'):
try:
data = json.loads(content)
except:
print "json decode failed: %s" % (content)
return False
if len(data) and i < self.max_pages:
concat_content += (data)
else:
self.last_status_id = concat_content[-1]['id']
return concat_content
if response.status in (500, 520, 503):
time.sleep(10)
trys += 1
if trys >= 3:
print "%s failed with internal errors" % (status_type)
return False
else:
print "%s failed: %s; Reason %s" % (status_type, twitter_id,
response.reason)
return False
