[twitter-dev] Re: Best way to pull/cache location based search results?
No one? On Jan 4, 3:20 pm, GeorgeMedia georgeme...@gmail.com wrote: Hello everyone! I sure hope you can help. I am developing a web based app that searches forlocationbased tweets using the search API with json results. I provide the longitude latitude via my own local database on my server. Presently I'm limited to just the US and Canada but I'm thinking I might be able to pull long/lat data dynamically from sites like hostip.info or even maybe the yahooapi? As you can imagine I am salivating in anticipation of the trends.locationapi because parsing the text from the individual tweets is a pain in the behind :) But back on point... I'm using the search api exclusively because that's the only way I know of to get localized tweets. I read the streaming/firehose search api docs but it doesn't seem to support geocode parameters. So my first question is, am I correct in assuming that presently the only way to getlocationbased tweets is using the geocode parameter on the search api? Secondly, I'm in a bit of a unique situation from what I can see form other apps in that it's a bit tough to run a cron because there are over 200,000 longitude/latitude sets in my database and that's just the US and Canada! So how could I possiblycachethe data so that when a user queries some random city so they can see local tweets for that city it's available without doing an api call every time a user types in a city? Because if my site gets popular that could become a problem very quickly. If I can think it I usually can code it but being new to using twitter's API and limits I'm a bit stuck.
[twitter-dev] Re: Best way to pull/cache location based search results?
On Jan 8, 9:29 am, GeorgeMedia georgeme...@gmail.com wrote: No one? I think you would be better off consuming the firehose, geocode the tweets yourself, and throw away any that aren’t in regions you care about, caching the rest for a period of time. The thing to remember about geocoding of tweets is that until very recently the geocoding was solely by the location field in a user’s profile. True geocoding of individual tweets is very recent and depends on the user enabling geo coding, and on the user agent posting the lat/lon with the tweet. So the firehose *does* contain the geo field, it's just mostly empty because most clients don’t populate it yet. So if the geo field is empty you’d have to geocode based on the location field which is a bit of a hairball and may contain any data up to 30 bytes. Alternately, do the cron job thing but enlarge the regions you’re searching on (search on the top N cities or metros for example, not 200,000 coordinates). Cache the data, and accept that it won’t be absolutely up to date (it’s already lost a lot of precision since the location field is completely arbitrary and even if it is a city or lat/lon pair, does not necessarily represent where the twitter user was at that moment in time). -- -ed costello
[twitter-dev] Search API suddenly stopped returning results for the same query
Is anyone else seeing the Search API not returning results that it had been returning recently? For example, this query returned results consistently this morning and then recently stopped: http://search.twitter.com/search.json...@aaronrankinpage=1rpp=100since_id=7452902654 It works without the since_id. And, I'm not having a rate limiting issue.
[twitter-dev] UPDATE: Social Graph API Deprecation
On December 22, 2009 we announced that the social graph method pagination of the followers/ids and friends/ids would finally be removed. We announced deprecation in September (http://bit.ly/46x1iL), November ( http://bit.ly/3UQ0LU) and December (http://bit.ly/5VPWk7) of last year. The page parameter will be completely removed 1/11/2010. However, the behavior of assuming that you want the first cursor page when passing no cursor parameter will not. In the December 2009 announcement, I explained that: You should always pass a cursor parameter. Starting soon, if you fail to pass a cursor, the data returned will be that of the first cursor (-1) and the next_cursor and previous_cursor elements will be included. In response to the feedback we received in a http://bit.ly/longDiscussionAboutTheSocialGraph we have decided not to immediately remove support for unreliably retrieving a complete friend or follower list (by passing neither page nor cursor parameters) on 1/11/2010. We understand that too many applications still depend on it. We're working on a better way to pull this data; expect another updated announcement on this list soon with further details. We know that the cursor-based social graph APIs can be improved -- we can provide richer functionality than we currently expose. To do this, we need your help; contribute your use cases for the social graph in response to this message on http://bit.ly/TwitterDevelopmentTalk. With better understanding of how you use the graph data, we can improve the quality and variety of APIs that we provide. Thanks!
[twitter-dev] Re: Social Graph API: Legacy data format will be eliminated 1/11/2010
We won't immediately remove the unbound search (defaulting no cursor
to the first).
Details:
http://groups.google.com/group/twitter-development-talk/browse_frm/thread/a0ba66db0e86941d
On Jan 7, 9:03 pm, Zaudio si...@z-audio.co.uk wrote:
Yes - Please can we have that urgently - yes or no?
Thanks
Simon
On Jan 6, 8:15 pm, PJB pjbmancun...@gmail.com wrote:
Can we please get some confirmation that the cursor-less calls won't
be going away this coming Monday?
On Dec 22 2009, 4:13 pm, Wilhelm Bierbaum wilh...@twitter.com wrote:
We noticed that some clients are still calling social graph methods
without cursor parameters. We wanted to take time to make sure that
people were calling the updated methods which return data with cursors
instead of the old formats that do not.
As previously announced in September (http://bit.ly/46x1iL) and
November (http://bit.ly/3UQ0LU), the legacy data formats returned
as a result of calling social graph endpoints without a cursor
parameter are deprecated and will be removed.
These formats have been removed from the API wiki since September.
You should always pass a cursor parameter. Starting soon, if you fail
to pass a cursor, the data returned will be that of the first cursor
(-1) and the next_cursor and previous_cursor elements will be included.
If you aren't seeing next_cursor and previous_cursor in your results,
you are getting data back in the old format. You will need to adjust
your parser to handle the new format.
We're going to start assuming you want data in the new format
(users_list / users / user or id_list / ids / id) instead of the old
format (users / user or ids / id) regardless of your passing a cursor
parameter as of 1/11/2010.
* The old formats will no longer be returned after 1/11/2010.
* Start using the new formats now by passing the 'cursor' parameter.
To recap, the old endpoints at
/statuses/friends.xml
/statuses/followers.xml
returned
users type=array
user
!-- ... omitted ... --
/user
/users
or JSON like [{/*user record*/ /*, .../]
whereas
/statuses/friends.xml?cursor=n
/statuses/followers.xml?cursor=n
return data that looks like
users_list
users type=array
user
!-- ... omitted ... --
/user
/users
next_cursor7128872798413429387/next_cursor
previous_cursor0/previous_cursor
/users_list
or, the JSON equivalent:
{users:[{/*user record*/} /*, ...*/], next_cursor:0,
previous_cursor:0}
and the old endpoints at
/friends/ids.xml
/followers/ids.xml
returned data that looks like
ids
id1/id
id2/id
id3/id
/ids
whereas
/friends/ids.xml?cursor=n
/followers/ids.xml?cursor=n
return data that looks like
id_list
ids
id1/id
id2/id
id3/id
/ids
next_cursor1288724293877798413/next_cursor
previous_cursor-1300794057949944903/previous_cursor
/id_list
or, the JSON equivalent:
{ids:[1, 2, 3], next_cursor:0, previous_cursor:0}
If you have any questions or comments, please feel free to post them
to twitter-development-talk.
Thanks!
--
Wilhelm Bierbaum
Twitter Platform Team- Hide quoted text -
- Show quoted text -
[twitter-dev] Re: Social Graph API: Legacy data format will be eliminated 1/11/2010
How much larger do you think makes it easier?
On Jan 7, 6:42 pm, st...@implu.com st...@implu.com wrote:
I would agree with several views expressed in various posts here.
1) A cursor-less call that returns all IDs makes for simpler code and
fewer API calls. i.e. less processing time.
2) If we must have a 'cursored' call then at least allow for cursor=-1
to return a larger number than 5k.
-Steve
On Jan 6, 10:15 pm, PJB pjbmancun...@gmail.com wrote:
Can we please get some confirmation that the cursor-less calls won't
be going away this coming Monday?
On Dec 22 2009, 4:13 pm, Wilhelm Bierbaum wilh...@twitter.com wrote:
We noticed that some clients are still calling social graph methods
without cursor parameters. We wanted to take time to make sure that
people were calling the updated methods which return data with cursors
instead of the old formats that do not.
As previously announced in September (http://bit.ly/46x1iL) and
November (http://bit.ly/3UQ0LU), the legacy data formats returned
as a result of calling social graph endpoints without a cursor
parameter are deprecated and will be removed.
These formats have been removed from the API wiki since September.
You should always pass a cursor parameter. Starting soon, if you fail
to pass a cursor, the data returned will be that of the first cursor
(-1) and the next_cursor and previous_cursor elements will be included.
If you aren't seeing next_cursor and previous_cursor in your results,
you are getting data back in the old format. You will need to adjust
your parser to handle the new format.
We're going to start assuming you want data in the new format
(users_list / users / user or id_list / ids / id) instead of the old
format (users / user or ids / id) regardless of your passing a cursor
parameter as of 1/11/2010.
* The old formats will no longer be returned after 1/11/2010.
* Start using the new formats now by passing the 'cursor' parameter.
To recap, the old endpoints at
/statuses/friends.xml
/statuses/followers.xml
returned
users type=array
user
!-- ... omitted ... --
/user
/users
or JSON like [{/*user record*/ /*, .../]
whereas
/statuses/friends.xml?cursor=n
/statuses/followers.xml?cursor=n
return data that looks like
users_list
users type=array
user
!-- ... omitted ... --
/user
/users
next_cursor7128872798413429387/next_cursor
previous_cursor0/previous_cursor
/users_list
or, the JSON equivalent:
{users:[{/*user record*/} /*, ...*/], next_cursor:0,
previous_cursor:0}
and the old endpoints at
/friends/ids.xml
/followers/ids.xml
returned data that looks like
ids
id1/id
id2/id
id3/id
/ids
whereas
/friends/ids.xml?cursor=n
/followers/ids.xml?cursor=n
return data that looks like
id_list
ids
id1/id
id2/id
id3/id
/ids
next_cursor1288724293877798413/next_cursor
previous_cursor-1300794057949944903/previous_cursor
/id_list
or, the JSON equivalent:
{ids:[1, 2, 3], next_cursor:0, previous_cursor:0}
If you have any questions or comments, please feel free to post them
to twitter-development-talk.
Thanks!
--
Wilhelm Bierbaum
Twitter Platform Team
[twitter-dev] Search Widget: Callback with # of returns?
I'm using the stock Search Widget at www.twimpathy.com. It's awesome, but I have one small problem: Sometimes a search returns nothing, but the widget of course still shows. Is there any kind of callback function that will stick the number of returns in a variable, so I can wrap the widget in a conditional to display or not? Thanks for any help!
Re: [twitter-dev] Re: Social Graph API: Legacy data format will be eliminated 1/11/2010
100k, at the minimum. On 1/8/10 3:35 PM, Wilhelm Bierbaum wrote: How much larger do you think makes it easier? On Jan 7, 6:42 pm, st...@implu.com st...@implu.com wrote: I would agree with several views expressed in various posts here. 1) A cursor-less call that returns all IDs makes for simpler code and fewer API calls. i.e. less processing time. 2) If we must have a 'cursored' call then at least allow for cursor=-1 to return a larger number than 5k. -- Dossy Shiobara | do...@panoptic.com | http://dossy.org/ Panoptic Computer Network | http://panoptic.com/ He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on. (p. 70)
Re: [twitter-dev] Any iPhone Twitter apps with OAuth login ?
I'm partly curious to know/confirm whether our app is the only iPhone (or mobile) app that uses Twitter OAuth login for posting tweets, but I also want to know what you think of the UI, if you've used Twitter OAuth login in any publicly released mobile app. I do and I sincerely thin Twitter could improve this design, it's not usable for me.
Re: [twitter-dev] Any iPhone Twitter apps with OAuth login ?
hey fabien (and the rest of the list). what do you think we could do to improve this for all of you? i'm actively interested in making it better. feel free to add to this thread, or mail me personally. thanks! On Fri, Jan 8, 2010 at 1:18 PM, Fabien Penso fabienpe...@gmail.com wrote: I'm partly curious to know/confirm whether our app is the only iPhone (or mobile) app that uses Twitter OAuth login for posting tweets, but I also want to know what you think of the UI, if you've used Twitter OAuth login in any publicly released mobile app. I do and I sincerely thin Twitter could improve this design, it's not usable for me. -- Raffi Krikorian Twitter Platform Team http://twitter.com/raffi
Re: [twitter-dev] Any iPhone Twitter apps with OAuth login ?
On Fri, Jan 8, 2010 at 11:24 PM, Raffi Krikorian ra...@twitter.com wrote: hey fabien (and the rest of the list). what do you think we could do to improve this for all of you? i'm actively interested in making it better. feel free to add to this thread, or mail me personally. Raffi, That would be such a great improvement. I think the required changes is small, I would go very simple and just display a login/password and the 2 buttons approve/deny in *large* without the sidebox on the right you have currently, as it's not needed. Something like what you get when you browse twitter.com/username from your iPhone.
[twitter-dev] Re: Social Graph API: Legacy data format will be eliminated 1/11/2010
As large as possible. 100k would be a huge improvement. For FriendOrFollow.com I need the user's entire social graph to effectively calculate who's not following them back, who they're not following back, and their mutual friendships. I can't really cache this data because user's make decisions on who to follow and unfollow based on my data. If the data is old, I start hearing complaints about how a user unfollowed someone who was really following them, etc. So the data really needs to be pulled on page load. The 5k at a time cursors pretty much cripples FriendOrFollow for anyone with an impressive amount of followers, and it also takes too many API calls to be rate limit effective. The more IDs that can be pulled at once, the better. If I could have the user's IDs streamed to me like the streaming API does tweets, that would be pretty hot. On Jan 8, 2:38 pm, Dossy Shiobara do...@panoptic.com wrote: 100k, at the minimum. On 1/8/10 3:35 PM, Wilhelm Bierbaum wrote: How much larger do you think makes it easier? On Jan 7, 6:42 pm, st...@implu.com st...@implu.com wrote: I would agree with several views expressed in various posts here. 1) A cursor-less call that returns all IDs makes for simpler code and fewer API calls. i.e. less processing time. 2) If we must have a 'cursored' call then at least allow for cursor=-1 to return a larger number than 5k. -- Dossy Shiobara | do...@panoptic.com |http://dossy.org/ Panoptic Computer Network |http://panoptic.com/ He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on. (p. 70)
Re: [twitter-dev] Re: Social Graph API: Legacy data format will be eliminated 1/11/2010
What proportion of your users have more than 5k followers? More than 25k followers? -John Kalucki http://twitter.com/jkalucki Services, Twitter Inc. On Fri, Jan 8, 2010 at 2:57 PM, DustyReagan dustyrea...@gmail.com wrote: As large as possible. 100k would be a huge improvement. For FriendOrFollow.com I need the user's entire social graph to effectively calculate who's not following them back, who they're not following back, and their mutual friendships. I can't really cache this data because user's make decisions on who to follow and unfollow based on my data. If the data is old, I start hearing complaints about how a user unfollowed someone who was really following them, etc. So the data really needs to be pulled on page load. The 5k at a time cursors pretty much cripples FriendOrFollow for anyone with an impressive amount of followers, and it also takes too many API calls to be rate limit effective. The more IDs that can be pulled at once, the better. If I could have the user's IDs streamed to me like the streaming API does tweets, that would be pretty hot. On Jan 8, 2:38 pm, Dossy Shiobara do...@panoptic.com wrote: 100k, at the minimum. On 1/8/10 3:35 PM, Wilhelm Bierbaum wrote: How much larger do you think makes it easier? On Jan 7, 6:42 pm, st...@implu.com st...@implu.com wrote: I would agree with several views expressed in various posts here. 1) A cursor-less call that returns all IDs makes for simpler code and fewer API calls. i.e. less processing time. 2) If we must have a 'cursored' call then at least allow for cursor=-1 to return a larger number than 5k. -- Dossy Shiobara | do...@panoptic.com |http://dossy.org/ Panoptic Computer Network |http://panoptic.com/ He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on. (p. 70)
[twitter-dev] Re: UPDATE: Social Graph API Deprecation
My use cases for the Social Graph API:
* Figure out mutual followers vs one-way followers, namely for my
Tweepsect application: http://tweepsect.com/
This requires a full graph dump, unless you include a parameter in the
statuses/{friends,followers} API calls that indicate whether said
friend is a mutual follower, or a just-friend (stalking) or a just-
follower (stalker). I need to load that data regardless, so if this is
included then I can omit loading the social graph altogether.
* For a Twitter client, for every tweet, figure out if the poster of
said tweet is a mutual follower or not.
Again, if this data was included under each tweet, I wouldn't need to
load the entire social graph.
* Crawling a user's social graph
If I could filter which results to get, such as by geolocation or by
mutualness or by when they last tweeted, it would reduce the amount of
work I have to do.
- shazow
On Jan 8, 12:29 pm, Wilhelm Bierbaum wilh...@twitter.com wrote:
On December 22, 2009 we announced that the social graph method pagination of
the followers/ids and friends/ids would finally be removed. We announced
deprecation in September (http://bit.ly/46x1iL), November
(http://bit.ly/3UQ0LU) and December (http://bit.ly/5VPWk7) of last year. The
page parameter will be completely removed 1/11/2010. However, the behavior
of assuming that you want the first cursor page when passing no cursor
parameter will not.
In the December 2009 announcement, I explained that:
You should always pass a cursor parameter. Starting soon, if you fail to
pass a cursor, the data returned will be that of the first cursor (-1) and
the next_cursor and previous_cursor elements will be included.
In response to the feedback we received in
ahttp://bit.ly/longDiscussionAboutTheSocialGraphwe have decided not to
immediately remove support for unreliably retrieving a complete friend or
follower list (by passing neither page nor cursor parameters) on 1/11/2010.
We understand that too many applications still depend on it. We're working
on a better way to pull this data; expect another updated announcement on
this list soon with further details.
We know that the cursor-based social graph APIs can be improved -- we can
provide richer functionality than we currently expose. To do this, we need
your help; contribute your use cases for the social graph in response to
this message onhttp://bit.ly/TwitterDevelopmentTalk. With better
understanding of how you use the graph data, we can improve the quality and
variety of APIs that we provide.
Thanks!
[twitter-dev] Re: UPDATE: Social Graph API Deprecation
I have two use cases: 1. Generating a list of all friends and followers. 2. Downloading the most recent 200 tweets of all friends and followers. The existing API functionality is adequate for the first. The second depends more on the rate limiting than the functionality. Right now, I have about 5000 contacts. At 150 calls per hour, this is over a day to complete. With the 1500 calls per hour I get with the coming oAuth boost, this will come down, but it's still going to be over an hour. On Jan 8, 12:29 pm, Wilhelm Bierbaum wilh...@twitter.com wrote: On December 22, 2009 we announced that the social graph method pagination of the followers/ids and friends/ids would finally be removed. We announced deprecation in September (http://bit.ly/46x1iL), November (http://bit.ly/3UQ0LU) and December (http://bit.ly/5VPWk7) of last year. The page parameter will be completely removed 1/11/2010. However, the behavior of assuming that you want the first cursor page when passing no cursor parameter will not. In the December 2009 announcement, I explained that: You should always pass a cursor parameter. Starting soon, if you fail to pass a cursor, the data returned will be that of the first cursor (-1) and the next_cursor and previous_cursor elements will be included. In response to the feedback we received in ahttp://bit.ly/longDiscussionAboutTheSocialGraphwe have decided not to immediately remove support for unreliably retrieving a complete friend or follower list (by passing neither page nor cursor parameters) on 1/11/2010. We understand that too many applications still depend on it. We're working on a better way to pull this data; expect another updated announcement on this list soon with further details. We know that the cursor-based social graph APIs can be improved -- we can provide richer functionality than we currently expose. To do this, we need your help; contribute your use cases for the social graph in response to this message onhttp://bit.ly/TwitterDevelopmentTalk. With better understanding of how you use the graph data, we can improve the quality and variety of APIs that we provide. Thanks!
Re: [twitter-dev] Re: Social Graph API: Legacy data format will be eliminated 1/11/2010
On 1/8/10 5:59 PM, John Kalucki wrote: What proportion of your users have more than 5k followers? More than 25k followers? Good point ... | grouping | percent | +--+-+ | 0-4,999 |72.7 | | 5,000-24,999 |22.3 | | 25,000+ | 5.0 | I think 27% of users is large enough to pay attention to ... ? -- Dossy Shiobara | do...@panoptic.com | http://dossy.org/ Panoptic Computer Network | http://panoptic.com/ He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on. (p. 70)
Re: [twitter-dev] about search result
Not currently. Hopefully once the APIs become more unified in_reply_to_status_id will be available. Abraham On Thu, Jan 7, 2010 at 22:58, lch1721 lch1...@gmail.com wrote: Hi everyone~ I want search result that include in_reply_to_status_id Is it impossible? I'm get status_ids from search result and show every tweet now. -- Abraham Williams | #doit | http://hashtagdoit.com Project | Intersect | http://intersect.labs.poseurtech.com Hacker | http://abrah.am | http://twitter.com/abraham This email is: [ ] shareable [x] ask first [ ] private.
Re: [twitter-dev] Search API suddenly stopped returning results for the same query
I'm seeing the 4 most recent statuses which is consistant with without since_id after 7452902654. Abraham On Fri, Jan 8, 2010 at 12:34, Aaron Rankin aran...@gmail.com wrote: Is anyone else seeing the Search API not returning results that it had been returning recently? For example, this query returned results consistently this morning and then recently stopped: http://search.twitter.com/search.json...@aaronrankinpage=1rpp=100since_id=7452902654 It works without the since_id. And, I'm not having a rate limiting issue. -- Abraham Williams | #doit | http://hashtagdoit.com Project | Intersect | http://intersect.labs.poseurtech.com Hacker | http://abrah.am | http://twitter.com/abraham This email is: [ ] shareable [x] ask first [ ] private. Sent from Madison, WI, United States
Re: [twitter-dev] Search API suddenly stopped returning results for the same query
Hello, Is there a way to search for a specific user using their email address on twitter.com? Either through the interface, or using the API. I don't want to give twitter the login to my account (gmail,hotmail,yahoo). I don't like the idea of 3rd party parsing through my contact list to find my friends on the network. Thanks mon.
[twitter-dev] Re: Social Graph API: Legacy data format will be eliminated 1/11/2010
Here's some rough numbers...x is the number of twitter user's with a follower count of... x = 100k 7140.007% 75k = x 100k 1510.001% 50k = x 75k 4110.004% 25k = x 50k 20440.020% 0 x 25k 10009489 96.529% Total: 10,369,396 So I would agree that 100k would be sufficient for our needs. -Steve On Jan 8, 3:38 pm, Dossy Shiobara do...@panoptic.com wrote: 100k, at the minimum. On 1/8/10 3:35 PM, Wilhelm Bierbaum wrote: How much larger do you think makes it easier? On Jan 7, 6:42 pm, st...@implu.com st...@implu.com wrote: I would agree with several views expressed in various posts here. 1) Acursor-less call that returns all IDs makes for simpler code and fewer API calls. i.e. less processing time. 2) If we must have a 'cursored' call then at least allow forcursor=-1 to return a larger number than 5k. -- Dossy Shiobara | do...@panoptic.com |http://dossy.org/ Panoptic Computer Network |http://panoptic.com/ He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on. (p. 70)
