[twitter-dev] Re: non json response
I think this is the same issue we've been having for a while at http://groups.google.com/group/twitter-development-talk/browse_thread/thread/e88f0f3182b0a0e7/e5f6b2a5678598f1 I never used to get it on search APIs but now I get it on both REST and search APIs On Sep 6, 8:35 pm, Rudifa rudi.far...@gmail.com wrote: I have seen this same http page with empty body !DOCTYPE html PUBLIC -//W3C//DTD HTML 4.01//EN http://www.w3.org/ TR/1999/REC-html401-19991224/strict.dtd !-- !DOCTYPE HTML PUBLIC -//W3C//DTD HTML 4.01//EN http://www.w3.org/TR/html4/strict.dtd; -- HTML HEAD META HTTP-EQUIV=Refresh CONTENT=0.1 META HTTP-EQUIV=Pragma CONTENT=no-cache META HTTP-EQUIV=Expires CONTENT=-1 TITLE/TITLE /HEAD BODYP/BODY /HTML a number of times in the last few days (but intermittently - a good response may come after several attempts), in response tohttp://twitter.com/users/show/rudifa.json The most recent one was on UTC time 2009-09-06 18:55:38.262 My IP is 84.227.186.88 as reported byhttp://www.whatismyip.com/ Could someone at twitter.com please tell us what does this mean? Server (s) overloaded? On Aug 30, 1:20 pm, Steven Wilkin iamthebisc...@gmail.com wrote: I'm consistently getting the same response when accessinghttp://search.twitter.com/trends.jsonfrom209.40.204.183 Steve On Aug 26, 5:27 pm, Ryan Sarver rsar...@twitter.com wrote: Ben, It's a known issue and we are trying to hunt it down. Can you please provide us with your source IP and an approximate time of when you saw it? Thanks, RyanOn Wed, Aug 26, 2009 at 7:00 AM, benben.apperr...@googlemail.com wrote: Occassionally i get back a 200 status html response from the json search api which look like this, most times the same search works fine, it just happens occassionally: !DOCTYPE html PUBLIC -//W3C//DTD HTML 4.01//EN http://www.w3.org/ TR/1999/REC-html401-19991224/strict.dtd !-- !DOCTYPE HTML PUBLIC -//W3C//DTD HTML 4.01//EN http://www.w3.org/TR/html4/strict.dtd; -- HTML HEAD META HTTP-EQUIV=Refresh CONTENT=0.1 META HTTP-EQUIV=Pragma CONTENT=no-cache META HTTP-EQUIV=Expires CONTENT=-1 TITLE/TITLE /HEAD BODYP/BODY /HTML Does anyone recognise what this kind of response means? Is it normal, or just beta-ish quirks?
[twitter-dev] Re: Paging (or cursoring) will always return unreliable (or jittery) results
Flat file generation and maintenance would be foolish at this stage. Seperating out the individual data sets purely for api to be served by different clusters with server side caching may fit the bill - but tbh if this isn't happening already I'll be shocked. On Sep 7, 5:40 am, Jesse Stay jesses...@gmail.com wrote: As far as retrieving the large graphs from a DB, flat files are one way - another is to just store the full graph (of ids) in a single column in the database and parse on retrieval. This is what FriendFeed is doing currently, so they've said. Dewald and I are both talking about this because we're also having to duplicate this on our own servers, so we too have to deal with the pains of the social graph. (and oh the pain it is!) On Sun, Sep 6, 2009 at 8:44 PM, Dewald Pretorius dpr...@gmail.com wrote: If I worked for Twitter, here's what I would have done. I would have grabbed the follower id list of the large accounts (those that usually kicked back 502s) and written them to flat files once every 5 or so minutes. When an API request comes in for that list, I'd just grab it from the flat file, instead of asking the DB to select 2+ million ids from amongst a few billion records, while it's trying to do a few thousand other selects at the same time. That's one way of getting rid of 502s on large social graph lists. Okay, the data is going to be 5 minutes out-dated. To that I say, so bloody what? Dewald- Hide quoted text - - Show quoted text -
[twitter-dev] Re: non json response
Either way an XML or JSON feed should NEVER return HTML! On Sep 7, 11:25 am, Ben Eliott ben.apperr...@googlemail.com wrote: IP: 67.23.28.168, time is Europe/London 2009-09-07 11:19:48,014 - twittersearch.models - CRITICAL - Search did not reutrn a json object! code = 200 answer = !DOCTYPE html PUBLIC -//W3C//DTD HTML 4.01//EN http://www.w3.org/TR/1999/REC-html401-19991224/strict.dtd !-- !DOCTYPE HTML PUBLIC -//W3C//DTD HTML 4.01//EN http://www.w3.org/TR/html4/strict.dtd; -- HTML HEAD META HTTP-EQUIV=Refresh CONTENT=0.1 META HTTP-EQUIV=Pragma CONTENT=no-cache META HTTP-EQUIV=Expires CONTENT=-1 TITLE/TITLE /HEAD BODYP/BODY /HTML Starting to wonder whether this is connected to auth/user agent. On 6 Sep 2009, at 20:35, Rudifa wrote: I have seen this same http page with empty body !DOCTYPE html PUBLIC -//W3C//DTD HTML 4.01//EN http://www.w3.org/ TR/1999/REC-html401-19991224/strict.dtd !-- !DOCTYPE HTML PUBLIC -//W3C//DTD HTML 4.01//EN http://www.w3.org/TR/html4/strict.dtd; -- HTML HEAD META HTTP-EQUIV=Refresh CONTENT=0.1 META HTTP-EQUIV=Pragma CONTENT=no-cache META HTTP-EQUIV=Expires CONTENT=-1 TITLE/TITLE /HEAD BODYP/BODY /HTML a number of times in the last few days (but intermittently - a good response may come after several attempts), in response tohttp://twitter.com/users/show/rudifa.json The most recent one was on UTC time 2009-09-06 18:55:38.262 My IP is 84.227.186.88 as reported byhttp://www.whatismyip.com/ Could someone at twitter.com please tell us what does this mean? Server (s) overloaded? On Aug 30, 1:20 pm, Steven Wilkin iamthebisc...@gmail.com wrote: I'm consistently getting the same response when accessinghttp://search.twitter.com/trends.jsonfrom 209.40.204.183 Steve On Aug 26, 5:27 pm, Ryan Sarver rsar...@twitter.com wrote: Ben, It's a known issue and we are trying to hunt it down. Can you please provide us with your source IP and an approximate time of when you saw it? Thanks, RyanOn Wed, Aug 26, 2009 at 7:00 AM, benben.apperr...@googlemail.com wrote: Occassionally i get back a 200 status html response from the json search api which look like this, most times the same search works fine, it just happens occassionally: !DOCTYPE html PUBLIC -//W3C//DTD HTML 4.01//EN http://www.w3.org/ TR/1999/REC-html401-19991224/strict.dtd !-- !DOCTYPE HTML PUBLIC -//W3C//DTD HTML 4.01//EN http://www.w3.org/TR/html4/strict.dtd; -- HTML HEAD META HTTP-EQUIV=Refresh CONTENT=0.1 META HTTP-EQUIV=Pragma CONTENT=no-cache META HTTP-EQUIV=Expires CONTENT=-1 TITLE/TITLE /HEAD BODYP/BODY /HTML Does anyone recognise what this kind of response means? Is it normal, or just beta-ish quirks?
[twitter-dev] SUP (Simple Update Protocol), FriendFeed and Twitter
Hi all, I've recently been doing some research on how FriendFeed manages to push user's twitter updates to users FriendFeed profile so fast. I was very impressed at the speed these updates were delivered to FriendFeed and appears on my profile (within 5 seconds) so I started looking into how it works. I've learnt quite a lot about SUP by Googling it: Lots of relevant links here http://blog.friendfeed.com/2008/12/simple-update-protocol-update.html tl;dr FriendFeed faced problems updating their profiles because they were issuing millions of API calls to keep everyones profiles up to date. They came up with a proposal for SUP - a new kind of API that services provide that only lists accounts that have been updated recently. This saves FriendFeed requesting ALL users so frequently - they now only need to request the API for accounts that have new content. According to that blog post various services have already setup SUP feeds to help FriendFeed update things in close-to-real-time. My question is: Does Twitter have a SUP feed that can publicly be used? We are starting development on a site with similar real-time functionality to FriendFeed and currently face the same problem FriendFeed had (before they devised the SUP proposal and implementation). If Twitter does not provide a public SUP feed? can someone please try to explain how FriendFeed push user's twitter updates within seconds?? My only guess is that they may be constantly connected to the streaming API's firehose method and monitoring updates from twitter accounts that are also associated with FriendFeed profiles. Hoping someone can shed some light on this for me. Thank you! -Sam
[twitter-dev] Re: SUP (Simple Update Protocol), FriendFeed and Twitter
Friendfeed consumes the Twitter Streaming API to update Twitter status. SUP is not employed. All Twitter accounts have access to the Streaming API, documented here: http://apiwiki.twitter.com/Streaming-API-Documentation -John Kalucki http://twitter.com/jkalucki Services, Twitter Inc. On Sep 7, 5:08 am, Sam Street sam...@gmail.com wrote: Hi all, I've recently been doing some research on how FriendFeed manages to push user's twitter updates to users FriendFeed profile so fast. I was very impressed at the speed these updates were delivered to FriendFeed and appears on my profile (within 5 seconds) so I started looking into how it works. I've learnt quite a lot about SUP by Googling it: Lots of relevant links herehttp://blog.friendfeed.com/2008/12/simple-update-protocol-update.html tl;dr FriendFeed faced problems updating their profiles because they were issuing millions of API calls to keep everyones profiles up to date. They came up with a proposal for SUP - a new kind of API that services provide that only lists accounts that have been updated recently. This saves FriendFeed requesting ALL users so frequently - they now only need to request the API for accounts that have new content. According to that blog post various services have already setup SUP feeds to help FriendFeed update things in close-to-real-time. My question is: Does Twitter have a SUP feed that can publicly be used? We are starting development on a site with similar real-time functionality to FriendFeed and currently face the same problem FriendFeed had (before they devised the SUP proposal and implementation). If Twitter does not provide a public SUP feed? can someone please try to explain how FriendFeed push user's twitter updates within seconds?? My only guess is that they may be constantly connected to the streaming API's firehose method and monitoring updates from twitter accounts that are also associated with FriendFeed profiles. Hoping someone can shed some light on this for me. Thank you! -Sam
[twitter-dev] Re: SUP (Simple Update Protocol), FriendFeed and Twitter
SUP will not work for Twitter or any other service that deals with very large data sets. In essence, a Twitter SUP feed would be one JSON array of all the Twitter users who have posted a status update in the past 60 seconds. a) The SUP feed will consistently contain a few million array entries. b) To build that feed you must do a select against the tweets table, which contains a few billion records, and extract all the user ids with a tweet that has a published time greater than now() - 60. Good luck asking any DB to do that kind of select once every 60 seconds. Dewald
[twitter-dev] Re: 200 errors
Can we please hear something from someone at Twitter about this, it's becoming unusable with constant XML errors On Sep 7, 4:51 am, Naveen A knig...@gmail.com wrote: We are seeing this HTML META REFRESH as well from our clients. We are a mobile application and seeing this issue more and more frequently to the point that application is not functioning properly, its hard for use to provide any specific ip data as the carriers are most likely proxying the requests from the device. It is not limited to a specific api call either, it is a systemic issue across a wide range of calls we make. There was a ticket related to the issue in the bug tracker for search, but it has been closed and I think it should be re-opened as it is still a problemhttp://code.google.com/p/twitter-api/issues/detail?id=968 Any feedback would be appreciated. On Sep 6, 3:01 pm, Rich rhyl...@gmail.com wrote: Yeah it's happening to me again, same as my previous email, except the time stamp will be around 2 minutes ago On Sep 6, 4:05 pm, twittme_mobi nlupa...@googlemail.com wrote: Hi Ryan, I am getting the same error - i can found it in the logs of my app every day - at least 20 times. 1. The IP of the machine making requests to the Twitter API. If you're behind NAT, please be sure to send us your *external* IP. --- Name: twittme.mobi Address: 67.222.129.154 2. The IP address of the machine you're contacting in the Twitter cluster. You can find this on UNIX machines via the host or nslookup commands, and on Windows machines via the nslookup command. --- Name: twitter.com Address: 128.121.146.100 3. The Twitter API URL (method) you're requesting and any other details about the request (GET vs. POST, parameters, headers, etc.). --- 'account/rate_limit_status.xml' 4. Your host operating system, browser (including version), relevant cookies, and any other pertinent information about your environment. --- Linux, mobile browser,firefox, no cookies used. 5. What kind of network connection you have and from which provider, and what kind of network connectivity devices you're using. --- devices are mostly mobile..probably using mobile connections or wireless. Thanks! On Sep 5, 2:54 pm, Alex hyc...@gmail.com wrote: hi Ryan, any update on this issue ?
[twitter-dev] Re: SUP (Simple Update Protocol), FriendFeed and Twitter
Not necessarily. See this document (which I've posted earlier on this list) for details: http://code.google.com/p/pubsubhubbub/wiki/PublisherEfficiency In essence, with PSHB (Pubsub Hubbub), Twitter would only have to retrieve the latest data, add it to flat files on the server or a single column in a database somewhere as a static RSS format. Then, using a combination of persistent connections, HTTP Pipelining, and multiple, cached and linked ATOM feeds, return those feeds to either a hub or the user. ATOM feeds can be linked, and Twitter doesn't need to return the entire dataset in each feed, just the latest data, linked to older data on the server (if I understand ATOM correctly - someone correct me if I'm wrong). So in essence Twitter only needs to retrieve, and return to the user or hub the latest (cached) data, and can do so in a persistent connection, multiple HTTP requests at a time. And of course this doesn't take into account the biggest advantage of PSHB - the hub. PSHB is built to be distributed. I know Twitter doesn't want to go there, but if they wanted to they could allow other authorized hubs to distribute the load of such data, and only the hubs would fetch data from Twitter, significantly reducing the load for Twitter regardless of the size of request and ensuring a) users own their data in a publicly owned format, and b) if Twitter ever goes down the content is still available via the API. IMO this is the only way Twitter will become a utility as Jack Dorsey wants it to be. I would love to see Twitter adopt a more publicly accepted standard like this. Or, if it's not meeting their needs, either create their own public standard and take the lead in open real-time stream standards, or join an existing one so the standards can be perfected to a manner a company like Twitter can handle. I know it would make my coding much easier as more companies begin to adopt these protocols and I'm stuck having to write the code for each one. Leaving the data retrieval in a closed, proprietary format benefits nobody. Jesse On Mon, Sep 7, 2009 at 7:52 AM, Dewald Pretorius dpr...@gmail.com wrote: SUP will not work for Twitter or any other service that deals with very large data sets. In essence, a Twitter SUP feed would be one JSON array of all the Twitter users who have posted a status update in the past 60 seconds. a) The SUP feed will consistently contain a few million array entries. b) To build that feed you must do a select against the tweets table, which contains a few billion records, and extract all the user ids with a tweet that has a published time greater than now() - 60. Good luck asking any DB to do that kind of select once every 60 seconds. Dewald
[twitter-dev] real time timeline. is it possible?
Hi everyone I wish to know if it's possible to get -nearly- real time timeline updates from my own account. I check stream.twitter.com but don't think provides user's timeline option. Regards, PD: i'm not english speaker :)
[twitter-dev] Extraneous tweets?
Hi!
I'm just getting started using the follow API. We got access to
shadow (thanks!) and I'm taking it for a spin now. I'm following
about 7k people.
Something weird I've found is that I seem to routinely get tweets from
users who were not included in my follow=parameter. I think I must be
doing something silly... or confused... anything I should look for?
Here's a code fragment
req = Net::HTTP::Post.new url.path
follow = Node.find(:all).map{|n| n.twitter_id}.join(,)
req.set_form_data :follow=follow
[twitter-dev] Re: Streaming API -- CHANGE REQUIRED -- URL rationalization
The point of it all would be performance. Obviously this would have to be done in a secure fashion but the stream api and privacy are not mutually exclusive. John do you think this will be possible ? Maybe passing some of the oAuth credentials ? On Aug 26, 8:18 pm, JDG ghil...@gmail.com wrote: I would hope they never expose protected tweets -- if they did, what would be ... you know, the point of it all? On Wed, Aug 26, 2009 at 17:02, Kevin Watters kevinwatt...@gmail.com wrote: Will the streaming API ever expose tweets from protected users?--or is it an infrastructure limitation that isn't going away anytime soon? Also, will we ever see the ability to get real time tweets based on search operators (http://search.twitter.com/operators)? On Aug 26, 3:06 pm, John Kalucki jkalu...@gmail.com wrote: The resources in the Streaming API have been rationalized. You'll need to update the URLs that streaming clients are using over the next two weeks. The old URLs will be deprecated on or after September 9, 2009. The change is documented in the Wiki: http://apiwiki.twitter.com/Streaming-API-Documentation, specifically inhttp:// apiwiki.twitter.com/Streaming-API-Documentation#Methods. The new scheme allows for API versioning, streams that contain objects other than statuses, separates access level control from URLs and allows multiple filter predicates to be specified on a single connection. The cute resource names have, sadly, been dropped we move towards pushing the service out of Alpha. Also, /track and friends have been merged with /follow and friends into a single resource. When you connect to a given resource, you will automatically be given the highest access level possible. The following is a mapping from the old URLs to the new URLs. Otherwise, you should notice only subtle changes to the Streaming API error handling behavior. All other functionality should continue to work as in the past. /firehose - /1/statuses/firehose /spritzer, /gardenhose - /1/statuses/sample /birddog, /shadow, /follow - /1/statuses/filter /partner/track, /restricted/track, /track - /1/statuses/filter For example, if you have been connecting to /gardenhose.json, connect to /1/statuses/sample.json. Note that the Streaming API is still in Alpha test. -John Kaluckihttp://twitter.com/jkalucki Services, Twitter Inc. -- Internets. Serious business.- Hide quoted text - - Show quoted text -
[twitter-dev] Re: 200 errors
It is happening on our site and I checked from one of our other sites using curl. It gives the 200 error one minute and then the status response the next. It does not matter if I request JSON it still returns the HTML.
[twitter-dev] jaisen mathai script - strange behaviour when posting status update
After a user has authenticated via oauth, the following does not work:
?
include 'EpiCurl.php';
include 'EpiOAuth.php';
include 'EpiTwitter.php';
include 'secret.php';
include 'connect.php';
$twitterObj2 = new EpiTwitter($consumer_key, $consumer_secret,
$oauth_token, $oauth_token_secret);
$tweet = testing;
$twitterObj2-post_statusesUpdate(array('status' = $tweet));
?
but the following does work even though the additional code should not
make a difference to the twitter api
?
include 'EpiCurl.php';
include 'EpiOAuth.php';
include 'EpiTwitter.php';
include 'secret.php';
include 'connect.php';
$twitterObj2 = new EpiTwitter($consumer_key, $consumer_secret,
$oauth_token, $oauth_token_secret);
$twitterInfo= $twitterObj1-get_statusesUser_timeline(array
('screen_name' = $username));
foreach($twitterInfo as $status) {
$url=http://www.ebay.com;;
$url = rawurlencode($url);
//USE IS.GD API
$url = http://is.gd/api.php?longurl=.$url;
$fh = fopen($url,'r');
$isgdurl = fread($fh, 140);
fclose($fh);
$tweet = testing;
$twitterObj2-post_statusesUpdate(array('status' = $tweet));
}
?
[twitter-dev] Re: non json response
I am able to consistently reproduce this error -- I get this response almost without fail after periods of inactivity greater than 2 hours. I am requesting XML via PHP, it's a GET request. The requests are coming from 96.30.16.192. Let me know if I can provide any additional info that might help, or if you have any suggestions about how best to handle this response. On Sep 6, 3:35 pm, Rudifa rudi.far...@gmail.com wrote: I have seen this same http page with empty body !DOCTYPE html PUBLIC -//W3C//DTD HTML 4.01//EN http://www.w3.org/ TR/1999/REC-html401-19991224/strict.dtd !-- !DOCTYPE HTML PUBLIC -//W3C//DTD HTML 4.01//EN http://www.w3.org/TR/html4/strict.dtd; -- HTML HEAD META HTTP-EQUIV=Refresh CONTENT=0.1 META HTTP-EQUIV=Pragma CONTENT=no-cache META HTTP-EQUIV=Expires CONTENT=-1 TITLE/TITLE /HEAD BODYP/BODY /HTML a number of times in the last few days (but intermittently - a good response may come after several attempts), in response tohttp://twitter.com/users/show/rudifa.json The most recent one was on UTC time 2009-09-06 18:55:38.262 My IP is 84.227.186.88 as reported byhttp://www.whatismyip.com/ Could someone at twitter.com please tell us what does this mean? Server (s) overloaded? On Aug 30, 1:20 pm, Steven Wilkin iamthebisc...@gmail.com wrote: I'm consistently getting the same response when accessinghttp://search.twitter.com/trends.jsonfrom209.40.204.183 Steve On Aug 26, 5:27 pm, Ryan Sarver rsar...@twitter.com wrote: Ben, It's a known issue and we are trying to hunt it down. Can you please provide us with your source IP and an approximate time of when you saw it? Thanks, RyanOn Wed, Aug 26, 2009 at 7:00 AM, benben.apperr...@googlemail.com wrote: Occassionally i get back a 200 status html response from the json search api which look like this, most times the same search works fine, it just happens occassionally: !DOCTYPE html PUBLIC -//W3C//DTD HTML 4.01//EN http://www.w3.org/ TR/1999/REC-html401-19991224/strict.dtd !-- !DOCTYPE HTML PUBLIC -//W3C//DTD HTML 4.01//EN http://www.w3.org/TR/html4/strict.dtd; -- HTML HEAD META HTTP-EQUIV=Refresh CONTENT=0.1 META HTTP-EQUIV=Pragma CONTENT=no-cache META HTTP-EQUIV=Expires CONTENT=-1 TITLE/TITLE /HEAD BODYP/BODY /HTML Does anyone recognise what this kind of response means? Is it normal, or just beta-ish quirks?
[twitter-dev] Re: non json response
I am able to consistently reproduce this error. I am making GET requests via PHP from IP: 96.30.16.192. I receive the error without fail after periods of inactivity lasting 2 hours or more. The header response code is 200. Please let me know if I can provide any additional info that might help you diagnose the problem, or if you have suggestions about how best to handle. Thanks. On Sep 6, 3:35 pm, Rudifa rudi.far...@gmail.com wrote: I have seen this same http page with empty body !DOCTYPE html PUBLIC -//W3C//DTD HTML 4.01//EN http://www.w3.org/ TR/1999/REC-html401-19991224/strict.dtd !-- !DOCTYPE HTML PUBLIC -//W3C//DTD HTML 4.01//EN http://www.w3.org/TR/html4/strict.dtd; -- HTML HEAD META HTTP-EQUIV=Refresh CONTENT=0.1 META HTTP-EQUIV=Pragma CONTENT=no-cache META HTTP-EQUIV=Expires CONTENT=-1 TITLE/TITLE /HEAD BODYP/BODY /HTML a number of times in the last few days (but intermittently - a good response may come after several attempts), in response tohttp://twitter.com/users/show/rudifa.json The most recent one was on UTC time 2009-09-06 18:55:38.262 My IP is 84.227.186.88 as reported byhttp://www.whatismyip.com/ Could someone at twitter.com please tell us what does this mean? Server (s) overloaded? On Aug 30, 1:20 pm, Steven Wilkin iamthebisc...@gmail.com wrote: I'm consistently getting the same response when accessinghttp://search.twitter.com/trends.jsonfrom209.40.204.183 Steve On Aug 26, 5:27 pm, Ryan Sarver rsar...@twitter.com wrote: Ben, It's a known issue and we are trying to hunt it down. Can you please provide us with your source IP and an approximate time of when you saw it? Thanks, RyanOn Wed, Aug 26, 2009 at 7:00 AM, benben.apperr...@googlemail.com wrote: Occassionally i get back a 200 status html response from the json search api which look like this, most times the same search works fine, it just happens occassionally: !DOCTYPE html PUBLIC -//W3C//DTD HTML 4.01//EN http://www.w3.org/ TR/1999/REC-html401-19991224/strict.dtd !-- !DOCTYPE HTML PUBLIC -//W3C//DTD HTML 4.01//EN http://www.w3.org/TR/html4/strict.dtd; -- HTML HEAD META HTTP-EQUIV=Refresh CONTENT=0.1 META HTTP-EQUIV=Pragma CONTENT=no-cache META HTTP-EQUIV=Expires CONTENT=-1 TITLE/TITLE /HEAD BODYP/BODY /HTML Does anyone recognise what this kind of response means? Is it normal, or just beta-ish quirks?
[twitter-dev] Implementing update via JS
Hi, I have to implement updating Twitter status through JS. Need pointers on how to get started
[twitter-dev] Re: 200 errors
I am seeing the 200 errors also from our sites. I tried getting status using curl and it returns the 200 HTML and then the status intermittently. If I specify JSON I still get the HTML on the errors and JSON data on the status. This is really affecting our website.
[twitter-dev] Re: OAuth changes?
I've been having some status updates fail using oAuth with .NET over the last few days. It seems to be an intermittent problem, and, like yours, my code's been working fine for months... Cheers, Rich. On Sep 5, 2:20 am, Bobby Gaza syml...@gmail.com wrote: Hi, I was curious if anyone has seen any calls to statuses/update stop working usingOAuth/PHP Pecl. I recently started getting errors of Incorrect Signature with some code that had been working perfectly fine for the past month. I'd be happy to elaborate more, but just shooting out this general inquiry in case if this is the wrong forum for it. Thanks Bobby
[twitter-dev] Re: 200 errors
The same also, blank 4.01 .
[twitter-dev] Application Not Posting
Hi, this application http://twitter.com/oauth_clients/details/16368 is not posting, I keep getting the following error message, please can you advise what might be wrong? Woah there! This page is no longer valid. It looks like someone already used the token information you provided. Please return to the site that sent you to this page and try again … it was probably an honest mistake. This link is used to add account: http://www.elertgadget.com/pub/twitteraccounts.php I then get the above error message on this URL: https://twitter.com/oauth/authorize?oauth_token=
[twitter-dev] Re: SUP (Simple Update Protocol), FriendFeed and Twitter
Why would you use the db ? Just do it all in memory right ? On Mon, Sep 7, 2009 at 6:52 AM, Dewald Pretoriusdpr...@gmail.com wrote: SUP will not work for Twitter or any other service that deals with very large data sets. In essence, a Twitter SUP feed would be one JSON array of all the Twitter users who have posted a status update in the past 60 seconds. a) The SUP feed will consistently contain a few million array entries. b) To build that feed you must do a select against the tweets table, which contains a few billion records, and extract all the user ids with a tweet that has a published time greater than now() - 60. Good luck asking any DB to do that kind of select once every 60 seconds. Dewald
[twitter-dev] Re: SUP (Simple Update Protocol), FriendFeed and Twitter
One question is how does friendfeed handle protected updates since the streaming api is for public statuses only ? On Mon, Sep 7, 2009 at 6:32 AM, John Kaluckijkalu...@gmail.com wrote: Friendfeed consumes the Twitter Streaming API to update Twitter status. SUP is not employed. All Twitter accounts have access to the Streaming API, documented here: http://apiwiki.twitter.com/Streaming-API-Documentation -John Kalucki http://twitter.com/jkalucki Services, Twitter Inc. On Sep 7, 5:08 am, Sam Street sam...@gmail.com wrote: Hi all, I've recently been doing some research on how FriendFeed manages to push user's twitter updates to users FriendFeed profile so fast. I was very impressed at the speed these updates were delivered to FriendFeed and appears on my profile (within 5 seconds) so I started looking into how it works. I've learnt quite a lot about SUP by Googling it: Lots of relevant links herehttp://blog.friendfeed.com/2008/12/simple-update-protocol-update.html tl;dr FriendFeed faced problems updating their profiles because they were issuing millions of API calls to keep everyones profiles up to date. They came up with a proposal for SUP - a new kind of API that services provide that only lists accounts that have been updated recently. This saves FriendFeed requesting ALL users so frequently - they now only need to request the API for accounts that have new content. According to that blog post various services have already setup SUP feeds to help FriendFeed update things in close-to-real-time. My question is: Does Twitter have a SUP feed that can publicly be used? We are starting development on a site with similar real-time functionality to FriendFeed and currently face the same problem FriendFeed had (before they devised the SUP proposal and implementation). If Twitter does not provide a public SUP feed? can someone please try to explain how FriendFeed push user's twitter updates within seconds?? My only guess is that they may be constantly connected to the streaming API's firehose method and monitoring updates from twitter accounts that are also associated with FriendFeed profiles. Hoping someone can shed some light on this for me. Thank you! -Sam
[twitter-dev] Re: SUP (Simple Update Protocol), FriendFeed and Twitter
+1 definitely ! I think everyone asks Twitter the same question but the problem was they developed the firehose prior to PHSB What are the main cons of PHSB ? On Mon, Sep 7, 2009 at 8:48 AM, Jesse Stayjesses...@gmail.com wrote: Not necessarily. See this document (which I've posted earlier on this list) for details: http://code.google.com/p/pubsubhubbub/wiki/PublisherEfficiency In essence, with PSHB (Pubsub Hubbub), Twitter would only have to retrieve the latest data, add it to flat files on the server or a single column in a database somewhere as a static RSS format. Then, using a combination of persistent connections, HTTP Pipelining, and multiple, cached and linked ATOM feeds, return those feeds to either a hub or the user. ATOM feeds can be linked, and Twitter doesn't need to return the entire dataset in each feed, just the latest data, linked to older data on the server (if I understand ATOM correctly - someone correct me if I'm wrong). So in essence Twitter only needs to retrieve, and return to the user or hub the latest (cached) data, and can do so in a persistent connection, multiple HTTP requests at a time. And of course this doesn't take into account the biggest advantage of PSHB - the hub. PSHB is built to be distributed. I know Twitter doesn't want to go there, but if they wanted to they could allow other authorized hubs to distribute the load of such data, and only the hubs would fetch data from Twitter, significantly reducing the load for Twitter regardless of the size of request and ensuring a) users own their data in a publicly owned format, and b) if Twitter ever goes down the content is still available via the API. IMO this is the only way Twitter will become a utility as Jack Dorsey wants it to be. I would love to see Twitter adopt a more publicly accepted standard like this. Or, if it's not meeting their needs, either create their own public standard and take the lead in open real-time stream standards, or join an existing one so the standards can be perfected to a manner a company like Twitter can handle. I know it would make my coding much easier as more companies begin to adopt these protocols and I'm stuck having to write the code for each one. Leaving the data retrieval in a closed, proprietary format benefits nobody. Jesse On Mon, Sep 7, 2009 at 7:52 AM, Dewald Pretorius dpr...@gmail.com wrote: SUP will not work for Twitter or any other service that deals with very large data sets. In essence, a Twitter SUP feed would be one JSON array of all the Twitter users who have posted a status update in the past 60 seconds. a) The SUP feed will consistently contain a few million array entries. b) To build that feed you must do a select against the tweets table, which contains a few billion records, and extract all the user ids with a tweet that has a published time greater than now() - 60. Good luck asking any DB to do that kind of select once every 60 seconds. Dewald
[twitter-dev] Re: Random 408 errors having been appearing the last 48 hours
We've been seeing 408's since the DOS attack back in July/August. They feel like rate limiting on Twitter's part when overloaded. Cannot tell since 408 isn't listed as an error they throw at api.twitter.com JEff On Sep 6, 8:51 am, bosher bhellm...@gmail.com wrote: Random 408 errors are being returned when users are attempting to Sign in with Twitter on my site TweetMeNews.com. Has anyone else been seeing this? Twitter, is there any way you can expand on the error message instead of just saying 408? That would help us better Understand Report what's breaking... Thanks, Bretthttp://twitter.com/TweetMeNews/
[twitter-dev] Re: Finding the most popular trending topics
Hi, Adding some kind of weight value or sorting of the returned trending topics would be a very good feature, for example to be able to create nice word clouds. /Håkan On Aug 24, 6:21 am, Chad Etzel jazzyc...@gmail.com wrote: Hi, There is currently no way to get the number of retweets/tweets of trending topics through the API. Thanks, -Chad On Sun, Aug 23, 2009 at 5:34 PM, TrixJotri...@gmail.com wrote: Trying to find the most popular trending topics. Currently using: http://search.twitter.com/trends/current.json?exclude=hashtags Is there a way that I can find the most popular trending topics as well as the number of retweets for each topic? That way I can save the trending topics in my database as well as the number of retweets currently retweeted for each topic and I can rank each trending topic myself based on the number or retweets tahnks
[twitter-dev] tweets
Als ik een tweet zend kom ik deze naderhand niet tegen bij degene die ik volg. Als er een tweet gestuurd woord door de gene die ik volg komt hij wel op mijn site maar niet andersom. heb ik iets verkeerd ingesteld soms? Dick Hofman
[twitter-dev] Re: Random 408 errors having been appearing the last 48 hours
I am having the same issue, most of the times I cannot connect to Twitter, I get 408 error and the API is mostly unusable form my side. I am able to connect just a couple of times every 36-48 hours! Are we the only people having this issue? How that can be possible? Is there any way to contact Twitter folks about this issue? Are they aware of this? Any more thoughts and testimonials about this issue would be appreciated. Thank you for sharing. Best,
[twitter-dev] Re: Random 408 errors having been appearing the last 48 hours
If you are having connection problems like this, please send your IP address, account(s), a curl(1) -vvv trace, and a tcpdump of the failure to a...@twitter.com. -John Kalucki http://twitter.com/jkalucki Services, Twitter inc. On Sep 7, 5:02 pm, fablau fabri...@virtualsheetmusic.com wrote: I am having the same issue, most of the times I cannot connect to Twitter, I get 408 error and the API is mostly unusable form my side. I am able to connect just a couple of times every 36-48 hours! Are we the only people having this issue? How that can be possible? Is there any way to contact Twitter folks about this issue? Are they aware of this? Any more thoughts and testimonials about this issue would be appreciated. Thank you for sharing. Best,
[twitter-dev] Re: SUP (Simple Update Protocol), FriendFeed and Twitter
This issue was aired considerably in this thread: http://groups.google.com/group/twitter-development-talk/browse_thread/thread/8665766f5e262d60 On Sep 7, 9:33 am, Monica Keller monica.kel...@gmail.com wrote: +1 definitely ! I think everyone asks Twitter the same question but the problem was they developed the firehose prior to PHSB What are the main cons of PHSB ? On Mon, Sep 7, 2009 at 8:48 AM, Jesse Stayjesses...@gmail.com wrote: Not necessarily. See this document (which I've posted earlier on this list) for details: http://code.google.com/p/pubsubhubbub/wiki/PublisherEfficiency In essence, with PSHB (Pubsub Hubbub), Twitter would only have to retrieve the latest data, add it to flat files on the server or a single column in a database somewhere as a static RSS format. Then, using a combination of persistent connections, HTTP Pipelining, and multiple, cached and linked ATOM feeds, return those feeds to either a hub or the user. ATOM feeds can be linked, and Twitter doesn't need to return the entire dataset in each feed, just the latest data, linked to older data on the server (if I understand ATOM correctly - someone correct me if I'm wrong). So in essence Twitter only needs to retrieve, and return to the user or hub the latest (cached) data, and can do so in a persistent connection, multiple HTTP requests at a time. And of course this doesn't take into account the biggest advantage of PSHB - the hub. PSHB is built to be distributed. I know Twitter doesn't want to go there, but if they wanted to they could allow other authorized hubs to distribute the load of such data, and only the hubs would fetch data from Twitter, significantly reducing the load for Twitter regardless of the size of request and ensuring a) users own their data in a publicly owned format, and b) if Twitter ever goes down the content is still available via the API. IMO this is the only way Twitter will become a utility as Jack Dorsey wants it to be. I would love to see Twitter adopt a more publicly accepted standard like this. Or, if it's not meeting their needs, either create their own public standard and take the lead in open real-time stream standards, or join an existing one so the standards can be perfected to a manner a company like Twitter can handle. I know it would make my coding much easier as more companies begin to adopt these protocols and I'm stuck having to write the code for each one. Leaving the data retrieval in a closed, proprietary format benefits nobody. Jesse On Mon, Sep 7, 2009 at 7:52 AM, Dewald Pretorius dpr...@gmail.com wrote: SUP will not work for Twitter or any other service that deals with very large data sets. In essence, a Twitter SUP feed would be one JSON array of all the Twitter users who have posted a status update in the past 60 seconds. a) The SUP feed will consistently contain a few million array entries. b) To build that feed you must do a select against the tweets table, which contains a few billion records, and extract all the user ids with a tweet that has a published time greater than now() - 60. Good luck asking any DB to do that kind of select once every 60 seconds. Dewald
[twitter-dev] Re: Extraneous tweets?
Do the apparently extraneous tweets happen to be in_reply_to a
specified user_id?
-John Kalucki
http://twitter.com/jkalucki
Services, Twitter Inc.
On Sep 5, 10:17 pm, Steve Farrell st...@farrell.org wrote:
Hi!
I'm just getting started using the follow API. We got access to
shadow (thanks!) and I'm taking it for a spin now. I'm following
about 7k people.
Something weird I've found is that I seem to routinely get tweets from
users who were not included in my follow=parameter. I think I must be
doing something silly... or confused... anything I should look for?
Here's a code fragment
req = Net::HTTP::Post.new url.path
follow = Node.find(:all).map{|n| n.twitter_id}.join(,)
req.set_form_data :follow=follow
[twitter-dev] Re: Paging (or cursoring) will always return unreliable (or jittery) results
This describes what I'd call row-based pagination. Cursor-based pagination does not suffer from the same jitter issues. A cursor-based approach returns a unique, within the total set, ordered opaque value that is indexed for constant time access. Removals in pages before or after do not affect the stability of next page retrieval. For a user with a large following, you'll never have a point-in-time snapshot of their followings with any approach, but you can retrieve a complete unique set of users that were followers throughout the duration of the query. Additions made while the query is running may or may not be returned, as chance allows. A row-based approach with OFFSET and LIMIT is doomed for reasons beyond correctness. The latency and CPU consumption, in MySQL at least, tends to O(N^2). The first few blocks aren't bad. The last few blocks for a 10M, or even 1M set are miserable. The jitter demonstrated by the current API is due to a minor and correctable design flaw in the allocation of the opaque cursor values. A fix is scheduled. -John Kalucki http://twitter.com/jkalucki Services, Twitter Inc. On Sep 6, 7:27 pm, Dewald Pretorius dpr...@gmail.com wrote: There is no way that paging through a large and volatile data set can ever return results that are 100% accurate. Let's say one wants to page through @aplusk's followers list. That's going to take between 3 and 5 minutes just to collect the follower ids with page (or the new cursors). It is likely that some of the follower ids that you have gone past and have already colledted, have unfollowed @aplusk while you are still collecting the rest. I assume that the Twitter system does paging by doing a standard SQL LIMIT clause. If you do LIMIT 100, 20 and some of the ids that you have already paged past have been deleted, the result set is going to shift to the left and you are going to miss the ones that were above 100 but have subsequently moved left to below 100. There really are only two solutions to this problem: a) we need to have the capability to reliably retrieve the entire result set in one API call, or b) everyone has to accept that the result set cannot be guaranteed to be 100% accurate. Dewald
[twitter-dev] Re: real time timeline. is it possible?
Personally, I think it would be great if the Streaming API could support streaming the with_friends timeline. There are many compelling use cases. You can simulate streaming the with_friends timeline by grabbing your following list to populate a follow parameter to the /1/statuses/ filter.format resource. You can also simulate mentions by defining a track query for all the followed screen_names on the same connection. Yes, you won't get protected statuses, but you could poll for those whenever a certain time period had passed and only after new status arrived. This would allow you to place protected statuses in correct chronological order without hitting rate limits, but with higher latency. -John Kalucki http://twitter.com/jkalucki Services, Twitter Inc. On Sep 5, 4:22 pm, Rolando Espinoza La fuente dark...@gmail.com wrote: Hi everyone I wish to know if it's possible to get -nearly- real time timeline updates from my own account. I check stream.twitter.com but don't think provides user's timeline option. Regards, PD: i'm not english speaker :)
[twitter-dev] Re: Streaming API -- CHANGE REQUIRED -- URL rationalization
Technically possible or not, streaming protected statuses isn't a current priority. In my opinion, and in my opinion only, it's also not a good idea, regardless of the safeguards employed. -John Kalucki http://twitter.com/jkalucki Services, Twitter Inc. On Sep 6, 11:16 am, Monica Keller monica.kel...@gmail.com wrote: The point of it all would be performance. Obviously this would have to be done in a secure fashion but the stream api and privacy are not mutually exclusive. John do you think this will be possible ? Maybe passing some of the oAuth credentials ? On Aug 26, 8:18 pm, JDG ghil...@gmail.com wrote: I would hope they never expose protected tweets -- if they did, what would be ... you know, the point of it all? On Wed, Aug 26, 2009 at 17:02, Kevin Watters kevinwatt...@gmail.com wrote: Will the streaming API ever expose tweets from protected users?--or is it an infrastructure limitation that isn't going away anytime soon? Also, will we ever see the ability to get real time tweets based on search operators (http://search.twitter.com/operators)? On Aug 26, 3:06 pm, John Kalucki jkalu...@gmail.com wrote: The resources in the Streaming API have been rationalized. You'll need to update the URLs that streaming clients are using over the next two weeks. The old URLs will be deprecated on or after September 9, 2009. The change is documented in the Wiki: http://apiwiki.twitter.com/Streaming-API-Documentation, specifically inhttp:// apiwiki.twitter.com/Streaming-API-Documentation#Methods. The new scheme allows for API versioning, streams that contain objects other than statuses, separates access level control from URLs and allows multiple filter predicates to be specified on a single connection. The cute resource names have, sadly, been dropped we move towards pushing the service out of Alpha. Also, /track and friends have been merged with /follow and friends into a single resource. When you connect to a given resource, you will automatically be given the highest access level possible. The following is a mapping from the old URLs to the new URLs. Otherwise, you should notice only subtle changes to the Streaming API error handling behavior. All other functionality should continue to work as in the past. /firehose - /1/statuses/firehose /spritzer, /gardenhose - /1/statuses/sample /birddog, /shadow, /follow - /1/statuses/filter /partner/track, /restricted/track, /track - /1/statuses/filter For example, if you have been connecting to /gardenhose.json, connect to /1/statuses/sample.json. Note that the Streaming API is still in Alpha test. -John Kaluckihttp://twitter.com/jkalucki Services, Twitter Inc. -- Internets. Serious business.- Hide quoted text - - Show quoted text -
[twitter-dev] Re: Recent Following and Follower Issues and Some Background on Social Graph
Hi John, On Sep 6, 3:59 pm, John Kalucki jkalu...@gmail.com wrote: resources. There is minor pagination jitter in one case and a certain class of row-count-based queries have to be deprecated (or limited) and replaced with cursor-based queries to be practical. For now, we're sending the row-count-queries queries back to the second system, which is otherwise idle, but isn't consistent with the first or third system. I am getting several emails per day at the moment from users telling me my app's results are wrong. The application currently asks for the entire follower/following ID list at once, using /followers/ids and / friends/ids. Does this count as a row-count-query? David
[twitter-dev] Re: Recent Following and Follower Issues and Some Background on Social Graph
I don't know all the details, but my general understanding is that these bulk followers calls have been heavily returning 503s for quite some time now, and this is long established, but bad, behavior. These bulk calls are hard to support and they need to be moved over to some form of practical pagination scheme. Ideally, we'd offer a stream of social graph deltas on the Streaming API and this polling business could be tightly restricted. Bluntly, until further back-end work is in place, we can return 5k followers reliably from the third system, or we can attempt to return large result sets, but often throw 503s -- really, timeouts, from the second system. We cannot return bulk operations, or use row-based cursors, from the third system. Scraping the social graph is certainly valuable in some cases, but generally it's a low value proposition for users, and scraping is often is used to support abusive behavior. -John Kalucki http://twitter.com/jkalucki Services, Twitter Inc. On Sep 7, 9:27 pm, David W. d...@botanicus.net wrote: Hi John, On Sep 6, 3:59 pm, John Kalucki jkalu...@gmail.com wrote: resources. There is minor pagination jitter in one case and a certain class of row-count-based queries have to be deprecated (or limited) and replaced with cursor-based queries to be practical. For now, we're sending the row-count-queries queries back to the second system, which is otherwise idle, but isn't consistent with the first or third system. I am getting several emails per day at the moment from users telling me my app's results are wrong. The application currently asks for the entire follower/following ID list at once, using /followers/ids and / friends/ids. Does this count as a row-count-query? David
[twitter-dev] Re: Recent Following and Follower Issues and Some Background on Social Graph
I might add that, as ever, a message on status.twitter mentioning this would really go a long way. David. On Sep 8, 5:27 am, David W. d...@botanicus.net wrote: Hi John, On Sep 6, 3:59 pm, John Kalucki jkalu...@gmail.com wrote: resources. There is minor pagination jitter in one case and a certain class of row-count-based queries have to be deprecated (or limited) and replaced with cursor-based queries to be practical. For now, we're sending the row-count-queries queries back to the second system, which is otherwise idle, but isn't consistent with the first or third system. I am getting several emails per day at the moment from users telling me my app's results are wrong. The application currently asks for the entire follower/following ID list at once, using /followers/ids and / friends/ids. Does this count as a row-count-query? David
[twitter-dev] Re: Paging (or cursoring) will always return unreliable (or jittery) results
I could really go for jittery right now... instead I'm getting totally broken! I'm getting two pages of results, using ?page=x, then empty. To me, it looks like all my accounts have max 10K followers. I'd love some kind of official response from Twitter on the status of paging (John?). Example: user @starbucks has nearly 300K followers, however: http://twitter.com/followers/ids.xml?id=30973page=3 returns empty result. - Waldron On Sep 7, 10:24 pm, John Kalucki jkalu...@gmail.com wrote: This describes what I'd call row-based pagination. Cursor-based pagination does not suffer from the same jitter issues. A cursor-based approach returns a unique, within the total set, ordered opaque value that is indexed for constant time access. Removals in pages before or after do not affect the stability of next page retrieval. For a user with a large following, you'll never have a point-in-time snapshot of their followings with any approach, but you can retrieve a complete unique set of users that were followers throughout the duration of the query. Additions made while the query is running may or may not be returned, as chance allows. A row-based approach with OFFSET and LIMIT is doomed for reasons beyond correctness. The latency and CPU consumption, in MySQL at least, tends to O(N^2). The first few blocks aren't bad. The last few blocks for a 10M, or even 1M set are miserable. The jitter demonstrated by the current API is due to a minor and correctable design flaw in the allocation of the opaque cursor values. A fix is scheduled. -John Kaluckihttp://twitter.com/jkalucki Services, Twitter Inc. On Sep 6, 7:27 pm, Dewald Pretorius dpr...@gmail.com wrote: There is no way that paging through a large and volatile data set can ever return results that are 100% accurate. Let's say one wants to page through @aplusk's followers list. That's going to take between 3 and 5 minutes just to collect the follower ids with page (or the new cursors). It is likely that some of the follower ids that you have gone past and have already colledted, have unfollowed @aplusk while you are still collecting the rest. I assume that the Twitter system does paging by doing a standard SQL LIMIT clause. If you do LIMIT 100, 20 and some of the ids that you have already paged past have been deleted, the result set is going to shift to the left and you are going to miss the ones that were above 100 but have subsequently moved left to below 100. There really are only two solutions to this problem: a) we need to have the capability to reliably retrieve the entire result set in one API call, or b) everyone has to accept that the result set cannot be guaranteed to be 100% accurate. Dewald
