[twitter-dev] Re: Upcoming changes to the way status IDs are sequenced
We're relying on the ID being sequention for a number of purposes: 1) Counting elapsed tweets to estimate tweet rates to feed back into count parameter to backtrack when restarting streaming API/Shadow - how will we be able to do that without sequential IDs??? 2) Indexing and sorting pages of tweets to be display by our application - moving away from sequential IDs to break our sorting algarithms... and require recoding to sort exclusively by date alone 3) Polling for new mentions to merge within streamed tweets - we use the ID as a last placeholder - again chnages there would break our app unless recoded. Zaudio Developer BullsOnWallStreet.com To unsubscribe from this group, send email to twitter-development-talk+unsubscribegooglegroups.com or reply to this email with the words REMOVE ME as the subject.
[twitter-dev] Urgent: Major issues with 400 errors on social graph friends and followers
Hi, I'm getting this back and just do not understand why: TWITTER LOG GENERATED ON 28/01/2010 3:57:57 PM ExecuteGetCommand: Twitter reported Error (OAuth): The remote server returned an error: (400) Bad Request. Request url: http://api.twitter.com/1/friends/ids.xml?user_id=35034625cursor=-1 ?xml version=1.0 encoding=UTF-8? hash request/1/friends/ids.xml? cursor=-1amp;oauth_consumer_key=Xamp;oauth_nonce=35034625546_490163amp;oauth_signature_method=HMAC- SHA1amp;oauth_timestamp=1264719478amp;oauth_token=35034625- amp;oauth_version=1.0amp;user_id=35034625amp;oauth_signature=/ request errorRate limit exceeded. Clients may not make more than 0 requests per hour./error /hash Response Headers: X-RateLimit-Limit: 0 Status: 400 Bad Request X-RateLimit-Remaining: 0 X-Runtime: 0.02385 X-RateLimit-Class: api_identified X-RateLimit-Reset: 1264723053 Vary: Accept-Encoding Connection: close Content-Length: 507 Cache-Control: no-cache, max-age=1800 Content-Type: application/xml; charset=utf-8 Date: Thu, 28 Jan 2010 22:58:35 GMT Expires: Thu, 28 Jan 2010 23:28:34 GMT Set-Cookie: _twitter_sess=BAh7BzoHaWQiJTdhNzJkYmNkYWQzZTM3OGMzOGZiODc3ODllMmUzMTFhIgpm %250AbGFzaElDOidBY3Rpb25Db250cm9sbGVyOjpGbGFzaDo6Rmxhc2hIYXNoewAG %250AOgpAdXNlZHsA--5b92ace9ec11f25bd06a6b1812e05e9550852992; domain=.twitter.com; path=/ Server: hi - I've tested rate limits for my local testing IP (216.45.124.14 ) where I'm getting this, and my testing account Zaudio... and the test user Zaudio2 - all looks good Please urgently address this. I thought it a blacklisting issue... but maybe it's a bug? Thanks Simon (Zaudio) Developer www.bullsonwallstreet.com
[twitter-dev] Re: Social Graph API: Legacy data format will be eliminated 1/11/2010
Yes - Please can we have that urgently - yes or no?
Thanks
Simon
On Jan 6, 8:15 pm, PJB pjbmancun...@gmail.com wrote:
Can we please get some confirmation that the cursor-less calls won't
be going away this coming Monday?
On Dec 22 2009, 4:13 pm, Wilhelm Bierbaum wilh...@twitter.com wrote:
We noticed that some clients are still calling social graph methods
without cursor parameters. We wanted to take time to make sure that
people were calling the updated methods which return data with cursors
instead of the old formats that do not.
As previously announced in September (http://bit.ly/46x1iL) and
November (http://bit.ly/3UQ0LU), the legacy data formats returned
as a result of calling social graph endpoints without a cursor
parameter are deprecated and will be removed.
These formats have been removed from the API wiki since September.
You should always pass a cursor parameter. Starting soon, if you fail
to pass a cursor, the data returned will be that of the first cursor
(-1) and the next_cursor and previous_cursor elements will be included.
If you aren't seeing next_cursor and previous_cursor in your results,
you are getting data back in the old format. You will need to adjust
your parser to handle the new format.
We're going to start assuming you want data in the new format
(users_list / users / user or id_list / ids / id) instead of the old
format (users / user or ids / id) regardless of your passing a cursor
parameter as of 1/11/2010.
* The old formats will no longer be returned after 1/11/2010.
* Start using the new formats now by passing the 'cursor' parameter.
To recap, the old endpoints at
/statuses/friends.xml
/statuses/followers.xml
returned
users type=array
user
!-- ... omitted ... --
/user
/users
or JSON like [{/*user record*/ /*, .../]
whereas
/statuses/friends.xml?cursor=n
/statuses/followers.xml?cursor=n
return data that looks like
users_list
users type=array
user
!-- ... omitted ... --
/user
/users
next_cursor7128872798413429387/next_cursor
previous_cursor0/previous_cursor
/users_list
or, the JSON equivalent:
{users:[{/*user record*/} /*, ...*/], next_cursor:0,
previous_cursor:0}
and the old endpoints at
/friends/ids.xml
/followers/ids.xml
returned data that looks like
ids
id1/id
id2/id
id3/id
/ids
whereas
/friends/ids.xml?cursor=n
/followers/ids.xml?cursor=n
return data that looks like
id_list
ids
id1/id
id2/id
id3/id
/ids
next_cursor1288724293877798413/next_cursor
previous_cursor-1300794057949944903/previous_cursor
/id_list
or, the JSON equivalent:
{ids:[1, 2, 3], next_cursor:0, previous_cursor:0}
If you have any questions or comments, please feel free to post them
to twitter-development-talk.
Thanks!
--
Wilhelm Bierbaum
Twitter Platform Team- Hide quoted text -
- Show quoted text -
[twitter-dev] Re: Rate Limiting question
Hey Kyle, Is that true re the rate limits? I've always assumed that ANY request on an IP is xounted against that IP's rate limit - so comes of the 20,000; and each user has their own rate limit... normally 150 per hour. So an authenticated request comes off both the user's and and IP's rate limit? In the very least, the user's limit of 150 per hour is affected first, and then if that is maxed out, then the IP limit will come into play? Could someone from Twitter clarify this, and Kyle's suggestion, as the API docs are very vague in this area Thanks Simon On Jan 1, 5:21 pm, Kyle Mulka repalvigla...@yahoo.com wrote: My experience with rate limiting shows that each authenticated request is counted against that user's limit on your IP. So, you get 20,000 requests per IP, per user, per hour assuming all your requests are authenticated. Any unauthenticated requests go towards the 20,000 request limit per IP, per hour. In my case, all Twitter API calls are authenticated and cached for an hour. The way my app is set up, under normal usage, no user will use more than 20,000 Twitter API requests. -- Kyle Mulka Founder, Congo Labshttp://twilk.com On Jan 1, 5:43 pm, jojet j...@jojet.com wrote: Hi all, I was feeling a little clever after working on some Twitter API stuff but then thought oh! I'd better think about Twitters rate limiting...and then that's where my brain started to melt! A few bits of info: my web app needs people to authenticate (OAUTH) and, from then on, the app analyses their tweets and occasionally updates registered user's statuses. I've applied for the webserver IP to be white listed which I believe gives the app 20,000 requests per hour. I've not found the search API to be great when looking for a hashtag (sometimes tweets just don't seem to get indexed) so I've gone a stage further and am analysing the individual timelines of all my registered users via a cron job (the cron job sucks in all of a persons tweets greater than the last analysed tweet of the user). This call is made via OAUTH/authenticated so I believe such a call depletes the user's rate limit quota rather than the IP/authenticated account of the webserver quota? Is that correct? Thanks for any thoughts here Joel
[twitter-dev] Re: Time zone support
I actually totally prefer time's to all be UNC / GMT as they are now... it's the same erevywhere, and very easily converted to local timezones for display - which is what I imagine most end users want to see... they don't want to be working out timezones from around the world in their heads! If tweets all start having variable timezones, it is just another thing that has to be calculated around to get times into the local timezone for end users. Please leave as it is. Just my thoughts Simon On Nov 8, 11:33 pm, Emrah e...@ekanet.net wrote: Raffi Krikorian wrote: hi emrah. this sounds interesting -- how do you handle people who are traveling and may not be in their home timezone when they say good morning? :) Timezone code could be set per Tweet as a parameter. E.g.: on mobile phones, the time is usually updated from the network operator and on most computers from an NNTP. Obviously the user experience should not be changed and the timezone must be discovered and set automatically. Regards, Emrah
[twitter-dev] Re: C# + OAuth + account/update_profile_image = 500 Internal Server Error
Thanks Thomas... I did look at TwitterVB as a possible way to find the answer to this particular function call... but the API says it doesn't yet cover account/updateprofileimage... Thanks Simon On Oct 19, 3:39 pm, Thomas Hübner thueb...@gmx.de wrote: You can use TwitterVB which covers nearly the complete API in .NET (OAuth included). U find it on codeplexhttp://twittervb.codeplex.com/ Cheers, Thomas Nicholas Granado schrieb: Simon, You would sign the request with all of the usual oauth param suspects. If I recall correctly this endpoint has no other params other than the 'image' param in the multi-part post body whose value would be the bytes of the image file. Typically I've only seen the post params passed into the oauth signing rigmarole when the post body is urlencoded. I hope this helps, this whole OAuth thing can be very confusing at first glance. If you are in C# I have my own lib for twitter basic auth/oauth that I've baked up, if you like I could pass you the bits. Nicholas --- Nicholas Granado email: ngran...@gmail.com mailto:ngran...@gmail.com twitter: heatxsink web:http://nickgranado.com On Mon, Oct 19, 2009 at 6:38 AM, Zaudio si...@z-audio.co.uk mailto:si...@z-audio.co.uk wrote: Nicholas, That's great feedback! In you opinion, how do I then sign the request? Do I use all the usual for the signaturebase... ie postmethodurlnonceetc etc or just postmethodurl as David suggested? I trust that the image data does not come into the signing process, and that I still can post the data using iso-8859-1 encoding as I would normally do for uploading files? If you have these answers, then I should be able to nail this for our .net case.Oauth's been working great for us until this hitch... Thanks Simon On Oct 18, 6:11 pm, Nicholas Granado ngran...@gmail.com mailto:ngran...@gmail.com wrote: Simon, I believe the body of your post might be incorrect. It should look like this: POST /account/update_profile_image.xml HTTP/1.1 Content-Type: multipart/form-data; boundary=8cbed79c91b24f3 Host: twitter.com http://twitter.com Content-Length: 3863(this will probably change now..) --8cbed79c91b24f3 Content-Disposition: form-data; name=image; filename=test.jpg Content-Type: image/jpeg (there's a few K of binary data here, the contents of the file) --8cbed79c91b24f3 The rest of the OAuth variables should be passed on the query string. I hope this helps. Cheers, Nicholas --- Nicholas Granado email: ngran...@gmail.com mailto:ngran...@gmail.com twitter: heatxsink web:http://nickgranado.com On Sun, Oct 18, 2009 at 2:42 PM, Zaudio si...@z-audio.co.uk mailto:si...@z-audio.co.uk wrote: Hi David, I found your excellent post hoping that it would solve the same challenge for my app: updating profile image via Oauth... using similar .net base to yourself... BUT I just get the 401 all the time... despite taking your advice to just sign with the HTTPmethod URL My post data is laid out much like yours... though I never got that 500 error... I've tried all sorts... dropping the off the end different encodings... What encoding did you use to encode your image, and then to post the request? Does it still work for you... or did this get broken when Twitter 'fixed' their Oauth implementation? Can anyone else advise if they have got this working and where I might be going wrong? Thanks Simon (Zaudio) On Aug 19, 11:40 pm, David Carson carson63...@gmail.com mailto:carson63...@gmail.com wrote: Got this sorted out and working, and thought I should share the two pitfalls which were causing me problems. First of all, unbelievably, the 500 Internal Server Error was being caused by an extra carriage return between my last HTTP header and the first multipart boundary. Seriously. I had two blank lines in there instead of one. Removed the extra carriage return, and my 500 vanished, being replaced by a more reasonable (401) Unauthorized - Incorrect signature error. Secondly, the OAuth documentation seems a bit shaky when it comes to multipart/form-data POSTs. But basically, you do NOT use any of the POST parameters when creating your signature. And this includes all of the OAuth-specific parameters like oauth_consumer_key, oauth_signature_method, etc. Bit of a security hole imho, OAuth
[twitter-dev] Re: C# + OAuth + account/update_profile_image = 500 Internal Server Error
Hi Nicholas, Sounds like you have this particular method cracked (account/ updateprofileimage) in Oauth Our code does use C# for all twitter layers... And thanks for your offer.. all I'm really after is the logic to get the correctly signed POST for this method... You seem to suggest I don't need to sign the image data as it is not urlencoded... So I'll try from there. If you can send me snippets that show the logic to make this work... that might help. Thanks again Simon On Oct 19, 8:32 am, Nicholas Granado ngran...@gmail.com wrote: Simon, You would sign the request with all of the usual oauth param suspects. If I recall correctly this endpoint has no other params other than the 'image' param in the multi-part post body whose value would be the bytes of the image file. Typically I've only seen the post params passed into the oauth signing rigmarole when the post body is urlencoded. I hope this helps, this whole OAuth thing can be very confusing at first glance. If you are in C# I have my own lib for twitter basic auth/oauth that I've baked up, if you like I could pass you the bits. Nicholas --- Nicholas Granado email: ngran...@gmail.com twitter: heatxsink web:http://nickgranado.com On Mon, Oct 19, 2009 at 6:38 AM, Zaudio si...@z-audio.co.uk wrote: Nicholas, That's great feedback! In you opinion, how do I then sign the request? Do I use all the usual for the signaturebase... ie postmethodurlnonceetc etc or just postmethodurl as David suggested? I trust that the image data does not come into the signing process, and that I still can post the data using iso-8859-1 encoding as I would normally do for uploading files? If you have these answers, then I should be able to nail this for our .net case.Oauth's been working great for us until this hitch... Thanks Simon On Oct 18, 6:11 pm, Nicholas Granado ngran...@gmail.com wrote: Simon, I believe the body of your post might be incorrect. It should look like this: POST /account/update_profile_image.xml HTTP/1.1 Content-Type: multipart/form-data; boundary=8cbed79c91b24f3 Host: twitter.com Content-Length: 3863(this will probably change now..) --8cbed79c91b24f3 Content-Disposition: form-data; name=image; filename=test.jpg Content-Type: image/jpeg (there's a few K of binary data here, the contents of the file) --8cbed79c91b24f3 The rest of the OAuth variables should be passed on the query string. I hope this helps. Cheers, Nicholas --- Nicholas Granado email: ngran...@gmail.com twitter: heatxsink web:http://nickgranado.com On Sun, Oct 18, 2009 at 2:42 PM, Zaudio si...@z-audio.co.uk wrote: Hi David, I found your excellent post hoping that it would solve the same challenge for my app: updating profile image via Oauth... using similar .net base to yourself... BUT I just get the 401 all the time... despite taking your advice to just sign with the HTTPmethod URL My post data is laid out much like yours... though I never got that 500 error... I've tried all sorts... dropping the off the end different encodings... What encoding did you use to encode your image, and then to post the request? Does it still work for you... or did this get broken when Twitter 'fixed' their Oauth implementation? Can anyone else advise if they have got this working and where I might be going wrong? Thanks Simon (Zaudio) On Aug 19, 11:40 pm, David Carson carson63...@gmail.com wrote: Got this sorted out and working, and thought I should share the two pitfalls which were causing me problems. First of all, unbelievably, the 500 Internal Server Error was being caused by an extra carriage return between my last HTTP header and the first multipart boundary. Seriously. I had two blank lines in there instead of one. Removed the extra carriage return, and my 500 vanished, being replaced by a more reasonable (401) Unauthorized - Incorrect signature error. Secondly, the OAuth documentation seems a bit shaky when it comes to multipart/form-data POSTs. But basically, you do NOT use any of the POST parameters when creating your signature. And this includes all of the OAuth-specific parameters like oauth_consumer_key, oauth_signature_method, etc. Bit of a security hole imho, OAuth implements all this complexity to avoid man-in-the-middle or replay attacks, and as soon as you do a multipart POST it's all negated. So, my signature base was literally: POSThttp%3A%2F%2Ftwitter.com%2Faccount%2Fupdate_profile_image.xml Just the HTTP method and the URL. No parameters. Once I made that change to the signature generation, my request went through fine and my avatar changed. Hope this helps
[twitter-dev] Re: C# + OAuth + account/update_profile_image = 500 Internal Server Error
Nicholas, That's great feedback! In you opinion, how do I then sign the request? Do I use all the usual for the signaturebase... ie postmethodurlnonceetc etc or just postmethodurl as David suggested? I trust that the image data does not come into the signing process, and that I still can post the data using iso-8859-1 encoding as I would normally do for uploading files? If you have these answers, then I should be able to nail this for our .net case.Oauth's been working great for us until this hitch... Thanks Simon On Oct 18, 6:11 pm, Nicholas Granado ngran...@gmail.com wrote: Simon, I believe the body of your post might be incorrect. It should look like this: POST /account/update_profile_image.xml HTTP/1.1 Content-Type: multipart/form-data; boundary=8cbed79c91b24f3 Host: twitter.com Content-Length: 3863(this will probably change now..) --8cbed79c91b24f3 Content-Disposition: form-data; name=image; filename=test.jpg Content-Type: image/jpeg (there's a few K of binary data here, the contents of the file) --8cbed79c91b24f3 The rest of the OAuth variables should be passed on the query string. I hope this helps. Cheers, Nicholas --- Nicholas Granado email: ngran...@gmail.com twitter: heatxsink web:http://nickgranado.com On Sun, Oct 18, 2009 at 2:42 PM, Zaudio si...@z-audio.co.uk wrote: Hi David, I found your excellent post hoping that it would solve the same challenge for my app: updating profile image via Oauth... using similar .net base to yourself... BUT I just get the 401 all the time... despite taking your advice to just sign with the HTTPmethod URL My post data is laid out much like yours... though I never got that 500 error... I've tried all sorts... dropping the off the end different encodings... What encoding did you use to encode your image, and then to post the request? Does it still work for you... or did this get broken when Twitter 'fixed' their Oauth implementation? Can anyone else advise if they have got this working and where I might be going wrong? Thanks Simon (Zaudio) On Aug 19, 11:40 pm, David Carson carson63...@gmail.com wrote: Got this sorted out and working, and thought I should share the two pitfalls which were causing me problems. First of all, unbelievably, the 500 Internal Server Error was being caused by an extra carriage return between my last HTTP header and the first multipart boundary. Seriously. I had two blank lines in there instead of one. Removed the extra carriage return, and my 500 vanished, being replaced by a more reasonable (401) Unauthorized - Incorrect signature error. Secondly, the OAuth documentation seems a bit shaky when it comes to multipart/form-data POSTs. But basically, you do NOT use any of the POST parameters when creating your signature. And this includes all of the OAuth-specific parameters like oauth_consumer_key, oauth_signature_method, etc. Bit of a security hole imho, OAuth implements all this complexity to avoid man-in-the-middle or replay attacks, and as soon as you do a multipart POST it's all negated. So, my signature base was literally: POSThttp%3A%2F%2Ftwitter.com%2Faccount%2Fupdate_profile_image.xml Just the HTTP method and the URL. No parameters. Once I made that change to the signature generation, my request went through fine and my avatar changed. Hope this helps someone! Cheers, David...
[twitter-dev] Re: Updates to the retweet API payload
Sure, I'll justify this mroe... One of our apps receives updates via the Streaming API and the various REST api methods (mentions, user timelines, friends/home timelines). We collect data as JSON as it's to date been faster and mor compact than alternatives... we can also then go on to use this directly client side in Jscript if we wish... We are caching all updates of interest in a db... thus parse them for the required data fields before storing them there. Currently the parser has to pull out the user node, and is then left with the root status node it is then simple to parse the separated nodes for all fields pertinant to the current operation. We make quick checks initially to determine the relevance of the message to the app's cached stream we want to check things like JUST created_at for the status (and not user) and then check the text property for certain markers. It is easy to find these efficiently in a JSON string without parsing the entire thing to objects as things stand... so we save a lot of server cpu cycles. It's fastest to this from the inner node outwards... this is where the 'wasted' cpu cycles are coming in here with the change for retweeted_status Status's of interest only have the further fields parsed that we want for out db copy... for example.. say you want to quickly check the id of the status to confirm if you have it in you db already or not... currently we just excluded the user node, and thent he id is in the remainder without conflict. Now add retweeted_status with it's user subnode To now get the id of the root status without parsing the entire string to objects... we pull it apart again from the inner node outwards... we now need to exclude the retweeted_status user subnode... this no longer has a unique start tag/definition... as there are TWO identical start tags in the string so we do a lot more work to ensure we get the retweeted_status and it's user node that we would if it had an alternative start tag. If it were instead retweeted_user, then we could extract that directly and easily, exclude it, then exclude the surrounding retweeted_status tag... and we've got the retweeted_status node separated... then we can procede as we do now with the rest... and if necessary use the retweeted_status as well. Hope that makes sense I agree that keeping it as user also makes good object sense... but then the retweeted_status is not 'status' anymore... and it is a status I'm suggesting soemthing similar for it's inner node.. Simon (Zaudio) On Oct 6, 12:01 pm, Marco Kaiser kaiser.ma...@gmail.com wrote: No, please don't change that to retweeted_user ... the data structure included as the retweeted status is a status, and that data structure has a user property. That's a very clear object model, and should map very well to JSON, as it's nested, not at the same level as the main user the retweet is received from. So by doing that change, you'd break the data model for a status, in that there are two version that need to be taken care of. Or can you explain in more depth why this would cause problems with reasonable JSON parsers that map strings to objects? 2009/10/6 Zaudio si...@z-audio.co.uk Another significant thought... could you 'please' consider changing the name of the user node INSIDE the retweeted_status node to say retweeted_user ? Thius will make JSON parsing way simpler... especially if the goal is to extract the retweeted_status when present; or do things like quickly find the date of the tweet... I alreayd have to contend with a created_at field in the user and status nodes... now that could double up... so owuld appreciate an easier to find retweeted user node for JSON parsability Thanks Simon (Zaudio) On Oct 4, 8:16 am, John Kalucki jkalu...@gmail.com wrote: Retweetis an invasive feature with many deep dependency paths. Firm dates would be useful, but they aren't possible in this particular situation. This makes planning for downstream folks difficult. I'd be ready for the slight possibility of low-volume retweets mid-to- late week, with a high chance the following week, and perhaps a near- unity chance of low-volume retweets the week after that. So, for critical code, any time now. As for full-roll-out, that would be even more of a guessing game. -John Kaluckihttp://twitter.com/jkalucki Services, Twitter Inc. On Oct 4, 6:43 am, Zaudio si...@z-audio.co.uk wrote: Hey John, Thanks for that... can you put an earliest date on 'very soon' please - just so I know how long we've got? Thanks Simon (Zaudio) On Oct 3, 8:15 pm, John Kalucki jkalu...@gmail.com wrote: There are plans to filter retweets from various resource, see the documentation. However, it would be most prudent to assume that retweets will eventually show up everywhere, and handle them appropriately, or at least tolerate them wherever they are found
[twitter-dev] Re: Updates to the retweet API payload
We use JSON thoughout... as JSON and also parsing to db fields... I love that JSON converts directly to OO but then again what works for one app, may hinder others... no ideal solution. I suspect that things will stay as they are anyway... so your our app lose on this one and will just have to do more during our to db parsing operations ... Simon On Oct 7, 7:52 am, Marco Kaiser kaiser.ma...@gmail.com wrote: Well, you're not making use of JSON as JSON, and instead use brute-force methods to extract parts of it... I think it's a bit unfair to request that change to be made, as it would complicate things for everyone doing real JSON-to-OO mapping. Just my 2c. And no, retweet_status is perfectly valid - it's the property name in a status model, and it is assigned a status data model. Just a nested object. You don't have to name a member status just because it is a Status data type... Marco 2009/10/7 Zaudio si...@z-audio.co.uk Sure, I'll justify this mroe... One of our apps receives updates via the Streaming API and the various REST api methods (mentions, user timelines, friends/home timelines). We collect data as JSON as it's to date been faster and mor compact than alternatives... we can also then go on to use this directly client side in Jscript if we wish... We are caching all updates of interest in a db... thus parse them for the required data fields before storing them there. Currently the parser has to pull out the user node, and is then left with the root status node it is then simple to parse the separated nodes for all fields pertinant to the current operation. We make quick checks initially to determine the relevance of the message to the app's cached stream we want to check things like JUST created_at for the status (and not user) and then check the text property for certain markers. It is easy to find these efficiently in a JSON string without parsing the entire thing to objects as things stand... so we save a lot of server cpu cycles. It's fastest to this from the inner node outwards... this is where the 'wasted' cpu cycles are coming in here with the change for retweeted_status Status's of interest only have the further fields parsed that we want for out db copy... for example.. say you want to quickly check the id of the status to confirm if you have it in you db already or not... currently we just excluded the user node, and thent he id is in the remainder without conflict. Now add retweeted_status with it's user subnode To now get the id of the root status without parsing the entire string to objects... we pull it apart again from the inner node outwards... we now need to exclude the retweeted_status user subnode... this no longer has a unique start tag/definition... as there are TWO identical start tags in the string so we do a lot more work to ensure we get the retweeted_status and it's user node that we would if it had an alternative start tag. If it were instead retweeted_user, then we could extract that directly and easily, exclude it, then exclude the surrounding retweeted_status tag... and we've got the retweeted_status node separated... then we can procede as we do now with the rest... and if necessary use the retweeted_status as well. Hope that makes sense I agree that keeping it as user also makes good object sense... but then the retweeted_status is not 'status' anymore... and it is a status I'm suggesting soemthing similar for it's inner node.. Simon (Zaudio) On Oct 6, 12:01 pm, Marco Kaiser kaiser.ma...@gmail.com wrote: No, please don't change that to retweeted_user ... the data structure included as the retweeted status is a status, and that data structure has a user property. That's a very clear object model, and should map very well to JSON, as it's nested, not at the same level as the main user the retweet is received from. So by doing that change, you'd break the data model for a status, in that there are two version that need to be taken care of. Or can you explain in more depth why this would cause problems with reasonable JSON parsers that map strings to objects? 2009/10/6 Zaudio si...@z-audio.co.uk Another significant thought... could you 'please' consider changing the name of the user node INSIDE the retweeted_status node to say retweeted_user ? Thius will make JSON parsing way simpler... especially if the goal is to extract the retweeted_status when present; or do things like quickly find the date of the tweet... I alreayd have to contend with a created_at field in the user and status nodes... now that could double up... so owuld appreciate an easier to find retweeted user node for JSON parsability Thanks Simon (Zaudio) On Oct 4, 8:16 am, John Kalucki jkalu...@gmail.com wrote: Retweetis an invasive feature with many deep dependency paths
[twitter-dev] Re: Updates to the retweet API payload
Another significant thought... could you 'please' consider changing the name of the user node INSIDE the retweeted_status node to say retweeted_user ? Thius will make JSON parsing way simpler... especially if the goal is to extract the retweeted_status when present; or do things like quickly find the date of the tweet... I alreayd have to contend with a created_at field in the user and status nodes... now that could double up... so owuld appreciate an easier to find retweeted user node for JSON parsability Thanks Simon (Zaudio) On Oct 4, 8:16 am, John Kalucki jkalu...@gmail.com wrote: Retweetis an invasive feature with many deep dependency paths. Firm dates would be useful, but they aren't possible in this particular situation. This makes planning for downstream folks difficult. I'd be ready for the slight possibility of low-volume retweets mid-to- late week, with a high chance the following week, and perhaps a near- unity chance of low-volume retweets the week after that. So, for critical code, any time now. As for full-roll-out, that would be even more of a guessing game. -John Kaluckihttp://twitter.com/jkalucki Services, Twitter Inc. On Oct 4, 6:43 am, Zaudio si...@z-audio.co.uk wrote: Hey John, Thanks for that... can you put an earliest date on 'very soon' please - just so I know how long we've got? Thanks Simon (Zaudio) On Oct 3, 8:15 pm, John Kalucki jkalu...@gmail.com wrote: There are plans to filter retweets from various resource, see the documentation. However, it would be most prudent to assume that retweets will eventually show up everywhere, and handle them appropriately, or at least tolerate them wherever they are found. They should start appearing in low volumes in all /1/statuses/* resources on the Streaming API very soon. -John Kaluckihttp://twitter.com/jkalucki Services, Twitter Inc. On Oct 3, 10:33 am, Zaudio si...@z-audio.co.uk wrote: This sounds a lot more sensible... Importantly, where can we expect this newpayloadto be returned... any of the following methods as well? REST API statuses/mentions statuses/user Streaming API/Shadow I need to know in advance of such an addition to any of these, as otherwise the parser on one of our apps gets broken until it's recoded to handle theretweetpayload... or is the ONLY was to get these vie theretweetmethods and the new home_timeline ? If so, what about apps that mainly make use of the Streaming API for tweets? Thanks Simon (Zaudio)- Hide quoted text - - Show quoted text -
[twitter-dev] Re: Updates to the retweet API payload
Hey John, Thanks for that... can you put an earliest date on 'very soon' please - just so I know how long we've got? Thanks Simon (Zaudio) On Oct 3, 8:15 pm, John Kalucki jkalu...@gmail.com wrote: There are plans to filter retweets from various resource, see the documentation. However, it would be most prudent to assume that retweets will eventually show up everywhere, and handle them appropriately, or at least tolerate them wherever they are found. They should start appearing in low volumes in all /1/statuses/* resources on the Streaming API very soon. -John Kaluckihttp://twitter.com/jkalucki Services, Twitter Inc. On Oct 3, 10:33 am, Zaudio si...@z-audio.co.uk wrote: This sounds a lot more sensible... Importantly, where can we expect this newpayloadto be returned... any of the following methods as well? REST API statuses/mentions statuses/user Streaming API/Shadow I need to know in advance of such an addition to any of these, as otherwise the parser on one of our apps gets broken until it's recoded to handle theretweetpayload... or is the ONLY was to get these vie theretweetmethods and the new home_timeline ? If so, what about apps that mainly make use of the Streaming API for tweets? Thanks Simon (Zaudio)
[twitter-dev] Re: Updates to the retweet API payload
This sounds a lot more sensible... Importantly, where can we expect this new payload to be returned... any of the following methods as well? REST API statuses/mentions statuses/user Streaming API/Shadow I need to know in advance of such an addition to any of these, as otherwise the parser on one of our apps gets broken until it's recoded to handle the retweet payload... or is the ONLY was to get these vie the retweet methods and the new home_timeline ? If so, what about apps that mainly make use of the Streaming API for tweets? Thanks Simon (Zaudio)
[twitter-dev] Re: Invalid / used nonce
My trial has worked so far today... not a single invalid nonce error but it's only been 12 hours Simon On Aug 12, 12:59 pm, Rob O'Brien r...@zepoid.com wrote: The interesting thing with my situation is that I'm still in development, so there's only a single person (me) hitting the app. Further, I'm only attempting a single call to Twitter. Also, I get a 401 on everything that requires authentication, but not on something like a rateLimitStatus check. Further, a call to /followers/ids.xml *works* on my local dev box, but not on the production server. The only difference I can think of would be IP address. I've been able to trace 3 separate requests being generated by Twitter4J and here are the values: [Wed Aug 12 10:19:56 PDT 2009] oauth_timestamp=1250097596,oauth_nonce=329444963 [Wed Aug 12 10:20:20 PDT 2009] oauth_timestamp=1250097620,oauth_nonce=173112023 [Wed Aug 12 10:24:39 PDT 2009] oauth_timestamp=1250097879,oauth_nonce=3202768030 Each timestamp is larger than the last and eachnonceis unique. Knowing that my values are legit makes me think there's another problem, but Twitter hasn't responded to my api@ email. Rob O'Brien Web Application Developer Consultant r...@zepoid.com -Original Message- From: twitter-development-talk@googlegroups.com [mailto:twitter-development-t...@googlegroups.com] On Behalf Of Zaudio Sent: Tuesday, August 11, 2009 12:04 PM To: Twitter Development Talk Subject: [twitter-dev] Re: Invalid /usednonce We're having the same issue in our app, occurs sporadically in our logs - but I believe the cause with us is that: We're generatingnoncevalues as a timestamp seeded sequence of random numbers We're creating an instance of the Oauth class that does this for each logged in user for the app Thus, for a single timestamp, it IS possible that the time seedednoncevalues are the same So - corrrective action being trialled: I'm prefixing the 'random'noncevalue with the userID stripped from the start of the token, padded to a fixed length of chars... this should guarantee then that thenonce/timestamp combo is indeed unique for every request made from our app Simon On Aug 11, 6:45 am, Dan Borthwick dan.borthw...@playfish.com wrote: For our app, we successfully call request_token from our server. When we then call statuses/update from the client, we get a 401 'Invalid / usednonce' response. If the request_token call comes directly from the client, the update call succeeds. The nonces have been sanity checked and are definitely different for each call. GET requests to users/show succeed regardless of whether the request_token comes from the proxy server or client. Code is based on MGTwitterEngine-1.0.8-OAuth. The same code was working ok prior to the recent DoS downtime. Perhaps something has been changed on Twitter's side that might result in the 401 response? On Aug 11, 8:38 am, graceawalker grace_blo...@hotmail.com wrote: No, mynonceis definately new every time. Surely if there was something wrong with the way it was being generated it would error during requestToken/accessToken/VerifyCredentials too?? All the code ive looked through is doing it exactly the same way. Is the 'status' parameter beingusedjust like all the oauth parameters? is an 'invalidnonce' error, definately an invalidnonceor could it be to do with the timestamp and timezones. Clutching at straws here... On Aug 11, 3:12 am, Chris Babcock cbabc...@asciiking.com wrote: On Mon, 10 Aug 2009 04:14:43 -0700 (PDT) graceawalker grace_blo...@hotmail.com wrote: I am calling and getting the whole way up to getting the access token just fine in my app (one im writing myself in c#), but when i try and call the update status URL im getting an 'Invalid/usednonce' error in my response data. Im not sure why this is, im calling the update method in the exact same way that i called request token apart from the new 'status' parameter in the query string. I call 'verify credentials' with my access token to ensure that it is working and it sends me back all of the correct data, but it is erroring when trying to update my status. Is there any obvious solution to this, or am i not supposed to be signing and organising the parameters in the same way that i did before? Im really stuck here guys and need help! Right, thenonceis a numberusedonce. Its purpose is to prevent replay attacks. If you use the samenoncefor more than one call to the API then you *should* be getting an error. Chris- Hide quoted text - - Show quoted text -- Hide quoted text - - Show quoted text -
[twitter-dev] Re: Rate Limit Change?
I get the same with my apps; an authenticated and unauthenticated call to get rate limits returns the same hits available out of the 20K... never managed to get an answer why I'm guessing the authentication is being ignored, and I just get IP limits all the time? Just go to www.bullsonwallstreet.com?rate_limits=1 and look just above the footer... after the call completes, you see the two rate limits display and refresh every 10 sec or so... weird Simon On Aug 12, 1:51 pm, shiplu shiplu@gmail.com wrote: If its called from a whitelisted IP, It will be charged to IP. Not account. -- A K M Mokaddimhttp://talk.cmyweb.nethttp://twitter.com/shiplu Stop Top Posting !! বাংলিশ লেখার চাইতে বাংলা লেখা অনেক ভাল Sent from Dhaka, Bangladesh
[twitter-dev] Re: Invalid / used nonce
We're having the same issue in our app, occurs sporadically in our logs - but I believe the cause with us is that: We're generating nonce values as a timestamp seeded sequence of random numbers We're creating an instance of the Oauth class that does this for each logged in user for the app Thus, for a single timestamp, it IS possible that the time seeded nonce values are the same So - corrrective action being trialled: I'm prefixing the 'random' nonce value with the userID stripped from the start of the token, padded to a fixed length of chars... this should guarantee then that the nonce/timestamp combo is indeed unique for every request made from our app Simon On Aug 11, 6:45 am, Dan Borthwick dan.borthw...@playfish.com wrote: For our app, we successfully call request_token from our server. When we then call statuses/update from the client, we get a 401 'Invalid / usednonce' response. If the request_token call comes directly from the client, the update call succeeds. The nonces have been sanity checked and are definitely different for each call. GET requests to users/show succeed regardless of whether the request_token comes from the proxy server or client. Code is based on MGTwitterEngine-1.0.8-OAuth. The same code was working ok prior to the recent DoS downtime. Perhaps something has been changed on Twitter's side that might result in the 401 response? On Aug 11, 8:38 am, graceawalker grace_blo...@hotmail.com wrote: No, mynonceis definately new every time. Surely if there was something wrong with the way it was being generated it would error during requestToken/accessToken/VerifyCredentials too?? All the code ive looked through is doing it exactly the same way. Is the 'status' parameter being used just like all the oauth parameters? is an 'invalidnonce' error, definately an invalidnonceor could it be to do with the timestamp and timezones. Clutching at straws here... On Aug 11, 3:12 am, Chris Babcock cbabc...@asciiking.com wrote: On Mon, 10 Aug 2009 04:14:43 -0700 (PDT) graceawalker grace_blo...@hotmail.com wrote: I am calling and getting the whole way up to getting the access token just fine in my app (one im writing myself in c#), but when i try and call the update status URL im getting an 'Invalid/usednonce' error in my response data. Im not sure why this is, im calling the update method in the exact same way that i called request token apart from the new 'status' parameter in the query string. I call 'verify credentials' with my access token to ensure that it is working and it sends me back all of the correct data, but it is erroring when trying to update my status. Is there any obvious solution to this, or am i not supposed to be signing and organising the parameters in the same way that i did before? Im really stuck here guys and need help! Right, thenonceis a number used once. Its purpose is to prevent replay attacks. If you use the samenoncefor more than one call to the API then you *should* be getting an error. Chris- Hide quoted text - - Show quoted text -
[twitter-dev] Re: API Calls During DoS Attack
I'm getting the ame problem with bullsonwallstreet.com - previous whitelisted rates of 2 now down to 150... not recovered yet. And I throttle all requests to a pretty low level for the REST API... but still down at 150! Let's hope that this attack ends soon, and honest users can have the performance needed back again soon! Simon On Aug 7, 7:48 am, Hedley Robertson hedley.robert...@gmail.com wrote: Yes seems like this is some sort of IP based blocking that they introduced, since one of my production servers started failing yesterday, then the other server, on a different IP, which was consistantly working, started failing later in the evening. Any suggestions on who can I contact directly to get this resolved? I filled out the 'whitelisting form' just now, but never had to worry about it in the past as my application is not abusive with rate limits, and not sure if this is the best channel anyway, since its more of an incorrect / misapplied blacklisting issue, it would seem? Hedley On Fri, Aug 7, 2009 at 3:42 AM, Dewald Pretorius dpr...@gmail.com wrote: They are definitely still actively blocking all volume requests. I noticed this morning that my website was working. Checked, and my rate limit was back to 20,000. So, I switched on one of my cron jobs, and within less than 5 minutes all requests from my IP were being completely blocked again. Wonder just how big are these woods that Twitter has to come out of. Dewald- Hide quoted text - - Show quoted text -
[twitter-dev] Re: twitter api server seems to be down (getting invalid signature) since 5.15 pm pst
Mine were all to do with urlencoding, and ensuring all post data is sent as post data, and none on the querystring for the URL; and thus ensuring I use the modified URLencoding method required for Oauth - which was previously being used, but not on the post data! Once I did that, that test updates with all sorts of characters now succeed every time... including spacesetc etc Seems good now... hope it stays that way! Good luck to you guys! On Jul 27, 10:46 pm, Duane Roelands duane.roela...@gmail.com wrote: From my experimenting, it appears that posting a tweet is successful if the text contains no spaces. Once you have a space in the tweet, it fails. Researching... On Jul 28, 12:29 am, winrich winric...@gmail.com wrote: ok guys. so my calls were failing on the verify_credentials call and not on the update or timeline calls. the only difference i saw was the the verify_credential call wasn't secured. i changed it to https and it worked. ??? lol On Jul 27, 9:19 pm, Chad Etzel jazzyc...@gmail.com wrote: On Mon, Jul 27, 2009 at 11:55 PM, Duane Roelandsduane.roela...@gmail.com wrote: RTFM is not a helpful answer, especially when many developers are relying on libraries that they did not write. That's a risk you run when using code you didn't write. I'm not saying that this situation doesn't suck for those affected. I'm sure that it does. But, for a technology so new as OAuth, the libraries may not be mature yet. Officially, Twitter OAuth is still in Public Beta and has never been officially recommended to integrate into production code. That being said, there could still be a problem on Twitter's end with their signature verification mechanism and the libraries could all be valid. I don't have a way of knowing. I do agree that at least a note that a security change was pushed today would be nice, though. -Chad
[twitter-dev] Re: twitter api server seems to be down (getting invalid signature) since 5.15 pm pst
I don't myself think that twitter are doing anything more than enforcing the standard. I spent 3 hours 'fixing' my code for our application (uses Shannon Whitley's c# library as a base); I only found two bugs in thelibrary that caused any problem... the use of httputility.urlencode in place of the modified urlencode already implemented for use with Oauth, as required by the spec... the library just wasn't using it in two places. Following that, I found I was passing some bad strings to the library... so the methods in it were not urldecoding and re- urlencoding the postdata as required to get it to meet the spec. Once I fixed that also, all seems good. So if our app now posts all data fine over Oauth, it suggests that twitter's interface is OK? Simon On Jul 28, 8:16 am, BlueSkies scarter28m-goo...@yahoo.com wrote: I'm the developer who reported this problem to Twitter offline. I had been preparing a test case for them when I noticed this post:http://groups.google.com/group/twitter-development-talk/browse_thread... I then skipped the preparation of the test case and sent e-mail directly to Twitter since this was what I considered to be a high risk issue. Since the fix from Twitter went in for this last night, I have also seen some strange behaviour with OAuth. I'm getting a lot of 401 responses for users who were already logged in. I'm also getting sporadic bad responses for accesses that ask for a request token (BTW - I'm using https if that makes any difference). I think that there is still a problem. I've switched my application back to using Basic Auth for now. - Scott @scott_carterhttp://bigtweet.com
