[twitter-dev] Re: Slow response to twitter updates for a third party app
Hi Glenn, Sounds like a good idea. It would be nice to have some sort of monitoring on the API. It would make it easier to determine where something is going wrong. Someone just mentioned http://api-status.com/ which seems to go into the direction you propose, even for several APIs. Also see http://api-status.com/6404/113966/Twitter-friend-timeline-API http://twitter.com/api_status/ Jaap
[twitter-dev] Re: Slow response to twitter updates for a third party app
John, can you already tell us something about the latency issue? Is it your opinion that the many connect failures I am seeing are something on my end? Or could that be correlated with the timeouts? Thanks, Jaap
Re: [twitter-dev] Re: Slow response to twitter updates for a third party app
You might want to take a look at the service that Apigee provides at apigee.com, Glenn. Taylor Singletary Developer Advocate, Twitter http://twitter.com/episod On Tue, May 11, 2010 at 1:21 AM, glenn gillen gl...@rubypond.com wrote: On May 11, 4:45 am, John Kalucki j...@twitter.com wrote: Now that we have a reasonable idea about what is transpiring, I'd venture to say that the latency distribution will be widest between about 6:30am to 10:30am PDT (13:30-17:30 UTC), and considerably less so until perhaps 5pm PDT. The balance of the day should be OK. At Warblecamp on the weekend someone (@mario) mentioned it would be nice have a means of identifying (or at least being aware of) these sorts of issues. And given the size of twitter these days and the considerable usage of the API, there are so many variables that can impact on performance differently for every user that it can be difficult to know if a problem is isolated to your code or is affecting a wide population. So floating an idea to promote some further discussion and see if there is any interest, obvious issues with the approach, input, etc. Would it be worthwhile having an independent service that allowed developers to programmatically log their current API performance and issues? For those that use Rails I'm thinking something along the lines of rpm.newrelic but specifically twitter focussed. It could post any 5xx error responses as they occur and regularly ping details about the process usage (CPU utilization, RAM, etc.). In isolation they're not very useful stats, in aggregation they'd help identify specific areas suffering problems like 80% of our users in The Netherlands are currently experiencing severe latency issues or system wide issues like a particular call failing, So would it be of any use? I'm not a consumer of the API anywhere near the scale of Twitterfeed and so I don't currently see the same requirement for such a service, I'm more inclined to believe if I have a problem it's almost always mine to deal with. The great thing about services like rpm.newrelic and hoptoad is that they give you actionable information, and while I think this would be an interesting technical challenge I wonder if it's actually providing users anything actionable. Thoughts? -- Glenn Gillen http://glenngillen.com/
Re: [twitter-dev] Re: Slow response to twitter updates for a third party app
It's probably on our end. I'll post some advice. On Wed, May 12, 2010 at 12:24 AM, Tjaap jdmeij...@gmail.com wrote: John, can you already tell us something about the latency issue? Is it your opinion that the many connect failures I am seeing are something on my end? Or could that be correlated with the timeouts? Thanks, Jaap
[twitter-dev] Re: Slow response to twitter updates for a third party app
FWIW, our application has been experiencing these same issues for several weeks now. I've seen repeated overcapacity errors being returned (the too many tweets HTML page) on both GET and POST requests. I don't have specifics on likely times of the day, though US Central nighttime seems much less likely than weekly business hours. Our servers are located in Dallas, TX. Let me know if there's anything else that would be helpful for me to provide. Thanks, Aaron Sprout Social On May 12, 9:54 am, John Kalucki j...@twitter.com wrote: It's probably on our end. I'll post some advice. On Wed, May 12, 2010 at 12:24 AM, Tjaap jdmeij...@gmail.com wrote: John, can you already tell us something about the latency issue? Is it your opinion that the many connect failures I am seeing are something on my end? Or could that be correlated with the timeouts? Thanks, Jaap
[twitter-dev] Re: Slow response to twitter updates for a third party app
On May 11, 4:45 am, John Kalucki j...@twitter.com wrote: Now that we have a reasonable idea about what is transpiring, I'd venture to say that the latency distribution will be widest between about 6:30am to 10:30am PDT (13:30-17:30 UTC), and considerably less so until perhaps 5pm PDT. The balance of the day should be OK. At Warblecamp on the weekend someone (@mario) mentioned it would be nice have a means of identifying (or at least being aware of) these sorts of issues. And given the size of twitter these days and the considerable usage of the API, there are so many variables that can impact on performance differently for every user that it can be difficult to know if a problem is isolated to your code or is affecting a wide population. So floating an idea to promote some further discussion and see if there is any interest, obvious issues with the approach, input, etc. Would it be worthwhile having an independent service that allowed developers to programmatically log their current API performance and issues? For those that use Rails I'm thinking something along the lines of rpm.newrelic but specifically twitter focussed. It could post any 5xx error responses as they occur and regularly ping details about the process usage (CPU utilization, RAM, etc.). In isolation they're not very useful stats, in aggregation they'd help identify specific areas suffering problems like 80% of our users in The Netherlands are currently experiencing severe latency issues or system wide issues like a particular call failing, So would it be of any use? I'm not a consumer of the API anywhere near the scale of Twitterfeed and so I don't currently see the same requirement for such a service, I'm more inclined to believe if I have a problem it's almost always mine to deal with. The great thing about services like rpm.newrelic and hoptoad is that they give you actionable information, and while I think this would be an interesting technical challenge I wonder if it's actually providing users anything actionable. Thoughts? -- Glenn Gillen http://glenngillen.com/
[twitter-dev] Re: Slow response to twitter updates for a third party app
I’m also seeing a ~5 second delay in responses to requests for http://api.twitter.com/1/statuses/user_timeline.xml. It’s consistently 5-7 seconds, but appears to happen both before the response is sent as well as midway, it sort of feels like both a timeout waiting for something to happen as well as a buffering thing. I can send a tcpdump report additional details offlist. -- -ed costello
[twitter-dev] Re: Slow response to twitter updates for a third party app
John, Chart of failed pings originating in Moab, Utah... http://tweetprobe.tumblr.com/post/587169206
[twitter-dev] Re: Slow response to twitter updates for a third party app
delayed tweet:ping traceroute 64 bytes from 128.242.240.61: icmp_seq=9 ttl=244 time=36.851 ms --- api.twitter.com ping statistics --- 10 packets transmitted, 8 packets received, 20% packet loss round-trip min/avg/max/stddev = 36.851/37.725/39.607/0.902 ms -- traceroute to api.twitter.com (128.242.240.93), 64 hops max, 40 byte packets 1 10.0.1.1 (10.0.1.1) 1.337 ms 0.661 ms 0.678 ms 2 192.168.254.254 (192.168.254.254) 2.009 ms 1.609 ms 0.977 ms 3 dr01.moab.ut.frontiernet.net (74.40.8.109) 8.835 ms 7.860 ms 9.103 ms 4 74.40.41.233 (74.40.41.233) 14.145 ms 13.365 ms 14.576 ms 5 xe--0-2-0---0.cor01.slkc.ut.frontiernet.net (74.40.4.1) 13.959 ms 14.443 ms 14.768 ms 6 ae1---0.cor02.plal.ca.frontiernet.net (74.40.5.61) 34.496 ms 32.226 ms 32.357 ms 7 ae1---0.cbr01.plal.ca.frontiernet.net (74.40.3.170) 60.735 ms 65.466 ms 35.187 ms 8 xe-0.paix.plalca01.us.bb.gin.ntt.net (198.32.176.14) 34.242 ms 36.110 ms 33.274 ms 9 xe-1-1-0.r21.mlpsca01.us.bb.gin.ntt.net (129.250.3.50) 30.914 ms 37.517 ms 30.781 ms 10 mg-2.c20.mlpsca01.us.da.verio.net (129.250.29.81) 34.434 ms 36.171 ms 34.288 ms 11 128.241.122.213 (128.241.122.213) 36.377 ms 34.334 ms 33.771 ms 12 * 128.241.122.213 (128.241.122.213) 34.956 ms !X * 13 * * * -
Re: [twitter-dev] Re: Slow response to twitter updates for a third party app
We're pretty sure that this isn't a connectivity issue. At least, it's not *just* a connectivity issue. -John Kalucki http://twitter.com/jkalucki Infrastructure, Twitter Inc. On Mon, May 10, 2010 at 2:06 PM, mikawhite mikawh...@me.com wrote: delayed tweet:ping traceroute 64 bytes from 128.242.240.61: icmp_seq=9 ttl=244 time=36.851 ms --- api.twitter.com ping statistics --- 10 packets transmitted, 8 packets received, 20% packet loss round-trip min/avg/max/stddev = 36.851/37.725/39.607/0.902 ms -- traceroute to api.twitter.com (128.242.240.93), 64 hops max, 40 byte packets 1 10.0.1.1 (10.0.1.1) 1.337 ms 0.661 ms 0.678 ms 2 192.168.254.254 (192.168.254.254) 2.009 ms 1.609 ms 0.977 ms 3 dr01.moab.ut.frontiernet.net (74.40.8.109) 8.835 ms 7.860 ms 9.103 ms 4 74.40.41.233 (74.40.41.233) 14.145 ms 13.365 ms 14.576 ms 5 xe--0-2-0---0.cor01.slkc.ut.frontiernet.net (74.40.4.1) 13.959 ms 14.443 ms 14.768 ms 6 ae1---0.cor02.plal.ca.frontiernet.net (74.40.5.61) 34.496 ms 32.226 ms 32.357 ms 7 ae1---0.cbr01.plal.ca.frontiernet.net (74.40.3.170) 60.735 ms 65.466 ms 35.187 ms 8 xe-0.paix.plalca01.us.bb.gin.ntt.net (198.32.176.14) 34.242 ms 36.110 ms 33.274 ms 9 xe-1-1-0.r21.mlpsca01.us.bb.gin.ntt.net (129.250.3.50) 30.914 ms 37.517 ms 30.781 ms 10 mg-2.c20.mlpsca01.us.da.verio.net (129.250.29.81) 34.434 ms 36.171 ms 34.288 ms 11 128.241.122.213 (128.241.122.213) 36.377 ms 34.334 ms 33.771 ms 12 * 128.241.122.213 (128.241.122.213) 34.956 ms !X * 13 * * * -
[twitter-dev] Re: Slow response to twitter updates for a third party app
Hi John and Ryan, Thanks for looking into this. The good news is that now I am seeing faster responses for my website. I did not change anything on my end, but maybe you guys did, or the traffic was more well-behaved :) In any case, here are my impressions of using the twitter APIs. Use case: Posting updates to Twitter using oAuth API for the case when the consumer app already has the access token. Weekdays mornings (Pacific Time) are generally slower than evenings or weekends. Two timed experiments showed it took 3 sec, and 2 sec for posting two consecutive updates on Monday evening. The first message often takes longer than subsequent updates for the same access token (any token caching going on here ?). Here is a traceroute output taken Monday evening: traceroute to twitter.com (168.143.162.36), 30 hops max, 40 byte packets 1 ip-173-201-183-251.ip.secureserver.net (173.201.183.251) 0.918 ms 0.977 ms 1.106 ms 2 ip-208-109-113-169.ip.secureserver.net (208.109.113.169) 1.505 ms 1.490 ms 1.499 ms 3 ip-208-109-113-158.ip.secureserver.net (208.109.113.158) 1.445 ms 1.424 ms 1.412 ms 4 ip-208-109-112-162.ip.secureserver.net (208.109.112.162) 1.386 ms 1.395 ms 1.429 ms 5 ip-208-109-112-138.ip.secureserver.net (208.109.112.138) 1.463 ms 1.515 ms 1.584 ms 6 xe-0-2-0.mpr3.phx2.us.above.net (64.124.196.37) 1.092 ms 1.184 ms 1.142 ms 7 ge-0-3-0.mpr3.lax9.us.above.net (64.125.28.70) 37.857 ms 37.838 ms 37.811 ms 8 xe-0-1-0.er1.lax9.us.above.net (64.125.31.89) 38.317 ms 38.306 ms 38.290 ms 9 xe-0-1-0.mpr1.lax12.us.above.net (64.125.31.189) 9.359 ms 9.476 ms 9.461 ms 10 * * * 11 ae-1.r20.lsanca03.us.bb.gin.ntt.net (129.250.5.253) 10.309 ms * * 12 * * * 13 ae-0.r20.snjsca04.us.bb.gin.ntt.net (129.250.2.96) 89.869 ms 56.178 ms * 14 ae-2.r20.mlpsca01.us.bb.gin.ntt.net (129.250.5.6) 62.913 ms 62.933 ms * 15 * * * 16 128.241.122.117 (128.241.122.117) 60.817 ms 59.320 ms 59.353 ms 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * I would be interested in any pointers you guys might have related to faster response times. Thanks for all your help! -Raj On May 10, 2:51 pm, John Kalucki j...@twitter.com wrote: We're pretty sure that this isn't a connectivity issue. At least, it's not *just* a connectivity issue. -John Kaluckihttp://twitter.com/jkalucki Infrastructure, Twitter Inc. On Mon, May 10, 2010 at 2:06 PM, mikawhite mikawh...@me.com wrote: delayed tweet:ping traceroute 64 bytes from 128.242.240.61: icmp_seq=9 ttl=244 time=36.851 ms --- api.twitter.com ping statistics --- 10 packets transmitted, 8 packets received, 20% packet loss round-trip min/avg/max/stddev = 36.851/37.725/39.607/0.902 ms -- traceroute to api.twitter.com (128.242.240.93), 64 hops max, 40 byte packets 1 10.0.1.1 (10.0.1.1) 1.337 ms 0.661 ms 0.678 ms 2 192.168.254.254 (192.168.254.254) 2.009 ms 1.609 ms 0.977 ms 3 dr01.moab.ut.frontiernet.net (74.40.8.109) 8.835 ms 7.860 ms 9.103 ms 4 74.40.41.233 (74.40.41.233) 14.145 ms 13.365 ms 14.576 ms 5 xe--0-2-0---0.cor01.slkc.ut.frontiernet.net (74.40.4.1) 13.959 ms 14.443 ms 14.768 ms 6 ae1---0.cor02.plal.ca.frontiernet.net (74.40.5.61) 34.496 ms 32.226 ms 32.357 ms 7 ae1---0.cbr01.plal.ca.frontiernet.net (74.40.3.170) 60.735 ms 65.466 ms 35.187 ms 8 xe-0.paix.plalca01.us.bb.gin.ntt.net (198.32.176.14) 34.242 ms 36.110 ms 33.274 ms 9 xe-1-1-0.r21.mlpsca01.us.bb.gin.ntt.net (129.250.3.50) 30.914 ms 37.517 ms 30.781 ms 10 mg-2.c20.mlpsca01.us.da.verio.net (129.250.29.81) 34.434 ms 36.171 ms 34.288 ms 11 128.241.122.213 (128.241.122.213) 36.377 ms 34.334 ms 33.771 ms 12 * 128.241.122.213 (128.241.122.213) 34.956 ms !X * 13 * * * -
Re: [twitter-dev] Re: Slow response to twitter updates for a third party app
Now that we have a reasonable idea about what is transpiring, I'd venture to say that the latency distribution will be widest between about 6:30am to 10:30am PDT (13:30-17:30 UTC), and considerably less so until perhaps 5pm PDT. The balance of the day should be OK. -John Kalucki http://twitter.com/jkalucki Infrastructure, Twitter Inc. On Mon, May 10, 2010 at 8:25 PM, raj raj4...@yahoo.com wrote: Hi John and Ryan, Thanks for looking into this. The good news is that now I am seeing faster responses for my website. I did not change anything on my end, but maybe you guys did, or the traffic was more well-behaved :) In any case, here are my impressions of using the twitter APIs. Use case: Posting updates to Twitter using oAuth API for the case when the consumer app already has the access token. Weekdays mornings (Pacific Time) are generally slower than evenings or weekends. Two timed experiments showed it took 3 sec, and 2 sec for posting two consecutive updates on Monday evening. The first message often takes longer than subsequent updates for the same access token (any token caching going on here ?). Here is a traceroute output taken Monday evening: traceroute to twitter.com (168.143.162.36), 30 hops max, 40 byte packets 1 ip-173-201-183-251.ip.secureserver.net (173.201.183.251) 0.918 ms 0.977 ms 1.106 ms 2 ip-208-109-113-169.ip.secureserver.net (208.109.113.169) 1.505 ms 1.490 ms 1.499 ms 3 ip-208-109-113-158.ip.secureserver.net (208.109.113.158) 1.445 ms 1.424 ms 1.412 ms 4 ip-208-109-112-162.ip.secureserver.net (208.109.112.162) 1.386 ms 1.395 ms 1.429 ms 5 ip-208-109-112-138.ip.secureserver.net (208.109.112.138) 1.463 ms 1.515 ms 1.584 ms 6 xe-0-2-0.mpr3.phx2.us.above.net (64.124.196.37) 1.092 ms 1.184 ms 1.142 ms 7 ge-0-3-0.mpr3.lax9.us.above.net (64.125.28.70) 37.857 ms 37.838 ms 37.811 ms 8 xe-0-1-0.er1.lax9.us.above.net (64.125.31.89) 38.317 ms 38.306 ms 38.290 ms 9 xe-0-1-0.mpr1.lax12.us.above.net (64.125.31.189) 9.359 ms 9.476 ms 9.461 ms 10 * * * 11 ae-1.r20.lsanca03.us.bb.gin.ntt.net (129.250.5.253) 10.309 ms * * 12 * * * 13 ae-0.r20.snjsca04.us.bb.gin.ntt.net (129.250.2.96) 89.869 ms 56.178 ms * 14 ae-2.r20.mlpsca01.us.bb.gin.ntt.net (129.250.5.6) 62.913 ms 62.933 ms * 15 * * * 16 128.241.122.117 (128.241.122.117) 60.817 ms 59.320 ms 59.353 ms 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * I would be interested in any pointers you guys might have related to faster response times. Thanks for all your help! -Raj On May 10, 2:51 pm, John Kalucki j...@twitter.com wrote: We're pretty sure that this isn't a connectivity issue. At least, it's not *just* a connectivity issue. -John Kaluckihttp://twitter.com/jkalucki Infrastructure, Twitter Inc. On Mon, May 10, 2010 at 2:06 PM, mikawhite mikawh...@me.com wrote: delayed tweet:ping traceroute 64 bytes from 128.242.240.61: icmp_seq=9 ttl=244 time=36.851 ms --- api.twitter.com ping statistics --- 10 packets transmitted, 8 packets received, 20% packet loss round-trip min/avg/max/stddev = 36.851/37.725/39.607/0.902 ms -- traceroute to api.twitter.com (128.242.240.93), 64 hops max, 40 byte packets 1 10.0.1.1 (10.0.1.1) 1.337 ms 0.661 ms 0.678 ms 2 192.168.254.254 (192.168.254.254) 2.009 ms 1.609 ms 0.977 ms 3 dr01.moab.ut.frontiernet.net (74.40.8.109) 8.835 ms 7.860 ms 9.103 ms 4 74.40.41.233 (74.40.41.233) 14.145 ms 13.365 ms 14.576 ms 5 xe--0-2-0---0.cor01.slkc.ut.frontiernet.net (74.40.4.1) 13.959 ms 14.443 ms 14.768 ms 6 ae1---0.cor02.plal.ca.frontiernet.net (74.40.5.61) 34.496 ms 32.226 ms 32.357 ms 7 ae1---0.cbr01.plal.ca.frontiernet.net (74.40.3.170) 60.735 ms 65.466 ms 35.187 ms 8 xe-0.paix.plalca01.us.bb.gin.ntt.net (198.32.176.14) 34.242 ms 36.110 ms 33.274 ms 9 xe-1-1-0.r21.mlpsca01.us.bb.gin.ntt.net (129.250.3.50) 30.914 ms 37.517 ms 30.781 ms 10 mg-2.c20.mlpsca01.us.da.verio.net (129.250.29.81) 34.434 ms 36.171 ms 34.288 ms 11 128.241.122.213 (128.241.122.213) 36.377 ms 34.334 ms 33.771 ms 12 * 128.241.122.213 (128.241.122.213) 34.956 ms !X * 13 * * * -
[twitter-dev] Re: Slow response to twitter updates for a third party app
Hi, sorry I didn't get back to you earlier. I was only monitoring this thread: http://code.google.com/p/twitter-api/issues/detail?id=1543 Right now about 10% of our post fails immediately. Latency is between 0.7s and 3s, with peaks to 7 or 8 seconds. We are not logging tcp dumps at the moment, but I will see what I can do. I also will try to get some more data from our logs to see when this is happening the most. Thanks Ryan and John for looking into it!
[twitter-dev] Re: Slow response to twitter updates for a third party app
The friend I am developing the app with, tells me that we can't log tcp traffic ourselves. We'd have to get our host provider involved. What I do have, is the logfile of updates: http://dl.dropbox.com/u/6030727/httpresponse_latency.xls I started this log on Friday. As you can see the connections (in seconds in the third column) took very long on Friday (times are Central European Time). It had been like this since Wednesday or Thursday. The times are limited by the timeout we set in cURL, we played around with that value a little to find out what was going on. The http response sometimes is 0, which cURL returns in 0 seconds. Sometimes the http response is 0 after a long wait. Hope this helps. If you need anything else, let me know, I'm @tjaap. Jaap
Re: [twitter-dev] Re: Slow response to twitter updates for a third party app
When you say that 10% fail immediately, do you mean that 10% fail in less than a second? What is the failure indication? This sounds a lot like a problem on your server, perhaps a shared resource exhaustion -- connections, memory, file descriptors, etc. I see the similar latency from here in San Francisco, on a weekend--- say 80% 0 seconds, 5% 1 second, 4% 2 seconds, 1% 3 seconds, and none beyond 3 seconds. During a weekday, I could see how latency might increase somewhat beyond this for very short periods. I don't see any test tweets failing, with curl or with a Scala program, after hundreds of POSTs. Are you seeing requests fail that take longer than 8 seconds? Also, when the latency increases to 8 seconds, can you run a ping to www.twitter.com and post the summary information at the end of the run? Where, geographically, are your servers located? On Sun, May 9, 2010 at 4:23 AM, Tjaap jdmeij...@gmail.com wrote: Hi, sorry I didn't get back to you earlier. I was only monitoring this thread: http://code.google.com/p/twitter-api/issues/detail?id=1543 Right now about 10% of our post fails immediately. Latency is between 0.7s and 3s, with peaks to 7 or 8 seconds. We are not logging tcp dumps at the moment, but I will see what I can do. I also will try to get some more data from our logs to see when this is happening the most. Thanks Ryan and John for looking into it!
Re: [twitter-dev] Re: Slow response to twitter updates for a third party app
Where are your servers located geographically? If you don't see an hourly correlation, do you see a strong time-of-day correlation or day-of-wee correlation? Can you run a ping from your servers during a period of instability and send the summary at the end of the run? On Sat, May 8, 2010 at 3:00 PM, Naveen Ayyagari nav...@getsocialscope.com wrote: I dont have tcp dumps available.. Ill try to get some captures later this weekend.. Ill also run through the logs to see if they have any correlation with top of the hour, but my gut instinct is they do not. Anywhere specific I should send tcp dumps, as they may contain username and passwords of our users (we are still in the process of switching over to OAuth)? Should I just XXX those out? On May 8, 1:50 pm, Ryan Sarver rsar...@twitter.com wrote: Raj, Naveen, @tjaap, Do any of you still have tcp dumps of the calls you were making that were getting long timeouts? On Sat, May 8, 2010 at 12:08 AM, Naveen Ayyagari nav...@getsocialscope.comwrote: We see the same huge latency and timeouts as well (our timeouts are also at 30 seconds). We running out of a US data center on multiple machines, we see this issue on all if our servers. I agree with @tjaap, would like to hear twitters reaction as well. On May 7, 6:02 pm, Tjaap jdmeij...@gmail.com wrote: I'm seeing exactly the same: big latency and a lot of timeouts. When posting a tweet, Twitter sometimes does not return a response within 30 seconds. After my curl times out, the app has no way of knowing if the post made it or not. Sometimes it will appear on Twitter and sometimes it won't. This effectively breaks our app. Replicated the error on two different servers (in the Netherlands) and with different accounts. I would very much appreciate a reaction from Twitter on this. We are in the process of switching to OAuth, but I would like to know if I have to implement a workaround while we're on the old system. Thanks, @tjaap
Re: [twitter-dev] Re: Slow response to twitter updates for a third party app
Same questions for everyone having issues: Where are your servers located geographically? If you don't see an hourly correlation, do you see a strong time-of-day correlation or day-of-wee correlation? Can you run a ping from your servers during a period of instability and send the summary at the end of the run? If the ping times are long, a traceroute may be useful too. On Sat, May 8, 2010 at 1:15 PM, mikawhite mikawh...@me.com wrote: john - I just scanned DM's and tweets that were slow or had no response from twitter - none were at top of hour - hth
[twitter-dev] Re: Slow response to twitter updates for a third party app
Our servers are in Amsterdam, in the Netherlands. Response time varies a lot. When I ping www.twitter.com from the server, this is what I get back: PING twitter.com (168.143.162.68) 56(84) bytes of data. 2 packets transmitted, 2 received, 0% packet loss, time 1014ms rtt min/avg/max/mdev = 203.355/209.525/215.696/6.187 ms And when I ping again immediately: rtt min/avg/max/mdev = 154.659/157.198/159.738/2.570 ms What's funny, is that when I ping from my home computer (Nijmegen, the Netherlands) it resolves www.twitter.com to a different IP and it is consistently faster: Ping statistics for 128.121.243.228: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss) Approximate round trip times in milli-seconds: Minimum = 178ms, Maximum = 183ms, Average = 179ms
[twitter-dev] Re: Slow response to twitter updates for a third party app
We see the same huge latency and timeouts as well (our timeouts are also at 30 seconds). We running out of a US data center on multiple machines, we see this issue on all if our servers. I agree with @tjaap, would like to hear twitters reaction as well. On May 7, 6:02 pm, Tjaap jdmeij...@gmail.com wrote: I'm seeing exactly the same: big latency and a lot of timeouts. When posting a tweet, Twitter sometimes does not return a response within 30 seconds. After my curl times out, the app has no way of knowing if the post made it or not. Sometimes it will appear on Twitter and sometimes it won't. This effectively breaks our app. Replicated the error on two different servers (in the Netherlands) and with different accounts. I would very much appreciate a reaction from Twitter on this. We are in the process of switching to OAuth, but I would like to know if I have to implement a workaround while we're on the old system. Thanks, @tjaap
Re: [twitter-dev] Re: Slow response to twitter updates for a third party app
Raj, Naveen, @tjaap, Do any of you still have tcp dumps of the calls you were making that were getting long timeouts? On Sat, May 8, 2010 at 12:08 AM, Naveen Ayyagari nav...@getsocialscope.comwrote: We see the same huge latency and timeouts as well (our timeouts are also at 30 seconds). We running out of a US data center on multiple machines, we see this issue on all if our servers. I agree with @tjaap, would like to hear twitters reaction as well. On May 7, 6:02 pm, Tjaap jdmeij...@gmail.com wrote: I'm seeing exactly the same: big latency and a lot of timeouts. When posting a tweet, Twitter sometimes does not return a response within 30 seconds. After my curl times out, the app has no way of knowing if the post made it or not. Sometimes it will appear on Twitter and sometimes it won't. This effectively breaks our app. Replicated the error on two different servers (in the Netherlands) and with different accounts. I would very much appreciate a reaction from Twitter on this. We are in the process of switching to OAuth, but I would like to know if I have to implement a workaround while we're on the old system. Thanks, @tjaap
Re: [twitter-dev] Re: Slow response to twitter updates for a third party app
Any chance that the timeouts are correlated with the top of the hour? -John Kalucki http://twitter.com/jkalucki Infrastructure, Twitter Inc. On Sat, May 8, 2010 at 12:08 AM, Naveen Ayyagari nav...@getsocialscope.com wrote: We see the same huge latency and timeouts as well (our timeouts are also at 30 seconds). We running out of a US data center on multiple machines, we see this issue on all if our servers. I agree with @tjaap, would like to hear twitters reaction as well. On May 7, 6:02 pm, Tjaap jdmeij...@gmail.com wrote: I'm seeing exactly the same: big latency and a lot of timeouts. When posting a tweet, Twitter sometimes does not return a response within 30 seconds. After my curl times out, the app has no way of knowing if the post made it or not. Sometimes it will appear on Twitter and sometimes it won't. This effectively breaks our app. Replicated the error on two different servers (in the Netherlands) and with different accounts. I would very much appreciate a reaction from Twitter on this. We are in the process of switching to OAuth, but I would like to know if I have to implement a workaround while we're on the old system. Thanks, @tjaap
[twitter-dev] Re: Slow response to twitter updates for a third party app
john - I just scanned DM's and tweets that were slow or had no response from twitter - none were at top of hour - hth
[twitter-dev] Re: Slow response to twitter updates for a third party app
I dont have tcp dumps available.. Ill try to get some captures later this weekend.. Ill also run through the logs to see if they have any correlation with top of the hour, but my gut instinct is they do not. Anywhere specific I should send tcp dumps, as they may contain username and passwords of our users (we are still in the process of switching over to OAuth)? Should I just XXX those out? On May 8, 1:50 pm, Ryan Sarver rsar...@twitter.com wrote: Raj, Naveen, @tjaap, Do any of you still have tcp dumps of the calls you were making that were getting long timeouts? On Sat, May 8, 2010 at 12:08 AM, Naveen Ayyagari nav...@getsocialscope.comwrote: We see the same huge latency and timeouts as well (our timeouts are also at 30 seconds). We running out of a US data center on multiple machines, we see this issue on all if our servers. I agree with @tjaap, would like to hear twitters reaction as well. On May 7, 6:02 pm, Tjaap jdmeij...@gmail.com wrote: I'm seeing exactly the same: big latency and a lot of timeouts. When posting a tweet, Twitter sometimes does not return a response within 30 seconds. After my curl times out, the app has no way of knowing if the post made it or not. Sometimes it will appear on Twitter and sometimes it won't. This effectively breaks our app. Replicated the error on two different servers (in the Netherlands) and with different accounts. I would very much appreciate a reaction from Twitter on this. We are in the process of switching to OAuth, but I would like to know if I have to implement a workaround while we're on the old system. Thanks, @tjaap
[twitter-dev] Re: Slow response to twitter updates for a third party app
I'm seeing exactly the same: big latency and a lot of timeouts. When posting a tweet, Twitter sometimes does not return a response within 30 seconds. After my curl times out, the app has no way of knowing if the post made it or not. Sometimes it will appear on Twitter and sometimes it won't. This effectively breaks our app. Replicated the error on two different servers (in the Netherlands) and with different accounts. I would very much appreciate a reaction from Twitter on this. We are in the process of switching to OAuth, but I would like to know if I have to implement a workaround while we're on the old system. Thanks, @tjaap
