[twitter-dev] Re: The silence is deafening....
I remember another service that used to have constant problems due to their astronomical growth ... what was it called? Oh, yeah, Friendster. What ever happened to them? Oh, there was also a service that became wildly popular and grew faster than it could handle ... remember playing a Flash version of Pac-Man every time there was an error ... what was it called? Oh, yeah, MySpace. What ever happened to them? I think I'm starting to see a pattern, here. On 8/9/09 2:00 PM, David Fisher wrote: The reason they are still facing issues is because of the DDoS. Oh, and they are still also coping with scaling overall and astronomical growth. -- Dossy Shiobara | do...@panoptic.com | http://dossy.org/ Panoptic Computer Network | http://panoptic.com/ "He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on." (p. 70)
[twitter-dev] Re: The silence is deafening....
I'm disgusted by you guys. Were any of you around in 2007 when Twitter was down nonstop? Did you bitch and moan as much then? You're acting like children. The reason they are still facing issues is because of the DDoS. Oh, and they are still also coping with scaling overall and astronomical growth. A DDoS that is shifting and trying to thwart them isn't easy to stand up to- moreso if you were just able to handle the load of your standard business traffic anyway. Its very clear that many of you have NEVER worked in a startup or technology environment. Its no like everyone from Biz down to the receptionist can focus all their magical energy on Twitter at the same time to make it work. I've worked for a startup before when our service/site has gone down. There were one, maybe two people in the company who had the ability, authorization or knowledge to do it. The rest of us were as frustrated as hell, but even if one of us was the cause (a bad query to the database or something) that didn't mean that we could all instantly fix it. Companies have different roles for people and it doesn't help at all for EVERYONE to SSH to the same box and try to muck with the settings to fix things. All hands are on deck that need to be. Everyone else at Twitter can't do anything about it, but is stressing out too. Chad not working today isn't a sign of them not working on the stuff. He just joined the team, and isn't in the engineering/operations department. Him flying out to California to stand and stare at the others working on the issue won't help. Different people have different roles and his role isn't to fix this stuff. If something happens at your office, does everyone from the CEO to the receptionist jump on the issue all at once? No, they can't. It doesn't make sense. Stop bitching and complaining. You're making yourselves look unprofessional and bad. -David Fisher Web Ecology Project http://webecologyproject.org On Aug 9, 2:51 am, chinaski007 wrote: > And, by the way, if you're a deckhand on a submarine going down, you > think you would go to a movie because it's your time off, or do > whatever you can to help out? > > On Aug 8, 11:47 pm, chinaski007 wrote: > > > Part of it is DDOS response, part of it is API issues... as one of the > > updates most recently noted with the "known issues" list. > > > On Aug 8, 11:42 pm, Bill Kocik wrote: > > > > On Aug 9, 2:28 am, chinaski007 wrote: > > > > > You're wrong. > > > > > If you check the tweets of the other main Twitter developers, you will > > > > see that they are doing sushi, rock concerts, weddings, watching > > > > movies on Saturday afternoon, etc. And while "getting married" is > > > > certainly a legitimate excuse, some of the other activities, during > > > > this major upheaval, are certainly questionable! > > > > Oh...I get it, you think that the developers are the right people to > > > handle a DDoS attack. I'm afraid I have some bad news for you: you're > > > wrong. > > > > And because not *all* the developers are sweating it out at Twitter > > > HQ, that means no one's working on it. I can't argue with that logic. > > > Seriously, I can't.
[twitter-dev] Re: The silence is deafening....
And just to add, managing the issue in that manner also facilitates external communications. Then regular status updates can be crafted, and approved by senior management, to go out to all affected parties including the developer community. And Chad won't have to run around like crazy trying to chase down people for an update. Dewald On Aug 9, 12:18 pm, Dewald Pretorius wrote: > Why is it important that staff members from all departments need to be > at the office during issues like these? > > If not for anything else, it is important for internal communication > and coordination, and for efficient management of the issue. > > Take the spate of META REFRESH issues that have come up as an example. > > Someone, probably in the Operations team, decided it would be a good > idea to also do META refreshes or redirects, to flush out the bots > that cannot handle those requests. > > Except, they did not understand how it was going to impact the rest of > the Twitter environment, specifically third-party app scripts that > also cannot handle those requests. In their minds, it was probably a > good idea because they only thought about normal browser behavior. > > If you have someone in senior management in the office to manage and > coordinate, and key people from all departments, you can have status > and coordination meetings throughout the day, and night for that > matter, where people can ask, "This is what we want to do, how will it > impact you guys?" > > Dewald
[twitter-dev] Re: The silence is deafening....
Why is it important that staff members from all departments need to be at the office during issues like these? If not for anything else, it is important for internal communication and coordination, and for efficient management of the issue. Take the spate of META REFRESH issues that have come up as an example. Someone, probably in the Operations team, decided it would be a good idea to also do META refreshes or redirects, to flush out the bots that cannot handle those requests. Except, they did not understand how it was going to impact the rest of the Twitter environment, specifically third-party app scripts that also cannot handle those requests. In their minds, it was probably a good idea because they only thought about normal browser behavior. If you have someone in senior management in the office to manage and coordinate, and key people from all departments, you can have status and coordination meetings throughout the day, and night for that matter, where people can ask, "This is what we want to do, how will it impact you guys?" Dewald
[twitter-dev] Re: The silence is deafening....
On 8/9/09 2:42 AM, Bill Kocik wrote: Oh...I get it, you think that the developers are the right people to handle a DDoS attack. I'm afraid I have some bad news for you: you're wrong. So, wait, you believe that Ops people are writing code and putting it in place in an emergency to defend against this DDoS? You think Ops people are the best suited to writing code under pressure and putting it directly into production? I'm afraid I have some bad news for you: you're wrong. :-) -- Dossy Shiobara | do...@panoptic.com | http://dossy.org/ Panoptic Computer Network | http://panoptic.com/ "He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on." (p. 70)
[twitter-dev] Re: The silence is deafening....
On 8/9/09 1:58 AM, chinaski007 wrote: Too, while we may love programming, many of us do this sort of thing as our primary means of surviving and paying the mortgage. Here's some free business consulting for you: DON'T BUILD YOUR PRIMARY BUSINESS ON TOP OF TWITTER. Like it even needed to be said in the first place ... -- Dossy Shiobara | do...@panoptic.com | http://dossy.org/ Panoptic Computer Network | http://panoptic.com/ "He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on." (p. 70)
[twitter-dev] Re: The silence is deafening....
Sandros, I think you are very mistaken, I would say the same if Twitter wasn't running a business based off of growing their base using a Free API, Twitter chose to have a free API and it is supported as such, no guarantees or warranties - however that isn't the point - the Free API is the lifeblood to the service and without it all the applications built on it coursing through its veins there isn't much of a service, just a website, so in my opinion the Free API is probably the most important part of their business and it is broken. Lots of businesses have grown up around Twitter and these business are unable to operate - in much the same way as when a postal strike occurs. I have refunded or given service credits in the order of £250 this weekend alone for access to my service. I can't complain too much, by good grace I have managed to build a profitable venture, but on the flip side the entire situation is so frustrating, I have just taken a 20% pay cut at my current employer to help them through some tougher times so my Twitter business was covering that 20%, if this situation continues for another week I will probably shut down Twollo, it won't be worth running anymore. I am pretty sure that Twitter are working as hard as they can on sorting the problem, but the situation is a valid one and we are right to openly complain, to be honest I am totally surprised this whole situation hasn't been on the likes of techcrunch as it is an ongoing issue that is causing so much consternation it is unbelievable. Your comment about laconica is a factious one, how can one have a fallback for a Twitter service The fallback at the moment is no business at all. I personally thank Chad for his efforts so far, he has been thrown in at the deep end and is probably in the same boat as most of us. Paul 2009/8/9 Sandro Ducceschi > > I can't believe all of you people. > The API is a free service and if it's down or not working for a > while, > you just sit down and take it like a grown up instead of complaining > and demanding that they send in all forces on a weekend. > > Some people do have lifes outside of twitter i heard being said.. > > And if it's such a big deal that your application / system needs to > run 24/7, you > should have thought of it beforehand and built some sort of fallback > system (laconica yeh?). > > In all honesty, if i had a say, i would make sure some of you would be > purposely blocked for a while just because of your statements. > > Have a nice and relaxing Sunday.
[twitter-dev] Re: The silence is deafening....
I can't believe all of you people. The API is a free service and if it's down or not working for a while, you just sit down and take it like a grown up instead of complaining and demanding that they send in all forces on a weekend. Some people do have lifes outside of twitter i heard being said.. And if it's such a big deal that your application / system needs to run 24/7, you should have thought of it beforehand and built some sort of fallback system (laconica yeh?). In all honesty, if i had a say, i would make sure some of you would be purposely blocked for a while just because of your statements. Have a nice and relaxing Sunday.
[twitter-dev] Re: The silence is deafening....
Chad, you'd definitely be captain of my sinking submarine! ;) On Aug 9, 12:33 am, Chad Etzel wrote: > Hi There, > > First, in my own defense, I feel that my original "part-time/day off" > statement was misconstrued. I was trying to make that point that even > though I am way over my hours on my contract, I still monitored the > situation all day from both sides (dev list and internal ops > information). I think you will find that I posted in several threads, > and I did help several individuals offlist. I'm still awake monitoring > things (but not for much longer at this rate). > > I did this because: > a) I am a fellow third-party app dev on the same sinking submarine. > b) I also feel a lack of communication sucks. > > Again, nothing new really to report, so no "official updates" have > been made aside from the Streaming API update this morning. Basically, > SNAFU. > > Second, you obviously don't know all of the usernames of people that > work here. There is an incredible ops team here working as we type all > of these emails. I'm not going to give names because they don't need > any more distractions or you checking up on them. Call that > non-transparent if you wish; I call it letting them get their job > done. > > We have top men working on it right now. Top. Men. > > Thanks you, as always, for your continued patience in the matter > -Chad > > On Sun, Aug 9, 2009 at 3:19 AM, chinaski007 wrote: > > > My point was that my browsing of the tweetstreams of the Twitter > > engineers I am familiar with, ops and otherwise, reveals another > > normal weekend, with all the loveliness that the Bay Area has to > > offer... and while there may be a bunch of Keebler elves drinking > > coffee and working hard, I don't see much evidence of that from my > > sampling. Do you? > > > On Aug 9, 12:08 am, Bill Kocik wrote: > > >> What was yours? > >
[twitter-dev] Re: The silence is deafening....
On Aug 9, 3:19 am, chinaski007 wrote: > My point was that my browsing of the tweetstreams of the Twitter > engineers I am familiar with, ops and otherwise, reveals another > normal weekend, with all the loveliness that the Bay Area has to > offer... and while there may be a bunch of Keebler elves drinking > coffee and working hard, I don't see much evidence of that from my > sampling. Do you? If you mean sampling of their tweets, I don't follow any of them. If you mean sampling of the current API behavior, I have to admit I've been extremely lucky. I haven't really seen any issues since Thursday other than inability to initiate new sessions via OAuth - and everyone who uses my app already has a live session open, so as long as they don't hit the sign out button they should be fine. I'll say that I'm damn glad I didn't send out a bunch of beta invites Wednesday night as I was on the verge of doing, but instead decided to wait until Thursday morning. Now I'm waiting 'til Monday, or until things are back online, whichever comes later. I guess I'd actually have to say that yes, I have seen evidence of their work, though, in that the service has managed to make some considerable comebacks while the DDoS intensity has increased (or so we're told). Does that mean they're huddled around glowing monitors right this minute? No, it doesn't, and I don't think we've seen significant improvement since things largely came back on Friday. But honestly, I don't know what they can do in short order (mostly because I'm just a simple software engineer with a background in Unix systems administration, not a network guy).
[twitter-dev] Re: The silence is deafening....
Hi There, First, in my own defense, I feel that my original "part-time/day off" statement was misconstrued. I was trying to make that point that even though I am way over my hours on my contract, I still monitored the situation all day from both sides (dev list and internal ops information). I think you will find that I posted in several threads, and I did help several individuals offlist. I'm still awake monitoring things (but not for much longer at this rate). I did this because: a) I am a fellow third-party app dev on the same sinking submarine. b) I also feel a lack of communication sucks. Again, nothing new really to report, so no "official updates" have been made aside from the Streaming API update this morning. Basically, SNAFU. Second, you obviously don't know all of the usernames of people that work here. There is an incredible ops team here working as we type all of these emails. I'm not going to give names because they don't need any more distractions or you checking up on them. Call that non-transparent if you wish; I call it letting them get their job done. We have top men working on it right now. Top. Men. Thanks you, as always, for your continued patience in the matter -Chad On Sun, Aug 9, 2009 at 3:19 AM, chinaski007 wrote: > > > My point was that my browsing of the tweetstreams of the Twitter > engineers I am familiar with, ops and otherwise, reveals another > normal weekend, with all the loveliness that the Bay Area has to > offer... and while there may be a bunch of Keebler elves drinking > coffee and working hard, I don't see much evidence of that from my > sampling. Do you? > > On Aug 9, 12:08 am, Bill Kocik wrote: > >> What was yours?
[twitter-dev] Re: The silence is deafening....
Ok, I think everyone sees your point, chinasky. On Sun, Aug 9, 2009 at 12:26 AM, chinaski007 wrote: > > > See here: > > > http://groups.google.com/group/twitter-development-talk/browse_thread/thread/acdcb4baf76037c8/d5f4d4204b65d617#d5f4d4204b65d617 > > On Aug 9, 12:17 am, Bill Kocik wrote: > > > I remember seeing "200 errors" somewhere, but I didn't read the > > details. 200 means status okay, what's the indication of error? >
[twitter-dev] Re: The silence is deafening....
See here: http://groups.google.com/group/twitter-development-talk/browse_thread/thread/acdcb4baf76037c8/d5f4d4204b65d617#d5f4d4204b65d617 On Aug 9, 12:17 am, Bill Kocik wrote: > I remember seeing "200 errors" somewhere, but I didn't read the > details. 200 means status okay, what's the indication of error?
[twitter-dev] Re: The silence is deafening....
My point was that my browsing of the tweetstreams of the Twitter engineers I am familiar with, ops and otherwise, reveals another normal weekend, with all the loveliness that the Bay Area has to offer... and while there may be a bunch of Keebler elves drinking coffee and working hard, I don't see much evidence of that from my sampling. Do you? On Aug 9, 12:08 am, Bill Kocik wrote: > What was yours?
[twitter-dev] Re: The silence is deafening....
On Aug 9, 3:03 am, chinaski007 wrote: > Yep, for sure. And maybe the rash of new 200 errors I remember seeing "200 errors" somewhere, but I didn't read the details. 200 means status okay, what's the indication of error? > But WHO in API is "day on" to communicate with us? Now *that's* a fair question.
[twitter-dev] Re: The silence is deafening....
On Aug 9, 2:51 am, chinaski007 wrote: > And, by the way, if you're a deckhand on a submarine going down, you > think you would go to a movie because it's your time off, or do > whatever you can to help out? Submarines are supposed to go down. And I don't think you can really go to a movie if you're on one. See what I did there? I completely missed your point - as you missed mine. Mine was that Chad is not the one working on the problem, and that his being "off" has no bearing on how soon solutions will arive; he's not even in (either of) the right department(s). What was yours?
[twitter-dev] Re: The silence is deafening....
Yep, for sure. And maybe the rash of new 200 errors, and OAuth errors, etc, are not under the purview of API (though I would doubt it). But, at the very least, it's over another, what, 48 hours since the last official update. And those developer updates are presumably the responsibility of API. And all we've got is breadcrumbs dropped from the Carolinas by a new part-timer who tells us it is his day off. Okay, fair. But WHO in API is "day on" to communicate with us? On Aug 8, 11:55 pm, Cameron Kaiser wrote: > > Part of it is DDOS response, part of it is API issues... as one of the > > updates most recently noted with the "known issues" list. > > And, by the way, if you're a deckhand on a submarine going down, you > > think you would go to a movie because it's your time off, or do > > whatever you can to help out? > > If the API is down because of the DDoS pounding on ops, then it doesn't matter > what the API devs are doing or not doing: it's not going out. I'm not sure > if you mean that the API devs should be working on the network problem, but > where I came from, crossing the discipline streams usually turns out to be > staggeringly unhelpful. I didn't tell the network guys how to run their > infrastructure, and they didn't tell me how to run my servers. > > -- > personal:http://www.cameronkaiser.com/-- > Cameron Kaiser * Floodgap Systems *www.floodgap.com* ckai...@floodgap.com > -- "Use gun kata for fun! Because you worth it!" > --
[twitter-dev] Re: The silence is deafening....
> Part of it is DDOS response, part of it is API issues... as one of the > updates most recently noted with the "known issues" list. > And, by the way, if you're a deckhand on a submarine going down, you > think you would go to a movie because it's your time off, or do > whatever you can to help out? If the API is down because of the DDoS pounding on ops, then it doesn't matter what the API devs are doing or not doing: it's not going out. I'm not sure if you mean that the API devs should be working on the network problem, but where I came from, crossing the discipline streams usually turns out to be staggeringly unhelpful. I didn't tell the network guys how to run their infrastructure, and they didn't tell me how to run my servers. -- personal: http://www.cameronkaiser.com/ -- Cameron Kaiser * Floodgap Systems * www.floodgap.com * ckai...@floodgap.com -- "Use gun kata for fun! Because you worth it!" --
[twitter-dev] Re: The silence is deafening....
And, by the way, if you're a deckhand on a submarine going down, you think you would go to a movie because it's your time off, or do whatever you can to help out? On Aug 8, 11:47 pm, chinaski007 wrote: > Part of it is DDOS response, part of it is API issues... as one of the > updates most recently noted with the "known issues" list. > > On Aug 8, 11:42 pm, Bill Kocik wrote: > > > On Aug 9, 2:28 am, chinaski007 wrote: > > > > You're wrong. > > > > If you check the tweets of the other main Twitter developers, you will > > > see that they are doing sushi, rock concerts, weddings, watching > > > movies on Saturday afternoon, etc. And while "getting married" is > > > certainly a legitimate excuse, some of the other activities, during > > > this major upheaval, are certainly questionable! > > > Oh...I get it, you think that the developers are the right people to > > handle a DDoS attack. I'm afraid I have some bad news for you: you're > > wrong. > > > And because not *all* the developers are sweating it out at Twitter > > HQ, that means no one's working on it. I can't argue with that logic. > > Seriously, I can't. > >
[twitter-dev] Re: The silence is deafening....
Part of it is DDOS response, part of it is API issues... as one of the updates most recently noted with the "known issues" list. On Aug 8, 11:42 pm, Bill Kocik wrote: > On Aug 9, 2:28 am, chinaski007 wrote: > > > You're wrong. > > > If you check the tweets of the other main Twitter developers, you will > > see that they are doing sushi, rock concerts, weddings, watching > > movies on Saturday afternoon, etc. And while "getting married" is > > certainly a legitimate excuse, some of the other activities, during > > this major upheaval, are certainly questionable! > > Oh...I get it, you think that the developers are the right people to > handle a DDoS attack. I'm afraid I have some bad news for you: you're > wrong. > > And because not *all* the developers are sweating it out at Twitter > HQ, that means no one's working on it. I can't argue with that logic. > Seriously, I can't.
[twitter-dev] Re: The silence is deafening....
On Aug 9, 2:28 am, chinaski007 wrote: > You're wrong. > > If you check the tweets of the other main Twitter developers, you will > see that they are doing sushi, rock concerts, weddings, watching > movies on Saturday afternoon, etc. And while "getting married" is > certainly a legitimate excuse, some of the other activities, during > this major upheaval, are certainly questionable! Oh...I get it, you think that the developers are the right people to handle a DDoS attack. I'm afraid I have some bad news for you: you're wrong. And because not *all* the developers are sweating it out at Twitter HQ, that means no one's working on it. I can't argue with that logic. Seriously, I can't.
[twitter-dev] Re: The silence is deafening....
You're wrong. If you check the tweets of the other main Twitter developers, you will see that they are doing sushi, rock concerts, weddings, watching movies on Saturday afternoon, etc. And while "getting married" is certainly a legitimate excuse, some of the other activities, during this major upheaval, are certainly questionable! On Aug 8, 11:20 pm, Bill Kocik wrote: > On Aug 8, 6:33 pm, Dewald Pretorius wrote: > > > However, I would hope that Twitter engineers are all in force at the > > office on a day like this to solve this issue and get our applications > > back up and running, regardless of whether it is Saturday, Sunday, or > > Christmas Day. > > I think you're making the assumption that because Chad is off the > clock today, all of Twitter engineering is as well. Hint: Chad isn't > Twitter engineering (or operations, for that matter), he's part-time > support for API consumers. He's also in North Carolina, not SFO. Chad > is not the one working to fend off the DDoS. > > I imagine there are a whole bunch of people on the West Coast running > on very little sleep and way too much coffee trying to fight this > thing. > > (None of this is meant to indict or vindicate anyone - I'm simply > saying, don't take "Chad has today off" to mean "Twitter's not doing > anything.")
[twitter-dev] Re: The silence is deafening....
On Aug 8, 6:33 pm, Dewald Pretorius wrote: > However, I would hope that Twitter engineers are all in force at the > office on a day like this to solve this issue and get our applications > back up and running, regardless of whether it is Saturday, Sunday, or > Christmas Day. I think you're making the assumption that because Chad is off the clock today, all of Twitter engineering is as well. Hint: Chad isn't Twitter engineering (or operations, for that matter), he's part-time support for API consumers. He's also in North Carolina, not SFO. Chad is not the one working to fend off the DDoS. I imagine there are a whole bunch of people on the West Coast running on very little sleep and way too much coffee trying to fight this thing. (None of this is meant to indict or vindicate anyone - I'm simply saying, don't take "Chad has today off" to mean "Twitter's not doing anything.")
[twitter-dev] Re: The silence is deafening....
As other media have noted, when Twitter goes down, people swap to FriendFeed, Facebook, etc. The same thing happens when Twitter apps go down. The problem with this outage is that it largely effects third-party Web-based apps. And so when our apps go down, for whatever reason, people swap to desktop competitors. As most of us are smaller Mum n Pop shops, such an exodus, even fractional, is significant. Too, while we may love programming, many of us do this sort of thing as our primary means of surviving and paying the mortgage. On Aug 8, 8:40 pm, Adam Cloud wrote: > Some of us program because we love to do it, not because of the revenue we > could make off the third-party app we use. > > Man up and just tell your users to be patient, it's not like they're going > to stop using your app because of some well publicized downtime, and if they > are, then it wasn't that great of an app to begin with.
[twitter-dev] Re: The silence is deafening....
Some of us program because we love to do it, not because of the revenue we could make off the third-party app we use. Man up and just tell your users to be patient, it's not like they're going to stop using your app because of some well publicized downtime, and if they are, then it wasn't that great of an app to begin with.
[twitter-dev] Re: The silence is deafening....
third-party app we created* On Sat, Aug 8, 2009 at 9:40 PM, Adam Cloud wrote: > Some of us program because we love to do it, not because of the revenue we > could make off the third-party app we use. > > Man up and just tell your users to be patient, it's not like they're going > to stop using your app because of some well publicized downtime, and if they > are, then it wasn't that great of an app to begin with. >
[twitter-dev] Re: The silence is deafening....
If spoofing of white-listed IP addresses is a concern to Twitter (and it probably is), I have a proxy infrastructure in place with already white-listed IP addresses that can make API calls from IP addresses that are not the same as my website IP address. It will take one hell of a lucky guess by anyone to guess those IP addresses. It is perhaps something all larger apps should put in place, namely the ability to use white-listed IP addresses that are not the same as your easily identifiable website IP address. Dewald
[twitter-dev] Re: The silence is deafening....
I agree. I also think it is very important to recognize Twitter made a strong move with such an open API. As a result, it is just as important to recognize, Twitter very well may not be where it is today were it not for third party apps. I may go as far as to say the API should be a higher priority than the Twitter.com site. I can say with zero doubt, I myself would not have learned to love Twitter were it not for tweetie on the iPhone. I know every single one of my friends can say the same, though it may be a different app and a different phone. Stats from Twitter on this would be interesting. I bet majority use is Twitter.com . However, I bet majority use where there is real interaction is all third party apps. Web user traffic is the casual login once a week user. Third party apps are what made Twitter I household name. To me, the API should be equal or more a priority, especially since a down API gets the third party app blamed, not Twitter. -- Scott Iphone says hello. On Aug 8, 2009, at 5:03 PM, chinaski007 wrote: If Twitter.com itself were down, you know that they would stay there until it was back up. But since it is just a large number of third party apps that are down... well, hey, it's a weekend in August!
[twitter-dev] Re: The silence is deafening....
Can anyone guess how long it will take for this problem to be fixed.. Apps totally dependent on the api are suffering very badly... lot of revenue loss also... On Aug 8, 6:37 pm, Neeraj Mathur wrote: > I just can't believe this is for real... > > On Aug 8, 2009, at 5:07 PM, Dossy Shiobara wrote: > > > > > On 8/8/09 8:03 PM, chinaski007 wrote: > >> If Twitter.com itself were down, you know that they would stay there > >> until it was back up. > > >> But since it is just a large number of third party apps that are > >> down... well, hey, it's a weekend in August! > > > Uh, my ability to access Twitter.com has been severely degraded > > throughout the day. It's not just third-party apps that are down. > > > I guess this is the one "good" thing about Twitter not having a real > > revenue model in place: they don't have to measure downtime in > > "dollars lost per hour" so there's really no actual pressure, > > compared to working for other companies that do and will FIRE YOUR > > ASS for outage like this. > > > -- > > Dossy Shiobara | do...@panoptic.com |http://dossy.org/ > > Panoptic Computer Network |http://panoptic.com/ > > "He realized the fastest way to change is to laugh at your own > > folly -- then you can let go and quickly move on." (p. 70)
[twitter-dev] Re: The silence is deafening....
I just can't believe this is for real... On Aug 8, 2009, at 5:07 PM, Dossy Shiobara wrote: > > On 8/8/09 8:03 PM, chinaski007 wrote: >> If Twitter.com itself were down, you know that they would stay there >> until it was back up. >> >> But since it is just a large number of third party apps that are >> down... well, hey, it's a weekend in August! > > Uh, my ability to access Twitter.com has been severely degraded > throughout the day. It's not just third-party apps that are down. > > I guess this is the one "good" thing about Twitter not having a real > revenue model in place: they don't have to measure downtime in > "dollars lost per hour" so there's really no actual pressure, > compared to working for other companies that do and will FIRE YOUR > ASS for outage like this. > > -- > Dossy Shiobara | do...@panoptic.com | http://dossy.org/ > Panoptic Computer Network | http://panoptic.com/ > "He realized the fastest way to change is to laugh at your own >folly -- then you can let go and quickly move on." (p. 70)
Re: Possibly curmudgeonly thoughts about the DDoS and architecture... (was Re: [twitter-dev] Re: The silence is deafening....)
Nick, Yes, they have very competent people. My criticism is not leveled against the API team. They are not the ones responsible for the edge defenses. But this thing has happened every single time so far. Twitter comes under attack, and the response is to simply swing the machine gun in a 360 degree arc. That's probably what I would do, but I am a lone guy, I do not have a company full of super competent and smart people. And the after the first time, I would make damn certain that I don't do it again, and I would make a list of who not to shoot the next time around. Dewald On Aug 8, 10:41 pm, Nick Arnett wrote: > On Sat, Aug 8, 2009 at 5:40 PM, Dewald Pretorius wrote: > > > Twitter needs to realize that our apps are NOT still down because of > > the ongoing denial-of-service attack. That's a cop-out to blame the > > attack. > > > Our apps are still down because they cannot allow known, white-listed > > IP addresses through the defenses. > > > And that is why I am getting frustrated, because I have asked multiple > > times months ago that they distinguish between friend and foe, and not > > kill everyone on sight when they are attacked. > > What make you think that they can? What if the DDoS attacks are spoofing > white-listed IP addresses sometimes? That would totally fit with using 302s > as a response. > > It's not a good idea to make assumptions about what they can and cannot do. > For Twitter to have grown as large as it is, I assume that they have some > very competent IT people, who surely are doing the best they can. Even > though Twitter isn't taking a direct revenue hit on this, I'm sure that they > know that the damage to their reputation could cost them more and more as > this continues. > > Hmmm... now does the idea of publishing tweetstreams as distributed RSS > feeds sound more attractive? If there's a criticism to be leveled, seems to > me it should be at the dependence on a single point of failure, not their > inability to cope with the inevitable sophisticated attack. DDoS and such > would have a far harder time causing this kind of trouble on a distributed > system. > > As I've said before, this isn't really a criticism of Twitter - what they've > created shows the demand for this kind of service. But imagine if right now > all the dead applications could fall back to reading RSS-published > twitterstreams instead of depending entirely on Twitter for them? > > Hope that doesn't sound like I'm taking advantage of a bad situation, but I > really think this points out the serious limitations of their architecture, > not the competence of their IT people. And no, those aren't the same > things. > > Nick
Possibly curmudgeonly thoughts about the DDoS and architecture... (was Re: [twitter-dev] Re: The silence is deafening....)
On Sat, Aug 8, 2009 at 5:40 PM, Dewald Pretorius wrote: > > Twitter needs to realize that our apps are NOT still down because of > the ongoing denial-of-service attack. That's a cop-out to blame the > attack. > > Our apps are still down because they cannot allow known, white-listed > IP addresses through the defenses. > > And that is why I am getting frustrated, because I have asked multiple > times months ago that they distinguish between friend and foe, and not > kill everyone on sight when they are attacked. What make you think that they can? What if the DDoS attacks are spoofing white-listed IP addresses sometimes? That would totally fit with using 302s as a response. It's not a good idea to make assumptions about what they can and cannot do. For Twitter to have grown as large as it is, I assume that they have some very competent IT people, who surely are doing the best they can. Even though Twitter isn't taking a direct revenue hit on this, I'm sure that they know that the damage to their reputation could cost them more and more as this continues. Hmmm... now does the idea of publishing tweetstreams as distributed RSS feeds sound more attractive? If there's a criticism to be leveled, seems to me it should be at the dependence on a single point of failure, not their inability to cope with the inevitable sophisticated attack. DDoS and such would have a far harder time causing this kind of trouble on a distributed system. As I've said before, this isn't really a criticism of Twitter - what they've created shows the demand for this kind of service. But imagine if right now all the dead applications could fall back to reading RSS-published twitterstreams instead of depending entirely on Twitter for them? Hope that doesn't sound like I'm taking advantage of a bad situation, but I really think this points out the serious limitations of their architecture, not the competence of their IT people. And no, those aren't the same things. Nick
[twitter-dev] Re: The silence is deafening....
Twitter needs to realize that our apps are NOT still down because of the ongoing denial-of-service attack. That's a cop-out to blame the attack. Our apps are still down because they cannot allow known, white-listed IP addresses through the defenses. And that is why I am getting frustrated, because I have asked multiple times months ago that they distinguish between friend and foe, and not kill everyone on sight when they are attacked.
[twitter-dev] Re: The silence is deafening....
On 8/8/09 8:03 PM, chinaski007 wrote: If Twitter.com itself were down, you know that they would stay there until it was back up. But since it is just a large number of third party apps that are down... well, hey, it's a weekend in August! Uh, my ability to access Twitter.com has been severely degraded throughout the day. It's not just third-party apps that are down. I guess this is the one "good" thing about Twitter not having a real revenue model in place: they don't have to measure downtime in "dollars lost per hour" so there's really no actual pressure, compared to working for other companies that do and will FIRE YOUR ASS for outage like this. -- Dossy Shiobara | do...@panoptic.com | http://dossy.org/ Panoptic Computer Network | http://panoptic.com/ "He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on." (p. 70)
[twitter-dev] Re: The silence is deafening....
If Twitter.com itself were down, you know that they would stay there until it was back up. But since it is just a large number of third party apps that are down... well, hey, it's a weekend in August! Grrr. On Aug 8, 4:55 pm, Dossy Shiobara wrote: > Seriously, anyone who has worked an Ops gig and hasn't worked a 30+ hour > day when a serious outage occurred just doesn't deserve respect. > > On 8/8/09 6:44 PM, Dewald Pretorius wrote: > > > When my app is down, that is exactly what I do to get it up and > > running again. > > > On Aug 8, 7:40 pm, Cameron Kaiser wrote: > >>> > > However, I would hope that Twitter engineers are all in force at the > >>> > > office on a day like this to solve this issue and get our > >>> > applications > >>> > > back up and running, regardless of whether it is Saturday, Sunday, > >>> > or > >>> > > Christmas Day. > > >> > Make sure they work all night too. > > -- > Dossy Shiobara | do...@panoptic.com |http://dossy.org/ > Panoptic Computer Network |http://panoptic.com/ > "He realized the fastest way to change is to laugh at your own > folly -- then you can let go and quickly move on." (p. 70)
[twitter-dev] Re: The silence is deafening....
Seriously, anyone who has worked an Ops gig and hasn't worked a 30+ hour day when a serious outage occurred just doesn't deserve respect. On 8/8/09 6:44 PM, Dewald Pretorius wrote: When my app is down, that is exactly what I do to get it up and running again. On Aug 8, 7:40 pm, Cameron Kaiser wrote: > > However, I would hope that Twitter engineers are all in force at the > > office on a day like this to solve this issue and get our applications > > back up and running, regardless of whether it is Saturday, Sunday, or > > Christmas Day. > > Make sure they work all night too. -- Dossy Shiobara | do...@panoptic.com | http://dossy.org/ Panoptic Computer Network | http://panoptic.com/ "He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on." (p. 70)
[twitter-dev] Re: The silence is deafening....
On 8/8/09 6:23 PM, Chad Etzel wrote: I'm not supposed to be working today, but in the interest of developer relations, I have been responding to occasional dev-list emails while I was out today. That's funny - as unpaid consumers of Twitter, we work 24/7 and do it with zest. Become an employee ... and there's now a clock to punch? -- Dossy Shiobara | do...@panoptic.com | http://dossy.org/ Panoptic Computer Network | http://panoptic.com/ "He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on." (p. 70)
[twitter-dev] Re: The silence is deafening....
When my app is down, that is exactly what I do to get it up and running again. On Aug 8, 7:40 pm, Cameron Kaiser wrote: > > However, I would hope that Twitter engineers are all in force at the > > office on a day like this to solve this issue and get our applications > > back up and running, regardless of whether it is Saturday, Sunday, or > > Christmas Day. > > Make sure they work all night too. > > -- > personal:http://www.cameronkaiser.com/-- > Cameron Kaiser * Floodgap Systems *www.floodgap.com* ckai...@floodgap.com > -- Intel outside -- 6502 inside! > --
[twitter-dev] Re: The silence is deafening....
> However, I would hope that Twitter engineers are all in force at the > office on a day like this to solve this issue and get our applications > back up and running, regardless of whether it is Saturday, Sunday, or > Christmas Day. Make sure they work all night too. -- personal: http://www.cameronkaiser.com/ -- Cameron Kaiser * Floodgap Systems * www.floodgap.com * ckai...@floodgap.com -- Intel outside -- 6502 inside! --
[twitter-dev] Re: The silence is deafening....
Chad, Thank you for your reply. However, I would hope that Twitter engineers are all in force at the office on a day like this to solve this issue and get our applications back up and running, regardless of whether it is Saturday, Sunday, or Christmas Day. Having the Twitter website operational is no reason for things to wait until Monday, not when all your third-party applications are borked. And yes, I am beginning to get just a tad frustrated. When we get no communication, we have to draw our own conclusions. Rather over-communicate by saying "it is still borked and we have X number of people working furiously on it" than creating the impression, through silence, that everyone has taken the weekend off. Dewald On Aug 8, 7:23 pm, Chad Etzel wrote: > On Sat, Aug 8, 2009 at 5:39 PM, Dewald Pretorius wrote: > > > tick tock tick tock tick tock tick tock tick tock tick tock > > To quote my last email: > > *** > We will also be monitoring the situation and giving out new > information as we have it. Please remain patient with us. As much as > you want it to be fixed, we want it to be fixed more. We're all going > to have to ride this out together. Communications may be slowed over > the weekend, but please know that we are not ignoring the situation. > *** > > John K provided an update on the Streaming API this morning. Other > than that, we have no new information and the situation remains the > same. Thus, no new emails saying "yep, still borked." > > I'm not supposed to be working today, but in the interest of developer > relations, I have been responding to occasional dev-list emails while > I was out today. > > Be patient, please. > > -Chad
[twitter-dev] Re: The silence is deafening....
Ask me "Are we there yet?" one more time and I'll turn this car around and you won't go to Disneyland at all! ;-) Nick On Sat, Aug 8, 2009 at 2:39 PM, Dewald Pretorius wrote: > > tick tock tick tock tick tock tick tock tick tock tick tock >
[twitter-dev] Re: The silence is deafening....
On Sat, Aug 8, 2009 at 5:39 PM, Dewald Pretorius wrote: > > tick tock tick tock tick tock tick tock tick tock tick tock To quote my last email: *** We will also be monitoring the situation and giving out new information as we have it. Please remain patient with us. As much as you want it to be fixed, we want it to be fixed more. We're all going to have to ride this out together. Communications may be slowed over the weekend, but please know that we are not ignoring the situation. *** John K provided an update on the Streaming API this morning. Other than that, we have no new information and the situation remains the same. Thus, no new emails saying "yep, still borked." I'm not supposed to be working today, but in the interest of developer relations, I have been responding to occasional dev-list emails while I was out today. Be patient, please. -Chad
[twitter-dev] Re: The silence is deafening....
If it is really important to you, how long does it take you to exclude known white-listed IP addresses from the defenses, if you put your mind and resources to it? On Aug 8, 6:42 pm, Cameron Kaiser wrote: > > tick tock tick tock tick tock tick tock tick tock tick tock > > I'm sure it's being worked on. > > -- > personal:http://www.cameronkaiser.com/-- > Cameron Kaiser * Floodgap Systems *www.floodgap.com* ckai...@floodgap.com > -- Anything that can be put into a nutshell belongs there. -- F. G. Brauer >
[twitter-dev] Re: The silence is deafening....
> tick tock tick tock tick tock tick tock tick tock tick tock I'm sure it's being worked on. -- personal: http://www.cameronkaiser.com/ -- Cameron Kaiser * Floodgap Systems * www.floodgap.com * ckai...@floodgap.com -- Anything that can be put into a nutshell belongs there. -- F. G. Brauer
