Re: [silk] Youtube an .pk Telecom
Those interested in the incident are no doubt already clear on what happened. I did however find the RIPE case study interesting if only as an indication of how they do a case study on such an event: http://www.ripe.net/news/study-youtube-hijacking.html - rather nicely done I thought.
Re: [silk] Youtube an .pk Telecom
More interestingly - is there a caste system in the IT sector in which working for a foreign multinational telecom company makes an employee less dumb that the average MTNL or BSNL employee? well in this case it was clearly PCCW (not really a poor 3rd world provider) that was pretty dumb; i get the impression they were actually dumber than pk tel (in technical terms; i think blocking youtube is dumb but not sure pk tel had a choice in the matter)
Re: [silk] Youtube an .pk Telecom
it's efficient to set rules in order of increasing specificity, with the more specific rules (by definition for a smaller address range) taking priority. e.g. in this case you could say route the 256-address-space to a black hole + route the 64-address-subset to the real youtube, with the latter rule taking precedence. doing the same thing without precedence would require at least three rules, route the X addresses before the 64-address-subset to a black hole, route the X addresses after the 64-address-subset to a black hole, route the 64-address-space to the real youtube. -rishab On Tue, 2008-02-26 at 22:36 +0530, Suresh Ramasubramanian wrote: Lawnun wrote: Why does shrinking the number of addresses create 'priority' as far as the BGP is concerned? Is there some merit to fewer addresses, as opposed to more? Something about specific routes being preferred. That's stuff you learn in cisco router classes. Oh, it didn't work - not for all the cases.
Re: [silk] Youtube an .pk Telecom
PTCL was dumb to block youtube. PCCW was even dumber to ignore best practices and not filter route annoucements from customer ISPs against routing registries etc [long standing best practice to guard against typos in router configs or something similar]. And still dumber to cut off the entire PTCL AS till the issue got fixed. I never did say PCCW wasnt dumb. They're our upstreams in HKG and I know how dumb, painfully, from first hand experience (but well, they're a bit more professional and good at the basics of service than vsnl / ptcl etc would ever be in their lives, that's an entirely different, stratospheric element of dumbness) srs Rishab Aiyer Ghosh [02/03/08 12:46 -0600]: More interestingly - is there a caste system in the IT sector in which working for a foreign multinational telecom company makes an employee less dumb that the average MTNL or BSNL employee? well in this case it was clearly PCCW (not really a poor 3rd world provider) that was pretty dumb; i get the impression they were actually dumber than pk tel (in technical terms; i think blocking youtube is dumb but not sure pk tel had a choice in the matter)
Re: [silk] Youtube an .pk Telecom
Lawnun wrote, [on 2/26/2008 10:27 PM]: Thanks Udhay. The article was extremely helpful for those of us coming to the matter from a non-engineering perspective. Also on (approximately) the same topic, a mixture of engineering and philosophy, from Rohit Khare, whom some of you also know: http://www.ics.uci.edu/~rohit/IEEE-L7-names-trust.html Udhay -- ((Udhay Shankar N)) ((udhay @ pobox.com)) ((www.digeratus.com))
Re: [silk] Youtube an .pk Telecom
IIRC, the ISP published bogus routes, and the cooperating providers accepted the BGP-pushed routes for some strange reason. A very simple and stupid reason. Several transit providers ignore long standing best practices, and don't filter route announcements. And at least one of them (PCCW) was upstream of Pakistani telecom. And so they leaked out those announcements. And a bunch of other providers picked up those routes from PCCW, still believing them. There's a lot of very well developed routing best practices (just walk into any nanog, ripe, apricot etc meeting for discussions, tutorials etc on these, or troll through google). Pity is that some providers are just too dumb to follow these.
Re: [silk] Youtube an .pk Telecom
On Tuesday 26 Feb 2008 11:51:32 am Suresh Ramasubramanian wrote: I mean, even if a big dumb 3rd world telco (like these guys, or like some of our own homegrown ISPs Off topic - but suppose an official Australian or other Western entity had made this statement, it would be dubbed racism and there would be a hue and cry in parliament (in India) and people would burn effigies of computers painted in some national colors on the streets. Or would people from these dumb 3rd world countires who provide the dumb employees for their telcos take it lying down? More interestingly - is there a caste system in the IT sector in which working for a foreign multinational telecom company makes an employee less dumb that the average MTNL or BSNL employee? shiv
Re: [silk] Youtube an .pk Telecom
On Tue, Feb 26, 2008 at 01:39:33PM +0530, Suresh Ramasubramanian wrote: A very simple and stupid reason. Several transit providers ignore long standing best practices, and don't filter route announcements. And at least one of them (PCCW) was upstream of Pakistani telecom. And so they leaked out those announcements. And a bunch of other providers picked up those routes from PCCW, still believing them. IIRC, there have been such incidents in the U.S. in the past, where a single party on dialup or cable modem could fux0r up their entire ISP. Try running a BGP daemon on your ISP's account, chances are, you can publish some bogus information as well. There's a lot of very well developed routing best practices (just walk into any nanog, ripe, apricot etc meeting for discussions, tutorials etc on these, or troll through google). Pity is that some providers are just too dumb to follow these. Look at the amount of best practices a voxel of vacuum has to follow to route electromagnetic radiation. In principle, routing packets from here to there can be done by very minimalistic decorations on top of that physics. The network is not nearly dumb enough yet. -- Eugen* Leitl a href=http://leitl.org;leitl/a http://leitl.org __ ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE
Re: [silk] Youtube an .pk Telecom
ss [26/02/08 14:33 +0530]: Or would people from these dumb 3rd world countires who provide the dumb employees for their telcos take it lying down? Oh I dont know. Having been at the receiving end of this dumbness yourself (in the form of slow / unreliable / overpriced connectivity, for starters) I suppose you can indulge yourself in a bit of techno racism and profiling More interestingly - is there a caste system in the IT sector in which working for a foreign multinational telecom company makes an employee less dumb that the average MTNL or BSNL employee? The local people working for places like nokia / samsung are just as bad as the ones working for the local companies (well, slightly less bad). The good ones get cherrypicked to head abroad. Or scout for high paying jobs abroad.
Re: [silk] Youtube an .pk Telecom
Eugen Leitl [26/02/08 10:13 +0100]: Try running a BGP daemon on your ISP's account, chances are, you can publish some bogus information as well. cable / dsl ISPs are used to kiddies injecting fake arp packets, bgp routes etc. And tend to guard against it. At least stateside. physics. The network is not nearly dumb enough yet. No. It wont be dumb enough to suit your highly advanced tastes in theoretical physics. Without possibly existing in some alternate universe. Engineering trumps physics every time.
Re: [silk] Youtube an .pk Telecom
Unfortunately (and I am not accusing you) there is a tendency to consider these people as being somewhat inferior in the same manner that IIT graduates sometimes refer to non IIT types. I wont consider them inferior. I have met some very technically competent people working for ISPs in the region (given I award fellowships for two large workshops, one asiapac wide and the other focused on the saarc region .. apricot.net and sanog.org) Especially the Pakistanis - there are several people in various Pakistan ISPs that strike me as much smarter than their peers elsewhere about these best practices Unfortunately, government owned telcos don't tend to retain smart people, and whatever smartness there is gets damped down by mediocrity and incompetence at senior levels. And any trips to foreign places, to attend even teaching conferences like these, tend to get sanctioned for less than competent senior management, who proceed to treat it as a paid vacation. Similar thing with government agencies .. saw a nice old gentleman whose ticket to a high level, expert conference had been paid for by a certain very large corporation. Senior official. Who didn't, unfortunately, know very much at all about the subject of the conference. And earnestly tried to show his interest by asking very simple, very basic questions. Sort of like the deputy director of a premier medical institution turning up at an international conference on cancer, funded by one of the large anti cancer drug firms, and then asking questions that'd count for 1 mark in a 10th standard biology exam. suresh
Re: [silk] Youtube an .pk Telecom
On Tuesday 26 Feb 2008 5:31:33 pm Suresh Ramasubramanian wrote: The local people working for places like nokia / samsung are just as bad as the ones working for the local companies (well, slightly less bad). The good ones get cherrypicked to head abroad. Or scout for high paying jobs abroad. Don't want to indulge in complete speculative nonsense here, but I have, in various forum discussions came across similar discussions about similar situations unrelated to the IT industry that may have a bearing on the issue. I would just like to bring them up as thoughts thunk while thinking. Knowing that all generalizations are wrong, I will still go ahead and state that the typical engineer or techie who hunts for higher paying jobs abroad is one who has family encouragement to do that and family support to do that. In other words those who start off being socially privileged in the first place (often but not necessarily forward caste)go ahead and achieve greater things The less privileged people often have family pressure to 1) Start earning soon to recoup investment 2) Inability to put in that extra investment to travel abroad 3) Family presssure to stay on and to fulfil family obligations and not go away These people often do not already have others who have done the same thing in their extended family, and can very often be the only technically educated person in the family. There are other bells and whistles that may be associated with this, such as backward caste, widowed mother, only son with three sisters to be married, only educated person in the family, easily available job nearby in local mofussil town in a government establishment that gels in with all the other social commitments. The salary is often much higher than anyone else in the extended family earns despite being much lower than the multinational/foreign type job. Like I said similar situations exist in other industries and vocations, including medicine and industrial research and production. Unfortunately (and I am not accusing you) there is a tendency to consider these people as being somewhat inferior in the same manner that IIT graduates sometimes refer to non IIT types. shiv
Re: [silk] Youtube an .pk Telecom
On Tue, Feb 26, 2008 at 04:03:15AM -0800, Suresh Ramasubramanian wrote: No. It wont be dumb enough to suit your highly advanced tastes in theoretical physics. Without possibly existing in some alternate universe. You're confusing engineering with physics. Engineering trumps physics every time. Do you think header layout is irrelevant for relativistic cut-through? -- Eugen* Leitl a href=http://leitl.org;leitl/a http://leitl.org __ ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE
Re: [silk] Youtube an .pk Telecom
Thanks Udhay. The article was extremely helpful for those of us coming to the matter from a non-engineering perspective. Quick question. The article said: YouTube took countermeasures within minutes, first trying to reclaim its network by narrowing its 1,024 broadcast to 256 addresses. Eleven minutes later, YouTube added an even more specific additional broadcast claiming just 64 addresses--which, under the Border Gateway Protocol, is more specific and therefore should overrule the Pakistani one. Over two hours after the initial false broadcast, Pakistan Telecom finally stopped. Why does shrinking the number of addresses create 'priority' as far as the BGP is concerned? Is there some merit to fewer addresses, as opposed to more? On a side note -- I'm totally curious if there's any legal implication for parties that are, as you all have indicated, lax in their enforcement of net standards? I mean, for one site, I can see it being as big of a deal, but what about the earlier example cited in the news.com piece about Turkey pretending to be the entire internet? That smacks of negligence to me. C On Tue, Feb 26, 2008 at 1:18 AM, Udhay Shankar N [EMAIL PROTECTED] wrote: Gautam John wrote: [ on 09:58 AM 2/26/2008 ] http://arstechnica.com/news.ars/post/20080225-insecure-routing-redirects-youtube-to-pakistan.html What the heck does this stuff mean? It escaped? So anyone can 'escape' routing information to shut down the 'tubes? In essence, yes. Sometimes. This piece may be of help: http://www.news.com/8301-10784_3-9878655-7.html?tag=nl.e498 Udhay -- ((Udhay Shankar N)) ((udhay @ pobox.com)) ((www.digeratus.com))
Re: [silk] Youtube an .pk Telecom
Lawnun wrote: Why does shrinking the number of addresses create 'priority' as far as the BGP is concerned? Is there some merit to fewer addresses, as opposed to more? Something about specific routes being preferred. That's stuff you learn in cisco router classes. Oh, it didn't work - not for all the cases. What did work was when PCCW pulled the plug on Pakistan Telecom's entire AS. And then kept the plug pulled till the Pakistanis fixed whatever was causing them to leak youtube prefixes. Silly of pccw - they could easily have filtered out the bogus prefixes that were being announced by the Pakistanis, instead of their whole AS number. On a side note -- I'm totally curious if there's any legal implication for parties that are, as you all have indicated, lax in their enforcement of net standards? I mean, for one site, I can see it being as big of a deal, Umm.. nothing legal as in international prosecutions take huge amounts of time and effort. All for some engineer who could use some (re)training? Maybe not. They learnt a hard, sharp lesson though, the Pakistanis .. if you screw up on the internet you risk having your connectivity disrupted by heavier than necessary mitigation measures. srs
Re: [silk] Youtube an .pk Telecom
At 2008-02-26 11:57:19 -0500, [EMAIL PROTECTED] wrote: Why does shrinking the number of addresses create 'priority' as far as the BGP is concerned? Is there some merit to fewer addresses, as opposed to more? Routers give priority to more specific routes over less-specific ones. Announcing a route for 10/24 (aka the network containing 255 addresses from 10.0.0.1 to 10.0.0.255) is more specific than announcing a route for 10/16 (i.e. the 65535 addresses from 10.0.0.1 to 10.0.255.255). This is so that an ISP can say Send traffic for this whole network to me, while the ISP's customers can say Send traffic for my small part of the ISP's network to me (that is, if they run BGP at all) and it all works. -- ams
Re: [silk] Youtube an .pk Telecom
From the archives: http://lists.ucc.gu.uwa.edu.au/pipermail/lore/2006-August/40.html ...then, suddenly, the internet stopped working. Network Operators everywhere sprang into action to discover the cause of the lack of traffic. And there it was. As far as the routing protocols were concerned, the entire internet existed in one location - some crappy Bay Networks router in AS7007...
Re: [silk] Youtube an .pk Telecom
Thanks Abhijit! That makes sense. The article wasn't quite clear enough, is all. On Tue, Feb 26, 2008 at 12:11 PM, Abhijit Menon-Sen [EMAIL PROTECTED] wrote: At 2008-02-26 11:57:19 -0500, [EMAIL PROTECTED] wrote: Why does shrinking the number of addresses create 'priority' as far as the BGP is concerned? Is there some merit to fewer addresses, as opposed to more? Routers give priority to more specific routes over less-specific ones. Announcing a route for 10/24 (aka the network containing 255 addresses from 10.0.0.1 to 10.0.0.255) is more specific than announcing a route for 10/16 (i.e. the 65535 addresses from 10.0.0.1 to 10.0.255.255). This is so that an ISP can say Send traffic for this whole network to me, while the ISP's customers can say Send traffic for my small part of the ISP's network to me (that is, if they run BGP at all) and it all works. -- ams
[silk] Youtube an .pk Telecom
http://news.bbc.co.uk/1/hi/technology/7262071.stm The BBC News website's technology editor, Darren Waters, says that to block Pakistan's citizens from accessing YouTube it is believed Pakistan Telecom hijacked the web server address of the popular video site. Those details were then passed on to the country's internet service providers so that anyone in Pakistan attempting to go to YouTube was instead re-directed to a different address. But the details of the hijack were leaked out into the wider internet from PCCW and as a result YouTube was mistakenly blocked by internet service providers around the world. snip What does that mean and is it that trivial to knock a site off-line?
Re: [silk] Youtube an .pk Telecom
On Mon, Feb 25, 2008 at 9:55 AM, Gautam John [EMAIL PROTECTED] wrote: What does that mean and is it that trivial to knock a site off-line? ...Pakistan Telecom routed the address block that YouTube's servers are into a black hole as a simple measure to filter access to the service. However, this routing information escaped from Pakistan Telecom to its ISP PCCW in Hong Kong, which propagated the route to the rest of the world. So any packets for YouTube would end up in Pakistan Telecom's black hole instead. http://arstechnica.com/news.ars/post/20080225-insecure-routing-redirects-youtube-to-pakistan.html What the heck does this stuff mean? It escaped? So anyone can 'escape' routing information to shut down the 'tubes?
Re: [silk] Youtube an .pk Telecom
Gautam John wrote: What does that mean and is it that trivial to knock a site off-line? Well, for a chronology of what went on, take a look at this - http://www.renesys.com/blog/2008/02/pakistan_hijacks_youtube_1.shtml Or for a simpler overview, see Brian Krebs' article in the Washington post (in his securityfix blog) - http://blog.washingtonpost.com/securityfix/2008/02/pakistan_censorship_order _take.html My comment there on that blog post might put it into simpler terms. For something more complex, please get some cisco / other routing classes. Those who are already CCIEs and reading this, please forgive me for a really bad and oversimplified analogy What this is like, is someone in a phone company in Atlanta screwing up and announcing that everything in (say) the 781 area code belongs to his telephone exchange, and is routed out of Atlanta rather than out of Boston (which is where 781 is). And then that Atlanta phone company sends out an update so that several other phone companies actually believe it. So, calls for boston people with 781 area codes end up getting rerouted to Atlanta instead of Boston. And either getting lost in thin air (as theres nobody in Atlanta who has a 781 area code phone) or making random unrelated phones ring. Its a crude analogy and phone switching doesnt really work this way .. but thought I would try at least making one.
Re: [silk] Youtube an .pk Telecom
Gautam John wrote: [ on 09:58 AM 2/26/2008 ] http://arstechnica.com/news.ars/post/20080225-insecure-routing-redirects-youtube-to-pakistan.html What the heck does this stuff mean? It escaped? So anyone can 'escape' routing information to shut down the 'tubes? In essence, yes. Sometimes. This piece may be of help: http://www.news.com/8301-10784_3-9878655-7.html?tag=nl.e498 Udhay -- ((Udhay Shankar N)) ((udhay @ pobox.com)) ((www.digeratus.com))
Re: [silk] Youtube an .pk Telecom
What the heck does this stuff mean? It escaped? So anyone can 'escape' routing information to shut down the 'tubes? In essence, yes. Sometimes. This piece may be of help: In essence - yes, but most providers that provide connectivity (transit etc) - have best practices that include filtering route announcements for sanity checks, to make sure stuff like this just doesn't happen. I mean, even if a big dumb 3rd world telco (like these guys, or like some of our own homegrown ISPs) screw up router announcements, these are caught and filtered out so that this damage doesn't occur Unfortunately - PCCW (big hong kong ISP - pacific century cyberworks) that provides connectivity to lots of smaller asian ISPs - doesn't do this. And youtube suffered as a result. What pccw did to stop this happening was even dumber, they cut off all of Pakistan telecom for a few hours.
Re: [silk] Youtube an .pk Telecom
Suresh Ramasubramanian wrote: [ on 11:44 AM 2/26/2008 ] What this is like, is someone in a phone company in Atlanta screwing up and announcing that everything in (say) the 781 area code belongs to his telephone exchange, and is routed out of Atlanta rather than out of Boston (which is where 781 is). And then that Atlanta phone company sends out an update so that several other phone companies actually believe it. So, calls for boston people with 781 area codes end up getting rerouted to Atlanta instead of Boston. And either getting lost in thin air (as theres nobody in Atlanta who has a 781 area code phone) or making random unrelated phones ring. As a completely irrelevant addendum to your explanation: in the above context, I take a deep and unholy delight in the fact that Atalanta's area code is 404. (as in, document not found) g, d, r Udhay -- ((Udhay Shankar N)) ((udhay @ pobox.com)) ((www.digeratus.com))
