Re: raging bulls
On Tue, Aug 07, 2012 at 05:15:51PM -1000, Michael Painter wrote: Eugen Leitl wrote: http://www.wired.com/business/2012/08/ff_wallstreet_trading/all/ Some interesting, network-relevant content there (but for the neutrino and drone rubbish). 'Rubbish' might be a pretty strong word when you're talking about the players in this space. If you want to shave off ms, using a source that takes at least minutes to accrue enough signal for a single bit is definitely not what you want. And drones across the Atlantic is way too Rube Goldbergesque to contemplate. My favorite from the article: But perhaps not even Einstein fully appreciated the degree to which electromagnetic waves bend in the presence of money. Maybe they should invest into a dense, a really low LEO sat constellation, and go by way of LoS laser.
Re: next hop packet loss
On Tue, Aug 7, 2012 at 4:12 PM, Jim Ray j...@neuse.net wrote: Sorry, I do not give verbose responses via iPhone on that small device with my tired old eyes. I ran Wireshark this morning. Without sniffing packets, the layman's description of problem is I can't get to vendor web site, http://www.CheckPoint.com, on Time Warner Business Class network I use. Hence, http is blocked in addition to ICMP. Yesterday, Check Point's website was unreachable from other parts of the world for some time with intermittent access for around an hour or so I believe. Eugeniu
Re: Trouble with IPv6 setup on Quagga
On Tuesday 07 August 2012 01:08:24 Anurag Bhatia wrote: router bgp 54456 bgp router-id 199.116.78.28 redistribute connected metric 1 redistribute static metric 1 neighbor 2607:1b00:10:a::1 remote-as 54456 neighbor 2607:1b00:10:a::1 next-hop-self address-family ipv6 network 2607:1b00:d1::/48 network 2607:1b00:d2::/48 neighbor 2607:1b00:10:a::1 activate exit-address-family Specifying next-hop-self in the general BGP router config section is equivalent to specifying it purely for IPv4 routes; you need to specify next- hop-self in the IPv6 address-family section. Regards, Oliver
Re: Trouble with IPv6 setup on Quagga
On 08/08/2012 09:37 AM, Oliver wrote: On Tuesday 07 August 2012 01:08:24 Anurag Bhatia wrote: router bgp 54456 bgp router-id 199.116.78.28 redistribute connected metric 1 redistribute static metric 1 neighbor 2607:1b00:10:a::1 remote-as 54456 neighbor 2607:1b00:10:a::1 next-hop-self address-family ipv6 network 2607:1b00:d1::/48 network 2607:1b00:d2::/48 neighbor 2607:1b00:10:a::1 activate exit-address-family Specifying next-hop-self in the general BGP router config section is equivalent to specifying it purely for IPv4 routes; you need to specify next- hop-self in the IPv6 address-family section. And you might want to disable (no neighbor ... activate) for the default-protocol (IPv4) as otherwise Quagga tries to advertise IPv4 over the same session as well - which you usually wouldn't want to. I've seen cases where both sides ran Quagga and wondered where all the (unfiltered) IPv4-routes came from :-) Regards, Stefan
RE: raging bulls
It seems to me that all the markets have been doing this the wrong way. Would it now be more fair to use some kind of signed timestamp and process all transactions in the order that they originated? Perhaps each trade could have a signed GPS tag with the absolute time on it. It would keep everyone's trades in order no matter how latent their connection to the market was. All you would have to do is introduce a couple of seconds delay to account for the longest circuit and then take them in order. They could certainly use less expensive connections and ensure that international traders get a fair shake. Steven Naslund -Original Message- From: Eugen Leitl [mailto:eu...@leitl.org] Sent: Wednesday, August 08, 2012 2:02 AM To: nanog@nanog.org Subject: Re: raging bulls On Tue, Aug 07, 2012 at 05:15:51PM -1000, Michael Painter wrote: Eugen Leitl wrote: http://www.wired.com/business/2012/08/ff_wallstreet_trading/all/ Some interesting, network-relevant content there (but for the neutrino and drone rubbish). 'Rubbish' might be a pretty strong word when you're talking about the players in this space. If you want to shave off ms, using a source that takes at least minutes to accrue enough signal for a single bit is definitely not what you want. And drones across the Atlantic is way too Rube Goldbergesque to contemplate. My favorite from the article: But perhaps not even Einstein fully appreciated the degree to which electromagnetic waves bend in the presence of money. Maybe they should invest into a dense, a really low LEO sat constellation, and go by way of LoS laser.
RE: raging bulls
There should be some sorts of way to authenticate a GPS timestamp. GPS may not be able to do it today but a satellite network could in theory cryptographically sign a time stamp so that is can only be decrypted by the receiver at the market data center. Either that or some kind of ground based hardware device that syncs with a satellite and generates a key stream so that the receiver can tell when something happened by where the device is in the stream. I am thinking of something along the lines of SecureID where the keys change every so often, I would just have to be lots faster. Steve -Original Message- From: Chu, Yi [NTK] [mailto:yi@sprint.com] Sent: Wednesday, August 08, 2012 9:01 AM To: Naslund, Steve; nanog@nanog.org Subject: RE: raging bulls What prevents someone to fake an earlier timestamp? Money can bend light, sure can a few msec. yi -Original Message- From: Naslund, Steve [mailto:snasl...@medline.com] Sent: Wednesday, August 08, 2012 9:53 AM To: nanog@nanog.org Subject: RE: raging bulls It seems to me that all the markets have been doing this the wrong way. Would it now be more fair to use some kind of signed timestamp and process all transactions in the order that they originated? Perhaps each trade could have a signed GPS tag with the absolute time on it. It would keep everyone's trades in order no matter how latent their connection to the market was. All you would have to do is introduce a couple of seconds delay to account for the longest circuit and then take them in order. They could certainly use less expensive connections and ensure that international traders get a fair shake. Steven Naslund -Original Message- From: Eugen Leitl [mailto:eu...@leitl.org] Sent: Wednesday, August 08, 2012 2:02 AM To: nanog@nanog.org Subject: Re: raging bulls On Tue, Aug 07, 2012 at 05:15:51PM -1000, Michael Painter wrote: Eugen Leitl wrote: http://www.wired.com/business/2012/08/ff_wallstreet_trading/all/ Some interesting, network-relevant content there (but for the neutrino and drone rubbish). 'Rubbish' might be a pretty strong word when you're talking about the players in this space. If you want to shave off ms, using a source that takes at least minutes to accrue enough signal for a single bit is definitely not what you want. And drones across the Atlantic is way too Rube Goldbergesque to contemplate. My favorite from the article: But perhaps not even Einstein fully appreciated the degree to which electromagnetic waves bend in the presence of money. Maybe they should invest into a dense, a really low LEO sat constellation, and go by way of LoS laser. This e-mail may contain Sprint Nextel proprietary information intended for the sole use of the recipient(s). Any use by others is prohibited. If you are not the intended recipient, please contact the sender and delete all copies of the message.
RE: raging bulls
Also, we are only talking about a delay long enough to satisfy the longest circuit so you could not push your timestamp very far back and would have to get the fake one done pretty quickly in order for it to be worthwhile. The real question is could you fake a cryptographic timestamp fast enough to actually gain time on the system. Possibly but it would be a very tall order. Steve -Original Message- From: Chu, Yi [NTK] [mailto:yi@sprint.com] Sent: Wednesday, August 08, 2012 9:01 AM To: Naslund, Steve; nanog@nanog.org Subject: RE: raging bulls What prevents someone to fake an earlier timestamp? Money can bend light, sure can a few msec. yi -Original Message- From: Naslund, Steve [mailto:snasl...@medline.com] Sent: Wednesday, August 08, 2012 9:53 AM To: nanog@nanog.org Subject: RE: raging bulls It seems to me that all the markets have been doing this the wrong way. Would it now be more fair to use some kind of signed timestamp and process all transactions in the order that they originated? Perhaps each trade could have a signed GPS tag with the absolute time on it. It would keep everyone's trades in order no matter how latent their connection to the market was. All you would have to do is introduce a couple of seconds delay to account for the longest circuit and then take them in order. They could certainly use less expensive connections and ensure that international traders get a fair shake. Steven Naslund
RE: raging bulls
It is a tough technical problem to be sure but not insurmountable. Think about a system in which the real time market data is also encrypted in such a way that it can only be decrypted at a particular point in time. Essentially it would be like each trading system receiving an envelope that must be opened simultaneously. Picture a satellite network that is time synchronized to transmit a key stream used to decrypt the data that is received over a terrestrial network. I am not talking about easy to implement here just what is possible. It is probably easier than faster than light travel although I supposed real estate on Mt Everest could get very valuable (closer to the satellites) :) Steve -Original Message- From: Brett Frankenberger [mailto:rbf+na...@panix.com] Sent: Wednesday, August 08, 2012 9:08 AM To: Naslund, Steve Cc: nanog@nanog.org Subject: Re: raging bulls On Wed, Aug 08, 2012 at 08:52:51AM -0500, Naslund, Steve wrote: It seems to me that all the markets have been doing this the wrong way. Would it now be more fair to use some kind of signed timestamp and process all transactions in the order that they originated? Perhaps each trade could have a signed GPS tag with the absolute time on it. It would keep everyone's trades in order no matter how latent their connection to the market was. All you would have to do is introduce a couple of seconds delay to account for the longest circuit and then take them in order. They could certainly use less expensive connections and ensure that international traders get a fair shake. This isn't about giving international traders a fair shake. This sort of latency is only relevant to high speed program trading, and the international traders can locate their servers in NYC just as easily as the US-based traders. What it's about is allowing traders to arbitrage between markets. When product A is traded in, say, London, and product B is traded in New York, and their prices are correlated, you can make money if your program running in NY can learn the price of product B in London a few milliseconds before the other guy's program. And you can make money if your program running in London can learn the price of product A in NY a few milliseconds before the other guy's program. Even if you execute the trades based on a GPS timestamp (I'm ignoring all the logistics of preventing cheating here), it doesn't matter, because the computer that got the information first will make the trading decision first. -- Brett
Re: raging bulls
On Wed, Aug 08, 2012 at 08:52:51AM -0500, Naslund, Steve wrote: It seems to me that all the markets have been doing this the wrong way. Would it now be more fair to use some kind of signed timestamp and process all transactions in the order that they originated? Perhaps each trade could have a signed GPS tag with the absolute time on it. It would keep everyone's trades in order no matter how latent their connection to the market was. All you would have to do is introduce a couple of seconds delay to account for the longest circuit and then take them in order. They could certainly use less expensive connections and ensure that international traders get a fair shake. This isn't about giving international traders a fair shake. This sort of latency is only relevant to high speed program trading, and the international traders can locate their servers in NYC just as easily as the US-based traders. What it's about is allowing traders to arbitrage between markets. When product A is traded in, say, London, and product B is traded in New York, and their prices are correlated, you can make money if your program running in NY can learn the price of product B in London a few milliseconds before the other guy's program. And you can make money if your program running in London can learn the price of product A in NY a few milliseconds before the other guy's program. Even if you execute the trades based on a GPS timestamp (I'm ignoring all the logistics of preventing cheating here), it doesn't matter, because the computer that got the information first will make the trading decision first. -- Brett
Re: raging bulls
On Wed, Aug 08, 2012 at 09:08:18AM -0500, Naslund, Steve wrote: Also, we are only talking about a delay long enough to satisfy the longest circuit so you could not push your timestamp very far back and would have to get the fake one done pretty quickly in order for it to be worthwhile. The real question is could you fake a cryptographic timestamp fast enough to actually gain time on the system. Possibly but it would be a very tall order. Why would generating a fake timestamp take longer than generating a real one? I assume you're proposing an architecture where if I were a trader, I'd have to buy a secure physical box from a third party trusted by the market, and I'd send my trade to that box and then it would timestamp it and sign it and then I'd send it to the market. Obvious failure modes include: (a) spoofing the GPS signal received by the box, so the box thinks the current time is some number of milliseconds before the actual time (how to do this is well understood and solved, and because GPS is one-way, even if the satellites started signing their time updates, that would only prevent spoofing times in the future, it wouldn't prevent spoofing times on the past), and (b) generating 10 trades at time X, then holding on to the signed messages until X+Y, and then only sending the ones that are profitable based on the new information you learned between (X) and (X+Y). Yes, there are some solutions. But most of those solutions have problems of their own. It's overwhelmingly difficult. (But also irrelevant, as I noted in my other post). If you think this through to what a working implementation would look like in detail, I think the failures become more obvious ... -- Brett
RE: raging bulls
Naslund, Steve SNaslund () medline com wrote: It seems to me that all the markets have been doing this the wrong way. Would it now be more fair to use some kind of signed timestamp and process all transactions in the order that they originated? Perhaps each trade could have a signed GPS tag with the absolute time on it. It would keep everyone's trades in order no matter how latent their connection to the market was. All you would have to do is introduce a couple of seconds delay to account for the longest circuit and then take them in order. They could certainly use less expensive connections and ensure that international traders get a fair shake. I can't see any incentive for any *influential* party involved (the big firms or the exchanges) to make the process fair. The exchange gets more money for low-latency network access and expensive co-location. The moneyed firms with HFT capability of course do not want anyone else to have their advantage. Even governments don't want long-distance traders to have fair access, as that reduces the advantage of local tax-paying firms, thereby reducing tax revenue, jobs, etc. HFT is not just a US phenomenon; all major exchanges have basically the same sort of phenomenon. So UK-based trading firms with HFT setups very close to the FTSE exchanges have advantage over US-based firms that don't have HFT setups in London. -- RPM
Re: raging bulls
On 8/8/12 6:52 AM, Naslund, Steve wrote: It seems to me that all the markets have been doing this the wrong way. Would it now be more fair to use some kind of signed timestamp and process all transactions in the order that they originated? Given an uneven distribution of sizes it's kind of hard to fill orders in the order in which they arrived (unmatched orders are part of a normally functioning market). A large bid may require the accumulation of sell orders while smaller orders may be more easily matched. Some HF trading strategies of course rely on this. Today large orders may be filled on more than one ecn at a time so the notion of central agency in clearance is also a little challenging. Perhaps each trade could have a signed GPS tag with the absolute time on it. It would keep everyone's trades in order no matter how latent their connection to the market was. All you would have to do is introduce a couple of seconds delay to account for the longest circuit and then take them in order. They could certainly use less expensive connections and ensure that international traders get a fair shake. it's simpler to just locate the trading platforms in the same place and give everyone the same length cable. The incentives are in the wrong place too deliberately induce delay without some externality (like a regulator) guiding behavior. If one sees current behavior as undesirable there are other methods such as the adjustment of transaction costs that might be more effective.
RE: raging bulls
Here is another thought. Many people think that the rapid computer trading does not really add any value to the market in any case since there is no long term investment. That point is debatable but if you really believed that, you could end all of that by adding a randomized delay to data transmission and processing of transactions to make the entire debate pointless and ensure that no one has any consistent advantage at all. Steve -Original Message- From: Naslund, Steve [mailto:snasl...@medline.com] Sent: Wednesday, August 08, 2012 9:08 AM To: nanog@nanog.org Subject: RE: raging bulls Also, we are only talking about a delay long enough to satisfy the longest circuit so you could not push your timestamp very far back and would have to get the fake one done pretty quickly in order for it to be worthwhile. The real question is could you fake a cryptographic timestamp fast enough to actually gain time on the system. Possibly but it would be a very tall order. Steve -Original Message- From: Chu, Yi [NTK] [mailto:yi@sprint.com] Sent: Wednesday, August 08, 2012 9:01 AM To: Naslund, Steve; nanog@nanog.org Subject: RE: raging bulls What prevents someone to fake an earlier timestamp? Money can bend light, sure can a few msec. yi -Original Message- From: Naslund, Steve [mailto:snasl...@medline.com] Sent: Wednesday, August 08, 2012 9:53 AM To: nanog@nanog.org Subject: RE: raging bulls It seems to me that all the markets have been doing this the wrong way. Would it now be more fair to use some kind of signed timestamp and process all transactions in the order that they originated? Perhaps each trade could have a signed GPS tag with the absolute time on it. It would keep everyone's trades in order no matter how latent their connection to the market was. All you would have to do is introduce a couple of seconds delay to account for the longest circuit and then take them in order. They could certainly use less expensive connections and ensure that international traders get a fair shake. Steven Naslund
Re: raging bulls
On Wed, Aug 8, 2012 at 8:08 AM, Brett Frankenberger rbf+na...@panix.com wrote: Even if you execute the trades based on a GPS timestamp (I'm ignoring all the logistics of preventing cheating here), it doesn't matter, because the computer that got the information first will make the trading decision first. -- Brett Such a system would be pretty complicated because it would also have to prevent intentional 'backdating' of trades as well. Then you've got the market data itself (as just mentioned) -- getting the information first is a big part of the latency problem for the quants. -- Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds. -- Samuel Butler
RE: raging bulls
Are there not mechanisms to handle replay attacks? There is also the minor matter of fraud and regulatory concerns. You might get away with it a few times but not often enough to avoid a potential death penalty of being disconnected. Steve -Original Message- From: Alexandre Snarskii [mailto:s...@snar.spb.ru] Sent: Wednesday, August 08, 2012 9:46 AM To: Naslund, Steve Cc: Alexandre Snarskii Subject: Re: raging bulls On Wed, Aug 08, 2012 at 09:08:18AM -0500, Naslund, Steve wrote: Also, we are only talking about a delay long enough to satisfy the longest circuit so you could not push your timestamp very far back and would have to get the fake one done pretty quickly in order for it to be worthwhile. The real question is could you fake a cryptographic timestamp fast enough to actually gain time on the system. Possibly but it would be a very tall order. Looks like replay attack works here: attacker can easily record encrypted timestamps and reuse them some milliseconds later, claiming I had no knowledge on how market changed during this time, it's my provider had to re-route my traffic!! Steve -Original Message- From: Chu, Yi [NTK] [mailto:yi@sprint.com] Sent: Wednesday, August 08, 2012 9:01 AM To: Naslund, Steve; nanog@nanog.org Subject: RE: raging bulls What prevents someone to fake an earlier timestamp? Money can bend light, sure can a few msec. yi -Original Message- From: Naslund, Steve [mailto:snasl...@medline.com] Sent: Wednesday, August 08, 2012 9:53 AM To: nanog@nanog.org Subject: RE: raging bulls It seems to me that all the markets have been doing this the wrong way. Would it now be more fair to use some kind of signed timestamp and process all transactions in the order that they originated? Perhaps each trade could have a signed GPS tag with the absolute time on it. It would keep everyone's trades in order no matter how latent their connection to the market was. All you would have to do is introduce a couple of seconds delay to account for the longest circuit and then take them in order. They could certainly use less expensive connections and ensure that international traders get a fair shake. Steven Naslund -- In theory, there is no difference between theory and practice. But, in practice, there is.
RE: raging bulls
No, not ever shorter under-see cables no. NEUTRINOS -shooting information at speed of light right through the earth (not around it) Should there be any high speed traders in here this is what you should invest all your money in to gain advantage against your competition First it was cold war times which gave birth to the Internet, that has changed our lives from the ground up Maybe this time it will be the stock markets and scent of money that will give the communication development spiral an unimaginable momentum adam -Original Message- From: Naslund, Steve [mailto:snasl...@medline.com] Sent: Wednesday, August 08, 2012 4:14 PM To: nanog@nanog.org Subject: RE: raging bulls It is a tough technical problem to be sure but not insurmountable. Think about a system in which the real time market data is also encrypted in such a way that it can only be decrypted at a particular point in time. Essentially it would be like each trading system receiving an envelope that must be opened simultaneously. Picture a satellite network that is time synchronized to transmit a key stream used to decrypt the data that is received over a terrestrial network. I am not talking about easy to implement here just what is possible. It is probably easier than faster than light travel although I supposed real estate on Mt Everest could get very valuable (closer to the satellites) :) Steve -Original Message- From: Brett Frankenberger [mailto:rbf+na...@panix.com] Sent: Wednesday, August 08, 2012 9:08 AM To: Naslund, Steve Cc: nanog@nanog.org Subject: Re: raging bulls On Wed, Aug 08, 2012 at 08:52:51AM -0500, Naslund, Steve wrote: It seems to me that all the markets have been doing this the wrong way. Would it now be more fair to use some kind of signed timestamp and process all transactions in the order that they originated? Perhaps each trade could have a signed GPS tag with the absolute time on it. It would keep everyone's trades in order no matter how latent their connection to the market was. All you would have to do is introduce a couple of seconds delay to account for the longest circuit and then take them in order. They could certainly use less expensive connections and ensure that international traders get a fair shake. This isn't about giving international traders a fair shake. This sort of latency is only relevant to high speed program trading, and the international traders can locate their servers in NYC just as easily as the US-based traders. What it's about is allowing traders to arbitrage between markets. When product A is traded in, say, London, and product B is traded in New York, and their prices are correlated, you can make money if your program running in NY can learn the price of product B in London a few milliseconds before the other guy's program. And you can make money if your program running in London can learn the price of product A in NY a few milliseconds before the other guy's program. Even if you execute the trades based on a GPS timestamp (I'm ignoring all the logistics of preventing cheating here), it doesn't matter, because the computer that got the information first will make the trading decision first. -- Brett
RE: raging bulls
This is a bit like an arms race. The markets will most likely have to level their own playing field. That is up to them. The markets may like high frequency trading but if more and more traders become disadvantaged they will act to level things out. They also would not like the government to step in which they are always apt to do. The government in general I think seems highly negative on high frequency trading because there is some systemic risk in systems handling large amounts of transactions at such a high rate. We have seen tremendous errors in the past and we are almost reaching the point where a firm will not be able to survive a system error or could cause a cascade effect. The question the markets and regulators always have to ask themselves is whether the market is fundamentally fair. The government has the additional duty of determining whether a market activity is detrimental to the economy of the nation involved. It is not for me to answer the question of whether this should be implemented, I am just saying that it is technically feasible to do so. As far as locating all the servers in the same place on the same length of cable, that apparently is not in the cards or you would not see the high cost specialized networks from Chicago to NYC. Steve -Original Message- From: joel jaeggli [mailto:joe...@bogus.com] Sent: Wednesday, August 08, 2012 9:23 AM To: Naslund, Steve Cc: nanog@nanog.org Subject: Re: raging bulls On 8/8/12 6:52 AM, Naslund, Steve wrote: It seems to me that all the markets have been doing this the wrong way. Would it now be more fair to use some kind of signed timestamp and process all transactions in the order that they originated? Given an uneven distribution of sizes it's kind of hard to fill orders in the order in which they arrived (unmatched orders are part of a normally functioning market). A large bid may require the accumulation of sell orders while smaller orders may be more easily matched. Some HF trading strategies of course rely on this. Today large orders may be filled on more than one ecn at a time so the notion of central agency in clearance is also a little challenging. Perhaps each trade could have a signed GPS tag with the absolute time on it. It would keep everyone's trades in order no matter how latent their connection to the market was. All you would have to do is introduce a couple of seconds delay to account for the longest circuit and then take them in order. They could certainly use less expensive connections and ensure that international traders get a fair shake. it's simpler to just locate the trading platforms in the same place and give everyone the same length cable. The incentives are in the wrong place too deliberately induce delay without some externality (like a regulator) guiding behavior. If one sees current behavior as undesirable there are other methods such as the adjustment of transaction costs that might be more effective.
RE: raging bulls
It might be complicated. I am just saying it is probably not as complicated as a permanent transatlantic aerial drone network or owning your own particle accelerator. I think all the anti-replay, anti-backdating concerns have probably been solved in the various public/private key networks, if not, there are certainly ways to do so. My only point was to say that there are technical ways to make high freq trading more inherently fair and there does not need to be a connectivity arms race unless that is what the markets want to do. The markets created this monster, they can't really complain about something they have the power to eliminate any time they want. If the majority of traders feel they are getting screwed, the policies will change. Looking at this from a strictly engineer's point of view, it seems like a very, very, risky way to handle extremely large sums of money. The systems are already outpacing the speeds they can be controlled and managed. At some point the systemic risk to the entire system will be the regulating factor. Let's hope they can head it off before it happens. I can very well see a point where a major system meltdown will cause an entire day of trading to be unwound because there is no other choice. You could always require all the trading systems to be in a single place but there will always be backend systems to control and monitor them somewhere else. Hey, putting everyone in the same place is how markets worked before when you needed a guy standing on the floor in order to operate. Steve -Original Message- From: Michael Loftis [mailto:mlof...@wgops.com] Sent: Wednesday, August 08, 2012 9:56 AM To: nanog@nanog.org Subject: Re: raging bulls On Wed, Aug 8, 2012 at 8:08 AM, Brett Frankenberger rbf+na...@panix.com wrote: Even if you execute the trades based on a GPS timestamp (I'm ignoring all the logistics of preventing cheating here), it doesn't matter, because the computer that got the information first will make the trading decision first. -- Brett Such a system would be pretty complicated because it would also have to prevent intentional 'backdating' of trades as well. Then you've got the market data itself (as just mentioned) -- getting the information first is a big part of the latency problem for the quants. -- Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds. -- Samuel Butler
Re: raging bulls
Here is another thought. Many people think that the rapid computer trading does not really add any value to the market in any case since there is no long term investment. It clearly doesn't. A proposal that's been kicking around for a while is to clear all trades once a second, so everyone has plenty of time to get them in no matter where they are. This has no chance of going anywhere, of course, until there's enough software disasters to provide political pushback against the leeches doing high speed trading. There is a new data center in Keflavik, Iceland. They advertise all of its fabulous green characteristics, e.g., power is from geothermal, and A/C is by opening the window, but it also happens to be closer to New York than anywhere in Europe, and closer to London than anywhere in North America, with good cable connections to both. R's, John
RE: raging bulls
Just leaving room for disagreement on the value of HFT. It would seem to add nothing but more volatility to the market and make small events more extreme. There are also big risks of systems making convention wisdom decisions in unconventional situations. Can someone pull the plug fast enough if some unpredicted event makes the conventional wisdom wrong? The article in question uses an example like oil price goes up, airline stocks go down. This sounds true enough but how many assumptions like this exist that might not ALWAYS be true. Is it fair to crater the airline industry or any other one because some convention like that causes a huge fast momentum swing. I guess the danger is that all the assumptions are those of a human but done at the speed of computing without natures built in safety catch of time to reconsider the assumption before pulling the trigger. We worry greatly over the software that controls aircraft and nuclear power plants but this software has a much greater potential for worldwide disaster and being a competitive market is probably changed many, many times more often in a less controlled way. You have to decide whether a global market meltdown and an aircraft crash can be compared but both are bad events. Again, this is a risk / reward situation that the markets and regulators need to deal with. I am normally not a big advocate for government control of anything but clearly there is a need for regulating an industry that has again and again done some very risky things that have very tangible effects on the world economy. When the speed limit of light becomes a major worry for your system, it peaks my radar as being a system that is running on the edge. We are getting a bit off the NANOG subject which would be the network implications of this so I will curtail the general discussing of HFT. Steve -Original Message- From: John Levine [mailto:jo...@iecc.com] Sent: Wednesday, August 08, 2012 10:54 AM To: nanog@nanog.org Cc: Naslund, Steve Subject: Re: raging bulls Here is another thought. Many people think that the rapid computer trading does not really add any value to the market in any case since there is no long term investment. It clearly doesn't. A proposal that's been kicking around for a while is to clear all trades once a second, so everyone has plenty of time to get them in no matter where they are. This has no chance of going anywhere, of course, until there's enough software disasters to provide political pushback against the leeches doing high speed trading. There is a new data center in Keflavik, Iceland. They advertise all of its fabulous green characteristics, e.g., power is from geothermal, and A/C is by opening the window, but it also happens to be closer to New York than anywhere in Europe, and closer to London than anywhere in North America, with good cable connections to both. R's, John
MXLogic outage
Probably old news by now, but MXLogic folks are having some major issues today and not reliably receiving inbound mail. Several of our customers are talking with MXLogic about it. FYI. -- Duane Toler deto...@gmail.com
the topic (was: raging bulls)
On Wed, Aug 08, 2012 at 11:10:41AM -0500, Naslund, Steve wrote: We are getting a bit off the NANOG subject You think? A
RE: MXLogic outage
We are the same way. Phones going nuts ringing as we are an MXLogic partner. I am slowly getting email with about a 2-3 hour delay right now. Anyone know any more? -Original Message- From: Duane Toler [mailto:deto...@gmail.com] Sent: Wednesday, August 08, 2012 10:34 AM To: nanog@nanog.org Subject: MXLogic outage Probably old news by now, but MXLogic folks are having some major issues today and not reliably receiving inbound mail. Several of our customers are talking with MXLogic about it. FYI. -- Duane Toler deto...@gmail.com
Re: MXLogic outage
On Wed, Aug 08, 2012 at 04:39:04PM +, Blake Pfankuch wrote: We are the same way. Phones going nuts ringing as we are an MXLogic partner. I am slowly getting email with about a 2-3 hour delay right now. Anyone know any more? -Original Message- From: Duane Toler [mailto:deto...@gmail.com] Sent: Wednesday, August 08, 2012 10:34 AM To: nanog@nanog.org Subject: MXLogic outage Probably old news by now, but MXLogic folks are having some major issues today and not reliably receiving inbound mail. Several of our customers are talking with MXLogic about it. FYI. What MX servers are your affected domains using? Ours are: 208.65.145.3 208.65.145.2 208.65.144.2 208.65.144.3 And no obvious delays currently. Ray
RE: the topic (was: raging bulls)
Thanks for such an intelligent comment that really adds to the discussion. For the record, 1. An article was posted talking about high speed network options. 2. I discussed why they might not be necessary 3. Various comments talked about high speed trading 4. I ended a subject that obviously veered off of networking So yeah I think to answer your question. Steve -Original Message- From: Andrew Sullivan [mailto:asulli...@dyn.com] Sent: Wednesday, August 08, 2012 11:38 AM To: nanog@nanog.org Subject: the topic (was: raging bulls) On Wed, Aug 08, 2012 at 11:10:41AM -0500, Naslund, Steve wrote: We are getting a bit off the NANOG subject You think? A
Re: raging bulls
On Wed, 08 Aug 2012 09:08:27 -0500, Brett Frankenberger said: What it's about is allowing traders to arbitrage between markets. When product A is traded in, say, London, and product B is traded in New York, and their prices are correlated, you can make money if your program running in NY can learn the price of product B in London a few milliseconds before the other guy's program. And you can make money if your program running in London can learn the price of product A in NY a few milliseconds before the other guy's program. If you can money off those milliseconds, then some government supervision is in order - that market is too damned volatile. I see a lot of people proposing ways to make it work, when what modern civilization needs is some market controls to make it *NOT* work. Didn't we learn our lesson the *last* time the financial system almost collapsed because financial wizards found a new way to slice and dice stuff? pgpguKS561Pml.pgp Description: PGP signature
RE: MXLogic outage
We are on .11 and .12. Our email is still a little delayed, but getting better. -Original Message- From: Ray Van Dolson [mailto:rvandol...@esri.com] Sent: Wednesday, August 08, 2012 10:43 AM To: nanog@nanog.org Subject: Re: MXLogic outage On Wed, Aug 08, 2012 at 04:39:04PM +, Blake Pfankuch wrote: We are the same way. Phones going nuts ringing as we are an MXLogic partner. I am slowly getting email with about a 2-3 hour delay right now. Anyone know any more? -Original Message- From: Duane Toler [mailto:deto...@gmail.com] Sent: Wednesday, August 08, 2012 10:34 AM To: nanog@nanog.org Subject: MXLogic outage Probably old news by now, but MXLogic folks are having some major issues today and not reliably receiving inbound mail. Several of our customers are talking with MXLogic about it. FYI. What MX servers are your affected domains using? Ours are: 208.65.145.3 208.65.145.2 208.65.144.2 208.65.144.3 And no obvious delays currently. Ray
Bell Canada outage?
Anyone at Bell Canada / Sympatico can tell us what's going on? Our routing table is going nuts with Bell advertising a lot of routes they shouldn't be
Re: Bell Canada outage?
On Wed, Aug 8, 2012 at 2:31 PM, Zachary McGibbon zachary.mcgibbon+na...@gmail.com wrote: Anyone at Bell Canada / Sympatico can tell us what's going on? Our routing table is going nuts with Bell advertising a lot of routes they shouldn't be Bell leaked a full table. To add to the fun, it seems that TATA took the full table and releaked it. -- Darius Jahandarie
Re: Bell Canada outage?
On Wed, Aug 8, 2012 at 12:31 PM, Zachary McGibbon zachary.mcgibbon+na...@gmail.com wrote: Anyone at Bell Canada / Sympatico can tell us what's going on? Our routing table is going nuts with Bell advertising a lot of routes they shouldn't be Outages mailing list is reporting that Tata is having problems in Montreal affecting 'many routers'...maybe this is related? Chris -- Chris Stone AxisInternet, Inc. www.axint.net
Re: Bell Canada outage?
That's what it looked like. We are connected (@ McGill University) to RISQ and a lot of our routes started to go via RISQ-Bell instead of RISQ-CANET/Canarie On Wed, Aug 8, 2012 at 2:35 PM, Chris Stone axi...@gmail.com wrote: On Wed, Aug 8, 2012 at 12:31 PM, Zachary McGibbon zachary.mcgibbon+na...@gmail.com wrote: Anyone at Bell Canada / Sympatico can tell us what's going on? Our routing table is going nuts with Bell advertising a lot of routes they shouldn't be Outages mailing list is reporting that Tata is having problems in Montreal affecting 'many routers'...maybe this is related? Chris -- Chris Stone AxisInternet, Inc. www.axint.net
Re: Bell Canada outage?
On Wed, Aug 8, 2012 at 2:35 PM, Chris Stone axi...@gmail.com wrote: Outages mailing list is reporting that Tata is having problems in Montreal affecting 'many routers'...maybe this is related? I am a transit customer of both TATA and Bell Canada. We saw route churn and heavy packet loss via both Bell and TATA beginning at approximately 13:25 ET. It took us some time to assess the situation and deactivate both our TATA and Bell BGP sessions. It also took over 10 minutes for my BGP withdraws to propagate from Bell to their neighbors, including Level3. I would guess Bell Canada's routers all have very busy CPU. No information from either NOC yet. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: Bell Canada outage?
On 8/8/2012 2:50 PM, Jeff Wheeler wrote: It also took over 10 minutes for my BGP withdraws to propagate from Bell to their neighbors, including Level3. I would guess Bell Canada's routers all have very busy CPU. I saw the same sort of behaviour from TATA. I shut my peer with them, yet TATA was still advertising to Level3 a good 15min after !??! route-views traceroute 199.212.134.1 Type escape sequence to abort. Tracing the route to 199.212.134.1 1 vl-51.uonet1-gw.uoregon.edu (128.223.51.2) [AS 3582] 280 msec 364 msec 280 msec 2 3.xe-1-3-0.uonet10-gw.uoregon.edu (128.223.3.10) [AS 3582] 252 msec 272 msec 272 msec 3 eugn-car1-gw.nero.net (207.98.68.177) [AS 3701] 272 msec 268 msec 244 msec 4 eugn-core1-gw.nero.net (207.98.64.161) [AS 3701] 260 msec 272 msec 280 msec 5 ge-6-21.car1.Sacramento1.Level3.net (4.53.200.1) [AS 3356] 296 msec 260 msec 312 msec 6 ae-11-11.car2.Sacramento1.Level3.net (4.69.132.150) [AS 3356] [MPLS: Label 83 Exp 0] 304 msec 328 msec 376 msec 7 ae-4-4.ebr2.SanJose1.Level3.net (4.69.132.158) [AS 3356] [MPLS: Label 1123 Exp 0] 328 msec 252 msec 308 msec 8 ae-3-3.ebr1.Denver1.Level3.net (4.69.132.58) [AS 3356] [MPLS: Label 1191 Exp 0] 332 msec 240 msec 232 msec 9 ae-1-100.ebr2.Denver1.Level3.net (4.69.151.182) [AS 3356] [MPLS: Label 1189 Exp 0] 252 msec 56 msec 40 msec 10 ae-3-3.ebr1.Chicago2.Level3.net (4.69.132.62) [AS 3356] [MPLS: Label 1186 Exp 0] 72 msec 72 msec 248 msec 11 ae-1-51.edge3.Chicago3.Level3.net (4.69.138.136) [AS 3356] 200 msec 344 msec 200 msec 12 tata-level3.chicago3.level3.net (4.68.110.194) [AS 3356] 204 msec 200 msec 216 msec 13 * * * 14 if-4-2.tcore1.MTT-Montreal.as6453.net (64.86.31.17) [AS 6453] 80 msec if-13-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.32.2) [AS 6453] [MPLS: Label 3208 Exp 0] 84 msec 228 msec 15 * * * 16 if-5-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.31.22) [AS 6453] [MPLS: Label 1678 Exp 0] 260 msec 244 msec 236 msec 17 if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 272 msec if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 272 msec if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 260 msec 18 if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 224 msec 208 msec 204 msec 19 if-5-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.31.22) [AS 6453] [MPLS: Label 1678 Exp 0] 260 msec 344 msec 300 msec 20 if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 256 msec if-5-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.31.22) [AS 6453] [MPLS: Label 1678 Exp 0] 240 msec if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 268 msec 21 if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 232 msec if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 248 msec if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 256 msec 22 if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 232 msec 240 msec 240 msec 23 if-5-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.31.22) [AS 6453] [MPLS: Label 1678 Exp 0] 244 msec * * 24 if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 284 msec 208 msec 220 msec 25 if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 204 msec 256 msec if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 204 msec 26 if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 224 msec 424 msec 208 msec 27 if-5-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.31.22) [AS 6453] [MPLS: Label 1678 Exp 0] 220 msec * * 28 if-5-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.31.22) [AS 6453] [MPLS: Label 1678 Exp 0] 136 msec 128 msec if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 116 msec 29 if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 128 msec 124 msec 116 msec 30 * 152 msec 120 msec route-views -- --- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, m...@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/
Re: Bell Canada outage?
This is what we saw via our upstream provider, it happened twice: [image: Inline image 1] On Wed, Aug 8, 2012 at 2:58 PM, Mike Tancsa m...@sentex.net wrote: On 8/8/2012 2:50 PM, Jeff Wheeler wrote: It also took over 10 minutes for my BGP withdraws to propagate from Bell to their neighbors, including Level3. I would guess Bell Canada's routers all have very busy CPU. I saw the same sort of behaviour from TATA. I shut my peer with them, yet TATA was still advertising to Level3 a good 15min after !??! route-views traceroute 199.212.134.1 Type escape sequence to abort. Tracing the route to 199.212.134.1 1 vl-51.uonet1-gw.uoregon.edu (128.223.51.2) [AS 3582] 280 msec 364 msec 280 msec 2 3.xe-1-3-0.uonet10-gw.uoregon.edu (128.223.3.10) [AS 3582] 252 msec 272 msec 272 msec 3 eugn-car1-gw.nero.net (207.98.68.177) [AS 3701] 272 msec 268 msec 244 msec 4 eugn-core1-gw.nero.net (207.98.64.161) [AS 3701] 260 msec 272 msec 280 msec 5 ge-6-21.car1.Sacramento1.Level3.net (4.53.200.1) [AS 3356] 296 msec 260 msec 312 msec 6 ae-11-11.car2.Sacramento1.Level3.net (4.69.132.150) [AS 3356] [MPLS: Label 83 Exp 0] 304 msec 328 msec 376 msec 7 ae-4-4.ebr2.SanJose1.Level3.net (4.69.132.158) [AS 3356] [MPLS: Label 1123 Exp 0] 328 msec 252 msec 308 msec 8 ae-3-3.ebr1.Denver1.Level3.net (4.69.132.58) [AS 3356] [MPLS: Label 1191 Exp 0] 332 msec 240 msec 232 msec 9 ae-1-100.ebr2.Denver1.Level3.net (4.69.151.182) [AS 3356] [MPLS: Label 1189 Exp 0] 252 msec 56 msec 40 msec 10 ae-3-3.ebr1.Chicago2.Level3.net (4.69.132.62) [AS 3356] [MPLS: Label 1186 Exp 0] 72 msec 72 msec 248 msec 11 ae-1-51.edge3.Chicago3.Level3.net (4.69.138.136) [AS 3356] 200 msec 344 msec 200 msec 12 tata-level3.chicago3.level3.net (4.68.110.194) [AS 3356] 204 msec 200 msec 216 msec 13 * * * 14 if-4-2.tcore1.MTT-Montreal.as6453.net (64.86.31.17) [AS 6453] 80 msec if-13-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.32.2) [AS 6453] [MPLS: Label 3208 Exp 0] 84 msec 228 msec 15 * * * 16 if-5-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.31.22) [AS 6453] [MPLS: Label 1678 Exp 0] 260 msec 244 msec 236 msec 17 if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 272 msec if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 272 msec if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 260 msec 18 if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 224 msec 208 msec 204 msec 19 if-5-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.31.22) [AS 6453] [MPLS: Label 1678 Exp 0] 260 msec 344 msec 300 msec 20 if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 256 msec if-5-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.31.22) [AS 6453] [MPLS: Label 1678 Exp 0] 240 msec if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 268 msec 21 if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 232 msec if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 248 msec if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 256 msec 22 if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 232 msec 240 msec 240 msec 23 if-5-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.31.22) [AS 6453] [MPLS: Label 1678 Exp 0] 244 msec * * 24 if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 284 msec 208 msec 220 msec 25 if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 204 msec 256 msec if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 204 msec 26 if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 224 msec 424 msec 208 msec 27 if-5-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.31.22) [AS 6453] [MPLS: Label 1678 Exp 0] 220 msec * * 28 if-5-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.31.22) [AS 6453] [MPLS: Label 1678 Exp 0] 136 msec 128 msec if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 116 msec 29 if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 128 msec 124 msec 116 msec 30 * 152 msec 120 msec route-views -- --- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, m...@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ image.png
Re: Bell Canada outage?
Hi, .-- My secret spy satellite informs me that at 12-08-08 11:35 AM Darius Jahandarie wrote: On Wed, Aug 8, 2012 at 2:31 PM, Zachary McGibbon zachary.mcgibbon+na...@gmail.com wrote: Anyone at Bell Canada / Sympatico can tell us what's going on? Our routing table is going nuts with Bell advertising a lot of routes they shouldn't be Bell leaked a full table. To add to the fun, it seems that TATA took the full table and releaked it. A quick analysis leads met to believe AS46618 ( Dery Telecom Inc) is the cause of this. AS46618 is dual homed to VIDEOTRON and Bell. What seems to have happened is that they leaked routes learned from VIDEOTRON to Bell. Based on BGP data I see that at 17:27 UTC AS46618 ( Dery Telecom Inc) started to leak a 'full table', or at least a significant chunk of it to its provider Bell AS577. Bell propagated that to it's peers. Tata was one of the ones that accepted all of that. I can see that Bell propagated at least 74,109 prefixes learned from AS46618 to Tata. Tata selected 70,160 of those routes. Cheers, Andree
Re: Bell Canada outage?
On Aug 8, 2012, at 3:50 PM, Andree Toonk wrote: Hi, .-- My secret spy satellite informs me that at 12-08-08 11:35 AM Darius Jahandarie wrote: On Wed, Aug 8, 2012 at 2:31 PM, Zachary McGibbon zachary.mcgibbon+na...@gmail.com wrote: Anyone at Bell Canada / Sympatico can tell us what's going on? Our routing table is going nuts with Bell advertising a lot of routes they shouldn't be Bell leaked a full table. To add to the fun, it seems that TATA took the full table and releaked it. A quick analysis leads met to believe AS46618 ( Dery Telecom Inc) is the cause of this. AS46618 is dual homed to VIDEOTRON and Bell. What seems to have happened is that they leaked routes learned from VIDEOTRON to Bell. Based on BGP data I see that at 17:27 UTC AS46618 ( Dery Telecom Inc) started to leak a 'full table', or at least a significant chunk of it to its provider Bell AS577. Bell propagated that to it's peers. Tata was one of the ones that accepted all of that. I can see that Bell propagated at least 74,109 prefixes learned from AS46618 to Tata. Tata selected 70,160 of those routes. Taking a cursory look at the data processed by my now (very old) leak auditing tool, this should give you an idea of the size of the impact... bgp=# select count(*) from leakinfo where aprox_time::date='2012-08-08'; count --- 21212 bgp=# select count(*) from leakinfo where aprox_time::date='2012-08-07'; count --- 3681 bgp=# select count(*) from leakinfo where aprox_time::date='2012-08-06'; count --- 2428 You can see details of this at the leak info page here: http://puck.nether.net/bgp/leakinfo.cgi - Jared
IPTV, Multicast, and IPv6 Transition
This may be way off in the future for some of you, or others may be past it. In support of possible work on IPv6 transition in the IETF MBONED Working Group, I'm looking for an indication of what use cases are realistic for operators who: -- deliver IPTV to their customers -- use multicast for the purpose, either now or in the near future -- are making the transition from IPv4 to IPv6. First question: does anyone anticipate a situation where you will have to translate multicast content from a IPv4-only source so it can be delivered to an IPv6-only receiver, or vice versa? Second question: the current official view is that dual stack is the sensible way to transition your network, and failing that, you can use Automatic Multicast Tunneling, an almost-ready Internet Draft available at http://datatracker.ietf.org/doc/draft-ietf-mboned-auto-multicast/ to tunnel multicast packets through a single-stack network when necessary. Do you consider your requirements adequately addressed by this view? People who really want to be helpful can comment on the problem statement and use cases documented in http://datatracker.ietf.org/doc/draft-ietf-mboned-v4v6-mcast-ps/ Thanks for your attention. Privacy will be respected if people want to reply privately. Tom Taylor Consultant
RE: Bell Canada outage?
Could this be causing issue with XO in east coast ? -Original Message- From: Andree Toonk [mailto:andree+na...@toonk.nl] Sent: Wednesday, August 08, 2012 12:50 PM To: nanog@nanog.org Subject: Re: Bell Canada outage? Hi, .-- My secret spy satellite informs me that at 12-08-08 11:35 AM Darius Jahandarie wrote: On Wed, Aug 8, 2012 at 2:31 PM, Zachary McGibbon zachary.mcgibbon+na...@gmail.com wrote: Anyone at Bell Canada / Sympatico can tell us what's going on? Our routing table is going nuts with Bell advertising a lot of routes they shouldn't be Bell leaked a full table. To add to the fun, it seems that TATA took the full table and releaked it. A quick analysis leads met to believe AS46618 ( Dery Telecom Inc) is the cause of this. AS46618 is dual homed to VIDEOTRON and Bell. What seems to have happened is that they leaked routes learned from VIDEOTRON to Bell. Based on BGP data I see that at 17:27 UTC AS46618 ( Dery Telecom Inc) started to leak a 'full table', or at least a significant chunk of it to its provider Bell AS577. Bell propagated that to it's peers. Tata was one of the ones that accepted all of that. I can see that Bell propagated at least 74,109 prefixes learned from AS46618 to Tata. Tata selected 70,160 of those routes. Cheers, Andree
Re: Bell Canada outage?
Thanks for the info, looks like Bell needs to put some filtering on their customer links! Things seem to have calmed down now but I'm still watching my prefixes received from my upstream provider: while true; do snmpget -v2c -c community router 1.3.6.1.4.1.9.9.187.1.2.4.1.1.ip of bgp peer.1.128; sleep 1; done I also monitor all my prefixes and bgp messages in Cacti On Wed, Aug 8, 2012 at 3:50 PM, Andree Toonk andree+na...@toonk.nl wrote: Hi, .-- My secret spy satellite informs me that at 12-08-08 11:35 AM Darius Jahandarie wrote: On Wed, Aug 8, 2012 at 2:31 PM, Zachary McGibbon zachary.mcgibbon+na...@gmail.com wrote: Anyone at Bell Canada / Sympatico can tell us what's going on? Our routing table is going nuts with Bell advertising a lot of routes they shouldn't be Bell leaked a full table. To add to the fun, it seems that TATA took the full table and releaked it. A quick analysis leads met to believe AS46618 ( Dery Telecom Inc) is the cause of this. AS46618 is dual homed to VIDEOTRON and Bell. What seems to have happened is that they leaked routes learned from VIDEOTRON to Bell. Based on BGP data I see that at 17:27 UTC AS46618 ( Dery Telecom Inc) started to leak a 'full table', or at least a significant chunk of it to its provider Bell AS577. Bell propagated that to it's peers. Tata was one of the ones that accepted all of that. I can see that Bell propagated at least 74,109 prefixes learned from AS46618 to Tata. Tata selected 70,160 of those routes. Cheers, Andree
Re: Bell Canada outage?
On 8 August 2012 16:10, Zachary McGibbon zachary.mcgibbon+na...@gmail.comwrote: Thanks for the info, looks like Bell needs to put some filtering on their customer links! I remember when AS577 had those... ;) -- Harald
Re: Bell Canada outage?
CPU's were pegged for a customer of mine in California. tracked it down to 2 events that went down at that time with a large message volume. 1) Peering between GLBX and Level3 dopped somewhere, causing many prefixes to shift away from L3 paths. 2) Some IPv6 prefixes were aggressively bouncing from HE starting at 11:25. Can't trace it any further upstream for HE unfortunately. This cause lots of CPU churn on one of my customers networks May or may not be related. -- Steve
Re: Bell Canada outage?
Further analysis shows that there were actually 107,409 prefixes affected of 14,391 unique origin ASn's. Interested if your prefixes was affected? I've uploaded a list of prefixes and ASn's that were leaked here: http://www.bgpmon.net/bell-leak.txt Cheers, Andree .-- My secret spy satellite informs me that at 12-08-08 12:50 PM Andree Toonk wrote: Hi, .-- My secret spy satellite informs me that at 12-08-08 11:35 AM Darius Jahandarie wrote: On Wed, Aug 8, 2012 at 2:31 PM, Zachary McGibbon zachary.mcgibbon+na...@gmail.com wrote: Anyone at Bell Canada / Sympatico can tell us what's going on? Our routing table is going nuts with Bell advertising a lot of routes they shouldn't be Bell leaked a full table. To add to the fun, it seems that TATA took the full table and releaked it. A quick analysis leads met to believe AS46618 ( Dery Telecom Inc) is the cause of this. AS46618 is dual homed to VIDEOTRON and Bell. What seems to have happened is that they leaked routes learned from VIDEOTRON to Bell. Based on BGP data I see that at 17:27 UTC AS46618 ( Dery Telecom Inc) started to leak a 'full table', or at least a significant chunk of it to its provider Bell AS577. Bell propagated that to it's peers. Tata was one of the ones that accepted all of that. I can see that Bell propagated at least 74,109 prefixes learned from AS46618 to Tata. Tata selected 70,160 of those routes. Cheers, Andree
Re: Bell Canada outage?
Ouch!! That's a lot! What do you think the outcome of this will be? What do you think that Bell did/will do (hopefully) to fix this so it doesn't happen again? On Wed, Aug 8, 2012 at 4:31 PM, Andree Toonk andree+na...@toonk.nl wrote: Further analysis shows that there were actually 107,409 prefixes affected of 14,391 unique origin ASn's. Interested if your prefixes was affected? I've uploaded a list of prefixes and ASn's that were leaked here: http://www.bgpmon.net/bell-leak.txt Cheers, Andree .-- My secret spy satellite informs me that at 12-08-08 12:50 PM Andree Toonk wrote: Hi, .-- My secret spy satellite informs me that at 12-08-08 11:35 AM Darius Jahandarie wrote: On Wed, Aug 8, 2012 at 2:31 PM, Zachary McGibbon zachary.mcgibbon+na...@gmail.com wrote: Anyone at Bell Canada / Sympatico can tell us what's going on? Our routing table is going nuts with Bell advertising a lot of routes they shouldn't be Bell leaked a full table. To add to the fun, it seems that TATA took the full table and releaked it. A quick analysis leads met to believe AS46618 ( Dery Telecom Inc) is the cause of this. AS46618 is dual homed to VIDEOTRON and Bell. What seems to have happened is that they leaked routes learned from VIDEOTRON to Bell. Based on BGP data I see that at 17:27 UTC AS46618 ( Dery Telecom Inc) started to leak a 'full table', or at least a significant chunk of it to its provider Bell AS577. Bell propagated that to it's peers. Tata was one of the ones that accepted all of that. I can see that Bell propagated at least 74,109 prefixes learned from AS46618 to Tata. Tata selected 70,160 of those routes. Cheers, Andree
Re: Bell Canada outage?
On 8/8/2012 4:14 PM, Steve Dalberg wrote: CPU's were pegged for a customer of mine in California. tracked it down to 2 events that went down at that time with a large message volume. 1) Peering between GLBX and Level3 dopped somewhere, causing many prefixes to shift away from L3 paths. FYI: We saw significant route churn on GBLX (AS3549) as well as on Tata. 2) Some IPv6 prefixes were aggressively bouncing from HE starting at 11:25. Can't trace it any further upstream for HE unfortunately. This cause lots of CPU churn on one of my customers networks May or may not be related.
RE: next hop packet loss
They've had me blocked for a few weeks now. I've always been able to reach it on Verizon network with iPhone, just not with Time Warner Business Class. I plan to take advice from kind members of group that offered it and investigate a little more with Wireshark yet have been in middle of client migration of aging Exchange 2003 server to 2010 version in the cloud since Friday. http://www.CheckPoint.com can wait. I had a great face to face meeting with person from another UTM company this morning http://www.sophos.com Regards, Jim Ray, President Neuse River Networks 2 Davis Drive, PO Box 13169 Research Triangle Park, NC 27709 919-838-1672 x100 www.NeuseRiverNetworks.com -Original Message- From: Eugeniu Patrascu [mailto:eu...@imacandi.net] Sent: Wednesday, August 08, 2012 5:07 AM To: Jim Ray Cc: t...@ninjabadger.net; nanog@nanog.org Subject: Re: next hop packet loss On Tue, Aug 7, 2012 at 4:12 PM, Jim Ray j...@neuse.net wrote: Sorry, I do not give verbose responses via iPhone on that small device with my tired old eyes. I ran Wireshark this morning. Without sniffing packets, the layman's description of problem is I can't get to vendor web site, http://www.CheckPoint.com, on Time Warner Business Class network I use. Hence, http is blocked in addition to ICMP. Yesterday, Check Point's website was unreachable from other parts of the world for some time with intermittent access for around an hour or so I believe. Eugeniu
Re: Bell Canada outage?
We have been advised that TATA/6453 is back to normal, and re-activated our BGP to them. Everything seems okay on this front. No update from Bell Canada yet. On Wed, Aug 8, 2012 at 4:11 PM, Harald Koch c...@pobox.com wrote: On 8 August 2012 16:10, Zachary McGibbon Thanks for the info, looks like Bell needs to put some filtering on their customer links! I remember when AS577 had those... ;) We actually have asked Bell Canada not to filter routes from us, and use prefix-limit only, because they were not able to build a prefix-list for us if we have any downstream customer ASNs, which we do. :-/ If someone at Bell Canada is reading and cared to contact me off-list, I would sure love to get my own route filtering fixed. I have had little success through the normal channels. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: Bell Canada outage?
Hi, .-- My secret spy satellite informs me that at 12-08-08 11:35 AM Darius Jahandarie wrote: On Wed, Aug 8, 2012 at 2:31 PM, Zachary McGibbon zachary.mcgibbon+na...@gmail.com wrote: Anyone at Bell Canada / Sympatico can tell us what's going on? Our routing table is going nuts with Bell advertising a lot of routes they shouldn't be Bell leaked a full table. To add to the fun, it seems that TATA took the full table and releaked it. A quick analysis leads met to believe AS46618 ( Dery Telecom Inc) is the cause of this. AS46618 is dual homed to VIDEOTRON and Bell. What seems to have happened is that they leaked routes learned from VIDEOTRON to Bell. Based on BGP data I see that at 17:27 UTC AS46618 ( Dery Telecom Inc) started to leak a 'full table', or at least a significant chunk of it to its provider Bell AS577. Bell propagated that to it's peers. Tata was one of the ones that accepted all of that. I can see that Bell propagated at least 74,109 prefixes learned from AS46618 to Tata. Tata selected 70,160 of those routes. Interesting. I have a server hosted on Bell Canada's network and I saw an outage of about 30 minutes today, but it ONLY affected connections from Verizon's network. This includes my own FIOS connection. I still could connect to the server through Comcast, Level 3 and XO with no problems. Traceroutes from my Verizon IP only got 2 hops, stopping at a philly router and traceroutes back to the same IP from that server got as far as NYC.
RE: next hop packet loss
telnet www.checkpoint.com 80 GET / HTTP/1.1 Host: www.checkpoint.com ...resolved some information and then lost connection according to this trailer from the screen scrape: !-- Column 2 -- div class=column !--- h2a href=https://supportcenter.checkpoint.com/supportcenter/p ortal?ev Connection to host lost. Site resolves fine on Verizon network with my iPhone and not on Time Warner network. Maybe Check Point is mad because my network is behind a Sonic Wall and not their product. Regards, Jim Ray, President Neuse River Networks 2 Davis Drive, PO Box 13169 Research Triangle Park, NC 27709 919-838-1672 x100 www.NeuseRiverNetworks.com -Original Message- From: wher...@gmail.com [mailto:wher...@gmail.com] On Behalf Of William Herrin Sent: Tuesday, August 07, 2012 10:51 AM To: Jim Ray Cc: nanog@nanog.org Subject: Re: next hop packet loss On Mon, Aug 6, 2012 at 11:27 AM, Jim Ray j...@neuse.net wrote: I have a Time Warner Business Class connection and am unable to reach http://www.checkpoint.com to research product line I wish to carry. I did a trace route and confirmed packets are past my network, Time Warner network and onto next hop where they execute jump to nowhere instruction. Here is the tracert just now (it has been failing for weeks): That's an artifact of Checkpoint blocking pings. Note the difference between ICMP and TCP-based traceroutes: traceroute -I 216.200.241.66 traceroute to 216.200.241.66 (216.200.241.66), 30 hops max, 60 byte packets 1 sark.dirtside.com (70.182.189.216) 0.462 ms 0.494 ms 0.555 ms 2 10.1.192.1 (10.1.192.1) 9.023 ms 9.197 ms 9.247 ms 3 ip72-196-255-1.dc.dc.cox.net (72.196.255.1) 15.210 ms 15.497 ms 15.548 ms 4 mrfddsrj01gex070003.rd.dc.cox.net (68.100.0.141) 13.594 ms 13.765 ms 13.817 ms 5 68.1.4.139 (68.1.4.139) 14.752 ms 15.016 ms 14.951 ms 6 ge-8-0-7.er2.iad10.us.above.net (64.125.12.241) 15.075 ms 9.565 ms 9.384 ms 7 xe-5-1-0.cr2.dca2.us.above.net (64.125.29.77) 33.238 ms 26.629 ms 26.554 ms 8 xe-2-2-0.cr2.iah1.us.above.net (64.125.30.53) 45.079 ms 45.230 ms 45.264 ms 9 xe-0-3-0.cr2.lax112.us.above.net (64.125.30.50) 75.982 ms 76.212 ms 76.154 ms 10 xe-2-1-0.cr2.sjc2.us.above.net (64.125.26.30) 93.901 ms 94.044 ms 88.715 ms 11 xe-1-1-0.er2.sjc2.us.above.net (64.125.26.202) 88.542 ms 88.885 ms 90.094 ms 12 64.124.201.230.b709.above.net (64.124.201.230) 89.691 ms 89.060 ms 88.895 ms 13 * * * 14 * * * 15 * * * traceroute -T -p 80 216.200.241.66 traceroute to 216.200.241.66 (216.200.241.66), 30 hops max, 60 byte packets 1 sark.dirtside.com (70.182.189.216) 0.487 ms 0.520 ms 0.568 ms 2 10.1.192.1 (10.1.192.1) 20.018 ms 24.851 ms 25.144 ms 3 ip72-196-255-1.dc.dc.cox.net (72.196.255.1) 25.415 ms 25.502 ms 25.591 ms 4 mrfddsrj01gex070003.rd.dc.cox.net (68.100.0.141) 25.139 ms 25.178 ms 25.260 ms 5 68.1.4.139 (68.1.4.139) 37.509 ms 37.437 ms 37.362 ms 6 ge-5-3-0.mpr2.iad10.us.above.net (64.125.13.57) 91.097 ms 89.808 ms ge-8-0-7.er2.iad10.us.above.net (64.125.12.241) 24.078 ms 7 xe-5-1-0.cr2.dca2.us.above.net (64.125.29.77) 26.324 ms 11.950 ms 12.477 ms 8 xe-2-2-0.cr2.iah1.us.above.net (64.125.30.53) 74.680 ms 74.575 ms 74.355 ms 9 xe-0-3-0.cr2.lax112.us.above.net (64.125.30.50) 76.781 ms 76.330 ms 76.118 ms 10 xe-2-1-0.cr2.sjc2.us.above.net (64.125.26.30) 100.310 ms 100.026 ms 98.495 ms 11 xe-1-1-0.er2.sjc2.us.above.net (64.125.26.202) 98.631 ms 93.570 ms 94.380 ms 12 64.124.201.230.b709.above.net (64.124.201.230) 94.420 ms 97.053 ms 95.015 ms 13 208.185.174.208 (208.185.174.208) 96.208 ms 96.541 ms 96.384 ms 14 www.checkpoint.com (216.200.241.66) 97.406 ms 97.534 ms 97.891 ms Since you get all the way to the Checkpoint border, try some basic diagnostics like: telnet www.checkpoint.com 80 GET / HTTP/1.1 Host: www.checkpoint.com Wait for the telnet to succeed before you type GET. Make sure you press enter twice after the last line. You're hand-jamming an HTTP request. If you don't connect then checkpoint is blocking your IP address for one reason or another. Maybe there are hackers in your neighborhood. Take it up with them by phone. If you do connect but get no response to the get http request then most likely checkpoint is blocking all ICMP packets and your path MTU is smaller than 1500 bytes. The ICMP block prevents the fragmentation needed message from reaching their web server, so it never figures out it needs to shorten its packets. If, as a firewall company, they have made this beginner mistake... 'nuff said. And of course if you do get complete content back from the web server then you have some other problem with your PC that's getting in the way. Regards, Bill Herrin -- William D. Herrin her...@dirtside.com b...@herrin.us 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004
Re: next hop packet loss
On Wed, Aug 8, 2012 at 7:39 PM, Jim Ray j...@neuse.net wrote: telnet www.checkpoint.com 80 GET / HTTP/1.1 Host: www.checkpoint.com ...resolved some information and then lost connection according to this trailer from the screen scrape: !-- Column 2 -- div class=column !--- h2a href=https://supportcenter.checkpoint.com/supportcenter/p ortal?ev Connection to host lost. Site resolves fine on Verizon network with my iPhone and not on Time Warner network. Maybe Check Point is mad because my network is behind a Sonic Wall and not their product. Hi Jim, Immediately lost or lost after a couple of minutes of no output? If there's a long delay, see path mtu detection in my prior post. If not... If immediate, try it from in front of the sonic wall instead of behind it. If it works, your sonic wall is malfunctioning (maybe Sonic Wall is mad that you're checking out a competitor ;) ). If you get the same result (lose the connection 75% of the way through), dig out wireshark and see what packets you send and receive right around the connection lost. It would help to know whether you're getting a TCP FIN, a TCP RST or a destination unreachable, and if the latter which one and from what IP. Regards, Bill Herrin -- William D. Herrin her...@dirtside.com b...@herrin.us 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004
Packets dropped due to ICMP off
Awe, man, don't laugh too hard. Turned out to be problem with Firefox. Safari on iPhone and IE on PC work. I learned something, too, and appreciate the input: tracert using ICMP is not valid test. Not everyone has ping enabled. So, what looks like packet loss at next hop is really ICMP turned off. Sent from my iPhone
