Re: IPv6 Security [Was: Re: misunderstanding scale]
On 03/26/2014 11:14 PM, Owen DeLong wrote: Why not just use private VLAN layer 2 controls for the privacy you describe? The technology I know of is what cisco calls 'protected ports' - My understanding is that those simply mean you can't pass traffic to or from other 'protected ports' - I use that capability when, say, putting a bunch of IPMIs on a private network, it works great, as if one of the IPMI ports is trying to talk to another, something is very wrong and it gets blocked. They are commonly used in the dedicated server hosting world to do what you are describing, but they have a big downside when being used on the public side;customer 1 can't talk to customer 2.Now, this isn't usually a big deal, except in one very common case; what if one entity buys two hosts? now those two hosts can't talk to oneanother. This is a very common problem for dedicated hosting providers (and why I give my dedicated hosts a vlan and a routed subnet, wasting IPv4.) For my virtuals, though, I have a much more clever switch as it's just some software running in the Dom0, so at least in the IPv4 world, filtering just their /32 in and out is a much better solution. Yes, you risk customer A spoofing customer B, but is that really a problem in your environment? Really? If so, one could argue you might want to consider getting a better class of customers. You wouldn't feel uncomfortable if some other company could come in and not only spoof your IP, but receive the return traffic? Keep in mind that they could do this in a way that is quite difficult to detect or trace, if they were clever about it. I may trust my provider, to a certain extent, but I certainly don't trust everyone who gives my provider money.
Re: IPv6 Security [Was: Re: misunderstanding scale]
It might make sense to just give everyone their own vlan and their own /64; that would, of course, bring its own problems and complexities (namely that I've gotta have the capability to deal with more customers than I can have native vlans - not impossible to get around, but significant added complexity.) I don’t see the point of that. why not? After carefully considering everything you have told me, this sounds like the way forward to do it the IPv6 way - privacy IPs would work fine, and I could filter every port such that only packets from that /64 were allowed out and only addresses to that /64 would be allowed in.Nobody would be able to spoof or listen in on their neighbor; yeah, my router would have to send a lot of RAs, but routers that handle the amount of traffic my customers send are cheap. I have a lot of customers, sure, but they are small. Sure, it's going to cost me in routing complexity, but it looks like the only thing I can do that will actually solve my problems and use IPv6 the way IPv6 is expecting to be used. I'd then have to figure out how to make their ipv4 /32 work, but I can think of several possibilities that might work. If nothing else, I could give them one interface for IPv6 and one for IPv4, and leave the IPv4 interface the current system.
Re: IPv6 Security [Was: Re: misunderstanding scale]
On 03/24/2014 06:18 PM, Owen DeLong wrote: DHCPv6 is no less robust in my experience than DHCPv4. ARP and ND have mostly equivalent issues. This depends a lot on what you mean by 'robust' Now, I have dealt with NAT, and I see IPv6 as a technology with the potential to make my life less unpleasant. I really want IPv6 to succeed. However, DHCPv6 isn't anywhere near as useful for me, as someone who normally deals with IPs that don't change, as DHCPv4 is. With DHCPv4, my customers all get an address based on their mac that doesn't change if their box is re-installed. I configure this on the DHCP server, and the customer can run whatever dhcp client they like on whatever OS they like and they get the same IP every time. With DHCPv6 there is a time-based identifier that is added to the mac that makes it impossible, as far as I can tell, to give the customer a consistent IP across OS wipes without doing significant client configuration. There are many ways to skin this cat; stateless autoconfig looks like it mostly works, but privacy extensions seem to be the default in many places; outgoing IPv6 from those random addresses will trip my BCP38 filters. That, and reading the standard, it sure doesn't sound like consistency was a goal, even though it seems fairly consistent experimentally. there's a lot of generally and may in the text about what it adds to the mac in order to get the local identifier. It might make sense to just give everyone their own vlan and their own /64; that would, of course, bring its own problems and complexities (namely that I've gotta have the capability to deal with more customers than I can have native vlans - not impossible to get around, but significant added complexity.) I suppose I can also just keep DHCPv4 around, and if folks want IPv6, well, they have to wire down the address themselves. That's how I'm doing it now.
Re: IPv6 Security [Was: Re: misunderstanding scale]
On 03/26/2014 03:49 PM, Matt Palmer wrote: On Wed, Mar 26, 2014 at 10:55:03AM -0700, Luke S. Crawford wrote: There are many ways to skin this cat; stateless autoconfig looks like it mostly works, but privacy extensions seem to be the default in many places; outgoing IPv6 from those random addresses will trip my BCP38 filters. Your what-now? You do realise SLAAC works entirely within a single /64, which shouldn't be difficult to decide is either routable or not in one hit, right? If you give every customer their own vlan and /64, sure. That can be done, and there are many advantages to doing it that way. But it's quite a bit more complex than my current setup. The way I'm setup now, I've got an IPv4 address block on a vlan, and an IPv6/64 on the same vlan. I have many customers on that vlan. Each customer has one (or more) IPv4 /32 addresses and one IPv6 /128 addresses. (if the customer wants more IPv6, we just route a /64 to the /128 they are allowed.) There are firewall rules that only allow appropriate packets in and out of the interface.These rules are important for privacy as well as preventing spoofing; they prevent sniffing of most traffic bound for other guests. This is in production on many of my hosts, and because I give every user both an IPv4 and an IPv6 address, this mostly works. My setup scripts wire down both the v4 and v6 addresses before I hand it off to the user; if the user wants re-install, well, they can wire down the IPv6 address by hand if they want it, and IPv4 works regardless. It is valid to say that I'm trying to use IPv6 the way I use IPv4, and perhaps that is the wrong thing to do. Perhaps IPv6 needs to be thought of in a different way from IPv4; Perhaps in IPv6, a /64 should be the smallest block I give to a user, and the smallest block I filter on, and I just need to eat the network complexity costs inherent to giving each user a vlan. My original comment and complaint, though, was in response to the assertion that DHCPv6 is as robust as DHCPv4. My point is that DHCPv6 does not fill the role that DHCPv4 fills, if you care about tying an IP to a MAC and you want that connection to persist across OS installs by customers.
Re: Evaluating Tier 1 Internet providers
On 08/29/2013 07:43 PM, Blake Dunlap wrote: +10 Good explanation. This is a lot of why I have someone like Cogent/L3/etc and some random transit provider in most of my pops I spec, plus a backhaul to another node. ... One thing to keep in mind is that for major Tier 1s, it's not at all uncommon to see some very large percentages of traffic (like say well north of 50%) stay completely on-net, going from customer to customer. In this type of model, capacity to other third party peers (typically the other Tier 1's) becomes secondary to other considerations like backbone capacity, which is why those huge Tier 1 networks often have much less peering capacity than you might otherwise expect. a major problem here is that some providers try too hard to be tier 1... - my pager has gone off many times because $lowcost_tier1 decided to route a packet from them in san jose destined for them in Sacramento through texas. Problem is, often that is still fewer hops, (even if it's many more ms) than going through my tier2 provider, so having the backup did not help me. Nor would taking customer-only routes from $lowcost_tier1... the shortest path, in terms of hops, was through them, through texas. There was nothing to be done short of switching to my tier2. I have no idea how to solve this sort of problem automatically. Ideally, if someone has a congested or down link, I'd prefer that they not announce routes to that part of the internet, as I do have a backup, but that isn't how it works.
Re: PDU recommendations
I also have had good experience with (used) servertech/century/power tower (I think all the same brand) - very inexpensive; if you are in santa clara I have some spare 2u 16 port 208v (20a/c19) units. Here is something a buddy wrote up when we were wiring them to the user-accessable power on/off menu: http://blog.prgmr.com/xenophobia/2012/02/notes-on-setting-up-a-sentry-p.html My new rack is all avocent PM3001-401 units.Used, of course; but the feature I was after was per-port power monitoring. I haven't quite gotten 'em all the way figured out yet. One thing I see as a negative (but might be positive?) is that they have fuses, not breakers.I don't know if this provides better protection; I do know that when my buddy overloaded one of them in testing, I had to replace fuses, rather than just switching a breaker back. (also, when a different buddy plugged a ancient desktop (so old the PSU wasn't auto-switch) with the power input switch set to 110 in, it blew some of the fuses in the PDU (and took out the rack) - it didn't damage any of the other servers on the pdu (other than taking out the PDU; but everything came back up when I swapped it with my spare.) Also note, uh, the servertech and the avocent and I think all the other PDUs I've seen can reboot the management interface without flipping the outlets. I did it a bunch when I was getting familiar with the avocent. Yeah. I think I need to give fewer buddies access to production. Nobody takes hardware seriously enough. I can find people I trust with root, and that trust doesn't seem misplaced.But I let them touch the hardware? and they fuck it up.So I end up doing almost all the hardware stuff myself. On 06/23/2013 04:48 PM, Trey Valenta wrote: I'll also throw out recommendations for ServTech PDUs. They have an affordable line of PDUs with static transfer switches that are particularly attractive for all your single-power-supply devices.
Re: LinkedIn password database compromised
On Wed, Jun 06, 2012 at 07:43:42PM -0700, Aaron C. de Bruyn wrote: Why haven't we taken this out of the hands of website operators yet? Why can't I use my ssh-agent to sign in to a website just like I do for about hundred servers, workstations, and my PCs at home? One local password used everywhere that can't be compromised through website stupidity... This is the way to go. The problem here is that x.509 is the only similar thing for browsers, and x509 requires a ca, which makes the whole process a whole lot more complext than the 'just give me the public half of the key you want to use to authenticate to this service' I mean, unless everyone trusts the same (few) CAs, which has a different set of problems. I haven't found any way that is as simple and as portable as using ssh that works in a web browser. I'm considering re-writing my billing application to be libcurses based or something, and letting users access that through ssh, too. (It would be silly, but it might work for me; it goes along with my schtick.)This would be somewhat suboptimal for things like bandwidth graphs, but eh. but yeah, if someone wants to pass the hat to get an apache module and a firefox addon written to do public key authentication over http using ssh keys, I'd put a couple hundred bucks in the hat.
Re: Industry practice for BGP costs - one time or fixed/monthly?
On Sun, May 27, 2012 at 12:34:22PM +1000, Matthew Palmer wrote: On Sat, May 26, 2012 at 09:39:16PM -0400, Luke S. Crawford wrote: On Sat, May 26, 2012 at 10:06:03AM +1000, Matthew Palmer wrote: ... Feel free to turn the process around -- decide what the service is worth to you, tell the provider of the service that price, and let them decide if they want to provide it to you at that price. Don't be too surprised if you get monkeys in exchange for your peanuts, though. Are you suggesting that you get worse service after you negotiate a better deal with a particular provider? To a certain extent, yes. It has been my experience (from both the service provider and the customer point-of-view) that customers who aren't worth as much to a supplier don't get as much love, because the cost of losing their business to a competitor is much less (or, in some cases, would be a net win). How is this communicated to the people doing the support? is there a 'cheap jerk' bit in the database? Until I got my own company, working as the technical guy in another company, I was never told what a customer was paying. I mean, I could see what plan they were on, but as a technical person I don't see the price they are paying unless I go directly into the billing database, and that sort of thing is usually super secret. From a business logic perspective, I agree that it seems like you describe the way it ought to be. Sometimes at very small companies? this is the case. I remember a few times, the boss telling me customer X is a big deal. go out of your way for them - but that happens less and less as the company gets bigger. Think, for a moment; if it's not in the database in an easily accessible manner, even if you do all the negotiation yourself, how many customers would you need before you lost track of who underpaid and who overpaid? for me, the limit would be around 5. I bet even a professional relationship manager would have a hard time around 100 or so. Of all my current providers, the worst response I get from sales is from the provider that I'm paying full price for. I mean, maybe that's because they see themselves as premium and I'm small? but maybe that's because I showed weakness by accepting the list price. Or maybe it's because when I met the salesperson I was driving a cheap car and dressed for work. I don't know.
Re: Industry practice for BGP costs - one time or fixed/monthly?
On Sat, May 26, 2012 at 10:06:03AM +1000, Matthew Palmer wrote: We pay what our providers think they can get away with. Like most pricing decisions, they're not based on any technical logic, they're based on what the market will bear. Feel free to turn the process around -- decide what the service is worth to you, tell the provider of the service that price, and let them decide if they want to provide it to you at that price. Don't be too surprised if you get monkeys in exchange for your peanuts, though. Are you suggesting that you get worse service after you negotiate a better deal with a particular provider? I mean, certainly the different providers have different levels of quality, but my experience has been that with a particular provider, you get the same service regardless of how well you negotiated, assuming you eventually came to an agreement. (quite often, in fact, I see the customer that asks for more... quite often gets more, without paying more. We've all had that 'difficult customer' that consumes far more support hours than what they pay could possibly buy. Quite often, that same customer negotiated a better deal, too. It's a cost of selecting for customers that are good at negotiation that most businesspeople don't take into account.) Back to negotiating for initial prices: as far as I can tell, this is largely a matter of knowing the market. The high initial prices are there so that the people that are unable or unwilling to put in the effort to find what the real market prices are pay a lot more. I know as my own business has progressed? the prices I pay even for the same unit of commodities that don't fall in price (like rack space) goes down fairly dramatically every year, simply because I understand the market better.
Re: ISPs and full packet inspection
On Thu, May 24, 2012 at 08:50:47AM -0400, not common wrote: Hello, I am looking for some guidance on full packet inspection at the ISP level. Is there any regulations that prohibit or provide guidance on this? Unless you are absolutely huge, and maybe even then, you need to worry more about how your customers will perceive this than how law enforcement will perceive this. (I mean, you want to follow the law, sure, but even if it's legal, if it cheeses the customers? well, you have a problem.) More to the point, like most on this list, law isn't my field. In my experience? customers get really, really uncomfortable with you doing, well, almost anything below the headers. I was talking about doing a inward facing snort IDS (to detect compromised hosts before I got complaints) and got so far as a prototype where I shared the info I recorded about each IP with the customer in question, but talking to customers? this idea was extremely offensive, so the project was quashed. Now, generally speaking, customers are much more okay with you going through the IP headers. For instance, instead of using an IDS, I could, say, count the number of outgoing connections destined for port 22 or 25, or the same but count how many unique destinations they use (e.g. to avoid MX host or ssh tunneling false positives... both of those use cases would have a lot of connections on those ports, but to a small number of remote hosts.) From what I've heard customers say, this would likely cause less offense than using snort or the like to do full packet inspection. (it wouldn't be completely inoffensive, but I think that if I wiped the logs often and shared my data with the customer, it sounds like something that customers would tolerate.) I haven't prototyped that system yet, though, so eh, who knows.
Re: Squeezing IPs out of ARIN
On Tue, Apr 24, 2012 at 01:32:17PM -0400, ad...@thecpaneladmin.com wrote: Anyone have any tips for getting IPs from ARIN? For an end-user allocation they are requesting that we provide customer names for existing allocations, which is information that will take a while to obtain. They are insisting that this is standard process and something that everyone does when requesting IPs. Has anyone actually had to do this? I have. clearly, I should have asked, or looked closer, but when I started this mess? it was not at all clear to me that ARIN saw things that went into a home as 'residential' and everything else as 'business' - but from my reading and their reactions to my questions, that's how they see it. If it's in a data center and not in a residence, you need to give them a name (human or business) for every reassigned IP, even if the reassignment is a /32. Probably the majority of my VPSs? personal use, but not residential. I started with changing the privacy policy, and blogged about it, asking for at least 80% of the people to opt-in. Maybe 2% did. I gave it months, then I emailed everyone, asking them to opt-out. I gave them two weeks, maybe 2% did. So yeah; eh, nobody got mad at me for it, and I think some people were impressed that I emailed them when I made such a large change to the privacy policy (that isn't expected?) so I guess it all turned out okay, but yeah. ARIN wants a name of some sort for every /32. (Now, I just did a query against my billing database and returned the business name and only returned the human name if there was no business name.)
Re: Most energy efficient (home) setup
On Sun, Apr 15, 2012 at 10:52:51AM -0500, Jimmy Hess wrote: Consider that the probability 16GB of SDRAM experiences at least one single bit error at sea level, in a given 6 hour period exceeds 66% = 1 - (1 - 1.3e-12 * 6)^(16 * 2^30 * 8).In any given 24 hour period, the probability of at least one single bit error exceeds 98%.Assuming the memory is good and functioning correctly; It's expected to see on average approximately 3 to 4 1-bit errors per day. More are frequently seen. Now if most of this 16GB of memory is unused, you will never notice that over 30 days, 120 or so bits have been flipped from their proper value.. I think that is an overestimate, at least if single-bit (corrected) ecc errors are as common as flipped bits on non-ecc ram. Now, First, count me in the ECC is a must, full stop. crowd. I insist on ecc for even my customer's dedicated servers, even though most of the customers don't care that much. It's not for you, it's for me. With ECC? if you have EDAC/bluesmoke setup correctly on a supported motherboard, you get console spew whenever you have a single-bit error. This means I can do a very simple grep on the box conserver logs to and I can find all the failing ram modules I am responsible for. Without ecc, I have no real way of telling the difference between broken software and broken ram. That said, I still think the 120 bits a month estimate is large; I believe that ECC ram should report correctable errors (assuming a correctly configured EDAC/bluesmoke module and supported chipset) about as often as non-ecc ram would get a bit flip. In a past role, I did spend the time grepping through such a properly configured cluster, with tens of thousands of nodes, looking for failing hardware. I should have done a proper paper with statistics, but I did not. The vast majority of servers had zero correctable ecc errors, while a few had a lot, which is consistent with the theory that ECC errors are more often caused by bad ram. (Of course, all these servers were in proper cases in a proper data center, which probably gives you a fair bit of shielding.) On my current fleet (well under 100 servers) single bit errors are so rare that if I get one, I schedule that machine for removal from production.
Re: Question about peering
On Sat, Apr 07, 2012 at 06:16:30PM -0400, Robert E. Seastrom wrote: Sometimes making the AS path as short as possible makes a lot of sense (e.g. when trying to get an anycast network to do the right thing), but assumptions that peering results in lower costs are less true every day. I keep reading people say that. But wouldn't the same forces that push down the per-megabit cost of transit also push down the per-megabit cost of peering?
Re: Question about peering
On Sat, Apr 07, 2012 at 07:25:24PM -0400, Robert E. Seastrom wrote: Generally the costs of transit are pushed down by competition. As a vendor your costs for bandwidth/transport/port*bw may drop but you are unlikely to drop your prices to your customers merely because your costs have gone down unless prompted to by a competitor. ah, so it's not the cost of production that is the problem, it is the 'natural monopoly' state of an IX that is the problem. It seems like that problem could be overcome by making the IX a cooperative owned by the members, maybe? Consider the case of a peering n00b network (the target of this discussion after all) in hypothetical facility that charges $1000/month for a gigabit ethernet port on the peering fabric. You I am in almost that exact position (A peering n00b network) - Of couse, I'm fairly certain I'm paying sucker prices, but I can get a gigE to any2 at 55 s market for less than a third the price you quote. just a data point.
Re: last mile, regulatory incentives, etc (was: att fiber, et al)
On Sat, Mar 24, 2012 at 02:42:36PM -0500, Frank Bulk wrote: I've been many times where you were, frustrated that I didn't know the dark fiber options for a potential opportunity, but you have to remind yourself don't have a *right* to know where *private* fiber is. It's not just the physical property, the lack of documentation is a competitive advantage. Considering that nearly all of this fiber runs over public right of ways granted by the government (and sometimes through the use of force by the government) it's not really private in the sense that it would be if you bury fiber on land you own, or on land owned by private individuals that have given you the right to run fiber over or through the land through some voluntary exchange of value. The public right of ways are created by the government as a public good, and as such, I think the people have a right to know what goes on in them. (Actually, I was talking to a far more experienced friend the other day, and he says that I should be able to contact the PUC and get exactly this data, though often this, too, is somewhat difficult, so when I re-start this project in a few months, that's the direction I am going to attack first.) Legal issues aside, treating a lack of documentation as a competitive advantage makes any transaction vastly less efficient when you consider both parties. I don't do business that way, and when I have a choice? I don't do business with companies that do. Yes, it is legal, and I am not suggesting that should change. But it's still an asshole move that (from a perspective that considers both parties) destroys value. I talked to the silicon valley power people (the operators of the Santa Clara municipal fiber network) and they gave me a cost per mile and a very detailed map (down to what side of the street the fiber is on) - they wouldn't let me have a copy of the map that actually documented the 'pull boxes', but still, it was enough information that I could look at a building and tell pretty quickly if I was wasting their time or not by getting a quote. Talking to anyone else? no maps (or ridiculously vague maps) and no cost per mile. I have to pick two endpoints and ask how much. In my case, the endpoints depend almost entirely on how much it costs, this means I waste a whole lot of salesperson time, and my own time. It's a vastly less efficient way to do business.
Re: last mile, regulatory incentives, etc (was: att fiber, et al)
On Thu, Mar 22, 2012 at 01:31:47PM -0400, Jared Mauch wrote: You agree on a price per distance (e.g.: mile/foot/whatnot). Lets say the cable costs $25k to install for the distance of 5000 feet. That cable has 144 strands. You need access to one strand. If you install it yourself, it will cost you $25k. If you share the pro-rata cost, it comes out around $174 for that strand. Lets say they mark it up 10x (profit, unused strands), would you pay $1740 for access? What does emergency restoration cost? WDM/DWDM add cost to that strand, but also increase the capacity based on what your overall lit capacity may be on a route. There are various cwdm/dwdm systems that range the usual 10/20/40/80/100km ranges. You obviously need to do the math yourselves on this. You may find the ROI is better than you think... I'm trying to do just that right now, actually. 55 s. market to 250 Stockton in San Jose. I dono if it's five thousand feet, but it's not twice that. The cheapest fiber pair I can rent from someone else I've found is $5K/month; the cheapest build-out I've found is $150K, so even if I'm only using one pair in that, if I can get money at anything like a reasonable interest rate, if I plan on sticking around more than 5 years it makes sense to lay new fiber. Which is weird, as this is probably one of the densest masses of existing fiber in the world, going from a 'center of the universe' data center to a minor data center. Even the $5K/month rate isn't bad. If they asked for a third of that, I'd bite even though I don't need that much capacity quite yet. The big problem here, I think, is that it's quite difficult to figure out who has what fiber where, and even once you know who owns it, to find out who to talk to at a company that might know what 'dark fiber' is, much less know how much they might rent it to you for. I spent several hours last month on the phone with XO and I kept getting redirected to someone trying to sell me a T1. I've got other projects right now, but once I'm done with that, I'm going to be spending a bunch of time pestering the PUC and other people that might know who owns fiber between here and there. As for equipment cost, in my corner of the world, I can get used cisco 15540 systems for what I consider to be not very much money, and 32 10G waves is plenty for what I'm doing. I mean, they eat way more power than is required, and 10G/wave is not great these days, but if I could sell a reasonable number of waves, even at a whole order of magnitude below market, I'd be in good shape. The whole project seems dramatically cheaper than lit services. At quoted prices, 10G waves over the same distance cost about 1/2 what a full pair of dark fiber costs. Now, the big problem with the build out? as far as I can tell, I've gotta be a carrier to actually own fiber in the ground. From what I understand, that's not out of the question for me, but it's definitely a lot of work and red tape. There are, however, companies that will do a build out for you (of course, charging you for it up front) then they will lease you the fiber at a very low yearly rate - right now, that looks like the second-best option, where the best option is hunting down the owners of all the dead bundles of fiber going into the meetme room.(250 Stockton is ex-enron, it's got bundles coming in from MFN, quest, global crossing, MCI, enron broadband xo and others. I'd bet money that if I had the kind of access to the meetme at 55 s. Market that I have at 250 Stockton I could start shining light down empty strands and I'd see some of it come out the other side.) But from the amount of time it takes to just find someone at those companies that even knows what dark fiber is? I think I might be better off putting in the effort to do whatever regulatory red tape is required to own fiber in the ground. So yeah; really? in my corner of the world, the problem is the same problem you see everywhere else in this industry. Any useful information is guarded jealously. In this case, where does the fiber run? I mean, I have pretty good maps of the Santa Clara municipal fiber network; but the private networks are impossible.
Re: Flexible BGP liist?
On Thu, Mar 15, 2012 at 10:41:18PM -0400, Joe Maimon wrote: So we have a wiki list of 1U rack hosting. We do? where? all I see on http://nanog.cluepon.net is spam How about a list of SP's willing to configure BGP over whatever you got, including tunnels? And willing to allocate you space for same? Put me down there. me too.
Re: Botnet hunting resources (was: Re: DOS in progress ?)
goe...@anime.net writes: On Fri, 8 Aug 2009, Luke S Crawford wrote: 1. are there people who apply pressure to ISPs to get them to shut down botnets, like maps did for spam? sadly no. ... Why do you think this might be? Fear of (extralegal) retaliation by botnet owners? or fear of getting sued by listed network owners? or is the idea (shunning packets from ISPs that host botnets) fundamentally unsound? If someone sufficiently trustworthy produced a BGP feed of networks that were unresponsive to abuse complaints, do you think other networks would use it to block traffic? I mean, ultimately I think that having several providers of such feeds with differing levels of aggression would be the best case, but someone has got to go first. -- Luke S. Crawford http://prgmr.com/xen/ - Hosting for the technically adept http://nostarch.com/xen.htm - We don't assume you are stupid.
Botnet hunting resources (was: Re: DOS in progress ?)
Jorge Amodio jmamo...@gmail.com writes: Are folks seeing any major DOS in progress ? Twitter seems to be under one and FB is flaky. From what I understand, it's quite common. I got hammered last week. It took out some routers at my upstream (it was a tcp syn flood attack, a whole lot of really small packets. 20Kpps was the peak I saw before the upstream took me out.) Now, I've cleaned up the mess; (and for now, dropped the inexpensive upstream with the weak routers) I'm building out my monitoring infrastructure and generally preparing for next time. as far as stopping the attacks by 'finishing the job' - which is to say, blackholing the target, the way forward is pretty clear. I mean, I need to do more research and implement stuff, but I don't really need NANOG help for that. The thing is, I like my customers. I don't want to shut off people who are paying me just because they get attacked. I mean, if that's what I've got to do to keep my other paying customers up, I'll do it, but I'd really rather not. what is the 'best practice' here? I mean, most of this is scripted, so conceivably, I could get source addresses fast enough to block them upstream. (right now my provider is only allowing me to blackhole my own space, not blackhole source addresses, which while it keeps me in business, is not really what I want.) My provider does seem to be pretty responsive, so if I can bring them a tool, they might set it up for me. But yeah, I'm getting sidetracked. I guess there are two things I want to know: 1. are there people who apply pressure to ISPs to get them to shut down botnets, like maps did for spam? I've got 50 gigs of packet captures, and have been going through with perl to detect IPs who send me lots of tcp packets with 0 payloads, then manually sending abuse reports. Half the abuse reports bounce, and the other half are ignored. (most of the hosts in question are in china.) 2. is there a standard way to push a null-route on the attackers source IP upstream? I know the problem is difficult due to trust issues, but if I could null route the source, it's just a matter of detecting abusive traffic, and with this attack, that part was pretty easy. -- Luke S. Crawford http://prgmr.com/xen/ - Hosting for the technically adept http://nostarch.com/xen.htm - We don't assume you are stupid.
Re: Why choose 120 volts?
Seth Mattinen se...@rollernet.us writes: I have a pure curiosity question for the NANOG crowd here. If you run your facility/datacenter/cage/rack on 120 volts, why? I've spent the last several days going back and forth with salespeople, trying to find a rack with 208v power in the south bay, or a cheap 100M connection from market post tower to heraklesdata in Sacramento. (where I have cheap 208v power) From what I see, most places in the bay area just can't handle the kind of heat density that a 30a 208v circuit per rack would bring. (they won't sell me more than 2 20A 120v circuits, either, and many will only sell me a single 15a circuit per rack. I assume that's an effort to keep the heat output within cooling system capabilities.) But that still doesn't explain why they don't hand out 10a 208v circuits. I've also seen employers pick 208v over 120v even after I pointed out the cost per watt advantages of 208v, even without counting efficiency gains. In one case they provisioned one rack with 208v, because the vendor of some particularly expensive bit of equipment recommended it, then they left all the commodity servers on 120v. Why didn't they put everything on 208v? I pointed out that the cost per watt was lower. Maybe I blew my credibility by wanting to research 48v power supplies for our kit before that? (it was a telco facility, after all, and I was young.) 30a 208v is about perfect for a rack, if you ask me. (I imagine the guys who have to deal with cooling feel differently, but at my scale, that's all priced into the power.) -- Luke S. Crawford http://prgmr.com/xen/ - Hosting for the technically adept We don't assume you are stupid.
Re: integrated KVMoIP and serial console terminal server
Joe Abley jab...@hopcount.ca writes: What is everybody's favourite combination rack-mount VGA/USB KVM-over- IP and serial console concentrator in 2009? I'm looking for something that will accommodate 8 or so 9600bps serial devices and about 12 VGA/USB devices, all reachable over IP via sane means (ssh, https, etc). Being able to trunk everything through cat5E/ RJ45 plant is not necessary; in this application everything is in the same cabinet. I can't speak to the KVM over IP (for *NIX, they are obviously inferior to serial) but I do have some suggestions for the serial end. Personally, I use an opengear cm4148; it seems to work fairly well. If I only needed 8 ports, I'd still be using my solution from 2005, which was an 8 port rocketport serial card in a FreeBSD box. I only moved to the opengear because I need many more ports I like both the opengear and the freebsd box because I can use ssh auth, I can log, and I can lock down each user so that a given private key can only view a certain port.
Re: REVERSE DNS Practices.
bmann...@vacation.karoshi.com writes: or - the more modern approach is to let the node (w/ proper authorization) do a secure dynamic update of the revserse map - so the forward and reverse delegations match. ... a -VERY- useful technique. I have a question. Is this an abuse problem? some ISPs require their domain to be in the rdns in an effort to herd abuse reports to the correct org. Is this generally considered useless? Is it generally considered OK to hand relatively untrusted users the keys to their own rdns? (I'm forcing my own customers to have a rdns of something.xen.prgmr.com for several months, Much to the chagrin of many presumably innocent and legitimate customers. )
Re: Security Intelligence [Was: Re: Netblock reassigned from Chile to US ISP...]
Brandon Galbraith brandon.galbra...@gmail.com writes: But it's definitely not cool when my credit card company cuts off my card due to abnormal charges when I'm abroad and suddenly can't get ahold of customer service via their international phone number. Automation in the right places works wonders for both convenience and the bottom line. In the wrong places, it's a sawed off shotgun pointed at your feet. Yeah, in this case, I think getting the rules right is the hard part... I don't think it matters that much if the rules are executed by a level-1 person vs. a script (the script, I think, would be more consistent, at least.)Sure, if you can afford to page someone good to deal with it, that's probably an even better answer, assuming they can get to it quickly, but that's much more expensive than just blocking it. (I imagine the right approach depends a lot on what you happen to be charging the customers in question.) Even if you do decide to wait around for an abuse@ complaint to take action, having the IDS logs of the outgoing traffic makes corroberating an abuse complaint much easier. And it's easy enough to email a human instead of shutting off a customer automatically.
Re: Security Intelligence [Was: Re: Netblock reassigned from Chile to US ISP...]
Randy Bush ra...@psg.com writes: be specific, like if you run X tools the payoff will be Y. Yes. And where is the appropriate form for this?I find this sort of thing quite interesting; and yeah, it doesn't seem like the sort of thing NANOG is for, but most of the small ISP forms (like webhostingtalk, etc...) well, the average technical skill level seems to be ridicioulously low. Some people talk about ways to give spammers only one 'whack' at your service, such as requesting a faxed ID ahead of time, or putting more effort into preventing credit card fraud. Me, my focus has been on detecting abuse from my customers before the rest of the world starts complaining. speaking as a small provider, I can tell you that I find running snort against my inbound traffic does reduce the cost of running an abuse desk. I do catch offenders before I get abuse@ complaints, sometimes. Granted, my snort-fu is not awesome. just the other week I was reminded that I wasn't even checking for ssh dictionary attacks. There is a lot more work i need to do with snort before I can have it automatically switch off customers, or notify me at a high priority, rather than writing to a log I read once every few days.Still, I think I am on the right track, as even with my poor, neglected snort setup I still catch some problems before I get complaints. I don't see anyone else talking about doing anything similar... Everyone else seems to be focused on preventing spammers from signing up or going after them after the fact. It seems to me that some effort into detecting abuse as it happens (rather than waiting for an abuse@ complaint, something that, in my experience takes a rather large amount of abuse to trigger.) could yield quite a lot of 'low hanging fruit' simply because not much effort has been put out in that direction. On the other hand, I have a hard time believing I'm smarter than the guys running ec2. So maybe I'm missing something, and it's really not actually any cheaper than manning the abuse@ desk with a bunch of grunts. Or maybe other people are already doing this, and I've just missed the conversation. Maybe even if you tune snort optimally, it still can't detect enough of the attacks to be useful?
Re: Security Intelligence [Was: Re: Netblock reassigned from Chile to US ISP...]
Randy Bush ra...@psg.com writes: speaking as a small provider, I can tell you that I find running snort against my inbound traffic does reduce the cost of running an abuse desk. I do catch offenders before I get abuse@ complaints, sometimes. unfortunately snort does not really scale to a larger provider. and, to the best of my poor knowledge, good open source tools to black-hole/redirect botted users are not generally available. universities have some that are good at campus and enterprise scale. I can't speak to the scaling of snort (I only eat around 20Mbps, and snort on a 256Mb Xen VM handles it just fine) but I'm not sure what you are getting at with regards to open-source tools to blackhole or redirect botted users. I mean, we've all got hooks in our billing system (or some other procedure) to manually disable abusive (or non-paying) customers now, right? I guess I'm not seeing how it is any harder to have a script watching snort disable the customer than it is to have freeside disable the customer when they dont pay their bill. My current setup (and I'm not saying this is the right way, or even a good way to do it) is just snort logging to a file. I then have a perl script tailing that file and 'doing stuff' - right now, 'doing stuff' consists of figuring out if it is abuse from one of my customers (in which case it puts it in the log for me) or to one of my customers (in which case, it puts it in a log for that customer. I figure it doesn't cost me any extra, so I might as well let customers see incoming attacks.) If I sat down at it long enough to say 'alert X (or alert X, y times in Z seconds) means the customer is definately botted (or abusive)' setting the perl script to run a script that uses ssh to connect to my router and blackhole the customer (or or to connect to my freeside system and suspend the account) is if not trivial, at least fairly easy.It's certainly something I could give to the junior guy on my team, and while I'd want to check his work and test carefully before going live, I'm confident he could implement. If you really need something pre-built, check out: http://www.snortsam.net/ (I haven't used it, but I don't think it's the only tool of its kind.) the hard parts (as I see them) are going to be 1. identifying the snort attacks that mean a box should be shut down. I mean, I don't want to shut you down for a simple port-scan. Maybe you are checking one of your own networks? things like that are probably more of a 'warning' for the customers I target. This is probably easier on a network targeting 'normal' customers, 'cause you can prohibit many of those things in your AUP. Also, at this point, it's pretty important that you don't have a noticable number of false positives. you probably want to run your thing in notify-only mode for a while until you are comfortable. 2. making sure that your system doesn't turn in to an easy way to DOS another server on the same network. BCP38, if implemented tightly enough (something I'm doing quite well on IPv4, but not on IPv6) can largely fix this problem, and as you are watching for abuse behind your own router, is a realistic solution.But it still takes some effort.
Re: updating checking DNS zone files
[EMAIL PROTECTED] writes: Apart from using Bernstein's tinydns, anyone have any scripts for looking for problems in zone files or for incrementing the serial number reliably? If you are using BIND, your problem is solved by DDNS and nsupdate. this has the added advantage of making it significantly more difficult for the new dns guy (or a buggy script) to take out your nameserver.
Re: amazonaws.com?
Peter Beckman [EMAIL PROTECTED] writes: If you are taking card-not-present credit card transactions over the ...snip hard to charge fradulent customers and also verifying customer identity annoys the customer... points- The goal here is to give abuse a negative expected return. One way to do this is to charge (and collect) a fee that is greater than what the spammer can earn between when they sign up and when you shut then down. There are two ways to do this - 1. raise (and collect) the abuse fee, or 2. lower the amount they can earn before you shut them down. I am suggesting that we put some effort into 2- If we can reduce the amount of time between when a spammer signs up and when they are shut down, we raise the spammer's costs. I think there is low-hanging fruit in this area. I believe that the 'strongly authenticate customer, then take legal action' model is dictated by the fact that most abuse incidents are not actually reported to your abuse desk- some abusive customers can go days or weeks before you receive a complaint. to give abuse a negative expected return, then, you need to make the consequence expensive. (to say nothing of covering the costs of trying to get good logs/evidence out of those who are complaining, or trying to figure out if your customer is a spammer or if your customer was owned by a spammer, and the costs of collecting the fee.) I wanted to point out another option providers now have. IDS technology has matured. Snort is free and pretty standard. Personally, I find monitoring incoming traffic to be... of limited utility. However, I believe snort is an excellent tool for lowering the cost of running an abuse desk, if you run it on the outgoing traffic. Snort is pretty good about alerting you to outgoing abuse before people complain. Heck, if you trust it, you can have it automatically shut down the abusive customers.
Re: amazonaws.com?
Peter Beckman [EMAIL PROTECTED] writes: ...snip use snort suggestion This is what I think we should ALL be doing -- monitoring our own network to make sure we aren't the source, via customers, of the spam or DOS attacks. All outbound email from your own network should be scanned by some sort of best-practice system before delivery to prevent or limit spam from originating on your network. IMO. But let's be realistic -- the reality is that not everyone does, due to financial or resource or management constraints I believe that in the case of a VPS provider like ec2, monitoring outgoing traffic with an IDS is cheaper than not monitoring it. Abuse reports are expensive to process. You need people with both social and technical skills on your end, people with social and technical skills who are willing to do what amounts to technical support. Often the abuse reports are vague, requiring back-and-fourth. Even if your IDS only catches a small percentage of the abuse-generating complaints (and I bet the IDS can get a large percentage of the complaint-generating abuse- it takes a lot of abuse to generate a complaint) you are saving a lot of money on abuse desk services. Heck, I bet just the ability to search IDS logs after a abuse report would pay for the IDS.
Re: [NANOG] Charter Communications going to sniff traffic foradvertising?
Christopher Morrow [EMAIL PROTECTED] writes: Oh, how do you know you can trust the VPN folks anymore than the cable-modem folks though? eventually the same cost issues are going to arise for the VPN folks as did for cable-modem/dsl folks (downward pressure on pricing and infra/opex/capex costs going up/not-decreasing). Unlike running fiber to your door, renting a VPS and setting up a vpn server is quite inexpensive to do yourself. ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
