RE: Reliable Cloud host ?
-Original Message- From: Tony Patti [mailto:t...@swalter.com] Sent: 27 February 2012 02:42 To: 'david raistrick'; 'Randy Carpenter' Cc: 'Nanog' Subject: RE: Reliable Cloud host ? -Original Message- From: david raistrick [mailto:dr...@icantclick.org] Sent: Sunday, February 26, 2012 7:19 PM To: Randy Carpenter Cc: Nanog Subject: Re: Reliable Cloud host ? On Sun, 26 Feb 2012, Randy Carpenter wrote: I don't need that kind of HA, and understand that it is not going to be available. 15 minutes of downtime is fine. 6 hours is completely unacceptable, and it false advertising to say you have a Cloud service, and then have the realization that you could have *indefinite* downtime. Um. You and I apparently work in different clouds. Since it is the weekend, I can't resist writing down a little equation: Marketing(cloud) Technology(cloud) For some values of cloud perhaps? Well indeed that is a valid point. All cloud to me means is that there is some abstracted instance of x and that it does not always relate to a particular physical device, indeed, it may well be spread around a few physical devices. I don't think there is any implied magic redundancy automatic failover move your instance to another bit of metal if something breaks in there unless that's specifically stated. caveat emptor -- Leigh __ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com __
Re: Reliable Cloud host ?
Hello, On Sun, Feb 26, 2012 at 10:56 PM, Randy Carpenter rcar...@network1.net wrote: Does anyone have any recommendation for a reliable cloud host? We require 1 or 2 very small virtual hosts to host some remote services to serve as backup to our main datacenter. One of these services is a DNS server, so it is important that it is up all the time. We have been using Rackspace Cloud Servers. We just realized that they have absolutely no redundancy or failover after experiencing a outage that lasted more than 6 hours yesterday. I am appalled that they would offer something called cloud without having any failover at all. Basic requirements: 1. Full redundancy with instant failover to other hypervisor hosts upon hardware failure (I thought this was a given!) 2. Actual support (with a phone number I can call) 3. reasonable pricing (No, $800/month is not reasonable when I need a tiny 256MB RAM Server with 1GB/mo of data transfers) Well, as everyone has been saying, unfortunately with infrastructure clouds, you have to engineer your set up to their standards to have failover. For example, Amazon (as mentioned in the thread) give a 99.95% uptime SLA *if* you set up failover yourself accros more than one Avaliability Zone within a region. Details are at http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html and http://blog.rightscale.com/2008/03/26/setting-up-a-fault-tolerant-site-using-amazons-availability-zones/ (though clearer, this one is a bit of an advert). As mentioned, with Amazon, you can use support if you pay for it, it's not included as standard. If you fancy some help though, people like RightScale sounds like exactly what you are after to make management much simpler for you http://www.rightscale.com/products/why-rightscale.php, but pricing for services like that can be a little high for small setups, though they do have a free edition that may be suitable. You can get the same kind of 99.95% SLA from other providers if you follow their deployment guidelines regarding their type of zones. Microsoft will do it for not too much )http://www.windowsazure.com/en-us/support/sla/) include online and telephone support in the price and are in the process of making Red Hat Linux available. But let's not forget simply buying the software as a service is also an option, where fail-over becomes Someone Else's Problem. For DNS, EasyDNS (https://web.easydns.com/DNS_hosting.php) are rather good and not too expensive, and you can get a 100% up-time guarantee if you want. A review of them regarding availability is at http://www.theregister.co.uk/2012/01/31/why_i_use_easydns/ Do let us know who you end up picking and how it goes. Alex
Re: Comcast / RCN Issues in Boston
There was an issue during much of the day on Friday 2/24 due to a code error on a Harvard Internet2 router at NOX. It was bounced around 4:35 pm and everything has been fine since. Andy On Fri, Feb 24, 2012 at 2:37 PM, Hashem, Sherif Rakhaa sherif_has...@hms.harvard.edu wrote: Are there any ongoing issues with Comcast and/or RCN in the Boston Metro Area? Thanks, Sherif Hashem Harvard Medical School | Network Operations 25 Shattuck Street | Gordon Hall Suite 500 | Boston, MA, 02115 d: (617)432-7534 | c: (617)999-7818 | f: (617)432-6804 -- Andy Grosser andy (at) meniscus [dot] org ---
Re: Reliable Cloud host ?
Linode.com is not cloud based but they offer IP failover between VPS instances at no additonal charge - their pricing is excellent, I have had no down time issues with them in 3+ years with 3 different customers using them and they have nice OOB and programmatic API access for controlling VPs instances as well. Max On 2/26/12, Randy Carpenter rcar...@network1.net wrote: Does anyone have any recommendation for a reliable cloud host? We require 1 or 2 very small virtual hosts to host some remote services to serve as backup to our main datacenter. One of these services is a DNS server, so it is important that it is up all the time. We have been using Rackspace Cloud Servers. We just realized that they have absolutely no redundancy or failover after experiencing a outage that lasted more than 6 hours yesterday. I am appalled that they would offer something called cloud without having any failover at all. Basic requirements: 1. Full redundancy with instant failover to other hypervisor hosts upon hardware failure (I thought this was a given!) 2. Actual support (with a phone number I can call) 3. reasonable pricing (No, $800/month is not reasonable when I need a tiny 256MB RAM Server with 1GB/mo of data transfers) thanks, -Randy
Re: Reliable Cloud host ?
On Feb 26, 2012, at 5:56 PM, Randy Carpenter wrote: We require 1 or 2 very small virtual hosts to host some remote services to serve as backup to our main datacenter. One of these services is a DNS server, so it is important that it is up all the time. We have been using Rackspace Cloud Servers. We just realized that they have absolutely no redundancy or failover after experiencing a outage that lasted more than 6 hours yesterday. I am appalled that they would offer something called cloud without having any failover at all. Pardon the weird question: Is the DNS service authoritative or recursive? If auth, you can solve this a few ways, either by giving the DNS name people point to multiple (and A) records pointing at a diverse set of instances. DNS is designed to work around a host being down. Same goes for MX and several other services. While it may make the service slightly slower, it's certainly not the end of the world. Taking a mesh of services from Rackspace, EC2, The Planet, or any other number of hosting providers will allow you to roll-your-own. The other solution is to go to a professional DNS service provider, e.g.: Dyn, Verisign, EveryDNS or NeuStar. While you can run your own infrastructure, the barrier for operating it properly is getting a bit higher each year in doing it right. I was recently shown an attack graph of a ~200Gb/s attack against a DNS server. *ouch*. Sometimes being professional is knowing when to say I can't do this justice myself, perhaps it's better/easier/cheaper to pay someone to do it right. - Jared (Disclosure: I work for one of the above named companies, but not in a capacity related to anything in this email).
FCoE/CNA Deployment w/ Nexus 5K, HP 580s, QLogic
Hi Everyone! I had several requests for more feedback on our FCoE experience, based on my comments from a thread last week, so I'm writing here with a bit more background on our project in hopes that it saves some pain for others :-). I'm with a sizable health insurance provider in the mid-west, and we've typically focused on technology vs. headcount as an overal strategy. Based on that, we upgrade much more often than some of our peers in the industry because techology is still cheaper than long-term staffing costs. Last fall, we were faced with an issue of both power and rack capacity constraints in our primary datacenter, which is just three years old now. As various ideas were on the table, which included taking out a section of IT cubes to expand the DC, the most appealing idea was to consolidate our server and network infrastructure into what was coined our High Density Row. We transitioned from Cat6500s as access to a Nexus 5K deployment, using 5Ks as both distribution and access for the new HD row. We didn't like how oversubscription is handled on 2K FEXs when it comes to 10G links, so for the situation here all 5Ks made the most sense. Our capacity needs couldn't justify 7Ks and while they would have been cool to have, we didn't want to blow money just because. Our SAN is an EMC Symmetrix with Cisco MDS switches in between it and the hosts (Fiber Channel). In the new row, we deployed all hosts with CNAs (converged net adapters), which combine both FCoE storage and network in a single 10Gb connection. Since FCoE was new to all of us, we use a phased approach that the Nexus offered where we brough straight fiber channel connections into our distibution layer 5Ks and used the Nexus' FCoE proxy functionality to convert between true FC to FCoE. From the host perpsective, it was only aware of FCoE connectivity to the Nexus. VSANs had to be created on the Nexus to map back to the FC VSANs on the MDS side, Virtual Fiber Channel (VFC) interfaces were created on the Nexus side, and a few other settings had to be configured. Overall though, the config wasn't huge, but the biggest hurdle for was that as the network guys, we had to learn the storage side to be able to properly set this up. So new terms like WWN (world wide name), floggy database, VSAN (a VLAN for storage), etc. Also, on the Nexus side, you have to enable the feature of FCOE, as Nexus OS is very modulular and leaves most options disabled during the initial setup. The painful part, which is probably what might be of most interest here, is that we hit a very strange and catrastrophic issue specific to QLogic's 8242 Copper-based (twinax) CNA adapter. As part of the burn-in testing, we were working with our server team to simulate the loss of a link/card/switch (all hosts were dual-connected with dual-CNAs to separate 5Ks). We were using the Cisco branded twinax cabling and QLogic's 8242 card (brand new HP DL580s in this case, new card, new 5K, new cabling). When a single link was dropped/diconnected PHYSICALLY (a shut/no shut is not the same here), the host's throughput on BOTH storage and network went to crap. Our baseline was showing nearly 400MB/s on storage (raw disk IO) tests prior to a link drop and 1-8 MB/s after! This siutation would not recover until you fully rebooted/power cycled the server. We had the same results accross every HP DL 580 tested, which was 5-6 of them I belive. We replaced CNAs, cables, and even moved ports across 5Ks. It didn't matter which cable, 5K, port, of card we used, all reacted the same! The hosts were all Windows 2008 Datacenter, simliar hardware, Nexus 5K on current code, twinax cabling. This situation led to a sev 2 w/ Cisco, the equivalant w/ HP, EMC, and QLogic. We used both the straight QLogic 8242 and the HP OEM'd version and the results were identical. QLogic acknowledged the issue but could not resolve it due not being able to grab a hardware level trace of the connection (required some type of test equipment that they couldn't provide and we didn't have). As part of our trail/error testing, we had our re-seller ship us the fiber versions of the same QLogic cards, becuase we eventually got down to a gut instinct of this being a copper/electrical anomoly. That instict was dead-on. Switching to the fiber versions, with fiber SFPs on the 5K side resolved the situation entirely. We are now able to drop a link with NO noticable degradation, back and forth, and eveyrthing is consistent again. We originally went the twinax route because it was signifiantly cheaper than the fiber, but in retrospect, as a whole, the danger posed was not worth it. You might ask, well... why would you intentially drop the cable? Think about a situation of doing a code upgrade on the 5K, since it's not a dual-sup box, you physcailly go through a reboot to upgrade it. That reboot right htere would have hosed our entire environment (keep in mind, the HD row's intent was to replace a signifiant
Re: Provider WAAS service for multiple MPLS VPN customers, possible?
Hi there, Just want to know if anybody out there has tried to put a pair of Cisco WAAS cards on two PEs to optimize the traffic of multiple VRFs between them ? Is that actually possible ? If it's possible, how does the WAAS module card forward the optimized traffic back to the correct VRF? Any hints or sample configurations are most appreciated. Frank.
BBC reports Kenya fiber break
Is anyone seeing this ? http://www.bbc.co.uk/news/world-africa-17179544 East Africa's high-speed internet access has been severely disrupted after a ship dropped its anchor onto fibre-optic cables off Kenya's coast. Regards Marshall
Re: Reliable Cloud host ?
On 2/27/2012 10:25 AM, Jason Gurtz wrote: [...] For DNS, EasyDNS (https://web.easydns.com/DNS_hosting.php) are rather good and not too expensive, and you can get a 100% up-time guarantee if you want. A review of them regarding availability is at http://www.theregister.co.uk/2012/01/31/why_i_use_easydns/ I have been a very satisfied EasyDNS customer for about a decade and concur with the article. Nothing is perfect, but the rapid response and support I've received have always been top-notch. I have been a satisfied DNS Made Easy customer for many years. Note: I am also an employee of DNS Made Easy. I was a customer for years before I became an employee. Do let us know who you end up picking and how it goes. Indeed. Cloud outside of references to mists and objects in the sky is a completely meaningless term for operators. In fact, it has made it harder to differentiate between services (which I'm sure is the point). As an operator (knowing how things can be subject to accelerated roll-out when $business feels they are missing out), I wonder if a lot of these cloud service bumps-in-the-road aren't just a symptom of not being fully baked in. It depends on what you mean by bumps-in-the-road... If you mean issues experienced by customers of cloud service providers, then the most common issues are a symptom of not implementing redundancy (anticipating failure) in their usage of the platform. There are a whole lot of folks who believe that they can buy an instance from Vendor =~ /.*cloud.*/ and all of their DR worries will magically be taken care of by the platform. That isn't the case. Amazon is usually pretty good at providing RFOs after issues. All of their RFOs (that I have seen) include pointers to all of the Amazon redundancy configuration documents that customers who did experience an issue regarding the RFO did not follow (which caused them to experience an outage due to a platform issue). DR in using cloud services is the same as DR has always been - look at all potential failures and then implement redundancy where the cost/benefit works out in favor of the redundancy. Document, test, rinse, lather, repeat. Rightscale and other services like it provide tools to help. -DMM
Re: BBC reports Kenya fiber break
On 27/02/2012 18:11, Marshall Eubanks wrote: Is anyone seeing this ? http://www.bbc.co.uk/news/world-africa-17179544 Along with: http://mybroadband.co.za/news/telecoms/44263-triple-whammy-hits-eassy.html The east is struggling with outages. -- Graham Beneke
Re: Reliable Cloud host ?
On Feb 27, 2012, at 10:28 AM, William Herrin wrote: On Mon, Feb 27, 2012 at 9:39 AM, Jared Mauch ja...@puck.nether.net wrote: Is the DNS service authoritative or recursive? If auth, you can solve this a few ways, either by giving the DNS name people point to multiple (and A) records pointing at a diverse set of instances. DNS is designed to work around a host being down. Same goes for MX and several other services. While it may make the service slightly slower, it's certainly not the end of the world. Hi Jared, How DNS is designed to work and how it actually works is not the same. Look up DNS Pinning for example. For most kinds of DR you need IP level failover where the IP address is rerouted to the available site. If you want a system with 0 loss and 0 delay, start building your private network. I'm never claimed your response would be perfect, but it will certainly work well enough to avoid major problems. Or you can pay someone to do it for you. I'm not sure what a DNS hosted solution costs, and I'm geeky and run my own DNS on beta/RC quality software as well ;). What I do know is that my domain hasn't disappeared from the net wholesale as the name servers are diverse-enough. Is DNS performance important? Sure. Should everyone set their TTL to 30? No. Reaching a high percentage of the internet doesn't require such a high SLA. Note, I didn't say reaching the top sites. While super-old, http://www.zooknic.com/Domains/counts.html says 111m named sites in a few gTLDs. I'm sure there are better stats, but most of them don't need the same dns infrastructure that a google, bing, Facebook, etc require. If your DNS fits on a VM in someone else's cloud, you likely won't notice the difference. A few extra NS records will likely do the right thing and go unnoticed. - Jared
Re: Reliable Cloud host ?
Pardon the weird question: Is the DNS service authoritative or recursive? If auth, you can solve this a few ways, either by giving the DNS name people point to multiple (and A) records pointing at a diverse set of instances. Authoritative. But, also not the only thing that we are running that needs some geographic and route diversity. DNS is designed to work around a host being down. Same goes for MX and several other services. While it may make the service slightly slower, it's certainly not the end of the world. Oh, how I wish this were true in practice. If I had a dollar for every time we had serious issues because one of a few authoritative DNS servers was not responding... OK, I wouldn't be rich, but this happens all the time. Caching servers out on the net get a non-answer because the server they chose to ask was down, and it caches that. They shouldn't do that, but they do, and there's nothing that can be done about it. -Randy
Re: BBC reports Kenya fiber break
On Mon, Feb 27, 2012 at 11:46 AM, Graham Beneke gra...@apolix.co.za wrote: On 27/02/2012 18:11, Marshall Eubanks wrote: Is anyone seeing this ? http://www.bbc.co.uk/news/world-africa-17179544 Along with: http://mybroadband.co.za/news/telecoms/44263-triple-whammy-hits-eassy.html The east is struggling with outages. -- Graham Beneke Most of the ISP's in Malawi have been having issues since the 17th due to a severed cable in the Red Sea. Oliver
Re: BBC reports Kenya fiber break
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 02/27/2012 08:11 AM, Marshall Eubanks wrote: Is anyone seeing this ? http://www.bbc.co.uk/news/world-africa-17179544 East Africa's high-speed internet access has been severely disrupted after a ship dropped its anchor onto fibre-optic cables off Kenya's coast. Regards Marshall - -- I don't have a direct feedback into this disruption but from what I gather they were able to (manually) re-route traffic (alternative submarine cable and /or satellite systems) whether its slow that's a different story but having performance degradation, as opposed to complete service outage is still workable, IMO. Hopefully diversity will help minimize localized damages as the global economy (communications, education, business, entertainment, banking commerce) continues to be dependent on undersea cables. Typically the GPS navigation suite has undersea cables well documented. I for one am interested to know how this was overlooked or maybe human error? regards, /virendra -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iF4EAREIAAYFAk9LyVkACgkQ3HuimOHfh+FZggD/a+LIEBXANdItl2NGbaTCRQsh +5/l0RvFRL3EMws8IsAA/jlV2gFGzCB1SM8pFAmKnK6sgS38tnxDFDj/4KqIUFky =40jD -END PGP SIGNATURE-
Re: Provider WAAS service for multiple MPLS VPN customers, possible?
I have never done it between MPLS like what you are referring to, but for the best optimization you will need edge WAE units on each end of the connection. - Original Message - From: Frank Ho completea...@gmail.com To: nanog@nanog.org Sent: Monday, February 27, 2012 11:04:35 AM Subject: Re: Provider WAAS service for multiple MPLS VPN customers, possible? Hi there, Just want to know if anybody out there has tried to put a pair of Cisco WAAS cards on two PEs to optimize the traffic of multiple VRFs between them ? Is that actually possible ? If it's possible, how does the WAAS module card forward the optimized traffic back to the correct VRF? Any hints or sample configurations are most appreciated. Frank.
Re: Reliable Cloud host ?
On Mon, Feb 27, 2012 at 7:28 AM, William Herrin b...@herrin.us wrote: On Sun, Feb 26, 2012 at 7:02 PM, Randy Carpenter rcar...@network1.net wrote: On Feb 26, 2012, at 4:56 PM, Randy Carpenter wrote: 1. Full redundancy with instant failover to other hypervisor hosts upon hardware failure (I thought this was a given!) This is actually a much harder problem to solve than it sounds, and gets progressively harder depending on what you mean by failover. At the very least, having two physical hosts capable of running your VM requires that your VM be stored on some kind of SAN (usually iSCSI based) storage system. Otherwise, two hosts have no way of accessing your VM's data if one were to die. This makes things an order of magnitude or higher more expensive. This does not have to be true at all. Even having a fully fault-tolerant SAN in addition to spare servers should not cost much more than having separate RAID arrays inside each of the server, when you are talking about 1,000s of server (which Rackspace certainly has) Randy, You're kidding, right? SAN storage costs the better part of an order of magnitude more than server storage, which itself is several times more expensive than workstation storage. That's before you duplicate the SAN and set up the replication process so that cabinet and room level failures don't take you out. This is clearly becoming a not-NANOG-ish thread, however... Failing to have central shared storage (iSCSI, NAS, SAN, whatever you prefer) fails the smell test on a local enterprise-grade virtualization cluster, much less a shared cloud service. Some people have done tricks with distributing the data using one of the research-ish shared filesystems, rather than separate shared storage. That can be made to work if the host OS model and its available shared filesystems work for you. Doesn't work for Vmware Vcenter / Vmotion-ish stuff as far as I know. There are plenty of people doing non-enterprise-grade virtualization. There's no mandate that you have the ability to migrate a virtual to another node in realtime or restart it immediately on another node if the first node dies suddenly. But anyone saying we have a cloud and not providing that type of service, is in marketing not engineering. From a systems architecture point of view, you can't do that. -- -george william herbert george.herb...@gmail.com
do not filter your customers - part2
If we are gonna start to get somewhere with this issue, how about to make sure the routing/prefix databases is correct first? Please see: https://www.fredan.se/temp/prefixes.tar In that file you will find 'not_allowed_to_announce6' which contains about 2307 prefixes of ipv6 which is not in any routing/prefix databases OR the prefix that was submitted to it was wrong (probably the syntax of that prefix). Which bring us to the next question. Why on earth is it possible to submit a faulty prefix into a database today? Why is there (basically) no verification at all? Please take a look at 'databases_to_prefixes.sh' see what's going on (ok, some of the databases is probably for internal use only and we need to filter that - but it is so much more that needs to be filtered). Also in that file you will find 'prefixes4' and 'prefixes6' which contains all the prefixes after all the checking has been made (One prefix per line). These two files could be really useful for everybody in this community if someone (like the RIR:s) made those available to all of us, so we don't have to download all the databases, just the prefixes (And I know that AS52011 is announce to two prefixes which is not in the databases. Thank you very much). -- //fredan
Re: Reliable Cloud host ?
On Mon, 27 Feb 2012 14:02:04 EST, William Herrin said: The net result is that when you switch the IP address of your server, a percentage of your users (declining over time) will be unable to access it for hours, days, weeks or even years regardless of the DNS TTL setting. Amen brother. So just for grins, after seeing William's I set up a listener on an address that had an NTP server on it many moons ago. As in the machine was shut down around 2002/06/30 22:49 and we didn't re-assign the IP address ever since *because* it kept getting hit with NTP packets.. Yes, a decade ago. In the first 15 minutes, 234 different IP's have tried to NTP to that address. And the winner for most confused host, which in addition to trying to NTP also did this: 14:23:24.518136 IP 74.254.73.90.68 128.173.14.71.123: BOOTP/DHCP, unknown (0xdb), length 48 14:23:57.395525 IP 74.254.73.90.53 128.173.14.71.123: 56064 [b23=0x6ee] [3494a] [0q] [307au] (48) 14:24:28.536351 IP 74.254.73.90.68 128.173.14.71.123: BOOTP/DHCP, unknown (0xdb), length 48 14:24:53.382719 IP 74.254.73.90.500 128.173.14.71.123: isakmp: 14:25:01.391268 IP 74.254.73.90.53 128.173.14.71.123: 56064 [b23=0x6ee] [3494a] [0q] [307au] (48) 14:25:32.522313 IP 74.254.73.90.68 128.173.14.71.123: BOOTP/DHCP, unknown (0xdb), length 48 14:26:05.399885 IP 74.254.73.90.53 128.173.14.71.123: 56064 [b23=0x6ee] [3494a] [0q] [307au] (48) 14:26:36.529713 IP 74.254.73.90.68 128.173.14.71.123: BOOTP/DHCP, unknown (0xdb), length 48 14:27:09.405922 IP 74.254.73.90.53 128.173.14.71.123: 56064 [b23=0x6ee] [3494a] [0q] [307au] (48) 14:27:40.528381 IP 74.254.73.90.68 128.173.14.71.123: BOOTP/DHCP, unknown (0xdb), length 48 14:28:13.393794 IP 74.254.73.90.53 128.173.14.71.123: 56064 [b23=0x6ee] [3494a] [0q] [307au] (48) 14:28:20.971269 IP 74.254.73.90.69 128.173.14.71.123: 48 tftp-#6914 14:28:37.907704 IP 74.254.73.90.161 128.173.14.71.123: [id?P/x/27] 14:28:44.525585 IP 74.254.73.90.68 128.173.14.71.123: BOOTP/DHCP, unknown (0xdb), length 48 14:29:17.399784 IP 74.254.73.90.53 128.173.14.71.123: 56064 [b23=0x6ee] [3494a] [0q] [307au] (48) 14:29:48.531804 IP 74.254.73.90.68 128.173.14.71.123: BOOTP/DHCP, unknown (0xdb), length 48 14:30:21.398360 IP 74.254.73.90.53 128.173.14.71.123: 56064 [b23=0x6ee] [3494a] [0q] [307au] (48) 14:30:52.530148 IP 74.254.73.90.68 128.173.14.71.123: BOOTP/DHCP, unknown (0xdb), length 48 14:31:25.403931 IP 74.254.73.90.53 128.173.14.71.123: 56064 [b23=0x6ee] [3494a] [0q] [307au] (48) 14:31:56.536594 IP 74.254.73.90.68 128.173.14.71.123: BOOTP/DHCP, unknown (0xdb), length 48 14:32:29.404457 IP 74.254.73.90.53 128.173.14.71.123: 56064 [b23=0x6ee] [3494a] [0q] [307au] (48) 14:33:00.534956 IP 74.254.73.90.68 128.173.14.71.123: BOOTP/DHCP, unknown (0xdb), length 48 14:33:33.402336 IP 74.254.73.90.53 128.173.14.71.123: 56064 [b23=0x6ee] [3494a] [0q] [307au] (48) Somewhere in BellSouth territory, a machine desperately needs to be whacked upside the head. pgpeSvLCciXmj.pgp Description: PGP signature
RE: Programmers with network engineering skills
Generalists are hard to come by these days. They are people who learn less and less about more and more till they know nothing about everything. People today are specializing in the left and right halves of the bytes They learn more and more about less and less till they know everything about nothing. And BTW, they are worthless unless you have five of them working on a problem because none of them know enough to fix it. Worse, you can replace the word five with fifty and it may be still true. I know of three of these, all gainfully employed at this time and could each find at least a couple jobs if they wanted. I am one, my son is two and a guy we worked with is the third. At one time (40 years ago) the mantra in IS was train for expertise, now it is hire for it. Somewhere there has to be a happy medium. I suggest this, find a good coder, not a mediocre who writes shit code but a good one who can think and learn and when you talk about branching out with his skill set he or she lights up. His first thing on site is take the A+ networking course. No, I do not sell the courses. But I have seen this kind of approach work when nothing else was. Ralph Brandt Communications Engineer HP Enterprise Services Telephone +1 717.506.0802 FAX +1 717.506.4358 Email ralph.bra...@pateam.com 5095 Ritter Rd Mechanicsburg PA 17055 -Original Message- From: A. Pishdadi [mailto:apishd...@gmail.com] Sent: Sunday, February 26, 2012 8:27 PM To: NANOG Subject: Programmers with network engineering skills Hello All, i have been looking for quite some time now a descent coder (c,php) who has a descent amount of system admin / netadmin experience. Doesn't necessarily need to be an expert at network engineering but being acclimated in understanding the basic fundamentals of networking. Understanding basic routing concepts, how to diagnose using tcpdump / pcap, understanding subnetting and how bgp works (not necessarily setting up bgp). I've posted job listings on the likes of dice and monster and have not found any good canidates, most of them ASP / Java guys. If anyone can point me to a site they might recommend for job postings or know of any consulting firms that might provide these services that would be greatly appreciated.
Re: Programmers with network engineering skills
I think you're more likely to find a network engineer with (possibly limited) programming skills. That's certainly where I would categorize myself. Owen On Feb 27, 2012, at 12:02 PM, Brandt, Ralph wrote: Generalists are hard to come by these days. They are people who learn less and less about more and more till they know nothing about everything. People today are specializing in the left and right halves of the bytes They learn more and more about less and less till they know everything about nothing. And BTW, they are worthless unless you have five of them working on a problem because none of them know enough to fix it. Worse, you can replace the word five with fifty and it may be still true. I know of three of these, all gainfully employed at this time and could each find at least a couple jobs if they wanted. I am one, my son is two and a guy we worked with is the third. At one time (40 years ago) the mantra in IS was train for expertise, now it is hire for it. Somewhere there has to be a happy medium. I suggest this, find a good coder, not a mediocre who writes shit code but a good one who can think and learn and when you talk about branching out with his skill set he or she lights up. His first thing on site is take the A+ networking course. No, I do not sell the courses. But I have seen this kind of approach work when nothing else was. Ralph Brandt Communications Engineer HP Enterprise Services Telephone +1 717.506.0802 FAX +1 717.506.4358 Email ralph.bra...@pateam.com 5095 Ritter Rd Mechanicsburg PA 17055 -Original Message- From: A. Pishdadi [mailto:apishd...@gmail.com] Sent: Sunday, February 26, 2012 8:27 PM To: NANOG Subject: Programmers with network engineering skills Hello All, i have been looking for quite some time now a descent coder (c,php) who has a descent amount of system admin / netadmin experience. Doesn't necessarily need to be an expert at network engineering but being acclimated in understanding the basic fundamentals of networking. Understanding basic routing concepts, how to diagnose using tcpdump / pcap, understanding subnetting and how bgp works (not necessarily setting up bgp). I've posted job listings on the likes of dice and monster and have not found any good canidates, most of them ASP / Java guys. If anyone can point me to a site they might recommend for job postings or know of any consulting firms that might provide these services that would be greatly appreciated.
Re: Programmers with network engineering skills
On Mon, 27 Feb 2012, Owen DeLong wrote: I think you're more likely to find a network engineer with (possibly limited) programming skills. While I'll agree about the more likely, if I needed a coder who had a firm grasp of networking I'd rather teach a good coder networking, than try to teach the art and magic of good development to a network guy. I think it really comes down to which you need: a hardcore network engineer/architect who can hack up code, or a hardcore developer who has or can obtain enough of a grasp of networking fundementals and specifics to build you the software you need him to develop. The ones who already know both ends extremely well are going to be -very- hard to find, but finding one who can learn enough of the other to accomplish what you need shouldn't be hard at all. oh wait, that's an echo I hear isn't it. ...d (who is not exactly the former though I've played one for TV, and not at all the later) -- david raistrickhttp://www.netmeister.org/news/learn2quote.html dr...@icantclick.org http://www.expita.com/nomime.html
dns and software, was Re: Reliable Cloud host ?
On Mon, 27 Feb 2012, William Herrin wrote: In some cases this is because of carelessness: The application does a gethostbyname once when it starts, grabs the first IP address in the list and retains it indefinitely. The gethostbyname function doesn't even pass the TTL to the application. Ntpd is/used to be one of the notable offenders, continuing to poll the dead address for years after the server moved. While yes it often is carelessness - it's been reported by hardcore development sorts that I trust that there is no standardized API to obtain the TTL... What needs to get fixed is get[hostbyname,addrinfo,etc] so programmers have better tools. -- david raistrickhttp://www.netmeister.org/news/learn2quote.html dr...@icantclick.org http://www.expita.com/nomime.html
Re: Reliable Cloud host ?
On Mon, Feb 27, 2012 at 11:19:27AM -0800, George Herbert wrote: On Mon, Feb 27, 2012 at 7:28 AM, William Herrin b...@herrin.us wrote: On Sun, Feb 26, 2012 at 7:02 PM, Randy Carpenter rcar...@network1.net wrote: On Feb 26, 2012, at 4:56 PM, Randy Carpenter wrote: 1. Full redundancy with instant failover to other hypervisor hosts upon hardware failure (I thought this was a given!) This is actually a much harder problem to solve than it sounds, and gets progressively harder depending on what you mean by failover. At the very least, having two physical hosts capable of running your VM requires that your VM be stored on some kind of SAN (usually iSCSI based) storage system. Otherwise, two hosts have no way of accessing your VM's data if one were to die. This makes things an order of magnitude or higher more expensive. This does not have to be true at all. Even having a fully fault-tolerant SAN in addition to spare servers should not cost much more than having separate RAID arrays inside each of the server, when you are talking about 1,000s of server (which Rackspace certainly has) Randy, You're kidding, right? SAN storage costs the better part of an order of magnitude more than server storage, which itself is several times more expensive than workstation storage. That's before you duplicate the SAN and set up the replication process so that cabinet and room level failures don't take you out. This is clearly becoming a not-NANOG-ish thread, however... Failing to have central shared storage (iSCSI, NAS, SAN, whatever you prefer) fails the smell test on a local enterprise-grade virtualization cluster, much less a shared cloud service. Some people have done tricks with distributing the data using one of the research-ish shared filesystems, rather than separate shared storage. That can be made to work if the host OS model and its available shared filesystems work for you. Doesn't work for Vmware Vcenter / Vmotion-ish stuff as far as I know. There are plenty of people doing non-enterprise-grade virtualization. There's no mandate that you have the ability to migrate a virtual to another node in realtime or restart it immediately on another node if the first node dies suddenly. But anyone saying we have a cloud and not providing that type of service, is in marketing not engineering. From a systems architecture point of view, you can't do that. Cloud is utterly meaningless drivel. Your idea of cloud is different from mine, which is different from my co-workers, bosses, people in marketing etc. etc. It's a vague useless term that could mean everything from a bog standard mail server through to full on 'deploy your app' things like Heroku. It would be more accurate to focus on IaaS, PaaS, SaaS et al For what little it's probably worth mentioning, Amazon provides a shared storage platform in the form of EBS, Elastic Block Storage, which you can choose to use as your root device on your server if you so wish (wouldn't advise you do, latency is unpredictable), or you can have it mounted wherever is relevant for your data (the most common route). That's their non-physical server dependent storage provision. If you pay extra it'll replicate, or even replicate between availability zones. You can also choose to have Amazon monitor and ensure sufficient numbers of your server are running through autoscale. Paul
RE: Programmers with network engineering skills
Generalists are hard to come by these days. They are people who learn less and less about more and more till they know nothing about everything. People today are specializing in the left and right halves of the bytes They learn more and more about less and less till they know everything about nothing. And BTW, they are worthless unless you have five of them working on a problem because none of them know enough to fix it. Worse, you can replace the word five with fifty and it may be still true. I know of three of these, all gainfully employed at this time and could each find at least a couple jobs if they wanted. I am one, my son is two and a guy we worked with is the third. At one time (40 years ago) the mantra in IS was train for expertise, now it is hire for it. Somewhere there has to be a happy medium. I suggest this, find a good coder, not a mediocre who writes bad code but a good one who can think and learn and when you talk about branching out with his skill set he or she lights up. His first thing on site is take the A+ networking course. No, I do not sell the courses. But I have seen this kind of approach work when nothing else was. Ralph Brandt Communications Engineer HP Enterprise Services Telephone +1 717.506.0802 FAX +1 717.506.4358 Email ralph.bra...@pateam.com 5095 Ritter Rd Mechanicsburg PA 17055 -Original Message- From: A. Pishdadi [mailto:apishd...@gmail.com] Sent: Sunday, February 26, 2012 8:27 PM To: NANOG Subject: Programmers with network engineering skills Hello All, i have been looking for quite some time now a descent coder (c,php) who has a descent amount of system admin / netadmin experience. Doesn't necessarily need to be an expert at network engineering but being acclimated in understanding the basic fundamentals of networking. Understanding basic routing concepts, how to diagnose using tcpdump / pcap, understanding subnetting and how bgp works (not necessarily setting up bgp). I've posted job listings on the likes of dice and monster and have not found any good canidates, most of them ASP / Java guys. If anyone can point me to a site they might recommend for job postings or know of any consulting firms that might provide these services that would be greatly appreciated.
Re: Programmers with network engineering skills
On Feb 27, 2012, at 12:31 PM, david raistrick wrote: On Mon, 27 Feb 2012, Owen DeLong wrote: I think you're more likely to find a network engineer with (possibly limited) programming skills. While I'll agree about the more likely, if I needed a coder who had a firm grasp of networking I'd rather teach a good coder networking, than try to teach the art and magic of good development to a network guy. Well, I won't call myself a hard-core coder, but, I think I have a reasonable grasp on the art and magic of good development. What I mostly lack is speed and efficiency in the language of choice for whatever project. I can write good code, it just takes me longer than it would take a hard-core coder. OTOH, having done both, I would say that I think you are not necessarily correct about which direction of teaching is harder. Yes, if you start with a network engineer that knows nothing about writing code or doesn't understand the principles of good coding, you're probably right. However, starting with a network engineer that can write decent code slowly, I think you will get a better result in most cases than if you try to teach network engineering to a hard-core coder that has only a minimal understanding of networking. I think it really comes down to which you need: a hardcore network engineer/architect who can hack up code, or a hardcore developer who has or can obtain enough of a grasp of networking fundementals and specifics to build you the software you need him to develop. I'm guessing that someone who needed a hard-core developer that could grasp fundamentals would have grabbed an existing coder and handed him a copy of Comer. The fact that this person posted to NANOG instead implies to me that he needs someone that has a better grasp than just the fundamentals. Of course I am speculating about that and I could be wrong. The ones who already know both ends extremely well are going to be -very- hard to find, but finding one who can learn enough of the other to accomplish what you need shouldn't be hard at all. Depends on what you need. However, I think it's faster to go from limited coding skills with a good basis in the fundamentals to usable development than to go from limited networking skills to a firm grasp on how networks behave in the real world. To the best of my knowledge, nothing but experience will teach you the latter. Even with 20+ years experience networks do still occasionally manage to surprise me. ...d (who is not exactly the former though I've played one for TV, and not at all the later) I am admittedly lost given the three choices as to which constitutes former or latter at this point. 1. Strong coder with limited networking 2. Strong networker with limited coding 3. Strong in both Owen Who is a strong network engineer Who has been a professional software engineer (though many years ago and my skills are rusty and out of date)
Re: Programmers with network engineering skills
- Original Message - From: Owen DeLong o...@delong.com I think you're more likely to find a network engineer with (possibly limited) programming skills. That's certainly where I would categorize myself. And you're the first I've seen suggest, or even imply, that going that direction instead might be more fruitful; seemed to me that the skills necessary to make a decent network engineer would support learning programming better than the other way round -- though in fact I personally did it the other way. Cheers, -- jra -- Jay R. Ashworth Baylink j...@baylink.com Designer The Things I Think RFC 2100 Ashworth Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
Re: Programmers with network engineering skills
On 2/27/2012 2:23 PM, Jay Ashworth wrote: - Original Message - From: Owen DeLong o...@delong.com I think you're more likely to find a network engineer with (possibly limited) programming skills. That's certainly where I would categorize myself. And you're the first I've seen suggest, or even imply, that going that direction instead might be more fruitful; seemed to me that the skills necessary to make a decent network engineer would support learning programming better than the other way round -- though in fact I personally did it the other way. I think it depends on what level of coding you're talking about. If you want someone that can whip up a few scripts to easily manage routine tasks, then sure, network guy - coder is usually a safe and easy path. OTOH, if you're talking professional application developer working on a project with more than one moving part, and/or more than one person on the team, you really need someone who thinks like a developer, and can be trained to understand network concepts. and yes, the latter is the path that I've taken, so I have a built-in bias. Doug -- It's always a long day; 86400 doesn't fit into a short. Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/
Re: Programmers with network engineering skills
Le lundi 27 février 2012 à 14:14 -0800, Owen DeLong a écrit : On Feb 27, 2012, at 12:31 PM, david raistrick wrote: On Mon, 27 Feb 2012, Owen DeLong wrote: I think you're more likely to find a network engineer with (possibly limited) programming skills. While I'll agree about the more likely, if I needed a coder who had a firm grasp of networking I'd rather teach a good coder networking, than try to teach the art and magic of good development to a network guy. Well, I won't call myself a hard-core coder, but, I think I have a reasonable grasp on the art and magic of good development. What I mostly lack is speed and efficiency in the language of choice for whatever project. I can write good code, it just takes me longer than it would take a hard-core coder. OTOH, having done both, I would say that I think you are not necessarily correct about which direction of teaching is harder. Yes, if you start with a network engineer that knows nothing about writing code or doesn't understand the principles of good coding, you're probably right. However, starting with a network engineer that can write decent code slowly, I think you will get a better result in most cases than if you try to teach network engineering to a hard-core coder that has only a minimal understanding of networking. I think it really comes down to which you need: a hardcore network engineer/architect who can hack up code, or a hardcore developer who has or can obtain enough of a grasp of networking fundementals and specifics to build you the software you need him to develop. I'm guessing that someone who needed a hard-core developer that could grasp fundamentals would have grabbed an existing coder and handed him a copy of Comer. The fact that this person posted to NANOG instead implies to me that he needs someone that has a better grasp than just the fundamentals. Of course I am speculating about that and I could be wrong. The ones who already know both ends extremely well are going to be -very- hard to find, but finding one who can learn enough of the other to accomplish what you need shouldn't be hard at all. Depends on what you need. However, I think it's faster to go from limited coding skills with a good basis in the fundamentals to usable development than to go from limited networking skills to a firm grasp on how networks behave in the real world. To the best of my knowledge, nothing but experience will teach you the latter. Even with 20+ years experience networks do still occasionally manage to surprise me. ...d (who is not exactly the former though I've played one for TV, and not at all the later) I am admittedly lost given the three choices as to which constitutes former or latter at this point. 1.Strong coder with limited networking 2.Strong networker with limited coding 3.Strong in both It's all about KISS, to appreciate sound abstraction, in other words. Cheers, mh Owen Who is a strong network engineer Who has been a professional software engineer (though many years ago and my skills are rusty and out of date)
Re: Programmers with network engineering skills
On 2/27/2012 2:31 PM, Doug Barton wrote: then sure, network guy - coder is usually a safe and easy path. Sorry, looking at this again it reads a lot more derogatory on paper than I meant it to. There is a lot of value in being able to automate repetitive tasks ... my point was simply that doing that is a different development model than working on a larger scale project; where scope, structure, etc. come into play. Doug (who either needs more caffeine, or less ...) -- It's always a long day; 86400 doesn't fit into a short. Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/
Re: Programmers with network engineering skills
On Feb 26, 2012, at 8:27 PM, A. Pishdadi apishd...@gmail.com wrote: Hello All, i have been looking for quite some time now a descent coder (c,php) who has a descent amount of system admin / netadmin experience. Doesn't necessarily need to be an expert at network engineering but being acclimated in understanding the basic fundamentals of networking. Understanding basic routing concepts, how to diagnose using tcpdump / pcap, understanding subnetting and how bgp works (not necessarily setting up bgp). I've posted job listings on the likes of dice and monster and have not found any good canidates, most of them ASP / Java guys. If anyone can point me to a site they might recommend for job postings or know of any consulting firms that might provide these services that would be greatly appreciated. Good Luck guys like these are being scooped up by large financial firms and hedgefunds and they don't come cheap ~$250k easy!
Re: Reliable Cloud host ?
On Mon, Feb 27, 2012 at 2:19 PM, George Herbert george.herb...@gmail.com wrote: Failing to have central shared storage (iSCSI, NAS, SAN, whatever you prefer) fails the smell test on a local enterprise-grade virtualization cluster, much less a shared cloud service. Hi George, Why would you imagine that a $30/month virtual private server is built on an enterprise-grade virtualization cluster? You know what it costs to builds fibre channel SANs and blade servers and DR. In what universe does $30/mo per customer recover that cost during the useful life of the equipment? A VPS is 2012's version of 2002's web server + CGI and a unix shell. Quite useful but don't expect magic from it. Regards, Bill Herrin -- William D. Herrin her...@dirtside.com b...@herrin.us 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004
Re: dns and software, was Re: Reliable Cloud host ?
On Feb 27, 2012, at 3:50 PM, William Herrin wrote: On Mon, Feb 27, 2012 at 3:43 PM, david raistrick dr...@icantclick.org wrote: On Mon, 27 Feb 2012, William Herrin wrote: In some cases this is because of carelessness: The application does a gethostbyname once when it starts, grabs the first IP address in the list and retains it indefinitely. The gethostbyname function doesn't even pass the TTL to the application. Ntpd is/used to be one of the notable offenders, continuing to poll the dead address for years after the server moved. While yes it often is carelessness - it's been reported by hardcore development sorts that I trust that there is no standardized API to obtain the TTL... What needs to get fixed is get[hostbyname,addrinfo,etc] so programmers have better tools. Meh. What should be fixed is that connect() should receive a name instead of an IP address. Having an application deal directly with the IP address should be the exception rather than the rule. Then, deal with the TTL issues once in the standard libraries instead of repeatedly in every single application. In theory, that'd even make the app code protocol agnostic so that it doesn't have to be rewritten yet again for IPv12. While I agree with the principle of what you are trying to say, I would argue that it should be dealt with in getnameinfo() / getaddrinfo() and not connect(). It is perfectly reasonable for connect() to deal with an address structure. If people are not using getnameinfo()/getaddrinfo() from the standard libraries, then, I don't see any reason to believe that they would use connect() from the standard libraries if it incorporated their functionality. Owen
Re: Programmers with network engineering skills
On Mon, Feb 27, 2012 at 3:22 PM, Owen DeLong o...@delong.com wrote: On Feb 27, 2012, at 12:02 PM, Brandt, Ralph wrote: Generalists are hard to come by these days. I think you're more likely to find a network engineer with (possibly limited) programming skills. I wish. For the past three months I've been trying to find a network engineer with a deep TCP/IP protocol understanding, network security expertise, some Linux experience, minor programming skill with sockets and a TS/SCI clearance. The clearance is killing me. The two generalists didn't have a clearance and the cleared applicants are programmers or admins but never both. On Mon, Feb 27, 2012 at 6:12 PM, Rodrick Brown rodrick.br...@gmail.com wrote: Good Luck guys like these are being scooped up by large financial firms and hedgefunds and they don't come cheap ~$250k easy! Not all of them. I've been approached a few times but there is something sleazy about helping a bunch of tycoons do millisecond timing attacks against the market. The money doesn't magically appear. Every dollar they squeeze out that way is stolen from some grandmother who has held the stock for 20 years. Regards, Bill Herrin -- William D. Herrin her...@dirtside.com b...@herrin.us 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004
Re: dns and software, was Re: Reliable Cloud host ?
On Mon, Feb 27, 2012 at 7:07 PM, Owen DeLong o...@delong.com wrote: On Feb 27, 2012, at 3:50 PM, William Herrin wrote: Meh. What should be fixed is that connect() should receive a name instead of an IP address. Having an application deal directly with the IP address should be the exception rather than the rule. Then, deal with the TTL issues once in the standard libraries instead of repeatedly in every single application. In theory, that'd even make the app code protocol agnostic so that it doesn't have to be rewritten yet again for IPv12. While I agree with the principle of what you are trying to say, I would argue that it should be dealt with in getnameinfo() / getaddrinfo() and not connect(). It is perfectly reasonable for connect() to deal with an address structure. Yes, well, that's why we're still using a layer 4 protocol (TCP) that can't dynamically rebind to the protocol level below (IP). God help us when folks start overriding the ethernet MAC address to force machines to keep the same IPv6 address that's been hardcoded somewhere or is otherwise too much trouble to change. Regards, Bill Herrin -- William D. Herrin her...@dirtside.com b...@herrin.us 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004
Re: Reliable Cloud host ?
On Mon, Feb 27, 2012 at 3:45 PM, William Herrin b...@herrin.us wrote: On Mon, Feb 27, 2012 at 2:19 PM, George Herbert george.herb...@gmail.com wrote: Failing to have central shared storage (iSCSI, NAS, SAN, whatever you prefer) fails the smell test on a local enterprise-grade virtualization cluster, much less a shared cloud service. Hi George, Why would you imagine that a $30/month virtual private server is built on an enterprise-grade virtualization cluster? You know what it costs to builds fibre channel SANs and blade servers and DR. In what universe does $30/mo per customer recover that cost during the useful life of the equipment? As I stated, one can either do it with SANs or with alternate storage. Amazon hit those price points with a custom distributed filesystem that's more akin to the research distributed filesystems than anything else. It's using node storage, but not single-node locked; if the physical dies it should not lose the data. Amazon wrote that filesystem, but one could approach the problem with an OTS research / labs distributed FS using blade or 1U internal disks and duplicate what they did. In the enterprise space, there's a lot more variety and flexibility too. I bought a 100 TB (raw) NAS / storage unit for well under $30k not that long ago. Even accounting for RAID6 and duplicate units on the network (network RAID1 across two units doing RAID6 internally), that would cover something like 250 standard AWS instances, or about $100/unit for the storage. At typical useful amortization (24 to 48 months) that's about $2 to $4/month/server. That's not an EMC, a Hitachi, a BlueArc, a NetApp, a Compellant, even a Nexsan. But one can walk up the curve relatively smoothly from that low end point to the bestest brightest highest-tier stuff depending on one's customers' needs. A VPS is 2012's version of 2002's web server + CGI and a unix shell. Quite useful but don't expect magic from it. There are plenty of services that know what they should do and do it reasonably well. AWS, above. There are also a lot of services that (without naming names) are floating out there in sketchy-land. One should both know better and expect better. It's possible to design reliable services - with geographical redundancy and the like between service providers, in case one corks - out of unreliable services. One should do some of that anyways, with clouds. But the quality of the underlying service varies a lot. If you're paying AWS prices for non-replicated storage, think carefully about what you're doing. If you're paying half of what AWS costs, and duplicating locations to handle outages, then you're probably ok. If you're paying more and getting better service, ok. -- -george william herbert george.herb...@gmail.com
Re: Programmers with network engineering skills
On Feb 27, 2012, at 7:53 PM, William Herrin b...@herrin.us wrote: On Mon, Feb 27, 2012 at 3:22 PM, Owen DeLong o...@delong.com wrote: On Feb 27, 2012, at 12:02 PM, Brandt, Ralph wrote: Generalists are hard to come by these days. I think you're more likely to find a network engineer with (possibly limited) programming skills. I wish. For the past three months I've been trying to find a network engineer with a deep TCP/IP protocol understanding, network security expertise, some Linux experience, minor programming skill with sockets and a TS/SCI clearance. The clearance is killing me. The two generalists didn't have a clearance and the cleared applicants are programmers or admins but never both. On Mon, Feb 27, 2012 at 6:12 PM, Rodrick Brown rodrick.br...@gmail.com wrote: Good Luck guys like these are being scooped up by large financial firms and hedgefunds and they don't come cheap ~$250k easy! Not all of them. I've been approached a few times but there is something sleazy about helping a bunch of tycoons do millisecond timing attacks against the market. The money doesn't magically appear. Every dollar they squeeze out that way is stolen from some grandmother who has held the stock for 20 years. Try explaining the number of ex-Bell Lab RD folks working on trading desks these days. A major financial firm I worked for in the past directly targeted candidates from the telecom industry. In recent news a russian programmer who allegedly stole Goldman Sachs proprietary code was making $400k/year and he's probably still on the market looking for work :-) Regards, Bill Herrin -- William D. Herrin her...@dirtside.com b...@herrin.us 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004
Re: Programmers with network engineering skills
On 2/27/2012 7:53 PM, William Herrin wrote: I think you're more likely to find a network engineer with (possibly limited) programming skills. I wish. For the past three months I've been trying to find a network engineer with a deep TCP/IP protocol understanding, network security expertise, some Linux experience, minor programming skill with sockets and a TS/SCI clearance. Is clearance the problem, or the ability to obtain clearance due to something in their background? If your work requires it, you should have some recourse for applicants to obtain the required clearance, no? /Jason
Re: dns and software, was Re: Reliable Cloud host ?
On Mon, Feb 27, 2012 at 4:59 PM, William Herrin b...@herrin.us wrote: Yes, well, that's why we're still using a layer 4 protocol (TCP) that can't dynamically rebind to the protocol level below (IP). This is somewhat irritating, but on the scale of 0 (all is well) to 10 (you want me to do WHAT with DHCPv6???) this is about a 2. The application can re-connect from the TCP layer if something wiggy happens to the layer below. This is an application layer solution, is well established, and works fine. One just has to notice something's amiss and retry connection rather than abort the application. God help us when folks start overriding the ethernet MAC address to force machines to keep the same IPv6 address that's been hardcoded somewhere or is otherwise too much trouble to change. It could be worse. Back in the day I worked for a company that did one of the earlier two-on-motherboard ethernet chip servers. The Boot PROM (from another vendor) had no clue about multiple ethernet interfaces. It came up with both interfaces set to the same NVRAM-set MAC. We wanted to fix it in firmware but kept having issues with that. I had to get an init script to rotate the MAC for the second interface up one, and ensure that it was in the OS and run before the interfaces got plumbed, get it bundled into the OS distribution, and ensure that factory MACs were only set to even numbers to start with. One of these steps ultimately failed rather spectacularly. -- -george william herbert george.herb...@gmail.com
Re: Programmers with network engineering skills
On Mon, Feb 27, 2012 at 5:07 PM, Jason Bertoch ja...@i6ix.com wrote: On 2/27/2012 7:53 PM, William Herrin wrote: I think you're more likely to find a network engineer with (possibly limited) programming skills. I wish. For the past three months I've been trying to find a network engineer with a deep TCP/IP protocol understanding, network security expertise, some Linux experience, minor programming skill with sockets and a TS/SCI clearance. Is clearance the problem, or the ability to obtain clearance due to something in their background? If your work requires it, you should have some recourse for applicants to obtain the required clearance, no? My understanding is that while primary and subcontractor companies can put people in the sponsoring organization's clearance granting queue, it takes so long to get someone through the queue that for high-level positions they essentially make having the clearance already a prerequisite. -- -george william herbert george.herb...@gmail.com
RE: Programmers with network engineering skills
Doug, I think the difference is that network engineers typically find themselves wanting to learn some form of programming to automate routine tasks while doing their job as a network engineer. They've actually managed to be interested in programming while pursuing a career in networking out of necessity. On the other hand, I think it's very rare for a hard-core programmer/developer to want to learn more about networking because it typically doesn't come up in their job when coding a professional application / large product with many moving parts and more than one person on the team. I'm sure it can happen either way and has (as many people have posted going either direction in this thread), but there needs to be some desire to learn for the individual. I think you'll find a network engineer desiring to improve their programming skills much easier than a developer that wants to learn improve their networking skills beyond plugging a router into their home network. -Jared -Original Message- From: Doug Barton [mailto:do...@dougbarton.us] Sent: Monday, February 27, 2012 2:31 PM To: Jay Ashworth Cc: NANOG Subject: Re: Programmers with network engineering skills On 2/27/2012 2:23 PM, Jay Ashworth wrote: - Original Message - From: Owen DeLong o...@delong.com I think you're more likely to find a network engineer with (possibly limited) programming skills. That's certainly where I would categorize myself. And you're the first I've seen suggest, or even imply, that going that direction instead might be more fruitful; seemed to me that the skills necessary to make a decent network engineer would support learning programming better than the other way round -- though in fact I personally did it the other way. I think it depends on what level of coding you're talking about. If you want someone that can whip up a few scripts to easily manage routine tasks, then sure, network guy - coder is usually a safe and easy path. OTOH, if you're talking professional application developer working on a project with more than one moving part, and/or more than one person on the team, you really need someone who thinks like a developer, and can be trained to understand network concepts. and yes, the latter is the path that I've taken, so I have a built-in bias. Doug -- It's always a long day; 86400 doesn't fit into a short. Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/
Re: Programmers with network engineering skills
--- george.herb...@gmail.com wrote: From: George Herbert george.herb...@gmail.com My understanding is that while primary and subcontractor companies can put people in the sponsoring organization's clearance granting queue, it takes so long to get someone through the queue that for high-level positions they essentially make having the clearance already a prerequisite. -- For TS maybe, but the Secret wasn't so bad. Also, it depends on how old you are. The younger you are the less they have to check. scott
Re: Provider WAAS service for multiple MPLS VPN customers, possible?
Theoretically if both WAEs are placed inline on MPLS uplink, then it should work -- unless WAAS code can only recognize IP/Ethernet but not IP/MPLS/Ethernet traffic. I don't think WAAS is VRF aware and can maintain a multi-VRF routing table. On Mon, Feb 27, 2012 at 8:04 AM, Frank Ho completea...@gmail.com wrote: Hi there, Just want to know if anybody out there has tried to put a pair of Cisco WAAS cards on two PEs to optimize the traffic of multiple VRFs between them ? Is that actually possible ? If it's possible, how does the WAAS module card forward the optimized traffic back to the correct VRF? Any hints or sample configurations are most appreciated. Frank.
RE: Programmers with network engineering skills
What about the case of the strong coder who decides that networking is more interesting as a life's work, moves into networking, will not consider employment where coding is even a remote possibility, and will successfully land another networking job elsewhere if management even brings up the subject of coding? I think this describes the great majority of networking professionals. -Original Message- From: Owen DeLong [mailto:o...@delong.com] Sent: Monday, February 27, 2012 2:14 PM To: david raistrick Cc: NANOG Subject: Re: Programmers with network engineering skills On Feb 27, 2012, at 12:31 PM, david raistrick wrote: On Mon, 27 Feb 2012, Owen DeLong wrote: I think you're more likely to find a network engineer with (possibly limited) programming skills. While I'll agree about the more likely, if I needed a coder who had a firm grasp of networking I'd rather teach a good coder networking, than try to teach the art and magic of good development to a network guy. Well, I won't call myself a hard-core coder, but, I think I have a reasonable grasp on the art and magic of good development. What I mostly lack is speed and efficiency in the language of choice for whatever project. I can write good code, it just takes me longer than it would take a hard-core coder. OTOH, having done both, I would say that I think you are not necessarily correct about which direction of teaching is harder. Yes, if you start with a network engineer that knows nothing about writing code or doesn't understand the principles of good coding, you're probably right. However, starting with a network engineer that can write decent code slowly, I think you will get a better result in most cases than if you try to teach network engineering to a hard-core coder that has only a minimal understanding of networking. I think it really comes down to which you need: a hardcore network engineer/architect who can hack up code, or a hardcore developer who has or can obtain enough of a grasp of networking fundementals and specifics to build you the software you need him to develop. I'm guessing that someone who needed a hard-core developer that could grasp fundamentals would have grabbed an existing coder and handed him a copy of Comer. The fact that this person posted to NANOG instead implies to me that he needs someone that has a better grasp than just the fundamentals. Of course I am speculating about that and I could be wrong. The ones who already know both ends extremely well are going to be -very- hard to find, but finding one who can learn enough of the other to accomplish what you need shouldn't be hard at all. Depends on what you need. However, I think it's faster to go from limited coding skills with a good basis in the fundamentals to usable development than to go from limited networking skills to a firm grasp on how networks behave in the real world. To the best of my knowledge, nothing but experience will teach you the latter. Even with 20+ years experience networks do still occasionally manage to surprise me. ...d (who is not exactly the former though I've played one for TV, and not at all the later) I am admittedly lost given the three choices as to which constitutes former or latter at this point. 1. Strong coder with limited networking 2. Strong networker with limited coding 3. Strong in both Owen Who is a strong network engineer Who has been a professional software engineer (though many years ago and my skills are rusty and out of date) This communication, together with any attachments or embedded links, is for the sole use of the intended recipient(s) and may contain information that is confidential or legally protected. If you are not the intended recipient, you are hereby notified that any review, disclosure, copying, dissemination, distribution or use of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by return e-mail message and delete the original and all copies of the communication, along with any attachments or embedded links, from your system.
Re: dns and software, was Re: Reliable Cloud host ?
In message CAP-guGVA4eHv0K=U=x2b-wpydy2rq7ze1di2ahc+dma_huy...@mail.gmail.com, William Herrin writes: On Mon, Feb 27, 2012 at 3:43 PM, david raistrick dr...@icantclick.org wro= te: On Mon, 27 Feb 2012, William Herrin wrote: In some cases this is because of carelessness: The application does a gethostbyname once when it starts, grabs the first IP address in the list and retains it indefinitely. The gethostbyname function doesn't even pass the TTL to the application. Ntpd is/used to be one of the notable offenders, continuing to poll the dead address for years after the server moved. While yes it often is carelessness - it's been reported by hardcore development sorts that I trust that there is no standardized API to obtai= n the TTL... =A0What needs to get fixed is get[hostbyname,addrinfo,etc] so programmers have better tools. Meh. What should be fixed is that connect() should receive a name instead of an IP address. Having an application deal directly with the IP address should be the exception rather than the rule. Then, deal with the TTL issues once in the standard libraries instead of repeatedly in every single application. No. connect() should stay the way it is. Most developers cut and paste the connection code. It's just that the code they cut and paste is very old and is often IPv4 only. In theory, that'd even make the app code protocol agnostic so that it doesn't have to be rewritten yet again for IPv12. getaddrinfo() man page has IP version agnostic code examples. It is however simplistic code which doesn't behave well when a address is unreachable. For examples of how to behave better for TCP see: https://www.isc.org/community/blog/201101/how-to-connect-to-a-multi-homed-server-over-tcp Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org
Re: Programmers with network engineering skills
programming is not being able to write a hundred lines of unreadable perl. a real programmer can be productive in networking tools in a matter of a month or two. i have seen it multiple times. a networker can become a useful real progammer in a year or three. randy
Re: Programmers with network engineering skills
On 02/27/2012 06:23 PM, Randy Bush wrote: programming is not being able to write a hundred lines of unreadable perl. a real programmer can be productive in networking tools in a matter of a month or two. i have seen it multiple times. a networker can become a useful real progammer in a year or three. I agree. Programmers aren't born understanding some fields and not others. In my case, I didn't have a clue about networking coming out of school but picked it up because I thought it was neat, and there was something intoxicating about the smell of the printed out RFC's. Mike, weird i know
RE: Programmers with network engineering skills
But my point is that each person who is capable to do so generally chooses their life's work, after working in and trying out several capacities, and this is extremely common in IT environments where a person could have cycled through programming, system admin, dba, networking, security, etc. For me, I prefer networking, and even a substantial raise would not get me to design and write computer programs again. Life is short, networking professionals generally are in high demand, and are in networking because they like it. Yes Perl scripting may become a temporary, necessary evil at some point, but if the subject of coding comes up, many will move on. -Original Message- From: Randy Bush [mailto:ra...@psg.com] Sent: Monday, February 27, 2012 6:23 PM To: Holmes,David A Cc: North American Network Operators' Group Subject: Re: Programmers with network engineering skills programming is not being able to write a hundred lines of unreadable perl. a real programmer can be productive in networking tools in a matter of a month or two. i have seen it multiple times. a networker can become a useful real progammer in a year or three. randy This communication, together with any attachments or embedded links, is for the sole use of the intended recipient(s) and may contain information that is confidential or legally protected. If you are not the intended recipient, you are hereby notified that any review, disclosure, copying, dissemination, distribution or use of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by return e-mail message and delete the original and all copies of the communication, along with any attachments or embedded links, from your system.
Re: Reliable Cloud host ?
On Sun, Feb 26, 2012 at 4:56 PM, Randy Carpenter rcar...@network1.net wrote: We have been using Rackspace Cloud Servers. We just realized that they have absolutely no redundancy or failover after experiencing a outage that lasted more than 6 hours yesterday. I am appalled that they would offer something called cloud without having any failover at all. Disclaimer: I work for Rackspace in a network architect capacity. We have plenty of redundancy where it is needed. We have all sorts of solutions, for all sorts of intersections of problems, budgets and customers. Sometimes finding the 'correct' solution is not as easy as it could or should be. The menu is simply getting crowded :) I don't know the specifics of your issue, but if you contact me privately I can look into the specifics. You can also use my work email address if you don't think I am legit (email me at this address to get it). I do find that that impact is quite extreme, and certainly an exception and something that there are many folks probably still working on root causes and lessons learned. We take this stuff seriously. 1. Full redundancy with instant failover to other hypervisor hosts upon hardware failure (I thought this was a given!) As others have mentioned, you will not be able to find some of these features for 1.5c/hr. They quickly spiral out of control for large-scale deployments. Every penny matters at 1.5c/hr . I would ask that you look in your product portfolio and see if you have anything at that price that you can answer a support phone call for :) . This is not meant to be antagonistic, just to have a clear mindset understanding of the $$ we are talking about and how careful you have to be. What these cloud price points allow you to do tho is to turn it from one type of a problem to another type of problem that you can have more control over. As others have mentioned, spreading out with many different providers is one example. They (cloud, VMs, VPS, whatever you want to call them) are cheap, disposable computing resources - don't treat them as anything else! As with anything, you get what you pay for, and I am sure we have all had 'that customer' that claims $1,000,000 in losses for every hour of impact, and they have a single whitebox server deployed. 2. Actual support (with a phone number I can call) This is where the providers will typically start to differentiate themselves from each other. As a company, we pride ourselves on support. Full support has a price. I don't want to turn this into a sale-ish email tho. 3. reasonable pricing (No, $800/month is not reasonable when I need a tiny 256MB RAM Server with 1GB/mo of data transfers) 1.5c / hr is what our basic linux image starts at IIRC. Again, I am not in sales, so I don't really keep track of how that compares to some of the other folks out there, I would guess it is about the going rate. I have used Linode.com as well as EC2 as well and they both have some great feature sets and offers. Both also have areas that could use improvements. I do agree that there is general misconceptions of what 'cloud' means. That is simply a byproduct of the amount of folks involved in such trends, and yes, the marketing folks getting involved as well. This is unavoidable in the world today. If you have any other questions or concerns that I can help with, please let me know... cheers, -- jason
Re: Programmers with network engineering skills
a real programmer can be productive in networking tools in a matter of a month or two. i have seen it multiple times. a networker can become a useful real progammer in a year or three. Thank you! I always wonder when someone distinguishes between a networker and a programmer as if they came from completely different worlds. I find these fields to be highly related. They are algorithmic at the core and you need a good understanding of architecture and design to successfully make the concepts work. If you have ever tried to find a bug in a badly structured network, you should be able to understand that implementing all of your application's use cases in one module is not a good idea. After implementing a good serialization scheme for your class data, network protocols are not that strange anymore (I know I'm exaggerating on simple examples here, but I hope the idea comes across). My point is, if someone has a good understanding of applying architectural patterns to a problem and isolating error causes while debugging, it shouldn't matter if he wrote mostly software the last years or if she administered a large scale network. A good sysadmin can learn to write software and a good programmer can learn to love the datacenter.
RE: Programmers with network engineering skills
Yes, a theoretical understanding of algorithms is a common element in programming and networking. But the thread seems to assume that highly capable programmers/network engineers are mere serfs, unable to forge their own destiny, at the beck and call of whomever they work for, instead of independent beings who are doing what they are doing because they like it and choose to continue doing so, even at the expense of foregoing substantial financial gain. -Original Message- From: Daniel Schauenberg [mailto:d...@unwiredcouch.com] Sent: Monday, February 27, 2012 7:09 PM To: Randy Bush Cc: Holmes,David A; North American Network Operators' Group Subject: Re: Programmers with network engineering skills a real programmer can be productive in networking tools in a matter of a month or two. i have seen it multiple times. a networker can become a useful real progammer in a year or three. Thank you! I always wonder when someone distinguishes between a networker and a programmer as if they came from completely different worlds. I find these fields to be highly related. They are algorithmic at the core and you need a good understanding of architecture and design to successfully make the concepts work. If you have ever tried to find a bug in a badly structured network, you should be able to understand that implementing all of your application's use cases in one module is not a good idea. After implementing a good serialization scheme for your class data, network protocols are not that strange anymore (I know I'm exaggerating on simple examples here, but I hope the idea comes across). My point is, if someone has a good understanding of applying architectural patterns to a problem and isolating error causes while debugging, it shouldn't matter if he wrote mostly software the last years or if she administered a large scale network. A good sysadmin can learn to write software and a good programmer can learn to love the datacenter. This communication, together with any attachments or embedded links, is for the sole use of the intended recipient(s) and may contain information that is confidential or legally protected. If you are not the intended recipient, you are hereby notified that any review, disclosure, copying, dissemination, distribution or use of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by return e-mail message and delete the original and all copies of the communication, along with any attachments or embedded links, from your system.
Re: dns and software, was Re: Reliable Cloud host ?
On Feb 27, 2012, at 19:10, Owen DeLong o...@delong.com wrote: On Feb 27, 2012, at 3:50 PM, William Herrin wrote: On Mon, Feb 27, 2012 at 3:43 PM, david raistrick dr...@icantclick.org wrote: On Mon, 27 Feb 2012, William Herrin wrote: In some cases this is because of carelessness: The application does a gethostbyname once when it starts, grabs the first IP address in the list and retains it indefinitely. The gethostbyname function doesn't even pass the TTL to the application. Ntpd is/used to be one of the notable offenders, continuing to poll the dead address for years after the server moved. While yes it often is carelessness - it's been reported by hardcore development sorts that I trust that there is no standardized API to obtain the TTL... What needs to get fixed is get[hostbyname,addrinfo,etc] so programmers have better tools. Meh. What should be fixed is that connect() should receive a name instead of an IP address. Having an application deal directly with the IP address should be the exception rather than the rule. Then, deal with the TTL issues once in the standard libraries instead of repeatedly in every single application. In theory, that'd even make the app code protocol agnostic so that it doesn't have to be rewritten yet again for IPv12. While I agree with the principle of what you are trying to say, I would argue that it should be dealt with in getnameinfo() / getaddrinfo() and not connect(). It is perfectly reasonable for connect() to deal with an address structure. If people are not using getnameinfo()/getaddrinfo() from the standard libraries, then, I don't see any reason to believe that they would use connect() from the standard libraries if it incorporated their functionality. gai/gni do not return TTL values on any platforms I'm aware of, the only way to get TTL currently is to use a non standard resolver (e.g. lwres). The issue is application developers not calling gai every time they connect (due to aforementioned security concerns, at least in the browser realm), instead opting to hold onto the original resolved address for unreasonable amounts of time. Modifying gai to provide TTL has been proposed in the past (dnsop '04) but afaik was shot down to prevent inconsistencies in the API. Maybe when happy eyeballs stabilizes someone will propose an API for inclusion in the standard library that implements HE style connections. Looks like there was already some talk on v6ops headed this way, but as always there's resistance to standardizing it. ~Matt
Re: dns and software, was Re: Reliable Cloud host ?
getaddrinfo was designed to be extensible as was struct addrinfo. Part of the problem with TTL is not data sources used by getaddrinfo have TTL information. Additionally for many uses you want to reconnect to the same server rather than the same name. Note there is nothing to prevent a getaddrinfo implementation maintaining its own cache though if I was implementing such a cache I would have a flag to to force a refresh. -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org
Re: Programmers with network engineering skills
On Mon, Feb 27, 2012 at 12:27 PM, A. Pishdadi apishd...@gmail.com wrote: Hello All, i have been looking for quite some time now a descent coder (c,php) who has Just a practical comment here; part of your problem may be offering c and php together. I don't want to start a war, but I know that at the very least all the c programmers I know would considered php to be ... horribly offensive. So, maybe seperating out these two roles (c and php programming) will help you. It is definitely true (speaking as a programmer, C# for several years) that seeing +PHP would instantly turn me off. Further, I'm sure that almost anyone who is still programming in c these days would have the level of networking knowledge you care about (and can train on top of). a descent amount of system admin / netadmin experience. Doesn't necessarily need to be an expert at network engineering but being acclimated in understanding the basic fundamentals of networking. Understanding basic routing concepts, how to diagnose using tcpdump / pcap, understanding subnetting and how bgp works (not necessarily setting up bgp). I've posted job listings on the likes of dice and monster and have not found any good canidates, most of them ASP / Java guys. If anyone can point me to a site they might recommend for job postings or know of any consulting firms that might provide these services that would be greatly appreciated. -- Noon Silk Fancy a quantum lunch? https://sites.google.com/site/quantumlunch/ Every morning when I wake up, I experience an exquisite joy — the joy of being this signature.
