Re: Managing IOS Configuration Snippets
On 2/28/2014 9:19 PM, Dale W. Carder wrote:
If I'm understanding what you're trying to do, you could script around
our rather unsophisticated 'sgrep' (stanza grep) tool combined with
scripting around rancid rcs to do what I think you are looking for.
http://net.doit.wisc.edu/~dwcarder/scripts/sgrep
sgrep can dump out a stanza of ios-like config, then you can rcsdiff
that to your master, per 'chunk' of config.
Dale
I'm digging the idea of your command. Along the same lines I've got
this awk snippet that I made and then forgot about. It functions like
the cisco pipe begin/end commands:
#!/bin/sh
if [ x${2} = x ]; then
awk /${1}/{temp=1}; temp==1{print}
else
awk /${1}/{temp=1};/${2}/{temp=0}; temp==1{print}
fi
Usage:
cat router-confg | begin_regex 'line vty' '!'
If you omit the second argument it just shows you from your match until
the end of file.
Re: Managing IOS Configuration Snippets
Robert,
sgrep can dump out a stanza of ios-like config, then you can rcsdiff
that to your master, per 'chunk' of config.
Dale
I'm digging the idea of your command. Along the same lines I've got this
awk snippet that I made and then forgot about. It functions like the cisco
pipe begin/end commands:
The clever part comes when you write something to tweak sgrep to spit out files
with the commands you need to run to reconfigure multiple devices, and then use
the make command to invoke clogin in parallel to push those changes. Showing
a complete example would get tedious, I have a tarball of the minimum fileset
needed to do this lying around somewhere. Dale probably has one as well.
The basic workflow is as follows:
1) Given a set of configuration files named device.conf, generate a set of
configuration changes to be made, and place those changes in a set of files
named device.cmd. Device is the valid hostname of a router/switch you need to
configure. Device.conf is the configuration file for that device, for example
from rancid. The contents of device.cmd are valid config commands including
config term, exit and write mem, that you intend to run on the device.
We will talk about what to do with those command files in a later step.
So if you wanted to edit the name of vlan100 on all of your routers, you might
do this:
for device in `ls r-*.conf | cut -d. -f1`;
do sgrep -is ^vlan 10\n $device.conf | sed -e 's/name .*/name
new-name/' | grep -v Found $device.cmd;
done
r-myrouter-100.cmd might contain:
vlan 10
name new-name
!
So you would need to make your loop more complex to add the conf t, exit
and wr mem parts, or just use a script. I write a script called mkcmdfile to
wrap sgrep every time I need to do work like this. mkcmdfile means make
command file. If you want to work from the command line, add a second sed
rule to look for Found and replace it with conf t, eliminate the trailing
grep -v Found and edit the original sed to replace vlan new-name with vlan
new-name\nexit\nwr mem/. I don't recommend doing that, but you could. I put
that into a wrapper script, or I post process the cmd files with a second for
loop.
Now that is the less hard way to change 4,000 devices. Except for made up
examples like renaming a vlan on 4,000 devices, that is only a little less
complicated than just doing it by hand, but hopefully you get the idea.
One huge win though, even when doing it by hand as above, I can generate the
files ahead of time, ensure they are correct, and then when I push them at 4AM
I do not have to worry about typographic errors. If you have change approval
boards, the CAB can examine the .cmd files if they choose to, this is a great
way to sanity check your work at 2PM when your mind is clear.
Loops are cool, but in reality, I am far more likely to edit sgrep with some
reasonable defaults, hand it a list of filenames, and then rely on $ARGV
changing to reflect the name of the file it is currently working on, so I know
that when I have a diamond loop (for () { do stuff }) in my Perl, $ARGV.conf
is what sgrep is reading from, and $ARGV.cmd is the file sgrep should be
writing to. In fact, I have something I call mkcmdfile that does pretty much
that.
Writing a custom sgrep for each change allows me to consider the entire
contents of the router, including its name, the business rules associated with
it, anything I keep in a database, etc, when making configuration changes to it.
Don't worry too much about how to do step 1. The lesson you must understand,
is that step 1 is finished when you have a directory full of files, all named
device.cmd, where device is the name of the device to work on, and the contents
are the commands to execute.
2) Given a directory full of command files (named device.cmd) invoke clogin on
each file:
The hard way would be another for loop:
for device in `ls *.cmd | cut -d. -f1`;
do clogin -s $device.cmd $device $device.log;
done
I go into how to do this easily using make later.
3) Look at the resulting $device.log files you generated, and make sure nothing
crazy happened.
It really is that simple. I can create cmd files for 4,000 devices, ensure
the cmd files are changing the correct interface properties, ntp server names
or whatever, and push them to all 4,000 devices, in a few hours. Doing that by
hand would take a week. In some cases, I can push them in 30 minutes.
I have the minimum fileset you need to do that lying around somewhere, but
basically that is it.
When I wrote sgrep, I was asking myself, what would happen if I make a tool
that had more computing power than a tool that works on regular expressions?
grep gets its name from: global - 'regular expression' - print. But regular
expressions are limited in the computing complexity they can reflect. I really
wanted something turing complete. But I never found a way to do it, that
Re: Managing IOS Configuration Snippets
On Feb 28, 2014, at 9:11 AM, Ryan Shea ryans...@google.com wrote: Keegan, don't get me wrong, I am not suggesting that even if version numbers were happily encoded in robust comments that this would be the same as actually digesting the configuration. If the function of checking using 'fancy versioning' is not an operational best practice, what do you suggest (all-knowing/singing/dancing tool which understands the configuration and your intent aside)? You say IF NTP or CPP were configured differently - with a large enough network there will always be configurations which have differences. With that as an operational constant, how do you determine which devices have the latest iteration of your line vty configuration. That’s what I mean. The things that lend well to to version tracking don’t tend to change much. How many ways are there to configure VTY lines, or NTP, or CPP, or even OSPF and if we’re talking about an access ACL why not just audit the configs to make sure that all the entries are there. Am I really going to care that one router has version 1.0 versus another router that has version 2.2.12 build9? It’s not source code.. How often will a change not be rolled out to every router. This is again related to the size and churn of the network, but my practical estimation is that once you get into thousands of routers there will almost always be some that get missed. Again, a router that was missed is a reason for audit and remediation not versioning. If you find a router with config missing does it really matter what version it’s on and when that version was valid? Not in my experience. Comprehensive auditing is very important, and arguably more useful than version checking - but it requires that you make knowledgeable and complete assertions. I assert the my snmp config should look like the snmp snippet version 77 is easier to grok than make sure our community string is not set to public (and repeat hand-crafted audit logic for every segment of the config). There may be some differences, but those are normally due to equipment lifecycle, mergers/consolidations and such. It’s easy to refer to something as the config for a particular platform or company than a version number. This can be tracked in GIT or SVN. Even then there will not be constant changes. I’d lean towards standardization. So the equipment that cannot adhere to the defined standards probably won’t evolve much on it’s own. What if some of the configs don't match the defined versions? This is why it may make sense to break snippets into functional areas. Just fix it might be sane for a banner, but squashing an interface mtu change that was put there for a reason could end in tears. I consider this bit out of the scope of the question, but yes it is another important problem. I wasn’t saying just fix it. I was saying that router configs don’t lend well to versioning. With software for example, if something is different it might be a different version of that application with compatibility issues, dependencies, library issues, etc. When it’s a router config chances are someone fat-fingered something. Most of the time the best thing to do is to fix or at least alert on the error, not to record it as a valid config version. Again, this is for things that lend themselves to snippets. ACL’s, NTP, SNMP, CPP, even Spanning-tree. Not for things like interface IP’s or static routes that may be different across different boxes or location. If you’re referring to the latter I may have misunderstood your question.. On Thu, Feb 27, 2014 at 10:03 PM, Christopher Morrow morrowc.li...@gmail.com wrote: On Thu, Feb 27, 2014 at 8:38 PM, Keegan Holley no.s...@comcast.net wrote: Putting aside the fact that snippets aren't a good way to conceptualize deployed router code, my gut still tells me to question the question here. The first is does this stuff change often enough to warrant a fancy versioning solution? I have yet to see NTP deployed in a different way than when I first learned to configure it. Next, when it does change how often is it not rolled out to every router. If NTP or CPP or SNMP or some other administrative option were configured differently across my sure, so you're saying that a large bit (maybe) of the router config is 'one size fits all' and 'never changes' where 'never' is really 'very infrequently'. sure, agreed... but there are parts of the config that do change more frequently (depending on the network perhaps)... how do you go about seeing which version / setup is deployed EXCEPT by building a home-grown 'config parser' and seeing that 'what is deployed matches mostly what I have in my config store for this router/class-of-router/network' ? It's a shame that vendors of network equipment don't have to manage large networks of their own equipment under constrained opex environments (no
Re: Managing IOS Configuration Snippets
On Feb 27, 2014, at 7:38 PM, Keegan Holley no.s...@comcast.net wrote: Putting aside the fact that snippets aren’t a good way to conceptualize deployed router code, my gut still tells me to question the question here. What I have always wanted is a way to group configuration, in particular by customer. Ideally with the ability to see it both as a unified view, and also as a per-customer view. For instance: customer A interface GigabitEthernet1/2/3.10 description A ip address 10.0.1.1 255.255.255.0 router bgp 1 neighbor 10.0.1.2 prefix-list A-in in ip prefix-list A-in 10.1.0.0/24 end customer B interface GigabitEthernet1/2/3.11 description B ip address 10.0.2.1 255.255.255.0 router bgp 1 neighbor 10.0.2.2 prefix-list B-in in ip prefix-list B-in 10.2.0.0/24 end Then I should be able to do: show run - Normal output like we see today, the device view. customer A show run - Same format as I have above, just config relevant to customer A. I can even see extending the tag to work with some other commands: customer A show int customer A show bgp ipv4 uni sum customer A show ip prefix-list The same functionality would work for snippets: customer ntp-servers-v1.0 ntp server 1.2.3.4 ntp server 1.2.3.5 ntp server 1.2.3.6 end Basically this follows the two modes in which engineers look at a device. Most of the time is configuring a specific customer, and wanting to be sure they are configured right; including the hard case of no customer A, that is making sure all configuration for a specific customer is removed. The rest of the time is typically troubleshooting a network level problem where you want the device view we have today, I see interface Gig1/2/3 is dropping packets, show run to see who's configure on it sort of operations. I don't know of any platform that has implemented this sort of config framework though. -- Leo Bicknell - bickn...@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ signature.asc Description: Message signed with OpenPGP using GPGMail
Re: Managing IOS Configuration Snippets
On 2/28/14, 10:24 , Leo Bicknell wrote: What I have always wanted is a way to group configuration, in particular by customer. Ideally with the ability to see it both as a unified view, and also as a per-customer view. For instance: customer A interface GigabitEthernet1/2/3.10 description A ip address 10.0.1.1 255.255.255.0 router bgp 1 neighbor 10.0.1.2 prefix-list A-in in ip prefix-list A-in 10.1.0.0/24 end ... Basically this follows the two modes in which engineers look at a device. Most of the time is configuring a specific customer, and wanting to be sure they are configured right; including the hard case of no customer A, that is making sure all configuration for a specific customer is removed. The rest of the time is typically troubleshooting a network level problem where you want the device view we have today, I see interface Gig1/2/3 is dropping packets, show run to see who's configure on it sort of operations. I don't know of any platform that has implemented this sort of config framework though. It's not the cleanest, but in junos you can pretty much get this by defining a configuration group per customer and applying them all. Your RE may hate you at commit time, but I've seen this approach work quite well. -e
Re: Managing IOS Configuration Snippets
Thus spake Ryan Shea (ryans...@google.com) on Thu, Feb 27, 2014 at 09:38:33AM -0500: Now, I hand you the 'show run' output and ask you if version 77 of the vty config is on this device. Can you answer the question? Now I hand you the 'show run' from 10,000 more device configs - and 100 more configuration chunks from revision control. Can you still answer the question? Assume a magical revision-history-aware configuration cross reference parser (while a noble and lovely goal) is not available. Hi Ryan, If I'm understanding what you're trying to do, you could script around our rather unsophisticated 'sgrep' (stanza grep) tool combined with scripting around rancid rcs to do what I think you are looking for. http://net.doit.wisc.edu/~dwcarder/scripts/sgrep sgrep can dump out a stanza of ios-like config, then you can rcsdiff that to your master, per 'chunk' of config. ex: sgrep -s vty r-cssc-b280c-1-core.conf Found stanza in r-cssc-b280c-1-core.conf size:9 line vty 0 4 access-class G-A-AdminAccess in exec-timeout 30 0 ipv6 access-class G-A-v6AdminAccess in line vty 5 24 access-class G-A-AdminAccess in exec-timeout 30 0 ipv6 access-class G-A-v6AdminAccess in ! See the -s and -e options for our sgrep. Add 'xargs -P' around your glue, and I think you'd be in luck. If you were building configs around this model, you could use m4. Dale
Re: Managing IOS Configuration Snippets
Thus spake Keegan Holley (no.s...@comcast.net) on Fri, Feb 28, 2014 at 09:49:19AM -0500: I wasn’t saying just fix it. I was saying that router configs don’t lend well to versioning. Um, what? $ rlog r-cssc-b280c-1-core.conf | grep 'total revision' total revisions: 2009; selected revisions: 2009 When it’s a router config chances are someone fat-fingered something. Most of the time the best thing to do is to fix or at least alert on the error, not to record it as a valid config version. We have our operators manually check in revisions (think in rcs terms: co -l router, go do work, verify it, ci -u router) rather than unsolicited / cron-triggered checkins. Then the check-in message contains the operator's description text of the change and often a ticket number. So there slightly fewer fat-finger configs checked in. Dale
Re: Managing IOS Configuration Snippets
On Feb 28, 2014, at 9:35 PM, Dale W. Carder dwcar...@wisc.edu wrote: Thus spake Keegan Holley (no.s...@comcast.net) on Fri, Feb 28, 2014 at 09:49:19AM -0500: I wasn’t saying just fix it. I was saying that router configs don’t lend well to versioning. Um, what? $ rlog r-cssc-b280c-1-core.conf | grep 'total revision' total revisions: 2009;selected revisions: 2009 I wish you were here to see my eyes rolling.. 2009 versions of something are no more grok-able than one current version. Congrats, you have a config backup system. When it’s a router config chances are someone fat-fingered something. Most of the time the best thing to do is to fix or at least alert on the error, not to record it as a valid config version. We have our operators manually check in revisions (think in rcs terms: co -l router, go do work, verify it, ci -u router) rather than unsolicited / cron-triggered checkins. Then the check-in message contains the operator's description text of the change and often a ticket number. So there slightly fewer fat-finger configs checked in. That’s not what the OP was looking for AFAIK. This is just change management. Dale
Re: Managing IOS Configuration Snippets
For a large install I set up a solution that might help. I utilized a Mediawiki install and its API to create, update and pull the configuration on many IOS devices. A wiki page for the host name was dynamically created and the configuration was placed there daily or hourly. This allowed support to review the configuration and advise customers quicker. Additional hacks for updating the devices via the wiki were used. The goal was transparency for the support team and the side effect was wiki page history showing what day and what lines changed. As mentioned the answer to your question would likely make a good article. On Wed, Feb 26, 2014 at 3:22 PM, Ryan Shea ryans...@google.com wrote: Howdy network operator cognoscenti, I'd love to hear your creative and workable solutions for a way to track in-line the configuration revisions you have on your cisco-like devices. Let me clearify/frame: You have a set of tested/approved configurations for your routers which use IOS style configuration. These configurations of course are always refined and updated. You break these pieces of configuration into logical sections, for example a configuration file for NTP configuration, a file for control plane filter and store these in some revision control system. Put aside for the moment whether this is a reasonable way to comprehend deployed configurations. What methods do some of you use to know which version of a configuration you have deployed to a given router for auditing and update purposes? Remarks are a convenient way to do this for ACLs - but I don't have similar mechanics for top level configurations. About a decade ago I thought I'd be super clever and encode versioning information into the snmp location - but that is just awful and there is a much better way everyone is using, right? Flexible commenting on other vendors/platforms make this a bit easier. Assume that this version encoding perfectly captures what is on the router and that no person is monkeying with the config... version 77 of the control plane filter is the same everywhere. -- ~ Andrew lathama Latham lath...@gmail.com http://lathama.net ~
Re: Managing IOS Configuration Snippets
Wow, this sounds fantastic! Have any code you can share? Cheers, Harry On Feb 27, 2014 6:52 AM, Andrew Latham lath...@gmail.com wrote: For a large install I set up a solution that might help. I utilized a Mediawiki install and its API to create, update and pull the configuration on many IOS devices. A wiki page for the host name was dynamically created and the configuration was placed there daily or hourly. This allowed support to review the configuration and advise customers quicker. Additional hacks for updating the devices via the wiki were used. The goal was transparency for the support team and the side effect was wiki page history showing what day and what lines changed. As mentioned the answer to your question would likely make a good article. On Wed, Feb 26, 2014 at 3:22 PM, Ryan Shea ryans...@google.com wrote: Howdy network operator cognoscenti, I'd love to hear your creative and workable solutions for a way to track in-line the configuration revisions you have on your cisco-like devices. Let me clearify/frame: You have a set of tested/approved configurations for your routers which use IOS style configuration. These configurations of course are always refined and updated. You break these pieces of configuration into logical sections, for example a configuration file for NTP configuration, a file for control plane filter and store these in some revision control system. Put aside for the moment whether this is a reasonable way to comprehend deployed configurations. What methods do some of you use to know which version of a configuration you have deployed to a given router for auditing and update purposes? Remarks are a convenient way to do this for ACLs - but I don't have similar mechanics for top level configurations. About a decade ago I thought I'd be super clever and encode versioning information into the snmp location - but that is just awful and there is a much better way everyone is using, right? Flexible commenting on other vendors/platforms make this a bit easier. Assume that this version encoding perfectly captures what is on the router and that no person is monkeying with the config... version 77 of the control plane filter is the same everywhere. -- ~ Andrew lathama Latham lath...@gmail.com http://lathama.net ~
Re: Managing IOS Configuration Snippets
Rancid with the git plugin can be used to attain pretty much the exact same thing a lot more easily, if you're after an existing implementation of it. Cheers, Paul On 2/27/2014 午後 09:44, Harry Hoffman wrote: Wow, this sounds fantastic! Have any code you can share? Cheers, Harry On Feb 27, 2014 6:52 AM, Andrew Latham lath...@gmail.com wrote: For a large install I set up a solution that might help. I utilized a Mediawiki install and its API to create, update and pull the configuration on many IOS devices. A wiki page for the host name was dynamically created and the configuration was placed there daily or hourly. This allowed support to review the configuration and advise customers quicker. Additional hacks for updating the devices via the wiki were used. The goal was transparency for the support team and the side effect was wiki page history showing what day and what lines changed. As mentioned the answer to your question would likely make a good article. On Wed, Feb 26, 2014 at 3:22 PM, Ryan Shea ryans...@google.com wrote: Howdy network operator cognoscenti, I'd love to hear your creative and workable solutions for a way to track in-line the configuration revisions you have on your cisco-like devices. Let me clearify/frame: You have a set of tested/approved configurations for your routers which use IOS style configuration. These configurations of course are always refined and updated. You break these pieces of configuration into logical sections, for example a configuration file for NTP configuration, a file for control plane filter and store these in some revision control system. Put aside for the moment whether this is a reasonable way to comprehend deployed configurations. What methods do some of you use to know which version of a configuration you have deployed to a given router for auditing and update purposes? Remarks are a convenient way to do this for ACLs - but I don't have similar mechanics for top level configurations. About a decade ago I thought I'd be super clever and encode versioning information into the snmp location - but that is just awful and there is a much better way everyone is using, right? Flexible commenting on other vendors/platforms make this a bit easier. Assume that this version encoding perfectly captures what is on the router and that no person is monkeying with the config... version 77 of the control plane filter is the same everywhere. -- ~ Andrew lathama Latham lath...@gmail.com http://lathama.net ~
Re: Managing IOS Configuration Snippets
On (2014-02-26 17:37 -0500), Robert Drake wrote: Consider looking at Tail-F's NCS, which according to marketing presentations appears to do everything I want right now. I'd like to believe them but I don't have any money so I can't test it out. :) Tail-F is probably least bad option out there. In configuration management, this is super easy: DB = Template = Network This is super hard: Network = DB The first one keeps all platform specific logic in flat ascii files filled with variables from template. When you introduce new product, feature, vendor to network, you only add new ascii templates, extremely easy, no platform-specific logic in DB. The second one every little change in network, requires parser changes trying to model it back to DB. This is not sustainable. We can kid ourselves that NetCONF/YANG will solve this, but they won't. SNMP is old technology, when new feature comes to vendor, it may take _years_ before MIB comes. There is no reason to suspect you will be able to get feature out via NetCONF just because it is there. And if you can't do it 100% then you have to write parser which can understand it. You only need the second one, in case 100% is not from DB. But it is actually trivial to produce 100% from DB. You don't want DB to model base configuration, that's lot of work for no gain, that'll come from template or at most DB vendor-specific-blob. Then after you push configuration from DB to network, you immediately collect configuration and create relation of DB-config 2 network-config, now you can keep ensuring network has correct config. If it does not have, you don't know why not, you can't fix the error itself, but you can repovision whole box, so you do get configuration conformance check, it's just very crude. But the alternative, trying to understand network config, is just never ending path to to pain. If someone is going to do it, model it to python or ruby ORM and put it in github so others can contribute and we don't need to do it alone. -- ++ytti
Re: Managing IOS Configuration Snippets
On Thu, Feb 27, 2014 at 8:58 AM, Saku Ytti s...@ytti.fi wrote: On (2014-02-26 17:37 -0500), Robert Drake wrote: Consider looking at Tail-F's NCS, which according to marketing presentations appears to do everything I want right now. I'd like to believe them but I don't have any money so I can't test it out. :) Tail-F is probably least bad option out there. In configuration management, this is super easy: DB = Template = Network This is super hard: Network = DB The first one keeps all platform specific logic in flat ascii files filled with variables from template. When you introduce new product, feature, vendor to network, you only add new ascii templates, extremely easy, no platform-specific logic in DB. The second one every little change in network, requires parser changes trying to model it back to DB. This is not sustainable. We can kid ourselves that NetCONF/YANG will solve this, but they won't. SNMP is old technology, when new feature comes to vendor, it may take _years_ before MIB comes. There is no reason to suspect you will be able to get feature out via NetCONF just because it is there. And if you can't do it 100% then you have to write parser which can understand it. You only need the second one, in case 100% is not from DB. But it is actually trivial to produce 100% from DB. You don't want DB to model base configuration, that's lot of work for no gain, that'll come from template or at most DB vendor-specific-blob. Then after you push configuration from DB to network, you immediately collect configuration and create relation of DB-config 2 network-config, now you can keep ensuring network has correct config. If it does not have, you don't know why not, you can't fix the error itself, but you can repovision whole box, so you do get configuration conformance check, it's just very crude. But the alternative, trying to understand network config, is just never ending path to to pain. If someone is going to do it, model it to python or ruby ORM and put it in github so others can contribute and we don't need to do it alone. -- ++ytti Agree with this. We started out with rancid, quickly moved to a homebrew scp and git backed system with webgit/cgit as the user interface. If you are lucky your network equipment supports advanced features like ssh keys. If not, you might be stuck using sshpass to ease config collection. Built a config parsing system that would decompose monolithic configs into configlet files. Md5sum the file and use as part of the filename. You can then see version information for parts of the config tree. Quickly realized that maintaining this system is a full time job, due to the advanced status of network equipment software... Now looking at Tail-F NCS. Demo is impressive. I'm hopeful. Stating the obvious: the software running on most network equipment is of poor quality. The tools to manage this are a combination of high quality engineers and homebrew tools. Vendor tools are of a similar quality to the equipment software. I'd like to think SDN is an attempt to improve this, but I have my doubts. -- Tim:
Re: Managing IOS Configuration Snippets
On Thu, Feb 27, 2014 at 9:50 AM, Ryan Shea ryans...@google.com wrote: A couple more thoughts, regarding Network = DB I completely agree that trying to use the network config itself as the authority for what we intend to be on a device is not the right long-term approach. There is still a problem with Network = DB that I see. Assuming you have *many* devices, that may or may not be up at a given time, or may be in various stages of turn-up / burn-in / decom it is expected that a config change will not successfully make it to all devices. There are other timing issues, like a config built for a device being turned up, followed by a push of an update to all devices that succeeds, followed by the final turn-up of this device. Even if you have a fancy config pushing engine, let's just take as a given that you'll need to scrub through your rancid-git backups to determine what needs to be updated. Regarding the MD5 approach, let's also think that configlets could have no commands in them. In the NTP example I had before, if we wanted to remove an NTP server the configlet would need the no version, but the rancid backup obviously would not have this. I'm not trying to work a unit test assertion framework here either. Some vendors have more robust commenting, and this can be quite convenient for explicitly stating what was pushed to the device. What are you using in your network... banner, snmp-location, hope, prayer? We don't do this, but the only flexible commenting in IOS style configs is ACLs. You could have an ACL that contains remarks only, and include version information: ip access-list CFG-VER remark CFG-VER-NTP 1.0.3 remark CFG-VER-VTY 4.3.2 end You could break this into individual ACLs if you prefer: ip access-list CFG-VER-NTP remark CFG-VER-NTP 1.0.3 end ip access-list CFG-VER-VTY remark CFG-VER-VTY 4.3.2 end Seems ridiculous, but that is the sorry state of the network OS. -- Tim:
Re: Managing IOS Configuration Snippets
On (2014-02-27 09:50 -0500), Ryan Shea wrote: Regarding the MD5 approach, let's also think that configlets could have no commands in them. In the NTP example I had before, if we wanted to For DB = Template = Network it's to me very easy, but yes, each template you make must have anti-template version. So let's say you have NTP model, which may contain some access restriction information, NTP version, NTP peers. When you apply this model to device, then some platform specific ntp template is called. If you remove this from device, you need to call 'anti' version of the template. Very simple and easy. You also wondered how do I know which version of config network device has, this is hard problem. To know exactly what is wrong and how to address just that. If you can relax requirement to know if configuration is correct or incorrect it becomes trivial. But fixing incorrect is either full reprovision of new config (at least in IOS and JunOS not a problem, won't break the unchanged bits). Or you have human resolve it (of course as custom dictates first you punish the responsible severely but swiftly) -- ++ytti
Re: Managing IOS Configuration Snippets
On 2/26/14, 16:22 , Ryan Shea wrote: Howdy network operator cognoscenti, I'd love to hear your creative and workable solutions for a way to track in-line the configuration revisions you have on your cisco-like devices. ... Assume that this version encoding perfectly captures what is on the router and that no person is monkeying with the config... version 77 of the control plane filter is the same everywhere. At a previous job, our roll-your-own solution was a template based system(*) generating full configs; all the version history for template sections, per-router local tweaks, and generated results was kept in RCS, and the actual last-configured version, plus any incremental changes, was kept in the login banner. So at login you'd see something like: blah blah authorized users blah blah $Id: routername-confg,v 1.23 2013/08/20 03:07:16 username Exp INCR: 1.2,1.3,1.4,1.5,1.6 and that version tracking made its way through to rancid for easy offline auditing. This made it nice and easy to tell when and what had been updated, though it still would take a couple steps to identify what exact subsections had been changed over time (since the incremental version tags were specific deltas in per-device configurations. You could probably do it in a more global way too - git commit ids, maybe? - but you also don't want to make the version string too wordy either). -e * based on ftp://ftp.cac.washington.edu/pub/config-generator/, but substantially enhanced beyond the last public domain version. I know I'd be really happy if the current version was ever open-sourced...
Re: Managing IOS Configuration Snippets
On Thu, Feb 27, 2014 at 10:34 PM, Erik Muller er...@buh.org wrote: At a previous job, our roll-your-own solution was a template based system(*) generating full configs; all the version history for template sections, per-router local tweaks, and generated results was kept in RCS, and the actual last-configured version, plus any incremental changes, was kept in the login banner. This has been around for several years now - http://sourceforge.net/projects/cisco-conf-rep/ -- Suresh Ramasubramanian (ops.li...@gmail.com)
RE: Managing IOS Configuration Snippets
Along those same lines, we've been using alias exec for the same thing for a while: Alias exec NTP 6500_NTP_V1.0.1 Alias exec bgp 6500_peer_V2.0.0 Thanks, Chuck -Original Message- From: Tim Durack [mailto:tdur...@gmail.com] Sent: Thursday, February 27, 2014 11:50 AM To: Ryan Shea Cc: nanog@nanog.org Subject: Re: Managing IOS Configuration Snippets On Thu, Feb 27, 2014 at 9:50 AM, Ryan Shea ryans...@google.com wrote: A couple more thoughts, regarding Network = DB I completely agree that trying to use the network config itself as the authority for what we intend to be on a device is not the right long-term approach. There is still a problem with Network = DB that I see. Assuming you have *many* devices, that may or may not be up at a given time, or may be in various stages of turn-up / burn-in / decom it is expected that a config change will not successfully make it to all devices. There are other timing issues, like a config built for a device being turned up, followed by a push of an update to all devices that succeeds, followed by the final turn-up of this device. Even if you have a fancy config pushing engine, let's just take as a given that you'll need to scrub through your rancid-git backups to determine what needs to be updated. Regarding the MD5 approach, let's also think that configlets could have no commands in them. In the NTP example I had before, if we wanted to remove an NTP server the configlet would need the no version, but the rancid backup obviously would not have this. I'm not trying to work a unit test assertion framework here either. Some vendors have more robust commenting, and this can be quite convenient for explicitly stating what was pushed to the device. What are you using in your network... banner, snmp-location, hope, prayer? We don't do this, but the only flexible commenting in IOS style configs is ACLs. You could have an ACL that contains remarks only, and include version information: ip access-list CFG-VER remark CFG-VER-NTP 1.0.3 remark CFG-VER-VTY 4.3.2 end You could break this into individual ACLs if you prefer: ip access-list CFG-VER-NTP remark CFG-VER-NTP 1.0.3 end ip access-list CFG-VER-VTY remark CFG-VER-VTY 4.3.2 end Seems ridiculous, but that is the sorry state of the network OS. -- Tim:
Re: Managing IOS Configuration Snippets
On 2/27/14, 12:21 , Suresh Ramasubramanian wrote: This has been around for several years now - http://sourceforge.net/projects/cisco-conf-rep/ But that's just archiving, like rancid, right? Still doesn't have any correlation to the template-management side of things. While having the backups makes it easy to check for simple things (do all my routers have the right syslog host set?), OP's question is about tracking what versions of templates may have been applied to routers; if there's any complex logic (like, are all active customer routes on this device included in the bcp38 acl on the upstream interface) or site-specific things, that can get a lot harder to audit without the metadata on how the configuration got there. -e
Re: Managing IOS Configuration Snippets
A lot of template management discussion focusses on using the network configs as the canonical model of the network. Storing the network model in the DB (whatever form that takes) is much more sane. There is the brownfields issue of populating that database and then building device state from there, but once that's done a lot of the problems go away. A solution like rancid/tail-f then simply becomes the mechanics to push your device state to the devices. Some good stuff here: https://www.nanog.org/meetings/nanog44/presentations/Monday/Gill_programatic_N44.pdf --Simon On 27 February 2014 09:46, Erik Muller er...@buh.org wrote: On 2/27/14, 12:21 , Suresh Ramasubramanian wrote: This has been around for several years now - http://sourceforge.net/projects/cisco-conf-rep/ But that's just archiving, like rancid, right? Still doesn't have any correlation to the template-management side of things. While having the backups makes it easy to check for simple things (do all my routers have the right syslog host set?), OP's question is about tracking what versions of templates may have been applied to routers; if there's any complex logic (like, are all active customer routes on this device included in the bcp38 acl on the upstream interface) or site-specific things, that can get a lot harder to audit without the metadata on how the configuration got there. -e
Re: Managing IOS Configuration Snippets
On 27 February 2014 10:39, Ryan Shea ryans...@google.com wrote: Very cool, thanks Erik. I can think of many ways to encode version metadata. Probably best to be somewhere in between overly verbose (full version $Id / date / author for every config chunk) and being unreadable (base64 encoded gzip of unique configlet identifiers and versions). Updating a banner feels a bit easier when you are pushing a full device-specific configuration from a templating system. Regardless of where it is stored, keeping the metadata in one of these fields (banner for example) means that checking the contents of the banner configlet now requires slightly more logic - which is fine. Chuck, interesting use of alias. Simon, completely agree that the network itself should not be the intent store. The real focus here is when your intent is in a DB/templating system thingy, how do you operationally ensure that intent matches reality. Again, with many devices going through upgrades, disabled/unreachable devices, new devices, pre-configured devices. The intent pusher is not blindly and constantly pushing to all devices, and it's likely not safe to do that. Absolutely. Expect/SNMP/config scraping is a solution here. It's tedious and boring to write the hooks, but it's not impossible. A solution like tail-f is a much better solution here. My personal preference would be to just push/pull JSON off the devices. I think there are two separate components here (which are often conflated): the mechanics to push/pull from devices into a data structure, and the network database to work with those data structures. There's a place for both device level models and network level models. --Simon On Thu, Feb 27, 2014 at 12:04 PM, Erik Muller er...@buh.org wrote: On 2/26/14, 16:22 , Ryan Shea wrote: Howdy network operator cognoscenti, I'd love to hear your creative and workable solutions for a way to track in-line the configuration revisions you have on your cisco-like devices. ... Assume that this version encoding perfectly captures what is on the router and that no person is monkeying with the config... version 77 of the control plane filter is the same everywhere. At a previous job, our roll-your-own solution was a template based system(*) generating full configs; all the version history for template sections, per-router local tweaks, and generated results was kept in RCS, and the actual last-configured version, plus any incremental changes, was kept in the login banner. So at login you'd see something like: blah blah authorized users blah blah $Id: routername-confg,v 1.23 2013/08/20 03:07:16 username Exp INCR: 1.2,1.3,1.4,1.5,1.6 and that version tracking made its way through to rancid for easy offline auditing. This made it nice and easy to tell when and what had been updated, though it still would take a couple steps to identify what exact subsections had been changed over time (since the incremental version tags were specific deltas in per-device configurations. You could probably do it in a more global way too - git commit ids, maybe? - but you also don't want to make the version string too wordy either). -e * based on ftp://ftp.cac.washington.edu/pub/config-generator/, but substantially enhanced beyond the last public domain version. I know I'd be really happy if the current version was ever open-sourced...
Re: Managing IOS Configuration Snippets
On 27 Feb 2014, at 12:46, Erik Muller er...@buh.org wrote: On 2/27/14, 12:21 , Suresh Ramasubramanian wrote: This has been around for several years now - http://sourceforge.net/projects/cisco-conf-rep/ But that's just archiving, like rancid, right? This is not any kind of sensible answer to the original question, but the general approach “give ops people a shell on a box with a rancid repository, encourage them to write scripts that do stuff” has the potential to cause all kinds of good things to happen faster than the time taken to organise a conference call to discuss requirements gathering for a “production” system. Examples, as ever: http://www.nanog.org/meetings/nanog26/presentations/stephen.pdf ftp://ftp.isc.org/isc/toolmakers/ (It’s not pretty. But it’s not supposed to be pretty.) Joe
Re: Managing IOS Configuration Snippets
On 2/27/14, 15:52 , Joe Abley wrote: This is not any kind of sensible answer to the original question, but the general approach “give ops people a shell on a box with a rancid repository, encourage them to write scripts that do stuff” has the potential to cause all kinds of good things to happen faster than the time taken to organise a conference call to discuss requirements gathering for a “production” system. +1000. And that applies equally to the backend. I have yet to meet a fancy, integrated, database-driven configuration management system that can beat a bunch of flat files and a few perl scripts. Hackability of a system can be a definite virtue here. -e
Re: Managing IOS Configuration Snippets
Definitely. Depends what form the database takes - I don't think SQL is the right answer here. Sticking with flat files and perl scripts as much as possible is good guidance. I'm biased, but I'd go with Python: http://www.youtube.com/watch?v=EGK5jjyUBCQ --Simon On 27 February 2014 13:05, Erik Muller er...@buh.org wrote: On 2/27/14, 15:52 , Joe Abley wrote: This is not any kind of sensible answer to the original question, but the general approach give ops people a shell on a box with a rancid repository, encourage them to write scripts that do stuff has the potential to cause all kinds of good things to happen faster than the time taken to organise a conference call to discuss requirements gathering for a production system. +1000. And that applies equally to the backend. I have yet to meet a fancy, integrated, database-driven configuration management system that can beat a bunch of flat files and a few perl scripts. Hackability of a system can be a definite virtue here. -e
Re: Managing IOS Configuration Snippets
Putting aside the fact that snippets aren’t a good way to conceptualize deployed router code, my gut still tells me to question the question here. The first is does this stuff change often enough to warrant a fancy versioning solution? I have yet to see NTP deployed in a different way than when I first learned to configure it. Next, when it does change how often is it not rolled out to every router. If NTP or CPP or SNMP or some other administrative option were configured differently across my network I would want to audit it and fix not version control. What if some of the configs don’t match the defined versions? It may be better to create standard templates and version them in SVN or GIT and then use config backups to track which devices have the standard configs. There are some for pay tools that can search for certain statements on various boxes and either alert or remediate when differences are found. On Feb 26, 2014, at 4:22 PM, Ryan Shea ryans...@google.com wrote: Howdy network operator cognoscenti, I'd love to hear your creative and workable solutions for a way to track in-line the configuration revisions you have on your cisco-like devices. Let me clearify/frame: You have a set of tested/approved configurations for your routers which use IOS style configuration. These configurations of course are always refined and updated. You break these pieces of configuration into logical sections, for example a configuration file for NTP configuration, a file for control plane filter and store these in some revision control system. Put aside for the moment whether this is a reasonable way to comprehend deployed configurations. What methods do some of you use to know which version of a configuration you have deployed to a given router for auditing and update purposes? Remarks are a convenient way to do this for ACLs - but I don't have similar mechanics for top level configurations. About a decade ago I thought I'd be super clever and encode versioning information into the snmp location - but that is just awful and there is a much better way everyone is using, right? Flexible commenting on other vendors/platforms make this a bit easier. Assume that this version encoding perfectly captures what is on the router and that no person is monkeying with the config... version 77 of the control plane filter is the same everywhere.
Re: Managing IOS Configuration Snippets
On Thu, Feb 27, 2014 at 8:38 PM, Keegan Holley no.s...@comcast.net wrote: Putting aside the fact that snippets aren't a good way to conceptualize deployed router code, my gut still tells me to question the question here. The first is does this stuff change often enough to warrant a fancy versioning solution? I have yet to see NTP deployed in a different way than when I first learned to configure it. Next, when it does change how often is it not rolled out to every router. If NTP or CPP or SNMP or some other administrative option were configured differently across my sure, so you're saying that a large bit (maybe) of the router config is 'one size fits all' and 'never changes' where 'never' is really 'very infrequently'. sure, agreed... but there are parts of the config that do change more frequently (depending on the network perhaps)... how do you go about seeing which version / setup is deployed EXCEPT by building a home-grown 'config parser' and seeing that 'what is deployed matches mostly what I have in my config store for this router/class-of-router/network' ? It's a shame that vendors of network equipment don't have to manage large networks of their own equipment under constrained opex environments (no fair comparing contracted work where you bill for time + materials, that's the wrong incentive set)... I bet that'd get them to fix stuff up right quick. network I would want to audit it and fix not version control. What if some of the configs don't match the defined versions? It may be better to create standard templates and version them in SVN or GIT and then use config backups to track which devices have the standard configs. There are some for pay tools that can search for certain statements on various boxes and either alert or remediate when differences are found. On Feb 26, 2014, at 4:22 PM, Ryan Shea ryans...@google.com wrote: Howdy network operator cognoscenti, I'd love to hear your creative and workable solutions for a way to track in-line the configuration revisions you have on your cisco-like devices. Let me clearify/frame: You have a set of tested/approved configurations for your routers which use IOS style configuration. These configurations of course are always refined and updated. You break these pieces of configuration into logical sections, for example a configuration file for NTP configuration, a file for control plane filter and store these in some revision control system. Put aside for the moment whether this is a reasonable way to comprehend deployed configurations. What methods do some of you use to know which version of a configuration you have deployed to a given router for auditing and update purposes? Remarks are a convenient way to do this for ACLs - but I don't have similar mechanics for top level configurations. About a decade ago I thought I'd be super clever and encode versioning information into the snmp location - but that is just awful and there is a much better way everyone is using, right? Flexible commenting on other vendors/platforms make this a bit easier. Assume that this version encoding perfectly captures what is on the router and that no person is monkeying with the config... version 77 of the control plane filter is the same everywhere.
Re: Managing IOS Configuration Snippets
On 2/26/2014 4:22 PM, Ryan Shea wrote: Howdy network operator cognoscenti, I'd love to hear your creative and workable solutions for a way to track in-line the configuration revisions you have on your cisco-like devices. Let me clearify/frame: You have a set of tested/approved configurations for your routers which use IOS style configuration. These configurations of course are always refined and updated. You break these pieces of configuration into logical sections, for example a configuration file for NTP configuration, a file for control plane filter and store these in some revision control system. Put aside for the moment whether this is a reasonable way to comprehend deployed configurations. What methods do some of you use to know which version of a configuration you have deployed to a given router for auditing and update purposes? Remarks are a convenient way to do this for ACLs - but I don't have similar mechanics for top level configurations. About a decade ago I thought I'd be super clever and encode versioning information into the snmp location - but that is just awful and there is a much better way everyone is using, right? Flexible commenting on other vendors/platforms make this a bit easier. Assume that this version encoding perfectly captures what is on the router and that no person is monkeying with the config... version 77 of the control plane filter is the same everywhere. I started a long email that really should just be a blog post. I need to get a blog or something. Short story is this: NETCONF is probably the future of change management on all types of routers and switches. It's not supported everywhere yet and is missing lots of features but they're working on it. Look at the talk given at NANOG60 for more information. There is a puppet module that is also incomplete. I'm not sure this is the right way to go (http://puppetlabs.com/blog/puppet-network-device-management) Most people roll their own solution. If you're looking to do that consider using augeas for parsing the configuration files. It can be really useful for documenting changes, and probably to diff parts of the config. You might also consider rabbitmq or another message queue to handle scheduling and deploying the changes. It can retry failed updates. You should work towards all or nothing commits (not all cisco gear supports this, but you can fake it in a couple of ways. Ultimately you want to rollback to a known good configuration if things go wrong) If you have money and want this right now: Consider looking at Tail-F's NCS, which according to marketing presentations appears to do everything I want right now. I'd like to believe them but I don't have any money so I can't test it out. :) Cheers, Robert
Re: Managing IOS Configuration Snippets
On 2/26/2014 5:37 PM, Robert Drake wrote: Most people roll their own solution. If you're looking to do that consider using augeas for parsing the configuration files. It can be really useful for documenting changes, and probably to diff parts of the config. You might also consider rabbitmq or another message queue to handle scheduling and deploying the changes. It can retry failed updates. You should work towards all or nothing commits (not all cisco gear supports this, but you can fake it in a couple of ways. Ultimately you want to rollback to a known good configuration if things go wrong) I should amend that even though I recommend all this I haven't used any of it for networking. I guess those are more shiny ball ideas than actual things I've used. We have perl scripts that wrap an in-house API to access our IPAM which generates initial configuration. The template files are a mix of m4 and Template::Toolkit. We use basically one-off perl scripts for auditing sections of the configs to find discrepancies. We use rancid to collect configs. We just started using netdot which is nice for topology discovery. TACACS and DHCP logs are parsed and stored in logstash. All of those tools provide the who, what, where and when but not the why. The why would require a bit more custom stuff and forcing people to use a frontend interface instead of directly touching the routers. We aren't ready for that yet.
Re: Managing IOS Configuration Snippets
On Wed, Feb 26, 2014 at 6:27 PM, Ryan Shea ryans...@google.com wrote: Robert - all great suggestions. Big cross-vendor configuration generation and deployment is outside the scope of what I was hoping for here. The goal is to have the version information somehow encoded into the configuration, and I'm not sure that NETCONF has anything to say about that matter. Certainly the same problem of which-versions-are-where exists in the puppet/chef world and there are platform specific ways to answer those puppet solves this by comparing a complete md5(file) with deployed md5(file)... not as simple to do that on: access-list 150 permit icmp any any access-list 150 permit tcp any eq 80 any access-list 150 deny ip any any it'd be super nice if you could grab out just the hermetic bit of config you care about, and md5sum() that, eh? provided your stored config was written out in the IOS version (specific?) spacing/etc manner, of course. questions. Deep analysis of the router configuration itself can give pretty strong hints about which version are deployed, but lets assume full config digestion and comparison is out of the question. From some off-list responses I am hearing that some folks do similar kludges with other text fields, wether they be remark/banner/snmp-foo/interface descriptions. this makes me sad... but go 'state of the art network equipment!' is it time to start asking vendors for more operable configuration storage and access?
Re: Managing IOS Configuration Snippets
We are evaluating a piece of software called Skybox: http://www.skyboxsecurity.com/ It's geared to security analytics, but it does allow you to define configurations that are expected on a device, what software version it is running, whether commands that aren't there are, and those that should be there aren't, e.t.c. It supports all major network equipment vendors, and also allows for simple or complex regular expressions that can be used to search configuration files more easily. It is an offline system, so all you do is regularly present it with a text file of the device's running configuration, and it will do the necessary checks per the policy you have defined. Based on the configuration files it has, it can also create a visual model of your network. Not something you'd rely on given you have other tools for that, but kind of cool, nonetheless. Worth a look, I'd say. Mark. signature.asc Description: This is a digitally signed message part.
