Re: Mass update deployment strategy
On 2007-01-09, Florian Weimer [EMAIL PROTECTED] wrote: * Javier Fernández-Sanguino Peña: If your installation where slightly bigger (maybe 100 systems) I would suggest you invest your time working with OVAL [1] and CVE [2]: a) deploy an OVAL agent at the nodes with apt-capabilities b) have a central OVAL server send new signatures to nodes so they can tell you wether they are vulnerable or not (and need to have a DSA applied or not) Does anyone publish Debian-specific OVAL signatures? Not to my knowledge. Do you think there is a need for them? No, too much beaucracy for too little gain. Cheers, Moritz -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Mass update deployment strategy
On Tue, Jan 09, 2007 at 08:08:36PM +0100, Florian Weimer wrote: Does anyone publish Debian-specific OVAL signatures? Do you think there is a need for them? Not that I know of, but I have a converter to OVAL signatures that can generate the XML files from the website contents. But somebody has to adapt the OVAL agent to do Debian checks too. I think there is a need for them, as there is currently no way to have a Debian systems integrate in heterogeneus patch management systems without too glue. Tools like update-notifier and debsecan are nice, but for desktop users or administrators handling a few systems, not for enterprise deployments. Regards Javier signature.asc Description: Digital signature
Re: Mass update deployment strategy
On Wed, Jan 10, 2007 at 07:23:36PM +0100, Moritz Muehlenhoff wrote: Do you think there is a need for them? No, too much beaucracy for too little gain. What bureaucracy? Unlike CVE names, each vendor can generate their own OVAL signatures. For example: http://people.redhat.com/mjc/oval/ for more information on Red Hat's OVAL strategy see: http://www.redhat.com/security/transparent/oval/ Regards Javier signature.asc Description: Digital signature
Re: Mass update deployment strategy
Dear George, A setup that works well, is to work with your own Debian and/or Ubuntu repository, to which you only commit (apt-move) packages when you tested and approved beforehand. On your test setup you will see exactly what the impact will be from update X on configuration Y and you will have a chance to script the answers to posed questions as well thus being able to fully automate the installation while maintaining an integrit infrastructure. This will also save quite a bit of bandwidth. One more advanced step (where you limit the SPOF for the installation server) is to use apt-proxy to get a peer2peer alike update mechanism. You may then configure your systems to download and install all updates at a set timely interval from your CRM-production mirror without thinking. If you setup two repo's, say production and testing, you could measure the impact of a (security)update on _one_ test system. Perhaps you could also combine its functionality with the suggested solution from Javier. A fun addition would be to use a hypvervisor (Xen) to run one instance of every deployed configuration from within your network on a separate virtual machine. Which adds even more advantages, because you can easily restore the state of your machines and you only need _one_ machine to measure the impact of updates. Have used this setup quite a few times, where there is always one (often combined with other services) update server per geographical site. Kind regards, Freek Kauffmann On 12/1/06, Javier Fernández-Sanguino Peña [EMAIL PROTECTED] wrote: On Mon, Nov 27, 2006 at 08:37:42PM +0100, mario wrote: Do you have a strategy or anything to automate this task a little more? The server farm is growing and i might have to look after 20 or 30 installations soon. I can already see myself updating ubuntu/debian installations all day long :(. Let me throw some ideas around... If your installation where slightly bigger (maybe 100 systems) I would suggest you invest your time working with OVAL [1] and CVE [2]: a) deploy an OVAL agent at the nodes with apt-capabilities b) have a central OVAL server send new signatures to nodes so they can tell you wether they are vulnerable or not (and need to have a DSA applied or not) c) priorise work based on the severity of the vulnerability (you can use both NIST's CVE DB [1] and the priorities set by the Debian Testing Security team in their tracker for this) d) request systems to update with a patch (remote ssh connection will do for this) Unfortunately, a) is not yet possible. I have working OVAL agents for Debian (i.e. they compile) but have not had the time yet to write an adapter for Apt (shouldn't be too difficult). I'm trying to finish the DSA to OVAL xml signature converter so that generating xml signatures from the DSAs published at the website will be a breeze, but I have not yet finished with that. Any help with that would be appreciated. The only thing I can find close to that is to use Nessus with Local Security Checks and use the feed which provides tests for Debian vulnerabilities (I believe this is possible with the free feed, but maybe they have changed it). This is hardly optimal as it requires the systems to be live in the network when you are 'scanning it'. You can also do it by hand, if you are so inclined: a) have the systems send you their status files when they are online to a central system b) have a central system pick up those files and compare that information with a database of known vulnerabilites in Debian c) priorise the systems and vulnerabilities based on the above criteria d) have the central system ask the other systems to install the patch from security.debian.org (or your local repository) based on c) The first two parts of that (by-hand) solution are already written in the scripts provided by Tiger (check the package source: systems/Linux/deb_checkadvisories and systems/Linux/2/update_advisories) Unfortunately, nobody has written off a free enterprise patch management for Debian. There are non-free commercial alternatives you can easily find, but I'm not going to publitize them in this mailing list (contact me in private if you are desperate looking for one). Regards Javier [1] http://oval.mitre.org [2] http://cve.mitre.org [1] Formerly ICAT, now it's the National Vulnerability Database -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (GNU/Linux) iD8DBQFFb3OtsandgtyBSwkRAosBAJ9dYAeZCMxCqbh4nZq0nVP6RZcq7ACfQdPv KgUoC58ZnelCJWGrO0o9Yi8= =hokl -END PGP SIGNATURE-
Re: Mass update deployment strategy
* Javier Fernández-Sanguino Peña: If your installation where slightly bigger (maybe 100 systems) I would suggest you invest your time working with OVAL [1] and CVE [2]: a) deploy an OVAL agent at the nodes with apt-capabilities b) have a central OVAL server send new signatures to nodes so they can tell you wether they are vulnerable or not (and need to have a DSA applied or not) Does anyone publish Debian-specific OVAL signatures? Do you think there is a need for them?
Re: Mass update deployment strategy
On Tue, Nov 28, 2006 at 05:32:35PM -0500, Joe Bouchard wrote: I would say that if you let your machines blindly to an apt-get update; apt-get upgrade every day, most of the time it won't be a problem, but someday it may be a problem and you might render half your cluster unbootable. There are various modifications to this blind update theme as others have suggested, but I think the basic issue remains. There will be some of those packages which ask a lot of questions. The thing I think If you know the answers to the questions, you can deploy the answers as well. See the preseeding section of the debian-installer package. I am getting ready to deploy hundreds of small embedded devices running Debian, and keeping them up to date is a potential nightmare, and I consider it carefully. First, there's no point in doing a blind update. Do an eyes-wide-open update: reserve a few of your devices in the lab as test units. When you see a package update come along that pertains to your setup, install it on a test unit. Figure out the answers to any questions. Run all your tests to see if it's still working properly. When you are convinced that it is good, move the package to your own repository. That should be the only repository these devices are configured to look at, and should contain every package they need, and none that they do not need. That repository is safe for auto-update, auto-upgrade. -dsr- -- .- -..- .-. --. ..- .-. --- -. --- .-.. - -. .-.. .--- ..- -. -.-- .-. ..-. ... -... . .-- .-. ..-. ..-. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Mass update deployment strategy
On Mon, Nov 27, 2006 at 08:37:42PM +0100, mario wrote: Do you have a strategy or anything to automate this task a little more? The server farm is growing and i might have to look after 20 or 30 installations soon. I can already see myself updating ubuntu/debian installations all day long :(. Let me throw some ideas around... If your installation where slightly bigger (maybe 100 systems) I would suggest you invest your time working with OVAL [1] and CVE [2]: a) deploy an OVAL agent at the nodes with apt-capabilities b) have a central OVAL server send new signatures to nodes so they can tell you wether they are vulnerable or not (and need to have a DSA applied or not) c) priorise work based on the severity of the vulnerability (you can use both NIST's CVE DB [1] and the priorities set by the Debian Testing Security team in their tracker for this) d) request systems to update with a patch (remote ssh connection will do for this) Unfortunately, a) is not yet possible. I have working OVAL agents for Debian (i.e. they compile) but have not had the time yet to write an adapter for Apt (shouldn't be too difficult). I'm trying to finish the DSA to OVAL xml signature converter so that generating xml signatures from the DSAs published at the website will be a breeze, but I have not yet finished with that. Any help with that would be appreciated. The only thing I can find close to that is to use Nessus with Local Security Checks and use the feed which provides tests for Debian vulnerabilities (I believe this is possible with the free feed, but maybe they have changed it). This is hardly optimal as it requires the systems to be live in the network when you are 'scanning it'. You can also do it by hand, if you are so inclined: a) have the systems send you their status files when they are online to a central system b) have a central system pick up those files and compare that information with a database of known vulnerabilites in Debian c) priorise the systems and vulnerabilities based on the above criteria d) have the central system ask the other systems to install the patch from security.debian.org (or your local repository) based on c) The first two parts of that (by-hand) solution are already written in the scripts provided by Tiger (check the package source: systems/Linux/deb_checkadvisories and systems/Linux/2/update_advisories) Unfortunately, nobody has written off a free enterprise patch management for Debian. There are non-free commercial alternatives you can easily find, but I'm not going to publitize them in this mailing list (contact me in private if you are desperate looking for one). Regards Javier [1] http://oval.mitre.org [2] http://cve.mitre.org [1] Formerly ICAT, now it's the National Vulnerability Database signature.asc Description: Digital signature
Re: Mass update deployment strategy
On Mon, Nov 27, 2006 at 03:37:22PM -0500, George Georgalis wrote: for n in host1 host2 hostz; do ssh [EMAIL PROTECTED] $ENV $UPD ; $UPG $UPC done Check out dsh and its option -c instead of this step :-) Marcin -- Marcin Owsiany [EMAIL PROTECTED] http://marcin.owsiany.pl/ GnuPG: 1024D/60F41216 FE67 DA2D 0ACA FC5E 3F75 D6F6 3A0D 8AA0 60F4 1216 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Mass update deployment strategy
I would say that if you let your machines blindly to an apt-get update; apt-get upgrade every day, most of the time it won't be a problem, but someday it may be a problem and you might render half your cluster unbootable. There are various modifications to this blind update theme as others have suggested, but I think the basic issue remains. There will be some of those packages which ask a lot of questions. The thing I think you really need to wonder about is kernel packages a few times I have used stock kernels (not ones I compiled), and when a new update comes out the apt-get upgrade tries to install the new kernel, update lilo.conf, run lilo, and advises you that reboot asap. My personal track record with this has been less than perfect, and a few times I've need to revert to an old kernel, or use an emergency boot CD to fix the problem and then I'm all set. Therefore, kernel upgrades are something I want to do manually at this point in my life, and I tend to stick with the same kernel for long periods of time. I am getting ready to deploy hundreds of small embedded devices running Debian, and keeping them up to date is a potential nightmare, and I consider it carefully. Here is my strategy thus far: - My devices are the same, hardware wise, and it's my intent to keep them the same software wise. - Remove anything I don't need. Anything that's purged won't need to be updated. A samba server doesn't need gcc, or X, or frozen bubble, or apache, or LaTex. Eliminate those packages, and you remove the maintenance concerns on those packages. - General security approaches which reduce exposure. Eliminate services, use tight firewall rules, network isolation, read-only filesystems, and all that. - Look at the security updates and decide for yourself how exposed you are, and if it's really necessary to upgrade a particular package right now. If they find a bug which is only exploitable at the console, and none of my systems have a console, I don't need to worry about it really, or at least I can put it off until I have a bunch that matter. Or in the case of some exotic (potential) risk with a race condition where my local users would have do something rediculously complex, and I know they aren't smart enough to know how to do that, I can weigh the cost/benefit of inaction and potentially postpone this patch until I really need it. - When I finally do have to make some updates, I will do a couple machines by hand, and then make sure it works, then write a script which hits each of the other boxes and does exactly the same. That's my plan, we'll see how well it works. These systems are small data collection appliances, with no proprietary data, and if one of these gets hijacked we have taken steps to prevent it from spreading, so the consequences of a vulnerability are rather small. If you have a public web server, that's a whole 'nother story. Good luck. Joe -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Mass update deployment strategy
Hello List, i am responsible for 10 (ubuntu and debian) installations so far. I have installed apticron which informs me about updates frequently. Actually, its that often that i sometimes need to invest 1h a day just doing updates. Do you have a strategy or anything to automate this task a little more? The server farm is growing and i might have to look after 20 or 30 installations soon. I can already see myself updating ubuntu/debian installations all day long :(. My installations are most of the time small firewalls and samba servers. Any comments or field reports about this? Thanks, Mario -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Fwd: Mass update deployment strategy
-- Forwarded message -- From: Manuel García [EMAIL PROTECTED] Date: Nov 27, 2006 3:46 PM Subject: Re: Mass update deployment strategy To: mario [EMAIL PROTECTED] Well, if every machine have the same hardware you may use systemimager to do the upgrade, read about systemimager in [1] [1] http://www.systemimager.org/documentation/ On 11/27/06, mario [EMAIL PROTECTED] wrote: Hello List, i am responsible for 10 (ubuntu and debian) installations so far. I have installed apticron which informs me about updates frequently. Actually, its that often that i sometimes need to invest 1h a day just doing updates. Do you have a strategy or anything to automate this task a little more? The server farm is growing and i might have to look after 20 or 30 installations soon. I can already see myself updating ubuntu/debian installations all day long :(. My installations are most of the time small firewalls and samba servers. Any comments or field reports about this? Thanks, Mario -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED] -- Manuel Garcia. Jefe de Informática CASEP. Administrador de redes Consultor independiente Debian GNU/Linux Testing codename Etch -- Manuel Garcia. Jefe de Informática CASEP. Administrador de redes Consultor independiente Debian GNU/Linux Testing codename Etch
Re: Mass update deployment strategy
On Mon, Nov 27, 2006 at 08:37:42PM +0100, mario wrote: Hello List, i am responsible for 10 (ubuntu and debian) installations so far. I have installed apticron which informs me about updates frequently. Actually, its that often that i sometimes need to invest 1h a day just doing updates. Do you have a strategy or anything to automate this task a little more? The server farm is growing and i might have to look after 20 or 30 installations soon. I can already see myself updating ubuntu/debian installations all day long :(. My installations are most of the time small firewalls and samba servers. Any comments or field reports about this? on your master computer you could run a script somthing like this... #!/bin/sh set -e set -x ENV=set -e export TERM=$TERM . /etc/profile UPD=echo hostname Updating Package Lists... apt-get -qq update || true UPG=apt-get upgrade --show-upgraded UPC=apt-get clean for n in host1 host2 hostz; do ssh [EMAIL PROTECTED] $ENV $UPD ; $UPG $UPC done don't forget to have ssh-agent working beforehand. // George -- George Georgalis, systems architect, administrator IXOYE -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
RE: Mass update deployment strategy
There is also a package called cron-apt which will automatically update your debian machines and send you an email regarding what it updated. ~Morgan -Original Message- From: George Georgalis [mailto:[EMAIL PROTECTED] Sent: Monday, November 27, 2006 3:37 PM To: debian-security@lists.debian.org Subject: Re: Mass update deployment strategy On Mon, Nov 27, 2006 at 08:37:42PM +0100, mario wrote: Hello List, i am responsible for 10 (ubuntu and debian) installations so far. I have installed apticron which informs me about updates frequently. Actually, its that often that i sometimes need to invest 1h a day just doing updates. Do you have a strategy or anything to automate this task a little more? The server farm is growing and i might have to look after 20 or 30 installations soon. I can already see myself updating ubuntu/debian installations all day long :(. My installations are most of the time small firewalls and samba servers. Any comments or field reports about this? on your master computer you could run a script somthing like this... #!/bin/sh set -e set -x ENV=set -e export TERM=$TERM . /etc/profile UPD=echo hostname Updating Package Lists... apt-get -qq update || true UPG=apt-get upgrade --show-upgraded UPC=apt-get clean for n in host1 host2 hostz; do ssh [EMAIL PROTECTED] $ENV $UPD ; $UPG $UPC done don't forget to have ssh-agent working beforehand. // George -- George Georgalis, systems architect, administrator IXOYE -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Mass update deployment strategy
On Mon, Nov 27, 2006 at 08:37:42PM +0100, mario wrote: i am responsible for 10 (ubuntu and debian) installations so far. I have installed apticron which informs me about updates frequently. Actually, its that often that i sometimes need to invest 1h a day just doing updates. Given the choice I'd much prefer identical distributions, even with a little pain. Since things differ between Ubuntu Debian (and Redhat/SuSE/etc). Having two or more security update schedules and two lots of testing is more painful. Do you have a strategy or anything to automate this task a little more? cfengine. I'm interested in puppet, but it wasn't (isn't yet?) stable at the time I started automation on a decent sized farm. Steve -- Debian GNU/Linux System Administration http://www.debian-administration.org/ signature.asc Description: Digital signature
Re: Mass update deployment strategy
On Mon, Nov 27, 2006 at 03:52:40PM -0500, Morgan Walker wrote: There is also a package called cron-apt which will automatically update your debian machines and send you an email regarding what it updated. And upgraded, if you really trust your package sources: $ cat /etc/cron-apt/action.d/9-dist-upgrade dist-upgrade -y -V -u -o Dpkg::Options::=--force-confold -Mikko -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
