Re: [CentOS] best parallel / cluster SSH
I mean if you go from supplying perl and some perl-scripted functionality, you either have to drop the functionality or some engineer has to rewrite the code in a different language - something that usually isn't cheap. I've never tried perl2c - if such a thing exists it probably embeds most of perl as a library. The few things we needed we rewrote in BASH - it was pretty easy (I did most of it) -- “Don't eat anything you've ever seen advertised on TV” - Michael Pollan, author of In Defense of Food ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] best parallel / cluster SSH
On 2/5/2010 5:22 PM, Alan McKay wrote: On Fri, Feb 5, 2010 at 1:31 PM, Les Mikeselllesmikes...@gmail.com wrote: Can you put a realistic price on what the extra resources would cost these days? Clearly you've never worked for a large company if you even ask that question. A $1 difference in cost over 100,000 units sold is $100,000 in your pocket. I recall the first model of BCM we decided not to put a power switch on it for just this reason. What I meant by the price is how much the price was reduced for the consumer. Manufacturers taking away functionality and not passing on the savings isn't very interesting, but I suppose there's a point in volume where you could pay someone to re-write perl or java code in C or some close-to-the-metal language to save a few bytes of flash and RAM. Rewrite perl? You mean, like using perl2c? mark ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] best parallel / cluster SSH
m.r...@5-cent.us wrote: Can you put a realistic price on what the extra resources would cost these days? Clearly you've never worked for a large company if you even ask that question. A $1 difference in cost over 100,000 units sold is $100,000 in your pocket. I recall the first model of BCM we decided not to put a power switch on it for just this reason. What I meant by the price is how much the price was reduced for the consumer. Manufacturers taking away functionality and not passing on the savings isn't very interesting, but I suppose there's a point in volume where you could pay someone to re-write perl or java code in C or some close-to-the-metal language to save a few bytes of flash and RAM. Rewrite perl? You mean, like using perl2c? I mean if you go from supplying perl and some perl-scripted functionality, you either have to drop the functionality or some engineer has to rewrite the code in a different language - something that usually isn't cheap. I've never tried perl2c - if such a thing exists it probably embeds most of perl as a library. -- Les Mikesell lesmikes...@gmail.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] best parallel / cluster SSH
On Sat, 2010-02-06 at 17:26 -0600, Les Mikesell wrote: I mean if you go from supplying perl and some perl-scripted functionality, you either have to drop the functionality or some engineer has to rewrite the code in a different language The perl interpreter is already written in C so there's nothing to rewrite in that regard. -- MELVILLE THEATRE ~ Melville Sask ~ http://www.melvilletheatre.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] best parallel / cluster SSH
Am Donnerstag, den 04.02.2010, 19:31 +0100 schrieb Alan McKay: It depends on what you need to do. If you really have enough machines or long-running jobs that a shell loop through them isn't practical, you might want something higher-level like cfengine or puppet, or at least something running under cron to make them independent. cfengine or puppet (or something else - slackmaster?) are where I want to be eventually - but in the immediate term something like this would help a lot.e.g bouncing my 4 front-end apache servers on 4 different boxes. That sort of thing. I like clusterssh (also named cssh) and mussh for this purpose. Chris financial.com AG Munich head office/Hauptsitz München: Maria-Probst-Str. 19 | 80939 München | Germany Frankfurt branch office/Niederlassung Frankfurt: Messeturm | Friedrich-Ebert-Anlage 49 | 60327 Frankfurt | Germany Management board/Vorstand: Dr. Steffen Boehnert | Dr. Alexis Eisenhofer | Dr. Yann Samson | Matthias Wiederwach Supervisory board/Aufsichtsrat: Dr. Dr. Ernst zur Linden (chairman/Vorsitzender) Register court/Handelsregister: Munich – HRB 128 972 | Sales tax ID number/St.Nr.: DE205 370 553 ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] best parallel / cluster SSH
In (HPC) clustering pdsh is very popular. It's available in .tgz with spec-file and rebuilds nicely on c5 with rpmbuild -tb ... https://computing.llnl.gov/linux/pdsh.html Coming from the HPC world I've been a long time PDSH user. I believe it is available in rpmforge, so there is no need to rebuild it if you don't want. I highly recommend it. In addition to the examples already cited, you can build a text file of commonly used groups of nodes and just use that to point PDSH at. If you really, really want to get fancy you can... but for more advanced uses just peek at the docs. -geoff - Geoff Galitz Blankenheim NRW, Germany http://www.galitz.org/ http://german-way.com/blog/ ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] best parallel / cluster SSH
Alan McKay wrote: It depends on what you need to do. If you really have enough machines or long-running jobs that a shell loop through them isn't practical, you might want something higher-level like cfengine or puppet, or at least something running under cron to make them independent. cfengine or puppet (or something else - slackmaster?) are where I want to be eventually - but in the immediate term something like this would help a lot.e.g bouncing my 4 front-end apache servers on 4 different boxes. That sort of thing. I was actually going to start another configuration management redux thread as a follow up to a thread I started a few months ago. Hi This is another interesting tool: https://fedorahosted.org/func/ Regards mg. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] best parallel / cluster SSH
On 2/5/2010 8:44 AM, Marcelo M. Garcia wrote: Alan McKay wrote: It depends on what you need to do. If you really have enough machines or long-running jobs that a shell loop through them isn't practical, you might want something higher-level like cfengine or puppet, or at least something running under cron to make them independent. cfengine or puppet (or something else - slackmaster?) are where I want to be eventually - but in the immediate term something like this would help a lot.e.g bouncing my 4 front-end apache servers on 4 different boxes. That sort of thing. I was actually going to start another configuration management redux thread as a follow up to a thread I started a few months ago. Hi This is another interesting tool: https://fedorahosted.org/func/ Python just seems like something that should be avoided in system management tools. Remember having to do special case things like having to 'yum update python\* yum\*' sometimes to keep the rest of an update from breaking? -- Les Mikesell lesmikes...@gmail.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] best parallel / cluster SSH
On Fri, Feb 5, 2010 at 11:03 AM, Les Mikesell lesmikes...@gmail.com wrote: Python just seems like something that should be avoided in system management tools. Remember having to do special case things like having to 'yum update python\* yum\*' sometimes to keep the rest of an update from breaking? Yes, I feel the same way about perl! -- “Don't eat anything you've ever seen advertised on TV” - Michael Pollan, author of In Defense of Food ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] best parallel / cluster SSH
On Fri, Feb 5, 2010 at 11:03 AM, Les Mikesell lesmikes...@gmail.com wrote: Python just seems like something that should be avoided in system management tools. Remember having to do special case things like having to 'yum update python\* yum\*' sometimes to keep the rest of an update from breaking? Yes, I feel the same way about perl! Never had a problem with perl updates breaking anything. I do remember a few years back, when it seemed as though any time I tried to install or upgrade something that was in python, it *ALWAYS* wanted a different subrelease, and upgrading that would break everything else in python mark, not a python fan ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] best parallel / cluster SSH
On Fri, Feb 5, 2010 at 12:45 PM, m.r...@5-cent.us wrote: Never had a problem with perl updates breaking anything. I do remember a few years back, when it seemed as though any time I tried to install or upgrade something that was in python, it *ALWAYS* wanted a different subrelease, and upgrading that would break everything else in python No problems with updates - just problems assuming it is on ever box out there. We removed it from the Nortel BCM because the perl installation accounted for more than half the space on our embedded Linux. -- “Don't eat anything you've ever seen advertised on TV” - Michael Pollan, author of In Defense of Food ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] best parallel / cluster SSH
On 2/5/2010 11:30 AM, Alan McKay wrote: On Fri, Feb 5, 2010 at 11:03 AM, Les Mikeselllesmikes...@gmail.com wrote: Python just seems like something that should be avoided in system management tools. Remember having to do special case things like having to 'yum update python\* yum\*' sometimes to keep the rest of an update from breaking? Yes, I feel the same way about perl! How so? In decades of using perl, I've only seen one case where even major version revs did not maintain complete backwards compatibility in the core language, that being when @ in double-quoted strings started to be interpolated. But yes, if I were to expect perl to control my machines, I'd provision a separate instance of it for that purpose so system updates can't be suicidal. Sort of like using static libs in C programs that need to always work. -- Les Mikesell lesmikes...@gmail.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] best parallel / cluster SSH
On 2/5/2010 12:09 PM, Alan McKay wrote: On Fri, Feb 5, 2010 at 12:45 PM,m.r...@5-cent.us wrote: Never had a problem with perl updates breaking anything. I do remember a few years back, when it seemed as though any time I tried to install or upgrade something that was in python, it *ALWAYS* wanted a different subrelease, and upgrading that would break everything else in python No problems with updates - just problems assuming it is on ever box out there. We removed it from the Nortel BCM because the perl installation accounted for more than half the space on our embedded Linux. Can you put a realistic price on what the extra resources would cost these days? -- Les Mikesell lesmikes...@gmail.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] best parallel / cluster SSH
Les Mikesell wrote: We removed it from the Nortel BCM because the perl installation accounted for more than half the space on our embedded Linux. Can you put a realistic price on what the extra resources would cost these days? when you're running on an embedded single chip processor that has a fixed amount of flash and ram, and the board goes into 10 units or more? lots. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] best parallel / cluster SSH
On Fri, Feb 5, 2010 at 1:31 PM, Les Mikesell lesmikes...@gmail.com wrote: Can you put a realistic price on what the extra resources would cost these days? Clearly you've never worked for a large company if you even ask that question. A $1 difference in cost over 100,000 units sold is $100,000 in your pocket. I recall the first model of BCM we decided not to put a power switch on it for just this reason. -- “Don't eat anything you've ever seen advertised on TV” - Michael Pollan, author of In Defense of Food ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] best parallel / cluster SSH
On 2/5/2010 5:22 PM, Alan McKay wrote: On Fri, Feb 5, 2010 at 1:31 PM, Les Mikeselllesmikes...@gmail.com wrote: Can you put a realistic price on what the extra resources would cost these days? Clearly you've never worked for a large company if you even ask that question. A $1 difference in cost over 100,000 units sold is $100,000 in your pocket. I recall the first model of BCM we decided not to put a power switch on it for just this reason. What I meant by the price is how much the price was reduced for the consumer. Manufacturers taking away functionality and not passing on the savings isn't very interesting, but I suppose there's a point in volume where you could pay someone to re-write perl or java code in C or some close-to-the-metal language to save a few bytes of flash and RAM. -- Les Mikesell lesmikes...@gmail.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] best parallel / cluster SSH
On 2/4/2010 11:45 AM, Alan McKay wrote: Hey folks, I stumbled upon this while looking for something else http://www.linux.com/archive/feature/151340 And it is something I could actually really make use of. But right on that site they list 3 different ones, and so I'm wondering what all is out there and what I should use. It depends on what you need to do. If you really have enough machines or long-running jobs that a shell loop through them isn't practical, you might want something higher-level like cfengine or puppet, or at least something running under cron to make them independent. -- Les Mikesell lesmikes...@gmail.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] best parallel / cluster SSH
It depends on what you need to do. If you really have enough machines or long-running jobs that a shell loop through them isn't practical, you might want something higher-level like cfengine or puppet, or at least something running under cron to make them independent. cfengine or puppet (or something else - slackmaster?) are where I want to be eventually - but in the immediate term something like this would help a lot.e.g bouncing my 4 front-end apache servers on 4 different boxes. That sort of thing. I was actually going to start another configuration management redux thread as a follow up to a thread I started a few months ago. -- “Don't eat anything you've ever seen advertised on TV” - Michael Pollan, author of In Defense of Food ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] best parallel / cluster SSH
Alan McKay wrote:
I was actually going to start another configuration management redux
thread as a follow up to a thread I started a few months ago.
As Les mentioned, it's far more common in that situation to use
ssh key authentication and a for loop, if your ssh key has a pass
phrase use a ssh agent.
I still use it quite often even though I do have a fairly extensive
cfengine setup, sometimes I need something done right now such
as a mass restart and can't wait for cfengine to run on each host.
If you have servers say
web01 - web30
sample script to restart apache -
for i in `seq -w 1 30`; do ssh r...@web${i} /etc/init.d/httpd restart; done
nate
___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] best parallel / cluster SSH
On Thu, Feb 04, 2010 at 01:31:39PM -0500, Alan McKay wrote: It depends on what you need to do. If you really have enough machines or long-running jobs that a shell loop through them isn't practical, you might want something higher-level like cfengine or puppet, or at least something running under cron to make them independent. cfengine or puppet (or something else - slackmaster?) are where I want to be eventually - but in the immediate term something like this would help a lot.e.g bouncing my 4 front-end apache servers on 4 different boxes. That sort of thing. I was actually going to start another configuration management redux thread as a follow up to a thread I started a few months ago. We've tried all the ones in that article you mentioned, and are currently using classh - http://freshmeat.net/projects/classh - which is pretty nice, in spite of being labelled alpha. I've packaged it for c5 here: http://www.openfusion.com.au/mrepo/centos5-i386/RPMS.of/classh-0.092-1.of.el5.noarch.rpm if you'd like to give it a whirl. Cheers, Gavin ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] best parallel / cluster SSH
On 2/4/2010 12:31 PM, Alan McKay wrote: It depends on what you need to do. If you really have enough machines or long-running jobs that a shell loop through them isn't practical, you might want something higher-level like cfengine or puppet, or at least something running under cron to make them independent. cfengine or puppet (or something else - slackmaster?) are where I want to be eventually - but in the immediate term something like this would help a lot.e.g bouncing my 4 front-end apache servers on 4 different boxes. That sort of thing. Usually for that sort of thing I explicitly _don't_ want all of a group to be out of service at the same moment so I much prefer to: for host in host1 host2 host3 host4 do echo $host ssh $host service httpd restart done which doesn't take much more thought then doing it locally and if I expect to repeat it I can save the commands in a script. I was actually going to start another configuration management redux thread as a follow up to a thread I started a few months ago. I'm interested in the topic, but most of what I've seen just add complexity for no particular reason. Clustering ssh commands might be an exception for a certain subset of things that need to happen at approximately the same time on multiple machines (and you are willing to risk breaking them all at once with a typo). -- Les Mikesell lesmikes...@gmail.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] best parallel / cluster SSH
I've been using clusterit for several years for multiple small clusters. It works well and was easy to install. I believe I got the Fedora source RPM and rebuilt it for CentOS. On Thu, Feb 4, 2010 at 1:50 PM, Gavin Carr ga...@openfusion.com.au wrote: On Thu, Feb 04, 2010 at 01:31:39PM -0500, Alan McKay wrote: It depends on what you need to do. If you really have enough machines or long-running jobs that a shell loop through them isn't practical, you might want something higher-level like cfengine or puppet, or at least something running under cron to make them independent. cfengine or puppet (or something else - slackmaster?) are where I want to be eventually - but in the immediate term something like this would help a lot. e.g bouncing my 4 front-end apache servers on 4 different boxes. That sort of thing. I was actually going to start another configuration management redux thread as a follow up to a thread I started a few months ago. We've tried all the ones in that article you mentioned, and are currently using classh - http://freshmeat.net/projects/classh - which is pretty nice, in spite of being labelled alpha. I've packaged it for c5 here: http://www.openfusion.com.au/mrepo/centos5-i386/RPMS.of/classh-0.092-1.of.el5.noarch.rpm if you'd like to give it a whirl. Cheers, Gavin ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos Cheers, -- Wade ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] best parallel / cluster SSH
On 2/4/2010 12:45 PM, nate wrote:
Alan McKay wrote:
I was actually going to start another configuration management redux
thread as a follow up to a thread I started a few months ago.
As Les mentioned, it's far more common in that situation to use
ssh key authentication and a for loop, if your ssh key has a pass
phrase use a ssh agent.
I still use it quite often even though I do have a fairly extensive
cfengine setup, sometimes I need something done right now such
as a mass restart and can't wait for cfengine to run on each host.
If you have servers say
web01 - web30
sample script to restart apache -
for i in `seq -w 1 30`; do ssh r...@web${i} /etc/init.d/httpd restart; done
Or keep the lists of servers in files so you can apply something like:
for i in `cat web-hosts`
to any ssh command loop
There's a big gray area between strict automation and manually logging
in to every machine for maintenance and I find it helpful to layer the
simple tools and steps you already know.
But, if someone ever gets cross-platform config management right or at
least close enough that it is worth learning yet another description
language I'd be very interested. Cfengine v3 might be getting there but
the windows version seems to be only available in the commercial build.
--
Les Mikesell
lesmikes...@gmail.com
___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] best parallel / cluster SSH
On 02/04/2010 06:31 PM, Alan McKay wrote: cfengine or puppet (or something else - slackmaster?) are where I want slackmaster, i dont know about - but are you refering to 'slack' ? its a fairly easy way to get started, and is essentially a wrapper around rsync. takes about 2 min to get setup and started, and will do all the basic things including managing services etc. The original centos buildsystem, which was spread over 12 machines, was all managed using slack scripts :) - KB ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] best parallel / cluster SSH
On Thursday 04 February 2010, Alan McKay wrote: Hey folks, I stumbled upon this while looking for something else http://www.linux.com/archive/feature/151340 And it is something I could actually really make use of. But right on that site they list 3 different ones, and so I'm wondering what all is out there and what I should use. In (HPC) clustering pdsh is very popular. It's available in .tgz with spec-file and rebuilds nicely on c5 with rpmbuild -tb ... https://computing.llnl.gov/linux/pdsh.html A few examples of what pdsh can do: - pick hosts with compact expressions: -w n[3-10,44] - settable fanout (run on X hosts in parallel) - remote command timeout - pdcp command that allows the copying of file - nifty post-processor that compats output: $ pdsh -w n[1-3],n5 uname -r | dshbak -c n[1-3,5] 2.6.18-164.11.1.el5 /Peter Is there one that is part of the standard CentOS? thanks, -Alan signature.asc Description: This is a digitally signed message part. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
