Re: [Linux-PowerEdge] FYI using pdsh for mass execution (Re: dsu woes)

2018-01-11 Thread James A. Peltier 
Yes, I am more than well aware of pdsh while handling dozens of clusters. The 
point is, we have a process in place already that does not involve these kinds 
of changes because all other vendors understand that some processes need to 
change with high impact security issues arise. I've had to revoke certificates 
for every research across the nation due to heatbleed, so I know this isn't 
always easy. The fact remains that Dell should have processes to deal with 
these issues like other vendors do. Full stop. 

- On 11 Jan, 2018, at 05:58, PJ Welsh <pjwe...@gmail.com> wrote: 

| ...

|| Date: Wed, 10 Jan 2018 17:03:25 -0700
|| From: Stephen Dowdy < [ mailto:sdo...@ucar.edu | sdo...@ucar.edu ] >
|| To: Patrick Boutilier < [ mailto:bouti...@ednet.ns.ca | bouti...@ednet.ns.ca 
]
|| >, [ mailto:linux-poweredge@dell.com | linux-poweredge@dell.com ]
|| Subject: [Linux-PowerEdge] FYI using pdsh for mass execution (Re: dsu
|| woes)
|| Message-ID: < [ mailto:e21300c5-f4c4-5abc-b150-fc4963985...@ucar.edu |
|| e21300c5-f4c4-5abc-b150-fc4963985...@ucar.edu ] >
|| Content-Type: text/plain; charset=utf-8

|| On 01/10/2018 03:27 PM, Patrick Boutilier wrote:
||> What I would do is download the BIOS update file once and then create a 
script
||> to rsync it to the 1500 hosts and run the update. I presume you
|| > would?have?a?script?to?run?dsu?on?the?1500?hosts?anyway?

|| First: i gave up on using Dell's enterprise management tools due to constant
|| heartache/headache/frustration. It definitely makes me sad that these tools
|| continually change, but never actually get much better. (okay, it looks like
|| they are finally attacking the BASHisms in some of their scripts that borked
|| Debian/Ubuntu systems badly, but the continual lack of 
correctness/current-cy,
|| etc just pains me).

|| Second:

|| FYI, LLNL's 'pdsh' works great for this. requires ssh public key trust for
|| 'root' in my following examples to do the following (running DUPs requires
|| 'root'):
|| ** note that your public key should be offline until loaded in your ssh-agent
|| (oof, Meltdown/Spectre, sigh)
|| WARNING: as any tool that allows mass-execution, if you screw up, you've now
|| multiplied that screwup to a large number of systems, so always be careful.

|| If you have a file of hostnames already:

|| pdsh -lroot -g {file} 'command-to-run-on-all-remote-systems'
|| (file is usually dshgroup module selected so ~/.dsh/group/{file} or
|| /etc/dsh/group/{file})

| pdsh is a great tool... but with 1500 systems, I bet he knows about it.

| DSU should be fixed, of course!

| A critical or high impact release update for BIOS/firmware *.BIN files should
| cause the Dell magic internal process to update the DSU repo!

| Yes the bosses likely decided once-per-whatever is good enough, but clearly 
it's
| not (good enough). All distributions have the concept of security/severity for
| packages and updates in repositories; utilize those options. Keep your stable
| once-per-whatever but give these high impact/high profile updates some legs to
| move out more quickly and effectively for your Dell customers.

| DSU has grown for the better... continue the growth in a positive direction.

| Thanks
| PJWelsh

| ___
| Linux-PowerEdge mailing list
| Linux-PowerEdge@dell.com
| https://lists.us.dell.com/mailman/listinfo/linux-poweredge

-- 
James A. Peltier 
Manager - Research Computing Group | IT Services 
Simon Fraser University | ASB10973 
 University Dr., Burnaby, B.C. V5A 1S6 
T: 778.782.6573 | M: 604.365.6432 | sfu.ca/itservices 
Twitter: @sfu_it 
___
Linux-PowerEdge mailing list
Linux-PowerEdge@dell.com
https://lists.us.dell.com/mailman/listinfo/linux-poweredge

Re: [Linux-PowerEdge] FYI using pdsh for mass execution (Re: dsu woes)

2018-01-11 Thread PJ Welsh 
...

> Date: Wed, 10 Jan 2018 17:03:25 -0700
> From: Stephen Dowdy <sdo...@ucar.edu>
> To: Patrick Boutilier <bouti...@ednet.ns.ca>, linux-poweredge@dell.com
> Subject: [Linux-PowerEdge] FYI using pdsh for mass execution (Re: dsu
> woes)
> Message-ID: <e21300c5-f4c4-5abc-b150-fc4963985...@ucar.edu>
> Content-Type: text/plain; charset=utf-8
>
> On 01/10/2018 03:27 PM, Patrick Boutilier wrote:
> > What I would do is download the BIOS update file once and then create a
> script to rsync it to the 1500 hosts and run the update. I presume you
> would?have?a?script?to?run?dsu?on?the?1500?hosts?anyway?
>
> First:  i gave up on using Dell's enterprise management tools due to
> constant heartache/headache/frustration.  It definitely makes me sad that
> these tools continually change, but never actually get much better.  (okay,
> it looks like they are finally attacking the BASHisms in some of their
> scripts that borked Debian/Ubuntu systems badly, but the continual lack of
> correctness/current-cy, etc just pains me).
>
> Second:
>
> FYI, LLNL's 'pdsh' works great for this.  requires ssh public key trust
> for 'root' in my following examples to do the following (running DUPs
> requires 'root'):
> ** note that your public key should be offline until loaded in your
> ssh-agent (oof, Meltdown/Spectre, sigh)
> WARNING: as any tool that allows mass-execution, if you screw up, you've
> now multiplied that screwup to a large number of systems, so always be
> careful.
>
> If you have a file of hostnames already:
>
>pdsh -lroot -g {file} 'command-to-run-on-all-remote-systems'
> (file is usually dshgroup module selected so ~/.dsh/group/{file} or
> /etc/dsh/group/{file})
>
>
pdsh is a great tool... but with 1500 systems, I bet he knows about it.

DSU should be fixed, of course!

A critical or high impact release update for BIOS/firmware *.BIN files
should cause the Dell magic internal process to update the DSU repo!

Yes the bosses likely decided once-per-whatever is good enough, but clearly
it's not (good enough). All distributions have the concept of
security/severity for packages and updates in repositories; utilize those
options. Keep your stable once-per-whatever but give these high impact/high
profile updates some legs to move out more quickly and effectively for your
Dell customers.

DSU has grown for the better... continue the growth in a positive direction.

Thanks
PJWelsh
___
Linux-PowerEdge mailing list
Linux-PowerEdge@dell.com
https://lists.us.dell.com/mailman/listinfo/linux-poweredge

Re: [Linux-PowerEdge] FYI using pdsh for mass execution (Re: dsu woes)

2018-01-10 Thread Klaus Steden 
This may also be useful if you're looking for a parallel command execution
tool:

https://github.com/kt97679/one-ssh

On Wed, Jan 10, 2018 at 4:11 PM, Vincent S. Cojot 
wrote:

> Speaking of mass execution on thousands of hosts, I've used Perl 'netrun'
> to a great extent in the recent past. Note that ansible core does great
> things in this fashion as well but may be a little slower than pdsh or
> netrun.
> My 2c,
> Vincent
>
> Sent from TypeApp 
> On Jan 10, 2018, at 19:03, Stephen Dowdy  wrote:
>>
>> On 01/10/2018 03:27 PM, Patrick Boutilier wrote:
>>
>>>  What I would do is download the BIOS update file once and then create a 
>>> script to rsync it to the 1500 hosts and run the update. I presume you 
>>> would have a script to run dsu on the 1500 hosts anyway?
>>>
>>
>> First:  i gave up on using Dell's enterprise management tools due to 
>> constant heartache/headache/frustration.  It definitely makes me sad that 
>> these tools continually change, but never actually get much better.  (okay, 
>> it looks like they are finally attacking the BASHisms in some of their 
>> scripts that borked Debian/Ubuntu systems badly, but the continual lack of 
>> correctness/current-cy, etc just pains me).
>>
>> Second:
>>
>> FYI, LLNL's 'pdsh' works great for this.  requires ssh public key trust for 
>> 'root' in my following examples to do the following (running DUPs requires 
>> 'root'):
>> ** note that your public key should be offline until loaded in your 
>> ssh-agent (oof, Meltdown/Spectre, sigh)
>> WARNING: as any tool that allows mass-execution, if you screw up, you've now 
>> multiplied that screwup to a large number of systems, so always be careful.
>>
>> If you have a file of hostnames already:
>>
>>pdsh -lroot -g {file} 'command-to-run-on-all-remote-systems'
>> (file is usually dshgroup module selected so ~/.dsh/group/{file} or 
>> /etc/dsh/group/{file})
>>
>> If you wanna hit everything in /etc/hosts, instead:
>>
>>awk '$0!~/^(#.*| *)$/{print$2}' /etc/hosts | WCOLL=- pdsh -Rexec scp 
>> FOO-2.7.0.BIN  root@%h:/dev/shm/
>> explained: all non-comment/blank lines, print hostname (field 2), setup 
>> pdsh's WCOLL envvar as the file containing hostnames to stdin (-), use the 
>> exec module to scp your DUP.BIN file, substituting %h for each hostname 
>> successively.
>>awk '$0!~/^(#.*| *)$/{print$2}' /etc/hosts | WCOLL=- pdsh -lroot 
>> '/dev/shm/FOO-2.7.0.BIN -q'
>> now, issue 'ssh' session to all hosts to run the DUP.BIN with '-q'.  ('-q' 
>> doesn't display changelog or prompt to run, also won't reboot after 
>> completion automatically)
>>
>> Note that 'pdsh' fans-out commands, running "N" jobs simultaneously (default 
>> 32).  I limit mine to 8 so i can use some special gateway-hop syntax with 
>> custom ./ssh/config rules to bounce past the admin nodes on clusters into 
>> the backend compute nodes.  This avoids the default 'sshd' connection 
>> throttling limits (usually 10 simultaneous connections)  (e.g.  
>> cluster1-admin!!node3) using the ssh-config rule:
>>
>>   Host*!!*
>> GatewayPorts no
>> ProxyCommand $(h="%h";p="%p" ; echo ssh -W ${h##*\!\!}:%p -l root 
>> ${h%%\!\!*})
>>
>> It's much easier to use libgenders or dshgroup style files for this kind of 
>> thing (than /etc/hosts and awk, etc), so you can use attribute selectors 
>> (genders) like:
>>
>> gpsh -lroot -g 'model=poweredge_r730' 'do-something'
>> (it's up to you to create a genders file with the right attributes filled in)
>>
>> records in my genders file, as created from a scripted MySQL asset database 
>> extraction look like:
>>
>> host99 
>> name=host99,manu=dell,model=precision_t3400,hwtype=desktop,sn=XXX,os=debian_linux,status=in_use,user=godot,responsible=godot,purpose=user_room_linux,sa1=sdowdy,project=unknown,location=fl2-2094
>>
>> Unfortunately, 'genders' doesn't support REGEX :-(   but you can use regex 
>> selection on hostnames in pdsh (just not attributes), like:
>>
>> pdsh -lroot -g 'hwtype=desktop' -w '/engr-.*/' ...
>>
>> to only hit the systems that are desktops and filter-down to only names with 
>> "engr-" in them.
>>
>> --stephen
>>
>>
> ___
> Linux-PowerEdge mailing list
> Linux-PowerEdge@dell.com
> https://lists.us.dell.com/mailman/listinfo/linux-poweredge
>
>
___
Linux-PowerEdge mailing list
Linux-PowerEdge@dell.com
https://lists.us.dell.com/mailman/listinfo/linux-poweredge

Re: [Linux-PowerEdge] FYI using pdsh for mass execution (Re: dsu woes)

2018-01-10 Thread Vincent S. Cojot 
Speaking of mass execution on thousands of hosts, I've used Perl 'netrun' to a 
great extent in the recent past. Note that ansible core does great things in 
this fashion as well but may be a little slower than pdsh or netrun.
My 2c,
Vincent

⁣Sent from TypeApp ​

On Jan 10, 2018, 19:03, at 19:03, Stephen Dowdy  wrote:
>On 01/10/2018 03:27 PM, Patrick Boutilier wrote:
>> What I would do is download the BIOS update file once and then create
>a script to rsync it to the 1500 hosts and run the update. I presume
>you would have a script to run dsu on the 1500 hosts anyway?
>
>First:  i gave up on using Dell's enterprise management tools due to
>constant heartache/headache/frustration.  It definitely makes me sad
>that these tools continually change, but never actually get much
>better.  (okay, it looks like they are finally attacking the BASHisms
>in some of their scripts that borked Debian/Ubuntu systems badly, but
>the continual lack of correctness/current-cy, etc just pains me).
>
>Second:
>
>FYI, LLNL's 'pdsh' works great for this.  requires ssh public key trust
>for 'root' in my following examples to do the following (running DUPs
>requires 'root'):
>** note that your public key should be offline until loaded in your
>ssh-agent (oof, Meltdown/Spectre, sigh)
>WARNING: as any tool that allows mass-execution, if you screw up,
>you've now multiplied that screwup to a large number of systems, so
>always be careful.
>
>If you have a file of hostnames already:
>
>   pdsh -lroot -g {file} 'command-to-run-on-all-remote-systems'
>(file is usually dshgroup module selected so ~/.dsh/group/{file} or
>/etc/dsh/group/{file})
>
>If you wanna hit everything in /etc/hosts, instead:
>
>awk '$0!~/^(#.*| *)$/{print$2}' /etc/hosts | WCOLL=- pdsh -Rexec scp
>FOO-2.7.0.BIN  root@%h:/dev/shm/
>explained: all non-comment/blank lines, print hostname (field 2), setup
>pdsh's WCOLL envvar as the file containing hostnames to stdin (-), use
>the exec module to scp your DUP.BIN file, substituting %h for each
>hostname successively.
>awk '$0!~/^(#.*| *)$/{print$2}' /etc/hosts | WCOLL=- pdsh -lroot
>'/dev/shm/FOO-2.7.0.BIN -q'
>now, issue 'ssh' session to all hosts to run the DUP.BIN with '-q'.
>('-q' doesn't display changelog or prompt to run, also won't reboot
>after completion automatically)
>
>Note that 'pdsh' fans-out commands, running "N" jobs simultaneously
>(default 32).  I limit mine to 8 so i can use some special gateway-hop
>syntax with custom ./ssh/config rules to bounce past the admin nodes on
>clusters into the backend compute nodes.  This avoids the default
>'sshd' connection throttling limits (usually 10 simultaneous
>connections)  (e.g.  cluster1-admin!!node3) using the ssh-config rule:
>
>  Host*!!*
>GatewayPorts no
>ProxyCommand $(h="%h";p="%p" ; echo ssh -W ${h##*\!\!}:%p -l root
>${h%%\!\!*})
>
>It's much easier to use libgenders or dshgroup style files for this
>kind of thing (than /etc/hosts and awk, etc), so you can use attribute
>selectors (genders) like:
>
>gpsh -lroot -g 'model=poweredge_r730' 'do-something'
>(it's up to you to create a genders file with the right attributes
>filled in)
>
>records in my genders file, as created from a scripted MySQL asset
>database extraction look like:
>
>host99
>name=host99,manu=dell,model=precision_t3400,hwtype=desktop,sn=XXX,os=debian_linux,status=in_use,user=godot,responsible=godot,purpose=user_room_linux,sa1=sdowdy,project=unknown,location=fl2-2094
>
>Unfortunately, 'genders' doesn't support REGEX :-(   but you can use
>regex selection on hostnames in pdsh (just not attributes), like:
>
>pdsh -lroot -g 'hwtype=desktop' -w '/engr-.*/' ...
>
>to only hit the systems that are desktops and filter-down to only names
>with "engr-" in them.
>
>--stephen
>
>--
>Stephen Dowdy  -  Systems Administrator  -  NCAR/RAL
>303.497.2869   -  sdo...@ucar.edu-
>http://www.ral.ucar.edu/~sdowdy/
>
>___
>Linux-PowerEdge mailing list
>Linux-PowerEdge@dell.com
>https://lists.us.dell.com/mailman/listinfo/linux-poweredge
___
Linux-PowerEdge mailing list
Linux-PowerEdge@dell.com
https://lists.us.dell.com/mailman/listinfo/linux-poweredge

[Linux-PowerEdge] FYI using pdsh for mass execution (Re: dsu woes)

2018-01-10 Thread Stephen Dowdy 
On 01/10/2018 03:27 PM, Patrick Boutilier wrote:
> What I would do is download the BIOS update file once and then create a 
> script to rsync it to the 1500 hosts and run the update. I presume you would 
> have a script to run dsu on the 1500 hosts anyway?

First:  i gave up on using Dell's enterprise management tools due to constant 
heartache/headache/frustration.  It definitely makes me sad that these tools 
continually change, but never actually get much better.  (okay, it looks like 
they are finally attacking the BASHisms in some of their scripts that borked 
Debian/Ubuntu systems badly, but the continual lack of correctness/current-cy, 
etc just pains me).

Second:

FYI, LLNL's 'pdsh' works great for this.  requires ssh public key trust for 
'root' in my following examples to do the following (running DUPs requires 
'root'):
** note that your public key should be offline until loaded in your ssh-agent 
(oof, Meltdown/Spectre, sigh)
WARNING: as any tool that allows mass-execution, if you screw up, you've now 
multiplied that screwup to a large number of systems, so always be careful.

If you have a file of hostnames already:

   pdsh -lroot -g {file} 'command-to-run-on-all-remote-systems'
(file is usually dshgroup module selected so ~/.dsh/group/{file} or 
/etc/dsh/group/{file})

If you wanna hit everything in /etc/hosts, instead:

   awk '$0!~/^(#.*| *)$/{print$2}' /etc/hosts | WCOLL=- pdsh -Rexec scp 
FOO-2.7.0.BIN  root@%h:/dev/shm/
explained: all non-comment/blank lines, print hostname (field 2), setup pdsh's 
WCOLL envvar as the file containing hostnames to stdin (-), use the exec module 
to scp your DUP.BIN file, substituting %h for each hostname successively.
   awk '$0!~/^(#.*| *)$/{print$2}' /etc/hosts | WCOLL=- pdsh -lroot 
'/dev/shm/FOO-2.7.0.BIN -q'
now, issue 'ssh' session to all hosts to run the DUP.BIN with '-q'.  ('-q' 
doesn't display changelog or prompt to run, also won't reboot after completion 
automatically)

Note that 'pdsh' fans-out commands, running "N" jobs simultaneously (default 
32).  I limit mine to 8 so i can use some special gateway-hop syntax with 
custom ./ssh/config rules to bounce past the admin nodes on clusters into the 
backend compute nodes.  This avoids the default 'sshd' connection throttling 
limits (usually 10 simultaneous connections)  (e.g.  cluster1-admin!!node3) 
using the ssh-config rule:

  Host*!!*
GatewayPorts no
ProxyCommand $(h="%h";p="%p" ; echo ssh -W ${h##*\!\!}:%p -l root 
${h%%\!\!*})

It's much easier to use libgenders or dshgroup style files for this kind of 
thing (than /etc/hosts and awk, etc), so you can use attribute selectors 
(genders) like:

gpsh -lroot -g 'model=poweredge_r730' 'do-something'
(it's up to you to create a genders file with the right attributes filled in)

records in my genders file, as created from a scripted MySQL asset database 
extraction look like:

host99 
name=host99,manu=dell,model=precision_t3400,hwtype=desktop,sn=XXX,os=debian_linux,status=in_use,user=godot,responsible=godot,purpose=user_room_linux,sa1=sdowdy,project=unknown,location=fl2-2094

Unfortunately, 'genders' doesn't support REGEX :-(   but you can use regex 
selection on hostnames in pdsh (just not attributes), like:

pdsh -lroot -g 'hwtype=desktop' -w '/engr-.*/' ...

to only hit the systems that are desktops and filter-down to only names with 
"engr-" in them.

--stephen

-- 
Stephen Dowdy  -  Systems Administrator  -  NCAR/RAL
303.497.2869   -  sdo...@ucar.edu-  http://www.ral.ucar.edu/~sdowdy/

___
Linux-PowerEdge mailing list
Linux-PowerEdge@dell.com
https://lists.us.dell.com/mailman/listinfo/linux-poweredge