Script 'mail_helper' called by obssrc
Hello community,
here is the log from the commit of package clustershell for openSUSE:Factory
checked in at 2021-11-09 23:54:47
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Comparing /work/SRC/openSUSE:Factory/clustershell (Old)
and /work/SRC/openSUSE:Factory/.clustershell.new.1890 (New)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "clustershell"
Tue Nov 9 23:54:47 2021 rev:7 rq:930223 version:1.8.4
Changes:
--------
--- /work/SRC/openSUSE:Factory/clustershell/clustershell.changes
2020-09-09 18:08:51.263575713 +0200
+++ /work/SRC/openSUSE:Factory/.clustershell.new.1890/clustershell.changes
2021-11-09 23:55:14.063971701 +0100
@@ -1,0 +2,15 @@
+Mon Nov 8 17:42:04 UTC 2021 - Stephane Thiell <sthi...@stanford.edu>
+
+- Update to upstream release 1.8.4:
+ * RangeSetND: fix padding info when slicing using __getitem__()
+ * Defaults: Allow out-of-tree worker modules
+ * NodeUtils: allow YAML list to declare node groups
+ * Tree: Use default local_worker and allow overriding Defaults
+ * Worker/Rsh: return maxrc properly for Rsh Worker
+ * xCAT binding: add support for spaces in group names
+ * CLI/Clush: Avoid python3 error with no stdin
+ * CLI/Clush: use os.read() in stdin thread
+ * CLI/Clush: Add maxrc option to clush.conf
+ * CLI/Display: Add support for NO_COLOR and CLICOLOR
+
+-------------------------------------------------------------------
Old:
----
ClusterShell-1.8.3.tar.gz
New:
----
ClusterShell-1.8.4.tar.gz
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Other differences:
------------------
++++++ clustershell.spec ++++++
--- /var/tmp/diff_new_pack.SWtu3I/_old 2021-11-09 23:55:14.551971949 +0100
+++ /var/tmp/diff_new_pack.SWtu3I/_new 2021-11-09 23:55:14.555971952 +0100
@@ -1,8 +1,8 @@
#
# spec file for package clustershell
#
-# Copyright (c) 2020 SUSE LLC
-# Copyright (c) 2017 Stephane Thiell <sthi...@stanford.edu>
+# Copyright (c) 2021 SUSE LLC
+# Copyright (c) 2021 Stephane Thiell <sthi...@stanford.edu>
#
# All modifications and additions to the file contributed by third parties
# remain the property of their copyright owners, unless otherwise agreed
@@ -47,7 +47,7 @@
%global srcname ClusterShell
Name: clustershell
-Version: 1.8.3
+Version: 1.8.4
Release: 1%{?dist}
Summary: Python framework for efficient cluster administration
License: LGPL-2.1-or-later
@@ -115,7 +115,6 @@
%description -n %{python3_pkgprefix}-%{name}
ClusterShell Python 3 module and related command line tools.
-
%prep
%setup -q -n %{srcname}-%{version}
@@ -197,6 +196,7 @@
%{python3_sitelib}/ClusterShell-*-py?.?.egg-info
%else
+
%files -n %{python3_pkgprefix}-%{name}
%if 0%{?rhel}
%defattr(-,root,root,-)
++++++ ClusterShell-1.8.3.tar.gz -> ClusterShell-1.8.4.tar.gz ++++++
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/ChangeLog
new/ClusterShell-1.8.4/ChangeLog
--- old/ClusterShell-1.8.3/ChangeLog 2019-12-07 21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/ChangeLog 2021-11-06 01:59:59.000000000 +0100
@@ -1,3 +1,27 @@
+2021-11-03 S. Thiell <sthi...@stanford.edu>
+
+ * Version 1.8.4 released. The main changes are listed below.
+
+ * RangeSetND: fix padding info when slicing using __getitem__() (#429)
+
+ * Defaults: Allow out-of-tree worker modules
+
+ * NodeUtils: allow YAML list to declare node groups (#438)
+
+ * Tree: Use default local_worker and allow overriding Defaults
+
+ * Worker/Rsh: return maxrc properly for Rsh Worker (#448)
+
+ * xCAT binding: add support for spaces in group names (#459)
+
+ * CLI/Clush: Avoid python3 error with no stdin (#460)
+
+ * CLI/Clush: use os.read() in stdin thread (#463)
+
+ * CLI/Clush: Add maxrc option to clush.conf (#451)
+
+ * CLI/Display: Add support for NO_COLOR and CLICOLOR (#428) (#432)
+
2019-12-01 S. Thiell <sthi...@stanford.edu>
* Version 1.8.3 released. The main changes are listed below.
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/PKG-INFO
new/ClusterShell-1.8.4/PKG-INFO
--- old/ClusterShell-1.8.3/PKG-INFO 2019-12-17 20:13:15.000000000 +0100
+++ new/ClusterShell-1.8.4/PKG-INFO 2021-11-06 23:20:43.000000000 +0100
@@ -1,12 +1,12 @@
Metadata-Version: 1.1
Name: ClusterShell
-Version: 1.8.3
+Version: 1.8.4
Summary: ClusterShell library and tools
Home-page: http://clustershell.sourceforge.net/
Author: Stephane Thiell
Author-email: sthi...@stanford.edu
License: LGPLv2+
-Download-URL:
http://sourceforge.net/projects/clustershell/files/clustershell/1.8.3/
+Download-URL:
http://sourceforge.net/projects/clustershell/files/clustershell/1.8.4/
Description: ClusterShell is an event-driven open source Python framework,
designed to run
local or distant commands in parallel on server farms or on large Linux
clusters. It will take care of common issues encountered on HPC
clusters, such
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/conf/clush.conf
new/ClusterShell-1.8.4/conf/clush.conf
--- old/ClusterShell-1.8.3/conf/clush.conf 2019-12-07 21:34:33.000000000
+0100
+++ new/ClusterShell-1.8.4/conf/clush.conf 2021-11-06 01:59:59.000000000
+0100
@@ -10,6 +10,7 @@
color: auto
fd_max: 8192
history_size: 100
+maxrc: no
node_count: yes
verbosity: 1
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/ClusterShell-1.8.3/conf/groups.conf.d/xcat.conf.example
new/ClusterShell-1.8.4/conf/groups.conf.d/xcat.conf.example
--- old/ClusterShell-1.8.3/conf/groups.conf.d/xcat.conf.example 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/conf/groups.conf.d/xcat.conf.example 2021-11-06
01:59:59.000000000 +0100
@@ -8,7 +8,7 @@
[xcat]
# list the nodes in the specified node group
-map: lsdef -s -t node $GROUP | cut -d' ' -f1
+map: lsdef -s -t node "$GROUP" | cut -d' ' -f1
# list all the nodes defined in the xCAT tables
all: lsdef -s -t node | cut -d' ' -f1
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/ClusterShell-1.8.3/conf/groups.d/cluster.yaml.example
new/ClusterShell-1.8.4/conf/groups.d/cluster.yaml.example
--- old/ClusterShell-1.8.3/conf/groups.d/cluster.yaml.example 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/conf/groups.d/cluster.yaml.example 2021-11-06
01:59:59.000000000 +0100
@@ -39,6 +39,11 @@
# and yes, ranges work for groups too!
old: '@rack[1,3]'
new: '@rack[2,4]'
+ # YAML lists
+ rack5:
+ - 'example[200-205]' # some comment about example[200-205]
+ - 'example245'
+ - 'example300,example[401-406]'
# Group source cpu:
# define groups @cpu:ivy, @cpu:hsw and @cpu:all
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/ClusterShell-1.8.3/doc/extras/vim/syntax/clushconf.vim
new/ClusterShell-1.8.4/doc/extras/vim/syntax/clushconf.vim
--- old/ClusterShell-1.8.3/doc/extras/vim/syntax/clushconf.vim 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/doc/extras/vim/syntax/clushconf.vim 2021-11-06
01:59:59.000000000 +0100
@@ -16,7 +16,7 @@
syn match clushComment ";.*$"
syn match clushHeader "\[\w\+\]"
-syn keyword clushKeys fanout command_timeout connect_timeout color
fd_max history_size node_count verbosity
+syn keyword clushKeys fanout command_timeout connect_timeout color
fd_max history_size node_count maxrc verbosity
syn keyword clushKeys ssh_user ssh_path ssh_options
syn keyword clushKeys rsh_path rcp_path rcp_options
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/doc/man/man1/clubak.1
new/ClusterShell-1.8.4/doc/man/man1/clubak.1
--- old/ClusterShell-1.8.3/doc/man/man1/clubak.1 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/doc/man/man1/clubak.1 2021-11-06
01:59:59.000000000 +0100
@@ -1,6 +1,6 @@
.\" Man page generated from reStructuredText.
.
-.TH CLUBAK 1 "2019-12-01" "1.8.3" "ClusterShell User Manual"
+.TH CLUBAK 1 "2021-11-03" "1.8.4" "ClusterShell User Manual"
.SH NAME
clubak \- format output from clush/pdsh-like output and more
.
@@ -87,7 +87,7 @@
message tree trace mode; switch to enable \fBClusterShell.MsgTree\fP trace
mode, all keys/nodes being kept for each message element of the tree, thus
allowing special output gathering
.TP
.BI \-\-color\fB= WHENCOLOR
-whether to use ANSI colors to surround node or nodeset prefix/header with
escape sequences to display them in color on the terminal. \fIWHENCOLOR\fP is
\fBnever\fP, \fBalways\fP or \fBauto\fP (which use color if standard output
refers to a terminal). Color is set to [34m (blue foreground text) and cannot
be modified.
+\fBclush\fP can use NO_COLOR, CLICOLOR and CLICOLOR_FORCE environment
variables. \fB\-\-color\fP command line option always takes precedence over
environment variables. NO_COLOR takes precedence over CLICOLOR_FORCE which
takes precedence over CLICOLOR. \fB\-\-color\fP tells whether to use ANSI
colors to surround node or nodeset prefix/header with escape sequences to
display them in color on the terminal. \fIWHENCOLOR\fP is \fBnever\fP,
\fBalways\fP or \fBauto\fP (which use color if standard output refers to a
terminal). Color is set to [34m (blue foreground text) and cannot be modified.
.TP
.B \-\-diff
show diff between gathered outputs
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/doc/man/man1/clush.1
new/ClusterShell-1.8.4/doc/man/man1/clush.1
--- old/ClusterShell-1.8.3/doc/man/man1/clush.1 2019-12-07 21:34:33.000000000
+0100
+++ new/ClusterShell-1.8.4/doc/man/man1/clush.1 2021-11-06 01:59:59.000000000
+0100
@@ -1,6 +1,6 @@
.\" Man page generated from reStructuredText.
.
-.TH CLUSH 1 "2019-12-01" "1.8.3" "ClusterShell User Manual"
+.TH CLUSH 1 "2021-11-03" "1.8.4" "ClusterShell User Manual"
.SH NAME
clush \- execute shell commands on a cluster
.
@@ -234,11 +234,11 @@
.B \-r\fP,\fB \-\-regroup
fold nodeset using node groups
.TP
-.B \-S
+.B \-S\fP,\fB \-\-maxrc
return the largest of command return codes
.TP
.BI \-\-color\fB= WHENCOLOR
-whether to use ANSI colors to surround node or nodeset prefix/header with
escape sequences to display them in color on the terminal. \fIWHENCOLOR\fP is
\fBnever\fP, \fBalways\fP or \fBauto\fP (which use color if standard
output/error refer to a terminal). Colors are set to [34m (blue foreground
text) for stdout and [31m (red foreground text) for stderr, and cannot be
modified.
+\fBclush\fP can use NO_COLOR, CLICOLOR and CLICOLOR_FORCE environment
variables. NO_COLOR takes precedence over CLICOLOR_FORCE which takes precedence
over CLICOLOR. When \fB\-\-color\fP option is used these environment variables
are not taken into account. \fB\-\-color\fP tells whether to use ANSI colors to
surround node or nodeset prefix/header with escape sequences to display them in
color on the terminal. \fIWHENCOLOR\fP is \fBnever\fP, \fBalways\fP or
\fBauto\fP (which use color if standard output/error refer to a terminal).
Colors are set to [34m (blue foreground text) for stdout and [31m (red
foreground text) for stderr, and cannot be modified.
.TP
.B \-\-diff
show diff between common outputs (find the best reference output by focusing
on largest nodeset and also smaller command return code)
@@ -281,7 +281,7 @@
limit time for command to run on the node
.TP
.BI \-R \ WORKER\fP,\fB \ \-\-worker\fB= WORKER
-worker name to use for connection (\fBexec\fP, \fBssh\fP, \fBrsh\fP,
\fBpdsh\fP), default is \fBssh\fP
+worker name to use for connection (\fBexec\fP, \fBssh\fP, \fBrsh\fP,
\fBpdsh\fP, or the name of a Python worker module), default is \fBssh\fP
.TP
.BI \-\-remote\fB= REMOTE
whether to enable remote execution: in tree mode, \(aqyes\(aq forces
connections to the leaf nodes for execution, \(aqno\(aq establishes connections
up to the leaf parent nodes for execution (default is \(aqyes\(aq)
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/doc/man/man1/nodeset.1
new/ClusterShell-1.8.4/doc/man/man1/nodeset.1
--- old/ClusterShell-1.8.3/doc/man/man1/nodeset.1 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/doc/man/man1/nodeset.1 2021-11-06
01:59:59.000000000 +0100
@@ -1,6 +1,6 @@
.\" Man page generated from reStructuredText.
.
-.TH NODESET 1 "2019-12-01" "1.8.3" "ClusterShell User Manual"
+.TH NODESET 1 "2021-11-03" "1.8.4" "ClusterShell User Manual"
.SH NAME
nodeset \- compute advanced nodeset operations
.
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/doc/man/man5/clush.conf.5
new/ClusterShell-1.8.4/doc/man/man5/clush.conf.5
--- old/ClusterShell-1.8.3/doc/man/man5/clush.conf.5 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/doc/man/man5/clush.conf.5 2021-11-06
01:59:59.000000000 +0100
@@ -1,6 +1,6 @@
.\" Man page generated from reStructuredText.
.
-.TH CLUSH.CONF 5 "2019-12-01" "1.8.3" "ClusterShell User Manual"
+.TH CLUSH.CONF 5 "2021-11-03" "1.8.4" "ClusterShell User Manual"
.SH NAME
clush.conf \- Configuration file for clush
.
@@ -82,12 +82,16 @@
( connect_timeout + command_timeout ). If set to \fI0\fP, no timeout occurs.
.TP
.B color
-Whether to use ANSI colors to surround node or nodeset prefix/header with
-escape sequences to display them in color on the terminal. Valid arguments
-are \fBnever\fP, \fBalways\fP or \fBauto\fP (which use color if standard
-output/error refer to a terminal). Colors are set to [34m (blue foreground
-text) for stdout and [31m (red foreground text) for stderr, and cannot be
-modified.
+\fBclush\fP can use NO_COLOR, CLICOLOR and CLICOLOR_FORCE
+environment variables. NO_COLOR takes precedence over CLICOLOR_FORCE which
+takes precedence over CLICOLOR. When the option is set in configuration file
+environment variables are taken into account only with \fIauto\fP argument.
+\fBcolor\fP tells whether to use ANSI colors to surround node or nodeset
+prefix/header with escape sequences to display them in color on the terminal.
+Valid arguments are \fBnever\fP, \fBalways\fP or \fBauto\fP (which use color if
+standard output/error refer to a terminal). Colors are set to [34m (blue
+foreground text) for stdout and [31m (red foreground text) for stderr, and
+cannot be modified.
.TP
.B fd_max
Maximum number of open file descriptors permitted per clush process (soft
@@ -103,6 +107,9 @@
Should \fBclush\fP display additional (node count) information in buffer
header? (\fIyes\fP/\fIno\fP)
.TP
+.B maxrc
+Should \fBclush\fP return the largest of command return codes? (yes/no)
+.TP
.B verbosity
Set the verbosity level: \fI0\fP (quiet), \fI1\fP (default), \fI2\fP (verbose)
or more
(debug).
@@ -145,6 +152,7 @@
history_size: 100
color: auto
fd_max: 10240
+maxrc: no
node_count: yes
.fi
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/doc/man/man5/groups.conf.5
new/ClusterShell-1.8.4/doc/man/man5/groups.conf.5
--- old/ClusterShell-1.8.3/doc/man/man5/groups.conf.5 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/doc/man/man5/groups.conf.5 2021-11-06
01:59:59.000000000 +0100
@@ -1,6 +1,6 @@
.\" Man page generated from reStructuredText.
.
-.TH GROUPS.CONF 5 "2019-12-01" "1.8.3" "ClusterShell User Manual"
+.TH GROUPS.CONF 5 "2021-11-03" "1.8.4" "ClusterShell User Manual"
.SH NAME
groups.conf \- Configuration file for ClusterShell node groups
.
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/doc/sphinx/conf.py
new/ClusterShell-1.8.4/doc/sphinx/conf.py
--- old/ClusterShell-1.8.3/doc/sphinx/conf.py 2019-12-07 21:34:33.000000000
+0100
+++ new/ClusterShell-1.8.4/doc/sphinx/conf.py 2021-11-06 01:59:59.000000000
+0100
@@ -48,9 +48,9 @@
# built documents.
#
# The short X.Y version.
-version = '1.8.3'
+version = '1.8.4'
# The full version, including alpha/beta/rc tags.
-release = '1.8.3'
+release = '1.8.4'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/doc/sphinx/config.rst
new/ClusterShell-1.8.4/doc/sphinx/config.rst
--- old/ClusterShell-1.8.3/doc/sphinx/config.rst 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/doc/sphinx/config.rst 2021-11-06
01:59:59.000000000 +0100
@@ -64,6 +64,13 @@
| node_count | Should *clush* display additional (node count) |
| | information in buffer header? (yes/no) |
+-----------------+----------------------------------------------------+
+| maxrc | Should *clush* return the largest of command |
+| | return codes? (yes/no) |
+| | If set to no (the default), *clush* exit status |
+| | gives no information about command return codes, |
+| | but rather reports on *clush* execution itself |
+| | (zero indicating a successful run). |
++-----------------+----------------------------------------------------+
| verbosity | Set the verbosity level: 0 (quiet), 1 (default), |
| | 2 (verbose) or more (debug). |
+-----------------+----------------------------------------------------+
@@ -236,6 +243,11 @@
compute: 'node[0001-0288]'
gpu: 'node[0001-0008]'
+ servers: # example of yaml list syntax for
nodes
+ - 'server001' # in a group
+ - 'server002,server101'
+ - 'server[003-006]'
+
cpu_only: '@compute!@gpu' # example of inline set operation
# define group @cpu_only with
node[0009-0288]
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/doc/sphinx/release.rst
new/ClusterShell-1.8.4/doc/sphinx/release.rst
--- old/ClusterShell-1.8.3/doc/sphinx/release.rst 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/doc/sphinx/release.rst 2021-11-06
01:59:59.000000000 +0100
@@ -13,6 +13,26 @@
.. warning:: Support for Python 2.5 and below has been dropped in this version.
+Version 1.8.4
+^^^^^^^^^^^^^
+
+This version contains a few bug fixes and improvements:
+
+* allow out-of-tree worker modules
+
+* use default local_worker and allow overriding :ref:`defaults-config` (tree
mode)
+
+* return maxrc properly in the case of the Rsh Worker
+
+* :ref:`clush-tool`: improve stdin support with Python 3
+
+* :ref:`clush-tool`: add maxrc option to :ref:`clush.conf <clush-config>`
+
+* :ref:`clush-tool`: add support for NO_COLOR and CLICOLOR
+
+For more details, please have a look at `GitHub Issues for 1.8.4 milestone`_.
+
+
Version 1.8.3
^^^^^^^^^^^^^
@@ -564,6 +584,7 @@
.. _GitHub Issues for 1.8.1 milestone:
https://github.com/cea-hpc/clustershell/issues?utf8=%E2%9C%93&q=is%3Aissue+milestone%3A1.8.1
.. _GitHub Issues for 1.8.2 milestone:
https://github.com/cea-hpc/clustershell/issues?utf8=%E2%9C%93&q=is%3Aissue+milestone%3A1.8.2
.. _GitHub Issues for 1.8.3 milestone:
https://github.com/cea-hpc/clustershell/issues?utf8=%E2%9C%93&q=is%3Aissue+milestone%3A1.8.3
+.. _GitHub Issues for 1.8.4 milestone:
https://github.com/cea-hpc/clustershell/issues?utf8=%E2%9C%93&q=is%3Aissue+milestone%3A1.8.4
.. _LGPL v2.1+: https://www.gnu.org/licenses/old-licenses/lgpl-2.1.en.html
.. _CeCILL-C V1: http://www.cecill.info/licences/Licence_CeCILL-C_V1-en.html
.. _xCAT: https://xcat.org/
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/doc/sphinx/tools/clush.rst
new/ClusterShell-1.8.4/doc/sphinx/tools/clush.rst
--- old/ClusterShell-1.8.3/doc/sphinx/tools/clush.rst 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/doc/sphinx/tools/clush.rst 2021-11-06
01:59:59.000000000 +0100
@@ -589,6 +589,8 @@
---------------
ok
+NO_COLOR, CLICOLOR_FORCE??and CLICOLOR environment variables can also
+be used to change the way *clush* uses colors to display messages.
.. _clush-worker:
@@ -633,6 +635,8 @@
installed; doesn't provide write support (eg. you cannot ``cat file | clush
--worker pdsh``); it is primarily an 1-to-n worker example.
+Worker modules distributed outside of ClusterShell are also supported by
+specifying the case-sensitive full Python module name of a worker module.
.. [#] LLNL parallel remote shell utility
(https://computing.llnl.gov/linux/pdsh.html)
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/doc/txt/clubak.txt
new/ClusterShell-1.8.4/doc/txt/clubak.txt
--- old/ClusterShell-1.8.3/doc/txt/clubak.txt 2019-12-07 21:34:33.000000000
+0100
+++ new/ClusterShell-1.8.4/doc/txt/clubak.txt 2021-11-06 01:59:59.000000000
+0100
@@ -7,9 +7,9 @@
--------------------------------------------------
:Author: Stephane Thiell <sthi...@stanford.edu>
-:Date: 2019-12-01
+:Date: 2021-11-03
:Copyright: GNU Lesser General Public License version 2.1 or later (LGPLv2.1+)
-:Version: 1.8.3
+:Version: 1.8.4
:Manual section: 1
:Manual group: ClusterShell User Manual
@@ -57,7 +57,7 @@
node / line content separator string (default: `:`)
-F, --fast faster but memory hungry mode (preload all messages per node)
-T, --tree message tree trace mode; switch to enable
``ClusterShell.MsgTree`` trace mode, all keys/nodes being kept for each message
element of the tree, thus allowing special output gathering
---color=WHENCOLOR whether to use ANSI colors to surround node or nodeset
prefix/header with escape sequences to display them in color on the terminal.
*WHENCOLOR* is ``never``, ``always`` or ``auto`` (which use color if standard
output refers to a terminal). Color is set to [34m (blue foreground text) and
cannot be modified.
+--color=WHENCOLOR ``clush`` can use NO_COLOR, CLICOLOR and CLICOLOR_FORCE
environment variables. ``--color`` command line option always takes precedence
over environment variables. NO_COLOR takes precedence over CLICOLOR_FORCE which
takes precedence over CLICOLOR. ``--color`` tells whether to use ANSI colors to
surround node or nodeset prefix/header with escape sequences to display them in
color on the terminal. *WHENCOLOR* is ``never``, ``always`` or ``auto`` (which
use color if standard output refers to a terminal). Color is set to [34m (blue
foreground text) and cannot be modified.
--diff show diff between gathered outputs
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/doc/txt/cluset.txt
new/ClusterShell-1.8.4/doc/txt/cluset.txt
--- old/ClusterShell-1.8.3/doc/txt/cluset.txt 2019-12-07 21:34:33.000000000
+0100
+++ new/ClusterShell-1.8.4/doc/txt/cluset.txt 2021-11-06 01:59:59.000000000
+0100
@@ -7,9 +7,9 @@
--------------------------------------------
:Author: Stephane Thiell <sthi...@stanford.edu>
-:Date: 2019-12-01
+:Date: 2021-11-03
:Copyright: GNU Lesser General Public License version 2.1 or later (LGPLv2.1+)
-:Version: 1.8.3
+:Version: 1.8.4
:Manual section: 1
:Manual group: ClusterShell User Manual
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/doc/txt/clush.conf.txt
new/ClusterShell-1.8.4/doc/txt/clush.conf.txt
--- old/ClusterShell-1.8.3/doc/txt/clush.conf.txt 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/doc/txt/clush.conf.txt 2021-11-06
01:59:59.000000000 +0100
@@ -7,9 +7,9 @@
------------------------------
:Author: Stephane Thiell, <sthi...@stanford.edu>
-:Date: 2019-12-01
+:Date: 2021-11-03
:Copyright: GNU Lesser General Public License version 2.1 or later (LGPLv2.1+)
-:Version: 1.8.3
+:Version: 1.8.4
:Manual section: 5
:Manual group: ClusterShell User Manual
@@ -54,12 +54,16 @@
ClusterShell library ensures that any commands complete in less than
( connect_timeout + command_timeout ). If set to *0*, no timeout occurs.
color
- Whether to use ANSI colors to surround node or nodeset prefix/header with
- escape sequences to display them in color on the terminal. Valid arguments
- are ``never``, ``always`` or ``auto`` (which use color if standard
- output/error refer to a terminal). Colors are set to [34m (blue foreground
- text) for stdout and [31m (red foreground text) for stderr, and cannot be
- modified.
+ ``clush`` can use NO_COLOR, CLICOLOR and CLICOLOR_FORCE
+ environment variables. NO_COLOR takes precedence over CLICOLOR_FORCE which
+ takes precedence over CLICOLOR. When the option is set in configuration file
+ environment variables are taken into account only with `auto` argument.
+ ``color`` tells whether to use ANSI colors to surround node or nodeset
+ prefix/header with escape sequences to display them in color on the terminal.
+ Valid arguments are ``never``, ``always`` or ``auto`` (which use color if
+ standard output/error refer to a terminal). Colors are set to [34m (blue
+ foreground text) for stdout and [31m (red foreground text) for stderr, and
+ cannot be modified.
fd_max
Maximum number of open file descriptors permitted per clush process (soft
resource limit for open files). This limit can never exceed the system
@@ -71,6 +75,8 @@
node_count
Should ``clush`` display additional (node count) information in buffer
header? (`yes`/`no`)
+maxrc
+ Should ``clush`` return the largest of command return codes? (yes/no)
verbosity
Set the verbosity level: `0` (quiet), `1` (default), `2` (verbose) or more
(debug).
@@ -108,6 +114,7 @@
| history_size: 100
| color: auto
| fd_max: 10240
+| maxrc: no
| node_count: yes
|
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/doc/txt/clush.txt
new/ClusterShell-1.8.4/doc/txt/clush.txt
--- old/ClusterShell-1.8.3/doc/txt/clush.txt 2019-12-07 21:34:33.000000000
+0100
+++ new/ClusterShell-1.8.4/doc/txt/clush.txt 2021-11-06 01:59:59.000000000
+0100
@@ -7,9 +7,9 @@
-----------------------------------
:Author: Stephane Thiell <sthi...@stanford.edu>
-:Date: 2019-12-01
+:Date: 2021-11-03
:Copyright: GNU Lesser General Public License version 2.1 or later (LGPLv2.1+)
-:Version: 1.8.3
+:Version: 1.8.4
:Manual section: 1
:Manual group: ClusterShell User Manual
@@ -159,8 +159,8 @@
-b, --dshbak display gathered results in a dshbak-like way (note: it
will only try to aggregate the output of commands with same return codes)
-B like -b but including standard error
-r, --regroup fold nodeset using node groups
- -S return the largest of command return codes
- --color=WHENCOLOR whether to use ANSI colors to surround node or nodeset
prefix/header with escape sequences to display them in color on the terminal.
*WHENCOLOR* is ``never``, ``always`` or ``auto`` (which use color if standard
output/error refer to a terminal). Colors are set to [34m (blue foreground
text) for stdout and [31m (red foreground text) for stderr, and cannot be
modified.
+ -S, --maxrc return the largest of command return codes
+ --color=WHENCOLOR ``clush`` can use NO_COLOR, CLICOLOR and CLICOLOR_FORCE
environment variables. NO_COLOR takes precedence over CLICOLOR_FORCE which
takes precedence over CLICOLOR. When ``--color`` option is used these
environment variables are not taken into account. ``--color`` tells whether to
use ANSI colors to surround node or nodeset prefix/header with escape sequences
to display them in color on the terminal. *WHENCOLOR* is ``never``, ``always``
or ``auto`` (which use color if standard output/error refer to a terminal).
Colors are set to [34m (blue foreground text) for stdout and [31m (red
foreground text) for stderr, and cannot be modified.
--diff show diff between common outputs (find the best
reference output by focusing on largest nodeset and also smaller command return
code)
File copying:
@@ -183,7 +183,7 @@
-u COMMAND_TIMEOUT, --command_timeout=COMMAND_TIMEOUT
limit time for command to run on the node
-R WORKER, --worker=WORKER
- worker name to use for connection (``exec``, ``ssh``,
``rsh``, ``pdsh``), default is ``ssh``
+ worker name to use for connection (``exec``, ``ssh``,
``rsh``, ``pdsh``, or the name of a Python worker module), default is ``ssh``
--remote=REMOTE whether to enable remote execution: in tree mode, 'yes'
forces connections to the leaf nodes for execution, 'no' establishes
connections up to the leaf parent nodes for execution (default is 'yes')
For a short explanation of these options, see ``-h, --help``.
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/doc/txt/groups.conf.txt
new/ClusterShell-1.8.4/doc/txt/groups.conf.txt
--- old/ClusterShell-1.8.3/doc/txt/groups.conf.txt 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/doc/txt/groups.conf.txt 2021-11-06
01:59:59.000000000 +0100
@@ -7,9 +7,9 @@
-----------------------------------------------
:Author: Stephane Thiell, <sthi...@stanford.edu>
-:Date: 2019-12-01
+:Date: 2021-11-03
:Copyright: GNU Lesser General Public License version 2.1 or later (LGPLv2.1+)
-:Version: 1.8.3
+:Version: 1.8.4
:Manual section: 5
:Manual group: ClusterShell User Manual
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/doc/txt/nodeset.txt
new/ClusterShell-1.8.4/doc/txt/nodeset.txt
--- old/ClusterShell-1.8.3/doc/txt/nodeset.txt 2019-12-07 21:34:33.000000000
+0100
+++ new/ClusterShell-1.8.4/doc/txt/nodeset.txt 2021-11-06 01:59:59.000000000
+0100
@@ -7,9 +7,9 @@
-----------------------------------
:Author: Stephane Thiell <sthi...@stanford.edu>
-:Date: 2019-12-01
+:Date: 2021-11-03
:Copyright: GNU Lesser General Public License version 2.1 or later (LGPLv2.1+)
-:Version: 1.8.3
+:Version: 1.8.4
:Manual section: 1
:Manual group: ClusterShell User Manual
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/lib/ClusterShell/CLI/Clubak.py
new/ClusterShell-1.8.4/lib/ClusterShell/CLI/Clubak.py
--- old/ClusterShell-1.8.3/lib/ClusterShell/CLI/Clubak.py 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/lib/ClusterShell/CLI/Clubak.py 2021-11-06
01:59:59.000000000 +0100
@@ -107,7 +107,7 @@
if options.interpret_keys == THREE_CHOICES[-1]: # auto?
enable_nodeset_key = None # AUTO
else:
- enable_nodeset_key = (options.interpret_keys == THREE_CHOICES[1])
+ enable_nodeset_key = (options.interpret_keys == THREE_CHOICES[2])
# Create new message tree
if options.trace_mode:
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/lib/ClusterShell/CLI/Clush.py
new/ClusterShell-1.8.4/lib/ClusterShell/CLI/Clush.py
--- old/ClusterShell-1.8.3/lib/ClusterShell/CLI/Clush.py 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/lib/ClusterShell/CLI/Clush.py 2021-11-06
01:59:59.000000000 +0100
@@ -82,15 +82,16 @@
self.master_worker.write(msg)
class OutputHandler(EventHandler):
- """Base class for clush output handlers."""
+ """Base class for generic output handlers."""
- def __init__(self):
+ def __init__(self, prog=None):
EventHandler.__init__(self)
self._runtimer = None
+ self._prog = prog if prog else os.path.basename(sys.argv[0])
def runtimer_init(self, task, ntotal=0):
"""Init timer for live command-completed progressmeter."""
- thandler = RunTimer(task, ntotal)
+ thandler = RunTimer(task, ntotal, prog=self._prog)
self._runtimer = task.timer(1.33, thandler, interval=1./3.,
autoclose=True)
@@ -134,8 +135,8 @@
class DirectOutputHandler(OutputHandler):
"""Direct output event handler class."""
- def __init__(self, display):
- OutputHandler.__init__(self)
+ def __init__(self, display, prog=None):
+ OutputHandler.__init__(self, prog=prog)
self._display = display
def ev_read(self, worker, node, sname, msg):
@@ -149,14 +150,15 @@
verb = VERB_QUIET
if self._display.maxrc:
verb = VERB_STD
- self._display.vprint_err(verb, "clush: %s: "
- "exited with exit code %d" % (node, rc))
+ self._display.vprint_err(verb, "%s: %s: exited with exit code %d" %
+ (self._prog, node, rc))
def ev_close(self, worker, timedout):
if timedout:
nodeset = NodeSet._fromlist1(worker.iter_keys_timeout())
self._display.vprint_err(VERB_QUIET,
- "clush: %s: command timeout" % nodeset)
+ "%s: %s: command timeout" %
+ (self._prog, nodeset))
self.update_prompt(worker)
class DirectProgressOutputHandler(DirectOutputHandler):
@@ -180,8 +182,8 @@
class CopyOutputHandler(DirectProgressOutputHandler):
"""Copy output event handler."""
- def __init__(self, display, reverse=False):
- DirectOutputHandler.__init__(self, display)
+ def __init__(self, display, reverse=False, prog=None):
+ DirectOutputHandler.__init__(self, display, prog=prog)
self.reverse = reverse
def ev_close(self, worker, timedout):
@@ -204,10 +206,10 @@
DirectOutputHandler.ev_close(self, worker, timedout)
class GatherOutputHandler(OutputHandler):
- """Gathered output event handler class (clush -b)."""
+ """Gathered output event handler class (e.g. clush -b)."""
- def __init__(self, display):
- OutputHandler.__init__(self)
+ def __init__(self, display, prog=None):
+ OutputHandler.__init__(self, prog=prog)
self._display = display
def ev_read(self, worker, node, sname, msg):
@@ -256,16 +258,16 @@
nsdisp = ns = NodeSet._fromlist1(nodelist)
if self._display.verbosity > VERB_QUIET and len(ns) > 1:
nsdisp = "%s (%d)" % (ns, len(ns))
- msgrc = "clush: %s: exited with exit code %d" % (nsdisp, rc)
+ msgrc = "%s: %s: exited with exit code %d" % (self._prog,
nsdisp, rc)
self._display.vprint_err(verbexit, msgrc)
# Display nodes that didn't answer within command timeout delay
if worker.num_timeout() > 0:
- self._display.vprint_err(verbexit, "clush: %s: command timeout" % \
- NodeSet._fromlist1(worker.iter_keys_timeout()))
+ self._display.vprint_err(verbexit, "%s: %s: command timeout" % \
+ (self._prog, NodeSet._fromlist1(worker.iter_keys_timeout())))
class SortedOutputHandler(GatherOutputHandler):
- """Sorted by node output event handler class (clush -L)."""
+ """Sorted by node output event handler class (e.g. clush -L)."""
def ev_close(self, worker, timedout):
# Overrides GatherOutputHandler.ev_close()
@@ -290,9 +292,9 @@
class LiveGatherOutputHandler(GatherOutputHandler):
"""Live line-gathered output event handler class (-bL)."""
- def __init__(self, display, nodes):
+ def __init__(self, display, nodes, prog=None):
assert nodes is not None, "cannot gather local command"
- GatherOutputHandler.__init__(self, display)
+ GatherOutputHandler.__init__(self, display, prog=prog)
self._nodes = NodeSet(nodes)
self._nodecnt = dict.fromkeys(self._nodes, 0)
self._mtreeq = []
@@ -346,7 +348,7 @@
class RunTimer(EventHandler):
"""Running progress timer event handler"""
- def __init__(self, task, total):
+ def __init__(self, task, total, prog=None):
EventHandler.__init__(self)
self.task = task
self.total = total
@@ -357,6 +359,7 @@
# updated by worker handler for progress
self.start_time = 0
self.bytes_written = 0
+ self._prog = prog if prog else os.path.basename(sys.argv[0])
def ev_timer(self, timer):
self.update()
@@ -390,9 +393,9 @@
if self.bytes_written > 0 or cnt != self.cnt_last:
self.cnt_last = cnt
# display completed/total clients
- towrite = 'clush: %*d/%*d%s%s\r' % (self.tslen, self.total - cnt,
- self.tslen, self.total, gwinfo,
- wrbwinfo)
+ towrite = '%s: %*d/%*d%s%s\r' % (self._prog, self.tslen,
+ self.total - cnt, self.tslen,
+ self.total, gwinfo, wrbwinfo)
self.wholelen = len(towrite)
sys.stderr.write(towrite)
self.started = True
@@ -403,12 +406,13 @@
return
self.erase_line()
# display completed/total clients
- fmt = 'clush: %*d/%*d'
+ fmt = '%s: %*d/%*d'
if force_cr:
fmt += '\n'
else:
fmt += '\r'
- sys.stderr.write(fmt % (self.tslen, self.total, self.tslen,
self.total))
+ sys.stderr.write(fmt % (self._prog, self.tslen, self.total, self.tslen,
+ self.total))
def signal_handler(signum, frame):
@@ -615,13 +619,14 @@
# 64k seems to be perfect with an openssh backend (they issue 64k
# reads) ; could consider making it an option for e.g. gsissh.
bufsize = 64 * 1024
- # thread loop: blocking read stdin + send messages to specified
- # port object
- buf = sys_stdin().read(bufsize) # use buffer in Python 3
- while buf:
+ # thread loop: read stdin + send messages to specified port object
+ # use os.read() to work around https://bugs.python.org/issue42717
+ while True:
+ buf = os.read(sys_stdin().fileno(), bufsize)
+ if not buf:
+ break
# send message to specified port object (with ack)
stdin_port.msg(buf)
- buf = sys_stdin().read(bufsize)
except IOError as ex:
display.vprint(VERB_VERB, "stdin: %s" % ex)
# send a None message to indicate EOF
@@ -631,7 +636,7 @@
"""Create a stdin->port->worker binding: connect specified worker
to stdin with the help of a reader thread and a ClusterShell Port
object."""
- assert not sys.stdin.isatty()
+ assert sys.stdin is not None and not sys.stdin.isatty()
# Create a ClusterShell Port object bound to worker's task. This object
# is able to receive messages in a thread-safe manner and then will safely
# trigger ev_msg() on a specified event handler.
@@ -928,7 +933,7 @@
interactive = not len(args) and \
not (options.copy or options.rcopy)
# check for foreground ttys presence (input)
- stdin_isafgtty = sys.stdin.isatty() and \
+ stdin_isafgtty = sys.stdin is not None and sys.stdin.isatty() and \
os.tcgetpgrp(sys.stdin.fileno()) == os.getpgrp()
# check for special condition (empty command and stdin not a tty)
if interactive and not stdin_isafgtty:
@@ -962,7 +967,8 @@
task.set_default("USER_handle_SIGUSR1", user_interaction)
task.excepthook = sys.excepthook
- task.set_default("USER_stdin_worker", not (sys.stdin.isatty() or \
+ task.set_default("USER_stdin_worker", not (sys.stdin is None or \
+ sys.stdin.isatty() or \
options.nostdin or \
user_interaction))
display.vprint(VERB_DEBUG, "Create STDIN worker: %s" % \
@@ -1090,7 +1096,7 @@
clush_exit(1, task)
rc = 0
- if options.maxrc:
+ if config.maxrc:
# Instead of clush return code, return commands retcode
rc = task.max_retcode()
if task.num_timeout() > 0:
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/lib/ClusterShell/CLI/Config.py
new/ClusterShell-1.8.4/lib/ClusterShell/CLI/Config.py
--- old/ClusterShell-1.8.3/lib/ClusterShell/CLI/Config.py 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/lib/ClusterShell/CLI/Config.py 2021-11-06
01:59:59.000000000 +0100
@@ -53,9 +53,10 @@
"connect_timeout": "%f" % DEFAULTS.connect_timeout,
"command_timeout": "%f" % DEFAULTS.command_timeout,
"history_size": "100",
- "color": THREE_CHOICES[-1], # auto
+ "color": THREE_CHOICES[0], # ''
"verbosity": "%d" % VERB_STD,
"node_count": "yes",
+ "maxrc": "no",
"fd_max": "8192"}
def __init__(self, options, filename=None):
@@ -94,6 +95,8 @@
self._set_main("command_timeout", options.command_timeout)
if options.whencolor:
self._set_main("color", options.whencolor)
+ if options.maxrc:
+ self._set_main("maxrc", options.maxrc)
try:
# -O/--option KEY=VALUE
@@ -213,6 +216,11 @@
return self.getboolean("Main", "node_count")
@property
+ def maxrc(self):
+ """maxrc value as a boolean"""
+ return self.getboolean("Main", "maxrc")
+
+ @property
def fd_max(self):
"""max number of open files (soft rlimit)"""
return self.getint("Main", "fd_max")
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/lib/ClusterShell/CLI/Display.py
new/ClusterShell-1.8.4/lib/ClusterShell/CLI/Display.py
--- old/ClusterShell-1.8.3/lib/ClusterShell/CLI/Display.py 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/lib/ClusterShell/CLI/Display.py 2021-11-06
01:59:59.000000000 +0100
@@ -25,6 +25,7 @@
import difflib
import sys
+import os
from ClusterShell.NodeSet import NodeSet
@@ -33,7 +34,7 @@
VERB_STD = 1
VERB_VERB = 2
VERB_DEBUG = 3
-THREE_CHOICES = ["never", "always", "auto"]
+THREE_CHOICES = ["", "never", "always", "auto"]
WHENCOLOR_CHOICES = THREE_CHOICES # deprecated; use THREE_CHOICES
if sys.getdefaultencoding() == 'ascii':
@@ -98,6 +99,18 @@
# display may change when 'max return code' option is set
self.maxrc = getattr(options, 'maxrc', False)
+ # Be compliant with NO_COLOR and CLI_COLORS trying to solve #428
+ # See https://no-color.org/ and https://bixense.com/clicolors/
+ # NO_COLOR takes precedence over CLI_COLORS. --color option always
+ # takes precedence over any environment variable.
+
+ if options.whencolor is None:
+ if (config is None) or (config.color == '' or config.color ==
'auto'):
+ if 'NO_COLOR' not in os.environ:
+ color = self._has_cli_color()
+ else:
+ color = False
+
if color is None:
# Should we use ANSI colors?
color = False
@@ -137,6 +150,26 @@
if hasattr(options, 'debug') and options.debug:
self.verbosity = VERB_DEBUG
+ def _has_cli_color(self):
+ """Tests CLICOLOR environment variable to determine wether to
+ use color or not on output."""
+ # When CLICOLOR_FORCE is set to something else than 0
+ # colors must be used.
+ if os.getenv("CLICOLOR_FORCE", "0") != "0":
+ return True
+
+ cli_color = os.getenv("CLICOLOR")
+
+ if cli_color is None:
+ return None
+ elif cli_color != "0":
+ # CLICOLOR is set and colored output should
+ # be used if stdout is a tty
+ return sys.stdout.isatty()
+ else:
+ # CLICOLOR is set to not display colors.
+ return False
+
def flush(self):
"""flush display object buffers"""
# only used to reset diff display for now
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/ClusterShell-1.8.3/lib/ClusterShell/CLI/OptionParser.py
new/ClusterShell-1.8.4/lib/ClusterShell/CLI/OptionParser.py
--- old/ClusterShell-1.8.3/lib/ClusterShell/CLI/OptionParser.py 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/lib/ClusterShell/CLI/OptionParser.py 2021-11-06
01:59:59.000000000 +0100
@@ -172,7 +172,7 @@
help="node / line content separator string "
"(default: ':')")
else:
- optgrp.add_option("-S", action="store_true", dest="maxrc",
+ optgrp.add_option("-S", "--maxrc", action="store_true",
dest="maxrc",
help="return the largest of command return
codes")
if msgtree_mode:
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/lib/ClusterShell/Defaults.py
new/ClusterShell-1.8.4/lib/ClusterShell/Defaults.py
--- old/ClusterShell-1.8.3/lib/ClusterShell/Defaults.py 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/lib/ClusterShell/Defaults.py 2021-11-06
01:59:59.000000000 +0100
@@ -55,18 +55,31 @@
"""
Return the class pointer matching `workername`.
+ This can be the 'short' name (such as `ssh`) or a fully-qualified
+ module path (such as ClusterShell.Worker.Ssh).
+
The module is loaded if not done yet.
"""
- modname = "ClusterShell.Worker.%s" % workername.capitalize()
+ # First try the worker name as a module under ClusterShell.Worker,
+ # but if that fails, try the worker name directly
+ try:
+ modname = "ClusterShell.Worker.%s" % workername.capitalize()
+ _import_module(modname)
+ except ImportError:
+ modname = workername
+ _import_module(modname)
+
+ # Get the class pointer
+ return sys.modules[modname].WORKER_CLASS
+
+def _import_module(modname):
+ """Import a python module if not done yet."""
# Iterate over a copy of sys.modules' keys to avoid RuntimeError
if modname.lower() not in [mod.lower() for mod in list(sys.modules)]:
# Import module if not yet loaded
__import__(modname)
- # Get the class pointer
- return sys.modules[modname].WORKER_CLASS
-
def _local_workerclass(defaults):
"""Return default local worker class."""
return _load_workerclass(defaults.local_workername)
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/lib/ClusterShell/Gateway.py
new/ClusterShell-1.8.4/lib/ClusterShell/Gateway.py
--- old/ClusterShell-1.8.3/lib/ClusterShell/Gateway.py 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/lib/ClusterShell/Gateway.py 2021-11-06
01:59:59.000000000 +0100
@@ -255,6 +255,11 @@
task._info.update(taskinfo)
task.set_info('print_debug', _gw_print_debug)
+ for infokey in taskinfo:
+ if infokey.startswith('tree_default:'):
+ self.logger.debug('Setting default %s to %s',
infokey[13:], taskinfo[infokey])
+ task.set_default(infokey[13:], taskinfo[infokey])
+
if task.info('debug'):
self.logger.setLevel(logging.DEBUG)
@@ -324,6 +329,14 @@
logger = logging.getLogger(__name__)
sys.excepthook = gateway_excepthook
+ if sys.stdin is None:
+ logger.critical('Gateway failure: sys.stdin is None')
+ sys.exit(1)
+
+ if sys.stdin.isatty():
+ logger.critical('Gateway failure: sys.stdin.isatty() is True')
+ sys.exit(1)
+
logger.debug('Starting gateway on %s', host)
logger.debug("environ=%s", os.environ)
@@ -338,10 +351,6 @@
task.set_default("stdout_msgtree", False)
task.set_default("stderr_msgtree", False)
- if sys.stdin.isatty():
- logger.critical('Gateway failure: sys.stdin.isatty() is True')
- sys.exit(1)
-
gateway = GatewayChannel(task)
worker = StreamWorker(handler=gateway)
# Define worker._fanout to not rely on the engine's fanout, and use
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/lib/ClusterShell/NodeUtils.py
new/ClusterShell-1.8.4/lib/ClusterShell/NodeUtils.py
--- old/ClusterShell-1.8.3/lib/ClusterShell/NodeUtils.py 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/lib/ClusterShell/NodeUtils.py 2021-11-06
01:59:59.000000000 +0100
@@ -445,6 +445,8 @@
result = []
assert source
raw = getattr(source, 'resolv_%s' % what)(*args)
+ if isinstance(raw, list):
+ raw = ','.join(raw)
for line in raw.splitlines():
[result.append(x) for x in line.strip().split()]
return result
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/lib/ClusterShell/RangeSet.py
new/ClusterShell-1.8.4/lib/ClusterShell/RangeSet.py
--- old/ClusterShell-1.8.3/lib/ClusterShell/RangeSet.py 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/lib/ClusterShell/RangeSet.py 2021-11-06
01:59:59.000000000 +0100
@@ -956,8 +956,7 @@
for rgvec in self._veclist:
iveclist += product(*rgvec)
assert(len(iveclist) == len(self))
- rnd = RangeSetND(iveclist[index],
- pads=[rg.padding for rg in self._veclist[0]],
+ rnd = RangeSetND(iveclist[index], pads=self.pads(),
autostep=self.autostep)
return rnd
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/lib/ClusterShell/Task.py
new/ClusterShell-1.8.4/lib/ClusterShell/Task.py
--- old/ClusterShell-1.8.3/lib/ClusterShell/Task.py 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/lib/ClusterShell/Task.py 2021-11-06
01:59:59.000000000 +0100
@@ -58,7 +58,7 @@
basestring = str
from ClusterShell.Defaults import config_paths, DEFAULTS
-from ClusterShell.Defaults import _local_workerclass, _distant_workerclass
+from ClusterShell.Defaults import _local_workerclass, _distant_workerclass,
_load_workerclass
from ClusterShell.Engine.Engine import EngineAbortException
from ClusterShell.Engine.Engine import EngineTimeoutException
from ClusterShell.Engine.Engine import EngineAlreadyRunningError
@@ -470,6 +470,10 @@
self._default_lock.acquire()
try:
self._default[default_key] = value
+ if default_key == 'local_workername':
+ self._default['local_worker'] = _load_workerclass(value)
+ elif default_key == 'distant_workername':
+ self._default['distant_worker'] = _load_workerclass(value)
finally:
self._default_lock.release()
@@ -510,6 +514,8 @@
- "command_timeout": Time in seconds to wait for a command to
complete before aborting (default: 0, which means
unlimited).
+ - "tree_default:<key>": In tree mode, overrides the key <key>
+ in Defaults (settings normally set in defaults.conf)
Threading considerations
========================
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/lib/ClusterShell/Worker/Rsh.py
new/ClusterShell-1.8.4/lib/ClusterShell/Worker/Rsh.py
--- old/ClusterShell-1.8.3/lib/ClusterShell/Worker/Rsh.py 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/lib/ClusterShell/Worker/Rsh.py 2021-11-06
01:59:59.000000000 +0100
@@ -26,6 +26,7 @@
import os
import shlex
+import re
from ClusterShell.Worker.Exec import ExecClient, CopyClient, ExecWorker
@@ -35,6 +36,12 @@
Rsh EngineClient.
"""
+ def __init__(self, node, command, worker, stderr, timeout, autoclose=False,
+ rank=None):
+ ExecClient.__init__(self, node, command, worker, stderr, timeout,
+ autoclose, rank)
+ self.rsh_rc = None
+
def _build_cmd(self):
"""
Build the shell command line to start the rsh commmand.
@@ -59,8 +66,26 @@
cmd_l.append("%s" % self.key) # key is the node
cmd_l.append("%s" % self.command)
+ # rsh does not properly return exit status
+ # force the exit status to be printed out
+ cmd_l.append("; echo XXRETCODE: $?")
+
return (cmd_l, None)
+ def _on_nodeset_msgline(self, nodes, msg, sname):
+ """Override _on_nodeset_msgline to parse magic return code"""
+ match = re.search(r"^XXRETCODE: (\d+)$", msg.decode())
+ if match:
+ self.rsh_rc = int(match.group(1))
+ else:
+ ExecClient._on_nodeset_msgline(self, nodes, msg, sname)
+
+ def _on_nodeset_close(self, nodes, rc):
+ """Override _on_nodeset_close to return rsh_rc"""
+ if (rc == 0 or rc == 1) and self.rsh_rc is not None:
+ rc = self.rsh_rc
+ ExecClient._on_nodeset_close(self, nodes, rc)
+
class RcpClient(CopyClient):
"""
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/lib/ClusterShell/Worker/Ssh.py
new/ClusterShell-1.8.4/lib/ClusterShell/Worker/Ssh.py
--- old/ClusterShell-1.8.3/lib/ClusterShell/Worker/Ssh.py 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/lib/ClusterShell/Worker/Ssh.py 2021-11-06
01:59:59.000000000 +0100
@@ -125,18 +125,18 @@
if self.reverse:
if user:
- cmd_l.append("%s@%s:%s" % (user, self.key, self.source))
+ cmd_l.append("%s@[%s]:%s" % (user, self.key, self.source))
else:
- cmd_l.append("%s:%s" % (self.key, self.source))
+ cmd_l.append("[%s]:%s" % (self.key, self.source))
cmd_l.append(os.path.join(self.dest, "%s.%s" % \
(os.path.basename(self.source), self.key)))
else:
cmd_l.append(self.source)
if user:
- cmd_l.append("%s@%s:%s" % (user, self.key, self.dest))
+ cmd_l.append("%s@[%s]:%s" % (user, self.key, self.dest))
else:
- cmd_l.append("%s:%s" % (self.key, self.dest))
+ cmd_l.append("[%s]:%s" % (self.key, self.dest))
return (cmd_l, None)
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/lib/ClusterShell/Worker/Tree.py
new/ClusterShell-1.8.4/lib/ClusterShell/Worker/Tree.py
--- old/ClusterShell-1.8.3/lib/ClusterShell/Worker/Tree.py 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/lib/ClusterShell/Worker/Tree.py 2021-11-06
01:59:59.000000000 +0100
@@ -280,11 +280,12 @@
tree=False)
else:
assert self.source is None
- worker = ExecWorker(nodes=targets,
- command=self.command,
- handler=self.metahandler,
- timeout=self.timeout,
- stderr=self.stderr)
+ workerclass = self.task.default('local_worker')
+ worker = workerclass(nodes=targets,
+ command=self.command,
+ handler=self.metahandler,
+ timeout=self.timeout,
+ stderr=self.stderr)
self.task.schedule(worker)
self.workers.append(worker)
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/lib/ClusterShell/__init__.py
new/ClusterShell-1.8.4/lib/ClusterShell/__init__.py
--- old/ClusterShell-1.8.3/lib/ClusterShell/__init__.py 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/lib/ClusterShell/__init__.py 2021-11-06
01:59:59.000000000 +0100
@@ -34,8 +34,8 @@
- ClusterShell.Task
"""
-__version__ = '1.8.3'
+__version__ = '1.8.4'
__version_info__ = tuple([ int(_n) for _n in __version__.split('.')])
-__date__ = '2019/12/01'
+__date__ = '2021/11/03'
__author__ = 'Stephane Thiell <sthi...@stanford.edu>'
__url__ = 'http://clustershell.readthedocs.org/'
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/ClusterShell-1.8.3/lib/ClusterShell.egg-info/PKG-INFO
new/ClusterShell-1.8.4/lib/ClusterShell.egg-info/PKG-INFO
--- old/ClusterShell-1.8.3/lib/ClusterShell.egg-info/PKG-INFO 2019-12-17
20:13:09.000000000 +0100
+++ new/ClusterShell-1.8.4/lib/ClusterShell.egg-info/PKG-INFO 2021-11-06
23:20:43.000000000 +0100
@@ -1,12 +1,12 @@
Metadata-Version: 1.1
Name: ClusterShell
-Version: 1.8.3
+Version: 1.8.4
Summary: ClusterShell library and tools
Home-page: http://clustershell.sourceforge.net/
Author: Stephane Thiell
Author-email: sthi...@stanford.edu
License: LGPLv2+
-Download-URL:
http://sourceforge.net/projects/clustershell/files/clustershell/1.8.3/
+Download-URL:
http://sourceforge.net/projects/clustershell/files/clustershell/1.8.4/
Description: ClusterShell is an event-driven open source Python framework,
designed to run
local or distant commands in parallel on server farms or on large Linux
clusters. It will take care of common issues encountered on HPC
clusters, such
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/setup.py
new/ClusterShell-1.8.4/setup.py
--- old/ClusterShell-1.8.3/setup.py 2019-12-07 21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/setup.py 2021-11-06 01:59:59.000000000 +0100
@@ -1,7 +1,7 @@
#!/usr/bin/env python
#
# Copyright (C) 2008-2016 CEA/DAM
-# Copyright (C) 2016-2019 Stephane Thiell <sthi...@stanford.edu>
+# Copyright (C) 2016-2021 Stephane Thiell <sthi...@stanford.edu>
#
# This file is part of ClusterShell.
#
@@ -23,7 +23,7 @@
from setuptools import setup, find_packages
-VERSION = '1.8.3'
+VERSION = '1.8.4'
# Default CFGDIR: in-prefix config install (rpmbuild or pip as user)
CFGDIR = 'etc/clustershell'
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/tests/CLIClushTest.py
new/ClusterShell-1.8.4/tests/CLIClushTest.py
--- old/ClusterShell-1.8.3/tests/CLIClushTest.py 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/tests/CLIClushTest.py 2021-11-06
01:59:59.000000000 +0100
@@ -275,6 +275,14 @@
self._clush_t(["-w", HOSTNAME, "/bin/false"], None, b"", 0, exp_err)
self._clush_t(["-w", HOSTNAME, "-b", "/bin/false"], None, b"", 0,
exp_err)
self._clush_t(["-S", "-w", HOSTNAME, "/bin/false"], None, b"", 1,
exp_err)
+ self._clush_t(["--maxrc", "-w", HOSTNAME, "/bin/false"], None, b"", 1,
exp_err)
+ self._clush_t(["-O", "maxrc=yes", "-w", HOSTNAME, "/bin/false"], None,
+ b"", 1, exp_err)
+ # -O takes precedence over --maxrc
+ self._clush_t(["--maxrc", "-O", "maxrc=no", "-w", HOSTNAME,
"/bin/false"], None,
+ b"", 0, exp_err)
+ self._clush_t(["-O", "maxrc=no", "--maxrc", "-w", HOSTNAME,
"/bin/false"], None,
+ b"", 0, exp_err)
for i in (1, 2, 127, 128, 255):
s = "clush: %s: exited with exit code %d\n" % (HOSTNAME, i)
self._clush_t(["-S", "-w", HOSTNAME, "exit %d" % i], None, b"", i,
@@ -606,7 +614,7 @@
class BrokenStdinMock(object):
def isatty(self):
return False
- def read(self, bufsize=1024):
+ def fileno(self):
raise IOError(errno.EINVAL, "Invalid argument")
sys.stdin = BrokenStdinMock()
@@ -670,3 +678,50 @@
self._clush_t(["--groupsconf", self.custf.name, "-R", "exec", "-w",
"@foo", "-bL", "echo ok"], None,
b"custom[7-42]: ok\n", 0, b"")
+
+
+class CLIClushTest_D_StdinFIFO(unittest.TestCase):
+ """Unit test class for testing CLI/Clush.py when stdin is a named pipe"""
+
+ def setUp(self):
+ # Create fifo
+ self.fname = tempfile.mktemp()
+ os.mkfifo(self.fname)
+
+ # Launch write thread
+ class FIFOThread(threading.Thread):
+ def run(self):
+ fifo_wr = open(self.fname, "w")
+ fifo_wr.write("0123456789")
+ fifo_wr.close()
+
+ fifoth = FIFOThread()
+ fifoth.fname = self.fname
+ fifoth.start()
+
+ # Use read end of fifo as stdin
+ sys.stdin = open(self.fname, "r")
+
+ def tearDown(self):
+ """cleanup all tasks"""
+ sys.stdin.close()
+ os.unlink(self.fname)
+ task_cleanup()
+ sys.stdin = sys.__stdin__
+
+ def _clush_t(self, args, stdin, expected_stdout, expected_rc=0,
+ expected_stderr=None):
+ CLI_main(self, main, ['clush'] + args, stdin, expected_stdout,
+ expected_rc, expected_stderr)
+
+ def test_300_fifo_stdin(self):
+ """test clush with stdin is a fifo (read)"""
+ s = "%s: 0123456789\n" % HOSTNAME
+ self._clush_t(["-w", HOSTNAME, "-v", "cat"], None,
+ s.encode(), 0, b"")
+
+ def test_301_fifo_stdin(self):
+ """test clush with stdin is a fifo (not read)"""
+ s = "%s: ok\n" % HOSTNAME
+ self._clush_t(["-w", HOSTNAME, "-v", "echo ok"], None,
+ s.encode(), 0, b"")
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/tests/CLIConfigTest.py
new/ClusterShell-1.8.4/tests/CLIConfigTest.py
--- old/ClusterShell-1.8.3/tests/CLIConfigTest.py 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/tests/CLIConfigTest.py 2021-11-06
01:59:59.000000000 +0100
@@ -35,9 +35,10 @@
parser.install_connector_options()
options, _ = parser.parse_args([])
config = ClushConfig(options, filename=f.name)
- self.assertEqual(config.color, WHENCOLOR_CHOICES[-1])
+ self.assertEqual(config.color, THREE_CHOICES[0])
self.assertEqual(config.verbosity, VERB_STD)
self.assertEqual(config.fanout, 64)
+ self.assertEqual(config.maxrc, False)
self.assertEqual(config.node_count, True)
self.assertEqual(config.connect_timeout, 10)
self.assertEqual(config.command_timeout, 0)
@@ -58,8 +59,9 @@
parser.install_connector_options()
options, _ = parser.parse_args([])
config = ClushConfig(options, filename=f.name)
- self.assertEqual(config.color, WHENCOLOR_CHOICES[-1])
+ self.assertEqual(config.color, THREE_CHOICES[0])
self.assertEqual(config.verbosity, VERB_STD)
+ self.assertEqual(config.maxrc, False)
self.assertEqual(config.node_count, True)
self.assertEqual(config.fanout, 64)
self.assertEqual(config.connect_timeout, 10)
@@ -94,8 +96,9 @@
display = Display(options, config)
display.vprint(VERB_STD, "test")
display.vprint(VERB_DEBUG, "shouldn't see this")
- self.assertEqual(config.color, WHENCOLOR_CHOICES[2])
+ self.assertEqual(config.color, THREE_CHOICES[-1])
self.assertEqual(config.verbosity, VERB_STD)
+ self.assertEqual(config.maxrc, False)
self.assertEqual(config.node_count, True)
self.assertEqual(config.fanout, 42)
self.assertEqual(config.connect_timeout, 14)
@@ -116,6 +119,7 @@
command_timeout: 0
history_size: 100
color: auto
+ maxrc: yes
node_count: yes
verbosity: 1
ssh_user: root
@@ -130,8 +134,9 @@
parser.install_connector_options()
options, _ = parser.parse_args([])
config = ClushConfig(options, filename=f.name)
- self.assertEqual(config.color, WHENCOLOR_CHOICES[2])
+ self.assertEqual(config.color, THREE_CHOICES[-1])
self.assertEqual(config.verbosity, VERB_STD)
+ self.assertEqual(config.maxrc, True)
self.assertEqual(config.node_count, True)
self.assertEqual(config.fanout, 42)
self.assertEqual(config.connect_timeout, 14)
@@ -293,7 +298,7 @@
display = Display(options, config)
display.vprint(VERB_STD, "test")
display.vprint(VERB_DEBUG, "test")
- self.assertEqual(config.color, WHENCOLOR_CHOICES[1])
+ self.assertEqual(config.color, THREE_CHOICES[2])
self.assertEqual(config.verbosity, VERB_DEBUG) # takes biggest
self.assertEqual(config.fanout, 36)
self.assertEqual(config.connect_timeout, 7)
@@ -355,7 +360,7 @@
parser.install_connector_options()
options, _ = parser.parse_args([])
config = ClushConfig(options) # filename=None to use defaults!
- self.assertEqual(config.color, WHENCOLOR_CHOICES[0])
+ self.assertEqual(config.color, THREE_CHOICES[1])
self.assertEqual(config.verbosity, VERB_VERB) # takes biggest
self.assertEqual(config.fanout, 42)
self.assertEqual(config.connect_timeout, 14)
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/tests/CLIDisplayTest.py
new/ClusterShell-1.8.4/tests/CLIDisplayTest.py
--- old/ClusterShell-1.8.3/tests/CLIDisplayTest.py 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/tests/CLIDisplayTest.py 2021-11-06
01:59:59.000000000 +0100
@@ -5,9 +5,10 @@
import tempfile
import unittest
+import os
from io import BytesIO
-from ClusterShell.CLI.Display import Display, WHENCOLOR_CHOICES, VERB_STD
+from ClusterShell.CLI.Display import Display, THREE_CHOICES, VERB_STD
from ClusterShell.CLI.OptionParser import OptionParser
from ClusterShell.MsgTree import MsgTree
@@ -38,27 +39,46 @@
mtree.add("hostfoo", b"message0")
mtree.add("hostfoo", b"message1")
- for whencolor in WHENCOLOR_CHOICES: # test whencolor switch
- for label in [True, False]: # test no-label switch
- options.label = label
- options.whencolor = whencolor
- disp = Display(options)
- # inhibit output
- disp.out = BytesIO()
- disp.err = BytesIO()
- # test print_* methods...
- disp.print_line(ns, b"foo bar")
- disp.print_line_error(ns, b"foo bar")
- disp.print_gather(ns, list(mtree.walk())[0][0])
- # test also string nodeset as parameter
- disp.print_gather("hostfoo", list(mtree.walk())[0][0])
- # test line_mode property
- self.assertEqual(disp.line_mode, False)
- disp.line_mode = True
- self.assertEqual(disp.line_mode, True)
- disp.print_gather("hostfoo", list(mtree.walk())[0][0])
- disp.line_mode = False
- self.assertEqual(disp.line_mode, False)
+ list_env_vars = []
+ list_env_vars.append(dict())
+ list_env_vars.append(dict(NO_COLOR='0'))
+ list_env_vars.append(dict(CLICOLOR='0'))
+ list_env_vars.append(dict(CLICOLOR='1'))
+ list_env_vars.append(dict(CLICOLOR='0', CLICOLOR_FORCE='0'))
+ list_env_vars.append(dict(CLICOLOR_FORCE='1'))
+
+ for env_vars in list_env_vars:
+ for var_name in env_vars:
+ var_value = env_vars[var_name]
+ os.environ[var_name] = var_value
+
+ for whencolor in THREE_CHOICES: # test whencolor switch
+ if whencolor == "":
+ options.whencolor = None
+ else:
+ options.whencolor = whencolor
+ for label in [True, False]: # test no-label switch
+ options.label = label
+ disp = Display(options)
+ # inhibit output
+ disp.out = BytesIO()
+ disp.err = BytesIO()
+ # test print_* methods...
+ disp.print_line(ns, b"foo bar")
+ disp.print_line_error(ns, b"foo bar")
+ disp.print_gather(ns, list(mtree.walk())[0][0])
+ # test also string nodeset as parameter
+ disp.print_gather("hostfoo", list(mtree.walk())[0][0])
+ # test line_mode property
+ self.assertEqual(disp.line_mode, False)
+ disp.line_mode = True
+ self.assertEqual(disp.line_mode, True)
+ disp.print_gather("hostfoo", list(mtree.walk())[0][0])
+ disp.line_mode = False
+ self.assertEqual(disp.line_mode, False)
+
+ for var_name in env_vars:
+ os.environ.pop(var_name)
def testDisplayRegroup(self):
"""test CLI.Display (regroup)"""
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/tests/DefaultsTest.py
new/ClusterShell-1.8.4/tests/DefaultsTest.py
--- old/ClusterShell-1.8.3/tests/DefaultsTest.py 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/tests/DefaultsTest.py 2021-11-06
01:59:59.000000000 +0100
@@ -3,10 +3,14 @@
"""Unit test for ClusterShell.Defaults"""
+import os
+import sys
+import shutil
+
from textwrap import dedent
import unittest
-from TLib import make_temp_file
+from TLib import make_temp_file, make_temp_dir
from ClusterShell.Defaults import Defaults, _task_print_debug
@@ -98,6 +102,21 @@
self.assertTrue(task.default("distant_worker") is WorkerSsh)
task_terminate()
+ dname = make_temp_dir()
+ modfile = open(os.path.join(dname, 'OutOfTree.py'), 'w')
+ modfile.write(dedent("""
+ class OutOfTreeWorker(object):
+ pass
+ WORKER_CLASS = OutOfTreeWorker"""))
+ modfile.flush()
+ modfile.close()
+ sys.path.append(dname)
+ self.defaults.distant_workername = 'OutOfTree'
+ task = task_self(self.defaults)
+ self.assertTrue(task.default("distant_worker").__name__ is
'OutOfTreeWorker')
+ task_terminate()
+ shutil.rmtree(dname, ignore_errors=True)
+
def test_005_misc_value_errors(self):
"""test Defaults misc value errors"""
task_terminate()
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/tests/NodeSetGroupTest.py
new/ClusterShell-1.8.4/tests/NodeSetGroupTest.py
--- old/ClusterShell-1.8.3/tests/NodeSetGroupTest.py 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/tests/NodeSetGroupTest.py 2021-11-06
01:59:59.000000000 +0100
@@ -1589,6 +1589,19 @@
self.assertRaises(GroupResolverConfigError, YAMLGroupLoader, f.name)
+ def test_list_group(self):
+ f = make_temp_file(dedent("""
+ rednecks:
+ bubba:
+ - pickup-1
+ - pickup-2
+ - tractor-[1-2]""").encode('ascii'))
+ loader = YAMLGroupLoader(f.name)
+ sources = list(loader)
+ resolver = GroupResolver(sources[0])
+ self.assertEqual(resolver.group_nodes('bubba'),
+ [ 'pickup-1,pickup-2,tractor-[1-2]' ])
+
class GroupResolverYAMLTest(unittest.TestCase):
def setUp(self):
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/tests/RangeSetNDTest.py
new/ClusterShell-1.8.4/tests/RangeSetNDTest.py
--- old/ClusterShell-1.8.3/tests/RangeSetNDTest.py 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/tests/RangeSetNDTest.py 2021-11-06
01:59:59.000000000 +0100
@@ -432,6 +432,9 @@
# steps
self.assertEqual(str(rn1[0:12:2]), "0-3; 1\n10; 10,12\n")
self.assertEqual(str(rn1[1:12:2]), "0-3; 2\n10; 11,13\n")
+ # GitHub #429
+ rn1 = RangeSetND([["110", "15-16"], ["107", "06"]])
+ self.assertEqual(str(rn1[0:3:2]), "107; 06\n110; 15\n")
def test_contiguous(self):
rn0 = RangeSetND()
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/tests/TLib.py
new/ClusterShell-1.8.4/tests/TLib.py
--- old/ClusterShell-1.8.3/tests/TLib.py 2019-12-07 21:34:33.000000000
+0100
+++ new/ClusterShell-1.8.4/tests/TLib.py 2021-11-06 01:59:59.000000000
+0100
@@ -39,6 +39,7 @@
def isatty(self):
return False
+
def load_cfg(name):
"""Load test configuration file as a new ConfigParser"""
cfgparser = configparser.ConfigParser()
@@ -88,14 +89,16 @@
# Capture standard streams
- # Input: if defined, stdin may either be a buffer or a string (with an
- # encoding).
+ # Input: if defined, the stdin argument specifies input data
if stdin is not None:
- if type(stdin) is bytes: # also works for str in Python 2
- sys.stdin = TBytesIO(stdin)
+ if type(stdin) is bytes:
+ # Use temporary file in Python 2 or with buffer (bytes) in Python 3
+ sys.stdin = tempfile.TemporaryFile()
+ sys.stdin.write(stdin)
+ sys.stdin.seek(0) # ready to be read
else:
# If stdin is a string in Python 3, use StringIO as sys.stdin
- # should be read in text mode for some tests.
+ # should be read in text mode for some tests (eg. Nodeset).
sys.stdin = StringIO(stdin)
# Output: ClusterShell sends bytes to sys_stdout()/sys_stderr() and when
@@ -110,7 +113,10 @@
finally:
sys.stdout = saved_stdout
sys.stderr = saved_stderr
- sys.stdin = saved_stdin
+ # close temporary file if we used one for stdin
+ if saved_stdin != sys.stdin:
+ sys.stdin.close()
+ sys.stdin = saved_stdin
try:
if expected_stdout is not None:
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/ClusterShell-1.8.3/tests/TreeWorkerTest.py
new/ClusterShell-1.8.4/tests/TreeWorkerTest.py
--- old/ClusterShell-1.8.3/tests/TreeWorkerTest.py 2019-12-07
21:34:33.000000000 +0100
+++ new/ClusterShell-1.8.4/tests/TreeWorkerTest.py 2021-11-06
01:59:59.000000000 +0100
@@ -191,6 +191,22 @@
self.assertEqual(teh.ev_close_cnt, 1)
self.assertEqual(teh.last_read, NODE_DISTANT.encode('ascii'))
+ def test_tree_run_noremote_alt_localworker(self):
+ """test tree run with remote=False and a non-exec localworker"""
+ teh = TEventHandler()
+ self.task.set_info('tree_default:local_workername', 'ssh')
+ self.task.run('echo %h', nodes=NODE_DISTANT, handler=teh, remote=False)
+ self.assertEqual(teh.ev_start_cnt, 1)
+ self.assertEqual(teh.ev_pickup_cnt, 1)
+ self.assertEqual(teh.ev_read_cnt, 1)
+ self.assertEqual(teh.ev_written_cnt, 0)
+ self.assertEqual(teh.ev_hup_cnt, 1)
+ self.assertEqual(teh.ev_timedout_cnt, 0)
+ self.assertEqual(teh.ev_close_cnt, 1)
+ # The exec worker will expand %h to the host, but ssh will just echo
'%h'
+ self.assertEqual(teh.last_read, '%h'.encode('ascii'))
+ del self.task._info['tree_default:local_workername']
+
def test_tree_run_direct(self):
"""test tree run with direct target, in topology"""
teh = TEventHandler()
