Hello community, here is the log from the commit of package salt for openSUSE:Factory checked in at 2020-03-08 22:22:24 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/salt (Old) and /work/SRC/openSUSE:Factory/.salt.new.26092 (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "salt" Sun Mar 8 22:22:24 2020 rev:99 rq:781848 version:2019.2.3 Changes: -------- --- /work/SRC/openSUSE:Factory/salt/salt.changes 2020-01-18 12:17:37.903122839 +0100 +++ /work/SRC/openSUSE:Factory/.salt.new.26092/salt.changes 2020-03-08 22:22:29.116016296 +0100 @@ -1,0 +2,66 @@ +Thu Mar 5 12:12:35 UTC 2020 - Jochen Breuer <[email protected]> + +- Use full option name instead of undocumented abbreviation for zypper +- Requiring python3-distro only for openSUSE/SLE >= 15 + +- Added: + * use-full-option-name-instead-of-undocumented-abbrevi.patch + +------------------------------------------------------------------- +Thu Mar 5 09:35:29 UTC 2020 - Pablo Suárez Hernández <[email protected]> + +- python-distro is only needed for > Python 3.7. Removing it for Python 2 + +------------------------------------------------------------------- +Wed Mar 4 16:51:34 UTC 2020 - Pablo Suárez Hernández <[email protected]> + +- Avoid possible user escalation upgrading salt-master (bsc#1157465) (CVE-2019-18897) + +------------------------------------------------------------------- +Wed Mar 4 10:29:13 UTC 2020 - Pablo Suárez Hernández <[email protected]> + +- Fix unit tests failures in test_batch_async tests + +- Added: + * fix-unit-tests-for-batch-async-after-refactor.patch + +------------------------------------------------------------------- +Mon Mar 2 10:49:09 UTC 2020 - Pablo Suárez Hernández <[email protected]> + +- Batch Async: Handle exceptions, properly unregister and close instances + after running async batching to avoid CPU starvation of the MWorkers (bsc#1162327) +- RHEL/CentOS 8 uses platform-python instead of python3 +- Enable build for Python 3.8 + +- Added: + * batch_async-avoid-using-fnmatch-to-match-event-217.patch + * apply-patch-from-upstream-to-support-python-3.8.patch + * batch-async-catch-exceptions-and-safety-unregister-a.patch + +------------------------------------------------------------------- +Wed Feb 12 09:16:12 UTC 2020 - Pablo Suárez Hernández <[email protected]> + +- Fix 'os_family' grain for Astra Linux Common Edition + +- Added: + * add-astra-linux-common-edition-to-the-os-family-list.patch + +------------------------------------------------------------------- +Mon Feb 3 10:42:42 UTC 2020 - Jochen Breuer <[email protected]> + +- Update to Salt version 2019.2.3 (CVE-2019-17361) (bsc#1163981) (bsc#1162504) + See release notes: https://docs.saltstack.com/en/latest/topics/releases/2019.2.3.html + +- Modified: + * use-adler32-algorithm-to-compute-string-checksums.patch + +------------------------------------------------------------------- +Wed Jan 29 15:38:36 UTC 2020 - Jochen Breuer <[email protected]> + +- Enable passing grains to start event based on 'start_event_grains' configuration parameter + +- Added: + * restrict-the-start_event_grains-only-to-the-start-ev.patch + * enable-passing-grains-to-start-event-based-on-start_.patch + +------------------------------------------------------------------- @@ -107 +173 @@ -- Replacing pycrypto with M2Crypto +- Replacing pycrypto with M2Crypto (bsc#1165425) Old: ---- v2019.2.2.tar.gz New: ---- add-astra-linux-common-edition-to-the-os-family-list.patch apply-patch-from-upstream-to-support-python-3.8.patch batch-async-catch-exceptions-and-safety-unregister-a.patch batch_async-avoid-using-fnmatch-to-match-event-217.patch enable-passing-grains-to-start-event-based-on-start_.patch fix-unit-tests-for-batch-async-after-refactor.patch restrict-the-start_event_grains-only-to-the-start-ev.patch use-full-option-name-instead-of-undocumented-abbrevi.patch v2019.2.3.tar.gz ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ salt.spec ++++++ --- /var/tmp/diff_new_pack.7wB7I2/_old 2020-03-08 22:22:31.860017989 +0100 +++ /var/tmp/diff_new_pack.7wB7I2/_new 2020-03-08 22:22:31.868017994 +0100 @@ -58,7 +58,7 @@ %bcond_with builddocs Name: salt -Version: 2019.2.2 +Version: 2019.2.3 Release: 0 Summary: A parallel remote execution system License: Apache-2.0 @@ -288,6 +288,22 @@ Patch103: list_downloaded-for-apt-module.patch # PATCH_FIX_OPENSUSE: https://github.com/openSUSE/salt/pull/200 Patch104: support-for-btrfs-and-xfs-in-parted-and-mkfs.patch +# PATCH_FIX_OPENSUSE: https://github.com/openSUSE/salt/pull/204 +Patch105: enable-passing-grains-to-start-event-based-on-start_.patch +# PATCH_FIX_OPENSUSE: https://github.com/openSUSE/salt/pull/205 +Patch106: restrict-the-start_event_grains-only-to-the-start-ev.patch +# PATCH-FIX_UPSTREAM: https://github.com/saltstack/salt/pull/56125 +Patch107: add-astra-linux-common-edition-to-the-os-family-list.patch +# PATCH_FIX_OPENSUSE: https://github.com/openSUSE/salt/pull/211 +Patch108: apply-patch-from-upstream-to-support-python-3.8.patch +# PATCH_FIX_OPENSUSE: https://github.com/openSUSE/salt/pull/217 +Patch109: batch_async-avoid-using-fnmatch-to-match-event-217.patch +# PATCH_FIX_OPENSUSE: https://github.com/openSUSE/salt/commit/8a23030d347b7487328c0395f5e30ef29daf1455 +Patch110: batch-async-catch-exceptions-and-safety-unregister-a.patch +# PATCH_FIX_OPENSUSE: https://github.com/openSUSE/salt/commit/a38adfa2efe40c2b1508b685af0b5d28a6bbcfc8 +Patch111: fix-unit-tests-for-batch-async-after-refactor.patch +# PATCH_FIX_OPENSUSE: https://github.com/openSUSE/salt/pull/218 +Patch112: use-full-option-name-instead-of-undocumented-abbrevi.patch BuildRoot: %{_tmppath}/%{name}-%{version}-build BuildRequires: logrotate @@ -477,7 +493,11 @@ Group: System/Management Requires: %{name} = %{version}-%{release} BuildRequires: python-rpm-macros +%if 0%{?rhel} == 8 +BuildRequires: platform-python +%else BuildRequires: python3 +%endif BuildRequires: python3-devel # requirements/base.txt %if 0%{?rhel} @@ -492,6 +512,7 @@ BuildRequires: python3-msgpack-python > 0.3 BuildRequires: python3-pyzmq >= 2.2.0 %if 0%{?suse_version} >= 1500 +BuildRequires: python3-distro BuildRequires: python3-M2Crypto %else BuildRequires: python3-pycrypto >= 2.6.1 @@ -524,7 +545,11 @@ %if %{with builddocs} BuildRequires: python3-sphinx %endif +%if 0%{?rhel} == 8 +Requires: platform-python +%else Requires: python3 +%endif # %if ! 0%{?suse_version} > 1110 Requires: python3-certifi @@ -548,6 +573,7 @@ Requires: python3-MarkupSafe Requires: python3-msgpack-python > 0.3 %if 0%{?suse_version} >= 1500 +Requires: python3-distro Requires: python3-M2Crypto %else Requires: python3-pycrypto >= 2.6.1 @@ -801,7 +827,7 @@ %prep # %setup -q -n salt-%{version} -%setup -q -n salt-2019.2.2-suse +%setup -q -n salt-2019.2.3-suse cp %{S:1} . cp %{S:5} ./.travis.yml %patch1 -p1 @@ -908,6 +934,14 @@ %patch102 -p1 %patch103 -p1 %patch104 -p1 +%patch105 -p1 +%patch106 -p1 +%patch107 -p1 +%patch108 -p1 +%patch109 -p1 +%patch110 -p1 +%patch111 -p1 +%patch112 -p1 %build %if 0%{?build_py2} @@ -1264,12 +1298,14 @@ # version to actually work. It seems a manual restart of salt-master may # still be required, but at least this will actually work given the file # ownership is correct. + # Symlinks are excluded to avoid possible user escalation (bsc#1157465) (CVE-2019-18897). for file in master.{pem,pub} ; do - [ -f /etc/salt/pki/master/$file ] && chown salt /etc/salt/pki/master/$file + [ -f /etc/salt/pki/master/$file ] && [ ! -L /etc/salt/pki/master/$file ] && chown --no-dereference salt /etc/salt/pki/master/$file done MASTER_CACHE_DIR="/var/cache/salt/master" - [ -d $MASTER_CACHE_DIR ] && chown -R salt:salt $MASTER_CACHE_DIR - [ -f $MASTER_CACHE_DIR/.root_key ] && chown root:root $MASTER_CACHE_DIR/.root_key + [ -d $MASTER_CACHE_DIR ] && find $MASTER_CACHE_DIR -type d | xargs -r chown --no-dereference salt:salt + [ -d $MASTER_CACHE_DIR ] && find $MASTER_CACHE_DIR -type f | xargs -r chown --no-dereference salt:salt + [ -f $MASTER_CACHE_DIR/.root_key ] && chown --no-dereference root:root $MASTER_CACHE_DIR/.root_key true fi %if %{with systemd} ++++++ _lastrevision ++++++ --- /var/tmp/diff_new_pack.7wB7I2/_old 2020-03-08 22:22:31.924018029 +0100 +++ /var/tmp/diff_new_pack.7wB7I2/_new 2020-03-08 22:22:31.924018029 +0100 @@ -1 +1 @@ -d3f65020201314619013243463c3fe8098529e3e \ No newline at end of file +d9f16c8fe9224267baa7b315699270521dda6162 \ No newline at end of file ++++++ _service ++++++ --- /var/tmp/diff_new_pack.7wB7I2/_old 2020-03-08 22:22:31.940018039 +0100 +++ /var/tmp/diff_new_pack.7wB7I2/_new 2020-03-08 22:22:31.940018039 +0100 @@ -3,7 +3,7 @@ <param name="url">https://github.com/openSUSE/salt-packaging.git</param> <param name="subdir">salt</param> <param name="filename">package</param> - <param name="revision">2019.2.2</param> + <param name="revision">2019.2.3</param> <param name="scm">git</param> </service> <service name="extract_file" mode="disabled"> @@ -12,8 +12,8 @@ </service> <service name="download_url" mode="disabled"> <param name="host">codeload.github.com</param> - <param name="path">openSUSE/salt/tar.gz/v2019.2.2-suse</param> - <param name="filename">v2019.2.2.tar.gz</param> + <param name="path">openSUSE/salt/tar.gz/v2019.2.3-suse</param> + <param name="filename">v2019.2.3.tar.gz</param> </service> <service name="update_changelog" mode="disabled"></service> </services> ++++++ add-astra-linux-common-edition-to-the-os-family-list.patch ++++++ >From 27770ed415e010055fea3a2cf599b56cd4aedf5e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Julio=20Gonz=C3=A1lez=20Gil?= <[email protected]> Date: Wed, 12 Feb 2020 10:05:45 +0100 Subject: [PATCH] Add Astra Linux Common Edition to the OS Family list (#209) --- salt/grains/core.py | 1 + tests/unit/grains/test_core.py | 20 ++++++++++++++++++++ 2 files changed, 21 insertions(+) diff --git a/salt/grains/core.py b/salt/grains/core.py index 31ed8a77aa..67e263d37b 100644 --- a/salt/grains/core.py +++ b/salt/grains/core.py @@ -1508,6 +1508,7 @@ _OS_FAMILY_MAP = { 'Funtoo': 'Gentoo', 'AIX': 'AIX', 'TurnKey': 'Debian', + 'AstraLinuxCE': 'Debian', } # Matches any possible format: diff --git a/tests/unit/grains/test_core.py b/tests/unit/grains/test_core.py index 09e197a2e4..700eac2b08 100644 --- a/tests/unit/grains/test_core.py +++ b/tests/unit/grains/test_core.py @@ -605,6 +605,26 @@ class CoreGrainsTestCase(TestCase, LoaderModuleMockMixin): } self._run_os_grains_tests("ubuntu-17.10", _os_release_map, expectation) + @skipIf(not salt.utils.platform.is_linux(), 'System is not Linux') + def test_astralinuxce_2_os_grains(self): + ''' + Test if OS grains are parsed correctly in Astra Linux CE 2.12.22 "orel" + ''' + _os_release_map = { + 'linux_distribution': ('AstraLinuxCE', '2.12.22', 'orel'), + } + expectation = { + 'os': 'AstraLinuxCE', + 'os_family': 'Debian', + 'oscodename': 'orel', + 'osfullname': 'AstraLinuxCE', + 'osrelease': '2.12.22', + 'osrelease_info': (2, 12, 22), + 'osmajorrelease': 2, + 'osfinger': 'AstraLinuxCE-2', + } + self._run_os_grains_tests("astralinuxce-2.12.22", _os_release_map, expectation) + @skipIf(not salt.utils.platform.is_windows(), 'System is not Windows') def test_windows_platform_data(self): ''' -- 2.23.0 ++++++ apply-patch-from-upstream-to-support-python-3.8.patch ++++++ >From 41a0d937511356dc75f4f041a08c1458b4e0ccf4 Mon Sep 17 00:00:00 2001 From: Steve Kowalik <[email protected]> Date: Mon, 17 Feb 2020 15:34:00 +1100 Subject: [PATCH] Apply patch from upstream to support Python 3.8 Apply saltstack/salt#56031 to support Python 3.8, which removed a deprecated module and changed some behaviour. Add a {Build,}Requires on python-distro, since it is now required. --- pkg/suse/salt.spec | 2 ++ salt/config/__init__.py | 4 +++- salt/grains/core.py | 16 ++++++++-------- salt/renderers/stateconf.py | 8 ++++---- tests/unit/modules/test_virt.py | 2 +- 5 files changed, 18 insertions(+), 14 deletions(-) diff --git a/pkg/suse/salt.spec b/pkg/suse/salt.spec index e3e678af3b..0f6a9bc012 100644 --- a/pkg/suse/salt.spec +++ b/pkg/suse/salt.spec @@ -62,6 +62,7 @@ BuildRequires: python-psutil BuildRequires: python-requests >= 1.0.0 BuildRequires: python-tornado >= 4.2.1 BuildRequires: python-yaml +BuildRequires: python-distro # requirements/opt.txt (not all) # BuildRequires: python-MySQL-python # BuildRequires: python-timelib @@ -112,6 +113,7 @@ Requires: python-psutil Requires: python-requests >= 1.0.0 Requires: python-tornado >= 4.2.1 Requires: python-yaml +Requires: python-distro %if 0%{?suse_version} # requirements/opt.txt (not all) Recommends: python-MySQL-python diff --git a/salt/config/__init__.py b/salt/config/__init__.py index 658128dac0..a372200892 100644 --- a/salt/config/__init__.py +++ b/salt/config/__init__.py @@ -3276,7 +3276,9 @@ def apply_cloud_providers_config(overrides, defaults=None): # Merge provided extends keep_looping = False for alias, entries in six.iteritems(providers.copy()): - for driver, details in six.iteritems(entries): + for driver in list(six.iterkeys(entries)): + # Don't use iteritems, because the values of the dictionary will be changed + details = entries[driver] if 'extends' not in details: # Extends resolved or non existing, continue! diff --git a/salt/grains/core.py b/salt/grains/core.py index 67e263d37b..0017c0f472 100644 --- a/salt/grains/core.py +++ b/salt/grains/core.py @@ -40,20 +40,20 @@ except ImportError: __proxyenabled__ = ['*'] __FQDN__ = None -# Extend the default list of supported distros. This will be used for the -# /etc/DISTRO-release checking that is part of linux_distribution() -from platform import _supported_dists -_supported_dists += ('arch', 'mageia', 'meego', 'vmware', 'bluewhite64', - 'slamd64', 'ovs', 'system', 'mint', 'oracle', 'void') - # linux_distribution deprecated in py3.7 try: from platform import linux_distribution as _deprecated_linux_distribution + # Extend the default list of supported distros. This will be used for the + # /etc/DISTRO-release checking that is part of linux_distribution() + from platform import _supported_dists + _supported_dists += ('arch', 'mageia', 'meego', 'vmware', 'bluewhite64', + 'slamd64', 'ovs', 'system', 'mint', 'oracle', 'void') + def linux_distribution(**kwargs): with warnings.catch_warnings(): warnings.simplefilter("ignore") - return _deprecated_linux_distribution(**kwargs) + return _deprecated_linux_distribution(supported_dists=_supported_dists, **kwargs) except ImportError: from distro import linux_distribution @@ -1961,7 +1961,7 @@ def os_data(): ) (osname, osrelease, oscodename) = \ [x.strip('"').strip("'") for x in - linux_distribution(supported_dists=_supported_dists)] + linux_distribution()] # Try to assign these three names based on the lsb info, they tend to # be more accurate than what python gets from /etc/DISTRO-release. # It's worth noting that Ubuntu has patched their Python distribution diff --git a/salt/renderers/stateconf.py b/salt/renderers/stateconf.py index 8cd0235c99..2ce888d5f3 100644 --- a/salt/renderers/stateconf.py +++ b/salt/renderers/stateconf.py @@ -224,10 +224,10 @@ def render(input, saltenv='base', sls='', argline='', **kws): tmplctx = STATE_CONF.copy() if tmplctx: prefix = sls + '::' - for k in six.iterkeys(tmplctx): # iterate over a copy of keys - if k.startswith(prefix): - tmplctx[k[len(prefix):]] = tmplctx[k] - del tmplctx[k] + tmplctx = { + k[len(prefix):] if k.startswith(prefix) else k: v + for k, v in six.iteritems(tmplctx) + } else: tmplctx = {} diff --git a/tests/unit/modules/test_virt.py b/tests/unit/modules/test_virt.py index 719f97a724..3079657a9b 100644 --- a/tests/unit/modules/test_virt.py +++ b/tests/unit/modules/test_virt.py @@ -1256,7 +1256,7 @@ class VirtTestCase(TestCase, LoaderModuleMockMixin): <alias name='net1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/> </interface> - <graphics type='spice' port='5900' autoport='yes' listen='127.0.0.1'> + <graphics type='spice' listen='127.0.0.1' autoport='yes'> <listen type='address' address='127.0.0.1'/> </graphics> <video> -- 2.16.4 ++++++ batch-async-catch-exceptions-and-safety-unregister-a.patch ++++++ >From cd0e4240e11dfe8a656de1c8d8f1c7f5c6655311 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pablo=20Su=C3=A1rez=20Hern=C3=A1ndez?= <[email protected]> Date: Fri, 28 Feb 2020 15:11:53 +0000 Subject: [PATCH] Batch Async: Catch exceptions and safety unregister and close instances --- salt/cli/batch_async.py | 156 +++++++++++++++++++++++----------------- 1 file changed, 89 insertions(+), 67 deletions(-) diff --git a/salt/cli/batch_async.py b/salt/cli/batch_async.py index da069b64bd..b8f272ed67 100644 --- a/salt/cli/batch_async.py +++ b/salt/cli/batch_async.py @@ -13,7 +13,6 @@ import salt.client # pylint: enable=import-error,no-name-in-module,redefined-builtin import logging -import fnmatch log = logging.getLogger(__name__) @@ -104,22 +103,25 @@ class BatchAsync(object): def __event_handler(self, raw): if not self.event: return - mtag, data = self.event.unpack(raw, self.event.serial) - for (pattern, op) in self.patterns: - if mtag.startswith(pattern[:-1]): - minion = data['id'] - if op == 'ping_return': - self.minions.add(minion) - if self.targeted_minions == self.minions: - self.event.io_loop.spawn_callback(self.start_batch) - elif op == 'find_job_return': - if data.get("return", None): - self.find_job_returned.add(minion) - elif op == 'batch_run': - if minion in self.active: - self.active.remove(minion) - self.done_minions.add(minion) - self.event.io_loop.spawn_callback(self.schedule_next) + try: + mtag, data = self.event.unpack(raw, self.event.serial) + for (pattern, op) in self.patterns: + if mtag.startswith(pattern[:-1]): + minion = data['id'] + if op == 'ping_return': + self.minions.add(minion) + if self.targeted_minions == self.minions: + self.event.io_loop.spawn_callback(self.start_batch) + elif op == 'find_job_return': + if data.get("return", None): + self.find_job_returned.add(minion) + elif op == 'batch_run': + if minion in self.active: + self.active.remove(minion) + self.done_minions.add(minion) + self.event.io_loop.spawn_callback(self.schedule_next) + except Exception as ex: + log.error("Exception occured while processing event: {}".format(ex)) def _get_next(self): to_run = self.minions.difference( @@ -146,54 +148,59 @@ class BatchAsync(object): if timedout_minions: self.schedule_next() - if running: + if self.event and running: self.find_job_returned = self.find_job_returned.difference(running) self.event.io_loop.spawn_callback(self.find_job, running) @tornado.gen.coroutine def find_job(self, minions): - not_done = minions.difference(self.done_minions).difference(self.timedout_minions) - - if not_done: - jid = self.jid_gen() - find_job_return_pattern = 'salt/job/{0}/ret/*'.format(jid) - self.patterns.add((find_job_return_pattern, "find_job_return")) - self.event.subscribe(find_job_return_pattern, match_type='glob') - - ret = yield self.local.run_job_async( - not_done, - 'saltutil.find_job', - [self.batch_jid], - 'list', - gather_job_timeout=self.opts['gather_job_timeout'], - jid=jid, - **self.eauth) - yield tornado.gen.sleep(self.opts['gather_job_timeout']) - self.event.io_loop.spawn_callback( - self.check_find_job, - not_done, - jid) + if self.event: + not_done = minions.difference(self.done_minions).difference(self.timedout_minions) + try: + if not_done: + jid = self.jid_gen() + find_job_return_pattern = 'salt/job/{0}/ret/*'.format(jid) + self.patterns.add((find_job_return_pattern, "find_job_return")) + self.event.subscribe(find_job_return_pattern, match_type='glob') + ret = yield self.local.run_job_async( + not_done, + 'saltutil.find_job', + [self.batch_jid], + 'list', + gather_job_timeout=self.opts['gather_job_timeout'], + jid=jid, + **self.eauth) + yield tornado.gen.sleep(self.opts['gather_job_timeout']) + if self.event: + self.event.io_loop.spawn_callback( + self.check_find_job, + not_done, + jid) + except Exception as ex: + log.error("Exception occured handling batch async: {}. Aborting execution.".format(ex)) + self.close_safe() @tornado.gen.coroutine def start(self): - self.__set_event_handler() - ping_return = yield self.local.run_job_async( - self.opts['tgt'], - 'test.ping', - [], - self.opts.get( - 'selected_target_option', - self.opts.get('tgt_type', 'glob') - ), - gather_job_timeout=self.opts['gather_job_timeout'], - jid=self.ping_jid, - metadata=self.metadata, - **self.eauth) - self.targeted_minions = set(ping_return['minions']) - #start batching even if not all minions respond to ping - yield tornado.gen.sleep(self.batch_presence_ping_timeout or self.opts['gather_job_timeout']) - self.event.io_loop.spawn_callback(self.start_batch) - + if self.event: + self.__set_event_handler() + ping_return = yield self.local.run_job_async( + self.opts['tgt'], + 'test.ping', + [], + self.opts.get( + 'selected_target_option', + self.opts.get('tgt_type', 'glob') + ), + gather_job_timeout=self.opts['gather_job_timeout'], + jid=self.ping_jid, + metadata=self.metadata, + **self.eauth) + self.targeted_minions = set(ping_return['minions']) + #start batching even if not all minions respond to ping + yield tornado.gen.sleep(self.batch_presence_ping_timeout or self.opts['gather_job_timeout']) + if self.event: + self.event.io_loop.spawn_callback(self.start_batch) @tornado.gen.coroutine def start_batch(self): @@ -206,7 +213,8 @@ class BatchAsync(object): "metadata": self.metadata } ret = self.event.fire_event(data, "salt/batch/{0}/start".format(self.batch_jid)) - self.event.io_loop.spawn_callback(self.run_next) + if self.event: + self.event.io_loop.spawn_callback(self.run_next) @tornado.gen.coroutine def end_batch(self): @@ -221,11 +229,21 @@ class BatchAsync(object): "metadata": self.metadata } self.event.fire_event(data, "salt/batch/{0}/done".format(self.batch_jid)) - for (pattern, label) in self.patterns: - if label in ["ping_return", "batch_run"]: - self.event.unsubscribe(pattern, match_type='glob') - del self - gc.collect() + + # release to the IOLoop to allow the event to be published + # before closing batch async execution + yield tornado.gen.sleep(1) + self.close_safe() + + def close_safe(self): + for (pattern, label) in self.patterns: + self.event.unsubscribe(pattern, match_type='glob') + self.event.remove_event_handler(self.__event_handler) + self.event = None + self.local = None + self.ioloop = None + del self + gc.collect() @tornado.gen.coroutine def schedule_next(self): @@ -233,7 +251,8 @@ class BatchAsync(object): self.scheduled = True # call later so that we maybe gather more returns yield tornado.gen.sleep(self.batch_delay) - self.event.io_loop.spawn_callback(self.run_next) + if self.event: + self.event.io_loop.spawn_callback(self.run_next) @tornado.gen.coroutine def run_next(self): @@ -254,17 +273,20 @@ class BatchAsync(object): metadata=self.metadata) yield tornado.gen.sleep(self.opts['timeout']) - self.event.io_loop.spawn_callback(self.find_job, set(next_batch)) + + # The batch can be done already at this point, which means no self.event + if self.event: + self.event.io_loop.spawn_callback(self.find_job, set(next_batch)) except Exception as ex: - log.error("Error in scheduling next batch: %s", ex) + log.error("Error in scheduling next batch: %s. Aborting execution", ex) self.active = self.active.difference(next_batch) + self.close_safe() else: yield self.end_batch() gc.collect() def __del__(self): self.local = None - self.event.remove_event_handler(self.__event_handler) self.event = None self.ioloop = None gc.collect() -- 2.23.0 ++++++ batch_async-avoid-using-fnmatch-to-match-event-217.patch ++++++ >From 885940513b7a5c800fcc33dd47f2e92e864ec230 Mon Sep 17 00:00:00 2001 From: Silvio Moioli <[email protected]> Date: Mon, 2 Mar 2020 11:23:59 +0100 Subject: [PATCH] batch_async: avoid using fnmatch to match event (#217) --- salt/cli/batch_async.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/salt/cli/batch_async.py b/salt/cli/batch_async.py index c4545e3ebc..da069b64bd 100644 --- a/salt/cli/batch_async.py +++ b/salt/cli/batch_async.py @@ -106,7 +106,7 @@ class BatchAsync(object): return mtag, data = self.event.unpack(raw, self.event.serial) for (pattern, op) in self.patterns: - if fnmatch.fnmatch(mtag, pattern): + if mtag.startswith(pattern[:-1]): minion = data['id'] if op == 'ping_return': self.minions.add(minion) -- 2.23.0 ++++++ enable-passing-grains-to-start-event-based-on-start_.patch ++++++ >From 79f566ba99f48d60e4948c9b8df9c6b6f497eced Mon Sep 17 00:00:00 2001 From: Abid Mehmood <[email protected]> Date: Thu, 1 Aug 2019 13:14:22 +0200 Subject: [PATCH] enable passing grains to start event based on 'start_event_grains' configuration parameter unit tests --- conf/minion | 5 +++++ doc/ref/configuration/minion.rst | 15 +++++++++++++++ salt/config/__init__.py | 1 + salt/minion.py | 5 +++++ tests/unit/test_minion.py | 32 ++++++++++++++++++++++++++++++++ 5 files changed, 58 insertions(+) diff --git a/conf/minion b/conf/minion index f2b6655932..cc7e962120 100644 --- a/conf/minion +++ b/conf/minion @@ -548,6 +548,11 @@ # - edit.vim # - hyper # +# List of grains to pass in start event when minion starts up: +#start_event_grains: +# - machine_id +# - uuid +# # Top file to execute if startup_states is 'top': #top_file: '' diff --git a/doc/ref/configuration/minion.rst b/doc/ref/configuration/minion.rst index 30343ebd8e..fe556ef9ce 100644 --- a/doc/ref/configuration/minion.rst +++ b/doc/ref/configuration/minion.rst @@ -2000,6 +2000,21 @@ List of states to run when the minion starts up if ``startup_states`` is set to - edit.vim - hyper +.. conf_minion:: start_event_grains + +``start_event_grains`` +---------------------- + +Default: ``[]`` + +List of grains to pass in start event when minion starts up. + +.. code-block:: yaml + + start_event_grains: + - machine_id + - uuid + .. conf_minion:: top_file ``top_file`` diff --git a/salt/config/__init__.py b/salt/config/__init__.py index ee90eb3120..788128b903 100644 --- a/salt/config/__init__.py +++ b/salt/config/__init__.py @@ -1283,6 +1283,7 @@ DEFAULT_MINION_OPTS = { 'state_top_saltenv': None, 'startup_states': '', 'sls_list': [], + 'start_event_grains': [], 'top_file': '', 'thoriumenv': None, 'thorium_top': 'top.sls', diff --git a/salt/minion.py b/salt/minion.py index 2c0a18604e..2b53da4f18 100644 --- a/salt/minion.py +++ b/salt/minion.py @@ -1456,6 +1456,11 @@ class Minion(MinionBase): else: return + if self.opts['start_event_grains']: + grains_to_add = dict( + [(k, v) for k, v in six.iteritems(self.opts.get('grains', {})) if k in self.opts['start_event_grains']]) + load['grains'] = grains_to_add + if sync: try: self._send_req_sync(load, timeout) diff --git a/tests/unit/test_minion.py b/tests/unit/test_minion.py index b78e0f6abd..83215151ee 100644 --- a/tests/unit/test_minion.py +++ b/tests/unit/test_minion.py @@ -305,6 +305,38 @@ class MinionTestCase(TestCase, AdaptedConfigurationTestCaseMixin): finally: minion.destroy() + def test_when_passed_start_event_grains(self): + mock_opts = self.get_config('minion', from_scratch=True) + mock_opts['start_event_grains'] = ["os"] + io_loop = tornado.ioloop.IOLoop() + io_loop.make_current() + minion = salt.minion.Minion(mock_opts, io_loop=io_loop) + try: + minion.tok = MagicMock() + minion._send_req_sync = MagicMock() + minion._fire_master('Minion has started', 'minion_start') + load = minion._send_req_sync.call_args[0][0] + + self.assertTrue('grains' in load) + self.assertTrue('os' in load['grains']) + finally: + minion.destroy() + + def test_when_not_passed_start_event_grains(self): + mock_opts = self.get_config('minion', from_scratch=True) + io_loop = tornado.ioloop.IOLoop() + io_loop.make_current() + minion = salt.minion.Minion(mock_opts, io_loop=io_loop) + try: + minion.tok = MagicMock() + minion._send_req_sync = MagicMock() + minion._fire_master('Minion has started', 'minion_start') + load = minion._send_req_sync.call_args[0][0] + + self.assertTrue('grains' not in load) + finally: + minion.destroy() + def test_minion_retry_dns_count(self): ''' Tests that the resolve_dns will retry dns look ups for a maximum of -- 2.16.4 ++++++ fix-unit-tests-for-batch-async-after-refactor.patch ++++++ >From 357865060d439e7bf54e5971deb68c0d70cfe91b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pablo=20Su=C3=A1rez=20Hern=C3=A1ndez?= <[email protected]> Date: Wed, 4 Mar 2020 10:13:43 +0000 Subject: [PATCH] Fix unit tests for batch async after refactor --- tests/unit/cli/test_batch_async.py | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/tests/unit/cli/test_batch_async.py b/tests/unit/cli/test_batch_async.py index f1d36a81fb..e1ce60859b 100644 --- a/tests/unit/cli/test_batch_async.py +++ b/tests/unit/cli/test_batch_async.py @@ -126,9 +126,10 @@ class AsyncBatchTestCase(AsyncTestCase, TestCase): self.batch.timedout_minions = {'bar'} self.batch.event = MagicMock() self.batch.metadata = {'mykey': 'myvalue'} + old_event = self.batch.event self.batch.end_batch() self.assertEqual( - self.batch.event.fire_event.call_args[0], + old_event.fire_event.call_args[0], ( { 'available_minions': set(['foo', 'bar']), @@ -146,6 +147,21 @@ class AsyncBatchTestCase(AsyncTestCase, TestCase): event = MagicMock() batch.event = event batch.__del__() + self.assertEqual(batch.local, None) + self.assertEqual(batch.event, None) + self.assertEqual(batch.ioloop, None) + + def test_batch_close_safe(self): + batch = BatchAsync(MagicMock(), MagicMock(), MagicMock()) + event = MagicMock() + batch.event = event + batch.patterns = { ('salt/job/1234/ret/*', 'find_job_return'), ('salt/job/4321/ret/*', 'find_job_return') } + batch.close_safe() + self.assertEqual(batch.local, None) + self.assertEqual(batch.event, None) + self.assertEqual(batch.ioloop, None) + self.assertEqual( + len(event.unsubscribe.mock_calls), 2) self.assertEqual( len(event.remove_event_handler.mock_calls), 1) -- 2.23.0 ++++++ restrict-the-start_event_grains-only-to-the-start-ev.patch ++++++ >From 127f0fa48ed2e5f05320c6c82f928f0d59a94bd9 Mon Sep 17 00:00:00 2001 From: Abid Mehmood <[email protected]> Date: Thu, 16 Jan 2020 11:28:04 +0100 Subject: [PATCH] Restrict the 'start_event_grains' only to the start events add test for custom events --- salt/minion.py | 11 ++++++++--- tests/unit/test_minion.py | 18 +++++++++++++++++- 2 files changed, 25 insertions(+), 4 deletions(-) diff --git a/salt/minion.py b/salt/minion.py index 2b53da4f18..a2fb203bfe 100644 --- a/salt/minion.py +++ b/salt/minion.py @@ -1437,7 +1437,7 @@ class Minion(MinionBase): finally: channel.close() - def _fire_master(self, data=None, tag=None, events=None, pretag=None, timeout=60, sync=True, timeout_handler=None): + def _fire_master(self, data=None, tag=None, events=None, pretag=None, timeout=60, sync=True, timeout_handler=None, include_startup_grains=False): ''' Fire an event on the master, or drop message if unable to send. ''' @@ -1456,7 +1456,7 @@ class Minion(MinionBase): else: return - if self.opts['start_event_grains']: + if include_startup_grains: grains_to_add = dict( [(k, v) for k, v in six.iteritems(self.opts.get('grains', {})) if k in self.opts['start_event_grains']]) load['grains'] = grains_to_add @@ -2149,6 +2149,9 @@ class Minion(MinionBase): }) def _fire_master_minion_start(self): + include_grains = False + if self.opts['start_event_grains']: + include_grains = True # Send an event to the master that the minion is live if self.opts['enable_legacy_startup_events']: # Old style event. Defaults to False in Sodium release. @@ -2157,7 +2160,8 @@ class Minion(MinionBase): self.opts['id'], time.asctime() ), - 'minion_start' + 'minion_start', + include_startup_grains=include_grains ) # send name spaced event self._fire_master( @@ -2166,6 +2170,7 @@ class Minion(MinionBase): time.asctime() ), tagify([self.opts['id'], 'start'], 'minion'), + include_startup_grains=include_grains ) def module_refresh(self, force_refresh=False, notify=False): diff --git a/tests/unit/test_minion.py b/tests/unit/test_minion.py index 83215151ee..41dfab0f45 100644 --- a/tests/unit/test_minion.py +++ b/tests/unit/test_minion.py @@ -314,7 +314,7 @@ class MinionTestCase(TestCase, AdaptedConfigurationTestCaseMixin): try: minion.tok = MagicMock() minion._send_req_sync = MagicMock() - minion._fire_master('Minion has started', 'minion_start') + minion._fire_master('Minion has started', 'minion_start', include_startup_grains=True) load = minion._send_req_sync.call_args[0][0] self.assertTrue('grains' in load) @@ -337,6 +337,22 @@ class MinionTestCase(TestCase, AdaptedConfigurationTestCaseMixin): finally: minion.destroy() + def test_when_other_events_fired_and_start_event_grains_are_set(self): + mock_opts = self.get_config('minion', from_scratch=True) + mock_opts['start_event_grains'] = ["os"] + io_loop = tornado.ioloop.IOLoop() + io_loop.make_current() + minion = salt.minion.Minion(mock_opts, io_loop=io_loop) + try: + minion.tok = MagicMock() + minion._send_req_sync = MagicMock() + minion._fire_master('Custm_event_fired', 'custom_event') + load = minion._send_req_sync.call_args[0][0] + + self.assertTrue('grains' not in load) + finally: + minion.destroy() + def test_minion_retry_dns_count(self): ''' Tests that the resolve_dns will retry dns look ups for a maximum of -- 2.16.4 ++++++ use-adler32-algorithm-to-compute-string-checksums.patch ++++++ --- /var/tmp/diff_new_pack.7wB7I2/_old 2020-03-08 22:22:32.244018226 +0100 +++ /var/tmp/diff_new_pack.7wB7I2/_new 2020-03-08 22:22:32.244018226 +0100 @@ -1,4 +1,4 @@ -From 815042712007e3bc9e5326b3a13b2880534630b9 Mon Sep 17 00:00:00 2001 +From c1e2410749479940275a217d598b22b24eef8758 Mon Sep 17 00:00:00 2001 From: Bo Maryniuk <[email protected]> Date: Sat, 28 Jul 2018 22:59:04 +0200 Subject: [PATCH] Use Adler32 algorithm to compute string checksums @@ -20,13 +20,13 @@ 2 files changed, 41 insertions(+), 19 deletions(-) diff --git a/salt/config/__init__.py b/salt/config/__init__.py -index b3dfc8e43d..ee90eb3120 100644 +index f4496f3a61..b272610fa4 100644 --- a/salt/config/__init__.py +++ b/salt/config/__init__.py -@@ -1217,6 +1217,10 @@ VALID_OPTS = { - - # Thorium top file location - 'thorium_top': six.string_types, +@@ -1221,6 +1221,10 @@ VALID_OPTS = { + # Allow raw_shell option when using the ssh + # client via the Salt API + 'netapi_allow_raw_shell': bool, + + # Use Adler32 hashing algorithm for server_id (default False until Sodium, "adler32" after) + # Possible values are: False, adler32, crc32 @@ -34,7 +34,7 @@ } # default configurations -@@ -1525,7 +1529,8 @@ DEFAULT_MINION_OPTS = { +@@ -1529,7 +1533,8 @@ DEFAULT_MINION_OPTS = { }, 'discovery': False, 'schedule': {}, ++++++ use-full-option-name-instead-of-undocumented-abbrevi.patch ++++++ >From fb82c59e6de2a31f60c9f8a23f1eed4e24009dcf Mon Sep 17 00:00:00 2001 From: Michael Calmer <[email protected]> Date: Sun, 1 Mar 2020 16:22:54 +0100 Subject: [PATCH] use full option name instead of undocumented abbreviation --- salt/modules/zypperpkg.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/salt/modules/zypperpkg.py b/salt/modules/zypperpkg.py index 8c1e05c21c..19d1fd96c7 100644 --- a/salt/modules/zypperpkg.py +++ b/salt/modules/zypperpkg.py @@ -2494,7 +2494,7 @@ def list_products(all=False, refresh=False, root=None): OEM_PATH = os.path.join(root, os.path.relpath(OEM_PATH, os.path.sep)) cmd = list() if not all: - cmd.append('--disable-repos') + cmd.append('--disable-repositories') cmd.append('products') if not all: cmd.append('-i') -- 2.16.4 ++++++ v2019.2.2.tar.gz -> v2019.2.3.tar.gz ++++++ /work/SRC/openSUSE:Factory/salt/v2019.2.2.tar.gz /work/SRC/openSUSE:Factory/.salt.new.26092/v2019.2.3.tar.gz differ: char 14, line 1
