Hello community, here is the log from the commit of package salt for openSUSE:Factory checked in at 2017-10-17 01:52:09 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/salt (Old) and /work/SRC/openSUSE:Factory/.salt.new (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "salt" Tue Oct 17 01:52:09 2017 rev:65 rq:533952 version:2017.7.2 Changes: -------- --- /work/SRC/openSUSE:Factory/salt/salt.changes 2017-08-28 15:10:56.717190891 +0200 +++ /work/SRC/openSUSE:Factory/.salt.new/salt.changes 2017-10-17 01:52:10.691389630 +0200 @@ -1,0 +2,57 @@ +Wed Oct 11 14:15:17 UTC 2017 - [email protected] + +- Add possibility to generate _version.py at the build time for + raw builds: https://github.com/saltstack/salt/pull/43955 + +- Added: + * enable-with-salt-version-parameter-for-setup.py-scri.patch + +------------------------------------------------------------------- +Tue Oct 10 15:31:36 UTC 2017 - [email protected] + +- Update to 2017.7.2 + See https://docs.saltstack.com/en/develop/topics/releases/2017.7.2.html + for full changelog +- Fix for CVE-2017-14695 (bsc#1062462) +- Fix for CVE-2017-14696 (bsc#1062464) +- Fix salt target-type field returns "String" for existing + jids but an empty "Array" for non existing jids. (issue #1711) + +- Added: + * bugfix-always-return-a-string-list-on-unknown-job-ta.patch + + +------------------------------------------------------------------- +Fri Oct 6 13:53:59 UTC 2017 - [email protected] + +- Fixed minion resource exhaustion when many functions are being + executed in parallel (bsc#1059758) + +- Added: + * introduce-process_count_max-minion-configuration-par.patch + * multiprocessing-minion-option-documentation-fixes.patch + +------------------------------------------------------------------- +Thu Oct 5 15:37:06 UTC 2017 - [email protected] + +- Remove 'TasksTask' attribute from salt-master.service in older + versions of systemd (bsc#985112) +- Provide custom SUSE salt-master.service file. + +------------------------------------------------------------------- +Tue Oct 3 11:54:02 UTC 2017 - [email protected] + +- Fix wrong version reported by Salt (bsc#1061407) + +------------------------------------------------------------------- +Tue Sep 12 07:26:03 UTC 2017 - [email protected] + +- list_pkgs: add parameter for returned attribute selection (bsc#1052264) +- Adding the leftover for zypper and yum list_pkgs functionality. +- Use $HOME to get the user home directory instead using '~' char (bsc#1042749) + +- Added: + * list_pkgs-add-parameter-for-returned-attribute-selec.patch + * use-home-to-get-the-user-home-directory-instead-usin.patch + +------------------------------------------------------------------- Old: ---- v2017.7.1.tar.gz New: ---- bugfix-always-return-a-string-list-on-unknown-job-ta.patch enable-with-salt-version-parameter-for-setup.py-scri.patch introduce-process_count_max-minion-configuration-par.patch list_pkgs-add-parameter-for-returned-attribute-selec.patch multiprocessing-minion-option-documentation-fixes.patch use-home-to-get-the-user-home-directory-instead-usin.patch v2017.7.2.tar.gz ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ salt.spec ++++++ --- /var/tmp/diff_new_pack.tIOwR8/_old 2017-10-17 01:52:12.591300632 +0200 +++ /var/tmp/diff_new_pack.tIOwR8/_new 2017-10-17 01:52:12.595300445 +0200 @@ -36,19 +36,36 @@ %bcond_with builddocs Name: salt -Version: 2017.7.1 +Version: 2017.7.2 Release: 0 Summary: A parallel remote execution system License: Apache-2.0 Group: System/Management Url: http://saltstack.org/ -Source: https://github.com/saltstack/salt/archive/v2017.7.1.tar.gz +Source: https://github.com/saltstack/salt/archive/v%{version}.tar.gz Source1: README.SUSE Source2: salt-tmpfiles.d Source3: html.tar.bz2 Source4: update-documentation.sh Source5: travis.yml +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/42310 +# https://github.com/saltstack/salt/pull/42718 +# https://github.com/saltstack/salt/pull/43195 +# https://github.com/saltstack/salt/pull/43214 +# https://github.com/saltstack/salt/pull/43281 +Patch1: list_pkgs-add-parameter-for-returned-attribute-selec.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/43441 +Patch2: use-home-to-get-the-user-home-directory-instead-usin.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/43663 +Patch3: multiprocessing-minion-option-documentation-fixes.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/43669 +Patch4: introduce-process_count_max-minion-configuration-par.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/ +Patch5: bugfix-always-return-a-string-list-on-unknown-job-ta.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/43955 +Patch6: enable-with-salt-version-parameter-for-setup.py-scri.patch + BuildRoot: %{_tmppath}/%{name}-%{version}-build BuildRequires: fdupes BuildRequires: logrotate @@ -386,8 +403,16 @@ cp %{S:1} . cp %{S:5} ./.travis.yml +%patch1 -p1 +%patch2 -p1 +%patch3 -p1 +%patch4 -p1 +%patch5 -p1 +%patch6 -p1 + %build -%{__python} setup.py --salt-transport=both build +%{__python} setup.py --with-salt-version=%{version} --salt-transport=both build +cp ./build/lib/salt/_version.py ./salt %if %{with docs} && %{without builddocs} # extract docs from the tarball @@ -459,7 +484,7 @@ ## install init and systemd scripts %if %{with systemd} -install -Dpm 0644 pkg/salt-master.service %{buildroot}%{_unitdir}/salt-master.service +install -Dpm 0644 pkg/suse/salt-master.service %{buildroot}%{_unitdir}/salt-master.service %if 0%{?suse_version} install -Dpm 0644 pkg/suse/salt-minion.service %{buildroot}%{_unitdir}/salt-minion.service %else @@ -692,6 +717,11 @@ true fi %if %{with systemd} +if [ `rpm -q systemd --queryformat="%{VERSION}"` -lt 228 ]; then + # On systemd < 228 the 'TasksTask' attribute is not available. + # Removing TasksMax from salt-master.service on SLE12SP1 LTSS (bsc#985112) + sed -i '/TasksMax=infinity/d' %{_unitdir}/salt-master.service +fi %if 0%{?suse_version} %service_add_post salt-master.service %fillup_only ++++++ bugfix-always-return-a-string-list-on-unknown-job-ta.patch ++++++ >From 49a4e807fb1cb844cec7b7c11b37f6c276f899e4 Mon Sep 17 00:00:00 2001 From: Bo Maryniuk <[email protected]> Date: Mon, 9 Oct 2017 17:57:48 +0200 Subject: [PATCH] Bugfix: always return a string "list" on unknown job target type. --- salt/returners/couchbase_return.py | 2 +- salt/returners/postgres_local_cache.py | 2 +- salt/runners/jobs.py | 2 +- salt/utils/jid.py | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/salt/returners/couchbase_return.py b/salt/returners/couchbase_return.py index 24c3a9105a..f5adecc2e7 100644 --- a/salt/returners/couchbase_return.py +++ b/salt/returners/couchbase_return.py @@ -309,7 +309,7 @@ def _format_job_instance(job): 'Arguments': list(job.get('arg', [])), # unlikely but safeguard from invalid returns 'Target': job.get('tgt', 'unknown-target'), - 'Target-type': job.get('tgt_type', []), + 'Target-type': job.get('tgt_type', 'list'), 'User': job.get('user', 'root')} if 'metadata' in job: diff --git a/salt/returners/postgres_local_cache.py b/salt/returners/postgres_local_cache.py index 422f8c77c7..28dc2f565c 100644 --- a/salt/returners/postgres_local_cache.py +++ b/salt/returners/postgres_local_cache.py @@ -180,7 +180,7 @@ def _format_job_instance(job): 'Arguments': json.loads(job.get('arg', '[]')), # unlikely but safeguard from invalid returns 'Target': job.get('tgt', 'unknown-target'), - 'Target-type': job.get('tgt_type', []), + 'Target-type': job.get('tgt_type', 'list'), 'User': job.get('user', 'root')} # TODO: Add Metadata support when it is merged from develop return ret diff --git a/salt/runners/jobs.py b/salt/runners/jobs.py index 82abd56eae..fae7942e38 100644 --- a/salt/runners/jobs.py +++ b/salt/runners/jobs.py @@ -542,7 +542,7 @@ def _format_job_instance(job): 'Arguments': list(job.get('arg', [])), # unlikely but safeguard from invalid returns 'Target': job.get('tgt', 'unknown-target'), - 'Target-type': job.get('tgt_type', []), + 'Target-type': job.get('tgt_type', 'list'), 'User': job.get('user', 'root')} if 'metadata' in job: diff --git a/salt/utils/jid.py b/salt/utils/jid.py index 3f4ef296a2..4dbf0d2c6f 100644 --- a/salt/utils/jid.py +++ b/salt/utils/jid.py @@ -65,7 +65,7 @@ def format_job_instance(job): 'Arguments': list(job.get('arg', [])), # unlikely but safeguard from invalid returns 'Target': job.get('tgt', 'unknown-target'), - 'Target-type': job.get('tgt_type', []), + 'Target-type': job.get('tgt_type', 'list'), 'User': job.get('user', 'root')} if 'metadata' in job: -- 2.14.2 ++++++ enable-with-salt-version-parameter-for-setup.py-scri.patch ++++++ >From 1949261a504fd01e057b41126d78f142f4977204 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pablo=20Su=C3=A1rez=20Hern=C3=A1ndez?= <[email protected]> Date: Fri, 6 Oct 2017 17:12:15 +0100 Subject: [PATCH] Enable '--with-salt-version' parameter for setup.py script --- setup.py | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/setup.py b/setup.py index effdc2f230..519f753401 100755 --- a/setup.py +++ b/setup.py @@ -183,17 +183,22 @@ class WriteSaltVersion(Command): ''' def run(self): - if not os.path.exists(SALT_VERSION_HARDCODED): + if not os.path.exists(SALT_VERSION_HARDCODED) or self.distribution.with_salt_version: # Write the version file if getattr(self.distribution, 'salt_version_hardcoded_path', None) is None: print('This command is not meant to be called on it\'s own') exit(1) + if not self.distribution.with_salt_version: + salt_version = __saltstack_version__ + else: + salt_version = SaltStackVersion.parse(self.distribution.with_salt_version) + # pylint: disable=E0602 open(self.distribution.salt_version_hardcoded_path, 'w').write( INSTALL_VERSION_TEMPLATE.format( date=DATE, - full_version_info=__saltstack_version__.full_info + full_version_info=salt_version.full_info ) ) # pylint: enable=E0602 @@ -731,6 +736,13 @@ class Build(build): def run(self): # Run build.run function build.run(self) + if getattr(self.distribution, 'with_salt_version', False): + # Write the hardcoded salt version module salt/_version.py + self.distribution.salt_version_hardcoded_path = os.path.join( + self.build_lib, 'salt', '_version.py' + ) + self.run_command('write_salt_version') + if getattr(self.distribution, 'running_salt_install', False): # If our install attribute is present and set to True, we'll go # ahead and write our install time python modules. @@ -839,6 +851,7 @@ class SaltDistribution(distutils.dist.Distribution): ('ssh-packaging', None, 'Run in SSH packaging mode'), ('salt-transport=', None, 'The transport to prepare salt for. Choices are \'zeromq\' ' '\'raet\' or \'both\'. Defaults to \'zeromq\'', 'zeromq')] + [ + ('with-salt-version=', None, 'Set a fixed version for Salt instead calculating it'), # Salt's Paths Configuration Settings ('salt-root-dir=', None, 'Salt\'s pre-configured root directory'), @@ -893,6 +906,9 @@ class SaltDistribution(distutils.dist.Distribution): self.salt_spm_pillar_dir = None self.salt_spm_reactor_dir = None + # Salt version + self.with_salt_version = None + self.name = 'salt-ssh' if PACKAGED_FOR_SALT_SSH else 'salt' self.salt_version = __version__ # pylint: disable=undefined-variable self.description = 'Portable, distributed, remote execution and configuration management system' -- 2.14.2 ++++++ introduce-process_count_max-minion-configuration-par.patch ++++++ >From dd3490c7e56a0aca17f73dde1684d469fe5582df Mon Sep 17 00:00:00 2001 From: Silvio Moioli <[email protected]> Date: Wed, 20 Sep 2017 14:33:33 +0200 Subject: [PATCH] Introduce process_count_max minion configuration parameter This allows users to limit the number of processes or threads a minion will start in response to published messages, prevents resource exhaustion in case a high number of concurrent jobs is scheduled in a short time. process_count_max: add defaults and documentation process_count_max: adapt existing unit tests process_count_max: add unit test process_count_max: disable by default --- conf/minion | 6 +++++ doc/ref/configuration/minion.rst | 17 +++++++++++++ salt/config/__init__.py | 4 +++ salt/minion.py | 10 ++++++++ tests/unit/test_minion.py | 53 +++++++++++++++++++++++++++++++++++++--- 5 files changed, 87 insertions(+), 3 deletions(-) diff --git a/conf/minion b/conf/minion index b1122c9e52..6d2b43da1b 100644 --- a/conf/minion +++ b/conf/minion @@ -671,6 +671,12 @@ # for a full explanation. #multiprocessing: True +# Limit the maximum amount of processes or threads created by salt-minion. +# This is useful to avoid resource exhaustion in case the minion receives more +# publications than it is able to handle, as it limits the number of spawned +# processes or threads. -1 is the default and disables the limit. +#process_count_max: -1 + ##### Logging settings ##### ########################################## diff --git a/doc/ref/configuration/minion.rst b/doc/ref/configuration/minion.rst index 5cc72f1daf..b935f86656 100644 --- a/doc/ref/configuration/minion.rst +++ b/doc/ref/configuration/minion.rst @@ -2179,6 +2179,23 @@ executed in a thread. multiprocessing: True +.. conf_minion:: process_count_max + +``process_count_max`` +------- + +.. versionadded:: Oxygen + +Default: ``-1`` + +Limit the maximum amount of processes or threads created by ``salt-minion``. +This is useful to avoid resource exhaustion in case the minion receives more +publications than it is able to handle, as it limits the number of spawned +processes or threads. ``-1`` is the default and disables the limit. + +.. code-block:: yaml + + process_count_max: -1 .. _minion-logging-settings: diff --git a/salt/config/__init__.py b/salt/config/__init__.py index b5b7f2a1f7..e843d8c79d 100644 --- a/salt/config/__init__.py +++ b/salt/config/__init__.py @@ -328,6 +328,9 @@ VALID_OPTS = { # Whether or not processes should be forked when needed. The alternative is to use threading. 'multiprocessing': bool, + # Maximum number of concurrently active processes at any given point in time + 'process_count_max': int, + # Whether or not the salt minion should run scheduled mine updates 'mine_enabled': bool, @@ -1193,6 +1196,7 @@ DEFAULT_MINION_OPTS = { 'auto_accept': True, 'autosign_timeout': 120, 'multiprocessing': True, + 'process_count_max': -1, 'mine_enabled': True, 'mine_return_job': False, 'mine_interval': 60, diff --git a/salt/minion.py b/salt/minion.py index 3c5046ee93..9e7301aafa 100644 --- a/salt/minion.py +++ b/salt/minion.py @@ -1288,6 +1288,7 @@ class Minion(MinionBase): self._send_req_async(load, timeout, callback=lambda f: None) # pylint: disable=unexpected-keyword-arg return True + @tornado.gen.coroutine def _handle_decoded_payload(self, data): ''' Override this method if you wish to handle the decoded data @@ -1319,6 +1320,15 @@ class Minion(MinionBase): self.functions, self.returners, self.function_errors, self.executors = self._load_modules() self.schedule.functions = self.functions self.schedule.returners = self.returners + + process_count_max = self.opts.get('process_count_max') + if process_count_max > 0: + process_count = len(salt.utils.minion.running(self.opts)) + while process_count >= process_count_max: + log.warn("Maximum number of processes reached while executing jid {0}, waiting...".format(data['jid'])) + yield tornado.gen.sleep(10) + process_count = len(salt.utils.minion.running(self.opts)) + # We stash an instance references to allow for the socket # communication in Windows. You can't pickle functions, and thus # python needs to be able to reconstruct the reference on the other diff --git a/tests/unit/test_minion.py b/tests/unit/test_minion.py index 535dfeedfc..6c9dca13cd 100644 --- a/tests/unit/test_minion.py +++ b/tests/unit/test_minion.py @@ -18,6 +18,7 @@ from salt.utils import event from salt.exceptions import SaltSystemExit import salt.syspaths import tornado +from salt.ext.six.moves import range __opts__ = {} @@ -69,7 +70,7 @@ class MinionTestCase(TestCase): mock_jid_queue = [123] try: minion = salt.minion.Minion(mock_opts, jid_queue=copy.copy(mock_jid_queue), io_loop=tornado.ioloop.IOLoop()) - ret = minion._handle_decoded_payload(mock_data) + ret = minion._handle_decoded_payload(mock_data).result() self.assertEqual(minion.jid_queue, mock_jid_queue) self.assertIsNone(ret) finally: @@ -98,7 +99,7 @@ class MinionTestCase(TestCase): # Call the _handle_decoded_payload function and update the mock_jid_queue to include the new # mock_jid. The mock_jid should have been added to the jid_queue since the mock_jid wasn't # previously included. The minion's jid_queue attribute and the mock_jid_queue should be equal. - minion._handle_decoded_payload(mock_data) + minion._handle_decoded_payload(mock_data).result() mock_jid_queue.append(mock_jid) self.assertEqual(minion.jid_queue, mock_jid_queue) finally: @@ -126,8 +127,54 @@ class MinionTestCase(TestCase): # Call the _handle_decoded_payload function and check that the queue is smaller by one item # and contains the new jid - minion._handle_decoded_payload(mock_data) + minion._handle_decoded_payload(mock_data).result() self.assertEqual(len(minion.jid_queue), 2) self.assertEqual(minion.jid_queue, [456, 789]) finally: minion.destroy() + + def test_process_count_max(self): + ''' + Tests that the _handle_decoded_payload function does not spawn more than the configured amount of processes, + as per process_count_max. + ''' + with patch('salt.minion.Minion.ctx', MagicMock(return_value={})), \ + patch('salt.utils.process.SignalHandlingMultiprocessingProcess.start', MagicMock(return_value=True)), \ + patch('salt.utils.process.SignalHandlingMultiprocessingProcess.join', MagicMock(return_value=True)), \ + patch('salt.utils.minion.running', MagicMock(return_value=[])), \ + patch('tornado.gen.sleep', MagicMock(return_value=tornado.concurrent.Future())): + process_count_max = 10 + mock_opts = salt.config.DEFAULT_MINION_OPTS + mock_opts['minion_jid_queue_hwm'] = 100 + mock_opts["process_count_max"] = process_count_max + + try: + io_loop = tornado.ioloop.IOLoop() + minion = salt.minion.Minion(mock_opts, jid_queue=[], io_loop=io_loop) + + # mock gen.sleep to throw a special Exception when called, so that we detect it + class SleepCalledEception(Exception): + """Thrown when sleep is called""" + pass + tornado.gen.sleep.return_value.set_exception(SleepCalledEception()) + + # up until process_count_max: gen.sleep does not get called, processes are started normally + for i in range(process_count_max): + mock_data = {'fun': 'foo.bar', + 'jid': i} + io_loop.run_sync(lambda data=mock_data: minion._handle_decoded_payload(data)) + self.assertEqual(salt.utils.process.SignalHandlingMultiprocessingProcess.start.call_count, i + 1) + self.assertEqual(len(minion.jid_queue), i + 1) + salt.utils.minion.running.return_value += [i] + + # above process_count_max: gen.sleep does get called, JIDs are created but no new processes are started + mock_data = {'fun': 'foo.bar', + 'jid': process_count_max + 1} + + self.assertRaises(SleepCalledEception, + lambda: io_loop.run_sync(lambda: minion._handle_decoded_payload(mock_data))) + self.assertEqual(salt.utils.process.SignalHandlingMultiprocessingProcess.start.call_count, + process_count_max) + self.assertEqual(len(minion.jid_queue), process_count_max + 1) + finally: + minion.destroy() -- 2.14.2 ++++++ list_pkgs-add-parameter-for-returned-attribute-selec.patch ++++++ ++++ 792 lines (skipped) ++++++ multiprocessing-minion-option-documentation-fixes.patch ++++++ >From 6c5b8be3d14814903abc70b5605c87277dad39db Mon Sep 17 00:00:00 2001 From: Silvio Moioli <[email protected]> Date: Wed, 20 Sep 2017 14:32:47 +0200 Subject: [PATCH] multiprocessing minion option: documentation fixes --- doc/man/salt.7 | 1 + doc/ref/configuration/minion.rst | 7 +++++-- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/doc/man/salt.7 b/doc/man/salt.7 index d6cfe937a1..86c463b771 100644 --- a/doc/man/salt.7 +++ b/doc/man/salt.7 @@ -10795,6 +10795,7 @@ cmd_whitelist_glob: .UNINDENT .UNINDENT .SS Thread Settings +.SS \fBmultiprocessing\fP .sp Default: \fBTrue\fP .sp diff --git a/doc/ref/configuration/minion.rst b/doc/ref/configuration/minion.rst index cd554268c1..5cc72f1daf 100644 --- a/doc/ref/configuration/minion.rst +++ b/doc/ref/configuration/minion.rst @@ -2164,11 +2164,14 @@ Thread Settings .. conf_minion:: multiprocessing +``multiprocessing`` +------- + Default: ``True`` -If `multiprocessing` is enabled when a minion receives a +If ``multiprocessing`` is enabled when a minion receives a publication a new process is spawned and the command is executed therein. -Conversely, if `multiprocessing` is disabled the new publication will be run +Conversely, if ``multiprocessing`` is disabled the new publication will be run executed in a thread. -- 2.14.2 ++++++ use-home-to-get-the-user-home-directory-instead-usin.patch ++++++ >From 2dfd28560f3825dc512822ba9d01d67070d5175b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pablo=20Su=C3=A1rez=20Hern=C3=A1ndez?= <[email protected]> Date: Mon, 11 Sep 2017 19:57:28 +0200 Subject: [PATCH] Use $HOME to get the user home directory instead using '~' char --- pkg/salt.bash | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/pkg/salt.bash b/pkg/salt.bash index 480361fe23..00174c072f 100644 --- a/pkg/salt.bash +++ b/pkg/salt.bash @@ -35,7 +35,8 @@ _salt_get_keys(){ } _salt(){ - local _salt_cache_functions=${SALT_COMP_CACHE_FUNCTIONS:='~/.cache/salt-comp-cache_functions'} + CACHE_DIR="$HOME/.cache/salt-comp-cache_functions" + local _salt_cache_functions=${SALT_COMP_CACHE_FUNCTIONS:=$CACHE_DIR} local _salt_cache_timeout=${SALT_COMP_CACHE_TIMEOUT:='last hour'} if [ ! -d "$(dirname ${_salt_cache_functions})" ]; then -- 2.12.2 ++++++ v2017.7.1.tar.gz -> v2017.7.2.tar.gz ++++++ /work/SRC/openSUSE:Factory/salt/v2017.7.1.tar.gz /work/SRC/openSUSE:Factory/.salt.new/v2017.7.2.tar.gz differ: char 13, line 1
