Hello community, here is the log from the commit of package salt for openSUSE:Factory checked in at 2019-01-28 20:46:12 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/salt (Old) and /work/SRC/openSUSE:Factory/.salt.new.28833 (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "salt" Mon Jan 28 20:46:12 2019 rev:81 rq:666637 version:2018.3.2 Changes: -------- --- /work/SRC/openSUSE:Factory/salt/salt.changes 2018-09-26 14:53:17.955118391 +0200 +++ /work/SRC/openSUSE:Factory/.salt.new.28833/salt.changes 2019-01-28 20:46:16.210042484 +0100 @@ -1,0 +2,250 @@ +Wed Jan 16 16:28:09 UTC 2019 - psuarezhernan...@suse.com + +- Do not restrict the Python version to < 3.7 + +------------------------------------------------------------------- +Tue Jan 15 09:47:12 UTC 2019 - b...@suse.de + +- Fix integration tests in state compiler (U#2068) + +- Added: + * fix-issue-2068-test.patch + +------------------------------------------------------------------- +Fri Jan 11 13:23:13 UTC 2019 - psuarezhernan...@suse.com + +- Fix "pkg.list_pkgs" output when using "attr" to take the arch into account (bsc#1114029) + +- Added: + * remove-arch-from-name-when-pkg.list_pkgs-is-called-w.patch + +------------------------------------------------------------------- +Thu Jan 10 12:52:09 UTC 2019 - mdinca <mdi...@suse.de> + +- Fix powerpc null server_id_arch (bsc#1117995) + +- Added: + * return-the-expected-powerpc-os-arch-bsc-1117995.patch + +------------------------------------------------------------------- +Thu Jan 10 09:53:33 UTC 2019 - b...@suse.de + +- Fix module 'azure.storage' has no attribute '__version__' + (bsc#1121091) + +- Added: + * azurefs-gracefully-handle-attributeerror.patch + +------------------------------------------------------------------- +Fri Jan 4 13:29:50 UTC 2019 - b...@suse.de + +- Add supportconfig module and states for minions and SaltSSH + +- Added: + * add-supportconfig-module-for-remote-calls-and-saltss.patch + +------------------------------------------------------------------- +Thu Jan 3 16:35:30 UTC 2019 - b...@suse.de + +- Fix FIPS enabled RES clients (bsc#1099887) + +- Added: + * retire-md5-checksum-for-pkg-mgmt-plugins.patch + +------------------------------------------------------------------- +Thu Jan 3 15:48:20 UTC 2019 - b...@suse.de + +- Add hold/unhold functions. Fix Debian repo "signed-by". + +- Added: + * decide-if-the-source-should-be-actually-skipped.patch + * add-hold-unhold-functions.patch + +------------------------------------------------------------------- +Tue Dec 4 16:28:21 UTC 2018 - psuarezhernan...@suse.com + +- Fix latin1 encoding problems on file module (bsc#1116837) + +- Added: + * fix-latin1-encoding-problems-on-file-module-bsc-1116.patch + +------------------------------------------------------------------- +Fri Nov 30 13:14:19 UTC 2018 - b...@suse.de + +- Don't error on retcode 0 in libcrypto.OPENSSL_init_crypto + +- Added: + * don-t-error-on-retcode-0-in-libcrypto.openssl_init_c.patch + +------------------------------------------------------------------- +Tue Nov 20 15:33:39 UTC 2018 - b...@suse.de + +- Debian info_installed compatibility (U#50453) + +- Added: + * debian-info_installed-compatibility-50453.patch + +------------------------------------------------------------------- +Fri Nov 16 14:17:45 UTC 2018 - b...@suse.de + +- Add compatibility with other package modules for "list_repos" function +- Bugfix: unable to detect os arch when RPM is not installed (bsc#1114197) + +- Added: + * make-aptpkg.list_repos-compatible-on-enabled-disable.patch + * get-os_arch-also-without-rpm-package-installed.patch + +------------------------------------------------------------------- +Thu Nov 8 09:32:49 UTC 2018 - psuarezhernan...@suse.com + +- Fix git_pillar merging across multiple __env__ repositories (bsc#1112874) + +- Added: + * fix-git_pillar-merging-across-multiple-__env__-repos.patch + +------------------------------------------------------------------- +Wed Oct 31 14:52:31 UTC 2018 - b...@suse.de + +- Fix LDAP authentication issue when a valid token is generated + by the salt-api even when invalid user credentials are passed. + (U#48901) + +- Added: + * fixing-issue-when-a-valid-token-is-generated-even-wh.patch + +------------------------------------------------------------------- +Tue Oct 30 10:48:23 UTC 2018 - Jochen Breuer <jbre...@suse.de> + +- Improved handling of LDAP group id. gid is no longer treated as a + string, which could have lead to faulty group creations. (bsc#1113784) + +- Added: + * improved-handling-of-ldap-group-id.patch + +------------------------------------------------------------------- +Thu Oct 25 13:04:42 UTC 2018 - psuarezhernan...@suse.com + +- Fix remote command execution and incorrect access control + when using salt-api. (bsc#1113699) (CVE-2018-15751) +- Fix Directory traversal vulnerability when using salt-api. + Allows an attacker to determine what files exist on + a server when querying /run or /events. (bsc#1113698) (CVE-2018-15750) + +- Added: + * fixes-cve-2018-15750-cve-2018-15751.patch + +------------------------------------------------------------------- +Thu Oct 18 13:17:33 UTC 2018 - b...@suse.de + +- Add multi-file support and globbing to the filetree (U#50018) + +- Added: + * add-multi-file-support-and-globbing-to-the-filetree-.patch + +------------------------------------------------------------------- +Wed Oct 17 15:21:17 UTC 2018 - b...@suse.de + +- Bugfix: supportconfig non-root permission issues (U#50095) + +- Added: + * support-config-non-root-permission-issues-fixes-u-50.patch + +------------------------------------------------------------------- +Wed Oct 17 14:18:09 UTC 2018 - b...@suse.de + +- Open profiles permissions to everyone for read-only + +------------------------------------------------------------------- +Tue Oct 16 15:26:16 UTC 2018 - b...@suse.de + +- Preserving signature in "module.run" state (U#50049) + +- Added: + * preserving-signature-in-module.run-state-u-50049.patch + +------------------------------------------------------------------- +Fri Oct 12 11:48:40 UTC 2018 - b...@suse.de + +- Install default salt-support profiles + +------------------------------------------------------------------- +Thu Oct 11 15:04:30 UTC 2018 - b...@suse.de + +- Fix unit tests due to merger failure +- Add CPE_NAME for osversion* grain parsing +- Get os_family for RPM distros from the RPM macros +- Install support profiles + +- Added: + * get-os_family-for-rpm-distros-from-the-rpm-macros.-u.patch + * add-cpe_name-for-osversion-grain-parsing-u-49946.patch + * make-profiles-a-package.patch + * fix-unit-test-for-grains-core.patch + +------------------------------------------------------------------- +Tue Oct 9 14:50:25 UTC 2018 - psuarezhernan...@suse.com + +- Bugfix: any unicode string of length 16 will raise TypeError + +- Added: + * bugfix-any-unicode-string-of-length-16-will-raise-ty.patch + +------------------------------------------------------------------- +Mon Oct 8 08:52:23 UTC 2018 - psuarezhernan...@suse.com + +- Fix async call to process manager (bsc#1110938) +- Early feature: Salt support-config (salt-support) + ++++ 53 more lines (skipped) ++++ between /work/SRC/openSUSE:Factory/salt/salt.changes ++++ and /work/SRC/openSUSE:Factory/.salt.new.28833/salt.changes New: ---- add-cpe_name-for-osversion-grain-parsing-u-49946.patch add-hold-unhold-functions.patch add-multi-file-support-and-globbing-to-the-filetree-.patch add-supportconfig-module-for-remote-calls-and-saltss.patch azurefs-gracefully-handle-attributeerror.patch bugfix-any-unicode-string-of-length-16-will-raise-ty.patch debian-info_installed-compatibility-50453.patch decide-if-the-source-should-be-actually-skipped.patch do-not-load-pip-state-if-there-is-no-3rd-party-depen.patch don-t-error-on-retcode-0-in-libcrypto.openssl_init_c.patch early-feature-support-config.patch fix-async-call-to-process-manager.patch fix-git_pillar-merging-across-multiple-__env__-repos.patch fix-index-error-when-running-on-python-3.patch fix-ipv6-scope-bsc-1108557.patch fix-issue-2068-test.patch fix-latin1-encoding-problems-on-file-module-bsc-1116.patch fix-unit-test-for-grains-core.patch fixes-cve-2018-15750-cve-2018-15751.patch fixing-issue-when-a-valid-token-is-generated-even-wh.patch get-os_arch-also-without-rpm-package-installed.patch get-os_family-for-rpm-distros-from-the-rpm-macros.-u.patch improved-handling-of-ldap-group-id.patch loosen-azure-sdk-dependencies-in-azurearm-cloud-driv.patch make-aptpkg.list_repos-compatible-on-enabled-disable.patch make-profiles-a-package.patch preserving-signature-in-module.run-state-u-50049.patch remove-arch-from-name-when-pkg.list_pkgs-is-called-w.patch retire-md5-checksum-for-pkg-mgmt-plugins.patch return-the-expected-powerpc-os-arch-bsc-1117995.patch support-config-non-root-permission-issues-fixes-u-50.patch support-use-of-gce-instance-credentials-109.patch update-error-list-for-zypper.patch ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ salt.spec ++++++ --- /var/tmp/diff_new_pack.eKktgk/_old 2019-01-28 20:46:17.642040978 +0100 +++ /var/tmp/diff_new_pack.eKktgk/_new 2019-01-28 20:46:17.646040973 +0100 @@ -15,7 +15,6 @@ # Please submit bugfixes or comments via http://bugs.opensuse.org/ # - %if 0%{?suse_version} >= 1320 # SLE15 %global build_py3 1 @@ -150,6 +149,76 @@ Patch42: use-adler32-algorithm-to-compute-string-checksums.patch # PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/49497 Patch43: x509-fixes-111.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/49605 +Patch44: support-use-of-gce-instance-credentials-109.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/49722 +Patch45: fix-index-error-when-running-on-python-3.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/49696 +Patch46: loosen-azure-sdk-dependencies-in-azurearm-cloud-driv.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/49737 +Patch47: do-not-load-pip-state-if-there-is-no-3rd-party-depen.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/49806 +Patch48: update-error-list-for-zypper.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/49815 +Patch49: fix-ipv6-scope-bsc-1108557.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/49480 +Patch50: early-feature-support-config.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/49085 +Patch51: fix-async-call-to-process-manager.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/49908 +Patch52: bugfix-any-unicode-string-of-length-16-will-raise-ty.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/49936 +Patch53: make-profiles-a-package.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/49930 +Patch54: get-os_family-for-rpm-distros-from-the-rpm-macros.-u.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/49946 +Patch55: add-cpe_name-for-osversion-grain-parsing-u-49946.patch +# PATCH-FIX_OPENSUSE: Fix unit test for grains core +Patch56: fix-unit-test-for-grains-core.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/50049 +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/50072 +Patch57: preserving-signature-in-module.run-state-u-50049.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/50095 +Patch58: support-config-non-root-permission-issues-fixes-u-50.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/50018 +Patch59: add-multi-file-support-and-globbing-to-the-filetree-.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/49761 +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/50201 +Patch60: fixes-cve-2018-15750-cve-2018-15751.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/48491 +Patch61: improved-handling-of-ldap-group-id.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/48901 +Patch62: fixing-issue-when-a-valid-token-is-generated-even-wh.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/50417 +Patch63: fix-git_pillar-merging-across-multiple-__env__-repos.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/50523 +Patch64: get-os_arch-also-without-rpm-package-installed.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/50392 +Patch65: make-aptpkg.list_repos-compatible-on-enabled-disable.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/50453 +Patch66: debian-info_installed-compatibility-50453.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/48580 +Patch67: don-t-error-on-retcode-0-in-libcrypto.openssl_init_c.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/48503 +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/48934 +Patch68: fix-latin1-encoding-problems-on-file-module-bsc-1116.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/50742 +Patch69: decide-if-the-source-should-be-actually-skipped.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/50773 +Patch70: add-hold-unhold-functions.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/49639 +Patch71: retire-md5-checksum-for-pkg-mgmt-plugins.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/50401 +# NOTE: This is a techpreview as well as in Fluorine! Release only in Neon. +Patch72: add-supportconfig-module-for-remote-calls-and-saltss.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/50567 +Patch73: azurefs-gracefully-handle-attributeerror.patch +# PATCH_FIX_OPENSUSE: https://github.com/openSUSE/salt/pull/116 +Patch74: return-the-expected-powerpc-os-arch-bsc-1117995.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/51108 +Patch75: remove-arch-from-name-when-pkg.list_pkgs-is-called-w.patch +# PATCH-FIX_UPSTREAM https://github.com/saltstack/salt/pull/51119 +Patch76: fix-issue-2068-test.patch # BuildRoot: %{_tmppath}/%{name}-%{version}-build BuildRoot: %{_tmppath}/%{name}-%{version}-build @@ -235,12 +304,12 @@ # requirements/base.txt %if 0%{?rhel} BuildRequires: python-jinja2 -BuildRequires: python-markupsafe BuildRequires: python-yaml +BuildRequires: python-markupsafe %else BuildRequires: python-Jinja2 -BuildRequires: python-MarkupSafe BuildRequires: python-PyYAML +BuildRequires: python-MarkupSafe %endif BuildRequires: python-futures >= 2.0 @@ -273,16 +342,16 @@ # requirements/base.txt %if 0%{?rhel} Requires: python-jinja2 -Requires: python-markupsafe Requires: python-yaml +Requires: python-markupsafe Requires: yum %if 0%{?rhel} == 6 Requires: yum-plugin-security %endif %else Requires: python-Jinja2 -Requires: python-MarkupSafe Requires: python-PyYAML +Requires: python-MarkupSafe %endif Requires: python-futures >= 2.0 @@ -322,7 +391,7 @@ Group: System/Management Requires: %{name} = %{version}-%{release} BuildRequires: python-rpm-macros -BuildRequires: python3 < 3.7 +BuildRequires: python3 BuildRequires: python3-devel # requirements/base.txt %if 0%{?rhel} @@ -353,7 +422,7 @@ %if %{with builddocs} BuildRequires: python3-sphinx %endif -Requires: python3 < 3.7 +Requires: python3 # %if ! 0%{?suse_version} > 1110 Requires: python3-certifi @@ -651,6 +720,39 @@ %patch41 -p1 %patch42 -p1 %patch43 -p1 +%patch44 -p1 +%patch45 -p1 +%patch46 -p1 +%patch47 -p1 +%patch48 -p1 +%patch49 -p1 +%patch50 -p1 +%patch51 -p1 +%patch52 -p1 +%patch53 -p1 +%patch54 -p1 +%patch55 -p1 +%patch56 -p1 +%patch57 -p1 +%patch58 -p1 +%patch59 -p1 +%patch60 -p1 +%patch61 -p1 +%patch62 -p1 +%patch63 -p1 +%patch64 -p1 +%patch65 -p1 +%patch66 -p1 +%patch67 -p1 +%patch68 -p1 +%patch69 -p1 +%patch70 -p1 +%patch71 -p1 +%patch72 -p1 +%patch73 -p1 +%patch74 -p1 +%patch75 -p1 +%patch76 -p1 %build %if 0%{?build_py2} @@ -739,6 +841,15 @@ install -Dd -m 0750 %{buildroot}%{_sysconfdir}/salt/pki/master/minions_rejected install -Dd -m 0750 %{buildroot}%{_sysconfdir}/salt/pki/minion +# Install salt-support profiles +%if 0%{?build_py2} +install -Dpm 0644 salt/cli/support/profiles/* %{buildroot}%{python_sitelib}/salt/cli/support/profiles +%endif +%if 0%{?build_py3} +install -Dpm 0644 salt/cli/support/profiles/* %{buildroot}%{python3_sitelib}/salt/cli/support/profiles +%endif + + ## Install Zypper plugins only on SUSE machines %if 0%{?suse_version} install -Dd -m 0750 %{buildroot}%{_prefix}/lib/zypp/plugins/commit @@ -1287,6 +1398,7 @@ %defattr(-,root,root,-) %{_bindir}/spm %{_bindir}/salt-call +%{_bindir}/salt-support %{_bindir}/salt-unity %{_mandir}/man1/salt-unity.1.gz %{_mandir}/man1/salt-call.1.gz ++++++ _lastrevision ++++++ --- /var/tmp/diff_new_pack.eKktgk/_old 2019-01-28 20:46:17.718040897 +0100 +++ /var/tmp/diff_new_pack.eKktgk/_new 2019-01-28 20:46:17.718040897 +0100 @@ -1 +1 @@ -ebc77d067d9fa300bdc5bb5dcccaa09e1787f688 \ No newline at end of file +2460cb78e6bda580f2567781e060a3e6c6ba25de \ No newline at end of file ++++++ add-cpe_name-for-osversion-grain-parsing-u-49946.patch ++++++ >From 3bad9e211c2e76ddac48f7c8ff1632e32e0a256e Mon Sep 17 00:00:00 2001 From: Bo Maryniuk <b...@suse.de> Date: Tue, 9 Oct 2018 14:08:50 +0200 Subject: [PATCH] Add CPE_NAME for osversion* grain parsing (U#49946) Remove unnecessary linebreak Override VERSION_ID from os-release, if CPE_NAME is given Add unit test for WFN format of CPE_NAME Add unit test for v2.3 of CPE format Add unit test for broken CPE_NAME Prevent possible crash if CPE_NAME is wrongly written in the distro Add part parsing Keep CPE_NAME only for opensuse series Remove linebreak Expand unit test to verify part name Fix proper part name in the string-bound CPE --- salt/grains/core.py | 43 +++++++++++++++++++++--- tests/unit/grains/test_core.py | 60 +++++++++++++++++++++++++++++----- 2 files changed, 90 insertions(+), 13 deletions(-) diff --git a/salt/grains/core.py b/salt/grains/core.py index 80eebd1c05..e41ab4e0ae 100644 --- a/salt/grains/core.py +++ b/salt/grains/core.py @@ -1355,6 +1355,34 @@ def _parse_os_release(os_release_files): return data +def _parse_cpe_name(cpe): + ''' + Parse CPE_NAME data from the os-release + + Info: https://csrc.nist.gov/projects/security-content-automation-protocol/scap-specifications/cpe + + :param cpe: + :return: + ''' + part = { + 'o': 'operating system', + 'h': 'hardware', + 'a': 'application', + } + ret = {} + cpe = (cpe or '').split(':') + if len(cpe) > 4 and cpe[0] == 'cpe': + if cpe[1].startswith('/'): # WFN to URI + ret['vendor'], ret['product'], ret['version'] = cpe[2:5] + ret['phase'] = cpe[5] if len(cpe) > 5 else None + ret['part'] = part.get(cpe[1][1:]) + elif len(cpe) == 13 and cpe[1] == '2.3': # WFN to a string + ret['vendor'], ret['product'], ret['version'], ret['phase'] = [x if x != '*' else None for x in cpe[3:7]] + ret['part'] = part.get(cpe[2]) + + return ret + + def os_data(): ''' Return grains pertaining to the operating system @@ -1554,13 +1582,20 @@ def os_data(): codename = codename_match.group(1) grains['lsb_distrib_codename'] = codename if 'CPE_NAME' in os_release: - if ":suse:" in os_release['CPE_NAME'] or ":opensuse:" in os_release['CPE_NAME']: + cpe = _parse_cpe_name(os_release['CPE_NAME']) + if not cpe: + log.error('Broken CPE_NAME format in /etc/os-release!') + elif cpe.get('vendor', '').lower() in ['suse', 'opensuse']: grains['os'] = "SUSE" # openSUSE `osfullname` grain normalization if os_release.get("NAME") == "openSUSE Leap": grains['osfullname'] = "Leap" elif os_release.get("VERSION") == "Tumbleweed": grains['osfullname'] = os_release["VERSION"] + # Override VERSION_ID, if CPE_NAME around + if cpe.get('version') and cpe.get('vendor') == 'opensuse': # Keep VERSION_ID for SLES + grains['lsb_distrib_release'] = cpe['version'] + elif os.path.isfile('/etc/SuSE-release'): grains['lsb_distrib_id'] = 'SUSE' version = '' @@ -1666,8 +1701,7 @@ def os_data(): # Commit introducing this comment should be reverted after the upstream bug is released. if 'CentOS Linux 7' in grains.get('lsb_distrib_codename', ''): grains.pop('lsb_distrib_release', None) - grains['osrelease'] = \ - grains.get('lsb_distrib_release', osrelease).strip() + grains['osrelease'] = grains.get('lsb_distrib_release', osrelease).strip() grains['oscodename'] = grains.get('lsb_distrib_codename', '').strip() or oscodename if 'Red Hat' in grains['oscodename']: grains['oscodename'] = oscodename @@ -1702,8 +1736,7 @@ def os_data(): r'((?:Open|Oracle )?Solaris|OpenIndiana|OmniOS) (Development)?' r'\s*(\d+\.?\d*|v\d+)\s?[A-Z]*\s?(r\d+|\d+\/\d+|oi_\S+|snv_\S+)?' ) - osname, development, osmajorrelease, osminorrelease = \ - release_re.search(rel_data).groups() + osname, development, osmajorrelease, osminorrelease = release_re.search(rel_data).groups() except AttributeError: # Set a blank osrelease grain and fallback to 'Solaris' # as the 'os' grain. diff --git a/tests/unit/grains/test_core.py b/tests/unit/grains/test_core.py index e973428add..2ab32ef41b 100644 --- a/tests/unit/grains/test_core.py +++ b/tests/unit/grains/test_core.py @@ -62,10 +62,11 @@ class CoreGrainsTestCase(TestCase, LoaderModuleMockMixin): def test_parse_etc_os_release(self, path_isfile_mock): path_isfile_mock.side_effect = lambda x: x == "/usr/lib/os-release" with salt.utils.files.fopen(os.path.join(OS_RELEASE_DIR, "ubuntu-17.10")) as os_release_file: - os_release_content = os_release_file.readlines() - with patch("salt.utils.files.fopen", mock_open()) as os_release_file: - os_release_file.return_value.__iter__.return_value = os_release_content - os_release = core._parse_os_release(["/etc/os-release", "/usr/lib/os-release"]) + os_release_content = os_release_file.read() + with patch("salt.utils.files.fopen", mock_open(read_data=os_release_content)): + os_release = core._parse_os_release( + '/etc/os-release', + '/usr/lib/os-release') self.assertEqual(os_release, { "NAME": "Ubuntu", "VERSION": "17.10 (Artful Aardvark)", @@ -81,10 +82,53 @@ class CoreGrainsTestCase(TestCase, LoaderModuleMockMixin): "UBUNTU_CODENAME": "artful", }) - @patch("os.path.isfile") - def test_missing_os_release(self, path_isfile_mock): - path_isfile_mock.return_value = False - os_release = core._parse_os_release(["/etc/os-release", "/usr/lib/os-release"]) + def test_parse_cpe_name_wfn(self): + ''' + Parse correct CPE_NAME data WFN formatted + :return: + ''' + for cpe, cpe_ret in [('cpe:/o:opensuse:leap:15.0', + {'phase': None, 'version': '15.0', 'product': 'leap', + 'vendor': 'opensuse', 'part': 'operating system'}), + ('cpe:/o:vendor:product:42:beta', + {'phase': 'beta', 'version': '42', 'product': 'product', + 'vendor': 'vendor', 'part': 'operating system'})]: + ret = core._parse_cpe_name(cpe) + for key in cpe_ret: + assert key in ret + assert cpe_ret[key] == ret[key] + + def test_parse_cpe_name_v23(self): + ''' + Parse correct CPE_NAME data v2.3 formatted + :return: + ''' + for cpe, cpe_ret in [('cpe:2.3:o:microsoft:windows_xp:5.1.601:beta:*:*:*:*:*:*', + {'phase': 'beta', 'version': '5.1.601', 'product': 'windows_xp', + 'vendor': 'microsoft', 'part': 'operating system'}), + ('cpe:2.3:h:corellian:millenium_falcon:1.0:*:*:*:*:*:*:*', + {'phase': None, 'version': '1.0', 'product': 'millenium_falcon', + 'vendor': 'corellian', 'part': 'hardware'}), + ('cpe:2.3:*:dark_empire:light_saber:3.0:beta:*:*:*:*:*:*', + {'phase': 'beta', 'version': '3.0', 'product': 'light_saber', + 'vendor': 'dark_empire', 'part': None})]: + ret = core._parse_cpe_name(cpe) + for key in cpe_ret: + assert key in ret + assert cpe_ret[key] == ret[key] + + def test_parse_cpe_name_broken(self): + ''' + Parse broken CPE_NAME data + :return: + ''' + for cpe in ['cpe:broken', 'cpe:broken:in:all:ways:*:*:*:*', + 'cpe:x:still:broken:123', 'who:/knows:what:is:here']: + assert core._parse_cpe_name(cpe) == {} + + def test_missing_os_release(self): + with patch('salt.utils.files.fopen', mock_open(read_data={})): + os_release = core._parse_os_release('/etc/os-release', '/usr/lib/os-release') self.assertEqual(os_release, {}) @skipIf(not salt.utils.platform.is_linux(), 'System is not Linux') -- 2.19.0 ++++++ add-hold-unhold-functions.patch ++++++ >From ba5171ce35b733a1f7997b4ea038998802b67298 Mon Sep 17 00:00:00 2001 From: Bo Maryniuk <b...@suse.de> Date: Thu, 6 Dec 2018 16:26:23 +0100 Subject: [PATCH] Add hold/unhold functions Add unhold function Add warnings --- salt/modules/zypper.py | 88 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 87 insertions(+), 1 deletion(-) diff --git a/salt/modules/zypper.py b/salt/modules/zypper.py index 6845e44ab6..773354b2f3 100644 --- a/salt/modules/zypper.py +++ b/salt/modules/zypper.py @@ -41,6 +41,7 @@ import salt.utils.pkg import salt.utils.pkg.rpm import salt.utils.stringutils import salt.utils.systemd +import salt.utils.versions from salt.utils.versions import LooseVersion from salt.exceptions import CommandExecutionError, MinionError, SaltInvocationError @@ -1738,7 +1739,7 @@ def clean_locks(): return out -def remove_lock(packages, **kwargs): # pylint: disable=unused-argument +def unhold(name=None, pkgs=None, **kwargs): ''' Remove specified package lock. @@ -1750,7 +1751,47 @@ def remove_lock(packages, **kwargs): # pylint: disable=unused-argument salt '*' pkg.remove_lock <package1>,<package2>,<package3> salt '*' pkg.remove_lock pkgs='["foo", "bar"]' ''' + ret = {} + if (not name and not pkgs) or (name and pkgs): + raise CommandExecutionError('Name or packages must be specified.') + elif name: + pkgs = [name] + + locks = list_locks() + try: + pkgs = list(__salt__['pkg_resource.parse_targets'](pkgs)[0].keys()) + except MinionError as exc: + raise CommandExecutionError(exc) + + removed = [] + missing = [] + for pkg in pkgs: + if locks.get(pkg): + removed.append(pkg) + ret[pkg]['comment'] = 'Package {0} is no longer held.'.format(pkg) + else: + missing.append(pkg) + ret[pkg]['comment'] = 'Package {0} unable to be unheld.'.format(pkg) + + if removed: + __zypper__.call('rl', *removed) + + return ret + + +def remove_lock(packages, **kwargs): # pylint: disable=unused-argument + ''' + Remove specified package lock. + + CLI Example: + + .. code-block:: bash + salt '*' pkg.remove_lock <package name> + salt '*' pkg.remove_lock <package1>,<package2>,<package3> + salt '*' pkg.remove_lock pkgs='["foo", "bar"]' + ''' + salt.utils.versions.warn_until('Sodium', 'This function is deprecated. Please use unhold() instead.') locks = list_locks() try: packages = list(__salt__['pkg_resource.parse_targets'](packages)[0].keys()) @@ -1771,6 +1812,50 @@ def remove_lock(packages, **kwargs): # pylint: disable=unused-argument return {'removed': len(removed), 'not_found': missing} +def hold(name=None, pkgs=None, **kwargs): + ''' + Add a package lock. Specify packages to lock by exact name. + + CLI Example: + + .. code-block:: bash + + salt '*' pkg.add_lock <package name> + salt '*' pkg.add_lock <package1>,<package2>,<package3> + salt '*' pkg.add_lock pkgs='["foo", "bar"]' + + :param name: + :param pkgs: + :param kwargs: + :return: + ''' + ret = {} + if (not name and not pkgs) or (name and pkgs): + raise CommandExecutionError('Name or packages must be specified.') + elif name: + pkgs = [name] + + locks = list_locks() + added = [] + try: + pkgs = list(__salt__['pkg_resource.parse_targets'](pkgs)[0].keys()) + except MinionError as exc: + raise CommandExecutionError(exc) + + for pkg in pkgs: + ret[pkg] = {'name': pkg, 'changes': {}, 'result': False, 'comment': ''} + if not locks.get(pkg): + added.append(pkg) + ret[pkg]['comment'] = 'Package {0} is now being held.'.format(pkg) + else: + ret[pkg]['comment'] = 'Package {0} is already set to be held.'.format(pkg) + + if added: + __zypper__.call('al', *added) + + return ret + + def add_lock(packages, **kwargs): # pylint: disable=unused-argument ''' Add a package lock. Specify packages to lock by exact name. @@ -1783,6 +1868,7 @@ def add_lock(packages, **kwargs): # pylint: disable=unused-argument salt '*' pkg.add_lock <package1>,<package2>,<package3> salt '*' pkg.add_lock pkgs='["foo", "bar"]' ''' + salt.utils.versions.warn_until('Sodium', 'This function is deprecated. Please use hold() instead.') locks = list_locks() added = [] try: -- 2.20.1 ++++++ add-multi-file-support-and-globbing-to-the-filetree-.patch ++++++ >From 671bb9d48e120c806ca1f6f176b0ada43b1e7594 Mon Sep 17 00:00:00 2001 From: Bo Maryniuk <b...@suse.de> Date: Fri, 12 Oct 2018 16:20:40 +0200 Subject: [PATCH] Add multi-file support and globbing to the filetree (U#50018) Add more possible logs Support multiple files grabbing Collect system logs and boot logs Support globbing in filetree --- salt/cli/support/intfunc.py | 49 ++++++++++++++++----------- salt/cli/support/profiles/default.yml | 7 ++++ 2 files changed, 37 insertions(+), 19 deletions(-) diff --git a/salt/cli/support/intfunc.py b/salt/cli/support/intfunc.py index 2727cd6394..f15f4d4097 100644 --- a/salt/cli/support/intfunc.py +++ b/salt/cli/support/intfunc.py @@ -6,6 +6,7 @@ Internal functions. from __future__ import absolute_import, print_function, unicode_literals import os +import glob from salt.cli.support.console import MessagesOutput import salt.utils.files @@ -13,7 +14,7 @@ import salt.utils.files out = MessagesOutput() -def filetree(collector, path): +def filetree(collector, *paths): ''' Add all files in the tree. If the "path" is a file, only that file will be added. @@ -21,22 +22,32 @@ def filetree(collector, path): :param path: File or directory :return: ''' - if not path: - out.error('Path not defined', ident=2) - else: - # The filehandler needs to be explicitly passed here, so PyLint needs to accept that. - # pylint: disable=W8470 - if os.path.isfile(path): - filename = os.path.basename(path) - try: - file_ref = salt.utils.files.fopen(path) # pylint: disable=W - out.put('Add {}'.format(filename), indent=2) - collector.add(filename) - collector.link(title=path, path=file_ref) - except Exception as err: - out.error(err, ident=4) - # pylint: enable=W8470 + _paths = [] + # Unglob + for path in paths: + _paths += glob.glob(path) + for path in set(_paths): + if not path: + out.error('Path not defined', ident=2) + elif not os.path.exists(path): + out.warning('Path {} does not exists'.format(path)) else: - for fname in os.listdir(path): - fname = os.path.join(path, fname) - filetree(collector, fname) + # The filehandler needs to be explicitly passed here, so PyLint needs to accept that. + # pylint: disable=W8470 + if os.path.isfile(path): + filename = os.path.basename(path) + try: + file_ref = salt.utils.files.fopen(path) # pylint: disable=W + out.put('Add {}'.format(filename), indent=2) + collector.add(filename) + collector.link(title=path, path=file_ref) + except Exception as err: + out.error(err, ident=4) + # pylint: enable=W8470 + else: + try: + for fname in os.listdir(path): + fname = os.path.join(path, fname) + filetree(collector, [fname]) + except Exception as err: + out.error(err, ident=4) diff --git a/salt/cli/support/profiles/default.yml b/salt/cli/support/profiles/default.yml index 01d9a26193..3defb5eef3 100644 --- a/salt/cli/support/profiles/default.yml +++ b/salt/cli/support/profiles/default.yml @@ -62,10 +62,17 @@ general-health: - ps.top: info: Top CPU consuming processes +boot_log: + - filetree: + info: Collect boot logs + args: + - /var/log/boot.* + system.log: # This works on any file system object. - filetree: info: Add system log args: - /var/log/syslog + - /var/log/messages -- 2.19.0 ++++++ add-supportconfig-module-for-remote-calls-and-saltss.patch ++++++ ++++ 1405 lines (skipped) ++++++ azurefs-gracefully-handle-attributeerror.patch ++++++ >From 326e649ef1f14b609916f0e9ce75e29a5e7f4d05 Mon Sep 17 00:00:00 2001 From: Robert Munteanu <romb...@apache.org> Date: Mon, 19 Nov 2018 17:52:34 +0100 Subject: [PATCH] azurefs: gracefully handle AttributeError It is possible that the azure.storage object has no __version__ defined. In that case, prevent console spam with unhandled AttributeError messages and instead consider that Azure support is not present. Problem was encountered on openSUSE Tumbleweed. --- salt/fileserver/azurefs.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/salt/fileserver/azurefs.py b/salt/fileserver/azurefs.py index c266323fbe..a42c10c594 100644 --- a/salt/fileserver/azurefs.py +++ b/salt/fileserver/azurefs.py @@ -68,7 +68,7 @@ try: if LooseVersion(azure.storage.__version__) < LooseVersion('0.20.0'): raise ImportError('azure.storage.__version__ must be >= 0.20.0') HAS_AZURE = True -except ImportError: +except (ImportError, AttributeError): HAS_AZURE = False # Import third party libs -- 2.20.1 ++++++ bugfix-any-unicode-string-of-length-16-will-raise-ty.patch ++++++ >From e82dc4c556497b612d31b65e60b34c979c957424 Mon Sep 17 00:00:00 2001 From: Bo Maryniuk <b...@suse.de> Date: Fri, 5 Oct 2018 12:02:08 +0200 Subject: [PATCH] Bugfix: any unicode string of length 16 will raise TypeError instead of ValueError --- salt/_compat.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/salt/_compat.py b/salt/_compat.py index 0576210afc..71963a4ead 100644 --- a/salt/_compat.py +++ b/salt/_compat.py @@ -192,7 +192,7 @@ class IPv6AddressScoped(ipaddress.IPv6Address): if len(data) == 16 and ':' not in data: try: packed = bool(int(str(bytearray(data)).encode('hex'), 16)) - except ValueError: + except (ValueError, TypeError): pass return packed -- 2.17.1 ++++++ debian-info_installed-compatibility-50453.patch ++++++ ++++ 642 lines (skipped) ++++++ decide-if-the-source-should-be-actually-skipped.patch ++++++ >From 5eacdf8fef35cdd05cae1b65485b3f820c86bc68 Mon Sep 17 00:00:00 2001 From: Bo Maryniuk <b...@suse.de> Date: Tue, 4 Dec 2018 16:39:08 +0100 Subject: [PATCH] Decide if the source should be actually skipped --- salt/modules/aptpkg.py | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/salt/modules/aptpkg.py b/salt/modules/aptpkg.py index dc27903230..42d606926f 100644 --- a/salt/modules/aptpkg.py +++ b/salt/modules/aptpkg.py @@ -1698,6 +1698,27 @@ def list_repo_pkgs(*args, **kwargs): # pylint: disable=unused-import return ret +def _skip_source(source): + ''' + Decide to skip source or not. + + :param source: + :return: + ''' + if source.invalid: + if source.uri and source.type and source.type in ("deb", "deb-src", "rpm", "rpm-src"): + pieces = source.mysplit(source.line) + if pieces[1].strip()[0] == "[": + options = pieces.pop(1).strip("[]").split() + if len(options) > 0: + log.debug("Source %s will be included although is marked invalid", source.uri) + return False + return True + else: + return True + return False + + def list_repos(): ''' Lists all repos in the sources.list (and sources.lists.d) files @@ -1713,7 +1734,7 @@ def list_repos(): repos = {} sources = sourceslist.SourcesList() for source in sources.list: - if source.invalid: + if _skip_source(source): continue repo = {} repo['file'] = source.file -- 2.20.1 ++++++ do-not-load-pip-state-if-there-is-no-3rd-party-depen.patch ++++++ >From 7727ab13e3492b722b316469cc912d9dd64f063e Mon Sep 17 00:00:00 2001 From: Bo Maryniuk <b...@suse.de> Date: Fri, 21 Sep 2018 17:31:39 +0200 Subject: [PATCH] Do not load pip state if there is no 3rd party dependencies Safe import 3rd party dependency --- salt/modules/pip.py | 12 ++++++++++-- salt/states/pip_state.py | 9 +++++---- 2 files changed, 15 insertions(+), 6 deletions(-) diff --git a/salt/modules/pip.py b/salt/modules/pip.py index f1a2e42433..85844f098b 100644 --- a/salt/modules/pip.py +++ b/salt/modules/pip.py @@ -79,7 +79,10 @@ from __future__ import absolute_import, print_function, unicode_literals # Import python libs import logging import os -import pkg_resources +try: + import pkg_resources +except ImportError: + pkg_resources = None import re import shutil import sys @@ -116,7 +119,12 @@ def __virtual__(): entire filesystem. If it's not installed in a conventional location, the user is required to provide the location of pip each time it is used. ''' - return 'pip' + if pkg_resources is None: + ret = False, 'Package dependency "pkg_resource" is missing' + else: + ret = 'pip' + + return ret def _clear_context(bin_env=None): diff --git a/salt/states/pip_state.py b/salt/states/pip_state.py index ab58fbd5fc..afe41d7fc8 100644 --- a/salt/states/pip_state.py +++ b/salt/states/pip_state.py @@ -23,7 +23,10 @@ requisite to a pkg.installed state for the package which provides pip from __future__ import absolute_import, print_function, unicode_literals import re import logging -import pkg_resources +try: + import pkg_resources +except ImportError: + pkg_resources = None # Import salt libs import salt.utils.versions @@ -71,9 +74,7 @@ def __virtual__(): ''' Only load if the pip module is available in __salt__ ''' - if 'pip.list' in __salt__: - return __virtualname__ - return False + return 'pip.list' in __salt__ and __virtualname__ or False def _find_key(prefix, pip_list): -- 2.19.0 ++++++ don-t-error-on-retcode-0-in-libcrypto.openssl_init_c.patch ++++++ >From 34089db15e7d3a1e361789f04613d0a13138dea0 Mon Sep 17 00:00:00 2001 From: rallytime <nic...@saltstack.com> Date: Fri, 13 Jul 2018 12:42:46 -0400 Subject: [PATCH] Don't error on retcode 0 in libcrypto.OPENSSL_init_crypto call Fixes #46884 --- salt/utils/rsax931.py | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/salt/utils/rsax931.py b/salt/utils/rsax931.py index 168c02734b..6bfef41bd3 100644 --- a/salt/utils/rsax931.py +++ b/salt/utils/rsax931.py @@ -71,10 +71,9 @@ def _init_libcrypto(): libcrypto.RSA_public_decrypt.argtypes = (c_int, c_char_p, c_char_p, c_void_p, c_int) try: - if libcrypto.OPENSSL_init_crypto(OPENSSL_INIT_NO_LOAD_CONFIG | - OPENSSL_INIT_ADD_ALL_CIPHERS | - OPENSSL_INIT_ADD_ALL_DIGESTS, None) != 1: - raise OSError("Failed to initialize OpenSSL library (OPENSSL_init_crypto failed)") + libcrypto.OPENSSL_init_crypto(OPENSSL_INIT_NO_LOAD_CONFIG | + OPENSSL_INIT_ADD_ALL_CIPHERS | + OPENSSL_INIT_ADD_ALL_DIGESTS, None) except AttributeError: # Support for OpenSSL < 1.1 (OPENSSL_API_COMPAT < 0x10100000L) libcrypto.OPENSSL_no_config() -- 2.19.2 ++++++ early-feature-support-config.patch ++++++ ++++ 1996 lines (skipped) ++++++ fix-async-call-to-process-manager.patch ++++++ >From b276ee7373e88d05c01912a9d9d3a44a5d17bab6 Mon Sep 17 00:00:00 2001 From: Daniel Wallace <danielwall...@gtmanfred.com> Date: Mon, 13 Aug 2018 13:55:37 -0500 Subject: [PATCH] fix async call to process manager --- salt/minion.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/salt/minion.py b/salt/minion.py index 9c05a646ea..8b8fd797d1 100644 --- a/salt/minion.py +++ b/salt/minion.py @@ -923,7 +923,7 @@ class MinionManager(MinionBase): install_zmq() self.io_loop = ZMQDefaultLoop.current() self.process_manager = ProcessManager(name='MultiMinionProcessManager') - self.io_loop.spawn_callback(self.process_manager.run, **{'async': True}) # Tornado backward compat + self.io_loop.spawn_callback(self.process_manager.run, **{'asynchronous': True}) # Tornado backward compat def __del__(self): self.destroy() @@ -1120,7 +1120,7 @@ class Minion(MinionBase): time.sleep(sleep_time) self.process_manager = ProcessManager(name='MinionProcessManager') - self.io_loop.spawn_callback(self.process_manager.run, **{'async': True}) + self.io_loop.spawn_callback(self.process_manager.run, **{'asynchronous': True}) # We don't have the proxy setup yet, so we can't start engines # Engines need to be able to access __proxy__ if not salt.utils.platform.is_proxy(): -- 2.17.1 ++++++ fix-git_pillar-merging-across-multiple-__env__-repos.patch ++++++ >From 49f8f296edf4655e2be7e564745931692ae939b7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pablo=20Su=C3=A1rez=20Hern=C3=A1ndez?= <psuarezhernan...@suse.com> Date: Tue, 6 Nov 2018 16:38:54 +0000 Subject: [PATCH] Fix git_pillar merging across multiple __env__ repositories (bsc#1112874) Resolve target branch when using __env__ Test git ext_pillar across multiple repos using __env__ Remove unicode references --- salt/utils/gitfs.py | 2 +- tests/integration/pillar/test_git_pillar.py | 144 ++++++++++++++++++++ tests/support/gitfs.py | 66 ++++++++- 3 files changed, 209 insertions(+), 3 deletions(-) diff --git a/salt/utils/gitfs.py b/salt/utils/gitfs.py index 6963f40226..11af741e35 100644 --- a/salt/utils/gitfs.py +++ b/salt/utils/gitfs.py @@ -2975,7 +2975,7 @@ class GitPillar(GitBase): if repo.env: env = repo.env else: - env = 'base' if repo.branch == repo.base else repo.branch + env = 'base' if repo.branch == repo.base else repo.get_checkout_target() if repo._mountpoint: if self.link_mountpoint(repo): self.pillar_dirs[repo.linkdir] = env diff --git a/tests/integration/pillar/test_git_pillar.py b/tests/integration/pillar/test_git_pillar.py index e97e720bab..e052782311 100644 --- a/tests/integration/pillar/test_git_pillar.py +++ b/tests/integration/pillar/test_git_pillar.py @@ -358,6 +358,38 @@ class GitPythonMixin(object): "available on the salt master"]} ) + def test_includes_enabled_solves___env___with_mountpoint(self): + ''' + Test with git_pillar_includes enabled and using "__env__" as the branch + name for the configured repositories. + The "gitinfo" repository contains top.sls file with a local reference + and also referencing external "nowhere.foo" which is provided by "webinfo" + repository mounted as "nowhere". + ''' + ret = self.get_pillar('''\ + file_ignore_regex: [] + file_ignore_glob: [] + git_pillar_provider: gitpython + cachedir: {cachedir} + extension_modules: {extmods} + ext_pillar: + - git: + - __env__ {url_extra_repo}: + - name: gitinfo + - __env__ {url}: + - name: webinfo + - mountpoint: nowhere + ''') + self.assertEqual( + ret, + {'branch': 'master', + 'motd': 'The force will be with you. Always.', + 'mylist': ['master'], + 'mydict': {'master': True, + 'nested_list': ['master'], + 'nested_dict': {'master': True}}} + ) + @destructiveTest @skipIf(NO_MOCK, NO_MOCK_REASON) @@ -413,7 +445,12 @@ class TestGitPythonAuthenticatedHTTP(TestGitPythonHTTP, GitPythonMixin): username=cls.username, password=cls.password, port=cls.nginx_port) + cls.url_extra_repo = 'http://{username}:{password}@127.0.0.1:{port}/extra_repo.git'.format( + username=cls.username, + password=cls.password, + port=cls.nginx_port) cls.ext_opts['url'] = cls.url + cls.ext_opts['url_extra_repo'] = cls.url_extra_repo cls.ext_opts['username'] = cls.username cls.ext_opts['password'] = cls.password @@ -1192,6 +1229,40 @@ class TestPygit2SSH(GitPillarSSHTestBase): ''') self.assertEqual(ret, expected) + def test_includes_enabled_solves___env___with_mountpoint(self): + ''' + Test with git_pillar_includes enabled and using "__env__" as the branch + name for the configured repositories. + The "gitinfo" repository contains top.sls file with a local reference + and also referencing external "nowhere.foo" which is provided by "webinfo" + repository mounted as "nowhere". + ''' + ret = self.get_pillar('''\ + file_ignore_regex: [] + file_ignore_glob: [] + git_pillar_provider: pygit2 + git_pillar_pubkey: {pubkey_nopass} + git_pillar_privkey: {privkey_nopass} + cachedir: {cachedir} + extension_modules: {extmods} + ext_pillar: + - git: + - __env__ {url_extra_repo}: + - name: gitinfo + - __env__ {url}: + - name: webinfo + - mountpoint: nowhere + ''') + self.assertEqual( + ret, + {'branch': 'master', + 'motd': 'The force will be with you. Always.', + 'mylist': ['master'], + 'mydict': {'master': True, + 'nested_list': ['master'], + 'nested_dict': {'master': True}}} + ) + @skipIf(NO_MOCK, NO_MOCK_REASON) @skipIf(_windows_or_mac(), 'minion is windows or mac') @@ -1439,6 +1510,38 @@ class TestPygit2HTTP(GitPillarHTTPTestBase): ''') self.assertEqual(ret, expected) + def test_includes_enabled_solves___env___with_mountpoint(self): + ''' + Test with git_pillar_includes enabled and using "__env__" as the branch + name for the configured repositories. + The "gitinfo" repository contains top.sls file with a local reference + and also referencing external "nowhere.foo" which is provided by "webinfo" + repository mounted as "nowhere". + ''' + ret = self.get_pillar('''\ + file_ignore_regex: [] + file_ignore_glob: [] + git_pillar_provider: pygit2 + cachedir: {cachedir} + extension_modules: {extmods} + ext_pillar: + - git: + - __env__ {url_extra_repo}: + - name: gitinfo + - __env__ {url}: + - name: webinfo + - mountpoint: nowhere + ''') + self.assertEqual( + ret, + {'branch': 'master', + 'motd': 'The force will be with you. Always.', + 'mylist': ['master'], + 'mydict': {'master': True, + 'nested_list': ['master'], + 'nested_dict': {'master': True}}} + ) + @skipIf(NO_MOCK, NO_MOCK_REASON) @skipIf(_windows_or_mac(), 'minion is windows or mac') @@ -1887,3 +1990,44 @@ class TestPygit2AuthenticatedHTTP(GitPillarHTTPTestBase): - env: base ''') self.assertEqual(ret, expected) + + def test_includes_enabled_solves___env___with_mountpoint(self): + ''' + Test with git_pillar_includes enabled and using "__env__" as the branch + name for the configured repositories. + The "gitinfo" repository contains top.sls file with a local reference + and also referencing external "nowhere.foo" which is provided by "webinfo" + repository mounted as "nowhere". + ''' + ret = self.get_pillar('''\ + file_ignore_regex: [] + file_ignore_glob: [] + git_pillar_provider: pygit2 + git_pillar_user: {user} + git_pillar_password: {password} + git_pillar_insecure_auth: True + cachedir: {cachedir} + extension_modules: {extmods} + ext_pillar: + - git: + - __env__ {url_extra_repo}: + - name: gitinfo + - user: {user} + - password: {password} + - insecure_auth: True + - __env__ {url}: + - name: webinfo + - mountpoint: nowhere + - user: {user} + - password: {password} + - insecure_auth: True + ''') + self.assertEqual( + ret, + {'branch': 'master', + 'motd': 'The force will be with you. Always.', + 'mylist': ['master'], + 'mydict': {'master': True, + 'nested_list': ['master'], + 'nested_dict': {'master': True}}} + ) diff --git a/tests/support/gitfs.py b/tests/support/gitfs.py index 2afd31539d..e645c50a86 100644 --- a/tests/support/gitfs.py +++ b/tests/support/gitfs.py @@ -133,9 +133,13 @@ class SSHDMixin(ModuleCase, ProcessManager, SaltReturnAssertsMixin): cls.url = 'ssh://{username}@127.0.0.1:{port}/~/repo.git'.format( username=cls.username, port=cls.sshd_port) + cls.url_extra_repo = 'ssh://{username}@127.0.0.1:{port}/~/extra_repo.git'.format( + username=cls.username, + port=cls.sshd_port) home = '/root/.ssh' cls.ext_opts = { 'url': cls.url, + 'url_extra_repo': cls.url_extra_repo, 'privkey_nopass': os.path.join(home, cls.id_rsa_nopass), 'pubkey_nopass': os.path.join(home, cls.id_rsa_nopass + '.pub'), 'privkey_withpass': os.path.join(home, cls.id_rsa_withpass), @@ -193,7 +197,8 @@ class WebserverMixin(ModuleCase, ProcessManager, SaltReturnAssertsMixin): # get_unused_localhost_port() return identical port numbers. cls.uwsgi_port = get_unused_localhost_port() cls.url = 'http://127.0.0.1:{port}/repo.git'.format(port=cls.nginx_port) - cls.ext_opts = {'url': cls.url} + cls.url_extra_repo = 'http://127.0.0.1:{port}/extra_repo.git'.format(port=cls.nginx_port) + cls.ext_opts = {'url': cls.url, 'url_extra_repo': cls.url_extra_repo} # Add auth params if present (if so this will trigger the spawned # server to turn on HTTP basic auth). for credential_param in ('user', 'password'): @@ -250,7 +255,7 @@ class GitTestBase(ModuleCase): Base class for all gitfs/git_pillar tests. Must be subclassed and paired with either SSHDMixin or WebserverMixin to provide the server. ''' - case = port = bare_repo = admin_repo = None + case = port = bare_repo = base_extra_repo = admin_repo = admin_extra_repo = None maxDiff = None git_opts = '-c user.name="Foo Bar" -c user.email=f...@bar.com' ext_opts = {} @@ -465,6 +470,61 @@ class GitPillarTestBase(GitTestBase, LoaderModuleMockMixin): ''')) _push('top_only', 'add top_only branch') + def make_extra_repo(self, root_dir, user='root'): + self.bare_extra_repo = os.path.join(root_dir, 'extra_repo.git') + self.admin_extra_repo = os.path.join(root_dir, 'admin_extra') + + for dirname in (self.bare_extra_repo, self.admin_extra_repo): + shutil.rmtree(dirname, ignore_errors=True) + + # Create bare extra repo + self.run_function( + 'git.init', + [self.bare_extra_repo], + user=user, + bare=True) + + # Clone bare repo + self.run_function( + 'git.clone', + [self.admin_extra_repo], + url=self.bare_extra_repo, + user=user) + + def _push(branch, message): + self.run_function( + 'git.add', + [self.admin_extra_repo, '.'], + user=user) + self.run_function( + 'git.commit', + [self.admin_extra_repo, message], + user=user, + git_opts=self.git_opts, + ) + self.run_function( + 'git.push', + [self.admin_extra_repo], + remote='origin', + ref=branch, + user=user, + ) + + with salt.utils.files.fopen( + os.path.join(self.admin_extra_repo, 'top.sls'), 'w') as fp_: + fp_.write(textwrap.dedent('''\ + "{{saltenv}}": + '*': + - motd + - nowhere.foo + ''')) + with salt.utils.files.fopen( + os.path.join(self.admin_extra_repo, 'motd.sls'), 'w') as fp_: + fp_.write(textwrap.dedent('''\ + motd: The force will be with you. Always. + ''')) + _push('master', 'initial commit') + class GitPillarSSHTestBase(GitPillarTestBase, SSHDMixin): ''' @@ -533,6 +593,7 @@ class GitPillarSSHTestBase(GitPillarTestBase, SSHDMixin): ) ) self.make_repo(root_dir, user=self.username) + self.make_extra_repo(root_dir, user=self.username) def get_pillar(self, ext_pillar_conf): ''' @@ -579,3 +640,4 @@ class GitPillarHTTPTestBase(GitPillarTestBase, WebserverMixin): self.spawn_server() # pylint: disable=E1120 self.make_repo(self.repo_dir) + self.make_extra_repo(self.repo_dir) -- 2.17.1 ++++++ fix-index-error-when-running-on-python-3.patch ++++++ >From 5502f05fac89330ab26d04e29d3aa6d36ab928c5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pablo=20Su=C3=A1rez=20Hern=C3=A1ndez?= <psuarezhernan...@suse.com> Date: Thu, 20 Sep 2018 11:51:58 +0100 Subject: [PATCH] Fix index error when running on Python 3 Fix wrong queryformat for zypper list_provides --- salt/modules/zypper.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/salt/modules/zypper.py b/salt/modules/zypper.py index 695bce4f4e..e4423cf1fc 100644 --- a/salt/modules/zypper.py +++ b/salt/modules/zypper.py @@ -2314,7 +2314,7 @@ def list_provides(**kwargs): ''' ret = __context__.get('pkg.list_provides') if not ret: - cmd = ['rpm', '-qa', '--queryformat', '[%{PROVIDES}_|-%{NAME}\n]'] + cmd = ['rpm', '-qa', '--queryformat', '%{PROVIDES}_|-%{NAME}\n'] ret = dict() for line in __salt__['cmd.run'](cmd, output_loglevel='trace', python_shell=False).splitlines(): provide, realname = line.split('_|-') @@ -2379,7 +2379,7 @@ def resolve_capabilities(pkgs, refresh, **kwargs): try: result = search(name, provides=True, match='exact') if len(result) == 1: - name = result.keys()[0] + name = next(iter(result.keys())) elif len(result) > 1: log.warn("Found ambiguous match for capability '%s'.", pkg) except CommandExecutionError as exc: -- 2.17.1 ++++++ fix-ipv6-scope-bsc-1108557.patch ++++++ ++++ 659 lines (skipped) ++++++ fix-issue-2068-test.patch ++++++ >From 2916f2f3e7c6af07148863281ffaf07df21f21da Mon Sep 17 00:00:00 2001 From: Bo Maryniuk <b...@suse.de> Date: Wed, 9 Jan 2019 16:08:19 +0100 Subject: [PATCH] Fix issue #2068 test Skip injecting `__call__` if chunk is not dict. This also fixes `integration/modules/test_state.py:StateModuleTest.test_exclude` that tests `include` and `exclude` state directives containing the only list of strings. Minor update: more correct is-dict check. --- salt/state.py | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/salt/state.py b/salt/state.py index b4b2a00601..815ebaec24 100644 --- a/salt/state.py +++ b/salt/state.py @@ -25,6 +25,7 @@ import traceback import re import time import random +import collections # Import salt libs import salt.loader @@ -2743,16 +2744,18 @@ class State(object): ''' for chunk in high: state = high[chunk] + if not isinstance(state, collections.Mapping): + continue for state_ref in state: needs_default = True + if not isinstance(state[state_ref], list): + continue for argset in state[state_ref]: if isinstance(argset, six.string_types): needs_default = False break if needs_default: - order = state[state_ref].pop(-1) - state[state_ref].append('__call__') - state[state_ref].append(order) + state[state_ref].insert(-1, '__call__') def call_high(self, high, orchestration_jid=None): ''' -- 2.20.1 ++++++ fix-latin1-encoding-problems-on-file-module-bsc-1116.patch ++++++ ++++ 941 lines (skipped) ++++++ fix-unit-test-for-grains-core.patch ++++++ >From 7ffa39cd80393f2a3ed5cd75793b134b9d939cf9 Mon Sep 17 00:00:00 2001 From: Bo Maryniuk <b...@suse.de> Date: Thu, 11 Oct 2018 16:20:40 +0200 Subject: [PATCH] Fix unit test for grains core --- tests/unit/grains/test_core.py | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/tests/unit/grains/test_core.py b/tests/unit/grains/test_core.py index 2ab32ef41b..4923ee00b0 100644 --- a/tests/unit/grains/test_core.py +++ b/tests/unit/grains/test_core.py @@ -62,11 +62,10 @@ class CoreGrainsTestCase(TestCase, LoaderModuleMockMixin): def test_parse_etc_os_release(self, path_isfile_mock): path_isfile_mock.side_effect = lambda x: x == "/usr/lib/os-release" with salt.utils.files.fopen(os.path.join(OS_RELEASE_DIR, "ubuntu-17.10")) as os_release_file: - os_release_content = os_release_file.read() - with patch("salt.utils.files.fopen", mock_open(read_data=os_release_content)): - os_release = core._parse_os_release( - '/etc/os-release', - '/usr/lib/os-release') + os_release_content = os_release_file.readlines() + with patch("salt.utils.files.fopen", mock_open()) as os_release_file: + os_release_file.return_value.__iter__.return_value = os_release_content + os_release = core._parse_os_release(["/etc/os-release", "/usr/lib/os-release"]) self.assertEqual(os_release, { "NAME": "Ubuntu", "VERSION": "17.10 (Artful Aardvark)", @@ -128,7 +127,7 @@ class CoreGrainsTestCase(TestCase, LoaderModuleMockMixin): def test_missing_os_release(self): with patch('salt.utils.files.fopen', mock_open(read_data={})): - os_release = core._parse_os_release('/etc/os-release', '/usr/lib/os-release') + os_release = core._parse_os_release(['/etc/os-release', '/usr/lib/os-release']) self.assertEqual(os_release, {}) @skipIf(not salt.utils.platform.is_linux(), 'System is not Linux') -- 2.19.0 ++++++ fixes-cve-2018-15750-cve-2018-15751.patch ++++++ >From 43b1f8fb6608c944812bc5bcd9da407624409ac7 Mon Sep 17 00:00:00 2001 From: Erik Johnson <paleh...@gmail.com> Date: Fri, 24 Aug 2018 10:35:55 -0500 Subject: [PATCH] Fixes: CVE-2018-15750, CVE-2018-15751 Ensure that tokens are hex to avoid hanging/errors in cherrypy Add empty token salt-api integration tests Handle Auth exceptions in run_job Update tornado test to correct authentication message --- salt/client/__init__.py | 8 ++++ salt/netapi/rest_cherrypy/app.py | 13 ++++++- .../netapi/rest_cherrypy/test_app.py | 39 +++++++++++++++++++ .../netapi/rest_tornado/test_app.py | 2 +- 4 files changed, 60 insertions(+), 2 deletions(-) diff --git a/salt/client/__init__.py b/salt/client/__init__.py index dcbc1473e1..77f2a963f7 100644 --- a/salt/client/__init__.py +++ b/salt/client/__init__.py @@ -349,6 +349,10 @@ class LocalClient(object): raise SaltClientError( 'The salt master could not be contacted. Is master running?' ) + except AuthenticationError as err: + raise AuthenticationError(err) + except AuthorizationError as err: + raise AuthorizationError(err) except Exception as general_exception: # Convert to generic client error and pass along message raise SaltClientError(general_exception) @@ -415,6 +419,10 @@ class LocalClient(object): raise SaltClientError( 'The salt master could not be contacted. Is master running?' ) + except AuthenticationError as err: + raise AuthenticationError(err) + except AuthorizationError as err: + raise AuthorizationError(err) except Exception as general_exception: # Convert to generic client error and pass along message raise SaltClientError(general_exception) diff --git a/salt/netapi/rest_cherrypy/app.py b/salt/netapi/rest_cherrypy/app.py index 78ea3c3fef..c272674146 100644 --- a/salt/netapi/rest_cherrypy/app.py +++ b/salt/netapi/rest_cherrypy/app.py @@ -1167,6 +1167,13 @@ class LowDataAdapter(object): if token: chunk['token'] = token + if 'token' in chunk: + # Make sure that auth token is hex + try: + int(chunk['token'], 16) + except (TypeError, ValueError): + raise cherrypy.HTTPError(401, 'Invalid token') + if client: chunk['client'] = client @@ -2167,7 +2174,11 @@ class Events(object): :return bool: True if valid, False if not valid. ''' - if auth_token is None: + # Make sure that auth token is hex. If it's None, or something other + # than hex, this will raise a ValueError. + try: + int(auth_token, 16) + except ValueError: return False # First check if the given token is in our session table; if so it's a diff --git a/tests/integration/netapi/rest_cherrypy/test_app.py b/tests/integration/netapi/rest_cherrypy/test_app.py index 000b7418bf..5865510fd7 100644 --- a/tests/integration/netapi/rest_cherrypy/test_app.py +++ b/tests/integration/netapi/rest_cherrypy/test_app.py @@ -124,6 +124,45 @@ class TestRun(cptc.BaseRestCherryPyTest): }) self.assertEqual(response.status, '401 Unauthorized') + def test_run_empty_token(self): + ''' + Test the run URL with empty token + ''' + cmd = dict(self.low, **{'token': ''}) + body = urlencode(cmd) + + request, response = self.request('/run', method='POST', body=body, + headers={ + 'content-type': 'application/x-www-form-urlencoded' + }) + assert response.status == '401 Unauthorized' + + def test_run_empty_token_upercase(self): + ''' + Test the run URL with empty token with upercase characters + ''' + cmd = dict(self.low, **{'ToKen': ''}) + body = urlencode(cmd) + + request, response = self.request('/run', method='POST', body=body, + headers={ + 'content-type': 'application/x-www-form-urlencoded' + }) + assert response.status == '401 Unauthorized' + + def test_run_wrong_token(self): + ''' + Test the run URL with incorrect token + ''' + cmd = dict(self.low, **{'token': 'bad'}) + body = urlencode(cmd) + + request, response = self.request('/run', method='POST', body=body, + headers={ + 'content-type': 'application/x-www-form-urlencoded' + }) + assert response.status == '401 Unauthorized' + class TestWebhookDisableAuth(cptc.BaseRestCherryPyTest): diff --git a/tests/integration/netapi/rest_tornado/test_app.py b/tests/integration/netapi/rest_tornado/test_app.py index beb085db1e..01abd354a7 100644 --- a/tests/integration/netapi/rest_tornado/test_app.py +++ b/tests/integration/netapi/rest_tornado/test_app.py @@ -237,7 +237,7 @@ class TestSaltAPIHandler(_SaltnadoIntegrationTestCase): self.assertEqual(len(ret), 3) # make sure we got 3 responses self.assertIn('jid', ret[0]) # the first 2 are regular returns self.assertIn('jid', ret[1]) - self.assertIn('Authentication error occurred.', ret[2]) # bad auth + self.assertIn('Failed to authenticate', ret[2]) # bad auth self.assertEqual(ret[0]['minions'], sorted(['minion', 'sub_minion', 'localhost'])) self.assertEqual(ret[1]['minions'], sorted(['minion', 'sub_minion', 'localhost'])) -- 2.17.1 ++++++ fixing-issue-when-a-valid-token-is-generated-even-wh.patch ++++++ >From 6c85da9a53e9dd022c96a199be4e3bdd280543d6 Mon Sep 17 00:00:00 2001 From: "Gareth J. Greenaway" <gar...@wiked.org> Date: Thu, 2 Aug 2018 15:35:24 -0700 Subject: [PATCH] Fixing issue when a valid token is generated even when invalid user credentials are passed. This change verifies that the binddn credentials are valid, then verifies that the username & password (if not None) are also valid. --- salt/auth/ldap.py | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/salt/auth/ldap.py b/salt/auth/ldap.py index cbfb03a2f2..0b9aa69fe4 100644 --- a/salt/auth/ldap.py +++ b/salt/auth/ldap.py @@ -283,9 +283,15 @@ def auth(username, password): log.error('LDAP authentication requires python-ldap module') return False - # If bind credentials are configured, use them instead of user's + # If bind credentials are configured, verify that we can a valid bind if _config('binddn', mandatory=False) and _config('bindpw', mandatory=False): bind = _bind_for_search(anonymous=_config('anonymous', mandatory=False)) + + # If username & password are not None, attempt to verify they are valid + if bind and username and password: + bind = _bind(username, password, + anonymous=_config('auth_by_group_membership_only', mandatory=False) + and _config('anonymous', mandatory=False)) else: bind = _bind(username, password, anonymous=_config('auth_by_group_membership_only', mandatory=False) -- 2.19.0 ++++++ get-os_arch-also-without-rpm-package-installed.patch ++++++ >From 2e0abe6d12aa2657a4febed3a80b8c4cf104487a Mon Sep 17 00:00:00 2001 From: Bo Maryniuk <b...@suse.de> Date: Wed, 14 Nov 2018 17:36:23 +0100 Subject: [PATCH] Get os_arch also without RPM package installed backport pkg.rpm test Add pkg.rpm unit test case Fix docstring Add UT for getting OS architecture fallback, when no RPM found (initrd, e.g.) Add UT for OS architecture detection on fallback, when no CPU arch can be determined Add UT for OS arch detection when no CPU arch or machine can be determined Remove unsupported testcase --- salt/utils/pkg/rpm.py | 18 ++++++--- tests/unit/utils/test_pkg.py | 72 ++++++++++++++++++++++++++++++++++++ 2 files changed, 84 insertions(+), 6 deletions(-) create mode 100644 tests/unit/utils/test_pkg.py diff --git a/salt/utils/pkg/rpm.py b/salt/utils/pkg/rpm.py index 94e231da4b..bb8c3fb589 100644 --- a/salt/utils/pkg/rpm.py +++ b/salt/utils/pkg/rpm.py @@ -9,7 +9,9 @@ import collections import datetime import logging import subprocess +import platform import salt.utils.stringutils +import salt.utils.path # Import 3rd-party libs from salt.ext import six @@ -42,12 +44,16 @@ def get_osarch(): ''' Get the os architecture using rpm --eval ''' - ret = subprocess.Popen( - 'rpm --eval "%{_host_cpu}"', - shell=True, - close_fds=True, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE).communicate()[0] + if salt.utils.path.which('rpm'): + ret = subprocess.Popen( + 'rpm --eval "%{_host_cpu}"', + shell=True, + close_fds=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE).communicate()[0] + else: + ret = ''.join(list(filter(None, platform.uname()[-2:]))[-1:]) + return salt.utils.stringutils.to_str(ret).strip() or 'unknown' diff --git a/tests/unit/utils/test_pkg.py b/tests/unit/utils/test_pkg.py new file mode 100644 index 0000000000..361e0bf92f --- /dev/null +++ b/tests/unit/utils/test_pkg.py @@ -0,0 +1,72 @@ +# -*- coding: utf-8 -*- + +from __future__ import absolute_import, unicode_literals, print_function + +from tests.support.unit import TestCase, skipIf +from tests.support.mock import Mock, MagicMock, patch, NO_MOCK, NO_MOCK_REASON +import salt.utils.pkg +from salt.utils.pkg import rpm + +try: + import pytest +except ImportError: + pytest = None + + +@skipIf(NO_MOCK, NO_MOCK_REASON) +@skipIf(pytest is None, 'PyTest is missing') +class PkgRPMTestCase(TestCase): + ''' + Test case for pkg.rpm utils + ''' + + @patch('salt.utils.path.which', MagicMock(return_value=True)) + def test_get_osarch_by_rpm(self): + ''' + Get os_arch if RPM package is installed. + :return: + ''' + subprocess_mock = MagicMock() + subprocess_mock.Popen = MagicMock() + subprocess_mock.Popen().communicate = MagicMock(return_value=['Z80']) + with patch('salt.utils.pkg.rpm.subprocess', subprocess_mock): + assert rpm.get_osarch() == 'Z80' + assert subprocess_mock.Popen.call_count == 2 # One within the mock + assert subprocess_mock.Popen.call_args[1]['close_fds'] + assert subprocess_mock.Popen.call_args[1]['shell'] + assert len(subprocess_mock.Popen.call_args_list) == 2 + assert subprocess_mock.Popen.call_args[0][0] == 'rpm --eval "%{_host_cpu}"' + + @patch('salt.utils.path.which', MagicMock(return_value=False)) + @patch('salt.utils.pkg.rpm.subprocess', MagicMock(return_value=False)) + @patch('salt.utils.pkg.rpm.platform.uname', MagicMock( + return_value=('Sinclair BASIC', 'motophone', '1982 Sinclair Research Ltd', '1.0', 'ZX81', 'Z80'))) + def test_get_osarch_by_platform(self): + ''' + Get os_arch if RPM package is not installed (inird image, for example). + :return: + ''' + assert rpm.get_osarch() == 'Z80' + + @patch('salt.utils.path.which', MagicMock(return_value=False)) + @patch('salt.utils.pkg.rpm.subprocess', MagicMock(return_value=False)) + @patch('salt.utils.pkg.rpm.platform.uname', MagicMock( + return_value=('Sinclair BASIC', 'motophone', '1982 Sinclair Research Ltd', '1.0', 'ZX81', ''))) + def test_get_osarch_by_platform_no_cpu_arch(self): + ''' + Get os_arch if RPM package is not installed (inird image, for example) but cpu arch cannot be determined. + :return: + ''' + assert rpm.get_osarch() == 'ZX81' + + @patch('salt.utils.path.which', MagicMock(return_value=False)) + @patch('salt.utils.pkg.rpm.subprocess', MagicMock(return_value=False)) + @patch('salt.utils.pkg.rpm.platform.uname', MagicMock( + return_value=('Sinclair BASIC', 'motophone', '1982 Sinclair Research Ltd', '1.0', '', ''))) + def test_get_osarch_by_platform_no_cpu_arch_no_machine(self): + ''' + Get os_arch if RPM package is not installed (inird image, for example) + where both cpu arch and machine cannot be determined. + :return: + ''' + assert rpm.get_osarch() == 'unknown' -- 2.19.1 ++++++ get-os_family-for-rpm-distros-from-the-rpm-macros.-u.patch ++++++ >From 943a258da3ed460f173968b0a92b95f2e63ab669 Mon Sep 17 00:00:00 2001 From: Bo Maryniuk <b...@suse.de> Date: Mon, 8 Oct 2018 12:48:24 +0200 Subject: [PATCH] Get os_family for RPM distros from the RPM macros. (U#49930) Strip and stringify the return for the osarch Fix imports --- salt/grains/core.py | 8 +++++--- salt/utils/pkg/rpm.py | 3 ++- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/salt/grains/core.py b/salt/grains/core.py index 6aaf38096d..80eebd1c05 100644 --- a/salt/grains/core.py +++ b/salt/grains/core.py @@ -49,6 +49,8 @@ import salt.utils.path import salt.utils.platform import salt.utils.stringutils import salt.utils.versions +import salt.utils.pkg.rpm + from salt.ext import six from salt.ext.six.moves import range @@ -1776,9 +1778,9 @@ def os_data(): # architecture. if grains.get('os_family') == 'Debian': osarch = __salt__['cmd.run']('dpkg --print-architecture').strip() - elif grains.get('os_family') == 'RedHat': - osarch = __salt__['cmd.run']('rpm --eval %{_host_cpu}').strip() - elif grains.get('os_family') == 'NILinuxRT': + elif grains.get('os_family') in ['RedHat', 'Suse']: + osarch = salt.utils.pkg.rpm.get_osarch() + elif grains.get('os_family') in ('NILinuxRT', 'Poky'): archinfo = {} for line in __salt__['cmd.run']('opkg print-architecture').splitlines(): if line.startswith('arch'): diff --git a/salt/utils/pkg/rpm.py b/salt/utils/pkg/rpm.py index 987edab894..94e231da4b 100644 --- a/salt/utils/pkg/rpm.py +++ b/salt/utils/pkg/rpm.py @@ -9,6 +9,7 @@ import collections import datetime import logging import subprocess +import salt.utils.stringutils # Import 3rd-party libs from salt.ext import six @@ -47,7 +48,7 @@ def get_osarch(): close_fds=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[0] - return ret or 'unknown' + return salt.utils.stringutils.to_str(ret).strip() or 'unknown' def check_32(arch, osarch=None): -- 2.19.0 ++++++ improved-handling-of-ldap-group-id.patch ++++++ >From d0234ed977ca860b3a3a6a587a6972bbaf5ae345 Mon Sep 17 00:00:00 2001 From: Raine Curtis <rcur...@suse.com> Date: Mon, 9 Jul 2018 09:55:30 -0600 Subject: [PATCH] Improved handling of LDAP group id gid is casted to int, which should be the case. Otherwise an error is returned. --- salt/states/group.py | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/salt/states/group.py b/salt/states/group.py index 6a720757e8..acf775134c 100644 --- a/salt/states/group.py +++ b/salt/states/group.py @@ -72,9 +72,16 @@ def _changes(name, delusers = [salt.utils.win_functions.get_sam_name(user).lower() for user in delusers] change = {} + ret = {} if gid: - if lgrp['gid'] != gid: - change['gid'] = gid + try: + gid = int(gid) + if lgrp['gid'] != gid: + change['gid'] = gid + except (TypeError, ValueError): + ret['result'] = False + ret['comment'] = 'Invalid gid' + return ret if members: # -- if new member list if different than the current -- 2.19.1 ++++++ loosen-azure-sdk-dependencies-in-azurearm-cloud-driv.patch ++++++ >From 9d9fb3fd787b40d9d27ad7c5eb69fa0cd4f5a304 Mon Sep 17 00:00:00 2001 From: Joachim Gleissner <jgleiss...@suse.com> Date: Tue, 18 Sep 2018 15:07:13 +0200 Subject: [PATCH] loosen azure sdk dependencies in azurearm cloud driver Remove dependency to azure-cli, which is not used at all. Use azure-storage-sdk as fallback if multiapi version is not available. remove unused import from azurearm driver --- salt/cloud/clouds/azurearm.py | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/salt/cloud/clouds/azurearm.py b/salt/cloud/clouds/azurearm.py index 8b9a9e8903..50e5ce1f62 100644 --- a/salt/cloud/clouds/azurearm.py +++ b/salt/cloud/clouds/azurearm.py @@ -67,6 +67,7 @@ import logging import pprint import base64 import collections +import pkgutil import salt.cache import salt.config as config import salt.utils.cloud @@ -74,7 +75,6 @@ import salt.utils.data import salt.utils.files import salt.utils.stringutils import salt.utils.yaml -from salt.utils.versions import LooseVersion from salt.ext import six import salt.version from salt.exceptions import ( @@ -125,9 +125,12 @@ try: from azure.mgmt.storage import StorageManagementClient from azure.mgmt.web import WebSiteManagementClient from msrestazure.azure_exceptions import CloudError - from azure.multiapi.storage.v2016_05_31 import CloudStorageAccount - from azure.cli import core - HAS_LIBS = LooseVersion(core.__version__) >= LooseVersion("2.0.12") + if pkgutil.find_loader('azure.multiapi'): + # use multiapi version if available + from azure.multiapi.storage.v2016_05_31 import CloudStorageAccount + else: + from azure.storage import CloudStorageAccount + HAS_LIBS = True except ImportError: pass # pylint: enable=wrong-import-position,wrong-import-order @@ -160,8 +163,7 @@ def __virtual__(): False, 'The following dependencies are required to use the AzureARM driver: ' 'Microsoft Azure SDK for Python >= 2.0rc5, ' - 'Microsoft Azure Storage SDK for Python >= 0.32, ' - 'Microsoft Azure CLI >= 2.0.12' + 'Microsoft Azure Storage SDK for Python >= 0.32' ) global cache # pylint: disable=global-statement,invalid-name -- 2.17.1 ++++++ make-aptpkg.list_repos-compatible-on-enabled-disable.patch ++++++ >From 350b0aa4ead80ac50047c08121bc09bddc05341d Mon Sep 17 00:00:00 2001 From: Bo Maryniuk <b...@suse.de> Date: Fri, 16 Nov 2018 10:54:12 +0100 Subject: [PATCH] Make aptpkg.list_repos compatible on enabled/disabled output --- salt/modules/aptpkg.py | 1 + 1 file changed, 1 insertion(+) diff --git a/salt/modules/aptpkg.py b/salt/modules/aptpkg.py index 175ef2ed06..90b99c44b9 100644 --- a/salt/modules/aptpkg.py +++ b/salt/modules/aptpkg.py @@ -1719,6 +1719,7 @@ def list_repos(): repo['file'] = source.file repo['comps'] = getattr(source, 'comps', []) repo['disabled'] = source.disabled + repo['enabled'] = not repo['disabled'] # This is for compatibility with the other modules repo['dist'] = source.dist repo['type'] = source.type repo['uri'] = source.uri.rstrip('/') -- 2.19.1 ++++++ make-profiles-a-package.patch ++++++ >From 155aa52dca9272db492990ad737256dada1c4364 Mon Sep 17 00:00:00 2001 From: Bo Maryniuk <b...@suse.de> Date: Mon, 8 Oct 2018 17:52:07 +0200 Subject: [PATCH] Make profiles a package. Add UTF-8 encoding Add a docstring --- salt/cli/support/profiles/__init__.py | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 salt/cli/support/profiles/__init__.py diff --git a/salt/cli/support/profiles/__init__.py b/salt/cli/support/profiles/__init__.py new file mode 100644 index 0000000000..b86aef30b8 --- /dev/null +++ b/salt/cli/support/profiles/__init__.py @@ -0,0 +1,4 @@ +# coding=utf-8 +''' +Profiles for salt-support. +''' -- 2.19.0 ++++++ preserving-signature-in-module.run-state-u-50049.patch ++++++ >From 318b4e0cd2efb02f26392bfe2d354a3ff5d21cbc Mon Sep 17 00:00:00 2001 From: Bo Maryniuk <b...@suse.de> Date: Mon, 15 Oct 2018 17:26:16 +0200 Subject: [PATCH] Preserving signature in "module.run" state (U#50049) Add unit test for _call_function on signature aligning named arguments Add unit test for _call_function routine for unnamed positional arguments Remove redundant docstrings Add different test function signature with the same outcome Replace standalone function with lambda-proxy for signatures only --- salt/states/module.py | 7 +++++-- tests/unit/states/test_module.py | 27 +++++++++++++++++++++++++++ 2 files changed, 32 insertions(+), 2 deletions(-) diff --git a/salt/states/module.py b/salt/states/module.py index 2190ffa3d2..90b1d0a5f5 100644 --- a/salt/states/module.py +++ b/salt/states/module.py @@ -323,7 +323,7 @@ def _call_function(name, returner=None, **kwargs): # func_args is initialized to a list of positional arguments that the function to be run accepts func_args = argspec.args[:len(argspec.args or []) - len(argspec.defaults or [])] - arg_type, na_type, kw_type = [], {}, False + arg_type, kw_to_arg_type, na_type, kw_type = [], {}, {}, False for funcset in reversed(kwargs.get('func_args') or []): if not isinstance(funcset, dict): # We are just receiving a list of args to the function to be run, so just append @@ -334,13 +334,16 @@ def _call_function(name, returner=None, **kwargs): # We are going to pass in a keyword argument. The trick here is to make certain # that if we find that in the *args* list that we pass it there and not as a kwarg if kwarg_key in func_args: - arg_type.append(funcset[kwarg_key]) + kw_to_arg_type[kwarg_key] = funcset[kwarg_key] continue else: # Otherwise, we're good and just go ahead and pass the keyword/value pair into # the kwargs list to be run. func_kw.update(funcset) arg_type.reverse() + for arg in func_args: + if arg in kw_to_arg_type: + arg_type.append(kw_to_arg_type[arg]) _exp_prm = len(argspec.args or []) - len(argspec.defaults or []) _passed_prm = len(arg_type) missing = [] diff --git a/tests/unit/states/test_module.py b/tests/unit/states/test_module.py index bf4ddcc5b4..25082d4bb4 100644 --- a/tests/unit/states/test_module.py +++ b/tests/unit/states/test_module.py @@ -324,3 +324,30 @@ class ModuleStateTest(TestCase, LoaderModuleMockMixin): self.assertIn(comment, ret['comment']) self.assertIn('world', ret['comment']) self.assertIn('hello', ret['comment']) + + def test_call_function_named_args(self): + ''' + Test _call_function routine when params are named. Their position ordering should not matter. + + :return: + ''' + with patch.dict(module.__salt__, + {'testfunc': lambda a, b, c, *args, **kwargs: (a, b, c, args, kwargs)}, clear=True): + assert module._call_function('testfunc', func_args=[{'a': 1}, {'b': 2}, {'c': 3}]) == (1, 2, 3, (), {}) + assert module._call_function('testfunc', func_args=[{'c': 3}, {'a': 1}, {'b': 2}]) == (1, 2, 3, (), {}) + + with patch.dict(module.__salt__, + {'testfunc': lambda c, a, b, *args, **kwargs: (a, b, c, args, kwargs)}, clear=True): + assert module._call_function('testfunc', func_args=[{'a': 1}, {'b': 2}, {'c': 3}]) == (1, 2, 3, (), {}) + assert module._call_function('testfunc', func_args=[{'c': 3}, {'a': 1}, {'b': 2}]) == (1, 2, 3, (), {}) + + def test_call_function_ordered_args(self): + ''' + Test _call_function routine when params are not named. Their position should matter. + + :return: + ''' + with patch.dict(module.__salt__, + {'testfunc': lambda a, b, c, *args, **kwargs: (a, b, c, args, kwargs)}, clear=True): + assert module._call_function('testfunc', func_args=[1, 2, 3]) == (1, 2, 3, (), {}) + assert module._call_function('testfunc', func_args=[3, 1, 2]) == (3, 1, 2, (), {}) -- 2.19.0 ++++++ remove-arch-from-name-when-pkg.list_pkgs-is-called-w.patch ++++++ ++++ 744 lines (skipped) ++++++ retire-md5-checksum-for-pkg-mgmt-plugins.patch ++++++ >From 0908344fae3edda3372ee03820ea30ebcfe8980e Mon Sep 17 00:00:00 2001 From: Bo Maryniuk <b...@suse.de> Date: Thu, 13 Sep 2018 12:00:55 +0200 Subject: [PATCH] Retire MD5 checksum for pkg mgmt plugins Use SHA256 algorithm for zyppnotify plugin Remove an empty line --- scripts/suse/yum/plugins/yumnotify.py | 2 +- scripts/suse/zypper/plugins/commit/zyppnotify | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/scripts/suse/yum/plugins/yumnotify.py b/scripts/suse/yum/plugins/yumnotify.py index 268e1e9531..dd2485c886 100644 --- a/scripts/suse/yum/plugins/yumnotify.py +++ b/scripts/suse/yum/plugins/yumnotify.py @@ -32,7 +32,7 @@ def _get_checksum(): Returns: hexdigest """ - digest = hashlib.md5() + digest = hashlib.sha256() with open(RPM_PATH, "rb") as rpm_db_fh: while True: buff = rpm_db_fh.read(0x1000) diff --git a/scripts/suse/zypper/plugins/commit/zyppnotify b/scripts/suse/zypper/plugins/commit/zyppnotify index 268298b108..b64badb119 100755 --- a/scripts/suse/zypper/plugins/commit/zyppnotify +++ b/scripts/suse/zypper/plugins/commit/zyppnotify @@ -35,7 +35,7 @@ class DriftDetector(Plugin): Returns: hexdigest ''' - digest = hashlib.md5() + digest = hashlib.sha256() with open(self.rpm_path, "rb") as rpm_db_fh: while True: buff = rpm_db_fh.read(0x1000) -- 2.20.1 ++++++ return-the-expected-powerpc-os-arch-bsc-1117995.patch ++++++ >From 2cbc403b422a699cd948ed6218fce28fa901f5fa Mon Sep 17 00:00:00 2001 From: Mihai Dinca <mdi...@suse.de> Date: Thu, 13 Dec 2018 12:17:35 +0100 Subject: [PATCH] Return the expected powerpc os arch (bsc#1117995) --- salt/utils/pkg/rpm.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/salt/utils/pkg/rpm.py b/salt/utils/pkg/rpm.py index bb8c3fb589..828b0cecda 100644 --- a/salt/utils/pkg/rpm.py +++ b/salt/utils/pkg/rpm.py @@ -53,8 +53,11 @@ def get_osarch(): stderr=subprocess.PIPE).communicate()[0] else: ret = ''.join(list(filter(None, platform.uname()[-2:]))[-1:]) - - return salt.utils.stringutils.to_str(ret).strip() or 'unknown' + ret = salt.utils.stringutils.to_str(ret).strip() or 'unknown' + ARCH_FIXES_MAPPING = { + "powerpc64le": "ppc64le" + } + return ARCH_FIXES_MAPPING.get(ret, ret) def check_32(arch, osarch=None): -- 2.20.1 ++++++ support-config-non-root-permission-issues-fixes-u-50.patch ++++++ >From 1113909fe9ab0509ebe439051238d6a4f95d3c54 Mon Sep 17 00:00:00 2001 From: Bo Maryniuk <b...@suse.de> Date: Wed, 17 Oct 2018 14:10:47 +0200 Subject: [PATCH] Support-config non-root permission issues fixes (U#50095) Do not crash if there is no configuration available at all Handle CLI and log errors Catch overwriting exiting archive error by other users Suppress excessive tracebacks on error log level --- salt/cli/support/collector.py | 39 ++++++++++++++++++++++++++++++++--- salt/utils/parsers.py | 2 +- 2 files changed, 37 insertions(+), 4 deletions(-) diff --git a/salt/cli/support/collector.py b/salt/cli/support/collector.py index 478d07e13b..a4343297b6 100644 --- a/salt/cli/support/collector.py +++ b/salt/cli/support/collector.py @@ -125,6 +125,31 @@ class SupportDataCollector(object): self.__current_section = [] self.__current_section_name = name + def _printout(self, data, output): + ''' + Use salt outputter to printout content. + + :return: + ''' + opts = {'extension_modules': '', 'color': False} + try: + printout = salt.output.get_printout(output, opts)(data) + if printout is not None: + return printout.rstrip() + except (KeyError, AttributeError, TypeError) as err: + log.debug(err, exc_info=True) + try: + printout = salt.output.get_printout('nested', opts)(data) + if printout is not None: + return printout.rstrip() + except (KeyError, AttributeError, TypeError) as err: + log.debug(err, exc_info=True) + printout = salt.output.get_printout('raw', opts)(data) + if printout is not None: + return printout.rstrip() + + return salt.output.try_printout(data, output, opts) + def write(self, title, data, output=None): ''' Add a data to the current opened section. @@ -138,7 +163,7 @@ class SupportDataCollector(object): try: if isinstance(data, dict) and 'return' in data: data = data['return'] - content = salt.output.try_printout(data, output, {'extension_modules': '', 'color': False}) + content = self._printout(data, output) except Exception: # Fall-back to just raw YAML content = None else: @@ -406,7 +431,11 @@ class SaltSupport(salt.utils.parsers.SaltSupportOptionParser): and self.config.get('support_archive') and os.path.exists(self.config['support_archive'])): self.out.warning('Terminated earlier, cleaning up') - os.unlink(self.config['support_archive']) + try: + os.unlink(self.config['support_archive']) + except Exception as err: + log.debug(err) + self.out.error('{} while cleaning up.'.format(err)) def _check_existing_archive(self): ''' @@ -418,7 +447,11 @@ class SaltSupport(salt.utils.parsers.SaltSupportOptionParser): if os.path.exists(self.config['support_archive']): if self.config['support_archive_force_overwrite']: self.out.warning('Overwriting existing archive: {}'.format(self.config['support_archive'])) - os.unlink(self.config['support_archive']) + try: + os.unlink(self.config['support_archive']) + except Exception as err: + log.debug(err) + self.out.error('{} while trying to overwrite existing archive.'.format(err)) ret = True else: self.out.warning('File {} already exists.'.format(self.config['support_archive'])) diff --git a/salt/utils/parsers.py b/salt/utils/parsers.py index 56a8961c3a..058346a9f4 100644 --- a/salt/utils/parsers.py +++ b/salt/utils/parsers.py @@ -1922,7 +1922,7 @@ class SaltSupportOptionParser(six.with_metaclass(OptionParserMeta, OptionParser, ''' _opts, _args = optparse.OptionParser.parse_args(self) configs = self.find_existing_configs(_opts.support_unit) - if cfg not in configs: + if configs and cfg not in configs: cfg = configs[0] return config.master_config(self.get_config_file_path(cfg)) -- 2.19.0 ++++++ support-use-of-gce-instance-credentials-109.patch ++++++ >From 4571116a54ff51683cb695ce795f04f8b318b440 Mon Sep 17 00:00:00 2001 From: jgleissner <jgleiss...@suse.com> Date: Wed, 19 Sep 2018 14:37:12 +0200 Subject: [PATCH] Support use of GCE instance credentials (#109) * Integration of MSI authentication with azurearm cloud driver (#105) * allow empty service_account_private_key in GCE driver Passing an emoty service_account_private_key to libcloud will enable authentication using instance credentials, which is used by CaaSP in GCE. --- salt/cloud/clouds/gce.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/salt/cloud/clouds/gce.py b/salt/cloud/clouds/gce.py index 75109491be..1018e36ed5 100644 --- a/salt/cloud/clouds/gce.py +++ b/salt/cloud/clouds/gce.py @@ -134,7 +134,8 @@ def __virtual__(): parameters = details['gce'] pathname = os.path.expanduser(parameters['service_account_private_key']) - if salt.utils.cloud.check_key_path_and_mode( + # empty pathname will tell libcloud to use instance credentials + if pathname and salt.utils.cloud.check_key_path_and_mode( provider, pathname ) is False: return False -- 2.17.1 ++++++ update-error-list-for-zypper.patch ++++++ >From 71e7ecfbb07cf14680a2a39de48a6e60cd20cb07 Mon Sep 17 00:00:00 2001 From: Bo Maryniuk <b...@suse.de> Date: Wed, 26 Sep 2018 17:54:53 +0200 Subject: [PATCH] Update error list for zypper Add error logging --- salt/modules/zypper.py | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-) diff --git a/salt/modules/zypper.py b/salt/modules/zypper.py index e4423cf1fc..6845e44ab6 100644 --- a/salt/modules/zypper.py +++ b/salt/modules/zypper.py @@ -75,7 +75,25 @@ class _Zypper(object): Allows serial zypper calls (first came, first won). ''' - SUCCESS_EXIT_CODES = [0, 100, 101, 102, 103] + SUCCESS_EXIT_CODES = { + 0: 'Successful run of zypper with no special info.', + 100: 'Patches are available for installation.', + 101: 'Security patches are available for installation.', + 102: 'Installation successful, reboot required.', + 103: 'Installation succesful, restart of the package manager itself required.', + } + + WARNING_EXIT_CODES = { + 6: 'No repositories are defined.', + 7: 'The ZYPP library is locked.', + 106: 'Some repository had to be disabled temporarily because it failed to refresh. ' + 'You should check your repository configuration (e.g. zypper ref -f).', + 107: 'Installation basically succeeded, but some of the packages %post install scripts returned an error. ' + 'These packages were successfully unpacked to disk and are registered in the rpm database, ' + 'but due to the failed install script they may not work as expected. The failed scripts output might ' + 'reveal what actually went wrong. Any scripts output is also logged to /var/log/zypp/history.' + } + LOCK_EXIT_CODE = 7 XML_DIRECTIVES = ['-x', '--xmlout'] ZYPPER_LOCK = '/var/run/zypp.pid' @@ -188,7 +206,15 @@ class _Zypper(object): :return: ''' - return self.exit_code not in self.SUCCESS_EXIT_CODES + if self.exit_code: + msg = self.SUCCESS_EXIT_CODES.get(self.exit_code) + if msg: + log.info(msg) + msg = self.WARNING_EXIT_CODES.get(self.exit_code) + if msg: + log.warning(msg) + + return self.exit_code not in self.SUCCESS_EXIT_CODES and self.exit_code not in self.WARNING_EXIT_CODES def _is_lock(self): ''' -- 2.19.0
