Script 'mail_helper' called by obssrc Hello community, here is the log from the commit of package salt for openSUSE:Factory checked in at 2022-07-12 11:12:13 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/salt (Old) and /work/SRC/openSUSE:Factory/.salt.new.1523 (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "salt" Tue Jul 12 11:12:13 2022 rev:130 rq:988366 version:3004 Changes: -------- --- /work/SRC/openSUSE:Factory/salt/salt.changes 2022-06-24 08:45:13.963133946 +0200 +++ /work/SRC/openSUSE:Factory/.salt.new.1523/salt.changes 2022-07-12 11:12:22.719693673 +0200 @@ -1,0 +2,41 @@ +Fri Jul 8 09:45:54 UTC 2022 - Pablo Su??rez Hern??ndez <[email protected]> + +- Add support for gpgautoimport in zypperpkg module +- Update Salt to work with Jinja >= and <= 3.1.0 (bsc#1198744) +- Fix salt.states.file.managed() for follow_symlinks=True and test=True (bsc#1199372) +- Make Salt 3004 compatible with pyzmq >= 23.0.0 (bsc#1201082) + +- Added: + * fix-jinja2-contextfuntion-base-on-version-bsc-119874.patch + * add-support-for-gpgautoimport-539.patch + * fix-62092-catch-zmq.error.zmqerror-to-set-hwm-for-zm.patch + * fix-salt.states.file.managed-for-follow_symlinks-tru.patch + +------------------------------------------------------------------- +Thu Jul 7 14:58:25 UTC 2022 - Pablo Su??rez Hern??ndez <[email protected]> + +- Add support for name, pkgs and diff_attr parameters to upgrade + function for zypper and yum (bsc#1198489) + +- Added: + * add-support-for-name-pkgs-and-diff_attr-parameters-t.patch + +------------------------------------------------------------------- +Tue Jun 28 07:40:48 UTC 2022 - Victor Zhestkov <[email protected]> + +- Fix ownership of salt thin directory when using the Salt Bundle +- Set default target for pip from VENV_PIP_TARGET environment variable +- Normalize package names once with pkg.installed/removed using yum (bsc#1195895) +- Save log to logfile with docker.build +- Use Salt Bundle in dockermod +- Ignore erros on reading license files with dpkg_lowpkg (bsc#1197288) + +- Added: + * normalize-package-names-once-with-pkg.installed-remo.patch + * use-salt-bundle-in-dockermod.patch + * fix-ownership-of-salt-thin-directory-when-using-the-.patch + * ignore-erros-on-reading-license-files-with-dpkg_lowp.patch + * set-default-target-for-pip-from-venv_pip_target-envi.patch + * save-log-to-logfile-with-docker.build.patch + +------------------------------------------------------------------- New: ---- add-support-for-gpgautoimport-539.patch add-support-for-name-pkgs-and-diff_attr-parameters-t.patch fix-62092-catch-zmq.error.zmqerror-to-set-hwm-for-zm.patch fix-jinja2-contextfuntion-base-on-version-bsc-119874.patch fix-ownership-of-salt-thin-directory-when-using-the-.patch fix-salt.states.file.managed-for-follow_symlinks-tru.patch ignore-erros-on-reading-license-files-with-dpkg_lowp.patch normalize-package-names-once-with-pkg.installed-remo.patch save-log-to-logfile-with-docker.build.patch set-default-target-for-pip-from-venv_pip_target-envi.patch use-salt-bundle-in-dockermod.patch ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ salt.spec ++++++ --- /var/tmp/diff_new_pack.v5BJIy/_old 2022-07-12 11:12:24.179695584 +0200 +++ /var/tmp/diff_new_pack.v5BJIy/_new 2022-07-12 11:12:24.183695589 +0200 @@ -308,7 +308,28 @@ Patch80: make-sure-saltcacheloader-use-correct-fileclient-519.patch # PATCH-FIX_UPSTREAM: https://github.com/saltstack/salt/commit/e068a34ccb2e17ae7224f8016a24b727f726d4c8 Patch81: fix-for-cve-2022-22967-bsc-1200566.patch - +# PATCH-FIX_UPSTREAM: https://github.com/saltstack/salt/pull/61827 +Patch82: ignore-erros-on-reading-license-files-with-dpkg_lowp.patch +# PATCH-FIX_UPSTREAM: https://github.com/saltstack/salt/pull/62109 +Patch83: use-salt-bundle-in-dockermod.patch +# PATCH-FIX_UPSTREAM: https://github.com/saltstack/salt/pull/61984 +Patch84: save-log-to-logfile-with-docker.build.patch +# PATCH-FIX_UPSTREAM: https://github.com/saltstack/salt/pull/62029 +Patch85: normalize-package-names-once-with-pkg.installed-remo.patch +# PATCH-FIX_UPSTREAM: https://github.com/saltstack/salt/pull/62089 +Patch86: set-default-target-for-pip-from-venv_pip_target-envi.patch +# PATCH-FIX_OPENSUSE: https://github.com/openSUSE/salt/pull/534 +Patch87: fix-ownership-of-salt-thin-directory-when-using-the-.patch +# PATCH-FIX_UPSTREAM: https://github.com/saltstack/salt/pull/62033 +Patch88: add-support-for-name-pkgs-and-diff_attr-parameters-t.patch +# PATCH-FIX_UPSTREAM: https://github.com/saltstack/salt/pull/62119 +Patch89: fix-62092-catch-zmq.error.zmqerror-to-set-hwm-for-zm.patch +# PATCH-FIX_UPSTREAM: https://github.com/saltstack/salt/pull/62067 +Patch90: fix-salt.states.file.managed-for-follow_symlinks-tru.patch +# PATCH-FIX_UPSTREAM: https://github.com/saltstack/salt/pull/61856 +Patch91: fix-jinja2-contextfuntion-base-on-version-bsc-119874.patch +# PATCH-FIX_UPSTREAM: https://github.com/saltstack/salt/pull/62209 +Patch92: add-support-for-gpgautoimport-539.patch BuildRoot: %{_tmppath}/%{name}-%{version}-build BuildRequires: logrotate ++++++ _lastrevision ++++++ --- /var/tmp/diff_new_pack.v5BJIy/_old 2022-07-12 11:12:24.239695663 +0200 +++ /var/tmp/diff_new_pack.v5BJIy/_new 2022-07-12 11:12:24.239695663 +0200 @@ -1,3 +1,3 @@ -f20138622e17e52fd49e531edd607b46d08a146c +e07459bfeea39239f6b446f40f6502e72dea488f (No newline at EOF) ++++++ add-support-for-gpgautoimport-539.patch ++++++ >From fbd5163bd0d5409a1823e9fb8e0cb623c22d6036 Mon Sep 17 00:00:00 2001 From: Michael Calmer <[email protected]> Date: Fri, 8 Jul 2022 10:15:37 +0200 Subject: [PATCH] add support for gpgautoimport (#539) * add support for gpgautoimport to refresh_db in the zypperpkg module * call refresh_db function from mod_repo * call refresh_db with kwargs where possible * ignore no repos defined exit code * fix zypperpkg test after adding more success return codes --- salt/modules/zypperpkg.py | 47 +++++++--- tests/unit/modules/test_zypperpkg.py | 124 +++++++++++++++++++++++---- 2 files changed, 140 insertions(+), 31 deletions(-) diff --git a/salt/modules/zypperpkg.py b/salt/modules/zypperpkg.py index 39d26f0e93..b622105e15 100644 --- a/salt/modules/zypperpkg.py +++ b/salt/modules/zypperpkg.py @@ -591,7 +591,7 @@ def list_upgrades(refresh=True, root=None, **kwargs): salt '*' pkg.list_upgrades """ if refresh: - refresh_db(root) + refresh_db(root, **kwargs) ret = dict() cmd = ["list-updates"] @@ -705,7 +705,7 @@ def info_available(*names, **kwargs): # Refresh db before extracting the latest package if kwargs.get("refresh", True): - refresh_db(root) + refresh_db(root, **kwargs) pkg_info = [] batch = names[:] @@ -1395,7 +1395,6 @@ def mod_repo(repo, **kwargs): cmd_opt.append("--name='{}'".format(kwargs.get("humanname"))) if kwargs.get("gpgautoimport") is True: - global_cmd_opt.append("--gpg-auto-import-keys") call_refresh = True if cmd_opt: @@ -1407,8 +1406,8 @@ def mod_repo(repo, **kwargs): # when used with "zypper ar --refresh" or "zypper mr --refresh" # --gpg-auto-import-keys is not doing anything # so we need to specifically refresh here with --gpg-auto-import-keys - refresh_opts = global_cmd_opt + ["refresh"] + [repo] - __zypper__(root=root).xml.call(*refresh_opts) + kwargs.update({"repos": repo}) + refresh_db(root=root, **kwargs) elif not added and not cmd_opt: comment = "Specified arguments did not result in modification of repo" @@ -1419,7 +1418,7 @@ def mod_repo(repo, **kwargs): return repo -def refresh_db(force=None, root=None): +def refresh_db(force=None, root=None, **kwargs): """ Trigger a repository refresh by calling ``zypper refresh``. Refresh will run with ``--force`` if the "force=True" flag is passed on the CLI or @@ -1430,6 +1429,17 @@ def refresh_db(force=None, root=None): {'<database name>': Bool} + gpgautoimport : False + If set to True, automatically trust and import public GPG key for + the repository. + + .. versionadded:: 3005 + + repos + Refresh just the specified repos + + .. versionadded:: 3005 + root operate on a different root directory. @@ -1450,11 +1460,22 @@ def refresh_db(force=None, root=None): salt.utils.pkg.clear_rtag(__opts__) ret = {} refresh_opts = ["refresh"] + global_opts = [] if force is None: force = __pillar__.get("zypper", {}).get("refreshdb_force", True) if force: refresh_opts.append("--force") - out = __zypper__(root=root).refreshable.call(*refresh_opts) + repos = kwargs.get("repos", []) + refresh_opts.extend([repos] if not isinstance(repos, list) else repos) + + if kwargs.get("gpgautoimport", False): + global_opts.append("--gpg-auto-import-keys") + + # We do the actual call to zypper refresh. + # We ignore retcode 6 which is returned when there are no repositories defined. + out = __zypper__(root=root).refreshable.call( + *global_opts, *refresh_opts, success_retcodes=[0, 6] + ) for line in out.splitlines(): if not line: @@ -1639,7 +1660,7 @@ def install( 'arch': '<new-arch>'}}} """ if refresh: - refresh_db(root) + refresh_db(root, **kwargs) try: pkg_params, pkg_type = __salt__["pkg_resource.parse_targets"]( @@ -1934,7 +1955,7 @@ def upgrade( cmd_update.insert(0, "--no-gpg-checks") if refresh: - refresh_db(root) + refresh_db(root, **kwargs) if dryrun: cmd_update.append("--dry-run") @@ -2844,7 +2865,7 @@ def search(criteria, refresh=False, **kwargs): root = kwargs.get("root", None) if refresh: - refresh_db(root) + refresh_db(root, **kwargs) cmd = ["search"] if kwargs.get("match") == "exact": @@ -2995,7 +3016,7 @@ def download(*packages, **kwargs): refresh = kwargs.get("refresh", False) if refresh: - refresh_db(root) + refresh_db(root, **kwargs) pkg_ret = {} for dld_result in ( @@ -3147,7 +3168,7 @@ def list_patches(refresh=False, root=None, **kwargs): salt '*' pkg.list_patches """ if refresh: - refresh_db(root) + refresh_db(root, **kwargs) return _get_patches(root=root) @@ -3241,7 +3262,7 @@ def resolve_capabilities(pkgs, refresh=False, root=None, **kwargs): salt '*' pkg.resolve_capabilities resolve_capabilities=True w3m_ssl """ if refresh: - refresh_db(root) + refresh_db(root, **kwargs) ret = list() for pkg in pkgs: diff --git a/tests/unit/modules/test_zypperpkg.py b/tests/unit/modules/test_zypperpkg.py index fea6eeb004..3f1560a385 100644 --- a/tests/unit/modules/test_zypperpkg.py +++ b/tests/unit/modules/test_zypperpkg.py @@ -358,7 +358,12 @@ class ZypperTestCase(TestCase, LoaderModuleMockMixin): run_out = {"stderr": "", "stdout": "\n".join(ref_out), "retcode": 0} zypper_mock = MagicMock(return_value=run_out) - call_kwargs = {"output_loglevel": "trace", "python_shell": False, "env": {}} + call_kwargs = { + "output_loglevel": "trace", + "python_shell": False, + "env": {}, + "success_retcodes": [0, 6], + } with patch.dict(zypper.__salt__, {"cmd.run_all": zypper_mock}): with patch.object(salt.utils.pkg, "clear_rtag", Mock()): result = zypper.refresh_db() @@ -376,6 +381,73 @@ class ZypperTestCase(TestCase, LoaderModuleMockMixin): zypper_mock.assert_called_with( ["zypper", "--non-interactive", "refresh", "--force"], **call_kwargs ) + zypper.refresh_db(gpgautoimport=True) + zypper_mock.assert_called_with( + [ + "zypper", + "--non-interactive", + "--gpg-auto-import-keys", + "refresh", + "--force", + ], + **call_kwargs + ) + zypper.refresh_db(gpgautoimport=True, force=True) + zypper_mock.assert_called_with( + [ + "zypper", + "--non-interactive", + "--gpg-auto-import-keys", + "refresh", + "--force", + ], + **call_kwargs + ) + zypper.refresh_db(gpgautoimport=True, force=False) + zypper_mock.assert_called_with( + [ + "zypper", + "--non-interactive", + "--gpg-auto-import-keys", + "refresh", + ], + **call_kwargs + ) + zypper.refresh_db( + gpgautoimport=True, + refresh=True, + repos="mock-repo-name", + root=None, + url="http://repo.url/some/path";, + ) + zypper_mock.assert_called_with( + [ + "zypper", + "--non-interactive", + "--gpg-auto-import-keys", + "refresh", + "--force", + "mock-repo-name", + ], + **call_kwargs + ) + zypper.refresh_db( + gpgautoimport=True, + repos="mock-repo-name", + root=None, + url="http://repo.url/some/path";, + ) + zypper_mock.assert_called_with( + [ + "zypper", + "--non-interactive", + "--gpg-auto-import-keys", + "refresh", + "--force", + "mock-repo-name", + ], + **call_kwargs + ) def test_info_installed(self): """ @@ -1555,18 +1627,23 @@ class ZypperTestCase(TestCase, LoaderModuleMockMixin): url = self.new_repo_config["url"] name = self.new_repo_config["name"] - with zypper_patcher: + with zypper_patcher, patch.object(zypper, "refresh_db", Mock()) as refreshmock: zypper.mod_repo(name, **{"url": url, "gpgautoimport": True}) self.assertEqual( zypper.__zypper__(root=None).xml.call.call_args_list, [ call("ar", url, name), - call("--gpg-auto-import-keys", "refresh", name), ], ) self.assertTrue( zypper.__zypper__(root=None).refreshable.xml.call.call_count == 0 ) + refreshmock.assert_called_once_with( + gpgautoimport=True, + repos=name, + root=None, + url="http://repo.url/some/path";, + ) def test_repo_noadd_nomod_ref(self): """ @@ -1585,15 +1662,17 @@ class ZypperTestCase(TestCase, LoaderModuleMockMixin): "salt.modules.zypperpkg", **self.zypper_patcher_config ) - with zypper_patcher: + with zypper_patcher, patch.object(zypper, "refresh_db", Mock()) as refreshmock: zypper.mod_repo(name, **{"url": url, "gpgautoimport": True}) - self.assertEqual( - zypper.__zypper__(root=None).xml.call.call_args_list, - [call("--gpg-auto-import-keys", "refresh", name)], - ) self.assertTrue( zypper.__zypper__(root=None).refreshable.xml.call.call_count == 0 ) + refreshmock.assert_called_once_with( + gpgautoimport=True, + repos=name, + root=None, + url="http://repo.url/some/path";, + ) def test_repo_add_mod_ref(self): """ @@ -1606,10 +1685,10 @@ class ZypperTestCase(TestCase, LoaderModuleMockMixin): zypper_patcher = patch.multiple( "salt.modules.zypperpkg", **self.zypper_patcher_config ) - url = self.new_repo_config["url"] name = self.new_repo_config["name"] - with zypper_patcher: + + with zypper_patcher, patch.object(zypper, "refresh_db", Mock()) as refreshmock: zypper.mod_repo( name, **{"url": url, "refresh": True, "gpgautoimport": True} ) @@ -1617,11 +1696,17 @@ class ZypperTestCase(TestCase, LoaderModuleMockMixin): zypper.__zypper__(root=None).xml.call.call_args_list, [ call("ar", url, name), - call("--gpg-auto-import-keys", "refresh", name), ], ) zypper.__zypper__(root=None).refreshable.xml.call.assert_called_once_with( - "--gpg-auto-import-keys", "mr", "--refresh", name + "mr", "--refresh", name + ) + refreshmock.assert_called_once_with( + gpgautoimport=True, + refresh=True, + repos=name, + root=None, + url="http://repo.url/some/path";, ) def test_repo_noadd_mod_ref(self): @@ -1641,16 +1726,19 @@ class ZypperTestCase(TestCase, LoaderModuleMockMixin): "salt.modules.zypperpkg", **self.zypper_patcher_config ) - with zypper_patcher: + with zypper_patcher, patch.object(zypper, "refresh_db", Mock()) as refreshmock: zypper.mod_repo( name, **{"url": url, "refresh": True, "gpgautoimport": True} ) - self.assertEqual( - zypper.__zypper__(root=None).xml.call.call_args_list, - [call("--gpg-auto-import-keys", "refresh", name)], - ) zypper.__zypper__(root=None).refreshable.xml.call.assert_called_once_with( - "--gpg-auto-import-keys", "mr", "--refresh", name + "mr", "--refresh", name + ) + refreshmock.assert_called_once_with( + gpgautoimport=True, + refresh=True, + repos=name, + root=None, + url="http://repo.url/some/path";, ) def test_wildcard_to_query_match_all(self): -- 2.36.1 ++++++ add-support-for-name-pkgs-and-diff_attr-parameters-t.patch ++++++ ++++ 1058 lines (skipped) ++++++ fix-62092-catch-zmq.error.zmqerror-to-set-hwm-for-zm.patch ++++++ >From df474d3cc0a5f02591fea093f9efc324c6feef46 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pablo=20Su=C3=A1rez=20Hern=C3=A1ndez?= <[email protected]> Date: Thu, 7 Jul 2022 11:38:09 +0100 Subject: [PATCH] Fix #62092: Catch zmq.error.ZMQError to set HWM for zmq >= 3 (#543) It looks like before release 23.0.0, when trying to access zmq.HWM it was raising ``AttributeError``, which is now wrapped under pyzmq's own ``zmq.error.ZMQError``. Simply caching that, should then set the HWM correctly for zmq >= 3 and therefore fix #62092. Co-authored-by: Mircea Ulinic <[email protected]> --- salt/transport/zeromq.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/salt/transport/zeromq.py b/salt/transport/zeromq.py index 9e61b23255..aa06298ee1 100644 --- a/salt/transport/zeromq.py +++ b/salt/transport/zeromq.py @@ -898,7 +898,7 @@ class ZeroMQPubServerChannel(salt.transport.server.PubServerChannel): try: pub_sock.setsockopt(zmq.HWM, self.opts.get("pub_hwm", 1000)) # in zmq >= 3.0, there are separate send and receive HWM settings - except AttributeError: + except (AttributeError, zmq.error.ZMQError): # Set the High Water Marks. For more information on HWM, see: # http://api.zeromq.org/4-1:zmq-setsockopt pub_sock.setsockopt(zmq.SNDHWM, self.opts.get("pub_hwm", 1000)) -- 2.36.1 ++++++ fix-jinja2-contextfuntion-base-on-version-bsc-119874.patch ++++++ >From 65494338f5a9bdaa0be27afab3da3a03a92d8cda Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pablo=20Su=C3=A1rez=20Hern=C3=A1ndez?= <[email protected]> Date: Fri, 8 Jul 2022 13:35:50 +0100 Subject: [PATCH] fix: jinja2 contextfuntion base on version (bsc#1198744) (#520) --- salt/utils/jinja.py | 16 ++++++++++++++-- tests/unit/utils/test_jinja.py | 8 +++++++- 2 files changed, 21 insertions(+), 3 deletions(-) diff --git a/salt/utils/jinja.py b/salt/utils/jinja.py index 0cb70bf64a..6b5b0d4e81 100644 --- a/salt/utils/jinja.py +++ b/salt/utils/jinja.py @@ -25,7 +25,7 @@ import salt.utils.json import salt.utils.stringutils import salt.utils.url import salt.utils.yaml -from jinja2 import BaseLoader, Markup, TemplateNotFound, nodes +from jinja2 import BaseLoader, TemplateNotFound, nodes from jinja2.environment import TemplateModule from jinja2.exceptions import TemplateRuntimeError from jinja2.ext import Extension @@ -34,6 +34,12 @@ from salt.utils.decorators.jinja import jinja_filter, jinja_global, jinja_test from salt.utils.odict import OrderedDict from salt.utils.versions import LooseVersion +try: + from markupsafe import Markup +except ImportError: + # jinja < 3.1 + from jinja2 import Markup + log = logging.getLogger(__name__) __all__ = ["SaltCacheLoader", "SerializerExtension"] @@ -706,7 +712,13 @@ def method_call(obj, f_name, *f_args, **f_kwargs): return getattr(obj, f_name, lambda *args, **kwargs: None)(*f_args, **f_kwargs) [email protected] +try: + contextfunction = jinja2.contextfunction +except AttributeError: + contextfunction = jinja2.pass_context + + +@contextfunction def show_full_context(ctx): return salt.utils.data.simple_types_filter( {key: value for key, value in ctx.items()} diff --git a/tests/unit/utils/test_jinja.py b/tests/unit/utils/test_jinja.py index 6502831aff..6bbcf9ef6f 100644 --- a/tests/unit/utils/test_jinja.py +++ b/tests/unit/utils/test_jinja.py @@ -22,7 +22,7 @@ import salt.utils.files import salt.utils.json import salt.utils.stringutils import salt.utils.yaml -from jinja2 import DictLoader, Environment, Markup, exceptions +from jinja2 import DictLoader, Environment, exceptions from salt.exceptions import SaltRenderError from salt.utils.decorators.jinja import JinjaFilter from salt.utils.jinja import ( @@ -46,6 +46,12 @@ try: except ImportError: HAS_TIMELIB = False +try: + from markupsafe import Markup +except ImportError: + # jinja < 3.1 + from jinja2 import Markup + BLINESEP = salt.utils.stringutils.to_bytes(os.linesep) -- 2.36.1 ++++++ fix-ownership-of-salt-thin-directory-when-using-the-.patch ++++++ >From 34a81d88db3862bcc03cdda4974e576723af7643 Mon Sep 17 00:00:00 2001 From: Victor Zhestkov <[email protected]> Date: Mon, 27 Jun 2022 18:03:49 +0300 Subject: [PATCH] Fix ownership of salt thin directory when using the Salt Bundle --- salt/client/ssh/ssh_py_shim.py | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/salt/client/ssh/ssh_py_shim.py b/salt/client/ssh/ssh_py_shim.py index 293ea1b7fa..95171f7aea 100644 --- a/salt/client/ssh/ssh_py_shim.py +++ b/salt/client/ssh/ssh_py_shim.py @@ -292,7 +292,30 @@ def main(argv): # pylint: disable=W0613 os.makedirs(OPTIONS.saltdir) cache_dir = os.path.join(OPTIONS.saltdir, "running_data", "var", "cache") os.makedirs(os.path.join(cache_dir, "salt")) - os.symlink("salt", os.path.relpath(os.path.join(cache_dir, "venv-salt-minion"))) + os.symlink( + "salt", os.path.relpath(os.path.join(cache_dir, "venv-salt-minion")) + ) + if os.path.exists(OPTIONS.saltdir) and ( + "SUDO_UID" in os.environ or "SUDO_GID" in os.environ + ): + try: + sudo_uid = int(os.environ.get("SUDO_UID", -1)) + except ValueError: + sudo_uid = -1 + try: + sudo_gid = int(os.environ.get("SUDO_GID", -1)) + except ValueError: + sudo_gid = -1 + dstat = os.stat(OPTIONS.saltdir) + if (sudo_uid != -1 and dstat.st_uid != sudo_uid) or ( + sudo_gid != -1 and dstat.st_gid != sudo_gid + ): + os.chown(OPTIONS.saltdir, sudo_uid, sudo_gid) + for dir_path, dir_names, file_names in os.walk(OPTIONS.saltdir): + for dir_name in dir_names: + os.lchown(os.path.join(dir_path, dir_name), sudo_uid, sudo_gid) + for file_name in file_names: + os.lchown(os.path.join(dir_path, file_name), sudo_uid, sudo_gid) if venv_salt_call is None: # Use Salt thin only if Salt Bundle (venv-salt-minion) is not available -- 2.36.1 ++++++ fix-salt.states.file.managed-for-follow_symlinks-tru.patch ++++++ >From 10705d922a11e5f2654d26e83e9f302862fafb18 Mon Sep 17 00:00:00 2001 From: Petr Pavlu <[email protected]> Date: Fri, 8 Jul 2022 10:11:52 +0200 Subject: [PATCH] Fix salt.states.file.managed() for follow_symlinks=True and test=True (bsc#1199372) (#535) When managing file /etc/test as follows: > file /etc/test: > file.managed: > - name: /etc/test > - source: salt://config/test > - mode: 644 > - follow_symlinks: True and with /etc/test being a symlink to a different file, an invocation of "salt-call '*' state.apply test=True" can report that the file should be updated even when a subsequent run of the same command without the test parameter makes no changes. The problem is that the test code path doesn't take correctly into account the follow_symlinks=True setting and ends up comparing permissions of the symlink instead of its target file. The patch addresses the problem by extending functions salt.modules.file.check_managed(), check_managed_changes() and check_file_meta() to have the follow_symlinks parameter which gets propagated to the salt.modules.file.stats() call and by updating salt.states.file.managed() to forward the same parameter to salt.modules.file.check_managed_changes(). Fixes #62066. [Cherry-picked from upstream commit 95bfbe31a2dc54723af3f1783d40de152760fe1a.] --- changelog/62066.fixed | 1 + salt/modules/file.py | 27 +++- salt/states/file.py | 1 + .../unit/modules/file/test_file_check.py | 144 ++++++++++++++++++ 4 files changed, 172 insertions(+), 1 deletion(-) create mode 100644 changelog/62066.fixed create mode 100644 tests/pytests/unit/modules/file/test_file_check.py diff --git a/changelog/62066.fixed b/changelog/62066.fixed new file mode 100644 index 0000000000..68216a03c1 --- /dev/null +++ b/changelog/62066.fixed @@ -0,0 +1 @@ +Fixed salt.states.file.managed() for follow_symlinks=True and test=True diff --git a/salt/modules/file.py b/salt/modules/file.py index 73619064ef..40c07455e3 100644 --- a/salt/modules/file.py +++ b/salt/modules/file.py @@ -5281,11 +5281,18 @@ def check_managed( serole=None, setype=None, serange=None, + follow_symlinks=False, **kwargs ): """ Check to see what changes need to be made for a file + follow_symlinks + If the desired path is a symlink, follow it and check the permissions + of the file to which the symlink points. + + .. versionadded:: 3005 + CLI Example: .. code-block:: bash @@ -5336,6 +5343,7 @@ def check_managed( serole=serole, setype=setype, serange=serange, + follow_symlinks=follow_symlinks, ) # Ignore permission for files written temporary directories # Files in any path will still be set correctly using get_managed() @@ -5372,6 +5380,7 @@ def check_managed_changes( setype=None, serange=None, verify_ssl=True, + follow_symlinks=False, **kwargs ): """ @@ -5387,6 +5396,12 @@ def check_managed_changes( .. versionadded:: 3002 + follow_symlinks + If the desired path is a symlink, follow it and check the permissions + of the file to which the symlink points. + + .. versionadded:: 3005 + CLI Example: .. code-block:: bash @@ -5456,6 +5471,7 @@ def check_managed_changes( serole=serole, setype=setype, serange=serange, + follow_symlinks=follow_symlinks, ) __clean_tmp(sfn) return changes @@ -5477,6 +5493,7 @@ def check_file_meta( setype=None, serange=None, verify_ssl=True, + follow_symlinks=False, ): """ Check for the changes in the file metadata. @@ -5553,6 +5570,12 @@ def check_file_meta( will not attempt to validate the servers certificate. Default is True. .. versionadded:: 3002 + + follow_symlinks + If the desired path is a symlink, follow it and check the permissions + of the file to which the symlink points. + + .. versionadded:: 3005 """ changes = {} if not source_sum: @@ -5560,7 +5583,9 @@ def check_file_meta( try: lstats = stats( - name, hash_type=source_sum.get("hash_type", None), follow_symlinks=False + name, + hash_type=source_sum.get("hash_type", None), + follow_symlinks=follow_symlinks, ) except CommandExecutionError: lstats = {} diff --git a/salt/states/file.py b/salt/states/file.py index 54e7decf86..a6288025e5 100644 --- a/salt/states/file.py +++ b/salt/states/file.py @@ -3038,6 +3038,7 @@ def managed( setype=setype, serange=serange, verify_ssl=verify_ssl, + follow_symlinks=follow_symlinks, **kwargs ) diff --git a/tests/pytests/unit/modules/file/test_file_check.py b/tests/pytests/unit/modules/file/test_file_check.py new file mode 100644 index 0000000000..bd0379ddae --- /dev/null +++ b/tests/pytests/unit/modules/file/test_file_check.py @@ -0,0 +1,144 @@ +import getpass +import logging +import os + +import pytest +import salt.modules.file as filemod +import salt.utils.files +import salt.utils.platform + +log = logging.getLogger(__name__) + + [email protected] +def configure_loader_modules(): + return {filemod: {"__context__": {}}} + + [email protected] +def tfile(tmp_path): + filename = str(tmp_path / "file-check-test-file") + + with salt.utils.files.fopen(filename, "w") as fp: + fp.write("Hi hello! I am a file.") + os.chmod(filename, 0o644) + + yield filename + + os.remove(filename) + + [email protected] +def a_link(tmp_path, tfile): + linkname = str(tmp_path / "a_link") + os.symlink(tfile, linkname) + + yield linkname + + os.remove(linkname) + + +def get_link_perms(): + if salt.utils.platform.is_linux(): + return "0777" + return "0755" + + [email protected]_on_windows(reason="os.symlink is not available on Windows") +def test_check_file_meta_follow_symlinks(a_link, tfile): + user = getpass.getuser() + lperms = get_link_perms() + + # follow_symlinks=False (default) + ret = filemod.check_file_meta( + a_link, tfile, None, None, user, None, lperms, None, None + ) + assert ret == {} + + ret = filemod.check_file_meta( + a_link, tfile, None, None, user, None, "0644", None, None + ) + assert ret == {"mode": "0644"} + + # follow_symlinks=True + ret = filemod.check_file_meta( + a_link, tfile, None, None, user, None, "0644", None, None, follow_symlinks=True + ) + assert ret == {} + + [email protected]_on_windows(reason="os.symlink is not available on Windows") +def test_check_managed_follow_symlinks(a_link, tfile): + user = getpass.getuser() + lperms = get_link_perms() + + # Function check_managed() ignores mode changes for files in the temp directory. + # Trick it to not recognize a_link as such. + a_link = "/" + a_link + + # follow_symlinks=False (default) + ret, comments = filemod.check_managed( + a_link, tfile, None, None, user, None, lperms, None, None, None, None, None + ) + assert ret is True + assert comments == "The file {} is in the correct state".format(a_link) + + ret, comments = filemod.check_managed( + a_link, tfile, None, None, user, None, "0644", None, None, None, None, None + ) + assert ret is None + assert comments == "The following values are set to be changed:\nmode: 0644\n" + + # follow_symlinks=True + ret, comments = filemod.check_managed( + a_link, + tfile, + None, + None, + user, + None, + "0644", + None, + None, + None, + None, + None, + follow_symlinks=True, + ) + assert ret is True + assert comments == "The file {} is in the correct state".format(a_link) + + [email protected]_on_windows(reason="os.symlink is not available on Windows") +def test_check_managed_changes_follow_symlinks(a_link, tfile): + user = getpass.getuser() + lperms = get_link_perms() + + # follow_symlinks=False (default) + ret = filemod.check_managed_changes( + a_link, tfile, None, None, user, None, lperms, None, None, None, None, None + ) + assert ret == {} + + ret = filemod.check_managed_changes( + a_link, tfile, None, None, user, None, "0644", None, None, None, None, None + ) + assert ret == {"mode": "0644"} + + # follow_symlinks=True + ret = filemod.check_managed_changes( + a_link, + tfile, + None, + None, + user, + None, + "0644", + None, + None, + None, + None, + None, + follow_symlinks=True, + ) + assert ret == {} -- 2.36.1 ++++++ ignore-erros-on-reading-license-files-with-dpkg_lowp.patch ++++++ >From 90d0e3ce40e46a9bff3e477b61e02cf3e87e8b9f Mon Sep 17 00:00:00 2001 From: Victor Zhestkov <[email protected]> Date: Mon, 27 Jun 2022 17:55:49 +0300 Subject: [PATCH] Ignore erros on reading license files with dpkg_lowpkg (bsc#1197288) * Ignore erros on reading license files with dpkg_lowpkg (bsc#1197288) * Add test for license reading with dpkg_lowpkg --- salt/modules/dpkg_lowpkg.py | 2 +- tests/pytests/unit/modules/test_dpkg_lowpkg.py | 18 ++++++++++++++++++ 2 files changed, 19 insertions(+), 1 deletion(-) create mode 100644 tests/pytests/unit/modules/test_dpkg_lowpkg.py diff --git a/salt/modules/dpkg_lowpkg.py b/salt/modules/dpkg_lowpkg.py index afbd619490..2c25b1fb2a 100644 --- a/salt/modules/dpkg_lowpkg.py +++ b/salt/modules/dpkg_lowpkg.py @@ -361,7 +361,7 @@ def _get_pkg_license(pkg): licenses = set() cpr = "/usr/share/doc/{}/copyright".format(pkg) if os.path.exists(cpr): - with salt.utils.files.fopen(cpr) as fp_: + with salt.utils.files.fopen(cpr, errors="ignore") as fp_: for line in salt.utils.stringutils.to_unicode(fp_.read()).split(os.linesep): if line.startswith("License:"): licenses.add(line.split(":", 1)[1].strip()) diff --git a/tests/pytests/unit/modules/test_dpkg_lowpkg.py b/tests/pytests/unit/modules/test_dpkg_lowpkg.py new file mode 100644 index 0000000000..1a89660c02 --- /dev/null +++ b/tests/pytests/unit/modules/test_dpkg_lowpkg.py @@ -0,0 +1,18 @@ +import os + +import salt.modules.dpkg_lowpkg as dpkg +from tests.support.mock import MagicMock, mock_open, patch + + +def test_get_pkg_license(): + """ + Test _get_pkg_license for ignore errors on reading license from copyright files + """ + license_read_mock = mock_open(read_data="") + with patch.object(os.path, "exists", MagicMock(return_value=True)), patch( + "salt.utils.files.fopen", license_read_mock + ): + dpkg._get_pkg_license("bash") + + assert license_read_mock.calls[0].args[0] == "/usr/share/doc/bash/copyright" + assert license_read_mock.calls[0].kwargs["errors"] == "ignore" -- 2.36.1 ++++++ normalize-package-names-once-with-pkg.installed-remo.patch ++++++ >From 09afcd0d04788ec4351c1c0b73a0c6eb3b0fd8c9 Mon Sep 17 00:00:00 2001 From: Victor Zhestkov <[email protected]> Date: Mon, 27 Jun 2022 18:01:21 +0300 Subject: [PATCH] Normalize package names once with pkg.installed/removed using yum (bsc#1195895) * Normalize the package names only once on install/remove * Add test for checking pkg.installed/removed with only normalisation * Fix split_arch conditions * Fix test_pkg --- salt/modules/yumpkg.py | 18 ++- salt/states/pkg.py | 3 +- tests/pytests/unit/states/test_pkg.py | 177 +++++++++++++++++++++++++- 3 files changed, 192 insertions(+), 6 deletions(-) diff --git a/salt/modules/yumpkg.py b/salt/modules/yumpkg.py index 9f8f548e5f..3138ac2e59 100644 --- a/salt/modules/yumpkg.py +++ b/salt/modules/yumpkg.py @@ -1449,7 +1449,12 @@ def install( try: pkg_params, pkg_type = __salt__["pkg_resource.parse_targets"]( - name, pkgs, sources, saltenv=saltenv, normalize=normalize, **kwargs + name, + pkgs, + sources, + saltenv=saltenv, + normalize=normalize and kwargs.get("split_arch", True), + **kwargs ) except MinionError as exc: raise CommandExecutionError(exc) @@ -1603,7 +1608,10 @@ def install( except ValueError: pass else: - if archpart in salt.utils.pkg.rpm.ARCHES: + if archpart in salt.utils.pkg.rpm.ARCHES and ( + archpart != __grains__["osarch"] + or kwargs.get("split_arch", True) + ): arch = "." + archpart pkgname = namepart @@ -2134,11 +2142,13 @@ def remove(name=None, pkgs=None, **kwargs): # pylint: disable=W0613 arch = "" pkgname = target try: - namepart, archpart = target.rsplit(".", 1) + namepart, archpart = pkgname.rsplit(".", 1) except ValueError: pass else: - if archpart in salt.utils.pkg.rpm.ARCHES: + if archpart in salt.utils.pkg.rpm.ARCHES and ( + archpart != __grains__["osarch"] or kwargs.get("split_arch", True) + ): arch = "." + archpart pkgname = namepart # Since we don't always have the arch info, epoch information has to parsed out. But diff --git a/salt/states/pkg.py b/salt/states/pkg.py index 0d601e1aaf..71298e6c7a 100644 --- a/salt/states/pkg.py +++ b/salt/states/pkg.py @@ -1901,6 +1901,7 @@ def installed( normalize=normalize, update_holds=update_holds, ignore_epoch=ignore_epoch, + split_arch=False, **kwargs ) except CommandExecutionError as exc: @@ -2973,7 +2974,7 @@ def _uninstall( } changes = __salt__["pkg.{}".format(action)]( - name, pkgs=pkgs, version=version, **kwargs + name, pkgs=pkgs, version=version, split_arch=False, **kwargs ) new = __salt__["pkg.list_pkgs"](versions_as_list=True, **kwargs) failed = [] diff --git a/tests/pytests/unit/states/test_pkg.py b/tests/pytests/unit/states/test_pkg.py index 17b91bcb39..10acae9f88 100644 --- a/tests/pytests/unit/states/test_pkg.py +++ b/tests/pytests/unit/states/test_pkg.py @@ -2,6 +2,8 @@ import logging import pytest import salt.modules.beacons as beaconmod +import salt.modules.pkg_resource as pkg_resource +import salt.modules.yumpkg as yumpkg import salt.states.beacon as beaconstate import salt.states.pkg as pkg import salt.utils.state as state_utils @@ -17,7 +19,7 @@ def configure_loader_modules(): pkg: { "__env__": "base", "__salt__": {}, - "__grains__": {"os": "CentOS"}, + "__grains__": {"os": "CentOS", "os_family": "RedHat"}, "__opts__": {"test": False, "cachedir": ""}, "__instance_id__": "", "__low__": {}, @@ -25,6 +27,15 @@ def configure_loader_modules(): }, beaconstate: {"__salt__": {}, "__opts__": {}}, beaconmod: {"__salt__": {}, "__opts__": {}}, + pkg_resource: { + "__salt__": {}, + "__grains__": {"os": "CentOS", "os_family": "RedHat"}, + }, + yumpkg: { + "__salt__": {}, + "__grains__": {"osarch": "x86_64", "osmajorrelease": 7}, + "__opts__": {}, + }, } @@ -715,3 +726,167 @@ def test_held_unheld(package_manager): hold_mock.assert_not_called() unhold_mock.assert_any_call(name="held-test", pkgs=["baz"]) unhold_mock.assert_any_call(name="held-test", pkgs=["bar"]) + + +def test_installed_with_single_normalize(): + """ + Test pkg.installed with preventing multiple package name normalisation + """ + + list_no_weird_installed = { + "pkga": "1.0.1", + "pkgb": "1.0.2", + "pkgc": "1.0.3", + } + list_no_weird_installed_ver_list = { + "pkga": ["1.0.1"], + "pkgb": ["1.0.2"], + "pkgc": ["1.0.3"], + } + list_with_weird_installed = { + "pkga": "1.0.1", + "pkgb": "1.0.2", + "pkgc": "1.0.3", + "weird-name-1.2.3-1234.5.6.test7tst.x86_64": "20220214-2.1", + } + list_with_weird_installed_ver_list = { + "pkga": ["1.0.1"], + "pkgb": ["1.0.2"], + "pkgc": ["1.0.3"], + "weird-name-1.2.3-1234.5.6.test7tst.x86_64": ["20220214-2.1"], + } + list_pkgs = MagicMock( + side_effect=[ + # For the package with version specified + list_no_weird_installed_ver_list, + {}, + list_no_weird_installed, + list_no_weird_installed_ver_list, + list_with_weird_installed, + list_with_weird_installed_ver_list, + # For the package with no version specified + list_no_weird_installed_ver_list, + {}, + list_no_weird_installed, + list_no_weird_installed_ver_list, + list_with_weird_installed, + list_with_weird_installed_ver_list, + ] + ) + + salt_dict = { + "pkg.install": yumpkg.install, + "pkg.list_pkgs": list_pkgs, + "pkg.normalize_name": yumpkg.normalize_name, + "pkg_resource.version_clean": pkg_resource.version_clean, + "pkg_resource.parse_targets": pkg_resource.parse_targets, + } + + with patch("salt.modules.yumpkg.list_pkgs", list_pkgs), patch( + "salt.modules.yumpkg.version_cmp", MagicMock(return_value=0) + ), patch( + "salt.modules.yumpkg._call_yum", MagicMock(return_value={"retcode": 0}) + ) as call_yum_mock, patch.dict( + pkg.__salt__, salt_dict + ), patch.dict( + pkg_resource.__salt__, salt_dict + ), patch.dict( + yumpkg.__salt__, salt_dict + ), patch.dict( + yumpkg.__grains__, {"os": "CentOS", "osarch": "x86_64", "osmajorrelease": 7} + ), patch.object( + yumpkg, "list_holds", MagicMock() + ): + + expected = { + "weird-name-1.2.3-1234.5.6.test7tst.x86_64": { + "old": "", + "new": "20220214-2.1", + } + } + ret = pkg.installed( + "test_install", + pkgs=[{"weird-name-1.2.3-1234.5.6.test7tst.x86_64.noarch": "20220214-2.1"}], + ) + call_yum_mock.assert_called_once() + assert ( + call_yum_mock.mock_calls[0].args[0][2] + == "weird-name-1.2.3-1234.5.6.test7tst.x86_64-20220214-2.1" + ) + assert ret["result"] + assert ret["changes"] == expected + + +def test_removed_with_single_normalize(): + """ + Test pkg.removed with preventing multiple package name normalisation + """ + + list_no_weird_installed = { + "pkga": "1.0.1", + "pkgb": "1.0.2", + "pkgc": "1.0.3", + } + list_no_weird_installed_ver_list = { + "pkga": ["1.0.1"], + "pkgb": ["1.0.2"], + "pkgc": ["1.0.3"], + } + list_with_weird_installed = { + "pkga": "1.0.1", + "pkgb": "1.0.2", + "pkgc": "1.0.3", + "weird-name-1.2.3-1234.5.6.test7tst.x86_64": "20220214-2.1", + } + list_with_weird_installed_ver_list = { + "pkga": ["1.0.1"], + "pkgb": ["1.0.2"], + "pkgc": ["1.0.3"], + "weird-name-1.2.3-1234.5.6.test7tst.x86_64": ["20220214-2.1"], + } + list_pkgs = MagicMock( + side_effect=[ + list_with_weird_installed_ver_list, + list_with_weird_installed, + list_no_weird_installed, + list_no_weird_installed_ver_list, + ] + ) + + salt_dict = { + "pkg.remove": yumpkg.remove, + "pkg.list_pkgs": list_pkgs, + "pkg.normalize_name": yumpkg.normalize_name, + "pkg_resource.parse_targets": pkg_resource.parse_targets, + "pkg_resource.version_clean": pkg_resource.version_clean, + } + + with patch("salt.modules.yumpkg.list_pkgs", list_pkgs), patch( + "salt.modules.yumpkg.version_cmp", MagicMock(return_value=0) + ), patch( + "salt.modules.yumpkg._call_yum", MagicMock(return_value={"retcode": 0}) + ) as call_yum_mock, patch.dict( + pkg.__salt__, salt_dict + ), patch.dict( + pkg_resource.__salt__, salt_dict + ), patch.dict( + yumpkg.__salt__, salt_dict + ): + + expected = { + "weird-name-1.2.3-1234.5.6.test7tst.x86_64": { + "old": "20220214-2.1", + "new": "", + } + } + ret = pkg.removed( + "test_remove", + pkgs=[{"weird-name-1.2.3-1234.5.6.test7tst.x86_64.noarch": "20220214-2.1"}], + ) + call_yum_mock.assert_called_once() + assert ( + call_yum_mock.mock_calls[0].args[0][2] + == "weird-name-1.2.3-1234.5.6.test7tst.x86_64-20220214-2.1" + ) + assert ret["result"] + assert ret["changes"] == expected -- 2.36.1 ++++++ save-log-to-logfile-with-docker.build.patch ++++++ >From c70db2e50599339118c9bf00c69f5cd38ef220bb Mon Sep 17 00:00:00 2001 From: Vladimir Nadvornik <[email protected]> Date: Mon, 27 Jun 2022 17:00:58 +0200 Subject: [PATCH] Save log to logfile with docker.build --- salt/modules/dockermod.py | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/salt/modules/dockermod.py b/salt/modules/dockermod.py index e6b81e7f09..1f871b40cf 100644 --- a/salt/modules/dockermod.py +++ b/salt/modules/dockermod.py @@ -3990,6 +3990,7 @@ def build( fileobj=None, dockerfile=None, buildargs=None, + logfile=None, ): """ .. versionchanged:: 2018.3.0 @@ -4043,6 +4044,9 @@ def build( buildargs A dictionary of build arguments provided to the docker build process. + logfile + Path to log file. Output from build is written to this file if not None. + **RETURN DATA** @@ -4117,6 +4121,20 @@ def build( stream_data = [] for line in response: stream_data.extend(salt.utils.json.loads(line, cls=DockerJSONDecoder)) + + if logfile: + try: + with salt.utils.files.fopen(logfile, "a") as f: + for item in stream_data: + try: + item_type = next(iter(item)) + except StopIteration: + continue + if item_type == "stream": + f.write(item[item_type]) + except OSError: + log.error("Unable to write logfile '%s'", logfile) + errors = [] # Iterate through API response and collect information for item in stream_data: -- 2.36.1 ++++++ set-default-target-for-pip-from-venv_pip_target-envi.patch ++++++ ++++ 1862 lines (skipped) ++++++ use-salt-bundle-in-dockermod.patch ++++++ >From ed53e3cbd62352b8d2af4d4b36c03e40981263bb Mon Sep 17 00:00:00 2001 From: Victor Zhestkov <[email protected]> Date: Mon, 27 Jun 2022 17:59:24 +0300 Subject: [PATCH] Use Salt Bundle in dockermod * Use Salt Bundle for salt calls in dockermod * Add test of performing a call with the Salt Bundle --- salt/modules/dockermod.py | 197 +++++++++++++++--- .../unit/modules/dockermod/test_module.py | 78 ++++++- 2 files changed, 241 insertions(+), 34 deletions(-) diff --git a/salt/modules/dockermod.py b/salt/modules/dockermod.py index fdded88dbb..e6b81e7f09 100644 --- a/salt/modules/dockermod.py +++ b/salt/modules/dockermod.py @@ -201,14 +201,19 @@ import copy import fnmatch import functools import gzip +import hashlib import json import logging import os +import pathlib import pipes import re import shutil import string import subprocess +import sys +import tarfile +import tempfile import time import uuid @@ -6682,6 +6687,111 @@ def _compile_state(sls_opts, mods=None): return st_.state.compile_high_data(high_data) +def gen_venv_tar(cachedir, venv_dest_dir, venv_name): + """ + Generate tarball with the Salt Bundle if required and return the path to it + """ + exec_path = pathlib.Path(sys.executable).parts + venv_dir_name = "venv-salt-minion" + if venv_dir_name not in exec_path: + return None + + venv_tar = os.path.join(cachedir, "venv-salt.tgz") + venv_hash = os.path.join(cachedir, "venv-salt.hash") + venv_lock = os.path.join(cachedir, ".venv-salt.lock") + + venv_path = os.path.join(*exec_path[0 : exec_path.index(venv_dir_name)]) + + with __utils__["files.flopen"](venv_lock, "w"): + start_dir = os.getcwd() + venv_hash_file = os.path.join(venv_path, venv_dir_name, "venv-hash.txt") + try: + with __utils__["files.fopen"](venv_hash_file, "r") as fh: + venv_hash_src = fh.readline().strip() + except Exception: # pylint: disable=broad-except + # It makes no sense what caused the exception + # Just calculate the hash different way + for cmd in ("rpm -qi venv-salt-minion", "dpkg -s venv-salt-minion"): + ret = __salt__["cmd.run_all"]( + cmd, + python_shell=True, + clean_env=True, + env={"LANG": "C", "LANGUAGE": "C", "LC_ALL": "C"}, + ) + if ret.get("retcode") == 0 and ret.get("stdout"): + venv_hash_src = hashlib.sha256( + "{}\n".format(ret.get("stdout")).encode() + ).hexdigest() + break + try: + with __utils__["files.fopen"](venv_hash, "r") as fh: + venv_hash_dest = fh.readline().strip() + except Exception: # pylint: disable=broad-except + # It makes no sense what caused the exception + # Set the hash to impossible value to force new tarball creation + venv_hash_dest = "UNKNOWN" + if venv_hash_src == venv_hash_dest and os.path.isfile(venv_tar): + return venv_tar + try: + tfd, tmp_venv_tar = tempfile.mkstemp( + dir=cachedir, + prefix=".venv-", + suffix=os.path.splitext(venv_tar)[1], + ) + os.close(tfd) + + os.chdir(venv_path) + tfp = tarfile.open(tmp_venv_tar, "w:gz") + + for root, dirs, files in salt.utils.path.os_walk( + venv_dir_name, followlinks=True + ): + for name in files: + if name == "python" and pathlib.Path(root).parts == ( + venv_dir_name, + "bin", + ): + tfd, tmp_python_file = tempfile.mkstemp( + dir=cachedir, + prefix=".python-", + ) + os.close(tfd) + try: + with __utils__["files.fopen"]( + os.path.join(root, name), "r" + ) as fh_in: + with __utils__["files.fopen"]( + tmp_python_file, "w" + ) as fh_out: + rd_lines = fh_in.readlines() + rd_lines = [ + 'export VIRTUAL_ENV="{}"\n'.format( + os.path.join(venv_dest_dir, venv_name) + ) + if line.startswith("export VIRTUAL_ENV=") + else line + for line in rd_lines + ] + fh_out.write("".join(rd_lines)) + os.chmod(tmp_python_file, 0o755) + tfp.add(tmp_python_file, arcname=os.path.join(root, name)) + continue + finally: + if os.path.isfile(tmp_python_file): + os.remove(tmp_python_file) + if not name.endswith((".pyc", ".pyo")): + tfp.add(os.path.join(root, name)) + + tfp.close() + shutil.move(tmp_venv_tar, venv_tar) + with __utils__["files.fopen"](venv_hash, "w") as fh: + fh.write("{}\n".format(venv_hash_src)) + finally: + os.chdir(start_dir) + + return venv_tar + + def call(name, function, *args, **kwargs): """ Executes a Salt function inside a running container @@ -6717,47 +6827,68 @@ def call(name, function, *args, **kwargs): if function is None: raise CommandExecutionError("Missing function parameter") - # move salt into the container - thin_path = __utils__["thin.gen_thin"]( - __opts__["cachedir"], - extra_mods=__salt__["config.option"]("thin_extra_mods", ""), - so_mods=__salt__["config.option"]("thin_so_mods", ""), - ) - ret = copy_to( - name, thin_path, os.path.join(thin_dest_path, os.path.basename(thin_path)) - ) + venv_dest_path = "/var/tmp" + venv_name = "venv-salt-minion" + venv_tar = gen_venv_tar(__opts__["cachedir"], venv_dest_path, venv_name) - # figure out available python interpreter inside the container (only Python3) - pycmds = ("python3", "/usr/libexec/platform-python") - container_python_bin = None - for py_cmd in pycmds: - cmd = [py_cmd] + ["--version"] - ret = run_all(name, subprocess.list2cmdline(cmd)) - if ret["retcode"] == 0: - container_python_bin = py_cmd - break - if not container_python_bin: - raise CommandExecutionError( - "Python interpreter cannot be found inside the container. Make sure Python is installed in the container" + if venv_tar is not None: + venv_python_bin = os.path.join(venv_dest_path, venv_name, "bin", "python") + dest_venv_tar = os.path.join(venv_dest_path, os.path.basename(venv_tar)) + copy_to(name, venv_tar, dest_venv_tar, overwrite=True, makedirs=True) + run_all( + name, + subprocess.list2cmdline( + ["tar", "zxf", dest_venv_tar, "-C", venv_dest_path] + ), + ) + run_all(name, subprocess.list2cmdline(["rm", "-f", dest_venv_tar])) + container_python_bin = venv_python_bin + thin_dest_path = os.path.join(venv_dest_path, venv_name) + thin_salt_call = os.path.join(thin_dest_path, "bin", "salt-call") + else: + # move salt into the container + thin_path = __utils__["thin.gen_thin"]( + __opts__["cachedir"], + extra_mods=__salt__["config.option"]("thin_extra_mods", ""), + so_mods=__salt__["config.option"]("thin_so_mods", ""), ) - # untar archive - untar_cmd = [ - container_python_bin, - "-c", - 'import tarfile; tarfile.open("{0}/{1}").extractall(path="{0}")'.format( - thin_dest_path, os.path.basename(thin_path) - ), - ] - ret = run_all(name, subprocess.list2cmdline(untar_cmd)) - if ret["retcode"] != 0: - return {"result": False, "comment": ret["stderr"]} + ret = copy_to( + name, thin_path, os.path.join(thin_dest_path, os.path.basename(thin_path)) + ) + + # figure out available python interpreter inside the container (only Python3) + pycmds = ("python3", "/usr/libexec/platform-python") + container_python_bin = None + for py_cmd in pycmds: + cmd = [py_cmd] + ["--version"] + ret = run_all(name, subprocess.list2cmdline(cmd)) + if ret["retcode"] == 0: + container_python_bin = py_cmd + break + if not container_python_bin: + raise CommandExecutionError( + "Python interpreter cannot be found inside the container. Make sure Python is installed in the container" + ) + + # untar archive + untar_cmd = [ + container_python_bin, + "-c", + 'import tarfile; tarfile.open("{0}/{1}").extractall(path="{0}")'.format( + thin_dest_path, os.path.basename(thin_path) + ), + ] + ret = run_all(name, subprocess.list2cmdline(untar_cmd)) + if ret["retcode"] != 0: + return {"result": False, "comment": ret["stderr"]} + thin_salt_call = os.path.join(thin_dest_path, "salt-call") try: salt_argv = ( [ container_python_bin, - os.path.join(thin_dest_path, "salt-call"), + thin_salt_call, "--metadata", "--local", "--log-file", diff --git a/tests/pytests/unit/modules/dockermod/test_module.py b/tests/pytests/unit/modules/dockermod/test_module.py index 47fe5d55e6..19c7f450d7 100644 --- a/tests/pytests/unit/modules/dockermod/test_module.py +++ b/tests/pytests/unit/modules/dockermod/test_module.py @@ -3,6 +3,7 @@ Unit tests for the docker module """ import logging +import sys import pytest import salt.config @@ -26,6 +27,7 @@ def configure_loader_modules(): whitelist=[ "args", "docker", + "files", "json", "state", "thin", @@ -880,13 +882,16 @@ def test_call_success(): client = Mock() client.put_archive = Mock() get_client_mock = MagicMock(return_value=client) + gen_venv_tar_mock = MagicMock(return_value=None) context = {"docker.exec_driver": "docker-exec"} salt_dunder = {"config.option": docker_config_mock} with patch.object(docker_mod, "run_all", docker_run_all_mock), patch.object( docker_mod, "copy_to", docker_copy_to_mock - ), patch.object(docker_mod, "_get_client", get_client_mock), patch.dict( + ), patch.object(docker_mod, "_get_client", get_client_mock), patch.object( + docker_mod, "gen_venv_tar", gen_venv_tar_mock + ), patch.dict( docker_mod.__opts__, {"cachedir": "/tmp"} ), patch.dict( docker_mod.__salt__, salt_dunder @@ -931,6 +936,11 @@ def test_call_success(): != docker_run_all_mock.mock_calls[9][1][1] ) + # check the parameters of gen_venv_tar call + assert gen_venv_tar_mock.mock_calls[0][1][0] == "/tmp" + assert gen_venv_tar_mock.mock_calls[0][1][1] == "/var/tmp" + assert gen_venv_tar_mock.mock_calls[0][1][2] == "venv-salt-minion" + assert {"retcode": 0, "comment": "container cmd"} == ret @@ -1352,3 +1362,69 @@ def test_port(): "bar": {"6666/tcp": ports["bar"]["6666/tcp"]}, "baz": {}, } + + [email protected]_test +def test_call_with_gen_venv_tar(): + """ + test module calling inside containers with the Salt Bundle + """ + ret = None + docker_run_all_mock = MagicMock( + return_value={ + "retcode": 0, + "stdout": '{"retcode": 0, "comment": "container cmd"}', + "stderr": "err", + } + ) + docker_copy_to_mock = MagicMock(return_value={"retcode": 0}) + docker_config_mock = MagicMock(return_value="") + docker_cmd_run_mock = MagicMock( + return_value={ + "retcode": 0, + "stdout": "test", + } + ) + client = Mock() + client.put_archive = Mock() + get_client_mock = MagicMock(return_value=client) + + context = {"docker.exec_driver": "docker-exec"} + salt_dunder = { + "config.option": docker_config_mock, + "cmd.run_all": docker_cmd_run_mock, + } + + with patch.object(docker_mod, "run_all", docker_run_all_mock), patch.object( + docker_mod, "copy_to", docker_copy_to_mock + ), patch.object(docker_mod, "_get_client", get_client_mock), patch.object( + sys, "executable", "/tmp/venv-salt-minion/bin/python" + ), patch.dict( + docker_mod.__opts__, {"cachedir": "/tmp"} + ), patch.dict( + docker_mod.__salt__, salt_dunder + ), patch.dict( + docker_mod.__context__, context + ): + ret = docker_mod.call("ID", "test.arg", 1, 2, arg1="val1") + + # Check that the directory is different each time + # [ call(name, [args]), ... + assert "mkdir" in docker_run_all_mock.mock_calls[0][1][1] + + assert ( + "tar zxf /var/tmp/venv-salt.tgz -C /var/tmp" + == docker_run_all_mock.mock_calls[1][1][1] + ) + + assert docker_run_all_mock.mock_calls[3][1][1].startswith( + "/var/tmp/venv-salt-minion/bin/python /var/tmp/venv-salt-minion/bin/salt-call " + ) + + # check remove the salt bundle tarball + assert docker_run_all_mock.mock_calls[2][1][1] == "rm -f /var/tmp/venv-salt.tgz" + + # check directory cleanup + assert docker_run_all_mock.mock_calls[4][1][1] == "rm -rf /var/tmp/venv-salt-minion" + + assert {"retcode": 0, "comment": "container cmd"} == ret -- 2.36.1
