** Description changed: [ Impact ] When using cloud-init to add a local user with a specific password/hash to the system at first boot, and sssd-tools also happens to be installed but sssd itself is not configured yet, a sequence of events leads to that user having no password set: - the user is created with the specified password hash just fine - cloud-init calls[1] "passwd -u" on the user - that passwd call ends up also calling "sss_cache -UG"[2], to reset the sssd cache, since the src:shadow package (where bin:passwd comes from) is built with sssd support[3] by default. If sss_cache is not available (bin:sssd-tools not installed), the bug does not manifest itself. - calling "sss_cache -UG" on a system with no valid sssd configuration yet prints an error message to stderr about a missing config.ldb, but exits with 0. This is since commits [4][5]. Those commits also changed sss_cache so that it no longer generates the config.ldb database, hence why it now always complains. - cloud-init checks exit status, and stderr, of the "passwd -u" call. It sees that stderr has content[6], and assumes a failure, proceeding to calling "passwd -d" on that account. This is what deletes the password hash from the user. For this SRU, I had originally opted to silence the error from the caller's side (src:shadow) if there is no config.ldb database present. That call is a best effort attempt to be helpful and clear the sssd cache, and the result of the call is being ignored anyway. But that has a much wider impact: src:shadow is installed everywhere. Even users not affected by this bug would get the update. Fedora (which is the sssd upstream) opted to entirely disable[7] sssd support in src:shadow. That is an option to consider for devel releases of ubuntu, but not an SRU. I then decided to silence the error from within sss_cache instead. It's the same fix in principle, and only users affected by the bug will receive the update. The one small change in behavior is that now when sss_cache doesn't find the config.ldb database, it's silent, so users might not be immediately aware that no action was taken by the tool. Note, however, that the exit code remains unchanged. And if you think about the end result of the tool run, it's the same: the cache was either expunged, or there was nothing to expunge. 1. https://github.com/canonical/cloud-init/blob/6193b2a0d241bed305e28a7aced1bd5c8c09de9a/cloudinit/distros/__init__.py#L1059 2. https://git.launchpad.net/ubuntu/+source/shadow/tree/src/passwd.c?h=applied/ubuntu/devel#n1175 3. https://git.launchpad.net/ubuntu/+source/shadow/tree/configure.ac?h=applied/ubuntu/devel#n202 4. https://github.com/SSSD/sssd/commit/a008accecd6d0b35e8d57d738ee3d05863aa7d0f 5. https://github.com/SSSD/sssd/commit/50b4579413c1146a2fced5ced19977e13790731c 6. https://github.com/canonical/cloud-init/blob/6193b2a0d241bed305e28a7aced1bd5c8c09de9a/cloudinit/distros/__init__.py#L1063 7. https://src.fedoraproject.org/rpms/shadow-utils/c/1820858d26f633482d3679d1c97324e25e3c1df8?branch=rawhide [ Test Plan ] There is a script attached to this bug which will reproduce the issue. It assumes a working lxd setup, and that "lxc launch ubuntu- daily:<ubuntu-release-name>" produces a working container. The script will exercise the following aspects of this bug: - provide the necessary cloud-init user-data to trigger the bug - check that the user has a password set or not in the end - check that "passwd -u" produces or not the stderr output that triggers the bug - check that "sss_cache -UG" is *always* silent, i.e., no stderr noise about the missing config.ldb To run the test, download the script to some directory and make it executable: wget https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/2153951/+attachment/6011062/+files/lp2153951-test.sh chmod +x lp2153951-test.sh Then run it targeting the ubuntu release under test. For example, for resolute: ./lp2153951-test.sh --release resolute That should reproduce the error, i.e., the script must indicate the test as having failed. To test the fixed packages, pass the "--pocket proposed" parameter. Again with the resolute example: ./lp2153951-test.sh --release resolute --pocket proposed This time the script should indicate that all checks have passed, and that completes the verification for this SRU. Here is run that shows the bug being reproduced: $ ./lp2153951-test.sh --release stonking Launching stonking container lp2153951-192830 (phase 1: repositories + packages) Launching lp2153951-192830 .....................................................status: done Resetting cloud-init state and rebooting (phase 2: user configuration) status: done === apt-cache policy passwd (package source used in this test) === passwd: - Installed: 1:4.19.3-2ubuntu2 - Candidate: 1:4.19.3-2ubuntu2 - Version table: - *** 1:4.19.3-2ubuntu2 500 - 500 http://archive.ubuntu.com/ubuntu stonking/main amd64 Packages - 100 /var/lib/dpkg/status + Installed: 1:4.19.3-2ubuntu2 + Candidate: 1:4.19.3-2ubuntu2 + Version table: + *** 1:4.19.3-2ubuntu2 500 + 500 http://archive.ubuntu.com/ubuntu stonking/main amd64 Packages + 100 /var/lib/dpkg/status === Checks === FAIL: tuser has no hashed password in /etc/shadow (got: '<empty>') FAIL: 'passwd -u tuser' complained about the SSSD config db on stderr: - [sss_cache] [sss_tool_confdb_init] (0x0010): Can't access '/var/lib/sss/db/config.ldb', probably SSSD isn't configured - Can't find configuration db, was SSSD configured and run? - [sss_cache] [sss_tool_confdb_init] (0x0010): Can't access '/var/lib/sss/db/config.ldb', probably SSSD isn't configured - Can't find configuration db, was SSSD configured and run? + [sss_cache] [sss_tool_confdb_init] (0x0010): Can't access '/var/lib/sss/db/config.ldb', probably SSSD isn't configured + Can't find configuration db, was SSSD configured and run? + [sss_cache] [sss_tool_confdb_init] (0x0010): Can't access '/var/lib/sss/db/config.ldb', probably SSSD isn't configured + Can't find configuration db, was SSSD configured and run? FAIL: 'sss_cache -UG' reported unexpected content on stderr - 'sss_cache -UG' stderr was: - [sss_cache] [sss_tool_confdb_init] (0x0010): Can't access '/var/lib/sss/db/config.ldb', probably SSSD isn't configured - Can't find configuration db, was SSSD configured and run? + 'sss_cache -UG' stderr was: + [sss_cache] [sss_tool_confdb_init] (0x0010): Can't access '/var/lib/sss/db/config.ldb', probably SSSD isn't configured + Can't find configuration db, was SSSD configured and run? RESULT: FAIL Destroying container lp2153951-192830 And here is a run showing the tests passing (I used a PPA for this example, but the real SRU verification must use "--pocket proposed"): $ ./lp2153951-test.sh --release stonking --pocket ppa:ahasenack/silence-sss-cache Launching stonking container lp2153951-196703 (phase 1: repositories + packages) Launching lp2153951-196703 ..........................................................................................status: done Resetting cloud-init state and rebooting (phase 2: user configuration) status: done === apt-cache policy passwd (package source used in this test) === passwd: - Installed: 1:4.19.3-2ubuntu2 - Candidate: 1:4.19.3-2ubuntu2 - Version table: - *** 1:4.19.3-2ubuntu2 500 - 500 http://archive.ubuntu.com/ubuntu stonking/main amd64 Packages - 100 /var/lib/dpkg/status + Installed: 1:4.19.3-2ubuntu2 + Candidate: 1:4.19.3-2ubuntu2 + Version table: + *** 1:4.19.3-2ubuntu2 500 + 500 http://archive.ubuntu.com/ubuntu stonking/main amd64 Packages + 100 /var/lib/dpkg/status === Checks === PASS: tuser has a hashed password in /etc/shadow (got: '$6$lp2153951$Iq7BfvEAR8qyz9eKAKs5sAGBBtyDoSRW0D2FnBJdG3Xy5.rLbSh29Hrds0VXSNzIx/1PNNmATPReqLjginX7/.') PASS: 'passwd -u tuser' produced no SSSD config-db complaint on stderr PASS: 'sss_cache -UG' was silent - 'sss_cache -UG' stderr was: - + 'sss_cache -UG' stderr was: RESULT: PASS Destroying container lp2153951-196703 [ Where problems could occur ] - An update to src:shadow affects all ubuntu users. The patch is contained - to the code that flushes the sssd cache, and only runs if the sss_cache - binary is installed. + The change to src:sssd is dropping an error message from the sss_cache + tool, and other callers of sss_tool_confdb_init(). The sss_cache message + was at level ERROR, whereas the one from sss_tool_confdb_ini() is at + level DEBUG, which is the default level set by sss_cache, that's why we + see it. - The cache flush, however, is not confined to the passwd command only. It - is used throughout the code, and also called from groupadd and others - (again, only if sss_cache is present). This is also the reason why - currently you see the same config.ldb errors messages when installing - sssd-tools fresh: + While we achieve the desired outcome of silencing the message when + sss_cache is called on a system where sssd is not yet configured, it + also silences the DEBUG message for other situations, including all + sssctl commands. + + The presented fix also has the added benefit of silencing these + sss_cache messages in the postinst output: ... Setting up sssd-common (2.12.0-1ubuntu5.5) ... Creating SSSD system user & group... [sss_cache] [sss_tool_confdb_init] (0x0010): Can't access '/var/lib/sss/db/config.ldb', probably SSSD isn't configured Can't find configuration db, was SSSD configured and run? [sss_cache] [sss_tool_confdb_init] (0x0010): Can't access '/var/lib/sss/db/config.ldb', probably SSSD isn't configured Can't find configuration db, was SSSD configured and run? ... [ Other Info ] See "Original Description" below for the original scenario where this bug appeared. [ Original Description ] Installed ubuntu-desktop in 26.04 server, and it didn't generate config.ldb file which kept deleting the password configured by cloud- init user data. ubuntu@ip-10-0-1-145:~$ sudo passwd -u ubuntu 2>&1 [sss_cache] [sss_tool_confdb_init] (0x0010): Can't access '/var/lib/sss/db/config.ldb', probably SSSD isn't configured Can't find configuration db, was SSSD configured and run? [sss_cache] [sss_tool_confdb_init] (0x0010): Can't access '/var/lib/sss/db/config.ldb', probably SSSD isn't configured Can't find configuration db, was SSSD configured and run? passwd: password changed. ubuntu@ip-10-0-1-145:~$ echo "exit: $?" exit: 0 there are output even though exit code 0 which triggers passwd -d in cloud-init script
** Description changed: [ Impact ] When using cloud-init to add a local user with a specific password/hash to the system at first boot, and sssd-tools also happens to be installed but sssd itself is not configured yet, a sequence of events leads to that user having no password set: - the user is created with the specified password hash just fine - cloud-init calls[1] "passwd -u" on the user - that passwd call ends up also calling "sss_cache -UG"[2], to reset the sssd cache, since the src:shadow package (where bin:passwd comes from) is built with sssd support[3] by default. If sss_cache is not available (bin:sssd-tools not installed), the bug does not manifest itself. - calling "sss_cache -UG" on a system with no valid sssd configuration yet prints an error message to stderr about a missing config.ldb, but exits with 0. This is since commits [4][5]. Those commits also changed sss_cache so that it no longer generates the config.ldb database, hence why it now always complains. - cloud-init checks exit status, and stderr, of the "passwd -u" call. It sees that stderr has content[6], and assumes a failure, proceeding to calling "passwd -d" on that account. This is what deletes the password hash from the user. For this SRU, I had originally opted to silence the error from the caller's side (src:shadow) if there is no config.ldb database present. That call is a best effort attempt to be helpful and clear the sssd cache, and the result of the call is being ignored anyway. But that has a much wider impact: src:shadow is installed everywhere. Even users not affected by this bug would get the update. Fedora (which is the sssd upstream) opted to entirely disable[7] sssd support in src:shadow. That is an option to consider for devel releases of ubuntu, but not an SRU. I then decided to silence the error from within sss_cache instead. It's the same fix in principle, and only users affected by the bug will receive the update. The one small change in behavior is that now when sss_cache doesn't find the config.ldb database, it's silent, so users might not be immediately aware that no action was taken by the tool. Note, however, that the exit code remains unchanged. And if you think about the end result of the tool run, it's the same: the cache was either expunged, or there was nothing to expunge. 1. https://github.com/canonical/cloud-init/blob/6193b2a0d241bed305e28a7aced1bd5c8c09de9a/cloudinit/distros/__init__.py#L1059 2. https://git.launchpad.net/ubuntu/+source/shadow/tree/src/passwd.c?h=applied/ubuntu/devel#n1175 3. https://git.launchpad.net/ubuntu/+source/shadow/tree/configure.ac?h=applied/ubuntu/devel#n202 4. https://github.com/SSSD/sssd/commit/a008accecd6d0b35e8d57d738ee3d05863aa7d0f 5. https://github.com/SSSD/sssd/commit/50b4579413c1146a2fced5ced19977e13790731c 6. https://github.com/canonical/cloud-init/blob/6193b2a0d241bed305e28a7aced1bd5c8c09de9a/cloudinit/distros/__init__.py#L1063 7. https://src.fedoraproject.org/rpms/shadow-utils/c/1820858d26f633482d3679d1c97324e25e3c1df8?branch=rawhide [ Test Plan ] There is a script attached to this bug which will reproduce the issue. It assumes a working lxd setup, and that "lxc launch ubuntu- daily:<ubuntu-release-name>" produces a working container. The script will exercise the following aspects of this bug: - provide the necessary cloud-init user-data to trigger the bug - check that the user has a password set or not in the end - check that "passwd -u" produces or not the stderr output that triggers the bug - check that "sss_cache -UG" is *always* silent, i.e., no stderr noise about the missing config.ldb To run the test, download the script to some directory and make it executable: wget https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/2153951/+attachment/6011062/+files/lp2153951-test.sh chmod +x lp2153951-test.sh Then run it targeting the ubuntu release under test. For example, for resolute: ./lp2153951-test.sh --release resolute That should reproduce the error, i.e., the script must indicate the test as having failed. To test the fixed packages, pass the "--pocket proposed" parameter. Again with the resolute example: ./lp2153951-test.sh --release resolute --pocket proposed This time the script should indicate that all checks have passed, and that completes the verification for this SRU. Here is run that shows the bug being reproduced: $ ./lp2153951-test.sh --release stonking Launching stonking container lp2153951-192830 (phase 1: repositories + packages) Launching lp2153951-192830 .....................................................status: done Resetting cloud-init state and rebooting (phase 2: user configuration) status: done === apt-cache policy passwd (package source used in this test) === passwd: Installed: 1:4.19.3-2ubuntu2 Candidate: 1:4.19.3-2ubuntu2 Version table: *** 1:4.19.3-2ubuntu2 500 500 http://archive.ubuntu.com/ubuntu stonking/main amd64 Packages 100 /var/lib/dpkg/status === Checks === FAIL: tuser has no hashed password in /etc/shadow (got: '<empty>') FAIL: 'passwd -u tuser' complained about the SSSD config db on stderr: [sss_cache] [sss_tool_confdb_init] (0x0010): Can't access '/var/lib/sss/db/config.ldb', probably SSSD isn't configured Can't find configuration db, was SSSD configured and run? [sss_cache] [sss_tool_confdb_init] (0x0010): Can't access '/var/lib/sss/db/config.ldb', probably SSSD isn't configured Can't find configuration db, was SSSD configured and run? FAIL: 'sss_cache -UG' reported unexpected content on stderr 'sss_cache -UG' stderr was: [sss_cache] [sss_tool_confdb_init] (0x0010): Can't access '/var/lib/sss/db/config.ldb', probably SSSD isn't configured Can't find configuration db, was SSSD configured and run? RESULT: FAIL Destroying container lp2153951-192830 And here is a run showing the tests passing (I used a PPA for this example, but the real SRU verification must use "--pocket proposed"): $ ./lp2153951-test.sh --release stonking --pocket ppa:ahasenack/silence-sss-cache Launching stonking container lp2153951-196703 (phase 1: repositories + packages) Launching lp2153951-196703 ..........................................................................................status: done Resetting cloud-init state and rebooting (phase 2: user configuration) status: done === apt-cache policy passwd (package source used in this test) === passwd: Installed: 1:4.19.3-2ubuntu2 Candidate: 1:4.19.3-2ubuntu2 Version table: *** 1:4.19.3-2ubuntu2 500 500 http://archive.ubuntu.com/ubuntu stonking/main amd64 Packages 100 /var/lib/dpkg/status === Checks === PASS: tuser has a hashed password in /etc/shadow (got: '$6$lp2153951$Iq7BfvEAR8qyz9eKAKs5sAGBBtyDoSRW0D2FnBJdG3Xy5.rLbSh29Hrds0VXSNzIx/1PNNmATPReqLjginX7/.') PASS: 'passwd -u tuser' produced no SSSD config-db complaint on stderr PASS: 'sss_cache -UG' was silent 'sss_cache -UG' stderr was: RESULT: PASS Destroying container lp2153951-196703 [ Where problems could occur ] The change to src:sssd is dropping an error message from the sss_cache tool, and other callers of sss_tool_confdb_init(). The sss_cache message was at level ERROR, whereas the one from sss_tool_confdb_ini() is at level DEBUG, which is the default level set by sss_cache, that's why we see it. While we achieve the desired outcome of silencing the message when sss_cache is called on a system where sssd is not yet configured, it also silences the DEBUG message for other situations, including all sssctl commands. + For example, before the update: + + # sssctl domain-list; echo $? + [sssd] [sss_tool_confdb_init] (0x0010): Can't access '/var/lib/sss/db/config.ldb', probably SSSD isn't configured + [sssd] [sss_tool_route] (0x0010): Initialization of command domain-list failed [2] No such file or directory + 1 + + + After the update: + + # sssctl domain-list; echo $? + [sssd] [sss_tool_route] (0x0010): Initialization of command domain-list failed [2] No such file or directory + 1 + + In both cases the exit status of sssctl is 1, but in the fixed case we + miss an important piece of information about why the command failed. + The presented fix also has the added benefit of silencing these sss_cache messages in the postinst output: ... Setting up sssd-common (2.12.0-1ubuntu5.5) ... Creating SSSD system user & group... [sss_cache] [sss_tool_confdb_init] (0x0010): Can't access '/var/lib/sss/db/config.ldb', probably SSSD isn't configured Can't find configuration db, was SSSD configured and run? [sss_cache] [sss_tool_confdb_init] (0x0010): Can't access '/var/lib/sss/db/config.ldb', probably SSSD isn't configured Can't find configuration db, was SSSD configured and run? ... [ Other Info ] See "Original Description" below for the original scenario where this bug appeared. [ Original Description ] Installed ubuntu-desktop in 26.04 server, and it didn't generate config.ldb file which kept deleting the password configured by cloud- init user data. ubuntu@ip-10-0-1-145:~$ sudo passwd -u ubuntu 2>&1 [sss_cache] [sss_tool_confdb_init] (0x0010): Can't access '/var/lib/sss/db/config.ldb', probably SSSD isn't configured Can't find configuration db, was SSSD configured and run? [sss_cache] [sss_tool_confdb_init] (0x0010): Can't access '/var/lib/sss/db/config.ldb', probably SSSD isn't configured Can't find configuration db, was SSSD configured and run? passwd: password changed. ubuntu@ip-10-0-1-145:~$ echo "exit: $?" exit: 0 there are output even though exit code 0 which triggers passwd -d in cloud-init script -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2153951 Title: no config.ldb file under /var/lib/sss/db in 26.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/shadow/+bug/2153951/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
