Re: [Bug 1919563] updated sssd with smart cards now brick systems without full cert chain
Marco, Great! This should be easy for me to test, and I’d be happy to do so. I may be able to do a regression test to make sure the automated NSSDB -> openssl upgrade works as well. This would mean however that the upgrade would need to drop the appropriate sssd.conf.d to configure the partial_chain config option on upgrade. I assume partial_chain will work even if the full chain is present? Karl > On Mar 28, 2021, at 4:15 PM, Marco Trevisan (Treviño) > <1919...@bugs.launchpad.net> wrote: > > So, I've done some work on SSSD upstream to make this to happen: > https://github.com/SSSD/sssd/pull/5558 > > With that we'll just be able to set on upgraders the option > `certification_verification = partial_chain`, and this will just make > the SSSD's PEM ring to work as the NSS db used to work: and so verify a > certificate if its only its issuer is in the SSSD's CA certificates DB. > > This comes with unit tests covering the case with generated > certificates, not sure if I can personally test this with real hardware > (for SRU purposes) though... We may still need to simulate it. > > At the end, it's just as doing: > openssl verify -partial_chain -CAfile intermediate_CA.pem > intermediate_CA_issued_cert.pem > > Karl, will this be enough for you? > > -- > You received this bug notification because you are subscribed to the bug > report. > https://bugs.launchpad.net/bugs/1919563 > > Title: > updated sssd with smart cards now brick systems without full cert > chain > > Status in sssd package in Ubuntu: > New > > Bug description: > With the latest sssd release supporting OpenSSL PKI authentication for > Ubuntu 20.04, the behavior between nssdb and OpenSSL has adversely > affected many systems which are configured for PKI only > authentication. > > The NSSDB implementation of sssd/p11_child ONLY requires the issuing > certificate to be populated to the nssdb and marked as trusted. While > this may be considered a poorly configured system, it is still > technically valid. > > The OpenSSL implementation of the sssd/p11_child requires the FULL > cert chain to the root cert (which is then also trusted by the system > root chain) in order to allow a certificate to authenticate. > > By upgrading to the latest packages, the conversion process from nssdb > to the OpenSSL pam file fails to check the chain of trust, thereby > creating a denial of service for some systems configured to require > smart card/PKI authentication in the pam stack via pam_sss and > require_cert_auth flag. > > Note that this is a popular configuration due to many organizations > are required to follow NIST 800-171 (and other) security derived > policy. Often policy requires PKI based authentication to be enforced > and all other authentication methods disabled. > > To manage notifications about this bug go to: > https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1919563/+subscriptions -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1919563 Title: updated sssd with smart cards now brick systems without full cert chain To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1919563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1919563] updated sssd with smart cards now brick systems without full cert chain
> On Mar 17, 2021, at 10:01 PM, Marco Trevisan (Treviño) > <1919...@bugs.launchpad.net> wrote: > > So, if I didn't get it wrong, if we'd just use /etc/ssl/certs/ca- > certificates.crt as the SSSD pam certificate in such case would work? While this would technically work, it would be really bad news. This would allow anyone with any user cert issued by a CA in the system wide cert store (by any CA in the world) to be trusted and pass authorization checks by p11_child. (Albeit, some directory attributes would have to line up, depending on your match rules) You only want the full chain of the issuing, intermediate and root CA(s) that are authorized to issue certs to users. Really, going to OpenSSL is a bit of a downgrade. Nssdb, allows one to flag which certs you want to trust, and which certs you don't. There’s no way that I’m aware of to flag a cert in the /etc/sssd/pki/sssd_auth_ca_db.pem to say I trust this cert, but not this other cert. Perhaps the right fix would be that p11_child is updated upstream to use the system cert store to complete chain of trust, and only the authorized issuing CAs should be put in the /etc/sssd/pki/sssd_auth_ca_db.pem file. Perhaps Summit would be willing to add this config option, for which could be set to default to the system store in Ubuntu in the Ubuntu packages. This still assumes a user has properly setup the system store to trust corporate root CAs (and intermediate, since there’s no way to differentiate between trusted root vs untrusted supporting certs in the system store in Ubuntu) > I mean having this in /etc/sssd/sssd.conf > > [pam] > pam_cert_db_path = /etc/ssl/certs/ca-certificates.crt > > And then what was into /etc/sssd/pki/sssd_auth_ca_db.pem to be added to > .crt's under /usr/local/share/ca-certificates/sssd_auth_ca_db/ and > eventually calling update-ca-certificates maybe? > > We could even do the other way around probably, by adding an hook to > /etc/ca-certificates/update.d/ so that we ensure that /etc/ssl/certs/ca- > certificates.crt is always in sync with the system ring? > > > As Robie said, we could revert this change but this would not be ideal for > various reasons IMHO: > 1. As you said this is going to be used more and more, and so we'll have to > end up to keep supporting >a growing number of systems with an outdated method that is going to be > dropped in future >(i.e. better to do it now that its usage is limited than having to do it > in future when the audience > is bigger) > 2. We would like to have a single documented method to have smartcard auth in > ubuntu using SSSD that can >be validated from 20.04 onward and that keep working in future LTSs (and > for sure next LTS will have to drop >NSS anyways, so it's just about delaying a problem making it bigger). Agreed. We just need something to prevent a user from bricking a system by just applying updates from upstream. This is kind of a high priority issue, given folks I’m sure are upgrading daily. By now this issue is starting to creep into offline synced repos. Karl -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1919563 Title: updated sssd with smart cards now brick systems without full cert chain To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1919563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1919563] Re: updated sssd with smart cards now brick systems without full cert chain
I agree and concur. Just need some checks here, as this is a pretty big change in behavior for a mid-life LTS release. That said, the new configuration is in line with RHEL8, and will help reduce the configuration scope for a working solution. I'll also comment, (and perhaps a bit of scope creap, but...) we've found a number of unfixed issues with sssd, specifically with PKINIT and LDAP optimizations. We're working with the upstream maintainers to help address these. We would like to see these brought into 20.04 LTS, as all directory users can benefit here. Are you or Marco the best to help us bring these into a general release down the road? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1919563 Title: updated sssd with smart cards now brick systems without full cert chain To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1919563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1919563] Re: updated sssd with smart cards now brick systems without full cert chain
To speak to real world assessment here - there's a big push across many (US) gov't orgs and industry to deploy MFA. These requirements are not new, but many have not been enforced due to lack of compliance checks/certifications. This is changing with new efforts in the US Gov't industry circles with regard to CMMC. This is an assessment/certification that industry must meet and maintain for contactual compliance, starting to roll out in the next year or so. Likewise there's been a lot of focus lately on unclassified compliance with NIST policy. We have a number of customers, working toward or maintaining an MFA solution. All are struggling. Many have lagged with pam_pkcs11 providing/satisfying most compliance requirements. But with RHEL8 and Ubuntu 20.04 adoption underway (with RHEL6 and 14.04 end of life) many are stuck working to cobble together an implementation. Of course with the uptick in remote work, MFA has also resurged, also pushing along adoption of sssd MFA. We noticed with the latest round of patching something was a-miss. and today tracked it down to this change. We're working with our customers to come up with a workaround. I think there's a larger number of folks impacted here, but unfortunately, the number of possible ways to do MFA is very large, and because no one maintainer has completely documented/supported MFA well, sysadmins typically develop their MFA craft using what they can. I don't discourage this change, in fact, will help push along the MFA adoption. However, I think perhaps some preflight checks in the package could solve someone bricking their machine. (or a large quantity of machines). I'd also suggest that MFA support in general should be considered a core requirement for future versions of the LTS, and well tested, supported and documented. Adoption will only grow with time, and become more critical. This will help reduce the variations of implementations, and help drive folks to a known and supported configuration. Reproduction of the issue: In our circles, we see a fully Microsoft AD integrated Smartcard (with kerberos and PKINIT) implementation. This also bleeds over into pam_sss configuration issues with U20.04, (for which I should file another ticket) Based on my diagnosis today, I think this is isolated to p11_child, and those with a nssdb with only issuing CA certs populated in the database. I don't think this issue matters for which directory is being used and if PKINIT is functioning, since all the MFA magic happens within p11_child. I'm going to assume that you folks have some way test AD with MFA, and will try to summarize. To reproduce, you'll need (at least) a 2 tier CA PKI chain. Root -> Issuing CA -> End user cert (with old sssd version) configure for smart card auth * do as you always do to join/setup sssd to a directory service * verify user ID lookups, and login works as expected with password * add any mapping/filter rules to the /etc/sssd/sssd.conf for p11_child * upadte /usr/share/pam-configs/sss to Priority 800, rebuild pam stack, dpkg-divert /usr/share/pam-configs/sss * add the root and issuing certs to /usr/local/share/ca-certificates, rebuild system trust store * generate a new, empty nssdb /usr/bin/certutil -N -d sql:/etc/pki/nssdb --empty-password * when adding the certs to nssdb, only add the Issuing CA WITH CT,C,C flags certutil -A -d /etc/pki/nssdb -n issuingCA.crt -t "CT,C,C" -i /usr/local/share/ca-certificates/issuingCA.pem * enable openSC modutil -force -dbdir /etc/pki/nssdb -add "OpenSC" -libfile opensc-pkcs11.so * test PKI auth works * login or: /usr/libexec/sssd/p11_child --nssdb=/etc/pki/nssdb --pre -d 10 --debug-fd=1 --verify no_ocsp * perform upgrade to latest sssd * verify the /etc/sssd/pki/sssd_auth_ca_db.pem is populated only with the issuingCA * test p11_child to see if it breaks /usr/libexec/sssd/p11_child --debug-microseconds=0 --debug-timestamps=1 --debug-fd=23 --debug-level=0xf7f0 --pre --verify no_ocsp --nssdb /etc/sssd/pki/sssd_auth_ca_db.pem fix it: * add the /usr/local/share/ca-certificates/rootCA.pem >> /etc/sssd/pki/sssd_auth_ca_db.pem * run p11_child again, observe that it works * try to login Brick your system procedure: After above test procedure works: * configure for MFA on old sssd * populate the below to /usr/share/pam-configs/sss-smartcardonly * pam-auth-update --package --enable sss-smartcardonly --remove sss --force * verify only smart card is allowed to login * apt upgrade * reboot, login no longer allowed Note that SSHing into the system may be allowed, depending on ssh configuration and if sss_ssh_authroizedkeys is enabled. Name: SSS authentication - Requires Smartcard Default: yes Conflicts: sss Priority: 800 Auth-Type: Primary Auth: [success=end default=ignore]pam_sss.so use_first_pass require_cert_auth Auth-Initial: [success=end default=ignore]pam_sss.so forward_pass require_cert_auth Account-Type: Additional Account: sufficient
[Bug 1905790] Re: Make SSSD in 20.04 using OpenSSL and p11-kit (instead of NSS) for p11_child
I don't know of a great way to test this without pulling apart p11_child, or using it as part of a pre-flight check somehow during the package update. The problem here is you'd need a PKI cert to test that preflight. As a failsafe, a dialog during upgrade with a preflight check of require_cert_auth in /etc/pam/common-password to throw a warning if the user continues with smart card enforcement. Force the user to ack to proceed, otherwise fail the package install. Perhaps adding a debconf flag to allow bypassing this dialog of this by sysadmins. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1905790 Title: Make SSSD in 20.04 using OpenSSL and p11-kit (instead of NSS) for p11_child To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1905790/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1905790] Re: Make SSSD in 20.04 using OpenSSL and p11-kit (instead of NSS) for p11_child
I've opened this as a new bug here. https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1919563 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1905790 Title: Make SSSD in 20.04 using OpenSSL and p11-kit (instead of NSS) for p11_child To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1905790/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1905790] Re: Make SSSD in 20.04 using OpenSSL and p11-kit (instead of NSS) for p11_child
This change had created a denial of service configuration bug for an untold number of smart card configured (and smart card requires) systems. p11_child requires with the OpenSSL PEM full cert chain to function. the NSSDB version does not. So for folks that have configured the minimum in the NSSDB by only adding the issuing certificate (and not chain of certs to the root), smart card authentication will now fail by simply updating to the latest Ubuntu 20.04 packages. The nssdb to pam conversion script does not check chain of trust in the new pam file. So when untold number of systems roll this out with require_cert_auth in the pam stack to be NIST 800-171 compliant, they will all be bricked and no way to login. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1905790 Title: Make SSSD in 20.04 using OpenSSL and p11-kit (instead of NSS) for p11_child To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1905790/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
