Launchpad has imported 4 comments from the remote bug at https://bugzilla.mozilla.org/show_bug.cgi?id=1721995.
If you reply to an imported comment from within Launchpad, your comment will be sent to the remote bug automatically. Read more about Launchpad's inter-bugtracker facilities at https://help.launchpad.net/InterBugTracking. ------------------------------------------------------------------------ On 2021-07-23T10:23:56+00:00 Paride Legovini wrote: User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:90.0) Gecko/20100101 Firefox/90.0 Steps to reproduce: When compiling nss with LTO enabled (gcc -flto) on s390x the resulting libnss3 is not fully functional. I noticed this as the build causes a regression in the dogtag-pki tests which are part of the dogtag-pki Ubuntu package. Newer releases of Ubuntu enable LTO by default when building packages. This specific issue will be worked around by disabling the optimizations specifically for this package and on s390x, however the problem is worth investigating upstream. The error printout doesn't immediately point to optimization issues, however this is always reproducible, and reliably goes away by turning LTO off. Steps to reproduce: - Build nss on s390x with LTO enabled. - Install dogtag-pki and ensure it uses the just built libnss3. - Exercise the following tests: https://salsa.debian.org/freeipa-team/dogtag-pki/-/blob/master/debian/tests/pkispawn. Actual results: The tests fail: autopkgtest [09:34:17]: test pkispawn: [----------------------- >>>> IP address is 10.226.183.135 >>>> Hostname was: >>>> /etc/hosts now has: 127.0.0.1 localhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts 10.226.183.135 autopkgtest.debci autopkgtest Starting installation... Completed installation for pki-tomcat Notice: Trust flag u is set automatically if the private key is present. /usr/lib/python3/dist-packages/urllib3/connection.py:455: SubjectAltNameWarning: Certificate for autopkgtest.debci has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/urllib3/urllib3/issues/497 for details.) warnings.warn( ERROR: ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) File "/usr/lib/python3/dist-packages/pki/server/pkispawn.py", line 575, in main scriptlet.spawn(deployer) File "/usr/lib/python3/dist-packages/pki/server/deployment/scriptlets/configuration.py", line 995, in spawn cert = deployer.setup_cert(client, tag) File "/usr/lib/python3/dist-packages/pki/server/deployment/__init__.py", line 355, in setup_cert return client.setupCert(request) File "/usr/lib/python3/dist-packages/pki/system.py", line 389, in setupCert response = self.connection.post( File "/usr/lib/python3/dist-packages/pki/client.py", line 55, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/pki/client.py", line 293, in post r = self.session.post( File "/usr/lib/python3/dist-packages/requests/sessions.py", line 590, in post return self.request('POST', url, data=data, json=json, **kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 655, in send r = adapter.send(request, **kwargs) File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send raise ConnectionError(err, request=request) Loading deployment configuration from debian/tests/deploy.cfg. Installation log: /var/log/pki/pki-ca-spawn.20210723093512.log Installing CA into /var/lib/pki/pki-tomcat. Installation failed: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) >>>> CA spawn failed: 2021-07-23 09:35:38 ERROR: ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) File "/usr/lib/python3/dist-packages/pki/server/pkispawn.py", line 575, in main scriptlet.spawn(deployer) File "/usr/lib/python3/dist-packages/pki/server/deployment/scriptlets/configuration.py", line 995, in spawn cert = deployer.setup_cert(client, tag) File "/usr/lib/python3/dist-packages/pki/server/deployment/__init__.py", line 355, in setup_cert return client.setupCert(request) File "/usr/lib/python3/dist-packages/pki/system.py", line 389, in setupCert response = self.connection.post( File "/usr/lib/python3/dist-packages/pki/client.py", line 55, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/pki/client.py", line 293, in post r = self.session.post( File "/usr/lib/python3/dist-packages/requests/sessions.py", line 590, in post return self.request('POST', url, data=data, json=json, **kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 655, in send r = adapter.send(request, **kwargs) File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send raise ConnectionError(err, request=request) autopkgtest [09:35:38]: test pkispawn: -----------------------] autopkgtest [09:35:39]: test pkispawn: - - - - - - - - - - results - - - - - - - - - - pkispawn FAIL non-zero exit status 1 autopkgtest [09:35:39]: @@@@@@@@@@@@@@@@@@@@ summary pkispawn FAIL non-zero exit status 1 Expected results: The tests all pass: autopkgtest [22:28:02]: test pkispawn: [----------------------- >>>> IP address is 10.226.183.148 >>>> Hostname was: >>>> /etc/hosts now has: 127.0.0.1 localhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts 10.226.183.148 autopkgtest.debci autopkgtest Starting installation... Completed installation for pki-tomcat Notice: Trust flag u is set automatically if the private key is present. /usr/lib/python3/dist-packages/urllib3/connection.py:455: SubjectAltNameWarning: Certificate for autopkgtest.debci has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/urllib3/urllib3/issues/497 for details.) warnings.warn( Loading deployment configuration from debian/tests/deploy.cfg. Installation log: /var/log/pki/pki-ca-spawn.20210722222833.log Installing CA into /var/lib/pki/pki-tomcat. ========================================================================== INSTALLATION SUMMARY ========================================================================== Administrator's username: caadmin Administrator's PKCS #12 file: /root/.dogtag/pki-tomcat/ca_admin_cert.p12 To check the status of the subsystem: systemctl status pki-tomcatd@pki-tomcat.service To restart the subsystem: systemctl restart pki-tomcatd@pki-tomcat.service The URL for the subsystem is: https://autopkgtest.debci:8443/ca PKI instances will be enabled upon system boot ========================================================================== WARNING: Directory already exists: /etc/pki/pki-tomcat log4j:WARN No appenders could be found for logger (org.jboss.logging). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. log4j:WARN No appenders could be found for logger (org.jboss.logging). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Loading deployment configuration from debian/tests/deploy.cfg. Installation log: /var/log/pki/pki-kra-spawn.20210722222939.log Installing KRA into /var/lib/pki/pki-tomcat. ========================================================================== INSTALLATION SUMMARY ========================================================================== Administrator's username: kraadmin To check the status of the subsystem: systemctl status pki-tomcatd@pki-tomcat.service To restart the subsystem: systemctl restart pki-tomcatd@pki-tomcat.service The URL for the subsystem is: https://autopkgtest.debci:8443/kra PKI instances will be enabled upon system boot ========================================================================== WARNING: Directory already exists: /etc/pki/pki-tomcat log4j:WARN No appenders could be found for logger (org.jboss.logging). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. log4j:WARN No appenders could be found for logger (org.jboss.logging). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Loading deployment configuration from debian/tests/deploy.cfg. Installation log: /var/log/pki/pki-ocsp-spawn.20210722223039.log Installing OCSP into /var/lib/pki/pki-tomcat. ========================================================================== INSTALLATION SUMMARY ========================================================================== Administrator's username: ocspadmin To check the status of the subsystem: systemctl status pki-tomcatd@pki-tomcat.service To restart the subsystem: systemctl restart pki-tomcatd@pki-tomcat.service The URL for the subsystem is: https://autopkgtest.debci:8443/ocsp PKI instances will be enabled upon system boot ========================================================================== WARNING: Directory already exists: /etc/pki/pki-tomcat log4j:WARN No appenders could be found for logger (org.jboss.logging). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. log4j:WARN No appenders could be found for logger (org.jboss.logging). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Loading deployment configuration from debian/tests/deploy.cfg. Installation log: /var/log/pki/pki-tks-spawn.20210722223141.log Installing TKS into /var/lib/pki/pki-tomcat. ========================================================================== INSTALLATION SUMMARY ========================================================================== Administrator's username: tksadmin To check the status of the subsystem: systemctl status pki-tomcatd@pki-tomcat.service To restart the subsystem: systemctl restart pki-tomcatd@pki-tomcat.service The URL for the subsystem is: https://autopkgtest.debci:8443/tks PKI instances will be enabled upon system boot ========================================================================== Loading deployment configuration from /var/lib/pki/pki-tomcat/tks/registry/tks/deployment.cfg. Uninstallation log: /var/log/pki/pki-tks-destroy.20210722223248.log Uninstalling TKS from /var/lib/pki/pki-tomcat. Uninstallation complete. WARNING: this 'OCSP' entry will NOT be deleted from security domain 'debci Security Domain'! WARNING: security domain 'debci Security Domain' may be offline or unreachable! ERROR: subprocess.CalledProcessError: Command '['/usr/bin/sslget', '-n', 'subsystemCert cert-pki-tomcat', '-p', '~EGeO^i!Ai4^', '-d', '/etc/pki/pki-tomcat/alias', '-e', 'name="/var/lib/pki/pki-tomcat"&type=OCSP&list=ocspList&host=autopkgtest.debci&sport=8443&ncsport=8443&adminsport=8443&agentsport=8443&operation=remove', '-v', '-r', '/ca/agent/ca/updateDomainXML', 'autopkgtest.debci:8443']' returned non-zero exit status 6.! Loading deployment configuration from /var/lib/pki/pki-tomcat/ocsp/registry/ocsp/deployment.cfg. Uninstallation log: /var/log/pki/pki-ocsp-destroy.20210722223255.log Uninstalling OCSP from /var/lib/pki/pki-tomcat. Uninstallation complete. ERROR: unable to access security domain. Continuing .. HTTPSConnectionPool(host='autopkgtest.debci', port=8443): Max retries exceeded with url: /ca/rest/securityDomain/domainInfo (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x3ff8af5a2b0>: Failed to establish a new connection: [Errno 111] Connection refused')) WARNING: this 'KRA' entry will NOT be deleted from security domain 'debci Security Domain'! WARNING: security domain 'debci Security Domain' may be offline or unreachable! ERROR: subprocess.CalledProcessError: Command '['/usr/bin/sslget', '-n', 'subsystemCert cert-pki-tomcat', '-p', '~EGeO^i!Ai4^', '-d', '/etc/pki/pki-tomcat/alias', '-e', 'name="/var/lib/pki/pki-tomcat"&type=KRA&list=kraList&host=autopkgtest.debci&sport=8443&ncsport=8443&adminsport=8443&agentsport=8443&operation=remove', '-v', '-r', '/ca/agent/ca/updateDomainXML', 'autopkgtest.debci:8443']' returned non-zero exit status 6.! Loading deployment configuration from /var/lib/pki/pki-tomcat/kra/registry/kra/deployment.cfg. Uninstallation log: /var/log/pki/pki-kra-destroy.20210722223257.log Uninstalling KRA from /var/lib/pki/pki-tomcat. Uninstallation complete. WARNING: this 'CA' entry will NOT be deleted from security domain 'debci Security Domain'! WARNING: security domain 'debci Security Domain' may be offline or unreachable! ERROR: subprocess.CalledProcessError: Command '['/usr/bin/sslget', '-n', 'subsystemCert cert-pki-tomcat', '-p', '~EGeO^i!Ai4^', '-d', '/etc/pki/pki-tomcat/alias', '-e', 'name="/var/lib/pki/pki-tomcat"&type=CA&list=caList&host=autopkgtest.debci&sport=8443&ncsport=8443&adminsport=8443&agentsport=8443&operation=remove', '-v', '-r', '/ca/agent/ca/updateDomainXML', 'autopkgtest.debci:8443']' returned non-zero exit status 6.! Loading deployment configuration from /var/lib/pki/pki-tomcat/ca/registry/ca/deployment.cfg. Uninstallation log: /var/log/pki/pki-ca-destroy.20210722223258.log Uninstalling CA from /var/lib/pki/pki-tomcat. Uninstallation complete. >>>> All done! autopkgtest [22:32:59]: test pkispawn: -----------------------] autopkgtest [22:33:00]: test pkispawn: - - - - - - - - - - results - - - - - - - - - - pkispawn PASS autopkgtest [22:33:00]: @@@@@@@@@@@@@@@@@@@@ summary pkispawn PASS Reply at: https://bugs.launchpad.net/ubuntu/+source/nss/+bug/1931104/comments/14 ------------------------------------------------------------------------ On 2021-07-28T00:08:53+00:00 Rrelyea wrote: Evidently there is a similiar issue in fedora: https://bugzilla.redhat.com/show_bug.cgi?id=1986627 Reply at: https://bugs.launchpad.net/ubuntu/+source/nss/+bug/1931104/comments/17 ------------------------------------------------------------------------ On 2021-08-10T16:59:48+00:00 Bbeurdouche wrote: Bob, I am marking this P3 for now as this is not a supported platform for us, but feel free to update the priority. Reply at: https://bugs.launchpad.net/ubuntu/+source/nss/+bug/1931104/comments/18 ------------------------------------------------------------------------ On 2021-08-10T20:27:32+00:00 Rrelyea wrote: I did a scratch build of nss with LTO on in fedora, so the tests were working correctly. I haven't tested it against dogtag yet. Once NSS 3.69 builds are complete, I'll drop the LTO changes into fedora and see if our dogtag team has any issues. Reply at: https://bugs.launchpad.net/ubuntu/+source/nss/+bug/1931104/comments/19 ** Changed in: nss Status: Unknown => New ** Bug watch added: github.com/urllib3/urllib3/issues #497 https://github.com/urllib3/urllib3/issues/497 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to nss in Ubuntu. https://bugs.launchpad.net/bugs/1931104 Title: Test of dogtag-pki is failing on s390x due to LTO Status in NSS: New Status in nss package in Ubuntu: Triaged Status in nss package in Fedora: Unknown Bug description: The test of dogtag-pki is failing on the nss 3.63 that is in impish proposed. Example: https://autopkgtest.ubuntu.com/results/autopkgtest-impish/impish/s390x/d/dogtag-pki/20210516_212719_e6522@/log.gz Bad: Installing CA into /var/lib/pki/pki-tomcat. Installation failed: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) ERROR: ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) File "/usr/lib/python3/dist-packages/pki/server/pkispawn.py", line 575, in main scriptlet.spawn(deployer) File "/usr/lib/python3/dist-packages/pki/server/deployment/scriptlets/configuration.py", line 995, in spawn cert = deployer.setup_cert(client, tag) File "/usr/lib/python3/dist-packages/pki/server/deployment/__init__.py", line 355, in setup_cert return client.setupCert(request) File "/usr/lib/python3/dist-packages/pki/system.py", line 389, in setupCert response = self.connection.post( File "/usr/lib/python3/dist-packages/pki/client.py", line 55, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/pki/client.py", line 293, in post r = self.session.post( File "/usr/lib/python3/dist-packages/requests/sessions.py", line 590, in post return self.request('POST', url, data=data, json=json, **kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 655, in send r = adapter.send(request, **kwargs) File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send raise ConnectionError(err, request=request) >>>> CA spawn failed: Good: nstalling CA into /var/lib/pki/pki-tomcat. Notice: Trust flag u is set automatically if the private key is present. /usr/lib/python3/dist-packages/urllib3/connection.py:455: SubjectAltNameWarning: Certificate for i-dogtag has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/urllib3/urllib3/issues/497 for details.) warnings.warn( ========================================================================== INSTALLATION SUMMARY ========================================================================== ... The good test above was with: ii libnss3:s390x 2:3.61-1ubuntu2 s390x Network Security Service libraries ii 389-ds-base 1.4.4.11-2 s390x 389 Directory Server suite - server Worth to know, the good case test still fails later on with: IOException: SocketException cannot write on socket: Failed to write to socket: (-5938) Encountered end of file. ERROR: CalledProcessError: Command '['pki', '-d', '/etc/pki/pki-tomcat/alias', '-f', '/etc/pki/pki-tomcat/password.conf', '-U', 'https://i-dogtag:8443', 'securitydomain-join', '--session', '4717921475119312283', '--type', 'TKS', '--hostname', 'i-dogtag', '--unsecure-port', '8080', '--secure-port', '8443', 'TKS i-dogtag 8443']' returned non-zero exit status 255. File "/usr/lib/python3/dist-packages/pki/server/pkispawn.py", line 575, in main scriptlet.spawn(deployer) File "/usr/lib/python3/dist-packages/pki/server/deployment/scriptlets/configuration.py", line 1038, in spawn subsystem.join_security_domain( File "/usr/lib/python3/dist-packages/pki/server/subsystem.py", line 1201, in join_security_domain subprocess.check_call(cmd) File "/usr/lib/python3.9/subprocess.py", line 373, in check_call raise CalledProcessError(retcode, cmd) Installation failed: Command failed: pki -d /etc/pki/pki-tomcat/alias -f /etc/pki/pki-tomcat/password.conf -U https://i-dogtag:8443 securitydomain-join --session 4717921475119312283 --type TKS --hostname i-dogtag --unsecure-port 8080 --secure-port 8443 TKS i-dogtag 8443 Please check pkispawn logs in /var/log/pki/pki-tks-spawn.20210607093926.log Well one issue at a time ... the current install issue first. Since it worked with the nss in -release I was upgrading this to the new nss. ii 389-ds-base 1.4.4.11-2 s390x 389 Directory Server suite - server ii libnss3:s390x 2:3.63-1ubuntu1 s390x Network Security Service libraries With this the install fail is reprodicible. So we can switch in/out bad case by up/downgrading libnss3. Comparing those two cases until they reach the first successful install message I've seen a crash: pki-tomcat[37160]: # pki-tomcat[37160]: # A fatal error has been detected by the Java Runtime Environment: pki-tomcat[37160]: # pki-tomcat[37160]: # SIGSEGV (0xb) at pc=0x000003ff9ce9ec02, pid=37160, tid=37246 pki-tomcat[37160]: # pki-tomcat[37160]: # JRE version: OpenJDK Runtime Environment (11.0.12+4) (build 11.0.12-ea+4-Ubuntu-0ubuntu2) pki-tomcat[37160]: # Java VM: OpenJDK 64-Bit Server VM (11.0.12-ea+4-Ubuntu-0ubuntu2, mixed mode, tiered, compressed oops, serial gc, linux-s390x) pki-tomcat[37160]: # Problematic frame: pki-tomcat[37160]: # C [libnss3.so+0x11ec02] pki-tomcat[37160]: # pki-tomcat[37160]: # Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /var/lib/pki/pki-tomcat/core.37160) pki-tomcat[37160]: # pki-tomcat[37160]: # An error report file with more information is saved as: pki-tomcat[37160]: # /var/lib/pki/pki-tomcat/hs_err_pid37160.log pki-tomcat[37160]: # pki-tomcat[37160]: # If you would like to submit a bug report, please visit: pki-tomcat[37160]: # https://bugs.launchpad.net/ubuntu/+source/openjdk-lts pki-tomcat[37160]: # The crash happened outside the Java Virtual Machine in native code. pki-tomcat[37160]: # See problematic frame for where to report the bug. A few extra runs had also shown: # Problematic frame: # C [libnssutil3.so+0x1b60c] PORT_FreeArena_Util+0xc And while I could not get a core dump out as the config required to be changed is written on the fly and then started I was able to find the code. Obviously there has to be a lot of abstraction but plenty of recent changes fixed double frees and dangling pointer values. For example https://github.com/nss-dev/nss/commit/350807b3a70f60928ea3f2bc95fd1795aae9b753 This is all (this and more similar fixes) in 3.66 which is released and in Debian unstable. It might be worth to re-merge that, throw it into a PPA and re-run the tests. To manage notifications about this bug go to: https://bugs.launchpad.net/nss/+bug/1931104/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
