[Touch-packages] [Bug 1835464] Re: nginx service fails after libssl update due to low entropy at boot
@racb I'm not sure that I would consider it normal or expected, though, for system services to suddenly stop working due to regular updates, and for a server to suddenly become unreachable and unresponsive just because it was updated. On the other hand, it's certainly not desirable for a system to silently operate with poor entropy and poor encryption quality. In my case, this is easily resolved due to the hardware RNG on the TI AM335X chip. However, AFAIK a Raspberry PI does not have a hardware RNG, nor do many embedded processors / systems - meaning they would have low entropy at boot, and rng-tools most likely won't help. Without looking at any code, here are a few observations. Does nginx really need to make this blocking call to openssl when the service starts? or only when the first https request is made to the service? That is, if no https request comes in for 2 min, or 10 min, maybe there would be sufficient entropy by then due to system activity. Does openssl really need to block on initialization until sufficient entropy exists? Or could it defer that until some subsequent call that does actually need adequate entropy? In other words, would moving this blocking behavior to a different function satisfy the security need that led to its implementation, without potentially blocking systemd services at boot time? Finally, I have a couple of the same devices that do not exhibit this blocking behavior. I'm not sure exactly why, but the difference appears somehow related to the way updates are applied. I've noticed a file '/.rnd' (from memory) which is used and/or generated by openssl. Looks like this file is used as an entropy seed. Once deleted (and the hardware RNG is not used), the nginx systemd service will start blocking and timing out. Attempts to create this file manually using openssl do not allow the nginx service to start successfully at boot. Maybe the simple fix is to find the right way to create and manage the /.rnd file on devices with low entropy? -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to openssl in Ubuntu. https://bugs.launchpad.net/bugs/1835464 Title: nginx service fails after libssl update due to low entropy at boot Status in nginx package in Ubuntu: Opinion Status in openssl package in Ubuntu: New Status in nginx source package in Bionic: Opinion Status in openssl source package in Bionic: New Bug description: After updating libssl and related packages, nginx will no longer autostart at system boot. Immediately after boot, nginx.service is in a failed state. # service nginx status ● nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Active: failed (Result: timeout) since Fri 2018-08-24 21:27:51 UTC; 32min ago Docs: man:nginx(8) systemd[1]: Starting A high performance web server and a reverse proxy server... systemd[1]: nginx.service: Start-pre operation timed out. Terminating. systemd[1]: nginx.service: Failed with result 'timeout'. systemd[1]: Failed to start A high performance web server and a reverse proxy server. The service can be manually started after boot. # service nginx start # service nginx status ● nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2018-08-24 22:02:06 UTC; 2s ago Docs: man:nginx(8) Process: 2704 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS) Process: 2703 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS) Main PID: 2705 (nginx) CGroup: /system.slice/nginx.service ├─2705 nginx: master process /usr/sbin/nginx -g daemon on; master_process on; └─2706 nginx: worker process systemd[1]: Starting A high performance web server and a reverse proxy server... systemd[1]: nginx.service: Failed to parse PID from file /run/nginx.pid: Invalid argument systemd[1]: Started A high performance web server and a reverse proxy server. This happens on an ARMHF based microcontroller running ubuntu 18.04.2 raspi server distribution with a stock kernel.org 4.9-181 kernel. Ubuntu repositories are not accessible from the device, so packages are copied to the device, and apt install is used to upgrade them: apt install --no-install-recommends $dir/updates/system/*.deb | logger 2>&1 The following is a list of packages that, when upgraded, cause the nginx systemd service to fail to autostart at boot. 201,205c201,205 < ii libpython2.7:armhf 2.7.15-4ubuntu4~18.04 armhf Shared Python runtime library (version 2.7) < ii libpython2.7-minimal:armhf
[Touch-packages] [Bug 1835464] Re: nginx service fails after libssl update due to low entropy at boot
@teward No, I'm not sure whether it's an nginx bug. openssl packages were updated; nginx package is at the same version. Basically, it looks like an openssl call that previously succeeded (and probably gave questionable responses) now has become a blocking call that doesn't return until sufficient entropy is available to ensure a reasonably secure random result. Where before nginx completed in a timely manner, it appears to be now blocking, and failing to start within the systemd timeout period, If that's the case (which looks likely), then other services which depend on openssl may time out as well. (tomcat with APR comes to mind as one possibility.) -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to openssl in Ubuntu. https://bugs.launchpad.net/bugs/1835464 Title: nginx service fails after libssl update due to low entropy at boot Status in nginx package in Ubuntu: Incomplete Status in openssl package in Ubuntu: New Status in nginx source package in Bionic: Incomplete Status in openssl source package in Bionic: New Bug description: After updating libssl and related packages, nginx will no longer autostart at system boot. Immediately after boot, nginx.service is in a failed state. # service nginx status ● nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Active: failed (Result: timeout) since Fri 2018-08-24 21:27:51 UTC; 32min ago Docs: man:nginx(8) systemd[1]: Starting A high performance web server and a reverse proxy server... systemd[1]: nginx.service: Start-pre operation timed out. Terminating. systemd[1]: nginx.service: Failed with result 'timeout'. systemd[1]: Failed to start A high performance web server and a reverse proxy server. The service can be manually started after boot. # service nginx start # service nginx status ● nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2018-08-24 22:02:06 UTC; 2s ago Docs: man:nginx(8) Process: 2704 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS) Process: 2703 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS) Main PID: 2705 (nginx) CGroup: /system.slice/nginx.service ├─2705 nginx: master process /usr/sbin/nginx -g daemon on; master_process on; └─2706 nginx: worker process systemd[1]: Starting A high performance web server and a reverse proxy server... systemd[1]: nginx.service: Failed to parse PID from file /run/nginx.pid: Invalid argument systemd[1]: Started A high performance web server and a reverse proxy server. This happens on an ARMHF based microcontroller running ubuntu 18.04.2 raspi server distribution with a stock kernel.org 4.9-181 kernel. Ubuntu repositories are not accessible from the device, so packages are copied to the device, and apt install is used to upgrade them: apt install --no-install-recommends $dir/updates/system/*.deb | logger 2>&1 The following is a list of packages that, when upgraded, cause the nginx systemd service to fail to autostart at boot. 201,205c201,205 < ii libpython2.7:armhf 2.7.15-4ubuntu4~18.04 armhf Shared Python runtime library (version 2.7) < ii libpython2.7-minimal:armhf 2.7.15-4ubuntu4~18.04 armhf Minimal subset of the Python language (version 2.7) < ii libpython2.7-stdlib:armhf 2.7.15-4ubuntu4~18.04 armhf Interactive high-level object-oriented language (standard library, version 2.7) < ii libpython3.6-minimal:armhf 3.6.8-1~18.04.1 armhf Minimal subset of the Python language (version 3.6) < ii libpython3.6-stdlib:armhf 3.6.8-1~18.04.1 armhf Interactive high-level object-oriented language (standard library, version 3.6) --- > ii libpython2.7:armhf 2.7.15~rc1-1ubuntu0.1 armhf Shared Python runtime library (version 2.7) > ii libpython2.7-minimal:armhf 2.7.15~rc1-1ubuntu0.1 armhf Minimal subset of the Python language (version 2.7) > ii libpython2.7-stdlib:armhf 2.7.15~rc1-1ubuntu0.1 armhf Interactive high-level object-oriented language (standard library, version 2.7) > ii libpython3.6-minimal:armhf 3.6.7-1~18.04 armhf Minimal subset of the Python language (version 3.6) > ii libpython3.6-stdlib:armhf 3.6.7-1~18.04 armhf Interactive high-level object-oriented language (standard library, version 3.6) 225c225 < ii
[Touch-packages] [Bug 1835464] Re: nginx service fails after libssl update due to low entropy at boot
@xnox In my case, this is on a TI AM3352 processor. The key config item is: CONFIG_HW_RANDOM_OMAP=m TI's docs indicate that the following is important: CONFIG_CRYPTO_DEV_OMAP_SHAM=y And these may be related: CONFIG_CRYPTO_DEV_OMAP_AES=y CONFIG_CRYPTO_SHA256_ARM=y CONFIG_CRYPTO_SHA512_ARM=y In general, for devices having a hardware random number generator, I believe the following are needed: CONFIG_HW_RANDOM=m CONFIG_HW_RANDOM_TPM=m I started by building an ubuntu kernel for this ARM processor; but after some backported kernel patches broke the ubuntu kernel for my device, I switched to the kernel.org stock 4.9 LTS kernel. Incidentally, that's made it easier to get support from driver developers. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to openssl in Ubuntu. https://bugs.launchpad.net/bugs/1835464 Title: nginx service fails after libssl update due to low entropy at boot Status in nginx package in Ubuntu: Incomplete Status in openssl package in Ubuntu: New Status in nginx source package in Bionic: Incomplete Status in openssl source package in Bionic: New Bug description: After updating libssl and related packages, nginx will no longer autostart at system boot. Immediately after boot, nginx.service is in a failed state. # service nginx status ● nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Active: failed (Result: timeout) since Fri 2018-08-24 21:27:51 UTC; 32min ago Docs: man:nginx(8) systemd[1]: Starting A high performance web server and a reverse proxy server... systemd[1]: nginx.service: Start-pre operation timed out. Terminating. systemd[1]: nginx.service: Failed with result 'timeout'. systemd[1]: Failed to start A high performance web server and a reverse proxy server. The service can be manually started after boot. # service nginx start # service nginx status ● nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2018-08-24 22:02:06 UTC; 2s ago Docs: man:nginx(8) Process: 2704 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS) Process: 2703 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS) Main PID: 2705 (nginx) CGroup: /system.slice/nginx.service ├─2705 nginx: master process /usr/sbin/nginx -g daemon on; master_process on; └─2706 nginx: worker process systemd[1]: Starting A high performance web server and a reverse proxy server... systemd[1]: nginx.service: Failed to parse PID from file /run/nginx.pid: Invalid argument systemd[1]: Started A high performance web server and a reverse proxy server. This happens on an ARMHF based microcontroller running ubuntu 18.04.2 raspi server distribution with a stock kernel.org 4.9-181 kernel. Ubuntu repositories are not accessible from the device, so packages are copied to the device, and apt install is used to upgrade them: apt install --no-install-recommends $dir/updates/system/*.deb | logger 2>&1 The following is a list of packages that, when upgraded, cause the nginx systemd service to fail to autostart at boot. 201,205c201,205 < ii libpython2.7:armhf 2.7.15-4ubuntu4~18.04 armhf Shared Python runtime library (version 2.7) < ii libpython2.7-minimal:armhf 2.7.15-4ubuntu4~18.04 armhf Minimal subset of the Python language (version 2.7) < ii libpython2.7-stdlib:armhf 2.7.15-4ubuntu4~18.04 armhf Interactive high-level object-oriented language (standard library, version 2.7) < ii libpython3.6-minimal:armhf 3.6.8-1~18.04.1 armhf Minimal subset of the Python language (version 3.6) < ii libpython3.6-stdlib:armhf 3.6.8-1~18.04.1 armhf Interactive high-level object-oriented language (standard library, version 3.6) --- > ii libpython2.7:armhf 2.7.15~rc1-1ubuntu0.1 armhf Shared Python runtime library (version 2.7) > ii libpython2.7-minimal:armhf 2.7.15~rc1-1ubuntu0.1 armhf Minimal subset of the Python language (version 2.7) > ii libpython2.7-stdlib:armhf 2.7.15~rc1-1ubuntu0.1 armhf Interactive high-level object-oriented language (standard library, version 2.7) > ii libpython3.6-minimal:armhf 3.6.7-1~18.04 armhf Minimal subset of the Python language (version 3.6) > ii libpython3.6-stdlib:armhf 3.6.7-1~18.04 armhf Interactive high-level object-oriented language
