[Sts-sponsors] [Bug 1908473] Re: rsyslog-relp: imrelp module leaves sockets in CLOSE_WAIT state which leads to file descriptor leak
Performing verification for Focal.
I installed rsyslog-relp 8.2001.0-1ubuntu1.1 and librelp0 1.5.0-1ubuntu2
from -updates.
>From there I set up the configuration file, launched a new rsyslog
instance, and used netcat to set 100 packets to the relp port.
https://paste.ubuntu.com/p/HfSDvNJzpX/
As we can see, there are 100 sockets still open in the CLOSE_WAIT state.
>From there I enabled -proposed and installed librelp
1.5.0-1ubuntu2.20.04.2.
I started a new instance of rsyslog, and used netcat to send another 100
packets to the relp port. This time, all sockets were closed and not
left in CLOSE_WAIT.
https://paste.ubuntu.com/p/tjXHhQ2293/
I also ran the testcase from the upstream testsuite, imrelp-
sessionbreak-vg.sh.
I did this by:
1) pull-lp-source rsyslog focal
2) edit debian/rules, add --enable-valgrind, remove --without-valgrind-tests,
3) wget
https://github.com/rsyslog/rsyslog/commit/baee0bd5420649329793746f0daf87c4f59fe6a6.patch
4) quilt import baee0bd5420649329793746f0daf87c4f59fe6a6.patch
5) quilt push
6) chmod +x tests/imrelp-sessionbreak-vg.sh
6) debuild -uc -us -b
It will eventually build tests, and imrelp-sessionbreak-vg.sh passes:
make[5]: Entering directory '/home/ubuntu/rsyslog-8.2001.0/tests'
...
PASS: imrelp-sessionbreak-vg.sh
...
We pass both the upstream testsuite and the testcase from the bug
report.
The file descriptor leak has been fixed, happy to mark as verified for
Focal.
** Tags removed: verification-needed verification-needed-focal
** Tags added: verification-done-focal
--
You received this bug notification because you are a member of STS
Sponsors, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1908473
Title:
rsyslog-relp: imrelp module leaves sockets in CLOSE_WAIT state which
leads to file descriptor leak
Status in librelp package in Ubuntu:
Fix Released
Status in rsyslog package in Ubuntu:
Fix Released
Status in librelp source package in Focal:
Fix Committed
Status in rsyslog source package in Focal:
Won't Fix
Status in librelp source package in Groovy:
Fix Committed
Status in rsyslog source package in Groovy:
Fix Released
Status in librelp source package in Hirsute:
Fix Released
Status in rsyslog source package in Hirsute:
Fix Released
Bug description:
[Impact]
In recent versions of rsyslog and librelp, the imrelp module leaks
file descriptors due to a bug where it does not correctly close
sockets, and instead, leaves them in the CLOSE_WAIT state.
This causes rsyslogd on busy servers to eventually hit the limit of
maximum open files allowed, which locks rsyslogd up until it is
restarted.
A workaround is to restart rsyslogd every month or so to manually
close all of the open sockets.
Only users of the imrelp module are affected, and not rsyslog users in
general.
[Testcase]
Install the rsyslog-relp module like so:
$ sudo apt install rsyslog rsyslog-relp
Next, generate a working directory, and make a config file that loads
the relp module.
$ sudo mkdir /workdir
$ cat << EOF >> ./spool.conf
\$LocalHostName spool
\$AbortOnUncleanConfig on
\$PreserveFQDN on
global(
workDirectory="/workdir"
maxMessageSize="256k"
)
main_queue(queue.type="Direct")
module(load="imrelp")
input(
type="imrelp"
name="imrelp"
port="601"
ruleset="spool"
MaxDataSize="256k"
)
ruleset(name="spool" queue.type="direct") {
}
# Just so rsyslog doesn't whine that we do not have outputs
ruleset(name="noop" queue.type="direct") {
action(
type="omfile"
name="omfile"
file="/workdir/spool.log"
)
}
EOF
Verify that the config is valid, then start a rsyslog server.
$ sudo rsyslogd -f ./spool.conf -N9
$ sudo rsyslogd -f ./spool.conf -i /workdir/rsyslogd.pid
Fetch the rsyslogd PID and check for open files.
$ RLOGPID=$(cat /workdir/rsyslogd.pid)
$ sudo ls -l /proc/$RLOGPID/fd
total 0
lr-x-- 1 root root 64 Dec 17 01:22 0 -> /dev/urandom
lrwx-- 1 root root 64 Dec 17 01:22 1 -> 'socket:[41228]'
lrwx-- 1 root root 64 Dec 17 01:22 3 -> 'socket:[41222]'
lrwx-- 1 root root 64 Dec 17 01:22 4 -> 'socket:[41223]'
lrwx-- 1 root root 64 Dec 17 01:22 7 -> 'anon_inode:[eventpoll]'
We have 3 sockets open by default. Next, use netcat to open 100
connections:
$ for i in {1..100} ; do nc -z 127.0.0.1 601 ; done
Now check for open file descriptors, and there will be an extra 100 sockets
in the list:
$ sudo ls -l /proc/$RLOGPID/fd
https://paste.ubuntu.com/p/f6NQVNbZcR/
We can check the state of these sockets with:
$ ss -t
https://paste.ubuntu.com/p/7Ts2FbxJrg/
The listening sockets will be in CLOSE-WAIT, and the netcat sockets
will be in FIN-WAIT-2.
$ ss -t | grep CLOSE-WAIT | wc -l
100
If you install the test package available in the following ppa:
https://launchpad.net/~mruffell/+archive/ubuntu/sf299578-test
When you open
[Sts-sponsors] [Bug 1906720] Re: Fix the disable_ssl_certificate_validation option
** Description changed:
[Impact]
* On Bionic, MAAS CLI fails to work with apis over https with self-signed
certificates due to broken disable_ssl_certificate_validation option
with python 3.5 and later.
[Steps to Reproduce]
1. prepare a maas server (it doesn't have to be HA to reproduce)
2. prepare a set of certificate, key and ca-bundle
3. place a new conf in /etc/nginx/sites-enabled and `sudo systemctl
restart nginx`
4. add the ca certificates to the host
sudo mkdir /usr/share/ca-certificates/extra
sudo cp -v ca-bundle.crt /usr/share/ca-certificates/extra/
dpkg-reconfigure ca-certificates
5. login with a new profile over https url
6. if the certificate is not trusted by the root store, it fails to login
7. adding the '--insecure' flag should disable the certificate check
[Where problems could occur]
* Potential issues could happen if we disable certificate validation for
all TLS interactions, any connection https related.
* Should not break existing python3 versions.
* Should not affect previously working python2 versions.
[Other Info]
This change should fix the issue with python3, and you should be able
to connect with python2 as before.
python2 => python-httplib2_0.9.2+dfsg-1ubuntu0.3_all.deb
python3 => python3-httplib2_0.9.2+dfsg-1ubuntu0.3_all.deb
+ *both are build from the same source package
helpful urls:
https://maas.io/docs/deb/2.8/cli/installation
https://maas.io/docs/deb/2.8/cli/configuration-journey
https://maas.io/docs/deb/2.8/ui/configuration-journey
# create bionic VM/lxc container
lxc launch ubuntu:bionic lp1906720
# get source code from repo
pull-lp-source python-httplib2 bionic
# install maas-cli
apt-get install maas-cli
# install maas server
apt-get install maas
# init maas
sudo maas init
# answer questions
# generate self signed cert and key
openssl req -newkey rsa:4096 -x509 -sha256 -days 60 -nodes -out localhost.crt
-keyout localhost.key
# add certs
sudo cp -v localhost.crt /usr/share/ca-certificates/extra/
# add new cert to list
sudo dpkg-reconfigure ca-certificates
+ [1]
# select yes with spacebar
# save and it will reload with 1 new certificate
# create api key files
touch api_key
touch api-key-file
# remove any packages with this
# or this python3-httplib2
apt-cache search python-httplib2
apt-get remove python-httplib2
apt-get remove python3-httplib2
# create 2 admin users
sudo maas createadmin testadmin
sudo maas createadmin secureadmin
# generate maas api keys
sudo maas apikey --username=testadmin > api_key
sudo maas apikey --username=secureadmin > api-key-file
# setup nginx proxy
sudo apt update
sudo apt install nginx
touch /etc/nginx/sites-available/maas-https-default
# contents of maas-https-default
server {
listen 443 ssl http2;
server_name _;
ssl_certificate /home/ubuntu/localhost.crt;
ssl_certificate_key /home/ubuntu/localhost.key;
location / {
proxy_pass http://localhost:5240;
include /etc/nginx/proxy_params;
}
location /MAAS/ws {
proxy_pass http://127.0.0.1:5240/MAAS/ws;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
}
}
sudo service nginx restart
# make sure you can login to maas-cli without TLS
# by running this script
# this is for the non-tls user
# this goes into a script called maas-login.sh
touch maas-login.sh
sudo chmod +rwx maas-login.sh
#!/bin/sh
PROFILE=testadmin
API_KEY_FILE=/home/ubuntu/api_key
API_SERVER=127.0.0.1:5240
MAAS_URL=http://$API_SERVER/MAAS
maas login $PROFILE $MAAS_URL - < $API_KEY_FILE
sudo chmod +rwx https-maas.sh
# another script called https-maas.sh
# for the tls user
#!/bin/sh
PROFILE=secureadmin
API_KEY_FILE=/home/ubuntu/api-key-file
API_SERVER=127.0.0.1
MAAS_URL=https://$API_SERVER/MAAS
maas login $PROFILE $MAAS_URL - < $API_KEY_FILE
# try to login
./maas-login.sh
cd /etc/nginx/sites-enabled
sudo touch maas-https-default
#example nginx config for maas https
server {
listen 443 ssl http2;
server_name _;
ssl_certificate /home/ubuntu/localhost.crt;
ssl_certificate_key /home/ubuntu/localhost.key;
location / {
proxy_pass http://localhost:5240;
include /etc/nginx/proxy_params;
}
location /MAAS/ws {
proxy_pass http://127.0.0.1:5240/MAAS/ws;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
}
}
# create link
sudo ln -s /etc/nginx/sites-available/maas-https-default
/etc/nginx/sites-enabled
# look at errors
cat /var/log/maas/regiond.log
cat regiond.log | grep "Python-http"
*i didn't see
[Sts-sponsors] [Bug 1906720] Re: Fix the disable_ssl_certificate_validation option
resolved https maas-cli
** Attachment added: "Screenshot from 2021-01-25 09-31-14.png"
https://bugs.launchpad.net/ubuntu/bionic/+source/python-httplib2/+bug/1906720/+attachment/5456688/+files/Screenshot%20from%202021-01-25%2009-31-14.png
--
You received this bug notification because you are a member of STS
Sponsors, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1906720
Title:
Fix the disable_ssl_certificate_validation option
Status in python-httplib2 package in Ubuntu:
Fix Released
Status in python-httplib2 source package in Bionic:
In Progress
Status in python-httplib2 source package in Focal:
Fix Released
Status in python-httplib2 source package in Groovy:
Fix Released
Status in python-httplib2 source package in Hirsute:
Fix Released
Bug description:
[Impact]
* On Bionic, MAAS CLI fails to work with apis over https with self-signed
certificates due to broken disable_ssl_certificate_validation option
with python 3.5 and later.
[Steps to Reproduce]
1. prepare a maas server (it doesn't have to be HA to reproduce)
2. prepare a set of certificate, key and ca-bundle
3. place a new conf in /etc/nginx/sites-enabled and `sudo systemctl
restart nginx`
4. add the ca certificates to the host
sudo mkdir /usr/share/ca-certificates/extra
sudo cp -v ca-bundle.crt /usr/share/ca-certificates/extra/
dpkg-reconfigure ca-certificates
5. login with a new profile over https url
6. if the certificate is not trusted by the root store, it fails to login
7. adding the '--insecure' flag should disable the certificate check
[Where problems could occur]
* Potential issues could happen if we disable certificate validation for
all TLS interactions, any connection https related.
* Should not break existing python3 versions.
* Should not affect previously working python2 versions.
[Other Info]
This change should fix the issue with python3, and you should be able
to connect with python2 as before.
python2 => python-httplib2_0.9.2+dfsg-1ubuntu0.3_all.deb
python3 => python3-httplib2_0.9.2+dfsg-1ubuntu0.3_all.deb
*both are build from the same source package
helpful urls:
https://maas.io/docs/deb/2.8/cli/installation
https://maas.io/docs/deb/2.8/cli/configuration-journey
https://maas.io/docs/deb/2.8/ui/configuration-journey
# create bionic VM/lxc container
lxc launch ubuntu:bionic lp1906720
# get source code from repo
pull-lp-source python-httplib2 bionic
# install maas-cli
apt-get install maas-cli
# install maas server
apt-get install maas
# init maas
sudo maas init
# answer questions
# generate self signed cert and key
openssl req -newkey rsa:4096 -x509 -sha256 -days 60 -nodes -out localhost.crt
-keyout localhost.key
# add certs
sudo cp -v localhost.crt /usr/share/ca-certificates/extra/
# add new cert to list
sudo dpkg-reconfigure ca-certificates
[1]
# select yes with spacebar
# save and it will reload with 1 new certificate
# create api key files
touch api_key
touch api-key-file
# remove any packages with this
# or this python3-httplib2
apt-cache search python-httplib2
apt-get remove python-httplib2
apt-get remove python3-httplib2
# create 2 admin users
sudo maas createadmin testadmin
sudo maas createadmin secureadmin
# generate maas api keys
sudo maas apikey --username=testadmin > api_key
sudo maas apikey --username=secureadmin > api-key-file
# setup nginx proxy
sudo apt update
sudo apt install nginx
touch /etc/nginx/sites-available/maas-https-default
# contents of maas-https-default
server {
listen 443 ssl http2;
server_name _;
ssl_certificate /home/ubuntu/localhost.crt;
ssl_certificate_key /home/ubuntu/localhost.key;
location / {
proxy_pass http://localhost:5240;
include /etc/nginx/proxy_params;
}
location /MAAS/ws {
proxy_pass http://127.0.0.1:5240/MAAS/ws;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
}
}
sudo service nginx restart
# make sure you can login to maas-cli without TLS
# by running this script
# this is for the non-tls user
# this goes into a script called maas-login.sh
touch maas-login.sh
sudo chmod +rwx maas-login.sh
#!/bin/sh
PROFILE=testadmin
API_KEY_FILE=/home/ubuntu/api_key
API_SERVER=127.0.0.1:5240
MAAS_URL=http://$API_SERVER/MAAS
maas login $PROFILE $MAAS_URL - < $API_KEY_FILE
sudo chmod +rwx https-maas.sh
# another script called https-maas.sh
# for the tls user
#!/bin/sh
PROFILE=secureadmin
API_KEY_FILE=/home/ubuntu/api-key-file
API_SERVER=127.0.0.1
MAAS_URL=https://$API_SERVER/MAAS
maas login $PROFILE $MAAS_URL - < $API_KEY_FILE
# try to login
./maas-login.sh
cd /etc/nginx/sites-enabled
sudo touch maas-https-default
#example nginx config
[Sts-sponsors] [Bug 1906720] Re: Fix the disable_ssl_certificate_validation option
** Description changed:
[Impact]
* On Bionic, MAAS CLI fails to work with apis over https with self-signed
certificates due to broken disable_ssl_certificate_validation option
with python 3.5 and later.
[Steps to Reproduce]
1. prepare a maas server (it doesn't have to be HA to reproduce)
2. prepare a set of certificate, key and ca-bundle
3. place a new conf in /etc/nginx/sites-enabled and `sudo systemctl
restart nginx`
4. add the ca certificates to the host
sudo mkdir /usr/share/ca-certificates/extra
sudo cp -v ca-bundle.crt /usr/share/ca-certificates/extra/
dpkg-reconfigure ca-certificates
5. login with a new profile over https url
6. if the certificate is not trusted by the root store, it fails to login
7. adding the '--insecure' flag should disable the certificate check
[Where problems could occur]
* Potential issues could happen if we disable certificate validation for
all TLS interactions, any connection https related.
* Should not break existing python3 versions.
* Should not affect previously working python2 versions.
[Other Info]
This change should fix the issue with python3, and you should be able
to connect with python2 as before.
python2 => python-httplib2_0.9.2+dfsg-1ubuntu0.3_all.deb
python3 => python3-httplib2_0.9.2+dfsg-1ubuntu0.3_all.deb
helpful urls:
https://maas.io/docs/deb/2.8/cli/installation
https://maas.io/docs/deb/2.8/cli/configuration-journey
https://maas.io/docs/deb/2.8/ui/configuration-journey
# create bionic VM/lxc container
lxc launch ubuntu:bionic lp1906720
# get source code from repo
pull-lp-source python-httplib2 bionic
# install maas-cli
apt-get install maas-cli
# install maas server
apt-get install maas
# init maas
sudo maas init
# answer questions
# generate self signed cert and key
openssl req -newkey rsa:4096 -x509 -sha256 -days 60 -nodes -out localhost.crt
-keyout localhost.key
# add certs
- sudo cp -v test.crt /usr/share/ca-certificates/extra/
+ sudo cp -v localhost.crt /usr/share/ca-certificates/extra/
# add new cert to list
sudo dpkg-reconfigure ca-certificates
# select yes with spacebar
- # save
+ # save and it will reload with 1 new certificate
# create api key files
touch api_key
touch api-key-file
# remove any packages with this
# or this python3-httplib2
apt-cache search python-httplib2
apt-get remove python-httplib2
apt-get remove python3-httplib2
# create 2 admin users
sudo maas createadmin testadmin
sudo maas createadmin secureadmin
# generate maas api keys
sudo maas apikey --username=testadmin > api_key
sudo maas apikey --username=secureadmin > api-key-file
# setup nginx proxy
sudo apt update
sudo apt install nginx
touch /etc/nginx/sites-available/maas-https-default
# contents of maas-https-default
server {
listen 443 ssl http2;
server_name _;
ssl_certificate /home/ubuntu/localhost.crt;
ssl_certificate_key /home/ubuntu/localhost.key;
location / {
proxy_pass http://localhost:5240;
include /etc/nginx/proxy_params;
}
location /MAAS/ws {
proxy_pass http://127.0.0.1:5240/MAAS/ws;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
}
}
sudo service nginx restart
# make sure you can login to maas-cli without TLS
# by running this script
# this is for the non-tls user
# this goes into a script called maas-login.sh
touch maas-login.sh
sudo chmod +rwx maas-login.sh
#!/bin/sh
PROFILE=testadmin
API_KEY_FILE=/home/ubuntu/api_key
API_SERVER=127.0.0.1:5240
MAAS_URL=http://$API_SERVER/MAAS
maas login $PROFILE $MAAS_URL - < $API_KEY_FILE
sudo chmod +rwx https-maas.sh
# another script called https-maas.sh
# for the tls user
#!/bin/sh
PROFILE=secureadmin
API_KEY_FILE=/home/ubuntu/api-key-file
API_SERVER=127.0.0.1
MAAS_URL=https://$API_SERVER/MAAS
maas login $PROFILE $MAAS_URL - < $API_KEY_FILE
# try to login
./maas-login.sh
cd /etc/nginx/sites-enabled
sudo touch maas-https-default
#example nginx config for maas https
server {
listen 443 ssl http2;
server_name _;
ssl_certificate /home/ubuntu/localhost.crt;
ssl_certificate_key /home/ubuntu/localhost.key;
location / {
proxy_pass http://localhost:5240;
include /etc/nginx/proxy_params;
}
location /MAAS/ws {
proxy_pass http://127.0.0.1:5240/MAAS/ws;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
}
}
# create link
sudo ln -s /etc/nginx/sites-available/maas-https-default
/etc/nginx/sites-enabled
# look at errors
cat /var/log/maas/regiond.log
cat regiond.log | grep "Python-http"
*i
[Sts-sponsors] [Bug 1906720] Re: Fix the disable_ssl_certificate_validation option
** Description changed:
[Impact]
* On Bionic, MAAS CLI fails to work with apis over https with self-signed
certificates due to broken disable_ssl_certificate_validation option
with python 3.5 and later.
[Steps to Reproduce]
1. prepare a maas server (it doesn't have to be HA to reproduce)
2. prepare a set of certificate, key and ca-bundle
3. place a new conf in /etc/nginx/sites-enabled and `sudo systemctl
restart nginx`
4. add the ca certificates to the host
sudo mkdir /usr/share/ca-certificates/extra
sudo cp -v ca-bundle.crt /usr/share/ca-certificates/extra/
dpkg-reconfigure ca-certificates
5. login with a new profile over https url
6. if the certificate is not trusted by the root store, it fails to login
7. adding the '--insecure' flag should disable the certificate check
[Where problems could occur]
* Potential issues could happen if we disable certificate validation for
all TLS interactions, any connection https related.
* Should not break existing python3 versions.
* Should not affect previously working python2 versions.
[Other Info]
This change should fix the issue with python3, and you should be able
to connect with python2 as before.
python2 => python-httplib2_0.9.2+dfsg-1ubuntu0.3_all.deb
python3 => python3-httplib2_0.9.2+dfsg-1ubuntu0.3_all.deb
helpful urls:
https://maas.io/docs/deb/2.8/cli/installation
https://maas.io/docs/deb/2.8/cli/configuration-journey
https://maas.io/docs/deb/2.8/ui/configuration-journey
# create bionic VM/lxc container
lxc launch ubuntu:bionic lp1906720
# get source code from repo
pull-lp-source python-httplib2 bionic
# install maas-cli
apt-get install maas-cli
# install maas server
apt-get install maas
# init maas
sudo maas init
# answer questions
# generate self signed cert and key
openssl req -newkey rsa:4096 -x509 -sha256 -days 60 -nodes -out localhost.crt
-keyout localhost.key
# add certs
sudo cp -v test.crt /usr/share/ca-certificates/extra/
# add new cert to list
sudo dpkg-reconfigure ca-certificates
# select yes with spacebar
# save
# create api key files
touch api_key
touch api-key-file
# remove any packages with this
# or this python3-httplib2
apt-cache search python-httplib2
apt-get remove python-httplib2
apt-get remove python3-httplib2
# create 2 admin users
sudo maas createadmin testadmin
sudo maas createadmin secureadmin
# generate maas api keys
sudo maas apikey --username=testadmin > api_key
sudo maas apikey --username=secureadmin > api-key-file
+
+ # setup nginx proxy
+ sudo apt update
+ sudo apt install nginx
+ touch /etc/nginx/sites-available/maas-https-default
+ # contents of maas-https-default
+ server {
+ listen 443 ssl http2;
+
+ server_name _;
+ ssl_certificate /home/ubuntu/localhost.crt;
+ ssl_certificate_key /home/ubuntu/localhost.key;
+
+ location / {
+ proxy_pass http://localhost:5240;
+ include /etc/nginx/proxy_params;
+ }
+
+ location /MAAS/ws {
+ proxy_pass http://127.0.0.1:5240/MAAS/ws;
+ proxy_http_version 1.1;
+ proxy_set_header Upgrade $http_upgrade;
+ proxy_set_header Connection "Upgrade";
+ }
+ }
+
+
# make sure you can login to maas-cli without TLS
# by running this script
# this is for the non-tls user
# this goes into a script called maas-login.sh
touch maas-login.sh
sudo chmod +rwx maas-login.sh
#!/bin/sh
PROFILE=testadmin
API_KEY_FILE=/home/ubuntu/api_key
API_SERVER=127.0.0.1:5240
MAAS_URL=http://$API_SERVER/MAAS
maas login $PROFILE $MAAS_URL - < $API_KEY_FILE
sudo chmod +rwx https-maas.sh
# another script called https-maas.sh
# for the tls user
#!/bin/sh
PROFILE=secureadmin
API_KEY_FILE=/home/ubuntu/api-key-file
API_SERVER=127.0.0.1
MAAS_URL=https://$API_SERVER/MAAS
- maas login --insecure $PROFILE $MAAS_URL - < $API_KEY_FILE
+ maas login $PROFILE $MAAS_URL - < $API_KEY_FILE
# try to login
./maas-login.sh
cd /etc/nginx/sites-enabled
sudo touch maas-https-default
#example nginx config for maas https
server {
listen 443 ssl http2;
server_name _;
ssl_certificate /home/ubuntu/localhost.crt;
ssl_certificate_key /home/ubuntu/localhost.key;
location / {
proxy_pass http://localhost:5240;
include /etc/nginx/proxy_params;
}
location /MAAS/ws {
proxy_pass http://127.0.0.1:5240/MAAS/ws;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
}
}
# create link
sudo ln -s /etc/nginx/sites-available/maas-https-default
/etc/nginx/sites-enabled
# look at errors
cat /var/log/maas/regiond.log
cat regiond.log | grep "Python-http"
*i didn't see any 404's though
2020-12-15 13:24:48 regiond: [info]
[Sts-sponsors] [Bug 1908473] Re: rsyslog-relp: imrelp module leaves sockets in CLOSE_WAIT state which leads to file descriptor leak
Hello Matthew, or anyone else affected,
Accepted librelp into focal-proposed. The package will build now and be
available at
https://launchpad.net/ubuntu/+source/librelp/1.5.0-1ubuntu2.20.04.2 in a
few hours, and then in the -proposed repository.
Please help us by testing this new package. See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed. Your feedback will aid us getting this
update out to other Ubuntu users.
If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, what testing has been
performed on the package and change the tag from verification-needed-
focal to verification-done-focal. If it does not fix the bug for you,
please add a comment stating that, and change the tag to verification-
failed-focal. In either case, without details of your testing we will
not be able to proceed.
Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in
advance for helping!
N.B. The updated package will be released to -updates after the bug(s)
fixed by this package have been verified and the package has been in
-proposed for a minimum of 7 days.
** Tags removed: verification-done verification-done-focal
** Tags added: verification-needed verification-needed-focal
--
You received this bug notification because you are a member of STS
Sponsors, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1908473
Title:
rsyslog-relp: imrelp module leaves sockets in CLOSE_WAIT state which
leads to file descriptor leak
Status in librelp package in Ubuntu:
Fix Released
Status in rsyslog package in Ubuntu:
Fix Released
Status in librelp source package in Focal:
Fix Committed
Status in rsyslog source package in Focal:
Won't Fix
Status in librelp source package in Groovy:
Fix Committed
Status in rsyslog source package in Groovy:
Fix Released
Status in librelp source package in Hirsute:
Fix Released
Status in rsyslog source package in Hirsute:
Fix Released
Bug description:
[Impact]
In recent versions of rsyslog and librelp, the imrelp module leaks
file descriptors due to a bug where it does not correctly close
sockets, and instead, leaves them in the CLOSE_WAIT state.
This causes rsyslogd on busy servers to eventually hit the limit of
maximum open files allowed, which locks rsyslogd up until it is
restarted.
A workaround is to restart rsyslogd every month or so to manually
close all of the open sockets.
Only users of the imrelp module are affected, and not rsyslog users in
general.
[Testcase]
Install the rsyslog-relp module like so:
$ sudo apt install rsyslog rsyslog-relp
Next, generate a working directory, and make a config file that loads
the relp module.
$ sudo mkdir /workdir
$ cat << EOF >> ./spool.conf
\$LocalHostName spool
\$AbortOnUncleanConfig on
\$PreserveFQDN on
global(
workDirectory="/workdir"
maxMessageSize="256k"
)
main_queue(queue.type="Direct")
module(load="imrelp")
input(
type="imrelp"
name="imrelp"
port="601"
ruleset="spool"
MaxDataSize="256k"
)
ruleset(name="spool" queue.type="direct") {
}
# Just so rsyslog doesn't whine that we do not have outputs
ruleset(name="noop" queue.type="direct") {
action(
type="omfile"
name="omfile"
file="/workdir/spool.log"
)
}
EOF
Verify that the config is valid, then start a rsyslog server.
$ sudo rsyslogd -f ./spool.conf -N9
$ sudo rsyslogd -f ./spool.conf -i /workdir/rsyslogd.pid
Fetch the rsyslogd PID and check for open files.
$ RLOGPID=$(cat /workdir/rsyslogd.pid)
$ sudo ls -l /proc/$RLOGPID/fd
total 0
lr-x-- 1 root root 64 Dec 17 01:22 0 -> /dev/urandom
lrwx-- 1 root root 64 Dec 17 01:22 1 -> 'socket:[41228]'
lrwx-- 1 root root 64 Dec 17 01:22 3 -> 'socket:[41222]'
lrwx-- 1 root root 64 Dec 17 01:22 4 -> 'socket:[41223]'
lrwx-- 1 root root 64 Dec 17 01:22 7 -> 'anon_inode:[eventpoll]'
We have 3 sockets open by default. Next, use netcat to open 100
connections:
$ for i in {1..100} ; do nc -z 127.0.0.1 601 ; done
Now check for open file descriptors, and there will be an extra 100 sockets
in the list:
$ sudo ls -l /proc/$RLOGPID/fd
https://paste.ubuntu.com/p/f6NQVNbZcR/
We can check the state of these sockets with:
$ ss -t
https://paste.ubuntu.com/p/7Ts2FbxJrg/
The listening sockets will be in CLOSE-WAIT, and the netcat sockets
will be in FIN-WAIT-2.
$ ss -t | grep CLOSE-WAIT | wc -l
100
If you install the test package available in the following ppa:
https://launchpad.net/~mruffell/+archive/ubuntu/sf299578-test
When you open connections with netcat, these will be closed properly,
and the file descriptor leak will be fixed.
[Where problems could occur]
If a regression
[Sts-sponsors] [Bug 1906720] Re: Fix the disable_ssl_certificate_validation option
** Tags removed: sts-sponsor-ddstreet
** Tags added: sts-sponsor-slashd
--
You received this bug notification because you are a member of STS
Sponsors, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1906720
Title:
Fix the disable_ssl_certificate_validation option
Status in python-httplib2 package in Ubuntu:
Fix Released
Status in python-httplib2 source package in Bionic:
In Progress
Status in python-httplib2 source package in Focal:
Fix Released
Status in python-httplib2 source package in Groovy:
Fix Released
Status in python-httplib2 source package in Hirsute:
Fix Released
Bug description:
[Impact]
* On Bionic, MAAS CLI fails to work with apis over https with self-signed
certificates due to broken disable_ssl_certificate_validation option
with python 3.5 and later.
[Steps to Reproduce]
1. prepare a maas server (it doesn't have to be HA to reproduce)
2. prepare a set of certificate, key and ca-bundle
3. place a new conf in /etc/nginx/sites-enabled and `sudo systemctl
restart nginx`
4. add the ca certificates to the host
sudo mkdir /usr/share/ca-certificates/extra
sudo cp -v ca-bundle.crt /usr/share/ca-certificates/extra/
dpkg-reconfigure ca-certificates
5. login with a new profile over https url
6. if the certificate is not trusted by the root store, it fails to login
7. adding the '--insecure' flag should disable the certificate check
[Where problems could occur]
* Potential issues could happen if we disable certificate validation for
all TLS interactions, any connection https related.
* Should not break existing python3 versions.
* Should not affect previously working python2 versions.
[Other Info]
This change should fix the issue with python3, and you should be able
to connect with python2 as before.
python2 => python-httplib2_0.9.2+dfsg-1ubuntu0.3_all.deb
python3 => python3-httplib2_0.9.2+dfsg-1ubuntu0.3_all.deb
helpful urls:
https://maas.io/docs/deb/2.8/cli/installation
https://maas.io/docs/deb/2.8/cli/configuration-journey
https://maas.io/docs/deb/2.8/ui/configuration-journey
# create bionic VM/lxc container
lxc launch ubuntu:bionic lp1906720
# get source code from repo
pull-lp-source python-httplib2 bionic
# install maas-cli
apt-get install maas-cli
# install maas server
apt-get install maas
# init maas
sudo maas init
# answer questions
# generate self signed cert and key
openssl req -newkey rsa:4096 -x509 -sha256 -days 60 -nodes -out localhost.crt
-keyout localhost.key
# add certs
sudo cp -v test.crt /usr/share/ca-certificates/extra/
# add new cert to list
sudo dpkg-reconfigure ca-certificates
# select yes with spacebar
# save
# create api key files
touch api_key
touch api-key-file
# remove any packages with this
# or this python3-httplib2
apt-cache search python-httplib2
apt-get remove python-httplib2
apt-get remove python3-httplib2
# create 2 admin users
sudo maas createadmin testadmin
sudo maas createadmin secureadmin
# generate maas api keys
sudo maas apikey --username=testadmin > api_key
sudo maas apikey --username=secureadmin > api-key-file
# make sure you can login to maas-cli without TLS
# by running this script
# this is for the non-tls user
# this goes into a script called maas-login.sh
touch maas-login.sh
sudo chmod +rwx maas-login.sh
#!/bin/sh
PROFILE=testadmin
API_KEY_FILE=/home/ubuntu/api_key
API_SERVER=127.0.0.1:5240
MAAS_URL=http://$API_SERVER/MAAS
maas login $PROFILE $MAAS_URL - < $API_KEY_FILE
sudo chmod +rwx https-maas.sh
# another script called https-maas.sh
# for the tls user
#!/bin/sh
PROFILE=secureadmin
API_KEY_FILE=/home/ubuntu/api-key-file
API_SERVER=127.0.0.1
MAAS_URL=https://$API_SERVER/MAAS
maas login --insecure $PROFILE $MAAS_URL - < $API_KEY_FILE
# try to login
./maas-login.sh
cd /etc/nginx/sites-enabled
sudo touch maas-https-default
#example nginx config for maas https
server {
listen 443 ssl http2;
server_name _;
ssl_certificate /home/ubuntu/localhost.crt;
ssl_certificate_key /home/ubuntu/localhost.key;
location / {
proxy_pass http://localhost:5240;
include /etc/nginx/proxy_params;
}
location /MAAS/ws {
proxy_pass http://127.0.0.1:5240/MAAS/ws;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
}
}
# create link
sudo ln -s /etc/nginx/sites-available/maas-https-default
/etc/nginx/sites-enabled
# look at errors
cat /var/log/maas/regiond.log
cat regiond.log | grep "Python-http"
*i didn't see any 404's though
2020-12-15 13:24:48 regiond: [info] 127.0.0.1 GET
/MAAS/api/2.0/users/?op=whoami HTTP/1.1 --> 200 OK (referrer: -; agent:
Python-httplib2/0.9.2 (gzip))
2020-12-15 13:24:48 regiond: [info] 127.0.0.1
[Sts-sponsors] [Bug 1908473] Re: rsyslog-relp: imrelp module leaves sockets in CLOSE_WAIT state which leads to file descriptor leak
Thanks again, Matthew.
I've reviewed and sponsored the new debdiff for Focal, with the existing
changes (so we have .20.04.1 and .2 in the same upload, going in focal-
proposed again.)
Further details added in bug 1912969.
--
You received this bug notification because you are a member of STS
Sponsors, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1908473
Title:
rsyslog-relp: imrelp module leaves sockets in CLOSE_WAIT state which
leads to file descriptor leak
Status in librelp package in Ubuntu:
Fix Released
Status in rsyslog package in Ubuntu:
Fix Released
Status in librelp source package in Focal:
Fix Committed
Status in rsyslog source package in Focal:
Won't Fix
Status in librelp source package in Groovy:
Fix Committed
Status in rsyslog source package in Groovy:
Fix Released
Status in librelp source package in Hirsute:
Fix Released
Status in rsyslog source package in Hirsute:
Fix Released
Bug description:
[Impact]
In recent versions of rsyslog and librelp, the imrelp module leaks
file descriptors due to a bug where it does not correctly close
sockets, and instead, leaves them in the CLOSE_WAIT state.
This causes rsyslogd on busy servers to eventually hit the limit of
maximum open files allowed, which locks rsyslogd up until it is
restarted.
A workaround is to restart rsyslogd every month or so to manually
close all of the open sockets.
Only users of the imrelp module are affected, and not rsyslog users in
general.
[Testcase]
Install the rsyslog-relp module like so:
$ sudo apt install rsyslog rsyslog-relp
Next, generate a working directory, and make a config file that loads
the relp module.
$ sudo mkdir /workdir
$ cat << EOF >> ./spool.conf
\$LocalHostName spool
\$AbortOnUncleanConfig on
\$PreserveFQDN on
global(
workDirectory="/workdir"
maxMessageSize="256k"
)
main_queue(queue.type="Direct")
module(load="imrelp")
input(
type="imrelp"
name="imrelp"
port="601"
ruleset="spool"
MaxDataSize="256k"
)
ruleset(name="spool" queue.type="direct") {
}
# Just so rsyslog doesn't whine that we do not have outputs
ruleset(name="noop" queue.type="direct") {
action(
type="omfile"
name="omfile"
file="/workdir/spool.log"
)
}
EOF
Verify that the config is valid, then start a rsyslog server.
$ sudo rsyslogd -f ./spool.conf -N9
$ sudo rsyslogd -f ./spool.conf -i /workdir/rsyslogd.pid
Fetch the rsyslogd PID and check for open files.
$ RLOGPID=$(cat /workdir/rsyslogd.pid)
$ sudo ls -l /proc/$RLOGPID/fd
total 0
lr-x-- 1 root root 64 Dec 17 01:22 0 -> /dev/urandom
lrwx-- 1 root root 64 Dec 17 01:22 1 -> 'socket:[41228]'
lrwx-- 1 root root 64 Dec 17 01:22 3 -> 'socket:[41222]'
lrwx-- 1 root root 64 Dec 17 01:22 4 -> 'socket:[41223]'
lrwx-- 1 root root 64 Dec 17 01:22 7 -> 'anon_inode:[eventpoll]'
We have 3 sockets open by default. Next, use netcat to open 100
connections:
$ for i in {1..100} ; do nc -z 127.0.0.1 601 ; done
Now check for open file descriptors, and there will be an extra 100 sockets
in the list:
$ sudo ls -l /proc/$RLOGPID/fd
https://paste.ubuntu.com/p/f6NQVNbZcR/
We can check the state of these sockets with:
$ ss -t
https://paste.ubuntu.com/p/7Ts2FbxJrg/
The listening sockets will be in CLOSE-WAIT, and the netcat sockets
will be in FIN-WAIT-2.
$ ss -t | grep CLOSE-WAIT | wc -l
100
If you install the test package available in the following ppa:
https://launchpad.net/~mruffell/+archive/ubuntu/sf299578-test
When you open connections with netcat, these will be closed properly,
and the file descriptor leak will be fixed.
[Where problems could occur]
If a regression were to occur, it would be limited to users of the
imrelp module, which is a part of the rsyslogd-relp package, and
depends on librelp.
rsyslog-relp is not part of a default installation of rsyslog, and is
opt in by changing a configuration file to enable imrelp.
The changes to rsyslog implement a testcase which exercises the problematic
code to ensure things are working as expected; this
can be enabled manually on build, and has been verified to pass (#7).
[Other]
Upstream bug list:
https://github.com/rsyslog/rsyslog/issues/4350
https://github.com/rsyslog/rsyslog/issues/4005
https://github.com/rsyslog/librelp/issues/188
https://github.com/rsyslog/librelp/pull/193
The following commits fix the problem:
rsyslogd
commit baee0bd5420649329793746f0daf87c4f59fe6a6
Author: Andre lorbach
Date: Thu Apr 9 13:00:35 2020 +0200
Subject: testbench: Add test for imrelp to check broken session handling.
Link:
https://github.com/rsyslog/rsyslog/commit/baee0bd5420649329793746f0daf87c4f59fe6a6
librelp
===
commit 7907c9c57f6ed94c8ce5a4e63c3c4e019f71cff0
Author: Andre lorbach
Date: Mon
