Public bug reported:

[Impact]

In recent versions of rsyslog and librelp, the imrelp module leaks file
descriptors due to a bug where it does not correctly close sockets, and
instead, leaves them in the CLOSE_WAIT state.

This causes rsyslogd on busy servers to eventually hit the limit of
maximum open files allowed, which locks rsyslogd up until it is
restarted.

A workaround is to restart rsyslogd every month or so to manually close
all of the open sockets.

Only users of the imrelp module are affected, and not rsyslog users in
general.

[Testcase]

Install the rsyslog-relp module like so:

$ sudo apt install rsyslog rsyslog-relp

Next, generate a working directory, and make a config file that loads
the relp module.

$ sudo mkdir /workdir
$ cat << EOF >> ./spool.conf
\$LocalHostName spool
\$AbortOnUncleanConfig on
\$PreserveFQDN on

global(
    workDirectory="/workdir"
    maxMessageSize="256k"
)

main_queue(queue.type="Direct")
module(load="imrelp")
input(
    type="imrelp"
    name="imrelp"
    port="601"
    ruleset="spool"
    MaxDataSize="256k"
)

ruleset(name="spool" queue.type="direct") {
}

# Just so rsyslog doesn't whine that we do not have outputs
ruleset(name="noop" queue.type="direct") {
    action(
        type="omfile"
        name="omfile"
        file="/workdir/spool.log"
    )
}
EOF

Verify that the config is valid, then start a rsyslog server.

$ sudo rsyslogd -f ./spool.conf -N9
$ sudo rsyslogd -f ./spool.conf -i /workdir/rsyslogd.pid

Fetch the rsyslogd PID and check for open files.

$ RLOGPID=$(cat /workdir/rsyslogd.pid)
$ sudo ls -l /proc/$RLOGPID/fd
total 0
lr-x------ 1 root root 64 Dec 17 01:22 0 -> /dev/urandom
lrwx------ 1 root root 64 Dec 17 01:22 1 -> 'socket:[41228]'
lrwx------ 1 root root 64 Dec 17 01:22 3 -> 'socket:[41222]'
lrwx------ 1 root root 64 Dec 17 01:22 4 -> 'socket:[41223]'
lrwx------ 1 root root 64 Dec 17 01:22 7 -> 'anon_inode:[eventpoll]'

We have 3 sockets open by default. Next, use netcat to open 100
connections:

$ for i in {1..100} ; do nc -z 127.0.0.1 601 ; done

Now check for open file descriptors, and there will be an extra 100 sockets
in the list:

$ sudo ls -l /proc/$RLOGPID/fd

https://paste.ubuntu.com/p/f6NQVNbZcR/

We can check the state of these sockets with:

$ ss -t

https://paste.ubuntu.com/p/7Ts2FbxJrg/

The listening sockets will be in CLOSE-WAIT, and the netcat sockets will
be in FIN-WAIT-2.

If you install the test package available in the following ppa:

https://launchpad.net/~mruffell/+archive/ubuntu/sf299578-test

When you open connections with netcat, these will be closed properly,
and the file descriptor leak will be fixed.

[Where problems could occur]

If a regression were to occur, it would be limited to users of the
imrelp module, which is a part of the rsyslogd-relp package, and depends
on librelp.

rsyslog-relp is not part of a default installation of rsyslog, and is
opt in by changing a configuration file to enable imrelp.

The changes to rsyslog implement a testcase which exercises the
problematic code to ensure things are working as expected, and should
run during autopkgtest time.

[Other]

Upstream bug list:

https://github.com/rsyslog/rsyslog/issues/4350
https://github.com/rsyslog/rsyslog/issues/4005
https://github.com/rsyslog/librelp/issues/188

The following commits fix the problem:

rsyslogd
========

commit baee0bd5420649329793746f0daf87c4f59fe6a6
Author: Andre lorbach <alorb...@adiscon.com>
Date:   Thu Apr 9 13:00:35 2020 +0200
Subject: testbench: Add test for imrelp to check broken session handling.
Link: 
https://github.com/rsyslog/rsyslog/commit/baee0bd5420649329793746f0daf87c4f59fe6a6

librelp
=======

commit 7907c9c57f6ed94c8ce5a4e63c3c4e019f71cff0
Author: Andre lorbach <alorb...@adiscon.com>
Date:   Mon May 11 14:59:55 2020 +0200
Subject: fix memory leak on session break.
Link: 
https://github.com/rsyslog/librelp/commit/7907c9c57f6ed94c8ce5a4e63c3c4e019f71cff0

commit 4a6ad8637c244fd3a1caeb9a93950826f58e956a
Author: Andre lorbach <alorb...@adiscon.com>
Date:   Wed Apr 8 15:55:32 2020 +0200
Subject: replsess: fix double free of sendbuf in some cases.
Link: 
https://github.com/rsyslog/librelp/commit/4a6ad8637c244fd3a1caeb9a93950826f58e956a

** Affects: librelp (Ubuntu)
     Importance: Medium
     Assignee: Matthew Ruffell (mruffell)
         Status: In Progress

** Affects: rsyslog (Ubuntu)
     Importance: Medium
     Assignee: Matthew Ruffell (mruffell)
         Status: In Progress

** Affects: librelp (Ubuntu Focal)
     Importance: Medium
     Assignee: Matthew Ruffell (mruffell)
         Status: In Progress

** Affects: rsyslog (Ubuntu Focal)
     Importance: Medium
     Assignee: Matthew Ruffell (mruffell)
         Status: In Progress

** Affects: librelp (Ubuntu Groovy)
     Importance: Medium
     Assignee: Matthew Ruffell (mruffell)
         Status: In Progress

** Affects: rsyslog (Ubuntu Groovy)
     Importance: Medium
     Assignee: Matthew Ruffell (mruffell)
         Status: In Progress

** Affects: librelp (Ubuntu Hirsute)
     Importance: Medium
     Assignee: Matthew Ruffell (mruffell)
         Status: In Progress

** Affects: rsyslog (Ubuntu Hirsute)
     Importance: Medium
     Assignee: Matthew Ruffell (mruffell)
         Status: In Progress


** Tags: sts

** Also affects: rsyslog (Ubuntu Hirsute)
   Importance: Undecided
       Status: New

** Also affects: rsyslog (Ubuntu Focal)
   Importance: Undecided
       Status: New

** Also affects: rsyslog (Ubuntu Groovy)
   Importance: Undecided
       Status: New

** Also affects: librelp (Ubuntu)
   Importance: Undecided
       Status: New

** Changed in: librelp (Ubuntu Focal)
       Status: New => In Progress

** Changed in: librelp (Ubuntu Groovy)
       Status: New => In Progress

** Changed in: librelp (Ubuntu Hirsute)
       Status: New => In Progress

** Changed in: rsyslog (Ubuntu Focal)
       Status: New => In Progress

** Changed in: rsyslog (Ubuntu Groovy)
       Status: New => In Progress

** Changed in: rsyslog (Ubuntu Hirsute)
       Status: New => In Progress

** Changed in: librelp (Ubuntu Focal)
   Importance: Undecided => Medium

** Changed in: librelp (Ubuntu Groovy)
   Importance: Undecided => Medium

** Changed in: librelp (Ubuntu Hirsute)
   Importance: Undecided => Medium

** Changed in: rsyslog (Ubuntu Focal)
   Importance: Undecided => Medium

** Changed in: rsyslog (Ubuntu Groovy)
   Importance: Undecided => Medium

** Changed in: rsyslog (Ubuntu Hirsute)
   Importance: Undecided => Medium

** Changed in: librelp (Ubuntu Focal)
     Assignee: (unassigned) => Matthew Ruffell (mruffell)

** Changed in: librelp (Ubuntu Groovy)
     Assignee: (unassigned) => Matthew Ruffell (mruffell)

** Changed in: librelp (Ubuntu Hirsute)
     Assignee: (unassigned) => Matthew Ruffell (mruffell)

** Changed in: rsyslog (Ubuntu Focal)
     Assignee: (unassigned) => Matthew Ruffell (mruffell)

** Changed in: rsyslog (Ubuntu Groovy)
     Assignee: (unassigned) => Matthew Ruffell (mruffell)

** Changed in: rsyslog (Ubuntu Hirsute)
     Assignee: (unassigned) => Matthew Ruffell (mruffell)

** Tags added: sts

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to rsyslog in Ubuntu.
https://bugs.launchpad.net/bugs/1908473

Title:
  rsyslog-relp: imrelp module leaves sockets in CLOSE_WAIT state which
  leads to file descriptor leak

Status in librelp package in Ubuntu:
  In Progress
Status in rsyslog package in Ubuntu:
  In Progress
Status in librelp source package in Focal:
  In Progress
Status in rsyslog source package in Focal:
  In Progress
Status in librelp source package in Groovy:
  In Progress
Status in rsyslog source package in Groovy:
  In Progress
Status in librelp source package in Hirsute:
  In Progress
Status in rsyslog source package in Hirsute:
  In Progress

Bug description:
  [Impact]

  In recent versions of rsyslog and librelp, the imrelp module leaks
  file descriptors due to a bug where it does not correctly close
  sockets, and instead, leaves them in the CLOSE_WAIT state.

  This causes rsyslogd on busy servers to eventually hit the limit of
  maximum open files allowed, which locks rsyslogd up until it is
  restarted.

  A workaround is to restart rsyslogd every month or so to manually
  close all of the open sockets.

  Only users of the imrelp module are affected, and not rsyslog users in
  general.

  [Testcase]

  Install the rsyslog-relp module like so:

  $ sudo apt install rsyslog rsyslog-relp

  Next, generate a working directory, and make a config file that loads
  the relp module.

  $ sudo mkdir /workdir
  $ cat << EOF >> ./spool.conf
  \$LocalHostName spool
  \$AbortOnUncleanConfig on
  \$PreserveFQDN on

  global(
      workDirectory="/workdir"
      maxMessageSize="256k"
  )

  main_queue(queue.type="Direct")
  module(load="imrelp")
  input(
      type="imrelp"
      name="imrelp"
      port="601"
      ruleset="spool"
      MaxDataSize="256k"
  )

  ruleset(name="spool" queue.type="direct") {
  }

  # Just so rsyslog doesn't whine that we do not have outputs
  ruleset(name="noop" queue.type="direct") {
      action(
          type="omfile"
          name="omfile"
          file="/workdir/spool.log"
      )
  }
  EOF

  Verify that the config is valid, then start a rsyslog server.

  $ sudo rsyslogd -f ./spool.conf -N9
  $ sudo rsyslogd -f ./spool.conf -i /workdir/rsyslogd.pid

  Fetch the rsyslogd PID and check for open files.

  $ RLOGPID=$(cat /workdir/rsyslogd.pid)
  $ sudo ls -l /proc/$RLOGPID/fd
  total 0
  lr-x------ 1 root root 64 Dec 17 01:22 0 -> /dev/urandom
  lrwx------ 1 root root 64 Dec 17 01:22 1 -> 'socket:[41228]'
  lrwx------ 1 root root 64 Dec 17 01:22 3 -> 'socket:[41222]'
  lrwx------ 1 root root 64 Dec 17 01:22 4 -> 'socket:[41223]'
  lrwx------ 1 root root 64 Dec 17 01:22 7 -> 'anon_inode:[eventpoll]'

  We have 3 sockets open by default. Next, use netcat to open 100
  connections:

  $ for i in {1..100} ; do nc -z 127.0.0.1 601 ; done

  Now check for open file descriptors, and there will be an extra 100 sockets
  in the list:

  $ sudo ls -l /proc/$RLOGPID/fd

  https://paste.ubuntu.com/p/f6NQVNbZcR/

  We can check the state of these sockets with:

  $ ss -t

  https://paste.ubuntu.com/p/7Ts2FbxJrg/

  The listening sockets will be in CLOSE-WAIT, and the netcat sockets
  will be in FIN-WAIT-2.

  If you install the test package available in the following ppa:

  https://launchpad.net/~mruffell/+archive/ubuntu/sf299578-test

  When you open connections with netcat, these will be closed properly,
  and the file descriptor leak will be fixed.

  [Where problems could occur]

  If a regression were to occur, it would be limited to users of the
  imrelp module, which is a part of the rsyslogd-relp package, and
  depends on librelp.

  rsyslog-relp is not part of a default installation of rsyslog, and is
  opt in by changing a configuration file to enable imrelp.

  The changes to rsyslog implement a testcase which exercises the
  problematic code to ensure things are working as expected, and should
  run during autopkgtest time.

  [Other]

  Upstream bug list:

  https://github.com/rsyslog/rsyslog/issues/4350
  https://github.com/rsyslog/rsyslog/issues/4005
  https://github.com/rsyslog/librelp/issues/188

  The following commits fix the problem:

  rsyslogd
  ========

  commit baee0bd5420649329793746f0daf87c4f59fe6a6
  Author: Andre lorbach <alorb...@adiscon.com>
  Date:   Thu Apr 9 13:00:35 2020 +0200
  Subject: testbench: Add test for imrelp to check broken session handling.
  Link: 
https://github.com/rsyslog/rsyslog/commit/baee0bd5420649329793746f0daf87c4f59fe6a6

  librelp
  =======

  commit 7907c9c57f6ed94c8ce5a4e63c3c4e019f71cff0
  Author: Andre lorbach <alorb...@adiscon.com>
  Date:   Mon May 11 14:59:55 2020 +0200
  Subject: fix memory leak on session break.
  Link: 
https://github.com/rsyslog/librelp/commit/7907c9c57f6ed94c8ce5a4e63c3c4e019f71cff0

  commit 4a6ad8637c244fd3a1caeb9a93950826f58e956a
  Author: Andre lorbach <alorb...@adiscon.com>
  Date:   Wed Apr 8 15:55:32 2020 +0200
  Subject: replsess: fix double free of sendbuf in some cases.
  Link: 
https://github.com/rsyslog/librelp/commit/4a6ad8637c244fd3a1caeb9a93950826f58e956a

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/librelp/+bug/1908473/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to     : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

Reply via email to