date:20191010

[Sts-sponsors] [Bug 1846138] Re: backport mod_reqtimeout with handshake support

2019-10-10 Thread Eric Desrochers

[VERIFICATION XENIAL - Part 2]
* Feedback #3:
"
I also tested and now it works perfectly, I can count the seconds which I 
configure for the handshake timeout and the connection is terminated exactly 
when the handshake timeout expires 

Great job!
"

-- 
You received this bug notification because you are a member of STS
Sponsors, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1846138

Title:
  backport mod_reqtimeout with handshake support

Status in apache2 package in Ubuntu:
  Fix Released
Status in apache2 source package in Xenial:
  Fix Committed
Status in apache2 source package in Bionic:
  Fix Released
Status in apache2 source package in Disco:
  Fix Released

Bug description:
  [Impact]

  When running TCP Defensics suite which sends corrupt packages towards
  vip__public port 443, the suite is hanging after the half suite
  because there are no free connections. The connections will be in
  state "established" ~ 2 hours.

  1.2. Detailed trouble description
  # ip netns exec haproxy netstat -npea | grep XXX.XXX.XXX.XXX | grep -i 
establish | grep 443
  tcp 0 0 XXX.XXX.XXX.XXX:443 YYY.YY.YYY.YY:2940 ESTABLISHED 115 81148003 
29817/haproxy
  tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:24979 ESTABLISHED 115 81802005 
29817/haproxy
  tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:19394 ESTABLISHED 115 81782263 
29817/haproxy
  tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:13931 ESTABLISHED 115 81752052 
29817/haproxy
  tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:12668 ESTABLISHED 115 81743719 
29817/haproxy
  tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:2961 ESTABLISHED 115 81139548 
29817/haproxy
  tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:8918 ESTABLISHED 115 81738132 
29817/haproxy
  tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:2957 ESTABLISHED 115 81148041 
29817/haproxy
  tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:10552 ESTABLISHED 115 81744903 
29817/haproxy

  This issue can be resolved by enabling the parameter(mod_reqtimeout).
  This parameter is available in apache 2.4.39 (released on 2019-04-01).

  [Test Case]

  This test case has been brought to my attention by an impacted user:
  "
  You must have an apache2 server, with an haproxy in front of it, and you 
initiate SSL connections with "nc" between 50 and 8000 connections and because 
the SSL connection process is never finished all those connections get stucked 
and never timeout.
  "

  Reproducer (Thanks to Szilard):
  https://pastebin.ubuntu.com/p/6Hk64CDc7H/

  [Regression Potential]

  * The backport already exist in Bionic/Disco (done by security team
  via the security channel)

  * It is also backported upstream into 2.4 (branch : 2.4.x)

  * It was tested pre-release by an impacted user, and the outcome was
  positive:

  "I have tested the below packages for enabling handshake
  parameter(mod_reqtimeout) in apache. Looks the package is working
  fine. "

  * Local autopkgtest inside qemu, revealed no issues:
  autopkgtest [12:09:48]:  summary
  duplicate-module-load PASS
  htcacheclean PASS
  ssl-passphrase   PASS
  chroot   PASS

  [Other Info]

  [Original description]
  Backport the handshake feature in mod_reqtimeout (in Apache 2.4.39) to Apache 
2.4.18.

  Lack of this feature was exhausting free connections when sent
  corrupted packets.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1846138/+subscriptions

-- 
Mailing list: https://launchpad.net/~sts-sponsors
Post to : sts-sponsors@lists.launchpad.net
Unsubscribe : https://launchpad.net/~sts-sponsors
More help   : https://help.launchpad.net/ListHelp

[Sts-sponsors] [Bug 1846138] Re: backport mod_reqtimeout with handshake support

2019-10-10 Thread Eric Desrochers

[VERIFICATION XENIAL]

* Feedback #1:

>From an impacted user:
"
They confirmed that from their perspective the test is OK, and the apache2 
packages are delivering expected result
"

* Feedback #2:
>From SustEng Mauricio (mfo):
"
The backport in xenial-proposed worked exactly as eoan 
(with the AcceptFilter bits mentioned in previous comment)
...
"

** Description changed:

  [Impact]
  
  When running TCP Defensics suite which sends corrupt packages towards
  vip__public port 443, the suite is hanging after the half suite because
  there are no free connections. The connections will be in state
  "established" ~ 2 hours.
  
  1.2. Detailed trouble description
  # ip netns exec haproxy netstat -npea | grep XXX.XXX.XXX.XXX | grep -i 
establish | grep 443
  tcp 0 0 XXX.XXX.XXX.XXX:443 YYY.YY.YYY.YY:2940 ESTABLISHED 115 81148003 
29817/haproxy
  tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:24979 ESTABLISHED 115 81802005 
29817/haproxy
  tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:19394 ESTABLISHED 115 81782263 
29817/haproxy
  tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:13931 ESTABLISHED 115 81752052 
29817/haproxy
  tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:12668 ESTABLISHED 115 81743719 
29817/haproxy
  tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:2961 ESTABLISHED 115 81139548 
29817/haproxy
  tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:8918 ESTABLISHED 115 81738132 
29817/haproxy
  tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:2957 ESTABLISHED 115 81148041 
29817/haproxy
  tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:10552 ESTABLISHED 115 81744903 
29817/haproxy
  
  This issue can be resolved by enabling the parameter(mod_reqtimeout).
  This parameter is available in apache 2.4.39 (released on 2019-04-01).
  
  [Test Case]
  
  This test case has been brought to my attention by an impacted user:
  "
  You must have an apache2 server, with an haproxy in front of it, and you 
initiate SSL connections with "nc" between 50 and 8000 connections and because 
the SSL connection process is never finished all those connections get stucked 
and never timeout.
  "
  
+ Reproducer (Thanks to Szilard):
+ https://pastebin.ubuntu.com/p/6Hk64CDc7H/
+ 
  [Regression Potential]
  
  * The backport already exist in Bionic/Disco (done by security team via
  the security channel)
  
  * It is also backported upstream into 2.4 (branch : 2.4.x)
  
  * It was tested pre-release by an impacted user, and the outcome was
  positive:
  
  "I have tested the below packages for enabling handshake
  parameter(mod_reqtimeout) in apache. Looks the package is working fine.
  "
  
  * Local autopkgtest inside qemu, revealed no issues:
  autopkgtest [12:09:48]:  summary
  duplicate-module-load PASS
  htcacheclean PASS
  ssl-passphrase   PASS
  chroot   PASS
  
- 
  [Other Info]
  
  [Original description]
  Backport the handshake feature in mod_reqtimeout (in Apache 2.4.39) to Apache 
2.4.18.
  
  Lack of this feature was exhausting free connections when sent corrupted
  packets.

** Tags removed: verification-needed-xenial
** Tags added: verification-done-xenial

-- 
You received this bug notification because you are a member of STS
Sponsors, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1846138

Title:
  backport mod_reqtimeout with handshake support

Status in apache2 package in Ubuntu:
  Fix Released
Status in apache2 source package in Xenial:
  Fix Committed
Status in apache2 source package in Bionic:
  Fix Released
Status in apache2 source package in Disco:
  Fix Released

Bug description:
  [Impact]

  When running TCP Defensics suite which sends corrupt packages towards
  vip__public port 443, the suite is hanging after the half suite
  because there are no free connections. The connections will be in
  state "established" ~ 2 hours.

  1.2. Detailed trouble description
  # ip netns exec haproxy netstat -npea | grep XXX.XXX.XXX.XXX | grep -i 
establish | grep 443
  tcp 0 0 XXX.XXX.XXX.XXX:443 YYY.YY.YYY.YY:2940 ESTABLISHED 115 81148003 
29817/haproxy
  tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:24979 ESTABLISHED 115 81802005 
29817/haproxy
  tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:19394 ESTABLISHED 115 81782263 
29817/haproxy
  tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:13931 ESTABLISHED 115 81752052 
29817/haproxy
  tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:12668 ESTABLISHED 115 81743719 
29817/haproxy
  tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:2961 ESTABLISHED 115 81139548 
29817/haproxy
  tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:8918 ESTABLISHED 115 81738132 
29817/haproxy
  tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:2957 ESTABLISHED 115 81148041 
29817/haproxy
  tcp 0 0 XXX.XXX.XXX.XXX:443 YY.YY.YYY.YY:10552 ESTABLISHED 115 81744903 
29817/haproxy

  This issue can be resolved by enabling the parameter(mod_reqtimeout).
  This parameter is available in apache 2.4.39 (released on 2019-04-01).

  [Test Case]

  This test case has been brought to my attention by an impacted user:
  "
  You must h

[Sts-sponsors] [Bug 1846787] Re: systemd-logind leaves leftover sessions and scope files

2019-10-10 Thread Dan Streetman

uploaded dbus and systemd to xenial queue, thanks!

-- 
You received this bug notification because you are a member of STS
Sponsors, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1846787

Title:
  systemd-logind leaves leftover sessions and scope files

Status in dbus package in Ubuntu:
  Fix Released
Status in systemd package in Ubuntu:
  Fix Released
Status in dbus source package in Xenial:
  In Progress
Status in systemd source package in Xenial:
  In Progress

Bug description:
  [Impact]
  Scope file leakage can cause SSH delays and reduce performance in systemd

  [Description]
  The current systemd-logind version present in Xenial can leave abandoned SSH
  sessions and scope files in cases where the host sees a lot of concurrent SSH
  connections. These leftover sessions can slow down systemd performance
  greatly, and can have an impact on sshd handling a great number of concurrent
  connections.

  To fix this issue, patches are needed in both dbus and systemd. These improve 
the
  performance of the communication between dbus and systemd, so that they can
  handle a better volume of events (e.g. SSH logins). All of those patches are
  already present from Bionic onwards, so we only need those fixes for Xenial.

  == Systemd ==
  Upstream patches:
  - core: use an AF_UNIX/SOCK_DGRAM socket for cgroup agent notification 
(d8fdc62037b5)

  $ git describe --contains d8fdc62037b5
  v230~71^2~2

  $ rmadison systemd
   systemd | 229-4ubuntu4 | xenial  | source, ...
   systemd | 229-4ubuntu21.21 | xenial-security | source, ...
   systemd | 229-4ubuntu21.22 | xenial-updates  | source, ... <
   systemd | 237-3ubuntu10| bionic  | source, ...
   systemd | 237-3ubuntu10.29 | bionic-security | source, ...
   systemd | 237-3ubuntu10.29 | bionic-updates  | source, ...
   systemd | 237-3ubuntu10.31 | bionic-proposed | source, ...

  == DBus ==
  Upstream patches:
  - Only read one message at a time if there are fds pending (892f084eeda0)
  - bus: Fix timeout restarts  (529600397bca)
  - DBusMainLoop: ensure all required timeouts are restarted (446b0d9ac75a)

  $ git describe --contains 892f084eeda0 529600397bca 446b0d9ac75a
  dbus-1.11.10~44
  dbus-1.11.10~45
  dbus-1.11.16~2

  $ rmadison dbus
   dbus | 1.10.6-1ubuntu3| xenial   | source, ...
   dbus | 1.10.6-1ubuntu3.4  | xenial-security  | source, ...
   dbus | 1.10.6-1ubuntu3.4  | xenial-updates   | source, ... <
   dbus | 1.12.2-1ubuntu1| bionic   | source, ...
   dbus | 1.12.2-1ubuntu1.1  | bionic-security  | source, ...
   dbus | 1.12.2-1ubuntu1.1  | bionic-updates   | source, ...

  [Test Case]
  1) Simulate a lot of concurrent SSH connections with e.g. a for loop:
  multipass@xenial-logind:~$ for i in {1..1000}; do sleep 0.1; ssh localhost 
sleep 1 & done

  2) Check for leaked sessions in /run/systemd/system/:
  multipass@xenial-logind:~$ ls -ld /run/systemd/system/session-*.scope*
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-103.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-104.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-105.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-106.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-110.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-111.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-112.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-113.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-114.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-115.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-116.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-117.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-118.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-119.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-120.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-121.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-122.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-123.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-126.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-131.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-134.scope.d
  ...

  [Regression Potential]
  As the patches change the communication socket between dbus and systemd, 
possible regressions could cause systemd to not be notified of dbus events and 
vice-versa. We could see units not getting started properly, and communication 
between different services break down (e.g. between sy

[Sts-sponsors] [Bug 1846787] Re: systemd-logind leaves leftover sessions and scope files

2019-10-10 Thread Dan Streetman

The patches are on the medium-to-large size, but I have reviewed them
and as far as I can tell they appear correct.  The systemd patch is
needed to fix the cgroup-agent from overrunning the dbus socket
connection queue, and the dbus patches are needed to prevent a highly
loaded dbus message queue from timing out a long queue of incoming
messages.

-- 
You received this bug notification because you are a member of STS
Sponsors, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1846787

Title:
  systemd-logind leaves leftover sessions and scope files

Status in dbus package in Ubuntu:
  Fix Released
Status in systemd package in Ubuntu:
  Fix Released
Status in dbus source package in Xenial:
  In Progress
Status in systemd source package in Xenial:
  In Progress

Bug description:
  [Impact]
  Scope file leakage can cause SSH delays and reduce performance in systemd

  [Description]
  The current systemd-logind version present in Xenial can leave abandoned SSH
  sessions and scope files in cases where the host sees a lot of concurrent SSH
  connections. These leftover sessions can slow down systemd performance
  greatly, and can have an impact on sshd handling a great number of concurrent
  connections.

  To fix this issue, patches are needed in both dbus and systemd. These improve 
the
  performance of the communication between dbus and systemd, so that they can
  handle a better volume of events (e.g. SSH logins). All of those patches are
  already present from Bionic onwards, so we only need those fixes for Xenial.

  == Systemd ==
  Upstream patches:
  - core: use an AF_UNIX/SOCK_DGRAM socket for cgroup agent notification 
(d8fdc62037b5)

  $ git describe --contains d8fdc62037b5
  v230~71^2~2

  $ rmadison systemd
   systemd | 229-4ubuntu4 | xenial  | source, ...
   systemd | 229-4ubuntu21.21 | xenial-security | source, ...
   systemd | 229-4ubuntu21.22 | xenial-updates  | source, ... <
   systemd | 237-3ubuntu10| bionic  | source, ...
   systemd | 237-3ubuntu10.29 | bionic-security | source, ...
   systemd | 237-3ubuntu10.29 | bionic-updates  | source, ...
   systemd | 237-3ubuntu10.31 | bionic-proposed | source, ...

  == DBus ==
  Upstream patches:
  - Only read one message at a time if there are fds pending (892f084eeda0)
  - bus: Fix timeout restarts  (529600397bca)
  - DBusMainLoop: ensure all required timeouts are restarted (446b0d9ac75a)

  $ git describe --contains 892f084eeda0 529600397bca 446b0d9ac75a
  dbus-1.11.10~44
  dbus-1.11.10~45
  dbus-1.11.16~2

  $ rmadison dbus
   dbus | 1.10.6-1ubuntu3| xenial   | source, ...
   dbus | 1.10.6-1ubuntu3.4  | xenial-security  | source, ...
   dbus | 1.10.6-1ubuntu3.4  | xenial-updates   | source, ... <
   dbus | 1.12.2-1ubuntu1| bionic   | source, ...
   dbus | 1.12.2-1ubuntu1.1  | bionic-security  | source, ...
   dbus | 1.12.2-1ubuntu1.1  | bionic-updates   | source, ...

  [Test Case]
  1) Simulate a lot of concurrent SSH connections with e.g. a for loop:
  multipass@xenial-logind:~$ for i in {1..1000}; do sleep 0.1; ssh localhost 
sleep 1 & done

  2) Check for leaked sessions in /run/systemd/system/:
  multipass@xenial-logind:~$ ls -ld /run/systemd/system/session-*.scope*
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-103.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-104.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-105.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-106.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-110.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-111.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-112.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-113.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-114.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-115.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-116.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-117.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-118.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-119.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-120.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-121.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-122.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-123.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-126.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-131.scope.d
  drwxr-xr-x 2 root root 160 Oct 4 15:34 /run/systemd/system/session-134.scope.d
  ...

  [Regression Pot

[Sts-sponsors] [Bug 1846138] Re: backport mod_reqtimeout with handshake support

[Sts-sponsors] [Bug 1846138] Re: backport mod_reqtimeout with handshake support

[Sts-sponsors] [Bug 1846787] Re: systemd-logind leaves leftover sessions and scope files

[Sts-sponsors] [Bug 1846787] Re: systemd-logind leaves leftover sessions and scope files

4 matches

Site Navigation

Mail list logo

Footer information