Re: [Bug 1842037] Re: SUNRPC: Use after free when GSSD credentials are invalid causes oops

[Frank Burkhardt]

Hi,

- Ursprüngliche Mail -
> Von: "Bug 1842037" <1842...@bugs.launchpad.net>
> An: "burk" 
> Gesendet: Montag, 18. November 2019 07:41:30
> Betreff: [Bug 1842037] Re: SUNRPC: Use after free when GSSD credentials are 
> invalid causes oops

> Hi Frank,
> 
> Thanks for testing the kernel in -proposed, it lets us mark the bug as
> verified so the patches make it into the release, which will be
> happening in about two weeks going by https://kernel.ubuntu.com/
> 
> Just to be clear, you tested the bionic-hwe kernel 5.0.0-37~18.04.1 from
> bionic -proposed?

root@styx:~ > uname -a
Linux styx 5.0.0-37-generic #40~18.04.1-Ubuntu SMP Thu Nov 14 12:06:39 UTC 2019 
x86_64 x86_64 x86_64 GNU/Linux

Yes, I'm certain.

> I have updated the tag, which in the future, can be done by clicking the
> pencil icon at the Tags: section, at the end of the summary at the top
> of the page, right before the first comment.

Found it.

> Thanks again for reporting the bugs and helping us test. There is
> nothing more to do now but wait for the SRU complete. The kernel will be
> released somewhere around the 2nd of December, give or take a few days
> in case any CVEs turn up.

Thank you very much.

Best,

Frank

-- 
Frank Burkhardt 
IT Dept., Max Planck Institute for Human Cognitive
and Brain Sciences, Leipzig, Germany

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1842037

Title:
  SUNRPC: Use after free when GSSD credentials are invalid causes oops

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842037/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Re: [Bug 1842037] Re: SUNRPC: Use after free when GSSD credentials are invalid causes oops

[Frank Burkhardt]

hi,

- Ursprüngliche Mail -
> Von: "Bug 1842037" <1842...@bugs.launchpad.net>
> An: "burk" 
> Gesendet: Freitag, 8. November 2019 00:37:27
> Betreff: [Bug 1842037] Re: SUNRPC: Use after free when GSSD credentials are 
> invalid causes oops

> Hi Frank,
> 
> Just giving you an update on the status of these bugs. The kernel team
> has reviewed the patches I submitted, and each set of patches have
> received two acks each, meaning they will be built into the next kernel
> update.

thank you very much. I managed to install the proposed kernel on several
machines and had a lot of users testing them successfully. However, I can't
find a way to add a tag to a bug. Same goes for bug 1828978 which is fixed
in the same kernel.

Best,

Frank

-- 
Frank Burkhardt 
IT Dept., Max Planck Institute for Human Cognitive
and Brain Sciences, Leipzig, Germany

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1842037

Title:
  SUNRPC: Use after free when GSSD credentials are invalid causes oops

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842037/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1828978] Re: NFSv4.1: Interrupted connections cause high bandwidth RPC ping-pong between client and server

[Frank Burkhardt]

I installed the patched kernel on 6 machines with several of my useds
hitting them hard. The problem can no longer be reproduced.

tag verification-done-disco

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1828978

Title:
  NFSv4.1: Interrupted connections cause high bandwidth RPC ping-pong
  between client and server

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1828978/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1842037] Re: SUNRPC: Use after free when GSSD credentials are invalid causes oops

[Frank Burkhardt]

I installed the patched kernel on 6 machines with several of my useds
hitting them hard. The problem can no longer be reproduced.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1842037

Title:
  SUNRPC: Use after free when GSSD credentials are invalid causes oops

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842037/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1842037] Re: SUNRPC: Use after free when GSSD credentials are invalid causes oops

[Frank Burkhardt]

tags:added: verification-done-disco

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1842037

Title:
  SUNRPC: Use after free when GSSD credentials are invalid causes oops

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842037/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Re: [Bug 1842037] Re: Oops when Kerberos credentials are invalid

[Frank Burkhardt]

Hi Matthew,

- Ursprüngliche Mail -
> Von: "Bug 1842037" <1842...@bugs.launchpad.net>
> An: "burk" 
> Gesendet: Samstag, 19. Oktober 2019 09:56:12
> Betreff: [Bug 1842037] Re: Oops when Kerberos credentials are invalid

> Hi Frank,
> 
> It has been 9 days, or slightly over a week since you applied the respun
> test kernel to your systems.

It looks very promising. We had no more problems on the test systems.

> Are your systems stable now?

Yes, they are.

> Are they suffering any symptoms of Bug 1842037 or Bug 1828978?

No, they're not.

> Is the test kernel more stable than regular released Ubuntu kernels? Do
> you think the patches you requested fix the problems you were having?

I do think so. Please integrated the patches.

Thank you very much.

Best regards,

Frank

-- 
Frank Burkhardt 
IT Dept., Max Planck Institute for Human Cognitive
and Brain Sciences, Leipzig, Germany

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1842037

Title:
  Oops when Kerberos credentials are invalid

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842037/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Re: [Bug 1842037] Re: Oops when Kerberos credentials are invalid

[Frank Burkhardt]

Hi,

- Ursprüngliche Mail -
> Von: "Bug 1842037" <1842...@bugs.launchpad.net>
> An: "burk" 
> Gesendet: Donnerstag, 3. Oktober 2019 01:22:16
> Betreff: [Bug 1842037] Re: Oops when Kerberos credentials are invalid

> Hi Frank,
> 
> I have tested the new test kernel and it boots successfully.
> 
> Please note this kernel is NOT SUPPORTED by Canonical, and is for
> TESTING PURPOSES ONLY. ONLY Install in a dedicated test environment.
> 
> Instructions to install (on a bionic system):
> 
> 1) sudo add-apt-repository ppa:mruffell/sf241068-test
> 2) sudo apt-get update
> 3) sudo apt install linux-image-unsigned-5.0.0-27-generic
> linux-headers-5.0.0-27-generic linux-headers-5.0.0-27
> linux-modules-5.0.0-27-generic linux-modules-extra-5.0.0-27-generic
> 4) reboot
> 5) uname -rv
> 5.0.0-27-generic #28~18.04.1+hf241068v20191002b1-Ubuntu SMP Wed Oct 2 04:04:05
> UTC
> 
> If you get different output from uname -rv, you might be booted into the
> wrong kernel, and you will need to change your grub configuration. Let
> me know if you need help with this.
> 
> Otherwise, this test kernel contains the requested patches for Bug
> 1842037 and Bug 1829878.

Thank you very much.

> Let me know how this test kernel goes, and if it fixes your NFS
> problems. If it doesn't, it would be good to upload some logs so we can
> have a better look.

I applied the test kernel to 3 of our production systems (since there's no
way to trigger either bug artificially). I know I shouldn't but I'm willing
to take the heat for problems arising from doing that.

I'll write you as soon as there's a hint of a problem on these machines.

Best,

Frank

-- 
Frank Burkhardt 
IT Dept., Max Planck Institute for Human Cognitive
and Brain Sciences, Leipzig, Germany

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1842037

Title:
  Oops when Kerberos credentials are invalid

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842037/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Re: [Bug 1842037] Re: Oops when Kerberos credentials are invalid

[Frank Burkhardt]

Hi Matthew,

- Ursprüngliche Mail -
> Von: "Bug 1842037" <1842...@bugs.launchpad.net>
> An: "burk" 
> Gesendet: Donnerstag, 26. September 2019 01:27:15
> Betreff: [Bug 1842037] Re: Oops when Kerberos credentials are invalid

> Hi Frank,
> 
> Just checking in to see how the test kernel is going. Does it fix your
> problem of the kernel crashing when users have invalid kerberos
> credentials?

I've not been able to reproduce the problem. The suggested method of
triggering it doesn't seem to work because:

   * If there's no kerberos identity (just a local user), the server
 won't let me enter the mounted folder.
   * If the local user is not known network wide but has a Kerberos
 identity and the remote idmapd cannot resolve it, it's mapped to
 nobody on the server.
   * No Oopps in both cases.

> Did you try it on the original server which crashes frequently? Has it
> made things more stable?

I tried on several of the affected servers. They are not more stable but
suffer from different NFS related problems now.

> Did you have an opportunity to try the reproducer I linked you in my
> previous message?
> 
> Let me know how things are going, when you have had a chance to test the
> kernel.

NFS is still unstable but the problem seems to be in GSSD now plus in
Bug 1828978 . However, I can't tell you, if 1828978 happens in Xenial, only.
Both problems are triggered relatively seldom and only cause headaches because
they happen on very crowded computer servers. If in doubt, the admins here
try to get the servers running again ASAP which makes analysis very difficult.

However, bug 1842037 is very clearly visible in the logs so I'm quite sure,
I didn't miss it. I think the best solution for now is to park the ticket on
your side and I'll provide feedback as soon as it happens again.

Thank you very much.

Best,

Frank

-- 
Frank Burkhardt 
IT Dept., Max Planck Institute for Human Cognitive
and Brain Sciences, Leipzig, Germany

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1842037

Title:
  Oops when Kerberos credentials are invalid

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842037/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Re: [Bug 1842037] Re: Oops when Kerberos credentials are invalid

[Frank Burkhardt]

Hi,

- Ursprüngliche Mail -
> Von: "Bug 1842037" <1842...@bugs.launchpad.net>
> An: "burk" 
> Gesendet: Mittwoch, 11. September 2019 02:44:11
> Betreff: [Bug 1842037] Re: Oops when Kerberos credentials are invalid

> Hi Frank,
> 
> The test kernel has built, and I have tested it to ensure it boots, but
> I now need you to verify that the backported patches fix the issue that
> you are having.

unfortunately, I have been unable to reproduce the problem in my test
environment until now. I attached the stack trace which I took when
opening the bug. I'll open the compute server with the patched kernel to some
test users now.

Best,

Frank

-- 
Frank Burkhardt 
IT Dept., Max Planck Institute for Human Cognitive
and Brain Sciences, Leipzig, Germany


** Attachment added: "bug-sunrpc.txt"
   
https://bugs.launchpad.net/bugs/1842037/+attachment/5288913/+files/bug-sunrpc.txt

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1842037

Title:
  Oops when Kerberos credentials are invalid

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842037/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Re: [Bug 1842037] Re: Oops when Kerberos credentials are invalid

[Frank Burkhardt]

Hi Matthey,

- Ursprüngliche Mail -
> Von: "Matthew Ruffell" <1842...@bugs.launchpad.net>
> An: "burk" 
> Gesendet: Mittwoch, 11. September 2019 02:44:11
> Betreff: [Bug 1842037] Re: Oops when Kerberos credentials are invalid

> Hi Frank,
> 
> The test kernel has built, and I have tested it to ensure it boots, but
> I now need you to verify that the backported patches fix the issue that
> you are having.
> 
> Can you please install the test kernel onto a system which is having
> problems with NFS clients that are connecting with invalid kerberos
> credentials?
> 
> Please note this kernel is NOT SUPPORTED by Canonical, and is for
> TESTING PURPOSES ONLY. ONLY Install in a dedicated test environment.

thank you for the quick response. I'm deploying the kernel to
a test system right now.

Best regards,

Frank

-- 
Frank Burkhardt 
IT Dept., Max Planck Institute for Human Cognitive
and Brain Sciences, Leipzig, Germany

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1842037

Title:
  Oops when Kerberos credentials are invalid

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842037/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1842037] [NEW] Oops when Kerberos credentials are invalid

[Frank Burkhardt]

Public bug reported:

There's a bug in Linux Kernel 5.0 which is triggered by invalid
credentials when the NFS clients is trying to aquire them via GSSD. This
affects NFS-Shares that are protected by krb5* security. They become
unusable until the system is re-booted. The problem is quite severe on
terminal servers with multiple users - some of them not caring about
refreshing their kerberos tickets.

A fix is available here:



** Affects: linux-meta-hwe (Ubuntu)
 Importance: Undecided
 Status: New

** Description changed:

- There's a bug in Linux Kernels before 5.0 which is triggered by invalid
+ There's a bug in Linux Kernel 5.0 which is triggered by invalid
  credentials when the NFS clients is trying to aquire them via GSSD. This
  affects NFS-Shares that are protected by krb5* security. They become
  unusable until the system is re-booted. The problem is quite severe on
  terminal servers with multiple users - some of them not caring about
  refreshing their kerberos tickets.
  
  A fix is available here:
  
  


-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1842037

Title:
  Oops when Kerberos credentials are invalid

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-meta-hwe/+bug/1842037/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1837526] [NEW] Restarting services puts puppet into restart-loop

[Frank Burkhardt]

Public bug reported:

Puppet is a configuration and software management system that can be
used e.g. for installing and upgrading debian packages, modifying
configuration files, etc. Puppet will call apt-get/dpkg when needed to
handle packages.

When libssl1.1 is installed/upgraded and the installation process is
controlled by Puppet, the automatic re-start of ssl-dependant services
Puts puppet into a re-start loop:

   1. libssl1.1 's postinstall script re-starts puppet via systemd (systemctl 
restart puppet.service)
   2. systemd will terminate all processes associated with the Puppet service
   3. dpkg is among these processes since it's called by Puppet and inherits 
its cgroup-memberships.
   4. libssl1.1 's postinstall script will never succeed since it's killed by 
systemd
   5. puppet re-starts
   6. puppet runs "dpkg --configura -a" 
   7. libssl1.1 is unconfigured, the postinst script is run
   8. goto 1.

Puppet should be removed from hardcoded list of services to be re-
started in libssl1.1's postinst script. A patch to do so is attached.

** Affects: openssl (Ubuntu)
 Importance: Undecided
 Status: New


** Tags: bionic xenial

** Attachment added: "Removes puppet from list of to-be-re-restarted services"
   
https://bugs.launchpad.net/bugs/1837526/+attachment/5278651/+files/libssl.diff

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1837526

Title:
  Restarting services puts puppet into restart-loop

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1837526/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1831423] [NEW] Bogus warning message in Net::Ping in Bionic

[Frank Burkhardt]

Public bug reported:

In Ubuntu Bionic's Ping.pm a test of the used socket's version is done
numerically although the Socket.pm's version contains a special
character ("_") which causes this warning message:

Argument "2.020_03" isn't numeric in numeric ge (>=) at
/usr/share/perl/5.26/Net/Ping.pm line 1801,  line 755.


Since Socket.pm is part of the same package, the test is not really useful 
anymore. I suggest removing it quick'n dirty via the attached patch to get rid 
of the warning while conserving the "old way".

** Affects: perl (Ubuntu)
 Importance: Undecided
 Status: New


** Tags: bionic

** Patch added: "Fix to remove bogus socket version warning in Ping.pm"
   
https://bugs.launchpad.net/bugs/1831423/+attachment/5268480/+files/ping-warning-fix.diff

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1831423

Title:
  Bogus warning message in Net::Ping in Bionic

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/perl/+bug/1831423/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1828978] [NEW] NFS connections block while causing a high-bandwidth RPC-pingpong between client and server

[Frank Burkhardt]

Public bug reported:

There's a bug in kernels before Linux 5.0 that affects NFS 4.1 connections. The 
bug presents itself like this:
   * On NFS clients: Attempts to access mounted NFS shares associated with the 
affected server
 block indefinitely.
   * On the network: A storm of repeated RPCs between NFS client and server 
uses a lot
 of bandwidth. Each RPC is acknoledged by the server with an 
NFS4ERR_SEQ_MISORDERED error.
   * Other NFS clients connected to the same NFS server: Performance drops 
dramatically.

A patch is available to fix this problem:

<https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=3453d5708b33efe76f40eca1c0ed60923094b971>

Is is possible to integrate the patch into the 4.18 kernel series?
I'm using Ubuntu 18.04.2 LTS as NFS client an server.

Thank you.

Best regards,

Frank Burkhardt

** Affects: linux-meta-hwe (Ubuntu)
 Importance: Undecided
 Status: New


** Tags: bionic

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1828978

Title:
  NFS connections block while causing a high-bandwidth RPC-pingpong
  between client and server

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-meta-hwe/+bug/1828978/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1713149] Re: resolv.conf symlink is broken after clean debootstrap

[Frank Burkhardt]

Hi,

(installing Bionic here)

a possible solution is to make sure, /target/run is already mounted
(bind-mount to /run) when debootstrap is running which will make systemd
create its resolv.conf copy in the correct /run-filesystem. Otherwise,
it would write to the /target-partition and in-target would hide the
stuff in there when doint its bind-mounts.

For now I create (and make executable) the script /usr/lib/base-
installer.d/41init-run via early_command:

#!/bin/sh
mkdir -p /target/run
mount --bind /run /target/run

IMHO this should be done by the base-installer.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1713149

Title:
  resolv.conf symlink is broken after clean debootstrap

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/resolvconf/+bug/1713149/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs