*** This bug is a duplicate of bug 1759920 ***
https://bugs.launchpad.net/bugs/1759920
Marking this bug as a duplicate of bug #1759920, where deployment of the
fix will be tracked.
** This bug has been marked a duplicate of bug 1759920
intel-microcode 3.20180312.0 causes lockup at login
Thanks @walterzjunior. Those kernel versions linked in comment #50 are
proceeding towards release regardless, but additional test results from
other people affected by this bug would be very much appreciated.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which
That's good news!
I have tested both kernels in multiple C5/M5 instances (more than 80)
with SSSD and AppArmor enabled in N.Virginia, and I could not face one
single failure. It seems that this new kernel version fixed the issue.
It would be good to get more feedbacks and push this kernel
Our next kernel version cycle includes the fix for the possibly-related
bug #1759920 (for both the Xenial-generic and linux-aws kernels). Those
kernels are available for testing now, hot off the builders. Please
test:
linux-aws:
I can confirm the suspicion for this issue to relate to Meltdown/Spectre
patching.
Running an Ubuntu 16.04 desktop in a virtual machine (VMware Workstation
12.5.9) on top of Windows 7.
The hardware is a Dell T1700 with an i7-4770, and it seems the issue arose
after BIOS and Microcode update.
This bug seems to be the closest match to what I'm experiencing. I'm
hitting similar trouble on desktop/graphical Ubuntu on some Dell
laptops.
Ubuntu 16.04 AMD64.
Happening on kernel versions 4.13.0-38 and 4.4.0-116.
I've tried version 4.4.0-117 as posted by @kamalmostafa - same result.
The
** Changed in: linux-aws (Ubuntu)
Status: In Progress => Confirmed
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1746806
Title:
sssd appears to crash AWS c5 and m5 instances, cause 100% CPU
Im seeing this issue on Ubuntu server 16.04.3 LTS in VM's under Hyper-V.
sssd is 1.13.4-1ubuntu1.10 amd64
kernal 4.4.0-87-generic x86_64 : Stable
kernal 4.4.0-116-generic x86_64 : Unstable
crashes randomly occur during boot after completing a realm join on a
stable kernel, or while performing
@kamalmostafa - Adding another datapoint: installed 4.4.0-117.141 on a
vm guest on ESXi 6.0 on a host running Intel Xeon CPU E5-2695 v4
(Broadwell) and the issue still occurs. Disabling the apparmor profile
for sssd allows boot to complete and restores sssd functionality.
--
You received this
@kamalmostafa - I have installed the suggested linux-
image-4.4.0-117-generic_4.4.0-117.141_amd64.deb and launched multiple M5
and C5 (more than 40) instances across multiple availability zones in N.
Virginia and unfortunately the issue still happening with this kernel.
--
You received this bug
@kamalmostafa please find attached
** Attachment added: "cpuinfo from Dell Optiplex 5040 i5-6500"
https://bugs.launchpad.net/cloud-images/+bug/1746806/+attachment/5078995/+files/cpuinfo
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to
Folks who can reproduce this reliably -- Please try this new
4.4.0-117.141 kernel:
https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa/+build/14450934
( linux-image-4.4.0-117-generic_4.4.0-117.141_amd64.deb )
--
You received this bug notification because you are a member of
@davidjmemmett- David, could we ask you attach a copy of your
/proc/cpuinfo file from that Optiplex?
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1746806
Title:
sssd appears to crash AWS c5 and
This issue is not limited to EC2. I'm running artful on a Dell Optiplex
5040 with an i5-6500 CPU (which is Skylake); I've just installed sssd
and hit the full system freeze. Even REISUB didn't work!
kernel 4.13.0-36-generic
sssd 1.15.3-2ubuntu1
apparmor 2.11.0-2ubuntu17.1
--
You received this
If it helps I am using Kernalcare so it pushes me to the latest version
which I am seeing might not be a good idea.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1746806
Title:
sssd appears to
I am having the same issue. I tested the C5 instance when it first came
out and it would randomly lock up at 100% CPU. I contacted AWS support
and they could not figure it out. They sent a message to the devs to see
if they could. So I switched back to c4 instances. I decided to try it
again
** Changed in: linux (Ubuntu)
Status: Confirmed => In Progress
** Changed in: linux (Ubuntu Xenial)
Status: Confirmed => In Progress
** Changed in: linux-aws (Ubuntu)
Status: Confirmed => In Progress
** Changed in: linux-aws (Ubuntu Xenial)
Status: Confirmed => In
Apparently not fixed (according to original reporter), so reopening.
** Changed in: linux (Ubuntu)
Status: Fix Released => Confirmed
** Changed in: linux (Ubuntu Xenial)
Status: Fix Released => Confirmed
** Changed in: linux-aws (Ubuntu Xenial)
Status: Fix Released =>
@davidjmemmett I tested the new kernel on one c5.xl yesterday and it
worked fine. Deployed the new kernel to all of our environments today
and we are seeing intermittent repro of the same behavior we saw in the
past (box fails to boot, no SSH available, CPU at 100%).
We reverted to the 20180109
I'm still experiencing this issue in eu-west-1 using an m5.large -
machine will not boot.
Kernel is 4.4.0-1052-aws
sssd is 1.13.4
apparmor is 2.10.95
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
** Changed in: cloud-images
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1746806
Title:
sssd appears to crash AWS c5 and m5 instances, cause 100% CPU
To
The fix for this issue has been released in the linux-aws and standard
Ubuntu kernels. Fixed versions are:
linux-aws (4.4.0-1052.61) xenial
linux-aws (4.4.0-1014.14) trusty
linux (4.4.0-116.140) xenial
** Changed in: linux (Ubuntu)
Status: Confirmed => Fix Released
** Changed in: linux
@kamalmostafa I installed that kernel package and it worked fine with
sssd running on a c5.xl
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1746806
Title:
sssd appears to crash AWS c5 and m5
** Changed in: cloud-images
Status: New => In Progress
** Changed in: cloud-images
Importance: Undecided => High
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1746806
Title:
sssd appears
I tried out the 1052 proposed kernel on a C5 instance that runs
sssd/apparmor and it did not lock up.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1746806
Title:
sssd appears to crash AWS c5 and
** Tags removed: kernel-key pti
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1746806
Title:
sssd appears to crash AWS c5 and m5 instances, cause 100% CPU
To manage notifications about this bug go
The linux-aws (4.4.0-1052.xx) release has been delayed by the merge of
additional necessary retpoline fixes. The packages have now been built
and are available in the -proposed archive; current ETA for final
release of this kernel is early next week. In the meantime, the .deb
package from
** Changed in: linux (Ubuntu Xenial)
Status: In Progress => Fix Committed
** Changed in: linux-aws (Ubuntu Xenial)
Status: In Progress => Fix Committed
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
@kamalmostafa: Is the latest kernel released? Your assistance is
appreciated
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1746806
Title:
sssd appears to crash AWS c5 and m5 instances, cause 100%
@Paul, I cannot provide an exact date, but it will most likely be within
the next five days.
For the record, we've decided to pull in an additional group of patches
so the next release linux-aws kernel will now be version
(4.4.0-1052.61). I'll post here once an official build of that becomes
@kamalmostafa: Do you all have a target date for when the new linux-aws
kernel (4.4.0-1051.60) will be released?
Thanks everyone for your help in quickly tracking down this issue.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
Thanks for the confirmation Paul, and thanks to all for the assistance
with testing and isolation.
We will be deploying this fix (the 'retpoline' patch set) in the next
version of the linux-aws kernel (4.4.0-1051.60) and the Xenial generic
kernel (4.4.0-113.136).
We'll post here once those
** Changed in: linux (Ubuntu)
Assignee: Joseph Salisbury (jsalisbury) => Kamal Mostafa (kamalmostafa)
** Changed in: linux-aws (Ubuntu)
Assignee: Joseph Salisbury (jsalisbury) => Kamal Mostafa (kamalmostafa)
** Also affects: linux (Ubuntu Xenial)
Importance: Undecided
Status:
@kamalmostafa I installed the rtp0 kernel and verified it boots fine
using c5.xl and m5.xl instances with apparmor & sssd enabled.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1746806
Title:
sssd
Here's a test kernel which incorporates the 'retpoline' patch set (soon
to be released in Xenial):
http://kernel.ubuntu.com/~kamal/linux-aws-rtp0/
(only the linux-image .deb is required).
My smoke test indicates that this kernel fixes the problem. Please
test, and provide feedback here.
--
@jsalisbury: I tested the kernel you provided above (commit
7de295e2a47849488acec80fc7c9973a4dca204e) and it boots fine on both a
c5.xl and a m5.xl.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
I built the next test kernel, up to the following commit:
7de295e2a47849488acec80fc7c9973a4dca204e
The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1746806
Can you test that kernel and report back if it has the bug or not? I
will build the next test kernel based on
Given that the c5 instances are Skylake, and I read somewhere that
there’s some special edge-case for Skylake and later CPUs, is this
reproducible on proper hardware, or is it limited to KVM (prior to AWS’s
c5s, the hypervisors were xen).
I’ve got a couple of Skylake machines available, and will
@jsalisbury: I installed that kernel and rebooted using a c5.xl, it
froze. I booted into a c4.xl and it booted fine, disabled the apparmor
service and rebooted into a c5.xl and it booted fine. Re-enabled
apparmor and rebooted into the c5.xl again and it froze on boot.
--
You received this bug
I started a kernel bisect between Ubuntu-4.4.0-109 and Ubuntu-4.4.0-110. The
kernel bisect will require testing of about 5 test kernels.
I built the first test kernel, up to the following commit:
d3d0f0a209ee29cf553b8b5580eb954b0d4aa970
The test kernel can be downloaded from:
** Tags added: pti
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1746806
Title:
sssd appears to crash AWS c5 and m5 instances, cause 100% CPU
To manage notifications about this bug go to:
The are no changes to apparmor in that range, but that does cover the
kaiser changes. Since there were no apparmor changes and kaiser changes
the kernel userspace memory interaction my guess is that something is
triggering in the copy_from_user when policy is loaded.
--
You received this bug
I was able to reproduce the problem with the -generic Xenial kernel (on
a c5.large instance):
4.4.0-109.132 works ok
4.4.0-110.133 broken
4.4.0-112.135 still broken
Those results correspond as expected to the linux-aws kernel versions.
The problem appears to have been introduced by something in
@racb and @jjohansen
I installed kernel 4.4.0-1050-aws, disabled sssd and apparmor on boot,
and restarted on a c5 and it boots fine.. also boots fine just disabling
sssd on boot.
If I start sssd without apparmor running everything is fine. If I start
apparmor first, then start sssd it freezes
@Joseph, etc.
linux-aws 4.4.0-1048.57 does not appear to exhibit the problem. And I
confirm pfriel's findings:
1047 works ok
1048 works ok
1049 broken
1050 broken
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
I'd like to perform a kernel bisect to identify the commit that introduced this
regression. First, can you test the 4.4.0-1048 kernel to narrow it down a bit
more? That kernel can be downloaded from:
https://launchpad.net/~canonical-kernel-security-team/+archive/ubuntu/ppa/+build/14224482
**
** Changed in: linux (Ubuntu)
Importance: Undecided => Critical
** Tags added: kernel-key
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1746806
Title:
sssd appears to crash AWS c5 and m5
Maybe but we would more information to say for sure.
There have been no changes in apparmor between the reported working
20180109 and 20180126.
The warning
> "Warning failed to create cache: usr.sbin.sssd" before the instance
just means that apparmor was not able to cache the binary policy that
Status changed to 'Confirmed' because the bug affects multiple users.
** Changed in: linux (Ubuntu)
Status: New => Confirmed
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1746806
Title:
Thanks. That's useful information.
On Fri, Feb 02, 2018 at 10:17:37AM -, David JM Emmett wrote:
> "Warning failed to create cache: usr.sbin.sssd" before the instance
That sounds AppArmor related perhaps?
--
You received this bug notification because you are a member of Ubuntu
Bugs, which
I'm experiencing this issue too with artful on c5.large. During the
installation of sssd-common 1.15.3-2ubuntu1, the last line of output is
"Warning failed to create cache: usr.sbin.sssd" before the instance
becomes unresponsive. If I use t2.medium, I don't experience this issue.
Regions tested:
If this problem comes and goes as you switch kernel, this points to a
possible kernel regression. In that case perhaps sssd is just the
trigger rather than the buggy package. So I'll add the kernel package to
this bug.
Or perhaps it's a latent bug in sssd that only triggers with the newer
kernel
I took a few minutes this afternoon and tried several different kernels,
here is what I found:
works fine: 4.4.0.1047.49 (packaged with AWS Ubuntu Cloud Image from 20180109)
BROKEN: 4.4.0.1049.51 (installed with "apt update && apt install
linux-aws=4.4.0.1049.51 linux-image-aws=4.4.0.1049.51
Seeing this as well. The system runs freeipa-client/sssd. If I remove
sssd from the bootstrap the node does not lock up. I captured an strace
showing the sssd startup is the last thing that happens before the 100%
CPU and lock up.
** Attachment added: "sssd-startup.txt"
Status changed to 'Confirmed' because the bug affects multiple users.
** Changed in: sssd (Ubuntu)
Status: New => Confirmed
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1746806
Title:
sssd
** Attachment added: "lsb_release_and_apt-cache-policy-sssd.txt"
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1746806/+attachment/5047062/+files/lsb_release_and_apt-cache-policy-sssd.txt
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed
** Attachment added: "kern.log"
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1746806/+attachment/5047061/+files/kern.log
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1746806
Title:
sssd
** Attachment added: "sssd_startup_from_shell.log"
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1746806/+attachment/5047064/+files/sssd_startup_from_shell.log
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
Added the cloud-images project so we can more easily track this.
** Also affects: cloud-images
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1746806
** Attachment added: "apport.sssd-common.apport"
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1746806/+attachment/5047060/+files/apport.sssd-common.apport
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
** Attachment added: "syslog"
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1746806/+attachment/5047066/+files/syslog
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1746806
Title:
sssd
61 matches
Mail list logo
