[Bug 1546320] Re: crash starting at kernel v3.13.0-72 in timer code

** No longer affects: linux-lts-utopic (Ubuntu) ** Changed in: linux (Ubuntu) Status: Triaged => Fix Released ** Changed in: linux (Ubuntu Vivid) Importance: Undecided => High ** Changed in: linux-lts-utopic (Ubuntu Trusty) Importance: Undecided => High -- You received this bug

[Bug 1546320] Re: crash starting at kernel v3.13.0-72 in timer code

** Changed in: linux-lts-utopic (Ubuntu Trusty) Assignee: (unassigned) => Eric Desrochers (slashd) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1546320 Title: crash starting at kernel

[Bug 1546320] Re: crash starting at kernel v3.13.0-72 in timer code

This bug was fixed in the package linux-lts-utopic - 3.16.0-67.87~14.04.1 --- linux-lts-utopic (3.16.0-67.87~14.04.1) trusty; urgency=low [ Brad Figg ] * Release Tracking Bug - LP: #1555847 [ Florian Westphal ] * SAUCE: [nf,v2] netfilter: x_tables: don't rely on

[Bug 1546320] Re: crash starting at kernel v3.13.0-72 in timer code

This bug was fixed in the package linux - 3.13.0-83.127 --- linux (3.13.0-83.127) trusty; urgency=low [ Brad Figg ] * Release Tracking Bug - LP: #1555839 [ Florian Westphal ] * SAUCE: [nf,v2] netfilter: x_tables: don't rely on well-behaving userspace - LP:

[Bug 1546320] Re: crash starting at kernel v3.13.0-72 in timer code

This bug was fixed in the package linux - 3.19.0-56.62 --- linux (3.19.0-56.62) vivid; urgency=low [ Brad Figg ] * Release Tracking Bug - LP: #1555832 [ Florian Westphal ] * SAUCE: [nf,v2] netfilter: x_tables: don't rely on well-behaving userspace - LP:

[Bug 1546320] Re: crash starting at kernel v3.13.0-72 in timer code

** Tags removed: verification-needed-vivid ** Tags added: verification-done-vivid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1546320 Title: crash starting at kernel v3.13.0-72 in timer code To

[Bug 1546320] Re: crash starting at kernel v3.13.0-72 in timer code

** Branch linked: lp:ubuntu/trusty-proposed/linux-lts-vivid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1546320 Title: crash starting at kernel v3.13.0-72 in timer code To manage notifications

[Bug 1546320] Re: crash starting at kernel v3.13.0-72 in timer code

** Tags removed: verification-needed-trusty ** Tags added: verification-done-trusty -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1546320 Title: crash starting at kernel v3.13.0-72 in timer code

[Bug 1546320] Re: crash starting at kernel v3.13.0-72 in timer code

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- trusty' to 'verification-done-trusty'. If verification is not done by 5 working days from

[Bug 1546320] Re: crash starting at kernel v3.13.0-72 in timer code

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- vivid' to 'verification-done-vivid'. If verification is not done by 5 working days from

[Bug 1546320] Re: crash starting at kernel v3.13.0-72 in timer code

It also been brought to my attention the following about the hotfix kernel: -- Hi Eric, We have applied the update from your PPA to the controller nodes and they have been stable over the weekend. We will see if they remain stable under load over the rest of the week. Thanks -- Eric -- You

[Bug 1546320] Re: crash starting at kernel v3.13.0-72 in timer code

** Changed in: linux (Ubuntu Trusty) Status: In Progress => Fix Committed ** Changed in: linux (Ubuntu Vivid) Status: In Progress => Fix Committed ** Also affects: linux-lts-utopic (Ubuntu) Importance: Undecided Status: New ** No longer affects: linux-lts-utopic (Ubuntu

[Bug 1546320] Re: crash starting at kernel v3.13.0-72 in timer code

Rectification about the hotfix instruction procedure on comment #11. Instructions -- # Add the PPA to your system $ sudo add-apt-repository ppa:slashd/bug1546320-hotfix # Resynchronize the package index files from their sources $ sudo apt-get update # Installing the packages desired for

[Bug 1546320] Re: crash starting at kernel v3.13.0-72 in timer code

** No longer affects: linux (Ubuntu Wily) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1546320 Title: crash starting at kernel v3.13.0-72 in timer code To manage notifications about this bug go

[Bug 1546320] Re: crash starting at kernel v3.13.0-72 in timer code

** Changed in: linux (Ubuntu) Status: Confirmed => Triaged -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1546320 Title: crash starting at kernel v3.13.0-72 in timer code To manage

[Bug 1546320] Re: crash starting at kernel v3.13.0-72 in timer code

Here's a positive feedback by someone from the community about the trusty testfix kernel with commit 470ed447 reverted that I have provided yesterday: "I am able to reproduce the kernel panic consistently courtesy of the wqthrash kernel module on this LKML post:

[Bug 1546320] Re: crash starting at kernel v3.13.0-72 in timer code

** Changed in: linux (Ubuntu Trusty) Assignee: Kamal Mostafa (kamalmostafa) => Eric Desrochers (slashd) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1546320 Title: crash starting at kernel

[Bug 1546320] Re: crash starting at kernel v3.13.0-72 in timer code

I build a test kernel and made it available on a PPA[1] for verification in order to bring confidence for the SRU process. This test kernel revert the offending commit "470ed447: workqueue: make sure delayed work run in local cpu". Instructions -- # Add the PPA to your system $ sudo

[Bug 1546320] Re: crash starting at kernel v3.13.0-72 in timer code

So, Wily can be "more safe" with both: 874bbfe & 22b886d, since the upstream decision described in: "The local CPU guarantee was accidental more than anything else and we want to get rid of it anyway. As, with the vmstat case fixed, 874bbfe600a6 is causing more problems than it's fixing,

[Bug 1546320] Re: crash starting at kernel v3.13.0-72 in timer code

inaddy@rafael(~/Codes/kernel/linux/stable)$ git tag --contains 22b886d v4.4 v4.4-rc1 22b886d first appeared in 4.4-rc1 commit 22b886dd1018093920c4250dee2a9a3cb7cff7b8 Author: Tejun Heo Date: Wed Nov 4 12:15:33 2015 -0500 timers: Use proper base migration in

[Bug 1546320] Re: crash starting at kernel v3.13.0-72 in timer code

Working on removing "874bbfe" from Trusty, as Kamal well said: commit 874bbfe600a660cba9c776b3957b1ce393151b76 Author: Shaohua Li Date: Wed Sep 30 09:05:30 2015 -0700 workqueue: make sure delayed work run in local cpu Is commit: commit

[Bug 1546320] Re: crash starting at kernel v3.13.0-72 in timer code

# SUMMARY - recent changes (2016) broke timer guarantee of work items queue without explicit cpuu to be put on local cpus. - 874bbfe made sure delayed work run in local cpu (Sep 2015) - vmstat was broken and fixed by 176bed1de5bf (explicitly scheduling per-cpu work) (Oct 2015) - 874bbfe

[Bug 1546320] Re: crash starting at kernel v3.13.0-72 in timer code

** Tags added: cpec -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1546320 Title: crash starting at kernel v3.13.0-72 in timer code To manage notifications about this bug go to:

[Bug 1546320] Re: crash starting at kernel v3.13.0-72 in timer code

Eric comments that this may be fixed by the mainline stable commit: [mainline] 22b886d "timers: Use proper base migration in add_timer_on()" ... but that can't be applied to pre-4.2 kernels (since it would require 0eeda71 "timer: Replace timer base by a cpu index", which isn't suitable for

[Bug 1546320] Re: crash starting at kernel v3.13.0-72 in timer code

** Changed in: linux (Ubuntu Trusty) Importance: Undecided => High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1546320 Title: crash starting at kernel v3.13.0-72 in timer code To manage

[Bug 1546320] Re: crash starting at kernel v3.13.0-72 in timer code

** Also affects: linux (Ubuntu Trusty) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Wily) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Vivid) Importance: Undecided Status: New ** Changed in: linux (Ubuntu Trusty)

[Bug 1546320] Re: crash starting at kernel v3.13.0-72 in timer code

The problem has been reported by few different peoples of the community already. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1546320 Title: crash starting at kernel v3.13.0-72 in timer code To

[Bug 1546320] Re: crash starting at kernel v3.13.0-72 in timer code

See "dmesg_detach_if_pending.out" taken from the last crash into /var/crash/ ** Attachment added: "dmesg_detach_if_pending.out" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1546320/+attachment/4573109/+files/dmesg_detach_if_pending.out ** Changed in: linux (Ubuntu) Status:

[Bug 1546320] Re: crash starting at kernel v3.13.0-72 in timer code

It seems like commit: 22b886dd timers: Use proper base migration in add_timer_on() [1] Has been started to by apply : $ git tag --contains 22b886dd Ubuntu-lts-4.4.0-4.19_14.04.1 Ubuntu-lts-4.4.0-4.19_14.04.2 [1] $ git show 22b886dd commit 22b886dd1018093920c4250dee2a9a3cb7cff7b8 Author: Tejun

[Bug 1546320] Re: crash starting at kernel v3.13.0-72 in timer code

It seems to a problem in add_timer_on moving from a timer_base to a new timer_base It does look like 470ed44 exposed the problem by switching to add_timer_on from mod_timer; this is a fix to add_timer_on Reference: --- https://lkml.org/lkml/2016/2/3/295 https://lkml.org/lkml/2016/2/4/247 Patch: