[Bug 1461620] Re: NUMA task migration race condition due to stop task not being checked when balancing happens

2015-09-30 Thread Rafael David Tinoco
inaddy@mylinux  ~/Work/Kernel/Ubuntu/ubuntu-trusty   master  git tag --contains 64863995563d71836fa48b743148dce993154a4e Ubuntu-3.13.0-60.99 Ubuntu-3.13.0-62.101 Ubuntu-3.13.0-62.102 Ubuntu-3.13.0-63.103 Ubuntu-3.13.0-64.104 Ubuntu-3.13.0-65.105 linux-image-generic | 3.13.0.24.28 | trusty

[Bug 1461620] Re: NUMA task migration race condition due to stop task not being checked when balancing happens

2015-08-17 Thread Launchpad Bug Tracker
This bug was fixed in the package linux - 3.19.0-26.28 --- linux (3.19.0-26.28) vivid; urgency=low [ Luis Henriques ] * Release Tracking Bug - LP: #1483630 [ Upstream Kernel Changes ] * Revert Bluetooth: ath3k: Add support of 04ca:300d AR3012 device linux

[Bug 1461620] Re: NUMA task migration race condition due to stop task not being checked when balancing happens

2015-08-11 Thread Rafael David Tinoco
Started verifying the fix.. will provide results soon. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1461620 Title: NUMA task migration race condition due to stop task not being checked when

[Bug 1461620] Re: NUMA task migration race condition due to stop task not being checked when balancing happens

2015-08-11 Thread Rafael David Tinoco
Trusty verification: inaddy@sf00079894trusty:~$ uname -a Linux sf00079894trusty 3.13.0-62-generic #101-Ubuntu SMP Thu Jul 30 09:01:36 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux inaddy@sf00079894trusty:~$ sudo trace-cmd report | grep stop_two_cpus | wc -l 74 In 5 seconds the logic was executed 74

[Bug 1461620] Re: NUMA task migration race condition due to stop task not being checked when balancing happens

2015-08-11 Thread Rafael David Tinoco
Vivid verification: inaddy@sf00079894vivid:~$ uname -a Linux sf00079894vivid 3.19.0-26-generic #27-Ubuntu SMP Tue Jul 28 18:27:31 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux inaddy@sf00079894vivid:~$ sudo trace-cmd report | grep stop_two_cpus | wc -l 46 In 5 seconds the logic was executed 46

[Bug 1461620] Re: NUMA task migration race condition due to stop task not being checked when balancing happens

2015-08-11 Thread Rafael David Tinoco
** Tags added: sts ** Tags added: cts -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1461620 Title: NUMA task migration race condition due to stop task not being checked when balancing happens

[Bug 1461620] Re: NUMA task migration race condition due to stop task not being checked when balancing happens

2015-08-05 Thread Brad Figg
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- trusty' to 'verification-done-trusty'. If verification is not done by 5 working days from

[Bug 1461620] Re: NUMA task migration race condition due to stop task not being checked when balancing happens

2015-08-05 Thread Brad Figg
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- vivid' to 'verification-done-vivid'. If verification is not done by 5 working days from

[Bug 1461620] Re: NUMA task migration race condition due to stop task not being checked when balancing happens

2015-08-04 Thread Launchpad Bug Tracker
** Branch linked: lp:ubuntu/trusty-proposed/linux-lts-vivid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1461620 Title: NUMA task migration race condition due to stop task not being checked

[Bug 1461620] Re: NUMA task migration race condition due to stop task not being checked when balancing happens

2015-07-31 Thread Launchpad Bug Tracker
This bug was fixed in the package linux - 4.1.0-3.3 --- linux (4.1.0-3.3) wily; urgency=low [ Andy Whitcroft ] * Release Tracking Bug - LP: #1478897 [ Colin Ian King ] * SAUCE: KEYS: ensure we free the assoc array edit if edit is valid - CVE-2015-1333 [ Seth

[Bug 1461620] Re: NUMA task migration race condition due to stop task not being checked when balancing happens

2015-07-27 Thread Andy Whitcroft
** Changed in: linux (Ubuntu) Status: Invalid = Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1461620 Title: NUMA task migration race condition due to stop task not being

[Bug 1461620] Re: NUMA task migration race condition due to stop task not being checked when balancing happens

2015-07-27 Thread Luis Henriques
** Changed in: linux (Ubuntu Trusty) Status: In Progress = Fix Committed ** Changed in: linux (Ubuntu Vivid) Status: In Progress = Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1461620] Re: NUMA task migration race condition due to stop task not being checked when balancing happens

2015-07-23 Thread Chris J Arges
** Description changed: SRU Justification: - Impact: - - Deadlock when migrating processes in between NUMA domains. - - Came with 1 kernel dump given to me. - - Hard to trigger. + Impact: +  - Deadlock when migrating processes in between NUMA domains. +  - Came with 1 kernel dump given

[Bug 1461620] Re: NUMA task migration race condition due to stop task not being checked when balancing happens

2015-07-23 Thread Rafael David Tinoco
** Description changed: + SRU Justification: + + Impact: + - Deadlock when migrating processes in between NUMA domains. + - Came with 1 kernel dump given to me. + - Hard to trigger. + + Fix: + - Upstream development after upstream discussion. + - Discussion:

[Bug 1461620] Re: NUMA task migration race condition due to stop task not being checked when balancing happens

2015-07-23 Thread Chris J Arges
** Also affects: linux (Ubuntu Vivid) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1461620 Title: NUMA task migration race condition due to stop task

[Bug 1461620] Re: NUMA task migration race condition due to stop task not being checked when balancing happens

2015-07-23 Thread Rafael David Tinoco
** Changed in: linux (Ubuntu Vivid) Status: New = In Progress ** Changed in: linux (Ubuntu Vivid) Assignee: (unassigned) = Rafael David Tinoco (inaddy) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1461620] Re: NUMA task migration race condition due to stop task not being checked when balancing happens

2015-07-23 Thread Rafael David Tinoco
I'm running the NUMA tests on 3.13 for some time now and it looks like the change did not introduce any regression... $ uname -a Linux sf00079894trusty 3.13.11-ckt22-201507231149 #2 SMP Thu Jul 23 13:45:04 BRT 2015 x86_64 x86_64 x86_64 GNU/Linux I'm using a virtualized 16 Domains / 16 CPUs

[Bug 1461620] Re: NUMA task migration race condition due to stop task not being checked when balancing happens

2015-06-15 Thread Rafael David Tinoco
Just got an update from Peter: https://lkml.org/lkml/2015/6/15/531 asking for feedback on a patch: Subject: stop_machine: Fix deadlock between multiple stop_two_cpus() From: Peter Zijlstra pet...@infradead.org Date: Fri, 5 Jun 2015 17:30:23 +0200 Will try to test the latest builds + this patch

[Bug 1461620] Re: NUMA task migration race condition due to stop task not being checked when balancing happens

2015-06-03 Thread Rafael David Tinoco
Sasha pointed me the a fix for this particular behaviour in between 3.16 and 3.17: https://lkml.org/lkml/2014/4/10/297 [PATCH] sched: Checking for stop task appearance when balancing happens Saying that indeed mine previous observation: --- NMI exception stack --- #4 [883fd2907d98]

[Bug 1461620] Re: NUMA task migration race condition due to stop task not being checked when balancing happens

2015-06-03 Thread Rafael David Tinoco
Using ftrace I can make sure that we are triggering the logic that is responsible for the dead lock to happen (in a frequent basis) but until now without the success of making it to happen. root@numa:~# trace-cmd record -p function -l numa_migrate_preferred -l task_numa_migrate -l migrate_swap -l

[Bug 1461620] Re: NUMA task migration race condition due to stop task not being checked when balancing happens

2015-06-03 Thread Joseph Salisbury
** Changed in: linux (Ubuntu Trusty) Importance: Undecided = Medium -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1461620 Title: NUMA task migration race condition due to stop task not being

[Bug 1461620] Re: NUMA task migration race condition due to stop task not being checked when balancing happens

2015-06-03 Thread Rafael David Tinoco
You can follow my comments in LKML: https://lkml.org/lkml/2015/3/6/484 Basically in kernel 3.13 we are getting the follow situation: I have a core dump locked on the same place (state machine for powering cpu down for the task swap) from a 3.13 (+ upstream patches) and this commit wasn't

[Bug 1461620] Re: NUMA task migration race condition due to stop task not being checked when balancing happens

2015-06-03 Thread Rafael David Tinoco
To understand better if this bug was triggered easy I created the following test case: I've been using a KVM guest emulating a NUMA environment with 32 different domains (1 for each vCPU): root@numa:~# numactl -H available: 32 nodes (0-31) node 0 cpus: 0 node 0 size: 237 MB node 0 free: 82

[Bug 1461620] Re: NUMA task migration race condition due to stop task not being checked when balancing happens

2015-06-03 Thread Rafael David Tinoco
But unfortunately I could not reproduce the issue (although I know it is in there). I'll create a small logic similar to: Commit a1d9a3231eac4117cadaf4b6bba5b2902c15a33e Author: Kirill Tkhai tk...@yandex.ru Date: Thu Apr 10 17:38:36 2014 +0400 sched: Check for stop task appearance when

[Bug 1461620] Re: NUMA task migration race condition due to stop task not being checked when balancing happens

2015-06-03 Thread Brad Figg
** Also affects: linux (Ubuntu Trusty) Importance: Undecided Status: New ** Changed in: linux (Ubuntu) Status: In Progress = Invalid ** Changed in: linux (Ubuntu Trusty) Status: New = In Progress ** Changed in: linux (Ubuntu Trusty) Assignee: (unassigned) = Rafael

[Bug 1461620] Re: NUMA task migration race condition due to stop task not being checked when balancing happens

2015-06-03 Thread Rafael David Tinoco
It happens that the fix relies on checking if the stop worker needs task selection re-start: + if (need_pull_dl_task(rq, prev)) { pull_dl_task(rq); + /* +* pull_rt_task() can drop (and re-acquire) rq-lock; this +* means a stop