[Kernel-packages] [Bug 1607355] Re: Task (usually mongod) blocked more 120 seconds (lock-ups) in juju on lxc/lxd + zfs

2018-09-07 Thread Launchpad Bug Tracker
[Expired for linux (Ubuntu) because there has been no activity for 60
days.]

** Changed in: linux (Ubuntu)
   Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1607355

Title:
  Task (usually mongod) blocked more 120 seconds (lock-ups) in juju on
  lxc/lxd + zfs

Status in linux package in Ubuntu:
  Expired

Bug description:
  I was able to reproduce this 2 or 3 times last 2 days. I have the
  following setup:

  Containers for Trusty/kilo service machines:

  inaddy@workstation:~$ lxc-ls  | grep tk
  tkcephmon01  RUNNING 0 -  192.168.65.52 -
  tkcephmon02  RUNNING 0 -  192.168.65.51 -
  tkcephmon03  RUNNING 0 -  192.168.65.48 -
  tkcinder RUNNING 0 -  192.168.65.49 -
  tkdash   RUNNING 0 -  192.168.65.50 -
  tkglance RUNNING 0 -  192.168.65.53 -
  tkjuju   RUNNING 0 -  192.168.65.15 -
  tkkeystone   RUNNING 0 -  192.168.65.54 -
  tkmysql  RUNNING 0 -  192.168.65.55 -
  tknova   RUNNING 0 -  192.168.65.56 -
  tkrabbit RUNNING 0 -  192.168.65.57 -
  tkswiftproxy RUNNING 0 -  192.168.65.58 -

  And compute nodes + neutrongw as kvm guests:

  inaddy@workstation:~$ virsh list --all | grep tk
   21tkcompute01running
   22tkcompute02running
   23tkcompute03running
   24tkneutrongwrunning

  All my LXC containers are on top of ZFS:

  Linux workstation 4.4.0-32-generic #51-Ubuntu SMP Tue Jul 19 18:09:07
  UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

  And my KVM guests are on top of ext4 + 1.2 raid0 stripped volume.

  I'm getting the lockups bellow (usually for mongod, from tkjuju
  container, the juju controller). After the first lockup appears
  (schedule timeout coming from zfs sync logic most likely), JuJu
  controller starts giving me errors on "update-status". From "juju
  status":

  glance/0error  idle1.25.6  10  9292/tcp   
tkglance   hook failed: "update-status"
  keystone/0  error  idle1.25.6  11 
tkkeystone hook failed: "update-status"
  mysql/0 error  idle1.25.6  12 
tkmysqlhook failed: "config-changed"
  neutron-api/0   error  idle1.25.6  4   9696/tcp   
tkneutrongwhook failed: "update-status"
  nova-compute/0  error  idle1.25.6  1  
tkcompute01hook failed: "update-status"
  nova-compute/1  error  idle1.25.6  2  
tkcompute02hook failed: "update-status"
  nova-compute/2  error  idle1.25.6  3  
tkcompute03hook failed: "update-status"

  Lockups:

  [105601.816578] INFO: task mongod:14480 blocked for more than 120 seconds.
  [105601.816583]   Tainted: P   O4.4.0-32-generic #51-Ubuntu
  [105601.816584] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [105601.816586] mongod  D 88010ec47ba8 0 14480  16855 
0x0100
  [105601.816590]  88010ec47ba8 57992eeb 880108e5ee00 
880108e58dc0
  [105601.816592]  88010ec48000 88081ecd6d00 7fff 
8182a600
  [105601.816594]  88010ec47d08 88010ec47bc0 81829e05 

  [105601.816596] Call Trace:
  [105601.816603]  [] ? bit_wait+0x60/0x60
  [105601.816606]  [] schedule+0x35/0x80
  [105601.816608]  [] schedule_timeout+0x1b5/0x270
  [105601.816612]  [] ? find_get_pages_tag+0x109/0x190
  [105601.816614]  [] ? bit_wait+0x60/0x60
  [105601.816616]  [] io_schedule_timeout+0xa4/0x110
  [105601.816618]  [] bit_wait_io+0x1b/0x70
  [105601.816620]  [] __wait_on_bit+0x5d/0x90
  [105601.816622]  [] wait_on_page_bit+0xcb/0xf0
  [105601.816625]  [] ? autoremove_wake_function+0x40/0x40
  [105601.816628]  [] __filemap_fdatawait_range+0xf3/0x160
  [105601.816630]  [] filemap_fdatawait_range+0x14/0x30
  [105601.816632]  [] filemap_write_and_wait_range+0x3f/0x70
  [105601.816682]  [] zpl_fsync+0x38/0x90 [zfs]
  [105601.816685]  [] vfs_fsync_range+0x4b/0xb0
  [105601.816687]  [] SyS_msync+0x17e/0x1f0
  [105601.816689]  [] entry_SYSCALL_64_fastpath+0x16/0x71
  [117121.961545] INFO: task txg_sync:4589 blocked for more than 120 seconds.
  [117121.961549]   Tainted: P   O4.4.0-32-generic #51-Ubuntu
  [117121.961550] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [117121.961551] txg_syncD 8807e1fbbaa8 0  4589  2 
0x
  

[Kernel-packages] [Bug 1607355] Re: Task (usually mongod) blocked more 120 seconds (lock-ups) in juju on lxc/lxd + zfs

2018-07-09 Thread Simos Xenitellis 
This might be related to #1773392.

On the other hand, the issue in #1773392 appears to have been introduced
in more recent kernels (16.04 not affected).

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1607355

Title:
  Task (usually mongod) blocked more 120 seconds (lock-ups) in juju on
  lxc/lxd + zfs

Status in linux package in Ubuntu:
  Incomplete

Bug description:
  I was able to reproduce this 2 or 3 times last 2 days. I have the
  following setup:

  Containers for Trusty/kilo service machines:

  inaddy@workstation:~$ lxc-ls  | grep tk
  tkcephmon01  RUNNING 0 -  192.168.65.52 -
  tkcephmon02  RUNNING 0 -  192.168.65.51 -
  tkcephmon03  RUNNING 0 -  192.168.65.48 -
  tkcinder RUNNING 0 -  192.168.65.49 -
  tkdash   RUNNING 0 -  192.168.65.50 -
  tkglance RUNNING 0 -  192.168.65.53 -
  tkjuju   RUNNING 0 -  192.168.65.15 -
  tkkeystone   RUNNING 0 -  192.168.65.54 -
  tkmysql  RUNNING 0 -  192.168.65.55 -
  tknova   RUNNING 0 -  192.168.65.56 -
  tkrabbit RUNNING 0 -  192.168.65.57 -
  tkswiftproxy RUNNING 0 -  192.168.65.58 -

  And compute nodes + neutrongw as kvm guests:

  inaddy@workstation:~$ virsh list --all | grep tk
   21tkcompute01running
   22tkcompute02running
   23tkcompute03running
   24tkneutrongwrunning

  All my LXC containers are on top of ZFS:

  Linux workstation 4.4.0-32-generic #51-Ubuntu SMP Tue Jul 19 18:09:07
  UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

  And my KVM guests are on top of ext4 + 1.2 raid0 stripped volume.

  I'm getting the lockups bellow (usually for mongod, from tkjuju
  container, the juju controller). After the first lockup appears
  (schedule timeout coming from zfs sync logic most likely), JuJu
  controller starts giving me errors on "update-status". From "juju
  status":

  glance/0error  idle1.25.6  10  9292/tcp   
tkglance   hook failed: "update-status"
  keystone/0  error  idle1.25.6  11 
tkkeystone hook failed: "update-status"
  mysql/0 error  idle1.25.6  12 
tkmysqlhook failed: "config-changed"
  neutron-api/0   error  idle1.25.6  4   9696/tcp   
tkneutrongwhook failed: "update-status"
  nova-compute/0  error  idle1.25.6  1  
tkcompute01hook failed: "update-status"
  nova-compute/1  error  idle1.25.6  2  
tkcompute02hook failed: "update-status"
  nova-compute/2  error  idle1.25.6  3  
tkcompute03hook failed: "update-status"

  Lockups:

  [105601.816578] INFO: task mongod:14480 blocked for more than 120 seconds.
  [105601.816583]   Tainted: P   O4.4.0-32-generic #51-Ubuntu
  [105601.816584] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [105601.816586] mongod  D 88010ec47ba8 0 14480  16855 
0x0100
  [105601.816590]  88010ec47ba8 57992eeb 880108e5ee00 
880108e58dc0
  [105601.816592]  88010ec48000 88081ecd6d00 7fff 
8182a600
  [105601.816594]  88010ec47d08 88010ec47bc0 81829e05 

  [105601.816596] Call Trace:
  [105601.816603]  [] ? bit_wait+0x60/0x60
  [105601.816606]  [] schedule+0x35/0x80
  [105601.816608]  [] schedule_timeout+0x1b5/0x270
  [105601.816612]  [] ? find_get_pages_tag+0x109/0x190
  [105601.816614]  [] ? bit_wait+0x60/0x60
  [105601.816616]  [] io_schedule_timeout+0xa4/0x110
  [105601.816618]  [] bit_wait_io+0x1b/0x70
  [105601.816620]  [] __wait_on_bit+0x5d/0x90
  [105601.816622]  [] wait_on_page_bit+0xcb/0xf0
  [105601.816625]  [] ? autoremove_wake_function+0x40/0x40
  [105601.816628]  [] __filemap_fdatawait_range+0xf3/0x160
  [105601.816630]  [] filemap_fdatawait_range+0x14/0x30
  [105601.816632]  [] filemap_write_and_wait_range+0x3f/0x70
  [105601.816682]  [] zpl_fsync+0x38/0x90 [zfs]
  [105601.816685]  [] vfs_fsync_range+0x4b/0xb0
  [105601.816687]  [] SyS_msync+0x17e/0x1f0
  [105601.816689]  [] entry_SYSCALL_64_fastpath+0x16/0x71
  [117121.961545] INFO: task txg_sync:4589 blocked for more than 120 seconds.
  [117121.961549]   Tainted: P   O4.4.0-32-generic #51-Ubuntu
  [117121.961550] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [117121.961551] txg_syncD 8807e1fbbaa8 0  4589  2 
0x
  

[Kernel-packages] [Bug 1607355] Re: Task (usually mongod) blocked more 120 seconds (lock-ups) in juju on lxc/lxd + zfs

2018-05-18 Thread Rafael David Tinoco
** Changed in: linux (Ubuntu)
 Assignee: Rafael David Tinoco (inaddy) => (unassigned)

** Changed in: linux (Ubuntu)
   Status: In Progress => Incomplete

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1607355

Title:
  Task (usually mongod) blocked more 120 seconds (lock-ups) in juju on
  lxc/lxd + zfs

Status in linux package in Ubuntu:
  Incomplete

Bug description:
  I was able to reproduce this 2 or 3 times last 2 days. I have the
  following setup:

  Containers for Trusty/kilo service machines:

  inaddy@workstation:~$ lxc-ls  | grep tk
  tkcephmon01  RUNNING 0 -  192.168.65.52 -
  tkcephmon02  RUNNING 0 -  192.168.65.51 -
  tkcephmon03  RUNNING 0 -  192.168.65.48 -
  tkcinder RUNNING 0 -  192.168.65.49 -
  tkdash   RUNNING 0 -  192.168.65.50 -
  tkglance RUNNING 0 -  192.168.65.53 -
  tkjuju   RUNNING 0 -  192.168.65.15 -
  tkkeystone   RUNNING 0 -  192.168.65.54 -
  tkmysql  RUNNING 0 -  192.168.65.55 -
  tknova   RUNNING 0 -  192.168.65.56 -
  tkrabbit RUNNING 0 -  192.168.65.57 -
  tkswiftproxy RUNNING 0 -  192.168.65.58 -

  And compute nodes + neutrongw as kvm guests:

  inaddy@workstation:~$ virsh list --all | grep tk
   21tkcompute01running
   22tkcompute02running
   23tkcompute03running
   24tkneutrongwrunning

  All my LXC containers are on top of ZFS:

  Linux workstation 4.4.0-32-generic #51-Ubuntu SMP Tue Jul 19 18:09:07
  UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

  And my KVM guests are on top of ext4 + 1.2 raid0 stripped volume.

  I'm getting the lockups bellow (usually for mongod, from tkjuju
  container, the juju controller). After the first lockup appears
  (schedule timeout coming from zfs sync logic most likely), JuJu
  controller starts giving me errors on "update-status". From "juju
  status":

  glance/0error  idle1.25.6  10  9292/tcp   
tkglance   hook failed: "update-status"
  keystone/0  error  idle1.25.6  11 
tkkeystone hook failed: "update-status"
  mysql/0 error  idle1.25.6  12 
tkmysqlhook failed: "config-changed"
  neutron-api/0   error  idle1.25.6  4   9696/tcp   
tkneutrongwhook failed: "update-status"
  nova-compute/0  error  idle1.25.6  1  
tkcompute01hook failed: "update-status"
  nova-compute/1  error  idle1.25.6  2  
tkcompute02hook failed: "update-status"
  nova-compute/2  error  idle1.25.6  3  
tkcompute03hook failed: "update-status"

  Lockups:

  [105601.816578] INFO: task mongod:14480 blocked for more than 120 seconds.
  [105601.816583]   Tainted: P   O4.4.0-32-generic #51-Ubuntu
  [105601.816584] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [105601.816586] mongod  D 88010ec47ba8 0 14480  16855 
0x0100
  [105601.816590]  88010ec47ba8 57992eeb 880108e5ee00 
880108e58dc0
  [105601.816592]  88010ec48000 88081ecd6d00 7fff 
8182a600
  [105601.816594]  88010ec47d08 88010ec47bc0 81829e05 

  [105601.816596] Call Trace:
  [105601.816603]  [] ? bit_wait+0x60/0x60
  [105601.816606]  [] schedule+0x35/0x80
  [105601.816608]  [] schedule_timeout+0x1b5/0x270
  [105601.816612]  [] ? find_get_pages_tag+0x109/0x190
  [105601.816614]  [] ? bit_wait+0x60/0x60
  [105601.816616]  [] io_schedule_timeout+0xa4/0x110
  [105601.816618]  [] bit_wait_io+0x1b/0x70
  [105601.816620]  [] __wait_on_bit+0x5d/0x90
  [105601.816622]  [] wait_on_page_bit+0xcb/0xf0
  [105601.816625]  [] ? autoremove_wake_function+0x40/0x40
  [105601.816628]  [] __filemap_fdatawait_range+0xf3/0x160
  [105601.816630]  [] filemap_fdatawait_range+0x14/0x30
  [105601.816632]  [] filemap_write_and_wait_range+0x3f/0x70
  [105601.816682]  [] zpl_fsync+0x38/0x90 [zfs]
  [105601.816685]  [] vfs_fsync_range+0x4b/0xb0
  [105601.816687]  [] SyS_msync+0x17e/0x1f0
  [105601.816689]  [] entry_SYSCALL_64_fastpath+0x16/0x71
  [117121.961545] INFO: task txg_sync:4589 blocked for more than 120 seconds.
  [117121.961549]   Tainted: P   O4.4.0-32-generic #51-Ubuntu
  [117121.961550] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [117121.961551] txg_syncD 8807e1fbbaa8 0  4589  2 

[Kernel-packages] [Bug 1607355] Re: Task (usually mongod) blocked more 120 seconds (lock-ups) in juju on lxc/lxd + zfs

2016-08-01 Thread Rafael David Tinoco
Got some more stack traces:

[22201.670647] INFO: task mongod:1917 blocked for more than 120 seconds.
[22201.670650]   Tainted: P   O4.4.0-34-generic #53-Ubuntu
[22201.670651] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this 
message.
[22201.670653] mongod  D 8805aecefba8 0  1917   5819 0x0100
[22201.670655]  8805aecefba8 579b1d9a 8807faf06040 
8806d6238000
[22201.670657]  8805aecf 88081edd6d00 7fff 
8182a610
[22201.670659]  8805aecefd08 8805aecefbc0 81829e15 

[22201.670661] Call Trace:
[22201.670666]  [] ? bit_wait+0x60/0x60
[22201.670668]  [] schedule+0x35/0x80
[22201.670670]  [] schedule_timeout+0x1b5/0x270
[22201.670673]  [] ? find_get_pages_tag+0x109/0x190
[22201.670675]  [] ? bit_wait+0x60/0x60
[22201.670676]  [] io_schedule_timeout+0xa4/0x110
[22201.670678]  [] bit_wait_io+0x1b/0x70
[22201.670680]  [] __wait_on_bit+0x5d/0x90
[22201.670682]  [] wait_on_page_bit+0xcb/0xf0
[22201.670684]  [] ? autoremove_wake_function+0x40/0x40
[22201.670686]  [] __filemap_fdatawait_range+0xf3/0x160
[22201.670688]  [] ? __filemap_fdatawrite_range+0xd1/0x100
[22201.670690]  [] filemap_fdatawait_range+0x14/0x30
[22201.670691]  [] filemap_write_and_wait_range+0x3f/0x70
[22201.670732]  [] zpl_fsync+0x38/0x90 [zfs]
[22201.670734]  [] vfs_fsync_range+0x4b/0xb0
[22201.670736]  [] SyS_msync+0x17e/0x1f0
[22201.670738]  [] entry_SYSCALL_64_fastpath+0x16/0x71
[85684.344389] INFO: task mongod:1917 blocked for more than 120 seconds.
[85684.344393]   Tainted: P   O4.4.0-34-generic #53-Ubuntu
[85684.344394] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this 
message.
[85684.344395] mongod  D 8805aecefba8 0  1917   5819 0x0100
[85684.344398]  8805aecefba8 0003 81e11500 
8806d6238000
[85684.344400]  8805aecf 88081ec16d00 7fff 
8182a610
[85684.344402]  8805aecefd08 8805aecefbc0 81829e15 

[85684.344403] Call Trace:
[85684.344409]  [] ? bit_wait+0x60/0x60
[85684.344411]  [] schedule+0x35/0x80
[85684.344413]  [] schedule_timeout+0x1b5/0x270
[85684.344416]  [] ? find_get_pages_tag+0x109/0x190
[85684.344418]  [] ? bit_wait+0x60/0x60
[85684.344419]  [] io_schedule_timeout+0xa4/0x110
[85684.344421]  [] bit_wait_io+0x1b/0x70
[85684.344423]  [] __wait_on_bit+0x5d/0x90
[85684.344425]  [] wait_on_page_bit+0xcb/0xf0
[85684.344427]  [] ? autoremove_wake_function+0x40/0x40
[85684.344429]  [] __filemap_fdatawait_range+0xf3/0x160
[85684.344431]  [] ? __filemap_fdatawrite_range+0xd1/0x100
[85684.344432]  [] filemap_fdatawait_range+0x14/0x30
[85684.344434]  [] filemap_write_and_wait_range+0x3f/0x70
[85684.344477]  [] zpl_fsync+0x38/0x90 [zfs]
[85684.344479]  [] vfs_fsync_range+0x4b/0xb0
[85684.344481]  [] SyS_msync+0x17e/0x1f0
[85684.344483]  [] entry_SYSCALL_64_fastpath+0x16/0x71
[110165.396049] INFO: task mongod:1917 blocked for more than 120 seconds.
[110165.396053]   Tainted: P   O4.4.0-34-generic #53-Ubuntu
[110165.396054] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
[110165.396055] mongod  D 8805aecefba8 0  1917   5819 0x0100
[110165.396058]  8805aecefba8 579c74f7 8807faf02940 
8806d6238000
[110165.396060]  8805aecf 88081ecd6d00 7fff 
8182a610
[110165.396062]  8805aecefd08 8805aecefbc0 81829e15 

[110165.396064] Call Trace:
[110165.396070]  [] ? bit_wait+0x60/0x60
[110165.396072]  [] schedule+0x35/0x80
[110165.396074]  [] schedule_timeout+0x1b5/0x270
[110165.396076]  [] ? find_get_pages_tag+0x109/0x190
[110165.396078]  [] ? bit_wait+0x60/0x60
[110165.396080]  [] io_schedule_timeout+0xa4/0x110
[110165.396082]  [] bit_wait_io+0x1b/0x70
[110165.396083]  [] __wait_on_bit+0x5d/0x90
[110165.396085]  [] wait_on_page_bit+0xcb/0xf0
[110165.396088]  [] ? autoremove_wake_function+0x40/0x40
[110165.396090]  [] __filemap_fdatawait_range+0xf3/0x160
[110165.396091]  [] ? __filemap_fdatawrite_range+0xd1/0x100
[110165.396093]  [] filemap_fdatawait_range+0x14/0x30
[110165.396094]  [] filemap_write_and_wait_range+0x3f/0x70
[110165.396137]  [] zpl_fsync+0x38/0x90 [zfs]
[110165.396139]  [] vfs_fsync_range+0x4b/0xb0
[110165.396141]  [] SyS_msync+0x17e/0x1f0
[110165.396143]  [] entry_SYSCALL_64_fastpath+0x16/0x71
[132726.355047] INFO: task mongod:1917 blocked for more than 120 seconds.
[132726.355062]   Tainted: P   O4.4.0-34-generic #53-Ubuntu
[132726.355063] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
[132726.355065] mongod  D 8805aecefba8 0  1917   5819 0x0100
[132726.355067]  8805aecefba8 0003 8806d623e040 
8806d6238000
[132726.355069]  8805aecf 88081ed56d00 7fff 
8182a610
[132726.355071]  8805aecefd08 8805aecefbc0 

[Kernel-packages] [Bug 1607355] Re: Task (usually mongod) blocked more 120 seconds (lock-ups) in juju on lxc/lxd + zfs

2016-07-28 Thread Rafael David Tinoco
** Changed in: linux (Ubuntu)
   Status: Incomplete => In Progress

** Changed in: linux (Ubuntu)
   Importance: Undecided => Medium

** Changed in: linux (Ubuntu)
 Assignee: (unassigned) => Rafael David Tinoco (inaddy)

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1607355

Title:
  Task (usually mongod) blocked more 120 seconds (lock-ups) in juju on
  lxc/lxd + zfs

Status in linux package in Ubuntu:
  In Progress

Bug description:
  I was able to reproduce this 2 or 3 times last 2 days. I have the
  following setup:

  Containers for Trusty/kilo service machines:

  inaddy@workstation:~$ lxc-ls  | grep tk
  tkcephmon01  RUNNING 0 -  192.168.65.52 -
  tkcephmon02  RUNNING 0 -  192.168.65.51 -
  tkcephmon03  RUNNING 0 -  192.168.65.48 -
  tkcinder RUNNING 0 -  192.168.65.49 -
  tkdash   RUNNING 0 -  192.168.65.50 -
  tkglance RUNNING 0 -  192.168.65.53 -
  tkjuju   RUNNING 0 -  192.168.65.15 -
  tkkeystone   RUNNING 0 -  192.168.65.54 -
  tkmysql  RUNNING 0 -  192.168.65.55 -
  tknova   RUNNING 0 -  192.168.65.56 -
  tkrabbit RUNNING 0 -  192.168.65.57 -
  tkswiftproxy RUNNING 0 -  192.168.65.58 -

  And compute nodes + neutrongw as kvm guests:

  inaddy@workstation:~$ virsh list --all | grep tk
   21tkcompute01running
   22tkcompute02running
   23tkcompute03running
   24tkneutrongwrunning

  All my LXC containers are on top of ZFS:

  Linux workstation 4.4.0-32-generic #51-Ubuntu SMP Tue Jul 19 18:09:07
  UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

  And my KVM guests are on top of ext4 + 1.2 raid0 stripped volume.

  I'm getting the lockups bellow (usually for mongod, from tkjuju
  container, the juju controller). After the first lockup appears
  (schedule timeout coming from zfs sync logic most likely), JuJu
  controller starts giving me errors on "update-status". From "juju
  status":

  glance/0error  idle1.25.6  10  9292/tcp   
tkglance   hook failed: "update-status"
  keystone/0  error  idle1.25.6  11 
tkkeystone hook failed: "update-status"
  mysql/0 error  idle1.25.6  12 
tkmysqlhook failed: "config-changed"
  neutron-api/0   error  idle1.25.6  4   9696/tcp   
tkneutrongwhook failed: "update-status"
  nova-compute/0  error  idle1.25.6  1  
tkcompute01hook failed: "update-status"
  nova-compute/1  error  idle1.25.6  2  
tkcompute02hook failed: "update-status"
  nova-compute/2  error  idle1.25.6  3  
tkcompute03hook failed: "update-status"

  Lockups:

  [105601.816578] INFO: task mongod:14480 blocked for more than 120 seconds.
  [105601.816583]   Tainted: P   O4.4.0-32-generic #51-Ubuntu
  [105601.816584] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [105601.816586] mongod  D 88010ec47ba8 0 14480  16855 
0x0100
  [105601.816590]  88010ec47ba8 57992eeb 880108e5ee00 
880108e58dc0
  [105601.816592]  88010ec48000 88081ecd6d00 7fff 
8182a600
  [105601.816594]  88010ec47d08 88010ec47bc0 81829e05 

  [105601.816596] Call Trace:
  [105601.816603]  [] ? bit_wait+0x60/0x60
  [105601.816606]  [] schedule+0x35/0x80
  [105601.816608]  [] schedule_timeout+0x1b5/0x270
  [105601.816612]  [] ? find_get_pages_tag+0x109/0x190
  [105601.816614]  [] ? bit_wait+0x60/0x60
  [105601.816616]  [] io_schedule_timeout+0xa4/0x110
  [105601.816618]  [] bit_wait_io+0x1b/0x70
  [105601.816620]  [] __wait_on_bit+0x5d/0x90
  [105601.816622]  [] wait_on_page_bit+0xcb/0xf0
  [105601.816625]  [] ? autoremove_wake_function+0x40/0x40
  [105601.816628]  [] __filemap_fdatawait_range+0xf3/0x160
  [105601.816630]  [] filemap_fdatawait_range+0x14/0x30
  [105601.816632]  [] filemap_write_and_wait_range+0x3f/0x70
  [105601.816682]  [] zpl_fsync+0x38/0x90 [zfs]
  [105601.816685]  [] vfs_fsync_range+0x4b/0xb0
  [105601.816687]  [] SyS_msync+0x17e/0x1f0
  [105601.816689]  [] entry_SYSCALL_64_fastpath+0x16/0x71
  [117121.961545] INFO: task txg_sync:4589 blocked for more than 120 seconds.
  [117121.961549]   Tainted: P   O4.4.0-32-generic #51-Ubuntu
  [117121.961550] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.