[Kernel-packages] [Bug 1676635] Re: [Hyper-V] Implement Hyper-V PTP Source

[Alex Ng]

Hi Marcelo,

I kicked the tires on 4.4.0-85-generic kernel from xenial-proposed. The
fixes look good. I see the PTP device and TimeSync is not causing "Time
has been changed" messages in systemd. I also see that apt-daily timer
is no longer being randomly delayed due to the clock changes.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1676635

Title:
  [Hyper-V] Implement Hyper-V PTP Source

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Xenial:
  Fix Committed
Status in linux source package in Yakkety:
  Fix Released
Status in linux source package in Zesty:
  Fix Released

Bug description:
  Please include the following upstream commit into lts-xenial, 16.04
  HWE, Yakkity, and Zesty. This will improve the behavior of timesync on
  Hyper-V hosts while simultaneously using network time sync protocols
  like NTP.

  
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/hv/hv_util.c?id=3716a49a81ba19dda7202633a68b28564ba95eb5

  commit 3716a49a81ba19dda7202633a68b28564ba95eb5
  Author: Vitaly Kuznetsov 
  Date:   Sat Feb 4 09:57:14 2017 -0700

  hv_utils: implement Hyper-V PTP source

  With TimeSync version 4 protocol support we started updating system time
  continuously through the whole lifetime of Hyper-V guests. Every 5 seconds
  there is a time sample from the host which triggers do_settimeofday[64]().
  While the time from the host is very accurate such adjustments may cause
  issues:
  - Time is jumping forward and backward, some applications may misbehave.
  - In case an NTP server runs in parallel and uses something else for time
    sync (network, PTP,...) system time will never converge.
  - Systemd starts annoying you by printing "Time has been changed" every 5
    seconds to the system log.

  Instead of doing in-kernel time adjustments offload the work to an
  NTP client by exposing TimeSync messages as a PTP device. Users may now
  decide what they want to use as a source.

  I tested the solution with chrony, the config was:

   refclock PHC /dev/ptp0 poll 3 dpoll -2 offset 0

  The result I'm seeing is accurate enough, the time delta between the guest
  and the host is almost always within [-10us, +10us], the in-kernel 
solution
  was giving us comparable results.

  I also tried implementing PPS device instead of PTP by using not currently
  used Hyper-V synthetic timers (we use only one of four for clockevent) but
  with PPS source only chrony wasn't able to give me the required accuracy,
  the delta often more that 100us.

  Signed-off-by: Vitaly Kuznetsov 
  Signed-off-by: K. Y. Srinivasan 
  Signed-off-by: Greg Kroah-Hartman 

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1676635/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1676635] Re: [Hyper-V] Implement Hyper-V PTP Source

[Alex Ng]

Please ignore my last comment #13, I intended it for another bug.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1676635

Title:
  [Hyper-V] Implement Hyper-V PTP Source

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Xenial:
  In Progress
Status in linux source package in Yakkety:
  Fix Released
Status in linux source package in Zesty:
  Fix Released

Bug description:
  Please include the following upstream commit into lts-xenial, 16.04
  HWE, Yakkity, and Zesty. This will improve the behavior of timesync on
  Hyper-V hosts while simultaneously using network time sync protocols
  like NTP.

  
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/hv/hv_util.c?id=3716a49a81ba19dda7202633a68b28564ba95eb5

  commit 3716a49a81ba19dda7202633a68b28564ba95eb5
  Author: Vitaly Kuznetsov 
  Date:   Sat Feb 4 09:57:14 2017 -0700

  hv_utils: implement Hyper-V PTP source

  With TimeSync version 4 protocol support we started updating system time
  continuously through the whole lifetime of Hyper-V guests. Every 5 seconds
  there is a time sample from the host which triggers do_settimeofday[64]().
  While the time from the host is very accurate such adjustments may cause
  issues:
  - Time is jumping forward and backward, some applications may misbehave.
  - In case an NTP server runs in parallel and uses something else for time
    sync (network, PTP,...) system time will never converge.
  - Systemd starts annoying you by printing "Time has been changed" every 5
    seconds to the system log.

  Instead of doing in-kernel time adjustments offload the work to an
  NTP client by exposing TimeSync messages as a PTP device. Users may now
  decide what they want to use as a source.

  I tested the solution with chrony, the config was:

   refclock PHC /dev/ptp0 poll 3 dpoll -2 offset 0

  The result I'm seeing is accurate enough, the time delta between the guest
  and the host is almost always within [-10us, +10us], the in-kernel 
solution
  was giving us comparable results.

  I also tried implementing PPS device instead of PTP by using not currently
  used Hyper-V synthetic timers (we use only one of four for clockevent) but
  with PPS source only chrony wasn't able to give me the required accuracy,
  the delta often more that 100us.

  Signed-off-by: Vitaly Kuznetsov 
  Signed-off-by: K. Y. Srinivasan 
  Signed-off-by: Greg Kroah-Hartman 

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1676635/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1676635] Re: [Hyper-V] Implement Hyper-V PTP Source

[Alex Ng]

Here are a few upstream commits that should also be included.

Drivers: hv: util: don't forget to init host_ts.lock
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/drivers/hv/hv_util.c?id=5a16dfc855127906fcd2935fb039bc8989313915

hv_utils: drop .getcrosststamp() support from PTP driver
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/drivers/hv/hv_util.c?id=4f9bac039a64f6306b613a0d90e6b7e75d7ab0c4

hv_utils: fix TimeSync work on pre-TimeSync-v4 hosts
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/drivers/hv/hv_util.c?id=1d10602d306cb7f70545b5e1166efc9409e7d384

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1676635

Title:
  [Hyper-V] Implement Hyper-V PTP Source

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Xenial:
  In Progress
Status in linux source package in Yakkety:
  Fix Released
Status in linux source package in Zesty:
  Fix Released

Bug description:
  Please include the following upstream commit into lts-xenial, 16.04
  HWE, Yakkity, and Zesty. This will improve the behavior of timesync on
  Hyper-V hosts while simultaneously using network time sync protocols
  like NTP.

  
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/hv/hv_util.c?id=3716a49a81ba19dda7202633a68b28564ba95eb5

  commit 3716a49a81ba19dda7202633a68b28564ba95eb5
  Author: Vitaly Kuznetsov 
  Date:   Sat Feb 4 09:57:14 2017 -0700

  hv_utils: implement Hyper-V PTP source

  With TimeSync version 4 protocol support we started updating system time
  continuously through the whole lifetime of Hyper-V guests. Every 5 seconds
  there is a time sample from the host which triggers do_settimeofday[64]().
  While the time from the host is very accurate such adjustments may cause
  issues:
  - Time is jumping forward and backward, some applications may misbehave.
  - In case an NTP server runs in parallel and uses something else for time
    sync (network, PTP,...) system time will never converge.
  - Systemd starts annoying you by printing "Time has been changed" every 5
    seconds to the system log.

  Instead of doing in-kernel time adjustments offload the work to an
  NTP client by exposing TimeSync messages as a PTP device. Users may now
  decide what they want to use as a source.

  I tested the solution with chrony, the config was:

   refclock PHC /dev/ptp0 poll 3 dpoll -2 offset 0

  The result I'm seeing is accurate enough, the time delta between the guest
  and the host is almost always within [-10us, +10us], the in-kernel 
solution
  was giving us comparable results.

  I also tried implementing PPS device instead of PTP by using not currently
  used Hyper-V synthetic timers (we use only one of four for clockevent) but
  with PPS source only chrony wasn't able to give me the required accuracy,
  the delta often more that 100us.

  Signed-off-by: Vitaly Kuznetsov 
  Signed-off-by: K. Y. Srinivasan 
  Signed-off-by: Greg Kroah-Hartman 

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1676635/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1676635] Re: [Hyper-V] Implement Hyper-V PTP Source

[Alex Ng]

@Alberto Ornaghi.

Are you running in Azure or in your own Hyper-V host? If you have access
to the Hyper-V host, you can disable the TimeSync integration service
from the VM settings.

Otherwise, you can create a script that runs "echo 2dd1ce17-079e-
403c-b352-a1921ee207ee > /sys/bus/vmbus/drivers/hv_util/unbind" upon
startup. This disables TimeSync from within the guest VM.

Hope this helps,
Alex

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1676635

Title:
  [Hyper-V] Implement Hyper-V PTP Source

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Xenial:
  In Progress
Status in linux source package in Yakkety:
  Fix Committed
Status in linux source package in Zesty:
  Fix Released

Bug description:
  Please include the following upstream commit into lts-xenial, 16.04
  HWE, Yakkity, and Zesty. This will improve the behavior of timesync on
  Hyper-V hosts while simultaneously using network time sync protocols
  like NTP.

  
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/hv/hv_util.c?id=3716a49a81ba19dda7202633a68b28564ba95eb5

  commit 3716a49a81ba19dda7202633a68b28564ba95eb5
  Author: Vitaly Kuznetsov 
  Date:   Sat Feb 4 09:57:14 2017 -0700

  hv_utils: implement Hyper-V PTP source

  With TimeSync version 4 protocol support we started updating system time
  continuously through the whole lifetime of Hyper-V guests. Every 5 seconds
  there is a time sample from the host which triggers do_settimeofday[64]().
  While the time from the host is very accurate such adjustments may cause
  issues:
  - Time is jumping forward and backward, some applications may misbehave.
  - In case an NTP server runs in parallel and uses something else for time
    sync (network, PTP,...) system time will never converge.
  - Systemd starts annoying you by printing "Time has been changed" every 5
    seconds to the system log.

  Instead of doing in-kernel time adjustments offload the work to an
  NTP client by exposing TimeSync messages as a PTP device. Users may now
  decide what they want to use as a source.

  I tested the solution with chrony, the config was:

   refclock PHC /dev/ptp0 poll 3 dpoll -2 offset 0

  The result I'm seeing is accurate enough, the time delta between the guest
  and the host is almost always within [-10us, +10us], the in-kernel 
solution
  was giving us comparable results.

  I also tried implementing PPS device instead of PTP by using not currently
  used Hyper-V synthetic timers (we use only one of four for clockevent) but
  with PPS source only chrony wasn't able to give me the required accuracy,
  the delta often more that 100us.

  Signed-off-by: Vitaly Kuznetsov 
  Signed-off-by: K. Y. Srinivasan 
  Signed-off-by: Greg Kroah-Hartman 

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1676635/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

[Alex Ng]

The patch from @faulpeltz hasn't been mainlined because of some feedback
that it shouldn't have to close and reopen the /dev/vmbus/hv_vss device
after failure.

I addressed this comment this in a modified version of @faulpeltz's
patch (see comment #345).

I haven't heard from @faulpeltz whether he's tested it. From my testing,
it seems fine and I expect to resubmit it upstream. If we can make it a
SAUCE patch, that would be nice.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250

Title:
  [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based
  Backups

Status in linux package in Ubuntu:
  In Progress
Status in linux source package in Trusty:
  In Progress
Status in linux source package in Xenial:
  In Progress
Status in linux source package in Yakkety:
  In Progress
Status in linux source package in Zesty:
  In Progress

Bug description:
  Customers have reported running various versions of Ubuntu 14.04.2 LTS
  on Generation 2 Hyper-V Hosts.On a random Basis, the file system
  will be mounted Read-Only due to a "disk error" (which really isn't
  the case here).As a result, they must reboot the Ubuntu guest to
  get the file system to mount RW again.

  The Error seen are the following:
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed. The Linux SCSI layer does not automatically adjust these 
parameters.
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc 
vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc 
vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed.  The Linux SCSI layer does not automatically adjust these 
parameters.

  This relates to the VSS "Windows Server Backup" process that kicks off at 
midnight on the host and finishes an hour and half later.   
  Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct 
kernel version we have.   We're currently running kernel version 
3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- 
We see the same errors on both.
  As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 
hyper-v system because of this.   We can stop the backup process and all is 
good, but we need nightly backups to image all of our VM's.   All the windows 
guests have no issues of course.   We also have some CentOS based guests 
running without issues from what we've seen.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

[Alex Ng]

Hi @jsalisbury,

Any status update on the patches for this issue?

It appears the test kernels have resolved the issue.

Let us know if you need additional testing or have questions about the
patches.

Thanks,
Alex

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250

Title:
  [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based
  Backups

Status in linux package in Ubuntu:
  In Progress
Status in linux source package in Trusty:
  In Progress
Status in linux source package in Xenial:
  In Progress
Status in linux source package in Yakkety:
  In Progress
Status in linux source package in Zesty:
  In Progress

Bug description:
  Customers have reported running various versions of Ubuntu 14.04.2 LTS
  on Generation 2 Hyper-V Hosts.On a random Basis, the file system
  will be mounted Read-Only due to a "disk error" (which really isn't
  the case here).As a result, they must reboot the Ubuntu guest to
  get the file system to mount RW again.

  The Error seen are the following:
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed. The Linux SCSI layer does not automatically adjust these 
parameters.
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc 
vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc 
vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed.  The Linux SCSI layer does not automatically adjust these 
parameters.

  This relates to the VSS "Windows Server Backup" process that kicks off at 
midnight on the host and finishes an hour and half later.   
  Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct 
kernel version we have.   We're currently running kernel version 
3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- 
We see the same errors on both.
  As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 
hyper-v system because of this.   We can stop the backup process and all is 
good, but we need nightly backups to image all of our VM's.   All the windows 
guests have no issues of course.   We also have some CentOS based guests 
running without issues from what we've seen.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

[Alex Ng]

Thanks Joseph for compiling the list. The patches you've outlined should
be sufficient.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250

Title:
  [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based
  Backups

Status in linux package in Ubuntu:
  In Progress
Status in linux source package in Trusty:
  In Progress
Status in linux source package in Xenial:
  In Progress
Status in linux source package in Yakkety:
  In Progress
Status in linux source package in Zesty:
  In Progress

Bug description:
  Customers have reported running various versions of Ubuntu 14.04.2 LTS
  on Generation 2 Hyper-V Hosts.On a random Basis, the file system
  will be mounted Read-Only due to a "disk error" (which really isn't
  the case here).As a result, they must reboot the Ubuntu guest to
  get the file system to mount RW again.

  The Error seen are the following:
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed. The Linux SCSI layer does not automatically adjust these 
parameters.
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc 
vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc 
vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed.  The Linux SCSI layer does not automatically adjust these 
parameters.

  This relates to the VSS "Windows Server Backup" process that kicks off at 
midnight on the host and finishes an hour and half later.   
  Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct 
kernel version we have.   We're currently running kernel version 
3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- 
We see the same errors on both.
  As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 
hyper-v system because of this.   We can stop the backup process and all is 
good, but we need nightly backups to image all of our VM's.   All the windows 
guests have no issues of course.   We also have some CentOS based guests 
running without issues from what we've seen.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

[Alex Ng]

@faulpeltz
I agree. Your patch should also be included in case a FREEZE operation does 
exceed the increased timeout.

I've attached a modified version of your patch that addresses some
concerns we had offline. Could you give it a try?

** Patch added: 
"0004-Tools-hv-vss-Thaw-the-filesystem-and-continue-after-.patch"
   
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+attachment/4778824/+files/0004-Tools-hv-vss-Thaw-the-filesystem-and-continue-after-.patch

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250

Title:
  [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based
  Backups

Status in linux package in Ubuntu:
  In Progress
Status in linux source package in Trusty:
  In Progress
Status in linux source package in Xenial:
  In Progress
Status in linux source package in Yakkety:
  In Progress

Bug description:
  Customers have reported running various versions of Ubuntu 14.04.2 LTS
  on Generation 2 Hyper-V Hosts.On a random Basis, the file system
  will be mounted Read-Only due to a "disk error" (which really isn't
  the case here).As a result, they must reboot the Ubuntu guest to
  get the file system to mount RW again.

  The Error seen are the following:
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed. The Linux SCSI layer does not automatically adjust these 
parameters.
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc 
vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc 
vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed.  The Linux SCSI layer does not automatically adjust these 
parameters.

  This relates to the VSS "Windows Server Backup" process that kicks off at 
midnight on the host and finishes an hour and half later.   
  Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct 
kernel version we have.   We're currently running kernel version 
3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- 
We see the same errors on both.
  As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 
hyper-v system because of this.   We can stop the backup process and all is 
good, but we need nightly backups to image all of our VM's.   All the windows 
guests have no issues of course.   We also have some CentOS based guests 
running without issues from what we've seen.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

[Alex Ng]

I'll let @jsalisbury comment on the status of his backported patches.

@jsalisbury, I'd also take this recently submitted patch from upstream
kernel as well. It ensures that the VSS driver doesn't timeout any long
running FREEZE operations too early. This should preclude the need for
faulpeltz's patch in most cases, unless the FREEZE operation takes
extraordinarily long (> 15 mins).

https://git.kernel.org/cgit/linux/kernel/git/next/linux-
next.git/commit/drivers/hv?id=b357fd3908c1191f2f56e38aa77f2aecdae18bc8

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250

Title:
  [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based
  Backups

Status in linux package in Ubuntu:
  In Progress
Status in linux source package in Trusty:
  In Progress
Status in linux source package in Xenial:
  In Progress
Status in linux source package in Yakkety:
  In Progress

Bug description:
  Customers have reported running various versions of Ubuntu 14.04.2 LTS
  on Generation 2 Hyper-V Hosts.On a random Basis, the file system
  will be mounted Read-Only due to a "disk error" (which really isn't
  the case here).As a result, they must reboot the Ubuntu guest to
  get the file system to mount RW again.

  The Error seen are the following:
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed. The Linux SCSI layer does not automatically adjust these 
parameters.
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc 
vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc 
vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed.  The Linux SCSI layer does not automatically adjust these 
parameters.

  This relates to the VSS "Windows Server Backup" process that kicks off at 
midnight on the host and finishes an hour and half later.   
  Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct 
kernel version we have.   We're currently running kernel version 
3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- 
We see the same errors on both.
  As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 
hyper-v system because of this.   We can stop the backup process and all is 
good, but we need nightly backups to image all of our VM's.   All the windows 
guests have no issues of course.   We also have some CentOS based guests 
running without issues from what we've seen.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

[Alex Ng]

Thanks @faulpeltz for the info.

I sent you a private message with the list of maintainers to send the
patch (trying to avoid pasting it here in case spam bots can go through
this archive).

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250

Title:
  [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based
  Backups

Status in linux package in Ubuntu:
  In Progress
Status in linux source package in Trusty:
  In Progress
Status in linux source package in Xenial:
  In Progress
Status in linux source package in Yakkety:
  In Progress

Bug description:
  Customers have reported running various versions of Ubuntu 14.04.2 LTS
  on Generation 2 Hyper-V Hosts.On a random Basis, the file system
  will be mounted Read-Only due to a "disk error" (which really isn't
  the case here).As a result, they must reboot the Ubuntu guest to
  get the file system to mount RW again.

  The Error seen are the following:
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed. The Linux SCSI layer does not automatically adjust these 
parameters.
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc 
vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc 
vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed.  The Linux SCSI layer does not automatically adjust these 
parameters.

  This relates to the VSS "Windows Server Backup" process that kicks off at 
midnight on the host and finishes an hour and half later.   
  Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct 
kernel version we have.   We're currently running kernel version 
3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- 
We see the same errors on both.
  As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 
hyper-v system because of this.   We can stop the backup process and all is 
good, but we need nightly backups to image all of our VM's.   All the windows 
guests have no issues of course.   We also have some CentOS based guests 
running without issues from what we've seen.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

[Alex Ng]

Hi @faulpeltz,

A few questions/comments about your patch:

1) Can you submit your patch to the upstream kernel?
2) Under load, were you able to measure how long the FIFREEZE operation took 
before it succeeded? I'm trying to see if we can increase the timeout of the 
kernel driver before it hits the error condition that you encountered.

Thanks,
Alex

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250

Title:
  [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based
  Backups

Status in linux package in Ubuntu:
  In Progress
Status in linux source package in Trusty:
  In Progress
Status in linux source package in Xenial:
  In Progress
Status in linux source package in Yakkety:
  In Progress

Bug description:
  Customers have reported running various versions of Ubuntu 14.04.2 LTS
  on Generation 2 Hyper-V Hosts.On a random Basis, the file system
  will be mounted Read-Only due to a "disk error" (which really isn't
  the case here).As a result, they must reboot the Ubuntu guest to
  get the file system to mount RW again.

  The Error seen are the following:
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed. The Linux SCSI layer does not automatically adjust these 
parameters.
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc 
vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc 
vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed.  The Linux SCSI layer does not automatically adjust these 
parameters.

  This relates to the VSS "Windows Server Backup" process that kicks off at 
midnight on the host and finishes an hour and half later.   
  Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct 
kernel version we have.   We're currently running kernel version 
3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- 
We see the same errors on both.
  As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 
hyper-v system because of this.   We can stop the backup process and all is 
good, but we need nightly backups to image all of our VM's.   All the windows 
guests have no issues of course.   We also have some CentOS based guests 
running without issues from what we've seen.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1584597] Re: Hyper-V Memory Ballooning re-broken in 16.04

[Alex Ng]

Evan, from your screenshots, I can see that the memory demand values and
the /proc/meminfo values do match closely to each other.

On the 16.04.01 VM, /proc/meminfo shows Committed_AS value is ~1800MB.
The balloon driver then adds a buffer of ~300MB (this buffer is
calculated according to how much total memory is assigned to the VM). So
~1800MB + ~300MB = 2100MB matches closely with the 2088 MB memory demand
that you're seeing from the host.

A similar calculation on the 15.10 VM also shows that the meminfo values
match closely with the reported memory demand. (~2900MB committed +
~500MB buffer = 3400MB compared to 3600MB)

The main culprit in both cases appears to be the large amount of
committed memory. Would be worth investigating what the cause of this
memory usage is.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1584597

Title:
  Hyper-V Memory Ballooning re-broken in 16.04

Status in linux package in Ubuntu:
  Triaged
Status in linux source package in Xenial:
  Triaged

Bug description:
  Regression: #1294283 fixed memory ballooning in Hyper-V in 14.04, but
  the bug has returned in 16.04.

  
  Steps to reproduce:

  1. Create Gen2 Hyper-V VM
  2. Install Ubuntu 16.04
  3. modprobe hv_balloon
  Memory usage according to Hyper-V Manager will not change

  
  This has been reproduced on 2 fresh Ubuntu Server 16.04 instances.

  I installed linux-generic-xenial on a 14.04 instance, rebooted,
  modprobe'd hv_balloon and memory usage in Hyper-V Manager reduced to
  the correct level.

  Environment:
  Host: Windows NanoServer 2016

  Ubuntu Server 16.04 version_signature: `Ubuntu 4.4.0-21.37-generic 4.4.6`
  Ubuntu Server 14.04 version_signature: `Ubuntu 4.4.0-22.40~14.04.1-generic 
4.4.8`

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1584597/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1584597] Re: Hyper-V Memory Ballooning re-broken in 16.04

[Alex Ng]

Bernhard(galmok), the memory corruption you're seeing is likely a known
issue that happens when Linux guests are running on Windows Server 2012.
This was fixed recently in the upstream Linux kernels, so hopefully a
future update will have these patches.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1584597

Title:
  Hyper-V Memory Ballooning re-broken in 16.04

Status in linux package in Ubuntu:
  Triaged
Status in linux source package in Xenial:
  Triaged

Bug description:
  Regression: #1294283 fixed memory ballooning in Hyper-V in 14.04, but
  the bug has returned in 16.04.

  
  Steps to reproduce:

  1. Create Gen2 Hyper-V VM
  2. Install Ubuntu 16.04
  3. modprobe hv_balloon
  Memory usage according to Hyper-V Manager will not change

  
  This has been reproduced on 2 fresh Ubuntu Server 16.04 instances.

  I installed linux-generic-xenial on a 14.04 instance, rebooted,
  modprobe'd hv_balloon and memory usage in Hyper-V Manager reduced to
  the correct level.

  Environment:
  Host: Windows NanoServer 2016

  Ubuntu Server 16.04 version_signature: `Ubuntu 4.4.0-21.37-generic 4.4.6`
  Ubuntu Server 14.04 version_signature: `Ubuntu 4.4.0-22.40~14.04.1-generic 
4.4.8`

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1584597/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

[Alex Ng]

One reason you may see the timeout messages is due to having a mismatch
between the user-space hv_vss_daemon version and the kernel version.

Can you rebuild the user-space hv_vss_daemon under the source tree's
tools/hv directory and replace the one provided that's provided in
Ubuntu by default?

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250

Title:
  [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based
  Backups

Status in linux package in Ubuntu:
  In Progress
Status in linux source package in Trusty:
  In Progress
Status in linux source package in Vivid:
  In Progress
Status in linux source package in Wily:
  In Progress
Status in linux source package in Xenial:
  In Progress
Status in linux source package in Yakkety:
  In Progress

Bug description:
  Customers have reported running various versions of Ubuntu 14.04.2 LTS
  on Generation 2 Hyper-V Hosts.On a random Basis, the file system
  will be mounted Read-Only due to a "disk error" (which really isn't
  the case here).As a result, they must reboot the Ubuntu guest to
  get the file system to mount RW again.

  The Error seen are the following:
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed. The Linux SCSI layer does not automatically adjust these 
parameters.
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc 
vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc 
vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed.  The Linux SCSI layer does not automatically adjust these 
parameters.

  This relates to the VSS "Windows Server Backup" process that kicks off at 
midnight on the host and finishes an hour and half later.   
  Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct 
kernel version we have.   We're currently running kernel version 
3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- 
We see the same errors on both.
  As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 
hyper-v system because of this.   We can stop the backup process and all is 
good, but we need nightly backups to image all of our VM's.   All the windows 
guests have no issues of course.   We also have some CentOS based guests 
running without issues from what we've seen.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

[Alex Ng]

And in response to Joseph's comment #309, the second patch shouldn't be
required as it's related to a feature introduced in Windows Server 2016
(I'm assuming you folks are testing in Windows Server 2012 R2).

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250

Title:
  [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based
  Backups

Status in linux package in Ubuntu:
  In Progress
Status in linux source package in Trusty:
  In Progress
Status in linux source package in Vivid:
  In Progress
Status in linux source package in Wily:
  In Progress
Status in linux source package in Xenial:
  In Progress
Status in linux source package in Yakkety:
  In Progress

Bug description:
  Customers have reported running various versions of Ubuntu 14.04.2 LTS
  on Generation 2 Hyper-V Hosts.On a random Basis, the file system
  will be mounted Read-Only due to a "disk error" (which really isn't
  the case here).As a result, they must reboot the Ubuntu guest to
  get the file system to mount RW again.

  The Error seen are the following:
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed. The Linux SCSI layer does not automatically adjust these 
parameters.
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc 
vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc 
vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed.  The Linux SCSI layer does not automatically adjust these 
parameters.

  This relates to the VSS "Windows Server Backup" process that kicks off at 
midnight on the host and finishes an hour and half later.   
  Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct 
kernel version we have.   We're currently running kernel version 
3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- 
We see the same errors on both.
  As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 
hyper-v system because of this.   We can stop the backup process and all is 
good, but we need nightly backups to image all of our VM's.   All the windows 
guests have no issues of course.   We also have some CentOS based guests 
running without issues from what we've seen.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

[Alex Ng]

Might be worth trying this patchset: https://lkml.org/lkml/2016/8/18/859

The first patch in the set addresses some issues with VSS that would
cause it to take a long time to initiate backup (and even may timeout).

The second patch is not necessary (but will need VSS daemon to be
replaced if you choose to apply it).

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250

Title:
  [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based
  Backups

Status in linux package in Ubuntu:
  In Progress
Status in linux source package in Trusty:
  In Progress
Status in linux source package in Vivid:
  In Progress
Status in linux source package in Wily:
  In Progress
Status in linux source package in Xenial:
  In Progress
Status in linux source package in Yakkety:
  In Progress

Bug description:
  Customers have reported running various versions of Ubuntu 14.04.2 LTS
  on Generation 2 Hyper-V Hosts.On a random Basis, the file system
  will be mounted Read-Only due to a "disk error" (which really isn't
  the case here).As a result, they must reboot the Ubuntu guest to
  get the file system to mount RW again.

  The Error seen are the following:
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed. The Linux SCSI layer does not automatically adjust these 
parameters.
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc 
vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc 
vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed.  The Linux SCSI layer does not automatically adjust these 
parameters.

  This relates to the VSS "Windows Server Backup" process that kicks off at 
midnight on the host and finishes an hour and half later.   
  Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct 
kernel version we have.   We're currently running kernel version 
3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- 
We see the same errors on both.
  As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 
hyper-v system because of this.   We can stop the backup process and all is 
good, but we need nightly backups to image all of our VM's.   All the windows 
guests have no issues of course.   We also have some CentOS based guests 
running without issues from what we've seen.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1584597] Re: Hyper-V Memory Ballooning re-broken in 16.04

[Alex Ng]

> Committed_AS: 7291440 kB

The amount of committed memory reported by meminfo is about 7GB. So it
shouldn't be surprising to see that the demand as seen by Hyper-V is
also high.

In general, the Hyper-V demand is calculated as Committed_AS value plus
some buffer(roughly 700MB if you have 8GB of total ram). So for the
above meminfo report, I would expect Hyper-V to show that about ~7-8GB
is in use.

Can you show /proc/meminfo when things have calmed down (as well as a
screenshot of what Hyper-V is showing at the same time)?

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1584597

Title:
  Hyper-V Memory Ballooning re-broken in 16.04

Status in linux package in Ubuntu:
  Triaged
Status in linux source package in Xenial:
  Triaged

Bug description:
  Regression: #1294283 fixed memory ballooning in Hyper-V in 14.04, but
  the bug has returned in 16.04.

  
  Steps to reproduce:

  1. Create Gen2 Hyper-V VM
  2. Install Ubuntu 16.04
  3. modprobe hv_balloon
  Memory usage according to Hyper-V Manager will not change

  
  This has been reproduced on 2 fresh Ubuntu Server 16.04 instances.

  I installed linux-generic-xenial on a 14.04 instance, rebooted,
  modprobe'd hv_balloon and memory usage in Hyper-V Manager reduced to
  the correct level.

  Environment:
  Host: Windows NanoServer 2016

  Ubuntu Server 16.04 version_signature: `Ubuntu 4.4.0-21.37-generic 4.4.6`
  Ubuntu Server 14.04 version_signature: `Ubuntu 4.4.0-22.40~14.04.1-generic 
4.4.8`

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1584597/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1584597] Re: Hyper-V Memory Ballooning re-broken in 16.04

[Alex Ng]

The hv_balloon driver hasn't changed between 15.10 and 16.04, so there
shouldn't be any difference in the way the driver reports demand to
Hyper-V.

To provide a further breakdown of the memory usage, can you show the
output of "cat /proc/meminfo"?

Might help to compare this info between 16.04 and 15.10.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1584597

Title:
  Hyper-V Memory Ballooning re-broken in 16.04

Status in linux package in Ubuntu:
  Triaged
Status in linux source package in Xenial:
  Triaged

Bug description:
  Regression: #1294283 fixed memory ballooning in Hyper-V in 14.04, but
  the bug has returned in 16.04.

  
  Steps to reproduce:

  1. Create Gen2 Hyper-V VM
  2. Install Ubuntu 16.04
  3. modprobe hv_balloon
  Memory usage according to Hyper-V Manager will not change

  
  This has been reproduced on 2 fresh Ubuntu Server 16.04 instances.

  I installed linux-generic-xenial on a 14.04 instance, rebooted,
  modprobe'd hv_balloon and memory usage in Hyper-V Manager reduced to
  the correct level.

  Environment:
  Host: Windows NanoServer 2016

  Ubuntu Server 16.04 version_signature: `Ubuntu 4.4.0-21.37-generic 4.4.6`
  Ubuntu Server 14.04 version_signature: `Ubuntu 4.4.0-22.40~14.04.1-generic 
4.4.8`

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1584597/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1584597] Re: Hyper-V Memory Ballooning re-broken in 16.04

[Alex Ng]

One other caveat I should mention:

Even if the VM has reduced its memory consumption, Hyper-V does not
necessarily reclaim the unused memory from that VM.

Generally, Hyper-V reclaims unused memory from a VM if it's seeing
memory pressure from other VMs or if the Hyper-V host itself is seeing
memory pressure.

The behavior you are seeing might be expected. Indeed, if your physical
host has a lot of physical memory, Hyper-V may decide that enough memory
is available to satisfy all VMs and it doesn't need to reclaim.

If say a bunch of other VMs come online that also need a lot of memory,
perhaps we would see Hyper-V start to reclaim the unused memory from the
existing VMs.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1584597

Title:
  Hyper-V Memory Ballooning re-broken in 16.04

Status in linux package in Ubuntu:
  Triaged
Status in linux source package in Xenial:
  Triaged

Bug description:
  Regression: #1294283 fixed memory ballooning in Hyper-V in 14.04, but
  the bug has returned in 16.04.

  
  Steps to reproduce:

  1. Create Gen2 Hyper-V VM
  2. Install Ubuntu 16.04
  3. modprobe hv_balloon
  Memory usage according to Hyper-V Manager will not change

  
  This has been reproduced on 2 fresh Ubuntu Server 16.04 instances.

  I installed linux-generic-xenial on a 14.04 instance, rebooted,
  modprobe'd hv_balloon and memory usage in Hyper-V Manager reduced to
  the correct level.

  Environment:
  Host: Windows NanoServer 2016

  Ubuntu Server 16.04 version_signature: `Ubuntu 4.4.0-21.37-generic 4.4.6`
  Ubuntu Server 14.04 version_signature: `Ubuntu 4.4.0-22.40~14.04.1-generic 
4.4.8`

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1584597/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1584597] Re: Hyper-V Memory Ballooning re-broken in 16.04

[Alex Ng]

Thanks. In both VMs, it looks like the buff/cached memory dropped by
2GB. After some time, did the Hyper-V host eventually reclaim this
memory? Can you check if the Hyper-V host's assigned memory also drops
after some time?

Also, can you tell us if you tried this on an older build where it
doesn't repro?

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1584597

Title:
  Hyper-V Memory Ballooning re-broken in 16.04

Status in linux package in Ubuntu:
  Triaged
Status in linux source package in Xenial:
  Triaged

Bug description:
  Regression: #1294283 fixed memory ballooning in Hyper-V in 14.04, but
  the bug has returned in 16.04.

  
  Steps to reproduce:

  1. Create Gen2 Hyper-V VM
  2. Install Ubuntu 16.04
  3. modprobe hv_balloon
  Memory usage according to Hyper-V Manager will not change

  
  This has been reproduced on 2 fresh Ubuntu Server 16.04 instances.

  I installed linux-generic-xenial on a 14.04 instance, rebooted,
  modprobe'd hv_balloon and memory usage in Hyper-V Manager reduced to
  the correct level.

  Environment:
  Host: Windows NanoServer 2016

  Ubuntu Server 16.04 version_signature: `Ubuntu 4.4.0-21.37-generic 4.4.6`
  Ubuntu Server 14.04 version_signature: `Ubuntu 4.4.0-22.40~14.04.1-generic 
4.4.8`

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1584597/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1584597] Re: Hyper-V Memory Ballooning re-broken in 16.04

[Alex Ng]

In both screenshots, it appears that the "buff/cache" value is very
high.

Perhaps the balloon driver is reporting the "buff/cache" as in use to
the host. Can you try running following commands to flush write buffers
and free page cache?

1) sync
2) echo 3 > /proc/sys/vm/drop_caches

After running the above, do you see that the "buff/cache" value
decreases and does this result in the memory being reclaimed by Hyper-V?

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1584597

Title:
  Hyper-V Memory Ballooning re-broken in 16.04

Status in linux package in Ubuntu:
  Triaged
Status in linux source package in Xenial:
  Triaged

Bug description:
  Regression: #1294283 fixed memory ballooning in Hyper-V in 14.04, but
  the bug has returned in 16.04.

  
  Steps to reproduce:

  1. Create Gen2 Hyper-V VM
  2. Install Ubuntu 16.04
  3. modprobe hv_balloon
  Memory usage according to Hyper-V Manager will not change

  
  This has been reproduced on 2 fresh Ubuntu Server 16.04 instances.

  I installed linux-generic-xenial on a 14.04 instance, rebooted,
  modprobe'd hv_balloon and memory usage in Hyper-V Manager reduced to
  the correct level.

  Environment:
  Host: Windows NanoServer 2016

  Ubuntu Server 16.04 version_signature: `Ubuntu 4.4.0-21.37-generic 4.4.6`
  Ubuntu Server 14.04 version_signature: `Ubuntu 4.4.0-22.40~14.04.1-generic 
4.4.8`

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1584597/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1584597] Re: Hyper-V Memory Ballooning re-broken in 16.04

[Alex Ng]

Hi Jason,

As Josh mentioned, the screenshot shows that demand as seen by Hyper-V
is still quite high compared to what is displayed inside the guest. If
this number doesn't settle, then Hyper-V would have no reason to reclaim
memory. Does the Hyper-V memory demand number ever settle after some
time?

With the "free" command, you should be seeing the following if memory is
being returned to the host successfully:

- Because Linux guests return memory via ballooning instead of hot-
unplug; you'll never see the "total" memory decrease (i.e. it'll always
reflect the spike)

- Instead, you should see that "used" memory increases and "free" memory
decreases. This indicates the kernel allocated some memory so no one can
use it, and that Hyper-V has reclaimed this memory. (i.e. the used
memory should increase by roughly the amount that the host has
reclaimed)

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1584597

Title:
  Hyper-V Memory Ballooning re-broken in 16.04

Status in linux package in Ubuntu:
  Triaged
Status in linux source package in Xenial:
  Triaged

Bug description:
  Regression: #1294283 fixed memory ballooning in Hyper-V in 14.04, but
  the bug has returned in 16.04.

  
  Steps to reproduce:

  1. Create Gen2 Hyper-V VM
  2. Install Ubuntu 16.04
  3. modprobe hv_balloon
  Memory usage according to Hyper-V Manager will not change

  
  This has been reproduced on 2 fresh Ubuntu Server 16.04 instances.

  I installed linux-generic-xenial on a 14.04 instance, rebooted,
  modprobe'd hv_balloon and memory usage in Hyper-V Manager reduced to
  the correct level.

  Environment:
  Host: Windows NanoServer 2016

  Ubuntu Server 16.04 version_signature: `Ubuntu 4.4.0-21.37-generic 4.4.6`
  Ubuntu Server 14.04 version_signature: `Ubuntu 4.4.0-22.40~14.04.1-generic 
4.4.8`

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1584597/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

[Alex Ng]

Also, to add one other thing. There were a bunch of commits made
upstream to the storvsc driver in the last few months.

Can we try them out to see if they have any impact on this issue? In
particular:

1) 81988a0e6b031bc80da15257201810ddcf989e64 - storvsc: get rid of bounce buffer
2) 3209f9d780d137cdb54c85e0a776cb19e723a170 - scsi: storvsc: Fix a bug in the 
handling of SRB status flags
3) 03996f2064a5c5b7c1bd942794d622179acf2d61 - storvsc: Tighten up the interrupt 
path

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250

Title:
  [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based
  Backups

Status in linux package in Ubuntu:
  In Progress
Status in linux source package in Trusty:
  In Progress
Status in linux source package in Vivid:
  In Progress
Status in linux source package in Wily:
  In Progress
Status in linux source package in Xenial:
  In Progress

Bug description:
  Customers have reported running various versions of Ubuntu 14.04.2 LTS
  on Generation 2 Hyper-V Hosts.On a random Basis, the file system
  will be mounted Read-Only due to a "disk error" (which really isn't
  the case here).As a result, they must reboot the Ubuntu guest to
  get the file system to mount RW again.

  The Error seen are the following:
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed. The Linux SCSI layer does not automatically adjust these 
parameters.
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc 
vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc 
vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed.  The Linux SCSI layer does not automatically adjust these 
parameters.

  This relates to the VSS "Windows Server Backup" process that kicks off at 
midnight on the host and finishes an hour and half later.   
  Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct 
kernel version we have.   We're currently running kernel version 
3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- 
We see the same errors on both.
  As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 
hyper-v system because of this.   We can stop the backup process and all is 
good, but we need nightly backups to image all of our VM's.   All the windows 
guests have no issues of course.   We also have some CentOS based guests 
running without issues from what we've seen.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

[Alex Ng]

To help see if this is an issue in the hv_storvsc driver; I took the
storvsc driver code from 3.13.0-34.60 (presumably a good build) and
applied it to the 4.2.0-27 kernel (presumably a bad build). Ran tiobench
with backups and was able to repro after about 48 hours.

This implies that either:
1) The bad commit is not in the hv_storvsc driver.

2) 3.13.0-34 is not a good build as we previously thought. I'll wait to
hear if other people can repro this issue on 3.13.0-34.

The storvsc driver code I took for this snapshot =>
http://kernel.ubuntu.com/git/ubuntu/ubuntu-
trusty.git/tree/drivers/scsi/storvsc_drv.c?id=3402ec8cbd984e8e2a1f99b51ec6a15a23328223

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250

Title:
  [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based
  Backups

Status in linux package in Ubuntu:
  In Progress
Status in linux source package in Trusty:
  In Progress
Status in linux source package in Vivid:
  In Progress
Status in linux source package in Wily:
  In Progress
Status in linux source package in Xenial:
  In Progress

Bug description:
  Customers have reported running various versions of Ubuntu 14.04.2 LTS
  on Generation 2 Hyper-V Hosts.On a random Basis, the file system
  will be mounted Read-Only due to a "disk error" (which really isn't
  the case here).As a result, they must reboot the Ubuntu guest to
  get the file system to mount RW again.

  The Error seen are the following:
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed. The Linux SCSI layer does not automatically adjust these 
parameters.
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc 
vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc 
vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed.  The Linux SCSI layer does not automatically adjust these 
parameters.

  This relates to the VSS "Windows Server Backup" process that kicks off at 
midnight on the host and finishes an hour and half later.   
  Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct 
kernel version we have.   We're currently running kernel version 
3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- 
We see the same errors on both.
  As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 
hyper-v system because of this.   We can stop the backup process and all is 
good, but we need nightly backups to image all of our VM's.   All the windows 
guests have no issues of course.   We also have some CentOS based guests 
running without issues from what we've seen.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

[Alex Ng]

Don't think this has been asked before, but has anyone had a repro when
backups were turned off? Or does this only happen when backups are
enabled?

I'm verifying this on my own as well, but if this happens regardless of
whether backup is enabled/disabled; then it'll help us narrow down the
cause further to either a storage issue (more likely) or a VSS backup
issue.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250

Title:
  [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based
  Backups

Status in linux package in Ubuntu:
  In Progress
Status in linux source package in Trusty:
  In Progress
Status in linux source package in Vivid:
  In Progress
Status in linux source package in Wily:
  In Progress
Status in linux source package in Xenial:
  In Progress

Bug description:
  Customers have reported running various versions of Ubuntu 14.04.2 LTS
  on Generation 2 Hyper-V Hosts.On a random Basis, the file system
  will be mounted Read-Only due to a "disk error" (which really isn't
  the case here).As a result, they must reboot the Ubuntu guest to
  get the file system to mount RW again.

  The Error seen are the following:
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed. The Linux SCSI layer does not automatically adjust these 
parameters.
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc 
vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc 
vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed.  The Linux SCSI layer does not automatically adjust these 
parameters.

  This relates to the VSS "Windows Server Backup" process that kicks off at 
midnight on the host and finishes an hour and half later.   
  Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct 
kernel version we have.   We're currently running kernel version 
3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- 
We see the same errors on both.
  As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 
hyper-v system because of this.   We can stop the backup process and all is 
good, but we need nightly backups to image all of our VM's.   All the windows 
guests have no issues of course.   We also have some CentOS based guests 
running without issues from what we've seen.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

[Alex Ng]

Any updates on whether adbb4e646 test kernel is able to repro this
issue?

In any case, if we can get a next test kernel to try, I can help try
repro on it as well.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250

Title:
  [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based
  Backups

Status in linux package in Ubuntu:
  In Progress
Status in linux source package in Trusty:
  In Progress
Status in linux source package in Vivid:
  In Progress
Status in linux source package in Wily:
  In Progress
Status in linux source package in Xenial:
  In Progress

Bug description:
  Customers have reported running various versions of Ubuntu 14.04.2 LTS
  on Generation 2 Hyper-V Hosts.On a random Basis, the file system
  will be mounted Read-Only due to a "disk error" (which really isn't
  the case here).As a result, they must reboot the Ubuntu guest to
  get the file system to mount RW again.

  The Error seen are the following:
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed. The Linux SCSI layer does not automatically adjust these 
parameters.
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc 
vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc 
vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed.  The Linux SCSI layer does not automatically adjust these 
parameters.

  This relates to the VSS "Windows Server Backup" process that kicks off at 
midnight on the host and finishes an hour and half later.   
  Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct 
kernel version we have.   We're currently running kernel version 
3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- 
We see the same errors on both.
  As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 
hyper-v system because of this.   We can stop the backup process and all is 
good, but we need nightly backups to image all of our VM's.   All the windows 
guests have no issues of course.   We also have some CentOS based guests 
running without issues from what we've seen.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

[Alex Ng]

I'm also going to try reverting this commit on a 4.2.0-27 kernel that I
was able to see this issue on; and see if I can repro it there as well.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250

Title:
  [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based
  Backups

Status in linux package in Ubuntu:
  In Progress
Status in linux source package in Trusty:
  In Progress
Status in linux source package in Vivid:
  In Progress
Status in linux source package in Wily:
  In Progress
Status in linux source package in Xenial:
  In Progress

Bug description:
  Customers have reported running various versions of Ubuntu 14.04.2 LTS
  on Generation 2 Hyper-V Hosts.On a random Basis, the file system
  will be mounted Read-Only due to a "disk error" (which really isn't
  the case here).As a result, they must reboot the Ubuntu guest to
  get the file system to mount RW again.

  The Error seen are the following:
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed. The Linux SCSI layer does not automatically adjust these 
parameters.
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc 
vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc 
vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed.  The Linux SCSI layer does not automatically adjust these 
parameters.

  This relates to the VSS "Windows Server Backup" process that kicks off at 
midnight on the host and finishes an hour and half later.   
  Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct 
kernel version we have.   We're currently running kernel version 
3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- 
We see the same errors on both.
  As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 
hyper-v system because of this.   We can stop the backup process and all is 
good, but we need nightly backups to image all of our VM's.   All the windows 
guests have no issues of course.   We also have some CentOS based guests 
running without issues from what we've seen.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

[Alex Ng]

Hi Joseph,

Any updates on the bisect?

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250

Title:
  [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based
  Backups

Status in linux package in Ubuntu:
  In Progress
Status in linux source package in Trusty:
  In Progress
Status in linux source package in Vivid:
  In Progress
Status in linux source package in Wily:
  In Progress
Status in linux source package in Xenial:
  In Progress

Bug description:
  Customers have reported running various versions of Ubuntu 14.04.2 LTS
  on Generation 2 Hyper-V Hosts.On a random Basis, the file system
  will be mounted Read-Only due to a "disk error" (which really isn't
  the case here).As a result, they must reboot the Ubuntu guest to
  get the file system to mount RW again.

  The Error seen are the following:
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed. The Linux SCSI layer does not automatically adjust these 
parameters.
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc 
vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc 
vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed.  The Linux SCSI layer does not automatically adjust these 
parameters.

  This relates to the VSS "Windows Server Backup" process that kicks off at 
midnight on the host and finishes an hour and half later.   
  Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct 
kernel version we have.   We're currently running kernel version 
3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- 
We see the same errors on both.
  As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 
hyper-v system because of this.   We can stop the backup process and all is 
good, but we need nightly backups to image all of our VM's.   All the windows 
guests have no issues of course.   We also have some CentOS based guests 
running without issues from what we've seen.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

[Alex Ng]

Thanks for the update @jsalisbury.

7af024a isn't likely to be the cause, as that commit only changes
behavior during VMBus shutdown (i.e. when we cleanup VMBus during guest
shutdown).

This leads me to think that commit c8c38b3 is more likely to be a
factor. Nonetheless, I'll withhold further comment until we see results
from testing commit 7af024a.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250

Title:
  [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based
  Backups

Status in linux package in Ubuntu:
  In Progress
Status in linux source package in Trusty:
  In Progress
Status in linux source package in Vivid:
  In Progress
Status in linux source package in Wily:
  In Progress

Bug description:
  Customers have reported running various versions of Ubuntu 14.04.2 LTS
  on Generation 2 Hyper-V Hosts.On a random Basis, the file system
  will be mounted Read-Only due to a "disk error" (which really isn't
  the case here).As a result, they must reboot the Ubuntu guest to
  get the file system to mount RW again.

  The Error seen are the following:
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed. The Linux SCSI layer does not automatically adjust these 
parameters.
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc 
vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc 
vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed.  The Linux SCSI layer does not automatically adjust these 
parameters.

  This relates to the VSS "Windows Server Backup" process that kicks off at 
midnight on the host and finishes an hour and half later.   
  Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct 
kernel version we have.   We're currently running kernel version 
3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- 
We see the same errors on both.
  As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 
hyper-v system because of this.   We can stop the backup process and all is 
good, but we need nightly backups to image all of our VM's.   All the windows 
guests have no issues of course.   We also have some CentOS based guests 
running without issues from what we've seen.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

[Alex Ng]

Thanks for going through this testing.

Based on your results, I looked at the Hyper-V related commits in
http://kernel.ubuntu.com/git/ubuntu/ubuntu-trusty.git between
3.13.0-26.48 (GOOD) and 3.13.0-35.62 (BAD).

>From reviewing these commits, there doesn't seem to be an obvious
culprit. But it's possible that anyone of the following could affect
storage driver behavior. Looking forward to seeing what you uncover in
your bisect.

- a61e9104ea3b183a3b3cfdaa43145e1cd26b21ff Drivers: hv: vmbus: Negotiate 
version 3.0 when running on ws2012r2 hosts
- c8c38b34f5790366d98306c1fc418ddff1b0357b Drivers: hv: vmbus: Fix a bug in the 
channel callback dispatch code

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250

Title:
  [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based
  Backups

Status in linux package in Ubuntu:
  In Progress
Status in linux source package in Trusty:
  In Progress
Status in linux source package in Vivid:
  In Progress
Status in linux source package in Wily:
  In Progress

Bug description:
  Customers have reported running various versions of Ubuntu 14.04.2 LTS
  on Generation 2 Hyper-V Hosts.On a random Basis, the file system
  will be mounted Read-Only due to a "disk error" (which really isn't
  the case here).As a result, they must reboot the Ubuntu guest to
  get the file system to mount RW again.

  The Error seen are the following:
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed. The Linux SCSI layer does not automatically adjust these 
parameters.
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc 
vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc 
vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed.  The Linux SCSI layer does not automatically adjust these 
parameters.

  This relates to the VSS "Windows Server Backup" process that kicks off at 
midnight on the host and finishes an hour and half later.   
  Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct 
kernel version we have.   We're currently running kernel version 
3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- 
We see the same errors on both.
  As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 
hyper-v system because of this.   We can stop the backup process and all is 
good, but we need nightly backups to image all of our VM's.   All the windows 
guests have no issues of course.   We also have some CentOS based guests 
running without issues from what we've seen.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

[Alex Ng]

Thanks for testing this.

We will continue looking at this internally. It's possible that the
issue's with the storage drivers.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250

Title:
  [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based
  Backups

Status in linux package in Ubuntu:
  In Progress
Status in linux source package in Trusty:
  In Progress
Status in linux source package in Vivid:
  In Progress
Status in linux source package in Wily:
  In Progress

Bug description:
  Customers have reported running various versions of Ubuntu 14.04.2 LTS
  on Generation 2 Hyper-V Hosts.On a random Basis, the file system
  will be mounted Read-Only due to a "disk error" (which really isn't
  the case here).As a result, they must reboot the Ubuntu guest to
  get the file system to mount RW again.

  The Error seen are the following:
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed. The Linux SCSI layer does not automatically adjust these 
parameters.
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc 
vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc 
vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed.  The Linux SCSI layer does not automatically adjust these 
parameters.

  This relates to the VSS "Windows Server Backup" process that kicks off at 
midnight on the host and finishes an hour and half later.   
  Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct 
kernel version we have.   We're currently running kernel version 
3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- 
We see the same errors on both.
  As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 
hyper-v system because of this.   We can stop the backup process and all is 
good, but we need nightly backups to image all of our VM's.   All the windows 
guests have no issues of course.   We also have some CentOS based guests 
running without issues from what we've seen.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

[Alex Ng]

While we're not completely certain this is the smoking gun, I've
observed that in the logs being posted here that there are no
freeze/thaw operations taking place when the issue occurs. The commits
that @jrp identified, fix messaging between the VSS utility driver and
the host; and will hopefully ensure expected freeze/thawing of the
filesystem during backup.

In our internal testing with the latest Linux-next kernel (which
contains these commits) we were unable to hit this issue using scripts
from @jsalisbury.

We look forward to hearing back from the testing.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250

Title:
  [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based
  Backups

Status in linux package in Ubuntu:
  In Progress
Status in linux source package in Trusty:
  In Progress
Status in linux source package in Vivid:
  In Progress
Status in linux source package in Wily:
  In Progress

Bug description:
  Customers have reported running various versions of Ubuntu 14.04.2 LTS
  on Generation 2 Hyper-V Hosts.On a random Basis, the file system
  will be mounted Read-Only due to a "disk error" (which really isn't
  the case here).As a result, they must reboot the Ubuntu guest to
  get the file system to mount RW again.

  The Error seen are the following:
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed. The Linux SCSI layer does not automatically adjust these 
parameters.
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc 
vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc 
vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed.  The Linux SCSI layer does not automatically adjust these 
parameters.

  This relates to the VSS "Windows Server Backup" process that kicks off at 
midnight on the host and finishes an hour and half later.   
  Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct 
kernel version we have.   We're currently running kernel version 
3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- 
We see the same errors on both.
  As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 
hyper-v system because of this.   We can stop the backup process and all is 
good, but we need nightly backups to image all of our VM's.   All the windows 
guests have no issues of course.   We also have some CentOS based guests 
running without issues from what we've seen.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

[Alex Ng]

In addition to what @jrp mentioned, I would also add:

b9830d120cbe155863399f25eaef6aa8353e767f "Drivers: hv: util: Pass the
channel information during the init call"

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250

Title:
  [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based
  Backups

Status in linux package in Ubuntu:
  In Progress
Status in linux source package in Trusty:
  In Progress
Status in linux source package in Vivid:
  In Progress
Status in linux source package in Wily:
  In Progress

Bug description:
  Customers have reported running various versions of Ubuntu 14.04.2 LTS
  on Generation 2 Hyper-V Hosts.On a random Basis, the file system
  will be mounted Read-Only due to a "disk error" (which really isn't
  the case here).As a result, they must reboot the Ubuntu guest to
  get the file system to mount RW again.

  The Error seen are the following:
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed. The Linux SCSI layer does not automatically adjust these 
parameters.
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc 
vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc 
vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense 
Key : Unit Attention [current] 
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. 
Sense: Changed operating definition
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: 
Warning! Received an indication that the operating parameters on this target 
have changed.  The Linux SCSI layer does not automatically adjust these 
parameters.

  This relates to the VSS "Windows Server Backup" process that kicks off at 
midnight on the host and finishes an hour and half later.   
  Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct 
kernel version we have.   We're currently running kernel version 
3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- 
We see the same errors on both.
  As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 
hyper-v system because of this.   We can stop the backup process and all is 
good, but we need nightly backups to image all of our VM's.   All the windows 
guests have no issues of course.   We also have some CentOS based guests 
running without issues from what we've seen.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1508706] Re: Networking hangs on azure using hv_netvsc; bisected

[Alex Ng]

Hi Jay,

Does this issue occur even when VXLAN is not configured between the two
instances?

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1508706

Title:
  Networking hangs on azure using hv_netvsc; bisected

Status in linux package in Ubuntu:
  Triaged

Bug description:
  
  Running Ubuntu instances on azure, testing basic networking between two 
instances.  This involves configuring VXLAN between the two instances and 
running iperf and rsync of the kernel tree between the instances, e.g.,

  ip link add vxlan0 type vxlan id 999 local 10.88.0.12 remote 10.88.0.11 dev 
eth0
  ip l set vxlan0 up
  ip addr add 242.0.0.12/8 dev vxlan0

  After some time (sometimes instantly, sometimes up to 30 minutes of
  activity), the networking will hang.  This hang takes two forms:  a
  complete loss of connectivity (all network, even the ssh session used
  to log in), or just a loss of connectivity between instances (the ssh
  session remains active).  Sometimes for the latter case, the ssh
  session will then later hang.

  This first appeared when testing with the Ubuntu 3.19 kernel, and I
  subsequently bisected this to:

  commit effa2012d207f78cbc5a8360e62d420a8860b7e9
  Author: KY Srinivasan 
  Date:   Mon May 11 15:39:46 2015 -0700

  hv_netvsc: Use the xmit_more skb flag to optimize signaling the
  host

  BugLink: http://bugs.launchpad.net/bugs/1454892

  Based on the information given to this driver (via the xmit_more skb 
flag),
  we can defer signaling the host if more packets are on the way. This will 
help
  make the host more efficient since it can potentially process a larger 
batch of
  packets. Implement this optimization.

  Signed-off-by: K. Y. Srinivasan 
  Signed-off-by: David S. Miller 
  Acked-by: Tim Gardner 
  Acked-by: Brad Figg 
  Signed-off-by: Brad Figg 

  I also tested the mainline kernel (net-next); it fails with the
  equivalent commit:

  commit 82fa3c776e5abba7ed6e4b4f4983d14731c37d6a
  Author: KY Srinivasan 
  Date:   Mon May 11 15:39:46 2015 -0700

  hv_netvsc: Use the xmit_more skb flag to optimize signaling the
  host

  For both kernel trees, I also tested the prior commit and it did not
  exhibit the failure after many hours.  For ubuntu, this was

  commit a4aeb290bd75af5e16a6144a418291476ac6140c
  Author: K. Y. Srinivasan 
  Date:   Wed Mar 18 12:29:29 2015 -0700

  Drivers: hv: vmbus: Export the vmbus_sendpacket_pagebuffer_ctl()

  and for mainline it was

  commit 9eea92226407e7a117ef1ceef45380ebd000a0e2
  Author: Alexei Starovoitov 
  Date:   Mon May 11 15:19:48 2015 -0700

  pktgen: fix packet generation

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1508706/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1439780] Re: [Hyper-V] Fiber Channel critical target error

[Alex Ng]

This test kernel fixes the issue.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1439780

Title:
  [Hyper-V] Fiber Channel critical target error

Status in linux package in Ubuntu:
  In Progress
Status in linux source package in Vivid:
  In Progress
Status in linux source package in Wily:
  In Progress

Bug description:
  System details:
  Linux version 3.19.0-11-generic (apw@gloin) (gcc version 4.9.2 (Ubuntu 
4.9.2-10ubuntu12) ) #11~lp1423343v201503311833 SMP Tue Mar 31 17:40:01 UTC 2015 
(Ubuntu 3.19.0-11.11~lp1423343v201503311833-generic 3.19.3)

  Ubuntu 3.19.0-11
  Ubuntu Vivid 15.04 64bit custom kernel
  Platform: WS2012R2

  Using a fiber channel disk partition ( dev/sde1 in this example) outputs the 
following error:
  ​
  [ 1307.744496] blk_update_request: critical target error, dev sde, sector 0
  [ 1396.052488] blk_update_request: critical target error, dev sde, sector 0
  [ 1456.216570] blk_update_request: critical target error, dev sde, sector 0

  Creating a partition works, reading too, but first time writing it
  hangs for around 20 seconds and after a couple more writes the
  partition goes into read-only mode:

  #echo test1  test1.txt
  *hangs for 30 seconds*
  blk_update_request: critical target error, dev sde, sector 0
  # echo test2  test2.txt
  blk_update_request: critical target error, dev sde, sector 289671192
  Aborting journal on device sde1-8.
  EXT4-fs error (device sde1): ext4_journal_check_start:56: Detected aborted 
journal
  EXT4-fs (sde1): Remounting filesystem read-only
  # echo test string  sample.txt
  -bash: sample.txt: Read-only file system

  
  Dmesg log:
  [  461.284362] blk_update_request: critical target error, dev sde, sector 0
  [  484.845659] EXT4-fs (sde1): mounted filesystem with ordered data mode. 
Opts: (null)
  [  548.988377] blk_update_request: critical target error, dev sde, sector 0
  [  609.216340] blk_update_request: critical target error, dev sde, sector 
289671192
  [  609.221812] Aborting journal on device sde1-8.
  [  612.237503] EXT4-fs error (device sde1): ext4_journal_check_start:56: 
Detected aborted journal
  [  612.249529] EXT4-fs (sde1): Remounting filesystem read-only

  Additional info:

  cat /proc/version_signature
  Ubuntu 3.19.0-11.11~lp1423343v201503311833-generic 3.19.3

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1439780/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1439780] Re: [Hyper-V] Fiber Channel critical target error

[Alex Ng]

Hi,

I'm finished with the bisect and the following commit will fix this
issue:

dc45708ca9988656d706940df5fd102672c5de92 [storvsc: Set the SRB flags
correctly when no data transfer is needed]

Looks like we'll want to include this patch.

Here's the rest of my bisect log, in case you're interested:

# bad: [ba155e2d21f6bf05de86a78dbe5bfd8757604a65] Linux 4.1-rc5
# good: [e26081808edadfd257c6c9d81014e3b25e9a6118] Linux 4.1-rc4
git bisect start 'v4.1-rc5' 'v4.1-rc4'
# good: [1d82b0baf9abd59ae6f53f3102f4e442043763a4] Merge branch 'for-linus' of 
git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input
git bisect good 1d82b0baf9abd59ae6f53f3102f4e442043763a4
# good: [0b6280c62026168f79ff4dd1437df131bdfd24f2] Merge 
git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
git bisect good 0b6280c62026168f79ff4dd1437df131bdfd24f2
# good: [b0f155ada4c819f06aa32b4c906e7e76350c7ec1] drm/exynos: dp: Lower level 
of EDID read success message
git bisect good b0f155ada4c819f06aa32b4c906e7e76350c7ec1
# good: [7ce14f6ff26460819345fe8495cf2dd6538b7cdc] Merge branch 'for-linus-4.1' 
of git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs
git bisect good 7ce14f6ff26460819345fe8495cf2dd6538b7cdc
# good: [c5db6a3bdeb72f4238e1faefa4ce4eab7a64baea] Merge branch 
'timers-urgent-for-linus' of 
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect good c5db6a3bdeb72f4238e1faefa4ce4eab7a64baea
# bad: [74856fbf441929918c49ff262ace9835048e4e6a] sd: Disable support for 256 
byte/sector disks
git bisect bad 74856fbf441929918c49ff262ace9835048e4e6a
# bad: [8b2564ec7410928639db5c09a34d7d8330f1d759] lpfc: Fix breakage on big 
endian kernels
git bisect bad 8b2564ec7410928639db5c09a34d7d8330f1d759
# bad: [dc45708ca9988656d706940df5fd102672c5de92] storvsc: Set the SRB flags 
correctly when no data transfer is needed
git bisect bad dc45708ca9988656d706940df5fd102672c5de92
# first bad commit: [dc45708ca9988656d706940df5fd102672c5de92] storvsc: Set the 
SRB flags correctly when no data transfer is needed

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1439780

Title:
  [Hyper-V] Fiber Channel critical target error

Status in linux package in Ubuntu:
  Confirmed

Bug description:
  System details:
  Linux version 3.19.0-11-generic (apw@gloin) (gcc version 4.9.2 (Ubuntu 
4.9.2-10ubuntu12) ) #11~lp1423343v201503311833 SMP Tue Mar 31 17:40:01 UTC 2015 
(Ubuntu 3.19.0-11.11~lp1423343v201503311833-generic 3.19.3)

  Ubuntu 3.19.0-11
  Ubuntu Vivid 15.04 64bit custom kernel
  Platform: WS2012R2

  Using a fiber channel disk partition ( dev/sde1 in this example) outputs the 
following error:
  ​
  [ 1307.744496] blk_update_request: critical target error, dev sde, sector 0
  [ 1396.052488] blk_update_request: critical target error, dev sde, sector 0
  [ 1456.216570] blk_update_request: critical target error, dev sde, sector 0

  Creating a partition works, reading too, but first time writing it
  hangs for around 20 seconds and after a couple more writes the
  partition goes into read-only mode:

  #echo test1  test1.txt
  *hangs for 30 seconds*
  blk_update_request: critical target error, dev sde, sector 0
  # echo test2  test2.txt
  blk_update_request: critical target error, dev sde, sector 289671192
  Aborting journal on device sde1-8.
  EXT4-fs error (device sde1): ext4_journal_check_start:56: Detected aborted 
journal
  EXT4-fs (sde1): Remounting filesystem read-only
  # echo test string  sample.txt
  -bash: sample.txt: Read-only file system

  
  Dmesg log:
  [  461.284362] blk_update_request: critical target error, dev sde, sector 0
  [  484.845659] EXT4-fs (sde1): mounted filesystem with ordered data mode. 
Opts: (null)
  [  548.988377] blk_update_request: critical target error, dev sde, sector 0
  [  609.216340] blk_update_request: critical target error, dev sde, sector 
289671192
  [  609.221812] Aborting journal on device sde1-8.
  [  612.237503] EXT4-fs error (device sde1): ext4_journal_check_start:56: 
Detected aborted journal
  [  612.249529] EXT4-fs (sde1): Remounting filesystem read-only

  Additional info:

  cat /proc/version_signature
  Ubuntu 3.19.0-11.11~lp1423343v201503311833-generic 3.19.3

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1439780/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1439780] Re: [Hyper-V] Fiber Channel critical target error

[Alex Ng]

Hi Joseph,

Kernel with commits up to 7ce14f6ff26460819345fe8495cf2dd6538b7cdc are
also seeing this issue.

I'm can run my own bisect as well so we can save some time going back
and forth. Are you basing your bisect off of Linux-stable repository?

Thanks,
Alex

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1439780

Title:
  [Hyper-V] Fiber Channel critical target error

Status in linux package in Ubuntu:
  Confirmed

Bug description:
  System details:
  Linux version 3.19.0-11-generic (apw@gloin) (gcc version 4.9.2 (Ubuntu 
4.9.2-10ubuntu12) ) #11~lp1423343v201503311833 SMP Tue Mar 31 17:40:01 UTC 2015 
(Ubuntu 3.19.0-11.11~lp1423343v201503311833-generic 3.19.3)

  Ubuntu 3.19.0-11
  Ubuntu Vivid 15.04 64bit custom kernel
  Platform: WS2012R2

  Using a fiber channel disk partition ( dev/sde1 in this example) outputs the 
following error:
  ​
  [ 1307.744496] blk_update_request: critical target error, dev sde, sector 0
  [ 1396.052488] blk_update_request: critical target error, dev sde, sector 0
  [ 1456.216570] blk_update_request: critical target error, dev sde, sector 0

  Creating a partition works, reading too, but first time writing it
  hangs for around 20 seconds and after a couple more writes the
  partition goes into read-only mode:

  #echo test1  test1.txt
  *hangs for 30 seconds*
  blk_update_request: critical target error, dev sde, sector 0
  # echo test2  test2.txt
  blk_update_request: critical target error, dev sde, sector 289671192
  Aborting journal on device sde1-8.
  EXT4-fs error (device sde1): ext4_journal_check_start:56: Detected aborted 
journal
  EXT4-fs (sde1): Remounting filesystem read-only
  # echo test string  sample.txt
  -bash: sample.txt: Read-only file system

  
  Dmesg log:
  [  461.284362] blk_update_request: critical target error, dev sde, sector 0
  [  484.845659] EXT4-fs (sde1): mounted filesystem with ordered data mode. 
Opts: (null)
  [  548.988377] blk_update_request: critical target error, dev sde, sector 0
  [  609.216340] blk_update_request: critical target error, dev sde, sector 
289671192
  [  609.221812] Aborting journal on device sde1-8.
  [  612.237503] EXT4-fs error (device sde1): ext4_journal_check_start:56: 
Detected aborted journal
  [  612.249529] EXT4-fs (sde1): Remounting filesystem read-only

  Additional info:

  cat /proc/version_signature
  Ubuntu 3.19.0-11.11~lp1423343v201503311833-generic 3.19.3

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1439780/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1439780] Re: [Hyper-V] Fiber Channel critical target error

[Alex Ng]

Hi,

The issue also shows up in this kernel.

I can take over the bisect. My current bisect log is the same as yours.

I'm currently building the next test kernel up to following commit: 
74856fbf441929918c49ff262ace9835048e4e6a

Will let you know when I narrow it down further.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1439780

Title:
  [Hyper-V] Fiber Channel critical target error

Status in linux package in Ubuntu:
  Confirmed

Bug description:
  System details:
  Linux version 3.19.0-11-generic (apw@gloin) (gcc version 4.9.2 (Ubuntu 
4.9.2-10ubuntu12) ) #11~lp1423343v201503311833 SMP Tue Mar 31 17:40:01 UTC 2015 
(Ubuntu 3.19.0-11.11~lp1423343v201503311833-generic 3.19.3)

  Ubuntu 3.19.0-11
  Ubuntu Vivid 15.04 64bit custom kernel
  Platform: WS2012R2

  Using a fiber channel disk partition ( dev/sde1 in this example) outputs the 
following error:
  ​
  [ 1307.744496] blk_update_request: critical target error, dev sde, sector 0
  [ 1396.052488] blk_update_request: critical target error, dev sde, sector 0
  [ 1456.216570] blk_update_request: critical target error, dev sde, sector 0

  Creating a partition works, reading too, but first time writing it
  hangs for around 20 seconds and after a couple more writes the
  partition goes into read-only mode:

  #echo test1  test1.txt
  *hangs for 30 seconds*
  blk_update_request: critical target error, dev sde, sector 0
  # echo test2  test2.txt
  blk_update_request: critical target error, dev sde, sector 289671192
  Aborting journal on device sde1-8.
  EXT4-fs error (device sde1): ext4_journal_check_start:56: Detected aborted 
journal
  EXT4-fs (sde1): Remounting filesystem read-only
  # echo test string  sample.txt
  -bash: sample.txt: Read-only file system

  
  Dmesg log:
  [  461.284362] blk_update_request: critical target error, dev sde, sector 0
  [  484.845659] EXT4-fs (sde1): mounted filesystem with ordered data mode. 
Opts: (null)
  [  548.988377] blk_update_request: critical target error, dev sde, sector 0
  [  609.216340] blk_update_request: critical target error, dev sde, sector 
289671192
  [  609.221812] Aborting journal on device sde1-8.
  [  612.237503] EXT4-fs error (device sde1): ext4_journal_check_start:56: 
Detected aborted journal
  [  612.249529] EXT4-fs (sde1): Remounting filesystem read-only

  Additional info:

  cat /proc/version_signature
  Ubuntu 3.19.0-11.11~lp1423343v201503311833-generic 3.19.3

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1439780/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1439780] Re: [Hyper-V] Fiber Channel critical target error

[Alex Ng]

The issue also shows in this kernel.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1439780

Title:
  [Hyper-V] Fiber Channel critical target error

Status in linux package in Ubuntu:
  Confirmed

Bug description:
  System details:
  Linux version 3.19.0-11-generic (apw@gloin) (gcc version 4.9.2 (Ubuntu 
4.9.2-10ubuntu12) ) #11~lp1423343v201503311833 SMP Tue Mar 31 17:40:01 UTC 2015 
(Ubuntu 3.19.0-11.11~lp1423343v201503311833-generic 3.19.3)

  Ubuntu 3.19.0-11
  Ubuntu Vivid 15.04 64bit custom kernel
  Platform: WS2012R2

  Using a fiber channel disk partition ( dev/sde1 in this example) outputs the 
following error:
  ​
  [ 1307.744496] blk_update_request: critical target error, dev sde, sector 0
  [ 1396.052488] blk_update_request: critical target error, dev sde, sector 0
  [ 1456.216570] blk_update_request: critical target error, dev sde, sector 0

  Creating a partition works, reading too, but first time writing it
  hangs for around 20 seconds and after a couple more writes the
  partition goes into read-only mode:

  #echo test1  test1.txt
  *hangs for 30 seconds*
  blk_update_request: critical target error, dev sde, sector 0
  # echo test2  test2.txt
  blk_update_request: critical target error, dev sde, sector 289671192
  Aborting journal on device sde1-8.
  EXT4-fs error (device sde1): ext4_journal_check_start:56: Detected aborted 
journal
  EXT4-fs (sde1): Remounting filesystem read-only
  # echo test string  sample.txt
  -bash: sample.txt: Read-only file system

  
  Dmesg log:
  [  461.284362] blk_update_request: critical target error, dev sde, sector 0
  [  484.845659] EXT4-fs (sde1): mounted filesystem with ordered data mode. 
Opts: (null)
  [  548.988377] blk_update_request: critical target error, dev sde, sector 0
  [  609.216340] blk_update_request: critical target error, dev sde, sector 
289671192
  [  609.221812] Aborting journal on device sde1-8.
  [  612.237503] EXT4-fs error (device sde1): ext4_journal_check_start:56: 
Detected aborted journal
  [  612.249529] EXT4-fs (sde1): Remounting filesystem read-only

  Additional info:

  cat /proc/version_signature
  Ubuntu 3.19.0-11.11~lp1423343v201503311833-generic 3.19.3

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1439780/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1439780] Re: [Hyper-V] Fiber Channel critical target error

[Alex Ng]

Still not working with the kernel up to this commit:
0b6280c62026168f79ff4dd1437df131bdfd24f2

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1439780

Title:
  [Hyper-V] Fiber Channel critical target error

Status in linux package in Ubuntu:
  Confirmed

Bug description:
  System details:
  Linux version 3.19.0-11-generic (apw@gloin) (gcc version 4.9.2 (Ubuntu 
4.9.2-10ubuntu12) ) #11~lp1423343v201503311833 SMP Tue Mar 31 17:40:01 UTC 2015 
(Ubuntu 3.19.0-11.11~lp1423343v201503311833-generic 3.19.3)

  Ubuntu 3.19.0-11
  Ubuntu Vivid 15.04 64bit custom kernel
  Platform: WS2012R2

  Using a fiber channel disk partition ( dev/sde1 in this example) outputs the 
following error:
  ​
  [ 1307.744496] blk_update_request: critical target error, dev sde, sector 0
  [ 1396.052488] blk_update_request: critical target error, dev sde, sector 0
  [ 1456.216570] blk_update_request: critical target error, dev sde, sector 0

  Creating a partition works, reading too, but first time writing it
  hangs for around 20 seconds and after a couple more writes the
  partition goes into read-only mode:

  #echo test1  test1.txt
  *hangs for 30 seconds*
  blk_update_request: critical target error, dev sde, sector 0
  # echo test2  test2.txt
  blk_update_request: critical target error, dev sde, sector 289671192
  Aborting journal on device sde1-8.
  EXT4-fs error (device sde1): ext4_journal_check_start:56: Detected aborted 
journal
  EXT4-fs (sde1): Remounting filesystem read-only
  # echo test string  sample.txt
  -bash: sample.txt: Read-only file system

  
  Dmesg log:
  [  461.284362] blk_update_request: critical target error, dev sde, sector 0
  [  484.845659] EXT4-fs (sde1): mounted filesystem with ordered data mode. 
Opts: (null)
  [  548.988377] blk_update_request: critical target error, dev sde, sector 0
  [  609.216340] blk_update_request: critical target error, dev sde, sector 
289671192
  [  609.221812] Aborting journal on device sde1-8.
  [  612.237503] EXT4-fs error (device sde1): ext4_journal_check_start:56: 
Detected aborted journal
  [  612.249529] EXT4-fs (sde1): Remounting filesystem read-only

  Additional info:

  cat /proc/version_signature
  Ubuntu 3.19.0-11.11~lp1423343v201503311833-generic 3.19.3

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1439780/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1439780] Re: [Hyper-V] Fiber Channel critical target error

[Alex Ng]

The test kernel still hits this issue.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1439780

Title:
  [Hyper-V] Fiber Channel critical target error

Status in linux package in Ubuntu:
  Confirmed

Bug description:
  System details:
  Linux version 3.19.0-11-generic (apw@gloin) (gcc version 4.9.2 (Ubuntu 
4.9.2-10ubuntu12) ) #11~lp1423343v201503311833 SMP Tue Mar 31 17:40:01 UTC 2015 
(Ubuntu 3.19.0-11.11~lp1423343v201503311833-generic 3.19.3)

  Ubuntu 3.19.0-11
  Ubuntu Vivid 15.04 64bit custom kernel
  Platform: WS2012R2

  Using a fiber channel disk partition ( dev/sde1 in this example) outputs the 
following error:
  ​
  [ 1307.744496] blk_update_request: critical target error, dev sde, sector 0
  [ 1396.052488] blk_update_request: critical target error, dev sde, sector 0
  [ 1456.216570] blk_update_request: critical target error, dev sde, sector 0

  Creating a partition works, reading too, but first time writing it
  hangs for around 20 seconds and after a couple more writes the
  partition goes into read-only mode:

  #echo test1  test1.txt
  *hangs for 30 seconds*
  blk_update_request: critical target error, dev sde, sector 0
  # echo test2  test2.txt
  blk_update_request: critical target error, dev sde, sector 289671192
  Aborting journal on device sde1-8.
  EXT4-fs error (device sde1): ext4_journal_check_start:56: Detected aborted 
journal
  EXT4-fs (sde1): Remounting filesystem read-only
  # echo test string  sample.txt
  -bash: sample.txt: Read-only file system

  
  Dmesg log:
  [  461.284362] blk_update_request: critical target error, dev sde, sector 0
  [  484.845659] EXT4-fs (sde1): mounted filesystem with ordered data mode. 
Opts: (null)
  [  548.988377] blk_update_request: critical target error, dev sde, sector 0
  [  609.216340] blk_update_request: critical target error, dev sde, sector 
289671192
  [  609.221812] Aborting journal on device sde1-8.
  [  612.237503] EXT4-fs error (device sde1): ext4_journal_check_start:56: 
Detected aborted journal
  [  612.249529] EXT4-fs (sde1): Remounting filesystem read-only

  Additional info:

  cat /proc/version_signature
  Ubuntu 3.19.0-11.11~lp1423343v201503311833-generic 3.19.3

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1439780/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1439780] Re: [Hyper-V] Fiber Channel critical target error

[Alex Ng]

So I tried to repro this on the latest upstream v4.1-fc5 kernel from
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.1-rc5-unstable/.

It looks like the issue's been fixed there.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1439780

Title:
  [Hyper-V] Fiber Channel critical target error

Status in linux package in Ubuntu:
  Confirmed

Bug description:
  System details:
  Linux version 3.19.0-11-generic (apw@gloin) (gcc version 4.9.2 (Ubuntu 
4.9.2-10ubuntu12) ) #11~lp1423343v201503311833 SMP Tue Mar 31 17:40:01 UTC 2015 
(Ubuntu 3.19.0-11.11~lp1423343v201503311833-generic 3.19.3)

  Ubuntu 3.19.0-11
  Ubuntu Vivid 15.04 64bit custom kernel
  Platform: WS2012R2

  Using a fiber channel disk partition ( dev/sde1 in this example) outputs the 
following error:
  ​
  [ 1307.744496] blk_update_request: critical target error, dev sde, sector 0
  [ 1396.052488] blk_update_request: critical target error, dev sde, sector 0
  [ 1456.216570] blk_update_request: critical target error, dev sde, sector 0

  Creating a partition works, reading too, but first time writing it
  hangs for around 20 seconds and after a couple more writes the
  partition goes into read-only mode:

  #echo test1  test1.txt
  *hangs for 30 seconds*
  blk_update_request: critical target error, dev sde, sector 0
  # echo test2  test2.txt
  blk_update_request: critical target error, dev sde, sector 289671192
  Aborting journal on device sde1-8.
  EXT4-fs error (device sde1): ext4_journal_check_start:56: Detected aborted 
journal
  EXT4-fs (sde1): Remounting filesystem read-only
  # echo test string  sample.txt
  -bash: sample.txt: Read-only file system

  
  Dmesg log:
  [  461.284362] blk_update_request: critical target error, dev sde, sector 0
  [  484.845659] EXT4-fs (sde1): mounted filesystem with ordered data mode. 
Opts: (null)
  [  548.988377] blk_update_request: critical target error, dev sde, sector 0
  [  609.216340] blk_update_request: critical target error, dev sde, sector 
289671192
  [  609.221812] Aborting journal on device sde1-8.
  [  612.237503] EXT4-fs error (device sde1): ext4_journal_check_start:56: 
Detected aborted journal
  [  612.249529] EXT4-fs (sde1): Remounting filesystem read-only

  Additional info:

  cat /proc/version_signature
  Ubuntu 3.19.0-11.11~lp1423343v201503311833-generic 3.19.3

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1439780/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1439780] Re: [Hyper-V] Fiber Channel critical target error

[Alex Ng]

To answer your question, it looks like the issue can be reproduced as
recently as v4.1-rc4.

So the last version of the kernel that we see this bug is: v4.1-rc4

And the first version of the kernel we don't see this bug is: v4.1-rc5

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1439780

Title:
  [Hyper-V] Fiber Channel critical target error

Status in linux package in Ubuntu:
  Confirmed

Bug description:
  System details:
  Linux version 3.19.0-11-generic (apw@gloin) (gcc version 4.9.2 (Ubuntu 
4.9.2-10ubuntu12) ) #11~lp1423343v201503311833 SMP Tue Mar 31 17:40:01 UTC 2015 
(Ubuntu 3.19.0-11.11~lp1423343v201503311833-generic 3.19.3)

  Ubuntu 3.19.0-11
  Ubuntu Vivid 15.04 64bit custom kernel
  Platform: WS2012R2

  Using a fiber channel disk partition ( dev/sde1 in this example) outputs the 
following error:
  ​
  [ 1307.744496] blk_update_request: critical target error, dev sde, sector 0
  [ 1396.052488] blk_update_request: critical target error, dev sde, sector 0
  [ 1456.216570] blk_update_request: critical target error, dev sde, sector 0

  Creating a partition works, reading too, but first time writing it
  hangs for around 20 seconds and after a couple more writes the
  partition goes into read-only mode:

  #echo test1  test1.txt
  *hangs for 30 seconds*
  blk_update_request: critical target error, dev sde, sector 0
  # echo test2  test2.txt
  blk_update_request: critical target error, dev sde, sector 289671192
  Aborting journal on device sde1-8.
  EXT4-fs error (device sde1): ext4_journal_check_start:56: Detected aborted 
journal
  EXT4-fs (sde1): Remounting filesystem read-only
  # echo test string  sample.txt
  -bash: sample.txt: Read-only file system

  
  Dmesg log:
  [  461.284362] blk_update_request: critical target error, dev sde, sector 0
  [  484.845659] EXT4-fs (sde1): mounted filesystem with ordered data mode. 
Opts: (null)
  [  548.988377] blk_update_request: critical target error, dev sde, sector 0
  [  609.216340] blk_update_request: critical target error, dev sde, sector 
289671192
  [  609.221812] Aborting journal on device sde1-8.
  [  612.237503] EXT4-fs error (device sde1): ext4_journal_check_start:56: 
Detected aborted journal
  [  612.249529] EXT4-fs (sde1): Remounting filesystem read-only

  Additional info:

  cat /proc/version_signature
  Ubuntu 3.19.0-11.11~lp1423343v201503311833-generic 3.19.3

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1439780/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp