The test kernel I built in comment #78 is failing to boot. I'm going to
review my backports and build another test kernel. I'll post a link to
it here shortly.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs
If those commits could just be fixes for the problem we are having, that
would be a really good start of 2016!
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250
Title:
[Hyper-V] Ubu
I built a Wily test kernel with those three patches. The patches also
required three prerequisite commits, so the test kernel has the
following six commits:
4246a0b6 block: add a bi_error field to struct bio
54efd50b block: make generic_make_request handle arbitrarily sized bios
03100aad block:
We should consider the following upstream storvsc commits to see if they
improve the situation:
storvsc_drv.c : commit 3209f9d780d137cdb54c85e0a776cb19e723a170 : scsi:
storvsc: Fix a bug in the handling of SRB status flags
storvsc_drv.c : commit 81988a0e6b031bc80da15257201810ddcf989e64 :
My update:
It looks the issue is somehow related to the backup, but I tend to think there
is a bug somewhere in the storvsc driver code -- it's very hard to track it
down because before the ext4 read-only issue happens, the ext4 file system may
have been somewhat corrupted.
--
You received t
I honestly just went the easy way doing that:
while true; do
mount | grep "sda1 on / type ext4 (ro" > /dev/null
if (($? == 0)); then
fsck -y /dev/sda1
reboot
fi
sleep 60
done
not the best practice for sure, but...
--
You received this bug notification because you are a member of
Just wanted to chip in, we experienced the same problem twice now on a
12.04 LTS Gen 1 VM. Unlike situation described in comment #38, we
haven't installed any daemons for the guest OS. Last time it happened,
we could definitely correlate it to the Avamar (VSS based) backup. At
the time, we were run
The patch mentioned in #72 can't help -- still bad luck. :-(
But I can confirm: before the issue happens, somehow athe host doesn't send us
freeze/thaw commands any longer.
we need further debugging...
--
You received this bug notification because you are a member of Kernel
Packages, which is s
** No longer affects: linux (Ubuntu Utopic)
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250
Title:
[Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based
Backups
St
I suspect the race condition may be in vss_on_msg() with the non-thread-
safe variable vss_transaction.state.
And I guess the below patch may have fixed the issue (the patch hasn't be in
the upstream yet):
http://lkml.iu.edu/hypermail/linux/kernel/1510.3/04218.html
I can only test the patch tomo
> "BTW, Since Ubuntu 15.04's "
typo.. 15.04 -> 15.10.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250
Title:
[Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based
B
@f-bosch @jsalisbury
I can reproduce the issue consistently within 5~6 hours with a Ubuntu 15.10 VM.
In /var/log/syslog, several minutes before the file system is remounted
as read-only, the hv_vss_daemon has stopped working: the daemon just
always hangs on the poll() , not receiving freeze/thaw
If you let the wrapper script run long enough, it will eventually fill
up the hard disk. This is because tiotest does not remove it's working
files when it finishes. I just added a line after the wait to clean up
the working directory. I would suggest using this new version of the
script, which
@decui Correct, ceteris paribus, the only changes were more of the same
(disk space) and some package changes through apt-get update (Ubuntu
14.04 LTS). My guess is that the downgrade is related to one of the two.
But I agree, let's postpone this one, until the read-only bug is fixed.
Afterwards, I
Thanks @f-bosch for your clarification in #62. So my understanding is:
the (temporary) I/O downgrade during the period of backup might be
caused by the fact the disk space has been almost used up (?) recently,
but it also might be somehow related to the backup. Let's focus on the
backup issue at
Unfortunately, the 4.3 Xenial kernel eventually did hit this bug, it
just took longer. I'll continue to investigate.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250
Title:
[Hyper
I have some good news. I'm unable to reproduce the bug with the
-proposed Xenial kernel, which is v4.3 based. I will continue to test
this kernel for some time to ensure it really doesn't exhibit the bug.
If needed, I'll continue to drive up the IO load. If it does not
exhibit the bug, I can per
I confirmed that the reproduce is repeatable. I can reproduce the bug
within 30 to 60 minutes. The bug can be reproduce with the following
steps:
1. Create a Virtual machine with 1 cpu and 2048M of memory.
2. Install Ubuntu on the VM. I installed 15.10(Wily).
3. Install the linux-cloud-tools
Wrapper script for tiotest.
** Attachment added: "run_tiotest.sh"
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+attachment/4529139/+files/run_tiotest.sh
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
@decui Regarding downgrade, I think I might not have been clear on this.
The downgrade occurs only when there are backups running. Our
performance otherwise is great. This (new) problem is not necessarily
related to this specific issue, but more to backups on Hyper-V in
general. Nonetheless, as I s
Frederik, Thanks for the new information!
About the I/O perf downgrade:
1. I don't think more occupied disk space should harm the perf so greatly (http
resopnse time: from <100ms to several seconds).
2. The perf downgrade appeared recently but the backup issue appeared
long long ago. Why do you
@decui @jsalisbury That is indeed good news. Since we did not make any
changes to our platform yet, we can help to test possible fixes. There
were also some questions.
1. Occurrence of the issue: with the fixes of bug #1445195 the amount of
read-only errors reduced drastically. But it is still the
@f-bosch Hi Frederik, about the second issue you mentioned ("The more
data getting on the disk, the higher the spikes are...requests that
involve write operations tend to have these characteristics") in #54,
it looks there is a storage I/O performance downgrade somehow.
Do you think if it's relat
@f-bosch Hi Frederik, we're really sorry that this backup-can-cause-
filesystem-remounted-readonly issue has been there for long time! We
definitely want to get it fixed as soon as possible. Unluckily according
to my understanding of the long thread here, I think we had difficulty
trying to reprod
I'm now able to reproduce this bug on demand. It takes just under an
hour to reproduce. I'm going to to try to reproduce a few more times to
make sure it's consistent. If the reproduce is consistent, I'll list
all the steps necessary.
Basically I'm using the tiotest benchmark from tiobench. Th
@Joseph Yes, I did, but it does not get past the "loading ramdisk" message
as @Frederik says, this is a problem which could really kill ubuntu's usage on
servers vm. I refused to move from ubuntu because I like it a lot, but I also
refuse to increment my backup intervals. Almost every morning I n
@Michele, can you confirm that you installed both the linux-image and
linux-image-extra .deb packages from the link I posted in #50?
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250
This problem started more than a year ago, I decided together with my
hoster to look at alternatives. Backup is the only thing that is causing
problems for us with HyperV. The problem is two fold. First, the read-
only state machines get into when generating the backup. Second, the
read/write spike
I can confirm the kernel from #50 does not boot
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250
Title:
[Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based
Backups
I'm on 15.10 and experiencing this problem. Windows Server 2012 r2
I'm available to test if needed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250
Title:
[Hyper-V] Ubuntu 14.04.2
Since I was on holidays, I will give this a try in next two weeks.
Thanks for the effort @jalisbury.
> On 20 Nov 2015, at 15:06, Joseph Salisbury
> wrote:
>
> I'll see if I can figured out why the 3.10 based kernel doesn't boot.
> However, it may just be easier to test the Precise kernel, which
I'll see if I can figured out why the 3.10 based kernel doesn't boot.
However, it may just be easier to test the Precise kernel, which is 3.2
based.
The latest Precise kernel can be downloaded from:
https://launchpad.net/~ubuntu-security/+archive/ubuntu/ppa/+build/8272116
--
You received this bu
I tested the kernel from #48 and it's not booting. I am attaching the
log.
** Attachment added: "icalog.txt"
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+attachment/4521296/+files/icalog.txt
--
You received this bug notification because you are a member of Kernel
Packages, w
I built an Ubuntu kernel based on 3.10, so it is the same base kernel
version as CentOS posted in comment #42. If possible, can this kernel
be tested to see if this issue is due to a regression introduced after
3.10?
The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1
Sorry for the long delay in posting results but I did want to share what we've
learned.
1) The failures happen during the VSS Snapshot/checkpoint creation process.
You can reproduce the errors by doing manual VSS snapshots.
2) The heavier the VM load, especially I/O, the more likely the error
@stuart-luscombe That is interesting. So if we combine both our
observations, then my conclusion is: there must be something in the
hyper v daemons. And one the things to look at is where the Ubuntu
daemons might differ from the CentOS daemons. Maybe @jsalisbury knows if
there is any difference bet
@f-bosch That's correct.
I just checked back through my backup logs and the VM was backing up
without error prior to following the technet instructions. I've attached
a grab of the status windows for both jobs.
** Attachment added: "Veeam Backup status before and after installing Hyper-V
tools"
@stuart-luscombe So if I understand correctly, the reason why you
installed the daemons was to get full file indexes and no complaints by
monitoring software. However, the downside is that the VMs started
crashing during backups after the install? More importantly, can you
confirm there was no prob
@f-bosch (#39)
I had been able to carry out backups on the VM prior to installing the
daemons but I was not able to produce full file indexes and my Hyper-V
monitoring software would complain about integration services not being
enabled.
--
You received this bug notification because you are a me
@jsalisbury uname -a for the CentOS machines is: Linux host.name
3.10.0-229.11.1.el7.x86_64 #1 SMP Thu Aug 6 01:06:18 UTC 2015 x86_64
x86_64 x86_64 GNU/Linux. This machine was never hit by the backup issue.
The technet link you posted contains the following quote: "As a result,
we've been hesitant
We are currently awaiting the patches posted in comment #35.
Can you post the specific CentOS kernel version you are using?
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250
Title:
A little remark that still has no answer yet: what are the Ubuntu
specifics that cause this issue? In the 10 months we have these machines
running: many crashes for Ubuntu with VSS snapshots while the CentOS
machines have had no crash at all. Maybe @jsalisbury has a explanation
for this?
--
You r
What do you mean with: did not occur until you followed those
instructions? You mean without the daemons everything was fine?
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250
Title:
I am experiencing this same issue when backing up a 14.04 LTS Gen 1 VM
using Veeam. The error seems to occur when the VSS snapshots are being
taken. The error did not occur until I had followed Microsoft's
instructions on packages to install for Ubuntu
(https://technet.microsoft.com/en-GB/library/d
Just wanted to report that I installed the Wily Kernel on Sept 3rd and the VM's
ran without errors until yesterday. The errors are different then before but
doing the same thing:
blk_update_request: I/O error, dev sda, sector 206963000
Aborting journal on device sda2-8
EXT4-fs errors (device sda
@jrp Thanks for that, I am happy to hear there is still being worked on. Let me
know if there is a build that I can test
@h-lbuntu-2 How are your results with Willy kernels?
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
We're getting ready for a new round of storvsc fixes that correspond to
LIS 4.0.11 we'll have to see if that improves things further.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250
Unfortunately, I did not have had time to test Willy kernels. But the
other 3.x stable kernels are still contain the problem.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250
Title:
@Joseph
Thank you. I upgraded the production server that is most often errors
out with the Wily kernel you referenced above and will report back
Please be patient, an individual VM can go several days without the
error.
--
You received this bug notification because you are a member of Kernel
Pa
@ubuntu
Is it possible for you to test the latest Wily kernel? The Wily kernel
has been rebased to the upstream 4.2 kernel, so it should have all the
latest Hyper-V updates in mainline.
The Wily kernel can be downloaded from:
https://launchpad.net/ubuntu/+source/linux/4.2.0-7.7/+build/7856238
-
To add to comment #30 I posted, the read-only bug continues to occur on
both Generation 1 and Generation 2 VMs.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250
Title:
[Hyper-V] Ub
Bug is still present in Vivid 3.19.0-26-generic.
Is there a workaround that avoids this problem? There's considerable
pressure to move off of Hyper-V and I'd rather not do it.
[57055.788468] sd 0:0:0:0: [storvsc] Sense Key : Unit Attention [current]
[57055.788561] sd 0:0:0:0: [storvsc] Add. Sens
@Frederik, have you tested with the Wily kernel yet, posted in comment
#9? The Wily kernel has since been rebased to 4.2, so testing of the
latest kernel by applying the latest Wily updates would be great.
--
You received this bug notification because you are a member of Kernel
Packages, which i
@dander88 That sounds great, but we had the same results for a test VPS
machine. As indicated by @jrp before: it depends on your IO load if the
machine goes into read-only. And test machines usually do not have that
many IO load.
--
You received this bug notification because you are a member of
So far so good after the kernal update. I have been backing up a test VM
4 times a day for almost 2 weeks. It has not gone into read only mode as
of yet.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchp
@F-Bosch we did not enter the "read-only" mode with the one test we
tried. I will keep a backup schedule going multiple times a day and see
if it ever goes into read only mode. I will report back in a few days to
let you know the results
--
You received this bug notification because you are a mem
@cohn By the way, I have not installed the 4.x kernels yet. So I have no
idea how many dependency problems you will run in to. Could you let me
know?
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.n
@dander88 According to @jrp the message "Received an indication that the
operating parameters on this target have changed. The Linux SCSI layer
does not automatically adjust these parameters." is beneign. He
mentioned that in bug 1445195.
Are you just seeing that message or do your systems also g
@John WIlkinson, You should only need the linux-image and linux-image-
extra .deb packages to install the latest kernel. The -headers .deb
packages should not be needed.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
htt
@Dustin, can you run "uname -a" to confirm your machine is running the
latest Wily kernel built from the current mainline kernel?
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250
Tit
@Chris -
1- No Local Backup at all - All through Altaro
2 - Standard Local on Scsi 0
3- Sent to a local NAS
3 - I dont know the exact load that they currently have - I know that it is
under 80% of system resources for sure. Not 100% on the services, I know MYSQL,
other than that not sure.
--
After running the patches, we are still seeing the same error in the
syslog. Options?
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250
Title:
[Hyper-V] Ubuntu 14.04.2 LTS Generatio
@Joseph Salisbury Is there a specific subset of those .deb packages that
need to be run, or are they all needed to patch the relevant bugs?
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/14
Hi Dustin,
Some questions on the topic, my apologize if these got replied before or in
other threads.
1. Is this repro using Windows Server Backup directly, and not through Altaro?
2. For the VM setup, is this a standard local vhdx on scsi controller 0 for a
Gen2 VM for OS disk? Or there are any
@Joseph - There are not too much, we use the backup program called
ALTARO. It will produce this error about every 10 or so backups.
We are working on getting that new kernel into some test units. I will
post results when we are done. Thanks for the follow up.
--
You received this bug notificatio
On the other bug (1445195) somebody reported seeing the same error on
Gen 1 devices. I wanted to report that we see the identical bug on both
Gen 1 and Gen 2 devices. Frequency does not appear to be any different
but I don't have precise data.
--
You received this bug notification because you
@Dustin it sounds like you have a reliable way to reproduce this bug?
If so, can you list those steps here for others to try?
Also, if you can reproduce this, would it be possible for you to test
the kernel posted in comment #9?
Thanks!
--
You received this bug notification because you are a me
We have the same VSS Issues in 14.04 LTS but hyper v (2012) gen 1. Has
this been seen before? We can reproduce the error on command.
Received an indication that the operating parameters on this target have
changed. The Linux SCSI layer does not automatically adjust these
parameters.
--
You recei
@jsalisbury I will start testing this week. However, I feel the latest
reports were pretty clear: the issue is there. While I was the only one
at first that still had problems, after a while more people reported (in
bug 1441595) that the new build still contains the issue. In my opinion,
it is now
That should have been bug 1445195.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250
Title:
[Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based
Backups
Status in l
@Frederik, yes that is correct. This kernel basically has all HV
related commits that are currently in mainline.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250
Title:
[Hyper-V]
@jsalisbury I did not find any new commits on this subject in the
current kernel master (https://github.com/torvalds/linux). And I believe
you already included all HV commits in the last test build from bug
1441595.
So testing this test kernel would mean I am testing whether another
commit (not sp
The Wily kernel has been rebased to upstream 4.1, which has all the
current Hyper-V commits in mainline. Can you give this test kernel a
test to see if it still exhibits this issue, or if it is resolved.
If it still exhibits the issue, we know that a new fix is needed. If
this test kernel fixes
** Also affects: linux (Ubuntu Trusty)
Importance: Undecided
Status: New
** Also affects: linux (Ubuntu Vivid)
Importance: Undecided
Status: New
** Also affects: linux (Ubuntu Wily)
Importance: Critical
Assignee: Joseph Salisbury (jsalisbury)
Status: In Progress
Prior comments regarding this issue can be found in bug 1445195
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250
Title:
[Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS
@jrp @jsalisbury I am using this kernel:
http://kernel.ubuntu.com/~jsalisbury/lp1445195/vivid/ on 14.04.2. So
that build is much more stable but NOT a complete fix of the problem.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ub
@jrp @jsalisbury I am using this kernel:
http://kernel.ubuntu.com/~jsalisbury/lp1445195/vivid/ on 14.04.2. So
that build is much more stable but a complete fix of the problem.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu
I believe he is running 14.04.2, which means at least the HWE kernel.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250
Title:
[Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors
@Frederik Bosch Can you post what kernel version you are currently
using?
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250
Title:
[Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Er
Error message that happens when this bug occurs:
[154272.293488] sd 2:0:0:0: [storvsc] Sense Key : Unit Attention [current]
[154272.293508] sd 2:0:0:0: [storvsc] Add. Sense: Changed operating definition
[154272.293665] sd 2:0:0:0: Warning! Received an indication that the operating
parameters on t
My latest report was that latest builds with patches are much more
stable but are also not a complete fix for the problem. It is still
there and occurs randomly. The error message is not changed. I have no
real indication what causes the read-only state. During the latest RO
state I noticed there w
Some additional details:
https://social.technet.microsoft.com/Forums/windowsserver/en-US
/8807f61c-565e-45bc-abc4-af09abf59de2/ubuntu-14042-lts-generation-2
-scsi-errors-on-vss-based-backups
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to
301 - 382 of 382 matches
Mail list logo