[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- trusty' to 'verification-done-trusty'. If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you! ** Tags added: verification-needed-trusty -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: In Progress Status in “linux” source package in Trusty: Fix Committed Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
** Also affects: linux (Ubuntu Precise) Importance: Undecided Status: New ** No longer affects: linux (Ubuntu Precise) ** Also affects: linux-lts-trusty (Ubuntu) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Precise) Importance: Undecided Status: New ** Also affects: linux-lts-trusty (Ubuntu Precise) Importance: Undecided Status: New ** Changed in: linux-lts-trusty (Ubuntu Trusty) Status: New = Invalid ** Changed in: linux (Ubuntu Precise) Status: New = Invalid ** Changed in: linux-lts-trusty (Ubuntu Precise) Status: New = Triaged -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: In Progress Status in “linux-lts-trusty” package in Ubuntu: New Status in “linux” source package in Precise: Invalid Status in “linux-lts-trusty” source package in Precise: Triaged Status in “linux” source package in Trusty: Fix Committed Status in “linux-lts-trusty” source package in Trusty: Invalid Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
I've nominated Precise as well based on the description indicating a recreate there. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: In Progress Status in “linux” source package in Trusty: Fix Committed Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
** Branch linked: lp:ubuntu/trusty-proposed/linux-keystone -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: In Progress Status in “linux” source package in Trusty: Fix Committed Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
** Branch linked: lp:ubuntu/precise-proposed/linux-lts-trusty -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: Fix Released Status in “linux” source package in Trusty: Fix Committed Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
Hi Chris,
This is Arvind Kumar from VMware. Recently the issue discussed in this
bug was brought into VMware's notice. We looked at the patch
(https://lkml.org/lkml/2014/9/23/509) which was done to address the
issue. Since the patch is done in mptsas driver, it addresses the issue
only on lsilogic controller, if user uses some other controller e.g.
pvscsi or buslogic then the issue remains. Moreover the patch disables
the WRITE SAME completely on the lsilogic which indicates that VMware
will never be able to support WRITE SAME on lsilogic. As I understand
from the bug, it is concluded that the WRITE SAME is not properly
implemented by VMware. Actually we don't support WRITE SAME at all.
We internally investigated the issue and as per our understanding the
issue is not VMware specific and rather seems to be with the kernel,
which could very well happen on real hardware too in case the disk
doesn't support WRITE SAME command. Below are the details of the
investigation by Petr Vandrovec.
--
In blk-lib.c on line 294 it checks whether bdev supports write_same.
With LVM, bdev here is dm-0. It says yes, it is supported, and so
write_same is invoked (note that check is racy in case device loses
write_same capability between test and moment bio is issued):
291 int blkdev_issue_zeroout(struct block_device *bdev, sector_t sector,
292 sector_t nr_sects, gfp_t gfp_mask)
293 {
294 if (bdev_write_same(bdev)) {
295 unsigned char bdn[BDEVNAME_SIZE];
296
297 if (!blkdev_issue_write_same(bdev, sector, nr_sects,
gfp_mask,
298 ZERO_PAGE(0)))
299 return 0;
300
301 bdevname(bdev, bdn);
302 pr_err(%s: WRITE SAME failed. Manually zeroing.\n,
bdn);
303 }
304
305 return __blkdev_issue_zeroout(bdev, sector, nr_sects,
gfp_mask);
306 }
307 EXPORT_SYMBOL(blkdev_issue_zeroout);
Then it gets to LVM, and LVM forwards request to sda. When it fails,
kernel clears bdev_write_same() on sda, and returns -121 (EREMOTEIO).
Now next request comes. Nobody cleared bdev_write_same() on dm-0, it got
cleared only on sda, so request gets to LVM, which forwards it to sda.
Where it hits a snag in blk-core.c:
1824 if (bio-bi_rw REQ_WRITE_SAME
!bdev_write_same(bio-bi_bdev)) {
1825 err = -EOPNOTSUPP;
1826 goto end_io;
1827 }
bi_bdev here is sda, and I/O fails with EOPNOTSUPP, without WRITE_SAME
ever being issued. And then it hits completion code that treats
EOPNOTSUPP as success:
18 static void bio_batch_end_io(struct bio *bio, int err)
19 {
20 struct bio_batch *bb = bio-bi_private;
21
22 if (err (err != -EOPNOTSUPP))
23 clear_bit(BIO_UPTODATE, bb-flags);
24 if (atomic_dec_and_test(bb-done))
25 complete(bb-wait);
26 bio_put(bio);
27 }
So everybody outside of blkdev_issue_write_same() thinks that I/O
succeeded, while in reality kernel even did not issue request!
Fix should:
1. Use different error code if WRITE_SAME request is thrown away. Or
remove special EOPNOTSUPP handling from end_io - I assume EOPNOTSUPP is
supposed to ignore failures from discarded commands, but then nobody
else should be using EOPNOTSUPP, and
2. WRITE_SAME failure should propagate from sda to dm-0.
--
Our understanding is that we should revert the fix in mptsas driver and
try to do the right fix as described above. I am attaching the patch
from Petr who did the investigation. CC'ing all involved people from
VMware too. Could you please evaluate the patch and suggest on further
steps?
Thanks!
Arvind
** Patch added: 0001-Do-not-silently-discard-WRITE_SAME-requests.patch
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+attachment/4230953/+files/0001-Do-not-silently-discard-WRITE_SAME-requests.patch
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1371591
Title:
file not initialized to 0s under some conditions on VMWare
Status in “linux” package in Ubuntu:
Fix Released
Status in “linux” source package in Trusty:
Fix Committed
Bug description:
Under some conditions, after fallocate() the file is observed not to
be completely initilized to 0s: some 4KB pages have left-over data
from previous files that occupied those pages. Note that in addition
to causing functional problems for applications expecting files to be
initialized to 0s, this is a security issue because it allows data to
leak from one file to another, bypassing file access controls.
The problem has been seen running under the following VMWare-based virtual
environments:
Fusion 6.0.2
ESXi 5.1.0
And under the
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
Hi Chris, can you revert original patch which blacklist VMware for no good reason, and instead apply attached patch, or its equivalent? As explained above, bug affects ALL disks that do not support WRITE_SAME when used with stackable block devices, like LVM. Due to the bug Linux kernel stops issuing WRITE_SAME after first failure, treating them as succeeding, instead of falling back to non-write-same code. I have doubts about handling discards too, but I'm not Linux storage guru, so I left discard handling returning EOPNOTSUPP, rather than switching it to EREMOTEIO too. Unfortunately I could not yet figure out how to reopen this bug so that correct fix can tracked. Thanks, Petr Vandrovec Disclosure: I'm VMware employee. ** Patch added: Do not ignore WRITE_SAME failures https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+attachment/4231085/+files/0001-Do-not-silently-discard-WRITE_SAME-requests.patch -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: Fix Released Status in “linux” source package in Trusty: Fix Committed Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
@petr-vmware, arvindkumar: Thanks for looking into this! I think your analysis makes more sense, because we'd expect the manually zeroing to actually work instead of causing corruption. What kind of testing and configurations have you done this on? Can you submit this patch upstream and reference my original patch (or reply to the thread with the original bug), once this receives some acks upstream and my patch gets reverted, we can do the same in Ubuntu kernel. I'll reopen the bug. Thanks, ** Changed in: linux (Ubuntu) Status: Fix Released = In Progress ** Changed in: linux (Ubuntu) Importance: Critical = High -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: In Progress Status in “linux” source package in Trusty: Fix Committed Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
** Tags added: patch -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: In Progress Status in “linux” source package in Trusty: Fix Committed Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
This bug was fixed in the package linux - 3.16.0-20.27 --- linux (3.16.0-20.27) utopic; urgency=low [ Tim Gardner ] * [Config] CONFIG_CXL=m * Release Tracking Bug - LP: #1376354 [ Andi Kleen ] * SAUCE: perf tools: Fix perf record as non root with kptr_restrict == 1 - LP: #1375441 [ Chris J Arges ] * SAUCE: Revert sd: don't use scsi_setup_blk_pc_cmnd for flush requests - LP: #1375452 [ Ian Munsie ] * SAUCE: (no-up) powerpc/cell: Move spu_handle_mm_fault() out of cell platform * SAUCE: (no-up) powerpc/cell: Move data segment faulting code out of cell platform * SAUCE: (no-up) powerpc/msi: Improve IRQ bitmap allocator * SAUCE: (no-up) powerpc/mm: Export mmu_kernel_ssize and mmu_linear_psize * SAUCE: (no-up) powerpc/powernv: Split out set MSI IRQ chip code * SAUCE: (no-up) cxl: Add new header for call backs and structs * SAUCE: (no-up) powerpc/powerpc: Add new PCIe functions for allocating cxl interrupts * SAUCE: (no-up) powerpc/mm: Add new hash_page_mm() * SAUCE: (no-up) powerpc/opal: Add PHB to cxl mode call * SAUCE: (no-up) powerpc/mm: Add hooks for cxl * SAUCE: (no-up) cxl: Add base builtin support * SAUCE: (no-up) cxl: Driver code for powernv PCIe based cards for userspace access * SAUCE: (no-up) cxl: Userspace header file. * SAUCE: (no-up) cxl: Add driver to Kbuild and Makefiles * SAUCE: (no-up) cxl: Add documentation for userspace APIs -- Tim Gardner tim.gard...@canonical.com Tue, 30 Sep 2014 13:05:27 -0600 ** Changed in: linux (Ubuntu) Status: Fix Committed = Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: Fix Released Status in “linux” source package in Trusty: Fix Committed Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
** Changed in: linux (Ubuntu) Status: In Progress = Fix Committed ** Changed in: linux (Ubuntu Trusty) Status: New = Fix Committed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: Fix Committed Status in “linux” source package in Trusty: Fix Committed Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
Patch here: http://git.infradead.org/users/hch/scsi-queue.git/commit/4089b71cc820a426d601283c92fcd4ffeb5139c2 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: In Progress Status in “linux” source package in Trusty: New Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
https://lkml.org/lkml/2014/9/23/509 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: In Progress Status in “linux” source package in Trusty: New Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
Re: [Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
Thanks Chris. I take it from the other thread that bubbling the setting up from the lower layer to the dm-* layer won't be possible. Do you know if this patch will fix it for ESXi as well as for the VMWare desktop products? I don't have access to ESXi, but the issue was originally reported to us by a customer running ESXi. I could ask them to try the patched kernel, but maybe we could do a simpler check first. Is there a way to determine the vendor id for their virtual SCSI disks on a running system so we can verify that the vendor id is the same? Thanks, Bruce On Wed, Sep 24, 2014 at 10:00 AM, Chris J Arges 1371...@bugs.launchpad.net wrote: https://lkml.org/lkml/2014/9/23/509 -- You received this bug notification because you are subscribed to the bug report. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: In Progress Status in “linux” source package in Trusty: New Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: In Progress Status in “linux” source package in Trusty: New Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern *
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
Bruce,
The following should get some of the info needed:
tail /sys/class/scsi_device/*/device/{vendor,model}
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1371591
Title:
file not initialized to 0s under some conditions on VMWare
Status in “linux” package in Ubuntu:
In Progress
Status in “linux” source package in Trusty:
New
Bug description:
Under some conditions, after fallocate() the file is observed not to
be completely initilized to 0s: some 4KB pages have left-over data
from previous files that occupied those pages. Note that in addition
to causing functional problems for applications expecting files to be
initialized to 0s, this is a security issue because it allows data to
leak from one file to another, bypassing file access controls.
The problem has been seen running under the following VMWare-based virtual
environments:
Fusion 6.0.2
ESXi 5.1.0
And under the following versions of Ubuntu:
Ubuntu 12.04, 3.11.0-26-generic
Ubuntu 14.04.1, 3.13.0-32-generic
Ubuntu 14.04.1, 3.13.0-35-generic
But did not reproduce under the following version:
Ubuntu 10.04, 2.6.32-38-server
The problem reproduced under LVM, but did not reproduce without LVM.
I reproduced the problem as follows under VMWare Fusion:
set up custom VM with default disk size (20 GB) and memory size (1 GB)
attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up
select all defaults during installation _including_ LVM
install gcc
unpack the attached repro.tgz
run repro.sh
what it does:
* fills the disk with a file containing bytes of 0xcc then deletes it
* repeatedly runs the repro program which creates two files and accesses them
in a certain pattern
* checks the file f0 with hexdump; it should contain all 0s, but if pages
0x1000-0x7000 contain 0xcc you have reproduced the problem
If the problem does not appear to reproduce, please try waiting a bit
and checking the f0 files with hexdump again. This behavior was
observed by a customer reproducing the problem under ESXi. I since
added an sync after the running the repro binary which I think will
fix that.
If you still can't reproduce the problem please let me know if there's
anything I can do to help. For example can we trace the disk accesses
at the SCSI level to verify whether the appropriate SCSI commands are
being sent? This may help determine whether the problem is in Linux or
in VMWare.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
Re: [Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
This is what the customer reports. Ignoring the CD-ROM, this is slightly
different from what I see on my VMWare Fusion installation, which reports
VMware, (with a comma and a space). Presumably this is an insignificant
difference?
Bruce
tail /sys/class/scsi_device/*/device/{vendor,model}
== /sys/class/scsi_device/1:0:0:0/device/vendor ==
NECVMWar
== /sys/class/scsi_device/2:0:0:0/device/vendor ==
VMware
== /sys/class/scsi_device/1:0:0:0/device/model ==
VMware IDE CDR10
== /sys/class/scsi_device/2:0:0:0/device/model ==
Virtual disk
On Wed, Sep 24, 2014 at 11:25 AM, Chris J Arges 1371...@bugs.launchpad.net
wrote:
Bruce,
The following should get some of the info needed:
tail /sys/class/scsi_device/*/device/{vendor,model}
--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1371591
Title:
file not initialized to 0s under some conditions on VMWare
Status in “linux” package in Ubuntu:
In Progress
Status in “linux” source package in Trusty:
New
Bug description:
Under some conditions, after fallocate() the file is observed not to
be completely initilized to 0s: some 4KB pages have left-over data
from previous files that occupied those pages. Note that in addition
to causing functional problems for applications expecting files to be
initialized to 0s, this is a security issue because it allows data to
leak from one file to another, bypassing file access controls.
The problem has been seen running under the following VMWare-based
virtual environments:
Fusion 6.0.2
ESXi 5.1.0
And under the following versions of Ubuntu:
Ubuntu 12.04, 3.11.0-26-generic
Ubuntu 14.04.1, 3.13.0-32-generic
Ubuntu 14.04.1, 3.13.0-35-generic
But did not reproduce under the following version:
Ubuntu 10.04, 2.6.32-38-server
The problem reproduced under LVM, but did not reproduce without LVM.
I reproduced the problem as follows under VMWare Fusion:
set up custom VM with default disk size (20 GB) and memory size (1 GB)
attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up
select all defaults during installation _including_ LVM
install gcc
unpack the attached repro.tgz
run repro.sh
what it does:
* fills the disk with a file containing bytes of 0xcc then deletes it
* repeatedly runs the repro program which creates two files and accesses
them in a certain pattern
* checks the file f0 with hexdump; it should contain all 0s, but if
pages 0x1000-0x7000 contain 0xcc you have reproduced the problem
If the problem does not appear to reproduce, please try waiting a bit
and checking the f0 files with hexdump again. This behavior was
observed by a customer reproducing the problem under ESXi. I since
added an sync after the running the repro binary which I think will
fix that.
If you still can't reproduce the problem please let me know if there's
anything I can do to help. For example can we trace the disk accesses
at the SCSI level to verify whether the appropriate SCSI commands are
being sent? This may help determine whether the problem is in Linux or
in VMWare.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1371591
Title:
file not initialized to 0s under some conditions on VMWare
Status in “linux” package in Ubuntu:
In Progress
Status in “linux” source package in Trusty:
New
Bug description:
Under some conditions, after fallocate() the file is observed not to
be completely initilized to 0s: some 4KB pages have left-over data
from previous files that occupied those pages. Note that in addition
to causing functional problems for applications expecting files to be
initialized to 0s, this is a security issue because it allows data to
leak from one file to another, bypassing file access controls.
The problem has been seen running under the following VMWare-based virtual
environments:
Fusion 6.0.2
ESXi 5.1.0
And under the following versions of Ubuntu:
Ubuntu 12.04, 3.11.0-26-generic
Ubuntu 14.04.1, 3.13.0-32-generic
Ubuntu 14.04.1, 3.13.0-35-generic
But did not reproduce under the following version:
Ubuntu 10.04, 2.6.32-38-server
The problem reproduced under LVM, but did not reproduce without LVM.
I reproduced the problem as follows under VMWare Fusion:
set up custom VM with default disk size (20 GB) and memory size (1 GB)
attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up
select all defaults during installation _including_ LVM
install gcc
unpack the attached repro.tgz
run repro.sh
what it does:
* fills the disk with a file containing bytes of 0xcc then deletes it
* repeatedly runs the repro program which creates two
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
Ok my experiment doing the following (and I know launchpad will mangle
my spacing...):
diff --git a/block/blk-lib.c b/block/blk-lib.c
index 9b5b561..03ad981 100644
--- a/block/blk-lib.c
+++ b/block/blk-lib.c
@@ -283,6 +283,7 @@ int __blkdev_issue_zeroout(struct block_device *bdev,
sector_t sector,
int blkdev_issue_zeroout(struct block_device *bdev, sector_t sector,
sector_t nr_sects, gfp_t gfp_mask)
{
+#if 0
if (bdev_write_same(bdev)) {
unsigned char bdn[BDEVNAME_SIZE];
@@ -293,7 +294,7 @@ int blkdev_issue_zeroout(struct block_device *bdev,
sector_t sector,
bdevname(bdev, bdn);
pr_err(%s: WRITE SAME failed. Manually zeroing.\n, bdn);
}
-
+#endif
return __blkdev_issue_zeroout(bdev, sector, nr_sects, gfp_mask);
}
EXPORT_SYMBOL(blkdev_issue_zeroout);
Forcing manual zeroout works fine, so the 'hardware' zeroout is the problem.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1371591
Title:
file not initialized to 0s under some conditions on VMWare
Status in “linux” package in Ubuntu:
In Progress
Status in “linux” source package in Trusty:
New
Bug description:
Under some conditions, after fallocate() the file is observed not to
be completely initilized to 0s: some 4KB pages have left-over data
from previous files that occupied those pages. Note that in addition
to causing functional problems for applications expecting files to be
initialized to 0s, this is a security issue because it allows data to
leak from one file to another, bypassing file access controls.
The problem has been seen running under the following VMWare-based virtual
environments:
Fusion 6.0.2
ESXi 5.1.0
And under the following versions of Ubuntu:
Ubuntu 12.04, 3.11.0-26-generic
Ubuntu 14.04.1, 3.13.0-32-generic
Ubuntu 14.04.1, 3.13.0-35-generic
But did not reproduce under the following version:
Ubuntu 10.04, 2.6.32-38-server
The problem reproduced under LVM, but did not reproduce without LVM.
I reproduced the problem as follows under VMWare Fusion:
set up custom VM with default disk size (20 GB) and memory size (1 GB)
attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up
select all defaults during installation _including_ LVM
install gcc
unpack the attached repro.tgz
run repro.sh
what it does:
* fills the disk with a file containing bytes of 0xcc then deletes it
* repeatedly runs the repro program which creates two files and accesses them
in a certain pattern
* checks the file f0 with hexdump; it should contain all 0s, but if pages
0x1000-0x7000 contain 0xcc you have reproduced the problem
If the problem does not appear to reproduce, please try waiting a bit
and checking the f0 files with hexdump again. This behavior was
observed by a customer reproducing the problem under ESXi. I since
added an sync after the running the repro binary which I think will
fix that.
If you still can't reproduce the problem please let me know if there's
anything I can do to help. For example can we trace the disk accesses
at the SCSI level to verify whether the appropriate SCSI commands are
being sent? This may help determine whether the problem is in Linux or
in VMWare.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
As a workaround, you can use IDE/SATA disks instead of SCSI (default), I'm not sure of performance implications, but the test cases passes. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: In Progress Status in “linux” source package in Trusty: New Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
Can you try this build and see if it fixes the issue? Please install it into the guest VM: http://people.canonical.com/~arges/lp1371591/ Thanks, -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: In Progress Status in “linux” source package in Trusty: New Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
I can confirm that this fixes the issue, both for the mongod repro and for the standalone repro attached to this report. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: In Progress Status in “linux” source package in Trusty: New Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
** Summary changed: - FS Corruption with Ubuntu and VMWare + file not initialized to 0s under some conditions on VMWare ** Attachment added: reproducer https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+attachment/4208902/+files/repro.tgz -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: Incomplete Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _except_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
So are you sure 20GB is enough disk space for the VM? I tried in a KVM VM (just to try the reproducer) and I get 'No space left on device' errors from fallocate. In addition can you post the machine information someone in this bug so I can reproduce the exact setup on my end with vmware. Also to be clear, LVM _is_ required to reproduce the issue. Thanks for the overall great bug description and reproducer script! --chris -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: Incomplete Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _except_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
** Attachment added: sample repro script output https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+attachment/4208971/+files/repro.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: Incomplete Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
20 GB should be more than enough. It should run the repro binary several times, using more disk space each time since it leaves the each run in place when it goes on to the next, and then get a failure from fallocate on the last one when the disk is filled up. I've attached a file showing a successful repro run on a 20 GB disk. Could the no space errors you are seeing be a ulimit issue? Yes, LVM is required to reproduce the issue; sorry about the error in the repro steps; fixed now. Can you clarify what machine information you're looking for? ** Description changed: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up - select all defaults during installation _except_ LVM + select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: Incomplete Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
When creating the VM, did you use the default settings, or was the hardware configuration changed at all? I.e. did you add extra processors, or just use 1? I'll look into my errors a bit more, getting VMWare Workstation setup; has this been only reproduced in VMWare Fusion? Also are there specific versions of VMWare * you have used to repo the issue? Also what kernel version can repo this issue? (uname -a)? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: In Progress Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
** Changed in: linux (Ubuntu) Status: Incomplete = In Progress -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: In Progress Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
@bruce-lucas: Ok! I can reproduce this issue in VMWare Workstation. I'll start investigating more deeply. In my KVM instance previously I did not reproduce the issue. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: In Progress Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
I accepted all the default settings when creating the VM (which for Fusion was 20 GB disk, 1 GB memory, single processor). On Fusion at least It is important to do the manual install: select installation method / more options / create a custom VM, mount the CD, set as boot device, bot up, go through the Ubuntu installer, accept the default to use LVM. If you go through select installation method / install from disc it does not take you through the Ubuntu installer UI, and it does not use LVM. At least that's what I see in Fusion; can't say for sure about Workstation. I reproduced it on VMWare Fusion, a customer reproduced it on VMWare ESXi (versions listed in original description). The kernel versions where it has been reproduced are also listed above next to the Ubuntu distro version. Some more information: * was able to reproduce the problem on Fedora 20 Fedora 20 (3.11.10-301, with LVM) * did not reproduce the problem on Centos 7.0 (3.10.0-123, with LVM) So so far all repros have been on 3.11 kernels or later. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: In Progress Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
** Information type changed from Public to Public Security -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: In Progress Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
Awesome, thanks. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: In Progress Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
By the way, a couple more pieces of information that may be relevant: The problem is sensitive to the particular pattern of access to map0. If you remove either of the two writes (at 0x0 and 0x7000) the problem disappears, or if you change 0x7000 to a higher page it also disappears. We also observed a message about a WRITE SAME failure in syslog. I believe this means that the platform does not implement SCSI WRITE SAME, which I imagine would be used to zero pages under some circumstances, but instead uses a fallback of manually zeroing the pages. Perhaps there is a problem in this area? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: In Progress Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
** Tags added: kernel-da-key -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: In Progress Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
@bruce-lucas OK I have reproduced this with the latest 3.17-rc kernel. So most likely this is an upstream issue. However I'll start testing previous versions to see if this is a regression between 3.10,3.11; this will help us zero in on the code changes that may have introduced this behavior. I do get the same error on my runs: [ 176.092640] dm-0: WRITE SAME failed. Manually zeroing. I'll investigate this in parallel. ** Tags added: kernel-bug-exists-upstream ** Also affects: linux (Ubuntu Trusty) Importance: Undecided Status: New -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: In Progress Status in “linux” source package in Trusty: New Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
The 'regression' is between 3.9 and 3.10-rc1. I'll bisect between these tags to see where the issue is. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: In Progress Status in “linux” source package in Trusty: New Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
The bisect resulted in the following: dc019b21fb92d620a3b52ccecc135ac968a7c7ec is the first bad commit commit dc019b21fb92d620a3b52ccecc135ac968a7c7ec Author: Mike Snitzer snit...@redhat.com Date: Fri May 10 14:37:16 2013 +0100 dm table: fix write same support If device_not_write_same_capable() returns true then the iterate_devices loop in dm_table_supports_write_same() should return false. Reported-by: Bharata B Rao bharata@gmail.com Signed-off-by: Mike Snitzer snit...@redhat.com Cc: sta...@vger.kernel.org # v3.8+ Signed-off-by: Alasdair G Kergon a...@redhat.com :04 04 d8b62d18789b5c9e5b52c076abcf4c8c066b5d59 71a5511a8ea76f43bd167524a9186c1d78407bce M drivers -- However, I don't think the issue is with this patch. The function 'device_not_write_same_capable()' correctly returns: return q !q-limits.max_write_same_sectors; If max_write_same_sectors is 0 (write_same not supported), then true is returned and thus 'not_write_same_capable'. Likewise the function 'dm_table_supports_write_same' iterates through dm tables and checks if (!ti-type-iterate_devices || ti-type-iterate_devices(ti, device_not_write_same_capable, NULL)) return false; So if iterate_devices is NULL, this if returns false, otherwise if iterate_devices exist, then device_not_write_same_capable is called, if it returns 'true' then the function returns 'false' (A bit confusing, but essentially the parent function is 'supports_write_same' and uses a 'not_write_same_capable' function to check this fact. ) That logic was introduced in: d54eaa5a0fde0a202e4e91f200f818edcef15bee (v3.8-rc1), which means that previous to that we might not see the same behavior which could account for 2.6.38 not failing this test case. Relevant thread: http://www.spinics.net/lists/dm-devel/msg19583.html -- Looking at the affected VM: Now this makes sense why LVM is only affected, and explains the helpful kernel message output. If we check the dm's for our LVM vg's we see the following: ubuntu@ubuntu:~$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda8:0020G 0 disk ├─sda1 8:10 243M 0 part /boot ├─sda2 8:20 1K 0 part └─sda5 8:50 19.8G 0 part ├─ubuntu--vg-root (dm-0) 252:00 18.8G 0 lvm / └─ubuntu--vg-swap_1 (dm-1) 252:10 1020M 0 lvm [SWAP] sr0 11:01 572M 0 rom ubuntu@ubuntu:~$ cat /sys/dev/block/252\:1/queue/write_same_max_bytes 33553920 ubuntu@ubuntu:~$ cat /sys/dev/block/252\:0/queue/write_same_max_bytes 33553920 So write_same support is enabled, but then that causes the failure. So at this point, I wonder if the underlying virtual SCSI is at fault. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: In Progress Status in “linux” source package in Trusty: New Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
FWIW, I also tested this on a similar KVM instance using a SCSI disk device and installed Ubuntu using LVM, running the same test case does not result in the failure; even though write_same is enabled for those devices. ubuntu@lp1371591:~$ lsblk NAMEMAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0030G 0 disk ├─sda18:10 243M 0 part /boot ├─sda28:20 1K 0 part └─sda58:50 29.8G 0 part ├─lp1371591--vg-root (dm-0) 252:00 28.8G 0 lvm / └─lp1371591--vg-swap_1 (dm-1) 252:10 1G 0 lvm [SWAP] sr0 11:01 1024M 0 rom ubuntu@lp1371591:~$ cat /sys/block/dm-*/queue/write_same_max_bytes 33553920 33553920 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: In Progress Status in “linux” source package in Trusty: New Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
The bisect resulted in the following: dc019b21fb92d620a3b52ccecc135ac968a7c7ec is the first bad commit commit dc019b21fb92d620a3b52ccecc135ac968a7c7ec Author: Mike Snitzer snit...@redhat.com Date: Fri May 10 14:37:16 2013 +0100 dm table: fix write same support If device_not_write_same_capable() returns true then the iterate_devices loop in dm_table_supports_write_same() should return false. Reported-by: Bharata B Rao bharata@gmail.com Signed-off-by: Mike Snitzer snit...@redhat.com Cc: sta...@vger.kernel.org # v3.8+ Signed-off-by: Alasdair G Kergon a...@redhat.com :04 04 d8b62d18789b5c9e5b52c076abcf4c8c066b5d59 71a5511a8ea76f43bd167524a9186c1d78407bce M drivers -- However, I don't think the issue is with this patch. The function 'device_not_write_same_capable()' correctly returns: return q !q-limits.max_write_same_sectors; If max_write_same_sectors is 0 (write_same not supported), then true is returned and thus 'not_write_same_capable'. Likewise the function 'dm_table_supports_write_same' iterates through dm tables and checks if (!ti-type-iterate_devices || ti-type-iterate_devices(ti, device_not_write_same_capable, NULL)) return false; So if iterate_devices is NULL, this if returns false, otherwise if iterate_devices exist, then device_not_write_same_capable is called, if it returns 'true' then the function returns 'false' (A bit confusing, but essentially the parent function is 'supports_write_same' and uses a 'not_write_same_capable' function to check this fact. ) That logic was introduced in: d54eaa5a0fde0a202e4e91f200f818edcef15bee (v3.8-rc1), which means that previous to that we might not see the same behavior which could account for 2.6.38 not failing this test case. Relevant thread: http://www.spinics.net/lists/dm-devel/msg19583.html -- Looking at the affected VM: Now this makes sense why LVM is only affected, and explains the helpful kernel message output. If we check the dm's for our LVM vg's we see the following: ubuntu@ubuntu:~$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda8:0020G 0 disk ├─sda1 8:10 243M 0 part /boot ├─sda2 8:20 1K 0 part └─sda5 8:50 19.8G 0 part ├─ubuntu--vg-root (dm-0) 252:00 18.8G 0 lvm / └─ubuntu--vg-swap_1 (dm-1) 252:10 1020M 0 lvm [SWAP] sr0 11:01 572M 0 rom ubuntu@ubuntu:~$ cat /sys/dev/block/252\:1/queue/write_same_max_bytes 33553920 ubuntu@ubuntu:~$ cat /sys/dev/block/252\:0/queue/write_same_max_bytes 33553920 So write_same support is enabled, but then that causes the failure. So at this point, I wonder if the underlying virtual SCSI is at fault. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: In Progress Status in “linux” source package in Trusty: New Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I
[Kernel-packages] [Bug 1371591] Re: file not initialized to 0s under some conditions on VMWare
As you mentioned in #10, blkdev_issue_zeroout (the function that prints 'WRITE SAME failed. Manually zeroing.'), still causes the test to show failures even if we should be manually zeroing out the block device when the hardware write_same fails. A test I could try is to short circuit BLKZEROOUT to always use the manual case and see if there are still failures. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1371591 Title: file not initialized to 0s under some conditions on VMWare Status in “linux” package in Ubuntu: In Progress Status in “linux” source package in Trusty: New Bug description: Under some conditions, after fallocate() the file is observed not to be completely initilized to 0s: some 4KB pages have left-over data from previous files that occupied those pages. Note that in addition to causing functional problems for applications expecting files to be initialized to 0s, this is a security issue because it allows data to leak from one file to another, bypassing file access controls. The problem has been seen running under the following VMWare-based virtual environments: Fusion 6.0.2 ESXi 5.1.0 And under the following versions of Ubuntu: Ubuntu 12.04, 3.11.0-26-generic Ubuntu 14.04.1, 3.13.0-32-generic Ubuntu 14.04.1, 3.13.0-35-generic But did not reproduce under the following version: Ubuntu 10.04, 2.6.32-38-server The problem reproduced under LVM, but did not reproduce without LVM. I reproduced the problem as follows under VMWare Fusion: set up custom VM with default disk size (20 GB) and memory size (1 GB) attach Ubuntu 14.04.1 ISO to CDROM, set it as boot device, boot up select all defaults during installation _including_ LVM install gcc unpack the attached repro.tgz run repro.sh what it does: * fills the disk with a file containing bytes of 0xcc then deletes it * repeatedly runs the repro program which creates two files and accesses them in a certain pattern * checks the file f0 with hexdump; it should contain all 0s, but if pages 0x1000-0x7000 contain 0xcc you have reproduced the problem If the problem does not appear to reproduce, please try waiting a bit and checking the f0 files with hexdump again. This behavior was observed by a customer reproducing the problem under ESXi. I since added an sync after the running the repro binary which I think will fix that. If you still can't reproduce the problem please let me know if there's anything I can do to help. For example can we trace the disk accesses at the SCSI level to verify whether the appropriate SCSI commands are being sent? This may help determine whether the problem is in Linux or in VMWare. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371591/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
