Re: [zfs-discuss] zero out block / sectors
On 2010-01-28 at 00:30 +0100 Ragnar Sundblad sent off: Are there any plans to add unwritten extent support into ZFS or any reason why not? I have no idea, but just out of curiosity - when do you want that? when you have many data streams being simultaneous written to disk over a long time you end up with highly fragmented files. When you write to a file with random seeks (think of torrent) you end up with an extremely fragmented file. When you have a virtual disk image which grows over time, you will get a fragmented disk image which affects performance bad. When you write to a file and you want to be sure you have enough space to write all the data, even if writing to that file takes some time, preallocation takes care the space is yours. Threre are plenty of reasons to do preallocation. With unwritten extent support preallocation is a cheap thing to do. Björn ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zero out block / sectors
On 28 jan 2010, at 12.11, Björn JACKE wrote: On 2010-01-28 at 00:30 +0100 Ragnar Sundblad sent off: Are there any plans to add unwritten extent support into ZFS or any reason why not? I have no idea, but just out of curiosity - when do you want that? when you have many data streams being simultaneous written to disk over a long time you end up with highly fragmented files. When you write to a file with random seeks (think of torrent) you end up with an extremely fragmented file. When you have a virtual disk image which grows over time, you will get a fragmented disk image which affects performance bad. When you write to a file and you want to be sure you have enough space to write all the data, even if writing to that file takes some time, preallocation takes care the space is yours. Threre are plenty of reasons to do preallocation. With unwritten extent support preallocation is a cheap thing to do. Ok, I now understand what you want, thanks! On zfs, your files will get fragmented anyway, preallocated or not, because all new data and metadata in a transaction group is written to unallocated blocks on the disk. As far as I know, there is no way to have zfs write to the same place as the data you are rewriting. /ragge ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zero out block / sectors
On 2010-01-25 at 08:31 -0600 Mike Gerdts sent off: You are missing the point. Compression and dedup will make it so that the blocks in the devices are not overwritten with zeroes. The goal is to overwrite the blocks so that a back-end storage device or back-end virtualization platform can recognize that the blocks are not in use and as such can reclaim the space. a filesystem that is able to do that fast would have to implement something like unwritten extents. Some days ago I experimented to create and allocate huge files on ZFS ontop of OpenSolaris using fnctl and F_ALLOCSP which is basically the same thing that you want to do when you zero out space. It takes ages because it actually writes zeroes to the disk. A filesystem that knows the concept of unwritten extents finishes the job immediately. There are no real zeros on the disk but the extent is tagged to be unwritten (you get zeros when you read it). Are there any plans to add unwritten extent support into ZFS or any reason why not? Björn ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zero out block / sectors
On 2010-01-27 at 09:50 + Darren J Moffat sent off: The whole point of the original question wasn't about consumers of ZFS but where ZFS is the consumer of block storage provided by something else that expects to see zeros on disk. This thread is about thin provisioning *to* ZFS not *on* it. you're right, indeed the original question is actually a different problem that unwritten extents support wouldn't address. Björn ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zero out block / sectors
On 27 jan 2010, at 10.44, Björn JACKE wrote:
On 2010-01-25 at 08:31 -0600 Mike Gerdts sent off:
You are missing the point. Compression and dedup will make it so that
the blocks in the devices are not overwritten with zeroes. The goal
is to overwrite the blocks so that a back-end storage device or
back-end virtualization platform can recognize that the blocks are not
in use and as such can reclaim the space.
a filesystem that is able to do that fast would have to implement something
like unwritten extents.
Rather what is needed is files with holes, as what is expected here is more
free space in the file system when the unused parts of the file is punched out.
With F_ALLOCSP, you would still not be able to use the space and there would be
no gain.
Some days ago I experimented to create and allocate
huge files on ZFS ontop of OpenSolaris using fnctl and F_ALLOCSP which is
basically the same thing that you want to do when you zero out space. It takes
ages because it actually writes zeroes to the disk. A filesystem that knows
the
concept of unwritten extents finishes the job immediately. There are no real
zeros on the disk but the extent is tagged to be unwritten (you get zeros when
you read it).
Files with holes are implemented, and as far as I know they are fast too:
-bash-4.0$ cat hole.py
f = open('foo', 'w')
f.write('x')
f.seek(2**62)
f.write('y')
f.close()
-bash-4.0$ time python hole.py
real0m0.019s
user0m0.010s
sys 0m0.009s
-bash-4.0$ ls -la foo
-rw-r--r-- 1 raggestaff4611686018427387905 Jan 28 00:26 foo
Are there any plans to add unwritten extent support into ZFS or any reason why
not?
I have no idea, but just out of curiosity - when do you want that?
/ragge
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zero out block / sectors
On Mon, Jan 25, 2010 at 2:32 AM, Kjetil Torgrim Homme kjeti...@linpro.no wrote: Mike Gerdts mger...@gmail.com writes: John Hoogerdijk wrote: Is there a way to zero out unused blocks in a pool? I'm looking for ways to shrink the size of an opensolaris virtualbox VM and using the compact subcommand will remove zero'd sectors. I've long suspected that you should be able to just use mkfile or dd if=/dev/zero ... to create a file that consumes most of the free space then delete that file. Certainly it is not an ideal solution, but seems quite likely to be effective. you'll need to (temporarily) enable compression for this to have an effect, AFAIK. (dedup will obviously work, too, if you dare try it.) You are missing the point. Compression and dedup will make it so that the blocks in the devices are not overwritten with zeroes. The goal is to overwrite the blocks so that a back-end storage device or back-end virtualization platform can recognize that the blocks are not in use and as such can reclaim the space. -- Mike Gerdts http://mgerdts.blogspot.com/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zero out block / sectors
Mike Gerdts mger...@gmail.com writes: Kjetil Torgrim Homme wrote: Mike Gerdts mger...@gmail.com writes: John Hoogerdijk wrote: Is there a way to zero out unused blocks in a pool? I'm looking for ways to shrink the size of an opensolaris virtualbox VM and using the compact subcommand will remove zero'd sectors. I've long suspected that you should be able to just use mkfile or dd if=/dev/zero ... to create a file that consumes most of the free space then delete that file. Certainly it is not an ideal solution, but seems quite likely to be effective. you'll need to (temporarily) enable compression for this to have an effect, AFAIK. (dedup will obviously work, too, if you dare try it.) You are missing the point. Compression and dedup will make it so that the blocks in the devices are not overwritten with zeroes. The goal is to overwrite the blocks so that a back-end storage device or back-end virtualization platform can recognize that the blocks are not in use and as such can reclaim the space. aha, I was assuming the OP's VirtualBox image was stored on ZFS, but I realise now that it's the other way around -- he's running ZFS inside a VirtualBox image hosted on a traditional filesystem. in that case you're right, and I'm wrong :-) -- Kjetil T. Homme Redpill Linpro AS - Changing the game ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zero out block / sectors
cs == Cindy Swearingen cindy.swearin...@sun.com writes: re == Richard Elling richard.ell...@gmail.com writes: cs http://defect.opensolaris.org/bz/show_bug.cgi?id=5993 the procedure described here is like doing FLAR by hand, so it'll probably be useful in quite a lot of situations, especially with the frequent genunix livecd's that understand new pool and zfs version numbers. re make a new virtual disk and attach it as a mirror. Once the re silver is complete, detach and destroy the old virtual disk. sounds heaps easier than the method in the bug. pgp5VNnGSKoPx.pgp Description: PGP signature ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zero out block / sectors
On Fri, Jan 22, 2010 at 2:42 PM, Cindy Swearingen cindy.swearin...@sun.com wrote: Hi John, You might check with the virtualguru, Rudolf Kutina. Unfortunately, Rudolfs last day at Sun was Jan 15th: http://blogs.sun.com/VirtualGuru/entry/kiss_of_dead_my_last you can still catch up with him at his new blog: http://virtguru.wordpress.com/ . .snip .. Regards, -- Al Hopper Logical Approach Inc,Plano,TX a...@logical-approach.com Voice: 972.379.2133 Timezone: US CDT OpenSolaris Governing Board (OGB) Member - Apr 2005 to Mar 2007 http://www.opensolaris.org/os/community/ogb/ogb_2005-2007/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zero out block / sectors
Mike Gerdts wrote: On Fri, Jan 22, 2010 at 1:00 PM, John Hoogerdijk john.hoogerd...@sun.com wrote: Is there a way to zero out unused blocks in a pool? I'm looking for ways to shrink the size of an opensolaris virtualbox VM and using the compact subcommand will remove zero'd sectors. I've long suspected that you should be able to just use mkfile or dd if=/dev/zero ... to create a file that consumes most of the free space then delete that file. Certainly it is not an ideal solution, but seems quite likely to be effective. I tried this with mkfile - no joy. jmh ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zero out block / sectors
On Sat, Jan 23, 2010 at 11:55 AM, John Hoogerdijk john.hoogerd...@sun.com wrote: Mike Gerdts wrote: On Fri, Jan 22, 2010 at 1:00 PM, John Hoogerdijk john.hoogerd...@sun.com wrote: Is there a way to zero out unused blocks in a pool? I'm looking for ways to shrink the size of an opensolaris virtualbox VM and using the compact subcommand will remove zero'd sectors. I've long suspected that you should be able to just use mkfile or dd if=/dev/zero ... to create a file that consumes most of the free space then delete that file. Certainly it is not an ideal solution, but seems quite likely to be effective. I tried this with mkfile - no joy. Let me ask a couple of the questions that come just after are you sure your computer is plugged in? Did you wait enough time for the data to be flushed to disk (or do sync and wait for it to complete) prior to removing the file? You did mkfile $huge /var/tmp/junk not mkfile -n $huge /var/tmp/junk, right? If not, I suspect that zpool replace to a thin provisioned disk is going to be your best bet (as suggested in another message). -- Mike Gerdts http://mgerdts.blogspot.com/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zero out block / sectors
Mike Gerdts wrote: On Sat, Jan 23, 2010 at 11:55 AM, John Hoogerdijk john.hoogerd...@sun.com wrote: Mike Gerdts wrote: On Fri, Jan 22, 2010 at 1:00 PM, John Hoogerdijk john.hoogerd...@sun.com wrote: Is there a way to zero out unused blocks in a pool? I'm looking for ways to shrink the size of an opensolaris virtualbox VM and using the compact subcommand will remove zero'd sectors. I've long suspected that you should be able to just use mkfile or dd if=/dev/zero ... to create a file that consumes most of the free space then delete that file. Certainly it is not an ideal solution, but seems quite likely to be effective. I tried this with mkfile - no joy. Let me ask a couple of the questions that come just after are you sure your computer is plugged in? :-) Did you wait enough time for the data to be flushed to disk (or do sync and wait for it to complete) prior to removing the file? yep - on the order of minutes and sync'd on the host as well. host is a Mac. You did mkfile $huge /var/tmp/junk not mkfile -n $huge /var/tmp/junk, right? yep. If not, I suspect that zpool replace to a thin provisioned disk is going to be your best bet (as suggested in another message). next on my list. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] zero out block / sectors
Is there a way to zero out unused blocks in a pool? I'm looking for ways to shrink the size of an opensolaris virtualbox VM and using the compact subcommand will remove zero'd sectors. Thanks, John Hoogerdijk Sun Microsystems of Canada IMO Network Computer NC Ltd. 808 240 Graham Avenue Winnipeg, Manitoba, R3C 0J7, Canada Phone: 204.927.1932 Cell: 204.230.6720 Fax: 204.927.1939 Email: john.hoogerd...@sun.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zero out block / sectors
John Hoogerdijk wrote: Is there a way to zero out unused blocks in a pool? I'm looking for ways to shrink the size of an opensolaris virtualbox VM and using the compact subcommand will remove zero'd sectors. Not yet, but this has been discussed here before. It is something I want to look at after I've got encryption support integrated. No promises but I want it for this and other purposes. -- Darren J Moffat ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zero out block / sectors
On Fri, Jan 22, 2010 at 1:00 PM, John Hoogerdijk john.hoogerd...@sun.com wrote: Is there a way to zero out unused blocks in a pool? I'm looking for ways to shrink the size of an opensolaris virtualbox VM and using the compact subcommand will remove zero'd sectors. I've long suspected that you should be able to just use mkfile or dd if=/dev/zero ... to create a file that consumes most of the free space then delete that file. Certainly it is not an ideal solution, but seems quite likely to be effective. -- Mike Gerdts http://mgerdts.blogspot.com/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
