Re: [zfs-discuss] how to know available disk space
On Wed, Feb 06, 2013 at 08:03:13PM -0700, Jan Owoc wrote: On Wed, Feb 6, 2013 at 4:26 PM, Edward Ned Harvey (opensolarisisdeadlongliveopensolaris) opensolarisisdeadlongliveopensola...@nedharvey.com wrote: When I used zpool status after the system crashed, I saw this: NAME SIZE ALLOC FREE EXPANDSZCAP DEDUP HEALTH ALTROOT storage 928G 568G 360G -61% 1.00x ONLINE - I did some cleanup, so I could turn things back on ... Freed up about 4G. Now, when I use zpool status I see this: NAME SIZE ALLOC FREE EXPANDSZCAP DEDUP HEALTH ALTROOT storage 928G 564G 364G -60% 1.00x ONLINE - When I use zfs list storage I see this: NAME USED AVAIL REFER MOUNTPOINT storage 909G 4.01G 32.5K /storage So I guess the lesson is (a) refreservation and zvol alone aren't enough to ensure your VM's will stay up. and (b) if you want to know how much room is *actually* available, as in usable, as in, how much can I write before I run out of space, you should use zfs list and not zpool status Could you run zfs list -o space storage? It will show how much is used by the data, the snapshots, refreservation, and children (if any). I read somewhere that one should always use zfs list to determine how much space is actually available to be written on a given filesystem. I have an idea, but it's a long shot. If you created more than one zfs on that pool, and added a reservation to each one, then that space is still technically unallocated as far as zpool list is concerned, but is not available to writing when you do zfs list. I would imagine you have one or more of your VMs that grew outside of their refreservation and now crashed for lack of free space on their zfs. Some of the other VMs aren't using their refreservation (yet), so they could, between them, still write 360GB of stuff to the drive. I'm seeing weird output aswell: # zpool list foo NAME SIZE ALLOC FREECAP DEDUP HEALTH ALTROOT foo 5.44T 4.44T 1023G81% 14.49x ONLINE - # zfs list | grep foo foo 62.9T 0 250G /volumes/foo foo/.nza-reserve 31K 100M31K none foo/foo 62.6T 0 62.6T /volumes/foo/foo # zfs list -o space foo NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD foo 0 62.9T 0250G 0 62.7T # zfs list -o space foo/foo NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD foo/foo 0 62.6T 0 62.6T 0 0 What's the correct way of finding out what actually uses/reserves that 1023G of FREE in the zpool? At this point the filesystems are full, and it's not possible to write to them anymore. Also creating new filesystems to the pool fail: Operation completed with error: cannot create 'foo/Test': out of space So the zpool is full for real. I'd like to better understand what actually uses that 1023G of FREE space reported by zpool.. 1023G out of 4.32T is around 22% overhead.. zpool foo consists of 3x mirror vdevs, so there's no raidz involved. 62.6T / 14.49x dedup-ratio = 4.32T Which is pretty close to the ALLOC value reported by zpool.. Data on the filesystem is VM images written over NFS. Thanks, -- Pasi ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] how to know available disk space
From: Pasi Kärkkäinen [mailto:pa...@iki.fi] What's the correct way of finding out what actually uses/reserves that 1023G of FREE in the zpool? Maybe this isn't exactly what you need, but maybe: for fs in `zfs list -H -o name` ; do echo $fs ; zfs get reservation,refreservation,usedbyrefreservation $fs ; done At this point the filesystems are full, and it's not possible to write to them anymore. You'll have to either reduce your reservations, or destroy old snapshots. Or add more disks. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] how to know available disk space
On Fri, Feb 08, 2013 at 09:47:38PM +, Edward Ned Harvey (opensolarisisdeadlongliveopensolaris) wrote: From: Pasi Kärkkäinen [mailto:pa...@iki.fi] What's the correct way of finding out what actually uses/reserves that 1023G of FREE in the zpool? Maybe this isn't exactly what you need, but maybe: for fs in `zfs list -H -o name` ; do echo $fs ; zfs get reservation,refreservation,usedbyrefreservation $fs ; done I checked this and there are no reservations configured (or well, there are the 100MB defaults, but not more than that). So reservations don't explain this.. At this point the filesystems are full, and it's not possible to write to them anymore. You'll have to either reduce your reservations, or destroy old snapshots. Or add more disks. There aren't any snapshots either.. I know adding disks will fix the problem, but I'd like to understand why zpool says there is almost 1TB of FREE space when clearly there isn't.. Thanks for the reply! -- Pasi ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] how to know available disk space
On 2013-02-08 22:47, Edward Ned Harvey
(opensolarisisdeadlongliveopensolaris) wrote:
Maybe this isn't exactly what you need, but maybe:
for fs in `zfs list -H -o name` ; do echo $fs ; zfs get
reservation,refreservation,usedbyrefreservation $fs ; done
What is the sacramental purpose of such construct in comparison to:
zfs list -H -o reservation,refreservation,usedbyrefreservation,name \
-t filesystem {-r pool/interesting/dataset}
Just asking - or suggesting a simpler way to do stuff ;)
//Jim
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] how to know available disk space
On Feb 6, 2013, at 5:17 PM, Gregg Wonderly gregg...@gmail.com wrote: This is one of the greatest annoyances of ZFS. I don't really understand how, a zvol's space can not be accurately enumerated from top to bottom of the tree in 'df' output etc. Why does a zvol divorce the space used from the root of the volume? Thick (with reservation) or thin provisioning behave differently. Also, depending on how you created the reservation, it might or might not account for the metadata overhead needed. By default, space for metadata is reserved, but if you use the -s (sparse aka thin provisioning) option, then later reservation changes are set as absolute. Also, metadata space, compression, copies, and deduplication must be accounted for. The old notions of free/available don't match very well with these modern features. Gregg Wonderly On Feb 6, 2013, at 5:26 PM, Edward Ned Harvey (opensolarisisdeadlongliveopensolaris) opensolarisisdeadlongliveopensola...@nedharvey.com wrote: I have a bunch of VM's, and some samba shares, etc, on a pool. I created the VM's using zvol's, specifically so they would have an appropriaterefreservation and never run out of disk space, even with snapshots. Today, I ran out of disk space, and all the VM's died. So obviously it didn't work. When I used zpool status after the system crashed, I saw this: NAME SIZE ALLOC FREE EXPANDSZCAP DEDUP HEALTH ALTROOT storage 928G 568G 360G -61% 1.00x ONLINE - I did some cleanup, so I could turn things back on ... Freed up about 4G. Now, when I use zpool status I see this: NAME SIZE ALLOC FREE EXPANDSZCAP DEDUP HEALTH ALTROOT storage 928G 564G 364G -60% 1.00x ONLINE - When I use zfs list storage I see this: NAME USED AVAIL REFER MOUNTPOINT storage 909G 4.01G 32.5K /storage So I guess the lesson is (a) refreservation and zvol alone aren't enough to ensure your VM's will stay up. and (b) if you want to know how much room is *actually* available, as in usable, as in, how much can I write before I run out of space, you should use zfs list and not zpoolstatus Correct. zpool status does not show dataset space available. -- richard -- richard.ell...@richardelling.com +1-760-896-4422 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] how to know available disk space
I have a bunch of VM's, and some samba shares, etc, on a pool. I created the VM's using zvol's, specifically so they would have an appropriate refreservation and never run out of disk space, even with snapshots. Today, I ran out of disk space, and all the VM's died. So obviously it didn't work. When I used zpool status after the system crashed, I saw this: NAME SIZE ALLOC FREE EXPANDSZCAP DEDUP HEALTH ALTROOT storage 928G 568G 360G -61% 1.00x ONLINE - I did some cleanup, so I could turn things back on ... Freed up about 4G. Now, when I use zpool status I see this: NAME SIZE ALLOC FREE EXPANDSZCAP DEDUP HEALTH ALTROOT storage 928G 564G 364G -60% 1.00x ONLINE - When I use zfs list storage I see this: NAME USED AVAIL REFER MOUNTPOINT storage 909G 4.01G 32.5K /storage So I guess the lesson is (a) refreservation and zvol alone aren't enough to ensure your VM's will stay up. and (b) if you want to know how much room is *actually* available, as in usable, as in, how much can I write before I run out of space, you should use zfs list and not zpool status ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] how to know available disk space
This is one of the greatest annoyances of ZFS. I don't really understand how, a zvol's space can not be accurately enumerated from top to bottom of the tree in 'df' output etc. Why does a zvol divorce the space used from the root of the volume? Gregg Wonderly On Feb 6, 2013, at 5:26 PM, Edward Ned Harvey (opensolarisisdeadlongliveopensolaris) opensolarisisdeadlongliveopensola...@nedharvey.com wrote: I have a bunch of VM's, and some samba shares, etc, on a pool. I created the VM's using zvol's, specifically so they would have an appropriaterefreservation and never run out of disk space, even with snapshots. Today, I ran out of disk space, and all the VM's died. So obviously it didn't work. When I used zpool status after the system crashed, I saw this: NAME SIZE ALLOC FREE EXPANDSZCAP DEDUP HEALTH ALTROOT storage 928G 568G 360G -61% 1.00x ONLINE - I did some cleanup, so I could turn things back on ... Freed up about 4G. Now, when I use zpool status I see this: NAME SIZE ALLOC FREE EXPANDSZCAP DEDUP HEALTH ALTROOT storage 928G 564G 364G -60% 1.00x ONLINE - When I use zfs list storage I see this: NAME USED AVAIL REFER MOUNTPOINT storage 909G 4.01G 32.5K /storage So I guess the lesson is (a) refreservation and zvol alone aren't enough to ensure your VM's will stay up. and (b) if you want to know how much room is *actually* available, as in usable, as in, how much can I write before I run out of space, you should use zfs list and not zpoolstatus ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] how to know available disk space
From: Gregg Wonderly [mailto:gregg...@gmail.com] This is one of the greatest annoyances of ZFS. I don't really understand how, a zvol's space can not be accurately enumerated from top to bottom of the tree in 'df' output etc. Why does a zvol divorce the space used from the root of the volume? The way I would say that is: Intuitively, I think people expect reservations to count against Alloc and Used. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] how to know available disk space
On Wed, Feb 6, 2013 at 4:26 PM, Edward Ned Harvey (opensolarisisdeadlongliveopensolaris) opensolarisisdeadlongliveopensola...@nedharvey.com wrote: When I used zpool status after the system crashed, I saw this: NAME SIZE ALLOC FREE EXPANDSZCAP DEDUP HEALTH ALTROOT storage 928G 568G 360G -61% 1.00x ONLINE - I did some cleanup, so I could turn things back on ... Freed up about 4G. Now, when I use zpool status I see this: NAME SIZE ALLOC FREE EXPANDSZCAP DEDUP HEALTH ALTROOT storage 928G 564G 364G -60% 1.00x ONLINE - When I use zfs list storage I see this: NAME USED AVAIL REFER MOUNTPOINT storage 909G 4.01G 32.5K /storage So I guess the lesson is (a) refreservation and zvol alone aren't enough to ensure your VM's will stay up. and (b) if you want to know how much room is *actually* available, as in usable, as in, how much can I write before I run out of space, you should use zfs list and not zpool status Could you run zfs list -o space storage? It will show how much is used by the data, the snapshots, refreservation, and children (if any). I read somewhere that one should always use zfs list to determine how much space is actually available to be written on a given filesystem. I have an idea, but it's a long shot. If you created more than one zfs on that pool, and added a reservation to each one, then that space is still technically unallocated as far as zpool list is concerned, but is not available to writing when you do zfs list. I would imagine you have one or more of your VMs that grew outside of their refreservation and now crashed for lack of free space on their zfs. Some of the other VMs aren't using their refreservation (yet), so they could, between them, still write 360GB of stuff to the drive. Jan ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
