Re: [zfs-discuss] [Fwd: ZFS user/group quotas space accounting [PSARC/2009/204 FastTrack timeout 04/08/2009]]
On Tue, Mar 31, 2009 at 8:47 PM, River Tarnell ri...@loreley.flyingparchment.org.uk wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Matthew Ahrens: does this mean that without an account on the NFS server, a user cannot see his current disk use / quota? That's correct. in this case, might i suggest at least an RFE to add ZFS quota support to rquotad? i'm sure we aren't the only site where non-privileged users don't have login rights to the NFS server, and it would be easier not to have to reimplement rquota locally to support ZFS. - river. For the benefit of those finding this conversation in the archives, this looks like it will be fixed in snv_114. http://bugs.opensolaris.org/view_bug.do?bug_id=6824968 http://hg.genunix.org/onnv-gate.hg/rev/4f68f041ddcd -- Mike Gerdts http://mgerdts.blogspot.com/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [Fwd: ZFS user/group quotas space accounting [PSARC/2009/204 FastTrack timeout 04/08/2009]]
Will this work with Linux rquota clients, too?
Olga
On 4/1/09, Matthew Ahrens matthew.ahr...@sun.com wrote:
Mike Gerdts wrote:
On Tue, Mar 31, 2009 at 7:12 PM, Matthew Ahrens matthew.ahr...@sun.com
wrote:
River Tarnell wrote:
Matthew Ahrens:
ZFS user quotas (like other zfs properties) will not be accessible
over
NFS;
you must be on the machine running zfs to manipulate them.
does this mean that without an account on the NFS server, a user
cannot
see his
current disk use / quota?
That's correct.
Do you have a reason for not wanting this to be implemented, or are
you just avoiding scope creep?
The latter -- I just don't have time to tack on any more features with
this. We've filed RFE 6824968 to add support to rquotad to report on zfs
user quotas. I'll note this in the PSARC case as well.
This additional RFE is staffed, but we don't have an ETA right now.
--matt
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
--
, __ ,
{ \/`o;-Olga Kryzhanovska -;o`\/ }
.'-/`-/ olga.kryzhanov...@gmail.com \-`\-'.
`'-..-| / Solaris/BSD//C/C++ programmer \ |-..-'`
/\/\ /\/\
`--` `--`
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [Fwd: ZFS user/group quotas space accounting [PSARC/2009/204 FastTrack timeout 04/08/2009]]
River Tarnell wrote: Matthew Ahrens: ZFS user quotas (like other zfs properties) will not be accessible over NFS; you must be on the machine running zfs to manipulate them. does this mean that without an account on the NFS server, a user cannot see his current disk use / quota? That's correct. So that's different from ufs with NFS where rquotad and the NFS client code makes sure that you can see the melbourne quota from the UFS server. I know that this is one of the additional protocols developed for NFSv2 and NFSv3; does NFSv4 has a similar mechanism to get the quota? Is there any reason why rquota is not supported? (It's not about manipulating quota; only displaying) Casper ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [Fwd: ZFS user/group quotas space accounting [PSARC/2009/204 FastTrack timeout 04/08/2009]]
casper@sun.com wrote: River Tarnell wrote: Matthew Ahrens: ZFS user quotas (like other zfs properties) will not be accessible over NFS; you must be on the machine running zfs to manipulate them. does this mean that without an account on the NFS server, a user cannot see his current disk use / quota? That's correct. So that's different from ufs with NFS where rquotad and the NFS client code makes sure that you can see the melbourne quota from the UFS server. I know that this is one of the additional protocols developed for NFSv2 and NFSv3; does NFSv4 has a similar mechanism to get the quota? Is there any reason why rquota is not supported? (It's not about manipulating quota; only displaying) If we had the .zfs/props/propname RFE implemented that would allow users to see this regardless of what file sharing protocol they use. As well as lots of other very interesting info about the filesystem. -- Darren J Moffat ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [Fwd: ZFS user/group quotas space accounting [PSARC/2009/204 FastTrack timeout 04/08/2009]]
Hello Matthew, Tuesday, March 31, 2009, 9:16:42 PM, you wrote: MA Robert Milkowski wrote: Hello Matthew, Excellent news. Wouldn't it be better if logical disk usage would be accounted and not physical - I mean when compression is enabled should quota be accounted based by a logical file size or physical as in du? MA ] MA The compressed space *is* the amount of space charged, same as struct stat's MA st_blocks and du(1) (and the referenced property, and the used property, MA etc). I don't think that we ever report the uncompressed size; it's only MA available indirectly by multiplying by the compressratio property. What I mean is: assume user joe have a quota of 1GB. So he creates a file fill in with 1's which is 5GB in size. ls utility will confirm it is 5GB in size while du will say it is 100MB (haven't done the actual tes but you get the idea). So from a user perspective isn't it a little bit confusing as he managed to write more data than he thinks he is allowed to. From a sysdmin perspective Nicilas is probably right that in most cases they would care more about physical usage than logical, or would they? -- Best regards, Robert Milkowski http://milek.blogspot.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [Fwd: ZFS user/group quotas space accounting [PSARC/2009/204 FastTrack timeout 04/08/2009]]
Robert Milkowski wrote: Hello Matthew, Tuesday, March 31, 2009, 9:16:42 PM, you wrote: MA Robert Milkowski wrote: Hello Matthew, Excellent news. Wouldn't it be better if logical disk usage would be accounted and not physical - I mean when compression is enabled should quota be accounted based by a logical file size or physical as in du? MA ] MA The compressed space *is* the amount of space charged, same as struct stat's MA st_blocks and du(1) (and the referenced property, and the used property, MA etc). I don't think that we ever report the uncompressed size; it's only MA available indirectly by multiplying by the compressratio property. What I mean is: assume user joe have a quota of 1GB. So he creates a file fill in with 1's which is 5GB in size. ls utility will confirm it is 5GB in size while du will say it is 100MB (haven't done the actual tes but you get the idea). So from a user perspective isn't it a little bit confusing as he managed to write more data than he thinks he is allowed to. From a sysdmin perspective Nicilas is probably right that in most cases they would care more about physical usage than logical, or would they? I think what this says is that from a practical perspective, quotas are either ineffective or incomprehensible for modern systems. By ineffective I mean that you cannot limit a user's use of space, you can only limit that to which the user is accounted. Perhaps we should change it from quota to goodwill, to borrow a term from the accounting world :-) -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [Fwd: ZFS user/group quotas space accounting [PSARC/2009/204 FastTrack timeout 04/08/2009]]
Robert Milkowski wrote: Hello Matthew, Tuesday, March 31, 2009, 9:16:42 PM, you wrote: MA Robert Milkowski wrote: Hello Matthew, Excellent news. Wouldn't it be better if logical disk usage would be accounted and not physical - I mean when compression is enabled should quota be accounted based by a logical file size or physical as in du? MA ] MA The compressed space *is* the amount of space charged, same as struct stat's MA st_blocks and du(1) (and the referenced property, and the used property, MA etc). I don't think that we ever report the uncompressed size; it's only MA available indirectly by multiplying by the compressratio property. What I mean is: assume user joe have a quota of 1GB. So he creates a file fill in with 1's which is 5GB in size. ls utility will confirm it is 5GB in size while du will say it is 100MB (haven't done the actual tes but you get the idea). So from a user perspective isn't it a little bit confusing as he managed to write more data than he thinks he is allowed to. Pleasant surprises tend to be tolerated :-) --matt ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [Fwd: ZFS user/group quotas space accounting [PSARC/2009/204 FastTrack timeout 04/08/2009]]
Hello Richard, Wednesday, April 1, 2009, 5:32:25 PM, you wrote: RE Robert Milkowski wrote: Hello Matthew, Tuesday, March 31, 2009, 9:16:42 PM, you wrote: MA Robert Milkowski wrote: Hello Matthew, Excellent news. Wouldn't it be better if logical disk usage would be accounted and not physical - I mean when compression is enabled should quota be accounted based by a logical file size or physical as in du? MA ] MA The compressed space *is* the amount of space charged, same as struct stat's MA st_blocks and du(1) (and the referenced property, and the used property, MA etc). I don't think that we ever report the uncompressed size; it's only MA available indirectly by multiplying by the compressratio property. What I mean is: assume user joe have a quota of 1GB. So he creates a file fill in with 1's which is 5GB in size. ls utility will confirm it is 5GB in size while du will say it is 100MB (haven't done the actual tes but you get the idea). So from a user perspective isn't it a little bit confusing as he managed to write more data than he thinks he is allowed to. From a sysdmin perspective Nicilas is probably right that in most cases they would care more about physical usage than logical, or would they? RE I think what this says is that from a practical perspective, quotas are RE either ineffective or incomprehensible for modern systems. By ineffective RE I mean that you cannot limit a user's use of space, you can only limit RE that to which the user is accounted. Perhaps we should change it from RE quota to goodwill, to borrow a term from the accounting world :-) After giving it a little bit more thought I think the current approach is better in most cases. In most cases user could compress a file by using external program anyway and physical disk space is the main issue being tried to be address by user/group quotas. -- Best regards, Robert Milkowski http://milek.blogspot.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [Fwd: ZFS user/group quotas space accounting [PSARC/2009/204 FastTrack timeout 04/08/2009]]
On Wed, Apr 01, 2009 at 10:58:34AM +0200, casper@sun.com wrote: I know that this is one of the additional protocols developed for NFSv2 and NFSv3; does NFSv4 has a similar mechanism to get the quota? Yes, NFSv4.0 and 4.1 both provide the same quota information retrieval interface, three file/directory attributes: - quota_avail_hard - quota_avail_soft - quota_used It's not clear if the values returned for these attributes are supposed to specific to the credentials of the caller or what, but I assume it's the former. I don't know if the Solaris NFSv4 client and server support this feature; the attributes are REQUIRED to implement in v4.1, but I'm not sure if that's also true in v4.0). Nico -- ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [Fwd: ZFS user/group quotas space accounting [PSARC/2009/204 FastTrack timeout 04/08/2009]]
On Wed, Apr 01, 2009 at 10:04:47AM +0100, Darren J Moffat wrote: If we had the .zfs/props/propname RFE implemented that would allow users to see this regardless of what file sharing protocol they use. As well as lots of other very interesting info about the filesystem. Indeed! ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [Fwd: ZFS user/group quotas space accounting [PSARC/2009/204 FastTrack timeout 04/08/2009]]
Mike Gerdts wrote: On Tue, Mar 31, 2009 at 7:12 PM, Matthew Ahrens matthew.ahr...@sun.com wrote: River Tarnell wrote: Matthew Ahrens: ZFS user quotas (like other zfs properties) will not be accessible over NFS; you must be on the machine running zfs to manipulate them. does this mean that without an account on the NFS server, a user cannot see his current disk use / quota? That's correct. Do you have a reason for not wanting this to be implemented, or are you just avoiding scope creep? The latter -- I just don't have time to tack on any more features with this. We've filed RFE 6824968 to add support to rquotad to report on zfs user quotas. I'll note this in the PSARC case as well. This additional RFE is staffed, but we don't have an ETA right now. --matt ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [Fwd: ZFS user/group quotas space accounting [PSARC/2009/204 FastTrack timeout 04/08/2009]]
On Wed, 1 Apr 2009, Matthew Ahrens wrote: So from a user perspective isn't it a little bit confusing as he managed to write more data than he thinks he is allowed to. Pleasant surprises tend to be tolerated :-) Until it comes time to back that data up. It is conceivable for users to create a DOS for the backup system by intentionally creating many huge files which compress perfectly with ZFS, but not so perfectly for backup (assuming that backup even supports compression). Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [Fwd: ZFS user/group quotas space accounting [PSARC/2009/204 FastTrack timeout 04/08/2009]]
Hello Bob, Wednesday, April 1, 2009, 7:14:46 PM, you wrote: BF On Wed, 1 Apr 2009, Matthew Ahrens wrote: So from a user perspective isn't it a little bit confusing as he managed to write more data than he thinks he is allowed to. Pleasant surprises tend to be tolerated :-) BF Until it comes time to back that data up. It is conceivable for users BF to create a DOS for the backup system by intentionally creating many BF huge files which compress perfectly with ZFS, but not so perfectly for BF backup (assuming that backup even supports compression). c'mon - they can do it anyway even with file system with no built-in compression as they can use any external program to compress their data. And if you are really worried about it then do not use compression in zfs. Not to mention that such a case is rather unrealistic (well, maybe except dd if=/dev/zero). -- Best regards, Robert Milkowsk http://milek.blogspot.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [Fwd: ZFS user/group quotas space accounting [PSARC/2009/204 FastTrack timeout 04/08/2009]]
[ re-sending to the list address - stupid thunderbird still doesn't have reply-to-list :-( ] Robert Milkowski wrote: Hello Bob, Wednesday, April 1, 2009, 7:14:46 PM, you wrote: ... BF Until it comes time to back that data up. It is conceivable for users BF to create a DOS for the backup system by intentionally creating many BF huge files which compress perfectly with ZFS, but not so perfectly for BF backup (assuming that backup even supports compression). c'mon - they can do it anyway even with file system with no built-in compression as they can use any external program to compress their data. And if you are really worried about it then do not use compression in zfs. But then the compressed file is backed up. Different case entirely. And don't forget zfs send/recv - sadly they send the uncompressed data, so they'd also suffer from this. Not that I think it's worth counting virtual space, mind you - too much effort for too little benefit, and adding more knobs adds complexity / bugs / cost. But the potential for problems is real - more so if you don't rule out malicious users. -- Carson ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] [Fwd: ZFS user/group quotas space accounting [PSARC/2009/204 FastTrack timeout 04/08/2009]]
FYI, I filed this PSARC case yesterday, and expect to integrate into OpenSolaris in April. Your comments are welcome. http://arc.opensolaris.org/caselog/PSARC/2009/204/ --matt ---BeginMessage--- Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI This information is Copyright 2009 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: ZFS user/group quotas space accounting 1.2. Name of Document Author/Supplier: Author: Matthew Ahrens 1.3 Date of This Document: 30 March, 2009 4. Technical Description ZFS user/group space accounting A. SUMMARY This case adds support to ZFS for user/group quotas per-uid/gid space tracking. B. PROBLEM Enterprise customers often want to know who is using space, based on what uid and gid owns each file. Education customers often want to apply per-user quotas to hundreds of thousands of users. In these situations, the number of users and/or existing infrastructure prohibits using one filesystem per user and setting filesystem-wide quotas. C. PROPOSED SOLUTION 1. Overview Each filesystem keeps track of how much space inside it is owned by each user (uid) and group (gid). This is the amount of space referenced, so relationships between filesystems, descendents, clones, and snapshots are ignored, and each tracks their user used and group used independently. This is the same policy behind the referenced, refquota, and refreservation properties. The amount of space charged is the amount of space reported by struct stat's st_blocks and du(1). Both POSIX ids (uid gid) and untranslated SIDs are supported (eg, when sharing filesystems over SMB without a name service translation set up). ZFS will now enforce quotas on the amount of space referenced by files owned by particular users and groups. Enforcement may be delayed by several seconds. In other words, users may go a bit over their quota before the system notices that they are over quota and begins to refuse additional writes with EDQUOT. This decision was made to get the feature to market in a reasonable time, with a minimum of engineering resources expended. The design and implementation do not preclude implementing strict enforcement at a later date. User space accounting and quotas stick with each dataset (snapshot, filesystem, and clone). This means that user quotas (and space accounting) are not inherited. They will be copied to a new snapshot, and keep the values they had at the time the snapshot was taken. Likewise, user quotas will be copied to a clone (from its origin snapshot), and they will be copied with zfs send (even without -R). (User accounting and quota information is not actually copied to snapshots and clones, just referenced and copied-on-write like other filesystem contents.) The user space accounting and quotas is reported by the new userused@user, groupused@group, userquota@user, and groupquota@group properties, and by the new zfs userspace and zfs groupspace subcommands, which are detailed below. 2. Version Compatibility To use these features, the pool must be upgraded to a new on-disk version (15). Old filesystems must have their space accounting information initialized by running zfs userspace fs or upgrading the old filesystem to a new on-disk version (4). To set user quotas, the pool and filesystem must both be upgraded. 3. Permissions Setting or changing user quotas are administrative actions, subject to the same privilege requirements as other zfs subcommands. There are new userquota and groupquota permissions which can be granted with zfs allow, to allow those properties to be viewed and changed. Unprivileged users can only view their own userquota and userused, and the groupquota and groupused of any groups they belong to. The new userused and groupused permissions can be granted with zfs allow to permit users to view these properties. The existing version permission (granted with zfs allow) permits the accounting information to be initialized by zfs userspace. 4. New Properties user/group space accounting information and quotas can be manipulated with 4 new properties: zfs get userused@user fs|snap zfs get groupused@group fs|snap zfs get userquota@user fs|snap zfs get groupquota@group fs|snap zfs set userquota@user=quota fs zfs set groupquota@user=quota fs The user or group is specified using one of the following forms: posix name (eg. ahrens) posix numeric id (eg. 126829) sid name (eg. ahr...@sun) sid numeric id (eg. S-1-12345-12423-125829) For zfs set, if a nonexistent name is specified, an error is generated. Any numeric ID is permitted. For zfs get, if a nonexistent name is specified, - is printed for the value, indicating that there is no value available (like zfs get nonexistent:userproperty). As with filesystem quotas (quota and refquota properties), user quotas can be set to a value larger than the available space. User quotas can also be set to a value less than the amount of space used
Re: [zfs-discuss] [Fwd: ZFS user/group quotas space accounting [PSARC/2009/204 FastTrack timeout 04/08/2009]]
2009/3/31 Matthew Ahrens matthew.ahr...@sun.com: 4. New Properties user/group space accounting information and quotas can be manipulated with 4 new properties: zfs get userused@user fs|snap zfs get groupused@group fs|snap zfs get userquota@user fs|snap zfs get groupquota@group fs|snap zfs set userquota@user=quota fs zfs set groupquota@user=quota fs The user or group is specified using one of the following forms: posix name (eg. ahrens) posix numeric id (eg. 126829) sid name (eg. ahr...@sun) sid numeric id (eg. S-1-12345-12423-125829) How does this work with zones? Suppose in the global zone I have passwd entries like: jill:x:123:123:Jill Admin:/home/jill:/bin/bash joe:x:124:124:Joe Admin:/home/joe:/bin/bash And in a non-global zone (called bedrock) I have: fred:x:123:123:Fred Flintstone:/home/fred:/bin/bash barney:x:124:124:Barney Rubble:/home/barney:/bin/bash Dataset rpool/quarry is delegated to the zone bedrock. Does zfs get all rpool/quarry report the same thing whether it is run in the global zone or the non-global zone? Has there been any thought to using a UID resolution mechanism similar to that used by ps? That is, if zfs get ... dataset is run in the global zone and the dataset is deleted to a non-global zone, display the UID rather than a possibly mistaken username. -- Mike Gerdts http://mgerdts.blogspot.com/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [Fwd: ZFS user/group quotas space accounting [PSARC/2009/204 FastTrack timeout 04/08/2009]]
Hello Matthew, Excellent news. Wouldn't it be better if logical disk usage would be accounted and not physical - I mean when compression is enabled should quota be accounted based by a logical file size or physical as in du? I'm not saying which one is better just raising the question. -- Best regards, Robert Milkowski http://milek.blogspot.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [Fwd: ZFS user/group quotas space accounting [PSARC/2009/204 FastTrack timeout 04/08/2009]]
much cheering ensues! 2009/3/31 Matthew Ahrens matthew.ahr...@sun.com: FYI, I filed this PSARC case yesterday, and expect to integrate into OpenSolaris in April. Your comments are welcome. http://arc.opensolaris.org/caselog/PSARC/2009/204/ --matt -- Forwarded message -- From: Matthew Ahrens ahr...@dm-eng-01.sfbay.sun.com To: psarc-...@sun.com Date: Mon, 30 Mar 2009 20:39:05 -0700 (PDT) Subject: ZFS user/group quotas space accounting [PSARC/2009/204 FastTrack timeout 04/08/2009] Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI This information is Copyright 2009 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: ZFS user/group quotas space accounting 1.2. Name of Document Author/Supplier: Author: Matthew Ahrens 1.3 Date of This Document: 30 March, 2009 4. Technical Description ZFS user/group space accounting A. SUMMARY This case adds support to ZFS for user/group quotas per-uid/gid space tracking. B. PROBLEM Enterprise customers often want to know who is using space, based on what uid and gid owns each file. Education customers often want to apply per-user quotas to hundreds of thousands of users. In these situations, the number of users and/or existing infrastructure prohibits using one filesystem per user and setting filesystem-wide quotas. C. PROPOSED SOLUTION 1. Overview Each filesystem keeps track of how much space inside it is owned by each user (uid) and group (gid). This is the amount of space referenced, so relationships between filesystems, descendents, clones, and snapshots are ignored, and each tracks their user used and group used independently. This is the same policy behind the referenced, refquota, and refreservation properties. The amount of space charged is the amount of space reported by struct stat's st_blocks and du(1). Both POSIX ids (uid gid) and untranslated SIDs are supported (eg, when sharing filesystems over SMB without a name service translation set up). ZFS will now enforce quotas on the amount of space referenced by files owned by particular users and groups. Enforcement may be delayed by several seconds. In other words, users may go a bit over their quota before the system notices that they are over quota and begins to refuse additional writes with EDQUOT. This decision was made to get the feature to market in a reasonable time, with a minimum of engineering resources expended. The design and implementation do not preclude implementing strict enforcement at a later date. User space accounting and quotas stick with each dataset (snapshot, filesystem, and clone). This means that user quotas (and space accounting) are not inherited. They will be copied to a new snapshot, and keep the values they had at the time the snapshot was taken. Likewise, user quotas will be copied to a clone (from its origin snapshot), and they will be copied with zfs send (even without -R). (User accounting and quota information is not actually copied to snapshots and clones, just referenced and copied-on-write like other filesystem contents.) The user space accounting and quotas is reported by the new userused@user, groupused@group, userquota@user, and groupquota@group properties, and by the new zfs userspace and zfs groupspace subcommands, which are detailed below. 2. Version Compatibility To use these features, the pool must be upgraded to a new on-disk version (15). Old filesystems must have their space accounting information initialized by running zfs userspace fs or upgrading the old filesystem to a new on-disk version (4). To set user quotas, the pool and filesystem must both be upgraded. 3. Permissions Setting or changing user quotas are administrative actions, subject to the same privilege requirements as other zfs subcommands. There are new userquota and groupquota permissions which can be granted with zfs allow, to allow those properties to be viewed and changed. Unprivileged users can only view their own userquota and userused, and the groupquota and groupused of any groups they belong to. The new userused and groupused permissions can be granted with zfs allow to permit users to view these properties. The existing version permission (granted with zfs allow) permits the accounting information to be initialized by zfs userspace. 4. New Properties user/group space accounting information and quotas can be manipulated with 4 new properties: zfs get userused@user fs|snap zfs get groupused@group fs|snap zfs get userquota@user fs|snap zfs get groupquota@group fs|snap zfs set userquota@user=quota fs zfs set groupquota@user=quota fs The user or group is specified using one of the following forms: posix name (eg. ahrens) posix numeric id (eg. 126829) sid name (eg. ahr...@sun) sid numeric id (eg. S-1-12345-12423-125829) For zfs set, if a nonexistent name is specified, an error is
Re: [zfs-discuss] [Fwd: ZFS user/group quotas space accounting [PSARC/2009/204 FastTrack timeout 04/08/2009]]
On Tue, Mar 31, 2009 at 02:37:02PM -0500, Mike Gerdts wrote:
The user or group is specified using one of the following forms:
posix name (eg. ahrens)
posix numeric id (eg. 126829)
sid name (eg. ahr...@sun)
sid numeric id (eg. S-1-12345-12423-125829)
How does this work with zones? Suppose in the global zone I have
passwd entries like:
ZFS stores UIDs, GIDs and SIDs on disk. The POSIX ID/SID - name
resolution happens in user-land. As such the answer to your question is
the same as for any other operating system facility that deals with
POSIX ID/SID - name resolution: it depends on each zone's
configuration.
(In general kernel code never deals with user/group names directly, but
with UIDs/GIDs/SIDs. One exception is the NFSv4 code, which upcalls to
user-land to resolve NFSv4 n...@domain user/group names and vice versa.)
jill:x:123:123:Jill Admin:/home/jill:/bin/bash
joe:x:124:124:Joe Admin:/home/joe:/bin/bash
And in a non-global zone (called bedrock) I have:
fred:x:123:123:Fred Flintstone:/home/fred:/bin/bash
barney:x:124:124:Barney Rubble:/home/barney:/bin/bash
Dataset rpool/quarry is delegated to the zone bedrock.
Does zfs get all rpool/quarry report the same thing whether it is
run in the global zone or the non-global zone?
If you use the -n option, yes :)
Oh, but then, the -n option is for the new zfs {user|group}space
sub-command. I don't think zfs get is getting that option; maybe it
should.
Has there been any thought to using a UID resolution mechanism similar
to that used by ps? That is, if zfs get ... dataset is run in the
global zone and the dataset is deleted to a non-global zone, display
the UID rather than a possibly mistaken username.
That seems like a good idea to me. You should send that comment to the
ARC case record (send an e-mail to psarc-...@sun.com with
PSARC/2009/204 somewhere in the Subject: header).
Nico
--
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [Fwd: ZFS user/group quotas space accounting [PSARC/2009/204 FastTrack timeout 04/08/2009]]
Robert Milkowski wrote: Hello Matthew, Excellent news. Wouldn't it be better if logical disk usage would be accounted and not physical - I mean when compression is enabled should quota be accounted based by a logical file size or physical as in du? ] The compressed space *is* the amount of space charged, same as struct stat's st_blocks and du(1) (and the referenced property, and the used property, etc). I don't think that we ever report the uncompressed size; it's only available indirectly by multiplying by the compressratio property. --matt ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [Fwd: ZFS user/group quotas space accounting [PSARC/2009/204 FastTrack timeout 04/08/2009]]
On Tue, Mar 31, 2009 at 01:16:42PM -0700, Matthew Ahrens wrote: Robert Milkowski wrote: Hello Matthew, Excellent news. Wouldn't it be better if logical disk usage would be accounted and not physical - I mean when compression is enabled should quota be accounted based by a logical file size or physical as in du? ] The compressed space *is* the amount of space charged, same as struct stat's st_blocks and du(1) (and the referenced property, and the used property, etc). I think sysadmins will generally care more about physical space consumption than uncompressed space consumption. I don't think that we ever report the uncompressed size; it's only available indirectly by multiplying by the compressratio property. But compressratio is a dataset-wide property -- it seems wrong to assume that it will be the same for each user/group's data inside a given dataset. A fast way to obtain the actual uncompressed data size might be useful, but I also think it should not be a priority if indeed sysadmins care [or ought to care!] more about physical space consumption than uncompressed space consumption. Nico -- ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [Fwd: ZFS user/group quotas space accounting [PSARC/2009/204 FastTrack timeout 04/08/2009]]
Nicolas Williams wrote:
On Tue, Mar 31, 2009 at 02:37:02PM -0500, Mike Gerdts wrote:
The user or group is specified using one of the following forms:
posix name (eg. ahrens)
posix numeric id (eg. 126829)
sid name (eg. ahr...@sun)
sid numeric id (eg. S-1-12345-12423-125829)
How does this work with zones? Suppose in the global zone I have
passwd entries like:
ZFS stores UIDs, GIDs and SIDs on disk. The POSIX ID/SID - name
resolution happens in user-land. As such the answer to your question is
the same as for any other operating system facility that deals with
POSIX ID/SID - name resolution: it depends on each zone's
configuration.
Right, so you'll get the same answer as if you did a ls on those files (if
the local zone's files were available to the global zone).
(In general kernel code never deals with user/group names directly, but
with UIDs/GIDs/SIDs. One exception is the NFSv4 code, which upcalls to
user-land to resolve NFSv4 n...@domain user/group names and vice versa.)
jill:x:123:123:Jill Admin:/home/jill:/bin/bash
joe:x:124:124:Joe Admin:/home/joe:/bin/bash
And in a non-global zone (called bedrock) I have:
fred:x:123:123:Fred Flintstone:/home/fred:/bin/bash
barney:x:124:124:Barney Rubble:/home/barney:/bin/bash
Dataset rpool/quarry is delegated to the zone bedrock.
Does zfs get all rpool/quarry report the same thing whether it is
run in the global zone or the non-global zone?
If you use the -n option, yes :)
Oh, but then, the -n option is for the new zfs {user|group}space
sub-command. I don't think zfs get is getting that option; maybe it
should.
Since zfs get all doesn't report the {user|group}{used|quo...@...
properties, you will definitely get the same answer ;-)
quote case materials
These new properties are not printed by zfs get all, since that could
generate a huge amount of output, which would not be very well
organized. The new zfs userspace subcommand should be used instead.
Has there been any thought to using a UID resolution mechanism similar
to that used by ps? That is, if zfs get ... dataset is run in the
global zone and the dataset is deleted to a non-global zone, display
the UID rather than a possibly mistaken username.
That seems like a good idea to me. You should send that comment to the
ARC case record (send an e-mail to psarc-...@sun.com with
PSARC/2009/204 somewhere in the Subject: header).
Indeed, that might be a good idea. I wasn't aware of that convention.
Though note that this only applies to zfs userspace -- we would simply say
that if the fs is zoned, that implies the -n option. We could also disallow
them from doing zfs get useru...@name pool/zoned/fs, just make it an error
to prevent them from seeing something other than what they intended.
--matt
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [Fwd: ZFS user/group quotas space accounting [PSARC/2009/204 FastTrack timeout 04/08/2009]]
On Tue, Mar 31, 2009 at 01:25:35PM -0700, Matthew Ahrens wrote: quote case materials These new properties are not printed by zfs get all, since that could generate a huge amount of output, which would not be very well organized. The new zfs userspace subcommand should be used instead. Ah, I missed that. Has there been any thought to using a UID resolution mechanism similar to that used by ps? That is, if zfs get ... dataset is run in the global zone and the dataset is deleted to a non-global zone, display the UID rather than a possibly mistaken username. That seems like a good idea to me. You should send that comment to the ARC case record (send an e-mail to psarc-...@sun.com with PSARC/2009/204 somewhere in the Subject: header). Indeed, that might be a good idea. I wasn't aware of that convention. Though note that this only applies to zfs userspace -- we would simply say that if the fs is zoned, that implies the -n option. We could also disallow them from doing zfs get useru...@name pool/zoned/fs, just make it an error to prevent them from seeing something other than what they intended. I don't see why the g-z admin should not get this data. Having to zlogin to get it seems wrong to me. Though, obviously, you'd have to zlogin to get zone-local _names_ -- one might want extended getXbyYs to do ID-zone-local name resolution in the g-z, though one then worries about ngz admins attacking the g-z admin through user/group names that are not properly encoded for the g-z admin's locale... (so we're likely to never do this). Nico -- ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [Fwd: ZFS user/group quotas space accounting [PSARC/2009/204 FastTrack timeout 04/08/2009]]
Nicolas Williams wrote: We could also disallow them from doing zfs get useru...@name pool/zoned/fs, just make it an error to prevent them from seeing something other than what they intended. I don't see why the g-z admin should not get this data. They can of course still get the data by doing: zfs get userused@numeric-id pool/zoned/fs or zfs userspace pool/zoned/fs, which will imply the -n flag. --matt ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [Fwd: ZFS user/group quotas space accounting [PSARC/2009/204 FastTrack timeout 04/08/2009]]
On 31 March, 2009 - Matthew Ahrens sent me these 10K bytes:
FYI, I filed this PSARC case yesterday, and expect to integrate into
OpenSolaris in April. Your comments are welcome.
http://arc.opensolaris.org/caselog/PSARC/2009/204/
Quota reporting over NFS or for userland apps like Samba? Using the old
school rquotad, something else or not at all? v3+v4?
Did i understand things right; If I assign 4gb quota to user X, and me
as sysadm wants to take extra snapshots to protect from mistakes or
speed up backup restores, that snapshot space is not counted into the
4gb quota? That is, is it like 'zfs set quota=N blah/X' or
'zfs set refquota=N blah/X' ?
Other than that, great news! The original idea of use filesystems
instead, they are cheap was good in theory.. unfortunately, it didn't
scale as well..
/Tomas - wants to get rid of the last UFS's
--
Tomas Ögren, st...@acc.umu.se, http://www.acc.umu.se/~stric/
|- Student at Computing Science, University of Umeå
`- Sysadmin at {cs,acc}.umu.se
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [Fwd: ZFS user/group quotas space accounting [PSARC/2009/204 FastTrack timeout 04/08/2009]]
Tomas Ögren wrote: On 31 March, 2009 - Matthew Ahrens sent me these 10K bytes: FYI, I filed this PSARC case yesterday, and expect to integrate into OpenSolaris in April. Your comments are welcome. http://arc.opensolaris.org/caselog/PSARC/2009/204/ Quota reporting over NFS or for userland apps like Samba? Using the old school rquotad, something else or not at all? v3+v4? ZFS user quotas (like other zfs properties) will not be accessible over NFS; you must be on the machine running zfs to manipulate them. Of course, like other zfs quotas, the user quotas apply to files created/written over NFS or SMB or locally. Did i understand things right; If I assign 4gb quota to user X, and me as sysadm wants to take extra snapshots to protect from mistakes or speed up backup restores, that snapshot space is not counted into the 4gb quota? That is, is it like 'zfs set quota=N blah/X' or 'zfs set refquota=N blah/X' ? It's like refquota. --matt ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [Fwd: ZFS user/group quotas space accounting [PSARC/2009/204 FastTrack timeout 04/08/2009]]
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Matthew Ahrens: ZFS user quotas (like other zfs properties) will not be accessible over NFS; you must be on the machine running zfs to manipulate them. does this mean that without an account on the NFS server, a user cannot see his current disk use / quota? - river. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (HP-UX) iEYEARECAAYFAknSrxMACgkQIXd7fCuc5vKESgCfbJ6pRIZM9imv7qLc1+et1GOf KvkAn2uA2QB+Qbg3HGcDIHIXf4bFsQrc =kVc1 -END PGP SIGNATURE- ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [Fwd: ZFS user/group quotas space accounting [PSARC/2009/204 FastTrack timeout 04/08/2009]]
River Tarnell wrote: Matthew Ahrens: ZFS user quotas (like other zfs properties) will not be accessible over NFS; you must be on the machine running zfs to manipulate them. does this mean that without an account on the NFS server, a user cannot see his current disk use / quota? That's correct. --matt ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [Fwd: ZFS user/group quotas space accounting [PSARC/2009/204 FastTrack timeout 04/08/2009]]
On Tue, Mar 31, 2009 at 7:12 PM, Matthew Ahrens matthew.ahr...@sun.com wrote: River Tarnell wrote: Matthew Ahrens: ZFS user quotas (like other zfs properties) will not be accessible over NFS; you must be on the machine running zfs to manipulate them. does this mean that without an account on the NFS server, a user cannot see his current disk use / quota? That's correct. Do you have a reason for not wanting this to be implemented, or are you just avoiding scope creep? In the past, this was a big pain point for NFS servers that used VxFS. I used one of Sun's source available programs to get the rquotad source to implement this in the Solaris 7 days. Google suggests others have done the same using the opensolaris code as a starting point. Still others have written wrappers around quota(1M) that invoke rsh or ssh to the appropriate NFS server. It seems as though this was eventually addressed by Veritas with 110434-02. We really shouldn't repeat this for long. It should be fairly straight-forward to modify rquotad to support this, so long as the zfs end of it is not overly complicated. Is now too early to file the RFE? For some reason it feels like the person on the other end of bugs.opensolars.org will get confused by the request to enhance a feature that doesn't yet exist. -- Mike Gerdts http://mgerdts.blogspot.com/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [Fwd: ZFS user/group quotas space accounting [PSARC/2009/204 FastTrack timeout 04/08/2009]]
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Matthew Ahrens: does this mean that without an account on the NFS server, a user cannot see his current disk use / quota? That's correct. in this case, might i suggest at least an RFE to add ZFS quota support to rquotad? i'm sure we aren't the only site where non-privileged users don't have login rights to the NFS server, and it would be easier not to have to reimplement rquota locally to support ZFS. - river. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (HP-UX) iEYEARECAAYFAknSx6oACgkQIXd7fCuc5vLTMgCdG9xW159lMD5NLLf8VUpJZud9 CLYAoKux7H/88gsJaCtroCd3IEnwH/ge =Evew -END PGP SIGNATURE- ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
