Re: [zfs-discuss] Issue with simultaneous IO to lots of ZFS pools

2008-05-05 Thread Chris Siebenmann
[Jeff Bonwick:] | That said, I suspect I know the reason for the particular problem | you're seeing: we currently do a bit too much vdev-level caching. | Each vdev can have up to 10MB of cache. With 132 pools, even if | each pool is just a single iSCSI device, that's 1.32GB of cache. | | We need

Re: [zfs-discuss] Issue with simultaneous IO to lots of ZFS pools

2008-05-01 Thread Darren J Moffat
Chris Siebenmann wrote: | Still, I'm curious -- why lots of pools? Administration would be | simpler with a single pool containing many filesystems. The short answer is that it is politically and administratively easier to use (at least) one pool per storage-buying group in our

Re: [zfs-discuss] Issue with simultaneous IO to lots of ZFS pools

2008-05-01 Thread David Collier-Brown
Darren J Moffat [EMAIL PROTECTED] wrote: Chris Siebenmann wrote: | Still, I'm curious -- why lots of pools? Administration would be | simpler with a single pool containing many filesystems. The short answer is that it is politically and administratively easier to use (at least) one pool per

Re: [zfs-discuss] Issue with simultaneous IO to lots of ZFS pools

2008-05-01 Thread Chris Siebenmann
| I think the root cause of the issue is that multiple groups are buying | physical rather than virtual storage yet it is all being attached to a | single system. They're actually buying constant-sized chunks of virtual storage, which is provided through a pool of SAN-based disk space. This

Re: [zfs-discuss] Issue with simultaneous IO to lots of ZFS pools

2008-05-01 Thread Richard Elling
David Collier-Brown wrote: Darren J Moffat [EMAIL PROTECTED] wrote: Chris Siebenmann wrote: | Still, I'm curious -- why lots of pools? Administration would be | simpler with a single pool containing many filesystems. The short answer is that it is politically and administratively

Re: [zfs-discuss] Issue with simultaneous IO to lots of ZFS pools

2008-05-01 Thread Chris Siebenmann
| There are two issues here. One is the number of pools, but the other | is the small amount of RAM in the server. To be honest, most laptops | today come with 2 GBytes, and most servers are in the 8-16 GByte range | (hmmm... I suppose I could look up the average size we sell...) Speaking as a

Re: [zfs-discuss] Issue with simultaneous IO to lots of ZFS pools

2008-05-01 Thread Bart Smaalders
Chris Siebenmann wrote: | There are two issues here. One is the number of pools, but the other | is the small amount of RAM in the server. To be honest, most laptops | today come with 2 GBytes, and most servers are in the 8-16 GByte range | (hmmm... I suppose I could look up the average size

Re: [zfs-discuss] Issue with simultaneous IO to lots of ZFS pools

2008-05-01 Thread David Collier-Brown
Chris Siebenmann [EMAIL PROTECTED] wrote: | Speaking as a sysadmin (and a Sun customer), why on earth would I have | to provision 8 GB+ of RAM on my NFS fileservers? I would much rather | have that memory in the NFS client machines, where it can actually be | put to work by user programs. | | (If

Re: [zfs-discuss] Issue with simultaneous IO to lots of ZFS pools

2008-05-01 Thread Richard Elling
Bart Smaalders wrote: Chris Siebenmann wrote: | There are two issues here. One is the number of pools, but the other | is the small amount of RAM in the server. To be honest, most laptops | today come with 2 GBytes, and most servers are in the 8-16 GByte range | (hmmm... I suppose I

[zfs-discuss] Issue with simultaneous IO to lots of ZFS pools

2008-04-30 Thread Chris Siebenmann
I have a test system with 132 (small) ZFS pools[*], as part of our work to validate a new ZFS-based fileserver environment. In testing, it appears that we can produce situations that will run the kernel out of memory, or at least out of some resource such that things start complaining 'bash:

Re: [zfs-discuss] Issue with simultaneous IO to lots of ZFS pools

2008-04-30 Thread Bill Moore
A silly question: Why are you using 132 ZFS pools as opposed to a single ZFS pool with 132 ZFS filesystems? --Bill On Wed, Apr 30, 2008 at 01:53:32PM -0400, Chris Siebenmann wrote: I have a test system with 132 (small) ZFS pools[*], as part of our work to validate a new ZFS-based fileserver

Re: [zfs-discuss] Issue with simultaneous IO to lots of ZFS pools

2008-04-30 Thread Jeff Bonwick
Indeed, things should be simpler with fewer (generally one) pool. That said, I suspect I know the reason for the particular problem you're seeing: we currently do a bit too much vdev-level caching. Each vdev can have up to 10MB of cache. With 132 pools, even if each pool is just a single iSCSI

Re: [zfs-discuss] Issue with simultaneous IO to lots of ZFS pools

2008-04-30 Thread Chris Siebenmann
| Still, I'm curious -- why lots of pools? Administration would be | simpler with a single pool containing many filesystems. The short answer is that it is politically and administratively easier to use (at least) one pool per storage-buying group in our environment. This got discussed in more