Re: [zfs-discuss] Compression
After additional digging and investigation, looks like it's showing me the compressed size, which is good. I've regroomed the storage pools and started moving things back, and I'm seeing files deflate back to their expected size. What really tipped me off was when I decided to log in to one of the VM's. VMware showed 120GB provisioned, 92GB allocated on the standard block storage. After storage vmotioning back to NFS, it showed 62GB allocated. The key was when I logged in to the machine, Windows said it was using 90GB. All said and done, it was a good adventure, and I got the info that I needed. Thanks to all that took the time to reply. -Matt Breitbach -Original Message- From: Donal Farrell [mailto:vmlinuz...@gmail.com] Sent: Wednesday, November 23, 2011 10:42 AM To: Matt Breitbach Subject: Re: [zfs-discuss] Compression is this on esx 3.5.x? or 4.x or greater? The reason i ask is that svmotion to and from nfs datastores did not work correctly in esx 3.5.x Also can you past the output of cat /vmfs/volumes/datastore/vmname.vmdk here? On Wed, Nov 23, 2011 at 1:55 PM, Matt Breitbach matth...@flash.shanje.com wrote: Currently using NFS to access the datastore. -Matt -Original Message- From: Richard Elling [mailto:richard.ell...@gmail.com] Sent: Tuesday, November 22, 2011 11:10 PM To: Matt Breitbach Cc: zfs-discuss@opensolaris.org Subject: Re: [zfs-discuss] Compression Hi Matt, On Nov 22, 2011, at 7:39 PM, Matt Breitbach wrote: So I'm looking at files on my ZFS volume that are compressed, and I'm wondering to myself, self, are the values shown here the size on disk, or are they the pre-compressed values. Google gives me no great results on the first few pages, so I headed here. This really relates to my VMware environment. I had some things happen on my platform that required me to Storage Vmotion everything off of a particular zpool. When I did that, I saw most VM's inflate to nearly their thick provisioned size. What didn't swell to that size went to about 2/3 provisioned (non-Nexenta storage). I have been seeing 1.3-1.5x compression ratios on pretty much everything I turn compression on for (these are general use VM's - webservers,SQL,firewall,etc). My question is this - when I'm looking in the file structure, or in the datastore browser in VMware, am I seeing the uncompressed file size, or the compressed filesize? My gut tells me that since they inflated _so_ badly when I storage vmotioned them, that they are the compressed values, but I would love to know for sure. How are you measuring the space? Are you using block (iscsi/fc) or NFS to access the datastores from ESXi? -- richard -- ZFS and performance consulting http://www.RichardElling.com LISA '11, Boston, MA, December 4-9 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Compression
Currently using NFS to access the datastore. -Matt -Original Message- From: Richard Elling [mailto:richard.ell...@gmail.com] Sent: Tuesday, November 22, 2011 11:10 PM To: Matt Breitbach Cc: zfs-discuss@opensolaris.org Subject: Re: [zfs-discuss] Compression Hi Matt, On Nov 22, 2011, at 7:39 PM, Matt Breitbach wrote: So I'm looking at files on my ZFS volume that are compressed, and I'm wondering to myself, self, are the values shown here the size on disk, or are they the pre-compressed values. Google gives me no great results on the first few pages, so I headed here. This really relates to my VMware environment. I had some things happen on my platform that required me to Storage Vmotion everything off of a particular zpool. When I did that, I saw most VM's inflate to nearly their thick provisioned size. What didn't swell to that size went to about 2/3 provisioned (non-Nexenta storage). I have been seeing 1.3-1.5x compression ratios on pretty much everything I turn compression on for (these are general use VM's - webservers,SQL,firewall,etc). My question is this - when I'm looking in the file structure, or in the datastore browser in VMware, am I seeing the uncompressed file size, or the compressed filesize? My gut tells me that since they inflated _so_ badly when I storage vmotioned them, that they are the compressed values, but I would love to know for sure. How are you measuring the space? Are you using block (iscsi/fc) or NFS to access the datastores from ESXi? -- richard -- ZFS and performance consulting http://www.RichardElling.com LISA '11, Boston, MA, December 4-9 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Compression
So I'm looking at files on my ZFS volume that are compressed, and I'm wondering to myself, self, are the values shown here the size on disk, or are they the pre-compressed values. Google gives me no great results on the first few pages, so I headed here. This really relates to my VMware environment. I had some things happen on my platform that required me to Storage Vmotion everything off of a particular zpool. When I did that, I saw most VM's inflate to nearly their thick provisioned size. What didn't swell to that size went to about 2/3 provisioned (non-Nexenta storage). I have been seeing 1.3-1.5x compression ratios on pretty much everything I turn compression on for (these are general use VM's - webservers,SQL,firewall,etc). My question is this - when I'm looking in the file structure, or in the datastore browser in VMware, am I seeing the uncompressed file size, or the compressed filesize? My gut tells me that since they inflated _so_ badly when I storage vmotioned them, that they are the compressed values, but I would love to know for sure. -Matt Breitbach ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Compression
2011-11-23 7:39, Matt Breitbach wrote: So I'm looking at files on my ZFS volume that are compressed, and I'm wondering to myself, self, are the values shown here the size on disk, or are they the pre-compressed values. Google gives me no great results on the first few pages, so I headed here. Alas, I can't give a good hint about VMWare - which values it uses. But here are some numbers it might see (likely du or ls sizes are in play): Locally on a ZFS-enabled system you can use ls to normally list your files. This would show you the logical POSIX file size, including any referenced-but-not-allocated sparse blocks (logical size = big, physical size = zero), etc. Basically, this just gives a range of byte numbers that you can address in the file, and depending on the underlying FS all or not all of these bytes are backed by physical storage 1:1. If you use du on the ZFS filesystem, you'll see the logical storage size, which takes into account compression and sparse bytes. So the du size should be not greater than ls size. Harder to see would be the physical allocation size, which refers to your data pool's redundancy (raidzN level, copies=N and so on). But you can more or less calculate that from du size. Also your files on ZFS indirectly consume space by requiring some metadata blocks (usually one per data block, and usually they are comparatively small) which is pool metadata and does not show up easily as file size as well. If you're too interested, you might search for howtos on zdb command usage to debug your ZFS pool and gather stats. If your new storage system does support some sort of compression, at least for contiguous ranges of zero-bytes, you might write some large zero-filled files into your VM's filesystems. This should empty the blocks previously used by files now deleted in the VM and may give temporary space savings. This really relates to my VMware environment. I had some things happen on my platform that required me to Storage Vmotion everything off of a particular zpool. When I did that, I saw most VM's inflate to nearly their thick provisioned size. What didn't swell to that size went to about 2/3 provisioned (non-Nexenta storage). I have been seeing 1.3-1.5x compression ratios on pretty much everything I turn compression on for (these are general use VM's - webservers,SQL,firewall,etc). My question is this - when I'm looking in the file structure, or in the datastore browser in VMware, am I seeing the uncompressed file size, or the compressed filesize? My gut tells me that since they inflated _so_ badly when I storage vmotioned them, that they are the compressed values, but I would love to know for sure. -Matt Breitbach HTH, //Jim Klimov ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Compression
On 11/23/11 04:58 PM, Jim Klimov wrote: 2011-11-23 7:39, Matt Breitbach wrote: So I'm looking at files on my ZFS volume that are compressed, and I'm wondering to myself, self, are the values shown here the size on disk, or are they the pre-compressed values. Google gives me no great results on the first few pages, so I headed here. Alas, I can't give a good hint about VMWare - which values it uses. But here are some numbers it might see (likely du or ls sizes are in play): Locally on a ZFS-enabled system you can use ls to normally list your files. This would show you the logical POSIX file size, including any referenced-but-not-allocated sparse blocks (logical size = big, physical size = zero), etc. Basically, this just gives a range of byte numbers that you can address in the file, and depending on the underlying FS all or not all of these bytes are backed by physical storage 1:1. If you use du on the ZFS filesystem, you'll see the logical storage size, which takes into account compression and sparse bytes. So the du size should be not greater than ls size. It can be significantly bigger: ls -sh x 2 x du -sh x 1Kx -- Ian. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Compression
2011-11-23 8:21, Ian Collins wrote: If you use du on the ZFS filesystem, you'll see the logical storage size, which takes into account compression and sparse bytes. So the du size should be not greater than ls size. It can be significantly bigger: ls -sh x 2 x du -sh x 1K x Pun accepted ;) Ian is right, that du size also reflects block-size usage, and that's how many bytes are actually used on the FS (over redundancy layer). Even if your files are smaller than a single block, that's the minimum they will bite off your disk anyway. However, the original question was about VM datastores, so large files were assumed. //Jim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Compression
Hi Matt, On Nov 22, 2011, at 7:39 PM, Matt Breitbach wrote: So I'm looking at files on my ZFS volume that are compressed, and I'm wondering to myself, self, are the values shown here the size on disk, or are they the pre-compressed values. Google gives me no great results on the first few pages, so I headed here. This really relates to my VMware environment. I had some things happen on my platform that required me to Storage Vmotion everything off of a particular zpool. When I did that, I saw most VM's inflate to nearly their thick provisioned size. What didn't swell to that size went to about 2/3 provisioned (non-Nexenta storage). I have been seeing 1.3-1.5x compression ratios on pretty much everything I turn compression on for (these are general use VM's - webservers,SQL,firewall,etc). My question is this - when I'm looking in the file structure, or in the datastore browser in VMware, am I seeing the uncompressed file size, or the compressed filesize? My gut tells me that since they inflated _so_ badly when I storage vmotioned them, that they are the compressed values, but I would love to know for sure. How are you measuring the space? Are you using block (iscsi/fc) or NFS to access the datastores from ESXi? -- richard -- ZFS and performance consulting http://www.RichardElling.com LISA '11, Boston, MA, December 4-9 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Compression block sizes
On Wed, 15 Sep 2010, Brandon High wrote: When using compression, are the on-disk record sizes determined before or after compression is applied? In other words, if record size is set to 128k, is that the amount of data fed into the compression engine, or is the output size trimmed to fit? I think it's the former, but I'm not certain. We have been told before that the blocksize is applied to the uncompressed data and that when compression is applied, short blocks may be written to disk. This does not mean that the short blocks don't start at a particular alignment. When using raidz, the zfs blocks are already broken up into smaller chunks, using a smaller alignment than the zfs record size. For zfs send, the data is uncompressed to full records prior to sending. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Compression block sizes
When using compression, are the on-disk record sizes determined before or after compression is applied? In other words, if record size is set to 128k, is that the amount of data fed into the compression engine, or is the output size trimmed to fit? I think it's the former, but I'm not certain. This could affect block alignment for SSDs, leading to worse write performance. It could also have a negative effect on 4k sector disks that lie about the sector size, such as the recent WD EARS drives. -B -- Brandon High : bh...@freaks.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression property not received
On Wed, Apr 7, 2010 at 10:47 AM, Daniel Bakken dan...@economicmodeling.com wrote: When I send a filesystem with compression=gzip to another server with compression=on, compression=gzip is not set on the received filesystem. I am using: Is compression set on the dataset, or is it being inherited from a parent dataset? I think only locally set properties are preserved. -B -- Brandon High : bh...@freaks.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression property not received
Hi Daniel, D'oh... I found a related bug when I looked at this yesterday but I didn't think it was your problem because you didn't get a busy message. See this RFE: http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6700597 Cindy On 04/07/10 17:59, Daniel Bakken wrote: We have found the problem. The mountpoint property on the sender was at one time changed from the default, then later changed back to defaults using zfs set instead of zfs inherit. Therefore, zfs send included these local non-default properties in the stream, even though the local properties are effectively set at defaults. This caused the receiver to stop processing subsequent properties in the stream because the mountpoint isn't valid on the receiver. I tested this theory with a spare zpool. First I used zfs inherit mountpoint promise1/archive to remove the local setting (which was exactly the same value as the default). This time the compression=gzip property was correctly received. It seems like a bug to me that one failed property in a stream prevents the rest from being applied. I should have used zfs inherit, but it would be best if zfs receive handled failures more gracefully, and attempted to set as many properties as possible. Thanks to Cindy and Tom for their help. Daniel On Wed, Apr 7, 2010 at 2:31 AM, Tom Erickson thomas.erick...@oracle.com mailto:thomas.erick...@oracle.com wrote: Now I remember that 'zfs receive' used to give up after the first property it failed to set. If I'm remembering correctly, then, in this case, if the mountpoint was invalid on the receive side, 'zfs receive' would not even try to set the remaining properties. I'd try the following in the source dataset: zfs inherit mountpoint promise1/archive to clear the explicit mountpoint and prevent it from being included in the send stream. Later set it back the way it was. (Soon there will be an option to take care of that; see CR 6883722 want 'zfs recv -o prop=value' to set initial property values of received dataset.) Then see if you receive the compression property successfully. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression property not received
On 08 April, 2010 - Cindy Swearingen sent me these 2,6K bytes:
Hi Daniel,
D'oh...
I found a related bug when I looked at this yesterday but I didn't think
it was your problem because you didn't get a busy message.
See this RFE:
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6700597
Solaris 10 'man zfs', under 'receive':
-uFile system that is associated with the received
stream is not mounted.
/Tomas
--
Tomas Ögren, st...@acc.umu.se, http://www.acc.umu.se/~stric/
|- Student at Computing Science, University of Umeå
`- Sysadmin at {cs,acc}.umu.se
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression property not received
Daniel, Which Solaris release is this? I can't reproduce this on my lab system that runs the Solaris 10 10/09 release. See the output below. Thanks, Cindy # zfs destroy -r tank/test # zfs create -o compression=gzip tank/test # zfs snapshot tank/t...@now # zfs send -R tank/t...@now | zfs receive -vd rpool receiving full stream of tank/t...@now into rpool/t...@now received 249KB stream in 2 seconds (125KB/sec) # zfs list -r rpool NAMEUSED AVAIL REFER MOUNTPOINT rpool 39.4G 27.5G 47.1M /rpool rpool/ROOT 4.89G 27.5G21K legacy rpool/ROOT/s10s_u8wos_08a 4.89G 27.5G 4.89G / rpool/dump 1.50G 27.5G 1.50G - rpool/export 44K 27.5G23K /export rpool/export/home21K 27.5G21K /export/home rpool/snaps31.0G 27.5G 31.0G /rpool/snaps rpool/swap2G 29.5G16K - rpool/test 21K 27.5G21K /rpool/test rpool/t...@now 0 -21K - # zfs get compression rpool/test NAMEPROPERTY VALUE SOURCE rpool/test compression gzip local On 04/07/10 11:47, Daniel Bakken wrote: When I send a filesystem with compression=gzip to another server with compression=on, compression=gzip is not set on the received filesystem. I am using: zfs send -R promise1/arch...@daily.1 | zfs receive -vd sas The zfs manpage says regarding the -R flag: When received, all properties, snapshots, descendent file systems, and clones are preserved. Snapshots are preserved, but the compression property is not. Any ideas why this doesn't work as advertised? Thanks, Daniel Bakken ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression property not received
I worked around the problem by first creating a filesystem of the same name with compression=gzip on the target server. Like this: zfs create sas/archive zfs set compression=gzip sas/archive Then I used zfs receive with the -F option: zfs send -vR promise1/arch...@daily.1 | zfs send zfs receive -vFd sas And now I have gzip compression enabled locally: zfs get compression sas/archive NAME PROPERTY VALUE SOURCE sas/archive compression gzip local Not pretty, but it works. Daniel Bakken On Wed, Apr 7, 2010 at 12:51 PM, Cindy Swearingen cindy.swearin...@oracle.com wrote: Hi Daniel, I tried to reproduce this by sending from a b130 system to a s10u9 system, which vary in pool versions, but this shouldn't matter. I've been sending/receiving streams between latest build systems and older s10 systems for a long time. The zfs send -R option to send a recursive snapshot and all properties integrated into b77 so that isn't your problem either. The above works as expected. See below. I also couldn't find any recent bugs related to this, but bug searching is not an exact science. Mystified as well... Cindy ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression property not received
The receive side is running build 111b (2009.06), so I'm not sure if your advice actually applies to my situation. Daniel Bakken On Tue, Apr 6, 2010 at 10:57 PM, Tom Erickson thomas.erick...@oracle.comwrote: After build 128, locally set properties override received properties, and this would be the expected behavior. In that case, the value was received and you can see it like this: % zfs get -o all compression tank NAME PROPERTY VALUE RECEIVED SOURCE tank compression ongzip local % You could make the received value the effective value (clearing the local value) like this: % zfs inherit -S compression tank % zfs get -o all compression tank NAME PROPERTY VALUE RECEIVED SOURCE tank compression gzip gzip received % If the receive side is below the version that supports received properties, then I would expect the receive to set compression=gzip. After build 128 'zfs receive' prints an error message for every property it fails to set. Before that version, 'zfs receive' is silent when it fails to set a property so long as everything else is successful. I might check whether I have permission to set compression with 'zfs allow'. You could pipe the send stream to zstreamdump to verify that compression=gzip is in the send stream, but I think before build 125 you will not have zstreamdump. Tom ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression property not received
Here is the info from zstreamdump -v on the sending side:
BEGIN record
hdrtype = 2
features = 0
magic = 2f5bacbac
creation_time = 0
type = 0
flags = 0x0
toguid = 0
fromguid = 0
toname = promise1/arch...@daily.1
nvlist version: 0
tosnap = daily.1
fss = (embedded nvlist)
nvlist version: 0
0xcfde021e56c8fc = (embedded nvlist)
nvlist version: 0
name = promise1/archive
parentfromsnap = 0x0
props = (embedded nvlist)
nvlist version: 0
mountpoint = /promise1/archive
compression = 0xa
dedup = 0x2
(end props)
I assume that compression = 0xa means gzip. I wonder if the dedup property
is causing the receiver (build 111b) to disregard all other properties,
since the receiver doesn't support dedup. Dedup was enabled in the past on
the sending filesystem, but is now disabled for reasons of sanity.
I'd like to try the dtrace debugging, but it would destroy the progress I've
made so far transferring the filesystem.
Thanks,
Daniel
On Wed, Apr 7, 2010 at 12:52 AM, Tom Erickson thomas.erick...@oracle.comwrote:
The advice regarding received vs local properties definitely does not
apply. You could still confirm the presence of the compression property in
the send stream with zstreamdump, since the send side is running build 129.
To debug the receive side I might dtrace the zap_update() function with the
fbt provider, something like
zfs send -R promise1/arch...@daily.1 | dtrace -c 'zfs receive -vd sas' \
-n 'fbt::zap_update:entry / stringof(args[2]) == compression || \
stringof(args[2]) == compression$recvd / { self-trace = 1; }' \
-n 'fbt::zap_update:return / self-trace / { trace(args[1]); \
self-trace = 0; }'
and look for non-zero return values.
I'd also redirect 'zdb -vvv poolname' to a file and search it for
compression to check the value in the ZAP.
I assume you have permission to set the compression property on the receive
side, but I'd check anyway.
Tom
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression property not received
We have found the problem. The mountpoint property on the sender was at one time changed from the default, then later changed back to defaults using zfs set instead of zfs inherit. Therefore, zfs send included these local non-default properties in the stream, even though the local properties are effectively set at defaults. This caused the receiver to stop processing subsequent properties in the stream because the mountpoint isn't valid on the receiver. I tested this theory with a spare zpool. First I used zfs inherit mountpoint promise1/archive to remove the local setting (which was exactly the same value as the default). This time the compression=gzip property was correctly received. It seems like a bug to me that one failed property in a stream prevents the rest from being applied. I should have used zfs inherit, but it would be best if zfs receive handled failures more gracefully, and attempted to set as many properties as possible. Thanks to Cindy and Tom for their help. Daniel On Wed, Apr 7, 2010 at 2:31 AM, Tom Erickson thomas.erick...@oracle.comwrote: Now I remember that 'zfs receive' used to give up after the first property it failed to set. If I'm remembering correctly, then, in this case, if the mountpoint was invalid on the receive side, 'zfs receive' would not even try to set the remaining properties. I'd try the following in the source dataset: zfs inherit mountpoint promise1/archive to clear the explicit mountpoint and prevent it from being included in the send stream. Later set it back the way it was. (Soon there will be an option to take care of that; see CR 6883722 want 'zfs recv -o prop=value' to set initial property values of received dataset.) Then see if you receive the compression property successfully. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression property not received
Daniel Bakken wrote: When I send a filesystem with compression=gzip to another server with compression=on, compression=gzip is not set on the received filesystem. I am using: zfs send -R promise1/arch...@daily.1 | zfs receive -vd sas The zfs manpage says regarding the -R flag: When received, all properties, snapshots, descendent file systems, and clones are preserved. Snapshots are preserved, but the compression property is not. Any ideas why this doesn't work as advertised? After build 128, locally set properties override received properties, and this would be the expected behavior. In that case, the value was received and you can see it like this: % zfs get -o all compression tank NAME PROPERTY VALUE RECEIVED SOURCE tank compression ongzip local % You could make the received value the effective value (clearing the local value) like this: % zfs inherit -S compression tank % zfs get -o all compression tank NAME PROPERTY VALUE RECEIVED SOURCE tank compression gzip gzip received % If the receive side is below the version that supports received properties, then I would expect the receive to set compression=gzip. After build 128 'zfs receive' prints an error message for every property it fails to set. Before that version, 'zfs receive' is silent when it fails to set a property so long as everything else is successful. I might check whether I have permission to set compression with 'zfs allow'. You could pipe the send stream to zstreamdump to verify that compression=gzip is in the send stream, but I think before build 125 you will not have zstreamdump. Tom ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression property not received
Daniel Bakken wrote:
The receive side is running build 111b (2009.06), so I'm not sure if
your advice actually applies to my situation.
The advice regarding received vs local properties definitely does not
apply. You could still confirm the presence of the compression property
in the send stream with zstreamdump, since the send side is running
build 129. To debug the receive side I might dtrace the zap_update()
function with the fbt provider, something like
zfs send -R promise1/arch...@daily.1 | dtrace -c 'zfs receive -vd sas' \
-n 'fbt::zap_update:entry / stringof(args[2]) == compression || \
stringof(args[2]) == compression$recvd / { self-trace = 1; }' \
-n 'fbt::zap_update:return / self-trace / { trace(args[1]); \
self-trace = 0; }'
and look for non-zero return values.
I'd also redirect 'zdb -vvv poolname' to a file and search it for
compression to check the value in the ZAP.
I assume you have permission to set the compression property on the
receive side, but I'd check anyway.
Tom
On Tue, Apr 6, 2010 at 10:57 PM, Tom Erickson
thomas.erick...@oracle.com mailto:thomas.erick...@oracle.com wrote:
After build 128, locally set properties override received
properties, and this would be the expected behavior. In that case,
the value was received and you can see it like this:
% zfs get -o all compression tank
NAME PROPERTY VALUE RECEIVED SOURCE
tank compression ongzip local
%
You could make the received value the effective value (clearing the
local value) like this:
% zfs inherit -S compression tank
% zfs get -o all compression tank
NAME PROPERTY VALUE RECEIVED SOURCE
tank compression gzip gzip received
%
If the receive side is below the version that supports received
properties, then I would expect the receive to set compression=gzip.
After build 128 'zfs receive' prints an error message for every
property it fails to set. Before that version, 'zfs receive' is
silent when it fails to set a property so long as everything else is
successful. I might check whether I have permission to set
compression with 'zfs allow'. You could pipe the send stream to
zstreamdump to verify that compression=gzip is in the send stream,
but I think before build 125 you will not have zstreamdump.
Tom
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression property not received
Daniel Bakken wrote: Here is the info from zstreamdump -v on the sending side: BEGIN record hdrtype = 2 features = 0 magic = 2f5bacbac creation_time = 0 type = 0 flags = 0x0 toguid = 0 fromguid = 0 toname = promise1/arch...@daily.1 nvlist version: 0 tosnap = daily.1 fss = (embedded nvlist) nvlist version: 0 0xcfde021e56c8fc = (embedded nvlist) nvlist version: 0 name = promise1/archive parentfromsnap = 0x0 props = (embedded nvlist) nvlist version: 0 mountpoint = /promise1/archive compression = 0xa dedup = 0x2 (end props) I assume that compression = 0xa means gzip. Yep, that's ZIO_COMPRESS_GZIP_6, the default gzip. I wonder if the dedup property is causing the receiver (build 111b) to disregard all other properties, since the receiver doesn't support dedup. Dedup was enabled in the past on the sending filesystem, but is now disabled for reasons of sanity. Now I remember that 'zfs receive' used to give up after the first property it failed to set. If I'm remembering correctly, then, in this case, if the mountpoint was invalid on the receive side, 'zfs receive' would not even try to set the remaining properties. You could try 'zfs get mountpoint' (or 'zdb -vvv poolname file' and search the file for 'mountpoint') to see if that was set. I'd like to try the dtrace debugging, but it would destroy the progress I've made so far transferring the filesystem. Maybe you could try receiving into a new pool that you can later throw away. zpool create bogustestpool c0t0d0 zfs send -R promise1/arch...@daily.1 | zfs receive -vd bogustestpool I'd try the following in the source dataset: zfs inherit mountpoint promise1/archive to clear the explicit mountpoint and prevent it from being included in the send stream. Later set it back the way it was. (Soon there will be an option to take care of that; see CR 6883722 want 'zfs recv -o prop=value' to set initial property values of received dataset.) Then see if you receive the compression property successfully. Tom ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression property not received
Daniel Bakken wrote: We have found the problem. The mountpoint property on the sender was at one time changed from the default, then later changed back to defaults using zfs set instead of zfs inherit. Therefore, zfs send included these local non-default properties in the stream, even though the local properties are effectively set at defaults. This caused the receiver to stop processing subsequent properties in the stream because the mountpoint isn't valid on the receiver. I tested this theory with a spare zpool. First I used zfs inherit mountpoint promise1/archive to remove the local setting (which was exactly the same value as the default). This time the compression=gzip property was correctly received. It seems like a bug to me that one failed property in a stream prevents the rest from being applied. I should have used zfs inherit, but it would be best if zfs receive handled failures more gracefully, and attempted to set as many properties as possible. Yes, that was fixed in build 128. Thanks to Cindy and Tom for their help. Glad to hear we identified the problem. Sorry for the trouble. Tom ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] compression ratio
With the default compression scheme (LZJB ), how does one calculate the ratio or amount compressed ahead of time when allocating storage? -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression at zfs filesystem creation
On Wed, 2009-06-17 at 12:35 +0200, casper@sun.com wrote: I still use disk swap because I have some bad experiences with ZFS swap. (ZFS appears to cache and that is very wrong) I'm experimenting with running zfs swap with the primarycache attribute set to metadata instead of the default all. aka: zfs set primarycache=metadata rpool/swap seems like that would be more likely to behave appropriately. - Bill ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression at zfs filesystem creation
Bill Sommerfeld wrote: On Wed, 2009-06-17 at 12:35 +0200, casper@sun.com wrote: I still use disk swap because I have some bad experiences with ZFS swap. (ZFS appears to cache and that is very wrong) I'm experimenting with running zfs swap with the primarycache attribute set to metadata instead of the default all. aka: zfs set primarycache=metadata rpool/swap seems like that would be more likely to behave appropriately. Agreed, and for the just incase scenario secondarycache=none - but then again using an SSD as swap could be interesting -- Darren J Moffat ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression at zfs filesystem creation
Bob Friesenhahn wrote: On Wed, 17 Jun 2009, Haudy Kazemi wrote: usable with very little CPU consumed. If the system is dedicated to serving files rather than also being used interactively, it should not matter much what the CPU usage is. CPU cycles can't be stored for later use. Ultimately, it (mostly*) does not matter if Clearly you have not heard of the software flywheel: http://www.simplesystems.org/users/bfriesen/software_flywheel.html I had not heard of such a device, however from the description it appears to be made from virtual unobtanium :) My line of reasoning is that unused CPU cycles are to some extent a wasted resource, paralleling the idea that having system RAM sitting empty/unused is also a waste and should be used for caching until the system needs that RAM for other purposes (how the ZFS cache is supposed to work). This isn't a perfect parallel as CPU power consumption and heat outlet do vary by load much more than does RAM. I'm sure someone could come up with a formula for the optimal CPU loading to maximize energy efficiency. There has been work on this the paper 'Dynamic Data Compression in Multi-hop Wireless Networks' at http://enl.usc.edu/~abhishek/sigmpf03-sharma.pdf . If I understand the blog entry correctly, for text data the task took up to 3.5X longer to complete, and for media data, the task took about 2.2X longer to complete with a maximum storage compression ratio of 2.52X. For my backup drive using lzjb compression I see a compression ratio of only 1.53x. I linked to several blog posts. It sounds like you are referring to ' http://blogs.sun.com/dap/entry/zfs_compression#comments '? This blog's test results show that on their quad core platform (Sun 7410 have quad core 2.3 ghz AMD Opteron cpus*) : * http://sunsolve.sun.com/handbook_pub/validateUser.do?target=Systems/7410/spec for text data, LZJB compression had negligible performance benefits (task times were unchanged or marginally better) and less storage space was consumed (1.47:1). for media data, LZJB compression had negligible performance benefits (task times were unchanged or marginally worse) and storage space consumed was unchanged (1:1). Take away message: as currently configured, their system has nothing to lose from enabling LZJB. for text data, GZIP compression at any setting, had a significant negative impact on write times (CPU bound), no performance impact on read times, and significant positive improvements in compression ratio. for media data, GZIP compression at any setting, had a significant negative impact on write times (CPU bound), no performance impact on read times, and marginal improvements in compression ratio. Take away message: With GZIP as their system is currently configured, write performance would suffer in exchange for a higher compression ratio. This may be acceptable if the system fulfills a role that has a read heavy usage profile of compressible content. (An archive.org backend would be such an example.) This is similar to the tradeoff made when comparing RAID1 or RAID10 vs RAID5. Automatic benchmarks could be used to detect and select the optimal compression settings for best performance, with the basic case assuming the system is a dedicated file server and more advanced cases accounting for the CPU needs of other processes run on the same platform. Another way would be to ask the administrator what the usage profile for the machine will be and preconfigure compression settings suitable for that use case. Single and dual core systems are more likely to become CPU bound from enabling compression than a quad core. All systems have bottlenecks in them somewhere by virtue of design decisions. One or more of these bottlenecks will be the rate limiting factor for any given workload, such that even if you speed up the rest of the system the process will still take the same amount of time to complete. The LZJB compression benchmarks on the quad core above demonstrate that LZJB is not the rate limiter either in writes or reads. The GZIP benchmarks show that it is a rate limiter, but only during writes. On a more powerful platform (6x faster CPU), GZIP writes may no longer be the bottleneck (assuming that the network bandwidth and drive I/O bandwidth remain unchanged). System component balancing also plays a role. If the server is connected via a 100 Mbps CAT5e link, and all I/O activity is from client computers on that link, does it make any difference if the server is actually capable of GZIP writes at 200 Mbps, 500 Mbps, or 1500 Mbps? If the network link is later upgraded to Gigabit ethernet, now only the system capable of GZIPing at 1500 Mbps can keep up. The rate limiting factor changes as different components are upgraded. In many systems for many workloads, hard drive I/O bandwidth is the rate limiting factor that has the most significant performance impact, such that a 20% boost
Re: [zfs-discuss] compression at zfs filesystem creation
On Thu, 18 Jun 2009, Haudy Kazemi wrote: for text data, LZJB compression had negligible performance benefits (task times were unchanged or marginally better) and less storage space was consumed (1.47:1). for media data, LZJB compression had negligible performance benefits (task times were unchanged or marginally worse) and storage space consumed was unchanged (1:1). Take away message: as currently configured, their system has nothing to lose from enabling LZJB. My understanding is that these tests were done with NFS and one client over gigabit ethernet (a file server scenario). So in this case, the system is able to keep up with NFS over gigabit ethernet when LZJB is used. In a stand-alone power-user desktop scenario, the situtation may be quite different. In this case application CPU usage may be competing with storage CPU usage. Since ZFS often defers writes, it may be that the compression is performed at the same time as application compute cycles. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression at zfs filesystem creation
Hello Richard, Monish Shah wrote: What about when the compression is performed in dedicated hardware? Shouldn't compression be on by default in that case? How do I put in an RFE for that? Is there a bugs.intel.com? :-) I may have misled you. I'm not asking for Intel to add hardware compression. Actually, we already have gzip compression boards that we have integrated into OpenSolaris / ZFS and they are also supported under NexentaStor. What I'm saying is that if such a card is installed, compression should be enabled by default. NB, Solaris already does this for encryption, which is often a more computationally intensive operation. Actually, compression is more compute intensive than symmetric encryption (such as AES). Public key encryption, on the other hand, is horrendously compute intensive, much more than compression or symmectric encryption. But, nobody uses public key encryption for bulk data encryption, so that doesn't apply. Your mileage may vary. You can always come up with compression algorithms that don't do a very good job of compressing, but which are light on CPU utilization. Monish I think the general cases are performed well by current hardware, and it is already multithreaded. The bigger issue is, as Bob notes, resource management. There is opportunity for people to work here, especially since the community has access to large amounts of varied hardware. Should we spin up a special interest group of some sort? -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression at zfs filesystem creation
David Magda dma...@ee.ryerson.ca writes: On Tue, June 16, 2009 15:32, Kyle McDonald wrote: So the cache saves not only the time to access the disk but also the CPU time to decompress. Given this, I think it could be a big win. Unless you're in GIMP working on JPEGs, or doing some kind of MPEG video editing--or ripping audio (MP3 / AAC / FLAC) stuff. All of which are probably some of the largest files in most people's homedirs nowadays. indeed. I think only programmers will see any substantial benefit from compression, since both the code itself and the object files generated are easily compressible. 1 GB of e-mail is a lot (probably my entire personal mail collection for a decade) and will compress well; 1 GB of audio files is nothing, and won't compress at all. Perhaps compressing /usr could be handy, but why bother enabling compression if the majority (by volume) of user data won't do anything but burn CPU? So the correct answer on whether compression should be enabled by default is it depends. (IMHO :) ) I'd be interested to see benchmarks on MySQL/PostgreSQL performance with compression enabled. my *guess* would be it isn't beneficial since they usually do small reads and writes, and there is little gain in reading 4 KiB instead of 8 KiB. what other uses cases can benefit from compression? -- Kjetil T. Homme Redpill Linpro AS - Changing the game ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression at zfs filesystem creation
On Wed, Jun 17, 2009 at 5:03 PM, Kjetil Torgrim Hommekjeti...@linpro.no wrote: indeed. I think only programmers will see any substantial benefit from compression, since both the code itself and the object files generated are easily compressible. Perhaps compressing /usr could be handy, but why bother enabling compression if the majority (by volume) of user data won't do anything but burn CPU? How do you define substantial? My opensolaris snv_111b installation has 1.47x compression ratio for /, with the default compression. It's well worthed for me. -- Fajar ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression at zfs filesystem creation
On Wed, Jun 17, 2009 at 5:03 PM, Kjetil Torgrim Hommekjeti...@linpro= .no wrote: indeed. =A0I think only programmers will see any substantial benefi= t from compression, since both the code itself and the object files generated are easily compressible. Perhaps compressing /usr could be handy, but why bother enabling compression if the majority (by volume) of user data won't do anything but burn CPU? How do you define substantial? My opensolaris snv_111b installation has 1.47x compression ratio for /, with the default compression. It's well worthed for me. Indeed; I've had a few systems with: UFS (boot env 1) UFS (boot env 2) swap lucreate couldn't fix everything in one (old UFS) partition because of dump and swap; with compression I can fit multiple environments (more than two). I still use disk swap because I have some bad experiences with ZFS swap. (ZFS appears to cache and that is very wrong) Now I use: rpool (using both the UFS partitions, now concatenated into one slice) and real swap. My ZFS/Solaris wish list is this: - when you convert from UFS to ZFS, zpool create fails and requires create if; I'd like zpool create about *all* errors, not just one so you know exactly what collateral damage you would do) has a UFS filesystem s2 overlaps s0 etc - zpool upgrade should fail if one of the available boot environments doesn't support the new version (or upgrade to the lowest supported zfs version) Casper ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression at zfs filesystem creation
Fajar A. Nugraha fa...@fajar.net writes: Kjetil Torgrim Homme wrote: indeed. I think only programmers will see any substantial benefit from compression, since both the code itself and the object files generated are easily compressible. Perhaps compressing /usr could be handy, but why bother enabling compression if the majority (by volume) of user data won't do anything but burn CPU? How do you define substantial? My opensolaris snv_111b installation has 1.47x compression ratio for /, with the default compression. It's well worthed for me. I don't really care if my / is 5 GB or 3 GB. how much faster is your system operating? what's the compression rate on your data areas? -- Kjetil T. Homme Redpill Linpro AS - Changing the game ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression at zfs filesystem creation
Unless you're in GIMP working on JPEGs, or doing some kind of MPEG video editing--or ripping audio (MP3 / AAC / FLAC) stuff. All of which are probably some of the largest files in most people's homedirs nowadays. indeed. I think only programmers will see any substantial benefit from compression, since both the code itself and the object files generated are easily compressible. If we are talking about data on people's desktops and laptops, yes, it is not very common to see a lot of compressible data. There will be some other examples, such as desktops being used for engineering drawings. The CAD files do tend to be compressible and they tend to be big. In any case, the really interesting case for compression is for business data (databases, e-mail servers, etc.) which tends to be quite compressible. ... I'd be interested to see benchmarks on MySQL/PostgreSQL performance with compression enabled. my *guess* would be it isn't beneficial since they usually do small reads and writes, and there is little gain in reading 4 KiB instead of 8 KiB. OK, now you have switched from compressibility of data to performance advantage. As I said above, this kind of data usually compresses pretty well. I agree that for random reads, there wouldn't be any gain from compression. For random writes, in a copy-on-write file system, there might be gains, because the blocks may be arranged in sequential fashion anyway. We are in the process of doing some performance tests to prove or disprove this. Now, if you are using SSDs for this type of workload, I'm pretty sure that compression will help writes. The reason is that the flash translation layer in the SSD has to re-arrange the data and write it page by page. If there is less data to write, there will be fewer program operations. Given that write IOPS rating in an SSD is often much less than read IOPS, using compression to improve that will surely be of great value. At this point, this is educated guesswork. I'm going to see if I can get my hands on an SSD to prove this. Monish what other uses cases can benefit from compression? -- Kjetil T. Homme Redpill Linpro AS - Changing the game ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression at zfs filesystem creation
Monish Shah mon...@indranetworks.com writes: I'd be interested to see benchmarks on MySQL/PostgreSQL performance with compression enabled. my *guess* would be it isn't beneficial since they usually do small reads and writes, and there is little gain in reading 4 KiB instead of 8 KiB. OK, now you have switched from compressibility of data to performance advantage. As I said above, this kind of data usually compresses pretty well. the thread has been about I/O performance since the first response, as far as I can tell. I agree that for random reads, there wouldn't be any gain from compression. For random writes, in a copy-on-write file system, there might be gains, because the blocks may be arranged in sequential fashion anyway. We are in the process of doing some performance tests to prove or disprove this. Now, if you are using SSDs for this type of workload, I'm pretty sure that compression will help writes. The reason is that the flash translation layer in the SSD has to re-arrange the data and write it page by page. If there is less data to write, there will be fewer program operations. Given that write IOPS rating in an SSD is often much less than read IOPS, using compression to improve that will surely be of great value. not necessarily, since a partial SSD write is much more expensive than a full block write (128 KiB?). in a write intensive application, that won't be an issue since the data is flowing steadily, but for the right mix of random reads and writes, this may exacerbate the bottleneck. At this point, this is educated guesswork. I'm going to see if I can get my hands on an SSD to prove this. that'd be great! -- Kjetil T. Homme Redpill Linpro AS - Changing the game ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression at zfs filesystem creation
On Wed, June 17, 2009 06:15, Fajar A. Nugraha wrote: Perhaps compressing /usr could be handy, but why bother enabling compression if the majority (by volume) of user data won't do anything but burn CPU? How do you define substantial? My opensolaris snv_111b installation has 1.47x compression ratio for /, with the default compression. It's well worthed for me. And how many GB is that? ~1.5x is quite good, but if you're talking about a 7.5 GB install using only 3 GB of space, but your homedir is 50 GB, it's not a lot in relative terms. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression at zfs filesystem creation
Bob Friesenhahn wrote: On Mon, 15 Jun 2009, Bob Friesenhahn wrote: On Mon, 15 Jun 2009, Rich Teer wrote: You actually have that backwards. :-) In most cases, compression is very desirable. Performance studies have shown that today's CPUs can compress data faster than it takes for the uncompressed data to be read or written. Do you have a reference for such an analysis based on ZFS? I would be interested in linear read/write performance rather than random access synchronous access. Perhaps you are going to make me test this for myself. Ok, I tested this for myself on a Solaris 10 system with 4 3GHz AMD64 cores and see that we were both right. I did an iozone run with compression and do see a performance improvement. I don't know what the data iozone produces looks like, but it clearly must be quite compressable. Testing was done with a 64GB file: KB reclen write rewrite read reread uncompressed: 67108864 128 359965 354854 550869 554271 lzjb: 67108864 128 851336 924881 1289059 1362625 Unfortunately, during the benchmark run with lzjb the system desktop was essentially unusable with misbehaving mouse and keyboard as well as reported 55% CPU consumption. Without the compression the system is fully usable with very little CPU consumed. If the system is dedicated to serving files rather than also being used interactively, it should not matter much what the CPU usage is. CPU cycles can't be stored for later use. Ultimately, it (mostly*) does not matter if one option consumes more CPU resources than another if those CPU resources were otherwise going to go unused. Changes (increases) in latencies are a consideration but probably depend more on process scheduler choice and policies. *Higher CPU usage will increase energy consumption, heat output, and cooling costs...these may be important considerations in some specialized dedicated file server applications, depending on operational considerations. The interactivity hit may pose a greater challenge for any other processes/databases/virtual machines run on hardware that also serves files. The interactivity hit may also be evidence that the process scheduler is not fairly or effectively sharing CPU resources amongst the running processes. If scheduler tweaks aren't effective, perhaps dedicating a processor core(s) to interactive GUI stuff and the other cores to filesystem duties would help smooth things out. Maybe zones be used for that? With a slower disk subsystem the CPU overhead would surely be less since writing is still throttled by the disk. It would be better to test with real data rather than iozone. There are 4 sets of articles with links and snippets from their test data below. Follow the links for the full discussion: First article: http://blogs.sun.com/dap/entry/zfs_compression#comments Hardware: Sun Storage 7000 # The server is a quad-core 7410 with 1 JBOD (configured with mirrored storage) and 16GB of RAM. No SSD. # The client machine is a quad-core 7410 with 128GB of DRAM. Summary: text data set Compression Ratio Total Write Read off 1.00x 3:30 2:08 1:22 lzjb 1.47x 3:26 2:04 1:22 gzip-2 2.35x 6:12 4:50 1:22 gzip 2.52x 11:18 9:56 1:22 gzip-9 2.52x 12:16 10:54 1:22 Summary: media data set Compression Ratio Total Write Read off 1.00x 3:29 2:07 1:22 lzjb 1.00x 3:31 2:09 1:22 gzip-2 1.01x 6:59 5:37 1:22 gzip 1.01x 7:18 5:57 1:21 gzip-9 1.01x 7:37 6:15 1:22 Second article/discussion: http://ekschi.com/technology/2009/04/28/zfs-compression-a-win-win/ http://blogs.sun.com/observatory/entry/zfs_compression_a_win_win Third article summary: ZFS and MySQL/InnoDB shows that gzip is often cpu-bound on current processors; lzjb improves performance. http://blogs.smugmug.com/don/2008/10/13/zfs-mysqlinnodb-compression-update/ Hardware: SunFire X2200 M2 w/64GB of RAM and 2 x dual-core 2.6GHz Opterons Dell MD3000 w/15 x 15K SCSI disks and mirrored 512MB battery-backed write caches "Also note that this is writing to two DAS enclosures with 15 x 15K SCSI disks apiece (28 spindles in a striped+mirrored configuration) with 512MB of write cache apiece." TABLE1 compression size ratio time uncompressed 172M 1 0.207s lzjb 79M 2.18X 0.234s gzip-1 50M 3.44X 0.24s gzip-9 46M 3.73X 0.217s
Re: [zfs-discuss] compression at zfs filesystem creation
David Magda wrote: On Tue, June 16, 2009 15:32, Kyle McDonald wrote: So the cache saves not only the time to access the disk but also the CPU time to decompress. Given this, I think it could be a big win. Unless you're in GIMP working on JPEGs, or doing some kind of MPEG video editing--or ripping audio (MP3 / AAC / FLAC) stuff. All of which are probably some of the largest files in most people's homedirs nowadays. 1 GB of e-mail is a lot (probably my entire personal mail collection for a decade) and will compress well; 1 GB of audio files is nothing, and won't compress at all. Perhaps compressing /usr could be handy, but why bother enabling compression if the majority (by volume) of user data won't do anything but burn CPU? So the correct answer on whether compression should be enabled by default is it depends. (IMHO :) ) The performance tests I've found almost universally show LZJB as not being cpu-bound on recent equipment. A few years from now GZIP may get away from being cpu-bound. As performance tests on current hardware show that enabling LZJB improves overall performance it would make sense to enable it by default. In the future when GZIP is no longer cpu-bound, it might become the default (or there could be another algorithm). There is a long history of previously formidable tasks starting out as cpu-bound but quickly progressing to an 'easily handled in the background' task. Decoding MP3 and MPEG1, MPEG2 (DVD resolutions), softmodems (and other host signal processor devices), and RAID are all tasks that can easily be handled by recent equipment. Another option/idea to consider is using LZJB as the default compression method, and then performing a background scrub-recompress during otherwise idle times. Technique ideas: 1.) A performance neutral/performance enhancing technique: use any algorithm that is not CPU bound on your hardware, and rarely if ever has worse performance than the uncompressed state 2.) Adaptive technique 1: rarely used blocks could be given the strongest compression (using an algorithm tuned for the data type detected), while frequently used blocks would be compressed at a performance neutral or performance improving levels. 3.) Adaptive technique 2: rarely used blocks could be given the strongest compression (using an algorithm tuned for the data type detected), while frequently used blocks would be compressed at a performance neutral or performance improving levels. As the storage device gets closer to its native capacity, start applying compression both proactively (to new data) and retroactively (to old data), progressively using more powerful compression techniques as the maximum native capacity is approached. Compression could delay the users from reaching the 80-95% capacity point where system performance curves often have their knees (a massive performance degradation with each additional unit). 4.) Maximize space technique: detect the data type and use the best available algorithm for the block. As a counterpoint, if drive capacities keep growing at their current pace it seems they ultimately risk obviating the need to give much thought to the compression algorithm, except to choose one that boosts system performance. (I.e. in hard drives, compression may primarily be used to improve performance rather than gain extra storage space, as drive capacity has grown many times faster than drive performance.) JPEGs often CAN be /losslessly/ compressed further by useful amounts (e.g. 25% space savings). There is more on this here: Tests: http://www.maximumcompression.com/data/jpg.php http://compression.ca/act/act-jpeg.html http://www.downloadsquad.com/2008/09/11/winzip-12-supports-lossless-jpg-compression/ http://download.cnet.com/8301-2007_4-10038172-12.html http://www.online-tech-tips.com/software-reviews/winzip-vs-7-zip-best-compression-method/ These have source code available: http://sylvana.net/jpeg-ari/ PAQ8R http://www.cs.fit.edu/~mmahoney/compression/ (general info http://en.wikipedia.org/wiki/PAQ ) This one says source code is not yet available (implying it may become available): http://www.elektronik.htw-aalen.de/packjpg/packjpg_m.htm ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression at zfs filesystem creation
On Wed, 17 Jun 2009, Haudy Kazemi wrote: usable with very little CPU consumed. If the system is dedicated to serving files rather than also being used interactively, it should not matter much what the CPU usage is. CPU cycles can't be stored for later use. Ultimately, it (mostly*) does not matter if Clearly you have not heard of the software flywheel: http://www.simplesystems.org/users/bfriesen/software_flywheel.html If I understand the blog entry correctly, for text data the task took up to 3.5X longer to complete, and for media data, the task took about 2.2X longer to complete with a maximum storage compression ratio of 2.52X. For my backup drive using lzjb compression I see a compression ratio of only 1.53x. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression at zfs filesystem creation
Hello, I would like to add one more point to this. Everyone seems to agree that compression is useful for reducing load on the disks and the disagreement is about the impact on CPU utilization, right? What about when the compression is performed in dedicated hardware? Shouldn't compression be on by default in that case? How do I put in an RFE for that? Monish On Mon, 15 Jun 2009, dick hoogendijk wrote: IF at all, it certainly should not be the DEFAULT. Compression is a choice, nothing more. I respectfully disagree somewhat. Yes, compression shuould be a choice, but I think the default should be for it to be enabled. I agree that Compression is a choice and would add : Compression is a choice and it is the default. Just my feelings on the issue. Dennis Clarke ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression at zfs filesystem creation
On Mon, 15 Jun 2009, Bob Friesenhahn wrote: On Mon, 15 Jun 2009, Thommy M. wrote: In most cases compression is not desireable. It consumes CPU and results in uneven system performance. IIRC there was a blog about I/O performance with ZFS stating that it was faster with compression ON as it didn't have to wait for so much data from the disks and that the CPU was fast at unpacking data. But sure, it uses more CPU (and probably memory). I'll believe this when I see it. :-) With really slow disks and a fast CPU it is possible that reading data the first time is faster. However, Solaris is really good at caching data so any often-accessed data is highly likely to be cached and therefore read just one time. The main point of using compression for the root pool would be so that the OS can fit on an abnormally small device such as a FLASH disk. I would use it for a read-mostly device or an archive (backup) device. Well, it depends on your working set and how much memory you have. I came across systems with lots of CPU left to spare but a working set is much bigger than the amount of memory and enabling lzjb gave over 2x compression ratio and make an application to run faster. Seen it with ldap, mysql and couple of other apps. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression at zfs filesystem creation
On Mon, 15 Jun 2009, Bob Friesenhahn wrote: On Mon, 15 Jun 2009, Rich Teer wrote: You actually have that backwards. :-) In most cases, compression is very desirable. Performance studies have shown that today's CPUs can compress data faster than it takes for the uncompressed data to be read or written. Do you have a reference for such an analysis based on ZFS? I would be interested in linear read/write performance rather than random access synchronous access. Perhaps you are going to make me test this for myself. Ok, I tested this for myself on a Solaris 10 system with 4 3GHz AMD64 cores and see that we were both right. I did an iozone run with compression and do see a performance improvement. I don't know what the data iozone produces looks like, but it clearly must be quite compressable. Testing was done with a 64GB file: KB reclen write rewritereadreread uncompressed: 67108864 128 359965 354854 550869 554271 lzjb: 67108864 128 851336 924881 1289059 1362625 Unfortunately, during the benchmark run with lzjb the system desktop was essentially unusable with misbehaving mouse and keyboard as well as reported 55% CPU consumption. Without the compression the system is fully usable with very little CPU consumed. With a slower disk subsystem the CPU overhead would surely be less since writing is still throttled by the disk. It would be better to test with real data rather than iozone. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression at zfs filesystem creation
Bob Friesenhahn wrote: On Mon, 15 Jun 2009, Thommy M. wrote: In most cases compression is not desireable. It consumes CPU and results in uneven system performance. IIRC there was a blog about I/O performance with ZFS stating that it was faster with compression ON as it didn't have to wait for so much data from the disks and that the CPU was fast at unpacking data. But sure, it uses more CPU (and probably memory). I'll believe this when I see it. :-) With really slow disks and a fast CPU it is possible that reading data the first time is faster. However, Solaris is really good at caching data so any often-accessed data is highly likely to be cached and therefore read just one time. One thing I'm cuious about... When reading compressed data, is it cached before or after it is uncompressed? If before, then while you've save re-reading it from the disk, there is still (redundant) overhead for uncompressing it over and over. If the uncompressed data is cached, then I agree it sounds like a total win for read-mostly filesystems. -Kyle The main point of using compression for the root pool would be so that the OS can fit on an abnormally small device such as a FLASH disk. I would use it for a read-mostly device or an archive (backup) device. On desktop systems the influence of compression on desktop response is quite noticeable when writing, even with very fast CPUs and multiple cores. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression at zfs filesystem creation
Kyle McDonald wrote: Bob Friesenhahn wrote: On Mon, 15 Jun 2009, Thommy M. wrote: In most cases compression is not desireable. It consumes CPU and results in uneven system performance. IIRC there was a blog about I/O performance with ZFS stating that it was faster with compression ON as it didn't have to wait for so much data from the disks and that the CPU was fast at unpacking data. But sure, it uses more CPU (and probably memory). I'll believe this when I see it. :-) With really slow disks and a fast CPU it is possible that reading data the first time is faster. However, Solaris is really good at caching data so any often-accessed data is highly likely to be cached and therefore read just one time. One thing I'm cuious about... When reading compressed data, is it cached before or after it is uncompressed? The decompressed (and decrypted) data is what is cached in memory. Currently the L2ARC stores decompressed (but encrypted) data on the cache devices. -- Darren J Moffat ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression at zfs filesystem creation
Monish Shah wrote: Hello, I would like to add one more point to this. Everyone seems to agree that compression is useful for reducing load on the disks and the disagreement is about the impact on CPU utilization, right? What about when the compression is performed in dedicated hardware? Shouldn't compression be on by default in that case? How do I put in an RFE for that? Is there a bugs.intel.com? :-) NB, Solaris already does this for encryption, which is often a more computationally intensive operation. I think the general cases are performed well by current hardware, and it is already multithreaded. The bigger issue is, as Bob notes, resource management. There is opportunity for people to work here, especially since the community has access to large amounts of varied hardware. Should we spin up a special interest group of some sort? -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression at zfs filesystem creation
Darren J Moffat wrote: Kyle McDonald wrote: Bob Friesenhahn wrote: On Mon, 15 Jun 2009, Thommy M. wrote: In most cases compression is not desireable. It consumes CPU and results in uneven system performance. IIRC there was a blog about I/O performance with ZFS stating that it was faster with compression ON as it didn't have to wait for so much data from the disks and that the CPU was fast at unpacking data. But sure, it uses more CPU (and probably memory). I'll believe this when I see it. :-) With really slow disks and a fast CPU it is possible that reading data the first time is faster. However, Solaris is really good at caching data so any often-accessed data is highly likely to be cached and therefore read just one time. One thing I'm cuious about... When reading compressed data, is it cached before or after it is uncompressed? The decompressed (and decrypted) data is what is cached in memory. Currently the L2ARC stores decompressed (but encrypted) data on the cache devices. So the cache saves not only the time to access the disk but also the CPU time to decompress. Given this, I think it could be a big win. -Kyle ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression at zfs filesystem creation
On Tue, June 16, 2009 15:32, Kyle McDonald wrote: So the cache saves not only the time to access the disk but also the CPU time to decompress. Given this, I think it could be a big win. Unless you're in GIMP working on JPEGs, or doing some kind of MPEG video editing--or ripping audio (MP3 / AAC / FLAC) stuff. All of which are probably some of the largest files in most people's homedirs nowadays. 1 GB of e-mail is a lot (probably my entire personal mail collection for a decade) and will compress well; 1 GB of audio files is nothing, and won't compress at all. Perhaps compressing /usr could be handy, but why bother enabling compression if the majority (by volume) of user data won't do anything but burn CPU? So the correct answer on whether compression should be enabled by default is it depends. (IMHO :) ) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] compression at zfs filesystem creation
Hi, I just installed 2009.06 and found that compression isn't enabled by default when filesystems are created. Does is make sense to have an RFE open for this? (I'll open one tonight if need be.) We keep telling people to turn on compression. Are there any situations where turning on compression doesn't make sense, like rpool/swap? what about rpool/dump? Thanks, ~~sa Shannon A. Fiume System Administrator, Infrastructure and Lab Management, Cloud Computing shannon dot fiume at sun dot com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression at zfs filesystem creation
Bob Friesenhahn wrote: On Mon, 15 Jun 2009, Shannon Fiume wrote: I just installed 2009.06 and found that compression isn't enabled by default when filesystems are created. Does is make sense to have an RFE open for this? (I'll open one tonight if need be.) We keep telling people to turn on compression. Are there any situations where turning on compression doesn't make sense, like rpool/swap? what about rpool/dump? In most cases compression is not desireable. It consumes CPU and results in uneven system performance. IIRC there was a blog about I/O performance with ZFS stating that it was faster with compression ON as it didn't have to wait for so much data from the disks and that the CPU was fast at unpacking data. But sure, it uses more CPU (and probably memory). ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression at zfs filesystem creation
* Shannon Fiume (shannon.fi...@sun.com) wrote: Hi, I just installed 2009.06 and found that compression isn't enabled by default when filesystems are created. Does is make sense to have an RFE open for this? (I'll open one tonight if need be.) We keep telling people to turn on compression. Are there any situations where turning on compression doesn't make sense, like rpool/swap? what about rpool/dump? That would be enhancement request #86. http://defect.opensolaris.org/bz/show_bug.cgi?id=86 Cheers, -- Glenn ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression at zfs filesystem creation
On Mon, 15 Jun 2009 22:51:12 +0200 Thommy M. thommy.m.malmst...@gmail.com wrote: IIRC there was a blog about I/O performance with ZFS stating that it was faster with compression ON as it didn't have to wait for so much data from the disks and that the CPU was fast at unpacking data. But sure, it uses more CPU (and probably memory). IF at all, it certainly should not be the DEFAULT. Compression is a choice, nothing more. -- Dick Hoogendijk -- PGP/GnuPG key: 01D2433D + http://nagual.nl/ | nevada / OpenSolaris 2009.06 release + All that's really worth doing is what we do for others (Lewis Carrol) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression at zfs filesystem creation
On Mon, 15 Jun 2009, dick hoogendijk wrote: IF at all, it certainly should not be the DEFAULT. Compression is a choice, nothing more. I respectfully disagree somewhat. Yes, compression shuould be a choice, but I think the default should be for it to be enabled. -- Rich Teer, SCSA, SCNA, SCSECA URLs: http://www.rite-group.com/rich http://www.linkedin.com/in/richteer ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression at zfs filesystem creation
On Mon, 15 Jun 2009, Thommy M. wrote: In most cases compression is not desireable. It consumes CPU and results in uneven system performance. IIRC there was a blog about I/O performance with ZFS stating that it was faster with compression ON as it didn't have to wait for so much data from the disks and that the CPU was fast at unpacking data. But sure, it uses more CPU (and probably memory). I'll believe this when I see it. :-) With really slow disks and a fast CPU it is possible that reading data the first time is faster. However, Solaris is really good at caching data so any often-accessed data is highly likely to be cached and therefore read just one time. The main point of using compression for the root pool would be so that the OS can fit on an abnormally small device such as a FLASH disk. I would use it for a read-mostly device or an archive (backup) device. On desktop systems the influence of compression on desktop response is quite noticeable when writing, even with very fast CPUs and multiple cores. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression at zfs filesystem creation
On Mon, 15 Jun 2009, Bob Friesenhahn wrote: In most cases compression is not desireable. It consumes CPU and results in uneven system performance. You actually have that backwards. :-) In most cases, compression is very desirable. Performance studies have shown that today's CPUs can compress data faster than it takes for the uncompressed data to be read or written. That is, the time to read or write compressed data + the time to compress or decompress it is less than the time read or write the uncompressed data. Such is the difference between CPUs and I/O! You are correct that the compression/decompression uses CPU, but most systems have an abundance of CPU, especially when performing I/O. -- Rich Teer, SCSA, SCNA, SCSECA URLs: http://www.rite-group.com/rich http://www.linkedin.com/in/richteer ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression at zfs filesystem creation
On Mon, 15 Jun 2009, dick hoogendijk wrote: IF at all, it certainly should not be the DEFAULT. Compression is a choice, nothing more. I respectfully disagree somewhat. Yes, compression shuould be a choice, but I think the default should be for it to be enabled. I agree that Compression is a choice and would add : Compression is a choice and it is the default. Just my feelings on the issue. Dennis Clarke ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression at zfs filesystem creation
On Mon, 15 Jun 2009, Rich Teer wrote: You actually have that backwards. :-) In most cases, compression is very desirable. Performance studies have shown that today's CPUs can compress data faster than it takes for the uncompressed data to be read or written. Do you have a reference for such an analysis based on ZFS? I would be interested in linear read/write performance rather than random access synchronous access. Perhaps you are going to make me test this for myself. You are correct that the compression/decompression uses CPU, but most systems have an abundance of CPU, especially when performing I/O. I assume that you are talking about single-user systems with little else to do? Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Compression/copies on root pool RFE
Richard Elling wrote: Miles Nordin wrote: AIUI the later BE's are clones of the first, and not all blocks will be rewritten, so it's still an issue. no? In practice, yes, they are clones. But whether it is an issue depends on what the issue is. As I see it, the issue is that someone wants to make install more complex, and thus lose the gains Caiman brought to the product. So, I'll stand by my comment, you can change policies later. -- richard No, the issue is wanting a compress root FS. So if you can't set it for the first OS BE, and other BEs are clones, then it's still an issue. Useful tools in the real world are complex. Sorry, but that's reality. You can have a stupid and easy mode, you can even have it be the default, but if it's the only mode you've made the product useless for serious work. -- Carson ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Compression/copies on root pool RFE
Carson Gaspar wrote: Richard Elling wrote: Miles Nordin wrote: AIUI the later BE's are clones of the first, and not all blocks will be rewritten, so it's still an issue. no? In practice, yes, they are clones. But whether it is an issue depends on what the issue is. As I see it, the issue is that someone wants to make install more complex, and thus lose the gains Caiman brought to the product. So, I'll stand by my comment, you can change policies later. -- richard No, the issue is wanting a compress root FS. So if you can't set it for the first OS BE, and other BEs are clones, then it's still an issue. Useful tools in the real world are complex. Sorry, but that's reality. You can have a stupid and easy mode, you can even have it be the default, but if it's the only mode you've made the product useless for serious work. I'll call bull* on that. Microsoft has an admirably simple installation and 88% of the market. Apple has another admirably simple installation and 10% of the market. Solaris has less than 1% of the market and has had a very complex installation process. You can't win that battle by increasing complexity. It is relatively simple to make this happen without changing the interactive installer at all: http://blogs.sun.com/glagasse/entry/howto_enable_zfs_compression_when But for most sites who will be doing large scale installations at sites, they will not be using interactive installers. -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Compression/copies on root pool RFE
I'll call bull* on that. Microsoft has an admirably simple installation and 88% of the market. Apple has another admirably simple installation and 10% of the market. Solaris has less than 1% of the market and has had a very complex installation process. You can't win that battle by increasing complexity. I've never installed Apple OS X but I have installed Microsoft; it's NOT at all easy. Or perhaps you are referring to no-one really installs Microsoft at all? This is true; systems come with MS Windows installed and a CD (if provided) which will reinstalls ON YOUR HARDWARE (but not on other). It is relatively simple to make this happen without changing the interactive installer at all: http://blogs.sun.com/glagasse/entry/howto_enable_zfs_compression_when But for most sites who will be doing large scale installations at sites, they will not be using interactive installers. But you will need to provide an escape hatch. (And I'm fine for interactive install to allow you to start a shell window but AI install needs to provide an escape hatch) Casper ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Compression/copies on root pool RFE
On Tue, May 5, 2009 at 6:09 PM, Ellis, Mike mike.el...@fmr.com wrote: PS: At one point the old JumpStart code was encumbered, and the community wasn't able to assist. I haven't looked at the next-gen jumpstart framework that was delivered as part of the OpenSolaris SPARC preview. Can anyone provide any background/doc-link on that new JumpStart framework? I think you are looking for the Caiman project. The replacement for jumpstart is the Automated Installer (AI). http://src.opensolaris.org/source/xref/caiman/AI_source/usr/src/ http://www.opensolaris.org/os/project/caiman/auto_install/ This is mostly code developed in the open. I'm not aware of any closed pieces that are specific to the new installer. I don't remember a lot of closed parts in the old jumpstart. (lu is different) Casper ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Compression/copies on root pool RFE
Ellis, Mike wrote: How about a generic zfs options field in the JumpStart profile? (essentially an area where options can be specified that are all applied to the boot-pool (with provisions to deal with a broken-out-var)) We had this discussion a while back and, IIRC, it was expected that AI would offer this capability (I haven't verified.) But as far as the interactive install, the goal was to ask fewer questions, not more. The existing and previous Solaris installers were considered far too complicated for people and the competition is fierce with all of the popular interactive installers much more simplified. I agree that interactive installation needs to remain as simple as possible. Note: in the Caiman world, this is only an issue for the first BE. Later BEs can easily have other policies. -- richard That should future proof things to some extent allowing for compression=x, copies=x, blocksize=x, zfsPool-version=x, checksum=sha256, future-dedup, future-crypto, and other such goodies. (by just passing the keywords and having ZFS deal with it on the other end, the jumpstart code can remain quite static, while the zfs-side gradually introduces the new features. Just a thought, -- MikeE PS: At one point the old JumpStart code was encumbered, and the community wasn't able to assist. I haven't looked at the next-gen jumpstart framework that was delivered as part of the OpenSolaris SPARC preview. Can anyone provide any background/doc-link on that new JumpStart framework? -Original Message- From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss-boun...@opensolaris.org] On Behalf Of Torrey McMahon Sent: Tuesday, May 05, 2009 6:38 PM To: zfs-discuss@opensolaris.org Subject: [zfs-discuss] Compression/copies on root pool RFE Before I put one in ... anyone else seen one? Seems we support compression on the root pool but there is no way to enable it at install time outside of a custom script you run before the installer. I'm thinking it should be a real install time option, have a jumpstart keyword, etc. Same with copies=2 Thanks. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Compression/copies on root pool RFE
On Wed, 6 May 2009, Richard Elling wrote: popular interactive installers much more simplified. I agree that interactive installation needs to remain as simple as possible. How about offering a choice an installation time: Custom or default?? Those that don't want/need the interactive flexibility can pick default whereas others who want more flexibility (but still want or need an interactive installation) can pick the custom option. Just a thought... -- Rich Teer, SCSA, SCNA, SCSECA URLs: http://www.rite-group.com/rich http://www.linkedin.com/in/richteer ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Compression/copies on root pool RFE
This sounds like a good idea to me, but it should be brought up on the caiman-disc...@opensolaris.org mailing list, since this is not just, or even primarily, a zfs issue. Lori Rich Teer wrote: On Wed, 6 May 2009, Richard Elling wrote: popular interactive installers much more simplified. I agree that interactive installation needs to remain as simple as possible. How about offering a choice an installation time: Custom or default?? Those that don't want/need the interactive flexibility can pick default whereas others who want more flexibility (but still want or need an interactive installation) can pick the custom option. Just a thought... ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Compression/copies on root pool RFE
On Wed, May 6, 2009 at 11:14 AM, Rich Teer rich.t...@rite-group.com wrote: On Wed, 6 May 2009, Richard Elling wrote: popular interactive installers much more simplified. I agree that interactive installation needs to remain as simple as possible. How about offering a choice an installation time: Custom or default?? Those that don't want/need the interactive flexibility can pick default whereas others who want more flexibility (but still want or need an interactive installation) can pick the custom option. Just a thought... If you do propose this on caiman-discuss, I'd suggest an option to mirror 2 boot devices as well. Doing the slice attach/installgrub is nontrivial for, say, a user who's primarily a dev. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Compression/copies on root pool RFE
re == Richard Elling richard.ell...@gmail.com writes: re Note: in the Caiman world, this is only an issue for the first re BE. Later BEs can easily have other policies. -- richard AIUI the later BE's are clones of the first, and not all blocks will be rewritten, so it's still an issue. no? pgpsh8Yk5oiQD.pgp Description: PGP signature ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Compression/copies on root pool RFE
On Wed, May 6, 2009 at 2:54 AM, casper@sun.com wrote: On Tue, May 5, 2009 at 6:09 PM, Ellis, Mike mike.el...@fmr.com wrote: PS: At one point the old JumpStart code was encumbered, and the community wasn't able to assist. I haven't looked at the next-gen jumpstart framework that was delivered as part of the OpenSolaris SPARC preview. Can anyone provide any background/doc-link on that new JumpStart framework? I think you are looking for the Caiman project. The replacement for jumpstart is the Automated Installer (AI). http://src.opensolaris.org/source/xref/caiman/AI_source/usr/src/ http://www.opensolaris.org/os/project/caiman/auto_install/ This is mostly code developed in the open. I'm not aware of any closed pieces that are specific to the new installer. I don't remember a lot of closed parts in the old jumpstart. (lu is different) Perhaps not closed for the same reasons, but I've not seen the open sourcing of anything related to installation other than pkgadd/pkgrm and associated libraries. Perhaps it would be possible to open source jumpstart but I suspect the motivation of anyone to spend the time to do so right now is almost none due to the focus on Caiman. This is probably better discussed at install-discuss, so I have redirected it there. -- Mike Gerdts http://mgerdts.blogspot.com/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Compression/copies on root pool RFE
Miles Nordin wrote: re == Richard Elling richard.ell...@gmail.com writes: re Note: in the Caiman world, this is only an issue for the first re BE. Later BEs can easily have other policies. -- richard AIUI the later BE's are clones of the first, and not all blocks will be rewritten, so it's still an issue. no? In practice, yes, they are clones. But whether it is an issue depends on what the issue is. As I see it, the issue is that someone wants to make install more complex, and thus lose the gains Caiman brought to the product. So, I'll stand by my comment, you can change policies later. -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Compression/copies on root pool RFE
Before I put one in ... anyone else seen one? Seems we support compression on the root pool but there is no way to enable it at install time outside of a custom script you run before the installer. I'm thinking it should be a real install time option, have a jumpstart keyword, etc. Same with copies=2 Thanks. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Compression/copies on root pool RFE
How about a generic zfs options field in the JumpStart profile? (essentially an area where options can be specified that are all applied to the boot-pool (with provisions to deal with a broken-out-var)) That should future proof things to some extent allowing for compression=x, copies=x, blocksize=x, zfsPool-version=x, checksum=sha256, future-dedup, future-crypto, and other such goodies. (by just passing the keywords and having ZFS deal with it on the other end, the jumpstart code can remain quite static, while the zfs-side gradually introduces the new features. Just a thought, -- MikeE PS: At one point the old JumpStart code was encumbered, and the community wasn't able to assist. I haven't looked at the next-gen jumpstart framework that was delivered as part of the OpenSolaris SPARC preview. Can anyone provide any background/doc-link on that new JumpStart framework? -Original Message- From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss-boun...@opensolaris.org] On Behalf Of Torrey McMahon Sent: Tuesday, May 05, 2009 6:38 PM To: zfs-discuss@opensolaris.org Subject: [zfs-discuss] Compression/copies on root pool RFE Before I put one in ... anyone else seen one? Seems we support compression on the root pool but there is no way to enable it at install time outside of a custom script you run before the installer. I'm thinking it should be a real install time option, have a jumpstart keyword, etc. Same with copies=2 Thanks. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Compression/copies on root pool RFE
On Tue, May 5, 2009 at 6:09 PM, Ellis, Mike mike.el...@fmr.com wrote: PS: At one point the old JumpStart code was encumbered, and the community wasn't able to assist. I haven't looked at the next-gen jumpstart framework that was delivered as part of the OpenSolaris SPARC preview. Can anyone provide any background/doc-link on that new JumpStart framework? I think you are looking for the Caiman project. The replacement for jumpstart is the Automated Installer (AI). http://src.opensolaris.org/source/xref/caiman/AI_source/usr/src/ http://www.opensolaris.org/os/project/caiman/auto_install/ This is mostly code developed in the open. I'm not aware of any closed pieces that are specific to the new installer. -- Mike Gerdts http://mgerdts.blogspot.com/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression on for zpool boot disk?
Hello Krzys, Wednesday, November 5, 2008, 5:41:16 AM, you wrote: K compression is not supported for rootpool? K # zpool create rootpool c1t1d0s0 K # zfs set compression=gzip-9 rootpool K # lucreate -c ufsBE -n zfsBE -p rootpool K Analyzing system configuration. K ERROR: ZFS pool rootpool does not support boot environments K # K why? are there any plans to have compression on that disk available? how about K encryption will that be available on zfs boot disk at some point too? only lzjb compression is supported on rpools now. -- Best regards, Robertmailto:[EMAIL PROTECTED] http://milek.blogspot.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] compression on for zpool boot disk?
compression is not supported for rootpool? # zpool create rootpool c1t1d0s0 # zfs set compression=gzip-9 rootpool # lucreate -c ufsBE -n zfsBE -p rootpool Analyzing system configuration. ERROR: ZFS pool rootpool does not support boot environments # why? are there any plans to have compression on that disk available? how about encryption will that be available on zfs boot disk at some point too? Thank you. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression on for zpool boot disk?
Krzys wrote: compression is not supported for rootpool? # zpool create rootpool c1t1d0s0 # zfs set compression=gzip-9 rootpool I think gzip compression is not supported on zfs root. Try compression=on. Regards, Fajar smime.p7s Description: S/MIME Cryptographic Signature ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression=on and zpool attach
On 11/09/2007, Mike DeMarco [EMAIL PROTECTED] wrote: I've got 12Gb or so of db+web in a zone on a ZFS filesystem on a mirrored zpool. Noticed during some performance testing today that its i/o bound but using hardly any CPU, so I thought turning on compression would be a quick win. If it is io bound won't compression make it worse? Well, the CPUs are sat twiddling their thumbs. I thought reducing the amount of data going to disk might help I/O - is that unlikely? IO bottle necks are usually caused by a slow disk or one that has heavy workloads reading many small files. Two factors that need to be considered are Head seek latency and spin latency. Head seek latency is the amount of time it takes for the head to move to the track that is to be written, this is a eternity for the system (usually around 4 or 5 milliseconds). Spin latency is the amount of time it takes for the spindle to spin the track to be read or written over the head. Ideally you only want to pay the latency penalty once. If you have large reads and writes going to the disk then compression may help a little but if you have many small reads or writes it will do nothing more than to burden your CPU with a no gain amount of work to do since your are going to be paying Mr latency for each read or write. Striping several disks together with a stripe width that is tuned for your data model is how you could get your performance up. Stripping has been left out of the ZFS model for some reason. Where it is true that RAIDZ will stripe the data across a given drive set it does not give you the option to tune the stripe width. Do to the write performance problems of RAIDZ you may not get a performance boost from it stripping if your write to read ratio is too high since the driver has to calculate parity for each write. benefit of compression on the blocks that are copied by the mirror being resilvered? No! Since you are doing a block for block mirror of the data, this would not could not compress the data. No problem, another job for rsync then :) -- Rasputin :: Jack of All Trades - Master of Nuns http://number9.hellooperator.net/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discu ss This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression=on and zpool attach
On 9/12/07, Mike DeMarco [EMAIL PROTECTED] wrote: Striping several disks together with a stripe width that is tuned for your data model is how you could get your performance up. Stripping has been left out of the ZFS model for some reason. Where it is true that RAIDZ will stripe the data across a given drive set it does not give you the option to tune the stripe width. Do to the write performance problems of RAIDZ you may not get a performance boost from it stripping if your write to read ratio is too high since the driver has to calculate parity for each write. I am not sure why you think striping has been left out of the ZFS model. If you create a ZFS pool without the raidz or mirror keywords, the pool will be striped. Also, the recordsize tunable can be useful for matching up application I/O to physical I/O. Thanks, - Ryan -- UNIX Administrator http://prefetch.net ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression=on and zpool attach
Mike DeMarco wrote: IO bottle necks are usually caused by a slow disk or one that has heavy workloads reading many small files. Two factors that need to be considered are Head seek latency and spin latency. Head seek latency is the amount of time it takes for the head to move to the track that is to be written, this is a eternity for the system (usually around 4 or 5 milliseconds). For most modern disks, writes are cached in a buffer. But reads still take the latency hit. The trick is to ensure that writes are committed to media, which ZFS will do for you. Spin latency is the amount of time it takes for the spindle to spin the track to be read or written over the head. Ideally you only want to pay the latency penalty once. If you have large reads and writes going to the disk then compression may help a little but if you have many small reads or writes it will do nothing more than to burden your CPU with a no gain amount of work to do since your are going to be paying Mr latency for each read or write. Striping several disks together with a stripe width that is tuned for your data model is how you could get your performance up. Stripping has been left out of the ZFS model for some reason. Where it is true that RAIDZ will stripe the data across a given drive set it does not give you the option to tune the stripe width. It is called dynamic striping and a write is not compelled to be spread across all vdevs. This opens up an interesting rat hole conversation about whether stochastic spreading is always better than an efficient, larger block write. Our grandchildren might still be arguing this when they enter the retirement home. In general, for ZFS, the top-level dynamic stripe interlace is 1 MByte which seems to fit well with the 128kByte block size. YMMV. Do to the write performance problems of RAIDZ you may not get a performance boost from it stripping if your write to read ratio is too high since the driver has to calculate parity for each write. Write performance for raidz is generally quite good, better than most other RAID-5 implementations which are bit by the read-modify-write cycle (added latency). raidz can pay for this optimization when doing small, random reads, TANSTAAFL. -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression=on and zpool attach
On 9/12/07, Mike DeMarco [EMAIL PROTECTED] wrote: Striping several disks together with a stripe width that is tuned for your data model is how you could get your performance up. Stripping has been left out of the ZFS model for some reason. Where it is true that RAIDZ will stripe the data across a given drive set it does not give you the option to tune the stripe width. Do to the write performance problems of RAIDZ you may not get a performance boost from it stripping if your write to read ratio is too high since the driver has to calculate parity for each write. I am not sure why you think striping has been left out of the ZFS model. If you create a ZFS pool without the raidz or mirror keywords, the pool will be striped. Also, the recordsize tunable can be useful for matching up application I/O to physical I/O. Thanks, - Ryan -- UNIX Administrator http://prefetch.net ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discu ss Oh... How right you are. I dug into the PDFs and read up on Dynamic striping. My bad. ZFS rocks. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] compression=on and zpool attach
I've got 12Gb or so of db+web in a zone on a ZFS filesystem on a mirrored zpool. Noticed during some performance testing today that its i/o bound but using hardly any CPU, so I thought turning on compression would be a quick win. I know I'll have to copy files for existing data to be compressed, so I was going to make a new filesystem, enable compression and rysnc everything in, then drop the old filesystem and mount the new one (with compressed blocks) in its place. But I'm going to be hooking in faster LUNs later this week. The plan was to remove half of the mirror, attach a new disk, remove the last old disk and attach the second half of the mirror (again on a faster disk). Will this do the same job? i.e. will I see the benefit of compression on the blocks that are copied by the mirror being resilvered? -- Rasputin :: Jack of All Trades - Master of Nuns http://number9.hellooperator.net/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression=on and zpool attach
On 9/11/07, Dick Davies [EMAIL PROTECTED] wrote: I've got 12Gb or so of db+web in a zone on a ZFS filesystem on a mirrored zpool. Noticed during some performance testing today that its i/o bound but using hardly any CPU, so I thought turning on compression would be a quick win. I know I'll have to copy files for existing data to be compressed, so I was going to make a new filesystem, enable compression and rysnc everything in, then drop the old filesystem and mount the new one (with compressed blocks) in its place. But I'm going to be hooking in faster LUNs later this week. The plan was to remove half of the mirror, attach a new disk, remove the last old disk and attach the second half of the mirror (again on a faster disk). Will this do the same job? i.e. will I see the benefit of compression on the blocks that are copied by the mirror being resilvered? No; resilvering just re-copies the existing blocks, in whatever compression state they are in. You need to re-write the files *at the filesystem layer* to get the blocks compressed. Cheers, - jonathan ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression=on and zpool attach
I've got 12Gb or so of db+web in a zone on a ZFS filesystem on a mirrored zpool. Noticed during some performance testing today that its i/o bound but using hardly any CPU, so I thought turning on compression would be a quick win. If it is io bound won't compression make it worse? I know I'll have to copy files for existing data to be compressed, so I was going to make a new filesystem, enable compression and rysnc everything in, then drop the old filesystem and mount the new one (with compressed blocks) in its place. But I'm going to be hooking in faster LUNs later this week. The plan was to remove half of the mirror, attach a new disk, remove the last old disk and attach the second half of the mirror (again on a faster disk). Will this do the same job? i.e. will I see the benefit of compression on the blocks that are copied by the mirror being resilvered? No! Since you are doing a block for block mirror of the data, this would not could not compress the data. -- Rasputin :: Jack of All Trades - Master of Nuns http://number9.hellooperator.net/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discu ss This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] compression=on and zpool attach
On 11/09/2007, Mike DeMarco [EMAIL PROTECTED] wrote: I've got 12Gb or so of db+web in a zone on a ZFS filesystem on a mirrored zpool. Noticed during some performance testing today that its i/o bound but using hardly any CPU, so I thought turning on compression would be a quick win. If it is io bound won't compression make it worse? Well, the CPUs are sat twiddling their thumbs. I thought reducing the amount of data going to disk might help I/O - is that unlikely? benefit of compression on the blocks that are copied by the mirror being resilvered? No! Since you are doing a block for block mirror of the data, this would not could not compress the data. No problem, another job for rsync then :) -- Rasputin :: Jack of All Trades - Master of Nuns http://number9.hellooperator.net/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
