Here's an updated spec (diffs followed by the actual
spec), Tim will be updating the case materials as
well.
Alok
--
------- clofi.psarc -------
--- /tmp/geta7790 Thu Oct 4 16:05:24 2007
+++ /tmp/getb7790 Thu Oct 4 16:05:24 2007
@@ -20,11 +20,12 @@
This project aims to extend lofi(7D) to support reading from a
compressed file via on-the-fly decompression.
This project will also add extensions to the lofiadm(1M) command to
provide for the ability to compress a file in a suitable format
- and to decompress it entirely.
+ and to decompress it entirely. The ability to decompress is provided
+ for debugging purposes.
Design Overview
The basic design is to compress a file in fixed size segments and
store the segments sequentially. The information about where each
@@ -38,10 +39,15 @@
structure. The compression metadata thus read is then used to
decompress the requested data when a read request comes in.
If the decompression fails for some reason, an EIO
is returned.
+ It should be noted that a compressed file can only be
+ accessed in a read-only mode and no changes are allowed to it
+ once it is compressed. An attempt to access the file in read/write
+ or mount it in read/write mode will fail with an ENOTSUP.
+
On disk format
A graphic representation of the compression header stored at the
start of the file can be found under materials/ondisk_format.png
@@ -54,10 +60,20 @@
On the fly decompression
A graphic representation of reading from compressed versus regular
files can be found under materials/compress_io.jpg
+Interaction with lofi encryption
+
+ PSARC/2007/001 proposes to add encryption to lofi(7D). Since
+ compressed lofi images are read-only, it makes sense to allow
+ either compression or encryption on a file but not both at
+ the same time.
+
+ Depending upon when PSARC/2007/001 delivers, support will need
+ to be added to prevent compressing encrypted files and vice-versa.
+
Compression Algorithms
The lofi(7D) driver will support compression algorithms present
in the Solaris kernel. Currently, lzjb and gzip are the two
algorithms available.
@@ -65,13 +81,14 @@
The lzjb algorithm provides very fast compression as compared
to gzip. gzip, even though it is slower than lzjb, is fast
enough for the typical use case of a LiveCD. In addition, gzip
provides a much better compression ratio as compared to lzjb
(on the order of ~25% in some cases) and it allows for more data
- to be packed on to the media. These characteristics make gzip
- well suited for a LiveCD. Thus, gzip will be the only supported
- algorithm at this time.
+ to be packed on to the media. These characteristics make gzip,
+ specifically gzip level 9, well suited for a LiveCD. Thus, gzip
+ will be the only supported algorithm at this time with levels 6
+ and 9 being supported. gzip level 6 will be the default.
The changes being made as part of this project will allow for
compression algorithms to be plugged into lofi with relative
ease as more of these algorithms are made available in the
Solaris kernel.
@@ -78,40 +95,44 @@
Command Changes
lofiadm(1M) will take the following additional arguments -
- -C Compress the file with a specified compression algorithm
- or with the default compression algorithm if one isn't
- specified. The only supported compression algorithm at this
- time is gzip which is also the default.
+ -C gzip | gzip-N Compress the file with "gzip" compression
+ algorithm before mapping the file in. You can
+ specify the "gzip" level by using the value
+ "gzip-N" where N is 6 (fast) or 9 (best compression
+ ratio). Currently, "gzip" is equivalent to "gzip-6"
+ (which is also the default for zfs(1M) and gzip(1)).
- -s The segment size to use to divide the file being compressed.
- If no segment size is specified, a default segment size of
- 128k is used.
+ -s segsize The segment size to use to divide the file being
+ compressed. Segment size must be an integer
+ multiple of 512.
-U Uncompress a compressed file
The output of lofiadm(1M) will also change to add another field to
reflect if the mapped file(s) is a compressed file or a regular file.
Examples:
- 1. Compress a file with gzip
+ 1. Compress a file with gzip and add the compressed file as a block
+ device
- # lofiadm -C gzip /export/home/solaris.iso
+ # lofiadm -a -C gzip /export/home/solaris.iso
- 2. Compress a file with gzip and compress it in 64k chunks
+ 2. Compress a file with gzip in 64k chunks and add the compressed
+ file as a block device
# lofiadm -C gzip -s 64k /export/home/solaris.iso
3. Check if there are any compressed files that are mapped
# lofiadm
- Block Device File Type
+ Block Device File Options
/dev/lofi/1 /export/home/solaris.iso Compressed (gzip)
- /dev/lofi/2 /export/home/regular.iso Regular
+ /dev/lofi/2 /export/home/regular.iso -
4. Uncompress a compressed file
# lofiadm -U /export/home/solaris.iso
---
clofi.psarc 1.3 07/10/04 SMI
Template Version: @(#)sac_nextcase 1.3 10/04/07 SMI
1. Introduction
1.1 Project/Component Working Name:
lofi(7D) compression support
1.2 Name of Document Author/Supplier
Author: Alok Aggarwal
1.3 Date of this Document
28 September, 2007
4. Technical Description
There are cases where certain files containing a file system image
need to be compressed. A classic use case is that of a LiveCD.
Compression support in lofi is required to make LiveCDs practical.
It allows for more data to be packed on the media and improves
performance by reducing head seek times.
This project aims to extend lofi(7D) to support reading from a
compressed file via on-the-fly decompression.
This project will also add extensions to the lofiadm(1M) command to
provide for the ability to compress a file in a suitable format
and to decompress it entirely. The ability to decompress is provided
for debugging purposes.
Design Overview
The basic design is to compress a file in fixed size segments and
store the segments sequentially. The information about where each
of the segments start is maintained in an index. This index along
with a header that contains the compression algorithm used,
segment size, number of segments and the uncompressed size of the
last segment is stored at the start of the file.
When a compressed file is mapped, the header containing the
compression metadata is read in and stored in the lofi state
structure. The compression metadata thus read is then used to
decompress the requested data when a read request comes in.
If the decompression fails for some reason, an EIO
is returned.
It should be noted that a compressed file can only be
accessed in a read-only mode and no changes are allowed to it
once it is compressed. An attempt to access the file in read/write
or mount it in read/write mode will fail with an ENOTSUP.
On disk format
A graphic representation of the compression header stored at the
start of the file can be found under materials/ondisk_format.png
It should be noted that this header information is always stored
in network byte order. This allows for endian neutral filesystems
such as ZFS to be compressed on a little endian machine, and
subsequently be uncompressed on a big endian machine; and
vice-versa.
On the fly decompression
A graphic representation of reading from compressed versus regular
files can be found under materials/compress_io.jpg
Interaction with lofi encryption
PSARC/2007/001 proposes to add encryption to lofi(7D). Since
compressed lofi images are read-only, it makes sense to allow
either compression or encryption on a file but not both at
the same time.
Depending upon when PSARC/2007/001 delivers, support will need
to be added to prevent compressing encrypted files and vice-versa.
Compression Algorithms
The lofi(7D) driver will support compression algorithms present
in the Solaris kernel. Currently, lzjb and gzip are the two
algorithms available.
The lzjb algorithm provides very fast compression as compared
to gzip. gzip, even though it is slower than lzjb, is fast
enough for the typical use case of a LiveCD. In addition, gzip
provides a much better compression ratio as compared to lzjb
(on the order of ~25% in some cases) and it allows for more data
to be packed on to the media. These characteristics make gzip,
specifically gzip level 9, well suited for a LiveCD. Thus, gzip
will be the only supported algorithm at this time with levels 6
and 9 being supported. gzip level 6 will be the default.
The changes being made as part of this project will allow for
compression algorithms to be plugged into lofi with relative
ease as more of these algorithms are made available in the
Solaris kernel.
Command Changes
lofiadm(1M) will take the following additional arguments -
-C gzip | gzip-N Compress the file with "gzip" compression
algorithm before mapping the file in. You can
specify the "gzip" level by using the value
"gzip-N" where N is 6 (fast) or 9 (best compression
ratio). Currently, "gzip" is equivalent to "gzip-6"
(which is also the default for zfs(1M) and gzip(1)).
-s segsize The segment size to use to divide the file being
compressed. Segment size must be an integer
multiple of 512.
-U Uncompress a compressed file
The output of lofiadm(1M) will also change to add another field to
reflect if the mapped file(s) is a compressed file or a regular file.
Examples:
1. Compress a file with gzip and add the compressed file as a block
device
# lofiadm -a -C gzip /export/home/solaris.iso
2. Compress a file with gzip in 64k chunks and add the compressed
file as a block device
# lofiadm -C gzip -s 64k /export/home/solaris.iso
3. Check if there are any compressed files that are mapped
# lofiadm
Block Device File Options
/dev/lofi/1 /export/home/solaris.iso Compressed(gzip)
/dev/lofi/2 /export/home/regular.iso -
4. Uncompress a compressed file
# lofiadm -U /export/home/solaris.iso
References
PSARC/2007/492 - Project Indiana
http://sac.eng/PSARC/2007/492/discussion.materials/Indiana.txt
PSARC/1999/463 - lofi
http://sac.sfbay/Archives/CaseLog/arc/PSARC/1999/463
On the Fly Decompression in BeleniX
http://www.genunix.org/distributions/belenix_site/?q=compression
gzip for ZFS update
http://blogs.sun.com/ahl/entry/gzip_for_zfs_update
6. Resources and Schedule
6.4. Steering Committee requested information
6.4.1. Consolidation C-team Name:
ON
6.5. ARC review type: FastTrack
