I am sponsoring this fast-track for Alok Aggarwal.
Requested binding is micro, timeout 10/08/2007.
This information is Copyright 2007 Sun Microsystems
1. Introduction
1.1. Project/Component Working Name:
lofi(7D) compression support
1.2. Name of Document Author/Supplier:
Author: Alok Aggarwal
1.3 Date of This Document:
01 October, 2007
4. Technical Description
clofi.psarc 1.2 07/09/28 SMI
Template Version: @(#)sac_nextcase 1.2 09/28/07 SMI
1. Introduction
1.1 Project/Component Working Name:
lofi(7D) compression support
1.2 Name of Document Author/Supplier
Author: Alok Aggarwal
1.3 Date of this Document
28 September, 2007
4. Technical Description
There are cases where certain files containing a file system image
need to be compressed. A classic use case is that of a LiveCD.
Compression support in lofi is required to make LiveCDs practical.
It allows for more data to be packed on the media and improves
performance by reducing head seek times.
This project aims to extend lofi(7D) to support reading from a
compressed file via on-the-fly decompression.
This project will also add extensions to the lofiadm(1M) command to
provide for the ability to compress a file in a suitable format
and to decompress it entirely.
Design Overview
The basic design is to compress a file in fixed size segments and
store the segments sequentially. The information about where each
of the segments start is maintained in an index. This index along
with a header that contains the compression algorithm used,
segment size, number of segments and the uncompressed size of the
last segment is stored at the start of the file.
When a compressed file is mapped, the header containing the
compression metadata is read in and stored in the lofi state
structure. The compression metadata thus read is then used to
decompress the requested data when a read request comes in.
If the decompression fails for some reason, an EIO
is returned.
On disk format
A graphic representation of the compression header stored at the
start of the file can be found under materials/ondisk_format.png
It should be noted that this header information is always stored
in network byte order. This allows for endian neutral filesystems
such as ZFS to be compressed on a little endian machine, and
subsequently be uncompressed on a big endian machine; and
vice-versa.
On the fly decompression
A graphic representation of reading from compressed versus regular
files can be found under materials/compress_io.jpg
Compression Algorithms
The lofi(7D) driver will support compression algorithms present
in the Solaris kernel. Currently, lzjb and gzip are the two
algorithms available.
The lzjb algorithm provides very fast compression as compared
to gzip. gzip, even though it is slower than lzjb, is fast
enough for the typical use case of a LiveCD. In addition, gzip
provides a much better compression ratio as compared to lzjb
(on the order of ~25% in some cases) and it allows for more data
to be packed on to the media. These characteristics make gzip
well suited for a LiveCD. Thus, gzip will be the only supported
algorithm at this time.
The changes being made as part of this project will allow for
compression algorithms to be plugged into lofi with relative
ease as more of these algorithms are made available in the
Solaris kernel.
Command Changes
lofiadm(1M) will take the following additional arguments -
-C Compress the file with a specified compression algorithm
or with the default compression algorithm if one isn't
specified. The only supported compression algorithm at this
time is gzip which is also the default.
-s The segment size to use to divide the file being compressed.
If no segment size is specified, a default segment size of
128k is used.
-U Uncompress a compressed file
The output of lofiadm(1M) will also change to add another field to
reflect if the mapped file(s) is a compressed file or a regular file.
Examples:
1. Compress a file with gzip
# lofiadm -C gzip /export/home/solaris.iso
2. Compress a file with gzip and compress it in 64k chunks
# lofiadm -C gzip -s 64k /export/home/solaris.iso
3. Check if there are any compressed files that are mapped
# lofiadm
Block Device File Type
/dev/lofi/1 /export/home/solaris.iso Compressed (gzip)
/dev/lofi/2 /export/home/regular.iso Regular
4. Uncompress a compressed file
# lofiadm -U /export/home/solaris.iso
References
PSARC/2007/492 - Project Indiana
http://sac.eng/PSARC/2007/492/discussion.materials/Indiana.txt
PSARC/1999/463 - lofi
http://sac.sfbay/Archives/CaseLog/arc/PSARC/1999/463
On the Fly Decompression in BeleniX
http://www.genunix.org/distributions/belenix_site/?q=compression
gzip for ZFS update
http://blogs.sun.com/ahl/entry/gzip_for_zfs_update
6. Resources and Schedule
6.4. Steering Committee requested information
6.4.1. Consolidation C-team Name:
ON
6.5. ARC review type: FastTrack
6. Resources and Schedule
6.4. Steering Committee requested information
6.4.1. Consolidation C-team Name:
ON
6.5. ARC review type: FastTrack
6.6. ARC Exposure: open
