http://osxbook.com/software/hfsdebug/
A Debugger for HFS Plus Volumes
© Amit Singh. All Rights Reserved.
Written in May 2004
What's New?
To learn what's new with hfsdebug, go to the Further
Reading section of this page.
hfsdebug
A debugger is a program that facilitates debugging, which
may be casually defined as the process of finding and fixing "bugs" in
the object of interest: usually a piece of software, firmware, or
hardware (although debugging may apply to any domain).
hfsdebug is a filesystem
debugger for Apple's HFS Plus (hereafter referred to as HFS+)
volume format. The term "debugger" may be somewhat of a misnomer for hfsdebug,
because a key feature (rather, a limitation) of it is that it operates
on an HFS+ volume only in read-only
mode. Therefore, it cannot currently be used by itself to repair any
defects in a volume, or to even write to a volume in any manner
whatsoever. Nevertheless, hfsdebug is meant to be a
useful tool in exploring HFS+ internals, as it allows you to browse,
inspect, and analyze various aspects of HFS+ volumes. It can also
calculate and display certain filesystem statistics. My original intent
in creating hfsdebug was to quantify filesystem
fragmentation under HFS+.
hfsdebug will often display many more details than are
viewable via "usual" methods. Moreover, read-only operations that
involve going through all objects on a volume should be faster with hfsdebug
(for example, creating a list of the N largest, or the N most
fragmented files on a volume). Still, hfsdebug is not
supposed to be a replacement for standard file utilities, or of any
existing API.
Volume Formats Supported
hfsdebug supports only the HFS+ volume format. The older
HFS format is not supported. It does, however, support the
following HFS+ variants:
- HFS Plus
- Journaled HFS Plus
- HFS Plus (journaled or otherwise) embedded within an HFS wrapper
- HFSX
Note that hfsdebug should work with HFS+ volumes
regardless of whether they reside on a real, physical disk (or disk
slice), or on a "virtual" disk (such as a disk image).
Architecture
hfsdebug aims to provide accurate and complete details of
various HFS+ internal structures, as well as filesystem statistics,
most of which are not available through an API such as POSIX (or even
the Carbon File Manager, which is far more flexible than POSIX when
dealing with HFS+ volumes). hfsdebug works on an HFS+
volume by accessing its raw device directly. This means several things:
- You may need superuser access to use
hfsdebug on
certain volumes, in particular, the root volume on a Mac OS X system.
- You can use
hfsdebug even on a mounted (live) HFS+
volume, in particular, the root volume on a running Mac OS X system.
- Since
hfsdebug does not access the volume's data
through a block device, or some higher level API, its operation does not
trash the system's buffer cache (it wouldn't be a big deal in any case,
because hfsdebug is read-only).
- Although
hfsdebug is written on, and for Mac OS X,
it
does not use any Mac OS X specific features. It is a C program that
uses the POSIX API, and should be readily portable to any Unix-like
operating system.
When you run hfsdebug as a superuser (via sudo,
say), it "drops" its privileges once it has opened a volume's raw
device. Specifically, it sets its user and group id's to 99 and 99,
respectively (that is, to those of the "unknown" user on Mac OS X).
Features
You can use hfsdebug to:
- View the Volume Header and the Master Directory Block (in case of
"embedded" HFS+ volumes).
- View contents of the header nodes of HFS+ "special files". These
files include the Catalog, Overflow Extents, and Hot Files B-Trees.
- List and view details of one or more types of records contained
in the special file B-trees, such as:
- File records, file thread records, folder records, and folder
thread records (the Catalog B-Tree).
- Extent records (the Extents Overflow B-Tree).
- Hot file records and hot file thread records (the Hot File
Clustering B-Tree).
- View detailed information about filesystem objects, such as
files,
folders, aliases, symbolic links, and hard links. An object may be
looked up using its Catalog Node ID (typically, but not always, the
same as the inode number reported by the POSIX API), a Carbon style
specification (the object's "node name" and the Catalog Node ID of its
parent), or using its POSIX path.
- View detailed information contained in a volume's journal, if any.
- Calculate and display volume statistics, such as:
- Top N files ordered by size.
- Top N files ordered by their degree of fragmentation.
- Top N files ordered by how "hot" they are (in the context of
Hot File Clustering).
- A
summary of the number of various kinds of filesystem objects present on
a volume, their space usage, special cases such as invisible and empty
files, and so on.
- Display details of all fragmented files on a volume.
- Display location and size of all free extents on a
volume.
Usage
Specifying a volume
hfsdebug needs an HFS+ volume to work on. This volume may
be specified in three ways:
- Implicitly: if no volume is explicitly specified on the command
line,
hfsdebug will attempt to determine the
"appropriate" volume. If an option requires a filesystem object as an
argument, then hfsdebug
automatically uses the volume containing that object. If no volume is
specified, and no option is present from which the volume can be
deduced, hfsdebug uses the root volume. Thus, both the
following commands will result in the root volume being used:
# hfsdebug /mach_kernel OPTIONS ... # hfsdebug OPTIONS ...
- Using the
-d option which takes a string naming the
volume's device. For active (mounted) volumes, this must be
the raw device, but may be the block device for unmounted volumes.
# hfsdebug -d /dev/rdisk0s9 OPTIONS ...
- Using the
-V option which takes a pathname to the
volume's mount point, or any filesystem object within it.
# hfsdebug -V /Volumes/Music OPTIONS ... ... # hfsdebug -V
/Volumes/Music/somefile.mp3 OPTIONS ...
Viewing the Volume Header
The following command displays the volume header of the specified
device (in this example, the root device). If the volume has an HFS
wrapper, the master directory block is also displayed. This option is
similar to "hdiutil hfsanalyze", except that hfsdebug
shows some additional details, while not showing the "alternate"
structure(s).
# hfsdebug -v
# HFS Plus Volume with HFS Wrapper Embedded offset = 11984 bytes
Wrapper volume size = 78149032.00 KB/76317.41 MB/74.53 GB Embedded
volume size = 78143052.00 KB/76311.57 MB/74.52 GB
# HFS Wrapper Master Directory Block drSigWord = $4244 (BD)
...
# HFS Plus Volume Header signature = 0x482b (H+)
... # Finder Info finderInfo[0] = 0x1077 (Macintosh
HD:/System/Library/CoreServices) finderInfo[1] = 0 finderInfo[2] = 0
finderInfo[3] = 0x30894 (Macintosh HD:/System Folder)
... # Catalog File logicalSize = 251658240 bytes totalBlocks = 61440
fork temperature = /* Metadata Zone */ clumpSize = 8388608 bytes
extents = startBlock blockCount % of file 0xc01 0x5800 36.67 % 0xed8c
0x800 3.33 % 0x1e70a 0x800 3.33 % 0x2add4 0x1000 6.67 % 0x4d4b4 0x800
3.33 % 0x55c39 0x800 3.33 % 0xdddb2 0x800 3.33 % 0x140b0e 0x800 3.33 %
38912 allocation blocks in 8 extents total. 4864.00 allocation blocks
per extent on an average.
...
Viewing headers of the Volume B-Trees
HFS+ uses several B-Trees for storing metadata. hfsdebug
allows you to view the contents of the header nodes of these data
structures:
# hfsdebug -b catalog
...
# hfsdebug -b extent
...
# hfsdebug -b hotfile
# HFS+ Hot File Clustering (HFC) B-Tree
# B-Tree Node Descriptor fLink = 0 bLink = 0 kind = 1 (kBTHeaderNode)
height = 0 numRecords = 3 reserved = 0
# B-Tree Header Record treeDepth = 2 rootNode = 3 leafRecords = 21244
firstLeafNode = 88 lastLeafNode = 1 nodeSize = 4096 bytes maxKeyLength
= 10 bytes totalNodes = 144 freeNodes = 28 reserved1 = 0 clumpSize =
65536 (ignored) btreeType = 128 (kUserBTreeType) keyCompareType = 0
(unspecified/default) attributes = 00000000000000000000000000000010 .
kBTBigKeys (keyLength is UInt16)
# User Data Record magic = 0XFF28FF26 version = 1 duration = 216000
seconds timebase = Sun May 2 09:02:27 2004 timeleft = 79068 seconds
threshold = 16 maxfileblks = 2560 blocks maxfilecnt = 1000 tag =
CLUSTERED HOT FILES B-TREE
Note that hfsdebug only knows about the on-disk
structures -- not their in-memory counterparts. If some information
exists only in memory, hfsdebug will not display it until
it is flushed to disk.
Viewing records contained in the Volume B-Trees
hfsdebug lets you list details of all leaf records
residing in the Volume B-Trees. Since some trees have more than one
type of leaf records, you may also specify the type(s) of records to
list:
# hfsdebug -b catalog -l file ... # hfsdebug -b catalog -l file
-l folder ... # hfsdebug -b catalog -l filethread ... # hfsdebug -b
hotfile -l hfcfile ... # hfsdebug -b hotfile -l hfcthread ... #
hfsdebug -b extent -l any ...
The "any" keyword causes all records to be listed for the specified
B-Tree. Note that this may result in a very large amount of output. For
example, running "hfsdebug -b catalog -l any" produces
almost a Gigabyte of data for a volume with about 600,000 files.
Viewing details of filesystem objects
hfsdebug can be used to view details of files, folders,
and other kinds of objects on HFS+ volumes, including those that are
not exposed by the filesystem to the user (for example, indirect node
files, hard link files, journal files, etc.) You may specify
an object in three ways:
- Its POSIX visible path (specified implicitly as the
last argument, or using the
-p option)
# hfsdebug /usr/bin/gcc ... # hfsdebug -p
/bin/ls ...
- Its Catalog Node ID (specified using the
-c option)
# hfsdebug -c 2 ...
- Its name and the Catalog Node ID of its parent (specified using
the
-F option)
# hfsdebug -F 2:.journal ...
The third way is needed when an object's POSIX path is not visible at
the user-level, and you don't know its Catalog Node ID: for example,
the journal files.
# hfsdebug
/System/Library/CoreServices/BootX path = Macintosh
HD:/System/Library/CoreServices/BootX
# Catalog File Record type = file file ID = 186444 flags =
0000000000000010 . File has a thread record in the catalog. reserved1 =
0 createDate = Mon Oct 6 19:28:33 2003 contentModDate = Fri Mar 26
04:23:24 2004 attributeModDate = Fri Mar 26 04:23:24 2004 accessDate =
Mon Oct 6 20:08:00 2003 backupDate = Fri Jan 1 00:00:00 1904 # BSD Info
ownerID = 0 (root) groupID = 0 (wheel) adminFlags = 00000000 ownerFlags
= 00000000 fileMode = -rw-r--r-- linkCount = 0 textEncoding = 0
reserved2 = 0 # Finder Info fdType = 0x74627869 (tbxi) fdCreator =
0x63687270 (chrp) fdFlags = 0000000000000000 fdLocation = (v = 0, h =
0) opaque = 0 # Data Fork logicalSize = 157892 bytes totalBlocks = 39
fork temperature = not a hot file clumpSize = 0 bytes extents =
startBlock blockCount % of file 0x4bfd6e 0x27 100.00 % 39 allocation
blocks in 1 extents total. 39.00 allocation blocks per extent on an
average. # Resource Fork logicalSize = 0 bytes
Note that as would be the correct behavior for a "debugger" of this
kind, hfsdebug
does not follow hard links and symbolic links automatically. It does,
however, display the link target of a hard or symbolic link.
Viewing a journaled volume's journal
hfsdebug can be used to view information about, and dump
the contents of, a volume's journal.
# hfsdebug -j
# HFS+ Journal
# Journal Info Block flags = 00000000000000000000000000000001 . Journal
resides on local volume itself. device_signature = 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 offset = 104861696 bytes size = 8388608 bytes reserved =
0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000
# Journal Header magic = 0x4a4e4c78 endian = 0x12345678 start = 5090816
bytes end = 3207680 bytes size = 8388608 bytes blhdr_size = 12288 bytes
checksum = 0x802a54c0 jhdr_size = 512 bytes # Block List Header
max_blocks = 767 num_blocks = 2 bytes_used = 20480 checksum = 0 pad = 0
binfo[0].bp = 0 block_info[ 1] { bnum 0000000000a0b010 bsize 8192 bytes
bp 0x17d5477c } ... output continues ...
Viewing the "hottest" files on a volume
We saw that you can list hot file records using "hfsdebug -b
hotfile -l ...". You can also use hfsdebug
to list the top N hottest files, as tracked and determined by Apple's
Hot File Clustering scheme, that is available on Mac OS X 10.3.x boot
volumes:
# hfsdebug -H -t 10
# Top 10 Hottest Files on the Volume
rank temperature cnid path
1 469 9165 Macintosh HD:/private/var/db/netinfo/local.nidb/Store.128
2 372 237719 Macintosh HD:/usr/libexec/gcc/darwin/ppc/3.3/specs
3 359 1047016 Macintosh HD:/private/var/vm/app_profile/0_names
4 337 238241 Macintosh HD:/usr/libexec/gcc/darwin/ppc/3.3-fast/specs
5 305 9133 Macintosh HD:/usr/share/zoneinfo/US/Pacific
6 305 9166 Macintosh HD:/private/var/db/netinfo/local.nidb/Store.160
7 282 250281 Macintosh HD:/usr/include/ppc/types.h
8 282 250287 Macintosh HD:/usr/include/sys/appleapiopts.h
9 282 250295 Macintosh HD:/usr/include/sys/cdefs.h
10 282 250310 Macintosh HD:/usr/include/sys/fcntl.h
610 active Hot Files.
Viewing volume usage summary
hfsdebug can be used to calculate and display various
usage statistics about a volume. It can optionally be instructed to
list the top N largest sized files on the volume.
# Volume Summary Information files =
563142 folders = 89762 aliases = 4 hard links = 4394 symbolic links =
9856 invisible files = 602 empty files = 5269 # Data Forks non-zero
data forks = 556267 fragmented data forks = 408 allocation blocks used
= 11629019 allocated storage = 47632461824 bytes (46516076.00
KB/45425.86 MB/44.36 GB) actual usage = 46214263456 bytes (45131116.66
KB/44073.36 MB/43.04 GB) total extent records = 556319 total extent
descriptors = 557328 overflow extent records = 52 overflow extent
descriptors = 347 # Resource Forks non-zero resource forks = 7570
fragmented resource forks = 18 allocation blocks used = 93739 allocated
storage = 383954944 bytes (374956.00 KB/366.17 MB/0.36 GB) actual usage
= 363034266 bytes (354525.65 KB/346.22 MB/0.34 GB) total extent records
= 7570 total extent descriptors = 7598 overflow extent records = 0
overflow extent descriptors = 0
# Largest Files on the Volume
# Using 1 KB = 1024 bytes, 1 MB = 1024 KB, 1 GB = 1024 MB
rank size cnid path
1 700.20 MB 988570 Macintosh HD:/work/Linux/KNOPPIX.toast
... output continues ...
Quantifying fragmentation on a volume
hfsdebug can be used to quantify how fragmented an HFS+
volume is, which was my original reason for creating it.
The -f option causes hfsdebug to list all
files (forks, to be precise) that have non-zero fragmentation. Thus,
any data or resource fork that occupies more than one extent is listed,
along with the number and size of each of its fragments. Moreover, you
can instruct hfsdebug to create a list of the top N files
with the most extents.
Refer to Fragmentation in HFS+ Volumes
for a more detailed discussion on this issue.
# hfsdebug -f -t 5
...
cnid=2364913 fork=data map=:1728:773: bytes=10241935 blocks=2501
extents=2 \
avg=1250.50 blks/ext path=Macintosh HD:/Applications/3rd
Party/Microsoft \
Office 2004/Shared Applications/Proofing Tools/German Grammar
Dictionary
cnid=2364937 fork=data map=:192:168: bytes=1473424 blocks=360 extents=2
\
avg=180.00 blks/ext path=Macintosh HD:/Applications/3rd Party/Microsoft
\
Office 2004/Shared Applications/Proofing Tools/Norwegian NYN Dictionary
cnid=2364952 fork=data map=:240:333: bytes=2346969 blocks=573 extents=2
\
avg=286.50 blks/ext path=Macintosh HD:/Applications/3rd Party/Microsoft
\
Office 2004/Shared Applications/Proofing Tools/Portuguese Thesaurus
Dictionary
cnid=2365831 fork=data map=:33:5:97:1:30:67:3:1:2:1:7:2:3:7:1:5:36: \
bytes=1229704 blocks=301 extents=17 avg=17.71 blks/ext path=Macintosh \
HD:/private/var/vm/app_profile/0_data
...
# Top 10 Files with the Most Extents on the Volume
rank extents blk/extents cnid path
1 694 28.43 1881709 Macintosh HD:/sw/var/mysql/lxr/useage.MYI
2 371 71.37 1881710 Macintosh HD:/sw/var/mysql/lxr/useage.MYD
3 281 104.16 1881707 Macintosh HD:/sw/var/mysql/lxr/symbols.MYD
4 267 61.36 2369116 Macintosh HD:/private/tmp/hfs+.dmg
5 104 78.33 45746 Macintosh HD:/System/Library/Caches/fontTablesAnnex
Out of 527454 data forks total, 526428 (99.81 %) have no fragmentation.
Out of 8119 resource forks total, 8081 (99.53 %) have no fragmentation.
Qualifying All Free Space
hfsdebug can go through the HFS+ Allocation File and
enumerate all free extents: an indicator of "free space
fragmentation", if you will.
The -0 option will print a list of triplets for the
volume, with each triplet consisting of the number of free contiguous
extents, the block on the volume at which the extent starts, and the
amount of space it represents. For example:
# hfsdebug -0
# Free Contiguous Starting @ Space 128928 68520 503.62 MB 1 197456 4.00
KB 1 197462 4.00 KB 1 197920 4.00 KB 1 198191 4.00 KB
... 15412 12819025 60.20 MB 6701325 12834438 25.56 GB
Allocation block size = 4096 bytes
Allocation blocks total = 19535763 (0x12a1793)
Allocation blocks free = 8163575 (0x7c90f7)
Note that the free extents list is printed in block order: it is not
sorted by the amount of free space the extents represent. You can
obtain a sorted listed by piping the output through sort:
# hfsdebug -0 | sort -n
... 31850 12611790 124.41 MB 45861 12689954 179.14 MB 81709 213206
319.18 MB 128928 68520 503.62 MB 6701325 12834438 25.56 GB
This way, you can list the free extents with the biggest ones last.
In this example, we can see that the largest amount of contiguous free
space on the volume is 25.56 GB.
Further Reading
I add new features to hfsdebug every once in a while. To
know what more you can do with hfsdebug, take a look at
the following links.
Summary of Options
hfsdebug: Command-line Options
Download
You must read and agree to this site's terms
and conditions before downloading or using any software or other
material available from this site.
HFSDebug-Lite-4.33.dmg
The disk image contains a Rosetta-compatible executable that runs on
Tiger, Leopard, and Snow Leopard. The SHA1 sum of the hfsdebug
executable is:
5974f2c8ab03fc026a7a41e9c0942ced699f441d
