Package: encfs Version: 1.5.2-1 Severity: normal I am trying to use encfs in a way that has me rsync a --reverse encfs to an ext3 filesystem on an untrusted machine, using the script at the end of this bug report. The problem I am having would also apply in the non-reverse case, assuming that the underlying filesystem is ext2/3.
The problem is that rsync (or tar) fail to copy all entries in large
directories (50,000+ entries), because apparently the directory
index (dir_index feature of ext2/3) gets exhausted.
In that case, the kernel writes to dmesg:
EXT3-fs warning (device sdb1): ext3_dx_add_entry: Directory index full!
and tar/rsync report ENOSPC ("No space left on device") on open(),
rename(), mkstemp() and anything else that modifies the directory.
Strangely, however, I can create additional files manually in those
directories. I cannot explain that.
http://www.linux-archive.org/ext3-users/90496-ext3_dx_add_entry-directory-index-full.html
has the best discussion of the problem I know. I don't know of a way
to increase the directory hash size.
Anyway, the problem is a function of encfs, which inflates the
filenames. Notably, the problem occurs with block-encrypting
filenames, *as well as* stream encryption.
Arguably, encfs might simply not be usable for this use-case, but on
the other hand I think that it wouldn't be too hard to solve this
problem, for instance by hashing each directory transparently.
A trivial implementation might be the following: since encrypted
filenames seem to be made up of letters, digits, and some special
characters, let's assume the set of possible characters is
26+10+6==42. It would already help if each directory had 42
single-letter/digit subdirectories and files would be sorted into
those accordingly.
An alternative might be to store all files in a giant 3-4-level
directory hash structure and to maintain an (encrypted) database of
filename -> hashed file mappings. In --reverse mode, this database
would have to be virtual and simulated by the encfs code.
The backupninja script is this:
set -e
LOCKFILE=/var/lock/backupninja-rsync-sergei.lock
trap "rm -f $LOCKFILE" 0 1 2 3 4 5 6 7 8 10 11 12 13 14 15
if ! lockfile -0 -r0 -l 3600 "$LOCKFILE" 2>/dev/null; then
echo "Fatal: another rsync process is still holding the lock" >&2
trap - 0 1 2 3 4 5 6 7 8 10 11 12 13 14 15
exit 1
fi
die() {
echo Fatal: "$1" exited with an error >&2
exit 1
}
DIR="$(mktemp -d -t encfs-rsync-mountpoint.XXXXXXXX)"
trap "[ -d $DIR ] && fusermount -z -u $DIR && rmdir $DIR; rm -f $LOCKFILE" 0
1 2 3 4 5 6 7 8 10 11 12 13 14 15
echo xxxxxxxxx | \
encfs --reverse --stdinpass /srv/backups "$DIR" || die encfs mount
rsync --archive --one-file-system --hard-links --acls --xattrs \
--delete-during --rsync-path='rsync --fake-super' \
"$DIR"/ dest:/path/to/destination/dir \
|| die rsync
set +e
[ -d "$DIR" ] && fusermount -z -u "$DIR" && rmdir --ignore-fail-on-non-empty
$DIR
rm -f $LOCKFILE
trap - 0 1 2 3 4 5 6 7 8 10 11 12 13 14 15
-- System Information:
Debian Release: squeeze/sid
APT prefers unstable
APT policy: (500, 'unstable'), (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Kernel: Linux 2.6.30-rc5-amd64 (SMP w/1 CPU core)
Locale: LANG=en_GB, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Versions of packages encfs depends on:
ii fuse-utils 2.7.4-2 Filesystem in USErspace (utilities
ii libboost-filesystem1.38.0 1.38.0-7 filesystem operations (portable pa
ii libboost-serialization1.38.0 1.38.0-7 serialization library for C++
ii libc6 2.9-19 GNU C Library: Shared libraries
ii libfuse2 2.7.4-2 Filesystem in USErspace library
ii libgcc1 1:4.4.0-10 GCC support library
ii librlog1c2a 1.3.7-1.2 flexible message logging library
ii libssl0.9.8 0.9.8k-3 SSL shared libraries
ii libstdc++6 4.4.0-10 The GNU Standard C++ Library v3
encfs recommends no packages.
encfs suggests no packages.
-- no debconf information
--
.''`. martin f. krafft <[email protected]> Related projects:
: :' : proud Debian developer http://debiansystem.info
`. `'` http://people.debian.org/~madduck http://vcs-pkg.org
`- Debian - when you have better things to do than fixing systems
digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
