I wanted to experiment with having one mail message per file, but I'm
sad to report that, on my kernel, ext3fs becomes unusably slow with
random access to hundreds of thousands of files, even with
`dir_index`, even if the branching factor is only 100 files per
directory. (I'm using Linux 2.6.18-6-686 from Debian Etch with
dm-crypt filesystem encryption on a year-1999 700MHz PIII laptop.
Maybe it wouldn't be a problem with a real computer, or a more recent
kernel.)
An earlier version of this code was responsible for the directory with
800 000 files that I mentioned in a previous kragen-hacks post which
included a shell script to remove that directory. (I had given up on
`rm -rf` after 8 hours.) This version doesn't stress the filesystem
quite as badly.
What I had in mind was to try checking my mailbox into Git, to get
better compression, faster syncing, and automatic detection and
correction of data corruption. The 1.4 version of Git I have doesn't
seem to perform reasonably on the task, but maybe that's because the
underlying filesystem is sucking rocks.
I probably ought to try running this on murdererfs and see if it
performs better; after all, this is the kind of thing it's made for,
right?
Like everything else posted to kragen-hacks without any notice to the
contrary, this program is in the public domain; I abandon any
copyright in it.
#!/usr/bin/python
import sys, os
def makedirs(name):
try: os.makedirs(name)
except OSError, e:
if e.errno == 17: pass
else: raise
class Output:
def __init__(self, dirname):
self.dirname = dirname
self.counter = 0
self.go_to_new_file()
def go_to_new_file(self):
self.counter += 1
dirname = '%s/%02d/%04d' % (self.dirname,
self.counter % 100,
self.counter % 10000)
filename = '%s/message-%s' % (dirname, self.counter)
makedirs(dirname)
self.fo = file(filename, 'w')
def write(self, data): self.fo.write(data)
def close(self): self.fo.close()
def split(mbox, output):
for line in mbox:
if line.startswith('From '):
output.go_to_new_file()
output.write(line)
output.close()
if __name__ == '__main__':
split(sys.stdin, Output(sys.argv[1]))
