On Sun, Aug 21, 2016 at 06:37:04PM +0000, Eric Wong wrote: > "W. Trevor King" <[email protected]> wrote: > > On Fri, Nov 07, 2014 at 11:03:21AM -0800, W. Trevor King wrote: > > > Eric Wong has been working on some tools to store email in a Git > > > repository, and his client-side code is ssoma [1]. I wanted a bit > > > more metadata than the stock ssoma-mda [2], and ended up just > > > writing a ssoma-mda in Python [3]… > > Btw, for public-inbox, I'm using git-fast-import now, so imports are > a bit faster and $GIT_DIR/ssoma.index is no longer used. This was > crucial for getting git@vger archives imported in a reasonable time. > > public-inbox-* still keeps ssoma.index up-to-date for backwards > compatibility with ssoma, and will probably do so until 2020 or > later (there'll be a few years of deprecation notices) > > So I or someone else needs to update Perl ssoma to use fast-import > at some point, too; and I suggest your python version do the same.
ssoma-mda imports 22k notmuch messages in around 15 minutes (with
profiling enabled), and:
$ python -m cProfile -o profile import.py notmuch.mbox
$ python -c "import pstats; p=pstats.Stats('profile');
p.sort_stats('cumulative').print_stats(10)"
Sun Aug 21 12:56:49 2016 profile
101823722 function calls (99078415 primitive calls) in 885.069
seconds
Ordered by: cumulative time
List reduced from 1145 to 10 due to restriction <10>
ncalls tottime percall cumtime percall filename:lineno(function)
70/1 0.002 0.000 885.069 885.069 {built-in method exec}
1 0.111 0.111 885.069 885.069
/home/wking/src/notmuch/notmuch-archives.git/import.py:9(<module>)
1 0.400 0.400 884.915 884.915
/home/wking/src/notmuch/notmuch-archives.git/import.py:17(import_mbox)
22875 0.601 0.000 863.371 0.038
/home/wking/src/notmuch/notmuch-archives.git/ssoma_mda.py:362(deliver)
22875 8.943 0.000 810.459 0.035
/home/wking/src/notmuch/notmuch-archives.git/ssoma_mda.py:207(append)
22875 0.418 0.000 308.353 0.013
/home/wking/.local/lib64/python3.4/site-packages/pygit2/index.py:146(write_tree)
22875 307.855 0.013 307.855 0.013 {built-in method
git_index_write_tree}
22874 0.575 0.000 279.293 0.012
/home/wking/.local/lib64/python3.4/site-packages/pygit2/index.py:238(diff_to_tree)
22874 278.501 0.012 278.501 0.012 {built-in method
git_diff_tree_to_index}
22875 0.088 0.000 80.413 0.004
/home/wking/.local/lib64/python3.4/site-packages/pygit2/index.py:99(read)
38 ms per ssoma delivery is probably fast enough, especially if you
are invoking ssoma-mda once per message, since process setup will take a
similar amount of time:
$ time python -c 'print("hello")'
hello
real 0m0.016s
user 0m0.013s
sys 0m0.003s
It's possible that fast-import would shave a few ms off the pygit2
addition (I'm not sure, and maybe pygit2 is faster than fast-import).
But I doubt it matters enough either way to be worth changing unless
you are dealing with a really large corpus.
Cheers,
Trevor
--
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy
signature.asc
Description: OpenPGP digital signature
