Kenneth Lakin wrote:
All,
I'm running vss2svn and ssphys (from SVN rev 309) on a Windows XP Pro machine
w/ 2GB of RAM. I compiled perl from the ActiveState 5.8.8 sources. I've enabled
perl's native memory management.
I have two separate issues.
The first:
*Summary:
SanityChecker needs to start writing out its data to disk; perhaps in an SQLite
DB or something.
*Exposition:
I'm processing a SSafe database that has well over 3000 commits.
Everything's fine until the IMPORTSVN phase. As we work through the IMPORTSVN phase,
perl keeps using more and more memory. 'Round about the time that SVN revision 2947 is
being processed, perl's using 1.5GB of ram. vss2svn goes to load a ~250MB file, and
cannot allocate the memory to do so. perl halts, on line 719 of Dumpfile.pm (
$node->{text} = do { local( $/ ) ; <EXP> } ; ) and I'm left with a 9GB SVN dump
file that still has two years of commits to go!
I've looked all through Vss2Svn::DumpFile and associated classes, and made a
short run thru the debugger. Unless I've messed up my perl compilation, I can't
find anything other than SanityChecker that might be eating so much memory.
I'm going to get to work on making SanityChecker write to an SQLite database
today, unless someone has a better idea.
Hi Kenneth, thanks for writing. You're losing me a bit here though. The
issue in Dumpfile.pm in which it pulls the entire file into memory is
certainly an issue (as documented in ticket 25 which you mention) and
one that I would like to fix. Doing a buffered read isn't terribly
difficult, but rewriting it to write the data directly from source to
target file will take some more coding.
However, I don't understand where you're getting to SanityChecker, or
what else you're planning on writing to SQLite. It certainly is odd that
you are seeing steadily increasing memory though; I've never seen it get
to that high of usage although others might have. The number of commits
(~3000) you're dealing with is certainly not huge but if you have
numerous multi-hundred-megabyte files then that could definitely be a
problem. But again, that would be solved by fixing Dumpfile.pm.
Is there a reason you compiled Perl from source, instead of using
ActiveState's binary version?
The second issue:
I have a patch for DumpFile::get_export_contents() that (for me) works better
than the one here:
http://www.pumacode.org/projects/vss2svn/ticket/25
(I'll submit the patch to the list within 24 hrs.)
All in all, this is a really fine tool! Thanks for making it available.
Glad it is working well for you; please feel free to send the patches if
you get it working with less memory. My time to work on this project is
very rare these days but I would definitely be interested in reducing
the overall memory footprint.
toby
_______________________________________________
vss2svn-users mailing list
Project homepage:
http://www.pumacode.org/projects/vss2svn/
Subscribe/Unsubscribe/Admin:
http://lists.pumacode.org/mailman/listinfo/vss2svn-users-lists.pumacode.org
Mailing list web interface (with searchable archives):
http://dir.gmane.org/gmane.comp.version-control.subversion.vss2svn.user
