I wouldn't expect that you've got mem problems, probably just not enough of it! Try repeating the exercise and looking at the amount of swap ( /sbin/swapon -a ) and see if you used it all up.
I've recently tried to montage several hundred images into a multi-page page-sized image (thumbnail collation). Physical mem 0.5GB, swap 1GB. Apart from running for ages, when top showed there was nothing more to be had, montage terminated with a memory allocation error. Not the most optimised programming to read all images into memory forst before producing a single byte of output.
I've tried scripting to add one image at a time, rather than all at once, with no joy.
I'm installing swapd to see if that helps.
Roy.
