2008/11/21 Marcus Wolschon <[EMAIL PROTECTED]>: > 2008/11/21 Sascha Silbe <[EMAIL PROTECTED]>: >> On Fri, Nov 21, 2008 at 10:59:27AM +0100, Marcus Wolschon wrote: >> >>> I ran into an outOfMemory-Exception memory-mapping the file when importing >>> Baden-Württemberg. >> >> [...] >>> >>> (I did not think this would become an issue for file-sizes of about 100MB) >> >> Did you try it on Windows or a POSIX compatible OS? 32bit or 64bit >> architecture/OS/userspace? >> On 64bit Linux, I can easily mmap() even the whole planet database (~50GB >> total, with largest file being ~30GB) - on 32bit it's less, but still way >> more than a few hundred MBs. > > I will try on a Linux/x86-32 later. ATM I only have an XP on an Eeepc > at my hands. > I could raise the limit to a few hundred MB by using an undocumented argument > to the java-runtime but still , reserving 3GB only allows me to memory-map > 2 files with a sum of 450MB.
Okay, I "found" it. I ran my program on an eeepc while programming on the train. Now I wrote a small test-program that would memory-map a 1 GB-file and write to it. It works fine on a 2GB desktop-PC but fails to aquire the mapping on the eeepc with 2GB. Both run Windows XP. I added a fallback to use normal IO for the index-file if the memory cannot be mapped. It is much, much slower (about 1/10 the speed since in the index I change single integers at random locations in a large file) but it works. I'll do the large files on my big Linux-laptop now. Marcus _______________________________________________ dev mailing list [email protected] http://lists.openstreetmap.org/listinfo/dev
