Hi Brandon, if Tilman's hints don't help, you might try to spilt your site into different sections and check them one at a time. You can use the "Do not check any URLs beginning with this:" to do this, if the site is structured in any way. Cheers Thomas
_____ Von: [email protected] [mailto:[email protected]] Im Auftrag von brandonmbyers01 Gesendet: Mittwoch, 24. Juni 2009 22:52 An: [email protected] Betreff: [xenu-usergroup] crawling a large site I've used Xenu for years, and it's an outstanding program. I've run into a problem, though: when it gets much beyond 500,000 total URL's (only 15-35% visited), my computer tells me the virtual memory is too full. I've got 1 GB of RAM, and the task manager claims ~ 2 GB are being used. I'd like to know what others have experienced when crawling a site of that size (or larger). Is it my hardware? Should I increase the virtual memory? I tend to always run it 1 thread at a time, so that my computer (and the server) don't get too bogged down. I imagine if I ran it much higher, I'd have a bigger problem with giant sites ... right? Anyway, thanks to Tilman, and thanks to everyone who's helped with improvements to the software. - Brandon
