Bley, Josef's bits of Mon, 1 Jul 2002 translated to: >we try to index with htdig 3.1.5 an amount of about 20,000 files (mainly >PDF-Files) on SUN Solaris 2.8 workstation. with 1 GB RAM. >All the URLs of the files are written in a start_url file. Each URL is a >single line in that file. >If the start_url contains about 15,000 URLs (lines) the indexing with htdig >is successful. >But if we extend it to 20,000 URLs, the indexing stops short after the start >with an "xalloc error". >Is there a limit in the lenght of the start_urls or a context to problems >with memory ?
I don't believe there are any htdig limits in this regard. It is a matter of operating system, hardware, and configuration. The xalloc error itself is definitely the result of an OS constraint. >Has anyone an idea, how to avoid this ? Have you checked to see if there are any prohibitive memory (or other resource) limits imposed on the shell from which you are running htdig? For csh-type shells you can check this with the limit command. For sh-type shells you can use something like ulimit -a. If all else fails, you might consider indexing in batches and then merging the results. Jim ------------------------------------------------------- This sf.net email is sponsored by: Jabber - The world's fastest growing real-time communications platform! Don't just IM. Build it in! http://www.jabber.com/osdn/xim _______________________________________________ htdig-general mailing list <[EMAIL PROTECTED]> To unsubscribe, send a message to <[EMAIL PROTECTED]> with a subject of unsubscribe FAQ: http://htdig.sourceforge.net/FAQ.html
