Thanx to all of you who gave me some hints...
I got some further questions:
Geoff Hutchison:
> >because all the sites to dig are on the local machine. How can I speed
> up?
>
> This is easy. See the various local_url attributes...
> <http://www.htdig.org/attrs.html#local_urls>
>
> If you index directly over the filesystem, you'll avoid HTTP and
> proxy issues all together.
Hm, this does not work, even if I use local_user_urls. Example:
I want to dig basis.shacknet.nu . It is located in
/spool/wwwoffle/http/basis.shacknet.nu
So, I add
local_urls: http://basis.shacknet.nu/ /spool/wwwoffle/http/basis.shacknet.nu
in the htdig config. If I only allow only digging via file system, the site
couldn't be found. Also if I point to /spool/wwwoffle/http/
Could this be because I'm using wwwoffle? (if you don't know wwwoffle: I can
use it as a proxy, I can surf on my local mirrored basis.shacknet.nu like
I'm online)
> >If I run htdig with serverwait_time=0 (standard value), htdig does it job
in about 1 and 1,5 hour. So far so god, but after this, when I'm searching
in my database, a lot of site are missing. In the log I saw that several
>> servers were not digged, in the log-file was something written like "New
server:
> >basis.shacknet.nu no server running". According to this I've set the
> >serverwait_time=1, because I read somewhere this could solve the problem.
>
> Certainly indexing can be done at such a rate as to saturate network
> connections or bump the limit of the number of server processes
> available, etc. There may also be issues between the proxy and the
> htdig indexing (which would be good to know so we can fix them).
>
How can I change the thing with the server processes? I didn't found a
similar config-attribute. Is it a system issue?
Greg Holmes:
>> Why this "no server running"-error? Any suggestions?
>Try this:
>http://www.htdig.org/mail/2000/10/0215.htm
>Hope it helps.
Do you think it solves the problem with local saved sites too? (before I
play arround with sources :) )
greetz n' thanx
db
----------------------------
@ Warp speed do not open the Windows
--
GMX - Die Kommunikationsplattform im Internet.
http://www.gmx.net
_______________________________________________
htdig-general mailing list <[EMAIL PROTECTED]>
To unsubscribe, send a message to <[EMAIL PROTECTED]> with a
subject of unsubscribe
FAQ: http://htdig.sourceforge.net/FAQ.html
