[dataparksearch] [Forum] Re: robots.txt:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - Name: seirge Subject: Re: robots.txt: The same issue. here is -v5 output indexer[13496]: {00} indexer from dpsearch-4.48-mysql started with search/4.48//etc/indexer.conf' indexer[13496]: {00} Chinese dictionary with 0 entries indexer[13496]

[dataparksearch] [Forum] Re: robots.txt:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - Name: Maxime Subject: Re: robots.txt: Yes, absolutely right. - - - - - - - - - - - - - - - - - - - - - - - - - - - - Read the full topic here: http://www.dataparksearch.org/cgi-bin/simpleforum.cgi?fid=02;topic_id=1185818418;page=2

[dataparksearch] [Forum] Re: robots.txt:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - Name: mico Subject: Re: robots.txt: > At 15:49:42 02/08/07, Maxime wrote: >Allow/Disallow commands are looking in order of appearance, and only the first >found applies. So "Disallow *.cgi" will still exclude *.cgi in this case. So if I h

[dataparksearch] [Forum] Re: robots.txt:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - Name: Maxime Subject: Re: robots.txt: Allow/Disallow commands are looking in order of appearance, and only the first found applies. So "Disallow *.cgi" will still exclude *.cgi in this case. - - - - - - - - - - - - - - - - - - - - - - - - -

[dataparksearch] [Forum] Re: robots.txt:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - Name: mico Subject: Re: robots.txt: Results: Aug 2 11:51:07 prole indexer[17364]: {01} Allow method is used Aug 2 11:51:07 prole indexer[17364]: {01} No conditional subsection detected Aug 2 11:51:07 prole indexer[17364]: {01}

[dataparksearch] [Forum] Re: robots.txt:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - Name: mico Subject: Re: robots.txt: Thank you for that... I just tested and it works. Unfortunatly... All it changed is the message that now says: "robots.txt support is disallowed for 'my.site.com'" The result is the same. Example: I

[dataparksearch] [Forum] Re: robots.txt:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - Name: Maxime Subject: Re: robots.txt: Please reindex this page with -v5 option for indexer, this enables maximal debug output, include why every link is accepted or rejected. - - - - - - - - - - - - - - - - - - - - - - - - - - - - Read the

[dataparksearch] [Forum] Re: robots.txt:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - Name: Maxime Subject: Re: robots.txt: "Robots no" command has been fixed in latest snapshot: http://www.dataparksearch.org/dpsearch-4.48-01082007.tar.gz - - - - - - - - - - - - - - - - - - - - - - - - - - - - Read the full topic here: http:

[dataparksearch] [Forum] Re: robots.txt:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - Name: mico Subject: Re: robots.txt: Allright, no problem, thanks! - - - - - - - - - - - - - - - - - - - - - - - - - - - - Read the full topic here: http://www.dataparksearch.org/cgi-bin/simpleforum.cgi?fid=02;topic_id=1185818418

[dataparksearch] [Forum] Re: robots.txt:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - Name: Maxime Subject: Re: robots.txt: I need time to verify such behavior, I'll check it today later or tomorrow. - - - - - - - - - - - - - - - - - - - - - - - - - - - - Read the full topic here: http://www.dataparksearch.org/cgi-bin/simple

[dataparksearch] [Forum] Re: robots.txt:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - Name: mico Subject: Re: robots.txt: Any other suggestions? - - - - - - - - - - - - - - - - - - - - - - - - - - - - Read the full topic here: http://www.dataparksearch.org/cgi-bin/simpleforum.cgi?fid=02;topic_id=1185818418

[dataparksearch] [Forum] Re: robots.txt:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - Name: Maxime Subject: Re: robots.txt: Where you have placed "Robots no" in your indexer.conf ? It should be before Server/Realm command that it should affects. - - - - - - - - - - - - - - - - - - - - - - - - - - - - Read the full topic here

[dataparksearch] [Forum] Re: robots.txt:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - Name: mico Subject: Re: robots.txt: Actually, I got some everywhere (Beginning, Middle, Before Server, End of file, etc.) :) But here's what i just tested in indexer.conf: Robots no Server http://some.site.com/ I don't have any Realm

[dataparksearch] [Forum] Re: robots.txt:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - Name: mico Subject: Re: robots.txt: and... I dumped manually the table 'robots' in search DB. I deleted the whole DB and created it again. But nothing changes, robots.txt is still not ignored. Thx for help, cheers, mico - - - - -