- - - - - - - - - - - - - - - - - - - - - - - - - - - -
Name: mico
Subject: robots.txt: 'Disallow /'
I'm trying to get some pages from a Server that has in robots.txt:
User-agent: *
Disallow: /
I did put in indexer.conf:
Robots no
But it didn't change anything I still get the same: "robots.txt: 'Disallow /'"
from the indexer.
What could i do to index this site ?
LOGS:
------
indexer[9296]: {00} indexer from dpsearch-4.47-mysql started with
'/usr/local/dpsearch/etc/indexer.conf'
indexer[9296]: {01} URL: http://some.site.com/
indexer[9296]: {01} ROBOTS: http://some.site.com/robots.txt
indexer[9296]: {01} robots.txt: 'Disallow /'
indexer[9296]: {01} Deleting http://some.site.com/
indexer[9296]: {01} Done (1 seconds, 0 documents, 0 bytes, 0.00 Kbytes/sec.)
indexer[9296]: {00} Total 8 seconds, 0 documents, 0 bytes, 0.00 Kbytes/sec,
0.00 sec/doc, 0 bytes/doc.
indexer[9296]: {00} Neo PopRank: 0 documents, 0 pas, 0.00 Kpas/sec, 0.00
sec/doc, 0.00 pas/doc.
indexer[9296]: {00} Mon 30 20:40:48 [9296] Flushing all buffers...
- - - - - - - - - - - - - - - - - - - - - - - - - - - -
Read the full topic here:
http://www.dataparksearch.org/cgi-bin/simpleforum.cgi?fid=02;post=
