Robots.txt

[BlackIce]

Hi,

I've just seen on a website which tracks bots, that "Tarantula" ,  our
nutch 1.11 based crawler is being classified as not obeying robots.txt.

What's the solution?