Re: Crawling and parsing

[Michael Kelleher]

It had nothing to do with configuration at all.

The site had a robots.txt that DENIES all robots from crawling.


That was the issue.