Oops… I forgot to link to Microsoft’s page on the Bing robots.txt tester: https://www.bing.com/webmasters/help/robots-txt-tester-623520ca
-- Ron Pool From: [email protected] <[email protected]> Sent: Sunday, November 13, 2022 9:22 AM To: [email protected] Subject: RE: [M100] Bitchin' 100 on Bing robots.txt on bitchin100.com has this contents: User-agent: * Disallow: I’ve not done web publishing in many years. I’ve never tried using a blank string in robots.txt for a pattern to match for either Disallow: or Allow: in a robots.txt file. I wonder if bing/duckduckgo consider a blank pattern to be one that matches any path, and google considers a blank pattern to not match any path. Since you don’t seem to be using robots.txt to keep any parts of your site from being indexed, why not just get rid of your robots.txt file> Or have it be empty if you can’t get rid of it? Then if some site crawlers/indexers interpret the meaning of a blank pattern in different ways, you won’t end up telling them to ignore things you didn’t mean to tell them to ignore. Then again, this might not have anything at all to do with your site not being indexed by bing/ddg. Maybe you did get put on a list of sites they don’t like for some reason. Bing has a robots.txt tester that supposedly shows you if a robots.txt file will block a URL from being indexed. But you have to be able to prove to Microsoft that you have publishing access to your site before you can use Bing’s robots.txt tester with your site. (“Add your site to Bing Webmaster tools by making detectable changes on your site.”). -- Ron Pool
