Hi,
I have a general question. Is the Web connector supporting sitemap files
referenced by the robots.txt? In my use case the robots.txt is stored in
the root of the website and is referencing two compressed sitemaps.
Example of robots.txt
------------------------
User-Agent: *
Disallow:
Sitemap: https://www.example.de/sitemap/de-sitemap.xml.gz
Sitemap: https://www.example.de/sitemap/en-sitemap.xml.gz
When start crawling in „Simple History" there is an error log entry as
follows:
Unknown robots.txt line: 'Sitemap:
https://www.example.de/sitemap/en-sitemap.xml.gz'
Is there a general problem with sitemaps at all or with sitemaps
referenced in robots.txt or with compressed sitemaps?
Best regards
Sebastian
