Hi Sebastian, yes that I mean. Do you think there is a way to learn more about, how to crawl any website?!
>Hi Ayhan, >you mean? >https://stackoverflow.com/questions/69352136/nutch-does-not-crawl-sites-that-allows-all-crawler-by-robots-txt<https://stackoverflow.com/questions/69352136/nutch-does-not-crawl-sites-that-allows-all-crawler-by-robots-txt>