larsfuse <l...@cl.no> added the comment:
> (...) refers users, for file structure, to > http://www.robotstxt.org/orig.html. This says nothing about the effect of an > empty file, so I don't see this as a bug. That is incorrect. From that url you can find: > The presence of an empty "/robots.txt" file has no explicit associated > semantics, it will be treated as if it was not present, i.e. all robots will > consider themselves welcome. So this is definitely a bug. ---------- _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue35457> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
