This is an automated email from the ASF dual-hosted git repository. snagel pushed a commit to branch NUTCH-1999-nutch-site-robots-txt in repository https://gitbox.apache.org/repos/asf/nutch-site.git
commit 142489f5f05d955f2140bffa6a82df90cd5cc359 Author: Sebastian Nagel <sna...@apache.org> AuthorDate: Sun Jan 9 13:53:35 2022 +0100 NUTCH-1999 Add /robots.txt to Nutch site --- content/robots.txt | 2 ++ 1 file changed, 2 insertions(+) diff --git a/content/robots.txt b/content/robots.txt new file mode 100644 index 0000000..c2a49f4 --- /dev/null +++ b/content/robots.txt @@ -0,0 +1,2 @@ +User-agent: * +Allow: /