+100 from me. On 16 April 2015 at 14:23, Patrick Uiterwijk <[email protected]> wrote:
> +1 from me, though you might need to add a <Directory /var/www/html> with > allow from all. > > > ----- Original Message ----- > > ok. Next thing... I notice that some queries hitting particularly hard > > are from spiders. > > > > Back in the puppet days, koji disallowed all spidering. > > > > I never have seen useful search results from it. > > > > So, I'd like to block all the spiders and see if that helps out load > > issues. > > > > robots.txt would have: > > > > User-agent: * > > Disallow: / > > > > +1s? > > > > kevin > > -- > > diff --git a/roles/koji_hub/files/kojiweb.conf > > b/roles/koji_hub/files/kojiweb.conf index 86abd2e..8334274 100644 > > --- a/roles/koji_hub/files/kojiweb.conf > > +++ b/roles/koji_hub/files/kojiweb.conf > > @@ -6,6 +6,8 @@ KeepAlive On > > Alias /koji "/usr/share/koji-web/scripts/wsgi_publisher.py" > > #(configuration goes in /etc/kojiweb/web.conf) > > > > +Alias /robots.txt /var/www/html/robots.txt > > + > > <Directory "/usr/share/koji-web/scripts/"> > > Options ExecCGI > > SetHandler wsgi-script > > diff --git a/roles/koji_hub/tasks/main.yml > > b/roles/koji_hub/tasks/main.yml index 0b6cd82..a5fc795 100644 > > --- a/roles/koji_hub/tasks/main.yml > > +++ b/roles/koji_hub/tasks/main.yml > > @@ -171,6 +171,13 @@ > > notify: restart httpd > > when: env != "staging" > > > > +- name: koji robots.txt config > > + copy: src=robots.txt dest=/var/www/html/robots.txt > > + tags: > > + - config > > + - koji_hub > > + notify: restart httpd > > + > > - name: kojira log dir > > file: dest=/var/log/kojira owner=root group=root mode=0750 > > state=directory tags: > > > > _______________________________________________ > > infrastructure mailing list > > [email protected] > > https://admin.fedoraproject.org/mailman/listinfo/infrastructure > _______________________________________________ > infrastructure mailing list > [email protected] > https://admin.fedoraproject.org/mailman/listinfo/infrastructure > -- Stephen J Smoogen.
_______________________________________________ infrastructure mailing list [email protected] https://admin.fedoraproject.org/mailman/listinfo/infrastructure
