On Wed, Nov 08, 2017 at 05:19:25PM +0100, Stefan Weil wrote: > Am 08.11.2017 um 16:33 schrieb Stefan Hajnoczi: > > Hi Mike and Jeff, > > qemu.org's bandwidth usage is dominated by release tarball downloads. > > This puts qemu.org bandwidth usage in the 2+ TB/month range. > > Hi Stefan, > > how much of this traffic is caused by web spiders? > > From my own binaries I know that the bots of the > different search engines cause most of the traffic, > if they are allowed to do so. > > Usually they respect robots.txt. There is no > https://www.qemu.org/robots.txt currently. > Nor is there a https://download.qemu.org/robots.txt. > Adding both would reduce the downloads, maybe > enough to fix the problem. > > Or do you see an advantage from bots which download > QEMU tarballs? robots.txt can also block only > selected bots. > > Regards > Stefan > > PS. There is a https://git.qemu.org/robots.txt.
Great idea! It's an easy to try adding a robots.txt and check how bandwidth uses changes over the next month. Jeff: Want to try this? Stefan
signature.asc
Description: PGP signature
