Am 08.11.2017 um 16:33 schrieb Stefan Hajnoczi: > Hi Mike and Jeff, > qemu.org's bandwidth usage is dominated by release tarball downloads. > This puts qemu.org bandwidth usage in the 2+ TB/month range.
Hi Stefan, how much of this traffic is caused by web spiders? >From my own binaries I know that the bots of the different search engines cause most of the traffic, if they are allowed to do so. Usually they respect robots.txt. There is no https://www.qemu.org/robots.txt currently. Nor is there a https://download.qemu.org/robots.txt. Adding both would reduce the downloads, maybe enough to fix the problem. Or do you see an advantage from bots which download QEMU tarballs? robots.txt can also block only selected bots. Regards Stefan PS. There is a https://git.qemu.org/robots.txt.
