We had an e-mail exchange today with the FBI about a bot coming to Cryptome daily to grab new files and to check on old ones. We complained about the bot taking over the site during its visit. The FBI explained that the visits are part of a "web cache mirroring process:"
http://cryptome.org/fbi-web-mirror.htm We would appreciate hearing of other sites being mirrored by the FBI. The IP address range of the MAS program: 65.207.53.xxx We have no objection to the FBI mirroring or by anybody else, but misconfigured bots have been an annoyance for repeatedly downloading old files, racking up tens of thousands error logs. One today produced over 16MB of log files before we got the user to wake up. Misconfigured WGet, as often, was the culprit.
