This is an email to shell account holders on translatewiki.net and to wikitech-l, so that you are informed.
Today at 08:10 UTC Niklas noticed that the translatewiki.net server had been compromised. We saw some suspicious files in /tmp and a few processes that didn't belong: elastic+ 22862 0.0 0.0 2684 2388 ? S 04:53 0:00 /tmp/freeBSD /tmp/freeBSD 1 elastic+ 31575 0.0 0.0 2684 2388 ? S 06:38 0:00 /tmp/freeBSD /tmp/freeBSD 1 elastic+ 31580 16.7 0.0 90816 724 ? Ssl 06:38 16:26 [.Linux_time_y_2] We gathered data and looked at our recent traffic statistics. We drew the following conclusions: - Only the Elasticsearch account had been compromised. The intruder did not gain access to other accounts. - The attack could be made because the Elasticsearch process was bound to all interfaces, instead of only the localhost interface, and dynamic scripting was enabled, because it is required by CirrusSearch (CVE-2014-3120). - A virtual machine was started, and given the traffic that was generated (about 1TB in the past 4 days), we think this was a DDoS drone. The process reported to an IP address in China. - A server reinstall is the right thing to do (better safe than sorry). The compromised server was taken off-line around 10:00 UTC today. Actions taken: - Bind Elasticsearch only to localhost from now on: https://gerrit.wikimedia.org/r/#/c/145262/ - Reinstall the server Actions to be taken: - Configure a firewall to only allow expected traffic to enter and exit the translatewiki.net server so that something like the added virtual machine could not have communicated to the outside world. - As a precaution, shell account holders should change any secret that they have used on the translatewiki.net server in the past 7 days. We are thankful to the people in the MediaWiki security IRC channel and Henri Salo for helping us with data gathering on the attack, and how to proceed. We have re-installed the translatewiki.net server, and are currently re-importing the databases. We expect to be back online in a few hours. Once we come back online, we'll still have to rebuild some non-critical meta data stores, like populating the search database. Cheers! Siebrand _______________________________________________ Wikitech-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikitech-l
