For the sake of completeness I've included Alex Osborne's analysis of the situation below. (Alex runs Clojars.)
-Phil -------------------------------------------------------------------- The really annoying thing about security is it's impossible to conclusively prove at any time anything is secure if you assume a sufficiently sophisticated attacker. All we can do is best effort. Check my logic here. There are four categories of files to verify. "The obvious jar overwrite": When a file is overwritten such that it's mtime or size changed Ivan's backups move the original version into a ./overwritten/ directory for that timestamp. This means that any files that were obviosuly overwritten after April 1 are in this subset: https://gist.github.com/ato/459ee535125e6a5b5489[1] We could try to verify these with jar diffs. "The sneaky jar overwrite": However if a file was overwritten such that the mtime and size didn't change, then Ivan would never have fetched the new version. We can verify these by testings against Ivan's current sha1s. I have done this and found no discrepancies. The only files that didn't match were the expected, those updated today after Ivan had calculated his hashes: ./nsfw/lein-template/0.5.2/lein-template-0.5.2.jar ./antler/lein-caribou/2.1.2/lein-caribou-2.1.2.jar "The obvious new jar": The attacker may not have overwritten an existing jar but added a new one (new version of a popular library say). I don't see how we can verify these (as being legit or not). This would have a lower value to the attacker compared to overwriting a popular existing version. "The sneaky new jar": If a new file was created but backdated (with an mtime in the past) Ivan will have downloaded it but the "-a" option to rsync will mean his timestamps are also backdated. This might be detectable by diffing Ivan's copies of the all-jars metadata file I guess. ------ Now rather than modifying the repository immediately the attacker could have installed a mechanism for uploading a malicious file at a later date: * change passwords, ssh keys, groups or email addresses in clojars db * steal the SSL private key (for later use in MITM attacks) * backdoor the VM db: I don't have an offsite db backup recently predating the attack. The closest thing I have is a backup taken with Linode's own backup infrastructure on April 7 which is too late to conclude much from. I compared the April 7 db with todays and there were a number of password changes since then, including some of the people that contacted us for support about it. ssl: If the attacker can perform MITM attacks against them most users will be screwed anyway as they will be accessing other repositories like Maven Central over unprotected HTTP. However if we do discover any evidence of the VM being comprised I'll request it be revoked. vm backdoor: I've checked the obvious things (shadow entries, authorized_keys files, running processes). But there's no way to verify this with high confidence. We could do a fresh OS install I suppose, I was intending to upgrade to Ubuntu 12.04 sometime this year anyway. ------ Finally if we take Linode's statements at face value I think it's very unlikely Clojars was compromised. They state that the attackers gained access only to the web server not the VM hosts. The evidence presented by "ryan" (who claimed to be one of the attackers) also only referred to the web server and database. The most straightforward ways for the web server to compromise a customer VM: * There is a form to reset the root password of a target VM, this requires rebooting the VM as it modifies the filesystem offline. * You can use the LISH password (cleartext in db) to get to the VM's serial console. This doesn't directly get you in as the getty still prompts you to login. * You can reboot the VM against a rescue disk image which you control through the console. The clojars VM hasn't been rebooted in the last few months. If Linode is wrong and the VM host was rooted a (somewhat sophisticated) online attack would be possible via memory modification.
pgpwOufQwNng5.pgp
Description: PGP signature
