Re: [Haskell-cafe] Status update on {code, trac, projects, planet, community}.haskell.org
On Thu, 2011-02-17 at 19:30 +0100, Henning Thielemann wrote: Duncan Coutts schrieb: Several people have asked about the new host key. Yes, there is a new RSA host key for the community server, the fingerprint of which is: 21:b8:59:ff:39:69:58:7a:51:ef:c1:d8:c6:24:6e:f7 ssh will likely give you a scary warning and you'll need to delete the old entry in your ~/.ssh/known_hosts file. You don't need to enter a new one, just delete the old one. When you next log into the server, ssh will ask you if you're happy with the new key. If you're paranoid, you can double check that it matches the key fingerprint above. Do you think it is paranoid? Sorry, I didn't mean it literally (or pejoratively). Unfortunately it has become quite common to ignore SSH warnings because admins often do not care about restoring keys when updating the operating system or moving the machine, even not telling users that the host key has changed. But if I had ignored the SSH warning on code.haskell.org recently I might have logged in and from there maybe to other servers, thus giving my passwords to the attackers. I think generally that just deleting a host from known_hosts in response to an SSH warning and blindly accepting a new host key is not a fix. Am I too afraid? No, you're quite right. It was these warnings that initially alerted us to the problem. Duncan ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Status update on {code, trac, projects, planet, community}.haskell.org
I believe code.haskell.org has moved to a new machine. Its IP address also changed, which causes your ssh to issue a warning. You can fix it by deleting the code.haskell.org entry from your local ~/.ssh/known_hosts file. On 16 February 2011 18:58, Henning Thielemann lemm...@henning-thielemann.de wrote: Thank you a lot for bringing code.haskell.org back! I missed it a lot! However, I still get $ ssh code.haskell.org @@@ @ WARNING: POSSIBLE DNS SPOOFING DETECTED! @ @@@ The RSA host key for code.haskell.org has changed, and the key for the according IP address 178.63.91.44 is unknown. This could either mean that DNS SPOOFING is happening or the IP address for the host and its host key have changed at the same time. @@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that the RSA host key has just been changed. ... How can I verify that it is safe to continue logging in? Could you restore the old host key? ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Status update on {code, trac, projects, planet, community}.haskell.org
Duncan Coutts schrieb: Several people have asked about the new host key. Yes, there is a new RSA host key for the community server, the fingerprint of which is: 21:b8:59:ff:39:69:58:7a:51:ef:c1:d8:c6:24:6e:f7 ssh will likely give you a scary warning and you'll need to delete the old entry in your ~/.ssh/known_hosts file. You don't need to enter a new one, just delete the old one. When you next log into the server, ssh will ask you if you're happy with the new key. If you're paranoid, you can double check that it matches the key fingerprint above. Do you think it is paranoid? Unfortunately it has become quite common to ignore SSH warnings because admins often do not care about restoring keys when updating the operating system or moving the machine, even not telling users that the host key has changed. But if I had ignored the SSH warning on code.haskell.org recently I might have logged in and from there maybe to other servers, thus giving my passwords to the attackers. I think generally that just deleting a host from known_hosts in response to an SSH warning and blindly accepting a new host key is not a fix. Am I too afraid? ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Status update on {code, trac, projects, planet, community}.haskell.org
On Thu, Feb 17, 2011 at 07:30:23PM +0100, Henning Thielemann wrote: Do you think it is paranoid? Unfortunately it has become quite common to ignore SSH warnings because admins often do not care about restoring keys when updating the operating system or moving the machine, even not telling users that the host key has changed. But if I had ignored the SSH warning on code.haskell.org recently I might have logged in and from there maybe to other servers, thus giving my passwords to the attackers. I think generally that just deleting a host from known_hosts in response to an SSH warning and blindly accepting a new host key is not a fix. Am I too afraid? If sshd has been compromised, so is the original host private key. It would be kind of pointless (security wise) to restore it on the new server. -- Vincent ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Status update on {code, trac, projects, planet, community}.haskell.org
2011/2/17 Henning Thielemann lemm...@henning-thielemann.de: Duncan Coutts schrieb: Several people have asked about the new host key. Yes, there is a new RSA host key for the community server, the fingerprint of which is: 21:b8:59:ff:39:69:58:7a:51:ef:c1:d8:c6:24:6e:f7 ssh will likely give you a scary warning and you'll need to delete the old entry in your ~/.ssh/known_hosts file. You don't need to enter a new one, just delete the old one. When you next log into the server, ssh will ask you if you're happy with the new key. If you're paranoid, you can double check that it matches the key fingerprint above. Do you think it is paranoid? Unfortunately it has become quite common to ignore SSH warnings because admins often do not care about restoring keys when updating the operating system or moving the machine, even not telling users that the host key has changed. But if I had ignored the SSH warning on code.haskell.org recently I might have logged in and from there maybe to other servers, thus giving my passwords to the attackers. I think generally that just deleting a host from known_hosts in response to an SSH warning and blindly accepting a new host key is not a fix. Am I too afraid? Hi, Regarding you giving passwords when logging in other marchines, I think it would not be the case if you only use key authentication from machines to machines. Your private key can be only on your local machine and you can use an ssh agent to do log from machines to machines. Cheers, Thu ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Status update on {code, trac, projects, planet, community}.haskell.org
Thank you a lot for bringing code.haskell.org back! I missed it a lot!
However, I still get
$ ssh code.haskell.org
@@@
@ WARNING: POSSIBLE DNS SPOOFING DETECTED! @
@@@
The RSA host key for code.haskell.org has changed,
and the key for the according IP address 178.63.91.44
is unknown. This could either mean that
DNS SPOOFING is happening or the IP address for the host
and its host key have changed at the same time.
@@@
@WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
...
How can I verify that it is safe to continue logging in?
Could you restore the old host key?
On Wed, 16 Feb 2011, Duncan Coutts wrote:
Code repositories
-
public URL: http://code.haskell.org/$projname/
server-side: /srv/code/$projname/
or: /srv/srv-from-nun/code/{checked-failed,checked-strayfiles}/$projname/
Similarly, many code repositories (44) have not been re-enabled. Ones
that we could check automatically have already been restored.
If the /srv/code/$project directory for your project is empty or missing
then you will find it in one of the directories
in /srv/srv-from-nun/code/, either checked-failed/ if darcs check
failed on that repository, or in checked-strayfiles/ if the repository
contains extra unrecorded files that we could not check automatically.
You should check that you are satisfied that the repository contains
just what you expect and then email supp...@community.haskell.org to ask
for it to be moved back to the usual location.
This causes a lot manual work for you. It would be easier for you and us
to allow project maintainers to move the repositories from srv-from-nun to
/src/code.
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Status update on {code, trac, projects, planet, community}.haskell.org
On Wed, 2011-02-16 at 02:12 +, Duncan Coutts wrote: We have not yet re-enabled user login accounts, nor re-enabled access to code repositories. We will send a further update when these are re-enabled, or procedures for people to re-enable them are finalised. Logging in == We have restored ssh logins for around 250 user accounts (ie darcs push will work). Several people have asked about the new host key. Yes, there is a new RSA host key for the community server, the fingerprint of which is: 21:b8:59:ff:39:69:58:7a:51:ef:c1:d8:c6:24:6e:f7 ssh will likely give you a scary warning and you'll need to delete the old entry in your ~/.ssh/known_hosts file. You don't need to enter a new one, just delete the old one. When you next log into the server, ssh will ask you if you're happy with the new key. If you're paranoid, you can double check that it matches the key fingerprint above. Duncan ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Status update on {code, trac, projects, planet, community}.haskell.org
On Wed, 2011-02-02 at 01:33 +, Duncan Coutts wrote:
All,
As you will be aware, some of the *.haskell.org websites have been down
recently, specifically:
code.haskell.org
trac.haskell.org
projects.haskell.org
planet.haskell.org
community.haskell.org
[...]
We have not yet re-enabled user login accounts, nor re-enabled access
to code repositories. We will send a further update when these are
re-enabled, or procedures for people to re-enable them are finalised.
Logging in
==
We have restored ssh logins for around 250 user accounts (ie darcs push
will work).
If you are not one of those 250 and you cannot log in then you will need
to email supp...@community.haskell.org. Give your real name, your unix
user name and attach your current ssh public key.
Once you have logged in
===
Personal webspace
-
public URL: http://code.haskell.org/~$username/
server-side: ~/public_html(-disabled)
You will notice that your ~/public_html directory has been renamed to
~/public_html-disabled. There is a slim possibility that the data was
altered when the server was compromised. We recommend that you check it
first and then to restore use: mv ~/public_html-disabled ~/public_html
Code repositories
-
public URL: http://code.haskell.org/$projname/
server-side: /srv/code/$projname/
or: /srv/srv-from-nun/code/{checked-failed,checked-strayfiles}/$projname/
Similarly, many code repositories (44) have not been re-enabled. Ones
that we could check automatically have already been restored.
If the /srv/code/$project directory for your project is empty or missing
then you will find it in one of the directories
in /srv/srv-from-nun/code/, either checked-failed/ if darcs check
failed on that repository, or in checked-strayfiles/ if the repository
contains extra unrecorded files that we could not check automatically.
You should check that you are satisfied that the repository contains
just what you expect and then email supp...@community.haskell.org to ask
for it to be moved back to the usual location.
Project websites
public URL: http://projects.haskell.org/$projname/
server-side: /srv/projects/${projname/
or: /srv/srv-from-nun/projects/$projname/
If the /srv/projects/$project directory for your project is empty or
missing then will find the project website
in /srv/srv-from-nun/projects/$project.
You should check that you are satisfied that the website directory
contains just what you expect and then email
supp...@community.haskell.org to ask for it to be moved back to the
usual location.
Explanation
===
We believe that when the server was compromised, the attacker was mainly
interested in collecting usernames and passwords. Since we do not use
password based logins, we think the attacker was not successful in this.
However we are unable to trust any of the ~/.ssh/authorized_keys because
the attacker could have modified them to give access at a later date.
We were able to verify the ~/.ssh/authorized_keys for around 250 users
by comparing the current file against the key that was originally
submitted in the account creation request. People who have added keys or
changed keys since initial account creation have not had their login
access restored and they must resend their current key.
For html, css, javascript files etc, there was the slight concern that
the attacker may have defaced sites or made malicious files available
for download. While we have not found any instance of this so far, we
need the help of project owners to check this.
Duncan
(On behalf of the Haskell infrastructure team)
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Status update on {code, trac, projects, planet, community}.haskell.org
On Wed, 2011-02-02 at 01:33 +, Duncan Coutts wrote: As you will be aware, some of the *.haskell.org websites have been down recently, specifically: code.haskell.org trac.haskell.org projects.haskell.org planet.haskell.org community.haskell.org [...] We have not yet re-enabled user login accounts, nor re-enabled access to code repositories. The support ticket system is not yet enabled. We will send a further update when these are re-enabled, or procedures for people to re-enable them are finalised. We have restored read-only access to the majority of projects on: code.haskell.org(code repositories) projects.haskell.org(project webspace) A small number have been held back because they contain .js, .html or .tar.gz snapshots that are not recorded in any repository. We will ask project owners to check these before they are restored. Duncan (On behalf of the Haskell infrastructure team) ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Status update on {code, trac, projects, planet, community}.haskell.org
* Duncan Coutts duncan.cou...@googlemail.com [2011-02-02 01:33:22+] These are all hosted on the community server. The community server was hacked on the 26th January and we took it offline. The server was running an old version of debian that was no longer supported with security updates. (Ironically, two days previously the infrastructure team had been discussing the fact that nobody seemed to have any time available to do the planned migration to a new host). The hacker replaced sshd which we noticed because the ssh host signature changed and it started prompting for passwords (we use key-based rather than password based logins). Might be related: http://sourceforge.net/blog/sourceforge-attack-full-report/ -- Roman I. Cheplyaka :: http://ro-che.info/ Don't worry what people think, they don't do it very often. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Status update on {code, trac, projects, planet, community}.haskell.org
On Thu, 2011-02-03 at 10:37 +0200, Roman Cheplyaka wrote: * Duncan Coutts duncan.cou...@googlemail.com [2011-02-02 01:33:22+] These are all hosted on the community server. The community server was hacked on the 26th January and we took it offline. The server was running an old version of debian that was no longer supported with security updates. (Ironically, two days previously the infrastructure team had been discussing the fact that nobody seemed to have any time available to do the planned migration to a new host). The hacker replaced sshd which we noticed because the ssh host signature changed and it started prompting for passwords (we use key-based rather than password based logins). Might be related: http://sourceforge.net/blog/sourceforge-attack-full-report/ Yes, it's quite possible. One difference to note is that we use ssh key based logins, not passwords. We suspect this saved us from the worst case scenarios. Nevertheless, while we don't have to reset passwords, we are concerned about the potential that the attacker replaced or added to users ~/.ssh/authorized_keys lists, which is why we have not yet re-enabled user accounts. We will try and provide as full a picture as we can when we're satisfied we've got as much info and confidence as we're likely to get. Duncan ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
