It's not an OpenSSH problem, per se -- it's OpenSSL that's causing the slowdown. There's been quite a bit of discussion about it on this list (though I think it's been a while since the subject last came up).
I believe rebuilding OpenSSL from the source package, so that it adds the UltraSparc optimiztions, would speed things up. JJ Streicher-Bremer wrote: > I have woody installed on my ultra2 and have had this problem from the > beginning. When I'm sshing to my system my clients (every one I have > tried) pauses for approx 15 sec before coming back with a prompt (or > connecting in the case of key-based auth). I do have forward _and_ > reverse name lookups working on this host. I even tried compiling my > own openssh with the same problem. When I run sshd in debug mode this > is what I get.

