Resolved: ssh returns Permission denied (gssapi-keyex,gssapi-with-mic).
Well this has been a thorn in my side for months but I think I've figured it out. At least I found a plausible reason for it and it's been working longer than it has before. The problem turned out to be I had both gsisshd and sshd running and the fix was to use chkconfig to disable it. The really weird part that made it hard to figure out was that ssh would work for days then suddenly stop. sudo service sshd restart would get it to work again for a few days. I had installed the gsi server stuff because we will (hopefully) move to that certificate based access soon, not thinking that it would be enabled on install. The take home lesson is think before you install potentially conflicting services. Thanks, Joe On 11/21/2012 02:16 PM, Joseph Areeda wrote: I can't figure out what causes this error. I can fix it by regenerating the server key on the system I'm trying to connect to and restarting sshd but that seems to be temporary as the same problem comes back in a week or so. Rebooting the server does not fix it. Does anyone know what that error means? I am using ssh not gsissh although I do have globus toolkit installed to contact grid computers. I'm pretty sure it's a misconfiguration on my part but I can't figure out what I did or didn't do. Thanks, Joe
Re: ssh returns Permission denied (gssapi-keyex,gssapi-with-mic).
Well there is your problem The users home directory needs to be 700 unless you turn off strict key checking in the sshd configuration file. Also the public key should be 600 as well. Making home directories world or group readable isn't a good plan for collaberation because many applications store sensitive information like passwords and cached information like session data in the home directory. instead consider creating group directories an setting the setgid bit on it so the group permissions are inherited by any files created in the directories. Making home directories world or group readable is a lazy solution to an easily solved problem. Its a common mistake that causes loads of problems because many application which are written to be secure purposly break when you do it. I highly suggest you comeup with a better plan for collaberation than that. On Nov 21, 2012 11:10 PM, Joseph Areeda newsre...@areeda.com wrote: On 11/21/2012 07:08 PM, Alan Bartlett wrote: On 22 November 2012 01:18, Joseph Areeda newsre...@areeda.com wrote: The user's directory is 755 which is the convention for grid computers in our collaboration and the plan is for this machine to be on our soon to be delivered cluster. The .ssh directory is 700. This doesn't change between the working and non-working state. Good, you've checked the directory. Now what about the files within it? Hopefully they are all 600? Alan. Alan, The private keys are all 600 and the public keys are 644. I keep a few different ones for going to different systems. Joe
Re: ssh returns Permission denied (gssapi-keyex,gssapi-with-mic).
Thanks for the comments Paul. I was surprised when I joined the collaboration and saw home directories world readable but that decision was made long before I arrived and changing it remains above my pay grade. The reason I doubt that's my current problem is because regenerating the server key files works. I can log in fine today and I haven't changed permissions. I also don't have problem logging into other systems from that machine that are [supposed to be] set up the same way. When it happens again, I will check if changing permissions helps. Also for the record I waited until my existing Kerberos tickets expired. These are to other services not that machine. I can log in fine with an expired or valid TGT hanging around and after kdestroy. Happy holidays, Joe On 11/22/2012 08:32 AM, Paul Robert Marino wrote: Well there is your problem The users home directory needs to be 700 unless you turn off strict key checking in the sshd configuration file. Also the public key should be 600 as well. Making home directories world or group readable isn't a good plan for collaberation because many applications store sensitive information like passwords and cached information like session data in the home directory. instead consider creating group directories an setting the setgid bit on it so the group permissions are inherited by any files created in the directories. Making home directories world or group readable is a lazy solution to an easily solved problem. Its a common mistake that causes loads of problems because many application which are written to be secure purposly break when you do it. I highly suggest you comeup with a better plan for collaberation than that. On Nov 21, 2012 11:10 PM, Joseph Areeda newsre...@areeda.com mailto:newsre...@areeda.com wrote: On 11/21/2012 07:08 PM, Alan Bartlett wrote: On 22 November 2012 01:18, Joseph Areeda newsre...@areeda.com mailto:newsre...@areeda.com wrote: The user's directory is 755 which is the convention for grid computers in our collaboration and the plan is for this machine to be on our soon to be delivered cluster. The .ssh directory is 700. This doesn't change between the working and non-working state. Good, you've checked the directory. Now what about the files within it? Hopefully they are all 600? Alan. Alan, The private keys are all 600 and the public keys are 644. I keep a few different ones for going to different systems. Joe
ssh returns Permission denied (gssapi-keyex,gssapi-with-mic).
I can't figure out what causes this error. I can fix it by regenerating the server key on the system I'm trying to connect to and restarting sshd but that seems to be temporary as the same problem comes back in a week or so. Rebooting the server does not fix it. Does anyone know what that error means? I am using ssh not gsissh although I do have globus toolkit installed to contact grid computers. I'm pretty sure it's a misconfiguration on my part but I can't figure out what I did or didn't do. Thanks, Joe
Re: ssh returns Permission denied (gssapi-keyex,gssapi-with-mic).
What does the output of ssh -vv hostname give you? and what does /var/log/secure say on the server side? Permission denied could be a number of things (time not in sync, PAM configuration right, or other stuff. without knowing the server and client sshd_config and ssh_config respectively it is hard to tell. Steve Timm On Wed, 21 Nov 2012, Joseph Areeda wrote: I can't figure out what causes this error. I can fix it by regenerating the server key on the system I'm trying to connect to and restarting sshd but that seems to be temporary as the same problem comes back in a week or so. Rebooting the server does not fix it. Does anyone know what that error means? I am using ssh not gsissh although I do have globus toolkit installed to contact grid computers. I'm pretty sure it's a misconfiguration on my part but I can't figure out what I did or didn't do. Thanks, Joe -- Steven C. Timm, Ph.D (630) 840-8525 t...@fnal.gov http://home.fnal.gov/~timm/ Fermilab Computing Division, Scientific Computing Facilities, Grid Facilities Department, FermiGrid Services Group, Group Leader. Lead of FermiCloud project.
Re: ssh returns Permission denied (gssapi-keyex,gssapi-with-mic).
Hi Joe, Did you look at the sshd_config file? I ran into a similar error output but it may not necessarily be the same issue you're having. In my case, the sshd_conf file on one of my users machine was edited and renamed. I backup that file and copy a default sshd_config file, then test it. Good luck. -T On Wed, Nov 21, 2012 at 5:16 PM, Joseph Areeda newsre...@areeda.com wrote: I can't figure out what causes this error. I can fix it by regenerating the server key on the system I'm trying to connect to and restarting sshd but that seems to be temporary as the same problem comes back in a week or so. Rebooting the server does not fix it. Does anyone know what that error means? I am using ssh not gsissh although I do have globus toolkit installed to contact grid computers. I'm pretty sure it's a misconfiguration on my part but I can't figure out what I did or didn't do. Thanks, Joe
Re: ssh returns Permission denied (gssapi-keyex,gssapi-with-mic).
Thank you Tam, and Steven, I just confirmed that regenerating the keys (ssh-keygen -t dsa -f ssh_host_dsa_key ssh -t rsa -f ssh_host_rsa_key) in /etc/ssh fixes the problem So ssh -vv shows me how it's supposed to look. I'll save that and do a diff when it happens again. As I continue my googling I can report on a few things it's not Server machine has a fixed ip address and dns/rdns appears working. Time issue Steven mentioned does not seem to be it, although I may stop using pool machines and set up a local ntp server so everybody gets the same time. I can ssh and gsissh to other servers. Server: ntpq -p remote refid st t when poll reach delay offset jitter == *ping-audit-207- .ACTS. 1 u5 128 377 19.867 5.804 1.927 +10504.x.rootbsd 198.30.92.2 2 u 129 128 376 45.146 -28.571 5.558 +ntp.sunflower.c 132.236.56.250 3 u 77 128 355 63.836 -14.753 5.360 -ntp2.ResComp.Be 128.32.206.553 u 126 128 377 22.112 7.311 2.022 Client: ntpq -p remote refid st t when poll reach delay offset jitter == 64.147.116.229 .ACTS. 1 u 47 1280 13.543 0.567 0.000 *nist1-chi.ustim .ACTS. 1 u 25 128 377 106.619 14.458 5.896 +name3.glorb.com 69.36.224.15 2 u 64 128 377 88.564 -27.542 3.631 +131.211.8.244 .PPS.1 u 81 128 377 167.107 3.259 2.340 The only setting I change in sshd_config is to turn off password auth but this machine is being brought up behind a firewall and I haven't done that yet. Also if it was a config problem I doubt changing the key would fix it, even temporarily. I will report back with the ssh -vv stuff when it happens again. At least now I have a chance of figuring out what's going on. Best, Joe On 11/21/2012 02:30 PM, Tam Nguyen wrote: Hi Joe, Did you look at the sshd_config file? I ran into a similar error output but it may not necessarily be the same issue you're having. In my case, the sshd_conf file on one of my users machine was edited and renamed. I backup that file and copy a default sshd_config file, then test it. Good luck. -T On Wed, Nov 21, 2012 at 5:16 PM, Joseph Areeda newsre...@areeda.com mailto:newsre...@areeda.com wrote: I can't figure out what causes this error. I can fix it by regenerating the server key on the system I'm trying to connect to and restarting sshd but that seems to be temporary as the same problem comes back in a week or so. Rebooting the server does not fix it. Does anyone know what that error means? I am using ssh not gsissh although I do have globus toolkit installed to contact grid computers. I'm pretty sure it's a misconfiguration on my part but I can't figure out what I did or didn't do. Thanks, Joe
RE: ssh returns Permission denied (gssapi-keyex,gssapi-with-mic).
Shouldn't need to regenerate the keys.. once you get them generated once they should be good for the life of the machine. Save copies of the keys as they are now and if your system goes bad, do differences to see what changed, if anything. Steve Timm From: owner-scientific-linux-us...@listserv.fnal.gov [mailto:owner-scientific-linux-us...@listserv.fnal.gov] On Behalf Of Joseph Areeda Sent: Wednesday, November 21, 2012 5:46 PM To: owner-scientific-linux-us...@listserv.fnal.gov Cc: scientific-linux-users Subject: Re: ssh returns Permission denied (gssapi-keyex,gssapi-with-mic). Thank you Tam, and Steven, I just confirmed that regenerating the keys (ssh-keygen -t dsa -f ssh_host_dsa_key ssh -t rsa -f ssh_host_rsa_key) in /etc/ssh fixes the problem So ssh -vv shows me how it's supposed to look. I'll save that and do a diff when it happens again. As I continue my googling I can report on a few things it's not Server machine has a fixed ip address and dns/rdns appears working. Time issue Steven mentioned does not seem to be it, although I may stop using pool machines and set up a local ntp server so everybody gets the same time. I can ssh and gsissh to other servers. Server: ntpq -p remote refid st t when poll reach delay offset jitter == *ping-audit-207- .ACTS. 1 u5 128 377 19.8675.804 1.927 +10504.x.rootbsd 198.30.92.2 2 u 129 128 376 45.146 -28.571 5.558 +ntp.sunflower.c 132.236.56.250 3 u 77 128 355 63.836 -14.753 5.360 -ntp2.ResComp.Be 128.32.206.553 u 126 128 377 22.1127.311 2.022 Client: ntpq -p remote refid st t when poll reach delay offset jitter == 64.147.116.229 .ACTS. 1 u 47 1280 13.5430.567 0.000 *nist1-chi.ustim .ACTS. 1 u 25 128 377 106.619 14.458 5.896 +name3.glorb.com 69.36.224.15 2 u 64 128 377 88.564 -27.542 3.631 +131.211.8.244 .PPS.1 u 81 128 377 167.1073.259 2.340 The only setting I change in sshd_config is to turn off password auth but this machine is being brought up behind a firewall and I haven't done that yet. Also if it was a config problem I doubt changing the key would fix it, even temporarily. I will report back with the ssh -vv stuff when it happens again. At least now I have a chance of figuring out what's going on. Best, Joe On 11/21/2012 02:30 PM, Tam Nguyen wrote: Hi Joe, Did you look at the sshd_config file? I ran into a similar error output but it may not necessarily be the same issue you're having. In my case, the sshd_conf file on one of my users machine was edited and renamed. I backup that file and copy a default sshd_config file, then test it. Good luck. -T On Wed, Nov 21, 2012 at 5:16 PM, Joseph Areeda newsre...@areeda.commailto:newsre...@areeda.com wrote: I can't figure out what causes this error. I can fix it by regenerating the server key on the system I'm trying to connect to and restarting sshd but that seems to be temporary as the same problem comes back in a week or so. Rebooting the server does not fix it. Does anyone know what that error means? I am using ssh not gsissh although I do have globus toolkit installed to contact grid computers. I'm pretty sure it's a misconfiguration on my part but I can't figure out what I did or didn't do. Thanks, Joe
RE: ssh returns Permission denied (gssapi-keyex,gssapi-with-mic).
On Nov 21, 2012 7:57 PM, Paul Robert Marino prmari...@gmail.com wrote: Ok To be clear are you using kerberos or not If the answer is no and you are just using ssh keys the most common cause of this issue is that the useres home directory is group or world readable. In the most secure mode which is the default if the useres home and or the ~/.ssh directory is has a any thing other than 700 or 500 set as the permissions it will reject the public key (the one on the server you are trying to connect to) this become obvious with -vvv but not -vv On Nov 21, 2012 7:34 PM, Steven C Timm t...@fnal.gov wrote: Shouldn’t need to regenerate the keys.. once you get them generated once they should be good for the life of the machine. Save copies of the keys as they are now and if your system goes bad, do differences to see what changed, if anything. ** ** Steve Timm ** ** ** ** *From:* owner-scientific-linux-us...@listserv.fnal.gov [mailto: owner-scientific-linux-us...@listserv.fnal.gov] *On Behalf Of *Joseph Areeda *Sent:* Wednesday, November 21, 2012 5:46 PM *To:* owner-scientific-linux-us...@listserv.fnal.gov *Cc:* scientific-linux-users *Subject:* Re: ssh returns Permission denied (gssapi-keyex,gssapi-with-mic). ** ** Thank you Tam, and Steven, I just confirmed that regenerating the keys (ssh-keygen -t dsa -f ssh_host_dsa_key ssh -t rsa -f ssh_host_rsa_key) in /etc/ssh fixes the problem So ssh -vv shows me how it's supposed to look. I'll save that and do a diff when it happens again. As I continue my googling I can report on a few things it's not Server machine has a fixed ip address and dns/rdns appears working. Time issue Steven mentioned does not seem to be it, although I may stop using pool machines and set up a local ntp server so everybody gets the same time. I can ssh and gsissh to other servers. Server: ntpq -p remote refid st t when poll reach delay offset jitter == *ping-audit-207- .ACTS. 1 u5 128 377 19.8675.804 1.927 +10504.x.rootbsd 198.30.92.2 2 u 129 128 376 45.146 -28.571 5.558 +ntp.sunflower.c 132.236.56.250 3 u 77 128 355 63.836 -14.753 5.360 -ntp2.ResComp.Be 128.32.206.553 u 126 128 377 22.1127.311 2.022 Client: ntpq -p remote refid st t when poll reach delay offset jitter == 64.147.116.229 .ACTS. 1 u 47 1280 13.5430.567 0.000 *nist1-chi.ustim .ACTS. 1 u 25 128 377 106.619 14.458 5.896 +name3.glorb.com 69.36.224.15 2 u 64 128 377 88.564 -27.542 3.631 +131.211.8.244 .PPS.1 u 81 128 377 167.1073.259 2.340 The only setting I change in sshd_config is to turn off password auth but this machine is being brought up behind a firewall and I haven't done that yet. Also if it was a config problem I doubt changing the key would fix it, even temporarily. I will report back with the ssh -vv stuff when it happens again. At least now I have a chance of figuring out what's going on. Best, Joe On 11/21/2012 02:30 PM, Tam Nguyen wrote: Hi Joe, Did you look at the sshd_config file? I ran into a similar error output but it may not necessarily be the same issue you're having. In my case, the sshd_conf file on one of my users machine was edited and renamed. I backup that file and copy a default sshd_config file, then test it. ** ** Good luck. -T On Wed, Nov 21, 2012 at 5:16 PM, Joseph Areeda newsre...@areeda.com wrote: I can't figure out what causes this error. I can fix it by regenerating the server key on the system I'm trying to connect to and restarting sshd but that seems to be temporary as the same problem comes back in a week or so. Rebooting the server does not fix it. Does anyone know what that error means? I am using ssh not gsissh although I do have globus toolkit installed to contact grid computers. I'm pretty sure it's a misconfiguration on my part but I can't figure out what I did or didn't do. Thanks, Joe ** **
Re: ssh returns Permission denied (gssapi-keyex,gssapi-with-mic).
Thank you Paul, Steven and Steve, I think Kerberos may be the issue. I do NOT use Kerberos to access this machine, I have a lot to learn before I turn that and LDAP on. But I do use it to access several services in our collaboration so the client machine often has a valid Kerberos TGT (and probably more often an expired ticket). I think it's worth experimenting with the client in different states of Kerberosity (or whatever that word should be). The user's directory is 755 which is the convention for grid computers in our collaboration and the plan is for this machine to be on our soon to be delivered cluster. The .ssh directory is 700. This doesn't change between the working and non-working state. I tarred the /etc/ssh directory and saved it for next time but wouldn't generating new keys make them almost completely different? Generating new keys makes no sense to me either, but it does work. Well, at least it has been the only thing I've done coincident with resolving the problem the last 3 times this has happened. I also save the triple verbose ssh output. I really appreciate the discussion gentlemen, it helps a lot. Best, Joe On 11/21/2012 04:58 PM, Paul Robert Marino wrote: On Nov 21, 2012 7:57 PM, Paul Robert Marino prmari...@gmail.com mailto:prmari...@gmail.com wrote: Ok To be clear are you using kerberos or not If the answer is no and you are just using ssh keys the most common cause of this issue is that the useres home directory is group or world readable. In the most secure mode which is the default if the useres home and or the ~/.ssh directory is has a any thing other than 700 or 500 set as the permissions it will reject the public key (the one on the server you are trying to connect to) this become obvious with -vvv but not -vv On Nov 21, 2012 7:34 PM, Steven C Timm t...@fnal.gov mailto:t...@fnal.gov wrote: Shouldn’t need to regenerate the keys.. once you get them generated once they should be good for the life of the machine. Save copies of the keys as they are now and if your system goes bad, do differences to see what changed, if anything. Steve Timm *From:*owner-scientific-linux-us...@listserv.fnal.gov mailto:owner-scientific-linux-us...@listserv.fnal.gov [mailto:owner-scientific-linux-us...@listserv.fnal.gov mailto:owner-scientific-linux-us...@listserv.fnal.gov] *On Behalf Of *Joseph Areeda *Sent:* Wednesday, November 21, 2012 5:46 PM *To:* owner-scientific-linux-us...@listserv.fnal.gov mailto:owner-scientific-linux-us...@listserv.fnal.gov *Cc:* scientific-linux-users *Subject:* Re: ssh returns Permission denied (gssapi-keyex,gssapi-with-mic). Thank you Tam, and Steven, I just confirmed that regenerating the keys (ssh-keygen -t dsa -f ssh_host_dsa_key ssh -t rsa -f ssh_host_rsa_key) in /etc/ssh fixes the problem So ssh -vv shows me how it's supposed to look. I'll save that and do a diff when it happens again. As I continue my googling I can report on a few things it's not Server machine has a fixed ip address and dns/rdns appears working. Time issue Steven mentioned does not seem to be it, although I may stop using pool machines and set up a local ntp server so everybody gets the same time. I can ssh and gsissh to other servers. Server: ntpq -p remote refid st t when poll reach delay offset jitter == *ping-audit-207- .ACTS. 1 u5 128 377 19.8675.804 1.927 +10504.x.rootbsd 198.30.92.2 2 u 129 128 376 45.146 -28.571 5.558 +ntp.sunflower.c 132.236.56.250 3 u 77 128 355 63.836 -14.753 5.360 -ntp2.ResComp.Be http://ntp2.ResComp.Be 128.32.206.553 tel:128.32.206.55%C2%A0%C2%A0%C2%A0%203 u 126 128 377 22.1127.311 2.022 Client: ntpq -p remote refid st t when poll reach delay offset jitter == 64.147.116.229 .ACTS. 1 u 47 1280 13.5430.567 0.000 *nist1-chi.ustim .ACTS. 1 u 25 128 377 106.619 14.458 5.896 +name3.glorb.com http://name3.glorb.com 69.36.224.15 2 u 64 128 377 88.564 -27.542 3.631 +131.211.8.244 .PPS.1 u 81 128 377 167.1073.259 2.340 The only setting I change in sshd_config is to turn off password auth but this machine is being brought up behind a firewall and I haven't done
