thanks Jamis, The interesting thing is I can copy/paste the commands capistrano is trying to issue (from the console, I did a "dry-run") and then use this on the prod server and everything works fine (i.e. prod server to git repo server calls are working fine). Can I ask a couple of quick questions:
Is there anything different from an ssh call point of view between then: (a) the first call which works (i.e. executing locally: "git ls-remote [email protected]:/repos/equity/.git HEAD") and (b) second call which has authentication issue (i.e. "f [ -d /u/apps/equity/shared/cached-copy ]; then cd /u/apps/equity/shared/cached-copy && git fetch -q origin && git reset -q --hard 7e92223c93c6f9019acd54d042fc8d348aa62a53; else git clone -q [email protected]:/repos/equity/.git /u/apps/equity/shared/cached-copy && cd /u/apps/equity/shared/cached-copy && git checkout -q -b deploy 7e92223c93c6f9019acd54d042fc8d348aa62a53; fi") Q1 - Do you see any difference in this first & second call from an ssh connection point of view? Or is this also strange to you that the first call works fine, but then the 2nd call comes along and can't connect from the DEV box to the PROD box? (or actually from your response is the main difference that the 1st targets the REPO server, whilst the 2nd then targets the PROD server - hence in my case am I really getting an authentication issue for the first capistrano call that is targeting the PROD server) Q2 - I'm assuming the authentication issue is my DEV box authenticating to the PROD box? (i.e. not the PROD box trying to run GIT to contact the REPO server) Is what you believe based on the logs? And also can I ask when you give your answer why (i.e. so I can learn re how you interpreted things) You did mention ("Capistrano opens a new SSH "channel" for each request.") Q3 - If it turns out to be an ssh/linux issue somehow, have you a pointer re where to look in the capistrano code? (i.e. if I had to post the code that made the call onto an ssh/linux forum). I'm finding it kind of confusing as I run capistrano of Server A, which authenticates to Server B, but the scripts actually run on Server B, which then has to authenticate from Server B to Server C, and to complicate things I think you pointed out the last ssh connection (server B to server C[repo]) is not really capistrano but git. :) Thanks again Greg 2009/2/7 Jamis Buck <[email protected]>: > > Greg, > > 1. Capistrano opens a new SSH "channel" for each request. Essentially, > this means a new shell instance for each command, and means you cannot > use stateful commands (e.g., run("cd foo") followed by run("pwd") will > not display "foo"). If you need commands to execute from within a > specific directory, or with a specific set of environment variables, > or whatever, you must make sure to change directory or set environment > variables every time you run() something. > > 2. Yes. Capistrano needs to query the current revision, and does so > from your local machine (DevServer, in your terminology). Once it > knows the revision to deploy, it will push the code to the servers > using the deployment strategy you selected (which may or may not do a > code pull from the target machines). > > As for the "remote end hung up unexpectedly" error, I'm not sure. The > error is coming from git on the remote server, and not from > capistrano, so you need to make sure that git on the remote host can > connect to the repository. Beyond that, I'm not sure what > troubleshooting help I can offer. > > - Jamis > > On Fri, Feb 6, 2009 at 3:49 AM, Greg Hauptmann > <[email protected]> wrote: >> PSS. It seems like, in terms of the ssh connections: >> (1st connection) the initial ssh connection open by "cap deploy:update" >> actually *closes* prior to the next one [I'm guessing this is the problem?] >> (2nd connection) the connection from the prod app server to the git repo >> server then seems to occur from DevServer => ProdAppServer, as opposed to >> from ProdAppServer => ProdAppServer(where repo is). >> This would explain why for the repo connection password request I'm seeing >> output (e.g. Enter passphrase for /Users/greg/.ssh/id_rsa) that highlights >> the request is coming from my DevServer, and NOT the ProdAppServer. Make >> sense? >> QUESTION THEREFORE: If this is correct, why would the SSH connection after >> the initial "cap deploy:update" drop? >> Here's a copy/paste of output over time. I've pasted the first password >> request, and the session open/closing log statements, then I've copied paste >> the capistrano output for the 2nd password request phase, and it's >> corresponding log output: >> -------------- >> Macintosh-2:equity greg$ cap deploy:update -d >> * executing `deploy:update' >> ** transaction: start >> * executing `deploy:update_code' >> updating the cached checkout on all servers >> executing locally: "git ls-remote [email protected]:/repos/equity/.git HEAD" >> * executing "if [ -d /u/apps/equity/shared/cached-copy ]; then cd >> /u/apps/equity/shared/cached-copy && git fetch -q origin && git reset -q >> --hard 7e92223c93c6f9019acd54d042fc8d348aa62a53; else git clone -q >> [email protected]:/repos/equity/.git /u/apps/equity/shared/cached-copy && cd >> /u/apps/equity/shared/cached-copy && git checkout -q -b deploy >> 7e92223c93c6f9019acd54d042fc8d348aa62a53; fi" >> Preparing to execute command: "if [ -d /u/apps/equity/shared/cached-copy ]; >> then cd /u/apps/equity/shared/cached-copy && git fetch -q origin && git >> reset -q --hard 7e92223c93c6f9019acd54d042fc8d348aa62a53; else git clone -q >> [email protected]:/repos/equity/.git /u/apps/equity/shared/cached-copy && cd >> /u/apps/equity/shared/cached-copy && git checkout -q -b deploy >> 7e92223c93c6f9019acd54d042fc8d348aa62a53; fi" >> Execute ([Yes], No, Abort) ? |y| >> >> Feb 6 20:36:24 home sshd_local(pam_unix)[26758]: session opened for user >> root by (uid=0) >> Feb 6 20:36:25 home sshd_local(pam_unix)[26758]: session closed for user >> root >> -------------- >> servers: ["10.1.1.1"] >> Enter passphrase for /Users/greg/.ssh/id_rsa: >> [[email protected]] executing command >> ** [10.1.1.1 :: err] Permission denied >> (publickey,gssapi-with-mic,keyboard-interactive). >> ** [10.1.1.1 :: err] fatal: The remote end hung up unexpectedly >> command finished >> *** [deploy:update_code] rolling back >> * executing "rm -rf /u/apps/equity/releases/20090206103749; true" >> Preparing to execute command: "rm -rf >> /u/apps/equity/releases/20090206103749; true" >> Execute ([Yes], No, Abort) ? |y| >> >> Feb 6 20:37:41 home sshd_local(pam_unix)[7673]: session opened for user >> root by (uid=0) >> Feb 6 20:37:41 home sshd_local(pam_unix)[9423]: authentication failure; >> logname= uid=0 euid=0 tty=ssh ruser= rhost=home.gregnet.org user=root >> Feb 6 20:37:44 home sshd_local(pam_unix)[10530]: authentication failure; >> logname= uid=0 euid=0 tty=ssh ruser= rhost=home.gregnet.org user=root >> Feb 6 20:37:46 home sshd_local(pam_unix)[11605]: authentication failure; >> logname= uid=0 euid=0 tty=ssh ruser= rhost=home.gregnet.org user=root >> -------------- >> >> >> 2009/2/6 Greg Hauptmann <[email protected]> >>> >>> PS. >>> >>> I've now cloned my repository manually onto my prod Linux box (under >>> /repos) and updated deploy.config to reflect this. That is when the cap >>> scripts are running on my prod box it should call out to itself now for git >>> repository. I still get an authentication issue however (see below). There >>> must be a difference re ssh login between when a script runs the git >>> command, and when I manually run it? >>> >>> ----------prod messages log------------ >>> Feb 6 16:52:00 home sshd_local(pam_unix)[10814]: session opened for user >>> root by (uid=0) >>> Feb 6 16:52:00 home sshd_local(pam_unix)[10814]: session closed for user >>> root >>> >>> Feb 6 16:52:02 home sshd_local(pam_unix)[10833]: session opened for user >>> root by (uid=0) >>> Feb 6 16:52:03 home sshd_local(pam_unix)[10856]: authentication failure; >>> logname= uid=0 euid=0 tty=ssh ruser= rhost=home.gregnet.org user=root >>> Feb 6 16:52:05 home sshd_local(pam_unix)[10857]: authentication failure; >>> logname= uid=0 euid=0 tty=ssh ruser= rhost=home.gregnet.org user=root >>> Feb 6 16:52:07 home sshd_local(pam_unix)[10858]: authentication failure; >>> logname= uid=0 euid=0 tty=ssh ruser= rhost=home.gregnet.org user=root >>> Feb 6 16:52:10 home sshd_local(pam_unix)[10833]: session closed for user >>> root >>> >>> >>> >>> -------prod secure log ----------------- >>> Feb 6 16:55:12 home sshd_local[11041]: Accepted publickey for root from >>> ::ffff:10.1.1.145 port 51107 ssh2 >>> >>> Feb 6 16:55:17 home sshd_local[11060]: Accepted publickey for root from >>> ::ffff:10.1.1.145 port 51108 ssh2 >>> Feb 6 16:55:17 home sshd_local[11082]: Postponed keyboard-interactive for >>> root from ::ffff:10.1.1.1 port 43930 ssh2 >>> Feb 6 16:55:20 home sshd_local[11081]: error: PAM: Authentication failure >>> for root from home.gregnet.org >>> Feb 6 16:55:20 home sshd_local[11082]: Postponed keyboard-interactive for >>> root from ::ffff:10.1.1.1 port 43930 ssh2 >>> Feb 6 16:55:22 home sshd_local[11081]: error: PAM: Authentication failure >>> for root from home.gregnet.org >>> Feb 6 16:55:22 home sshd_local[11082]: Postponed keyboard-interactive for >>> root from ::ffff:10.1.1.1 port 43930 ssh2 >>> Feb 6 16:55:24 home sshd_local[11081]: error: PAM: Authentication failure >>> for root from home.gregnet.org >>> Feb 6 16:55:24 home sshd_local[11082]: Connection closed by >>> ::ffff:10.1.1.1 >>> >>> >>> >>> ----------capistrano output --------------- >>> Macintosh-2:equity greg$ cap deploy:update >>> * executing `deploy:update' >>> ** transaction: start >>> * executing `deploy:update_code' >>> updating the cached checkout on all servers >>> executing locally: "git ls-remote [email protected]:/repos/equity/.git >>> HEAD" >>> Enter passphrase for key '/Users/greg/.ssh/id_rsa': >>> * executing "if [ -d /u/apps/equity/shared/cached-copy ]; then cd >>> /u/apps/equity/shared/cached-copy && git fetch -q origin && git reset -q >>> --hard 7e92223c93c6f9019acd54d042fc8d348aa62a53; else git clone -q >>> [email protected]:/repos/equity/.git /u/apps/equity/shared/cached-copy && cd >>> /u/apps/equity/shared/cached-copy && git checkout -q -b deploy >>> 7e92223c93c6f9019acd54d042fc8d348aa62a53; fi" >>> servers: ["10.1.1.1"] >>> Enter passphrase for /Users/greg/.ssh/id_rsa: >>> [[email protected]] executing command >>> ** [10.1.1.1 :: err] Permission denied >>> (publickey,gssapi-with-mic,keyboard-interactive). >>> ** [10.1.1.1 :: err] fatal: The remote end hung up unexpectedly >>> command finished >>> *** [deploy:update_code] rolling back >>> * executing "rm -rf /u/apps/equity/releases/20090206065213; true" >>> servers: ["10.1.1.1"] >>> [[email protected]] executing command >>> command finished >>> failed: "sh -c \"if [ -d /u/apps/equity/shared/cached-copy ]; then cd >>> /u/apps/equity/shared/cached-copy && git fetch -q origin && git reset -q >>> --hard 7e92223c93c6f9019acd54d042fc8d348aa62a53; else git clone -q >>> [email protected]:/repos/equity/.git /u/apps/equity/shared/cached-copy && cd >>> /u/apps/equity/shared/cached-copy && git checkout -q -b deploy >>> 7e92223c93c6f9019acd54d042fc8d348aa62a53; fi\"" on [email protected] >>> >>> >>> >>> 2009/2/6 Greg Hauptmann <[email protected]> >>>> >>>> Hi guys, >>>> >>>> I'm stuck on this. I can ssh into my target prod server, and from there >>>> ssh into my repo server (for git) fine. When I run "cap deploy:update" it >>>> seems when the capistrano scripts running on my target prod server get a >>>> "failed password" when trying to access the repo server??? Any ideas??? >>>> Here's a tail of the secure.log on the repo server for both cases. >>>> >>>> ---------- repo server log when "manually ssh'ing in from prod server to >>>> repo server" ------------------- >>>> Feb 6 15:23:18 Macintosh-2 com.apple.SecurityServer[21]: checkpw() >>>> succeeded, creating credential for user greg >>>> Feb 6 15:23:18 Macintosh-2 com.apple.SecurityServer[21]: checkpw() >>>> succeeded, creating shared credential for user greg >>>> Feb 6 15:23:18 Macintosh-2 com.apple.SecurityServer[21]: Succeeded >>>> authorizing right system.login.tty by client /usr/sbin/sshd for >>>> authorization created by /usr/sbin/sshd. >>>> Feb 6 15:23:18 Macintosh-2 sshd[2372]: Accepted keyboard-interactive/pam >>>> for greg from 10.1.1.1 port 49636 ssh2 >>>> >>>> --------- repo server log when capistrano is trying to access repo server >>>> from prod server ------------ >>>> Feb 6 15:23:53 Macintosh-2 sshd[2414]: error: PAM: Authentication >>>> failure for greg from home.gregsdomainname.org >>>> Feb 6 15:23:53: --- last message repeated 2 times --- >>>> Feb 6 15:23:53 Macintosh-2 sshd[2414]: Failed password for greg from >>>> 10.1.1.1 port 50366 ssh2 >>>> >>>> --------- cap console out --------------------- >>>> Macintosh-2:equity greg$ cap deploy:update >>>> * executing `deploy:update' >>>> ** transaction: start >>>> * executing `deploy:update_code' >>>> updating the cached checkout on all servers >>>> executing locally: "git ls-remote . HEAD" >>>> * executing "if [ -d /u/apps/equity/shared/cached-copy ]; then cd >>>> /u/apps/equity/shared/cached-copy && git fetch -q origin && git reset -q >>>> --hard 581568057e9bc8d41a9681c15ad27d778faa551b; else git clone -q >>>> [email protected]:/Users/greg/source/equity/.git >>>> /u/apps/equity/shared/cached-copy && cd /u/apps/equity/shared/cached-copy && >>>> git checkout -q -b deploy 581568057e9bc8d41a9681c15ad27d778faa551b; fi" >>>> servers: ["10.1.1.1"] >>>> Enter passphrase for /Users/greg/.ssh/id_rsa: >>>> [[email protected]] executing command >>>> ** [10.1.1.1 :: err] Permission denied, please try again. >>>> ** [10.1.1.1 :: err] Permission denied, please try again. >>>> ** [10.1.1.1 :: err] Permission denied >>>> (publickey,password,keyboard-interactive). >>>> ** [10.1.1.1 :: err] fatal: The remote end hung up unexpectedly >>>> command finished >>>> *** [deploy:update_code] rolling back >>>> * executing "rm -rf /u/apps/equity/releases/20090206051539; true" >>>> servers: ["10.1.1.1"] >>>> [[email protected]] executing command >>>> command finished >>>> failed: "sh -c \"if [ -d /u/apps/equity/shared/cached-copy ]; then cd >>>> /u/apps/equity/shared/cached-copy && git fetch -q origin && git reset -q >>>> --hard 581568057e9bc8d41a9681c15ad27d778faa551b; else git clone -q >>>> [email protected]:/Users/greg/source/equity/.git >>>> /u/apps/equity/shared/cached-copy && cd /u/apps/equity/shared/cached-copy && >>>> git checkout -q -b deploy 581568057e9bc8d41a9681c15ad27d778faa551b; fi\"" on >>>> [email protected] >>>> >>>> Summary of Configuration >>>> ==================== >>>> * Have two machines: >>>> - MacBook = Development & Git Repository >>>> - Linux Box (Redhat) = Target Prod Server >>>> * So the Linux box is calling back to the same Macbook for the >>>> repository. >>>> "cap deploy"[MacBook"] ===> "runs commands"[Linux Box] ==> "Git >>>> Repo"[MacBook] >>>> >>>> My MacBook /etc/ssh_config >>>> ========================= >>>> Macintosh-2:etc greg$ cat /etc/ssh_config >>>> # Host * >>>> # ForwardAgent no >>>> # ForwardX11 no >>>> # RhostsRSAAuthentication no >>>> # RSAAuthentication yes >>>> PasswordAuthentication yes >>>> # HostbasedAuthentication no >>>> # GSSAPIAuthentication yes >>>> # GSSAPIDelegateCredentials no >>>> # GSSAPIKeyExchange yes >>>> # GSSAPITrustDNS no >>>> # BatchMode no >>>> # CheckHostIP yes >>>> # AddressFamily any >>>> # ConnectTimeout 0 >>>> # StrictHostKeyChecking ask >>>> # IdentityFile ~/.ssh/identity >>>> # IdentityFile ~/.ssh/id_rsa >>>> # IdentityFile ~/.ssh/id_dsa >>>> # Port 22 >>>> # Protocol 2,1 >>>> # Cipher 3des >>>> # Ciphers >>>> aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc >>>> # EscapeChar ~ >>>> # Tunnel no >>>> # TunnelDevice any:any >>>> # PermitLocalCommand no >>>> >>>> >>>> >>>> >>>> Thanks >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>> >>> >>> >>> -- >>> Greg >>> http://blog.gregnet.org/ >>> >>> >> >> >> >> -- >> Greg >> http://blog.gregnet.org/ >> >> >> >> > >> > > > > -- Greg http://blog.gregnet.org/ --~--~---------~--~----~------------~-------~--~----~ To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/capistrano -~----------~----~----~----~------~----~------~--~---
