Greg, First, let me standardize on some terminology, so we're both talking about the same things:
* "local machine" is the server you are running capistrano on. I'm assuming this is what you mean by "DEV box" and "DevServer". * "remote hosts" are the servers you are targeting with Capistrano. I'm assuming this is what you are meaning by "PROD box" and "ProdServer". * "repository host" or "repository url" is the server that is hosting the git repository. I'm assuming this is what you mean by "REPO server". Now, here's the output you originally shared, with annotations by me: > Macintosh-2:equity greg$ cap deploy:update -d > * executing `deploy:update' > ** transaction: start > * executing `deploy:update_code' > updating the cached checkout on all servers > executing locally: "git ls-remote [email protected]:/repos/equity/.git HEAD" "executing locally" means just that: it is executing the given command locally, on your local machine. In this case, Capistrano is executing "git ls-remote" on your local machine, connecting directly to the repository server. It succeeds, indicating that everything (public keys, etc.) is set up correctly for connections from your local host to the repository server. > * executing "if [ -d /u/apps/equity/shared/cached-copy ]; then cd > /u/apps/equity/shared/cached-copy && git fetch -q origin && git reset -q > --hard 7e92223c93c6f9019acd54d042fc8d348aa62a53; else git clone -q > [email protected]:/repos/equity/.git /u/apps/equity/shared/cached-copy && cd > /u/apps/equity/shared/cached-copy && git checkout -q -b deploy > 7e92223c93c6f9019acd54d042fc8d348aa62a53; fi" This is the next command that Capistrano is going to execute, and in this case, it is going to execute it on the remote hosts (or remote _host_, if you're deploying to only a single server). > Feb 6 20:36:24 home sshd_local(pam_unix)[26758]: session opened for user > root by (uid=0) > Feb 6 20:36:25 home sshd_local(pam_unix)[26758]: session closed for user > root This is really weird to me. It looks like your syslog is logging to the console? Anyway, I'm ignoring it, but if you don't know what it's from either, maybe you'll want to investigate it since it might be a further symptom of the problem you're having. > servers: ["10.1.1.1"] This line is capistrano telling you which remote servers it is going to execute the command on. In this case, it looks like it is the same as the repository server. > Enter passphrase for /Users/greg/.ssh/id_rsa: > [[email protected]] executing command Capistrano has successfully connected to the remote server at this point, and is commencing execution of the command. If you ever see this "executing command" bit, then you can rest assured that Capistrano _has_ connected and authenticated to your remote host(s) successfully. > ** [10.1.1.1 :: err] Permission denied > (publickey,gssapi-with-mic,keyboard-interactive). > ** [10.1.1.1 :: err] fatal: The remote end hung up unexpectedly > command finished So, given the fact that we know Capistrano connected from the local host to your remote host successfully, coupled with the fact that the log lines above are prefixed with "[10.1.1.1 :: err]", we can therefore know that the "permission denied" and "remote end hung up" messages are not from Capistrano, and are not about the connection from your local host to the remote host. The messages therefore must be output from the command that was just executed, the "git clone" (or "git fetch"), in this case. So, let's look closer at the clone command: git clone -q [email protected]:/repos/equity/.git /u/apps/equity/shared/cached-copy Well, there are two interesting things here, to me. The first is that you're using 'root' to log in for git. Not seeing the rest of your deploy.rb, I have to ask: are setting the :user variable to root, also? In other words, are the remote commands being executed as root, or as some other user? (I won't get into the severe security issues with ever ssh'ing into a server as root.) The other interesting thing is that the remote host and the repository host are the same. So, the first few things I'd suggest testing are this: * If you log into 10.1.1.1 as whichever user you're deploying as (root?) and then try to ssh FROM THERE to 10.1.1.1 AS ROOT, does it prompt you for a password? If not, does the log in succeed? If it does not, then the issue is entirely unrelated to capistrano, and is instead about configuring your public keys so that you can log in from 10.1.1.1 to 10.1.1.1. * Assuming you can ssh from 10.1.1.1 to 10.1.1.1 as root, can you execute the "git clone" command that cap is trying to do? Log into 10.1.1.1 (as your deploy user), and then try the execute the following command: "git clone -q [email protected]:/repos/equity/.git /tmp/test" Lastly, since the repo URL is the same as the remote host, you might want to consider this reconfiguration. Instead of setting the :repository variable to "[email protected]:/repos/equity/.git" (and, assuming your deploy user is "root", too), you might try this: set :repository, "/repos/equity/.git" set :local_repository, "[email protected]:/repos/equity/.git" If capistrano sees a :local_repository variable, it will use that URL for all requests from the local host. The :repository variable will then be used only for requests from the remote host. In this way you can tell git to skip the SSH transport when requesting from the remote host (since it's the same as the repo URL). Hope that helps some. - Jamis On Sat, Feb 7, 2009 at 12:51 AM, Greg Hauptmann <[email protected]> wrote: > thanks Jamis, > > The interesting thing is I can copy/paste the commands capistrano is trying > to issue (from the console, I did a "dry-run") and then use this on the prod > server and everything works fine (i.e. prod server to git repo server calls > are working fine). Can I ask a couple of quick questions: > > Is there anything different from an ssh call point of view between then: > > (a) the first call which works (i.e. executing locally: "git ls-remote > [email protected]:/repos/equity/.git HEAD") and > > (b) second call which has authentication issue (i.e. "f [ -d > /u/apps/equity/shared/cached-copy ]; then cd > /u/apps/equity/shared/cached-copy && git fetch -q origin && git reset -q > --hard 7e92223c93c6f9019acd54d042fc8d348aa62a53; else git clone -q > [email protected]:/repos/equity/.git /u/apps/equity/shared/cached-copy && cd > /u/apps/equity/shared/cached-copy && git checkout -q -b deploy > 7e92223c93c6f9019acd54d042fc8d348aa62a53; fi") > > Q1 - Do you see any difference in this first & second call from an ssh > connection point of view? Or is this also strange to you that the first > call works fine, but then the 2nd call comes along and can't connect from > the DEV box to the PROD box? (or actually from your response is the main > difference that the 1st targets the REPO server, whilst the 2nd then targets > the PROD server - hence in my case am I really getting an authentication > issue for the first capistrano call that is targeting the PROD server) > Q2 - I'm assuming the authentication issue is my DEV box authenticating to > the PROD box? (i.e. not the PROD box trying to run GIT to contact the REPO > server) Is what you believe based on the logs? And also can I ask when > you give your answer why (i.e. so I can learn re how you interpreted things) > You did mention ("Capistrano opens a new SSH "channel" for each request.") > Q3 - If it turns out to be an ssh/linux issue somehow, have you a pointer re > where to look in the capistrano code? (i.e. if I had to post the code that > made the call onto an ssh/linux forum). I'm finding it kind of confusing as > I run capistrano of Server A, which authenticates to Server B, but the > scripts actually run on Server B, which then has to authenticate from Server > B to Server C, and to complicate things I think you pointed out the last ssh > connection (server B to server C[repo]) is not really capistrano but git. > :) > > Thanks again > Greg > > > 2009/2/7 Jamis Buck <[email protected]>: >> >> Greg, >> >> 1. Capistrano opens a new SSH "channel" for each request. Essentially, >> this means a new shell instance for each command, and means you cannot >> use stateful commands (e.g., run("cd foo") followed by run("pwd") will >> not display "foo"). If you need commands to execute from within a >> specific directory, or with a specific set of environment variables, >> or whatever, you must make sure to change directory or set environment >> variables every time you run() something. >> >> 2. Yes. Capistrano needs to query the current revision, and does so >> from your local machine (DevServer, in your terminology). Once it >> knows the revision to deploy, it will push the code to the servers >> using the deployment strategy you selected (which may or may not do a >> code pull from the target machines). >> >> As for the "remote end hung up unexpectedly" error, I'm not sure. The >> error is coming from git on the remote server, and not from >> capistrano, so you need to make sure that git on the remote host can >> connect to the repository. Beyond that, I'm not sure what >> troubleshooting help I can offer. >> >> - Jamis >> >> On Fri, Feb 6, 2009 at 3:49 AM, Greg Hauptmann >> <[email protected]> wrote: >>> PSS. It seems like, in terms of the ssh connections: >>> (1st connection) the initial ssh connection open by "cap deploy:update" >>> actually *closes* prior to the next one [I'm guessing this is the >>> problem?] >>> (2nd connection) the connection from the prod app server to the git repo >>> server then seems to occur from DevServer => ProdAppServer, as opposed to >>> from ProdAppServer => ProdAppServer(where repo is). >>> This would explain why for the repo connection password request I'm >>> seeing >>> output (e.g. Enter passphrase for /Users/greg/.ssh/id_rsa) that >>> highlights >>> the request is coming from my DevServer, and NOT the ProdAppServer. >>> Make >>> sense? >>> QUESTION THEREFORE: If this is correct, why would the SSH connection >>> after >>> the initial "cap deploy:update" drop? >>> Here's a copy/paste of output over time. I've pasted the first password >>> request, and the session open/closing log statements, then I've copied >>> paste >>> the capistrano output for the 2nd password request phase, and it's >>> corresponding log output: >>> -------------- >>> Macintosh-2:equity greg$ cap deploy:update -d >>> * executing `deploy:update' >>> ** transaction: start >>> * executing `deploy:update_code' >>> updating the cached checkout on all servers >>> executing locally: "git ls-remote [email protected]:/repos/equity/.git >>> HEAD" >>> * executing "if [ -d /u/apps/equity/shared/cached-copy ]; then cd >>> /u/apps/equity/shared/cached-copy && git fetch -q origin && git reset -q >>> --hard 7e92223c93c6f9019acd54d042fc8d348aa62a53; else git clone -q >>> [email protected]:/repos/equity/.git /u/apps/equity/shared/cached-copy && cd >>> /u/apps/equity/shared/cached-copy && git checkout -q -b deploy >>> 7e92223c93c6f9019acd54d042fc8d348aa62a53; fi" >>> Preparing to execute command: "if [ -d /u/apps/equity/shared/cached-copy >>> ]; >>> then cd /u/apps/equity/shared/cached-copy && git fetch -q origin && git >>> reset -q --hard 7e92223c93c6f9019acd54d042fc8d348aa62a53; else git clone >>> -q >>> [email protected]:/repos/equity/.git /u/apps/equity/shared/cached-copy && cd >>> /u/apps/equity/shared/cached-copy && git checkout -q -b deploy >>> 7e92223c93c6f9019acd54d042fc8d348aa62a53; fi" >>> Execute ([Yes], No, Abort) ? |y| >>> >>> Feb 6 20:36:24 home sshd_local(pam_unix)[26758]: session opened for user >>> root by (uid=0) >>> Feb 6 20:36:25 home sshd_local(pam_unix)[26758]: session closed for user >>> root >>> -------------- >>> servers: ["10.1.1.1"] >>> Enter passphrase for /Users/greg/.ssh/id_rsa: >>> [[email protected]] executing command >>> ** [10.1.1.1 :: err] Permission denied >>> (publickey,gssapi-with-mic,keyboard-interactive). >>> ** [10.1.1.1 :: err] fatal: The remote end hung up unexpectedly >>> command finished >>> *** [deploy:update_code] rolling back >>> * executing "rm -rf /u/apps/equity/releases/20090206103749; true" >>> Preparing to execute command: "rm -rf >>> /u/apps/equity/releases/20090206103749; true" >>> Execute ([Yes], No, Abort) ? |y| >>> >>> Feb 6 20:37:41 home sshd_local(pam_unix)[7673]: session opened for user >>> root by (uid=0) >>> Feb 6 20:37:41 home sshd_local(pam_unix)[9423]: authentication failure; >>> logname= uid=0 euid=0 tty=ssh ruser= rhost=home.gregnet.org user=root >>> Feb 6 20:37:44 home sshd_local(pam_unix)[10530]: authentication failure; >>> logname= uid=0 euid=0 tty=ssh ruser= rhost=home.gregnet.org user=root >>> Feb 6 20:37:46 home sshd_local(pam_unix)[11605]: authentication failure; >>> logname= uid=0 euid=0 tty=ssh ruser= rhost=home.gregnet.org user=root >>> -------------- >>> >>> >>> 2009/2/6 Greg Hauptmann <[email protected]> >>>> >>>> PS. >>>> >>>> I've now cloned my repository manually onto my prod Linux box (under >>>> /repos) and updated deploy.config to reflect this. That is when the cap >>>> scripts are running on my prod box it should call out to itself now for >>>> git >>>> repository. I still get an authentication issue however (see below). >>>> There >>>> must be a difference re ssh login between when a script runs the git >>>> command, and when I manually run it? >>>> >>>> ----------prod messages log------------ >>>> Feb 6 16:52:00 home sshd_local(pam_unix)[10814]: session opened for >>>> user >>>> root by (uid=0) >>>> Feb 6 16:52:00 home sshd_local(pam_unix)[10814]: session closed for >>>> user >>>> root >>>> >>>> Feb 6 16:52:02 home sshd_local(pam_unix)[10833]: session opened for >>>> user >>>> root by (uid=0) >>>> Feb 6 16:52:03 home sshd_local(pam_unix)[10856]: authentication >>>> failure; >>>> logname= uid=0 euid=0 tty=ssh ruser= rhost=home.gregnet.org user=root >>>> Feb 6 16:52:05 home sshd_local(pam_unix)[10857]: authentication >>>> failure; >>>> logname= uid=0 euid=0 tty=ssh ruser= rhost=home.gregnet.org user=root >>>> Feb 6 16:52:07 home sshd_local(pam_unix)[10858]: authentication >>>> failure; >>>> logname= uid=0 euid=0 tty=ssh ruser= rhost=home.gregnet.org user=root >>>> Feb 6 16:52:10 home sshd_local(pam_unix)[10833]: session closed for >>>> user >>>> root >>>> >>>> >>>> >>>> -------prod secure log ----------------- >>>> Feb 6 16:55:12 home sshd_local[11041]: Accepted publickey for root from >>>> ::ffff:10.1.1.145 port 51107 ssh2 >>>> >>>> Feb 6 16:55:17 home sshd_local[11060]: Accepted publickey for root from >>>> ::ffff:10.1.1.145 port 51108 ssh2 >>>> Feb 6 16:55:17 home sshd_local[11082]: Postponed keyboard-interactive >>>> for >>>> root from ::ffff:10.1.1.1 port 43930 ssh2 >>>> Feb 6 16:55:20 home sshd_local[11081]: error: PAM: Authentication >>>> failure >>>> for root from home.gregnet.org >>>> Feb 6 16:55:20 home sshd_local[11082]: Postponed keyboard-interactive >>>> for >>>> root from ::ffff:10.1.1.1 port 43930 ssh2 >>>> Feb 6 16:55:22 home sshd_local[11081]: error: PAM: Authentication >>>> failure >>>> for root from home.gregnet.org >>>> Feb 6 16:55:22 home sshd_local[11082]: Postponed keyboard-interactive >>>> for >>>> root from ::ffff:10.1.1.1 port 43930 ssh2 >>>> Feb 6 16:55:24 home sshd_local[11081]: error: PAM: Authentication >>>> failure >>>> for root from home.gregnet.org >>>> Feb 6 16:55:24 home sshd_local[11082]: Connection closed by >>>> ::ffff:10.1.1.1 >>>> >>>> >>>> >>>> ----------capistrano output --------------- >>>> Macintosh-2:equity greg$ cap deploy:update >>>> * executing `deploy:update' >>>> ** transaction: start >>>> * executing `deploy:update_code' >>>> updating the cached checkout on all servers >>>> executing locally: "git ls-remote [email protected]:/repos/equity/.git >>>> HEAD" >>>> Enter passphrase for key '/Users/greg/.ssh/id_rsa': >>>> * executing "if [ -d /u/apps/equity/shared/cached-copy ]; then cd >>>> /u/apps/equity/shared/cached-copy && git fetch -q origin && git reset -q >>>> --hard 7e92223c93c6f9019acd54d042fc8d348aa62a53; else git clone -q >>>> [email protected]:/repos/equity/.git /u/apps/equity/shared/cached-copy && cd >>>> /u/apps/equity/shared/cached-copy && git checkout -q -b deploy >>>> 7e92223c93c6f9019acd54d042fc8d348aa62a53; fi" >>>> servers: ["10.1.1.1"] >>>> Enter passphrase for /Users/greg/.ssh/id_rsa: >>>> [[email protected]] executing command >>>> ** [10.1.1.1 :: err] Permission denied >>>> (publickey,gssapi-with-mic,keyboard-interactive). >>>> ** [10.1.1.1 :: err] fatal: The remote end hung up unexpectedly >>>> command finished >>>> *** [deploy:update_code] rolling back >>>> * executing "rm -rf /u/apps/equity/releases/20090206065213; true" >>>> servers: ["10.1.1.1"] >>>> [[email protected]] executing command >>>> command finished >>>> failed: "sh -c \"if [ -d /u/apps/equity/shared/cached-copy ]; then cd >>>> /u/apps/equity/shared/cached-copy && git fetch -q origin && git reset -q >>>> --hard 7e92223c93c6f9019acd54d042fc8d348aa62a53; else git clone -q >>>> [email protected]:/repos/equity/.git /u/apps/equity/shared/cached-copy && cd >>>> /u/apps/equity/shared/cached-copy && git checkout -q -b deploy >>>> 7e92223c93c6f9019acd54d042fc8d348aa62a53; fi\"" on [email protected] >>>> >>>> >>>> >>>> 2009/2/6 Greg Hauptmann <[email protected]> >>>>> >>>>> Hi guys, >>>>> >>>>> I'm stuck on this. I can ssh into my target prod server, and from >>>>> there >>>>> ssh into my repo server (for git) fine. When I run "cap >>>>> deploy:update" it >>>>> seems when the capistrano scripts running on my target prod server get >>>>> a >>>>> "failed password" when trying to access the repo server??? Any >>>>> ideas??? >>>>> Here's a tail of the secure.log on the repo server for both cases. >>>>> >>>>> ---------- repo server log when "manually ssh'ing in from prod server >>>>> to >>>>> repo server" ------------------- >>>>> Feb 6 15:23:18 Macintosh-2 com.apple.SecurityServer[21]: checkpw() >>>>> succeeded, creating credential for user greg >>>>> Feb 6 15:23:18 Macintosh-2 com.apple.SecurityServer[21]: checkpw() >>>>> succeeded, creating shared credential for user greg >>>>> Feb 6 15:23:18 Macintosh-2 com.apple.SecurityServer[21]: Succeeded >>>>> authorizing right system.login.tty by client /usr/sbin/sshd for >>>>> authorization created by /usr/sbin/sshd. >>>>> Feb 6 15:23:18 Macintosh-2 sshd[2372]: Accepted >>>>> keyboard-interactive/pam >>>>> for greg from 10.1.1.1 port 49636 ssh2 >>>>> >>>>> --------- repo server log when capistrano is trying to access repo >>>>> server >>>>> from prod server ------------ >>>>> Feb 6 15:23:53 Macintosh-2 sshd[2414]: error: PAM: Authentication >>>>> failure for greg from home.gregsdomainname.org >>>>> Feb 6 15:23:53: --- last message repeated 2 times --- >>>>> Feb 6 15:23:53 Macintosh-2 sshd[2414]: Failed password for greg from >>>>> 10.1.1.1 port 50366 ssh2 >>>>> >>>>> --------- cap console out --------------------- >>>>> Macintosh-2:equity greg$ cap deploy:update >>>>> * executing `deploy:update' >>>>> ** transaction: start >>>>> * executing `deploy:update_code' >>>>> updating the cached checkout on all servers >>>>> executing locally: "git ls-remote . HEAD" >>>>> * executing "if [ -d /u/apps/equity/shared/cached-copy ]; then cd >>>>> /u/apps/equity/shared/cached-copy && git fetch -q origin && git reset >>>>> -q >>>>> --hard 581568057e9bc8d41a9681c15ad27d778faa551b; else git clone -q >>>>> [email protected]:/Users/greg/source/equity/.git >>>>> /u/apps/equity/shared/cached-copy && cd >>>>> /u/apps/equity/shared/cached-copy && >>>>> git checkout -q -b deploy 581568057e9bc8d41a9681c15ad27d778faa551b; fi" >>>>> servers: ["10.1.1.1"] >>>>> Enter passphrase for /Users/greg/.ssh/id_rsa: >>>>> [[email protected]] executing command >>>>> ** [10.1.1.1 :: err] Permission denied, please try again. >>>>> ** [10.1.1.1 :: err] Permission denied, please try again. >>>>> ** [10.1.1.1 :: err] Permission denied >>>>> (publickey,password,keyboard-interactive). >>>>> ** [10.1.1.1 :: err] fatal: The remote end hung up unexpectedly >>>>> command finished >>>>> *** [deploy:update_code] rolling back >>>>> * executing "rm -rf /u/apps/equity/releases/20090206051539; true" >>>>> servers: ["10.1.1.1"] >>>>> [[email protected]] executing command >>>>> command finished >>>>> failed: "sh -c \"if [ -d /u/apps/equity/shared/cached-copy ]; then cd >>>>> /u/apps/equity/shared/cached-copy && git fetch -q origin && git reset >>>>> -q >>>>> --hard 581568057e9bc8d41a9681c15ad27d778faa551b; else git clone -q >>>>> [email protected]:/Users/greg/source/equity/.git >>>>> /u/apps/equity/shared/cached-copy && cd >>>>> /u/apps/equity/shared/cached-copy && >>>>> git checkout -q -b deploy 581568057e9bc8d41a9681c15ad27d778faa551b; >>>>> fi\"" on >>>>> [email protected] >>>>> >>>>> Summary of Configuration >>>>> ==================== >>>>> * Have two machines: >>>>> - MacBook = Development & Git Repository >>>>> - Linux Box (Redhat) = Target Prod Server >>>>> * So the Linux box is calling back to the same Macbook for the >>>>> repository. >>>>> "cap deploy"[MacBook"] ===> "runs commands"[Linux Box] ==> "Git >>>>> Repo"[MacBook] >>>>> >>>>> My MacBook /etc/ssh_config >>>>> ========================= >>>>> Macintosh-2:etc greg$ cat /etc/ssh_config >>>>> # Host * >>>>> # ForwardAgent no >>>>> # ForwardX11 no >>>>> # RhostsRSAAuthentication no >>>>> # RSAAuthentication yes >>>>> PasswordAuthentication yes >>>>> # HostbasedAuthentication no >>>>> # GSSAPIAuthentication yes >>>>> # GSSAPIDelegateCredentials no >>>>> # GSSAPIKeyExchange yes >>>>> # GSSAPITrustDNS no >>>>> # BatchMode no >>>>> # CheckHostIP yes >>>>> # AddressFamily any >>>>> # ConnectTimeout 0 >>>>> # StrictHostKeyChecking ask >>>>> # IdentityFile ~/.ssh/identity >>>>> # IdentityFile ~/.ssh/id_rsa >>>>> # IdentityFile ~/.ssh/id_dsa >>>>> # Port 22 >>>>> # Protocol 2,1 >>>>> # Cipher 3des >>>>> # Ciphers >>>>> >>>>> aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc >>>>> # EscapeChar ~ >>>>> # Tunnel no >>>>> # TunnelDevice any:any >>>>> # PermitLocalCommand no >>>>> >>>>> >>>>> >>>>> >>>>> Thanks >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> Greg >>>> http://blog.gregnet.org/ >>>> >>>> >>> >>> >>> >>> -- >>> Greg >>> http://blog.gregnet.org/ >>> >>> >>> >>> > >>> >> >> > > > -- > Greg > http://blog.gregnet.org/ > > > > > > > --~--~---------~--~----~------------~-------~--~----~ To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/capistrano -~----------~----~----~----~------~----~------~--~---
