git fetch pack freezes
Hello, We are using git with the smart HTTP protocol. We are seeing the client freeze with Jenkins poll. It stops the continuous build which is a bit of a pain. It happens with the latest release 1.8.0 on the client side. The server is using 1.7.0.4. On the server we are seeing the following error message: var/log/apache2/error.log.1:[Mon Nov 19 15:17:12 2012] [error] [client 192.168.50.15] fatal: git upload-pack: not our ref aa92ca2076c2421433f89ee0b0ae89cd36d27481 multi_ack_detailed side-band-64k thin-pack no-progress ofs-delta The error is located in upload-pack.c: * Hmph. We may later want to allow want line that * asks for something like master~10 (symbolic)... * would it make sense? I don't know. */ o = lookup_object(sha1_buf); if (!o || !(o-flags OUR_REF)) die(git upload-pack: not our ref %s, line+5); It calls die with I believe is doing the right thing(TM). On the client side I compiled git with debugging symbols. I then attached the debugger to the process that gets hosed. I get the following backtrace: #0 0x7fff85389af0 in read () #1 0x00010f900913 in xread (fd=0, buf=0x7fff6f376e2c, len=4) at wrapper.c:142 #2 0x00010f900a11 in read_in_full (fd=0, buf=0x7fff6f376e2c, count=4) at wrapper.c:171 #3 0x00010f8a14e4 in safe_read (fd=0, buffer=0x7fff6f376e2c, size=4, return_line_fail=0) at pkt-line.c:140 #4 0x00010f8a110e in packet_read_internal (fd=0, buffer=0x10f981eb0 , size=1000, return_line_fail=0) at pkt-line.c:183 #5 0x00010f8a1246 in packet_read_line (fd=0, buffer=0x10f981eb0 , size=1000) at pkt-line.c:211 #6 0x00010f7b5196 in get_ack (fd=0, result_sha1=0x7fff6f377450 ) at builtin/fetch-pack.c:227 #7 0x00010f7b4074 in find_common (fd=0x7fff6f377668, result_sha1=0x7fff6f377450 , refs=0x0) at builtin/fetch-pack.c:416 #8 0x00010f7b32a5 in do_fetch_pack (fd=0x7fff6f377668, orig_ref=0x10fb007a0, sought=0x7fff6f377670, pack_lockfile=0x7fff6f377660) at builtin/fetch-pack.c:832 #9 0x00010f7b2cbd in fetch_pack (my_args=0x10f973fe8, fd=0x7fff6f377668, conn=0x0, ref=0x10fb007a0, dest=0x7fff6f377a78 https://foo.visionobjects.com/git/a-project-name/;, sought=0x7fff6f377670, pack_lockfile=0x7fff6f377660) at builtin/fetch-pack.c:1069 #10 0x00010f7b2a25 in cmd_fetch_pack (argc=7, argv=0x7fff6f377890, prefix=0x0) at builtin/fetch-pack.c:1012 #11 0x00010f77aa5b in run_builtin (p=0x10f971990, argc=7, argv=0x7fff6f377890) at git.c:306 #12 0x00010f779ac7 in handle_internal_command (argc=7, argv=0x7fff6f377890) at git.c:467 #13 0x00010f77a299 in run_argv (argcp=0x7fff6f377858, argv=0x7fff6f377850) at git.c:513 #14 0x00010f7798fd in main (argc=7, argv=0x7fff6f377890) at git.c:588 The file descriptor being 0 in #6 seems a bit suspicious. I am not familiar with the git protocol. Could I get some guidance on how to troubleshoot some more so that I come up with a patch? -- Ivan Kanis, Release Manager, Vision Objects, Tel +33 2 28 01 49 50, Fax +33 2 40 25 89 20 http://www.visionobjects.com The essence of science: ask an impertinent question, and you are on the way to a pertinent answer. -- Jacob Bronowski -- To unsubscribe from this list: send the line unsubscribe git in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: git fetch pack freezes
On Wed, Nov 28, 2012 at 6:12 AM, Ivan Kanis ivan.ka...@googlemail.com wrote: We are using git with the smart HTTP protocol. We are seeing the client freeze with Jenkins poll. It stops the continuous build which is a bit of a pain. It happens with the latest release 1.8.0 on the client side. The server is using 1.7.0.4. On the server we are seeing the following error message: var/log/apache2/error.log.1:[Mon Nov 19 15:17:12 2012] [error] [client 192.168.50.15] fatal: git upload-pack: not our ref aa92ca2076c2421433f89ee0b0ae89cd36d27481 multi_ack_detailed side-band-64k thin-pack no-progress ofs-delta Upgrade your server. What happened here is a race condition that is only possible in the smart HTTP protocol. The client makes an initial request to ask where the branches are. Then it makes a second (and possibly more) subsequent request to negotiate a common ancestor and actually obtain the necessary objects. If a branch is updated between that initial request and any subsequent request, the server fails with this error message. More recent servers handle this more gracefully by looking to see if the object the client wants is reachable within some reasonable distance from a current branch. If so, the request is still allowed to continue, rather than die() with a failure. On the client side I compiled git with debugging symbols. I then attached the debugger to the process that gets hosed. I get the following backtrace: #5 0x00010f8a1246 in packet_read_line (fd=0, buffer=0x10f981eb0 , size=1000) at pkt-line.c:211 #6 0x00010f7b5196 in get_ack (fd=0, result_sha1=0x7fff6f377450 ) at builtin/fetch-pack.c:227 #7 0x00010f7b4074 in find_common (fd=0x7fff6f377668, result_sha1=0x7fff6f377450 , refs=0x0) at builtin/fetch-pack.c:416 The file descriptor being 0 in #6 seems a bit suspicious. I am not familiar with the git protocol. Could I get some guidance on how to troubleshoot some more so that I come up with a patch? There is actually a parent process in front of this one called git-remote-http (or git-remote-https depending on protocol). That parent is doing the HTTP transport and is sending whatever the server provides into this child's stdin. I'm not surprised there is yet another deadlock condition in here, its a rather heinous process setup and way to handle the protocol. fd in frame #7 is an address of an int array. fd in frame #6 is an actual fd, the one to read from, which in this case should be 0 to read from stdin, because its trying to get data from the HTTP server by way of reading it from the pipe the parent supplied to it on stdin. So the stack frames are correct. Its just a problem that the parent didn't identify the server crashing and closing its side of the pipe on stdin to force it to EOF to prevent the child from getting hung here in a read. -- To unsubscribe from this list: send the line unsubscribe git in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: git fetch pack freezes
Hi Shawn, Thanks for the quick reply. Shawn Pearce spea...@spearce.org wrote: On Wed, Nov 28, 2012 at 6:12 AM, Ivan Kanis ivan.ka...@googlemail.com wrote: On the server we are seeing the following error message: Upgrade your server. OK we'll look into it. I have a question: will a 1.8 server still serve 1.7 git client? So the stack frames are correct. Its just a problem that the parent didn't identify the server crashing and closing its side of the pipe on stdin to force it to EOF to prevent the child from getting hung here in a read. It sounds like a bug on the client, doesn't it? -- Ivan Kanis, Release Manager, Vision Objects, Tel +33 2 28 01 49 50, Fax +33 2 40 25 89 20 http://www.visionobjects.com Let a fool hold his tongue and he will pass for a sage. -- Publilius Syrus -- To unsubscribe from this list: send the line unsubscribe git in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: git fetch pack freezes
On Wed, Nov 28, 2012 at 9:25 AM, Ivan Kanis ivan.ka...@googlemail.com wrote: Shawn Pearce spea...@spearce.org wrote: On Wed, Nov 28, 2012 at 6:12 AM, Ivan Kanis ivan.ka...@googlemail.com wrote: On the server we are seeing the following error message: Upgrade your server. OK we'll look into it. I have a question: will a 1.8 server still serve 1.7 git client? Yes. So the stack frames are correct. Its just a problem that the parent didn't identify the server crashing and closing its side of the pipe on stdin to force it to EOF to prevent the child from getting hung here in a read. It sounds like a bug on the client, doesn't it? Yes. -- To unsubscribe from this list: send the line unsubscribe git in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
