Re: concurrent access to multiple local git repos is error prone
On Mon, Aug 6, 2018 at 9:38 AM Alexander Mills wrote: > Yeah this concurrency problem is real. Not only does it happen with > `git status` the same thing happens with `git rev-parse > --show-toplevel`. "git rev-parse --show-toplevel" having this same problem helps. This command should never make any update in the repository, not even taking any lock and very basic access to the repository (I think it just needs to resolve HEAD, it does not even access object database). It's so "simple" [1] that makes me think this problem is something not really related to git. Perhaps the process is terminated abnormally because it hits some system limits? [1] well the repo discovery steps are a bit messy but I think it's unlikely we have any racing issues there. > What happens is that I get no stdout when repos are accessed > concurrently (and no stderr). -- Duy
Re: concurrent access to multiple local git repos is error prone
(administrivia: please don't top-post) Hi Alex, Alexander Mills wrote: > Yeah this concurrency problem is real. Not only does it happen with > `git status` the same thing happens with `git rev-parse > --show-toplevel`. Sorry for the confusion --- I didn't mean to claim your experience was not real! What I wanted to make clear is that 1. Git is designed to allow concurrent reads of a repository (and pushes to a repository). If it doesn't work, that is a simple bug, not a design goal. 2. Plenty of people rely on concurrently accessing repositories, so if it doesn't work, then (i) we definitely want to know and (ii) we're going to need a lot of detail to figure out what's happening, so we can fix it. Does that make it clearer? > What happens is that I get no stdout when repos are accessed > concurrently (and no stderr). If I limit concurrency to 1, the problem > goes away. When I up the concurrency, the problem is sporadic, which > is the exact signal for a concurrency/race-condition related issue. > The signs are damn clear. I have seen this problem on MacOS I think a > year back on a different project, but I never reported it b/c I hadn't > really verified it. > > Like I said I am on Ubuntu. I have 3 git repos that are incorporated > into the tool that's generating the problem. For one repo I got this: > > $ git fsck > > Checking object directories: 100% (256/256), done. > dangling tree 4b825dc642cb6eb9a060e54bf8d69288fbee4904 > > For `$ git version --build-options` I have: > > git version 2.17.1 > cpu: x86_64 > no commit associated with this build > sizeof-long: 8 Thanks. My best idea for a next step is that if you can come up with a reproduction recipe, that would be very helpful. It doesn't have to reproduce 100% of the time, but e.g. if you have a script that reproduces it 50% of the time, I can run that script in a loop. Thanks, Jonathan
Re: concurrent access to multiple local git repos is error prone
To add something to the previous message, I have strong evidence that the problem occurs when *different* repos are accessed concurrency, not the same repo, as bizarre as that may be. -alex On Mon, Aug 6, 2018 at 12:36 AM, Alexander Mills wrote: > Hi Johnathan, > > Yeah this concurrency problem is real. Not only does it happen with > `git status` the same thing happens with `git rev-parse > --show-toplevel`. > What happens is that I get no stdout when repos are accessed > concurrently (and no stderr). If I limit concurrency to 1, the problem > goes away. When I up the concurrency, the problem is sporadic, which > is the exact signal for a concurrency/race-condition related issue. > The signs are damn clear. I have seen this problem on MacOS I think a > year back on a different project, but I never reported it b/c I hadn't > really verified it. > > Like I said I am on Ubuntu. I have 3 git repos that are incorporated > into the tool that's generating the problem. For one repo I got this: > > $ git fsck > > Checking object directories: 100% (256/256), done. > dangling tree 4b825dc642cb6eb9a060e54bf8d69288fbee4904 > > For `$ git version --build-options` I have: > > git version 2.17.1 > cpu: x86_64 > no commit associated with this build > sizeof-long: 8 > > > -alex > > > > On Sun, Aug 5, 2018 at 1:11 AM, Jonathan Nieder wrote: >> Alexander Mills wrote: >> >>> Yeah in this case what appeared to be happening is that if `git status` was >>> called concurrently, frequently I wouldnt get any stdout.. >> >> Thanks for reporting. A few questions: >> >> What platform are you on? What version of Git are you running? What >> is the output of "git version --build-options"? >> >> What is the exit status from these "git status" invocations that >> didn't write output? What options are you passing to "git status"? >> Are there any other interesting symptoms? >> >> Is this part of a larger tool or script? Are there other operations >> going on (e.g. something sending signals to these git processes)? >> >> Does the repository pass "git fsck"? Any other symptoms or hints that >> could help in tracking this down? >> >> Can you paste a transcript of the commands or script you ran and what >> output it produced? Is this something I should be able to reproduce? >> >> Thanks and hope that helps, >> Jonathan > > > > -- > Alexander D. Mills > ¡¡¡ New cell phone number: (415)730-1805 !!! > alexander.d.mi...@gmail.com > > www.linkedin.com/pub/alexander-mills/b/7a5/418/ -- Alexander D. Mills ¡¡¡ New cell phone number: (415)730-1805 !!! alexander.d.mi...@gmail.com www.linkedin.com/pub/alexander-mills/b/7a5/418/
Re: concurrent access to multiple local git repos is error prone
Hi Johnathan, Yeah this concurrency problem is real. Not only does it happen with `git status` the same thing happens with `git rev-parse --show-toplevel`. What happens is that I get no stdout when repos are accessed concurrently (and no stderr). If I limit concurrency to 1, the problem goes away. When I up the concurrency, the problem is sporadic, which is the exact signal for a concurrency/race-condition related issue. The signs are damn clear. I have seen this problem on MacOS I think a year back on a different project, but I never reported it b/c I hadn't really verified it. Like I said I am on Ubuntu. I have 3 git repos that are incorporated into the tool that's generating the problem. For one repo I got this: $ git fsck Checking object directories: 100% (256/256), done. dangling tree 4b825dc642cb6eb9a060e54bf8d69288fbee4904 For `$ git version --build-options` I have: git version 2.17.1 cpu: x86_64 no commit associated with this build sizeof-long: 8 -alex On Sun, Aug 5, 2018 at 1:11 AM, Jonathan Nieder wrote: > Alexander Mills wrote: > >> Yeah in this case what appeared to be happening is that if `git status` was >> called concurrently, frequently I wouldnt get any stdout.. > > Thanks for reporting. A few questions: > > What platform are you on? What version of Git are you running? What > is the output of "git version --build-options"? > > What is the exit status from these "git status" invocations that > didn't write output? What options are you passing to "git status"? > Are there any other interesting symptoms? > > Is this part of a larger tool or script? Are there other operations > going on (e.g. something sending signals to these git processes)? > > Does the repository pass "git fsck"? Any other symptoms or hints that > could help in tracking this down? > > Can you paste a transcript of the commands or script you ran and what > output it produced? Is this something I should be able to reproduce? > > Thanks and hope that helps, > Jonathan -- Alexander D. Mills ¡¡¡ New cell phone number: (415)730-1805 !!! alexander.d.mi...@gmail.com www.linkedin.com/pub/alexander-mills/b/7a5/418/
exit code in git diff-index [was: Re: concurrent access to multiple local git repos is error prone]
On 08/05, Alexander Mills wrote: > Also, as an aside, this seems to be a bug, but probably a known bug: > > $ git diff-index HEAD; echo $? > > :100755 100755 60e5d683c1eb3e61381b1a8ec2db822b94b9faec > M cli/npp_check_merge.sh > :100644 100644 35a453544de41e2227ab0afab31a396d299139e9 > M src/find-projects.ts > :100644 100644 c1ee7bc18e6604cbf0d16653e9366109d6ac2ec9 > M src/tables.ts > :100644 100644 29d9674fbb48f223f3434179d666b2aa991ad05a > M > src/vcs-helpers/git-helpers.ts > 0 > > $ git diff-index --quiet HEAD; echo $? > 1 > > different exit codes depending on whether --quiet was used. In this > case, the exit code should be consistent. > The bug is with the `git diff-index` command, as you can see. This is not a bug. 'git diff-index' (and 'git diff') only give an exit code other than 0 in the default case if something actually goes wrong with generating the diff, which in the usual case it shouldn't. To get an exit code from 'git diff-index' if there are differences, you'd have to pass the '--exit-code' flag. The '--quite' flag implies '--exit-code', as there's not much use in 'git diff --quiet' if there's not even an exit code showing whether there are differences or not. The original patch (and more importantly the reasoning why '--exit-code' is not the default behaviour for 'git diff') can be found at [1]. [1]: https://public-inbox.org/git/81b0412b0703131717k7106ee1cg964628f0bda2c...@mail.gmail.com/ > -alex
Re: concurrent access to multiple local git repos is error prone
I am on Ubuntu, I will run the checks that you mentioned. I assume it's operator error for now though. Like I said, it was just `git status` that was sometimes failing to send any stdout, presumably when access to a certain git repo was concurrent, but possibly due to concurrent access across multiple repos. Also, as an aside, this seems to be a bug, but probably a known bug: $ git diff-index HEAD; echo $? :100755 100755 60e5d683c1eb3e61381b1a8ec2db822b94b9faec M cli/npp_check_merge.sh :100644 100644 35a453544de41e2227ab0afab31a396d299139e9 M src/find-projects.ts :100644 100644 c1ee7bc18e6604cbf0d16653e9366109d6ac2ec9 M src/tables.ts :100644 100644 29d9674fbb48f223f3434179d666b2aa991ad05a M src/vcs-helpers/git-helpers.ts 0 $ git diff-index --quiet HEAD; echo $? 1 different exit codes depending on whether --quiet was used. In this case, the exit code should be consistent. The bug is with the `git diff-index` command, as you can see. -alex On Sun, Aug 5, 2018 at 1:11 AM, Jonathan Nieder wrote: > Alexander Mills wrote: > >> Yeah in this case what appeared to be happening is that if `git status` was >> called concurrently, frequently I wouldnt get any stdout.. > > Thanks for reporting. A few questions: > > What platform are you on? What version of Git are you running? What > is the output of "git version --build-options"? > > What is the exit status from these "git status" invocations that > didn't write output? What options are you passing to "git status"? > Are there any other interesting symptoms? > > Is this part of a larger tool or script? Are there other operations > going on (e.g. something sending signals to these git processes)? > > Does the repository pass "git fsck"? Any other symptoms or hints that > could help in tracking this down? > > Can you paste a transcript of the commands or script you ran and what > output it produced? Is this something I should be able to reproduce? > > Thanks and hope that helps, > Jonathan -- Alexander D. Mills ¡¡¡ New cell phone number: (415)730-1805 !!! alexander.d.mi...@gmail.com www.linkedin.com/pub/alexander-mills/b/7a5/418/
Re: concurrent access to multiple local git repos is error prone
Alexander Mills wrote: > Yeah in this case what appeared to be happening is that if `git status` was > called concurrently, frequently I wouldnt get any stdout.. Thanks for reporting. A few questions: What platform are you on? What version of Git are you running? What is the output of "git version --build-options"? What is the exit status from these "git status" invocations that didn't write output? What options are you passing to "git status"? Are there any other interesting symptoms? Is this part of a larger tool or script? Are there other operations going on (e.g. something sending signals to these git processes)? Does the repository pass "git fsck"? Any other symptoms or hints that could help in tracking this down? Can you paste a transcript of the commands or script you ran and what output it produced? Is this something I should be able to reproduce? Thanks and hope that helps, Jonathan
Re: concurrent access to multiple local git repos is error prone
Hi, Alexander Mills wrote: > Yeah as long as git shouldnt fumble wrt concurrent access across repos, it > was most likely the same repo being accessed concurrently and that's what > was causing the issue. > > that being said, it would be really nice if git itself could handle > concurrent requests to the same repo, that definitely seems to be a > limitation. Instead of all these guesses, I'd prefer if you describe the actual symptoms. Git is meant to support concurrent accesses of the same repo as well, and I haven't heard of any rumors of that being broken, either. Sincerely, Jonathan
Re: concurrent access to multiple local git repos is error prone
Hi Alexander, Alexander Mills wrote: > On Sat, Aug 4, 2018 at 2:47 PM, Alexander Mills > wrote: >> I assume that access more than 1 git repo concurrently on a local >> machine is not without errors. However this seems like a huge >> limitation or design flaw. >> >> Is my observation correct? Are there any plans to remove this limitation? [...] > Note, in my first paragraph, I should have said "If I have multiple > local git repos, and I run `git status -v` on them *concurrently*"... There's no known limitation of this kind, no. It sounds quite strange, and I've never heard of anything like that before. Do you have more details? Thanks and hope that helps, Jonathan
Re: concurrent access to multiple local git repos is error prone
Note, in my first paragraph, I should have said "If I have multiple local git repos, and I run `git status -v` on them *concurrently*"... -alex On Sat, Aug 4, 2018 at 2:47 PM, Alexander Mills wrote: > If I have multiple local git repos, and I run `git status -v` on them, > sometimes I don't get any stdout for the command. This is highly > reproducible. > > I assume that access more than 1 git repo concurrently on a local > machine is not without errors. However this seems like a huge > limitation or design flaw. > > Is my observation correct? Are there any plans to remove this limitation? > > My use case - I create a lot of developer tools and more than once I > have hit this limitation...I have to create a queue with concurrency > of 1 to run commands on git repos. It's very strange and > counterintuitive to have to do this. > > > -- > Alexander D. Mills > ¡¡¡ New cell phone number: (415)730-1805 !!! > alexander.d.mi...@gmail.com > > www.linkedin.com/pub/alexander-mills/b/7a5/418/ -- Alexander D. Mills ¡¡¡ New cell phone number: (415)730-1805 !!! alexander.d.mi...@gmail.com www.linkedin.com/pub/alexander-mills/b/7a5/418/
concurrent access to multiple local git repos is error prone
If I have multiple local git repos, and I run `git status -v` on them, sometimes I don't get any stdout for the command. This is highly reproducible. I assume that access more than 1 git repo concurrently on a local machine is not without errors. However this seems like a huge limitation or design flaw. Is my observation correct? Are there any plans to remove this limitation? My use case - I create a lot of developer tools and more than once I have hit this limitation...I have to create a queue with concurrency of 1 to run commands on git repos. It's very strange and counterintuitive to have to do this. -- Alexander D. Mills ¡¡¡ New cell phone number: (415)730-1805 !!! alexander.d.mi...@gmail.com www.linkedin.com/pub/alexander-mills/b/7a5/418/
