Re: cherry-pick very slow on big repository

2017-11-21 Thread Elijah Newren
On Tue, Nov 21, 2017 at 4:07 AM, Peter Krefting wrote: > Elijah Newren: > >> Sure, take a look at the big-repo-small-cherry-pick branch of >> https://github.com/newren/git > > > With those changes, the time usage is the same as if I set > merge.renameLimit=1 for the repository, and the end result

Re: cherry-pick very slow on big repository

2017-11-21 Thread Peter Krefting
Elijah Newren: Sure, take a look at the big-repo-small-cherry-pick branch of https://github.com/newren/git With those changes, the time usage is the same as if I set merge.renameLimit=1 for the repository, and the end result is identical: $ time /usr/local/stow/git-v2.15.0-323-g31fe956618/b

Re: cherry-pick very slow on big repository

2017-11-13 Thread Elijah Newren
On Mon, Nov 13, 2017 at 3:22 AM, Peter Krefting wrote: > Elijah Newren: > >> I would be very interested to hear how my rename detection performance >> patches work for you; this kind of usecase was the exact one it was designed >> to help the most. See >> https://public-inbox.org/git/201711102221

Re: cherry-pick very slow on big repository

2017-11-13 Thread Peter Krefting
Elijah Newren: I would be very interested to hear how my rename detection performance patches work for you; this kind of usecase was the exact one it was designed to help the most. See https://public-inbox.org/git/20171110222156.23221-1-new...@gmail.com/ I'd be happy to try them out. Is the

RE: cherry-pick very slow on big repository

2017-11-13 Thread Peter Krefting
Kevin Willford: Since this is happening during a merge, you might need to use merge.renameLimit or the merge strategy option of -Xno-renames. Although the code does fallback to use the diff.renameLimit but there is still a lot that is done before even checking the rename limit so I would first

Re: cherry-pick very slow on big repository

2017-11-10 Thread Elijah Newren
On Fri, Nov 10, 2017 at 6:05 AM, Peter Krefting wrote: > Derrick Stolee: > >> Git is spending time detecting renames, which implies you probably renamed >> a folder or added and deleted a large number of files. This rename detection >> is quadratic (# adds times # deletes). > > Yes, a couple of di

Re: cherry-pick very slow on big repository

2017-11-10 Thread Elijah Newren
Interesting timing. I have some performance patches specifically developed because rename detection during merges made a small cherry-pick in a large repo rather slow...in my case, I dropped the time for the cherry pick by a factor of about 30 (no guarantees you'll see the same; it's very history-

RE: cherry-pick very slow on big repository

2017-11-10 Thread Kevin Willford
renames turned off. Thanks, Kevin > -Original Message- > From: git-ow...@vger.kernel.org [mailto:git-ow...@vger.kernel.org] On Behalf > Of Peter Krefting > Sent: Friday, November 10, 2017 7:05 AM > To: Derrick Stolee > Cc: Jeff King ; Git Mailing List > Subject: Re: che

Re: cherry-pick very slow on big repository

2017-11-10 Thread Peter Krefting
Derrick Stolee: Git is spending time detecting renames, which implies you probably renamed a folder or added and deleted a large number of files. This rename detection is quadratic (# adds times # deletes). Yes, a couple of directories with a lot of template files have been renamed (and some

Re: cherry-pick very slow on big repository

2017-11-10 Thread Derrick Stolee
On 11/10/2017 7:37 AM, Peter Krefting wrote: Jeff King: Can you get a backtrace? I'd do something like: Seems that it spends most time in diffcore_count_changes(), that is where it hits whenever I hit Ctrl+C (various line numbers 199-207 in diffcore-delta.c; this is on the v2.15.0 tag). (

Re: cherry-pick very slow on big repository

2017-11-10 Thread Peter Krefting
Jeff King: Can you get a backtrace? I'd do something like: Seems that it spends most time in diffcore_count_changes(), that is where it hits whenever I hit Ctrl+C (various line numbers 199-207 in diffcore-delta.c; this is on the v2.15.0 tag). (gdb) bt #0 diffcore_count_changes (src=src@en

Re: cherry-pick very slow on big repository

2017-11-10 Thread Jeff King
On Fri, Nov 10, 2017 at 10:39:39AM +0100, Peter Krefting wrote: > Running strace, it seems like it is doing lstat(), open(), mmap(), close() > and munmap() on every single file in the repository, which takes a lot of > time. > > I thought it was just updating the status, but "git status" returns

cherry-pick very slow on big repository

2017-11-10 Thread Peter Krefting
Hi! On a big repository (57000 files, 2,5 gigabytes in .git/objects), git cherry-pick is very slow for me (v2.15.0). This is cherry-picking a one-file change, where the file is in the same place on both branches, and which applies cleanly (I am backporting a few fixes to a maintenance version