[PATCH] Use path comparison for patch ids before the file content

When limiting the list in a revision walk using cherry pick, patch ids are calculated by producing the diff of the content of the files. This would be more efficent by using a patch id looking at the paths that were changed in the commits and only if all the file changed are the same fall back to

[[PATCH v2] 0/4] Use header data patch ids for rebase to avoid loading file content

This patch series is to remove the hand rolled hashmap in the patch_ids and use the hashmap.h implementation. It also introduces the idea of having a header only patch id so that the contents of the files do not have to be loaded in order to determine if two commits are different. Kevin

[[PATCH v2] 3/4] patch-ids: add flag to create the diff patch id using header only data

From: Kevin Willford <kewi...@microsoft.com> This will allow a diff patch id to be created using only the header data so that the contents of the file will not have to be loaded. Signed-off-by: Kevin Willford <kcwillf...@gmail.com> --- diff.c | 16 ++-- diff

[[PATCH v2] 2/4] patch-ids: replace the seen indicator with a commit pointer

From: Kevin Willford <kewi...@microsoft.com> The cherry_pick_list was looping through the original side checking the seen indicator and setting the cherry_flag on the commit. If we save off the commit in the patch_id we can set the cherry_flag on the correct commit when running through the

[[PATCH v2] 1/4] patch-ids: stop using a hand-rolled hashmap implementation

From: Kevin Willford <kewi...@microsoft.com> This change will use the hashmap from the hashmap.h to keep track of the patch_ids that have been encountered instead of using an internal implementation. This simplifies the implementation of the patch ids. Signed-off-by: Kevin Willford &l

[[PATCH v2] 4/4] rebase: avoid computing unnecessary patch IDs

From: Kevin Willford <kewi...@microsoft.com> The `rebase` family of Git commands avoid applying patches that were already integrated upstream. They do that by using the revision walking option that computes the patch IDs of the two sides of the rebase (local-only patches vs upstream-onl

RE: [PATCH 3/3] reset.c: update files when using sparse to avoid data loss.

> -Original Message- > From: Duy Nguyen [mailto:pclo...@gmail.com] > Sent: Monday, April 10, 2017 4:24 AM > To: Kevin Willford <kcwillf...@gmail.com> > Cc: git@vger.kernel.org; gits...@pobox.com; p...@peff.net; Kevin Willford > <kewi...@microsoft.com> >

RE: [PATCH 3/3] reset.c: update files when using sparse to avoid data loss.

> -Original Message- > From: git-ow...@vger.kernel.org [mailto:git-ow...@vger.kernel.org] On > Behalf Of Duy Nguyen > Sent: Wednesday, April 12, 2017 7:21 AM > To: Kevin Willford <kewi...@microsoft.com> > Cc: Kevin Willford <kcwillf...@gmail.com>; git@vger.k

[PATCH 3/3] reset.c: update files when using sparse to avoid data loss.

directory and if it will be missing with the new index entry or was not missing in the previous version, we create the previous index version of the file in the working directory so that status will report correctly and the files will be availble for the user to deal with. Signed-off-by: Kevin

[PATCH 0/3] fix working directory file issues while using sparse-checkout

While using the sparse-checkout feature there are scenarios where the working directory should and should not be updated. This patch series addresses issues found in reset, apply, and merge conflicts. Kevin Willford (3): merge-recursive.c: conflict using sparse should update file apply.c

[PATCH 1/3] merge-recursive.c: conflict using sparse should update file

Update the file when there is a conflict with a modify/delete scenario when using the sparse-checkout feature since the file might not be on disk because the skip-worktree bit is on and the user will need the file and content to determine how to resolve the conflict. Signed-off-by: Kevin Willford

[PATCH 2/3] apply.c: do not checkout file when skip-worktree bit set

ile in the working directory, then update the index but not keep the working directory up to date with the changes that happened in the index. Signed-off-by: Kevin Willford <kewi...@microsoft.com> --- apply.c | 18 ++ 1 file changed, 18 insertions(+) diff --git a/apply.c b/appl

[PATCH v2] commit: skip discarding the index if there is no pre-commit hook

If there is not a pre-commit hook, there is no reason to discard the index and reread it. This change checks to presence of a pre-commit hook and then only discards the index if there was one. Signed-off-by: Kevin Willford <kewi...@microsoft.com> --- builtin/commit.c | 15 +---

RE: [PATCH v2] commit: skip discarding the index if there is no pre-commit hook

> On Mon, Aug 14, 2017 at 03:54:25PM -0600, Kevin Willford wrote: > > > If there is not a pre-commit hook, there is no reason to discard > > the index and reread it. > > > > This change checks to presence of a pre-commit hook and then only > > discards the in

Re: [PATCH] cache-tree: remove use of strbuf_addf in update_one

On 8/10/2017 3:03 PM, Jeff King wrote: On Thu, Aug 10, 2017 at 11:58:34AM -0700, Stefan Beller wrote: On Thu, Aug 10, 2017 at 11:47 AM, Kevin Willford <kcwillf...@gmail.com> wrote: String formatting can be a performance issue when there are hundreds of thousands of trees. When ch

[PATCH] commit: skip discarding the index if there is no pre-commit hook

If there is not a pre-commit hook, there is no reason to discard the index and reread it. This change checks to presence of a pre-commit hook and then only discards the index if there was one. Signed-off-by: Kevin Willford <kewi...@microsoft.com> --- builtin/commit.

[PATCH v2 1/2] format-patch: have progress option while generating patches

meter. This patch allows a progress option to be passed to format-patch so that the user can be informed the progress of generating the patch. This option is then used by the rebase command when calling format-patch. Signed-off-by: Kevin Willford <kewi...@microsoft.com> --- Documentati

[PATCH v2 2/2] rebase: turn on progress option by default for format-patch

This change passes the progress option of format-patch checking that stderr is attached and rebase is not being run in quiet mode. Signed-off-by: Kevin Willford <kewi...@microsoft.com> --- git-rebase--am.sh | 1 + git-rebase.sh | 6 ++ 2 files changed, 7 insertions(+) diff --git

[PATCH v2 0/2] Add progress for format-patch and rebase

not set before propagating the progress flag to format-patch Kevin Willford (2): format-patch: have progress option while generating patches rebase: turn on progress option by default for format-patch Documentation/git-format-patch.txt | 4 builtin/log.c

[PATCH] cache-tree: remove use of strbuf_addf in update_one

through that are not a known one for git. Signed-off-by: Kevin Willford <kewi...@microsoft.com> --- cache-tree.c | 24 +++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/cache-tree.c b/cache-tree.c index 2440d1dc89..41744b3db7 100644 --- a/cache-tree.c +++ b

[PATCH 2/2] rebase: turn on progress option by default for format-patch

This change passes the progress option of format-patch by default and passes the -q --quiet option through to the format-patch call so that it is respected as well. Signed-off-by: Kevin Willford <kewi...@microsoft.com> --- git-rebase--am.sh | 5 +++-- git-rebase.sh | 2 ++ 2 files chan

[PATCH 0/2] Add progress to format-patch and rebase

and makes it the default when running rebase. It will also respect the -q[uiet] flag and not show when it is present. Kevin Willford (2): format-patch: have progress option while generating patches rebase: turn on progress option by default for format-patch Documentation/git-format-patch.txt | 8

[PATCH 1/2] format-patch: have progress option while generating patches

be informed the progress of generating the patch. This option will then be used by the rebase command when calling format-patch. Signed-off-by: Kevin Willford <kewi...@microsoft.com> --- Documentation/git-format-patch.txt | 8 builtin/log.c | 10 ++ 2

RE: [PATCH 2/2] rebase: turn on progress option by default for format-patch

> -Original Message- > From: Stefan Beller [mailto:sbel...@google.com] > Sent: Wednesday, May 31, 2017 1:09 PM > To: Kevin Willford <kcwillf...@gmail.com> > Cc: git@vger.kernel.org; Junio C Hamano <gits...@pobox.com>; Kevin > Willford <kewi...@microso

RE: [PATCH 1/2] format-patch: have progress option while generating patches

> -Original Message- > From: git-ow...@vger.kernel.org [mailto:git-ow...@vger.kernel.org] On > Behalf Of Stefan Beller > Sent: Wednesday, May 31, 2017 12:40 PM > To: Kevin Willford <kcwillf...@gmail.com> > Cc: git@vger.kernel.org; Junio C Hamano <gits...@pobox.c

RE: [PATCH 1/1] reset: fix reset when using the sparse-checkout feature.

> From: Jacob Keller [mailto:jacob.kel...@gmail.com] > Sent: Tuesday, September 12, 2017 7:39 PM > > On Tue, Sep 12, 2017 at 4:30 PM, Kevin Willford <kewi...@microsoft.com> wrote: > > > > Sorry. It was not in the sparse-checkout file. > > $ git config co

RE: [PATCH 1/1] reset: fix reset when using the sparse-checkout feature.

> From: Junio C Hamano [mailto:gits...@pobox.com] > Sent: Wednesday, September 13, 2017 4:18 PM > > Jacob Keller writes: > > > By definition if you do a sparse checkout, you're telling git "I only > > care about the files in this sparse checkout, and do not concern me >

RE: [PATCH 1/1] reset: fix reset when using the sparse-checkout feature.

From: Junio C Hamano [mailto:gits...@pobox.com] Sent: Thursday, September 14, 2017 11:00 PM > > Kevin Willford <kewi...@microsoft.com> writes: > > > 1. Does this statement, "I only care about the files in this > > sparse checkout, and do not concern me with a

[PATCH 0/1] reset: fix mixed reset when using sparse-checkout

the skip-wortree bit so that the file contents before the reset is preserved on disk and status will reports the correct results. Kevin Willford (1): reset: fix reset when using the sparse-checkout feature. builtin/reset.c | 25 + t/t7114-reset-sparse-checkout.sh

[PATCH 1/1] reset: fix reset when using the sparse-checkout feature.

directory and if it will be missing with the new index entry or was not missing in the previous version, we create the previous index version of the file in the working directory so that status will report correctly and the files will be availble for the user to deal with. Signed-off-by: Kevin

RE: [PATCH 1/1] reset: fix reset when using the sparse-checkout feature.

From: Junio C Hamano [mailto:gits...@pobox.com] Sent: Friday, September 8, 2017 1:12 PM > Kevin Willford <kewi...@microsoft.com> writes: > > > When using the sparse checkout feature the git reset command will add > > entries to the index that will have t

RE: [PATCH 1/1] reset: fix reset when using the sparse-checkout feature.

From: Junio C Hamano [mailto:gits...@pobox.com] Sent: Friday, September 8, 2017 1:02 PM > Kevin Willford <kewi...@microsoft.com> writes: > > > diff --git a/builtin/reset.c b/builtin/reset.c > > index d72c7d1c96..1b8bb45989 100644 > > --- a/builtin/reset.c > >

RE: What's cooking in git.git (Sep 2017, #02; Mon, 11)

> > * kw/write-index-reduce-alloc (2017-09-08) 2 commits > - read-cache: fix index corruption with index v4 > - Add t/helper/test-write-cache to .gitignore > > Expecting a reroll. > cf. q...@mail.gmail.com> > I didn't submit these patches

RE: [PATCH] read-cache: fix index corruption with index v4

> From: Thomas Gummerer [mailto:t.gumme...@gmail.com] > Sent: Monday, September 4, 2017 4:58 PM > > ce012deb98 ("read-cache: avoid allocating every ondisk entry when > writing", 2017-08-21) changed the way cache entries are written to the > index file. While previously it wrote the name to an

[PATCH v3 3/3] merge-recursive: change current file dir string_lists to hashmap

will save some time when there are millions of paths that will be checked. Signed-off-by: Kevin Willford <kewi...@microsoft.com> --- merge-recursive.c | 56 --- merge-recursive.h | 3 +-- 2 files changed, 46 insertions(+), 13 deletions(-) diff

[PATCH v3 1/3] merge-recursive: fix memory leak

In merge_trees if process_renames or process_entry returns less than zero, the method will just return and not free re_merge, re_head, or entries. This change cleans up the allocated variables before returning to the caller. Signed-off-by: Kevin Willford <kewi...@microsoft.com> ---

[PATCH v3 2/3] merge-recursive: remove return value from get_files_dirs

if read_tree_recursive gets an error. Since the debug output has been removed and the caller isn't checking the return value there is no reason to keep calculating and returning a value. Signed-off-by: Kevin Willford <kewi...@microsoft.com> --- merge-recursive.c | 8 ++-- 1 file chan

[PATCH v3 0/3] merge-recursive: replace string_list with hashmap

the hashmap and why it is getting initialized and freed but not a local. 3. Use hashmap_get_from_hash and remove the dummy entry Kevin Willford (3): merge-recursive: fix memory leak merge-recursive: remove return value from get_files_dirs merge-recursive: change current file dir string_lists

[PATCH v2] merge-recursive: change current file dir string_lists to hashmap

will save some time when there are millions of paths that will be checked. Also cleaned up a memory leak and method where the return was not being used. Signed-off-by: Kevin Willford <kewi...@microsoft.com> --- merge-recursive.c | 76 ---

RE: [PATCH 1/1] reset: fix reset when using the sparse-checkout feature.

> From: Jacob Keller [mailto:jacob.kel...@gmail.com] > Sent: Tuesday, September 12, 2017 4:29 PM > > On Tue, Sep 12, 2017 at 1:20 PM, Kevin Willford <kewi...@microsoft.com> wrote: > > > > I think this is where I need to do a better job of explaining so here is a >

RE: [PATCH 1/1] reset: fix reset when using the sparse-checkout feature.

> From: Junio C Hamano [mailto:gits...@pobox.com] > Sent: Friday, September 8, 2017 9:18 PM > > Kevin Willford <kewi...@microsoft.com> writes: > > > 1. reset mixed when there were files that were added > > > > In this case the index will no longer have

RE: [PATCH 1/1] reset: fix reset when using the sparse-checkout feature.

> From: Junio C Hamano [mailto:gits...@pobox.com] > Sent: Monday, September 11, 2017 9:57 PM > > Let's imagine a path P that is outside the sparse checkout area. > And we check out a commit that has P. P would still be recorded in > the index but it would not materialize in the working tree.

[PATCH 1/3] perf: add test for writing the index

A performance test for writing the index to be able to determine if changes to allocating ondisk structure help. Signed-off-by: Kevin Willford <kewi...@microsoft.com> --- Makefile| 1 + t/helper/test-write-cache.c | 23 +++ t/perf/p0007-write-ca

[PATCH 2/3] read-cache: fix memory leak in do_write_index

The previous_name_buf was never getting released when there was an error in ce_write_entry or allow was false and execution was returned to the caller. Signed-off-by: Kevin Willford <kewi...@microsoft.com> --- read-cache.c | 12 +--- 1 file changed, 9 insertions(+), 3 deletions(-)

[PATCH 3/3] read-cache: avoid allocating every ondisk entry when writing

to fill the buffer before flushing to disk, we don't have to worry about doing multiple ce_write calls. Running the p0007-write-cache.sh tests would save anywhere between 3-7% when the index had over a million entries with no performance degradation on small repos. Signed-off-by: Kevin Willford <k

[PATCH 0/3] read-cache: use stack ondisk struct when writing index

) 5.86(0.01+0.03) -5.5% 0007.2: write_locked_index 3 times (4394531 files) 10.29(0.04+0.03) 9.75(0.04+0.01) -5.2% 0007.2: write_locked_index 3 times (4394531 files) 10.52(0.00+0.04) 9.79(0.03+0.03) -6.9% Kevin Willford (3): perf: add test for writing the index read-cache: fix memory

[PATCH 3/3] merge-recursive: change current file dir string_lists to hashmap

will save some time when there are millions of paths that will be checked. Signed-off-by: Kevin Willford <kewi...@microsoft.com> --- merge-recursive.c | 48 +--- merge-recursive.h | 3 +-- 2 files changed, 38 insertions(+), 13 deletions(-) diff --git a

[PATCH 2/3] merge-recursive: remove return value from get_files_dirs

if read_tree_recursive gets an error. Since the debug output has been removed and the caller isn't checking the return value there is no reason to keep calulating and returning a value. Signed-off-by: Kevin Willford <kewi...@microsoft.com> --- merge-recursive.c | 8 ++-- 1 file chan

[PATCH 0/3] merge-recursive: replace string_list with hashmap

some time when there are millions of paths that will be checked. Also cleaned up a memory leak and method where the return was not being used. Kevin Willford (3): merge-recursive: fix memory leak merge-recursive: remove return value from get_files_dirs merge-recursive: change current file dir

[PATCH 1/3] merge-recursive: fix memory leak

In merge_trees if process_renames or process_entry returns less than zero, the method will just return and not free re_merge, re_head, or entries. This change cleans up the allocated variables before returning to the caller. Signed-off-by: Kevin Willford <kewi...@microsoft.com> ---

RE: [PATCH 2/3] merge-recursive: remove return value from get_files_dirs

> > On Mon, Aug 28, 2017 at 02:28:28PM -0600, Kevin Willford wrote: > > > The return value of the get_files_dirs call is never being used. > > Looking at the history of the file and it was originally only > > being used for debug output statements. Also when > >

RE: cherry-pick very slow on big repository

Since this is happening during a merge, you might need to use merge.renameLimit or the merge strategy option of -Xno-renames. Although the code does fallback to use the diff.renameLimit but there is still a lot that is done before even checking the rename limit so I would first try getting