Re: [PATCH v2 9/9] pack-objects: improve partial packfile reuse

2019-10-22 Thread Jonathan Tan
First, the commit message could probably be reordered to start with the main point (reusing of packfiles at object granularity instead of allowing reuse of only one contiguous region). To specific points: > The dynamic array of `struct reused_chunk` is useful > because we need to know not just the

Re: [PATCH v2 3/9] ewah/bitmap: introduce bitmap_word_alloc()

2019-10-22 Thread Jonathan Tan
> From: Jeff King > > In a following commit we will need to allocate a variable > number of bitmap words, instead of always 32, so let's add > bitmap_word_alloc() for this purpose. > > We will also always access at least one word for each bitmap, > so we want to make sure that at least one is al

Re: [PATCH 1/1] config: add documentation to config.h

2019-10-18 Thread Jonathan Tan
> From: Heba Waly > > This commit is copying and summarizing the documentation from > documentation/technical/api-config.txt to comments in config.h Thanks for this commit! As for your commit message, as far as I know, the idea is to move the documentation, not to copy it. Also, write this in i

[PATCH v2 7/7] index-pack: make quantum of work smaller

2019-10-17 Thread Jonathan Tan
that this patch is a net improvement. Signed-off-by: Jonathan Tan --- builtin/index-pack.c | 336 --- 1 file changed, 190 insertions(+), 146 deletions(-) diff --git a/builtin/index-pack.c b/builtin/index-pack.c index 31607a77fc..072592a35d 100644 --- a/bu

[PATCH v2 5/7] index-pack: calculate {ref,ofs}_{first,last} early

2019-10-17 Thread Jonathan Tan
possible moment. This allowed the members of struct base_data to be populated in any order, superficially useful when we have the object contents before the struct object_entry. But this makes reasoning about the state of struct base_data more complicated, hence this patch. Signed-off-by: Jonathan Tan

[PATCH v2 6/7] index-pack: make resolve_delta() assume base data

2019-10-17 Thread Jonathan Tan
A subsequent commit will make the quantum of work smaller, necessitating more locking. This commit allows resolve_delta() to be called outside the lock. Signed-off-by: Jonathan Tan --- builtin/index-pack.c | 8 +--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/builtin/index

[PATCH v2 3/7] index-pack: remove redundant parameter

2019-10-17 Thread Jonathan Tan
find_{ref,ofs}_delta_{,children} take an enum object_type parameter, but the object type is already present in the name of the function. Remove that parameter from these functions. Signed-off-by: Jonathan Tan --- builtin/index-pack.c | 26 -- 1 file changed, 12

[PATCH v2 4/7] index-pack: remove redundant child field

2019-10-17 Thread Jonathan Tan
; pointer is redundant even now, remove it so that the aforementioned subsequent patch will be clearer. In the meantime, reclaim memory in the reverse order of the "base" pointers. Signed-off-by: Jonathan Tan --- builtin/index-pack.c | 41 ++--- 1 f

[PATCH v2 1/7] Documentation: deltaBaseCacheLimit is per-thread

2019-10-17 Thread Jonathan Tan
Clarify that core.deltaBaseCacheLimit is per-thread, as can be seen from the fact that cache usage (base_cache_used in struct thread_local in builtin/index-pack.c) is tracked individually for each thread and compared against delta_base_cache_limit. Signed-off-by: Jonathan Tan --- Documentation

[PATCH v2 2/7] index-pack: unify threaded and unthreaded code

2019-10-17 Thread Jonathan Tan
Signed-off-by: Jonathan Tan --- builtin/index-pack.c | 10 +- 1 file changed, 1 insertion(+), 9 deletions(-) diff --git a/builtin/index-pack.c b/builtin/index-pack.c index 60a5591039..df6b3b8cf6 100644 --- a/builtin/index-pack.c +++ b/builtin/index-pack.c @@ -1210,15 +1210,7 @@ static

[PATCH v2 0/7] Better threaded delta resolution in index-pack

2019-10-17 Thread Jonathan Tan
ve > puzzled all that out, it would be nice to make the argument in the > commit message. I've added an explanation in the commit message. Jonathan Tan (7): Documentation: deltaBaseCacheLimit is per-thread index-pack: unify threaded and unthreaded code index-pack: remove redundant parameter

Re: [PATCH 1/1] builtin/blame.c: constants into bit shift format

2019-10-16 Thread Jonathan Tan
> There was some discussion recently about converting these related > #defines to enums [0]. We might consider doing that here. > > If you read through that entire thread, you'd see that there were some > disagreements about whether using enums for sets of bits is a good idea > ([1] and [2]), b

[PATCH v2] fetch-pack: write fetched refs to .promisor

2019-10-14 Thread Jonathan Tan
TTP protocol v2 (fetch_refs_via_pack() in transport.c sets lock_pack) and with HTTP protocol v0/v1 (fetch_git() in remote-curl.c passes "--lock-pack" to "fetch-pack"). Signed-off-by: Jonathan Tan --- Changes from v1: - commit message explains scope of change ("it knows the name of t

Re: [PATCH] fetch-pack: write fetched refs to .promisor

2019-10-14 Thread Jonathan Tan
Thanks for your comments. Rearranging them: > This makes me wonder why we don't also change index-pack to write a > similar message to the .promisor. I guess there's potentially too much > information to shove all the refs on the command-line? index-pack already is capable of writing messages to

Re: [Outreachy] [PATCH] blame: Convert pickaxe_blame defined constants to enums

2019-10-14 Thread Jonathan Tan
> > Well, I agree that it could be better, but with the C language as we > > have it now, I still slightly prefer an enum to a list of #define. Both > > ways, we still have to manually enter values for each flag, but with > > enum, we get better debugger display (at least in gdb) and in the > > fun

Re: [Outreachy] [PATCH] blame: Convert pickaxe_blame defined constants to enums

2019-10-14 Thread Jonathan Tan
> Jonathan Tan writes: > > >> > Also, I have a slight preference for putting "= 02" on the BLAME_COPY > >> > line but that is not necessary. > >> > >> That is absolutely necessary; it is not like "we do not care what > >

Re: [PATCH v2] send-pack: never fetch when checking exclusions

2019-10-11 Thread Jonathan Tan
> As a general rule (and why I'm raising this issue in reply to Jonathan's > patch), I think most or all sites that want OBJECT_INFO_QUICK will want > SKIP_FETCH_OBJECT as well, and vice versa. The reasoning is generally > the same: > > - it's OK to racily have a false negative (we'll still be c

Re: [RFC PATCH 10/10] pack-objects: improve partial packfile reuse

2019-10-11 Thread Jonathan Tan
> > This makes sense - offsets may be different when we omit objects from > > the packfile. I think this can be computed by calculating the number of > > zero bits between the current object's index and the nth object prior > > (where n is the offset) in the bitmap resulting from > > reuse_partial_

Re: [Outreachy] [PATCH] blame: Convert pickaxe_blame defined constants to enums

2019-10-11 Thread Jonathan Tan
> > Any reason why the names are renamed to omit "PICKAXE_"? In particular, > > these names are still global, so it is good to retain the extra context. > > > > (This doesn't mean that you are wrong to remove them - I just gave my > > opinion, and a reason for my opinion. If you had a reason to re

Re: [Outreachy] [PATCH] blame: Convert pickaxe_blame defined constants to enums

2019-10-11 Thread Jonathan Tan
> Jonathan Tan writes: > > >> - if ((opt & PICKAXE_BLAME_COPY_HARDEST) > >> - || ((opt & PICKAXE_BLAME_COPY_HARDER) > >> + if ((opt & BLAME_COPY_HARDEST) > >> + || ((opt & BLAME_COPY_HARDER) > > > > Any reason

Re: [RFC PATCH 10/10] pack-objects: improve partial packfile reuse

2019-10-10 Thread Jonathan Tan
I'm going to start with pack-bitmap.h, then builtin/pack-objects.c. > int reuse_partial_packfile_from_bitmap(struct bitmap_index *, > struct packed_git **packfile, > -uint32_t *entries, off_t *up_to); > +

Re: [RFC PATCH 05/10] pack-bitmap: don't rely on bitmap_git->reuse_objects

2019-10-10 Thread Jonathan Tan
> As we now allocate 2 more words than necessary for each > bitmap to serve as marks telling us that we can stop > iterating over the words, we don't need to rely on > bitmap_git->reuse_objects to stop iterating over the words. As Peff states [1], this justification is probably incorrect as well.

Re: [RFC PATCH 04/10] ewah/bitmap: always allocate 2 more words

2019-10-10 Thread Jonathan Tan
> From: Jeff King > > In a following patch we will allocate a variable number > of words in some bitmaps. When iterating over the words we > will need a mark to tell us when to stop iterating. Let's > always allocate 2 more words, that will always contain 0, > as that mark. [snip] > if (b

Re: [PATCH 3/6] index-pack: remove redundant child field

2019-10-10 Thread Jonathan Tan
> On 10/9/2019 7:44 PM, Jonathan Tan wrote: > > Instead, recompute ancestry if we ever need to reclaim memory. > > I find this message lacking in important details: > > 1. Where do we recompute ancestry? > 2. What are the performance implications of this change? > 3. W

Re: [Outreachy] Outreachy initial contact

2019-10-10 Thread Jonathan Tan
> Hello Jonathan! > My name is Wambui and I'm one of the applicants in the current round of > Outreachy applications. I wanted to write an initial email to introduce > myself since I'm interested in working on your mentored projects! > > I have been going through the mailing list threads about Out

Re: [Outreachy] [PATCH] blame: Convert pickaxe_blame defined constants to enums

2019-10-10 Thread Jonathan Tan
> Convert pickaxe_blame preprocessor constants in blame.h to an enum. > Also replace previous instances of the constants with the new enum values. First of all, thanks for your initiative in finding a microproject and making a patch for it! About your commit message title, I know that 50 characte

[PATCH 4/6] index-pack: calculate {ref,ofs}_{first,last} early

2019-10-09 Thread Jonathan Tan
Whenever we make a struct base_data, immediately calculate its delta children. This eliminates confusion as to when the {ref,ofs}_{first,last} fields are initialized. Signed-off-by: Jonathan Tan --- builtin/index-pack.c | 125 +-- 1 file changed, 61

[PATCH 2/6] index-pack: remove redundant parameter

2019-10-09 Thread Jonathan Tan
Signed-off-by: Jonathan Tan --- builtin/index-pack.c | 26 -- 1 file changed, 12 insertions(+), 14 deletions(-) diff --git a/builtin/index-pack.c b/builtin/index-pack.c index 1a2600d4b3..99f6e2957f 100644 --- a/builtin/index-pack.c +++ b/builtin/index-pack.c @@ -614,7

[PATCH 3/6] index-pack: remove redundant child field

2019-10-09 Thread Jonathan Tan
Instead, recompute ancestry if we ever need to reclaim memory. Signed-off-by: Jonathan Tan --- builtin/index-pack.c | 41 ++--- 1 file changed, 22 insertions(+), 19 deletions(-) diff --git a/builtin/index-pack.c b/builtin/index-pack.c index 99f6e2957f

[PATCH 5/6] index-pack: make resolve_delta() assume base data

2019-10-09 Thread Jonathan Tan
A subsequent commit will make the quantum of work smaller, necessitating more locking. This commit allows resolve_delta() to be called outside the lock. Signed-off-by: Jonathan Tan --- builtin/index-pack.c | 8 +--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/builtin/index

[PATCH 1/6] index-pack: unify threaded and unthreaded code

2019-10-09 Thread Jonathan Tan
Signed-off-by: Jonathan Tan --- builtin/index-pack.c | 10 +- 1 file changed, 1 insertion(+), 9 deletions(-) diff --git a/builtin/index-pack.c b/builtin/index-pack.c index a23454da6e..1a2600d4b3 100644 --- a/builtin/index-pack.c +++ b/builtin/index-pack.c @@ -1210,15 +1210,7 @@ static

[RFC PATCH 0/6] Better threaded delta resolution in index-pack

2019-10-09 Thread Jonathan Tan
ens during repack. [1] https://public-inbox.org/git/20190926003300.195781-1-jonathanta...@google.com/ [2] https://public-inbox.org/git/20190704100530.smn4rpiekwtfy...@glandium.org/ Jonathan Tan (6): index-pack: unify threaded and unthreaded code index-pack: remove redundant parameter index-

[PATCH 6/6] index-pack: make quantum of work smaller

2019-10-09 Thread Jonathan Tan
Signed-off-by: Jonathan Tan --- builtin/index-pack.c | 267 --- 1 file changed, 127 insertions(+), 140 deletions(-) diff --git a/builtin/index-pack.c b/builtin/index-pack.c index 3908cd3115..f6318037ca 100644 --- a/builtin/index-pack.c +++ b/builtin/index

[PATCH v2] send-pack: never fetch when checking exclusions

2019-10-08 Thread Jonathan Tan
o not help at all in the typical case where the client is up-to-date with the upstream of the branch being pushed. Ensure that these lazy fetches do not occur. Signed-off-by: Jonathan Tan --- > For example, would this change mean that the resulting pack may > include stuff that are re

Re: [PATCH v2 0/2] add trace2 regions to fetch & push

2019-10-07 Thread Jonathan Tan
> Changes since V1: > * Use the repository struct argument in transport_push(), rather than > the global the_repository. Thanks, the patches now look good to me. I verified that the repository argument to the trace functions just cause a different repo ID to be printed, which is what we want (e.

Re: [PATCH 0/2] add trace2 regions to fetch & push

2019-10-07 Thread Jonathan Tan
> We'd like to collect better statistics about where the time is spent in > fetches and pushes so that we can hopefully identify some areas for > future optimization. So let's add some trace2 regions around some of the > fetch/push phases so we can break down their timing. Thanks. Patch 1 looks g

[PATCH] send-pack: never fetch when checking exclusions

2019-10-07 Thread Jonathan Tan
ch. Ensure that this lazy fetch does not occur. Signed-off-by: Jonathan Tan --- send-pack.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/send-pack.c b/send-pack.c index 6dc16c3211..34c77cbb1a 100644 --- a/send-pack.c +++ b/send-pack.c @@ -40,7 +40,8 @@ int option_parse_p

[PATCH] fetch: delay fetch_if_missing=0 until after config

2019-10-07 Thread Jonathan Tan
When running "git fetch" in a partial clone with no blobs, for example, by: git clone --filter=blob:none --no-checkout \ https://kernel.googlesource.com/pub/scm/git/git git -C git fetch "git fetch" will fail to load the config blob object, printing "unable to load config blob object". Th

Re: Common thread pool API in Git?

2019-09-25 Thread Jonathan Tan
> For those who want to know, this question was motivated by a big delta > tree occurring in [1]. Forgot to mention: this is not an issue of the server not repacking with a chain length limit. The tree in question is indeed not tall, but it is very wide.

Common thread pool API in Git?

2019-09-25 Thread Jonathan Tan
Does anyone have ideas or plans for this? I know that (at least) "grep" and "index-pack" have their own implementations, and it would be great to just have one that all code can use. For those who want to know, this question was motivated by a big delta tree occurring in [1]. index-pack does paral

Re: [PATCH] add a Code of Conduct document

2019-09-24 Thread Jonathan Tan
s stated better what I would have said, so to this patch: Acked-by: Jonathan Tan

Re: Git in Outreachy December 2019?

2019-09-24 Thread Jonathan Tan
> On Mon, Sep 23, 2019 at 01:38:54PM -0700, Jonathan Tan wrote: > > > I didn't have any concrete ideas so I didn't include those, but some > > unrefined ideas: > > One risk to a mentoring project like this is that the intern does a good > job of steps 1-5, a

Re: Git in Outreachy December 2019?

2019-09-23 Thread Jonathan Tan
> I approved this. I did leave some comments elsewhere in the thread, but > I think we can continue to iterate on the idea. Thanks. > > There was a "How can applicants make a contribution to your project?" > > question and a few questions about communication channels. I answered > > them as best

Re: Git in Outreachy December 2019?

2019-09-23 Thread Jonathan Tan
> I think this is an OK level of detail. I'm not sure quite sure about the > goal of the project, though. In particular: > > - I'm not clear what we'd hope to gain. I.e., what richer information > would we want to pass back and forth between index-pack and the > other processes? It might

Re: Git in Outreachy December 2019?

2019-09-23 Thread Jonathan Tan
> No need for user-facing benefits. Refactoring or improving the code in > other useful ways are very good subjects (as I already said in my > reply to Emily and Dscho). Thanks! > I think this is really great, both the idea and the description! No > need for more details. Thanks! I've just submi

Re: [PATCH v2] merge-recursive: symlink's descendants not in way

2019-09-20 Thread Jonathan Tan
> Jonathan Tan writes: > > >> OK. We notice that we need to newly create foo/bar but we > >> incorrectly find that there is "foo/bar" already because of the > >> careless use of bare lstat(2) makes "bar" visible as if it were also > >&

Re: [PATCH v2] merge-recursive: symlink's descendants not in way

2019-09-20 Thread Jonathan Tan
> OK. We notice that we need to newly create foo/bar but we > incorrectly find that there is "foo/bar" already because of the > careless use of bare lstat(2) makes "bar" visible as if it were also > "foo/bar". I wonder if the current code would be confused the same > way if the side branch added

Re: Git in Outreachy December 2019?

2019-09-20 Thread Jonathan Tan
> Prospective mentors need to sign up on that site, and should propose a > project they'd be willing to mentor. [snip] > I'm happy to discuss possible projects if anybody has an idea but isn't > sure how to develop it into a proposal. I'm new to Outreachy and programs like this, so does anyone h

Re: What's cooking in git.git (Sep 2019, #02; Wed, 18)

2019-09-18 Thread Jonathan Tan
> * jt/cache-tree-avoid-lazy-fetch-during-merge (2019-09-09) 1 commit > - cache-tree: do not lazy-fetch tentative tree > > The cache-tree code has been taught to be less aggressive in > attempting to see if a tree object it computed already exists in > the repository. > > Waiting for a respo

[PATCH v2] merge-recursive: symlink's descendants not in way

2019-09-18 Thread Jonathan Tan
Helped-by: Elijah Newren Signed-off-by: Jonathan Tan --- Changes from v1: - Used has_symlink_leading_path(). This drastically shortens the diff. - Updated commit message following suggestions from Junio, Szeder Gábor, and Elijah Newren. - Updated test to add prereq and verification that the workin

Re: [RFC PATCH] merge-recursive: symlink's descendants not in way

2019-09-17 Thread Jonathan Tan
> Yeah, I recall having to add has_symlink_leading_path() long time > ago in different codepaths (including "apply"). It is not surprising > to see a similar glitch remaining in merge-recursive (it's a tricky > issue, and it's a tricky code). Thanks for the pointer to has_symlink_leading_path() -

Re: [RFC PATCH] merge-recursive: symlink's descendants not in way

2019-09-17 Thread Jonathan Tan
> Jonathan Tan writes: > > > When the working tree has: > > - foo (symlink) > > - foo/bar (directory) > > Whoa, wait. I assume, since this is about merge, the assumption is > that the working tree is clean with respect to the index, so 'foo' >

[RFC PATCH] merge-recursive: symlink's descendants not in way

2019-09-17 Thread Jonathan Tan
one", and will handle it as "Add/delete" or "Modify/delete" appropriately (including reinstatement of the previously unlinked symlink with a new unique filename if necessary, again, just like if "foo" were a regular file instead). Helped-by: Elijah Newren S

merge-recursive thinks symlink's child dirs are "real"

2019-09-16 Thread Jonathan Tan
This was raised by a coworker at $DAYJOB. I run the following script: $GIT init test && cd test ln -s . foo mkdir bar && touch bar/file $GIT add foo bar/file $GIT commit -m "foo symlink" $GIT checkout -b branch1 $GIT commit --allow-empty -m "empty commit" $GIT checkout master

Re: Git in Outreachy December 2019?

2019-09-13 Thread Jonathan Tan
> Do we have interested mentors for the next round of Outreachy? > > The deadline for Git to apply to the program is September 5th. The > deadline for mentors to have submitted project descriptions is September > 24th. Intern applications would start on October 1st. > > If there are mentors who w

Re: [PATCH v2] cache-tree: do not lazy-fetch merge tree

2019-09-10 Thread Jonathan Tan
> Sidenote, just curious: did you originally intend to add this test > before the test script sources 'lib-httpd.sh', or you were about to > append it at the end as usual, but then noticed the warning comment > telling you not to do so? Honestly, I don't remember. I do try to put tests near simila

Re: [PATCH v2] cache-tree: do not lazy-fetch merge tree

2019-09-10 Thread Jonathan Tan
> Junio C Hamano writes: > > > Isn't that what is going on? I thought I dug up the original that > > introduced the has_object_file() call to this codepath to make sure > > we understand why we make the check (and I expected the person who > > is proposing this change to do the same and record t

[PATCH v2] cache-tree: do not lazy-fetch merge tree

2019-09-09 Thread Jonathan Tan
is both unnecessary (because we have the tree in the struct strbuf) and likely to fail (because the remote probably doesn't have this tree). Do not lazy fetch in this situation. Signed-off-by: Jonathan Tan --- As requested in What's Cooking [1], here's a patch with an updated

Re: [PATCH] fetch-pack: write fetched refs to .promisor

2019-09-05 Thread Jonathan Tan
> I'm not really opposed to what you're doing here, but I did recently > think of another possible use for .promisor files. So it seems like a > good time to bring it up, since presumably we'd have to choose one or > the other. Thanks for bringing it up - yes, we should discuss this. > I noticed

Re: [PATCH] cache-tree: do not lazy-fetch merge tree

2019-09-04 Thread Jonathan Tan
> On 9/3/2019 3:42 PM, Jonathan Tan wrote: > > When cherry-picking (for example), new trees may be constructed. During > > this process, Git checks whether these trees exist. However, in a > > partial clone, this causes a lazy fetch to occur, which is both > > unnecessa

[PATCH] cache-tree: do not lazy-fetch merge tree

2019-09-03 Thread Jonathan Tan
) and likely to fail (because the remote probably doesn't have this tree). Do not lazy fetch in this situation. Signed-off-by: Jonathan Tan --- Another partial clone bug. This raises the issue that failed fetches are currently fatal - if they weren't fatal, this cherry-pick would h

Re: [PATCH] fetch-pack: write fetched refs to .promisor

2019-08-27 Thread Jonathan Tan
> Jonathan Tan writes: > > > As written in the NEEDSWORK comment, repack does not preserve the > > contents of .promisor files, but I thought I'd send this out anyway as > > this change is already useful for users who don't run repack much. > > What

[PATCH] fetch-pack: write fetched refs to .promisor

2019-08-26 Thread Jonathan Tan
hashes against what the promisor remote reports now. Signed-off-by: Jonathan Tan --- As written in the NEEDSWORK comment, repack does not preserve the contents of .promisor files, but I thought I'd send this out anyway as this change is already useful for users who don't run r

Re: [PATCH 0/2] Skip ls-refs if possible for HTTP

2019-08-22 Thread Jonathan Tan
> This probably is totally off-tangent, but do any of these "let's > advertise fewer" changes at the protocol level have to take into > account the use of --prune option on the client side? I don't think so. According to what I understand from the documentation, the prune option prunes based on th

Re: [PATCH] diff: skip GITLINK when lazy fetching missing objs

2019-08-22 Thread Jonathan Tan
> I wondered what would happen when it does not succeed. It looks like the > whole diff process just dies. > > The batch fetch is purely an optimization, because we'd eventually fetch > the individual objects on demand. If the batch one fails, should we > continue with the operation? That leaves a

[PATCH 2/2] transport: teach all vtables to allow fetch first

2019-08-21 Thread Jonathan Tan
The only transport that does not allow fetch() to be called before get_refs_list() is the bundle transport. Clean up the code by teaching the bundle transport the ability to do this, and removing support for transports that don't support this order of invocation. Signed-off-by: Jonatha

[PATCH 0/2] Skip ls-refs if possible for HTTP

2019-08-21 Thread Jonathan Tan
re of the value of the test in patch 2, but that test does fail if I don't update fetch_refs_from_bundle() to first call get_refs_from_bundle() if it hasn't already been called. Jonathan Tan (2): transport-helper: skip ls-refs if unnecessary transport: teach all vtables to allow fetch

[PATCH 1/2] transport-helper: skip ls-refs if unnecessary

2019-08-21 Thread Jonathan Tan
if needed). If not, the remote helper interface will invoke get_refs_list() if it hasn't been invoked yet, preserving existing behavior. Signed-off-by: Jonathan Tan --- t/t5702-protocol-v2.sh | 13 + transport-helper.c | 39 +-- 2 file

Re: [PATCH] diff: skip GITLINK when lazy fetching missing objs

2019-08-20 Thread Jonathan Tan
> Jonathan Tan writes: > > > In 7fbbcb21b1 ("diff: batch fetching of missing blobs", 2019-04-08), > > diff was taught to batch the fetching of missing objects when operating > > on a partial clone, but was not taught to refrain from fetching > > GITLINK

[PATCH] diff: skip GITLINK when lazy fetching missing objs

2019-08-20 Thread Jonathan Tan
In 7fbbcb21b1 ("diff: batch fetching of missing blobs", 2019-04-08), diff was taught to batch the fetching of missing objects when operating on a partial clone, but was not taught to refrain from fetching GITLINKs. Teach diff to check if an object is a GITLINK before including it in the set to be f

Re: [PATCH v3] documentation: add tutorial for revision walking

2019-07-24 Thread Jonathan Tan
Thanks - I think this is a useful guide to what can be a complicated topic. It looks good overall; I just have some minor comments below. > diff --git a/Documentation/Makefile b/Documentation/Makefile > index 76f2ecfc1b..91e5da67c4 100644 > --- a/Documentation/Makefile > +++ b/Documentation/Makefi

Re: [PATCH] submodule: plumb --filter to cloned submodules

2019-07-24 Thread Jonathan Tan
> When cloning a repo with a --filter and with --recurse-submodules > enabled, the partial clone filter only applies to the top-level repo. > This can lead to unexpected bandwidth and disk usage for projects which > include large submodules. > > Fix this by plumbing the --filter argument from git-

Re: [PATCH v5 00/10] Filter combination

2019-06-28 Thread Jonathan Tan
> This applies suggestions made by Jonathan Tan, as well as fixes a > Coccinelle-breaking error in strbuf usage, and makes an additional string > localizable. > > Thanks, > > Matthew DeVore (10): > list-objects-filter: encapsulate filter components > list-objec

Re: What's cooking in git.git (Jun 2019, #06; Wed, 26)

2019-06-26 Thread Jonathan Tan
Junio, what do you think of this small patch that just updates a test: https://public-inbox.org/git/20190605192624.129677-1-jonathanta...@google.com/ For what it's worth, Stolee and Peff have replied and both of them seem to be OK with it.

Re: [PATCH v4 10/10] list-objects-filter-options: make parser void

2019-06-21 Thread Jonathan Tan
> This function always returns 0, so make it return void instead. And...patches 7-10 look straightforward and good to me. In summary, I don't think any changes need to be made to all 10 patches other than textual ones (commit messages, documentation, and function names).

Re: [PATCH v4 06/10] list-objects-filter-options: make filter_spec a string_list

2019-06-21 Thread Jonathan Tan
Patch 5 and this patch look good to me. > @@ -1134,27 +1134,25 @@ int cmd_clone(int argc, const char **argv, const char > *prefix) > transport_set_option(transport, TRANS_OPT_FOLLOWTAGS, "1"); > > if (option_upload_pack) > transport_set_option(transport, TRANS_

Re: [PATCH v4 04/10] list-objects-filter: implement composite filters

2019-06-21 Thread Jonathan Tan
> Allow combining filters such that only objects accepted by all filters > are shown. The motivation for this is to allow getting directory > listings without also fetching blobs. This can be done by combining > blob:none with tree:. There are massive repositories that have > larger-than-expected t

Re: [PATCH v4 01/10] list-objects-filter: make API easier to use

2019-06-21 Thread Jonathan Tan
> Make the list-objects-filter.h API more opaque and easier to use. This > prepares for combined filter support, where filters will be created and > used in a new context. > > Helped-by: Jeff Hostetler > Helped-by: Junio C Hamano > Signed-off-by: Matthew DeVore So what happens is that filter_f

Re: [PATCH 12/17] delta-islands: convert island_marks khash to use oids

2019-06-20 Thread Jonathan Tan
> @@ -105,7 +105,7 @@ int in_same_island(const struct object_id *trg_oid, const > struct object_id *src_ >* If we don't have a bitmap for the target, we can delta it >* against anything -- it's not an important object >*/ > - trg_pos = kh_get_sha1(island_marks, trg_oid-

Re: [PATCH v4] fetch-pack: support negotiation tip whitelist

2019-06-18 Thread Jonathan Tan
> > @@ -230,7 +246,7 @@ static int find_common(struct fetch_negotiator > > *negotiator, > > if (args->stateless_rpc && multi_ack == 1) > > die(_("--stateless-rpc requires multi_ack_detailed")); > > > > - for_each_ref(rev_list_insert_ref_oid, negotiator); > > + mark_tips(negotia

Re: [PATCH 03/10] t1450: make hash size independent

2019-06-11 Thread Jonathan Tan
> @@ -84,7 +86,7 @@ test_expect_success 'branch pointing to non-commit' ' > test_expect_success 'HEAD link pointing at a funny object' ' > test_when_finished "mv .git/SAVED_HEAD .git/HEAD" && > mv .git/HEAD .git/SAVED_HEAD && > - echo >.git/

[PATCH 1/2] t5616: use correct flag to check object is missing

2019-06-11 Thread Jonathan Tan
If we want to check whether an object is missing, the correct flag to pass to rev-list is --ignore-missing; --exclude-promisor-objects will exclude any object that came from the promisor remote, whether it is present or missing. Use the correct flag. Signed-off-by: Jonathan Tan --- t/t5616

[PATCH 0/2] Improve test code coverage on jt/partial-clone-missing-ref-delta-base

2019-06-11 Thread Jonathan Tan
paring the output - the relevant line indeed shows up as uncovered on "next", and covered on this. Jonathan Tan (2): t5616: use correct flag to check object is missing t5616: cover case of client having delta base t/t5616-partial-clone.sh | 41

[PATCH 2/2] t5616: cover case of client having delta base

2019-06-11 Thread Jonathan Tan
slightly to cover this case. [1] 8a30a1efd1 ("index-pack: prefetch missing REF_DELTA bases", 2019-05-15). [2] https://public-inbox.org/git/396091fc-5572-19a5-4f18-61c258590...@gmail.com/ Signed-off-by: Jonathan Tan --- t/t5616-partial-clone.sh | 39

[PATCH] t5551: test usage of chunked encoding explicitly

2019-06-05 Thread Jonathan Tan
of passing with GIT_TEST_PROTOCOL_VERSION=2. Signed-off-by: Jonathan Tan --- With this change, all tests pass at master with GIT_TEST_PROTOCOL_VERSION=2. --- t/t5551-http-fetch-smart.sh | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/t/t5551-http-fetch-smart.sh b/t/t

[PATCH] fetch-pack: send server options after command

2019-05-22 Thread Jonathan Tan
ich follows the documentation in this regard. Signed-off-by: Jonathan Tan --- fetch-pack.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fetch-pack.c b/fetch-pack.c index 3f24d0c8a6..1c10f54e78 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -1115,7 +1115,7 @@ static int send_fetch

Re: [PATCH 2/2] index-pack: prefetch missing REF_DELTA bases

2019-05-16 Thread Jonathan Tan
> > > Right, REF_DELTA is definitely correctly handled currently, and I don't > > > think that would break with your patch. It's just that your patch would > > > introduce a bunch of extra traffic as we request bases separately that > > > are already in the pack. > > > > Ah...I see. For this probl

Re: [RFC PATCH 0/3] implement composite filters

2019-05-16 Thread Jonathan Tan
> Here is a first stab at composite filters. It does not actually support omits, > but the biggest difficulties of the implementation are already addressed. So I > decided to send out an early version to give interested people an idea of just > what is needed to implement it, and then give them a c

Re: [PATCH 2/2] index-pack: prefetch missing REF_DELTA bases

2019-05-16 Thread Jonathan Tan
> On Thu, May 16, 2019 at 11:26:46AM -0700, Jonathan Tan wrote: > > > > Pretty unlikely, but should we put some kind of circuit-breaker into the > > > client to ensure this? > > > > I thought of this - such a server could, but it seems to me that it > &

Re: [PATCH 2/2] index-pack: prefetch missing REF_DELTA bases

2019-05-16 Thread Jonathan Tan
> On Tue, May 14, 2019 at 02:10:55PM -0700, Jonathan Tan wrote: > > > Support for lazy fetching should still generally be turned off in > > index-pack because it is used as part of the lazy fetching process > > itself (if not, infinite loops may occur), but we do need to

Re: [PATCH 2/2] index-pack: prefetch missing REF_DELTA bases

2019-05-15 Thread Jonathan Tan
> > To resolve this, prefetch all missing REF_DELTA bases before attempting > > to resolve them. This both ensures that all bases are attempted to be > > fetched, and ensures that we make only one request per index-pack > > invocation, and not one request per missing object. > > Hmm. I wonder whet

Re: [PATCH 1/2] t5616: refactor packfile replacement

2019-05-15 Thread Jonathan Tan
> > +# Converts bytes into their hexadecimal representation. For example, > > +# "printf 'ab\r\n' | hex_unpack" results in '61620d0a'. > > +hex_unpack () { > > + perl -e '$/ = undef; $input = <>; print unpack("H2" x length($input), > > $input)' > > +} > > + > > +# Inserts $1 at the start of the

[PATCH 1/2] t5616: refactor packfile replacement

2019-05-14 Thread Jonathan Tan
A subsequent patch will perform the same packfile replacement that is already done twice, so refactor it into its own function. Also, the same subsequent patch will use, in another way, part of the packfile replacement functionality, so extract those out too. Signed-off-by: Jonathan Tan --- t

[PATCH 0/2] Partial clone fix: handling received REF_DELTA

2019-05-14 Thread Jonathan Tan
There is an issue when fetching into a partial clone, and the server sends a REF_DELTA object that is based on a missing promisor object. Here is a fix; more details are in the commit message of patch 2. Jonathan Tan (2): t5616: refactor packfile replacement index-pack: prefetch missing

[PATCH 2/2] index-pack: prefetch missing REF_DELTA bases

2019-05-14 Thread Jonathan Tan
attempted to be fetched, and ensures that we make only one request per index-pack invocation, and not one request per missing object. Signed-off-by: Jonathan Tan --- builtin/index-pack.c | 26 +++-- t/t5616-partial-clone.sh | 61 2 fil

Re: Proposal: object negotiation for partial clones

2019-05-09 Thread Jonathan Tan
> > On 2019/05/07, at 11:34, Jonathan Tan wrote: > > > > To get an enumeration of available objects, don't you need to use only > > "blob:none"? Combining filters (once that's implemented) will get all > > objects only up to a certain depth. >

Re: [PATCH v5 1/2] documentation: add tutorial for first contribution

2019-05-08 Thread Jonathan Tan
> > And "official" is a phrase I have trouble with in this context. A > > mirror does not have to be blessed as official; that's the point of > > a mirror---anybody can make one to help users with better > > connectivity or availability, as long as its users trust the mirror > > maintainer for mir

Re: [PATCH v4] documentation: add tutorial for first contribution

2019-05-07 Thread Jonathan Tan
> On Mon, May 06, 2019 at 03:28:44PM -0700, Jonathan Tan wrote: > > Sorry for not looking at this sooner. > > > > Firstly, I'm not sure if this file should be named without the ".txt", > > like SubmittingPatches. > > I should mention that during

Re: Proposal: object negotiation for partial clones

2019-05-07 Thread Jonathan Tan
> > My main question is: we can get the same list of objects (in the form of > > tree objects) if we fetch with "blob:none" filter. Admittedly, we will > > get extra data (file names, etc.) - if the extra bandwidth saving is > > necessary, this should be called out. (And some of the savings will be

Re: [PATCH v4] documentation: add tutorial for first contribution

2019-05-06 Thread Jonathan Tan
Sorry for not looking at this sooner. Firstly, I'm not sure if this file should be named without the ".txt", like SubmittingPatches. As for my other comments below, the Makefile comment below is the only one I feel strongly about; feel free to disagree with the rest (which I think are subjective

  1   2   3   4   5   6   7   8   9   10   >