[PATCH v4 01/15] upload-pack: add object filtering for partial clone

From: Jeff Hostetler <jeffh...@microsoft.com> Teach upload-pack to negotiate object filtering over the protocol and to send filter parameters to pack-objects. This is intended for partial clone and fetch. The idea to make upload-pack configurable using uploadpack.allowFilter comes from Jo

[PATCH v4 01/10] extension.partialclone: introduce partial clone extension

From: Jonathan Tan Introduce new repository extension option: `extensions.partialclone` See the update to Documentation/technical/repository-version.txt in this patch for more information. Signed-off-by: Jonathan Tan ---

[PATCH v4 00/10] Partial clone part 2: fsck and promisors

From: Jeff Hostetler <jeffh...@microsoft.com> This is part 2 of a 3 part sequence for partial clone. Part 2 assumes part 1 is in place. This patch series is labeled V4 to keep it in sync with the V4 version of part 1. (There was no V3 of this part.) Part 2 is concerned with fsck, gc, i

[PATCH v4 09/10] rev-list: support termination at promisor objects

ed-off-by: Jonathan Tan <jonathanta...@google.com> Signed-off-by: Jeff Hostetler <jeffh...@microsoft.com> --- Documentation/rev-list-options.txt | 12 - builtin/rev-list.c | 74 --- list-objects.c | 29 +++

[PATCH v4 08/10] sha1_file: support lazily fetching missing objects

From: Jonathan Tan Teach sha1_file to fetch objects from the remote configured in extensions.partialclone whenever an object is requested but missing. The fetching of objects can be suppressed through a global variable. This is used by fsck and index-pack. However, by

[PATCH v4 04/10] fsck: support referenced promisor objects

From: Jonathan Tan Teach fsck to not treat missing promisor objects indirectly pointed to by refs as an error when extensions.partialclone is set. Signed-off-by: Jonathan Tan --- builtin/fsck.c | 11 +++

[PATCH v4 10/10] gc: do not repack promisor packfiles

ed-off-by: Jonathan Tan <jonathanta...@google.com> Signed-off-by: Jeff Hostetler <jeffh...@microsoft.com> --- Documentation/git-pack-objects.txt | 12 - builtin/gc.c | 3 +++ builtin/pack-objects.c | 36 +

[PATCH v4 07/10] introduce fetch-object: fetch one promisor object

From: Jonathan Tan Introduce fetch-object, providing the ability to fetch one object from a promisor remote. This uses fetch-pack. To do this, the transport mechanism has been updated with 2 flags, "from-promisor" to indicate that the resulting pack comes from a

[PATCH v4 02/10] fsck: introduce partialclone extension

From: Jonathan Tan Currently, Git does not support repos with very large numbers of objects or repos that wish to minimize manipulation of certain blobs (for example, because they are very large) very well, even if the user operates mostly on part of the repo, because

[PATCH v4 03/10] fsck: support refs pointing to promisor objects

From: Jonathan Tan Teach fsck to not treat refs referring to missing promisor objects as an error when extensions.partialclone is set. For the purposes of warning about no default refs, such refs are still treated as legitimate refs. Signed-off-by: Jonathan Tan

[PATCH v4 05/10] fsck: support promisor objects as CLI argument

From: Jonathan Tan Teach fsck to not treat missing promisor objects provided on the CLI as an error when extensions.partialclone is set. Signed-off-by: Jonathan Tan --- builtin/fsck.c | 2 ++ t/t0410-partial-clone.sh | 13

[PATCH v4 06/10] index-pack: refactor writing of .keep files

From: Jonathan Tan In a subsequent commit, index-pack will be taught to write ".promisor" files which are similar to the ".keep" files it knows how to write. Refactor the writing of ".keep" files, so that the implementation of writing ".promisor" files becomes easier.

[PATCH v4 4/6] list-objects: filter objects in traverse_commit_list

From: Jeff Hostetler <jeffh...@microsoft.com> Create traverse_commit_list_filtered() and add filtering interface to allow certain objects to be omitted from the traversal. Update traverse_commit_list() to be a wrapper for the above with a null filter to minimize the number of callers that

[PATCH v4 0/6] Partial clone part 1: object filtering

From: Jeff Hostetler <jeffh...@microsoft.com> Here is V4 of the list-object filtering, rev-list, and pack-objects. This version addresses comments on the V3 version series. This version replaces the code to scan and reject the filter-spec for injection characters with a new hex-en

[PATCH v4 6/6] pack-objects: add list-objects filtering

From: Jeff Hostetler <jeffh...@microsoft.com> Teach pack-objects to use the filtering provided by the traverse_commit_list_filtered() interface to omit unwanted objects from the resulting packfile. This feature is intended for partial clone/fetch. Filtering requires the use of the &qu

[PATCH v4 3/6] oidset: add iterator methods to oidset

From: Jeff Hostetler <jeffh...@microsoft.com> Add the usual iterator methods to oidset. Add oidset_remove(). Signed-off-by: Jeff Hostetler <jeffh...@microsoft.com> --- oidset.c | 10 ++ oidset.h | 36 2 files changed, 46 insertions(+)

[PATCH v4 1/6] dir: allow exclusions from blob in addition to file

From: Jeff Hostetler <jeffh...@microsoft.com> Refactor add_excludes() to separate the reading of the exclude file into a buffer and the parsing of the buffer into exclude_list items. Add add_excludes_from_blob_to_list() to allow an exclude file be specified with an OID without assuming a

[PATCH v4 2/6] oidmap: add oidmap iterator methods

From: Jeff Hostetler <jeffh...@microsoft.com> Add the usual map iterator functions to oidmap. Signed-off-by: Jeff Hostetler <jeffh...@microsoft.com> --- oidmap.h | 22 ++ 1 file changed, 22 insertions(+) diff --git a/oidmap.h b/oidmap.h index 18f54cd..d3

[PATCH v4 5/6] rev-list: add list-objects filtering support

From: Jeff Hostetler <jeffh...@microsoft.com> Teach rev-list to use the filtering provided by the traverse_commit_list_filtered() interface to omit unwanted objects from the result. This feature is intended to help with partial clone. Object filtering is only allowed when one of the &quo

Re: [PATCH 04/14] fetch: add object filtering for partial fetch

On 11/3/2017 4:38 PM, Jonathan Tan wrote: @@ -1242,6 +1249,20 @@ static int fetch_multiple(struct string_list *list) int i, result = 0; struct argv_array argv = ARGV_ARRAY_INIT; + if (filter_options.choice) { + /* +* We currently only support

Re: [PATCH 02/14] clone, fetch-pack, index-pack, transport: partial clone

On 11/8/2017 1:01 PM, Adam Dinwoodie wrote: On Friday 03 November 2017 at 01:32 pm -0700, Jonathan Tan wrote: On Thu, 2 Nov 2017 20:31:17 + Jeff Hostetler <g...@jeffhostetler.com> wrote: diff --git a/builtin/index-pack.c b/builtin/index-pack.c index a0a35e6..31cd5ba 100644 --- a/b

Re: [PATCH v3 4/6] list-objects: filter objects in traverse_commit_list

On 11/8/2017 12:01 AM, Junio C Hamano wrote: Jonathan Tan writes: Having said that, though, it might be safer to still introduce one, and relax it later if necessary - it is much easier to relax a constraint than to increase one. It would also be more error prone

Re: [PATCH 1/9] extension.partialclone: introduce partial clone extension

On 11/8/2017 4:51 PM, Jonathan Tan wrote: On Wed, 8 Nov 2017 15:32:21 -0500 Jeff Hostetler <g...@jeffhostetler.com> wrote: Thanks Jonathan. I moved my version of part 2 on top of yesterday's part 1. There are a few changes between my version and yours. Could you take a quick look a

Re: [PATCH v3 4/6] list-objects: filter objects in traverse_commit_list

On 11/7/2017 6:20 PM, Jonathan Tan wrote: On Tue, 7 Nov 2017 19:35:44 + Jeff Hostetler <g...@jeffhostetler.com> wrote: +/* + * Reject the arg if it contains any characters that might + * require quoting or escaping when handing to a sub-command. + */ +static int reject_injection

Re: [PATCH 1/9] extension.partialclone: introduce partial clone extension

On 11/8/2017 4:51 PM, Jonathan Tan wrote: On Wed, 8 Nov 2017 15:32:21 -0500 Jeff Hostetler <g...@jeffhostetler.com> wrote: Thanks Jonathan. I moved my version of part 2 on top of yesterday's part 1. There are a few changes between my version and yours. Could you take a quick look a

Re: Test failures on 'pu' branch

On 11/8/2017 3:36 PM, Stefan Beller wrote: On Wed, Nov 8, 2017 at 12:28 PM, Ramsay Jones wrote: t5300-pack-object.sh (Wstat: 256 Tests: 40 Failed: 2) t5500-fetch-pack.sh (Wstat: 256 Tests: 355 Failed:

Re: [PATCH 1/9] extension.partialclone: introduce partial clone extension

On 11/6/2017 2:16 PM, Jonathan Tan wrote: On Mon, 6 Nov 2017 12:32:45 -0500 Jeff Hostetler <g...@jeffhostetler.com> wrote: Yes, that is a point I wanted to ask about. I renamed the extensions.partialclone that you created and then I moved your remote..blob-max-bytes s

Re: [PATCH v2 0/6] Partial clone part 1: object filtering

On 11/7/2017 7:54 PM, Junio C Hamano wrote: Jonathan Tan writes: I can see some use for this parameter - for example, when doing a report for statistical purposes (percentage of objects missing, for example) or for a background task that downloads missing objects

[PATCH v3 4/6] list-objects: filter objects in traverse_commit_list

From: Jeff Hostetler <jeffh...@microsoft.com> Create traverse_commit_list_filtered() and add filtering interface to allow certain objects to be omitted from the traversal. Update traverse_commit_list() to be a wrapper for the above with a null filter to minimize the number of callers that

[PATCH v3 1/6] dir: allow exclusions from blob in addition to file

From: Jeff Hostetler <jeffh...@microsoft.com> Refactor add_excludes() to separate the reading of the exclude file into a buffer and the parsing of the buffer into exclude_list items. Add add_excludes_from_blob_to_list() to allow an exclude file be specified with an OID without assuming a

[PATCH v3 6/6] pack-objects: add list-objects filtering

From: Jeff Hostetler <jeffh...@microsoft.com> Teach pack-objects to use the filtering provided by the traverse_commit_list_filtered() interface to omit unwanted objects from the resulting packfile. This feature is intended for partial clone/fetch. Filtering requires the use of the &qu

[PATCH v3 5/6] rev-list: add list-objects filtering support

From: Jeff Hostetler <jeffh...@microsoft.com> Teach rev-list to use the filtering provided by the traverse_commit_list_filtered() interface to omit unwanted objects from the result. This feature is intended to help with partial clone. Object filtering is only allowed when one of the &quo

[PATCH v3 2/6] oidmap: add oidmap iterator methods

From: Jeff Hostetler <jeffh...@microsoft.com> Add the usual map iterator functions to oidmap. Signed-off-by: Jeff Hostetler <jeffh...@microsoft.com> --- oidmap.h | 22 ++ 1 file changed, 22 insertions(+) diff --git a/oidmap.h b/oidmap.h index 18f54cd..d3

[PATCH v3 3/6] oidset: add iterator methods to oidset

From: Jeff Hostetler <jeffh...@microsoft.com> Add the usual iterator methods to oidset. Add oidset_remove(). Signed-off-by: Jeff Hostetler <jeffh...@microsoft.com> --- oidset.c | 10 ++ oidset.h | 36 2 files changed, 46 insertions(+)

[PATCH v3 0/6] Partial clone part 1: object filtering

From: Jeff Hostetler <jeffh...@microsoft.com> Here is V3 of the list-object filtering. This addresses the comments on the mailing list for the V2 series as well as the various TODO items I left in the code. I also documented some of the bit flags and fields that I added. In the blo

Re: [PATCH v2 4/6] list-objects: filter objects in traverse_commit_list

On 11/2/2017 3:32 PM, Jonathan Tan wrote: On Thu, 2 Nov 2017 17:50:11 + Jeff Hostetler <g...@jeffhostetler.com> wrote: + if (skip_prefix(v0, "oid=", )) { + filter_options->choice = LOFC_SPARSE_OID; + if (!get_

Re: [PATCH v2 4/6] list-objects: filter objects in traverse_commit_list

On 11/2/2017 1:50 PM, Jeff Hostetler wrote: From: Jeff Hostetler <jeffh...@microsoft.com> Create traverse_commit_list_filtered() and add filtering interface to allow certain objects to be omitted from the traversal. ... diff --git a/list-objects-filter.c b/list-objects-filter.c new fil

Re: [PATCH 1/9] extension.partialclone: introduce partial clone extension

On 11/3/2017 2:39 PM, Jonathan Tan wrote: On Fri, 3 Nov 2017 09:57:18 -0400 Jeff Hostetler <g...@jeffhostetler.com> wrote: On 11/2/2017 6:24 PM, Jonathan Tan wrote: On Thu, 2 Nov 2017 20:20:44 +0000 Jeff Hostetler <g...@jeffhostetler.com> wrote: From: Jeff Host

Re: [PATCH v2 0/6] Partial clone part 1: object filtering

On 11/3/2017 11:05 AM, Junio C Hamano wrote: Jeff Hostetler <g...@jeffhostetler.com> writes: Yes, I thought we should have both (perhaps renamed or combined into 1 parameter with value, such as --exclude=missing vs --exclude=promisor) and let the user decide how strict the

Re: [PATCH] fix an 'dubious one-bit signed bitfield' error

d'oh. thanks! On 11/3/2017 1:05 PM, Ramsay Jones wrote: Signed-off-by: Ramsay Jones --- Hi Jeff, If you need to re-roll your 'jh/object-filtering' branch, could you please squash this into the relevant commit (b87fd93d81, "list-objects: filter objects in

Re: [PATCH 00/14] WIP Partial clone part 3: clone, fetch, fetch-pack, upload-pack, and tests

On 11/2/2017 7:41 PM, Jonathan Tan wrote: On Thu, 2 Nov 2017 20:31:15 + Jeff Hostetler <g...@jeffhostetler.com> wrote: From: Jeff Hostetler <jeffh...@microsoft.com> This is part 3 of 3 for partial clone. It assumes that part 1 [1] and part 2 [2] are in place. Part 3

Re: [PATCH 1/9] extension.partialclone: introduce partial clone extension

On 11/2/2017 6:24 PM, Jonathan Tan wrote: On Thu, 2 Nov 2017 20:20:44 + Jeff Hostetler <g...@jeffhostetler.com> wrote: From: Jeff Hostetler <jeffh...@microsoft.com> Introduce the ability to have missing objects in a repo. This functionality is guarded by new reposito

Re: [PATCH v2 0/6] Partial clone part 1: object filtering

On 11/2/2017 3:44 PM, Jonathan Tan wrote: On Thu, 2 Nov 2017 17:50:07 + Jeff Hostetler <g...@jeffhostetler.com> wrote: From: Jeff Hostetler <jeffh...@microsoft.com> Here is V2 of the list-object filtering. It replaces [1] and reflect a refactoring and simplification of

Re: [PATCH v2 4/6] list-objects: filter objects in traverse_commit_list

On 11/3/2017 7:54 AM, Johannes Schindelin wrote: Hi Jonathan, On Thu, 2 Nov 2017, Jonathan Tan wrote: On Thu, 2 Nov 2017 17:50:11 + Jeff Hostetler <g...@jeffhostetler.com> wrote: +int parse_list_objects_filter(struct list_objects_filter_options *filter_o

[PATCH 01/14] upload-pack: add object filtering for partial clone

From: Jeff Hostetler <jeffh...@microsoft.com> Teach upload-pack to negotiate object filtering over the protocol and to send filter parameters to pack-objects. This is intended for partial clone and fetch. The idea to make upload-pack configurable using uploadpack.allowFilter comes from Jo

[PATCH 07/14] fetch-pack: test support excluding large blobs

From: Jonathan Tan <jonathanta...@google.com> Created tests to verify fetch-pack and upload-pack support for excluding large blobs using --filter=blobs:limit= parameter. Signed-off-by: Jonathan Tan <jonathanta...@google.com> Signed-off-by: Jeff Hostetler <jeffh...@microsoft.co

[PATCH 03/14] fetch: refactor calculation of remote list

From: Jonathan Tan Separate out the calculation of remotes to be fetched from and the actual fetching. This will allow us to include an additional step before the actual fetching in a subsequent commit. Signed-off-by: Jonathan Tan ---

[PATCH 13/14] fetch-pack: restore save_commit_buffer after use

ithout the message (which is what the commit does), and did not mention parsing the unparsed commit, so I prefer to preserve the existing behavior. Signed-off-by: Jonathan Tan <jonathanta...@google.com> Signed-off-by: Jeff Hostetler <jeffh...@microsoft.com> --- fetch-pack.c | 4 +

[PATCH 06/14] pack-objects: test support for blob filtering

ies. Signed-off-by: Jonathan Tan <jonathanta...@google.com> Signed-off-by: Jeff Hostetler <jeffh...@microsoft.com> --- t/t5300-pack-object.sh | 45 + t/test-lib-functions.sh | 12 2 files changed, 57 insertions(+) diff --git

[PATCH 09/14] t5500: add fetch-pack tests for partial clone

From: Jonathan Tan <jonathanta...@google.com> Signed-off-by: Jonathan Tan <jonathanta...@google.com> Signed-off-by: Jeff Hostetler <jeffh...@microsoft.com> --- t/t5500-fetch-pack.sh | 36 1 file changed, 36 insertions(+) diff --git a/t/t55

[PATCH 12/14] unpack-trees: batch fetching of missing blobs

al clones, we already know what needs to be fetched based on the contents of the local repo alone, whereas for status=delayed, it is the filter process that tells us what needs to be checked in the end. Signed-off-by: Jonathan Tan <jonathanta...@google.com> Signed-off-by: Jeff Hostetler <

[PATCH 14/14] index-pack: silently assume missing objects are promisor

From: Jeff Hostetler <jeffh...@microsoft.com> Teach index-pack to not complain about missing objects when the --promisor flag is given. The assumption is that index-pack is currently building the idx and promisor data and the is_promisor_object() query would fail anyway. Signed-off-by

[PATCH 11/14] t5500: more tests for partial clone and fetch

From: Jonathan Tan <jonathanta...@google.com> Signed-off-by: Jonathan Tan <jonathanta...@google.com> Signed-off-by: Jeff Hostetler <jeffh...@microsoft.com> --- t/t5500-fetch-pack.sh | 60 +++ 1 file changed, 56 insertions(+), 4

[PATCH 00/14] WIP Partial clone part 3: clone, fetch, fetch-pack, upload-pack, and tests

From: Jeff Hostetler <jeffh...@microsoft.com> This is part 3 of 3 for partial clone. It assumes that part 1 [1] and part 2 [2] are in place. Part 3 is concerned with the commands: clone, fetch, upload-pack, fetch-pack, remote-curl, index-pack, and the pack-protocol. Jonathan and I indepen

[PATCH 02/14] clone, fetch-pack, index-pack, transport: partial clone

From: Jeff Hostetler <jeffh...@microsoft.com> Signed-off-by: Jeff Hostetler <jeffh...@microsoft.com> --- builtin/clone.c | 9 + builtin/fetch-pack.c | 4 builtin/index-pack.c | 10 ++ fetch-pack.c | 13 + fetch-pack.h | 2 +

[PATCH 04/14] fetch: add object filtering for partial fetch

From: Jeff Hostetler <jeffh...@microsoft.com> Teach fetch to use the list-objects filtering parameters to allow a "partial fetch" following a "partial clone". Signed-off-by: Jeff Hostetler <jeffh...@microsoft.c

[PATCH 08/14] fetch: add from_promisor and exclude-promisor-objects parameters

From: Jonathan Tan <jonathanta...@google.com> Teach fetch to use from-promisor and exclude-promisor-objects parameters with sub-commands. Initialize fetch_if_missing global variable. Signed-off-by: Jonathan Tan <jonathanta...@google.com> Signed-off-by: Jeff Hostetler <jeffh..

[PATCH 10/14] t5601: test for partial clone

From: Jonathan Tan <jonathanta...@google.com> Signed-off-by: Jonathan Tan <jonathanta...@google.com> Signed-off-by: Jeff Hostetler <jeffh...@microsoft.com> --- builtin/clone.c | 17 ++--- t/t5601-clone.sh | 49 + 2

[PATCH 05/14] remote-curl: add object filtering for partial clone

From: Jeff Hostetler <jeffh...@microsoft.com> Signed-off-by: Jeff Hostetler <jeffh...@microsoft.com> --- Documentation/gitremote-helpers.txt | 4 remote-curl.c | 10 -- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/Documentati

[PATCH 8/9] sha1_file: support lazily fetching missing objects

oogle.com> Signed-off-by: Jeff Hostetler <jeffh...@microsoft.com> --- builtin/cat-file.c | 3 + builtin/fetch-pack.c | 2 + builtin/fsck.c | 3 + builtin/index-pack.c | 6 ++ builtin/rev-list.c | 35 +-- cache.h | 8 +

[PATCH 1/9] extension.partialclone: introduce partial clone extension

From: Jeff Hostetler <jeffh...@microsoft.com> Introduce the ability to have missing objects in a repo. This functionality is guarded by new repository extension options: `extensions.partialcloneremote` and `extensions.partialclonefilter`. See the update to Documentation/tec

[PATCH 6/9] index-pack: refactor writing of .keep files

ing ".promisor" files becomes easier. Signed-off-by: Jonathan Tan <jonathanta...@google.com> Signed-off-by: Jeff Hostetler <jeffh...@microsoft.com> --- builtin/index-pack.c | 99 1 file changed, 53 insertions(+), 46 delet

[PATCH 3/9] fsck: support refs pointing to promisor objects

an Tan <jonathanta...@google.com> Signed-off-by: Jeff Hostetler <jeffh...@microsoft.com> --- builtin/fsck.c | 8 t/t0410-partial-clone.sh | 24 2 files changed, 32 insertions(+) diff --git a/builtin/fsck.c b/builtin/fsck.c index 2934299..ee937bb 10

[PATCH 2/9] fsck: introduce partialclone extension

reflog are not an error case; in future commits, fsck will be taught about other cases. Signed-off-by: Jonathan Tan <jonathanta...@google.com> Signed-off-by: Jeff Hostetler <jeffh...@microsoft.com> --- builtin/fsck.c | 2 +- cache.h | 3 +- packfile.c

[PATCH 9/9] gc: do not repack promisor packfiles

ed-off-by: Jonathan Tan <jonathanta...@google.com> Signed-off-by: Jeff Hostetler <jeffh...@microsoft.com> --- Documentation/git-pack-objects.txt | 4 +++ builtin/gc.c | 4 +++ builtin/pack-objects.c | 14 ++ builtin/prune.c|

[PATCH 5/9] fsck: support promisor objects as CLI argument

From: Jonathan Tan <jonathanta...@google.com> Teach fsck to not treat missing promisor objects provided on the CLI as an error when extensions.partialcloneremote is set. Signed-off-by: Jonathan Tan <jonathanta...@google.com> Signed-off-by: Jeff Hostetler <jeffh...@microsoft.co

[PATCH 7/9] introduce fetch-object: fetch one promisor object

ay to suppress the ref advertisement and officially allow any object type to be "want"-ed. Signed-off-by: Jonathan Tan <jonathanta...@google.com> Signed-off-by: Jeff Hostetler <jeffh...@microsoft.com> --- Documentation/gitremote-helpers.txt | 2 ++ Makefile

[PATCH 4/9] fsck: support referenced promisor objects

From: Jonathan Tan <jonathanta...@google.com> Teach fsck to not treat missing promisor objects indirectly pointed to by refs as an error when extensions.partialcloneremote is set. Signed-off-by: Jonathan Tan <jonathanta...@google.com> Signed-off-by: Jeff Hostetler <jeffh..

[PATCH 0/9] WIP Partial clone part 2: fsck and promisors

From: Jeff Hostetler <jeffh...@microsoft.com> This is part 2 of a proposed 3 part sequence for partial clone. Part 2 assumes part 1 [1] is in place. Part 2 is concerned with fsck, gc, initial support for dynamic object fetching, and tracking promisor objects. Jonathan Tan originally dev

[PATCH v2 4/6] list-objects: filter objects in traverse_commit_list

From: Jeff Hostetler <jeffh...@microsoft.com> Create traverse_commit_list_filtered() and add filtering interface to allow certain objects to be omitted from the traversal. Update traverse_commit_list() to be a wrapper for the above with a null filter to minimize the number of callers that

[PATCH v2 5/6] rev-list: add list-objects filtering support

From: Jeff Hostetler <jeffh...@microsoft.com> Teach rev-list to use the filtering provided by the traverse_commit_list_filtered() interface to omit unwanted objects from the result. This feature is intended to help with partial clone. Object filtering is only allowed when one of the &quo

[PATCH v2 3/6] oidset: add iterator methods to oidset

From: Jeff Hostetler <jeffh...@microsoft.com> Add the usual iterator methods to oidset. Add oidset_remove(). Signed-off-by: Jeff Hostetler <jeffh...@microsoft.com> --- oidset.c | 10 ++ oidset.h | 36 2 files changed, 46 insertions(+)

[PATCH v2 6/6] pack-objects: add list-objects filtering

From: Jeff Hostetler <jeffh...@microsoft.com> Teach pack-objects to use the filtering provided by the traverse_commit_list_filtered() interface to omit unwanted objects from the resulting packfile. This feature is intended for partial clone/fetch. Filtering requires the use of the &qu

[PATCH v2 1/6] dir: allow exclusions from blob in addition to file

From: Jeff Hostetler <jeffh...@microsoft.com> Refactor add_excludes() to separate the reading of the exclude file into a buffer and the parsing of the buffer into exclude_list items. Add add_excludes_from_blob_to_list() to allow an exclude file be specified with an OID without assuming a

[PATCH v2 0/6] Partial clone part 1: object filtering

From: Jeff Hostetler <jeffh...@microsoft.com> Here is V2 of the list-object filtering. It replaces [1] and reflect a refactoring and simplification of the original. After much discussion on the "list-object-filter-map" I've replaced it with a regular oidset -- the only

[PATCH v2 2/6] oidmap: add oidmap iterator methods

From: Jeff Hostetler <jeffh...@microsoft.com> Add the usual map iterator functions to oidmap. Signed-off-by: Jeff Hostetler <jeffh...@microsoft.com> --- oidmap.h | 22 ++ 1 file changed, 22 insertions(+) diff --git a/oidmap.h b/oidmap.h index 18f54cd..d3

RE: What's cooking in git.git (Oct 2017, #07; Mon, 30)

From: Junio C Hamano [mailto:gits...@pobox.com] Subject: Re: What's cooking in git.git (Oct 2017, #07; Mon, 30) > Jeff Hostetler <g...@jeffhostetler.com> writes: > >> I've been assuming that the jt/partial-clone-lazy-fetch is a >> placeholder for our next combin

Re: What's cooking in git.git (Oct 2017, #07; Mon, 30)

On 10/30/2017 1:31 PM, Johannes Schindelin wrote: Hi Junio, On Mon, 30 Oct 2017, Junio C Hamano wrote: * jt/partial-clone-lazy-fetch (2017-10-02) 18 commits - fetch-pack: restore save_commit_buffer after use - unpack-trees: batch fetching of missing blobs - clone: configure

Re: [PATCH] dir: allow exclusions from blob in addition to file

On 10/26/2017 9:20 PM, Junio C Hamano wrote: Jeff Hostetler <g...@jeffhostetler.com> writes: From: Jeff Hostetler <jeffh...@microsoft.com> Refactor add_excludes() to separate the reading of the exclude file into a buffer and the parsing of the buffer into exclude_list

Re: [PATCH 01/13] dir: allow exclusions from blob in addition to file

On 10/25/2017 11:47 PM, Junio C Hamano wrote: Jeff Hostetler <g...@jeffhostetler.com> writes: The existing code handles use cases where you want to read the exclusion list from a pathname in the worktree -- or from blob named in the index when the pathname is not populated (pres

[PATCH] dir: allow exclusions from blob in addition to file

From: Jeff Hostetler <jeffh...@microsoft.com> I pulled commit 01/13 from Tuesday's partial clone part 1 patch series [1] and refactored the changes in dir.c to try to address Junio's comments in [2] WRT sharing more code with the existing read_skip_worktree_file_from_index(). This pat

[PATCH] dir: allow exclusions from blob in addition to file

From: Jeff Hostetler <jeffh...@microsoft.com> Refactor add_excludes() to separate the reading of the exclude file into a buffer and the parsing of the buffer into exclude_list items. Add add_excludes_from_blob_to_list() to allow an exclude file be specified with an OID without assuming a

Re: [PATCH 10/13] rev-list: add list-objects filtering support

On 10/25/2017 12:41 AM, Jonathan Tan wrote: On Tue, Oct 24, 2017 at 11:53 AM, Jeff Hostetler <g...@jeffhostetler.com> wrote: static void finish_object(struct object *obj, const char *name, void *cb_data) { struct rev_list_info *info = cb_data; - if (obj->type ==

Re: [PATCH 08/13] list-objects: add traverse_commit_list_filtered method

On 10/25/2017 12:24 AM, Jonathan Tan wrote: On Tue, Oct 24, 2017 at 11:53 AM, Jeff Hostetler <g...@jeffhostetler.com> wrote: +void traverse_commit_list_filtered( + struct list_objects_filter_options *filter_options, + struct rev_info *revs, + show_commit_fn show_

Re: [PATCH 07/13] list-objects-filter-options: common argument parsing

On 10/25/2017 12:14 AM, Jonathan Tan wrote: On Tue, Oct 24, 2017 at 11:53 AM, Jeff Hostetler <g...@jeffhostetler.com> wrote: + * ::= blob:none + * blob:limit:[kmg] + * sparse:oid: + * sparse:path: I notice in the code below that there are some

Re: [PATCH 03/13] list-objects: filter objects in traverse_commit_list

On 10/25/2017 12:05 AM, Jonathan Tan wrote: On Tue, Oct 24, 2017 at 11:53 AM, Jeff Hostetler <g...@jeffhostetler.com> wrote: +enum list_objects_filter_result { + LOFR_ZERO = 0, + LOFR_MARK_SEEN = 1<<0, Probably worth documenting, something like /* Mark th

Re: [PATCH 02/13] list-objects-filter-map: extend oidmap to collect omitted objects

On 10/25/2017 3:10 AM, Junio C Hamano wrote: Jeff Hostetler <g...@jeffhostetler.com> writes: From: Jeff Hostetler <jeffh...@microsoft.com> Create helper class to extend oidmap to collect a list of omitted or missing objects during traversal. The reason why oidmap itself ca

Re: [PATCH 00/13] WIP Partial clone part 1: object filtering

we should, as I wrote in my reply to Jeff Hostetler [1]. As for how this patch set (excluding the partialclone patch) interacts with my fsck series, they are relatively independent, as far as I can tell. I'll rebase my fsck, gc, and lazy object fetch patches (but not the fetch and clone part

Re: [PATCH 01/13] dir: allow exclusions from blob in addition to file

On 10/25/2017 2:43 AM, Junio C Hamano wrote: Jeff Hostetler <g...@jeffhostetler.com> writes: +static int add_excludes_from_buffer(char *buf, size_t size, + const char *base, int baselen, + struct exclude_li

[PATCH 10/13] rev-list: add list-objects filtering support

From: Jeff Hostetler <jeffh...@microsoft.com> Teach rev-list to use the filtering provided by the traverse_commit_list_filtered() interface to omit unwanted objects from the result. This feature is only enabled when one of the "--objects*" options are used. Furthermore, when the

[PATCH 12/13] pack-objects: add list-objects filtering

From: Jeff Hostetler <jeffh...@microsoft.com> Teach pack-objects to use the filtering provided by the traverse_commit_list_filtered() interface to omit unwanted objects from the resulting packfile. This feature is intended for partial clone/fetch. Filtering requires the use of the &qu

[PATCH 09/13] extension.partialclone: introduce partial clone extension

From: Jeff Hostetler <jeffh...@microsoft.com> Introduce the ability to have missing objects in a repo. This functionality is guarded by new repository extension options: `extensions.partialcloneremote` and `extensions.partialclonefilter`. See the update to Documentation/tec

[PATCH 05/13] list-objects-filter-blobs-limit: add large blob filtering

From: Jeff Hostetler <jeffh...@microsoft.com> Create a filter for traverse_commit_list_worker() to omit blobs larger than a requested size from the result, but always include ".git*" special files. Signed-off-by: Jeff Hostetler <jeffh...@microsoft

[PATCH 04/13] list-objects-filter-blobs-none: add filter to omit all blobs

From: Jeff Hostetler <jeffh...@microsoft.com> Create a simple filter for traverse_commit_list_worker() to omit all blobs from the result. This filter will be used in a future commit by rev-list and pack-objects to create a "commits and trees" result. This is intended for partial

[PATCH 13/13] t5317: pack-objects object filtering test

From: Jeff Hostetler <jeffh...@microsoft.com> Signed-off-by: Jeff Hostetler <jeffh...@microsoft.com> --- t/t5317-pack-objects-filter-objects.sh | 384 + 1 file changed, 384 insertions(+) create mode 100755 t/t5317-pack-objects-filter-objects.sh di

[PATCH 01/13] dir: allow exclusions from blob in addition to file

From: Jeff Hostetler <jeffh...@microsoft.com> Refactor add_excludes() to separate the reading of the exclude file into a buffer and the parsing of the buffer into exclude_list items. Add add_excludes_from_blob_to_list() to allow an exclude file be specified with an OID. Signed-off-by

[PATCH 03/13] list-objects: filter objects in traverse_commit_list

From: Jeff Hostetler <jeffh...@microsoft.com> Create traverse_commit_list_filtered() and add filtering interface to allow certain objects to be omitted (not shown) during a traversal. Update traverse_commit_list() to be a wrapper for the above. Filtering will be used in a future commit

[PATCH 07/13] list-objects-filter-options: common argument parsing

From: Jeff Hostetler <jeffh...@microsoft.com> Create common routines and defines for parsing list-objects-filter-related command line arguments and pack-protocol fields. Signed-off-by: Jeff Hostetler <jeffh...@microsoft.com> --- Makefile | 1 + list-ob

[PATCH 11/13] t6112: rev-list object filtering test

From: Jeff Hostetler <jeffh...@microsoft.com> Signed-off-by: Jeff Hostetler <jeffh...@microsoft.com> --- t/t6112-rev-list-filters-objects.sh | 223 1 file changed, 223 insertions(+) create mode 100755 t/t6112-rev-list-filters-objects.sh diff --g

[PATCH 06/13] list-objects-filter-sparse: add sparse filter

From: Jeff Hostetler <jeffh...@microsoft.com> Create a filter for traverse_commit_list_worker() to only include the blobs the would be referenced by a sparse-checkout using the given specification. Signed-off-by: Jeff Hostetler <jeffh...@microsoft.com> --- Makefile

<    1   2   3   4   5   6   7   >