[PATCH v6 10/35] protocol: introduce enum protocol_version value protocol_v2

2018-03-15 Thread Brandon Williams
Introduce protocol_v2, a new value for 'enum protocol_version'. Subsequent patches will fill in the implementation of protocol_v2. Signed-off-by: Brandon Williams --- builtin/fetch-pack.c | 2 ++ builtin/receive-pack.c | 6 ++ builtin/send-pack.c| 3 +++ builtin/upload-p

[PATCH v6 09/35] transport: store protocol version

2018-03-15 Thread Brandon Williams
ned-off-by: Brandon Williams --- transport.c | 35 ++- 1 file changed, 26 insertions(+), 9 deletions(-) diff --git a/transport.c b/transport.c index 63c3dbab94..2378dcb38c 100644 --- a/transport.c +++ b/transport.c @@ -118,6 +118,7 @@ struct git_transport_data {

[PATCH v6 11/35] test-pkt-line: introduce a packet-line test helper

2018-03-15 Thread Brandon Williams
Introduce a packet-line test helper which can either pack or unpack an input stream into packet-lines and writes out the result to stdout. Signed-off-by: Brandon Williams --- Makefile | 1 + t/helper/test-pkt-line.c | 64 2 files changed

[PATCH v6 13/35] ls-refs: introduce ls-refs server command

2018-03-15 Thread Brandon Williams
based on provided ref-prefixes. Signed-off-by: Brandon Williams --- Documentation/technical/protocol-v2.txt | 31 +++ Makefile| 1 + ls-refs.c | 96 ls-refs.h | 10

[PATCH v6 16/35] transport: convert transport_get_remote_refs to take a list of ref prefixes

2018-03-15 Thread Brandon Williams
Teach transport_get_remote_refs() to accept a list of ref prefixes, which will be sent to the server for use in filtering when using protocol v2. (This list will be ignored when not using protocol v2.) Signed-off-by: Brandon Williams --- builtin/clone.c | 2 +- builtin/fetch.c | 4

[PATCH v6 22/35] fetch-pack: support shallow requests

2018-03-15 Thread Brandon Williams
Enable shallow clones and deepen requests using protocol version 2 if the server 'fetch' command supports the 'shallow' feature. Signed-off-by: Brandon Williams --- Documentation/technical/protocol-v2.txt | 18 --- connect.c | 22

[PATCH v6 19/35] push: pass ref prefixes when pushing

2018-03-15 Thread Brandon Williams
Construct a list of ref prefixes to be passed to 'get_refs_list()' from the refspec to be used during the push. This list of ref prefixes will be used to allow the server to filter the ref advertisement when communicating using protocol v2. Signed-off-by: Brandon Williams --- transp

[PATCH v6 29/35] remote-curl: create copy of the service name

2018-03-15 Thread Brandon Williams
Make a copy of the service name being requested instead of relying on the buffer pointed to by the passed in 'const char *' to remain unchanged. Currently, all service names are string constants, but a subsequent patch will introduce service names from external sources. Signed-off-b

[PATCH v6 30/35] remote-curl: store the protocol version the server responded with

2018-03-15 Thread Brandon Williams
Store the protocol version the server responded with when performing discovery. This will be used in a future patch to either change the 'Git-Protocol' header sent in subsequent requests or to determine if a client needs to fallback to using a different protocol version. Signed-off-b

[PATCH v6 18/35] fetch: pass ref prefixes when fetching

2018-03-15 Thread Brandon Williams
Construct a list of ref prefixes to be passed to 'transport_get_remote_refs()' from the refspec to be used during the fetch. This list of ref prefixes will be used to allow the server to filter the ref advertisement when communicating using protocol v2. Signed-off-by: Brando

[PATCH v6 27/35] transport-helper: introduce stateless-connect

2018-03-15 Thread Brandon Williams
y in our http remote-helper (remote-curl) so that protocol v2 can be used using the http transport. Signed-off-by: Brandon Williams --- Documentation/gitremote-helpers.txt | 32 + transport-helper.c | 11 ++ transport.c

[PATCH v6 33/35] http: eliminate "# service" line when using protocol v2

2018-03-15 Thread Brandon Williams
When an http info/refs request is made, requesting that protocol v2 be used, don't send a "# service" line since this line is not part of the v2 spec. Signed-off-by: Brandon Williams --- http-backend.c | 8 ++-- remote-curl.c | 3 +++ 2 files changed, 9 insertions(+), 2 del

[PATCH v6 34/35] remote-curl: implement stateless-connect command

2018-03-15 Thread Brandon Williams
as a pass through to convert requests to http. Signed-off-by: Brandon Williams --- remote-curl.c | 207 - t/t5702-protocol-v2.sh | 45 + 2 files changed, 251 insertions(+), 1 deletion(-) diff --git a/remote-curl.c b/remote-curl.c index

[PATCH v6 26/35] transport-helper: refactor process_connect_service

2018-03-15 Thread Brandon Williams
A future patch will need to take advantage of the logic which runs and processes the response of the connect command on a remote helper so factor out this logic from 'process_connect_service()' and place it into a helper function 'run_connect()'. Signed-off-by: Brandon Will

[PATCH v6 17/35] ls-remote: pass ref prefixes when requesting a remote's refs

2018-03-15 Thread Brandon Williams
Construct an argv_array of ref prefixes based on the patterns supplied via the command line and pass them to 'transport_get_remote_refs()' to be used when communicating protocol v2 so that the server can limit the ref advertisement based on those prefixes. Signed-off-by: Brando

[PATCH v6 20/35] upload-pack: introduce fetch server command

2018-03-15 Thread Brandon Williams
Introduce the 'fetch' server command. Signed-off-by: Brandon Williams --- Documentation/technical/protocol-v2.txt | 125 +++ serve.c | 2 + t/t5701-git-serve.sh| 1 + upload-pack.c

[PATCH v6 12/35] serve: introduce git-serve

2018-03-15 Thread Brandon Williams
t the completion of a command, a client can request that another command be executed or can terminate the connection by sending a flush packet. Signed-off-by: Brandon Williams --- .gitignore | 1 + Documentation/Makefile | 1 + Documentation/tech

[PATCH v6 35/35] remote-curl: don't request v2 when pushing

2018-03-15 Thread Brandon Williams
27;receive-pack' endpoint and fall back to using v0, but when push v2 is rolled out to servers, the 'receive-pack' endpoint will start responding using v2. So we don't want to get into a state where a client is requesting to push with v2 before they actually know how to pus

[PATCH v6 32/35] http: don't always add Git-Protocol header

2018-03-15 Thread Brandon Williams
Instead of always sending the Git-Protocol header with the configured version with every http request, explicitly send it when discovering refs and then only send it on subsequent http requests if the server understood the version requested. Signed-off-by: Brandon Williams --- http.c

[PATCH v6 28/35] pkt-line: add packet_buf_write_len function

2018-03-15 Thread Brandon Williams
Add the 'packet_buf_write_len()' function which allows for writing an arbitrary length buffer into a 'struct strbuf' and formatting it in packet-line format. Signed-off-by: Brandon Williams --- pkt-line.c | 16 pkt-line.h | 1 + 2 files changed, 17 inserti

[PATCH v6 31/35] http: allow providing extra headers for http requests

2018-03-15 Thread Brandon Williams
Add a way for callers to request that extra headers be included when making http requests. Signed-off-by: Brandon Williams --- http.c | 8 http.h | 7 +++ 2 files changed, 15 insertions(+) diff --git a/http.c b/http.c index 5977712712..e1757d62b2 100644 --- a/http.c +++ b/http.c

[PATCH v6 23/35] connect: refactor git_connect to only get the protocol version once

2018-03-15 Thread Brandon Williams
ful in the next patch where we can have centralized logic which determines if we need to request a different protocol version than what the user has configured. Signed-off-by: Brandon Williams --- connect.c | 27 +++ 1 file changed, 15 insertions(+), 12 deletions(-) di

[PATCH v6 21/35] fetch-pack: perform a fetch using v2

2018-03-15 Thread Brandon Williams
When communicating with a v2 server, perform a fetch by requesting the 'fetch' command. Signed-off-by: Brandon Williams --- Documentation/technical/protocol-v2.txt | 67 +- builtin/fetch-pack.c| 2 +- fetch-pack.c

[PATCH v6 25/35] transport-helper: remove name parameter

2018-03-15 Thread Brandon Williams
was removed the parameter was no longer necessary but wasn't removed. Clean up 'recvline_fh()' parameter list by removing the 'name' parameter. Signed-off-by: Brandon Williams --- transport-helper.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tra

[PATCH v6 24/35] connect: don't request v2 when pushing

2018-03-15 Thread Brandon Williams
27;receive-pack' endpoint and fall back to using v0, but when push v2 is rolled out to servers, the 'receive-pack' endpoint will start responding using v2. So we don't want to get into a state where a client is requesting to push with v2 before they actually know how to pus

[PATCH v6 15/35] transport: convert get_refs_list to take a list of ref prefixes

2018-03-15 Thread Brandon Williams
fs it sends based on the sent prefixes. This list will be ignored when not using protocol v2. Signed-off-by: Brandon Williams --- transport-helper.c | 5 +++-- transport-internal.h | 11 ++- transport.c | 18 +++--- 3 files changed, 24 insertions(+), 10 deleti

[PATCH v6 14/35] connect: request remote refs using v2

2018-03-15 Thread Brandon Williams
Teach the client to be able to request a remote's refs using protocol v2. This is done by having a client issue a 'ls-refs' request to a v2 server. Signed-off-by: Brandon Williams --- builtin/upload-pack.c | 10 +-- connect.c

[PATCH v6 05/35] upload-pack: factor out processing lines

2018-03-15 Thread Brandon Williams
Factor out the logic for processing shallow, deepen, deepen_since, and deepen_not lines into their own functions to simplify the 'receive_needs()' function in addition to making it easier to reuse some of this logic when implementing protocol_v2. Signed-off-by: Brandon Williams -

[PATCH v6 04/35] upload-pack: convert to a builtin

2018-03-15 Thread Brandon Williams
In order to allow for code sharing with the server-side of fetch in protocol-v2 convert upload-pack to be a builtin. Signed-off-by: Brandon Williams --- Makefile | 3 +- builtin.h | 1 + builtin/upload-pack.c | 67 ++ git.c

Re: [PATCH 01/44] repository: initialize the_repository in main()

2018-03-21 Thread Brandon Williams
t(struct repository *repo, const char *gitdir, const char > *worktree); I had to double check that this was ok, it indeed is as repo_init is only used in repository.c. > +extern void initialize_the_repository(void); > extern int repo_submodule_init(struct repository *submodule, > struct repository *superproject, > const char *path); > -- > 2.16.1.435.g8f24da2e1a > -- Brandon Williams

Re: [PATCH 02/44] repository.c: move env-related setup code back to environment.c

2018-03-21 Thread Brandon Williams
wly) in > the process of unifying the repo setup for the main repo and > submodules, which hopefully may kill this function or replace it with > something better. But it's too early to tell. Since this part of the > series has landed in 'next', I'll post a fixup patch soon with your > suggestion. > -- > Duy Yeah looking at the patch this is probably my only complaint. The rest looked good. -- Brandon Williams

Re: [PATCH 03/44] repository.c: delete dead functions

2018-03-21 Thread Brandon Williams
16.1.435.g8f24da2e1a > This patch needs to be squashed into the previous one, the build breaks otherwise (unless built with -Werror=unused-function). -- Brandon Williams

Re: [PATCH 04/44] sha1_file.c: move delayed getenv(altdb) back to setup_git_env()

2018-03-21 Thread Brandon Williams
cb) > > void prepare_alt_odb(void) > { > - const char *alt; > - > if (alt_odb_tail) > return; > > - alt = getenv(ALTERNATE_DB_ENVIRONMENT); > - > alt_odb_tail = &alt_odb_list; > - link_alt_odb_entries(alt, PATH_SEP, NULL, 0); > + link_alt_odb_entries(the_repository->alternate_db, PATH_SEP, NULL, 0); > > read_info_alternates(get_object_directory(), 0); > } > -- > 2.16.1.435.g8f24da2e1a > -- Brandon Williams

Re: [PATCH 05/44] repository: delete ignore_env member

2018-03-21 Thread Brandon Williams
which doesn't go through > - * the normal 'repo_init()' process. > - */ > - unsigned ignore_env:1; > > /* Indicate if a repository has a different 'commondir' from 'gitdir' */ > unsigned different_commondir:1; > -- > 2.16.1.435.g8f24da2e1a > -- Brandon Williams

Re: [PATCH 06/44] repository: introduce raw object store field

2018-03-21 Thread Brandon Williams
ude "object-store.h" > + > struct config_set; > struct index_state; > struct submodule_cache; > @@ -21,13 +23,9 @@ struct repository { > char *commondir; > > /* > - * Path to the repository's object store. > - * Cannot be NULL after initialization. > + * Holds any information related to accessing the raw object content. >*/ > - char *objectdir; > - > - /* Path to extra alternate object database if not NULL */ > - char *alternate_db; > + struct raw_object_store objects; > > /* >* Path to the repository's graft file. > diff --git a/sha1_file.c b/sha1_file.c > index 4af422e3cf..792bb21c15 100644 > --- a/sha1_file.c > +++ b/sha1_file.c > @@ -671,7 +671,8 @@ void prepare_alt_odb(void) > return; > > alt_odb_tail = &alt_odb_list; > - link_alt_odb_entries(the_repository->alternate_db, PATH_SEP, NULL, 0); > + link_alt_odb_entries(the_repository->objects.alternate_db, > + PATH_SEP, NULL, 0); > > read_info_alternates(get_object_directory(), 0); > } > -- > 2.16.1.435.g8f24da2e1a > -- Brandon Williams

Re: [PATCH 09/44] object-store: free alt_odb_list

2018-03-21 Thread Brandon Williams
entry itself hasn't been freed (as noted above). > > Is leaking the entries themselves intentional? > > > void raw_object_store_clear(struct raw_object_store *o) > > { > > FREE_AND_NULL(o->objectdir); > > FREE_AND_NULL(o->alternate_db); > > + > > + free_alt_odbs(o); > > + o->alt_odb_tail = NULL; > > } > > The commit message talks about freeing memory and resetting > alt_odb_list and alt_odb_tail, but this code only resets alt_odb_tail. -- Brandon Williams

Re: [PATCH 10/44] object-store: move packed_git and packed_git_mru to object store

2018-03-21 Thread Brandon Williams
gt; prepare_packed_git(); > - for (p = packed_git; p; p = p->next) { > + for (p = the_repository->objects.packed_git; p; p = p->next) { > if ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local) > continue; > if ((flags & FOR_EACH_OBJECT_PROMISOR_ONLY) && > diff --git a/packfile.h b/packfile.h > index a7fca598d6..76496226bb 100644 > --- a/packfile.h > +++ b/packfile.h > @@ -38,6 +38,9 @@ extern void prepare_packed_git(void); > extern void reprepare_packed_git(void); > extern void install_packed_git(struct packed_git *pack); > > +struct packed_git *get_packed_git(struct repository *r); > +struct list_head *get_packed_git_mru(struct repository *r); > + > /* > * Give a rough count of objects in the repository. This sacrifices accuracy > * for speed. > diff --git a/server-info.c b/server-info.c > index 26a6c20b7d..6fe64ede17 100644 > --- a/server-info.c > +++ b/server-info.c > @@ -1,4 +1,5 @@ > #include "cache.h" > +#include "repository.h" > #include "refs.h" > #include "object.h" > #include "commit.h" > @@ -200,7 +201,7 @@ static void init_pack_info(const char *infofile, int > force) > objdirlen = strlen(objdir); > > prepare_packed_git(); > - for (p = packed_git; p; p = p->next) { > + for (p = get_packed_git(the_repository); p; p = p->next) { > /* we ignore things on alternate path since they are >* not available to the pullers in general. >*/ > @@ -210,7 +211,7 @@ static void init_pack_info(const char *infofile, int > force) > } > num_pack = i; > info = xcalloc(num_pack, sizeof(struct pack_info *)); > - for (i = 0, p = packed_git; p; p = p->next) { > + for (i = 0, p = get_packed_git(the_repository); p; p = p->next) { > if (!p->pack_local) > continue; > info[i] = xcalloc(1, sizeof(struct pack_info)); > diff --git a/sha1_name.c b/sha1_name.c > index 957ce25680..bd4d7352ce 100644 > --- a/sha1_name.c > +++ b/sha1_name.c > @@ -196,7 +196,8 @@ static void find_short_packed_object(struct > disambiguate_state *ds) > struct packed_git *p; > > prepare_packed_git(); > - for (p = packed_git; p && !ds->ambiguous; p = p->next) > + for (p = get_packed_git(the_repository); p && !ds->ambiguous; > + p = p->next) > unique_in_pack(p, ds); > } > > @@ -566,7 +567,7 @@ static void find_abbrev_len_packed(struct min_abbrev_data > *mad) > struct packed_git *p; > > prepare_packed_git(); > - for (p = packed_git; p; p = p->next) > + for (p = get_packed_git(the_repository); p; p = p->next) > find_abbrev_len_for_pack(p, mad); > } > > -- > 2.16.1.435.g8f24da2e1a > -- Brandon Williams

Re: [PATCH 11/44] object-store: close all packs upon clearing the object store

2018-03-21 Thread Brandon Williams
;do-not-close'"); > else > diff --git a/packfile.h b/packfile.h > index 76496226bb..5b1ce00f84 100644 > --- a/packfile.h > +++ b/packfile.h > @@ -66,7 +66,7 @@ extern void close_pack_index(struct packed_git *); > > extern unsigned char *use_pack(struct packed_git *, struct pack_window **, > off_t, unsigned long *); > extern void close_pack_windows(struct packed_git *); > -extern void close_all_packs(void); > +extern void close_all_packs(struct raw_object_store *o); > extern void unuse_pack(struct pack_window **); > extern void clear_delta_base_cache(void); > extern struct packed_git *add_packed_git(const char *path, size_t path_len, > int local); > -- > 2.16.1.435.g8f24da2e1a > -- Brandon Williams

Re: [PATCH 12/44] pack: move prepare_packed_git_run_once to object store

2018-03-21 Thread Brandon Williams
; { > approximate_object_count_valid = 0; > - prepare_packed_git_run_once = 0; > + the_repository->objects.packed_git_initialized = 0; > prepare_packed_git(); > } > > -- > 2.16.1.435.g8f24da2e1a > -- Brandon Williams

Re: [PATCH 13/44] pack: move approximate object count to object store

2018-03-21 Thread Brandon Williams
but where, how, when does > > 'approximate_object_count_valid' ever get set to anything other than > > 0? Even in the existing code (without this patch), there doesn't seem > > to be anyplace which sets this to a non-zero value. Am I missing > > something obvious (or subtle)? > > Probably related to this > https://public-inbox.org/git/20180226085508.ga30...@sigill.intra.peff.net/#t Yeah as far as doing a conversion this should be fine, we can come by and clean it up at a later point. -- Brandon Williams

Re: [PATCH 19/44] sha1_file: allow link_alt_odb_entries to handle arbitrary repositories

2018-03-21 Thread Brandon Williams
mp;objdirbuf); > } > > -static void read_info_alternates_the_repository(const char *relative_base, > - int depth) > +static void read_info_alternates(struct repository *r, > + const char *relative_base, > + int depth) > { > char *path; > struct strbuf buf = STRBUF_INIT; > @@ -516,7 +512,7 @@ static void read_info_alternates_the_repository(const > char *relative_base, > return; > } > > - link_alt_odb_entries(the_repository, buf.buf, '\n', relative_base, > depth); > + link_alt_odb_entries(r, buf.buf, '\n', relative_base, depth); > strbuf_release(&buf); > free(path); > } > -- > 2.16.1.435.g8f24da2e1a > -- Brandon Williams

Re: [PATCH 32/44] sha1_file: allow sha1_loose_object_info to handle arbitrary repositories

2018-03-21 Thread Brandon Williams
*oi->disk_sizep = st.st_size; > return 0; > } > > - map = map_sha1_file(the_repository, sha1, &mapsize); > + map = map_sha1_file(r, sha1, &mapsize); > if (!map) > return -1; > > -- > 2.16.1.435.g8f24da2e1a > -- Brandon Williams

Re: [PATCH 40/44] packfile: allow prepare_packed_git to handle arbitrary repositories

2018-03-21 Thread Brandon Williams
pare_packed_git_##r() > -extern void prepare_packed_git_the_repository(void); > +extern void prepare_packed_git(struct repository *r); > #define reprepare_packed_git(r) reprepare_packed_git_##r() > extern void reprepare_packed_git_the_repository(void); > extern void install_packed_git(struct repository *r, struct packed_git > *pack); > -- > 2.16.1.435.g8f24da2e1a > -- Brandon Williams

Re: [PATCH 44/44] packfile: keep prepare_packed_git() private

2018-03-21 Thread Brandon Williams
packed_git(the_repository); p; p = p->next) { > /* we ignore things on alternate path since they are >* not available to the pullers in general. > diff --git a/sha1_name.c b/sha1_name.c > index cb42e398b2..f25a0970ca 100644 > --- a/sha1_name.c > +++ b/sha1_name.c > @@ -195,7 +195,6 @@ static void find_short_packed_object(struct > disambiguate_state *ds) > { > struct packed_git *p; > > - prepare_packed_git(the_repository); > for (p = get_packed_git(the_repository); p && !ds->ambiguous; >p = p->next) > unique_in_pack(p, ds); > @@ -566,7 +565,6 @@ static void find_abbrev_len_packed(struct min_abbrev_data > *mad) > { > struct packed_git *p; > > - prepare_packed_git(the_repository); > for (p = get_packed_git(the_repository); p; p = p->next) > find_abbrev_len_for_pack(p, mad); > } > -- > 2.16.1.435.g8f24da2e1a > -- Brandon Williams

Re: [PATCH 00/44] reroll nd/remove-ignore-env.. sb/object-store and sb/packfiles..

2018-03-21 Thread Brandon Williams
_packed_git_mru() introduced > earlier will call prepare_packed_git() automatically. > > The whole thing is also available at > > https://github.com/pclouds/git/tree/ignore-env-object-store-packfiles > > And interdiff of all three, compared to what is currently in 'pu'. > Looks pretty good in my opinon: The only major issue I could find is with patch 40 and must be fixed before this can be merged. The rest of the series looks good. -- Brandon Williams

Re: [PATCH 10/44] object-store: move packed_git and packed_git_mru to object store

2018-03-23 Thread Brandon Williams
On 03/23, Duy Nguyen wrote: > On Fri, Mar 23, 2018 at 6:03 PM, Duy Nguyen wrote: > > On Wed, Mar 21, 2018 at 11:18 PM, Brandon Williams > > wrote: > >> You're marking packed_git > >> as "private"...well C has no notion of private vs public fie

Re: [PATCH 00/27] sb/object-store updates

2018-03-23 Thread Brandon Williams
big due to the "objects." to "objects->" conversion > (blame Brandon for his suggestion, don't blame me :D) Haha, I'm guessing you prefer having a pointer too then? :P The interdiff looks good, though I'll set some time aside today to go through the series as a whole again. -- Brandon Williams

Re: [PATCH 04/27] object-store: free alt_odb_list

2018-03-23 Thread Brandon Williams
_list = next; > + } > +} > + > void raw_object_store_clear(struct raw_object_store *o) > { > FREE_AND_NULL(o->objectdir); > FREE_AND_NULL(o->alternate_db); > + > + free_alt_odbs(o); > + o->alt_odb_tail = NULL; > } > -- > 2.17.0.rc0.348.gd5a49e0b6f > -- Brandon Williams

Re: [PATCH 00/27] sb/object-store updates

2018-03-23 Thread Brandon Williams
On 03/23, Brandon Williams wrote: > On 03/23, Nguyễn Thái Ngọc Duy wrote: > > I think I have addressed all comments I've received so far. What I > > decided not to do, I have responded individually. One comment I did > > not respond nor do is the approximate thing, whi

Re: [PATCH 00/12] sb/packfiles-in-repository updates

2018-03-23 Thread Brandon Williams
ttp-backend.c | 1 - > http.c | 2 +- > pack-bitmap.c| 1 - > packfile.c | 76 +++- > packfile.h | 11 +++--- > server-info.c | 1 - > sha1_file.c | 8 ++--- > sha1_name.c | 2 -- > 17 files changed, 58 insertions(+), 67 deletions(-) > > -- > 2.17.0.rc0.348.gd5a49e0b6f > -- Brandon Williams

Re: Git Merge contributor summit notes

2018-03-26 Thread Brandon Williams
y fetching the object or asking an object-size-server for the > data. > > All of these cases could be eliminated if the type/size were available > in the OID. > > Just a thought. While we are converting to a new hash it seems like > this would be a good time to at least discuss it. Echoing what Stefan said. I don't think its a good idea to embed this sort of data into the OID. There are a lot of reasons but one of them being that would block having access to this data behind completing the hash transition (which could very well still be years away from completing). I think that a much better approach would be to create a meta-data data structure, much like the commit graph that stolee has been working on) which can store this data along side the objects (but not in the packfiles themselves). It could be a stacking structure which is periodically coalesced and we could add in a wire feature to fetch this meta data from the server upon fetching objects. -- Brandon Williams

Re: [PATCH v6 07/35] connect: convert get_remote_heads to use struct packet_reader

2018-03-27 Thread Brandon Williams
On 03/27, Duy Nguyen wrote: > On Tue, Mar 27, 2018 at 6:25 PM, Duy Nguyen wrote: > > On Tue, Mar 27, 2018 at 6:11 PM, Jeff King wrote: > >> On Tue, Mar 27, 2018 at 05:27:14PM +0200, Duy Nguyen wrote: > >> > >>> On Thu, Mar 15, 2018 at 10:31:14AM -0700, Bran

Re: [PATCH 2/5] submodule-config: allow submodule_free to handle arbitrary repositories

2018-03-27 Thread Brandon Williams
struct index_state > *index, > if (!state && ce->ce_flags & CE_WT_REMOVE) { > repo_read_gitmodules(the_repository); > } else if (state && (ce->ce_flags & CE_UPDATE)) { > - submodule_free(); > + submodule_free(the_repository); > checkout_entry(ce, state, NULL); > repo_read_gitmodules(the_repository); > } > -- > 2.17.0.rc1.321.gba9d0f2565-goog > -- Brandon Williams

Re: [PATCH 5/5] submodule: fixup nested submodules after moving the submodule

2018-03-27 Thread Brandon Williams
har > *prefix, > if (!sub) > die(_("could not lookup name for submodule '%s'"), > path); > connect_work_tree_and_git_dir(path, > - git_path("modules/%s", sub->name)); > + git_path("modules/%s", sub->name), 0); > } else { > /* Is it already absorbed into the superprojects git dir? */ > char *real_sub_git_dir = real_pathdup(sub_git_dir, 1); > diff --git a/t/t7001-mv.sh b/t/t7001-mv.sh > index d4e6485a26..ff70244620 100755 > --- a/t/t7001-mv.sh > +++ b/t/t7001-mv.sh > @@ -497,7 +497,7 @@ test_expect_success 'moving a submodule in nested > directories' ' > test_cmp expect actual > ' > > -test_expect_failure 'moving nested submodules' ' > +test_expect_success 'moving nested submodules' ' > git commit -am "cleanup commit" && > mkdir sub_nested_nested && > (cd sub_nested_nested && > -- > 2.17.0.rc1.321.gba9d0f2565-goog > -- Brandon Williams

Re: [PATCH 5/6] submodule: fixup nested submodules after moving the submodule

2018-03-28 Thread Brandon Williams
gitdir); > + connect_work_tree_and_git_dir(path, gitdir, 1); > free(gitdir); > } > } > @@ -1947,7 +1947,7 @@ void absorb_git_dir_into_superproject(const char > *prefix, > if (!sub) > die(_("could not lookup name for submodule '%s'"), > path); > connect_work_tree_and_git_dir(path, > - git_path("modules/%s", sub->name)); > + git_path("modules/%s", sub->name), 0); > } else { > /* Is it already absorbed into the superprojects git dir? */ > char *real_sub_git_dir = real_pathdup(sub_git_dir, 1); > diff --git a/t/t7001-mv.sh b/t/t7001-mv.sh > index d4e6485a26..ff70244620 100755 > --- a/t/t7001-mv.sh > +++ b/t/t7001-mv.sh > @@ -497,7 +497,7 @@ test_expect_success 'moving a submodule in nested > directories' ' > test_cmp expect actual > ' > > -test_expect_failure 'moving nested submodules' ' > +test_expect_success 'moving nested submodules' ' > git commit -am "cleanup commit" && > mkdir sub_nested_nested && > (cd sub_nested_nested && > -- > 2.17.0.rc1.321.gba9d0f2565-goog > -- Brandon Williams

Re: What's cooking in git.git (Mar 2018, #05; Wed, 28)

2018-03-28 Thread Brandon Williams
e next-gen transfer protocol. > > Is everybody happy with this version? One design decision with > larger consequence "to or not to build in?" has been settled in > favor of status quo, IIRC. I haven't heard anymore complaints with this version so I *think* it should be good. -- Brandon Williams

Re: [PATCH] {fetch,upload}-pack: clearly mark unreachable v2 code

2018-03-30 Thread Brandon Williams
create_pack_file(); > state = FETCH_DONE; > break; > - case FETCH_DONE: > - continue; > + default: > + BUG("Added a new fetch_state without updating switch"); > } > } > > -- > 2.16.2.804.g6dcf76e118 > -- Brandon Williams

[PATCH] commit: allow partial commits with relative paths

2018-04-03 Thread Brandon Williams
'pathspec_prefix' removed and the 'list_paths' function wasn't properly adjusted to cope with the change, resulting in over-eager pruning of the tree that is overlayed on the index. This fixes the regression and adds a regression test so this can be prevented in the future

Re: [PATCH 0/6] Compute and consume generation numbers

2018-04-03 Thread Brandon Williams
++ > commit.c | 53 > commit.h | 7 +++- > object.c | 4 +- > 6 files changed, 104 insertions(+), 16 deletions(-) > > -- > 2.17.0.20.g9f30ba16e1 > -- Brandon Williams

Re: [PATCH 2/6] commit: add generation number to struct commmit

2018-04-03 Thread Brandon Williams
+#define GENERATION_NUMBER_UNDEF 0x > +#define GENERATION_NUMBER_NONE 0 > > struct commit_list { > struct commit *item; > @@ -24,6 +26,7 @@ struct commit { > struct commit_list *parents; > struct tree *tree; > uint32_t graph_pos; > + uint32_t generation; > }; > > extern int save_commit_buffer; > -- > 2.17.0.rc0 > -- Brandon Williams

Re: [PATCH 2/6] commit: add generation number to struct commmit

2018-04-03 Thread Brandon Williams
On 04/03, Jeff King wrote: > On Tue, Apr 03, 2018 at 11:05:36AM -0700, Brandon Williams wrote: > > > On 04/03, Derrick Stolee wrote: > > > The generation number of a commit is defined recursively as follows: > > > > > > * If a commit A has no parents,

Re: [PATCH 8/6] commit: use generation numbers for in_merge_bases()

2018-04-04 Thread Brandon Williams
get commits. > > > > Performance tests were run on a copy of the Linux repository where > > HEAD is contained in v4.13 but no earlier tag. Also, all tags were > > copied to branches and 'git branch --contains' was tested: > > > > Before: 60.0s > > After: 0.4s > > Rel %: -99.3% Now that is an impressive speedup. -- Brandon Williams

Re: [PATCH 12/16] refs: store the main ref store inside the repository struct

2018-04-09 Thread Brandon Williams
pository.h > @@ -26,6 +26,9 @@ struct repository { >*/ > struct raw_object_store *objects; > > + /* The store in which the refs are held. */ > + struct ref_store *main_ref_store; > + > /* >* Path to the repository's graft file. >* Cannot be NULL after initialization. > -- > 2.17.0.484.g0c8726318c-goog > -- Brandon Williams

Re: [PATCHv2 00/16] object-store refactoring 3 (replace objects, main ref store)

2018-04-09 Thread Brandon Williams
ory.h | 3 + > revision.c| 5 +- > sha1_file.c | 7 ++- > streaming.c | 3 +- > t/helper/test-ref-store.c | 3 +- > 18 files changed, 183 insertions(+), 195 deletions(-) > create mode 100644 replace-object.c > create mode 100644 replace-object.h > delete mode 100644 replace_object.c > > -- > 2.17.0.484.g0c8726318c-goog > -- Brandon Williams

Re: [PATCH 12/16] refs: store the main ref store inside the repository struct

2018-04-09 Thread Brandon Williams
On 04/09, Stefan Beller wrote: > Hi Brandon, > > On Mon, Apr 9, 2018 at 4:24 PM, Brandon Williams wrote: > > >> - main_ref_store = ref_store_init(get_git_dir(), REF_STORE_ALL_CAPS); > >> - return main_ref_store; > >> + r->main_ref_store = re

Re: [PATCH 1/8] ssh test: make copy_ssh_wrapper_as clean up after itself

2017-11-20 Thread Brandon Williams
t;$TRASH_DIRECTORY/plink" && > GIT_SSH_VARIANT=plink \ > git clone "[myhost:123]:src" ssh-bracket-clone-variant-3 && > expect_ssh "-P 123" myhost src > ' > > test_expect_success 'GIT_SSH_VARIANT overrides plink to tortoiseplink' ' > + copy_ssh_wrapper_as "$TRASH_DIRECTORY/plink" && > GIT_SSH_VARIANT=tortoiseplink \ > git clone "[myhost:123]:src" ssh-bracket-clone-variant-4 && > expect_ssh "-batch -P 123" myhost src > @@ -449,9 +448,6 @@ test_expect_success 'clean failure on broken quoting' ' > git clone "[myhost:123]:src" sq-failure > ' > > -# Reset the GIT_SSH environment variable for clone tests. > -setup_ssh_wrapper > - > counter=0 > # $1 url > # $2 none|host > -- > 2.15.0.448.gf294e3d99a > -- Brandon Williams

Re: [PATCH 3/8] connect: split git:// setup into a separate function

2017-11-20 Thread Brandon Williams
equest, '\0'); > - strbuf_addf(&request, "version=%d%c", > - get_protocol_version_config(), '\0'); > - } > - > - packet_write(fd[1], request.buf, request.len); > - > - free(target_host); > - strbuf_release(&request); > + conn = git_connect_git(fd, hostandport, path, prog, flags); > } else { > struct strbuf cmd = STRBUF_INIT; > const char *const *var; > -- > 2.15.0.448.gf294e3d99a > -- Brandon Williams

Re: [PATCH 4/8] connect: split ssh command line options into separate function

2017-11-20 Thread Brandon Williams
iant != VARIANT_SIMPLE) { > - if (flags & CONNECT_IPV4) > - argv_array_push(&conn->args, "-4"); > - else if (flags & CONNECT_IPV6) > - argv_array_push(&conn->args, "-6"); >

Re: [PATCH 6/8] ssh: 'auto' variant to select between 'ssh' and 'simple'

2017-11-20 Thread Brandon Williams
ed as simple' ' > copy_ssh_wrapper_as "$TRASH_DIRECTORY/simple" && > git clone -4 "[myhost:123]:src" ssh-bracket-clone-simple && > @@ -381,6 +387,21 @@ test_expect_success 'uplink is treated as simple' ' > expect_ssh myhost src > ' > > +test_expect_success 'OpenSSH-like uplink is treated as ssh' ' > + write_script "$TRASH_DIRECTORY/uplink" <<-EOF && > + if test "\$1" = "-G" > + then > + exit 0 > + fi && > + exec "\$TRASH_DIRECTORY/ssh$X" "\$@" > + EOF > + test_when_finished "rm -f \"\$TRASH_DIRECTORY/uplink\"" && > + GIT_SSH="$TRASH_DIRECTORY/uplink" && > + test_when_finished "GIT_SSH=\"\$TRASH_DIRECTORY/ssh\$X\"" && > + git clone "[myhost:123]:src" ssh-bracket-clone-sshlike-uplink && > + expect_ssh "-p 123" myhost src > +' > + > test_expect_success 'plink is treated specially (as putty)' ' > copy_ssh_wrapper_as "$TRASH_DIRECTORY/plink" && > git clone "[myhost:123]:src" ssh-bracket-clone-plink-0 && > -- > 2.15.0.448.gf294e3d99a > -- Brandon Williams

Re: [PATCH 4/8] connect: split ssh command line options into separate function

2017-11-20 Thread Brandon Williams
On 11/20, Jonathan Nieder wrote: > Hi, > > Brandon Williams wrote: > > On 11/20, Jonathan Nieder wrote: > [long stream of quoted context snipped; please cut down the quoted > text to what you are replying to in the future] > >> @@ -972,16 +1031,13 @@ struct child

Re: [PATCH 3/8] connect: split git:// setup into a separate function

2017-11-20 Thread Brandon Williams
On 11/20, Jonathan Nieder wrote: > Brandon Williams wrote: > > On 11/20, Jonathan Nieder wrote: > > >> + /* These underlying connection commands die() if they > >> + * cannot connect. > >> + */ > > > > I know this is really just code motio

Re: [PATCH v3 0/8] Coping with unrecognized ssh wrapper scripts in GIT_SSH

2017-11-20 Thread Brandon Williams
> https://public-inbox.org/git/20171023231625.6mhcyqti7vdg6...@aiede.mtv.corp.google.com/ > [2] > https://public-inbox.org/git/20171115202516.hduhzsgeoff5a...@aiede.mtv.corp.google.com/ > [3] https://public-inbox.org/git/xmqq60b59toe@gitster.mtv.corp.google.com/ -- Brandon Williams

Re: Bug in pathspec handling (in conjunction with submodules)

2017-11-28 Thread Brandon Williams
blem, of course, is that `git rev-list HEAD -- '[bracket]'` never > recurses into the submodule. And therefore, the promised "more accurate > matching [...] in the submodule" is never performed. And the commit adding > the submodule is never pruned. > > Since I am not really familiar with all that tree diff code (and as a > general rule to protect my mental health, I try my best to stay away from > submodules, too), but you two are, may I ask you gentle people to have a > closer look to fix those bugs? > > Thanks, > Dscho -- Brandon Williams

[PATCH] pathspec: only match across submodule boundaries when requested

2017-11-28 Thread Brandon Williams
pec' and using this flag to determine if matches should be allowed to cross submodule boundaries. Signed-off-by: Brandon Williams --- builtin/grep.c| 1 + pathspec.h| 1 + t/t4208-log-magic-pathspec.sh | 17 + tree-walk.c

Re: git-clone ignores submodule.recurse configuration

2017-12-01 Thread Brandon Williams
ot work with clone because a recursive clone ends up pulling data from other sources aside from the URL the user explicitly provides. Historically there have been a number of security related bugs with respect to cloning submodules so we felt it was best to require a recursive clone to be requested explicitly, at least for the time being. -- Brandon Williams

Re: gitattributes not read for diff-tree anymore in 2.15?

2017-12-04 Thread Brandon Williams
On 12/04, Ben Boeckel wrote: > Hi, > > I've bisected a failure in our test suite to this commit: > > commit 557a5998df19faf8641acfc5b6b1c3c2ba64dca9 (HEAD, refs/bisect/bad) > Author: Brandon Williams > Date: Thu Aug 3 11:20:00 2017 -0700 >

[WIP 00/15] protocol version 2

2017-12-04 Thread Brandon Williams
deepening as a separate server command or to keep it in fetch like it is in v1, just something to think about. Any comments or criticism is welcome. Thanks! Brandon Williams (15): pkt-line: introduce packet_read_with_status pkt-line: introduce struct packet_reader pkt-line: add delim packet s

[WIP 04/15] upload-pack: convert to a builtin

2017-12-04 Thread Brandon Williams
In order to allow for code sharing with the server-side of fetch in protocol-v2 convert upload-pack to be a builtin. Signed-off-by: Brandon Williams --- Makefile | 3 ++- builtin.h | 1 + git.c | 1 + upload-pack.c | 2 +- 4 files changed, 5 insertions(+), 2 deletions(-) diff

[WIP 02/15] pkt-line: introduce struct packet_reader

2017-12-04 Thread Brandon Williams
enables a caller to be able to peek a single line at a time using 'packet_reader_peek()' and having a caller consume a line by calling 'packet_reader_read()'. Signed-off-by: Brandon Williams --- pkt-line.c | 61

[WIP 11/15] serve: introduce git-serve

2017-12-04 Thread Brandon Williams
request that another command be executed or can terminate the connection by sending a flush packet. Signed-off-by: Brandon Williams --- .gitignore | 1 + Makefile| 2 + builtin.h | 1 + builtin/serve.c | 25 git.c | 1 + s

[WIP 13/15] connect: request remote refs using v2

2017-12-04 Thread Brandon Williams
Teach the client to be able to request a remote's refs using protocol v2. This is done by having a client issue a 'ls-refs' request to a v2 server. Signed-off-by: Brandon Williams --- connect.c | 85 +- remote.h

[WIP 07/15] connect: convert get_remote_heads to use struct packet_reader

2017-12-04 Thread Brandon Williams
st line without consuming the first ref sent in the protocol_v0 case so that the protocol version the server is speaking can be determined outside of 'get_remote_heads()' in a future patch. Signed-off-by: Brandon Williams --- connect.c | 125 +++

[WIP 08/15] connect: discover protocol version outside of get_remote_heads

2017-12-04 Thread Brandon Williams
ned-off-by: Brandon Williams --- builtin/fetch-pack.c | 14 +- builtin/send-pack.c | 15 +-- connect.c| 13 - connect.h| 3 +++ remote-curl.c| 18 -- remote.h | 5 +++-- transport.c

[WIP 03/15] pkt-line: add delim packet support

2017-12-04 Thread Brandon Williams
eserved to indicate the end of a response. Signed-off-by: Brandon Williams --- pkt-line.c | 19 ++- pkt-line.h | 3 +++ 2 files changed, 21 insertions(+), 1 deletion(-) diff --git a/pkt-line.c b/pkt-line.c index 518109bbe..222e1e310 100644 --- a/pkt-line.c +++ b/pkt-line.c @@ -9

[WIP 01/15] pkt-line: introduce packet_read_with_status

2017-12-04 Thread Brandon Williams
_status' type. This allows for easily identifying between special and normal packets as well as errors. It also enables easily adding a new special packet in the future. Signed-off-by: Brandon Williams --- pkt-line.c | 55 ++- pkt-line

[WIP 09/15] transport: store protocol version

2017-12-04 Thread Brandon Williams
ned-off-by: Brandon Williams --- transport.c | 35 ++- 1 file changed, 26 insertions(+), 9 deletions(-) diff --git a/transport.c b/transport.c index 28b744ec2..4160c4167 100644 --- a/transport.c +++ b/transport.c @@ -118,6 +118,7 @@ struct git_transport_data {

[WIP 06/15] transport: use get_refs_via_connect to get refs

2017-12-04 Thread Brandon Williams
Remove code duplication and use the existing 'get_refs_via_connect()' function to retrieve a remote's heads in 'fetch_refs_via_pack()' and 'git_transport_push()'. Signed-off-by: Brandon Williams --- transport.c | 18 -- 1 file changed, 4 i

[WIP 15/15] fetch-pack: perform a fetch using v2

2017-12-04 Thread Brandon Williams
When communicating with a v2 server, perform a fetch by requesting the 'fetch' command. Signed-off-by: Brandon Williams --- builtin/fetch-pack.c | 2 +- fetch-pack.c | 237 - fetch-pack.h | 4 +- t/t5701-prot

[WIP 14/15] upload_pack: introduce fetch server command

2017-12-04 Thread Brandon Williams
Introduce the 'fetch' server command. Signed-off-by: Brandon Williams --- serve.c | 2 + upload-pack.c | 223 ++ upload-pack.h | 9 +++ 3 files changed, 234 insertions(+) create mode 100644 upload-pack.h diff --git a

[WIP 12/15] ls-refs: introduce ls-refs server command

2017-12-04 Thread Brandon Williams
based on provided ref-patterns. Signed-off-by: Brandon Williams --- Makefile | 1 + ls-refs.c | 96 +++ ls-refs.h | 9 ++ serve.c | 8 ++ 4 files changed, 114 insertions(+) create mode 100644 ls-refs.c create mode 100644

[WIP 05/15] upload-pack: factor out processing lines

2017-12-04 Thread Brandon Williams
Factor out the logic for processing shallow, deepen, deepen_since, and deepen_not lines into their own functions to simplify the 'receive_needs()' function in addition to making it easier to reuse some of this logic when implementing protocol_v2. Signed-off-by: Brandon Williams -

[WIP 10/15] protocol: introduce enum protocol_version value protocol_v2

2017-12-04 Thread Brandon Williams
Introduce protocol_v2, a new value for 'enum protocol_version'. Subsequent patches will fill in the implementation of protocol_v2. Signed-off-by: Brandon Williams --- builtin/fetch-pack.c | 3 +++ builtin/receive-pack.c | 3 +++ builtin/send-pack.c| 3 +++ connect.c

[PATCH v2] pathspec: only match across submodule boundaries when requested

2017-12-04 Thread Brandon Williams
pec' and using this flag to determine if matches should be allowed to cross submodule boundaries. This fixes https://github.com/git-for-windows/git/issues/1371. Signed-off-by: Brandon Williams --- builtin/grep.c| 1 + pathspec.h| 1 + t/t4208-log-mag

Re: [PATCH] pathspec: only match across submodule boundaries when requested

2017-12-04 Thread Brandon Williams
On 11/29, Johannes Schindelin wrote: > Hi Brandon, > > On Tue, 28 Nov 2017, Brandon Williams wrote: > > > Commit 74ed43711fd (grep: enable recurse-submodules to work on > > objects, 2016-12-16) taught 'tree_entry_interesting()' to be able to > > match acr

Re: gitattributes not read for diff-tree anymore in 2.15?

2017-12-05 Thread Brandon Williams
On 12/05, Ben Boeckel wrote: > On Mon, Dec 04, 2017 at 15:03:55 -0800, Brandon Williams wrote: > > Reading the attributes files should be done regardless if the gitmodules > > file is read. The gitmodules file should only come into play if you are > > dealing with subm

[PATCH] diff-tree: read the index so attribute checks work in bare repositories

2017-12-05 Thread Brandon Williams
ly reading the index during the setup of diff-tree. Reported-by: Ben Boeckel Signed-off-by: Brandon Williams --- This patch should fix the regression. Let me know if it doesn't solve the issue and I'll investigate some more. builtin/diff-tree.c| 1 + t/t4015-diff-whitespa

Re: [PATCH] diff-tree: read the index so attribute checks work in bare repositories

2017-12-05 Thread Brandon Williams
On 12/05, Ben Boeckel wrote: > On Tue, Dec 05, 2017 at 14:13:37 -0800, Brandon Williams wrote: > > This patch should fix the regression. Let me know if it doesn't solve the > > issue and I'll investigate some more. > > Our test suite passes again. Thanks

Re: [RFC] 'unsigned long' to 'size_t' conversion

2017-12-06 Thread Brandon Williams
the unsigned char[20] -> struct object_id conversion I would think that the best way to go about such a conversion would be to do it in small chunks as you've mentioned. That way you are only causing churn in hopefully small parts of the code base at a time instead of one monolithic change that is sure to cause conflicts. -- Brandon Williams

Re: [PATCH] diff-tree: read the index so attribute checks work in bare repositories

2017-12-06 Thread Brandon Williams
On 12/05, Stefan Beller wrote: > On Tue, Dec 5, 2017 at 2:13 PM, Brandon Williams wrote: > > A regression was introduced in 557a5998d (submodule: remove > > gitmodules_config, 2017-08-03) to how attribute processing was handled > > in bare repositories when running

Re: [PATCH] diff-tree: read the index so attribute checks work in bare repositories

2017-12-06 Thread Brandon Williams
On 12/05, Eric Sunshine wrote: > On Tue, Dec 5, 2017 at 5:13 PM, Brandon Williams wrote: > > A regression was introduced in 557a5998d (submodule: remove > > gitmodules_config, 2017-08-03) to how attribute processing was handled > > in bare repositories when running

<    4   5   6   7   8   9   10   11   12   13   >