Re: Resumable clone

2016-03-08 Thread Kevin Wern
Junio, > Yes, that is very close to what I said in the "what remains?" > section, but with a crucial difference in a detail. Perhaps reading > the message you are respoinding to again more carefully will clear > the confusion. This is what we want to allow the server to say > (from the message

Re: Resumable clone

2016-03-07 Thread Kevin Wern
Hey Junio and Duy, Thank you for your thorough responses! I'm new to git dev, so it's extremely helpful. > - The server side endpoint does not have to be, and I think it > should not be, implemented as an extension to the current > upload-pack protocol. It is perfectly fine to add a new "git >

Resumable clone

2016-03-05 Thread Kevin Wern
Hey, all, A while ago, I noticed that if a clone is interrupted, it has to be restarted completely. When I looked up the issue, I saw that someone suggested resumable clones as a feature as early as 2007. It doesn't seem like any progress has been made on this issue, so I began looking at

[PATCH 1/2] Resumable clone: create git-prime-clone (Draft)

2016-05-18 Thread Kevin Wern
Create a bare-bones version of git-prime-clone, which returns the location of an alternate resource specified by the server that the client should fetch and build before returning to perform an incremental fetch. At this point, no validation is performed of the file's existence, the file's

[PATCH 2/2] Resumable clone: add endpoints for prime clone (Draft)

2016-05-18 Thread Kevin Wern
Add endpoints for prime-clone in http service and git daemon --- daemon.c | 7 +++ http-backend.c | 1 + 2 files changed, 8 insertions(+) diff --git a/daemon.c b/daemon.c index 8d45c33..2ddc7f7 100644 --- a/daemon.c +++ b/daemon.c @@ -475,10 +475,17 @@ static int receive_pack(void)

[PATCH 0/2] git-prime-clone

2016-05-18 Thread Kevin Wern
considerations I missed, and I want to keep this as easily extensible to the other types as possible. Apologies for not keeping everyone in the loop. I should be working on this more actively from this point onwards. -Kevin Wern -- [PATCH 1/2] Resumable clone: create git-prime-clone (Draft

[PATCH 10/11] run command: add RUN_COMMAND_NO_STDOUT

2016-09-15 Thread Kevin Wern
Add option RUN_COMMAND_NO_STDOUT, which sets no_stdout on a child process. This will be used by git clone when calling index-pack on a downloaded packfile. Signed-off-by: Kevin Wern <kevin.m.w...@gmail.com> --- run-command.c | 1 + run-command.h | 1 + 2 files changed, 2 insertions(+)

[PATCH 06/11] Resumable clone: implement transport_prime_clone

2016-09-15 Thread Kevin Wern
verbosely. Therefore, all error parsing is done with remote-curl, and any protocol breach between remote-curl and transport-helper will treated as a bug and result in death. Signed-off-by: Kevin Wern <kevin.m.w...@gmail.com> --- transport-helper.

[PATCH 07/11] Resumable clone: add resumable download to http/curl

2016-09-15 Thread Kevin Wern
will be downloaded to. The url passed to remote-curl on invocation does not matter (git clone will use the resource url again here). Signed-off-by: Kevin Wern <kevin.m.w...@gmail.com> --- http.c| 86 ++- http.h| 7 - remote-curl.

[PATCH 11/11] Resumable clone: implement primer logic in git-clone

2016-09-15 Thread Kevin Wern
leaving a resumable directory on exit--the logic for when to do these still needs more work. Signed-off-by: Kevin Wern <kevin.m.w...@gmail.com> --- Documentation/git-clone.txt | 16 ++ builtin/clone.c | 590 +--- t/t9904-git-prime-clone.s

[PATCH 09/11] path: add resumable marker

2016-09-15 Thread Kevin Wern
Create function to get gitdir file RESUMABLE. Signed-off-by: Kevin Wern <kevin.m.w...@gmail.com> --- cache.h | 1 + path.c | 1 + 2 files changed, 2 insertions(+) diff --git a/cache.h b/cache.h index d7ff46e..1f4117c 100644 --- a/cache.h +++ b/cache.h @@ -811,6 +811,7 @@ cons

[PATCH 05/11] Resumable clone: add output parsing to connect.c

2016-09-15 Thread Kevin Wern
Add method for transport to call when parsing primeclone output from stdin. Suppress stderr when using git_connect with ssh, unless output is verbose. Signed-off-by: Kevin Wern <kevin.m.w...@gmail.com> --- connect.c | 47 +++ connect.

[PATCH 08/11] Resumable clone: create transport_download_primer

2016-09-15 Thread Kevin Wern
Create function transport_download_primer and components to invoke and pass commands to remote-curl. Signed-off-by: Kevin Wern <kevin.m.w...@gmail.com> --- transport-helper.c | 24 transport.c| 9 + transport.h| 7 +++ 3 files chang

[PATCH 01/11] Resumable clone: create service git-prime-clone

2016-09-15 Thread Kevin Wern
to BINDIR_PROGRAMS_NEED_X, in addition to the usual builtin places. Add git-prime-clone executable to gitignore, as well Signed-off-by: Kevin Wern <kevin.m.w...@gmail.com> --- .gitignore| 1 + Documentation/git-prime-clone.txt | 39 Ma

[PATCH 02/11] Resumable clone: add prime-clone endpoints

2016-09-15 Thread Kevin Wern
Add logic to serve git-prime-clone to git and http clients. Do not pass --stateless-rpc and --advertise-refs options to prime-clone. It is inherently stateless and an 'advertisement'. Signed-off-by: Kevin Wern <kevin.m.w...@gmail.com> --- Documentation/git-daemon.txt

[PATCH 03/11] pkt-line: create gentle packet_read_line functions

2016-09-15 Thread Kevin Wern
Create a functions that can read malformed messages without dying. Includes creation of flag PACKET_READ_GENTLE_ALL. For use handling prime-clone (or other server error) responses. Signed-off-by: Kevin Wern <kevin.m.w...@gmail.com> --- pkt-line.

[PATCH 04/11] Resumable clone: add prime-clone to remote-curl

2016-09-15 Thread Kevin Wern
in this case. Signed-off-by: Kevin Wern <kevin.m.w...@gmail.com> --- remote-curl.c | 165 ++ 1 file changed, 121 insertions(+), 44 deletions(-) diff --git a/remote-curl.c b/remote-curl.c index 15e48e2..8ebb587 100644 --- a/remote-curl.c +++ b/

[PATCH 00/11] Resumable clone

2016-09-15 Thread Kevin Wern
trumps completeness. Hopefully, the bulk of the 'learning and re-doing' is done and I can update more frequently in smaller increments. I will probably work on the git-daemon download service, the curl timeout issue, and supporting other filetypes next. Feedback is appreciated. Kevin Wern

Re: [PATCH 10/11] run command: add RUN_COMMAND_NO_STDOUT

2016-09-28 Thread Kevin Wern
On Wed, Sep 28, 2016 at 10:54:52AM -0700, Junio C Hamano wrote: > > I just got an impression that you were apologetic for having to add > this option that is otherwise useless and tried to suggest a simpler > solution that does not involve such an addition. Sorry, to be clear, I meant I was ok

Re: [PATCH 02/11] Resumable clone: add prime-clone endpoints

2016-09-27 Thread Kevin Wern
On Mon, Sep 19, 2016 at 08:15:00PM +0700, Duy Nguyen wrote: > We also have an exception for select_getanyfile() below. I think it's > time we add a function callback in struct rpc_service to run each > service the way they want. Then prime-clone won't need an exception > (neither does

Re: [PATCH 03/11] pkt-line: create gentle packet_read_line functions

2016-09-27 Thread Kevin Wern
On Fri, Sep 16, 2016 at 03:17:28PM -0700, Junio C Hamano wrote: > Kevin Wern <kevin.m.w...@gmail.com> writes: > > > /* And complain if we didn't get enough bytes to satisfy the read. */ > > if (ret < size) { > > - if (op

Re: [PATCH 11/11] Resumable clone: implement primer logic in git-clone

2016-09-27 Thread Kevin Wern
On Mon, Sep 19, 2016 at 09:04:40PM +0700, Duy Nguyen wrote: > On Fri, Sep 16, 2016 at 7:12 AM, Kevin Wern <kevin.m.w...@gmail.com> wrote: > > builtin/clone.c | 590 > > +--- > > Argh.. this is too big for my b

Re: [PATCH 11/11] Resumable clone: implement primer logic in git-clone

2016-09-27 Thread Kevin Wern
On Fri, Sep 16, 2016 at 04:32:05PM -0700, Junio C Hamano wrote: > > --- > > @@ -172,6 +173,12 @@ objects from the source repository into a pack in the > > cloned repository. > > via ssh, this specifies a non-default path for the command > > run on the other end. > > > >

Re: [PATCH 07/11] Resumable clone: add resumable download to http/curl

2016-09-28 Thread Kevin Wern
On Fri, Sep 16, 2016 at 03:45:44PM -0700, Junio C Hamano wrote: > > @@ -1136,7 +1138,10 @@ static int handle_curl_result(struct slot_results > > *results) > > curl_easy_strerror(results->curl_result), > > sizeof(curl_errorstr)); > > #endif

Re: [PATCH 01/11] Resumable clone: create service git-prime-clone

2016-09-27 Thread Kevin Wern
On Fri, Sep 16, 2016 at 01:53:15PM -0700, Junio C Hamano wrote: > Kevin Wern <kevin.m.w...@gmail.com> writes: > > > Create git-prime-clone, a program to be executed on the server that > > returns the location and type of static resource to download before > >

Re: [PATCH 10/11] run command: add RUN_COMMAND_NO_STDOUT

2016-09-27 Thread Kevin Wern
On Fri, Sep 16, 2016 at 04:07:00PM -0700, Junio C Hamano wrote: > Kevin Wern <kevin.m.w...@gmail.com> writes: > > > Add option RUN_COMMAND_NO_STDOUT, which sets no_stdout on a child > > process. > > > > This will be used by git clone when calling ind

Re: [PATCH 04/11] Resumable clone: add prime-clone to remote-curl

2016-09-28 Thread Kevin Wern
On Mon, Sep 19, 2016 at 08:52:34PM +0700, Duy Nguyen wrote: > > A brief overview for this service in > Documentation/technical/http-protocol.txt (and maybe > Documentation/gitremote-helpers.txt as well) would be great help. It's > a bit hard to follow because at this point I don't know anything