Re: [PATCH v8 6/6] convert: add "status=delayed" to filter process protocol

[Lars Schneider]

> On 30 Jun 2017, at 20:19, Junio C Hamano  wrote:
> 
> Lars Schneider  writes:
> 
>>> On 29 Jun 2017, at 20:34, Junio C Hamano  wrote:
>>> 
>>> You seem to have squashed an unrelated "table-driven" clean-up into
>>> this step.  While I think the end result looks good, I would have
>>> liked to see it as a separate step, either as a preparatory "now we
>>> are going to add the third one, let's make it table-driven before
>>> doing so" step, or a follow-up "now we have done everything we
>>> wanted to do, let's make this clean-up at the end" step.
>> 
>> I am sorry. That was a misunderstanding - I thought you want to have 
>> this change squashed. 
>> 
>> The "preparatory" patch sounds good! 
>> Would you be OK if I send a v9 with this change?
> 
> That would be more than OK.  I think we are pretty much ready to hit
> 'next'.

Done: 
http://public-inbox.org/git/20170630204128.48708-1-larsxschnei...@gmail.com/

My brain is fried already today ... I hope the commit message is OK :)

Thanks,
Lars

Re: [PATCH v8 6/6] convert: add "status=delayed" to filter process protocol

[Junio C Hamano]

Lars Schneider  writes:

>> On 29 Jun 2017, at 20:34, Junio C Hamano  wrote:
>> 
>> You seem to have squashed an unrelated "table-driven" clean-up into
>> this step.  While I think the end result looks good, I would have
>> liked to see it as a separate step, either as a preparatory "now we
>> are going to add the third one, let's make it table-driven before
>> doing so" step, or a follow-up "now we have done everything we
>> wanted to do, let's make this clean-up at the end" step.
>
> I am sorry. That was a misunderstanding - I thought you want to have 
> this change squashed. 
>
> The "preparatory" patch sounds good! 
> Would you be OK if I send a v9 with this change?

That would be more than OK.  I think we are pretty much ready to hit
'next'.

Thanks.

Re: [PATCH v8 6/6] convert: add "status=delayed" to filter process protocol

[Lars Schneider]

> On 29 Jun 2017, at 20:34, Junio C Hamano  wrote:
> 
> You seem to have squashed an unrelated "table-driven" clean-up into
> this step.  While I think the end result looks good, I would have
> liked to see it as a separate step, either as a preparatory "now we
> are going to add the third one, let's make it table-driven before
> doing so" step, or a follow-up "now we have done everything we
> wanted to do, let's make this clean-up at the end" step.

I am sorry. That was a misunderstanding - I thought you want to have 
this change squashed. 

The "preparatory" patch sounds good! 
Would you be OK if I send a v9 with this change?

Thanks,
Lars

Re: [PATCH v8 6/6] convert: add "status=delayed" to filter process protocol

[Junio C Hamano]

You seem to have squashed an unrelated "table-driven" clean-up into
this step.  While I think the end result looks good, I would have
liked to see it as a separate step, either as a preparatory "now we
are going to add the third one, let's make it table-driven before
doing so" step, or a follow-up "now we have done everything we
wanted to do, let's make this clean-up at the end" step.

Thanks.

[PATCH v8 6/6] convert: add "status=delayed" to filter process protocol

[Lars Schneider]

Some `clean` / `smudge` filters may require a significant amount of
time to process a single blob (e.g. the Git LFS smudge filter might
perform network requests). During this process the Git checkout
operation is blocked and Git needs to wait until the filter is done to
continue with the checkout.

Teach the filter process protocol, introduced in edcc8581 ("convert: add
filter..process option", 2016-10-16), to accept the status
"delayed" as response to a filter request. Upon this response Git
continues with the checkout operation. After the checkout operation Git
calls "finish_delayed_checkout" which queries the filter for remaining
blobs. If the filter is still working on the completion, then the filter
is expected to block. If the filter has completed all remaining blobs
then an empty response is expected.

Git has a multiple code paths that checkout a blob. Support delayed
checkouts only in `clone` (in unpack-trees.c) and `checkout` operations
for now. The optimization is most effective in these code paths as all
files of the tree are processed.

Signed-off-by: Lars Schneider 
---
 Documentation/gitattributes.txt |  69 +-
 builtin/checkout.c  |   3 +
 cache.h |   3 +
 convert.c   | 149 +++--
 convert.h   |  26 +
 entry.c | 132 +-
 t/t0021-conversion.sh   | 116 +++
 t/t0021/rot13-filter.pl | 204 +++-
 unpack-trees.c  |   2 +
 9 files changed, 602 insertions(+), 102 deletions(-)

diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt
index 4736483865..4049a0b9a8 100644
--- a/Documentation/gitattributes.txt
+++ b/Documentation/gitattributes.txt
@@ -425,8 +425,8 @@ packet:  git< capability=clean
 packet:  git< capability=smudge
 packet:  git< 
 
-Supported filter capabilities in version 2 are "clean" and
-"smudge".
+Supported filter capabilities in version 2 are "clean", "smudge",
+and "delay".
 
 Afterwards Git sends a list of "key=value" pairs terminated with
 a flush packet. The list will contain at least the filter command
@@ -512,12 +512,73 @@ the protocol then Git will stop the filter process and 
restart it
 with the next file that needs to be processed. Depending on the
 `filter..required` flag Git will interpret that as error.
 
-After the filter has processed a blob it is expected to wait for
-the next "key=value" list containing a command. Git will close
+After the filter has processed a command it is expected to wait for
+a "key=value" list containing the next command. Git will close
 the command pipe on exit. The filter is expected to detect EOF
 and exit gracefully on its own. Git will wait until the filter
 process has stopped.
 
+Delay
+^
+
+If the filter supports the "delay" capability, then Git can send the
+flag "can-delay" after the filter command and pathname. This flag
+denotes that the filter can delay filtering the current blob (e.g. to
+compensate network latencies) by responding with no content but with
+the status "delayed" and a flush packet.
+
+packet:  git> command=smudge
+packet:  git> pathname=path/testfile.dat
+packet:  git> can-delay=1
+packet:  git> 
+packet:  git> CONTENT
+packet:  git> 
+packet:  git< status=delayed
+packet:  git< 
+
+
+If the filter supports the "delay" capability then it must support the
+"list_available_blobs" command. If Git sends this command, then the
+filter is expected to return a list of pathnames representing blobs
+that have been delayed earlier and are now available.
+The list must be terminated with a flush packet followed
+by a "success" status that is also terminated with a flush packet. If
+no blobs for the delayed paths are available, yet, then the filter is
+expected to block the response until at least one blob becomes
+available. The filter can tell Git that it has no more delayed blobs
+by sending an empty list. As soon as the filter responds with an empty
+list, Git stops asking. All blobs that Git has not received at this
+point are considered missing and will result in an error.
+
+
+packet:  git> command=list_available_blobs
+packet:  git> 
+packet:  git< pathname=path/testfile.dat
+packet:  git< pathname=path/otherfile.dat
+packet:  git< 
+packet:  git< status=success
+packet:  git< 
+
+
+After Git received the pathnames, it will request the corresponding
+blobs again. These requests contain a pathname and an empty content
+section. The filter is expected to respond with the smudged content
+in the usual way as explained above.