Re: [PATCH v4 20/35] upload-pack: introduce fetch server command

[Brandon Williams]

On 03/13, Jonathan Tan wrote:
> On Wed, 28 Feb 2018 15:22:37 -0800
> Brandon Williams  wrote:
> 
> > +output = *section
> > +section = (acknowledgments | packfile)
> > + (flush-pkt | delim-pkt)
> > +
> > +acknowledgments = PKT-LINE("acknowledgments" LF)
> > + (nak | *ack)
> > + (ready)
> > +ready = PKT-LINE("ready" LF)
> > +nak = PKT-LINE("NAK" LF)
> > +ack = PKT-LINE("ACK" SP obj-id LF)
> > +
> > +packfile = PKT-LINE("packfile" LF)
> > +  [PACKFILE]
> 
> I should have noticed this earlier, but "PACKFILE" is not defined anywhere -
> it's probably better written as:
> 
> *PKT-LINE(%x01-03 *%x00-ff)"
> 
> or something like that.

I'll document it as you described.

> 
> > +acknowledgments section
> > +   * Always begins with the section header "acknowledgments"
> > +
> > +   * The server will respond with "NAK" if none of the object ids sent
> > + as have lines were common.
> > +
> > +   * The server will respond with "ACK obj-id" for all of the
> > + object ids sent as have lines which are common.
> > +
> > +   * A response cannot have both "ACK" lines as well as a "NAK"
> > + line.
> > +
> > +   * The server will respond with a "ready" line indicating that
> > + the server has found an acceptable common base and is ready to
> > + make and send a packfile (which will be found in the packfile
> > + section of the same response)
> > +
> > +   * If the client determines that it is finished with negotiations
> > + by sending a "done" line, the acknowledgments sections MUST be
> > + omitted from the server's response.
> > +
> > +   * If the server has found a suitable cut point and has decided
> > + to send a "ready" line, then the server can decide to (as an
> > + optimization) omit any "ACK" lines it would have sent during
> > + its response.  This is because the server will have already
> > + determined the objects it plans to send to the client and no
> > + further negotiation is needed.
> > +
> > +
> > +packfile section
> > +   * Always begins with the section header "packfile"
> > +
> > +   * The transmission of the packfile begins immediately after the
> > + section header
> > +
> > +   * The data transfer of the packfile is always multiplexed, using
> > + the same semantics of the 'side-band-64k' capability from
> > + protocol version 1.  This means that each packet, during the
> > + packfile data stream, is made up of a leading 4-byte pkt-line
> > + length (typical of the pkt-line format), followed by a 1-byte
> > + stream code, followed by the actual data.
> > +
> > + The stream code can be one of:
> > +   1 - pack data
> > +   2 - progress messages
> > +   3 - fatal error message just before stream aborts
> > +
> > +   * This section is only included if the client has sent 'want'
> > + lines in its request and either requested that no more
> > + negotiation be done by sending 'done' or if the server has
> > + decided it has found a sufficient cut point to produce a
> > + packfile.
> 
> For both the sections, I think that the conditions for
> inclusion/non-inclusion ("This section is only included if...") should
> be the first point.
> 
> > +static void upload_pack_data_init(struct upload_pack_data *data)
> > +{
> > +   struct object_array wants = OBJECT_ARRAY_INIT;
> > +   struct oid_array haves = OID_ARRAY_INIT;
> > +
> > +   memset(data, 0, sizeof(*data));
> > +   data->wants = wants;
> > +   data->haves = haves;
> > +}
> 
> Any reason to use a initializer function instead of a static literal?

Its much cleaner and easier to read than it was when i was using an
initializer.

-- 
Brandon Williams

Re: [PATCH v4 20/35] upload-pack: introduce fetch server command

[Jonathan Tan]

On Wed, 28 Feb 2018 15:22:37 -0800
Brandon Williams  wrote:

> +output = *section
> +section = (acknowledgments | packfile)
> +   (flush-pkt | delim-pkt)
> +
> +acknowledgments = PKT-LINE("acknowledgments" LF)
> +   (nak | *ack)
> +   (ready)
> +ready = PKT-LINE("ready" LF)
> +nak = PKT-LINE("NAK" LF)
> +ack = PKT-LINE("ACK" SP obj-id LF)
> +
> +packfile = PKT-LINE("packfile" LF)
> +[PACKFILE]

I should have noticed this earlier, but "PACKFILE" is not defined anywhere -
it's probably better written as:

*PKT-LINE(%x01-03 *%x00-ff)"

or something like that.

> +acknowledgments section
> + * Always begins with the section header "acknowledgments"
> +
> + * The server will respond with "NAK" if none of the object ids sent
> +   as have lines were common.
> +
> + * The server will respond with "ACK obj-id" for all of the
> +   object ids sent as have lines which are common.
> +
> + * A response cannot have both "ACK" lines as well as a "NAK"
> +   line.
> +
> + * The server will respond with a "ready" line indicating that
> +   the server has found an acceptable common base and is ready to
> +   make and send a packfile (which will be found in the packfile
> +   section of the same response)
> +
> + * If the client determines that it is finished with negotiations
> +   by sending a "done" line, the acknowledgments sections MUST be
> +   omitted from the server's response.
> +
> + * If the server has found a suitable cut point and has decided
> +   to send a "ready" line, then the server can decide to (as an
> +   optimization) omit any "ACK" lines it would have sent during
> +   its response.  This is because the server will have already
> +   determined the objects it plans to send to the client and no
> +   further negotiation is needed.
> +
> +
> +packfile section
> + * Always begins with the section header "packfile"
> +
> + * The transmission of the packfile begins immediately after the
> +   section header
> +
> + * The data transfer of the packfile is always multiplexed, using
> +   the same semantics of the 'side-band-64k' capability from
> +   protocol version 1.  This means that each packet, during the
> +   packfile data stream, is made up of a leading 4-byte pkt-line
> +   length (typical of the pkt-line format), followed by a 1-byte
> +   stream code, followed by the actual data.
> +
> +   The stream code can be one of:
> + 1 - pack data
> + 2 - progress messages
> + 3 - fatal error message just before stream aborts
> +
> + * This section is only included if the client has sent 'want'
> +   lines in its request and either requested that no more
> +   negotiation be done by sending 'done' or if the server has
> +   decided it has found a sufficient cut point to produce a
> +   packfile.

For both the sections, I think that the conditions for
inclusion/non-inclusion ("This section is only included if...") should
be the first point.

> +static void upload_pack_data_init(struct upload_pack_data *data)
> +{
> + struct object_array wants = OBJECT_ARRAY_INIT;
> + struct oid_array haves = OID_ARRAY_INIT;
> +
> + memset(data, 0, sizeof(*data));
> + data->wants = wants;
> + data->haves = haves;
> +}

Any reason to use a initializer function instead of a static literal?

[PATCH v4 20/35] upload-pack: introduce fetch server command

[Brandon Williams]

Introduce the 'fetch' server command.

Signed-off-by: Brandon Williams 
---
 Documentation/technical/protocol-v2.txt | 128 
 serve.c |   2 +
 t/t5701-git-serve.sh|   1 +
 upload-pack.c   | 267 
 upload-pack.h   |   5 +
 5 files changed, 403 insertions(+)

diff --git a/Documentation/technical/protocol-v2.txt 
b/Documentation/technical/protocol-v2.txt
index 7f50e6462..99c70a1e4 100644
--- a/Documentation/technical/protocol-v2.txt
+++ b/Documentation/technical/protocol-v2.txt
@@ -205,3 +205,131 @@ The output of ls-refs is as follows:
 ref-attribute = (symref | peeled)
 symref = "symref-target:" symref-target
 peeled = "peeled:" obj-id
+
+ fetch
+---
+
+`fetch` is the command used to fetch a packfile in v2.  It can be looked
+at as a modified version of the v1 fetch where the ref-advertisement is
+stripped out (since the `ls-refs` command fills that role) and the
+message format is tweaked to eliminate redundancies and permit easy
+addition of future extensions.
+
+Additional features not supported in the base command will be advertised
+as the value of the command in the capability advertisement in the form
+of a space separated list of features, e.g.  "=
+".
+
+A `fetch` request can take the following arguments:
+
+want 
+   Indicates to the server an object which the client wants to
+   retrieve.  Wants can be anything and are not limited to
+   advertised objects.
+
+have 
+   Indicates to the server an object which the client has locally.
+   This allows the server to make a packfile which only contains
+   the objects that the client needs. Multiple 'have' lines can be
+   supplied.
+
+done
+   Indicates to the server that negotiation should terminate (or
+   not even begin if performing a clone) and that the server should
+   use the information supplied in the request to construct the
+   packfile.
+
+thin-pack
+   Request that a thin pack be sent, which is a pack with deltas
+   which reference base objects not contained within the pack (but
+   are known to exist at the receiving end). This can reduce the
+   network traffic significantly, but it requires the receiving end
+   to know how to "thicken" these packs by adding the missing bases
+   to the pack.
+
+no-progress
+   Request that progress information that would normally be sent on
+   side-band channel 2, during the packfile transfer, should not be
+   sent.  However, the side-band channel 3 is still used for error
+   responses.
+
+include-tag
+   Request that annotated tags should be sent if the objects they
+   point to are being sent.
+
+ofs-delta
+   Indicate that the client understands PACKv2 with delta referring
+   to its base by position in pack rather than by an oid.  That is,
+   they can read OBJ_OFS_DELTA (ake type 6) in a packfile.
+
+The response of `fetch` is broken into a number of sections separated by
+delimiter packets (0001), with each section beginning with its section
+header.
+
+output = *section
+section = (acknowledgments | packfile)
+ (flush-pkt | delim-pkt)
+
+acknowledgments = PKT-LINE("acknowledgments" LF)
+ (nak | *ack)
+ (ready)
+ready = PKT-LINE("ready" LF)
+nak = PKT-LINE("NAK" LF)
+ack = PKT-LINE("ACK" SP obj-id LF)
+
+packfile = PKT-LINE("packfile" LF)
+  [PACKFILE]
+
+
+acknowledgments section
+   * Always begins with the section header "acknowledgments"
+
+   * The server will respond with "NAK" if none of the object ids sent
+ as have lines were common.
+
+   * The server will respond with "ACK obj-id" for all of the
+ object ids sent as have lines which are common.
+
+   * A response cannot have both "ACK" lines as well as a "NAK"
+ line.
+
+   * The server will respond with a "ready" line indicating that
+ the server has found an acceptable common base and is ready to
+ make and send a packfile (which will be found in the packfile
+ section of the same response)
+
+   * If the client determines that it is finished with negotiations
+ by sending a "done" line, the acknowledgments sections MUST be
+ omitted from the server's response.
+
+   * If the server has found a suitable cut point and has decided
+ to send a "ready" line, then the server can decide to (as an
+ optimization) omit any "ACK" lines it would have sent during
+ its response.  This is because the server will have already
+ determined the objects it plans to send to the client and no
+ further negotiation is needed.
+
+
+packfile section
+   * Always begins with the section header "packfile"
+
+