Re: [PATCH 4/6] upload-pack: delegate rev walking in shallow fetch to pack-objects

2013-08-29 Thread Duy Nguyen
On Wed, Aug 28, 2013 at 9:52 PM, Matthijs Kooijman matth...@stdin.nl wrote:
 Hi Nguy,

 On Fri, Aug 16, 2013 at 04:52:05PM +0700, Nguyễn Thái Ngọc Duy wrote:
 upload-pack has a special rev walking code for shallow recipients. It
 works almost like the similar code in pack-objects except:

 1. in upload-pack, graft points could be added for deepening

 2. also when the repository is deepened, the shallow point will be
moved further away from the tip, but the old shallow point will be
marked as edge to produce more efficient packs. See 6523078 (make
shallow repository deepening more network efficient - 2009-09-03)

 pass the file to pack-objects via --shallow-file. This will override
 $GIT_DIR/shallow and give pack-objects the exact repository shape that
 upload-pack has.

 mark edge commits by revision command arguments. Even if old shallow
 points are passed as --not revisions as in this patch, they will not
 be picked up by mark_edges_uninteresting() because this function looks
 up to parents for edges, while in this case the edge is the children,
 in the opposite direction. This will be fixed in the next patch when
 all given uninteresting commits are marked as edges.
 This says the next patch but it really refers to 6/6, not 5/6. Patch
 6/6 has the same problem (it says previous patch). Perhaps patches 4
 and 5 should just be swapped?

Yeah. I guess I reordered the patches before sending out and forgot
that the commit message needs a special order. Wil do.
-- 
Duy
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [PATCH 4/6] upload-pack: delegate rev walking in shallow fetch to pack-objects

2013-08-28 Thread Matthijs Kooijman
Hi Nguy,

On Fri, Aug 16, 2013 at 04:52:05PM +0700, Nguyễn Thái Ngọc Duy wrote:
 upload-pack has a special rev walking code for shallow recipients. It
 works almost like the similar code in pack-objects except:
 
 1. in upload-pack, graft points could be added for deepening
 
 2. also when the repository is deepened, the shallow point will be
moved further away from the tip, but the old shallow point will be
marked as edge to produce more efficient packs. See 6523078 (make
shallow repository deepening more network efficient - 2009-09-03)
 
 pass the file to pack-objects via --shallow-file. This will override
 $GIT_DIR/shallow and give pack-objects the exact repository shape that
 upload-pack has.
 
 mark edge commits by revision command arguments. Even if old shallow
 points are passed as --not revisions as in this patch, they will not
 be picked up by mark_edges_uninteresting() because this function looks
 up to parents for edges, while in this case the edge is the children,
 in the opposite direction. This will be fixed in the next patch when
 all given uninteresting commits are marked as edges.
This says the next patch but it really refers to 6/6, not 5/6. Patch
6/6 has the same problem (it says previous patch). Perhaps patches 4
and 5 should just be swapped?

Gr.

Matthijs
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


[PATCH 4/6] upload-pack: delegate rev walking in shallow fetch to pack-objects

2013-08-16 Thread Nguyễn Thái Ngọc Duy
upload-pack has a special rev walking code for shallow recipients. It
works almost like the similar code in pack-objects except:

1. in upload-pack, graft points could be added for deepening

2. also when the repository is deepened, the shallow point will be
   moved further away from the tip, but the old shallow point will be
   marked as edge to produce more efficient packs. See 6523078 (make
   shallow repository deepening more network efficient - 2009-09-03)

pass the file to pack-objects via --shallow-file. This will override
$GIT_DIR/shallow and give pack-objects the exact repository shape that
upload-pack has.

mark edge commits by revision command arguments. Even if old shallow
points are passed as --not revisions as in this patch, they will not
be picked up by mark_edges_uninteresting() because this function looks
up to parents for edges, while in this case the edge is the children,
in the opposite direction. This will be fixed in the next patch when
all given uninteresting commits are marked as edges.

Signed-off-by: Nguyễn Thái Ngọc Duy pclo...@gmail.com
---
 t/t5530-upload-pack-error.sh |   3 -
 upload-pack.c| 128 +++
 2 files changed, 32 insertions(+), 99 deletions(-)

diff --git a/t/t5530-upload-pack-error.sh b/t/t5530-upload-pack-error.sh
index c983d36..3932e79 100755
--- a/t/t5530-upload-pack-error.sh
+++ b/t/t5530-upload-pack-error.sh
@@ -54,9 +54,6 @@ test_expect_success 'upload-pack fails due to error in 
rev-list' '
printf 0032want %s\n0034shallow %s0009done\n \
$(git rev-parse HEAD) $(git rev-parse HEAD^) input 
test_must_fail git upload-pack . input /dev/null 2output.err 
-   # pack-objects survived
-   grep Total.*, reused output.err 
-   # but there was an error, which must have been in rev-list
grep bad tree object output.err
 '
 
diff --git a/upload-pack.c b/upload-pack.c
index 127e59a..d5a003a 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -68,87 +68,28 @@ static ssize_t send_client_data(int fd, const char *data, 
ssize_t sz)
return sz;
 }
 
-static FILE *pack_pipe = NULL;
-static void show_commit(struct commit *commit, void *data)
-{
-   if (commit-object.flags  BOUNDARY)
-   fputc('-', pack_pipe);
-   if (fputs(sha1_to_hex(commit-object.sha1), pack_pipe)  0)
-   die(broken output pipe);
-   fputc('\n', pack_pipe);
-   fflush(pack_pipe);
-   free(commit-buffer);
-   commit-buffer = NULL;
-}
-
-static void show_object(struct object *obj,
-   const struct name_path *path, const char *component,
-   void *cb_data)
-{
-   show_object_with_name(pack_pipe, obj, path, component);
-}
-
-static void show_edge(struct commit *commit)
-{
-   fprintf(pack_pipe, -%s\n, sha1_to_hex(commit-object.sha1));
-}
-
-static int do_rev_list(int in, int out, void *user_data)
-{
-   int i;
-   struct rev_info revs;
-
-   pack_pipe = xfdopen(out, w);
-   init_revisions(revs, NULL);
-   revs.tag_objects = 1;
-   revs.tree_objects = 1;
-   revs.blob_objects = 1;
-   if (use_thin_pack)
-   revs.edge_hint = 1;
-
-   for (i = 0; i  want_obj.nr; i++) {
-   struct object *o = want_obj.objects[i].item;
-   /* why??? */
-   o-flags = ~UNINTERESTING;
-   add_pending_object(revs, o, NULL);
-   }
-   for (i = 0; i  have_obj.nr; i++) {
-   struct object *o = have_obj.objects[i].item;
-   o-flags |= UNINTERESTING;
-   add_pending_object(revs, o, NULL);
-   }
-   setup_revisions(0, NULL, revs, NULL);
-   if (prepare_revision_walk(revs))
-   die(revision walk setup failed);
-   mark_edges_uninteresting(revs.commits, revs, show_edge);
-   if (use_thin_pack)
-   for (i = 0; i  extra_edge_obj.nr; i++)
-   fprintf(pack_pipe, -%s\n, sha1_to_hex(
-   extra_edge_obj.objects[i].item-sha1));
-   traverse_commit_list(revs, show_commit, show_object, NULL);
-   fflush(pack_pipe);
-   fclose(pack_pipe);
-   return 0;
-}
-
 static void create_pack_file(void)
 {
-   struct async rev_list;
struct child_process pack_objects;
char data[8193], progress[128];
char abort_msg[] = aborting due to possible repository 
corruption on the remote side.;
int buffered = -1;
ssize_t sz;
-   const char *argv[10];
-   int arg = 0;
+   const char *argv[12];
+   int i, arg = 0;
+   FILE *pipe_fd;
+   char *shallow_file = NULL;
 
-   argv[arg++] = pack-objects;
-   if (!shallow_nr) {
-   argv[arg++] = --revs;
-   if (use_thin_pack)
-   argv[arg++] = --thin;
+   if (shallow_nr) {
+   shallow_file = setup_temporary_shallow();
+