Re: [PATCH 14/23] pack v4: object data copy

2013-08-27 Thread Junio C Hamano
Nicolas Pitre n...@fluxnic.net writes:

 Blob and tag objects have no particular changes except for their object
 header.

 Delta objects are also copied as is, except for their delta base reference
 which is converted to the new way as used elsewhere in pack v4 encoding
 i.e. an index into the SHA1 table or a literal SHA1 prefixed by 0 if not
 found in the table (see add_sha1_ref).  This is true for both REF_DELTA
 as well as OFS_DELTA.

 Object payload is validated against the recorded CRC32 in the source
 pack index file when possible.

 Signed-off-by: Nicolas Pitre n...@fluxnic.net
 ---

The title somewhat confused me until I realized that this series is
building a program that would convert existing data from a single
pack into packv4 format, not a pack-objects --pack-verison=4.

  packv4-create.c | 66 
 +
  1 file changed, 66 insertions(+)

 diff --git a/packv4-create.c b/packv4-create.c
 index 6e0bb1d..a6dc818 100644
 --- a/packv4-create.c
 +++ b/packv4-create.c
 @@ -12,6 +12,7 @@
  #include object.h
  #include tree-walk.h
  #include pack.h
 +#include pack-revindex.h
  
  
  static int pack_compression_level = Z_DEFAULT_COMPRESSION;
 @@ -673,6 +674,71 @@ static unsigned int write_object_header(struct sha1file 
 *f, enum object_type typ
   return end - buf;
  }
  
 +static unsigned long copy_object_data(struct sha1file *f, struct packed_git 
 *p,
 +   off_t offset)
 +{
 + struct pack_window *w_curs = NULL;
 + struct revindex_entry *revidx;
 + enum object_type type;
 + unsigned long avail, size, datalen, written;
 + int hdrlen, idx_nr;
 + unsigned char *src, *end, buf[24];
 +
 + revidx = find_pack_revindex(p, offset);
 + idx_nr = revidx-nr;
 + datalen = revidx[1].offset - offset;
 +
 + src = use_pack(p, w_curs, offset, avail);
 + hdrlen = unpack_object_header_buffer(src, avail, type, size);
 +
 + written = write_object_header(f, type, size);
 +
 + if (type == OBJ_OFS_DELTA) {
 + unsigned char c = src[hdrlen++];
 + off_t base_offset = c  127;
 + while (c  128) {
 + base_offset += 1;
 + if (!base_offset || MSB(base_offset, 7))
 + die(delta offset overflow);
 + c = src[hdrlen++];
 + base_offset = (base_offset  7) + (c  127);
 + }
 + base_offset = offset - base_offset;
 + if (base_offset = 0 || base_offset = offset)
 + die(delta offset out of bound);
 + revidx = find_pack_revindex(p, base_offset);
 + end = add_sha1_ref(buf, nth_packed_object_sha1(p, revidx-nr));
 + sha1write(f, buf, end - buf);
 + written += end - buf;
 + } else if (type == OBJ_REF_DELTA) {
 + end = add_sha1_ref(buf, src + hdrlen);
 + hdrlen += 20;
 + sha1write(f, buf, end - buf);
 + written += end - buf;
 + }
 +
 + if (p-index_version  1 
 + check_pack_crc(p, w_curs, offset, datalen, idx_nr))
 + die(bad CRC for object at offset %PRIuMAX in %s,
 + (uintmax_t)offset, p-pack_name);
 +
 + offset += hdrlen;
 + datalen -= hdrlen;
 +
 + while (datalen) {
 + src = use_pack(p, w_curs, offset, avail);
 + if (avail  datalen)
 + avail = datalen;
 + sha1write(f, src, avail);
 + written += avail;
 + offset += avail;
 + datalen -= avail;
 + }
 + unuse_pack(w_curs);
 +
 + return written;
 +}
 +
  static struct packed_git *open_pack(const char *path)
  {
   char arg[PATH_MAX];
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [PATCH 14/23] pack v4: object data copy

2013-08-27 Thread Nicolas Pitre
On Tue, 27 Aug 2013, Junio C Hamano wrote:

 Nicolas Pitre n...@fluxnic.net writes:
 
  Blob and tag objects have no particular changes except for their object
  header.
 
  Delta objects are also copied as is, except for their delta base reference
  which is converted to the new way as used elsewhere in pack v4 encoding
  i.e. an index into the SHA1 table or a literal SHA1 prefixed by 0 if not
  found in the table (see add_sha1_ref).  This is true for both REF_DELTA
  as well as OFS_DELTA.
 
  Object payload is validated against the recorded CRC32 in the source
  pack index file when possible.
 
  Signed-off-by: Nicolas Pitre n...@fluxnic.net
  ---
 
 The title somewhat confused me until I realized that this series is
 building a program that would convert existing data from a single
 pack into packv4 format, not a pack-objects --pack-verison=4.

I initially started with extensions to pack-objects but that quickly 
became a big hassle to keep things working while experimenting.  So this 
tool is just a straight pack converter which in itself turned out to be 
quite complex already.  Eventually its code could be merged into 
pack-objects.


Nicolas
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html