Re: [PATCH] pack-objects: name pack files after trailer hash
Jeff King writes:
> On Mon, Dec 16, 2013 at 11:19:33AM -0800, Jonathan Nieder wrote:
>
>> > I was tempted to explicitly say something like "this is
>> > opaque and meaningless to you, don't rely on it", but I don't know that
>> > there is any need.
>> [...]
>> > On top of jk/name-pack-after-byte-representations, naturally.
>>
>> I think there is --- if someone starts caring about the SHA-1 used,
>> they won't be able to act on old packfiles that were created before
>> this change. How about something like the following instead?
>
> Right, my point was that I do not think anybody has ever cared, and I do
> not see them starting now. But that is just my intuition.
>
>> diff --git a/Documentation/git-pack-objects.txt
>> b/Documentation/git-pack-objects.txt
>> index d94edcd..cdab9ed 100644
>> --- a/Documentation/git-pack-objects.txt
>> +++ b/Documentation/git-pack-objects.txt
>> @@ -51,8 +51,7 @@ base-name::
>> to determine the name of the created file.
>> When this option is used, the two files are written in
>> -.{pack,idx} files. is a hash
>> -of the sorted object names to make the resulting filename
>> -based on the pack content, and written to the standard
>> +based on the pack content and is written to the standard
>
> I'm fine with that. I was worried it would get clunky, but the way you
> have worded it is good.
Our mails crossed; I think the above is good.
Thanks.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] pack-objects: name pack files after trailer hash
On Mon, Dec 16, 2013 at 11:33:11AM -0800, Junio C Hamano wrote:
> > to determine the name of the created file.
> > When this option is used, the two files are written in
> > -.{pack,idx} files. is a hash
> > + of the bytes of the packfile, and is written to the standard
>
> "hash of the bytes of the packfile" tempts one to do
>
> $ sha1sum .git/objects/pack/pack-*.pack
>
> but that is not what we expect. I wonder if there are better ways to
> phrase it (or alternatively perhaps we want to make that expectation
> hold by updating our code to hash)?
Yeah, I wondered about that, but didn't think it was worth the verbosity
to explain that the true derivation. I think Jonathan's suggestion takes
care of it, though.
-Peff
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] pack-objects: name pack files after trailer hash
Jeff King writes:
> I was tempted to explicitly say something like "this is
> opaque and meaningless to you, don't rely on it", but I don't know that
> there is any need.
Thanks.
When we did the original naming, it was envisioned that we may use
the name for fsck to make sure that the pack contains what it
contains in the name, but it never materialized. The most prominent
and useful characteristic of the new naming scheme is that two
packfiles with the same name must be identical, and we may want to
start using it some time later once everybody repacked their packs
with the updated pack-objects.
But until that time comes, some packs in existing repositories will
hash to their names while others do not, so spelling out how the new
names are derived without saying older pack-objects used to name
their output differently may add more confusion than it is worth.
>to determine the name of the created file.
> When this option is used, the two files are written in
> -.{pack,idx} files. is a hash
> + of the bytes of the packfile, and is written to the standard
"hash of the bytes of the packfile" tempts one to do
$ sha1sum .git/objects/pack/pack-*.pack
but that is not what we expect. I wonder if there are better ways to
phrase it (or alternatively perhaps we want to make that expectation
hold by updating our code to hash)?
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] pack-objects: name pack files after trailer hash
On Mon, Dec 16, 2013 at 11:19:33AM -0800, Jonathan Nieder wrote:
> > I was tempted to explicitly say something like "this is
> > opaque and meaningless to you, don't rely on it", but I don't know that
> > there is any need.
> [...]
> > On top of jk/name-pack-after-byte-representations, naturally.
>
> I think there is --- if someone starts caring about the SHA-1 used,
> they won't be able to act on old packfiles that were created before
> this change. How about something like the following instead?
Right, my point was that I do not think anybody has ever cared, and I do
not see them starting now. But that is just my intuition.
> diff --git a/Documentation/git-pack-objects.txt
> b/Documentation/git-pack-objects.txt
> index d94edcd..cdab9ed 100644
> --- a/Documentation/git-pack-objects.txt
> +++ b/Documentation/git-pack-objects.txt
> @@ -51,8 +51,7 @@ base-name::
>to determine the name of the created file.
> When this option is used, the two files are written in
> -.{pack,idx} files. is a hash
> - of the sorted object names to make the resulting filename
> - based on the pack content, and written to the standard
> + based on the pack content and is written to the standard
I'm fine with that. I was worried it would get clunky, but the way you
have worded it is good.
-Peff
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] pack-objects: name pack files after trailer hash
Jeff King wrote:
> The original patch is in next, so here's one on top. I just updated the
> description.
Thanks.
> I was tempted to explicitly say something like "this is
> opaque and meaningless to you, don't rely on it", but I don't know that
> there is any need.
[...]
> On top of jk/name-pack-after-byte-representations, naturally.
I think there is --- if someone starts caring about the SHA-1 used,
they won't be able to act on old packfiles that were created before
this change. How about something like the following instead?
-- >8 --
From: Jeff King
Subject: pack-objects doc: treat output filename as opaque
After 1190a1a (pack-objects: name pack files after trailer hash,
2013-12-05), the SHA-1 used to determine the filename is calculated
differently. Update the documentation to not guarantee anything more
than that the SHA-1 depends on the pack content somehow.
Hopefully this will discourage readers from depending on the old or
the new calculation.
Reported-by: Michael Haggerty
Signed-off-by: Jeff King
Signed-off-by: Jonathan Nieder
---
Documentation/git-pack-objects.txt | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/Documentation/git-pack-objects.txt
b/Documentation/git-pack-objects.txt
index d94edcd..cdab9ed 100644
--- a/Documentation/git-pack-objects.txt
+++ b/Documentation/git-pack-objects.txt
@@ -51,8 +51,7 @@ base-name::
to determine the name of the created file.
When this option is used, the two files are written in
-.{pack,idx} files. is a hash
- of the sorted object names to make the resulting filename
- based on the pack content, and written to the standard
+ based on the pack content and is written to the standard
output of the command.
--stdout::
--
1.8.5.1
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] pack-objects: name pack files after trailer hash
On Mon, Dec 16, 2013 at 08:41:38AM +0100, Michael Haggerty wrote:
> The old naming scheme is documented in
> Documentation/git-pack-objects.txt, under "OPTIONS" -> "base-name":
>
> > base-name::
> > Write into a pair of files (.pack and .idx), using
> > to determine the name of the created file.
> > When this option is used, the two files are written in
> > -.{pack,idx} files. is a hash
> > of the sorted object names to make the resulting filename
> > based on the pack content, and written to the standard
> > output of the command.
>
> The documentation should either be updated or the description of the
> naming scheme should be removed altogether.
Thanks. I looked in Documentation/technical for anything to update, but
didn't imagine we would be advertising the format in the user-facing
documentation. :)
The original patch is in next, so here's one on top. I just updated the
description. I was tempted to explicitly say something like "this is
opaque and meaningless to you, don't rely on it", but I don't know that
there is any need.
-- >8 --
Subject: docs: update pack-objects "base-name" description
As of 1190a1a, the SHA-1 used to determine the filename is
now calculated differently. Update the documentation to
reflect this.
Noticed-by: Michael Haggerty
Signed-off-by: Jeff King
---
On top of jk/name-pack-after-byte-representations, naturally.
Documentation/git-pack-objects.txt | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/Documentation/git-pack-objects.txt
b/Documentation/git-pack-objects.txt
index d94edcd..c69affc 100644
--- a/Documentation/git-pack-objects.txt
+++ b/Documentation/git-pack-objects.txt
@@ -51,8 +51,7 @@ base-name::
to determine the name of the created file.
When this option is used, the two files are written in
-.{pack,idx} files. is a hash
- of the sorted object names to make the resulting filename
- based on the pack content, and written to the standard
+ of the bytes of the packfile, and is written to the standard
output of the command.
--stdout::
--
1.8.5.524.g6743da6
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] pack-objects: name pack files after trailer hash
On 12/05/2013 09:28 PM, Jeff King wrote:
> [...]
> This patch simply uses the pack trailer sha1 as the pack
> name. It should be similarly unique, but covers the exact
> representation of the objects. Other parts of git should not
> care, as the pack name is returned by pack-objects and is
> essentially opaque.
> [...]
Peff,
The old naming scheme is documented in
Documentation/git-pack-objects.txt, under "OPTIONS" -> "base-name":
> base-name::
> Write into a pair of files (.pack and .idx), using
>to determine the name of the created file.
> When this option is used, the two files are written in
> -.{pack,idx} files. is a hash
> of the sorted object names to make the resulting filename
> based on the pack content, and written to the standard
> output of the command.
The documentation should either be updated or the description of the
naming scheme should be removed altogether.
Michael
--
Michael Haggerty
mhag...@alum.mit.edu
http://softwareswirl.blogspot.com/
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] pack-objects: name pack files after trailer hash
On Thu, Dec 05, 2013 at 02:59:45PM -0800, Junio C Hamano wrote: > > One test needs to be updated, because it actually corrupts a > > pack and expects that re-packing the corrupted bytes will > > use the same name. It won't anymore, but we can easily just > > use the name that pack-objects hands back. > > Re-reading the tests in that script, I am not sure if keeping these > tests is even a sane thing to do, by the way. It "expects" that > certain breakages are propagated, and anybody who breaks that > expectation by improving pack-objects etc. to catch such breakages > will be yelled at by breaking the test that used to pass. I had a similar thought, but I figured I would leave it for the person who _does_ make that change. The yelling will be a good signal that they've got it right, and they can clean up the test (either by dropping it, or modifying it to check the right thing) at that point. > Seeing that the way the test scripts are line-wrapped follows the > ancient convention, I suspect that this may be because it predates > our more recent best practice to document known breakages with > test_expect_failure. I read it more as "make sure that the v1 index breaks, so when we are testing v2 we know it is not an accident that we notice the breakage". But I also see your reason, and I think it would be fine to use test_expect_failure. -Peff -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] pack-objects: name pack files after trailer hash
Jeff King writes:
> The second half would be to simplify git-repack. The current behavior is
> to replace the old packfile with a tricky rename dance. Which is still
> correct, but overly complicated. We should be able to just drop the new
> packfile, since we know the bytes are identical (or rename the new one
> over the old, though I think keeping the old is probably kinder to the
> disk cache, especially if another process already has it mmap'd).
Concurred.
> One test needs to be updated, because it actually corrupts a
> pack and expects that re-packing the corrupted bytes will
> use the same name. It won't anymore, but we can easily just
> use the name that pack-objects hands back.
Re-reading the tests in that script, I am not sure if keeping these
tests is even a sane thing to do, by the way. It "expects" that
certain breakages are propagated, and anybody who breaks that
expectation by improving pack-objects etc. to catch such breakages
will be yelled at by breaking the test that used to pass.
Seeing that the way the test scripts are line-wrapped follows the
ancient convention, I suspect that this may be because it predates
our more recent best practice to document known breakages with
test_expect_failure.
> diff --git a/t/t5302-pack-index.sh b/t/t5302-pack-index.sh
> index fe82025..4bbb718 100755
> --- a/t/t5302-pack-index.sh
> +++ b/t/t5302-pack-index.sh
> @@ -174,11 +174,11 @@ test_expect_success \
> test_expect_success \
> '[index v1] 5) pack-objects happily reuses corrupted data' \
> 'pack4=$(git pack-objects test-4 - test -f "test-4-${pack1}.pack"'
> + test -f "test-4-${pack4}.pack"'
>
> test_expect_success \
> '[index v1] 6) newly created pack is BAD !' \
> -'test_must_fail git verify-pack -v "test-4-${pack1}.pack"'
> +'test_must_fail git verify-pack -v "test-4-${pack4}.pack"'
A good thing is that the above hunks are the right thing to do, even
if we are to modernise these tests so that they document a known
breakage with expect-failure.
Thanks.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] pack-objects: name pack files after trailer hash
On Thu, Dec 5, 2013 at 12:28 PM, Jeff King wrote: > Subject: pack-objects: name pack files after trailer hash > > Our current scheme for naming packfiles is to calculate the > sha1 hash of the sorted list of objects contained in the > packfile. This gives us a unique name, so we are reasonably > sure that two packs with the same name will contain the same > objects. Yay-by: Shawn Pearce > --- > pack-write.c | 8 +--- > pack.h| 2 +- > t/t5302-pack-index.sh | 4 ++-- > 3 files changed, 4 insertions(+), 10 deletions(-) Obviously this is correct given the diffstat. :-) -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] pack-objects: name pack files after trailer hash
On Thu, Dec 05, 2013 at 11:04:18AM -0500, Jeff King wrote:
> > Yes. And this is why the packfile name algorithm is horribly flawed. I
> > keep saying we should change it to name the pack using the last 20
> > bytes of the file but ... nobody has written the patch for that? :-)
>
> Totally agree. I think we could also get rid of the horrible hacks in
> repack where we pack to a tempfile, then have to do another tempfile
> dance (which is not atomic!) to move the same-named packfile out of the
> way. If the name were based on the content, we could just throw away our
> new pack if one of the same name is already there (just like we do for
> loose objects).
>
> I haven't looked at making such a patch, but I think it shouldn't be too
> complicated. My big worry would be weird fallouts from some hidden part
> of the code that we don't realize is depending on the current naming
> scheme. :)
So here's the first part, that actually changes the name. It passes the
test suite, so it must be good, right? And just look at that diffstat.
This actually applies on top of e74435a (sha1write: make buffer
const-correct, 2013-10-24), which is on another topic. Since the sha1
parameter to write_idx_file is no longer used as both an in- and out-
parameter (yuck), we can make it const.
The second half would be to simplify git-repack. The current behavior is
to replace the old packfile with a tricky rename dance. Which is still
correct, but overly complicated. We should be able to just drop the new
packfile, since we know the bytes are identical (or rename the new one
over the old, though I think keeping the old is probably kinder to the
disk cache, especially if another process already has it mmap'd).
-- >8 --
Subject: pack-objects: name pack files after trailer hash
Our current scheme for naming packfiles is to calculate the
sha1 hash of the sorted list of objects contained in the
packfile. This gives us a unique name, so we are reasonably
sure that two packs with the same name will contain the same
objects.
It does not, however, tell us that two such packs have the
exact same bytes. This makes things awkward if we repack the
same set of objects. Due to run-to-run variations, the bytes
may not be identical (e.g., changed zlib or git versions,
different source object reuse due to new packs in the
repository, or even different deltas due to races during a
multi-threaded delta search).
In theory, this could be helpful to a program that cares
that the packfile contains a certain set of objects, but
does not care about the particular representation. In
practice, no part of git makes use of that, and in many
cases it is potentially harmful. For example, if a dumb http
client fetches the .idx file, it must be sure to get the
exact .pack that matches it. Similarly, a partial transfer
of a .pack file cannot be safely resumed, as the actual
bytes may have changed. This could also affect a local
client which opened the .idx and .pack files, closes the
.pack file (due to memory or file descriptor limits), and
then re-opens a changed packfile.
In all of these cases, git can detect the problem, as we
have the sha1 of the bytes themselves in the pack trailer
(which we verify on transfer), and the .idx file references
the trailer from the matching packfile. But it would be
simpler and more efficient to actually get the correct
bytes, rather than noticing the problem and having to
restart the operation.
This patch simply uses the pack trailer sha1 as the pack
name. It should be similarly unique, but covers the exact
representation of the objects. Other parts of git should not
care, as the pack name is returned by pack-objects and is
essentially opaque.
One test needs to be updated, because it actually corrupts a
pack and expects that re-packing the corrupted bytes will
use the same name. It won't anymore, but we can easily just
use the name that pack-objects hands back.
Signed-off-by: Jeff King
---
pack-write.c | 8 +---
pack.h| 2 +-
t/t5302-pack-index.sh | 4 ++--
3 files changed, 4 insertions(+), 10 deletions(-)
diff --git a/pack-write.c b/pack-write.c
index ca9e63b..ddc174e 100644
--- a/pack-write.c
+++ b/pack-write.c
@@ -44,14 +44,13 @@ static int need_large_offset(off_t offset, const struct
pack_idx_option *opts)
*/
const char *write_idx_file(const char *index_name, struct pack_idx_entry
**objects,
int nr_objects, const struct pack_idx_option *opts,
- unsigned char *sha1)
+ const unsigned char *sha1)
{
struct sha1file *f;
struct pack_idx_entry **sorted_by_sha, **list, **last;
off_t last_obj_offset = 0;
uint32_t array[256];
int i, fd;
- git_SHA_CTX ctx;
uint32_t index_version;
if (nr_objects) {
@@ -114,9 +113,6 @@ const char *write_idx_file(const char *index_name, struct
pack_idx_entry **objec
}
sha1write(f, array, 256 * 4);
-
