Junio C Hamano wrote:
Dan Holmsand <[EMAIL PROTECTED]> writes:
Repacking all of that to a single pack file gives, somewhat
surprisingly, a pack size of 62M (+ 1.3M index). In other words, the
cost of getting all those branches, and all of the new stuff from
Linus, turns out to be *negative* (probably due to some strange
deltification coincidence).



We do _not_ want to optimize for initial slurps into empty
repositories.  Quite the opposite.  We want to optimize for
allowing quick updates of reasonably up-to-date developer repos.
If initial slurps are _also_ efficient then that is an added
bonus; that is something the baseline big pack (60M Linus pack)
would give us already.  So repacking everything into a single
pack nightly is _not_ what we want to do, even though that would
give the maximum compression ;-).  I know you understand this,
but just stating the second of the above paragraphs would give
casual readers a wrong impression.

I agree, to a point: I think the bonus is quite nice to have... As it is, it's actually faster on my machine to clone a fresh tree of Linus' than it is to "git clone" a local tree (without doing the hardlinking "cheating", that is). And it's kind of nice to have the option to start completely fresh.

Anyway, my point is this: to make pulling efficient, we should ideally have (1) as few object files to pull as possible, especially when using http, and (2) have as few packs as possible, to gain some compression for those who pull more seldom. Point 1 is obviously the most important one.

To make this happen, relatively frequent repacking and re-repacking (even if only on parts of the repository) would be necessary. Or at least nice to have...

Which was why I wanted the "dumb fetch" thingies to at least do some "relatively smart un/repacking" to avoid duplication. And, ideally, that they would avoid downloading entire packs that we just want the beginning of. That would lessen the cost of repacking, which I happen to think is a good thing.

Also, it's kind of strange when the ssh/local fetching *always* unpacks everything, and rsync/http *never* does this...

You are correct.  For somebody like Jeff, having the Linus
baseline pack with one pack of all of his head (incremental that
excludes what is already in the Linus baseline pack) would help
pullers.

That would work, of course. It, however, means that Linus becomes the "official repository maintainer" in a way that doesn't feel very distributed. Perhaps then Linus' packs should be marked "official" in some way?

The big problem, however, comes when Jeff (or anyone else) decides to
repack. Then, if you fetch both his repo and Linus', you might end up
with several really big pack files, that mostly overlap. That could
easily mean storing most objects many times, if you don't do some
smart selective un/repacking when fetching.


Indeed.  Overlapping packs is a possibility, but my gut feeling
is that it would not be too bad, if things are arranged so that
packs are expanded-and-then-repacked _very_ rarely if ever.
Instead, at least for your public repository, if you only repack
incrementally I think you would be OK.

To be exact, you're ok (in the meaning of avoiding duplicates) as long as you always rsync in the "official packs", and coordinate with others you're merging with, before you do any repacking of your own. Sure, this works. It just feels a bit "un-distributed" for my personal taste...

/dan
-
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Reply via email to