Re: upload-pack is slow with lots of refs

2012-10-09 Thread Shawn Pearce
On Mon, Oct 8, 2012 at 8:05 AM, Johannes Sixt j...@kdbg.org wrote: Am 05.10.2012 18:57, schrieb Shawn Pearce: On Thu, Oct 4, 2012 at 11:24 PM, Johannes Sixt j.s...@viscovery.net wrote: Upload-pack can just start advertising refs in the v1 way and announce a v2 capability and listen for

Re: upload-pack is slow with lots of refs

2012-10-09 Thread Johannes Sixt
Am 09.10.2012 08:46, schrieb Shawn Pearce: On Mon, Oct 8, 2012 at 8:05 AM, Johannes Sixt j...@kdbg.org wrote: Am 05.10.2012 18:57, schrieb Shawn Pearce: Smart HTTP is not bidirectional. The client can't cut off the server. Smart HTTP does not need it: you already posted a better solution (I'm

Re: upload-pack is slow with lots of refs

2012-10-09 Thread Johannes Sixt
Am 09.10.2012 22:30, schrieb Johannes Sixt: Am 09.10.2012 08:46, schrieb Shawn Pearce: As it turns out we don't really have this problem with git://. Clients can bury a v2 request in the extended headers where the host line appears today. I tried, but it seems that todays git-daemons are

Re: upload-pack is slow with lots of refs

2012-10-08 Thread Johannes Sixt
Am 05.10.2012 18:57, schrieb Shawn Pearce: On Thu, Oct 4, 2012 at 11:24 PM, Johannes Sixt j.s...@viscovery.net wrote: Upload-pack can just start advertising refs in the v1 way and announce a v2 capability and listen for response in parallel. A v2 capable client can start sending wants or some

Re: upload-pack is slow with lots of refs

2012-10-05 Thread Johannes Sixt
Am 10/3/2012 21:41, schrieb Shawn Pearce: On Wed, Oct 3, 2012 at 11:55 AM, Jeff King p...@peff.net wrote: On Wed, Oct 03, 2012 at 11:53:35AM -0700, Junio C Hamano wrote: Jeff King p...@peff.net writes: Has there been any work on extending the protocol so that the client tells the server what

Re: upload-pack is slow with lots of refs

2012-10-05 Thread Shawn Pearce
On Thu, Oct 4, 2012 at 11:24 PM, Johannes Sixt j.s...@viscovery.net wrote: Am 10/3/2012 21:41, schrieb Shawn Pearce: On Wed, Oct 3, 2012 at 11:55 AM, Jeff King p...@peff.net wrote: On Wed, Oct 03, 2012 at 11:53:35AM -0700, Junio C Hamano wrote: Jeff King p...@peff.net writes: Has there been

Re: upload-pack is slow with lots of refs

2012-10-04 Thread Sascha Cunz
Am Mittwoch, 3. Oktober 2012, 16:13:16 schrieb Jeff King: On Wed, Oct 03, 2012 at 12:41:38PM -0700, Shawn O. Pearce wrote: Out of curiosity, how are you thinking about triggering such a new behavior in a backwards-compatible way? Invoke git-upload-pack2, and fall back to reconnecting to

Re: upload-pack is slow with lots of refs

2012-10-04 Thread Jeff King
On Thu, Oct 04, 2012 at 11:52:13PM +0200, Sascha Cunz wrote: Would it be possible to use this workflow: - Every client connects per default to v1 - If server is capable of v2, it sends a flag along with the usual response (A v1 server will obviously not send that flag) That is more or

Re: upload-pack is slow with lots of refs

2012-10-03 Thread Nguyen Thai Ngoc Duy
On Wed, Oct 3, 2012 at 7:36 PM, Ævar Arnfjörð Bjarmason ava...@gmail.com wrote: I'm creating a system where a lot of remotes constantly fetch from a central repository for deployment purposes, but I've noticed that even with a remote.$name.fetch configuration to only get certain refs a git

Re: upload-pack is slow with lots of refs

2012-10-03 Thread Jeff King
On Wed, Oct 03, 2012 at 02:36:00PM +0200, Ævar Arnfjörð Bjarmason wrote: I'm creating a system where a lot of remotes constantly fetch from a central repository for deployment purposes, but I've noticed that even with a remote.$name.fetch configuration to only get certain refs a git fetch

Re: upload-pack is slow with lots of refs

2012-10-03 Thread Junio C Hamano
Jeff King p...@peff.net writes: Has there been any work on extending the protocol so that the client tells the server what refs it's interested in? I don't think so. It would be hard to do in a backwards-compatible way, because the advertisement is the first thing the server says, before it

Re: upload-pack is slow with lots of refs

2012-10-03 Thread Jeff King
On Wed, Oct 03, 2012 at 11:53:35AM -0700, Junio C Hamano wrote: Jeff King p...@peff.net writes: Has there been any work on extending the protocol so that the client tells the server what refs it's interested in? I don't think so. It would be hard to do in a backwards-compatible way,

Re: upload-pack is slow with lots of refs

2012-10-03 Thread Junio C Hamano
Ævar Arnfjörð Bjarmason ava...@gmail.com writes: I'm creating a system where a lot of remotes constantly fetch from a central repository for deployment purposes, but I've noticed that even with a remote.$name.fetch configuration to only get certain refs a git fetch will still call git-upload

Re: upload-pack is slow with lots of refs

2012-10-03 Thread Shawn Pearce
On Wed, Oct 3, 2012 at 11:55 AM, Jeff King p...@peff.net wrote: On Wed, Oct 03, 2012 at 11:53:35AM -0700, Junio C Hamano wrote: Jeff King p...@peff.net writes: Has there been any work on extending the protocol so that the client tells the server what refs it's interested in? I don't

Re: upload-pack is slow with lots of refs

2012-10-03 Thread Jeff King
On Wed, Oct 03, 2012 at 12:41:38PM -0700, Shawn O. Pearce wrote: Out of curiosity, how are you thinking about triggering such a new behavior in a backwards-compatible way? Invoke git-upload-pack2, and fall back to reconnecting to start git-upload-pack if it fails? Basically, yes. New

Re: upload-pack is slow with lots of refs

2012-10-03 Thread Ævar Arnfjörð Bjarmason
On Wed, Oct 3, 2012 at 8:03 PM, Jeff King p...@peff.net wrote: On Wed, Oct 03, 2012 at 02:36:00PM +0200, Ævar Arnfjörð Bjarmason wrote: I'm creating a system where a lot of remotes constantly fetch from a central repository for deployment purposes, but I've noticed that even with a

Re: upload-pack is slow with lots of refs

2012-10-03 Thread Jeff King
On Wed, Oct 03, 2012 at 10:16:56PM +0200, Ævar Arnfjörð Bjarmason wrote: I can't provide all the details now (not with access to that machine now), but briefly: * The git client/server version is 1.7.8 * The repository has around 50k refs, they're real refs, almost all of them (say

Re: upload-pack is slow with lots of refs

2012-10-03 Thread Ævar Arnfjörð Bjarmason
On Wed, Oct 3, 2012 at 11:20 PM, Jeff King p...@peff.net wrote: Thanks for all that info, it's really useful. * A co-worker who was working on this today tried it on 1.7.12 and claimed that it had the same performance characteristics. That's surprising to me. Can you try to verify those

Re: upload-pack is slow with lots of refs

2012-10-03 Thread Ævar Arnfjörð Bjarmason
On Wed, Oct 3, 2012 at 8:03 PM, Jeff King p...@peff.net wrote: What version of git are you using? In the past year or so, I've made several tweaks to speed up large numbers of refs, including: - cff38a5 (receive-pack: eliminate duplicate .have refs, v1.7.6); note that this only helps

Re: upload-pack is slow with lots of refs

2012-10-03 Thread Jeff King
On Thu, Oct 04, 2012 at 12:15:47AM +0200, Ævar Arnfjörð Bjarmason wrote: I think he was wrong, I tested this on git.git by first creating a lot of tags: parallel --eta git tag -a -m{} test-again-{} ::: $(git rev-list HEAD) Then doing: git pack-refs --all git repack -A -d

Re: upload-pack is slow with lots of refs

2012-10-03 Thread Jeff King
On Thu, Oct 04, 2012 at 12:32:35AM +0200, Ævar Arnfjörð Bjarmason wrote: On Wed, Oct 3, 2012 at 8:03 PM, Jeff King p...@peff.net wrote: What version of git are you using? In the past year or so, I've made several tweaks to speed up large numbers of refs, including: - cff38a5

Re: upload-pack is slow with lots of refs

2012-10-03 Thread Ævar Arnfjörð Bjarmason
On Thu, Oct 4, 2012 at 1:21 AM, Jeff King p...@peff.net wrote: On Thu, Oct 04, 2012 at 12:32:35AM +0200, Ævar Arnfjörð Bjarmason wrote: On Wed, Oct 3, 2012 at 8:03 PM, Jeff King p...@peff.net wrote: What version of git are you using? In the past year or so, I've made several tweaks to

Re: upload-pack is slow with lots of refs

2012-10-03 Thread Ævar Arnfjörð Bjarmason
On Thu, Oct 4, 2012 at 1:15 AM, Jeff King p...@peff.net wrote: On Thu, Oct 04, 2012 at 12:15:47AM +0200, Ævar Arnfjörð Bjarmason wrote: I think he was wrong, I tested this on git.git by first creating a lot of tags: parallel --eta git tag -a -m{} test-again-{} ::: $(git rev-list HEAD)