Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Jeff King
On Sun, May 05, 2013 at 05:38:53PM -0500, Felipe Contreras wrote: We don't care about blobs, or any object other than commits, but in order to find the type of object, we are parsing the whole thing, which is slow, specially in big repositories with lots of big files. I did a double-take on

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Junio C Hamano
Jeff King p...@peff.net writes: On Sun, May 05, 2013 at 05:38:53PM -0500, Felipe Contreras wrote: We don't care about blobs, or any object other than commits, but in order to find the type of object, we are parsing the whole thing, which is slow, specially in big repositories with lots of

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Junio C Hamano
Junio C Hamano gits...@pobox.com writes: By discarding marks on blobs, we may be robbing some optimization possibilities, and by discarding marks on tags, we may be robbing some features, from users of fast-export; we might want to add an option --use-object-marks={blob,commit,tag} or

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Jeff King
On Mon, May 06, 2013 at 08:08:45AM -0700, Junio C Hamano wrote: I'm also not sure why your claim we don't care about blobs is true, because naively we would want future runs of fast-export to avoid having to write out the whole blob content when mentioning the blob again. The existing

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Junio C Hamano
Jeff King p...@peff.net writes: On Mon, May 06, 2013 at 08:08:45AM -0700, Junio C Hamano wrote: I'm also not sure why your claim we don't care about blobs is true, because naively we would want future runs of fast-export to avoid having to write out the whole blob content when mentioning

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Jeff King
On Mon, May 06, 2013 at 09:32:41AM -0700, Junio C Hamano wrote: OK. If the argument is we do not write them, so do not bother reading them back in, I think that is reasonable. The way I read builtin/fast-export.c::import_marks() is that it is more like we do not write them, and we do not

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Junio C Hamano
Jeff King p...@peff.net writes: So yes, I think this is an obviously correct optimization. Thanks for clarifying, and sorry to be so slow. No need to be sorry. It just shows that the log message could have been more helpful. Here is what I tentatively queued. commit

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Jeff King
On Mon, May 06, 2013 at 10:17:41AM -0700, Junio C Hamano wrote: Jeff King p...@peff.net writes: So yes, I think this is an obviously correct optimization. Thanks for clarifying, and sorry to be so slow. No need to be sorry. It just shows that the log message could have been more

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Jeff King
On Mon, May 06, 2013 at 01:19:35PM -0400, Jeff King wrote: Here is what I tentatively queued. [...] Yeah, that is much for to understand (to me, at least). Ugh. That was supposed to be much easier to understand. Perhaps I will learn to type one day. -Peff -- To unsubscribe from this

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Felipe Contreras
On Mon, May 6, 2013 at 7:31 AM, Jeff King p...@peff.net wrote: On Sun, May 05, 2013 at 05:38:53PM -0500, Felipe Contreras wrote: We don't care about blobs, or any object other than commits, but in order to find the type of object, we are parsing the whole thing, which is slow, specially in

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Felipe Contreras
On Mon, May 6, 2013 at 10:08 AM, Junio C Hamano gits...@pobox.com wrote: Jeff King p...@peff.net writes: On Sun, May 05, 2013 at 05:38:53PM -0500, Felipe Contreras wrote: We don't care about blobs, or any object other than commits, but in order to find the type of object, we are parsing the

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Jeff King
On Mon, May 06, 2013 at 02:02:13PM -0500, Felipe Contreras wrote: I did a double-take on reading this subject line and first paragraph, thinking surely fast-export needs to actually output blobs?. If you think that, then you are not familiar with the code. --export-marks=file:: [...]

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Felipe Contreras
On Mon, May 6, 2013 at 11:20 AM, Jeff King p...@peff.net wrote: On Mon, May 06, 2013 at 08:08:45AM -0700, Junio C Hamano wrote: I'm also not sure why your claim we don't care about blobs is true, because naively we would want future runs of fast-export to avoid having to write out the

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Felipe Contreras
On Mon, May 6, 2013 at 2:11 PM, Jeff King p...@peff.net wrote: On Mon, May 06, 2013 at 02:02:13PM -0500, Felipe Contreras wrote: I did a double-take on reading this subject line and first paragraph, thinking surely fast-export needs to actually output blobs?. If you think that, then you

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Junio C Hamano
Felipe Contreras felipe.contre...@gmail.com writes: The story is different on the fast-import side, where we do say we dump the full table and a later run can depend on these marks. Yes, and gaining nothing but increased disk-space. I thought that the gaining nothing has already been refuted

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Junio C Hamano
Felipe Contreras felipe.contre...@gmail.com writes: How? * if we teach fast-import to optionally not write marks for blobs and trees out, your remote-bzr can take advantage of it, I already said remote-bzr is irrelevant. *Everybody* benefits. Everybody who does _not_ need to look at

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Felipe Contreras
On Mon, May 6, 2013 at 8:59 PM, Junio C Hamano gits...@pobox.com wrote: Felipe Contreras felipe.contre...@gmail.com writes: How? * if we teach fast-import to optionally not write marks for blobs and trees out, your remote-bzr can take advantage of it, I already said remote-bzr is