Re: non-smooth progress indication for git fsck and git gc

2018-09-06 Thread Jeff King
On Mon, Sep 03, 2018 at 06:48:54PM +0200, Ævar Arnfjörð Bjarmason wrote: > > And there are definitely a few nasty bits (like the way the progress is > > ended). I'm not planning on taking this further for now, but maybe > > you or somebody can find it interesting or useful. > > I think it would

Re: non-smooth progress indication for git fsck and git gc

2018-09-04 Thread Junio C Hamano
Jeff King writes: > That code isn't lib-ified enough to be run in process, but I think the > patch below should give similar behavior to what fsck currently does. > We'd need to tell index-pack to use our fsck.* config for its checks, I > imagine. The progress here is still per-pack, but I think

Re: non-smooth progress indication for git fsck and git gc

2018-09-03 Thread Ævar Arnfjörð Bjarmason
On Sun, Sep 02 2018, Jeff King wrote: > On Sun, Sep 02, 2018 at 03:55:28AM -0400, Jeff King wrote: > >> I still think the more interesting long-term thing here is to reuse the >> pack verification from index-pack, which actually hashes as it does the >> per-object countup. >> >> That code isn't

Re: non-smooth progress indication for git fsck and git gc

2018-09-02 Thread Jeff King
On Sun, Sep 02, 2018 at 03:55:28AM -0400, Jeff King wrote: > I still think the more interesting long-term thing here is to reuse the > pack verification from index-pack, which actually hashes as it does the > per-object countup. > > That code isn't lib-ified enough to be run in process, but I

Re: non-smooth progress indication for git fsck and git gc

2018-09-02 Thread Jeff King
On Sun, Sep 02, 2018 at 03:46:57AM -0400, Jeff King wrote: > Something like this, which chunks it there, uses a per-packfile meter > (though still does not give any clue how many packfiles there are), and > shows a throughput meter. Actually, in typical cases it would not matter how many

Re: non-smooth progress indication for git fsck and git gc

2018-09-02 Thread Jeff King
On Sat, Sep 01, 2018 at 02:53:28PM +0200, Ævar Arnfjörð Bjarmason wrote: > With this we'll get output like: > > $ ~/g/git/git -C ~/g/2015-04-03-1M-git/ --exec-path=$PWD fsck > Checking object directories: 100% (256/256), done. > Hashing: 100% (452634108/452634108), done. >

Re: non-smooth progress indication for git fsck and git gc

2018-09-01 Thread Ævar Arnfjörð Bjarmason
On Sat, Sep 01 2018, Ævar Arnfjörð Bjarmason wrote: > On Thu, Aug 16 2018, Ævar Arnfjörð Bjarmason wrote: > >> On Thu, Aug 16 2018, Jeff King wrote: >> >>> On Thu, Aug 16, 2018 at 08:54:25AM +0200, Ulrich Windl wrote: >>> I'd like to point out some minor issue observed while processing

Re: non-smooth progress indication for git fsck and git gc

2018-09-01 Thread Ævar Arnfjörð Bjarmason
On Thu, Aug 16 2018, Ævar Arnfjörð Bjarmason wrote: > On Thu, Aug 16 2018, Jeff King wrote: > >> On Thu, Aug 16, 2018 at 08:54:25AM +0200, Ulrich Windl wrote: >> >>> I'd like to point out some minor issue observed while processing some >>> 5-object repository with many binary objects, but

Re: Antw: Re: non-smooth progress indication for git fsck and git gc

2018-08-21 Thread Duy Nguyen
On Tue, Aug 21, 2018 at 3:13 AM Jeff King wrote: > I _think_ they should work together OK without further modification. > Once upon a time the caller had to say "don't show if we're past N% > after M seconds", but I think with the current code we'd just show it if > we're not completely finished

Re: Antw: Re: non-smooth progress indication for git fsck and git gc

2018-08-21 Thread Ulrich Windl
>>> Jeff King schrieb am 21.08.2018 um 03:07 in Nachricht <20180821010712.ga32...@sigill.intra.peff.net>: > On Mon, Aug 20, 2018 at 10:57:13AM +0200, Ævar Arnfjörð Bjarmason wrote: > [...] > So it really should just be a simple: > > progress = start_delayed_progress("Hashing packfile", 0); >

Re: Antw: Re: non-smooth progress indication for git fsck and git gc

2018-08-20 Thread Jeff King
On Mon, Aug 20, 2018 at 10:57:13AM +0200, Ævar Arnfjörð Bjarmason wrote: > > That seems to apply. BTW: Is there a way go get some repository statistics > > like a histogram of object sizes (or whatever that might be useful to help > > making decisions)? > > The git-sizer program is really

Re: Antw: Re: non-smooth progress indication for git fsck and git gc

2018-08-20 Thread Ulrich Windl
Hi! Here are some stats from the repository. First the fast import ones (which had good performance, but probably all cached, also): % git fast-import <../git-stream /usr/lib/git/git-fast-import statistics: - Alloc'd objects:

Re: Antw: Re: non-smooth progress indication for git fsck and git gc

2018-08-20 Thread Ævar Arnfjörð Bjarmason
On Mon, Aug 20 2018, Ulrich Windl wrote: Jeff King schrieb am 16.08.2018 um 22:55 in Nachricht > <20180816205556.ga8...@sigill.intra.peff.net>: >> On Thu, Aug 16, 2018 at 10:35:53PM +0200, Ævar Arnfjörð Bjarmason wrote: >> >>> This is all interesting, but I think unrelated to what Ulrich

Antw: Re: non-smooth progress indication for git fsck and git gc

2018-08-20 Thread Ulrich Windl
>>> Jeff King schrieb am 16.08.2018 um 22:55 in Nachricht <20180816205556.ga8...@sigill.intra.peff.net>: > On Thu, Aug 16, 2018 at 10:35:53PM +0200, Ævar Arnfjörð Bjarmason wrote: > >> This is all interesting, but I think unrelated to what Ulrich is talking >> about. Quote: >> >> Between

Antw: Re: non-smooth progress indication for git fsck and git gc

2018-08-20 Thread Ulrich Windl
>>> Duy Nguyen schrieb am 16.08.2018 um 17:18 in Nachricht : > On Thu, Aug 16, 2018 at 1:10 PM Ulrich Windl > wrote: >> >> Hi! >> >> I'd like to point out some minor issue observed while processing some > 5-object repository with many binary objects, but most are rather small: >> >> Between

Re: non-smooth progress indication for git fsck and git gc

2018-08-17 Thread Duy Nguyen
On Thu, Aug 16, 2018 at 11:08 PM Jeff King wrote: > > On Thu, Aug 16, 2018 at 04:55:56PM -0400, Jeff King wrote: > > > > * We spend the majority of the ~30s on this: > > > > > > https://github.com/git/git/blob/63749b2dea5d1501ff85bab7b8a7f64911d21dea/pack-check.c#L70-L79 > > > > This is

Re: non-smooth progress indication for git fsck and git gc

2018-08-16 Thread Junio C Hamano
Jeff King writes: > On Thu, Aug 16, 2018 at 11:57:14AM -0400, Jeff King wrote: > >> The only way to solve that is to count bytes. We don't have a total byte >> count in most cases, and it wouldn't always make sense (e.g., the >> "Compressing objects" meter can show the same issue, but it's not

Re: non-smooth progress indication for git fsck and git gc

2018-08-16 Thread Jeff King
On Thu, Aug 16, 2018 at 04:55:56PM -0400, Jeff King wrote: > > * We spend the majority of the ~30s on this: > > > > https://github.com/git/git/blob/63749b2dea5d1501ff85bab7b8a7f64911d21dea/pack-check.c#L70-L79 > > This is hashing the actual packfile. This is potentially quite long, >

Re: non-smooth progress indication for git fsck and git gc

2018-08-16 Thread Jeff King
On Thu, Aug 16, 2018 at 10:35:53PM +0200, Ævar Arnfjörð Bjarmason wrote: > This is all interesting, but I think unrelated to what Ulrich is talking > about. Quote: > > Between the two phases of "git fsck" (checking directories and > checking objects) there was a break of several seconds

Re: non-smooth progress indication for git fsck and git gc

2018-08-16 Thread Ævar Arnfjörð Bjarmason
On Thu, Aug 16 2018, Jeff King wrote: > On Thu, Aug 16, 2018 at 08:54:25AM +0200, Ulrich Windl wrote: > >> I'd like to point out some minor issue observed while processing some >> 5-object repository with many binary objects, but most are rather >> small: >> >> Between the two phases of

Re: non-smooth progress indication for git fsck and git gc

2018-08-16 Thread Jeff King
On Thu, Aug 16, 2018 at 11:57:14AM -0400, Jeff King wrote: > The only way to solve that is to count bytes. We don't have a total byte > count in most cases, and it wouldn't always make sense (e.g., the > "Compressing objects" meter can show the same issue, but it's not really > putting through

Re: non-smooth progress indication for git fsck and git gc

2018-08-16 Thread Jeff King
On Thu, Aug 16, 2018 at 05:18:51PM +0200, Duy Nguyen wrote: > > During "git gc" the writing objects phase did not update for some > > seconds, but then the percentage counter jumped like from 15% to > > 42%. > [...] > > Is it possible to make this repository public? You can also use "git >

Re: non-smooth progress indication for git fsck and git gc

2018-08-16 Thread Jeff King
On Thu, Aug 16, 2018 at 08:54:25AM +0200, Ulrich Windl wrote: > I'd like to point out some minor issue observed while processing some > 5-object repository with many binary objects, but most are rather > small: > > Between the two phases of "git fsck" (checking directories and > checking

Re: non-smooth progress indication for git fsck and git gc

2018-08-16 Thread Duy Nguyen
On Thu, Aug 16, 2018 at 1:10 PM Ulrich Windl wrote: > > Hi! > > I'd like to point out some minor issue observed while processing some > 5-object repository with many binary objects, but most are rather small: > > Between the two phases of "git fsck" (checking directories and checking >

non-smooth progress indication for git fsck and git gc

2018-08-16 Thread Ulrich Windl
Hi! I'd like to point out some minor issue observed while processing some 5-object repository with many binary objects, but most are rather small: Between the two phases of "git fsck" (checking directories and checking objects) there was a break of several seconds where no progress was