Re: clone performance regression
On Mon, Apr 26, 2021 at 01:48:48PM +, Kim Randell via Mercurial-devel wrote: > We have a large repository (>130,000 changesets, 165 heads, ~3.5GB > bundled with zstd). For the most part Mercurial performs very well for > us. However, we noticed a significant slowdown in cloning when > upgrading to Mercurial 5.7.1. Upon investigation I determined that this > was due to warming the tags cache, which was added (among updating all > caches) in this changeset: > https://www.mercurial-scm.org/repo/hg/rev/95a615dd77bf D10082 should fix this in principle, but it can't be used in the current form due to historic bugs. Joerg ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
mercurial@47011: 16 new changesets (16 on stable)
16 new changesets (16 on stable) in mercurial: https://www.mercurial-scm.org/repo/hg/rev/8d2b62d716b0 changeset: 46996:8d2b62d716b0 branch: stable tag: 5.8rc1 user:Joerg Sonnenberger date:Tue Apr 20 20:35:46 2021 +0200 summary: packaging: extract pre-computed version when running from plain tarball https://www.mercurial-scm.org/repo/hg/rev/106402245301 changeset: 46997:106402245301 branch: stable user:Matt Harbison date:Tue Apr 20 23:31:36 2021 -0400 summary: mail: add a TODO about proper mbox locking https://www.mercurial-scm.org/repo/hg/rev/65d18001cfbf changeset: 46998:65d18001cfbf branch: stable user:Pierre-Yves David date:Tue Apr 20 16:12:19 2021 +0200 summary: relnote: clarify the backward compatibility entry about p1/p2 swap https://www.mercurial-scm.org/repo/hg/rev/035c06f9fddf changeset: 46999:035c06f9fddf branch: stable user:Pierre-Yves David date:Tue Apr 20 16:18:43 2021 +0200 summary: relnode: document the move to `zstd` as default compression https://www.mercurial-scm.org/repo/hg/rev/c3eee5ad1de9 changeset: 47000:c3eee5ad1de9 branch: stable user:Pierre-Yves David date:Tue Apr 20 16:29:05 2021 +0200 summary: relnote: document the use of persistent-nodemap by default for Rust build https://www.mercurial-scm.org/repo/hg/rev/5be3e2edd449 changeset: 47001:5be3e2edd449 branch: stable user:Pierre-Yves David date:Tue Apr 20 16:31:13 2021 +0200 summary: relnote: mention a nodemap bug fixes https://www.mercurial-scm.org/repo/hg/rev/8fc2c867af54 changeset: 47002:8fc2c867af54 branch: stable parent: 46996:8d2b62d716b0 user:Augie Fackler date:Wed Apr 21 10:49:28 2021 -0400 summary: Added tag 5.8rc1 for changeset 8d2b62d716b0 https://www.mercurial-scm.org/repo/hg/rev/41f8f38d8377 changeset: 47003:41f8f38d8377 branch: stable user:Augie Fackler date:Wed Apr 21 10:49:29 2021 -0400 summary: Added signature for changeset 8d2b62d716b0 https://www.mercurial-scm.org/repo/hg/rev/3c8e4c6ec9bc changeset: 47004:3c8e4c6ec9bc branch: stable parent: 47003:41f8f38d8377 parent: 47001:5be3e2edd449 user:Augie Fackler date:Wed Apr 21 10:58:42 2021 -0400 summary: merge: stable heads https://www.mercurial-scm.org/repo/hg/rev/27602e030a1f changeset: 47005:27602e030a1f branch: stable user:Matt Harbison date:Sun Apr 18 00:56:09 2021 -0400 summary: incoming: use bytes for an error message https://www.mercurial-scm.org/repo/hg/rev/e050efe97fbe changeset: 47006:e050efe97fbe branch: stable user:Matt Harbison date:Wed Apr 14 09:49:36 2021 -0400 summary: typing: enable pytype processing on mercurial/upgrade_utils/actions.py https://www.mercurial-scm.org/repo/hg/rev/fceccc36873d changeset: 47007:fceccc36873d branch: stable user:Raphaël Gomès date:Wed Apr 21 17:59:14 2021 +0200 summary: lint: fix failing code check https://www.mercurial-scm.org/repo/hg/rev/77e73827a02d changeset: 47008:77e73827a02d branch: stable user:Joerg Sonnenberger date:Thu Apr 22 02:57:30 2021 +0200 summary: tests: don't hard-code /bin/bash https://www.mercurial-scm.org/repo/hg/rev/42eb8b7881b8 changeset: 47009:42eb8b7881b8 branch: stable user:Georges Racinet date:Sat Apr 24 15:46:39 2021 +0200 summary: repoview: style change in newtype() cache handling https://www.mercurial-scm.org/repo/hg/rev/76ae43d5b1db changeset: 47010:76ae43d5b1db branch: stable user:Georges Racinet date:Fri Apr 23 18:30:53 2021 +0200 summary: repoview: fix memory leak of filtered repo classes https://www.mercurial-scm.org/repo/hg/rev/b7e623ac98b6 changeset: 47011:b7e623ac98b6 branch: stable bookmark:@ tag: tip user:Georges Racinet date:Sat Apr 24 16:30:05 2021 +0200 summary: repoview: separate concerns in _filteredrepotypes comment -- Repository URL: https://www.mercurial-scm.org/repo/hg ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Failed pipeline for branch/stable | mercurial-devel | 0e3713d5
Pipeline #21231 has failed! Project: mercurial-devel ( https://foss.heptapod.net/octobus/mercurial-devel ) Branch: branch/stable ( https://foss.heptapod.net/octobus/mercurial-devel/-/commits/branch/stable ) Commit: 0e3713d5 ( https://foss.heptapod.net/octobus/mercurial-devel/-/commit/0e3713d50c775b94d607a19bef8089f7ae68a7d6 ) Commit Message: repoview: separate concerns in _filteredrepotyp... Commit Author: Georges Racinet ( https://foss.heptapod.net/gracinet ) Pipeline #21231 ( https://foss.heptapod.net/octobus/mercurial-devel/-/pipelines/21231 ) triggered by Raphaël Gomès ( https://foss.heptapod.net/raphael.gomes ) had 1 failed build. Job #190057 ( https://foss.heptapod.net/octobus/mercurial-devel/-/jobs/190057/raw ) Stage: tests Name: test-py2-rust -- You're receiving this email because of your account on foss.heptapod.net. ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Re: [PATCH 1 of 3 STABLE] repoview: style change in newtype() cache handling
Queued the series, thanks! On 4/26/21 12:19 PM, Georges Racinet wrote: # HG changeset patch # User Georges Racinet # Date 1619271999 -7200 # Sat Apr 24 15:46:39 2021 +0200 # Branch stable # Node ID 125fe190f484d396abc93428e3bd58b0b7455c24 # Parent 77e73827a02db25dc675b0afe552f24c807c741d # EXP-Topic memleak-repo-class repoview: style change in newtype() cache handling This way of writing it does not change the logic at all, but is more fit for the change we want to make in the next changeset. If anything, that's one dict lookup less in the hot path, but that should be non measurable. diff -r 77e73827a02d -r 125fe190f484 mercurial/repoview.py --- a/mercurial/repoview.py Thu Apr 22 02:57:30 2021 +0200 +++ b/mercurial/repoview.py Sat Apr 24 15:46:39 2021 +0200 @@ -472,10 +472,12 @@ def newtype(base): """Create a new type with the repoview mixin and the given base class""" -if base not in _filteredrepotypes: +cls = _filteredrepotypes.get(base) +if cls is not None: +return cls -class filteredrepo(repoview, base): -pass +class filteredrepo(repoview, base): +pass -_filteredrepotypes[base] = filteredrepo -return _filteredrepotypes[base] +_filteredrepotypes[base] = filteredrepo +return filteredrepo ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
clone performance regression
Hello, We have a large repository (>130,000 changesets, 165 heads, ~3.5GB bundled with zstd). For the most part Mercurial performs very well for us. However, we noticed a significant slowdown in cloning when upgrading to Mercurial 5.7.1. Upon investigation I determined that this was due to warming the tags cache, which was added (among updating all caches) in this changeset: https://www.mercurial-scm.org/repo/hg/rev/95a615dd77bf I posted about this in the general list, but didn't fully understand the issue at that point, and having investigated further, thought I'd try the developer list now - apologies if I'm using the wrong list. As you can see in the attached logs (using Powershell syntax), warming the cache adds ~10 seconds to a clone of the Mercurial source (https://www.mercurial-scm.org/repo/hg) on my machine, dominating the processing time for a hardlink clone. If the cache has already been warmed on the source repo, and it's a local non-pull clone, then the cache copy mostly eliminates this. However, it is impossible to eliminate this for a pull clone. In our case the additional time is over 20 minutes, approximately doubling the time to clone. For https://hg.mozilla.org/mozilla-central it adds around 3 minutes. Warming the tags cache doesn't seem to provide any benefit for a simple hg tags call, which takes under a tenth of a second in all cases; presumably there will be some situations in which it is beneficial (perhaps creating a new head from a non-head?), but it doesn't seem worth slowing down the clone this much. Warming the other caches doesn't appear in the profile, so perhaps those could still be done on clone, just not the tags. Best regards, Kim Randell > pip install mercurial==5.6.1 > hg clone -U https://www.mercurial-scm.org/repo/hg > hg -R hg debugtagscache 0 9117c6561b0bd7792fa13b50d28239d51b78e51f missing/invalid ... all missing/invalid 46994 e846261413d0dfc2c0bb696c830f10fe9e586e1d missing/invalid 46995 01264552a1dc163161a334c2568f3a68bf06e355 6379bb39250b74feb34799d2004fac938d859ed6 > hg -R hg tags --time > $null time: real 0.089 secs (user 0.047+0.000 sys 0.016+0.000) > hg clone -U --profile hg hg2 | 100.0% 1.11s dispatch.py:_callcatchline 43: dispatch.run() | 100.0% 1.11s scmutil.py: callcatch line 488: return scmutil.callcatch(ui... | 100.0% 1.11s dispatch.py:_runcatchfunc line 153: return func() | 100.0% 1.11s extensions.py: loadall line 469: return _dispatch(req) | 94.4% 1.05s share.py: clone line 189: return orig(ui, repo, *args... | 94.4% 1.05s commands.py:clone line 139: return orig(ui, source, *ar... | 94.4% 1.05s hg.py: clone line 1908: depth=opts.get(b'depth') or... | 87.3% 0.97s hg.py: copystore line 853: destlock = copystore(ui, sr... | 87.3% 0.97s util.py:copyfiles line 472: srcvfs.join(f), dstvfs.join... | 87.3% 0.97s util.py:copyfiles line 1967: hardlink, n = copyfiles(src... | 84.5% 0.94s util.py:copyfiles line 1967: hardlink, n = copyfiles(src... \ 42.3% 0.47s win32.py: oslinkline 1979: oslink(src, dst) \ 35.2% 0.39s util.py:copyfiles line 1967: hardlink, n = copyfiles(src... \ 19.7% 0.22s win32.py: oslink line 1979: oslink(src, dst) \ 11.3% 0.12s util.py:copyfiles line 1967: hardlink, n = copyfiles(src... | 8.5% 0.09s win32.py: oslinkline 1979: oslink(src, dst) --- Sample count: 60 Total time: 1.109375 seconds (0.00 wall) > hg -R hg2 tags --time > $null time: real 0.040 secs (user 0.047+0.000 sys 0.000+0.000) > hg clone -U --profile --pull hg hg3 requesting all changes adding changesets adding manifests adding file changes added 46996 changesets with 91887 changes to 3681 files new changesets 9117c6561b0b:01264552a1dc \ 86.5% 34.02s dispatch.py:_callcatchline 43: dispatch.run() | 86.5% 34.02s scmutil.py: callcatch line 488: return scmutil.callcatch(ui... | 86.5% 34.02s dispatch.py:_runcatchfunc line 153: return func() | 86.5% 34.02s extensions.py: loadall line 469: return _dispatch(req) | 86.3% 33.92s share.py: clone line 189: return orig(ui, repo, *args... | 86.3% 33.92s commands.py:clone line 139: return orig(ui, source, *ar... | 86.3% 33.92s hg.py: clone line 1908: depth=opts.get(b'depth') or... | 86.2% 33.89s exchange.py:pullline 935: depth=depth, | 82.7% 32.50s exchange.py:_fullpullbundle2line 1827: _fullpullbundle2(repo, pullop) | 82.7% 32.50s exchange.py:_pullbundle2line 1693: _pullbundle2(pullop) | 81.7% 32.12s bundle2.py: processbundle line 2030: bundle2.processbundle(pullo... | 81.7% 32.12s bundle2.py: processparts
