D4502: util: allow lrucachedict to track cost of entries
lothiraldan added inline comments. INLINE COMMENTS > indygreg wrote in util.py:1277 > Good catch! I'll send a revised patch. > > FWIW, cost accounting on this data structure opens up a lot of potential > around caching on revlogs. I have some alpha-quality commits to replace the > full revision cache on the revlog with an `lrucachedict` and to add a > decompressed chunk cache to revlogs. Such caches can speed up certain > operations drastically. However, we need to be careful about adding always-on > caches to revlogs because they can result in memory bloat. I was thinking > about adding context manager methods to revlogs to temporarily activate > certain aggressive caches in order to facilitate certain operations. e.g. a > fulltext or chunk cache when applying delta groups could make it drastically > faster to compute deltas during bulk insertion. A chunk cache could make > reverse walks significantly faster. Etc. I figured you'd be interested given > recent work in this area :) Having a weighted cache would combine well with our work on intermediate snapshots. If we can keep the right intermediate snapshots in the cache we will get a lot of useful cache hit. REPOSITORY rHG Mercurial REVISION DETAIL https://phab.mercurial-scm.org/D4502 To: indygreg, #hg-reviewers Cc: lothiraldan, mercurial-devel ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 19 of 19] sparse-revlog: set max delta chain length to on thousand
# HG changeset patch # User Boris Feld # Date 1536333525 14400 # Fri Sep 07 11:18:45 2018 -0400 # Node ID 113d5204173704d2ec31719eacdb40e2f88bee39 # Parent f8b71b0b9d035171dae9162f87747ce02da4b086 # EXP-Topic sparse-snapshot # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 113d52041737 sparse-revlog: set max delta chain length to on thousand The new snapshot system used in the sparse-revlog case gave us some small size benefit so far. However its most important property is to gracefully handle harder limit on delta chainlength. Long delta chain has a very detrimental impact on read (and write) performance in revlog. Being able to shorter them provide a great boost. However, shorting delta used to result significantly lower compression ratio. The intermediate snapshots effectively suppress most of this effect (even all in some case). # Effect on the test repository The repository we use for test is not "realistic" but can still show this in action using an unreasonably low chain limit. Limiting the chain length show a sizeable increase but stay under control: +6% for limit=15; +15% for limit=10. Without the snapshot system the increase is significantly bigger: +45% for limit=15; +80% for limit=10. Even slightly larger than without delta chain limit, the resulting size is still smaller than before we started doing snapshots. Here is a table for comparison. *Since the repository is not branchy, the initial sparse-revlog version does not bring much benefit compare to the non-sparse one): chain length limit | none | limit=15 | limit=10 | without sparse-revlog | 62 818 987 | 112 664 615 | 131 222 574 | without snapshot | 74 365 490 | 108 211 410 | 133 857 764 | withsnapshot | 59 230 936 | 63 002 924 | 68 415 329 | # Effect On Real Life Repositories The series provides significant benefits on all kind of repositories. Using `hg debugupgraderepo -o redeltaparent --run`, we recomputed delta chain for various repositories with different settings: - delta chain length: unlimited or 1000 limit - sparse-revlog: enabled or disabled - this series:applied or not applied We can observe multiple types of effect: - On very branchy repositories: * The delta chain limit as low impact on the repo size. * Intermediate snapshot greatly reduces manifest size: - pypy: -80% - netbeans: -95% * The delta chain limit is effective, without a size impact: - netbeans average: 613 -> 282 - private #1 average: 1 068 -> 307 - On more linear repository: * Intermediate snapshot limit the impact of delta chain limit: - mozilla: without the series: +360% with the series: +25% * The delta chain limit provides large improvement: - mozilla's average chain length: unlimited: 15 338 limited: 469 * Despite the chain length limit, the manifest size is reduced: - mercurial: -25% - mozilla: -30% It is clear that the use of chains of intermediate snapshots provide large benefits both in storage size and delta chains quality. We should now switch our effort toward making sure the write performance are acceptable. Then, `sparse-revlog` will be a suitable format for all new repository. # Raw Statistic * no-sparse: general delta repository not using sparse-revlog * no-snapshot: sparse-revlog repository not using this series * snapshot:sparse-revlog repository using this series mercurial Manifest Size: limit | none|1000 |-| no-sparse | 8 021 373 | 8 199 366 no-snapshot | 8 103 561 | 8 259 719 snapshot| 6 137 116 | 6 126 433 Manifest Chain length data limit || none|| 1000|| value || average | max || average | max || ||-|-||-|-|| no-sparse || 307 |1456 || 279 |1000 || no-snapshot || 312 |1456 || 283 |1000 || snapshot|| 248 |1208 || 241 |1000 || Full Store Size limit | none|1000 |-| no-sparse | 51 013 198 | 51 201 574 no-snapshot | 50 930 795 | 51 141 006 snapshot| 48 072 037 | 48 093 572 pypy Manifest Size: limit | none|1000 |-| no-sparse | 193 987 784 | 193 987 784 no-snapshot | 163 171 745 | 163 312 229 snapshot| 34 605 900 | 34 600 750 Manifest Chain length data limit || none|| 1000|| value || average | max || average | max ||
[PATCH 18 of 19] snapshot: also consider the snapshot chain of one unrelated revision
# HG changeset patch # User Boris Feld # Date 1536333525 14400 # Fri Sep 07 11:18:45 2018 -0400 # Node ID f8b71b0b9d035171dae9162f87747ce02da4b086 # Parent 535a778ae14a756b207901db5b3c0e5a078ae6cd # EXP-Topic sparse-snapshot # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r f8b71b0b9d03 snapshot: also consider the snapshot chain of one unrelated revision To maximize the chance of good delta chain reuse, we inject an unrelated delta chain into our search. To do so, we search for the highest revision unrelated to the parents of the current revision and use its snapshot chain too. Adding this extra snapshot into the mix can have a performance impact. We'll deal with performance impact in a later series. diff --git a/mercurial/revlogutils/deltas.py b/mercurial/revlogutils/deltas.py --- a/mercurial/revlogutils/deltas.py +++ b/mercurial/revlogutils/deltas.py @@ -719,6 +719,36 @@ def _rawgroups(revlog, p1, p2, cachedelt parents_snaps[idx].add(s) snapfloor = min(parents_snaps[0]) + 1 _findsnapshots(revlog, snapshots, snapfloor) +# search for the highest "unrelated" revision +# +# Adding snapshots used by "unrelated" revision increase the odd we +# reuse an independant, yet better snapshot chain. +# +# XXX instead of building a set of revisions, we could lazily enumerate +# over the chains. That would be more efficient, however we stick to +# simple code for now. +all_revs = set() +for chain in candidate_chains: +all_revs.update(chain) +other = None +for r in revlog.revs(prev, snapfloor): +if r not in all_revs: +other = r +break +if other is not None: +# To avoid unfair competition, we won't use unrelated intermediate +# snapshot that are deeper than the ones from the parent delta +# chain. +max_depth = max(parents_snaps.keys()) +chain = deltachain(other) +for idx, s in enumerate(chain): +if s < snapfloor: +continue +if max_depth < idx: +break +if not revlog.issnapshot(s): +break +parents_snaps[idx].add(s) # Test them as possible intermediate snapshot base # We test them from highest to lowest level. High level one are more # likely to result in small delta @@ -756,9 +786,10 @@ def _rawgroups(revlog, p1, p2, cachedelt # more and more snapshot as the repository grow. yield tuple(snapshots[nullrev]) -# other approach failed try against prev to hopefully save us a -# fulltext. -yield (prev,) +if not sparse: +# other approach failed try against prev to hopefully save us a +# fulltext. +yield (prev,) class deltacomputer(object): def __init__(self, revlog): diff --git a/tests/test-sparse-revlog.t b/tests/test-sparse-revlog.t --- a/tests/test-sparse-revlog.t +++ b/tests/test-sparse-revlog.t @@ -77,7 +77,7 @@ repeatedly while some of it changes rare $ f -s .hg/store/data/*.d - .hg/store/data/_s_p_a_r_s_e-_r_e_v_l_o_g-_t_e_s_t-_f_i_l_e.d: size=59302280 + .hg/store/data/_s_p_a_r_s_e-_r_e_v_l_o_g-_t_e_s_t-_f_i_l_e.d: size=59230936 $ hg debugrevlog * format : 1 flags : generaldelta @@ -89,45 +89,45 @@ repeatedly while some of it changes rare empty :0 ( 0.00%) text :0 (100.00%) delta :0 (100.00%) - snapshot : 168 ( 3.36%) -lvl-0 : 4 ( 0.08%) -lvl-1 : 18 ( 0.36%) -lvl-2 : 39 ( 0.78%) -lvl-3 : 54 ( 1.08%) -lvl-4 : 53 ( 1.06%) - deltas: 4833 (96.64%) - revision size : 59302280 - snapshot : 5833942 ( 9.84%) -lvl-0 : 804068 ( 1.36%) -lvl-1 :1378470 ( 2.32%) -lvl-2 :1608138 ( 2.71%) -lvl-3 :1222158 ( 2.06%) -lvl-4 : 821108 ( 1.38%) - deltas: 53468338 (90.16%) + snapshot : 176 ( 3.52%) +lvl-0 : 3 ( 0.06%) +lvl-1 : 17 ( 0.34%) +lvl-2 : 45 ( 0.90%) +lvl-3 : 56 ( 1.12%) +lvl-4 : 55 ( 1.10%) + deltas: 4825 (96.48%) + revision size : 59230936 + snapshot : 5770371 ( 9.74%) +lvl-0 : 602962 ( 1.02%) +lvl-1 :1534153 ( 2.59%) +lvl-2 :1604445 ( 2.71%) +lvl-3 :1218174 ( 2.06%) +lvl-4 : 810637 ( 1.37%) + deltas: 53460565 (90.26%) chunks: 5001 0x78 (x) : 5001 (100.00%) -
[PATCH 17 of 19] snapshot: extract parent chain computation
# HG changeset patch # User Boris Feld # Date 1536333457 14400 # Fri Sep 07 11:17:37 2018 -0400 # Node ID 535a778ae14a756b207901db5b3c0e5a078ae6cd # Parent 2ad2e932a5e2bb5bbf2bac7b7d3870ba969c07d6 # EXP-Topic sparse-snapshot # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 535a778ae14a snapshot: extract parent chain computation The final step of this series is to include chain related to "prev" in the search. Before adding that code we do some simple code movement to clarify the next diff. diff --git a/mercurial/revlogutils/deltas.py b/mercurial/revlogutils/deltas.py --- a/mercurial/revlogutils/deltas.py +++ b/mercurial/revlogutils/deltas.py @@ -711,8 +711,9 @@ def _rawgroups(revlog, p1, p2, cachedelt # search for snapshot in parents delta chain # map: snapshot-level: snapshot-rev parents_snaps = collections.defaultdict(set) -for p in parents: -for idx, s in enumerate(deltachain(p)): +candidate_chains = [deltachain(p) for p in parents] +for chain in candidate_chains: +for idx, s in enumerate(chain): if not revlog.issnapshot(s): break parents_snaps[idx].add(s) ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 16 of 19] snapshot: refine candidate snapshot base upward
# HG changeset patch # User Boris Feld # Date 1536333456 14400 # Fri Sep 07 11:17:36 2018 -0400 # Node ID 2ad2e932a5e2bb5bbf2bac7b7d3870ba969c07d6 # Parent 45a29cbc741ec23c1c3c98227c5ca2fcf6f0a3ff # EXP-Topic sparse-snapshot # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 2ad2e932a5e2 snapshot: refine candidate snapshot base upward Once we found a suitable snapshot base it is useful to check if it has a "children" snapshot that would provide a better diff. This is useful when base not directly related to stored revision are picked. In those case, we "jumped" to this new chain at an arbitrary point, checking if a higher point is more appropriate will help to provide better results and increase snapshot reuse. diff --git a/mercurial/revlogutils/deltas.py b/mercurial/revlogutils/deltas.py --- a/mercurial/revlogutils/deltas.py +++ b/mercurial/revlogutils/deltas.py @@ -658,6 +658,19 @@ def _refinedgroups(revlog, p1, p2, cache if base == nullrev: break good = yield (base,) +# refine snapshot up +# +# XXX the _findsnapshots call can be expensive and is "duplicated" with +# the one done in `_rawgroups`. Once we start working on performance, +# we should make the two logics share this computation. +snapshots = collections.defaultdict(list) +_findsnapshots(revlog, snapshots, good + 1) +previous = None +while good != previous: +previous = good +children = tuple(sorted(c for c in snapshots[good])) +good = yield children + # we have found nothing yield None diff --git a/tests/test-sparse-revlog.t b/tests/test-sparse-revlog.t --- a/tests/test-sparse-revlog.t +++ b/tests/test-sparse-revlog.t @@ -77,7 +77,7 @@ repeatedly while some of it changes rare $ f -s .hg/store/data/*.d - .hg/store/data/_s_p_a_r_s_e-_r_e_v_l_o_g-_t_e_s_t-_f_i_l_e.d: size=59303048 + .hg/store/data/_s_p_a_r_s_e-_r_e_v_l_o_g-_t_e_s_t-_f_i_l_e.d: size=59302280 $ hg debugrevlog * format : 1 flags : generaldelta @@ -89,45 +89,45 @@ repeatedly while some of it changes rare empty :0 ( 0.00%) text :0 (100.00%) delta :0 (100.00%) - snapshot : 165 ( 3.30%) + snapshot : 168 ( 3.36%) lvl-0 : 4 ( 0.08%) -lvl-1 : 17 ( 0.34%) -lvl-2 : 46 ( 0.92%) -lvl-3 : 62 ( 1.24%) -lvl-4 : 36 ( 0.72%) - deltas: 4836 (96.70%) - revision size : 59303048 - snapshot : 6105443 (10.30%) -lvl-0 : 804187 ( 1.36%) -lvl-1 :1476228 ( 2.49%) -lvl-2 :1752567 ( 2.96%) -lvl-3 :1461776 ( 2.46%) -lvl-4 : 610685 ( 1.03%) - deltas: 53197605 (89.70%) +lvl-1 : 18 ( 0.36%) +lvl-2 : 39 ( 0.78%) +lvl-3 : 54 ( 1.08%) +lvl-4 : 53 ( 1.06%) + deltas: 4833 (96.64%) + revision size : 59302280 + snapshot : 5833942 ( 9.84%) +lvl-0 : 804068 ( 1.36%) +lvl-1 :1378470 ( 2.32%) +lvl-2 :1608138 ( 2.71%) +lvl-3 :1222158 ( 2.06%) +lvl-4 : 821108 ( 1.38%) + deltas: 53468338 (90.16%) chunks: 5001 0x78 (x) : 5001 (100.00%) - chunks size : 59303048 - 0x78 (x) : 59303048 (100.00%) + chunks size : 59302280 + 0x78 (x) : 59302280 (100.00%) avg chain length : 17 max chain length : 45 - max chain reach : 26194433 + max chain reach : 22744720 compression ratio : 29 uncompressed data size (min/max/avg) : 346468 / 346472 / 346471 - full revision size (min/max/avg) : 200992 / 201080 / 201046 - inter-snapshot size (min/max/avg): 11610 / 172762 / 32927 - level-1 (min/max/avg) : 15619 / 172762 / 86836 - level-2 (min/max/avg) : 13055 / 85219 / 38099 - level-3 (min/max/avg) : 11610 / 42645 / 23577 - level-4 (min/max/avg) : 12928 / 20205 / 16963 - delta size (min/max/avg) : 10649 / 106863 / 11000 + full revision size (min/max/avg) : 200985 / 201050 / 201017 + inter-snapshot size (min/max/avg): 11598 / 163304 / 30669 + level-1 (min/max/avg) : 15616 / 163304 / 76581 + level-2 (min/max/avg) : 11602 / 86428 / 41234 + level-3 (min/max/avg) : 11598 / 42390 / 22632 + level-4 (min/max/avg) : 11603 / 19649 / 15492 + delta size (min/max/avg) : 10649 / 105465 / 11063 - deltas against prev : 4162 (86.06%) - where prev = p1 : 4120 (98.99%) + deltas against prev
[PATCH 15 of 19] snapshot: try to refine new snapshot base down the chain
# HG changeset patch # User Boris Feld # Date 1536333455 14400 # Fri Sep 07 11:17:35 2018 -0400 # Node ID 45a29cbc741ec23c1c3c98227c5ca2fcf6f0a3ff # Parent cc72a5b2135b94e1176eaf7a71b5b577bb7b7a33 # EXP-Topic sparse-snapshot # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 45a29cbc741e snapshot: try to refine new snapshot base down the chain There are cases where doing a diff against a snapshot's parent will be shorter than against the snapshot itself. Reusing snapshot not directly related to the revision we are trying to store increase this odd. So once we found a possible candidate, we check the snapshots lower in the chain. This will involve extra processing, but this extra processing will only happen when we are doing building a snapshot, a rare situation. diff --git a/mercurial/revlogutils/deltas.py b/mercurial/revlogutils/deltas.py --- a/mercurial/revlogutils/deltas.py +++ b/mercurial/revlogutils/deltas.py @@ -647,6 +647,17 @@ def _refinedgroups(revlog, p1, p2, cache good = yield candidates if good is not None: break + +# if we have a refinable value, try to refine it +if good is not None and good not in (p1, p2) and revlog.issnapshot(good): +# refine snapshot down +previous = None +while previous != good: +previous = good +base = revlog.deltaparent(good) +if base == nullrev: +break +good = yield (base,) # we have found nothing yield None ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 14 of 19] snapshot: make sure we'll never refine delta base from a reused source
# HG changeset patch # User Boris Feld # Date 1536333454 14400 # Fri Sep 07 11:17:34 2018 -0400 # Node ID cc72a5b2135b94e1176eaf7a71b5b577bb7b7a33 # Parent 4c849d34990166b1246426533efa517733d8a7fe # EXP-Topic sparse-snapshot # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r cc72a5b2135b snapshot: make sure we'll never refine delta base from a reused source The point of reusing delta from the source is to avoid doing computation when applying a bundle. Refining such delta would go against that spirit. We do not have refining logic in place yet. This code needed to be moved out of the way before we could start adding such logic. diff --git a/mercurial/revlogutils/deltas.py b/mercurial/revlogutils/deltas.py --- a/mercurial/revlogutils/deltas.py +++ b/mercurial/revlogutils/deltas.py @@ -630,6 +630,19 @@ def _findsnapshots(revlog, cache, start_ def _refinedgroups(revlog, p1, p2, cachedelta): good = None +# First we try to reuse a the delta contained in the bundle. +# (or from the source revlog) +# +# This logic only applies to general delta repositories and can be disabled +# through configuration. Disabling reuse source delta is useful when +# we want to make sure we recomputed "optimal" deltas. +if cachedelta and revlog._generaldelta and revlog._lazydeltabase: +# Assume what we received from the server is a good choice +# build delta will reuse the cache +good = yield (cachedelta[0],) +if good is not None: +yield None +return for candidates in _rawgroups(revlog, p1, p2, cachedelta): good = yield candidates if good is not None: @@ -651,17 +664,6 @@ def _rawgroups(revlog, p1, p2, cachedelt prev = curr - 1 deltachain = lambda rev: revlog._deltachain(rev)[0] -# First we try to reuse a the delta contained in the bundle. -# (or from the source revlog) -# -# This logic only applies to general delta repositories and can be disabled -# through configuration. Disabling reuse of source delta is useful when -# we want to make sure we recomputed "optimal" deltas. -if cachedelta and gdelta and revlog._lazydeltabase: -# Assume what we received from the server is a good choice -# build delta will reuse the cache -yield (cachedelta[0],) - if gdelta: # exclude already lazy tested base if any parents = [p for p in (p1, p2) if p != nullrev] ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 13 of 19] snapshot: turn _refinedgroups into a coroutine
# HG changeset patch # User Boris Feld # Date 1536333454 14400 # Fri Sep 07 11:17:34 2018 -0400 # Node ID 4c849d34990166b1246426533efa517733d8a7fe # Parent f79dc24022976ab6ba0a8f7183651e0e4d9877d5 # EXP-Topic sparse-snapshot # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 4c849d349901 snapshot: turn _refinedgroups into a coroutine We are now almost ready to start adding refining logic. diff --git a/mercurial/revlogutils/deltas.py b/mercurial/revlogutils/deltas.py --- a/mercurial/revlogutils/deltas.py +++ b/mercurial/revlogutils/deltas.py @@ -589,7 +589,7 @@ def _candidategroups(revlog, textlen, p1 tested = set([nullrev]) candidates = _refinedgroups(revlog, p1, p2, cachedelta) while True: -temptative = next(candidates) +temptative = candidates.send(good) if temptative is None: break group = [] @@ -618,8 +618,6 @@ def _candidategroups(revlog, textlen, p1 # impacting performances. Some bounding or slicing mecanism # would help to reduce this impact. good = yield tuple(group) -if good is not None: -break yield None def _findsnapshots(revlog, cache, start_rev): ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 12 of 19] snapshot: also use None as a stop value for `_refinegroup`
# HG changeset patch # User Boris Feld # Date 1536333453 14400 # Fri Sep 07 11:17:33 2018 -0400 # Node ID f79dc24022976ab6ba0a8f7183651e0e4d9877d5 # Parent db3775d10e1167be3487eed8b915ca2bf8c4bccb # EXP-Topic sparse-snapshot # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r f79dc2402297 snapshot: also use None as a stop value for `_refinegroup` This is yet another small step toward turning `_refinegroups` into a co-routine. diff --git a/mercurial/revlogutils/deltas.py b/mercurial/revlogutils/deltas.py --- a/mercurial/revlogutils/deltas.py +++ b/mercurial/revlogutils/deltas.py @@ -587,7 +587,11 @@ def _candidategroups(revlog, textlen, p1 deltas_limit = textlen * LIMIT_DELTA2TEXT tested = set([nullrev]) -for temptative in _refinedgroups(revlog, p1, p2, cachedelta): +candidates = _refinedgroups(revlog, p1, p2, cachedelta) +while True: +temptative = next(candidates) +if temptative is None: +break group = [] for rev in temptative: # skip over empty delta (no need to include them in a chain) @@ -632,6 +636,8 @@ def _refinedgroups(revlog, p1, p2, cache good = yield candidates if good is not None: break +# we have found nothing +yield None def _rawgroups(revlog, p1, p2, cachedelta): """Provides group of revision to be tested as delta base ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 11 of 19] snapshot: add refining logic at the findeltainfo level
# HG changeset patch # User Boris Feld # Date 1536333453 14400 # Fri Sep 07 11:17:33 2018 -0400 # Node ID db3775d10e1167be3487eed8b915ca2bf8c4bccb # Parent 256a998ee3bf645cc6eee3a144e6825d61406a7c # EXP-Topic sparse-snapshot # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r db3775d10e11 snapshot: add refining logic at the findeltainfo level Once we found a delta, we want to have the candidates logic challenge it, searching for a better candidate. The logic at the lower level is still missing. We'll introduce it later. Adding small changes in individual commits make it simpler to explain the code change. This is another small step toward turning `_refinegroups` into a co-routine. diff --git a/mercurial/revlogutils/deltas.py b/mercurial/revlogutils/deltas.py --- a/mercurial/revlogutils/deltas.py +++ b/mercurial/revlogutils/deltas.py @@ -582,6 +582,7 @@ def _candidategroups(revlog, textlen, p1 deltalength = revlog.length deltaparent = revlog.deltaparent +good = None deltas_limit = textlen * LIMIT_DELTA2TEXT @@ -612,7 +613,9 @@ def _candidategroups(revlog, textlen, p1 # XXX: in the sparse revlog case, group can become large, # impacting performances. Some bounding or slicing mecanism # would help to reduce this impact. -yield tuple(group) +good = yield tuple(group) +if good is not None: +break yield None def _findsnapshots(revlog, cache, start_rev): @@ -847,14 +850,20 @@ class deltacomputer(object): candidaterevs = next(groups) while candidaterevs is not None: nominateddeltas = [] +if deltainfo is not None: +# if we already found a good delta, +# challenge it against refined candidates +nominateddeltas.append(deltainfo) for candidaterev in candidaterevs: candidatedelta = self._builddeltainfo(revinfo, candidaterev, fh) if isgooddeltainfo(self.revlog, candidatedelta, revinfo): nominateddeltas.append(candidatedelta) if nominateddeltas: deltainfo = min(nominateddeltas, key=lambda x: x.deltalen) -break -candidaterevs = next(groups) +if deltainfo is not None: +candidaterevs = groups.send(deltainfo.base) +else: +candidaterevs = next(groups) if deltainfo is None: deltainfo = self._fullsnapshotinfo(fh, revinfo) ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 10 of 19] snapshot: use None as a stop value when looking for a good delta
# HG changeset patch # User Boris Feld # Date 1536333452 14400 # Fri Sep 07 11:17:32 2018 -0400 # Node ID 256a998ee3bf645cc6eee3a144e6825d61406a7c # Parent 9ea2ef9abc22291b021b80c8f88320026ae79618 # EXP-Topic sparse-snapshot # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 256a998ee3bf snapshot: use None as a stop value when looking for a good delta Having clear stop value should help keep clear logic around the co-routine. The alternative of using a StopIteration exception give a messier result. This is one small step toward turning `_refinegroups` into a co-routine. diff --git a/mercurial/revlogutils/deltas.py b/mercurial/revlogutils/deltas.py --- a/mercurial/revlogutils/deltas.py +++ b/mercurial/revlogutils/deltas.py @@ -577,6 +577,7 @@ def _candidategroups(revlog, textlen, p1 """ # should we try to build a delta? if not (len(revlog) and revlog._storedeltachains): +yield None return deltalength = revlog.length @@ -612,6 +613,7 @@ def _candidategroups(revlog, textlen, p1 # impacting performances. Some bounding or slicing mecanism # would help to reduce this impact. yield tuple(group) +yield None def _findsnapshots(revlog, cache, start_rev): """find snapshot from start_rev to tip""" @@ -842,7 +844,8 @@ class deltacomputer(object): p1r, p2r = revlog.rev(p1), revlog.rev(p2) groups = _candidategroups(self.revlog, revinfo.textlen, p1r, p2r, cachedelta) -for candidaterevs in groups: +candidaterevs = next(groups) +while candidaterevs is not None: nominateddeltas = [] for candidaterev in candidaterevs: candidatedelta = self._builddeltainfo(revinfo, candidaterev, fh) @@ -851,6 +854,7 @@ class deltacomputer(object): if nominateddeltas: deltainfo = min(nominateddeltas, key=lambda x: x.deltalen) break +candidaterevs = next(groups) if deltainfo is None: deltainfo = self._fullsnapshotinfo(fh, revinfo) ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 09 of 19] snapshot: introduce an intermediate `_refinedgroups` generator
# HG changeset patch # User Boris Feld # Date 1536333452 14400 # Fri Sep 07 11:17:32 2018 -0400 # Node ID 9ea2ef9abc22291b021b80c8f88320026ae79618 # Parent 979ae7a8db741f15531f76dded568e0792e57b10 # EXP-Topic sparse-snapshot # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 9ea2ef9abc22 snapshot: introduce an intermediate `_refinedgroups` generator This method will be used to improve the search for a good snapshot base. To keep things simpler, we introduce the necessary function before doing any delta base logic change. The next handful of commits will focus on refactoring the code to let that new logic land as clearly as possible. # General Idea Right now, the search for a good delta base stop whenever we found a good one. However, when using sparse-revlog, we should probably try a bit harder. We do significant effort to increase delta re-use by jumping on "unrelated" delta chains that provide better results. Moving to another chain for a better result is good, but we have no guarantee we jump at a reasonable point in that new chain. When we consider over the chains related to the parents, we start from the higher-level snapshots. This is a way to consider the snapshot closer to the current revision that has the best chance to produce a small delta. We do benefit from this walk order when jumping to a better "unrelated" stack. To counter-balance this, we'll introduce a way to "refine" the result. After a good delta have been found, we'll keep searching for a better delta, using the current best one as a starting point. # Target Setup The `finddeltainfo` method is responsible for the general search for a good delta. It requests candidates base from `_candidategroups` and decides which one are usable. The `_candidategroups` generator act as a top-level filter, it does not care about how we pick candidates, it just does basic filtering, excluding revisions that have been tested already or that are an obvious misfit. The `_rawgroups` generator is the one with the actual ancestors walking logic, It does not care about what would do a good delta and what was already tested, it just issues the initial candidates. We introduce a new `_refinedgroup` function to bridge the gap between `_candidategroups` and `_rawgroups`. It delegates the initial iteration logic and then performing relevant refining of the valid base once found. (This logic is yet to be added to function) All these logics are fairly independent and easier to understand when standing alone, not mixed with each other. It also makes it easy to test and try different approaches for one of those four layers without affecting the other ones. # Technical details To communicate `finddeltainfo` choice of "current best delta base" to the `_refinegroup` logic, we plan to use python co-routine feature. The `_candidategroups` and `_refinegroup` generators will become co-routine. This will allow `_refinegroup` to detect when a good delta have been found and triggers various refining steps. For now, `_candidategroups` will just pass the value down the stack. After poking at various option, the co-routine appears the best to keep each layers focus on its duty, without the need to spread implementation details across layers. diff --git a/mercurial/revlogutils/deltas.py b/mercurial/revlogutils/deltas.py --- a/mercurial/revlogutils/deltas.py +++ b/mercurial/revlogutils/deltas.py @@ -585,7 +585,7 @@ def _candidategroups(revlog, textlen, p1 deltas_limit = textlen * LIMIT_DELTA2TEXT tested = set([nullrev]) -for temptative in _rawgroups(revlog, p1, p2, cachedelta): +for temptative in _refinedgroups(revlog, p1, p2, cachedelta): group = [] for rev in temptative: # skip over empty delta (no need to include them in a chain) @@ -621,6 +621,13 @@ def _findsnapshots(revlog, cache, start_ if issnapshot(rev): cache[deltaparent(rev)].append(rev) +def _refinedgroups(revlog, p1, p2, cachedelta): +good = None +for candidates in _rawgroups(revlog, p1, p2, cachedelta): +good = yield candidates +if good is not None: +break + def _rawgroups(revlog, p1, p2, cachedelta): """Provides group of revision to be tested as delta base ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 08 of 19] snapshot: consider unrelated snapshots at a similar level first
# HG changeset patch # User Boris Feld # Date 1536333451 14400 # Fri Sep 07 11:17:31 2018 -0400 # Node ID 979ae7a8db741f15531f76dded568e0792e57b10 # Parent b0d159cf86730ba381b8b6a0e9031a40d52cb101 # EXP-Topic sparse-snapshot # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 979ae7a8db74 snapshot: consider unrelated snapshots at a similar level first This new step is inserted before considering using a level-N snapshot as a base for a level-N+1 snapshot. We first check if existing level-N+1 snapshots using the same base would be a suitable base for a level-N+2 snapshot. This increases snapshot reuse and limits the risk of snapshot explosion in very branchy repositories. Using a "deeper" snapshot as the base also results in a smaller snapshot since it builds a level-N+2 intermediate snapshot instead of an N+1 one. This logic is similar for the one we added in a previous commit. In that previous commit is only applied to level-0 "siblings". We can see this effect in the test repository. Snapshots moved from lower levels to higher levels. diff --git a/mercurial/revlogutils/deltas.py b/mercurial/revlogutils/deltas.py --- a/mercurial/revlogutils/deltas.py +++ b/mercurial/revlogutils/deltas.py @@ -616,9 +616,10 @@ def _candidategroups(revlog, textlen, p1 def _findsnapshots(revlog, cache, start_rev): """find snapshot from start_rev to tip""" deltaparent = revlog.deltaparent +issnapshot = revlog.issnapshot for rev in revlog.revs(start_rev): -if deltaparent(rev) == nullrev: -cache[nullrev].append(rev) +if issnapshot(rev): +cache[deltaparent(rev)].append(rev) def _rawgroups(revlog, p1, p2, cachedelta): """Provides group of revision to be tested as delta base @@ -673,11 +674,35 @@ def _rawgroups(revlog, p1, p2, cachedelt if not revlog.issnapshot(s): break parents_snaps[idx].add(s) +snapfloor = min(parents_snaps[0]) + 1 +_findsnapshots(revlog, snapshots, snapfloor) # Test them as possible intermediate snapshot base # We test them from highest to lowest level. High level one are more # likely to result in small delta +floor = None for idx, snaps in sorted(parents_snaps.items(), reverse=True): +siblings = set() +for s in snaps: +siblings.update(snapshots[s]) +# Before considering making a new intermediate snapshot, we check +# if an existing snapshot, children of base we consider, would be +# suitable. +# +# It give a change to reuse a delta chain "unrelated" to the +# current revision instead of starting our own. Without such +# re-use, topological branches would keep reopening new chains. +# Creating more and more snapshot as the repository grow. + +if floor is not None: +# We only do this for siblings created after the one in our +# parent's delta chain. Those created before has less chances +# to be valid base since our ancestors had to create a new +# snapshot. +siblings = [r for r in siblings if floor < r] +yield tuple(sorted(siblings)) +# then test the base from our parent's delta chain. yield tuple(sorted(snaps)) +floor = min(snaps) # No suitable base found in the parent chain, search if any full # snapshots emitted since parent's base would be a suitable base for an # intermediate snapshot. @@ -686,8 +711,6 @@ def _rawgroups(revlog, p1, p2, cachedelt # revisions instead of starting our own. Without such re-use, # topological branches would keep reopening new full chains. Creating # more and more snapshot as the repository grow. -snapfloor = min(parents_snaps[0]) + 1 -_findsnapshots(revlog, snapshots, snapfloor) yield tuple(snapshots[nullrev]) # other approach failed try against prev to hopefully save us a diff --git a/tests/test-sparse-revlog.t b/tests/test-sparse-revlog.t --- a/tests/test-sparse-revlog.t +++ b/tests/test-sparse-revlog.t @@ -77,7 +77,7 @@ repeatedly while some of it changes rare $ f -s .hg/store/data/*.d - .hg/store/data/_s_p_a_r_s_e-_r_e_v_l_o_g-_t_e_s_t-_f_i_l_e.d: size=63812493 + .hg/store/data/_s_p_a_r_s_e-_r_e_v_l_o_g-_t_e_s_t-_f_i_l_e.d: size=59303048 $ hg debugrevlog * format : 1 flags : generaldelta @@ -89,45 +89,45 @@ repeatedly while some of it changes rare empty :0 ( 0.00%) text :0 (100.00%) delta :0 (100.00%) - snapshot : 179 ( 3.58%) + snapshot : 165 ( 3.30%) lvl-0 : 4 ( 0.08%) -lvl-1 :
[PATCH 07 of 19] snapshot: consider all snapshots in the parents' chains
# HG changeset patch # User Boris Feld # Date 1536333450 14400 # Fri Sep 07 11:17:30 2018 -0400 # Node ID b0d159cf86730ba381b8b6a0e9031a40d52cb101 # Parent b7726577179f0cffc8a4753606dcc5feb858c22b # EXP-Topic sparse-snapshot # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r b0d159cf8673 snapshot: consider all snapshots in the parents' chains There are no reasons to only consider full snapshot as a possible base for an intermediate snapshot. Now that the basic principles have been set, we can start adding more levels of snapshots. We now consider all snapshots in the parent's chains (full or intermediate). This creates a chain of intermediate snapshots, each smaller than the previous one. # Effect On The Test Repository In the test repository, we can see a decrease in the revlog size and slightly shorter delta chain. However, that approach creates snapshots more frequently, increasing the risk of ending into problematic cases in very branchy repositories (not triggered by the test repository). The next changesets will remove that risk by adding logic that increases deltas reuse. diff --git a/mercurial/revlogutils/deltas.py b/mercurial/revlogutils/deltas.py --- a/mercurial/revlogutils/deltas.py +++ b/mercurial/revlogutils/deltas.py @@ -661,12 +661,23 @@ def _rawgroups(revlog, p1, p2, cachedelt yield parents if sparse and parents: +snapshots = collections.defaultdict(list) # map: base-rev: snapshot-rev # See if we can use an existing snapshot in the parent chains to use as # a base for a new intermediate-snapshot -bases = [] +# +# search for snapshot in parents delta chain +# map: snapshot-level: snapshot-rev +parents_snaps = collections.defaultdict(set) for p in parents: -bases.append(deltachain(p)[0]) -yield tuple(sorted(bases)) +for idx, s in enumerate(deltachain(p)): +if not revlog.issnapshot(s): +break +parents_snaps[idx].add(s) +# Test them as possible intermediate snapshot base +# We test them from highest to lowest level. High level one are more +# likely to result in small delta +for idx, snaps in sorted(parents_snaps.items(), reverse=True): +yield tuple(sorted(snaps)) # No suitable base found in the parent chain, search if any full # snapshots emitted since parent's base would be a suitable base for an # intermediate snapshot. @@ -675,8 +686,7 @@ def _rawgroups(revlog, p1, p2, cachedelt # revisions instead of starting our own. Without such re-use, # topological branches would keep reopening new full chains. Creating # more and more snapshot as the repository grow. -snapfloor = min(bases) + 1 -snapshots = collections.defaultdict(list) +snapfloor = min(parents_snaps[0]) + 1 _findsnapshots(revlog, snapshots, snapfloor) yield tuple(snapshots[nullrev]) diff --git a/tests/test-sparse-revlog.t b/tests/test-sparse-revlog.t --- a/tests/test-sparse-revlog.t +++ b/tests/test-sparse-revlog.t @@ -77,7 +77,7 @@ repeatedly while some of it changes rare $ f -s .hg/store/data/*.d - .hg/store/data/_s_p_a_r_s_e-_r_e_v_l_o_g-_t_e_s_t-_f_i_l_e.d: size=67810463 + .hg/store/data/_s_p_a_r_s_e-_r_e_v_l_o_g-_t_e_s_t-_f_i_l_e.d: size=63812493 $ hg debugrevlog * format : 1 flags : generaldelta @@ -89,39 +89,45 @@ repeatedly while some of it changes rare empty :0 ( 0.00%) text :0 (100.00%) delta :0 (100.00%) - snapshot : 126 ( 2.52%) + snapshot : 179 ( 3.58%) lvl-0 : 4 ( 0.08%) -lvl-1 :120 ( 2.40%) -lvl-2 : 2 ( 0.04%) - deltas: 4875 (97.48%) - revision size : 67810463 - snapshot : 14373347 (21.20%) -lvl-0 : 804235 ( 1.19%) -lvl-1 : 13535903 (19.96%) -lvl-2 : 33209 ( 0.05%) - deltas: 53437116 (78.80%) +lvl-1 : 49 ( 0.98%) +lvl-2 : 56 ( 1.12%) +lvl-3 : 63 ( 1.26%) +lvl-4 : 7 ( 0.14%) + deltas: 4822 (96.42%) + revision size : 63812493 + snapshot : 10745878 (16.84%) +lvl-0 : 804204 ( 1.26%) +lvl-1 :4986508 ( 7.81%) +lvl-2 :2858810 ( 4.48%) +lvl-3 :1958906 ( 3.07%) +lvl-4 : 137450 ( 0.22%) + deltas: 53066615 (83.16%) chunks: 5001 0x78 (x) : 5001 (100.00%) - chunks size : 67810463 - 0x78 (x) : 67810463 (100.00%) + chunks size : 63812493 + 0x78 (x) : 63812493 (100.00%) - avg chain length :
[PATCH 06 of 19] snapshot: search for unrelated but reusable full-snapshot
# HG changeset patch # User Boris Feld # Date 1536333450 14400 # Fri Sep 07 11:17:30 2018 -0400 # Node ID b7726577179f0cffc8a4753606dcc5feb858c22b # Parent c6dd1a01b22cd2649dc7c0dfe9981305b6e762d2 # EXP-Topic sparse-snapshot # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r b7726577179f snapshot: search for unrelated but reusable full-snapshot # New Strategy Step: Reusing Snapshot Outside Of Parents' Chain. If no suitable bases were found in the parent's chains, see if we could reuse a full snapshot not directly related to the current revision. Such search can be expensive, so we only search for snapshots appended to the revlog *after* the bases used by the parents of the current revision (the one we just tested). We assume the parent's bases were created because the previous snapshots were unsuitable, so there are low odds they would be useful now. This search gives a chance to reuse a delta chain unrelated to the current revision. Without this re-use, topological branches would keep reopening new full chains. Creating more and more snapshots as the repository grow. In repositories with many topological branches, the lack of delta reuse can create too many snapshots reducing overall compression to nothing. This results in a very large repository and other usability issues. For now, we still focus on creating level-1 snapshots. However, this principle will play a large part in how we avoid snapshot explosion once we have more snapshot levels. # Effects On The Test Repository In the test repository we created, we can see the beneficial effect of such reuse. We need very few level-0 snapshots and the overall revlog size has decreased. The `hg debugrevlog` call, show a "lvl-2" snapshot. It comes from the existing delta logic using the `prev` revision (revlog's tip) as the base. In this specific case, it turns out the tip was a level-1 snapshot. This is a coincidence that can be ignored. Finding and testing against all these unrelated snapshots can have a performance impact at write time. We currently focus on building good deltas chain we build. Performance concern will be dealt with later in another series. diff --git a/mercurial/revlogutils/deltas.py b/mercurial/revlogutils/deltas.py --- a/mercurial/revlogutils/deltas.py +++ b/mercurial/revlogutils/deltas.py @@ -9,6 +9,7 @@ from __future__ import absolute_import +import collections import heapq import struct @@ -607,8 +608,18 @@ def _candidategroups(revlog, textlen, p1 continue group.append(rev) if group: +# XXX: in the sparse revlog case, group can become large, +# impacting performances. Some bounding or slicing mecanism +# would help to reduce this impact. yield tuple(group) +def _findsnapshots(revlog, cache, start_rev): +"""find snapshot from start_rev to tip""" +deltaparent = revlog.deltaparent +for rev in revlog.revs(start_rev): +if deltaparent(rev) == nullrev: +cache[nullrev].append(rev) + def _rawgroups(revlog, p1, p2, cachedelta): """Provides group of revision to be tested as delta base @@ -656,6 +667,18 @@ def _rawgroups(revlog, p1, p2, cachedelt for p in parents: bases.append(deltachain(p)[0]) yield tuple(sorted(bases)) +# No suitable base found in the parent chain, search if any full +# snapshots emitted since parent's base would be a suitable base for an +# intermediate snapshot. +# +# It give a chance to reuse a delta chain unrelated to the current +# revisions instead of starting our own. Without such re-use, +# topological branches would keep reopening new full chains. Creating +# more and more snapshot as the repository grow. +snapfloor = min(bases) + 1 +snapshots = collections.defaultdict(list) +_findsnapshots(revlog, snapshots, snapfloor) +yield tuple(snapshots[nullrev]) # other approach failed try against prev to hopefully save us a # fulltext. diff --git a/tests/test-sparse-revlog.t b/tests/test-sparse-revlog.t --- a/tests/test-sparse-revlog.t +++ b/tests/test-sparse-revlog.t @@ -77,7 +77,7 @@ repeatedly while some of it changes rare $ f -s .hg/store/data/*.d - .hg/store/data/_s_p_a_r_s_e-_r_e_v_l_o_g-_t_e_s_t-_f_i_l_e.d: size=72315280 + .hg/store/data/_s_p_a_r_s_e-_r_e_v_l_o_g-_t_e_s_t-_f_i_l_e.d: size=67810463 $ hg debugrevlog * format : 1 flags : generaldelta @@ -89,36 +89,39 @@ repeatedly while some of it changes rare empty :0 ( 0.00%) text :0 (100.00%) delta :0 (100.00%) - snapshot : 145 ( 2.90%) -lvl-0 : 15 ( 0.30%) -lvl-1 :130 ( 2.60%) - deltas: 4856 (97.10%) - revision size :
[PATCH 05 of 19] snapshot: try intermediate snapshot against parents' base
# HG changeset patch # User Boris Feld # Date 1536333449 14400 # Fri Sep 07 11:17:29 2018 -0400 # Node ID c6dd1a01b22cd2649dc7c0dfe9981305b6e762d2 # Parent 607e4fcb774047629cea21bc4052ce8c35eab5d5 # EXP-Topic sparse-snapshot # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r c6dd1a01b22c snapshot: try intermediate snapshot against parents' base # Regarding The Series Started By This Changeset This is the first changesets of a group adjusting delta chain strategy to build a useful chain of intermediate snapshots. The series will introduce a full strategy to produce chains of multiple snapshots on top of which a "usual" delta chain will be built. That strategy will have multiple steps to maximize snapshot reuse, avoiding pathological cases and improving overall compression in very branchy repositories. An important property of sparse-revlog using such snapshot-chain is that they can use very short delta chain without problematic impact on the resulting compression. Shorter delta chains are important to achieve good performance. To make each step clear, we'll introduce them one by one. See the end of this series for full details. # Regarding This Changeset Before this change, if we cannot store the current revision as a delta against a "simple" candidate (p1, p2, prev), we created a new level-0 snapshot (also called full snapshot). As the first step, we introduce a simple strategy: try an intermediate level-1 snapshot against the chain base of the "current revision" parents. The "current revision" is the one we are currently trying to store in the revlog, triggering this search for a good delta base. The first item in the chain is always a level-0 snapshot. # Effect On The Test Repository We can already see the effect on the test-repository. Most of the snapshots have shifted from level 0 to level 1. The overall size has slightly decreased. (However, keep in mind that this repository only emulates real data) # Regarding Statistic The current series focuses on improving the chain built. Improving the performance of this logic will be done as a second step. Sparse-revlog is still experimental and disabled by default. We'll provide more statistic about resulting size and delta chain at the end of this series. diff --git a/mercurial/revlogutils/deltas.py b/mercurial/revlogutils/deltas.py --- a/mercurial/revlogutils/deltas.py +++ b/mercurial/revlogutils/deltas.py @@ -618,8 +618,10 @@ def _rawgroups(revlog, p1, p2, cachedelt The group order aims at providing fast or small candidates first. """ gdelta = revlog._generaldelta +sparse = revlog._sparserevlog curr = len(revlog) prev = curr - 1 +deltachain = lambda rev: revlog._deltachain(rev)[0] # First we try to reuse a the delta contained in the bundle. # (or from the source revlog) @@ -647,6 +649,14 @@ def _rawgroups(revlog, p1, p2, cachedelt # Test all parents (1 or 2), and keep the best candidate yield parents +if sparse and parents: +# See if we can use an existing snapshot in the parent chains to use as +# a base for a new intermediate-snapshot +bases = [] +for p in parents: +bases.append(deltachain(p)[0]) +yield tuple(sorted(bases)) + # other approach failed try against prev to hopefully save us a # fulltext. yield (prev,) diff --git a/tests/test-sparse-revlog.t b/tests/test-sparse-revlog.t --- a/tests/test-sparse-revlog.t +++ b/tests/test-sparse-revlog.t @@ -77,7 +77,7 @@ repeatedly while some of it changes rare $ f -s .hg/store/data/*.d - .hg/store/data/_s_p_a_r_s_e-_r_e_v_l_o_g-_t_e_s_t-_f_i_l_e.d: size=74365490 + .hg/store/data/_s_p_a_r_s_e-_r_e_v_l_o_g-_t_e_s_t-_f_i_l_e.d: size=72315280 $ hg debugrevlog * format : 1 flags : generaldelta @@ -89,33 +89,36 @@ repeatedly while some of it changes rare empty :0 ( 0.00%) text :0 (100.00%) delta :0 (100.00%) - snapshot : 101 ( 2.02%) -lvl-0 :101 ( 2.02%) - deltas: 4900 (97.98%) - revision size : 74365490 - snapshot : 20307865 (27.31%) -lvl-0 : 20307865 (27.31%) - deltas: 54057625 (72.69%) + snapshot : 145 ( 2.90%) +lvl-0 : 15 ( 0.30%) +lvl-1 :130 ( 2.60%) + deltas: 4856 (97.10%) + revision size : 72315280 + snapshot : 18481085 (25.56%) +lvl-0 :3016019 ( 4.17%) +lvl-1 : 15465066 (21.39%) + deltas: 53834195 (74.44%) chunks: 5001 0x78 (x) : 5001 (100.00%) - chunks size : 74365490 - 0x78 (x) : 74365490 (100.00%) + chunks size : 72315280 + 0x78 (x) : 72315280 (100.00%) - avg chain length : 23 + avg chain length : 18
[PATCH 04 of 19] sparse-revlog: add a test checking revlog deltas for a churning file
# HG changeset patch
# User Boris Feld
# Date 1536333449 14400
# Fri Sep 07 11:17:29 2018 -0400
# Node ID 607e4fcb774047629cea21bc4052ce8c35eab5d5
# Parent a4f94a5caf6f652ff4959daedddf1d88c6059f1b
# EXP-Topic sparse-snapshot
# Available At https://bitbucket.org/octobus/mercurial-devel/
# hg pull https://bitbucket.org/octobus/mercurial-devel/ -r
607e4fcb7740
sparse-revlog: add a test checking revlog deltas for a churning file
The test repository contains 5000 revisions and is therefore slow to build:
five minutes with CHG, over fifteen minutes without. It is too slow to build
during the test. Bundling all content produce a sizeable result, 20BM, too
large to be committed. Instead, we commit a script to build the expected
bundle and the test checks if the bundle is available. Any run of the script
will produce the same repository content, using resulting in the same hashes.
Using smaller repositories was tried, however, it misses most of the cases we
are planning to improve. Having them in a 5000 repository is already nice, we
usually see these case in repositories in the order of magnitude of one
million revisions.
This test will be very useful to check various changes strategy for building
delta to store in a sparse-revlog.
In this series we will focus our attention on the following metrics:
The ones that will impact the final storage performance (size, space):
* size of the revlog data file (".hg/store/data/*.d")
* chain length info
The ones that describe the deltas patterns:
* number of snapshot revision (and their level)
* size taken by snapshot revision (and their level)
diff --git a/.hgignore b/.hgignore
--- a/.hgignore
+++ b/.hgignore
@@ -19,6 +19,7 @@ syntax: glob
*.zip
\#*\#
.\#*
+tests/artifacts/cache/big-file-churn.hg
tests/.coverage*
tests/.testtimes*
tests/.hypothesis
diff --git a/tests/artifacts/cache/big-file-churn.hg.md5
b/tests/artifacts/cache/big-file-churn.hg.md5
new file mode 100644
--- /dev/null
+++ b/tests/artifacts/cache/big-file-churn.hg.md5
@@ -0,0 +1,1 @@
+fe0d0bb5979de50f4fed71bb9437764d
diff --git a/tests/artifacts/scripts/generate-churning-bundle.py
b/tests/artifacts/scripts/generate-churning-bundle.py
new file mode 100755
--- /dev/null
+++ b/tests/artifacts/scripts/generate-churning-bundle.py
@@ -0,0 +1,139 @@
+#!/usr/bin/env python
+#
+# generate-branchy-bundle - generate a branch for a "large" branchy repository
+#
+# Copyright 2018 Octobus, cont...@octobus.net
+#
+# This software may be used and distributed according to the terms of the
+# GNU General Public License version 2 or any later version.
+#
+# This script generates a repository suitable for testing delta computation
+# strategies.
+#
+# The repository update a single "large" file with many updates. One fixed part
+# of the files always get updated while the rest of the lines get updated over
+# time. This update happens over many topological branches, some getting merged
+# back.
+#
+# Running with `chg` in your path and `CHGHG` set is recommended for speed.
+
+from __future__ import absolute_import, print_function
+
+import hashlib
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+
+BUNDLE_NAME = 'big-file-churn.hg'
+
+# constants for generating the repository
+NB_CHANGESET = 5000
+PERIOD_MERGING = 8
+PERIOD_BRANCHING = 7
+MOVE_BACK_MIN = 3
+MOVE_BACK_RANGE = 5
+
+# constants for generating the large file we keep updating
+#
+# At each revision, the beginning on the file change,
+# and set of other lines changes too.
+FILENAME='SPARSE-REVLOG-TEST-FILE'
+NB_LINES = 10500
+ALWAYS_CHANGE_LINES = 500
+FILENAME = 'SPARSE-REVLOG-TEST-FILE'
+OTHER_CHANGES = 300
+
+def nextcontent(previous_content):
+"""utility to produce a new file content from the previous one"""
+return hashlib.md5(previous_content).hexdigest()
+
+def filecontent(iteridx, oldcontent):
+"""generate a new file content
+
+The content is generated according the iteration index and previous
+content"""
+
+# initial call
+if iteridx is None:
+current = ''
+else:
+current = str(iteridx)
+
+for idx in xrange(NB_LINES):
+do_change_line = True
+if oldcontent is not None and ALWAYS_CHANGE_LINES < idx:
+do_change_line = not ((idx - iteridx) % OTHER_CHANGES)
+
+if do_change_line:
+to_write = current + '\n'
+current = nextcontent(current)
+else:
+to_write = oldcontent[idx]
+yield to_write
+
+def updatefile(filename, idx):
+"""update to be at appropriate content for iteration """
+existing = None
+if idx is not None:
+with open(filename, 'rb') as old:
+existing = old.readlines()
+with open(filename, 'wb') as target:
+for line in filecontent(idx, existing):
+target.write(line)
+
+def hg(command, *args):
+"""call a mercurial command with appropriate config and argument"""
+env = os.environ.copy()
+if
[PATCH 03 of 19] tests: add a `tests/artifacts/` directory
# HG changeset patch # User Boris Feld # Date 1534589144 -7200 # Sat Aug 18 12:45:44 2018 +0200 # Node ID a4f94a5caf6f652ff4959daedddf1d88c6059f1b # Parent 2d50819efa01dbb1ba0e14e30486490cb90c002a # EXP-Topic sparse-snapshot # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r a4f94a5caf6f tests: add a `tests/artifacts/` directory That directory is meant to cache large items used by tests that are slow to generate. See 'PURPOSE' file for details and next changesets for a first user. diff --git a/tests/artifacts/PURPOSE b/tests/artifacts/PURPOSE new file mode 100644 --- /dev/null +++ b/tests/artifacts/PURPOSE @@ -0,0 +1,9 @@ +This directory is meant to cache artifacts useful for tests (such as bundle). + +Those artifacts need to be cached because they are slow to regenerate on each +test but too large to be tracked within the repository. They are not expected +to change between each run and can be cached. + +The `./scripts/` contains code to generate the artifact while the `cache` +directory contains resulting artifact. + ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 01 of 19] revlog: drop duplicated code
# HG changeset patch # User Boris Feld # Date 1536087921 -7200 # Tue Sep 04 21:05:21 2018 +0200 # Node ID bdcfd96d87254a508e85a6eb502ffe4c57845bc8 # Parent 6268fed317d04dd8f0430468818ea5d0529b6503 # EXP-Topic sparse-snapshot # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r bdcfd96d8725 revlog: drop duplicated code This code probably got duplicated by a rebase/evolve conflict. We drop the extra copy, the other copy is right below. This had no real effects since other logic ensure that we never test the same revision twice. diff --git a/mercurial/revlogutils/deltas.py b/mercurial/revlogutils/deltas.py --- a/mercurial/revlogutils/deltas.py +++ b/mercurial/revlogutils/deltas.py @@ -621,19 +621,6 @@ def _rawgroups(revlog, p1, p2, cachedelt curr = len(revlog) prev = curr - 1 -# should we try to build a delta? -if prev != nullrev and revlog._storedeltachains: -tested = set() -# This condition is true most of the time when processing -# changegroup data into a generaldelta repo. The only time it -# isn't true is if this is the first revision in a delta chain -# or if ``format.generaldelta=true`` disabled ``lazydeltabase``. -if cachedelta and gdelta and revlog._lazydeltabase: -# Assume what we received from the server is a good choice -# build delta will reuse the cache -yield (cachedelta[0],) -tested.add(cachedelta[0]) - # This condition is true most of the time when processing # changegroup data into a generaldelta repo. The only time it # isn't true is if this is the first revision in a delta chain ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 02 of 19] revlog: clarify the comment attached to delta reuse
# HG changeset patch # User Boris Feld # Date 1536089308 -7200 # Tue Sep 04 21:28:28 2018 +0200 # Node ID 2d50819efa01dbb1ba0e14e30486490cb90c002a # Parent bdcfd96d87254a508e85a6eb502ffe4c57845bc8 # EXP-Topic sparse-snapshot # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 2d50819efa01 revlog: clarify the comment attached to delta reuse The previous version was a bit complicated and referred to a deprecated configuration option. diff --git a/mercurial/revlogutils/deltas.py b/mercurial/revlogutils/deltas.py --- a/mercurial/revlogutils/deltas.py +++ b/mercurial/revlogutils/deltas.py @@ -621,10 +621,12 @@ def _rawgroups(revlog, p1, p2, cachedelt curr = len(revlog) prev = curr - 1 -# This condition is true most of the time when processing -# changegroup data into a generaldelta repo. The only time it -# isn't true is if this is the first revision in a delta chain -# or if ``format.generaldelta=true`` disabled ``lazydeltabase``. +# First we try to reuse a the delta contained in the bundle. +# (or from the source revlog) +# +# This logic only applies to general delta repositories and can be disabled +# through configuration. Disabling reuse of source delta is useful when +# we want to make sure we recomputed "optimal" deltas. if cachedelta and gdelta and revlog._lazydeltabase: # Assume what we received from the server is a good choice # build delta will reuse the cache ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D4516: contrib: use a monotonic timer in catapipe
lothiraldan created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.
REVISION SUMMARY
As spotted by Gregory, we should use a monotonic timer to get better timings.
REPOSITORY
rHG Mercurial
REVISION DETAIL
https://phab.mercurial-scm.org/D4516
AFFECTED FILES
contrib/catapipe.py
CHANGE DETAILS
diff --git a/contrib/catapipe.py b/contrib/catapipe.py
--- a/contrib/catapipe.py
+++ b/contrib/catapipe.py
@@ -29,17 +29,21 @@
from __future__ import absolute_import, print_function
import argparse
-import datetime
import json
import os
+import timeit
_TYPEMAP = {
'START': 'B',
'END': 'E',
}
_threadmap = {}
+# Timeit already contains the whole logic about which timer to use based on
+# Python version and OS
+timer = timeit.default_timer
+
def main():
parser = argparse.ArgumentParser()
parser.add_argument('pipe', type=str, nargs=1,
@@ -55,12 +59,12 @@
try:
with open(fn) as f, open(args.output, 'w') as out:
out.write('[\n')
-start = datetime.datetime.now()
+start = timer()
while True:
ev = f.readline().strip()
if not ev:
continue
-now = datetime.datetime.now()
+now = timer()
if args.debug:
print(ev)
verb, session, label = ev.split(' ', 2)
To: lothiraldan, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D4515: contrib: fix catapipe output argument documentation
lothiraldan created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.
REPOSITORY
rHG Mercurial
REVISION DETAIL
https://phab.mercurial-scm.org/D4515
AFFECTED FILES
contrib/catapipe.py
CHANGE DETAILS
diff --git a/contrib/catapipe.py b/contrib/catapipe.py
--- a/contrib/catapipe.py
+++ b/contrib/catapipe.py
@@ -45,7 +45,8 @@
parser.add_argument('pipe', type=str, nargs=1,
help='Path of named pipe to create and listen on.')
parser.add_argument('output', default='trace.json', type=str, nargs='?',
-help='Path of named pipe to create and listen on.')
+help='Path of json file to create where the traces '
+ 'will be stored.')
parser.add_argument('--debug', default=False, action='store_true',
help='Print useful debug messages')
args = parser.parse_args()
To: lothiraldan, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D4514: tracing: trace command function execution
lothiraldan created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.
REPOSITORY
rHG Mercurial
REVISION DETAIL
https://phab.mercurial-scm.org/D4514
AFFECTED FILES
mercurial/dispatch.py
CHANGE DETAILS
diff --git a/mercurial/dispatch.py b/mercurial/dispatch.py
--- a/mercurial/dispatch.py
+++ b/mercurial/dispatch.py
@@ -992,7 +992,8 @@
def _runcommand(ui, options, cmd, cmdfunc):
"""Run a command function, possibly with profiling enabled."""
try:
-return cmdfunc()
+with tracing.log("Running %s command" % cmd):
+return cmdfunc()
except error.SignatureError:
raise error.CommandError(cmd, _('invalid arguments'))
To: lothiraldan, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D4513: extension: add a summary of total loading time per extension
lothiraldan created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.
REPOSITORY
rHG Mercurial
REVISION DETAIL
https://phab.mercurial-scm.org/D4513
AFFECTED FILES
mercurial/extensions.py
tests/test-bad-extension.t
tests/test-extension-timing.t
CHANGE DETAILS
diff --git a/tests/test-extension-timing.t b/tests/test-extension-timing.t
--- a/tests/test-extension-timing.t
+++ b/tests/test-extension-timing.t
@@ -68,6 +68,7 @@
debug.extensions: > remaining aftercallbacks completed in * (glob)
debug.extensions: - loading extension registration objects
debug.extensions: > extension registration object loading took * (glob)
+ debug.extensions: > extension foobar take a total of * to load (glob)
debug.extensions: extension loading complete
debug.extensions: loading additional extensions
debug.extensions: - processing 1 entries
diff --git a/tests/test-bad-extension.t b/tests/test-bad-extension.t
--- a/tests/test-bad-extension.t
+++ b/tests/test-bad-extension.t
@@ -122,6 +122,8 @@
debug.extensions: > remaining aftercallbacks completed in * (glob)
debug.extensions: - loading extension registration objects
debug.extensions: > extension registration object loading took * (glob)
+ debug.extensions: > extension baddocext take a total of * to load (glob)
+ debug.extensions: > extension gpg take a total of * to load (glob)
debug.extensions: extension loading complete
#endif
diff --git a/mercurial/extensions.py b/mercurial/extensions.py
--- a/mercurial/extensions.py
+++ b/mercurial/extensions.py
@@ -166,7 +166,7 @@
_rejectunicode(t, o._table)
_validatecmdtable(ui, getattr(mod, 'cmdtable', {}))
-def load(ui, name, path, log=lambda *a: None):
+def load(ui, name, path, log=lambda *a: None, loadingtime=None):
if name.startswith('hgext.') or name.startswith('hgext/'):
shortname = name[6:]
else:
@@ -180,6 +180,8 @@
with util.timedcm('load extension %r', shortname) as stats:
mod = _importext(name, path, bind(_reportimporterror, ui))
log(' > %r extension loaded in %s\n', shortname, stats)
+if loadingtime is not None:
+loadingtime[shortname] += stats.elapsed
# Before we do anything with the extension, check against minimum stated
# compatibility. This gives extension authors a mechanism to have their
@@ -237,6 +239,7 @@
msg % values, label='debug.extensions')
else:
log = lambda *a, **kw: None
+loadingtime = collections.defaultdict(int)
result = ui.configitems("extensions")
if whitelist is not None:
result = [(k, v) for (k, v) in result if k in whitelist]
@@ -252,7 +255,7 @@
_disabledextensions[name] = path[1:]
continue
try:
-load(ui, name, path, log)
+load(ui, name, path, log, loadingtime)
except Exception as inst:
msg = stringutil.forcebytestr(inst)
if path:
@@ -292,6 +295,7 @@
log('- the %r extension uisetup failed\n', name)
broken.add(name)
log(' > uisetup for %r took %s\n', name, stats)
+loadingtime[name] += stats.elapsed
log('> all uisetup took %s\n', alluisetupstats)
log('- executing extsetup hooks\n')
@@ -305,6 +309,7 @@
log('- the %r extension extsetup failed\n', name)
broken.add(name)
log(' > extsetup for %r took %s\n', name, stats)
+loadingtime[name] += stats.elapsed
log('> all extsetup took %s\n', allextetupstats)
for name in broken:
@@ -360,6 +365,12 @@
with util.timedcm('load registration objects') as stats:
_loadextra(ui, newindex, extraloaders)
log('> extension registration object loading took %s\n', stats)
+
+# Report per extension loading time (except reposetup)
+for name in sorted(loadingtime):
+extension_msg = '> extension %s take a total of %s to load\n'
+log(extension_msg, name, util.timecount(loadingtime[name]))
+
log('extension loading complete\n')
def _loadextra(ui, newindex, extraloaders):
To: lothiraldan, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D4512: extensions: trace the total time of running all reposetup callbacks
lothiraldan created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.
REPOSITORY
rHG Mercurial
REVISION DETAIL
https://phab.mercurial-scm.org/D4512
AFFECTED FILES
mercurial/hg.py
tests/test-extension-timing.t
CHANGE DETAILS
diff --git a/tests/test-extension-timing.t b/tests/test-extension-timing.t
--- a/tests/test-extension-timing.t
+++ b/tests/test-extension-timing.t
@@ -87,6 +87,7 @@
reposetup called for a
ui == repo.ui
debug.extensions: > reposetup for 'foobar' took * (glob)
+ debug.extensions: > all reposetup took * (glob)
Foo
$ cd ..
diff --git a/mercurial/hg.py b/mercurial/hg.py
--- a/mercurial/hg.py
+++ b/mercurial/hg.py
@@ -171,13 +171,15 @@
for f in presetupfuncs or []:
f(ui, obj)
log('- executing reposetup hooks\n')
-for name, module in extensions.extensions(ui):
-log(' - running reposetup for %s\n' % (name,))
-hook = getattr(module, 'reposetup', None)
-if hook:
-with util.timedcm('reposetup %r', name) as stats:
-hook(ui, obj)
-log(' > reposetup for %r took %s\n', name, stats)
+with util.timedcm('all reposetup') as allreposetupstats:
+for name, module in extensions.extensions(ui):
+log(' - running reposetup for %s\n' % (name,))
+hook = getattr(module, 'reposetup', None)
+if hook:
+with util.timedcm('reposetup %r', name) as stats:
+hook(ui, obj)
+log(' > reposetup for %r took %s\n', name, stats)
+log('> all reposetup took %s\n', allreposetupstats)
if not obj.local():
for f in wirepeersetupfuncs:
f(ui, obj)
To: lothiraldan, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D4511: extensions: trace the total time of running all extsetup callbacks
lothiraldan created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.
REPOSITORY
rHG Mercurial
REVISION DETAIL
https://phab.mercurial-scm.org/D4511
AFFECTED FILES
mercurial/extensions.py
tests/test-bad-extension.t
tests/test-extension-timing.t
CHANGE DETAILS
diff --git a/tests/test-extension-timing.t b/tests/test-extension-timing.t
--- a/tests/test-extension-timing.t
+++ b/tests/test-extension-timing.t
@@ -63,6 +63,7 @@
debug.extensions: - executing extsetup hooks
debug.extensions: - running extsetup for 'foobar'
debug.extensions: > extsetup for 'foobar' took * (glob)
+ debug.extensions: > all extsetup took * (glob)
debug.extensions: - executing remaining aftercallbacks
debug.extensions: > remaining aftercallbacks completed in * (glob)
debug.extensions: - loading extension registration objects
@@ -75,6 +76,7 @@
debug.extensions: - executing uisetup hooks
debug.extensions: > all uisetup took * (glob)
debug.extensions: - executing extsetup hooks
+ debug.extensions: > all extsetup took * (glob)
debug.extensions: - executing remaining aftercallbacks
debug.extensions: > remaining aftercallbacks completed in * (glob)
debug.extensions: - loading extension registration objects
diff --git a/tests/test-bad-extension.t b/tests/test-bad-extension.t
--- a/tests/test-bad-extension.t
+++ b/tests/test-bad-extension.t
@@ -117,6 +117,7 @@
debug.extensions: > extsetup for 'gpg' took * (glob)
debug.extensions: - running extsetup for 'baddocext'
debug.extensions: > extsetup for 'baddocext' took * (glob)
+ debug.extensions: > all extsetup took * (glob)
debug.extensions: - executing remaining aftercallbacks
debug.extensions: > remaining aftercallbacks completed in * (glob)
debug.extensions: - loading extension registration objects
diff --git a/mercurial/extensions.py b/mercurial/extensions.py
--- a/mercurial/extensions.py
+++ b/mercurial/extensions.py
@@ -295,15 +295,17 @@
log('> all uisetup took %s\n', alluisetupstats)
log('- executing extsetup hooks\n')
-for name in _order[newindex:]:
-if name in broken:
-continue
-log(' - running extsetup for %r\n', name)
-with util.timedcm('extsetup %r', name) as stats:
-if not _runextsetup(name, ui):
-log('- the %r extension extsetup failed\n', name)
-broken.add(name)
-log(' > extsetup for %r took %s\n', name, stats)
+with util.timedcm('all extsetup') as allextetupstats:
+for name in _order[newindex:]:
+if name in broken:
+continue
+log(' - running extsetup for %r\n', name)
+with util.timedcm('extsetup %r', name) as stats:
+if not _runextsetup(name, ui):
+log('- the %r extension extsetup failed\n', name)
+broken.add(name)
+log(' > extsetup for %r took %s\n', name, stats)
+log('> all extsetup took %s\n', allextetupstats)
for name in broken:
log('- disabling broken %r extension\n', name)
To: lothiraldan, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D4510: extensions: trace the total time of running all uisetup callbacks
lothiraldan created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.
REPOSITORY
rHG Mercurial
REVISION DETAIL
https://phab.mercurial-scm.org/D4510
AFFECTED FILES
mercurial/extensions.py
tests/test-bad-extension.t
tests/test-extension-timing.t
CHANGE DETAILS
diff --git a/tests/test-extension-timing.t b/tests/test-extension-timing.t
--- a/tests/test-extension-timing.t
+++ b/tests/test-extension-timing.t
@@ -59,6 +59,7 @@
uisetup called
uisetup called [status]
debug.extensions: > uisetup for 'foobar' took * (glob)
+ debug.extensions: > all uisetup took * (glob)
debug.extensions: - executing extsetup hooks
debug.extensions: - running extsetup for 'foobar'
debug.extensions: > extsetup for 'foobar' took * (glob)
@@ -72,6 +73,7 @@
debug.extensions: > loaded 0 extensions, total time * (glob)
debug.extensions: - loading configtable attributes
debug.extensions: - executing uisetup hooks
+ debug.extensions: > all uisetup took * (glob)
debug.extensions: - executing extsetup hooks
debug.extensions: - executing remaining aftercallbacks
debug.extensions: > remaining aftercallbacks completed in * (glob)
diff --git a/tests/test-bad-extension.t b/tests/test-bad-extension.t
--- a/tests/test-bad-extension.t
+++ b/tests/test-bad-extension.t
@@ -111,6 +111,7 @@
debug.extensions: > uisetup for 'gpg' took * (glob)
debug.extensions: - running uisetup for 'baddocext'
debug.extensions: > uisetup for 'baddocext' took * (glob)
+ debug.extensions: > all uisetup took * (glob)
debug.extensions: - executing extsetup hooks
debug.extensions: - running extsetup for 'gpg'
debug.extensions: > extsetup for 'gpg' took * (glob)
diff --git a/mercurial/extensions.py b/mercurial/extensions.py
--- a/mercurial/extensions.py
+++ b/mercurial/extensions.py
@@ -284,13 +284,15 @@
broken = set()
log('- executing uisetup hooks\n')
-for name in _order[newindex:]:
-log(' - running uisetup for %r\n', name)
-with util.timedcm('uisetup %r', name) as stats:
-if not _runuisetup(name, ui):
-log('- the %r extension uisetup failed\n', name)
-broken.add(name)
-log(' > uisetup for %r took %s\n', name, stats)
+with util.timedcm('all uisetup') as alluisetupstats:
+for name in _order[newindex:]:
+log(' - running uisetup for %r\n', name)
+with util.timedcm('uisetup %r', name) as stats:
+if not _runuisetup(name, ui):
+log('- the %r extension uisetup failed\n', name)
+broken.add(name)
+log(' > uisetup for %r took %s\n', name, stats)
+log('> all uisetup took %s\n', alluisetupstats)
log('- executing extsetup hooks\n')
for name in _order[newindex:]:
To: lothiraldan, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D4509: extensions: add timing for extensions reposetup
lothiraldan created this revision.
Herald added subscribers: mercurial-devel, mjpieters.
Herald added a reviewer: hg-reviewers.
REPOSITORY
rHG Mercurial
REVISION DETAIL
https://phab.mercurial-scm.org/D4509
AFFECTED FILES
mercurial/hg.py
tests/test-extension-timing.t
tests/test-extension.t
CHANGE DETAILS
diff --git a/tests/test-extension.t b/tests/test-extension-timing.t
copy from tests/test-extension.t
copy to tests/test-extension-timing.t
--- a/tests/test-extension.t
+++ b/tests/test-extension-timing.t
@@ -40,1716 +40,51 @@
$ echo '[extensions]' >> $HGRCPATH
$ echo "foobar = $abspath" >> $HGRCPATH
- $ hg foo
+
+Test extension setup timings
+
+ $ hg foo --traceback --config devel.debug.extensions=yes --debug 2>&1
+ debug.extensions: loading extensions
+ debug.extensions: - processing 1 entries
+ debug.extensions: - loading extension: 'foobar'
+ debug.extensions: > 'foobar' extension loaded in * (glob)
+ debug.extensions: - validating extension tables: 'foobar'
+ debug.extensions: - invoking registered callbacks: 'foobar'
+ debug.extensions: > callbacks completed in * (glob)
+ debug.extensions: > loaded 1 extensions, total time * (glob)
+ debug.extensions: - loading configtable attributes
+ debug.extensions: - executing uisetup hooks
+ debug.extensions: - running uisetup for 'foobar'
+ uisetup called [debug]
uisetup called
uisetup called [status]
- reposetup called for a
- ui == repo.ui
- reposetup called for a (chg !)
- ui == repo.ui (chg !)
- Foo
- $ hg foo --quiet
- uisetup called (no-chg !)
- reposetup called for a (chg !)
- ui == repo.ui
- Foo
- $ hg foo --debug
- uisetup called [debug] (no-chg !)
- uisetup called (no-chg !)
- uisetup called [status] (no-chg !)
- reposetup called for a (chg !)
- ui == repo.ui
- Foo
-
- $ cd ..
- $ hg clone a b
- uisetup called (no-chg !)
- uisetup called [status] (no-chg !)
- reposetup called for a
- ui == repo.ui
- reposetup called for b
- ui == repo.ui
- updating to branch default
- 1 files updated, 0 files merged, 0 files removed, 0 files unresolved
-
- $ hg bar
- uisetup called (no-chg !)
- uisetup called [status] (no-chg !)
- Bar
- $ echo 'foobar = !' >> $HGRCPATH
-
-module/__init__.py-style
-
- $ echo "barfoo = $barfoopath" >> $HGRCPATH
- $ cd a
- $ hg foo
- uisetup called
- uisetup called [status]
+ debug.extensions: > uisetup for 'foobar' took * (glob)
+ debug.extensions: - executing extsetup hooks
+ debug.extensions: - running extsetup for 'foobar'
+ debug.extensions: > extsetup for 'foobar' took * (glob)
+ debug.extensions: - executing remaining aftercallbacks
+ debug.extensions: > remaining aftercallbacks completed in * (glob)
+ debug.extensions: - loading extension registration objects
+ debug.extensions: > extension registration object loading took * (glob)
+ debug.extensions: extension loading complete
+ debug.extensions: loading additional extensions
+ debug.extensions: - processing 1 entries
+ debug.extensions: > loaded 0 extensions, total time * (glob)
+ debug.extensions: - loading configtable attributes
+ debug.extensions: - executing uisetup hooks
+ debug.extensions: - executing extsetup hooks
+ debug.extensions: - executing remaining aftercallbacks
+ debug.extensions: > remaining aftercallbacks completed in * (glob)
+ debug.extensions: - loading extension registration objects
+ debug.extensions: > extension registration object loading took * (glob)
+ debug.extensions: extension loading complete
+ debug.extensions: - executing reposetup hooks
+ debug.extensions: - running reposetup for foobar
reposetup called for a
ui == repo.ui
- reposetup called for a (chg !)
- ui == repo.ui (chg !)
+ debug.extensions: > reposetup for 'foobar' took * (glob)
Foo
- $ echo 'barfoo = !' >> $HGRCPATH
-
-Check that extensions are loaded in phases:
-
- $ cat > foo.py < import os
- > name = os.path.basename(__file__).rsplit('.', 1)[0]
- > print("1) %s imported" % name)
- > def uisetup(ui):
- > print("2) %s uisetup" % name)
- > def extsetup():
- > print("3) %s extsetup" % name)
- > def reposetup(ui, repo):
- >print("4) %s reposetup" % name)
- >
- > # custom predicate to check registration of functions at loading
- > from mercurial import (
- > registrar,
- > smartset,
- > )
- > revsetpredicate = registrar.revsetpredicate()
- > @revsetpredicate(name, safe=True) # safe=True for query via hgweb
- > def custompredicate(repo, subset, x):
- > return smartset.baseset([r for r in subset if r in {0}])
- > EOF
-
- $ cp foo.py bar.py
- $ echo 'foo = foo.py' >> $HGRCPATH
- $ echo 'bar = bar.py' >> $HGRCPATH
-
-Check normal command's load order of extensions and registration of functions
-
- $ hg log -r "foo() and bar()" -q
- 1) foo imported
- 1) bar imported
- 2) foo uisetup
- 2) bar uisetup
- 3) foo extsetup
- 3) bar extsetup
- 4) foo reposetup
- 4) bar reposetup
- 0:c24b9ac61126
-
D4507: ancestors: add nullrev to set from the beginning
This revision was automatically updated to reflect the committed changes.
Closed by commit rHG8eb2145ff0fb: ancestors: add nullrev to set from the
beginning (authored by martinvonz, committed by ).
REPOSITORY
rHG Mercurial
CHANGES SINCE LAST UPDATE
https://phab.mercurial-scm.org/D4507?vs=10840=10842
REVISION DETAIL
https://phab.mercurial-scm.org/D4507
AFFECTED FILES
mercurial/ancestor.py
CHANGE DETAILS
diff --git a/mercurial/ancestor.py b/mercurial/ancestor.py
--- a/mercurial/ancestor.py
+++ b/mercurial/ancestor.py
@@ -311,15 +311,14 @@
If inclusive is True, the source revisions are also yielded. The
reverse revision number order is still enforced."""
-seen = set()
+seen = {nullrev}
revs = self._initrevs
parentrevs = self._parentrevs
stoprev = self._stoprev
schedule = heapq.heappush
nextitem = heapq.heappop
see = seen.add
-see(nullrev)
if self._inclusive:
visit = [-r for r in revs]
To: martinvonz, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Re: [PATCH] phase: report number of non-public changeset alongside the new range
On Fri, 07 Sep 2018 11:07:52 -0400, Boris Feld wrote:
> # HG changeset patch
> # User Boris Feld
> # Date 1535586801 -7200
> # Thu Aug 30 01:53:21 2018 +0200
> # Node ID 5f931c1d1422e9a8c08cca3a59804b25a1183449
> # Parent ab452995eafffa69c34e863e4d8c03e163d8f3ad
> # EXP-Topic phase-report
> # Available At https://bitbucket.org/octobus/mercurial-devel/
> # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r
> 5f931c1d1422
> phase: report number of non-public changeset alongside the new range
Queued, thanks.
> -repo.ui.status(_('new changesets %s\n') % revrange)
> +draft = len(repo.revs('%ld and draft()', revs))
> +secret = len(repo.revs('%ld and secret()', revs))
> +if not (draft or secret):
> +msg = _('new changesets %s\n') % revrange
> +elif draft and secret:
> +msg = _('new changesets %s (%d drafts, %d secrets)\n')
> +msg %= (revrange, draft, secret)
> +elif draft:
> +msg = _('new changesets %s (%d drafts)\n')
> +msg %= (revrange, draft)
> +elif secret:
> +msg = _('new changesets %s (%d secrets)\n')
> +msg %= (revrange, secret)
Added "else: raise ProgrammingError" to make sure msg is otherwise defined.
> +repo.ui.status(msg)
Can you add tests for 'draft and secret' and 'not draft and secret' cases?
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Re: [PATCH 1 of 2] tests: conditionalize an error message about unlinking a non empty directory
On Sat, 08 Sep 2018 00:04:41 -0400, Matt Harbison wrote: > # HG changeset patch > # User Matt Harbison > # Date 1536377989 14400 > # Fri Sep 07 23:39:49 2018 -0400 > # Node ID cf6130dd324b0eed4c902b45df503be429fe413f > # Parent a60dae060bc83e773bdbafb2432bc4cf387ebce2 > tests: conditionalize an error message about unlinking a non empty directory Queued, thanks. ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D4508: lazyancestors: reuse __iter__ implementation in __contains__
yuja added a comment. > +self._containsseen = set() > +self._containsiter = iter(self) Perhaps __iter__() needs to be extracted to a free function to avoid reference cycle. REPOSITORY rHG Mercurial REVISION DETAIL https://phab.mercurial-scm.org/D4508 To: martinvonz, #hg-reviewers Cc: yuja, mercurial-devel ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Re: D4508: lazyancestors: reuse __iter__ implementation in __contains__
> +self._containsseen = set() > +self._containsiter = iter(self) Perhaps __iter__() needs to be extracted to a free function to avoid reference cycle. ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D4508: lazyancestors: reuse __iter__ implementation in __contains__
martinvonz created this revision. Herald added a subscriber: mercurial-devel. Herald added a reviewer: hg-reviewers. REVISION SUMMARY There was a comment in the code that said "Trying to do both __iter__ and __contains__ using the same visit heap and seen set is complex enough that it slows down both. Keep them separate.". However, it seems easy and efficient to make __contains__ keep an iterator across calls. I couldn't measure any slowdown from `hg bundle --all` (which seem to call lazyancestors.__contains__ frequently). 1HG: -- REPOSITORY rHG Mercurial REVISION DETAIL https://phab.mercurial-scm.org/D4508 AFFECTED FILES mercurial/ancestor.py CHANGE DETAILS diff --git a/mercurial/ancestor.py b/mercurial/ancestor.py --- a/mercurial/ancestor.py +++ b/mercurial/ancestor.py @@ -277,18 +277,8 @@ self._stoprev = stoprev self._inclusive = inclusive -# Initialize data structures for __contains__. -# For __contains__, we use a heap rather than a deque because -# (a) it minimizes the number of parentrevs calls made -# (b) it makes the loop termination condition obvious -# Python's heap is a min-heap. Multiply all values by -1 to convert it -# into a max-heap. -self._containsvisit = [-rev for rev in revs] -heapq.heapify(self._containsvisit) -if inclusive: -self._containsseen = set(revs) -else: -self._containsseen = set() +self._containsseen = set() +self._containsiter = iter(self) def __nonzero__(self): """False if the set is empty, True otherwise.""" @@ -344,35 +334,29 @@ def __contains__(self, target): """Test whether target is an ancestor of self._initrevs.""" -# Trying to do both __iter__ and __contains__ using the same visit -# heap and seen set is complex enough that it slows down both. Keep -# them separate. seen = self._containsseen if target in seen: return True +iter = self._containsiter +if iter is None: +# Iterator exhausted +return False # Only integer target is valid, but some callers expect 'None in self' # to be False. So we explicitly allow it. if target is None: return False -parentrevs = self._parentrevs -visit = self._containsvisit -stoprev = self._stoprev -heappop = heapq.heappop -heappush = heapq.heappush see = seen.add - -targetseen = False - -while visit and -visit[0] > target and not targetseen: -for parent in parentrevs(-heappop(visit)): -if parent < stoprev or parent in seen: -continue -# We need to make sure we push all parents into the heap so -# that we leave it in a consistent state for future calls. -heappush(visit, -parent) -see(parent) -if parent == target: -targetseen = True - -return targetseen +try: +while True: +rev = next(iter) +see(rev) +if rev == target: +return True +if rev < target: +return False +except StopIteration: +# Set to None to indicate fast-path can be used next time, and to +# free up memory. +self._containsiter = None +return False To: martinvonz, #hg-reviewers Cc: mercurial-devel ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D4507: ancestors: add nullrev to set from the beginning
martinvonz created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.
REPOSITORY
rHG Mercurial
REVISION DETAIL
https://phab.mercurial-scm.org/D4507
AFFECTED FILES
mercurial/ancestor.py
CHANGE DETAILS
diff --git a/mercurial/ancestor.py b/mercurial/ancestor.py
--- a/mercurial/ancestor.py
+++ b/mercurial/ancestor.py
@@ -311,15 +311,14 @@
If inclusive is True, the source revisions are also yielded. The
reverse revision number order is still enforced."""
-seen = set()
+seen = {nullrev}
revs = self._initrevs
parentrevs = self._parentrevs
stoprev = self._stoprev
schedule = heapq.heappush
nextitem = heapq.heappop
see = seen.add
-see(nullrev)
if self._inclusive:
visit = [-r for r in revs]
To: martinvonz, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Re: [PATCH 2 of 2] ancestor: filter out initial revisions lower than stoprev
On Fri, Sep 7, 2018 at 9:50 PM Yuya Nishihara wrote: > On Fri, 7 Sep 2018 20:49:32 -0700, Martin von Zweigbergk wrote: > > Good catch. However, I think the docstring needs updating. I think it > > explicitly said that inclusive mode emits the initial revs ignoring > > stoprev. > > That part was for __iter__(), and was rewritten by Boris' patch. Ah, right, I had already forgotten that. Sorry about the noise. I'm queuing this series, thanks. ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
