[PATCH] lfs: explicitly add the Content-Length header when uploading blobs, for py3
# HG changeset patch # User Matt Harbison # Date 1548729306 18000 # Mon Jan 28 21:35:06 2019 -0500 # Node ID 1bc01490178a97d4af903c87581bc270362bc079 # Parent 7eb7637e34bf412374a165fbd234022a5a804683 lfs: explicitly add the Content-Length header when uploading blobs, for py3 This was the reason for test-lfs-test-server.t#git-server complaining about an "invalid byte in chunk length". For some reason if this isn't explicitly added, py3.7.1 is adding `transfer-encoding: chunked` as well as `Content-length: x`. Wireshark flagged this as malformed. However, if this is set, it doesn't bother with `transfer-encoding`. Before this patch with py3: PUT /objects/31cf46fbc4ecd458a0943c5b4881f1f5a6dd36c53d6167d5b69ac45149b38e5b HTTP/1.1 Accept-Encoding: identity Content-length: 12 accept: application/vnd.git-lfs content-type: application/octet-stream host: localhost:20062 transfer-encoding: chunked user-agent: git-lfs/2.3.4 (Mercurial 4.9rc0+149-7eb7637e34bf) Before this patch with py27: PUT /objects/31cf46fbc4ecd458a0943c5b4881f1f5a6dd36c53d6167d5b69ac45149b38e5b HTTP/1.1 Accept-Encoding: identity accept: application/vnd.git-lfs content-type: application/octet-stream content-length: 12 host: localhost:20062 user-agent: git-lfs/2.3.4 (Mercurial 4.9rc0+149-7eb7637e34bf+20190128) With this patch and py3, the content is the same as the py27 example. RFC2616 says to ignore `Content-Length` if `Transfer-Encoding` is present, so maybe there's nothing to do in the hg-server side (though I'm not sure which it is using if presented both). Maybe chunked encoding is better to do? If someone knows how to suppress the `Content-Length`, we can try that instead. diff --git a/contrib/python3-whitelist b/contrib/python3-whitelist --- a/contrib/python3-whitelist +++ b/contrib/python3-whitelist @@ -352,6 +352,7 @@ test-lfconvert.t test-lfs-bundle.t test-lfs-largefiles.t test-lfs-pointer.py +test-lfs-test-server.t test-lfs.t test-linelog.py test-linerange.py diff --git a/hgext/lfs/blobstore.py b/hgext/lfs/blobstore.py --- a/hgext/lfs/blobstore.py +++ b/hgext/lfs/blobstore.py @@ -428,6 +428,7 @@ class _gitlfsremote(object): request.data = filewithprogress(localstore.open(oid), None) request.get_method = lambda: r'PUT' request.add_header(r'Content-Type', r'application/octet-stream') +request.add_header(r'Content-Length', len(request.data)) for k, v in headers: request.add_header(pycompat.strurl(k), pycompat.strurl(v)) ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Re: [PATCH 1 of 7] perf: document perfheads
On Tue, Jan 29, 2019 at 1:45 AM Boris Feld wrote: > # HG changeset patch > # User Boris Feld > # Date 1548445951 18000 > # Fri Jan 25 14:52:31 2019 -0500 > # Node ID f39f3732ef42fe901b418e23433475076b599757 > # Parent 431cf2c8c83969a68d132d6ba92af1998e147927 > # EXP-Topic perf-ext > # Available At https://bitbucket.org/octobus/mercurial-devel/ > # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r > f39f3732ef42 > perf: document perfheads > Queued 1-6, many thanks for continuous work on performance testing. ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Re: [PATCH 5 of 7] perf: add a no-lookup variant to perfindex
On Tue, Jan 29, 2019 at 3:30 AM Pulkit Goyal <7895pul...@gmail.com> wrote: > > > On Tue, Jan 29, 2019 at 1:49 AM Boris Feld wrote: > >> # HG changeset patch >> # User Boris Feld >> # Date 1548459828 18000 >> # Fri Jan 25 18:43:48 2019 -0500 >> # Node ID 2518d5acdc4e994193ca5b70f945983a61e22ff5 >> # Parent 9a0d513d684855e3fbdd023c1fe84e7bad6579c4 >> # EXP-Topic perf-ext >> # Available At https://bitbucket.org/octobus/mercurial-devel/ >> # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r >> 2518d5acdc4e >> perf: add a no-lookup variant to perfindex >> >> It is useful to check how long it takes to create a index object without >> doing >> anything with it. We add a new flag dedicated to that. >> >> diff --git a/contrib/perf.py b/contrib/perf.py >> --- a/contrib/perf.py >> +++ b/contrib/perf.py >> @@ -1017,6 +1017,7 @@ def perfignore(ui, repo, **opts): >> >> @command(b'perfindex', [ >> (b'', b'rev', b'', b'revision to be looked up (default >> tip)'), >> +(b'', b'no-lookup', None, b'do not revision lookup post >> creation'), >> ] + formatteropts) >> def perfindex(ui, repo, **opts): >> """benchmark index creation time followed by a lookup >> @@ -1031,7 +1032,9 @@ def perfindex(ui, repo, **opts): >> opts = _byteskwargs(opts) >> timer, fm = gettimer(ui, opts) >> mercurial.revlog._prereadsize = 2**24 # disable lazy parser in old hg >> -if opts[b'rev'] is None: >> +if opts[b'no_lookup']: >> +n = None >> +elif opts[b'rev'] is None: >> n = repo[b"tip"].node() >> > > We should disallow passing both '--no-lookup' and '--rev'. > Ignore this. I see you have done that in next patch. ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Re: [PATCH 5 of 7] perf: add a no-lookup variant to perfindex
On Tue, Jan 29, 2019 at 1:49 AM Boris Feld wrote: > # HG changeset patch > # User Boris Feld > # Date 1548459828 18000 > # Fri Jan 25 18:43:48 2019 -0500 > # Node ID 2518d5acdc4e994193ca5b70f945983a61e22ff5 > # Parent 9a0d513d684855e3fbdd023c1fe84e7bad6579c4 > # EXP-Topic perf-ext > # Available At https://bitbucket.org/octobus/mercurial-devel/ > # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r > 2518d5acdc4e > perf: add a no-lookup variant to perfindex > > It is useful to check how long it takes to create a index object without > doing > anything with it. We add a new flag dedicated to that. > > diff --git a/contrib/perf.py b/contrib/perf.py > --- a/contrib/perf.py > +++ b/contrib/perf.py > @@ -1017,6 +1017,7 @@ def perfignore(ui, repo, **opts): > > @command(b'perfindex', [ > (b'', b'rev', b'', b'revision to be looked up (default tip)'), > +(b'', b'no-lookup', None, b'do not revision lookup post > creation'), > ] + formatteropts) > def perfindex(ui, repo, **opts): > """benchmark index creation time followed by a lookup > @@ -1031,7 +1032,9 @@ def perfindex(ui, repo, **opts): > opts = _byteskwargs(opts) > timer, fm = gettimer(ui, opts) > mercurial.revlog._prereadsize = 2**24 # disable lazy parser in old hg > -if opts[b'rev'] is None: > +if opts[b'no_lookup']: > +n = None > +elif opts[b'rev'] is None: > n = repo[b"tip"].node() > We should disallow passing both '--no-lookup' and '--rev'. ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 7 of 7] perf: add a perfnodemap command
# HG changeset patch
# User Boris Feld
# Date 1548460545 18000
# Fri Jan 25 18:55:45 2019 -0500
# Node ID 884df78c77a1c3b11ca2e5728ae6b5ebd7f6d2aa
# Parent 1e135ddcbdd182a6b7ae7b121d254fb58c7fd658
# EXP-Topic perf-ext
# Available At https://bitbucket.org/octobus/mercurial-devel/
# hg pull https://bitbucket.org/octobus/mercurial-devel/ -r
884df78c77a1
perf: add a perfnodemap command
The command focus on timing of the nodemap object itself.
diff --git a/contrib/perf.py b/contrib/perf.py
--- a/contrib/perf.py
+++ b/contrib/perf.py
@@ -1038,7 +1038,8 @@ def perfindex(ui, repo, **opts):
* -1:
* -1: + 0
-It is not currently possible to check for lookup of a missing node."""
+It is not currently possible to check for lookup of a missing node. For
+deeper lookup benchmarking, checkout the `perfnodemap` command."""
import mercurial.revlog
opts = _byteskwargs(opts)
timer, fm = gettimer(ui, opts)
@@ -1068,6 +1069,58 @@ def perfindex(ui, repo, **opts):
timer(d, setup=setup)
fm.end()
+@command(b'perfnodemap', [
+(b'', b'rev', [], b'revision to be looked up (default tip)'),
+] + formatteropts)
+def perfnodemap(ui, repo, **opts):
+"""benchmark the time necessary to look up revision from a cold nodemap
+
+Depending on the implementation, the amount and order of revision we look
+up can varies. Example of useful set to test:
+* tip
+* 0
+* -10:
+* :10
+* -10: + :10
+* :10: + -10:
+* -1:
+* -1: + 0
+
+The command currently focus on valid binary lookup. Benchmarking for
+hexlookup, prefix lookup and missing lookup would also be valuable.
+"""
+import mercurial.revlog
+opts = _byteskwargs(opts)
+timer, fm = gettimer(ui, opts)
+mercurial.revlog._prereadsize = 2**24 # disable lazy parser in old hg
+
+unfi = repo.unfiltered()
+# find the filecache func directly
+# This avoid polluting the benchmark with the filecache logic
+makecl = unfi.__class__.changelog.func
+if not opts[b'rev']:
+raise error.Abort('use --rev to specify revisions to look up')
+revs = scmutil.revrange(repo, opts[b'rev'])
+cl = repo.changelog
+nodes = [cl.node(r) for r in revs]
+
+# use a list to pass reference to a nodemap from one closure to the next
+nodeget = [None]
+def setnodeget():
+# probably not necessary, but for good measure
+clearchangelog(unfi)
+nodeget[0] = makecl(unfi).nodemap.get
+
+def setup():
+setnodeget()
+def d():
+get = nodeget[0]
+for n in nodes:
+get(n)
+
+timer(d, setup=setup)
+fm.end()
+
@command(b'perfstartup', formatteropts)
def perfstartup(ui, repo, **opts):
opts = _byteskwargs(opts)
diff --git a/tests/test-contrib-perf.t b/tests/test-contrib-perf.t
--- a/tests/test-contrib-perf.t
+++ b/tests/test-contrib-perf.t
@@ -109,6 +109,8 @@ perfstatus
perfmoonwalk benchmark walking the changelog backwards
perfnodelookup
(no help text available)
+ perfnodemap benchmark the time necessary to look up revision from a cold
+ nodemap
perfparents (no help text available)
perfpathcopies
benchmark the copy tracing logic
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 6 of 7] perf: support looking up multiple revisions
# HG changeset patch
# User Boris Feld
# Date 1548663631 18000
# Mon Jan 28 03:20:31 2019 -0500
# Node ID 1e135ddcbdd182a6b7ae7b121d254fb58c7fd658
# Parent 2518d5acdc4e994193ca5b70f945983a61e22ff5
# EXP-Topic perf-ext
# Available At https://bitbucket.org/octobus/mercurial-devel/
# hg pull https://bitbucket.org/octobus/mercurial-devel/ -r
1e135ddcbdd1
perf: support looking up multiple revisions
The nodemap code has optimisations around the number of lookup we actually made.
As a result, being able to specify multiple revisions to look up is important
when measuring performances. One can now specify full revspecs with the --rev
arguments.
diff --git a/contrib/perf.py b/contrib/perf.py
--- a/contrib/perf.py
+++ b/contrib/perf.py
@@ -1016,7 +1016,7 @@ def perfignore(ui, repo, **opts):
fm.end()
@command(b'perfindex', [
-(b'', b'rev', b'', b'revision to be looked up (default tip)'),
+(b'', b'rev', [], b'revision to be looked up (default tip)'),
(b'', b'no-lookup', None, b'do not revision lookup post creation'),
] + formatteropts)
def perfindex(ui, repo, **opts):
@@ -1025,7 +1025,18 @@ def perfindex(ui, repo, **opts):
The default is to look `tip` up. Depending on the index implementation,
the revision looked up can matters. For example, an implementation
scanning the index will have a faster lookup time for `--rev tip` than for
-`--rev 0`.
+`--rev 0`. The number of looked up revisions and their order can also
+matters.
+
+Example of useful set to test:
+* tip
+* 0
+* -10:
+* :10
+* -10: + :10
+* :10: + -10:
+* -1:
+* -1: + 0
It is not currently possible to check for lookup of a missing node."""
import mercurial.revlog
@@ -1033,12 +1044,15 @@ def perfindex(ui, repo, **opts):
timer, fm = gettimer(ui, opts)
mercurial.revlog._prereadsize = 2**24 # disable lazy parser in old hg
if opts[b'no_lookup']:
-n = None
-elif opts[b'rev'] is None:
-n = repo[b"tip"].node()
+if opts['rev']:
+raise error.Abort('--no-lookup and --rev are mutually exclusive')
+nodes = []
+elif not opts[b'rev']:
+nodes = [repo[b"tip"].node()]
else:
-rev = scmutil.revsingle(repo, opts[b'rev'])
-n = repo[rev].node()
+revs = scmutil.revrange(repo, opts[b'rev'])
+cl = repo.changelog
+nodes = [cl.node(r) for r in revs]
unfi = repo.unfiltered()
# find the filecache func directly
@@ -1049,7 +1063,7 @@ def perfindex(ui, repo, **opts):
clearchangelog(unfi)
def d():
cl = makecl(unfi)
-if n is not None:
+for n in nodes:
cl.rev(n)
timer(d, setup=setup)
fm.end()
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 5 of 7] perf: add a no-lookup variant to perfindex
# HG changeset patch # User Boris Feld # Date 1548459828 18000 # Fri Jan 25 18:43:48 2019 -0500 # Node ID 2518d5acdc4e994193ca5b70f945983a61e22ff5 # Parent 9a0d513d684855e3fbdd023c1fe84e7bad6579c4 # EXP-Topic perf-ext # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 2518d5acdc4e perf: add a no-lookup variant to perfindex It is useful to check how long it takes to create a index object without doing anything with it. We add a new flag dedicated to that. diff --git a/contrib/perf.py b/contrib/perf.py --- a/contrib/perf.py +++ b/contrib/perf.py @@ -1017,6 +1017,7 @@ def perfignore(ui, repo, **opts): @command(b'perfindex', [ (b'', b'rev', b'', b'revision to be looked up (default tip)'), +(b'', b'no-lookup', None, b'do not revision lookup post creation'), ] + formatteropts) def perfindex(ui, repo, **opts): """benchmark index creation time followed by a lookup @@ -1031,7 +1032,9 @@ def perfindex(ui, repo, **opts): opts = _byteskwargs(opts) timer, fm = gettimer(ui, opts) mercurial.revlog._prereadsize = 2**24 # disable lazy parser in old hg -if opts[b'rev'] is None: +if opts[b'no_lookup']: +n = None +elif opts[b'rev'] is None: n = repo[b"tip"].node() else: rev = scmutil.revsingle(repo, opts[b'rev']) @@ -1046,7 +1049,8 @@ def perfindex(ui, repo, **opts): clearchangelog(unfi) def d(): cl = makecl(unfi) -cl.rev(n) +if n is not None: +cl.rev(n) timer(d, setup=setup) fm.end() ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 4 of 7] perf: add some documentation to perfindex
# HG changeset patch # User Boris Feld # Date 1548668860 18000 # Mon Jan 28 04:47:40 2019 -0500 # Node ID 9a0d513d684855e3fbdd023c1fe84e7bad6579c4 # Parent 1e7a5bec6326c537b65ffb59b51c4bb34ea189cc # EXP-Topic perf-ext # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 9a0d513d6848 perf: add some documentation to perfindex It seems useful to document how the arguments can affect the benchmark. diff --git a/contrib/perf.py b/contrib/perf.py --- a/contrib/perf.py +++ b/contrib/perf.py @@ -1019,6 +1019,14 @@ def perfignore(ui, repo, **opts): (b'', b'rev', b'', b'revision to be looked up (default tip)'), ] + formatteropts) def perfindex(ui, repo, **opts): +"""benchmark index creation time followed by a lookup + +The default is to look `tip` up. Depending on the index implementation, +the revision looked up can matters. For example, an implementation +scanning the index will have a faster lookup time for `--rev tip` than for +`--rev 0`. + +It is not currently possible to check for lookup of a missing node.""" import mercurial.revlog opts = _byteskwargs(opts) timer, fm = gettimer(ui, opts) diff --git a/tests/test-contrib-perf.t b/tests/test-contrib-perf.t --- a/tests/test-contrib-perf.t +++ b/tests/test-contrib-perf.t @@ -93,7 +93,7 @@ perfstatus find statistic about potential parameters for the 'perftracecopies' perfignorebenchmark operation related to computing ignore - perfindex (no help text available) + perfindex benchmark index creation time followed by a lookup perflinelogedits (no help text available) perfloadmarkers ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 3 of 7] perf: move cache clearing in the `setup` step of `perfheads`
# HG changeset patch # User Boris Feld # Date 1548445999 18000 # Fri Jan 25 14:53:19 2019 -0500 # Node ID 1e7a5bec6326c537b65ffb59b51c4bb34ea189cc # Parent dbc0d2061b0584c423f4ecf007cd0125e16b11d1 # EXP-Topic perf-ext # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 1e7a5bec6326 perf: move cache clearing in the `setup` step of `perfheads` The cache clearing is pretty fast, but this seems more "correct". diff --git a/contrib/perf.py b/contrib/perf.py --- a/contrib/perf.py +++ b/contrib/perf.py @@ -539,10 +539,11 @@ def perfheads(ui, repo, **opts): opts = _byteskwargs(opts) timer, fm = gettimer(ui, opts) cl = repo.changelog +def s(): +clearcaches(cl) def d(): len(cl.headrevs()) -clearcaches(cl) -timer(d) +timer(d, setup=s) fm.end() @command(b'perftags', formatteropts+ ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 1 of 7] perf: document perfheads
# HG changeset patch # User Boris Feld # Date 1548445951 18000 # Fri Jan 25 14:52:31 2019 -0500 # Node ID f39f3732ef42fe901b418e23433475076b599757 # Parent 431cf2c8c83969a68d132d6ba92af1998e147927 # EXP-Topic perf-ext # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r f39f3732ef42 perf: document perfheads diff --git a/contrib/perf.py b/contrib/perf.py --- a/contrib/perf.py +++ b/contrib/perf.py @@ -535,6 +535,7 @@ def clearcaches(cl): @command(b'perfheads', formatteropts) def perfheads(ui, repo, **opts): +"""benchmark the computation of a changelog heads""" opts = _byteskwargs(opts) timer, fm = gettimer(ui, opts) cl = repo.changelog diff --git a/tests/test-contrib-perf.t b/tests/test-contrib-perf.t --- a/tests/test-contrib-perf.t +++ b/tests/test-contrib-perf.t @@ -88,7 +88,7 @@ perfstatus (no help text available) perffncachewrite (no help text available) - perfheads (no help text available) + perfheads benchmark the computation of a changelog heads perfhelper-pathcopies find statistic about potential parameters for the 'perftracecopies' ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 2 of 7] revlog: document cext oddities in terms of object/caches
# HG changeset patch
# User Boris Feld
# Date 1548458522 18000
# Fri Jan 25 18:22:02 2019 -0500
# Node ID dbc0d2061b0584c423f4ecf007cd0125e16b11d1
# Parent f39f3732ef42fe901b418e23433475076b599757
# EXP-Topic perf-ext
# Available At https://bitbucket.org/octobus/mercurial-devel/
# hg pull https://bitbucket.org/octobus/mercurial-devel/ -r
dbc0d2061b05
revlog: document cext oddities in terms of object/caches
This clarify why we just call clearcaches on a single object.
diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -607,6 +607,9 @@ class revlog(object):
self._pcache = {}
try:
+# If we are using the native C version, you are in a fun case
+# where self.index, self.nodemap and self._nodecaches is the same
+# object.
self._nodecache.clearcaches()
except AttributeError:
self._nodecache = {nullid: nullrev}
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
mercurial@41391: 2 new changesets
2 new changesets in mercurial: https://www.mercurial-scm.org/repo/hg/rev/91ec3d9a97f2 changeset: 41390:91ec3d9a97f2 user:Augie Fackler date:Fri Jan 25 21:23:16 2019 -0500 summary: py3: new tests from the ratchet https://www.mercurial-scm.org/repo/hg/rev/bc843e251134 changeset: 41391:bc843e251134 bookmark:@ tag: tip user:Martin von Zweigbergk date:Thu Jan 17 09:18:48 2019 -0800 summary: unshare: use context manager for locks -- Repository URL: https://www.mercurial-scm.org/repo/hg ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D5638: branchmap: encapsulate cache updating in the map itself
mjpieters updated this revision to Diff 13543.
REPOSITORY
rHG Mercurial
CHANGES SINCE LAST UPDATE
https://phab.mercurial-scm.org/D5638?vs=13502=13543
REVISION DETAIL
https://phab.mercurial-scm.org/D5638
AFFECTED FILES
contrib/perf.py
mercurial/branchmap.py
mercurial/localrepo.py
mercurial/statichttprepo.py
mercurial/streamclone.py
CHANGE DETAILS
diff --git a/mercurial/streamclone.py b/mercurial/streamclone.py
--- a/mercurial/streamclone.py
+++ b/mercurial/streamclone.py
@@ -13,7 +13,6 @@
from .i18n import _
from . import (
-branchmap,
cacheutil,
error,
narrowspec,
@@ -174,7 +173,7 @@
repo._writerequirements()
if rbranchmap:
-branchmap.replacecache(repo, rbranchmap)
+repo._branchcaches.replace(repo, rbranchmap)
repo.invalidate()
diff --git a/mercurial/statichttprepo.py b/mercurial/statichttprepo.py
--- a/mercurial/statichttprepo.py
+++ b/mercurial/statichttprepo.py
@@ -13,6 +13,7 @@
from .i18n import _
from . import (
+branchmap,
changelog,
error,
localrepo,
@@ -192,7 +193,7 @@
self.changelog = changelog.changelog(self.svfs)
self._tags = None
self.nodetagscache = None
-self._branchcaches = {}
+self._branchcaches = branchmap.BranchMapCache()
self._revbranchcache = None
self.encodepats = None
self.decodepats = None
diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -992,7 +992,7 @@
self._dirstatevalidatewarned = False
-self._branchcaches = {}
+self._branchcaches = branchmap.BranchMapCache()
self._revbranchcache = None
self._filterpats = {}
self._datafilters = {}
@@ -1520,8 +1520,7 @@
def branchmap(self):
'''returns a dictionary {branch: [branchheads]} with branchheads
ordered by increasing revision number'''
-branchmap.updatecache(self)
-return self._branchcaches[self.filtername]
+return self._branchcaches[self]
@unfilteredmethod
def revbranchcache(self):
@@ -2073,9 +2072,9 @@
return
if tr is None or tr.changes['origrepolen'] < len(self):
-# updating the unfiltered branchmap should refresh all the others,
+# accessing the 'ser ved' branchmap should refresh all the others,
self.ui.debug('updating the branch cache\n')
-branchmap.updatecache(self.filtered('served'))
+self.filtered('served').branchmap()
if full:
rbc = self.revbranchcache()
@@ -2093,7 +2092,7 @@
# can't use delattr on proxy
del self.__dict__[r'_tagscache']
-self.unfiltered()._branchcaches.clear()
+self._branchcaches.clear()
self.invalidatevolatilesets()
self._sparsesignaturecache.clear()
diff --git a/mercurial/branchmap.py b/mercurial/branchmap.py
--- a/mercurial/branchmap.py
+++ b/mercurial/branchmap.py
@@ -43,75 +43,89 @@
'served': 'immutable',
'immutable': 'base'}
-def updatecache(repo):
-"""Update the cache for the given filtered view on a repository"""
-# This can trigger updates for the caches for subsets of the filtered
-# view, e.g. when there is no cache for this filtered view or the cache
-# is stale.
+
+class BranchMapCache(object):
+"""Cache mapping"""
+def __init__(self):
+self._per_filter = {}
-cl = repo.changelog
-filtername = repo.filtername
-bcache = repo._branchcaches.get(filtername)
-if bcache is None or not bcache.validfor(repo):
-# cache object missing or cache object stale? Read from disk
-bcache = branchcache.fromfile(repo)
+def __getitem__(self, repo):
+self.updatecache(repo)
+return self._per_filter[repo.filtername]
+
+def updatecache(self, repo):
+"""Update the cache for the given filtered view on a repository"""
+# This can trigger updates for the caches for subsets of the filtered
+# view, e.g. when there is no cache for this filtered view or the cache
+# is stale.
-revs = []
-if bcache is None:
-# no (fresh) cache available anymore, perhaps we can re-use
-# the cache for a subset, then extend that to add info on missing
-# revisions.
-subsetname = subsettable.get(filtername)
-if subsetname is not None:
-subset = repo.filtered(subsetname)
-bcache = subset.branchmap().copy()
-extrarevs = subset.changelog.filteredrevs - cl.filteredrevs
-revs.extend(r for r in extrarevs if r <= bcache.tiprev)
-else:
-# nothing to fall back on, start empty.
-bcache = branchcache()
+cl = repo.changelog
+filtername = repo.filtername
+bcache = self._per_filter.get(filtername)
+if bcache
D5635: branchmap: make branchcache responsible for reading
mjpieters updated this revision to Diff 13542.
REPOSITORY
rHG Mercurial
CHANGES SINCE LAST UPDATE
https://phab.mercurial-scm.org/D5635?vs=13501=13542
REVISION DETAIL
https://phab.mercurial-scm.org/D5635
AFFECTED FILES
contrib/perf.py
mercurial/branchmap.py
CHANGE DETAILS
diff --git a/mercurial/branchmap.py b/mercurial/branchmap.py
--- a/mercurial/branchmap.py
+++ b/mercurial/branchmap.py
@@ -30,63 +30,6 @@
pack_into = struct.pack_into
unpack_from = struct.unpack_from
-def _filename(repo):
-"""name of a branchcache file for a given repo or repoview"""
-filename = "branch2"
-if repo.filtername:
-filename = '%s-%s' % (filename, repo.filtername)
-return filename
-
-def read(repo):
-f = None
-try:
-f = repo.cachevfs(_filename(repo))
-lineiter = iter(f)
-cachekey = next(lineiter).rstrip('\n').split(" ", 2)
-last, lrev = cachekey[:2]
-last, lrev = bin(last), int(lrev)
-filteredhash = None
-if len(cachekey) > 2:
-filteredhash = bin(cachekey[2])
-bcache = branchcache(tipnode=last, tiprev=lrev,
- filteredhash=filteredhash)
-if not bcache.validfor(repo):
-# invalidate the cache
-raise ValueError(r'tip differs')
-cl = repo.changelog
-for l in lineiter:
-l = l.rstrip('\n')
-if not l:
-continue
-node, state, label = l.split(" ", 2)
-if state not in 'oc':
-raise ValueError(r'invalid branch state')
-label = encoding.tolocal(label.strip())
-node = bin(node)
-if not cl.hasnode(node):
-raise ValueError(
-r'node %s does not exist' % pycompat.sysstr(hex(node)))
-bcache.setdefault(label, []).append(node)
-if state == 'c':
-bcache._closednodes.add(node)
-
-except (IOError, OSError):
-return None
-
-except Exception as inst:
-if repo.ui.debugflag:
-msg = 'invalid branchheads cache'
-if repo.filtername is not None:
-msg += ' (%s)' % repo.filtername
-msg += ': %s\n'
-repo.ui.debug(msg % pycompat.bytestr(inst))
-bcache = None
-
-finally:
-if f:
-f.close()
-
-return bcache
### Nearest subset relation
# Nearest subset of filter X is a filter Y so that:
@@ -107,7 +50,7 @@
revs = []
if bcache is None or not bcache.validfor(repo):
-bcache = read(repo)
+bcache = branchcache.fromfile(repo)
if bcache is None:
subsetname = subsettable.get(filtername)
if subsetname is None:
@@ -181,6 +124,64 @@
This field can be used to avoid changelog reads when determining if a
branch head closes a branch or not.
"""
+@classmethod
+def fromfile(cls, repo):
+f = None
+try:
+f = repo.cachevfs(cls._filename(repo))
+lineiter = iter(f)
+cachekey = next(lineiter).rstrip('\n').split(" ", 2)
+last, lrev = cachekey[:2]
+last, lrev = bin(last), int(lrev)
+filteredhash = None
+if len(cachekey) > 2:
+filteredhash = bin(cachekey[2])
+bcache = cls(tipnode=last, tiprev=lrev, filteredhash=filteredhash)
+if not bcache.validfor(repo):
+# invalidate the cache
+raise ValueError(r'tip differs')
+cl = repo.changelog
+for line in lineiter:
+line = line.rstrip('\n')
+if not line:
+continue
+node, state, label = line.split(" ", 2)
+if state not in 'oc':
+raise ValueError(r'invalid branch state')
+label = encoding.tolocal(label.strip())
+node = bin(node)
+if not cl.hasnode(node):
+raise ValueError(
+r'node %s does not exist' % pycompat.sysstr(hex(node)))
+bcache.setdefault(label, []).append(node)
+if state == 'c':
+bcache._closednodes.add(node)
+
+except (IOError, OSError):
+return None
+
+except Exception as inst:
+if repo.ui.debugflag:
+msg = 'invalid branchheads cache'
+if repo.filtername is not None:
+msg += ' (%s)' % repo.filtername
+msg += ': %s\n'
+repo.ui.debug(msg % pycompat.bytestr(inst))
+bcache = None
+
+finally:
+if f:
+f.close()
+
+return bcache
+
+@staticmethod
+def _filename(repo):
+"""name of a branchcache file for a given repo or repoview"""
+filename = "branch2"
+if repo.filtername:
+filename = '%s-%s' %
Re: [PATCH V3] extdiff: add --per-file and --confirm options
> I do understand however that it would be easy for users to get in a > messy situation so I guess I'll add a "--batch" option to "--per-file" > if that's OK? Another way to solve this would be to actually default to "--prompt" when "--per-file" is specified, and require "--no-prompt" if you want the "batch fire processes" mode for tools like Meld or BeyondCompare. It looks like that's how Git does it, and that would save us an option, which I like. -- l u d o . . 8 0 17 80 ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Re: [PATCH V3] extdiff: add --per-file and --confirm options
> What I pointed out last time was that we shouldn't spawn non-GUI tools > asynchronously. You can see the problem by the following command on Unix. > > $ hg --config extdiff.vimdiff= vimdiff -c. --per-file Mmmh so for this I figured people would use the "--confirm" flag because, if we use the usual (blocking) process launch, there's not much difference anymore (the only difference being that the next vimdiff process launches automatically instead of the user having the press Enter). It would also prevent using this command for programs that support showing multiple diffs, usually in a tabbed way. That's how BeyondCompare works (you can batch-fire multiple processes and it will just keep opening more diff tabs in the same instance). I'm also currently coming up with a script to do this using Vim's clientserver functionality (so you run the hg command and it will open Vim with each file-diff in a tab). I do understand however that it would be easy for users to get in a messy situation so I guess I'll add a "--batch" option to "--per-file" if that's OK? > And perhaps, it's too late to compare ui.config value based on self._cmd > here. If self._cmd is the key of [diff-tools] for example, we shouldn't look > for [merge-tools]. Ah good point, thanks. -- l u d o . . 8 0 17 80 ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Re: [PATCH STABLE] revlog: make sure we never use sparserevlog without general delta (issue6056)
On Fri, 25 Jan 2019 15:00:52 -0500, Boris Feld wrote:
> # HG changeset patch
> # User Boris Feld
> # Date 1548372167 18000
> # Thu Jan 24 18:22:47 2019 -0500
> # Branch stable
> # Node ID c7661fdcaeac457aa5b8ae6b6a10b259713934b9
> # Parent 26ee61c33dee366576a0f7266fea9834985e78ec
> # EXP-Topic issue6056
> # Available At https://bitbucket.org/octobus/mercurial-devel/
> # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r
> c7661fdcaeac
> revlog: make sure we never use sparserevlog without general delta (issue6056)
>
> We are getting user report where the delta code tries to use `sparse-revlog`
> logic on repository where `generaldelta` is disabled. This can't work so we
> ensure the two booleans have a consistent value.
>
> Creating this kind of repository is not expected to be possible and we weren't
> able to create one for testing purpose. But if users run into this, we better
> put stronger protection.
Mathias, do you have any idea how the broken repository could be created?
> --- a/mercurial/revlog.py
> +++ b/mercurial/revlog.py
> @@ -497,6 +497,9 @@ class revlog(object):
> else:
> raise error.RevlogError(_('unknown version (%d) in revlog %s') %
> (fmt, self.indexfile))
> +# sparse-revlog can't be on without general-delta (issue6056)
> +if not self._generaldelta:
> +self._sparserevlog = False
Maybe this can be handled at localrepo.ensurerequirements*() or
esolverevlogstorevfsoptions(), where ui is available so we can add
some debug/warning messages.
And, UnboundLocalError is most likely a coding error. If that's due to
contract violation, it would be better to raise a proper exception instead.
if sparse and not gdelta:
raise error.ProgrammingError(...) # or StorageError?
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Re: [PATCH 07 of 11] py3: byteify the decoded JSON responses upon receipt in the LFS blobstore
On Mon, 28 Jan 2019 00:20:53 -0500, Matt Harbison wrote: > # HG changeset patch > # User Matt Harbison > # Date 1548629295 18000 > # Sun Jan 27 17:48:15 2019 -0500 > # Node ID b98988169d4a9c7890b93091683fa4ec38d61a47 > # Parent 1d6f4c32abc28ea54e3d1d8487a1d773033aedf0 > py3: byteify the decoded JSON responses upon receipt in the LFS blobstore > -return response > +def encodestr(x): > +if isinstance(x, unicode): Fixed s/unicode/pycompat.unicode/ in flight. > +return x.encode(u'utf-8') > +return x > + > +return pycompat.rapply(encodestr, response) I assume here JSON strings are encoding agnostic (i.e. ASCII.) If the JSON had a filename for example, it wouldn't be always correct to encode data as UTF-8. ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Re: [PATCH 03 of 11] py3: raw stringify various things in the LFS server module
On Mon, 28 Jan 2019 00:20:49 -0500, Matt Harbison wrote:
> # HG changeset patch
> # User Matt Harbison
> # Date 1548568239 18000
> # Sun Jan 27 00:50:39 2019 -0500
> # Node ID 9a35d8754bdfaac20d1931dbd6a82233d11f4722
> # Parent 7768ad6f53d8e08e90918e50201fc2fab4f9eff6
> py3: raw stringify various things in the LFS server module
> -operation = lfsreq.get('operation')
> -if operation not in ('upload', 'download'):
> +operation = lfsreq.get(r'operation')
> +operation = pycompat.bytestr(operation)
If we don't care non-ASCII "operation" value and don't want Unicode exception,
using pycompat.sysbytes() is safer.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Re: [PATCH 10 of 11] py3: conditionalize some LFS test output
On Mon, 28 Jan 2019 00:33:21 -0500, Matt Harbison wrote: > Second is this: > >--- c:/Users/Matt/hg/tests/test-lfs-serve.t >+++ c:/Users/Matt/hg/tests/test-lfs-serve.t#lfsremote-on.err >@@ -665,7 +665,7 @@ > #if lfsremote-on > $ cat $TESTTMP/errors.log | grep '^[A-Z]' > Traceback (most recent call last): >- ValueError: no common changegroup version >+ ValueError: b'no common changegroup version' > #else > $ cat $TESTTMP/errors.log > #endif > > Some ValueErrors have translated strings, but most don't. Very few have > r'' prefixes. So IDK if it's a reasonable plan to add a subclass of > ValueError to the errors module that would take bytes in the constructor > and translate to str for the native class. Maybe we shouldn't abuse ValueError for our custom errors. ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Re: [PATCH 01 of 11] py3: add b'' and r'' prefixes to LFS tests
On Mon, 28 Jan 2019 00:20:47 -0500, Matt Harbison wrote: > # HG changeset patch > # User Matt Harbison > # Date 1548566777 18000 > # Sun Jan 27 00:26:17 2019 -0500 > # Node ID 2b6e1f299ac278097680b54134f87c12e90bf909 > # Parent 873a28d7e962e933f3f67dd263369c35450035f6 > py3: add b'' and r'' prefixes to LFS tests Queued the series, thanks. ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Re: [PATCH 04 of 11] py3: byteify the LFS blobstore module
On Mon, 28 Jan 2019 00:20:50 -0500, Matt Harbison wrote:
> # HG changeset patch
> # User Matt Harbison
> # Date 1548620368 18000
> # Sun Jan 27 15:19:28 2019 -0500
> # Node ID d639ccd9f7139991165d04b4601b06c3341a79cd
> # Parent 9a35d8754bdfaac20d1931dbd6a82233d11f4722
> py3: byteify the LFS blobstore module
> @@ -303,29 +305,32 @@ class _gitlfsremote(object):
> rawjson = rsp.read()
> except util.urlerr.httperror as ex:
> hints = {
> -400: _('check that lfs serving is enabled on %s and "%s" is '
> +400: _(b'check that lfs serving is enabled on %s and "%s" is
> '
> 'supported') % (self.baseurl, action),
> -404: _('the "lfs.url" config may be used to override %s')
> +404: _(b'the "lfs.url" config may be used to override %s')
> % self.baseurl,
> }
> -hint = hints.get(ex.code, _('api=%s, action=%s') % (url, action))
> -raise LfsRemoteError(_('LFS HTTP error: %s') % ex, hint=hint)
> +hint = hints.get(ex.code, _(b'api=%s, action=%s') % (url,
> action))
> +raise LfsRemoteError(
> +_(b'LFS HTTP error: %s') % stringutil.forcebytestr(ex),
> + hint=hint)
> except util.urlerr.urlerror as ex:
> -hint = (_('the "lfs.url" config may be used to override %s')
> +hint = (_(b'the "lfs.url" config may be used to override %s')
> % self.baseurl)
> -raise LfsRemoteError(_('LFS error: %s') % _urlerrorreason(ex),
> +raise LfsRemoteError(_(b'LFS error: %s') % _urlerrorreason(ex),
> hint=hint)
> try:
> response = json.loads(rawjson)
> except ValueError:
> -raise LfsRemoteError(_('LFS server returns invalid JSON: %s')
> - % rawjson)
> +raise LfsRemoteError(_(b'LFS server returns invalid JSON: %s')
> + % rawjson.encode("utf-8"))
Does rsp.read() return a unicode string? If it doesn't, this could crash if
rawjson wasn't ASCII bytes.
>
> if self.ui.debugflag:
> -self.ui.debug('Status: %d\n' % rsp.status)
> +self.ui.debug(b'Status: %d\n' % rsp.status)
> # lfs-test-server and hg serve return headers in different order
> -self.ui.debug('%s\n'
> - % '\n'.join(sorted(str(rsp.info()).splitlines(
> +headers = pycompat.bytestr(rsp.info())
> +self.ui.debug(b'%s\n'
> + % b'\n'.join(sorted(headers.splitlines(
I'm not sure if rsp.info() never contains non-ASCII bytes.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D5730: py3: more tests from the ratchet
This revision was automatically updated to reflect the committed changes. Closed by commit rHGd3a7c743ae3b: py3: more tests from the ratchet (authored by durin42, committed by ). REPOSITORY rHG Mercurial CHANGES SINCE LAST UPDATE https://phab.mercurial-scm.org/D5730?vs=13540=13541 REVISION DETAIL https://phab.mercurial-scm.org/D5730 AFFECTED FILES contrib/python3-whitelist CHANGE DETAILS diff --git a/contrib/python3-whitelist b/contrib/python3-whitelist --- a/contrib/python3-whitelist +++ b/contrib/python3-whitelist @@ -242,6 +242,7 @@ test-generaldelta.t test-getbundle.t test-git-export.t +test-githelp.t test-globalopts.t test-glog-beautifygraph.t test-glog-topological.t @@ -290,6 +291,7 @@ test-histedit-obsolete.t test-histedit-outgoing.t test-histedit-templates.t +test-http-api-httpv2.t test-http-api.t test-http-branchmap.t test-http-bundle1.t @@ -460,6 +462,7 @@ test-newcgi.t test-newercgi.t test-nointerrupt.t +test-notify-changegroup.t test-obshistory.t test-obsmarker-template.t test-obsmarkers-effectflag.t @@ -471,6 +474,7 @@ test-obsolete-divergent.t test-obsolete-tag-cache.t test-obsolete.t +test-oldcgi.t test-origbackup-conflict.t test-pager-legacy.t test-pager.t @@ -653,6 +657,7 @@ test-ssh.t test-sshserver.py test-stack.t +test-static-http.t test-status-color.t test-status-inprocess.py test-status-rev.t To: durin42, pulkit, #hg-reviewers Cc: mercurial-devel ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D5730: py3: more tests from the ratchet
pulkit added inline comments. INLINE COMMENTS > python3-whitelist:660 > test-stack.t > +test-static-http.t > test-status-color.t This does not pass on py3.7. --- /home/pulkit/repo/pushaccess/tests/test-static-http.t +++ /home/pulkit/repo/pushaccess/tests/test-static-http.t.err @@ -227,9 +227,9 @@ /.hg/requires /.hg/store/00changelog.i /.hg/store/00manifest.i - /.hg/store/data/%7E2ehgsub.i - /.hg/store/data/%7E2ehgsubstate.i /.hg/store/data/a.i + /.hg/store/data/~2ehgsub.i + /.hg/store/data/~2ehgsubstate.i /notarepo/.hg/00changelog.i /notarepo/.hg/requires /remote-with-names/.hg/bookmarks @@ -243,8 +243,8 @@ /remote-with-names/.hg/requires /remote-with-names/.hg/store/00changelog.i /remote-with-names/.hg/store/00manifest.i - /remote-with-names/.hg/store/data/%7E2ehgtags.i /remote-with-names/.hg/store/data/foo.i + /remote-with-names/.hg/store/data/~2ehgtags.i /remote/.hg/bookmarks /remote/.hg/bookmarks.current /remote/.hg/cache/branch2-base @@ -258,10 +258,10 @@ /remote/.hg/requires /remote/.hg/store/00changelog.i /remote/.hg/store/00manifest.i - /remote/.hg/store/data/%7E2edotfile%20with%20spaces.i - /remote/.hg/store/data/%7E2ehgtags.i /remote/.hg/store/data/bar.i /remote/.hg/store/data/quux.i + /remote/.hg/store/data/~2edotfile%20with%20spaces.i + /remote/.hg/store/data/~2ehgtags.i /remotempty/.hg/bookmarks /remotempty/.hg/bookmarks.current /remotempty/.hg/requires @@ -275,5 +275,5 @@ /sub/.hg/requires /sub/.hg/store/00changelog.i /sub/.hg/store/00manifest.i - /sub/.hg/store/data/%7E2ehgtags.i /sub/.hg/store/data/test.i + /sub/.hg/store/data/~2ehgtags.i ERROR: test-static-http.t output changed ! Since buildbot has 3.6, I have queued this. REPOSITORY rHG Mercurial REVISION DETAIL https://phab.mercurial-scm.org/D5730 To: durin42, pulkit, #hg-reviewers Cc: mercurial-devel ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
