Re: [PATCH 01 of 10] tests: use grep -E instead of obsolescent egrep

[Joerg Sonnenberger]

On Wed, Jun 28, 2023 at 02:06:46AM +0200, Mads Kiilerich wrote:
> # HG changeset patch
> # User Mads Kiilerich 
> # Date 1687782898 -7200
> #  Mon Jun 26 14:34:58 2023 +0200
> # Branch stable
> # Node ID 2790b07cd5da2be40fd5299ae114a49eb6196e55
> # Parent  42f761e97decbd95b5c3171ba83701828d60db24
> tests: use grep -E instead of obsolescent egrep
> 
> Testing on Fedora 38 failed with:
>   egrep: warning: egrep is obsolescent; using grep -E
> 
> The warning comes from
> https://git.savannah.gnu.org/cgit/grep.git/commit/?id=a9515624709865d480e3142fd959bccd1c9372d1
> . For further anecdotal evidence of the change, see
> https://www.phoronix.com/news/GNU-Grep-3.8-Stop-egrep-fgrep .

I'm mildly objecting to this change and would call on everyone to fill a
bug with GNU and any Linux distribution that keeps this idiocity...

Joerg
___
Mercurial-devel mailing list
Mercurial-devel@lists.mercurial-scm.org
https://lists.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] histedit: fix diff colors

[Joerg Sonnenberger]

Am Mon, Oct 24, 2022 at 05:05:18PM -0400 schrieb Jordi Gutiérrez Hermoso:
> I don't know what was up here, but seems like checking for the first
> byte isn't the same thing as "startswith", so the colours in chistedit
> diffs were broken.

Can you also please build a test case for this?

Joerg
___
Mercurial-devel mailing list
Mercurial-devel@lists.mercurial-scm.org
https://lists.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] histedit: fix diff colors

[Joerg Sonnenberger]

On Mon, Oct 24, 2022 at 05:05:18PM -0400, Jordi Gutiérrez Hermoso wrote:
> diff --git a/hgext/histedit.py b/hgext/histedit.py
> --- a/hgext/histedit.py
> +++ b/hgext/histedit.py
> @@ -1428,11 +1428,11 @@ pgup/K: move patch up, pgdn/J: move patc
>  for y in range(0, length):
>  line = output[y]
>  if diffcolors:
> -if line and line[0] == b'+':
> +if line and line.startswith(b'+'):
>  win.addstr(
>  y, 0, line, curses.color_pair(COLOR_DIFF_ADD_LINE)
>  )
> -elif line and line[0] == b'-':
> +elif line and line.startswith(b'-'):
>  win.addstr(
>  y, 0, line, curses.color_pair(COLOR_DIFF_DEL_LINE)
>  )

Do we still need the "line and" part of the check?

Joerg
___
Mercurial-devel mailing list
Mercurial-devel@lists.mercurial-scm.org
https://lists.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH stable] hghave: make black version regex work with newer versions of black

[Joerg Sonnenberger]

Am Sun, May 22, 2022 at 02:13:03AM +0200 schrieb Manuel Jacob:
> # HG changeset patch
> # User Manuel Jacob 
> # Date 1653176900 -7200
> #  Sun May 22 01:48:20 2022 +0200
> # Branch stable
> # Node ID 29f2716c5c54c7e0f7aa6d91979893f5d2078862
> # Parent  477b5145e1a02715f846ce017b460858a58e03b1
> # EXP-Topic black_version_regex
> hghave: make black version regex work with newer versions of black
> 
> Black commit 117891878e5be4d6b771ae5de299e51b679cea27 (included in black >=
> 21.11b0) dropped the string "version " from the output of "black --version". 
> To
> make the regex work with newer black versions, make matching of "version "
> optional.

I had a patch like this locally, but newer black versions insist on
incompatible output and that's where I stopped. There is also the issue
that the regex itself seems wrong, e.g. the unescaped "." in the [].

Joerg
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D12583: iblt: prototype for setdiscovery

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D12583

AFFECTED FILES
  mercurial/bundlerepo.py
  mercurial/debugcommands.py
  mercurial/exchange.py
  mercurial/iblt.py
  mercurial/interfaces/repository.py
  mercurial/localrepo.py
  mercurial/setdiscovery.py
  mercurial/wireprotov1peer.py
  mercurial/wireprotov1server.py

CHANGE DETAILS

diff --git a/mercurial/wireprotov1server.py b/mercurial/wireprotov1server.py
--- a/mercurial/wireprotov1server.py
+++ b/mercurial/wireprotov1server.py
@@ -275,6 +275,7 @@
 b'known',
 b'getbundle',
 b'unbundlehash',
+b'iblt-changelog',
 ]
 
 
@@ -426,6 +427,15 @@
 continue
 return None
 
+@wireprotocommand(b'getestimator', b'name', permission=b'pull')
+def getestimator(repo, proto, name):
+estimator = repo.peer().getestimator(name)
+return wireprototypes.bytesresponse(estimator.dump())
+
+@wireprotocommand(b'getiblt', b'name size seed', permission=b'pull')
+def getiblt(repo, proto, name, size, seed):
+inst = repo.peer().getiblt(name, int(size), int(seed))
+return wireprototypes.bytesresponse(inst.dump())
 
 @wireprotocommand(b'getbundle', b'*', permission=b'pull')
 def getbundle(repo, proto, others):
diff --git a/mercurial/wireprotov1peer.py b/mercurial/wireprotov1peer.py
--- a/mercurial/wireprotov1peer.py
+++ b/mercurial/wireprotov1peer.py
@@ -21,6 +21,7 @@
 changegroup as changegroupmod,
 encoding,
 error,
+iblt,
 pushkey as pushkeymod,
 pycompat,
 util,
@@ -503,6 +504,14 @@
 ret = bundle2.getunbundler(self.ui, stream)
 return ret
 
+def getestimator(self, name):
+d = self._call(b"getestimator", name=name)
+return iblt.estimator.load(d)
+
+def getiblt(self, name, size, seed):
+d = self._call(b"getiblt", name=name, size=b'%d' % size, seed=b'%d' % 
seed)
+return iblt.iblt.load(d)[0]
+
 # End of ipeercommands interface.
 
 # Begin of ipeerlegacycommands interface.
diff --git a/mercurial/setdiscovery.py b/mercurial/setdiscovery.py
--- a/mercurial/setdiscovery.py
+++ b/mercurial/setdiscovery.py
@@ -279,6 +279,70 @@
 'discovery', member='PartialDiscovery', default=partialdiscovery
 )
 
+import math
+iblt_sizes = [(1 << i) for i in range(5, 31)]
+iblt_sizes += [math.trunc(math.sqrt(2) * s) for s in iblt_sizes]
+iblt_sizes.sort()
+
+def round_iblt_size(size):
+size = size + size // 4
+for s in iblt_sizes:
+if s >= size:
+return s
+
+def findsetdifferences(ui, local, remote):
+if not remote.capable(b'iblt-changelog'):
+ui.status(b'no iblt support: %s\n' % b' 
'.join(list(remote.capabilities(
+return False, [], [], [], []
+myestimator = local.peer().getestimator(b'changelog')
+theirestimator = remote.getestimator(b'changelog')
+estimated_diff = myestimator.compare(theirestimator)
+# bail out if estimated_diff = O(len(repo)) and fallback to the classic 
mechanism?
+iblt_size = round_iblt_size(estimated_diff)
+ui.debug(b"expected difference is: %d, using IBLT size of %d\n" % 
(estimated_diff, iblt_size))
+
+attempt = 0
+while True:
+myiblt = local.peer().getiblt(b'changelog', iblt_size, 0)
+theiriblt = remote.getiblt(b'changelog', iblt_size, 0)
+theiriblt.subtract(myiblt)
+success, them_only, my_only = theiriblt.list()
+if not success:
+attempt += 1
+if attempt == 3:
+ui.debug(b'iblt extraction failed\n')
+return False, [], [], [], []
+iblt_size = round_iblt_size(iblt_size + 1)
+ui.debug(b'iblt extraction failed, retrying with size %d' % 
iblt_size)
+continue
+
+ui.status(b'iblt extraction worked, %d local changes and %d remote 
changes found\n' % (len(my_only), len(them_only)))
+break
+
+has_node = local.changelog.index.has_node
+nodelen = len(local.nullid)
+my_only = [node[:nodelen] for node in my_only]
+
+# first: find all parents and nodes
+parents = set()
+nodes = set()
+for row in them_only:
+node = row[:nodelen]
+if has_node(node):
+raise error.Abort(_(b"found already known remote change: %s") % 
node)
+nodes.add(node)
+parents.add(row[nodelen:2*nodelen])
+parents.add(row[2*nodelen:])
+# second: remote heads are all nodes that are not also parents
+remoteheads = nodes - parents
+# third: parent nodes that are not nodes themselve are the boundary
+# of the common set. Double check that they are known locally.
+commonheadscandidates = parents - nodes
+commonheads = [node for node in commonheadscandidates if has_node(node)]
+if len(commonheads) != len(commonheadscandidates):
+raise error.Abort(_(b"found remote changes 

D12391: pullbundle: fix file name in the help text

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  It is pullbundles.manifest and not pullbundle.manifest.

REPOSITORY
  rHG Mercurial

BRANCH
  stable

REVISION DETAIL
  https://phab.mercurial-scm.org/D12391

AFFECTED FILES
  i18n/ja.po
  i18n/pt_BR.po
  mercurial/helptext/config.txt
  tests/test-pull-bundle.t

CHANGE DETAILS

diff --git a/tests/test-pull-bundle.t b/tests/test-pull-bundle.t
--- a/tests/test-pull-bundle.t
+++ b/tests/test-pull-bundle.t
@@ -194,7 +194,7 @@
   * sending pullbundle "0.hg" (glob)
   $ rm repo/.hg/blackbox.log
 
-Test processing when nodes used in the pullbundle.manifest end up being hidden
+Test processing when nodes used in the pullbundles.manifest end up being hidden
 
   $ hg --repo repo debugobsolete ed1b79f46b9a29f5a6efa59cf12fcfca43bead5a
   1 new obsolescence markers
diff --git a/mercurial/helptext/config.txt b/mercurial/helptext/config.txt
--- a/mercurial/helptext/config.txt
+++ b/mercurial/helptext/config.txt
@@ -2293,7 +2293,7 @@
 effectively prevents concurrent pushes.
 
 ``pullbundle``
-When set, the server will check pullbundle.manifest for bundles
+When set, the server will check pullbundles.manifest for bundles
 covering the requested heads and common nodes. The first matching
 entry will be streamed to the client.
 
diff --git a/i18n/pt_BR.po b/i18n/pt_BR.po
--- a/i18n/pt_BR.po
+++ b/i18n/pt_BR.po
@@ -27460,12 +27460,12 @@
 
 msgid ""
 "``pullbundle``\n"
-"When set, the server will check pullbundle.manifest for bundles\n"
+"When set, the server will check pullbundles.manifest for bundles\n"
 "covering the requested heads and common nodes. The first matching\n"
 "entry will be streamed to the client."
 msgstr ""
 "``pullbundle``\n"
-"Se definido, o servidor verificará pullbundle.manifest para\n"
+"Se definido, o servidor verificará pullbundles.manifest para\n"
 "arquivos de bundle que contiverem as cabeças e nós comuns\n"
 "pedidos. A primeira entrada correspondente será enviadas para\n"
 "o cliente."
diff --git a/i18n/ja.po b/i18n/ja.po
--- a/i18n/ja.po
+++ b/i18n/ja.po
@@ -26642,7 +26642,7 @@
 
 msgid ""
 "``pullbundle``\n"
-"When set, the server will check pullbundle.manifest for bundles\n"
+"When set, the server will check pullbundles.manifest for bundles\n"
 "covering the requested heads and common nodes. The first matching\n"
 "entry will be streamed to the client."
 msgstr ""



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: pycompat.py strategy

[Joerg Sonnenberger]

Am Tue, Mar 01, 2022 at 11:20:41AM +0100 schrieb Raphaël Gomès:
> 
> On 2/22/22 23:06, Joerg Sonnenberger wrote:
> > Am Mon, Feb 21, 2022 at 11:47:36AM -0700 schrieb Gregory Szorc:
> > > Some options:
> > > 
> > > a) Attempt to delete as much as pycompat.py as possible (leaving only the
> > > pieces needed to abstract over differences in Python 3.5-3.x).
> > > b) Leave the ~complete API provided by pycompat.py and delete pycompat
> > > usage within the repo as much as possible.
> > > c) Leave the ~complete API provided by pycompat.py and still use pycompat
> > > heavily within the repo.
> > Personally, I'd prefer to start with (c) and gradually go to (b).
> Care to elaborate? :)

We can start just nuking the non-py3 case in pycompat and then slowly
inline it in the rest of the tree. Ideally using a tool for the
mechnical part of it.

Joerg
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

IBLT-base discovery

[Joerg Sonnenberger]

Hello all,
TIL: Prototype covering incoming/pull can be found at the end after the
hyphen line.

This mail is about a prototype for a discovery mechanism based on
Invertible Bloom Lookup Tables (IBLT). An IBLT is a generalized form of
a Bloom Filter and can be used to efficiently compute set differences.
The only requirement is that the set items are canonical and have the
same size, e.g. a cryptographic hash over a record can be used as
identifier. It is efficient in the follow senses:

(1) The data to be exchanged between two parties scales linear with the
size of the different (i.e. O(|A\B + B\A|) given sets A and B).

(2) The number of round trips is expected to be O(1), normally one for
the estimation step and a second for the actual exchange. Depending on
the details, both steps can be part of one round trip in opposite
directions.

For the changeset discovery, this approach is interesting because it
works independently of the structure of the commit graph as long as they
are similar enough.  This is different from the currently used sampling
protocol that scales based on the total and disjoint number of DAG
heads. This property also makes it attractive for exchanging additional
data attached to commits like changeset signatures or provenance records.

The expected O(1) derives from the probalistic nature of the algorithm,
e.g. it derives from properties of random 3-graphs. The principles are
also found in modern construction mechanisms for perfect hash tables or
error-correcting codes. The success rate increases with size and
practical simulations of the hypergraphs show that it works almost
always even for moderate sizes in thousand entries range.

Based on the natural and justified choices of (1) deterministically
keyed hash functions, (2) fixed size categories with exponential growth
and (3) retries with the next larger size as normal handling for failure
cases, the data structure can be pre-computed and easily updated
incrementally. Without trying for optimal encodings, space requirement
would be O(d * log d) with d being the maximum allowed difference size
and update cost of O(log d) when adding a new key.

---

A working prototype as alternative to the current set discovery algorithm
can be found in https://www.netbsd.org/~joerg/iblt-exchange.diff
There are a couple of hacks and short cuts in the code, but it works
well enough to illustrate the point. It is written to be minimally
intrusive. For example, there is no cache (in)validation or incremental
update implemented.

From a protocol perspective, it is also much more inefficient than
necessary. It reuses the current getbundle wire command. For this, it
needs to compute (or at least approximate a superset of) the common heads
and the remote sets. This is done by using both the node id of the commit
as well as the parent nodes as key, so it tripples the key size and
therefore the exchanged data. A better but more intrusive approach
replaces the parameters of getbundle directly with the IBLT.

Test case is my local pkgsrc repository with ~776k commits and a clone
without my local changesets. Incoming shows 71 changes. Test uses ssh
over localhost. All values are best of three runs.

  New protocol Old protocol
Run time 2.40s2.77s
Data to server2784   24,338 
Data from server19,218,995   19,181,398

For the no-change case of incoming against the same repository:

  New protocol Old protocol
Run time 0.62s0.62s
Data to server 5449,965
Data from server38,796   11,028

For reference, the estimator step uses 35,308 Bytes and results in an
IBLT of 181 entries for the first case and 32 entries for the second
case with a size of 11,958 Bytes and 2124 Bytes respectively.

Joerg
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: pycompat.py strategy

[Joerg Sonnenberger]

Am Mon, Feb 21, 2022 at 11:47:36AM -0700 schrieb Gregory Szorc:
> Some options:
> 
> a) Attempt to delete as much as pycompat.py as possible (leaving only the
> pieces needed to abstract over differences in Python 3.5-3.x).
> b) Leave the ~complete API provided by pycompat.py and delete pycompat
> usage within the repo as much as possible.
> c) Leave the ~complete API provided by pycompat.py and still use pycompat
> heavily within the repo.

Personally, I'd prefer to start with (c) and gradually go to (b).

Joerg
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D12207: zstd: hack include order to ensure that our zstd.h is found

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  If the regular Python CFLAGS include directories that already have the
  zstd headers available, a different and possible incompatible version
  can be picked up otherwise. Sadly, it seems like Python has no easy way
  to prefix flags before the rest.

REPOSITORY
  rHG Mercurial

BRANCH
  stable

REVISION DETAIL
  https://phab.mercurial-scm.org/D12207

AFFECTED FILES
  contrib/python-zstandard/setup_zstd.py

CHANGE DETAILS

diff --git a/contrib/python-zstandard/setup_zstd.py 
b/contrib/python-zstandard/setup_zstd.py
--- a/contrib/python-zstandard/setup_zstd.py
+++ b/contrib/python-zstandard/setup_zstd.py
@@ -145,9 +145,17 @@
 
 include_dirs = set([os.path.join(actual_root, d) for d in ext_includes])
 if not system_zstd:
-include_dirs.update(
-[os.path.join(actual_root, d) for d in zstd_includes]
-)
+from distutils import sysconfig
+try:
+from shlex import quote
+except ImportError:
+from pipes import quote
+includes = []
+for incdir in [os.path.join(actual_root, d) for d in zstd_includes]:
+   includes.append('-I' + quote(incdir))
+   include_dirs.add(incdir)
+config_vars = sysconfig.get_config_vars()
+config_vars['CFLAGS'] = ' '.join(includes + [config_vars.get('CFLAGS', 
'')])
 if support_legacy:
 include_dirs.update(
 [os.path.join(actual_root, d) for d in zstd_includes_legacy]



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D11956: exchange: add fast path for subrepo check on push

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  Try to check if .hgsub and .hgsubstate exist at all before looking
  for them in every changeset to be pushed. The latter can be quite
  expensive for large repositories and the existance check is almost free.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D11956

AFFECTED FILES
  mercurial/exchange.py

CHANGE DETAILS

diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -522,8 +522,21 @@
 
 def _checksubrepostate(pushop):
 """Ensure all outgoing referenced subrepo revisions are present locally"""
+
+repo = pushop.repo
+
+# If the repository does not use subrepos, skip the expensive
+# manifest checks.
+try:
+if not len(repo.file(b'.hgsub')) or not len(repo.file(b'.hgsubstate')):
+return
+except RuntimeError:
+# Alternative filelog implementations might not implement this,
+# so just fallback to the generic implementation.
+pass
+
 for n in pushop.outgoing.missing:
-ctx = pushop.repo[n]
+ctx = repo[n]
 
 if b'.hgsub' in ctx.manifest() and b'.hgsubstate' in ctx.files():
 for subpath in sorted(ctx.substate):



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: breach by trojan source code (python)

[Joerg Sonnenberger]

On Tue, Nov 02, 2021 at 02:51:01PM +0100, Uwe Brauer wrote:
> 
> Hi 
> 
> https://www.trojansource.codes/trojan-source.pdf
> 
> I skimmed the article, the authors claim that python is effect.
> 
> Any thoughts?

IMHO this is blown quite out of proportion and reasonably addressed by
filters in black or other code formatting tools.

Joerg
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 1 of 2] fixed compare filelog rev with file content, besides meta-info content

[Joerg Sonnenberger]

> * filelog.size() now returns rev textbody len, without meta-info

So, have you made sure this doesn't create a huge performance
regression? The whole point of the original logic is to try avoid a full
revision resolution as often as possible.

Joerg
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D11453: log: if there is a merge state, show conflictother for . [WIP]

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D11453

AFFECTED FILES
  i18n/de.po
  i18n/ja.po
  i18n/pt_BR.po
  mercurial/logcmdutil.py
  mercurial/templatekw.py
  tests/test-backout.t
  tests/test-merge7.t
  tests/test-parents.t
  tests/test-shelve.t

CHANGE DETAILS

diff --git a/tests/test-shelve.t b/tests/test-shelve.t
--- a/tests/test-shelve.t
+++ b/tests/test-shelve.t
@@ -1425,7 +1425,8 @@
   |  summary: changes to: add A to bars
   |
   | @  changeset:   4:fe451a778c81
-  |/   user:test
+  |/   merging: 5:f1d5f53e397b
+  |user:test
   |date:Thu Jan 01 00:00:00 1970 +
   |summary: add C to bars
   |
diff --git a/tests/test-parents.t b/tests/test-parents.t
--- a/tests/test-parents.t
+++ b/tests/test-parents.t
@@ -106,6 +106,7 @@
   (branch merge, don't forget to commit)
   $ hg parents c
   changeset:   3:02d851b7e549
+  merging: 4:48cee28d4b4e
   user:test
   date:Thu Jan 01 00:00:03 1970 +
   summary: c
diff --git a/tests/test-merge7.t b/tests/test-merge7.t
--- a/tests/test-merge7.t
+++ b/tests/test-merge7.t
@@ -126,6 +126,7 @@
   changeset:   3:50c3a7e29886
   parent:  1:d1e159716d41
   parent:  2:96b70246a118
+  merging: 4:40d11a4173a8
   user:test
   date:Thu Jan 01 00:00:00 1970 +
   summary: Merge 1
diff --git a/tests/test-backout.t b/tests/test-backout.t
--- a/tests/test-backout.t
+++ b/tests/test-backout.t
@@ -89,6 +89,7 @@
   $ hg log -G
   @  changeset:   4:ed7b793d
   |  tag: tip
+  |  merging: 1:22cb4f70d813
   |  user:test
   |  date:Thu Jan 01 00:00:05 1970 +
   |  summary: ypples
@@ -741,6 +742,7 @@
   $ hg log -G
   @  changeset:   2:b71750c4b0fd
   |  tag: tip
+  |  merging: 0:a30dd8addae3
   |  user:test
   |  date:Thu Jan 01 00:00:00 1970 +
   |  summary: capital ten
diff --git a/mercurial/templatekw.py b/mercurial/templatekw.py
--- a/mercurial/templatekw.py
+++ b/mercurial/templatekw.py
@@ -125,6 +125,7 @@
 b'files:   %s\n'
 b'instability: %s\n'
 b'manifest:%s\n'
+b'merging: %s\n'
 b'obsolete:%s\n'
 b'parent:  %s\n'
 b'phase:   %s\n'
diff --git a/mercurial/logcmdutil.py b/mercurial/logcmdutil.py
--- a/mercurial/logcmdutil.py
+++ b/mercurial/logcmdutil.py
@@ -279,6 +279,7 @@
 self.lastheader = None
 self.footer = None
 self._columns = templatekw.getlogcolumns()
+self._currentchangeset = repo[b'.']
 
 def flush(self, ctx):
 rev = ctx.rev()
@@ -348,6 +349,13 @@
 columns[b'parent'] % scmutil.formatchangeid(pctx), label=label
 )
 
+merging = self._getmerging(ctx)
+if merging:
+self.ui.write(
+columns[b'merging'] % scmutil.formatchangeid(merging),
+label='log.merging',
+)
+
 if self.ui.debugflag:
 mnode = ctx.manifestnode()
 if mnode is None:
@@ -425,6 +433,22 @@
 
 self._showpatch(ctx, graphwidth)
 
+def _getmerging(self, ctx):
+if ctx != self._currentchangeset:
+return None
+from . import mergestate as mergestatemod
+
+mergestate = mergestatemod.mergestate.read(self.repo)
+if not mergestate.active():
+return None
+try:
+other = mergestate.otherctx
+except KeyError:
+return None
+if other != ctx:
+return other
+return None
+
 def _showobsfate(self, ctx):
 # TODO: do not depend on templater
 tres = formatter.templateresources(self.repo.ui, self.repo)
@@ -520,6 +544,10 @@
 removed=fm.formatlist(files.removed, name=b'file'),
 )
 
+merging = self._getmerging(ctx)
+if merging:
+fm.data(merging=fm.hexfunc(merging.node()))
+
 verbose = not self.ui.debugflag and self.ui.verbose
 if verbose or b'files' in datahint:
 fm.data(files=fm.formatlist(ctx.files(), name=b'file'))
diff --git a/i18n/pt_BR.po b/i18n/pt_BR.po
--- a/i18n/pt_BR.po
+++ b/i18n/pt_BR.po
@@ -39515,6 +39515,7 @@
 "files:   %s\n"
 "instability: %s\n"
 "manifest:%s\n"
+"merging: %s\n"
 "obsolete:%s\n"
 "parent:  %s\n"
 "phase:   %s\n"
@@ -39533,6 +39534,7 @@
 "arquivos:  %s\n"
 "instabilidade: %s\n"
 "manifesto: %s\n"
+"merging:   %s\n"
 "obsolescência: %s\n"
 "pai:   %s\n"
 "fase:  %s\n"
diff --git a/i18n/ja.po b/i18n/ja.po
--- a/i18n/ja.po
+++ b/i18n/ja.po
@@ -38206,6 +38206,7 @@
 "files:   %s\n"
 "instability: %s\n"
 "manifest:%s\n"
+"merging: %s\n"
 "obsolete:%s\n"
 "parent:  %s\n"
 "phase:   %s\n"
@@ 

D7177: rebase: introduce optional parent mapping

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger added a comment.


  From the VC call today: The use case can be addressed by marking parents as 
to-be-preserved, so that for merge commits rebase can decide on the parent link 
is should operate on. It covers a general DAG and is somewhat easier to use in 
terms of UX than the parentmap, but can provide the same functionality with a 
multi-step rebase.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7177/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7177

To: joerg.sonnenberger, martinvonz, #hg-reviewers, baymax
Cc: mercurial-patches, marmoute, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D11252: debugupgraderepo: add fix-metaencoding-flag pass for issue6528

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.
marmoute added a comment.


  This is similar to D11239 , having 
this done during upgrade too seems useful, but having a simpler/more-focussed 
command for this seems important (and "on-fly-fix" during exchange too.
  
  Ideally we would take D11239  first 
and reuse its bulding block for that diff afterward.

REPOSITORY
  rHG Mercurial

BRANCH
  stable

REVISION DETAIL
  https://phab.mercurial-scm.org/D11252

AFFECTED FILES
  mercurial/revlog.py
  mercurial/upgrade_utils/actions.py
  mercurial/upgrade_utils/engine.py

CHANGE DETAILS

diff --git a/mercurial/upgrade_utils/engine.py 
b/mercurial/upgrade_utils/engine.py
--- a/mercurial/upgrade_utils/engine.py
+++ b/mercurial/upgrade_utils/engine.py
@@ -136,6 +136,10 @@
 ):
 """returns the new revlog object created"""
 newrl = None
+recheckmetaencoding = (
+upgrade_op.recheckmetaencoding
+and (rl_type & store.FILEFLAGS_FILELOG) != 0
+)
 if matchrevlog(upgrade_op.revlogs_to_process, rl_type):
 ui.note(
 _(b'cloning %d revisions from %s\n') % (len(old_revlog), unencoded)
@@ -148,6 +152,7 @@
 deltareuse=upgrade_op.delta_reuse_mode,
 forcedeltabothparents=upgrade_op.force_re_delta_both_parents,
 sidedatacompanion=sidedatacompanion,
+recheckmetaencoding=recheckmetaencoding,
 )
 else:
 msg = _(b'blindly copying %s containing %i revisions\n')
diff --git a/mercurial/upgrade_utils/actions.py 
b/mercurial/upgrade_utils/actions.py
--- a/mercurial/upgrade_utils/actions.py
+++ b/mercurial/upgrade_utils/actions.py
@@ -616,6 +616,22 @@
 )
 )
 
+register_optimization(
+improvement(
+name=b'fix-metaencoding-flag',
+type=OPTIMISATION,
+description=_(
+b'filelog entries with copy metadata may have been flagged '
+b'incorrectly in Mercurial 5.8. This option will look for such '
+b'revisions and fix them.'
+),
+upgrademessage=_(
+b'revision content will be recomputed; this will likely 
drastically '
+b'slow down execution time'
+),
+)
+)
+
 
 def findoptimizations(repo):
 """Determine optimisation that could be used during upgrade"""
@@ -710,6 +726,11 @@
 elif b're-delta-fulladd' in self._upgrade_actions_names:
 self.delta_reuse_mode = revlog.revlog.DELTAREUSEFULLADD
 
+# should this operation search for misordered parents of copy filelog 
entries
+self.recheckmetaencoding = (
+b'fix-metaencoding-flag' in self._upgrade_actions_names
+)
+
 # should this operation force re-delta of both parents
 self.force_re_delta_both_parents = (
 b're-delta-multibase' in self._upgrade_actions_names
diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -2779,6 +2779,7 @@
 deltareuse=DELTAREUSESAMEREVS,
 forcedeltabothparents=None,
 sidedatacompanion=None,
+recheckmetaencoding=False,
 ):
 """Copy this revlog to another, possibly with format changes.
 
@@ -2876,6 +2877,7 @@
 deltareuse,
 forcedeltabothparents,
 sidedatacompanion,
+recheckmetaencoding,
 )
 
 finally:
@@ -2891,10 +2893,18 @@
 deltareuse,
 forcedeltabothparents,
 sidedatacompanion,
+recheckmetaencoding,
 ):
 """perform the core duty of `revlog.clone` after parameter 
processing"""
+
+def ismetaencoded(text, flags):
+return text.startswith(b'\x01\n') and (
+flags & REVIDX_ISCENSORED == 0
+)
+
 deltacomputer = deltautil.deltacomputer(destrevlog)
 index = self.index
+
 for rev in self:
 entry = index[rev]
 
@@ -2914,7 +2924,11 @@
 # the revlog chunk is a delta.
 cachedelta = None
 rawtext = None
-if any(sidedataactions) or deltareuse == self.DELTAREUSEFULLADD:
+if (
+any(sidedataactions)
+or deltareuse == self.DELTAREUSEFULLADD
+or recheckmetaencoding
+):
 dropall = sidedataactions[0]
 filterout = sidedataactions[1]
 update = sidedataactions[2]
@@ -2928,6 +2942,13 @@
 sidedata.update(update)
 if not sidedata:
 sidedata = None
+if (
+recheckmetaencoding
+and ismetaencoded(text, flags)
+and p1 != self.nullid
+and p2 == self.nullid
+):
+p1, p2 = p2, p1
 

D11203: revlog: recommit 49fd21f32695 with a fix for issue6528

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: durin42.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  `filelog.size` currently special cases two forms of metadata encoding:
  
  - copy data via the parent order as flag bit
  - censor data by peaking into the raw delta
  
  All other forms of metadata encoding including the empty metadata block
  are mishandled. In `basefilectx.cmp` the empty metadata block is
  explicitly checked to compensate for this.
  
  Restore 49fd21f32695 
, 
but disable it for filelog, so that the original
  flag bit use contines to work. Document all this mess for now in
  preparation of a proper rework.

REPOSITORY
  rHG Mercurial

BRANCH
  stable

REVISION DETAIL
  https://phab.mercurial-scm.org/D11203

AFFECTED FILES
  mercurial/context.py
  mercurial/filelog.py
  mercurial/revlog.py
  tests/test-narrow-shallow-merges.t

CHANGE DETAILS

diff --git a/tests/test-narrow-shallow-merges.t 
b/tests/test-narrow-shallow-merges.t
--- a/tests/test-narrow-shallow-merges.t
+++ b/tests/test-narrow-shallow-merges.t
@@ -179,7 +179,7 @@
   
 
   $ hg log -T '{if(ellipsis,"...")}{node|short} {p1node|short} {p2node|short} 
{desc}\n' | sort
-  ...2a20009de83e  3ac1f5779de3 outside 10
+  ...2a20009de83e 3ac1f5779de3  outside 10
   ...3ac1f5779de3 bb96a08b062a 465567bdfb2d merge a/b/c/d 9
   ...8d874d57adea 7ef88b4dd4fa  outside 12
   ...b844052e7b3b   outside 2c
diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -434,6 +434,7 @@
 upperboundcomp=None,
 persistentnodemap=False,
 concurrencychecker=None,
+canonical_parent_order=True,
 ):
 """
 create a revlog object
@@ -491,6 +492,13 @@
 
 self._concurrencychecker = concurrencychecker
 
+# parent order is supposed to be semantically irrelevant, so we
+# normally resort parents to ensure that the first parent is non-null,
+# if there is a non-null parent at all.
+# filelog obuses the parent order as flag to mark some instances of
+# meta-encoded files, so allow it to disable this behavior.
+self.canonical_parent_order = canonical_parent_order
+
 def _loadindex(self):
 mmapindexthreshold = None
 opts = self.opener.options
@@ -885,7 +893,10 @@
 raise error.WdirUnsupported
 raise
 
-return entry[5], entry[6]
+if self.canonical_parent_order and entry[5] == nullrev:
+return entry[6], entry[5]
+else:
+return entry[5], entry[6]
 
 # fast parentrevs(rev) where rev isn't filtered
 _uncheckedparentrevs = parentrevs
@@ -907,6 +918,11 @@
 i = self.index
 d = i[self.rev(node)]
 return i[d[5]][7], i[d[6]][7]  # map revisions to nodes inline
+# inline node() to avoid function call overhead
+if self.canonical_parent_order and d[5] == nullid:
+return i[d[6]][7], i[d[5]][7]
+else:
+return i[d[5]][7], i[d[6]][7]
 
 def chainlen(self, rev):
 return self._chaininfo(rev)[0]
diff --git a/mercurial/filelog.py b/mercurial/filelog.py
--- a/mercurial/filelog.py
+++ b/mercurial/filelog.py
@@ -27,7 +27,10 @@
 class filelog(object):
 def __init__(self, opener, path):
 self._revlog = revlog.revlog(
-opener, b'/'.join((b'data', path + b'.i')), censorable=True
+opener,
+b'/'.join((b'data', path + b'.i')),
+censorable=True,
+canonical_parent_order=False,  # see comment in revlog.py
 )
 # Full name of the user visible file, relative to the repository root.
 # Used by LFS.
@@ -197,6 +200,7 @@
 return 0
 
 # XXX if self.read(node).startswith("\1\n"), this returns (size+4)
+# XXX See also basefilectx.cmp.
 return self._revlog.size(rev)
 
 def cmp(self, node, text):
diff --git a/mercurial/context.py b/mercurial/context.py
--- a/mercurial/context.py
+++ b/mercurial/context.py
@@ -988,6 +988,16 @@
 if self._repo._encodefilterpats:
 # can't rely on size() because wdir content may be decoded
 return self._filelog.cmp(self._filenode, fctx.data())
+# filelog.size() has two special cases:
+# - censored metadata
+# - copy/rename tracking
+# The first is detected by peaking into the delta,
+# the second is detected by abusing parent order
+# in the revlog index as flag bit. This leaves files using
+# the dummy encoding and non-standard meta attributes.
+# The following check is a special case for the empty
+# metadata block used if the raw file content 

Re: [PATCH STABLE] cext: fix memory leak in phases computation

[Joerg Sonnenberger]

On Mon, Jun 07, 2021 at 12:29:20AM +0200, Georges Racinet wrote:
> # HG changeset patch
> # User Georges Racinet 
> # Date 1622935470 -7200
> #  Sun Jun 06 01:24:30 2021 +0200
> # Branch stable
> # Node ID be560b55eb7cfe25c68fb6fab5417fab6688cf84
> # Parent  5ac0f2a8ba7205266a206ad8da89a79173e8efea
> # EXP-Topic memleak-phases
> cext: fix memory leak in phases computation

LGTM.

Joerg
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10726: recover: only apply last journal record per file

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This got broken in 2019 when the size check was introduced. It is most
  noticable when dealing with transactions that involve an inline to
  non-inline revlog storage transaction. It wasn't seen as much at the
  time because the in-memory journal actually de-duplicated the entry
  implicity, but since 63edc384d3b7 
 
the on-disk journal is used for
  rollback as well as recover.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10726

AFFECTED FILES
  mercurial/transaction.py
  tests/test-transaction-rollback-on-revlog-split.t

CHANGE DETAILS

diff --git a/tests/test-transaction-rollback-on-revlog-split.t 
b/tests/test-transaction-rollback-on-revlog-split.t
--- a/tests/test-transaction-rollback-on-revlog-split.t
+++ b/tests/test-transaction-rollback-on-revlog-split.t
@@ -4,9 +4,9 @@
 Helper extension to intercept renames.
 
   $ cat > $TESTTMP/intercept_rename.py << EOF
-  > from mercurial.extensions import wrapfunction
-  > from mercurial.util import atomictempfile
-  > import os, sys
+  > import os
+  > import sys
+  > from mercurial import extensions, util
   > 
   > def extsetup(ui):
   > def close(orig, *args, **kwargs):
@@ -14,7 +14,7 @@
   > if path.endswith(b'/.hg/store/data/file.i'):
   > os._exit(80)
   > return orig(*args, **kwargs)
-  > wrapfunction(atomictempfile, 'close', close)
+  > extensions.wrapfunction(util.atomictempfile, 'close', close)
   > EOF
 
 
@@ -62,6 +62,28 @@
   data/file.d 0
   data/file.d 1046
   data/file.i 128
+  $ hg recover
+  rolling back interrupted transaction
+  (verify step skipped, run `hg verify` to check your repository content)
+  $ f -s .hg/store/data/file*
+  .hg/store/data/file.d: size=1046
+  .hg/store/data/file.i: size=128
+  $ hg tip
+  changeset:   1:3ce491143aec
+  tag: tip
+  user:test
+  date:Thu Jan 01 00:00:00 1970 +
+  summary: _
+  
+  $ hg verify
+  checking changesets
+  checking manifests
+  crosschecking files in changesets and manifests
+  checking files
+   warning: revlog 'data/file.d' not in fncache!
+  checked 2 changesets with 2 changes to 1 files
+  1 warnings encountered!
+  hint: run "hg debugrebuildfncache" to recover from corrupt fncache
   $ cd ..
 
 Now retry the same but intercept the rename of the index and check that
@@ -83,4 +105,24 @@
   data/file.i 1174
   data/file.d 0
   data/file.d 1046
+
+  $ hg recover
+  rolling back interrupted transaction
+  (verify step skipped, run `hg verify` to check your repository content)
+  $ f -s .hg/store/data/file*
+  .hg/store/data/file.d: size=1046
+  .hg/store/data/file.i: size=1174
+  $ hg tip
+  changeset:   1:3ce491143aec
+  tag: tip
+  user:test
+  date:Thu Jan 01 00:00:00 1970 +
+  summary: _
+  
+  $ hg verify
+  checking changesets
+  checking manifests
+  crosschecking files in changesets and manifests
+  checking files
+  checked 2 changesets with 2 changes to 1 files
   $ cd ..
diff --git a/mercurial/transaction.py b/mercurial/transaction.py
--- a/mercurial/transaction.py
+++ b/mercurial/transaction.py
@@ -56,7 +56,7 @@
 unlink=True,
 checkambigfiles=None,
 ):
-for f, o in entries:
+for f, o in sorted(dict(entries).items()):
 if o or not unlink:
 checkambig = checkambigfiles and (f, b'') in checkambigfiles
 try:



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10725: revlog: update data file record before index rename

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: indygreg.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  Whe migrating from inline to non-inline data storage, the data file is
  recorded initially as zero sized so that it is removed on failure. But
  the record has to be updated before the index is renamed since otherwise
  data is lost on rollback.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10725

AFFECTED FILES
  mercurial/revlog.py
  tests/test-transaction-rollback-on-revlog-split.t

CHANGE DETAILS

diff --git a/tests/test-transaction-rollback-on-revlog-split.t 
b/tests/test-transaction-rollback-on-revlog-split.t
--- a/tests/test-transaction-rollback-on-revlog-split.t
+++ b/tests/test-transaction-rollback-on-revlog-split.t
@@ -1,7 +1,27 @@
 Test correctness of revlog inline -> non-inline transition
 --
 
+Helper extension to intercept renames.
+
+  $ cat > $TESTTMP/intercept_rename.py << EOF
+  > from mercurial.extensions import wrapfunction
+  > from mercurial.util import atomictempfile
+  > import os, sys
+  > 
+  > def extsetup(ui):
+  > def close(orig, *args, **kwargs):
+  > path = args[0]._atomictempfile__name
+  > if path.endswith(b'/.hg/store/data/file.i'):
+  > os._exit(80)
+  > return orig(*args, **kwargs)
+  > wrapfunction(atomictempfile, 'close', close)
+  > EOF
+
+
 Test offset computation to correctly factor in the index entries themselve.
+Also test that the new data size has the correct size if the transaction is 
aborted
+after the index has been replaced.
+
 Test repo has one small, one moderate and one big change. The clone has
 the small and moderate change and will transition to non-inline storage when
 adding the big change.
@@ -18,6 +38,12 @@
 
   $ hg clone -r 1 troffset-computation troffset-computation-copy --config 
format.revlog-compression=none -q
   $ cd troffset-computation-copy
+
+Reference size:
+
+  $ f -s .hg/store/data/file*
+  .hg/store/data/file.i: size=1174
+
   $ cat > .hg/hgrc < [hooks]
   > pretxnchangegroup = python:$TESTDIR/helper-killhook.py:killme
@@ -25,5 +51,36 @@
   $ hg pull ../troffset-computation
   pulling from ../troffset-computation
   [80]
-  $ cat .hg/store/journal | tr -s '\000' ' ' | grep data/file | tail -1
+
+The first file.i entry should match the size above.
+The first file.d entry is the temporary record during the split,
+the second entry after the split happened. The sum of the second file.d
+and the second file.i entry should match the first file.i entry.
+
+  $ cat .hg/store/journal | tr -s '\000' ' ' | grep data/file
+  data/file.i 1174
+  data/file.d 0
+  data/file.d 1046
   data/file.i 128
+  $ cd ..
+
+Now retry the same but intercept the rename of the index and check that
+the journal does not contain the new index size. This demonstrates the edge 
case
+where the data file is left as garbage.
+
+  $ hg clone -r 1 troffset-computation troffset-computation-copy2 --config 
format.revlog-compression=none -q
+  $ cd troffset-computation-copy2
+  $ cat > .hg/hgrc < [extensions]
+  > intercept_rename = $TESTTMP/intercept_rename.py
+  > [hooks]
+  > pretxnchangegroup = python:$TESTDIR/helper-killhook.py:killme
+  > EOF
+  $ hg pull ../troffset-computation
+  pulling from ../troffset-computation
+  [80]
+  $ cat .hg/store/journal | tr -s '\000' ' ' | grep data/file
+  data/file.i 1174
+  data/file.d 0
+  data/file.d 1046
+  $ cd ..
diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -1999,6 +1999,12 @@
 e = header + e
 fp.write(e)
 
+# There is a small transactional race here. If the rename of
+# the index fails, we should remove the datafile. It is more
+# important to ensure that the data file is not truncated
+# when the index is replaced as otherwise data is lost.
+tr.replace(self._datafile, self.start(trindex))
+
 # the temp file replace the real index when we exit the context
 # manager
 



To: joerg.sonnenberger, indygreg, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10724: revlog: fix index computation during inline->non-inline transition

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: indygreg.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  The computation 63edc384d3b7 
 
failed to factor in the index entries
  themselve as revlog.start() doesn't count them. Fonud by Valtenin
  Gatienbaron with a precise test case from me.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10724

AFFECTED FILES
  mercurial/revlog.py
  tests/helper-killhook.py
  tests/test-transaction-rollback-on-revlog-split.t

CHANGE DETAILS

diff --git a/tests/test-transaction-rollback-on-revlog-split.t 
b/tests/test-transaction-rollback-on-revlog-split.t
new file mode 100644
--- /dev/null
+++ b/tests/test-transaction-rollback-on-revlog-split.t
@@ -0,0 +1,29 @@
+Test correctness of revlog inline -> non-inline transition
+--
+
+Test offset computation to correctly factor in the index entries themselve.
+Test repo has one small, one moderate and one big change. The clone has
+the small and moderate change and will transition to non-inline storage when
+adding the big change.
+
+  $ hg init troffset-computation --config format.revlog-compression=none
+  $ cd troffset-computation
+  $ printf '% 20d' '1' > file
+  $ hg commit -Aqm_
+  $ printf '% 1024d' '1' > file
+  $ hg commit -Aqm_
+  $ dd if=/dev/zero of=file bs=1k count=128 > /dev/null 2>&1
+  $ hg commit -Aqm_
+  $ cd ..
+
+  $ hg clone -r 1 troffset-computation troffset-computation-copy --config 
format.revlog-compression=none -q
+  $ cd troffset-computation-copy
+  $ cat > .hg/hgrc < [hooks]
+  > pretxnchangegroup = python:$TESTDIR/helper-killhook.py:killme
+  > EOF
+  $ hg pull ../troffset-computation
+  pulling from ../troffset-computation
+  [80]
+  $ cat .hg/store/journal | tr -s '\000' ' ' | grep data/file | tail -1
+  data/file.i 128
diff --git a/tests/helper-killhook.py b/tests/helper-killhook.py
new file mode 100644
--- /dev/null
+++ b/tests/helper-killhook.py
@@ -0,0 +1,4 @@
+import os
+
+def killme(ui, repo, hooktype, **wkargs):
+os._exit(80)
diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -1985,7 +1985,7 @@
 with self._indexfp(b'r') as ifh, self._datafp(b'w') as dfh:
 for r in self:
 dfh.write(self._getsegmentforrevs(r, r, df=ifh)[1])
-if troffset <= self.start(r):
+if troffset <= self.start(r) + r * self.index.entry_size:
 trindex = r
 
 with self._indexfp(b'w') as fp:



To: joerg.sonnenberger, indygreg, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Manifest handling

[Joerg Sonnenberger]

Hello all,
this is a bit of a brain dump based on a recent discussion with
Pierre-Yves on IRC. I think it deserves some input from others and maybe
it can grow into a proper plan in time.

For the context of this discussion, see
  https://phab.mercurial-scm.org/D10082
and 
  https://bz.mercurial-scm.org/show_bug.cgi?id=6219
In short, the changelog entry itself is supposed to keep a list of files
touched by the change, but there are two major shortcomings:
(1) It doesn't show removals.
(2) Older versions notoriously mishandled merges.

This is painful not just for the review mentioned above, but various
other places that want to efficiently track what files could have been
affected by a changeset. This is made even worse by the inability to do
get fast manifest deltas. The current manifest code again has two major
shortcomings:
(1) It only works against the delta parent and that isn't necessarily
one of the changeset parents.
(2) It doesn't show removals.

Now I have been discussing "Bonsai" changesets as option for a while,
but that would certainly introduce a major breakage and is therefore
something that must be opt-in. But the idea by itself has enough merits
and brings the question on whether we can't compute the equivalent as
(non-optional) cache. What do we want in such a cache? We want to allow
walking from parent to child or the other way around and efficiently
compute the new file nodes. For merge commits, it should be possible to
traverse to either parent efficently. The first idea would be to store
lists of (path, old node, new node) for non-merges and (path, p1 node,
p2 node, new node) for merge commits. Added/removed files are using a
special marker node like  like in other places. This would be
require variable size entries (more parsing) and quite a bit of text
storage for big/deep repos. That's undesirable, so the first improvement
would be to have a side table of the path names similar to how we handle
it in the rev-branch-cache. The second option would be to store file
revisions and not nodes, but that has some concerns about strip handling.
It would reduce the size of the entries by factor of 4 to 7 though. Using
such a data structure for traversing is easy, just figure out which
parent is the desired parent and pick the appropiate old/new entries.

Computing this structure is bit tricky as we do not want to ever compute
the full text of parent or child. So we need to process the diffs
directly. For that, we need to know the starting position of each file.
The end of each line we compute easily based on the node len and the
path name. We can exploit the special nature of the manifest: all lines
are sorted and path is unique. Based on that we can build an augmented
binary tree that keeps track of the size of its subtrees. The tree can
be shared with the delta base and updated bottom-up, e.g. as
red-black-tree that would require O(d log n) new nodes to be written
where d is the number of lines touched and n the total number of lines
in the manifest. For merges or if the delta is not against the parent of
a changeset, a bit more work is necessary to compute the precise diff.
This can be done either by walking the path between the nodes or by
recursively comparing the trees. I would expect that in most cases the
recursive compare works quite well as most tree nodes should end up
being shared for the typical short term forks and the like.

Thoughts?

Joerg
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10536: core: don't hard-code node length

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: indygreg.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10536

AFFECTED FILES
  mercurial/localrepo.py
  mercurial/revlog.py

CHANGE DETAILS

diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -1433,7 +1433,7 @@
 if isinstance(id, int):
 # rev
 return self.node(id)
-if len(id) == 20:
+if len(id) == self.nodeconstants.nodelen:
 # possibly a binary node
 # odds of a binary node being all hex in ASCII are 1 in 10**25
 try:
diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -1840,7 +1840,7 @@
 # when we know that '.' won't be hidden
 node = self.dirstate.p1()
 rev = self.unfiltered().changelog.rev(node)
-elif len(changeid) == 20:
+elif len(changeid) == self.nodeconstants.nodelen:
 try:
 node = changeid
 rev = self.changelog.rev(changeid)



To: joerg.sonnenberger, indygreg, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10535: core: don't hard-code hex node lengths

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: indygreg.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10535

AFFECTED FILES
  mercurial/localrepo.py
  mercurial/revlog.py
  mercurial/revset.py
  mercurial/scmutil.py
  mercurial/templatefuncs.py

CHANGE DETAILS

diff --git a/mercurial/templatefuncs.py b/mercurial/templatefuncs.py
--- a/mercurial/templatefuncs.py
+++ b/mercurial/templatefuncs.py
@@ -764,9 +764,10 @@
 )
 
 repo = context.resource(mapping, b'repo')
-if len(hexnode) > 40:
+hexnodelen = 2 * repo.nodeconstants.nodelen
+if len(hexnode) > hexnodelen:
 return hexnode
-elif len(hexnode) == 40:
+elif len(hexnode) == hexnodelen:
 try:
 node = bin(hexnode)
 except TypeError:
diff --git a/mercurial/scmutil.py b/mercurial/scmutil.py
--- a/mercurial/scmutil.py
+++ b/mercurial/scmutil.py
@@ -643,7 +643,7 @@
 except (ValueError, OverflowError, IndexError):
 pass
 
-if len(symbol) == 40:
+if len(symbol) == 2 * repo.nodeconstants.nodelen:
 try:
 node = bin(symbol)
 rev = repo.changelog.rev(node)
diff --git a/mercurial/revset.py b/mercurial/revset.py
--- a/mercurial/revset.py
+++ b/mercurial/revset.py
@@ -1724,7 +1724,7 @@
 def _node(repo, n):
 """process a node input"""
 rn = None
-if len(n) == 40:
+if len(n) == 2 * repo.nodeconstants.nodelen:
 try:
 rn = repo.changelog.rev(bin(n))
 except error.WdirUnsupported:
diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -1454,7 +1454,7 @@
 return self.node(rev)
 except (ValueError, OverflowError):
 pass
-if len(id) == 40:
+if len(id) == 2 * self.nodeconstants.nodelen:
 try:
 # a full hex nodeid?
 node = bin(id)
diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -1861,7 +1861,7 @@
 changeid = hex(changeid)  # for the error message
 raise
 
-elif len(changeid) == 40:
+elif len(changeid) == 2 * self.nodeconstants.nodelen:
 node = bin(changeid)
 rev = self.changelog.rev(node)
 else:



To: joerg.sonnenberger, indygreg, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10534: tests: bump default timeout to 360s

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  A number of tests hit or almost hit the default limit even on modern
  hardware. While the tests are ideally split into smaller pieces, that's
  non-trivial work. HyperThreading and similar technologies can trigger
  this often, even without any other load on the machine.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10534

AFFECTED FILES
  tests/run-tests.py

CHANGE DETAILS

diff --git a/tests/run-tests.py b/tests/run-tests.py
--- a/tests/run-tests.py
+++ b/tests/run-tests.py
@@ -326,7 +326,7 @@
 
 default_defaults = {
 'jobs': ('HGTEST_JOBS', multiprocessing.cpu_count()),
-'timeout': ('HGTEST_TIMEOUT', 180),
+'timeout': ('HGTEST_TIMEOUT', 360),
 'slowtimeout': ('HGTEST_SLOWTIMEOUT', 1500),
 'port': ('HGTEST_PORT', 20059),
 'shell': ('HGTEST_SHELL', 'sh'),



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10533: manifests: push down expected node length into the parser

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This strictly enforces the node length in the manifest lines according
  to what the repository expects. One test case moves large hash testing
  into the non-treemanifest part as treemanifests don't provide an
  interface for overriding just the node length for now.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10533

AFFECTED FILES
  mercurial/cext/manifest.c
  mercurial/cext/parsers.c
  mercurial/cext/parsers.pyi
  mercurial/manifest.py
  mercurial/policy.py
  tests/test-manifest.py

CHANGE DETAILS

diff --git a/tests/test-manifest.py b/tests/test-manifest.py
--- a/tests/test-manifest.py
+++ b/tests/test-manifest.py
@@ -81,12 +81,12 @@
 raise NotImplementedError('parsemanifest not implemented by test case')
 
 def testEmptyManifest(self):
-m = self.parsemanifest(EMTPY_MANIFEST)
+m = self.parsemanifest(20, EMTPY_MANIFEST)
 self.assertEqual(0, len(m))
 self.assertEqual([], list(m))
 
 def testManifest(self):
-m = self.parsemanifest(A_SHORT_MANIFEST)
+m = self.parsemanifest(20, A_SHORT_MANIFEST)
 self.assertEqual([b'bar/baz/qux.py', b'foo'], list(m))
 self.assertEqual(BIN_HASH_2, m[b'bar/baz/qux.py'])
 self.assertEqual(b'l', m.flags(b'bar/baz/qux.py'))
@@ -95,20 +95,16 @@
 with self.assertRaises(KeyError):
 m[b'wat']
 
-def testManifestLongHashes(self):
-m = self.parsemanifest(b'a\0' + b'f' * 64 + b'\n')
-self.assertEqual(binascii.unhexlify(b'f' * 64), m[b'a'])
-
 def testSetItem(self):
 want = BIN_HASH_1
 
-m = self.parsemanifest(EMTPY_MANIFEST)
+m = self.parsemanifest(20, EMTPY_MANIFEST)
 m[b'a'] = want
 self.assertIn(b'a', m)
 self.assertEqual(want, m[b'a'])
 self.assertEqual(b'a\0' + HASH_1 + b'\n', m.text())
 
-m = self.parsemanifest(A_SHORT_MANIFEST)
+m = self.parsemanifest(20, A_SHORT_MANIFEST)
 m[b'a'] = want
 self.assertEqual(want, m[b'a'])
 self.assertEqual(b'a\0' + HASH_1 + b'\n' + A_SHORT_MANIFEST, m.text())
@@ -116,14 +112,14 @@
 def testSetFlag(self):
 want = b'x'
 
-m = self.parsemanifest(EMTPY_MANIFEST)
+m = self.parsemanifest(20, EMTPY_MANIFEST)
 # first add a file; a file-less flag makes no sense
 m[b'a'] = BIN_HASH_1
 m.setflag(b'a', want)
 self.assertEqual(want, m.flags(b'a'))
 self.assertEqual(b'a\0' + HASH_1 + want + b'\n', m.text())
 
-m = self.parsemanifest(A_SHORT_MANIFEST)
+m = self.parsemanifest(20, A_SHORT_MANIFEST)
 # first add a file; a file-less flag makes no sense
 m[b'a'] = BIN_HASH_1
 m.setflag(b'a', want)
@@ -133,7 +129,7 @@
 )
 
 def testCopy(self):
-m = self.parsemanifest(A_SHORT_MANIFEST)
+m = self.parsemanifest(20, A_SHORT_MANIFEST)
 m[b'a'] = BIN_HASH_1
 m2 = m.copy()
 del m
@@ -142,7 +138,7 @@
 def testCompaction(self):
 unhex = binascii.unhexlify
 h1, h2 = unhex(HASH_1), unhex(HASH_2)
-m = self.parsemanifest(A_SHORT_MANIFEST)
+m = self.parsemanifest(20, A_SHORT_MANIFEST)
 m[b'alpha'] = h1
 m[b'beta'] = h2
 del m[b'foo']
@@ -164,7 +160,7 @@
 m[b'foo']
 
 def testMatchException(self):
-m = self.parsemanifest(A_SHORT_MANIFEST)
+m = self.parsemanifest(20, A_SHORT_MANIFEST)
 match = matchmod.match(util.localpath(b'/repo'), b'', [b're:.*'])
 
 def filt(path):
@@ -177,7 +173,7 @@
 m._matches(match)
 
 def testRemoveItem(self):
-m = self.parsemanifest(A_SHORT_MANIFEST)
+m = self.parsemanifest(20, A_SHORT_MANIFEST)
 del m[b'foo']
 with self.assertRaises(KeyError):
 m[b'foo']
@@ -193,9 +189,9 @@
 addl = b'z-only-in-left\0' + HASH_1 + b'\n'
 addr = b'z-only-in-right\0' + HASH_2 + b'x\n'
 left = self.parsemanifest(
-A_SHORT_MANIFEST.replace(HASH_1, HASH_3 + b'x') + addl
+20, A_SHORT_MANIFEST.replace(HASH_1, HASH_3 + b'x') + addl
 )
-right = self.parsemanifest(A_SHORT_MANIFEST + addr)
+right = self.parsemanifest(20, A_SHORT_MANIFEST + addr)
 want = {
 b'foo': ((BIN_HASH_3, b'x'), (BIN_HASH_1, b'')),
 b'z-only-in-left': ((BIN_HASH_1, b''), MISSING),
@@ -208,14 +204,18 @@
 b'foo': (MISSING, (BIN_HASH_3, b'x')),
 b'z-only-in-left': (MISSING, (BIN_HASH_1, b'')),
 }
-self.assertEqual(want, self.parsemanifest(EMTPY_MANIFEST).diff(left))
+self.assertEqual(
+want, self.parsemanifest(20, EMTPY_MANIFEST).diff(left)
+)
 
 want = {
 b'bar/baz/qux.py': 

D10529: node: replace nullid and friends with nodeconstants class

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: indygreg.
Herald added a reviewer: durin42.
Herald added a reviewer: durin42.
Herald added a reviewer: martinvonz.
Herald added subscribers: mercurial-patches, Kwan.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  The introduction of 256bit hashes require changes to nullid and other
  constant magic values. Start pushing them down from repository and
  revlog where sensible.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10529

AFFECTED FILES
  hgext/absorb.py
  hgext/convert/git.py
  hgext/convert/hg.py
  hgext/git/dirstate.py
  hgext/git/gitlog.py
  hgext/git/gitutil.py
  hgext/git/index.py
  hgext/gpg.py
  hgext/hgk.py
  hgext/journal.py
  hgext/largefiles/basestore.py
  hgext/largefiles/lfcommands.py
  hgext/largefiles/lfutil.py
  hgext/lfs/wrapper.py
  hgext/mq.py
  hgext/narrow/narrowbundle2.py
  hgext/narrow/narrowcommands.py
  hgext/phabricator.py
  hgext/remotefilelog/contentstore.py
  hgext/remotefilelog/datapack.py
  hgext/remotefilelog/debugcommands.py
  hgext/remotefilelog/fileserverclient.py
  hgext/remotefilelog/historypack.py
  hgext/remotefilelog/metadatastore.py
  hgext/remotefilelog/remotefilectx.py
  hgext/remotefilelog/remotefilelog.py
  hgext/remotefilelog/remotefilelogserver.py
  hgext/remotefilelog/repack.py
  hgext/remotefilelog/shallowbundle.py
  hgext/remotefilelog/shallowrepo.py
  hgext/sqlitestore.py
  hgext/transplant.py
  hgext/uncommit.py
  mercurial/bookmarks.py
  mercurial/branchmap.py
  mercurial/bundle2.py
  mercurial/bundlerepo.py
  mercurial/changegroup.py
  mercurial/changelog.py
  mercurial/cmdutil.py
  mercurial/commands.py
  mercurial/commit.py
  mercurial/context.py
  mercurial/copies.py
  mercurial/debugcommands.py
  mercurial/dirstate.py
  mercurial/discovery.py
  mercurial/exchange.py
  mercurial/exchangev2.py
  mercurial/filelog.py
  mercurial/filemerge.py
  mercurial/hg.py
  mercurial/hgweb/webutil.py
  mercurial/interfaces/dirstate.py
  mercurial/localrepo.py
  mercurial/logcmdutil.py
  mercurial/manifest.py
  mercurial/merge.py
  mercurial/mergestate.py
  mercurial/metadata.py
  mercurial/obsolete.py
  mercurial/patch.py
  mercurial/phases.py
  mercurial/pure/parsers.py
  mercurial/revlog.py
  mercurial/scmutil.py
  mercurial/setdiscovery.py
  mercurial/shelve.py
  mercurial/sparse.py
  mercurial/strip.py
  mercurial/subrepo.py
  mercurial/tagmerge.py
  mercurial/tags.py
  mercurial/templatefuncs.py
  mercurial/templatekw.py
  mercurial/testing/storage.py
  mercurial/treediscovery.py
  mercurial/util.py
  mercurial/utils/storageutil.py
  mercurial/verify.py
  mercurial/wireprotov1server.py
  mercurial/wireprotov2server.py
  tests/drawdag.py
  tests/simplestorerepo.py
  tests/test-annotate.t
  tests/test-commit.t
  tests/test-fastannotate-hg.t
  tests/test-filelog.py
  tests/test-parseindex2.py
  tests/test-remotefilelog-datapack.py
  tests/test-remotefilelog-histpack.py
  tests/test-revlog-raw.py
  tests/testlib/ext-sidedata.py

To: joerg.sonnenberger, indygreg, durin42, martinvonz, #hg-reviewers
Cc: Kwan, mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10507: tests: don't hard-code /bin/bash

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.
Alphare accepted this revision.
This revision is now accepted and ready to land.
joerg.sonnenberger closed this revision.
joerg.sonnenberger added a comment.


  Merged as 77e73827a02d 


REVISION SUMMARY
  From Thomas Klausner in pkgsrc.

REPOSITORY
  rHG Mercurial

BRANCH
  stable

REVISION DETAIL
  https://phab.mercurial-scm.org/D10507

AFFECTED FILES
  tests/test-transaction-rollback-on-sigpipe.t

CHANGE DETAILS

diff --git a/tests/test-transaction-rollback-on-sigpipe.t 
b/tests/test-transaction-rollback-on-sigpipe.t
--- a/tests/test-transaction-rollback-on-sigpipe.t
+++ b/tests/test-transaction-rollback-on-sigpipe.t
@@ -22,14 +22,14 @@
 
   $ killable_pipe=`pwd`/killable_pipe.sh
   $ script $killable_pipe < #!/bin/bash
+  > #!/usr/bin/env bash
   > echo \$\$ >> $pidfile
   > exec cat
   > EOF
 
   $ remotecmd=`pwd`/remotecmd.sh
   $ script $remotecmd < #!/bin/bash
+  > #!/usr/bin/env bash
   > hg "\$@" 1> >($killable_pipe) 2> >($killable_pipe >&2)
   > EOF
 
@@ -38,7 +38,7 @@
 
   $ hook_script=`pwd`/pretxnchangegroup.sh
   $ script $hook_script < #!/bin/bash
+  > #!/usr/bin/env bash
   > for pid in \$(cat $pidfile) ; do
   >   kill \$pid
   >   while kill -0 \$pid 2>/dev/null ; do



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: clone performance regression

[Joerg Sonnenberger]

On Mon, Apr 26, 2021 at 01:48:48PM +, Kim Randell via Mercurial-devel wrote:
> We have a large repository (>130,000 changesets, 165 heads, ~3.5GB
> bundled with zstd). For the most part Mercurial performs very well for
> us. However, we noticed a significant slowdown in cloning when
> upgrading to Mercurial 5.7.1. Upon investigation I determined that this
> was due to warming the tags cache, which was added (among updating all
> caches) in this changeset:
> https://www.mercurial-scm.org/repo/hg/rev/95a615dd77bf

D10082 should fix this in principle, but it can't be used in the current
form due to historic bugs.

Joerg
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10502: packaging: extract pre-computed version when running from plain tarball

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  The tarballs created by setup.py are not including the usual archive
  markers as `hg export` leaves, so the rewrite of the version number
  computation actually left the empty version string around. This meant
  that installations from PyPI would use 0.0.0 as version string.

REPOSITORY
  rHG Mercurial

BRANCH
  stable

REVISION DETAIL
  https://phab.mercurial-scm.org/D10502

AFFECTED FILES
  setup.py

CHANGE DETAILS

diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -437,6 +437,9 @@
 version = '%(latesttag)s+hg%(latesttagdistance)s.%(node).12s' % kw
 else:
 version = '0+hg' + kw.get('node', '')[:12]
+elif os.path.exists('mercurial/__version__.py'):
+data = open('mercurial/__version__.py').read()
+version = re.search('version = b"(.*)"', data).group(1)
 
 if version:
 versionb = version



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10291: mergestate: remove unused import

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10291

AFFECTED FILES
  mercurial/mergestate.py

CHANGE DETAILS

diff --git a/mercurial/mergestate.py b/mercurial/mergestate.py
--- a/mercurial/mergestate.py
+++ b/mercurial/mergestate.py
@@ -10,7 +10,6 @@
 bin,
 hex,
 nullhex,
-nullid,
 nullrev,
 )
 from . import (



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10290: refactor: prefer checks against nullrev over nullid

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  A common pattern is using a changeset context and obtaining the node to
  compare against nullid. Change this to obtain the nullrev instead. In
  the future, the nullid becomes a property of the repository and is no
  longer a global constant, so using nullrev is much easier to reason
  about. Python function call overhead makes the difference moot, but
  future changes will result in more dictionary lookups otherwise, so
  prefer the simpler pattern.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10290

AFFECTED FILES
  hgext/extdiff.py
  hgext/split.py
  mercurial/context.py
  mercurial/copies.py
  mercurial/logcmdutil.py
  mercurial/mergestate.py
  mercurial/shelve.py
  mercurial/simplemerge.py

CHANGE DETAILS

diff --git a/mercurial/simplemerge.py b/mercurial/simplemerge.py
--- a/mercurial/simplemerge.py
+++ b/mercurial/simplemerge.py
@@ -19,7 +19,7 @@
 from __future__ import absolute_import
 
 from .i18n import _
-from .node import nullid
+from .node import nullrev
 from . import (
 error,
 mdiff,
@@ -427,7 +427,7 @@
 def is_not_null(ctx):
 if not util.safehasattr(ctx, "node"):
 return False
-return ctx.node() != nullid
+return ctx.rev() != nullrev
 
 
 def _mergediff(m3, name_a, name_b, name_base):
diff --git a/mercurial/shelve.py b/mercurial/shelve.py
--- a/mercurial/shelve.py
+++ b/mercurial/shelve.py
@@ -534,7 +534,7 @@
 parent = parents[0]
 origbranch = wctx.branch()
 
-if parent.node() != nullid:
+if parent.rev() != nullrev:
 desc = b"changes to: %s" % parent.description().split(b'\n', 1)[0]
 else:
 desc = b'(changes in empty repository)'
diff --git a/mercurial/mergestate.py b/mercurial/mergestate.py
--- a/mercurial/mergestate.py
+++ b/mercurial/mergestate.py
@@ -11,6 +11,7 @@
 hex,
 nullhex,
 nullid,
+nullrev,
 )
 from . import (
 error,
@@ -341,7 +342,7 @@
 flo = fco.flags()
 fla = fca.flags()
 if b'x' in flags + flo + fla and b'l' not in flags + flo + fla:
-if fca.node() == nullid and flags != flo:
+if fca.rev() == nullrev and flags != flo:
 if preresolve:
 self._repo.ui.warn(
 _(
diff --git a/mercurial/logcmdutil.py b/mercurial/logcmdutil.py
--- a/mercurial/logcmdutil.py
+++ b/mercurial/logcmdutil.py
@@ -14,6 +14,7 @@
 from .i18n import _
 from .node import (
 nullid,
+nullrev,
 wdirid,
 wdirrev,
 )
@@ -82,7 +83,7 @@
 If diff.merge is enabled, an overlayworkingctx of the auto-merged parents 
will be returned.
 """
 repo = ctx.repo()
-if repo.ui.configbool(b"diff", b"merge") and ctx.p2().node() != nullid:
+if repo.ui.configbool(b"diff", b"merge") and ctx.p2().rev() != nullrev:
 # avoid cycle context -> subrepo -> cmdutil -> logcmdutil
 from . import context
 
diff --git a/mercurial/copies.py b/mercurial/copies.py
--- a/mercurial/copies.py
+++ b/mercurial/copies.py
@@ -149,7 +149,7 @@
 # optimization, since the ctx.files() for a merge commit is not correct for
 # this comparison.
 forwardmissingmatch = match
-if b.p1() == a and b.p2().node() == nullid:
+if b.p1() == a and b.p2().rev() == nullrev:
 filesmatcher = matchmod.exact(b.files())
 forwardmissingmatch = matchmod.intersectmatchers(match, filesmatcher)
 if repo.ui.configbool(b'devel', b'copy-tracing.trace-all-files'):
diff --git a/mercurial/context.py b/mercurial/context.py
--- a/mercurial/context.py
+++ b/mercurial/context.py
@@ -2885,7 +2885,7 @@
 # "1 < len(self._parents)" can't be used for checking
 # existence of the 2nd parent, because "memctx._parents" is
 # explicitly initialized by the list, of which length is 2.
-if p2.node() != nullid:
+if p2.rev() != nullrev:
 man2 = p2.manifest()
 managing = lambda f: f in man1 or f in man2
 else:
@@ -2903,7 +2903,7 @@
 return scmutil.status(modified, added, removed, [], [], [], [])
 
 def parents(self):
-if self._parents[1].node() == nullid:
+if self._parents[1].rev() == nullrev:
 return [self._parents[0]]
 return self._parents
 
@@ -3052,7 +3052,7 @@
 # "1 < len(self._parents)" can't be used for checking
 # existence of the 2nd parent, because "metadataonlyctx._parents" is
 # explicitly initialized by the list, of which length is 2.
-if p2.node() != nullid:
+if p2.rev() != nullrev:
 man2 = p2.manifest()
 managing = lambda f: f in man1 or f in man2
 else:
diff --git a/hgext/split.py b/hgext/split.py
--- a/hgext/split.py
+++ b/hgext/split.py
@@ -12,7 +12,7 @@
 from mercurial.i18n import _
 
 from mercurial.node import (
-

D10289: setdiscovery: simplify by using tiprev directly

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  tip() uses tiprev() and reads the node from it, so drop a layer of
  indirection.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10289

AFFECTED FILES
  mercurial/setdiscovery.py

CHANGE DETAILS

diff --git a/mercurial/setdiscovery.py b/mercurial/setdiscovery.py
--- a/mercurial/setdiscovery.py
+++ b/mercurial/setdiscovery.py
@@ -390,7 +390,7 @@
 if audit is not None:
 audit[b'total-roundtrips'] = 1
 
-if cl.tip() == nullid:
+if cl.tiprev() == nullrev:
 if srvheadhashes != [nullid]:
 return [nullid], True, srvheadhashes
 return [nullid], False, []



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10287: shelve: simplify

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10287

AFFECTED FILES
  mercurial/shelve.py

CHANGE DETAILS

diff --git a/mercurial/shelve.py b/mercurial/shelve.py
--- a/mercurial/shelve.py
+++ b/mercurial/shelve.py
@@ -534,7 +534,7 @@
 parent = parents[0]
 origbranch = wctx.branch()
 
-if parent.node() != nullid:
+if parent.rev() != nullrev:
 desc = b"changes to: %s" % parent.description().split(b'\n', 1)[0]
 else:
 desc = b'(changes in empty repository)'



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10288: simplemerge: simplify

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10288

AFFECTED FILES
  mercurial/simplemerge.py

CHANGE DETAILS

diff --git a/mercurial/simplemerge.py b/mercurial/simplemerge.py
--- a/mercurial/simplemerge.py
+++ b/mercurial/simplemerge.py
@@ -19,7 +19,7 @@
 from __future__ import absolute_import
 
 from .i18n import _
-from .node import nullid
+from .node import nullrev
 from . import (
 error,
 mdiff,
@@ -427,7 +427,7 @@
 def is_not_null(ctx):
 if not util.safehasattr(ctx, "node"):
 return False
-return ctx.node() != nullid
+return ctx.rev() != nullrev
 
 
 def _mergediff(m3, name_a, name_b, name_base):



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10286: mergestate: simplify

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10286

AFFECTED FILES
  mercurial/mergestate.py

CHANGE DETAILS

diff --git a/mercurial/mergestate.py b/mercurial/mergestate.py
--- a/mercurial/mergestate.py
+++ b/mercurial/mergestate.py
@@ -11,6 +11,7 @@
 hex,
 nullhex,
 nullid,
+nullrev,
 )
 from . import (
 error,
@@ -341,7 +342,7 @@
 flo = fco.flags()
 fla = fca.flags()
 if b'x' in flags + flo + fla and b'l' not in flags + flo + fla:
-if fca.node() == nullid and flags != flo:
+if fca.rev() == nullrev and flags != flo:
 if preresolve:
 self._repo.ui.warn(
 _(



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10285: logcmdutil: simplify

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10285

AFFECTED FILES
  mercurial/logcmdutil.py

CHANGE DETAILS

diff --git a/mercurial/logcmdutil.py b/mercurial/logcmdutil.py
--- a/mercurial/logcmdutil.py
+++ b/mercurial/logcmdutil.py
@@ -14,6 +14,7 @@
 from .i18n import _
 from .node import (
 nullid,
+nullrev,
 wdirid,
 wdirrev,
 )
@@ -82,7 +83,7 @@
 If diff.merge is enabled, an overlayworkingctx of the auto-merged parents 
will be returned.
 """
 repo = ctx.repo()
-if repo.ui.configbool(b"diff", b"merge") and ctx.p2().node() != nullid:
+if repo.ui.configbool(b"diff", b"merge") and ctx.p2().rev() != nullrev:
 # avoid cycle context -> subrepo -> cmdutil -> logcmdutil
 from . import context
 



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10283: copies: simplify

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10283

AFFECTED FILES
  mercurial/copies.py

CHANGE DETAILS

diff --git a/mercurial/copies.py b/mercurial/copies.py
--- a/mercurial/copies.py
+++ b/mercurial/copies.py
@@ -149,7 +149,7 @@
 # optimization, since the ctx.files() for a merge commit is not correct for
 # this comparison.
 forwardmissingmatch = match
-if b.p1() == a and b.p2().node() == nullid:
+if b.p1() == a and b.p2().rev() == nullrev:
 filesmatcher = matchmod.exact(b.files())
 forwardmissingmatch = matchmod.intersectmatchers(match, filesmatcher)
 if repo.ui.configbool(b'devel', b'copy-tracing.trace-all-files'):



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10280: cmdutil: simplify

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  Prefer look-up via nullrev over nullid. We special case the latter, but
  testing an integer is still cheaper.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10280

AFFECTED FILES
  mercurial/cmdutil.py

CHANGE DETAILS

diff --git a/mercurial/cmdutil.py b/mercurial/cmdutil.py
--- a/mercurial/cmdutil.py
+++ b/mercurial/cmdutil.py
@@ -16,6 +16,7 @@
 from .node import (
 hex,
 nullid,
+nullrev,
 short,
 )
 from .pycompat import (
@@ -1936,12 +1937,12 @@
 ui.debug(b'message:\n%s\n' % (message or b''))
 
 if len(parents) == 1:
-parents.append(repo[nullid])
+parents.append(repo[nullrev])
 if opts.get(b'exact'):
 if not nodeid or not p1:
 raise error.InputError(_(b'not a Mercurial patch'))
 p1 = repo[p1]
-p2 = repo[p2 or nullid]
+p2 = repo[p2 or nullrev]
 elif p2:
 try:
 p1 = repo[p1]
@@ -1951,7 +1952,7 @@
 # first parent.
 if p1 != parents[0]:
 p1 = parents[0]
-p2 = repo[nullid]
+p2 = repo[nullrev]
 except error.RepoError:
 p1, p2 = parents
 if p2.node() == nullid:



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10279: split: simplify

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  Replace check with nullid with a check on the simpler nullrev.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10279

AFFECTED FILES
  hgext/split.py

CHANGE DETAILS

diff --git a/hgext/split.py b/hgext/split.py
--- a/hgext/split.py
+++ b/hgext/split.py
@@ -12,7 +12,7 @@
 from mercurial.i18n import _
 
 from mercurial.node import (
-nullid,
+nullrev,
 short,
 )
 
@@ -80,12 +80,12 @@
 raise error.InputError(_(b'cannot split multiple revisions'))
 
 rev = revs.first()
-ctx = repo[rev]
-# Handle nullid specially here (instead of leaving for precheck()
+# Handle nullrev specially here (instead of leaving for precheck()
 # below) so we get a nicer message and error code.
-if rev is None or ctx.node() == nullid:
+if rev is None or rev == nullrev:
 ui.status(_(b'nothing to split\n'))
 return 1
+ctx = repo[rev]
 if ctx.node() is None:
 raise error.InputError(_(b'cannot split working directory'))
 



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10278: extdiff: avoid nullid

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10278

AFFECTED FILES
  hgext/extdiff.py

CHANGE DETAILS

diff --git a/hgext/extdiff.py b/hgext/extdiff.py
--- a/hgext/extdiff.py
+++ b/hgext/extdiff.py
@@ -91,7 +91,7 @@
 
 from mercurial.i18n import _
 from mercurial.node import (
-nullid,
+nullrev,
 short,
 )
 from mercurial import (
@@ -565,18 +565,18 @@
 repo, [from_rev] + [to_rev], b'nowarn'
 )
 ctx1a = scmutil.revsingle(repo, from_rev, None)
-ctx1b = repo[nullid]
+ctx1b = repo[nullrev]
 ctx2 = scmutil.revsingle(repo, to_rev, None)
 else:
 ctx1a, ctx2 = scmutil.revpair(repo, revs)
 if not revs:
 ctx1b = repo[None].p2()
 else:
-ctx1b = repo[nullid]
+ctx1b = repo[nullrev]
 
 # Disable 3-way merge if there is only one parent
 if do3way:
-if ctx1b.node() == nullid:
+if ctx1b.rev() == nullrev:
 do3way = False
 
 matcher = scmutil.match(ctx2, pats, opts)



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10277: fix: merge imports

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10277

AFFECTED FILES
  hgext/fix.py

CHANGE DETAILS

diff --git a/hgext/fix.py b/hgext/fix.py
--- a/hgext/fix.py
+++ b/hgext/fix.py
@@ -131,8 +131,10 @@
 import subprocess
 
 from mercurial.i18n import _
-from mercurial.node import nullrev
-from mercurial.node import wdirrev
+from mercurial.node import (
+nullrev,
+wdirrev,
+)
 
 from mercurial.utils import procutil
 



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10244: tests: ask any chg instance to terminate before looking at sqlite dbs

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  There are spurious errors in CI where the database is still locked, so
  force the daemon to quit to get deterministic behavior. Since the kill
  command itself is racy, also sleep 2s to give the server time to wake up
  and exit.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10244

AFFECTED FILES
  tests/test-wireproto-exchangev2-shallow.t

CHANGE DETAILS

diff --git a/tests/test-wireproto-exchangev2-shallow.t 
b/tests/test-wireproto-exchangev2-shallow.t
--- a/tests/test-wireproto-exchangev2-shallow.t
+++ b/tests/test-wireproto-exchangev2-shallow.t
@@ -176,6 +176,10 @@
   updating the branch cache
   (sent 5 HTTP requests and * bytes; received * bytes in responses) (glob)
 
+#if chg
+  $ hg --kill-chg-daemon
+  $ sleep 2
+#endif
   $ sqlite3 -line client-shallow-1/.hg/store/db.sqlite << EOF
   > SELECT id, path, revnum, node, p1rev, p2rev, linkrev, flags FROM filedata 
ORDER BY id ASC;
   > EOF
@@ -347,6 +351,10 @@
   updating the branch cache
   (sent 5 HTTP requests and * bytes; received * bytes in responses) (glob)
 
+#if chg
+  $ hg --kill-chg-daemon
+  $ sleep 2
+#endif
   $ sqlite3 -line client-shallow-narrow-1/.hg/store/db.sqlite << EOF
   > SELECT id, path, revnum, node, p1rev, p2rev, linkrev, flags FROM filedata 
ORDER BY id ASC;
   > EOF



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10243: chg: kill trailing comma in SEE ALSO

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10243

AFFECTED FILES
  contrib/chg/chg.1

CHANGE DETAILS

diff --git a/contrib/chg/chg.1 b/contrib/chg/chg.1
--- a/contrib/chg/chg.1
+++ b/contrib/chg/chg.1
@@ -36,6 +36,6 @@
 .B \-\-kill\-chg\-daemon
 Terminate the background command servers.
 .SH SEE ALSO
-.BR hg (1),
+.BR hg (1)
 .SH AUTHOR
 Written by Yuya Nishihara .



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10242: tests: resort to fix test with newer git versions

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: durin42.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10242

AFFECTED FILES
  tests/test-git-interop.t

CHANGE DETAILS

diff --git a/tests/test-git-interop.t b/tests/test-git-interop.t
--- a/tests/test-git-interop.t
+++ b/tests/test-git-interop.t
@@ -28,9 +28,9 @@
   $ hg status
   abort: repository specified git format in .hg/requires but has no .git 
directory
   [255]
+  $ git config --global init.defaultBranch master
   $ git init
   Initialized empty Git repository in $TESTTMP/nogit/.git/
-  $ git config --global init.defaultBranch master
 This status invocation shows some hg gunk because we didn't use
 `hg init --git`, which fixes up .git/info/exclude for us.
   $ hg status



To: joerg.sonnenberger, durin42, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10241: git: fix missing case from 6266d19556ad (introduction of nodeconstants)

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: durin42.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10241

AFFECTED FILES
  hgext/git/gitlog.py

CHANGE DETAILS

diff --git a/hgext/git/gitlog.py b/hgext/git/gitlog.py
--- a/hgext/git/gitlog.py
+++ b/hgext/git/gitlog.py
@@ -218,7 +218,7 @@
 n = nodeorrev
 # handle looking up nullid
 if n == nullid:
-return hgchangelog._changelogrevision(extra={})
+return hgchangelog._changelogrevision(extra={}, manifest=nullid)
 hn = gitutil.togitnode(n)
 # We've got a real commit!
 files = [



To: joerg.sonnenberger, durin42, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10226: README: document requirement for builtin zstd

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10226

AFFECTED FILES
  README.rst

CHANGE DETAILS

diff --git a/README.rst b/README.rst
--- a/README.rst
+++ b/README.rst
@@ -18,3 +18,13 @@
 
 See https://mercurial-scm.org/ for detailed installation
 instructions, platform-specific notes, and Mercurial user information.
+
+Notes for packagers
+===
+
+Mercurial ships a copy of the python-zstandard sources. This is used to
+provide support for zstd compression and decompression functionality. The
+module is not intended to be replaced by the plain python-zstandard nor
+is it intended to use a system zstd library. Patches can result in hard
+to diagnose errors and are explicitly discouraged as unsupported
+configuration.



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10150: revlog: guarantee that p1 != null if a non-null parent exists

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: indygreg.
Herald added a reviewer: durin42.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This change does not affect the hashing (which already did this
  transformation), but can change the log output in the rare case where
  this behavior was observed in repositories. The change can simplify
  iteration code where regular changesets and merges are distinct
  branches.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10150

AFFECTED FILES
  mercurial/revlog.py
  relnotes/next
  tests/test-narrow-shallow-merges.t

CHANGE DETAILS

diff --git a/tests/test-narrow-shallow-merges.t 
b/tests/test-narrow-shallow-merges.t
--- a/tests/test-narrow-shallow-merges.t
+++ b/tests/test-narrow-shallow-merges.t
@@ -179,7 +179,7 @@
   
 
   $ hg log -T '{if(ellipsis,"...")}{node|short} {p1node|short} {p2node|short} 
{desc}\n' | sort
-  ...2a20009de83e  3ac1f5779de3 outside 10
+  ...2a20009de83e 3ac1f5779de3  outside 10
   ...3ac1f5779de3 bb96a08b062a 465567bdfb2d merge a/b/c/d 9
   ...8d874d57adea 7ef88b4dd4fa  outside 12
   ...b844052e7b3b   outside 2c
diff --git a/relnotes/next b/relnotes/next
--- a/relnotes/next
+++ b/relnotes/next
@@ -21,6 +21,13 @@
 
 == Backwards Compatibility Changes ==
 
+ * In normal repositories, the first parent of a changeset is not null,
+   unless both parents are null (like the first changeset). Some legacy
+   repositories violate this condition. The revlog code will now
+   silentely swap the parents if this condition is tested. This can
+   change the output of `hg log` when explicitly asking for first or
+   second parent.
+
 
 == Internal API Changes ==
 
diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -878,8 +878,10 @@
 if rev == wdirrev:
 raise error.WdirUnsupported
 raise
-
-return entry[5], entry[6]
+if entry[5] == nullrev:
+return entry[6], entry[5]
+else:
+return entry[5], entry[6]
 
 # fast parentrevs(rev) where rev isn't filtered
 _uncheckedparentrevs = parentrevs
@@ -900,7 +902,11 @@
 def parents(self, node):
 i = self.index
 d = i[self.rev(node)]
-return i[d[5]][7], i[d[6]][7]  # map revisions to nodes inline
+# inline node() to avoid function call overhead
+if d[5] == nullid:
+return i[d[6]][7], i[d[5]][7]
+else:
+return i[d[5]][7], i[d[6]][7]
 
 def chainlen(self, rev):
 return self._chaininfo(rev)[0]



To: joerg.sonnenberger, indygreg, durin42, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10103: relnotes: document a number of node->revision type changes

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10103

AFFECTED FILES
  relnotes/next

CHANGE DETAILS

diff --git a/relnotes/next b/relnotes/next
--- a/relnotes/next
+++ b/relnotes/next
@@ -26,3 +26,8 @@
 
  * `changelog.branchinfo` is deprecated and will be removed after 5.8.
It is superseded by `changelogrevision.branchinfo`.
+
+ * Callbacks for revlog.addgroup and the changelog._nodeduplicatecallback hook
+   now get a revision number as argument instead of a node.
+
+ * revlog.addrevision returns the revision number instead of the node.



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10082: tags: redo .hgtags file node cache to work more like the revbranchcache [WIP]

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  Computing tags requires parsing .hgtags for all heads. Mercurial
  therefore keeps a cache to efficiently find the .hgtags version of a
  revision without having to parse the manifest, but this cache is
  computed lazily and often incomplete.
  
  The new implementation of the test works a lot more like the
  revbranchcache and updates the cache in two stages:
  (1) When a new changeset is added, check if .hgtags is touched. The
  common case is that it didn't change and it is therefore inherited from
  the parent. Now the parent might not be fully resolved yet (missing
  manifest), so just keep a dictionary mapping to the parent revision that
  potentially touched it.
  (2) At the end of the transaction, resolve entries before writing the
  cache to disk. At this point, manifests are all known, so they can be
  parsed as necessary. The fast path here is reading just the delta, but
  this doesn't necessarily answer the question, since the delta could have
  been to a non-parent.
  
  If the cache logic hits an invalid or missing node, it will recheck all
  nodes. This is a bit more expensive, but simplifies the logic and avoids
  recursions. This penalty is normally hit only once, but read-only
  clients should run debugupdatecaches once and after strips.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10082

AFFECTED FILES
  mercurial/bundle2.py
  mercurial/debugcommands.py
  mercurial/interfaces/repository.py
  mercurial/localrepo.py
  mercurial/statichttprepo.py
  mercurial/tags.py

CHANGE DETAILS

diff --git a/mercurial/tags.py b/mercurial/tags.py
--- a/mercurial/tags.py
+++ b/mercurial/tags.py
@@ -492,7 +492,7 @@
 without a '.hgtags' file.
 """
 starttime = util.timer()
-fnodescache = hgtagsfnodescache(repo.unfiltered())
+fnodescache = repo.hgtagsfnodescache()
 cachefnode = {}
 for node in nodes:
 fnode = fnodescache.getfnode(node)
@@ -504,9 +504,7 @@
 duration = util.timer() - starttime
 ui.log(
 b'tagscache',
-b'%d/%d cache hits/lookups in %0.4f seconds\n',
-fnodescache.hitcount,
-fnodescache.lookupcount,
+b'cache update took %0.4f seconds\n',
 duration,
 )
 return cachefnode
@@ -695,186 +693,140 @@
 
 def __init__(self, repo):
 assert repo.filtername is None
-
 self._repo = repo
-
-# Only for reporting purposes.
-self.lookupcount = 0
-self.hitcount = 0
-
+self._delayed = {}
+self._delayed_clr = {}
 try:
-data = repo.cachevfs.read(_fnodescachefile)
-except (OSError, IOError):
-data = b""
-self._raw = bytearray(data)
+self._hgtagsrevs = bytearray(repo.cachevfs.read(_fnodescachefile))
+except (IOError, OSError):
+self._hgtagsrevs = bytearray()
+self._hgtagsrevslen = min(
+len(self._hgtagsrevs) // _fnodesrecsize, len(repo)
+)
+expected = self._hgtagsrevslen * _fnodesrecsize
+if len(self._hgtagsrevs) != expected:
+self._hgtagsrevs = self._hgtagsrevs[:expected]
+self._oldhgtagsrevslen = self._hgtagsrevslen
 
-# The end state of self._raw is an array that is of the exact length
-# required to hold a record for every revision in the repository.
-# We truncate or extend the array as necessary. self._dirtyoffset is
-# defined to be the start offset at which we need to write the output
-# file. This offset is also adjusted when new entries are calculated
-# for array members.
-cllen = len(repo.changelog)
-wantedlen = cllen * _fnodesrecsize
-rawlen = len(self._raw)
-
-self._dirtyoffset = None
+def _clear(self):
+self._hgtagsrevs = bytearray()
+self._oldhgtagsrevslen = 0
+self._hgtagsrevslen = 0
 
-rawlentokeep = min(
-wantedlen, (rawlen // _fnodesrecsize) * _fnodesrecsize
-)
-if rawlen > rawlentokeep:
-# There's no easy way to truncate array instances. This seems
-# slightly less evil than copying a potentially large array slice.
-for i in range(rawlen - rawlentokeep):
-self._raw.pop()
-rawlen = len(self._raw)
-self._dirtyoffset = rawlen
-if rawlen < wantedlen:
-if self._dirtyoffset is None:
-self._dirtyoffset = rawlen
-# TODO: zero fill entire record, because it's invalid not missing?
-self._raw.extend(b'\xff' * (wantedlen - rawlen))
-
-def getfnode(self, node, computemissing=True):
+def getfnode(self, nodeorrev, computemissing=True):
 """Obtain the filenode of the .hgtags file at a specified revision.
 
+If an .hgtags 

D10081: changelog: rename parameters to reflect semantics

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  `read` and `readfiles` can be used with a revision just as well, so
  follow the naming convention in revlog to reflect this.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10081

AFFECTED FILES
  mercurial/changelog.py

CHANGE DETAILS

diff --git a/mercurial/changelog.py b/mercurial/changelog.py
--- a/mercurial/changelog.py
+++ b/mercurial/changelog.py
@@ -507,7 +507,7 @@
 if not self._delayed:
 revlog.revlog._enforceinlinesize(self, tr, fp)
 
-def read(self, node):
+def read(self, nodeorrev):
 """Obtain data from a parsed changelog revision.
 
 Returns a 6-tuple of:
@@ -523,7 +523,7 @@
 ``changelogrevision`` instead, as it is faster for partial object
 access.
 """
-d, s = self._revisiondata(node)
+d, s = self._revisiondata(nodeorrev)
 c = changelogrevision(
 d, s, self._copiesstorage == b'changeset-sidedata'
 )
@@ -536,11 +536,11 @@
 text, sidedata, self._copiesstorage == b'changeset-sidedata'
 )
 
-def readfiles(self, node):
+def readfiles(self, nodeorrev):
 """
 short version of read that only returns the files modified by the cset
 """
-text = self.revision(node)
+text = self.revision(nodeorrev)
 if not text:
 return []
 last = text.index(b"\n\n")



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [RFC] Interaction between strip and caches

[Joerg Sonnenberger]

On Fri, Feb 26, 2021 at 10:52:52PM -0500, Augie Fackler wrote:
> 
> 
> > On Dec 14, 2020, at 5:03 PM, Joerg Sonnenberger  wrote:
> > 
> > Hello all,
> > while looking at the revbranchcache, I noticed that it is doing quite an
> > expensive probalistic invalidation dance. It is essentially looking up
> > the revision in the changelog again and compares the first 32bit to see
> > if they (still) match. Other caches are doing cheaper checks like
> > remembering the head revision and node and checking it again to match.
> > The goal is in all cases to detect one of two cases:
> > 
> > (1) Repository additions by a hg instance without support for the cache.
> > (2) Repository removals by strip without update support specific to the
> > cache in use.
> > 
> > The first part is generally handled reasonable well and cheap. Keep
> > track of the number of revisions and process to all missing changesets
> > is something code has to support anyway. The real difficult problem is
> > the second part. I would like us to adopt a more explicit way of dealing
> > with this and opt-in support via a repository requirement. Given that
> > the strip command has finally become part of core, it looks like a good
> > time to do this now.
> > 
> > The first option is to require strip to nuke all caches that it can't
> > update. This is easy to implement and works reliable by nature with all
> > existing caches. It is also the more blunt option.
> 
> Won’t the caches invalidate themselves an this defect happens today?

Only if the cache implementation hooks into strip and is active at the
time. As mentioned at the start, it is expensive and complex. I'd say
80% of the complexity of the new .hgtags cache version I am working on
is dealing with the current cache invalidation.

> > The second option is to keep a journal of strips. This can be a single
> > monotonically increasing counter and every cache just reads the counter
> > and rebuilds itself. Alternatively it could be a full journal that lists
> > the revisions and associated nodes removed. This requires changes to
> > existing caches but has the advantage that strip can be replayed by the
> > cache logic to avoid a full rebuild.
> 
> Potentially complicated, but could be worthwhile in a large repo with
> strips. Is that something you expect to encounter? For the most part
> we’ve historically considered strip an anti-pattern of sorts and not
> worried super hard about optimizing it.

My hope is that if we can handle additions by non-cache-aware clients as
we do now, it is good enough. Replaying changes is moderately cheap if
we don't have to deal with strip.

There is also the related issue of cache invalidation for obsstore, but
the same concerns apply -- replaying changes is easy as long as we don't
have to handle removal of entries.

Joerg
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: SHA1 replacement steps

[Joerg Sonnenberger]

On Fri, Feb 26, 2021 at 10:49:45PM -0500, Augie Fackler wrote:
> 
> 
> > On Feb 15, 2021, at 9:18 AM, Joerg Sonnenberger  wrote:
> > 
> > Hello all,
> > to help the review process along, here are the rough steps I see in
> > preparation for supporting 256bit hashes:
> > 
> > (1) Move the current 160bit constants from mercurial.node into a
> > subclass. Instead of a global constant, derive the correct constant from
> > the repo/revlog/... instance and pass it down as necessary. The API
> > change itself is in D9750. The expectation for this step is that a
> > repository has one hash size and one set of magic values, but it doesn't
> > change anything regarding the hash function itself. A follow-up change
> > is necessary to replace the global constants (approximately D9465 minus
> > D9750).
> 
> I was somewhat assuming we’d alter various codepaths to always emit
> 256-bit hashes, even if they end in all NUL. Your way sounds a little
> more complicated but is also fine, I don’t feel strongly.

I was considering it. It is harder to ensure that nothing breaks as we
would have to fix all external interactions at the same time from
output commands to network IO. Keeping the code parts somewhat separate
has the advantage that we can sort out the new hash first and the compat
migration as second step without having to worry about compat as much.

> > 
> > (2) Adjust various on-disk formats to switch between the current 160bit
> > and 256bit encoding based on the node constants in use. This would be a
> > non-functional change for existing repositories.
> 
> Are any on-disk formats not already using 256-bit storage?
> I know revlogs are, so I _think_ this is only going to matter for caches.

It's a mixed bag. The textual formats are fine, revlog is fine. Most of
the ad-hoc formats are 160bit only.

> > 
> > (3) Introduce the tagged 256(*) hash function. My plan here is to use
> > Blake2b configured for 248bit output and a suffix of b'\x01'. It is a
> > bit wasteful to reserve 8bit for the tag, but simplifies code. Biggest
> > downside is that the full Blake2b support is not available in Python 2.
> 
> Honestly I think new hash functions is exactly the kind of thing we
> should gate on Python 3. If someone is _really_ enthusiastic they can
> write a backport extension or something, for the users that are
> (bafflingly) caring about modern hashes but stuck on an ancient Python.

Yeah, I'm mostly fine with not supporting Python 2 here. Just needs some
small feature testing I think to not mess up the current status.

> > The tag would allow different hash functions to co-exist and embed
> > existing SHA1 hashes by zero padding.
> > 
> > (4) Adjust hash verification logic to derive the hash function from the
> > tag of a node, not just hard-coding it.
> > 
> > At the end of step 4, most repositories can be converted in a mostly
> > transparent way.
> 
> Notably, you can allow people to only upgrade new hashes if they’re so
> inclined, which lets you preserve gpg signatures etc.

Right, migration will not be forced in the near future. Maybe in a few
years, but not now.

> > Some additional changes might be necessary for allowing
> > "short" node ids for things like .hgtags, but overall, existing hashes
> > should just continue to work as before.
> 
> Overall +1. We can arm-wrestle later about if allowing a “new commits
> are blake2b” mode (vs convert-the-repo mode) is reasonable, I don’t
> think it’ll take a ton of code either way.
> 
> One request: I think we should reserve a couple of suffixes (0xff and
> 0xfe, perhaps?) for “private use” - this would be useful for large
> installations that do strange things with hashing out of necessity.

All F is currently used as marker entry by the hgtags cache for example,
so yeah, it certainly sounds sensible to reserve certain entries.

Joerg
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10075: ci: hook network-io tests into the pipeline

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This runs the "pip install" tests once for Python 2 and 3 each.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10075

AFFECTED FILES
  contrib/heptapod-ci.yml

CHANGE DETAILS

diff --git a/contrib/heptapod-ci.yml b/contrib/heptapod-ci.yml
--- a/contrib/heptapod-ci.yml
+++ b/contrib/heptapod-ci.yml
@@ -8,6 +8,7 @@
 PYTHON: python
 TEST_HGMODULEPOLICY: "allow"
 HG_CI_IMAGE_TAG: "latest"
+TEST_HGTESTS_ALLOW_NETIO: "0"
 
 .runtests_template: 
 stage: tests
@@ -23,7 +24,7 @@
 script:
 - echo "python used, $PYTHON"
 - echo "$RUNTEST_ARGS"
-- HGMODULEPOLICY="$TEST_HGMODULEPOLICY" "$PYTHON" tests/run-tests.py 
--color=always $RUNTEST_ARGS
+- HGTESTS_ALLOW_NETIO="$TEST_HGTESTS_ALLOW_NETIO" 
HGMODULEPOLICY="$TEST_HGMODULEPOLICY" "$PYTHON" tests/run-tests.py 
--color=always $RUNTEST_ARGS
 
 
 .rust_template: 
@@ -69,6 +70,7 @@
 variables:
 RUNTEST_ARGS: " --no-rust --blacklist /tmp/check-tests.txt"
 TEST_HGMODULEPOLICY: "c"
+TEST_HGTESTS_ALLOW_NETIO: "1"
 
 test-py3:
 <<: *runtests
@@ -76,6 +78,7 @@
 RUNTEST_ARGS: " --no-rust --blacklist /tmp/check-tests.txt"
 PYTHON: python3
 TEST_HGMODULEPOLICY: "c"
+TEST_HGTESTS_ALLOW_NETIO: "1"
 
 test-py2-pure:
 <<: *runtests



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

SHA1 replacement steps

[Joerg Sonnenberger]

Hello all,
to help the review process along, here are the rough steps I see in
preparation for supporting 256bit hashes:

(1) Move the current 160bit constants from mercurial.node into a
subclass. Instead of a global constant, derive the correct constant from
the repo/revlog/... instance and pass it down as necessary. The API
change itself is in D9750. The expectation for this step is that a
repository has one hash size and one set of magic values, but it doesn't
change anything regarding the hash function itself. A follow-up change
is necessary to replace the global constants (approximately D9465 minus
D9750).

(2) Adjust various on-disk formats to switch between the current 160bit
and 256bit encoding based on the node constants in use. This would be a
non-functional change for existing repositories.

(3) Introduce the tagged 256(*) hash function. My plan here is to use
Blake2b configured for 248bit output and a suffix of b'\x01'. It is a
bit wasteful to reserve 8bit for the tag, but simplifies code. Biggest
downside is that the full Blake2b support is not available in Python 2.

The tag would allow different hash functions to co-exist and embed
existing SHA1 hashes by zero padding.

(4) Adjust hash verification logic to derive the hash function from the
tag of a node, not just hard-coding it.

At the end of step 4, most repositories can be converted in a mostly
transparent way. Some additional changes might be necessary for allowing
"short" node ids for things like .hgtags, but overall, existing hashes
should just continue to work as before.

Joerg
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D9955: build: fake PEP440 versions

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  If the current version is not exactly a tag, use a local version
  specifier to fix it up. PEP 440 uses the "+" separator and only allows
  alphanumeric and dot, so use dot for further separations.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9955

AFFECTED FILES
  setup.py

CHANGE DETAILS

diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -419,9 +419,9 @@
 ltag = sysstr(hg.run(ltagcmd))
 changessincecmd = ['log', '-T', 'x\n', '-r', "only(.,'%s')" % ltag]
 changessince = len(hg.run(changessincecmd).splitlines())
-version = '%s+%s-%s' % (ltag, changessince, hgid)
+version = '%s+hg%s.%s' % (ltag, changessince, hgid)
 if version.endswith('+'):
-version += time.strftime('%Y%m%d')
+version = version[:-1] + 'local' + time.strftime('%Y%m%d')
 elif os.path.exists('.hg_archival.txt'):
 kw = dict(
 [[t.strip() for t in l.split(':', 1)] for l in 
open('.hg_archival.txt')]
@@ -430,11 +430,15 @@
 version = kw['tag']
 elif 'latesttag' in kw:
 if 'changessincelatesttag' in kw:
-version = '%(latesttag)s+%(changessincelatesttag)s-%(node).12s' % 
kw
+version = (
+'%(latesttag)s+.%(changessincelatesttag)s.%(node).12s' % kw
+)
 else:
-version = '%(latesttag)s+%(latesttagdistance)s-%(node).12s' % kw
+version = (
+'%(latesttag)s+.%(latesttagdistance)s.%(node).12s' % kw
+)
 else:
-version = kw.get('node', '')[:12]
+version = '0+' + kw.get('node', '')[:12]
 
 if version:
 versionb = version
@@ -451,20 +455,6 @@
 ),
 )
 
-try:
-oldpolicy = os.environ.get('HGMODULEPOLICY', None)
-os.environ['HGMODULEPOLICY'] = 'py'
-from mercurial import __version__
-
-version = __version__.version
-except ImportError:
-version = b'unknown'
-finally:
-if oldpolicy is None:
-del os.environ['HGMODULEPOLICY']
-else:
-os.environ['HGMODULEPOLICY'] = oldpolicy
-
 
 class hgbuild(build):
 # Insert hgbuildmo first so that files in mercurial/locale/ are found
@@ -1683,8 +1673,8 @@
 # unicode on Python 2 still works because it won't contain any
 # non-ascii bytes and will be implicitly converted back to bytes
 # when operated on.
-assert isinstance(version, bytes)
-setupversion = version.decode('ascii')
+assert isinstance(version, str)
+setupversion = version
 
 extra = {}
 



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D9950: ci: test real dependency installation for pip [WIP]

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  In the past, the pip smoke test inhibited actual dependency
  installation, but that fails in different environments for setuptools
  itself. Since it isn't what we actually want to test (which is pip
  install), allow this to call home.
  
  WIP as this part of the test should only be run during CI.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9950

AFFECTED FILES
  setup.py
  tests/test-install.t

CHANGE DETAILS

diff --git a/tests/test-install.t b/tests/test-install.t
--- a/tests/test-install.t
+++ b/tests/test-install.t
@@ -197,8 +197,10 @@
 
 Note: we use this weird path to run pip and hg to avoid platform differences,
 since it's bin on most platforms but Scripts on Windows.
-  $ ./installenv/*/pip install --no-index $TESTDIR/.. >> pip.log
+  $ ./installenv/*/pip install $TESTDIR/.. >> pip.log
 Failed building wheel for mercurial (?)
+  WARNING: You are using pip version *; however, version * is available. 
(glob) (?)
+  You should consider upgrading via the '$TESTTMP/installenv/bin/python* -m 
pip install --upgrade pip' command. (glob) (?)
   $ ./installenv/*/hg debuginstall || cat pip.log
   checking encoding (ascii)...
   checking Python executable (*) (glob)
@@ -226,13 +228,13 @@
 
 Note: --no-site-packages is the default for all versions enabled by hghave
 
-  $ "$PYTHON" -m virtualenv --never-download installenv >> pip.log
+  $ "$PYTHON" -m virtualenv installenv >> pip.log
   DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. 
Please upgrade your Python as Python 2.7 won't be maintained after that date. A 
future version of pip will drop support for Python 2.7. (?)
   DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. 
Please upgrade your Python as Python 2.7 won't be maintained after that date. A 
future version of pip will drop support for Python 2.7. More details about 
Python 2 support in pip, can be found at 
https://pip.pypa.io/en/latest/development/release-process/#python-2-support (?)
 
 Note: we use this weird path to run pip and hg to avoid platform differences,
 since it's bin on most platforms but Scripts on Windows.
-  $ ./installenv/*/pip install --no-index $TESTDIR/.. >> pip.log
+  $ ./installenv/*/pip install $TESTDIR/.. >> pip.log
   DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. 
Please upgrade your Python as Python 2.7 won't be maintained after that date. A 
future version of pip will drop support for Python 2.7. (?)
   DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. 
Please upgrade your Python as Python 2.7 won't be maintained after that date. A 
future version of pip will drop support for Python 2.7. More details about 
Python 2 support in pip, can be found at 
https://pip.pypa.io/en/latest/development/release-process/#python-2-support (?)
   DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. 
Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will 
drop support for Python 2.7 in January 2021. More details about Python 2 
support in pip can be found at 
https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 
21.0 will remove support for this functionality. (?)
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -419,9 +419,9 @@
 ltag = sysstr(hg.run(ltagcmd))
 changessincecmd = ['log', '-T', 'x\n', '-r', "only(.,'%s')" % ltag]
 changessince = len(hg.run(changessincecmd).splitlines())
-version = '%s+%s-%s' % (ltag, changessince, hgid)
+version = '%s+hg%s.%s' % (ltag, changessince, hgid)
 if version.endswith('+'):
-version += time.strftime('%Y%m%d')
+version = version[:-1] + 'local' + time.strftime('%Y%m%d')
 elif os.path.exists('.hg_archival.txt'):
 kw = dict(
 [[t.strip() for t in l.split(':', 1)] for l in 
open('.hg_archival.txt')]
@@ -430,11 +430,15 @@
 version = kw['tag']
 elif 'latesttag' in kw:
 if 'changessincelatesttag' in kw:
-version = '%(latesttag)s+%(changessincelatesttag)s-%(node).12s' % 
kw
+version = (
+'%(latesttag)s'  # +.%(changessincelatesttag)s.%(node).12s' % 
kw
+)
 else:
-version = '%(latesttag)s+%(latesttagdistance)s-%(node).12s' % kw
+version = (
+'%(latesttag)s'  # +.%(latesttagdistance)s.%(node).12s' % kw
+)
 else:
-version = kw.get('node', '')[:12]
+version = '0'  # + kw.get('node', '')[:12]
 
 if version:
 versionb = version
@@ -451,20 +455,6 @@
 ),
 )
 
-try:
-oldpolicy = os.environ.get('HGMODULEPOLICY', None)
-os.environ['HGMODULEPOLICY'] = 'py'
-from mercurial import __version__
-
-version = 

D9928: tests: drop pip test on the client

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  The PIP testing is fragile, various hacks are included already. It has
  been failing persistently in the CI and locally on OpenSuSE. Since
  general PIP use requires network access, it is better done during CI
  anyway.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9928

AFFECTED FILES
  tests/test-install.t

CHANGE DETAILS

diff --git a/tests/test-install.t b/tests/test-install.t
--- a/tests/test-install.t
+++ b/tests/test-install.t
@@ -170,93 +170,3 @@
   $ hg debuginstall --config extensions.fsmonitor= --config 
fsmonitor.watchman_exe=false -Tjson | grep atchman
 "fsmonitor-watchman": "false",
 "fsmonitor-watchman-error": "warning: Watchman unavailable: watchman 
exited with code 1",
-
-Verify that Mercurial is installable with pip. Note that this MUST be
-the last test in this file, because we do some nasty things to the
-shell environment in order to make the virtualenv work reliably.
-
-On Python 3, we use the venv module, which is part of the standard library.
-But some Linux distros strip out this module's functionality involving pip,
-so we have to look for the ensurepip module, which these distros strip out
-completely.
-On Python 2, we use the 3rd party virtualenv module, if available.
-
-  $ cd $TESTTMP
-  $ unset PYTHONPATH
-
-#if py3 ensurepip
-  $ "$PYTHON" -m venv installenv >> pip.log
-
-Hack: Debian does something a bit different in ensurepip.bootstrap. This makes
-it so that pip thinks the 'wheel' wheel is installed so it can build wheels;
-when it goes to try, however, it shells out to run `python3 -u `,
-that *doesn't* get the 'wheel' wheel, and it fails with an invalid command
-'bdist_wheel'. To fix this, we just delete the wheel from where Debian put it 
in
-our virtual env. Then pip doesn't think it's installed and doesn't try to 
build.
-  $ rm installenv/share/python-wheels/wheel-*.whl >/dev/null 2>&1 || true
-
-Note: we use this weird path to run pip and hg to avoid platform differences,
-since it's bin on most platforms but Scripts on Windows.
-  $ ./installenv/*/pip install --no-index $TESTDIR/.. >> pip.log
-Failed building wheel for mercurial (?)
-  $ ./installenv/*/hg debuginstall || cat pip.log
-  checking encoding (ascii)...
-  checking Python executable (*) (glob)
-  checking Python implementation (*) (glob)
-  checking Python version (3.*) (glob)
-  checking Python lib (*)... (glob)
-  checking Python security support (*) (glob)
-  checking Rust extensions \((installed|missing)\) (re)
-  checking Mercurial version (*) (glob)
-  checking Mercurial custom build (*) (glob)
-  checking module policy (*) (glob)
-  checking installed modules (*/mercurial)... (glob)
-  checking registered compression engines (*) (glob)
-  checking available compression engines (*) (glob)
-  checking available compression engines for wire protocol (*) (glob)
-  checking "re2" regexp engine \((available|missing)\) (re)
-  checking templates 
($TESTTMP/installenv/*/site-packages/mercurial/templates)... (glob)
-  checking default template 
($TESTTMP/installenv/*/site-packages/mercurial/templates/map-cmdline.default) 
(glob)
-  checking commit editor... (*) (glob)
-  checking username (test)
-  no problems detected
-#endif
-
-#if virtualenv no-py3
-
-Note: --no-site-packages is the default for all versions enabled by hghave
-
-  $ "$PYTHON" -m virtualenv --never-download installenv >> pip.log
-  DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. 
Please upgrade your Python as Python 2.7 won't be maintained after that date. A 
future version of pip will drop support for Python 2.7. (?)
-  DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. 
Please upgrade your Python as Python 2.7 won't be maintained after that date. A 
future version of pip will drop support for Python 2.7. More details about 
Python 2 support in pip, can be found at 
https://pip.pypa.io/en/latest/development/release-process/#python-2-support (?)
-
-Note: we use this weird path to run pip and hg to avoid platform differences,
-since it's bin on most platforms but Scripts on Windows.
-  $ ./installenv/*/pip install --no-index $TESTDIR/.. >> pip.log
-  DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. 
Please upgrade your Python as Python 2.7 won't be maintained after that date. A 
future version of pip will drop support for Python 2.7. (?)
-  DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. 
Please upgrade your Python as Python 2.7 won't be maintained after that date. A 
future version of pip will drop support for Python 2.7. More details about 
Python 2 support in pip, can be found at 
https://pip.pypa.io/en/latest/development/release-process/#python-2-support (?)
-  DEPRECATION: Python 2.7 reached the end of its life on 

D9883: revlog: change addgroup callbacks to take revision numbers

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: indygreg.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9883

AFFECTED FILES
  hgext/sqlitestore.py
  mercurial/changegroup.py
  mercurial/changelog.py
  mercurial/exchangev2.py
  mercurial/interfaces/repository.py
  mercurial/revlog.py
  mercurial/testing/storage.py
  tests/simplestorerepo.py

CHANGE DETAILS

diff --git a/tests/simplestorerepo.py b/tests/simplestorerepo.py
--- a/tests/simplestorerepo.py
+++ b/tests/simplestorerepo.py
@@ -550,7 +550,7 @@
 
 if node in self._indexbynode:
 if duplicaterevisioncb:
-duplicaterevisioncb(self, node)
+duplicaterevisioncb(self, self.rev(node))
 empty = False
 continue
 
@@ -560,12 +560,12 @@
 else:
 text = mdiff.patch(self.revision(deltabase), delta)
 
-self._addrawrevision(
+rev = self._addrawrevision(
 node, text, transaction, linkrev, p1, p2, flags
 )
 
 if addrevisioncb:
-addrevisioncb(self, node)
+addrevisioncb(self, rev)
 empty = False
 return not empty
 
diff --git a/mercurial/testing/storage.py b/mercurial/testing/storage.py
--- a/mercurial/testing/storage.py
+++ b/mercurial/testing/storage.py
@@ -1129,12 +1129,13 @@
 with self._maketransactionfn() as tr:
 nodes = []
 
-def onchangeset(cl, node):
+def onchangeset(cl, rev):
+node = cl.node(rev)
 nodes.append(node)
 cb(cl, node)
 
-def ondupchangeset(cl, node):
-nodes.append(node)
+def ondupchangeset(cl, rev):
+nodes.append(cl.node(rev))
 
 f.addgroup(
 [],
@@ -1163,12 +1164,13 @@
 with self._maketransactionfn() as tr:
 nodes = []
 
-def onchangeset(cl, node):
+def onchangeset(cl, rev):
+node = cl.node(rev)
 nodes.append(node)
 cb(cl, node)
 
-def ondupchangeset(cl, node):
-nodes.append(node)
+def ondupchangeset(cl, rev):
+nodes.append(cl.node(rev))
 
 f.addgroup(
 deltas,
@@ -1217,8 +1219,8 @@
 with self._maketransactionfn() as tr:
 newnodes = []
 
-def onchangeset(cl, node):
-newnodes.append(node)
+def onchangeset(cl, rev):
+newnodes.append(cl.node(rev))
 
 f.addgroup(
 deltas,
diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -2424,11 +2424,12 @@
 link = linkmapper(linknode)
 flags = flags or REVIDX_DEFAULT_FLAGS
 
-if self.index.has_node(node):
+rev = self.index.get_rev(node)
+if rev is not None:
 # this can happen if two branches make the same change
-self._nodeduplicatecallback(transaction, node)
+self._nodeduplicatecallback(transaction, rev)
 if duplicaterevisioncb:
-duplicaterevisioncb(self, node)
+duplicaterevisioncb(self, rev)
 empty = False
 continue
 
@@ -2466,7 +2467,7 @@
 # We're only using addgroup() in the context of changegroup
 # generation so the revision data can always be handled as raw
 # by the flagprocessor.
-self._addrevision(
+rev = self._addrevision(
 node,
 None,
 transaction,
@@ -2482,7 +2483,7 @@
 )
 
 if addrevisioncb:
-addrevisioncb(self, node)
+addrevisioncb(self, rev)
 empty = False
 
 if not dfh and not self._inline:
diff --git a/mercurial/interfaces/repository.py 
b/mercurial/interfaces/repository.py
--- a/mercurial/interfaces/repository.py
+++ b/mercurial/interfaces/repository.py
@@ -774,8 +774,9 @@
 This used to be the default when ``addrevisioncb`` was provided up to
 Mercurial 5.8.
 
-``addrevisioncb`` should be called for each node as it is committed.
-``duplicaterevisioncb`` should be called for each pre-existing node.
+``addrevisioncb`` should be called for each new rev as it is committed.
+``duplicaterevisioncb`` should be called for all revs with a
+pre-existing node.
 
 ``maybemissingparents`` is a bool indicating whether the incoming
 data may reference 

D9882: revlog: change addrevision to return the new revision, not node

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: indygreg.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9882

AFFECTED FILES
  mercurial/changelog.py
  mercurial/filelog.py
  mercurial/interfaces/repository.py
  mercurial/manifest.py
  mercurial/revlog.py

CHANGE DETAILS

diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -2111,13 +2111,14 @@
 )
 
 node = node or self.hash(rawtext, p1, p2)
-if self.index.has_node(node):
-return node
+rev = self.index.get_rev(node)
+if rev is not None:
+return rev
 
 if validatehash:
 self.checkhash(rawtext, node, p1=p1, p2=p2)
 
-rev = self.addrawrevision(
+return self.addrawrevision(
 rawtext,
 transaction,
 link,
@@ -2128,7 +2129,6 @@
 cachedelta=cachedelta,
 deltacomputer=deltacomputer,
 )
-return node
 
 def addrawrevision(
 self,
diff --git a/mercurial/manifest.py b/mercurial/manifest.py
--- a/mercurial/manifest.py
+++ b/mercurial/manifest.py
@@ -1704,9 +1704,10 @@
 arraytext, deltatext = m.fastdelta(self.fulltextcache[p1], work)
 cachedelta = self._revlog.rev(p1), deltatext
 text = util.buffer(arraytext)
-n = self._revlog.addrevision(
+rev = self._revlog.addrevision(
 text, transaction, link, p1, p2, cachedelta
 )
+n = self._revlog.node(rev)
 except FastdeltaUnavailable:
 # The first parent manifest isn't already loaded or the
 # manifest implementation doesn't support fastdelta, so
@@ -1724,7 +1725,8 @@
 arraytext = None
 else:
 text = m.text()
-n = self._revlog.addrevision(text, transaction, link, p1, p2)
+rev = self._revlog.addrevision(text, transaction, link, p1, p2)
+n = self._revlog.node(rev)
 arraytext = bytearray(text)
 
 if arraytext is not None:
@@ -1765,9 +1767,10 @@
 n = m2.node()
 
 if not n:
-n = self._revlog.addrevision(
+rev = self._revlog.addrevision(
 text, transaction, link, m1.node(), m2.node()
 )
+n = self._revlog.node(rev)
 
 # Save nodeid so parent manifest can calculate its nodeid
 m.setnode(n)
diff --git a/mercurial/interfaces/repository.py 
b/mercurial/interfaces/repository.py
--- a/mercurial/interfaces/repository.py
+++ b/mercurial/interfaces/repository.py
@@ -734,7 +734,7 @@
 flags=0,
 cachedelta=None,
 ):
-"""Add a new revision to the store.
+"""Add a new revision to the store and return its number.
 
 This is similar to ``add()`` except it operates at a lower level.
 
diff --git a/mercurial/filelog.py b/mercurial/filelog.py
--- a/mercurial/filelog.py
+++ b/mercurial/filelog.py
@@ -176,7 +176,8 @@
 def add(self, text, meta, transaction, link, p1=None, p2=None):
 if meta or text.startswith(b'\1\n'):
 text = storageutil.packmeta(meta, text)
-return self.addrevision(text, transaction, link, p1, p2)
+rev = self.addrevision(text, transaction, link, p1, p2)
+return self.node(rev)
 
 def renamed(self, node):
 return storageutil.filerevisioncopied(self, node)
diff --git a/mercurial/changelog.py b/mercurial/changelog.py
--- a/mercurial/changelog.py
+++ b/mercurial/changelog.py
@@ -598,9 +598,10 @@
 parseddate = b"%s %s" % (parseddate, extra)
 l = [hex(manifest), user, parseddate] + sortedfiles + [b"", desc]
 text = b"\n".join(l)
-return self.addrevision(
+rev = self.addrevision(
 text, transaction, len(self), p1, p2, sidedata=sidedata, 
flags=flags
 )
+return self.node(rev)
 
 def branchinfo(self, rev):
 """return the branch name and open/close state of a revision



To: joerg.sonnenberger, indygreg, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D9884: changegroup: don't convert revisions to node for duplicate handling

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  The only consumer can handle revision lists fine. Avoid
  materializing a range if there are no duplicates as optimization.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9884

AFFECTED FILES
  mercurial/changegroup.py

CHANGE DETAILS

diff --git a/mercurial/changegroup.py b/mercurial/changegroup.py
--- a/mercurial/changegroup.py
+++ b/mercurial/changegroup.py
@@ -316,11 +316,11 @@
 self.callback = progress.increment
 
 efilesset = set()
-cgnodes = []
+duprevs = []
 
 def ondupchangelog(cl, rev):
 if rev < clstart:
-cgnodes.append(cl.node(rev))
+duprevs.append(rev)
 
 def onchangelog(cl, rev):
 ctx = cl.changelogrevision(rev)
@@ -448,8 +448,12 @@
 if added:
 phases.registernew(repo, tr, targetphase, added)
 if phaseall is not None:
-phases.advanceboundary(repo, tr, phaseall, cgnodes, revs=added)
-cgnodes = []
+if duprevs:
+duprevs.extend(added)
+else:
+duprevs = added
+phases.advanceboundary(repo, tr, phaseall, [], revs=duprevs)
+duprevs = []
 
 if changesets > 0:
 



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D9881: revlog: change addrawrevision to return the revision

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: indygreg.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9881

AFFECTED FILES
  mercurial/revlog.py

CHANGE DETAILS

diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -2117,7 +2117,7 @@
 if validatehash:
 self.checkhash(rawtext, node, p1=p1, p2=p2)
 
-return self.addrawrevision(
+rev = self.addrawrevision(
 rawtext,
 transaction,
 link,
@@ -2128,6 +2128,7 @@
 cachedelta=cachedelta,
 deltacomputer=deltacomputer,
 )
+return node
 
 def addrawrevision(
 self,
@@ -2150,7 +2151,7 @@
 dfh = self._datafp(b"a+")
 ifh = self._indexfp(b"a+")
 try:
-self._addrevision(
+return self._addrevision(
 node,
 rawtext,
 transaction,
@@ -2163,7 +2164,6 @@
 dfh,
 deltacomputer=deltacomputer,
 )
-return node
 finally:
 if dfh:
 dfh.close()



To: joerg.sonnenberger, indygreg, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D9880: revlog: change _addrevision to return the new revision

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: indygreg.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  The node is passed as argument already, so returning it is quite
  pointless. The revision number on the other is useful as it decouples
  the caller from the revlog internals.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9880

AFFECTED FILES
  mercurial/revlog.py

CHANGE DETAILS

diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -2150,7 +2150,7 @@
 dfh = self._datafp(b"a+")
 ifh = self._indexfp(b"a+")
 try:
-return self._addrevision(
+self._addrevision(
 node,
 rawtext,
 transaction,
@@ -2163,6 +2163,7 @@
 dfh,
 deltacomputer=deltacomputer,
 )
+return node
 finally:
 if dfh:
 dfh.close()
@@ -2334,7 +2335,7 @@
 if type(rawtext) == bytes:  # only accept immutable objects
 self._revisioncache = (node, curr, rawtext)
 self._chainbasecache[curr] = deltainfo.chainbase
-return node
+return curr
 
 def _writeentry(self, transaction, ifh, dfh, entry, data, link, offset):
 # Files opened in a+ mode have inconsistent behavior on various



To: joerg.sonnenberger, indygreg, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D9878: revlog: initial version of phash index [POC]

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: indygreg.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  Integration is still somewhat hackish, but a reasonable start for a PoC.
  Comparing it to Rust is not entirely fair as the additional Python
  function in between dominates the runtime. The generator is pure Python
  at the moment and at least a factor of 25 slower than the comparable C
  extension. No fallback path for hash function yet to a real universal
  hash (as opposed to just assuming the node hash is well distributed
  enough). Most interesting baseline: pure Python lookup via the hash
  function is a factor of 20 slower than the current dictionary lookup.
  The index generation takes 37s for 2 * 570k revisions (manifest +
  changelog). Further testing to measure the incremental cache update cost
  is necessary, since initial testing shows a negative cost overall. It
  can be estimated to be bound by ~65ms on the test platform.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9878

AFFECTED FILES
  mercurial/cext/revlog.c
  mercurial/changelog.py
  mercurial/localrepo.py
  mercurial/pure/parsers.py
  mercurial/revlog.py
  mercurial/utils/phash_reader.py
  mercurial/utils/phash_writer.py

CHANGE DETAILS

diff --git a/mercurial/utils/phash_writer.py b/mercurial/utils/phash_writer.py
new file mode 100644
--- /dev/null
+++ b/mercurial/utils/phash_writer.py
@@ -0,0 +1,104 @@
+import struct
+
+be32 = struct.Struct('>I')
+
+
+def default_hash(key, seed):
+base = seed * 4
+return [
+be32.unpack(key[base : base + 4])[0],
+be32.unpack(key[base + 4 : base + 8])[0],
+be32.unpack(key[base + 8 : base + 12])[0],
+]
+
+
+class hashtable:
+def __init__(self, hash=default_hash):
+self.entries = []
+self.hash = hash
+
+def insert(self, key):
+self.entries.append(key)
+
+def _hash_entry(self, nv, seed, entry):
+hash = [x % nv for x in self.hash(entry, seed)]
+if hash[0] == hash[1]:
+hash[1] ^= 1
+if hash[0] == hash[2] or hash[1] == hash[2]:
+hash[2] ^= 1
+if hash[0] == hash[2] or hash[1] == hash[2]:
+hash[2] ^= 2
+
+return hash
+
+def _compute_graph(self, ne, nv, seed):
+edges = []
+vertices_edges = [0] * nv
+vertices_degrees = [0] * nv
+
+for i in range(ne):
+e = self._hash_entry(nv, seed, self.entries[i])
+edges.append(e)
+for v in e:
+vertices_edges[v] ^= i
+vertices_degrees[v] += 1
+
+output_order = []
+
+def remove_vertex(v):
+if vertices_degrees[v] != 1:
+return
+e = vertices_edges[v]
+output_order.append(e)
+for v in edges[e]:
+vertices_edges[v] ^= e
+vertices_degrees[v] -= 1
+
+for v in range(nv):
+remove_vertex(v)
+oi = 0
+while oi < len(output_order):
+for v in edges[output_order[oi]]:
+remove_vertex(v)
+oi += 1
+
+if len(output_order) == ne:
+return edges, output_order
+else:
+return None
+
+def write(self):
+seed = 0
+ne = len(self.entries)
+nv = max(128, ne * 124 // 100)
+nv = (nv | 3) + 1
+
+rv = None
+seed = -1
+while rv is None:
+seed += 1
+rv = self._compute_graph(ne, nv, seed)
+edges, output_order = rv
+
+g = [0] * nv
+visited = [False] * nv
+for idx in reversed(output_order):
+edge = edges[idx]
+if not visited[edge[0]]:
+g[edge[0]] = (idx - g[edge[1]] - g[edge[2]]) % ne
+elif not visited[edge[1]]:
+g[edge[1]] = (idx - g[edge[0]] - g[edge[2]]) % ne
+else:
+g[edge[2]] = (idx - g[edge[1]] - g[edge[0]]) % ne
+for v in edge:
+visited[v] = True
+
+output = bytearray(12 + 4 * len(g))
+be32.pack_into(output, 0, seed)
+be32.pack_into(output, 4, nv)
+be32.pack_into(output, 8, ne)
+pos = 12
+for v in g:
+be32.pack_into(output, pos, v)
+pos += 4
+return output
diff --git a/mercurial/utils/phash_reader.py b/mercurial/utils/phash_reader.py
new file mode 100644
--- /dev/null
+++ b/mercurial/utils/phash_reader.py
@@ -0,0 +1,39 @@
+import struct
+
+be32 = struct.Struct('>I')
+
+
+def default_hash(key, seed):
+base = seed * 4
+return [
+be32.unpack(key[base : base + 4])[0],
+be32.unpack(key[base + 4 : base + 8])[0],
+be32.unpack(key[base + 8 : base + 12])[0],
+]
+
+
+class hashtable:
+def __init__(self, data, hash=default_hash):
+self.data = data
+

D9866: debugshell: add a simple command for starting an interactive shell

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This simplifies interactive exploration of the Mercurial APIs.
  The ui and repo instances are provided as local variables.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9866

AFFECTED FILES
  mercurial/debugcommands.py

CHANGE DETAILS

diff --git a/mercurial/debugcommands.py b/mercurial/debugcommands.py
--- a/mercurial/debugcommands.py
+++ b/mercurial/debugcommands.py
@@ -3717,6 +3717,15 @@
 ui.writenoi18n(b' revision %s\n' % v[1])
 
 
+@command(b'debugshell')
+def debugshell(ui, repo):
+import code
+imported_objects = {
+'ui': ui,
+'repo': repo,
+}
+code.interact(local=imported_objects)
+
 @command(
 b'debugsuccessorssets',
 [(b'', b'closest', False, _(b'return closest successors sets only'))],



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: Getting the revisions of .hgtags programatically

[Joerg Sonnenberger]

On Sun, Jan 24, 2021 at 10:59:30PM +, Roy Marples wrote:
> On 24/01/2021 22:50, Joerg Sonnenberger wrote:
> > On Sun, Jan 24, 2021 at 10:16:46PM +, Roy Marples wrote:
> > > Hi Joerg
> > > 
> > > On 24/01/2021 22:03, Joerg Sonnenberger wrote:
> > > > Hello Roy,
> > > > isn't repo.tagslist() (and maybe filtering out local tags) already doing
> > > > what you want?
> > > 
> > > It is sort of. I use it to get a list of tag and yes, I filter out non
> > > globals already.
> > > However, it only returns the revision tagged with the name, not which
> > > revision created the tag.
> > 
> > Why do you care?
> 
> Because fast-import requires the author and date to create the equivalent tag.
> This could be different from the author and date of the commit tagged and if
> any commit message is given for the tag, that would likely be different
> also.
> 
> Currently to get by, I create a light-weight tag instead, but this lacks any
> author or date information and unlike the fast-import tag command is not
> versioned in the repository like our global tags.
> 
> But more importantly, why do I care? Because the information that should be
> exportable is then lost and I have zero tolerance for lost data.

Well, the point of the question is that it is a not very well defined
property. I mean just look at the rules to decide which value is
visible for conflicting defitions. tags._updatetags is where the merge
rules are defined. The relationship to the commit graph is
...interesting at best. So, no, I don't think "tagger" and "tag time"
are properties I would naturally associate with a tag.

Joerg
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: Getting the revisions of .hgtags programatically

[Joerg Sonnenberger]

On Sun, Jan 24, 2021 at 10:16:46PM +, Roy Marples wrote:
> Hi Joerg
> 
> On 24/01/2021 22:03, Joerg Sonnenberger wrote:
> > Hello Roy,
> > isn't repo.tagslist() (and maybe filtering out local tags) already doing
> > what you want?
> 
> It is sort of. I use it to get a list of tag and yes, I filter out non
> globals already.
> However, it only returns the revision tagged with the name, not which
> revision created the tag.

Why do you care?

Joerg
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: Getting the revisions of .hgtags programatically

[Joerg Sonnenberger]

Hello Roy,
isn't repo.tagslist() (and maybe filtering out local tags) already doing
what you want?

Joerg

On Sat, Jan 23, 2021 at 01:56:52AM +, Roy Marples via Mercurial-devel wrote:
> Hi List
> 
> I'm working on added tag support to the fastexport extension.
> I have little idea about the internals of mercurial so asking here for 
> guidance.
> 
> My idea is to walk the revisions of .hgtags backwards until I find a
> changeset which added the tag name.
> Once found I can export the user, date and message into the fastexport.
> 
> What is the best way of achieving this please?
> I am currently looking at the annotate code as that gets what I want, but
> it's slow going working it all out and would appreciate somem guidance if
> possible.
> 
> Please include me directly in replies as I'm not subbed to this list.
> Thanks
> 
> Roy
> ___
> Mercurial-devel mailing list
> Mercurial-devel@mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D9847: revlog: decouple caching from addrevision callback for addgroup

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: indygreg.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  For changesets, it is useful to cache the content as it will almost
  always be processed afterwards. For manifests on the other hand, the
  content is often not used directly as there is a fast path for deltas.
  Explicitly disable the cache in exchangev2's manifest handling for that
  reason.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9847

AFFECTED FILES
  mercurial/changegroup.py
  mercurial/exchangev2.py
  mercurial/interfaces/repository.py
  mercurial/manifest.py
  mercurial/revlog.py
  mercurial/unionrepo.py

CHANGE DETAILS

diff --git a/mercurial/unionrepo.py b/mercurial/unionrepo.py
--- a/mercurial/unionrepo.py
+++ b/mercurial/unionrepo.py
@@ -128,6 +128,7 @@
 deltas,
 linkmapper,
 transaction,
+alwayscache=False,
 addrevisioncb=None,
 duplicaterevisioncb=None,
 maybemissingparents=False,
diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -2375,6 +2375,7 @@
 deltas,
 linkmapper,
 transaction,
+alwayscache=False,
 addrevisioncb=None,
 duplicaterevisioncb=None,
 ):
@@ -2475,7 +2476,7 @@
 (baserev, delta),
 ifh,
 dfh,
-alwayscache=bool(addrevisioncb),
+alwayscache=alwayscache,
 deltacomputer=deltacomputer,
 )
 
diff --git a/mercurial/manifest.py b/mercurial/manifest.py
--- a/mercurial/manifest.py
+++ b/mercurial/manifest.py
@@ -1836,6 +1836,7 @@
 deltas,
 linkmapper,
 transaction,
+alwayscache=False,
 addrevisioncb=None,
 duplicaterevisioncb=None,
 ):
@@ -1843,6 +1844,7 @@
 deltas,
 linkmapper,
 transaction,
+alwayscache=alwayscache,
 addrevisioncb=addrevisioncb,
 duplicaterevisioncb=duplicaterevisioncb,
 )
diff --git a/mercurial/interfaces/repository.py 
b/mercurial/interfaces/repository.py
--- a/mercurial/interfaces/repository.py
+++ b/mercurial/interfaces/repository.py
@@ -769,7 +769,13 @@
 ``nullid``, in which case the header from the delta can be ignored
 and the delta used as the fulltext.
 
+``alwayscache`` instructs the lower layers to cache the content of the
+newly added revision, even if it needs to be explicitly computed.
+This used to be the default when ``addrevisioncb`` was provided up to
+Mercurial 5.7.
+
 ``addrevisioncb`` should be called for each node as it is committed.
+``duplicaterevisioncb`` should be called for each pre-existing node.
 
 ``maybemissingparents`` is a bool indicating whether the incoming
 data may reference parents/ancestor revisions that aren't present.
diff --git a/mercurial/exchangev2.py b/mercurial/exchangev2.py
--- a/mercurial/exchangev2.py
+++ b/mercurial/exchangev2.py
@@ -423,6 +423,7 @@
 iterrevisions(),
 linkrev,
 weakref.proxy(tr),
+alwayscache=True,
 addrevisioncb=onchangeset,
 duplicaterevisioncb=ondupchangeset,
 )
diff --git a/mercurial/changegroup.py b/mercurial/changegroup.py
--- a/mercurial/changegroup.py
+++ b/mercurial/changegroup.py
@@ -334,6 +334,7 @@
 deltas,
 csmap,
 trp,
+alwayscache=True,
 addrevisioncb=onchangelog,
 duplicaterevisioncb=ondupchangelog,
 ):



To: joerg.sonnenberger, indygreg, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D9842: tests: deal with more timing differences in output

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9842

AFFECTED FILES
  tests/test-wireproto-exchangev2-shallow.t

CHANGE DETAILS

diff --git a/tests/test-wireproto-exchangev2-shallow.t 
b/tests/test-wireproto-exchangev2-shallow.t
--- a/tests/test-wireproto-exchangev2-shallow.t
+++ b/tests/test-wireproto-exchangev2-shallow.t
@@ -100,10 +100,15 @@
   received frame(size=1170; request=1; stream=2; streamflags=encoded; 
type=command-response; flags=continuation)
   received frame(size=0; request=1; stream=2; streamflags=; 
type=command-response; flags=eos) (?)
   add changeset 3390ef850073
+  received frame(size=0; request=1; stream=2; streamflags=; 
type=command-response; flags=eos) (?)
   add changeset b709380892b1
+  received frame(size=0; request=1; stream=2; streamflags=; 
type=command-response; flags=eos) (?)
   add changeset 47fe012ab237
+  received frame(size=0; request=1; stream=2; streamflags=; 
type=command-response; flags=eos) (?)
   add changeset 97765fc3cd62
+  received frame(size=0; request=1; stream=2; streamflags=; 
type=command-response; flags=eos) (?)
   add changeset dc666cf9ecf3
+  received frame(size=0; request=1; stream=2; streamflags=; 
type=command-response; flags=eos) (?)
   add changeset 93a8bd067ed2
   received frame(size=0; request=1; stream=2; streamflags=; 
type=command-response; flags=eos) (?)
   checking for updated bookmarks
@@ -269,10 +274,15 @@
   received frame(size=1170; request=1; stream=2; streamflags=encoded; 
type=command-response; flags=continuation)
   received frame(size=0; request=1; stream=2; streamflags=; 
type=command-response; flags=eos) (?)
   add changeset 3390ef850073
+  received frame(size=0; request=1; stream=2; streamflags=; 
type=command-response; flags=eos) (?)
   add changeset b709380892b1
+  received frame(size=0; request=1; stream=2; streamflags=; 
type=command-response; flags=eos) (?)
   add changeset 47fe012ab237
+  received frame(size=0; request=1; stream=2; streamflags=; 
type=command-response; flags=eos) (?)
   add changeset 97765fc3cd62
+  received frame(size=0; request=1; stream=2; streamflags=; 
type=command-response; flags=eos) (?)
   add changeset dc666cf9ecf3
+  received frame(size=0; request=1; stream=2; streamflags=; 
type=command-response; flags=eos) (?)
   add changeset 93a8bd067ed2
   received frame(size=0; request=1; stream=2; streamflags=; 
type=command-response; flags=eos) (?)
   checking for updated bookmarks
@@ -407,8 +417,11 @@
   received frame(size=783; request=1; stream=2; streamflags=encoded; 
type=command-response; flags=continuation)
   received frame(size=0; request=1; stream=2; streamflags=; 
type=command-response; flags=eos) (?)
   add changeset 3390ef850073
+  received frame(size=0; request=1; stream=2; streamflags=; 
type=command-response; flags=eos) (?)
   add changeset b709380892b1
+  received frame(size=0; request=1; stream=2; streamflags=; 
type=command-response; flags=eos) (?)
   add changeset 47fe012ab237
+  received frame(size=0; request=1; stream=2; streamflags=; 
type=command-response; flags=eos) (?)
   add changeset 97765fc3cd62
   received frame(size=0; request=1; stream=2; streamflags=; 
type=command-response; flags=eos) (?)
   checking for updated bookmarks
@@ -522,6 +535,7 @@
   received frame(size=400; request=1; stream=2; streamflags=encoded; 
type=command-response; flags=continuation)
   received frame(size=0; request=1; stream=2; streamflags=; 
type=command-response; flags=eos) (?)
   add changeset dc666cf9ecf3
+  received frame(size=0; request=1; stream=2; streamflags=; 
type=command-response; flags=eos) (?)
   add changeset 93a8bd067ed2
   received frame(size=0; request=1; stream=2; streamflags=; 
type=command-response; flags=eos) (?)
   checking for updated bookmarks



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D9831: exchangev2: avoid second look-up by node

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  Accessing the revlog by node is slightly more expensive than by
  revision, so look up the revision first and use it afterwards.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9831

AFFECTED FILES
  mercurial/exchangev2.py

CHANGE DETAILS

diff --git a/mercurial/exchangev2.py b/mercurial/exchangev2.py
--- a/mercurial/exchangev2.py
+++ b/mercurial/exchangev2.py
@@ -364,12 +364,13 @@
 def onchangeset(cl, node):
 progress.increment()
 
-revision = cl.changelogrevision(node)
+rev = cl.rev(node)
+revision = cl.changelogrevision(rev)
 added.append(node)
 
 # We need to preserve the mapping of changelog revision to node
 # so we can set the linkrev accordingly when manifests are added.
-manifestnodes[cl.rev(node)] = revision.manifest
+manifestnodes[rev] = revision.manifest
 
 nodesbyphase = {phase: set() for phase in phases.phasenames.values()}
 remotebookmarks = {}



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D9830: commit: look-up new revision once

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  Look-up by node is slightly more expensive, so since it is necessary
  more than once, do it explicitly.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9830

AFFECTED FILES
  mercurial/commit.py

CHANGE DETAILS

diff --git a/mercurial/commit.py b/mercurial/commit.py
--- a/mercurial/commit.py
+++ b/mercurial/commit.py
@@ -96,6 +96,7 @@
 ctx.date(),
 extra,
 )
+rev = repo[n].rev()
 xp1, xp2 = p1.hex(), p2 and p2.hex() or b''
 repo.hook(
 b'pretxncommit',
@@ -108,7 +109,7 @@
 targetphase = subrepoutil.newcommitphase(repo.ui, ctx)
 
 # prevent unmarking changesets as public on recommit
-waspublic = oldtip == repo.changelog.tiprev() and not repo[n].phase()
+waspublic = oldtip == repo.changelog.tiprev() and not repo[rev].phase()
 
 if targetphase and not waspublic:
 # retract boundary do not alter parent changeset.
@@ -116,7 +117,7 @@
 # be compliant anyway
 #
 # if minimal phase was 0 we don't need to retract anything
-phases.registernew(repo, tr, targetphase, [repo[n].rev()])
+phases.registernew(repo, tr, targetphase, [rev])
 return n
 
 



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D9821: sqlitestore: disable test with chg

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  There are known issues with transactions not being closed in a timely
  fashion, making the test flakey.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9821

AFFECTED FILES
  tests/test-sqlitestore.t

CHANGE DETAILS

diff --git a/tests/test-sqlitestore.t b/tests/test-sqlitestore.t
--- a/tests/test-sqlitestore.t
+++ b/tests/test-sqlitestore.t
@@ -1,4 +1,4 @@
-#require sqlite
+#require sqlite no-chg
 
   $ cat >> $HGRCPATH < [extensions]



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D9782: localrepo: fix comment typo

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9782

AFFECTED FILES
  mercurial/localrepo.py

CHANGE DETAILS

diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -2618,7 +2618,7 @@
 return
 
 if tr is None or tr.changes[b'origrepolen'] < len(self):
-# accessing the 'ser ved' branchmap should refresh all the others,
+# accessing the 'served' branchmap should refresh all the others,
 self.ui.debug(b'updating the branch cache\n')
 self.filtered(b'served').branchmap()
 self.filtered(b'served.hidden').branchmap()



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D9781: branchmap: update rev-branch-cache incrementally

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  Historically, the revision to branch mapping cache was updated on demand
  and shared via bundle2 to avoid the cost of rebuilding on first use.
  
  Use the new `register_changeset` callback and update rbc directly on
  every change. Make the transfer of the bundle part redundant, but keep
  it for the moment to avoid the test churn.
  
  Over all, "hg unbundle" for large bundles is less than 1.8% slower for
  different larger repositories and that seems to a reasonable trade off.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9781

AFFECTED FILES
  mercurial/branchmap.py
  mercurial/bundle2.py
  mercurial/localrepo.py
  relnotes/next
  tests/test-acl.t
  tests/test-inherit-mode.t
  tests/test-rebase-conflicts.t

CHANGE DETAILS

diff --git a/tests/test-rebase-conflicts.t b/tests/test-rebase-conflicts.t
--- a/tests/test-rebase-conflicts.t
+++ b/tests/test-rebase-conflicts.t
@@ -318,10 +318,10 @@
   bundle2-input-part: total payload size 1686
   bundle2-input-part: "cache:rev-branch-cache" (advisory) supported
   bundle2-input-part: total payload size 74
-  truncating cache/rbc-revs-v1 to 56
   bundle2-input-part: "phase-heads" supported
   bundle2-input-part: total payload size 24
   bundle2-input-bundle: 3 parts total
+  truncating cache/rbc-revs-v1 to 72
   added 2 changesets with 2 changes to 1 files
   updating the branch cache
   invalid branch cache (served): tip differs
diff --git a/tests/test-inherit-mode.t b/tests/test-inherit-mode.t
--- a/tests/test-inherit-mode.t
+++ b/tests/test-inherit-mode.t
@@ -134,6 +134,8 @@
   00660 ../push/.hg/00changelog.i
   00770 ../push/.hg/cache/
   00660 ../push/.hg/cache/branch2-base
+  00660 ../push/.hg/cache/rbc-names-v1
+  00660 ../push/.hg/cache/rbc-revs-v1
   00660 ../push/.hg/dirstate
   00660 ../push/.hg/requires
   00770 ../push/.hg/store/
diff --git a/tests/test-acl.t b/tests/test-acl.t
--- a/tests/test-acl.t
+++ b/tests/test-acl.t
@@ -204,6 +204,7 @@
   bundle2-input-part: "phase-heads" supported
   bundle2-input-part: total payload size 24
   bundle2-input-bundle: 5 parts total
+  truncating cache/rbc-revs-v1 to 8
   updating the branch cache
   added 3 changesets with 3 changes to 3 files
   bundle2-output-bundle: "HG20", 1 parts total
@@ -283,6 +284,7 @@
   bundle2-input-part: "phase-heads" supported
   bundle2-input-part: total payload size 24
   bundle2-input-bundle: 5 parts total
+  truncating cache/rbc-revs-v1 to 8
   updating the branch cache
   added 3 changesets with 3 changes to 3 files
   bundle2-output-bundle: "HG20", 1 parts total
@@ -806,6 +808,7 @@
   acl: acl.deny.bookmarks not enabled
   acl: bookmark access granted: "ef1ea85a6374b77d6da9dcda9541f498f2d17df7" on 
bookmark "moving-bookmark"
   bundle2-input-bundle: 7 parts total
+  truncating cache/rbc-revs-v1 to 8
   updating the branch cache
   invalid branch cache (served.hidden): tip differs
   added 1 changesets with 1 changes to 1 files
@@ -982,6 +985,7 @@
   bundle2-input-part: "phase-heads" supported
   bundle2-input-part: total payload size 24
   bundle2-input-bundle: 5 parts total
+  truncating cache/rbc-revs-v1 to 8
   updating the branch cache
   added 3 changesets with 3 changes to 3 files
   bundle2-output-bundle: "HG20", 1 parts total
@@ -1318,6 +1322,7 @@
   bundle2-input-part: "phase-heads" supported
   bundle2-input-part: total payload size 24
   bundle2-input-bundle: 5 parts total
+  truncating cache/rbc-revs-v1 to 8
   updating the branch cache
   added 3 changesets with 3 changes to 3 files
   bundle2-output-bundle: "HG20", 1 parts total
@@ -1408,6 +1413,7 @@
   bundle2-input-part: "phase-heads" supported
   bundle2-input-part: total payload size 24
   bundle2-input-bundle: 5 parts total
+  truncating cache/rbc-revs-v1 to 8
   updating the branch cache
   added 3 changesets with 3 changes to 3 files
   bundle2-output-bundle: "HG20", 1 parts total
@@ -1577,6 +1583,7 @@
   bundle2-input-part: "phase-heads" supported
   bundle2-input-part: total payload size 24
   bundle2-input-bundle: 5 parts total
+  truncating cache/rbc-revs-v1 to 8
   updating the branch cache
   added 3 changesets with 3 changes to 3 files
   bundle2-output-bundle: "HG20", 1 parts total
diff --git a/relnotes/next b/relnotes/next
--- a/relnotes/next
+++ b/relnotes/next
@@ -42,6 +42,9 @@
  * The `branchmap` cache is updated more intelligently and can be
significantly faster for repositories with many branches and changesets.
 
+ * The `rev-branch-cache` is now updated incrementally whenever changesets
+   are added.
+
 
 == New Experimental Features ==
 
diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -1990,7 +1990,7 @@
 return self._revbranchcache
 
 def register_changeset(self, node, changelogrevision):
-pass

D9780: repository: introduce register_changeset callback

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  The new callback is called whenever a changeset is added to the repository
  (commit, unbundle or exchange). Since the bulk operations already parse
  the changeset (readfiles or full changesetrevision), always use the
  latter to avoid redundant lookups. The first consumer of the new
  interface needs to look at extra.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9780

AFFECTED FILES
  mercurial/changegroup.py
  mercurial/commit.py
  mercurial/exchangev2.py
  mercurial/interfaces/repository.py
  mercurial/localrepo.py

CHANGE DETAILS

diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -1989,6 +1989,9 @@
 self._revbranchcache = branchmap.revbranchcache(self.unfiltered())
 return self._revbranchcache
 
+def register_changeset(self, node, changelogrevision):
+pass
+
 def branchtip(self, branch, ignoremissing=False):
 """return the tip node for a given branch
 
diff --git a/mercurial/interfaces/repository.py 
b/mercurial/interfaces/repository.py
--- a/mercurial/interfaces/repository.py
+++ b/mercurial/interfaces/repository.py
@@ -1641,6 +1641,13 @@
 def revbranchcache():
 pass
 
+def register_changeset(node, changelogrevision):
+"""Extension point for caches for new nodes.
+
+The changelogrevision object is provided as optimisation to
+avoid duplicate lookups."""
+pass
+
 def branchtip(branchtip, ignoremissing=False):
 """Return the tip node for a given branch."""
 
diff --git a/mercurial/exchangev2.py b/mercurial/exchangev2.py
--- a/mercurial/exchangev2.py
+++ b/mercurial/exchangev2.py
@@ -371,6 +371,8 @@
 # so we can set the linkrev accordingly when manifests are added.
 manifestnodes[cl.rev(node)] = revision.manifest
 
+repo.register_changeset(node, revision)
+
 nodesbyphase = {phase: set() for phase in phases.phasenames.values()}
 remotebookmarks = {}
 
diff --git a/mercurial/commit.py b/mercurial/commit.py
--- a/mercurial/commit.py
+++ b/mercurial/commit.py
@@ -96,6 +96,8 @@
 ctx.date(),
 extra,
 )
+repo.register_changeset(n, repo.changelog.changelogrevision(n))
+
 xp1, xp2 = p1.hex(), p2 and p2.hex() or b''
 repo.hook(
 b'pretxncommit',
diff --git a/mercurial/changegroup.py b/mercurial/changegroup.py
--- a/mercurial/changegroup.py
+++ b/mercurial/changegroup.py
@@ -323,7 +323,9 @@
 cgnodes.append(node)
 
 def onchangelog(cl, node):
-efilesset.update(cl.readfiles(node))
+ctx = cl.changelogrevision(node)
+efilesset.update(ctx.files)
+repo.register_changeset(node, ctx)
 
 self.changelogheader()
 deltas = self.deltaiter()



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D9779: changelog: move branchinfo to changelogrevision

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  The function parses the extra dictionary after looking up the
  changelogrevision. To avoid duplicated look up, it is better to provide
  it as property of changelogrevision instead. Keep the function for a
  release cycle as at least the topic extension depends on it.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9779

AFFECTED FILES
  mercurial/changelog.py
  relnotes/next

CHANGE DETAILS

diff --git a/relnotes/next b/relnotes/next
--- a/relnotes/next
+++ b/relnotes/next
@@ -63,4 +63,5 @@
 
 == Internal API Changes ==
 
-
+ * `changelog.branchinfo` is deprecated and will be removed after 5.7.
+   It is superseded by `changelogrevision.branchinfo`.
diff --git a/mercurial/changelog.py b/mercurial/changelog.py
--- a/mercurial/changelog.py
+++ b/mercurial/changelog.py
@@ -200,6 +200,7 @@
 p1copies = attr.ib(default=None)
 p2copies = attr.ib(default=None)
 description = attr.ib(default=b'')
+branchinfo = attr.ib(default=(_defaultextra[b'branch'], False))
 
 
 class changelogrevision(object):
@@ -372,6 +373,11 @@
 def description(self):
 return encoding.tolocal(self._text[self._offsets[3] + 2 :])
 
+@property
+def branchinfo(self):
+extra = self.extra
+return encoding.tolocal(extra.get(b"branch")), b'close' in extra
+
 
 class changelog(revlog.revlog):
 def __init__(self, opener, trypending=False):
@@ -601,8 +607,7 @@
 
 This function exists because creating a changectx object
 just to access this is costly."""
-extra = self.changelogrevision(rev).extra
-return encoding.tolocal(extra.get(b"branch")), b'close' in extra
+return self.changelogrevision(rev).branchinfo
 
 def _nodeduplicatecallback(self, transaction, node):
 # keep track of revisions that got "re-added", eg: unbunde of know rev.



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D9778: reverse-branch-cache: switch to doubling allocating scheme

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  In preperation for updating the reverse-branch-cache incrementally
  whenever a new changeset comes in, avoid bad performance on resize with
  Python 3.7 (and likely other 3.x versions).

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9778

AFFECTED FILES
  mercurial/branchmap.py

CHANGE DETAILS

diff --git a/mercurial/branchmap.py b/mercurial/branchmap.py
--- a/mercurial/branchmap.py
+++ b/mercurial/branchmap.py
@@ -566,6 +566,7 @@
 # [4 byte hash prefix][4 byte branch name number with sign bit indicating open]
 _rbcrecfmt = b'>4sI'
 _rbcrecsize = calcsize(_rbcrecfmt)
+_rbcmininc = 64 * _rbcrecsize
 _rbcnodelen = 4
 _rbcbranchidxmask = 0x7FFF
 _rbccloseflag = 0x8000
@@ -730,11 +731,15 @@
 if rev == nullrev:
 return
 rbcrevidx = rev * _rbcrecsize
-if len(self._rbcrevs) < rbcrevidx + _rbcrecsize:
-self._rbcrevs.extend(
-b'\0'
-* (len(self._repo.changelog) * _rbcrecsize - 
len(self._rbcrevs))
-)
+requiredsize = rbcrevidx + _rbcrecsize
+rbccur = len(self._rbcrevs)
+if rbccur < requiredsize:
+# bytearray doesn't allocate extra space at least in Python 3.7.
+# When multiple changesets are added in a row, precise resize would
+# result in quadratic complexity. Overallocate to compensate by
+# use the classic doubling technique for dynamic arrays instead.
+# If there was a gap in the map before, less space will be 
reserved.
+self._rbcrevs.extend(b'\0' * max(_rbcmininc, requiredsize))
 pack_into(_rbcrecfmt, self._rbcrevs, rbcrevidx, node, branchidx)
 self._rbcrevslen = min(self._rbcrevslen, rev)
 



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D9750: node: introduce nodeconstants class

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: indygreg.
Herald added a reviewer: durin42.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.
joerg.sonnenberger added a comment.


  This is the API changing part of D9465  
without all the followup changes.

REVISION SUMMARY
  In preparing for moving from SHA1 hashes to a modern hash function,
  place nullid and other constant magic vules in a class. Provide the
  active set of constants in the repository and push it down. Provide
  nullid directly in strategic places like the repository as it is
  accessed very often. This changeset introduces the API change, but not
  the mechanical replacement of the node.py attributes itself.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9750

AFFECTED FILES
  contrib/perf.py
  hgext/absorb.py
  hgext/git/gitlog.py
  hgext/largefiles/lfutil.py
  hgext/sqlitestore.py
  mercurial/bookmarks.py
  mercurial/branchmap.py
  mercurial/bundle2.py
  mercurial/bundlerepo.py
  mercurial/changegroup.py
  mercurial/changelog.py
  mercurial/dirstate.py
  mercurial/discovery.py
  mercurial/exchange.py
  mercurial/filelog.py
  mercurial/interfaces/dirstate.py
  mercurial/interfaces/repository.py
  mercurial/localrepo.py
  mercurial/manifest.py
  mercurial/node.py
  mercurial/obsolete.py
  mercurial/revlog.py
  mercurial/statichttprepo.py
  mercurial/store.py
  mercurial/unionrepo.py
  mercurial/upgrade_utils/engine.py
  relnotes/next
  tests/simplestorerepo.py
  tests/test-check-interfaces.py
  tests/test-manifest.py

CHANGE DETAILS

diff --git a/tests/test-manifest.py b/tests/test-manifest.py
--- a/tests/test-manifest.py
+++ b/tests/test-manifest.py
@@ -6,6 +6,8 @@
 import unittest
 import zlib
 
+from mercurial.node import sha1nodeconstants
+
 from mercurial import (
 manifest as manifestmod,
 match as matchmod,
@@ -436,7 +438,7 @@
 
 class testtreemanifest(unittest.TestCase, basemanifesttests):
 def parsemanifest(self, text):
-return manifestmod.treemanifest(b'', text)
+return manifestmod.treemanifest(sha1nodeconstants, b'', text)
 
 def testWalkSubtrees(self):
 m = self.parsemanifest(A_DEEPER_MANIFEST)
diff --git a/tests/test-check-interfaces.py b/tests/test-check-interfaces.py
--- a/tests/test-check-interfaces.py
+++ b/tests/test-check-interfaces.py
@@ -243,7 +243,10 @@
 
 # Conforms to imanifestlog.
 ml = manifest.manifestlog(
-vfs, repo, manifest.manifestrevlog(repo.svfs), repo.narrowmatch()
+vfs,
+repo,
+manifest.manifestrevlog(repo.nodeconstants, repo.svfs),
+repo.narrowmatch(),
 )
 checkzobject(ml)
 checkzobject(repo.manifestlog)
@@ -258,7 +261,7 @@
 # Conforms to imanifestdict.
 checkzobject(mctx.read())
 
-mrl = manifest.manifestrevlog(vfs)
+mrl = manifest.manifestrevlog(repo.nodeconstants, vfs)
 checkzobject(mrl)
 
 ziverify.verifyClass(repository.irevisiondelta, revlog.revlogrevisiondelta)
diff --git a/tests/simplestorerepo.py b/tests/simplestorerepo.py
--- a/tests/simplestorerepo.py
+++ b/tests/simplestorerepo.py
@@ -106,7 +106,9 @@
 
 _flagserrorclass = simplestoreerror
 
-def __init__(self, svfs, path):
+def __init__(self, repo, svfs, path):
+self.nullid = repo.nullid
+self._repo = repo
 self._svfs = svfs
 self._path = path
 
@@ -687,7 +689,7 @@
 
 class simplestorerepo(repo.__class__):
 def file(self, f):
-return filestorage(self.svfs, f)
+return filestorage(repo, self.svfs, f)
 
 repo.__class__ = simplestorerepo
 
diff --git a/relnotes/next b/relnotes/next
--- a/relnotes/next
+++ b/relnotes/next
@@ -57,4 +57,6 @@
 
 == Internal API Changes ==
 
-
+ * `nodes.nullid` and related constants are being phased out as part of
+   the deprecation of SHA1. Repository instances and related classes
+   provide access via `nodeconstants` and in some cases `nullid` attributes.
diff --git a/mercurial/upgrade_utils/engine.py 
b/mercurial/upgrade_utils/engine.py
--- a/mercurial/upgrade_utils/engine.py
+++ b/mercurial/upgrade_utils/engine.py
@@ -35,7 +35,9 @@
 return changelog.changelog(repo.svfs)
 elif path.endswith(b'00manifest.i'):
 mandir = path[: -len(b'00manifest.i')]
-return manifest.manifestrevlog(repo.svfs, tree=mandir)
+return manifest.manifestrevlog(
+repo.nodeconstants, repo.svfs, tree=mandir
+)
 else:
 # reverse of "/".join(("data", path + ".i"))
 return filelog.filelog(repo.svfs, path[5:-2])
diff --git a/mercurial/unionrepo.py b/mercurial/unionrepo.py
--- a/mercurial/unionrepo.py
+++ b/mercurial/unionrepo.py
@@ -152,9 +152,9 @@
 
 
 class unionmanifest(unionrevlog, manifest.manifestrevlog):
-def __init__(self, opener, opener2, linkmapper):
-manifest.manifestrevlog.__init__(self, opener)
-   

[RFC] Perfect hash table plan

[Joerg Sonnenberger]

Hello all,
I've been looking at different indexing methods for use in different
parts of store and cache. One technique that I had part of the code base
around already is the generation of perfect hash tables. The plan can be
found in the Wiki:

https://www.mercurial-scm.org/wiki/PhashPlan

The proof of concept doesn't include the generator as I've been too lazy
so far to port that code, but a small x86_64 Linux binary is included
with the Python glue around it for testing. The test case is the exact
matching for nodes to revision and I've outlined how the prefix matching
could be implemented as well.

Joerg
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: New revlog format, plan page

[Joerg Sonnenberger]

On Mon, Jan 11, 2021 at 01:12:30PM +0100, Pierre-Yves David wrote:
> (1) Some of the current cache we have would fit well in such index
> * The hgtagsfnodes cache: taking 4 bytes to cache the `.hgtags` revision
> number associated with a changelog revisions. (This will requires some
> bookkeeping while adding/stripping),
> * the `rbc-revs-v1`: using an integer (4bytes) and an external list to store
> the branch on which each revision is,
> * (probably another 4 bytes to store the sub-branch/topic,)

I'd be reluctant to move them into the revlog. If anything, it would
call for a more variant friendly format specification. Ultimately, we
should figure out first how "hot" the various caches are before dedicing
to tie them tighter to changelog. Also, at the very least in the case of
rbc-revs-v1, it would also prevent some useful optimisations. When we
sort out the cache invalidation story, having a strict linear mapping of
32bit entries would make queries for all revisions of a given branch
easier than if it is part of a more complex data structure.

> (2) Some cache key mechanism. Right now a lot of cache validate their
> content using a (tip-rev, tip-node) pair. That pair is fragile as it does
> not garantee that the content before the tip is the same. Having "some"
> bytes that gather some kind of accumulated value from the previously added
> nodes. It does not have to be too many bytes, as the (tip-node, tip-rev,
> cache-key) should be good enough. We can probably build it using a series of
> shift and xor of the hash we are adding.

See my mail from Dec, 14th. Having done a few more things in the mean
time, I'd add phases and obslog as cache keys on top and that's
something we don't handle well right now at all. At that point the
current invalidation strategy just becomes way too fragile.

Joerg
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: New revlog format, plan page

[Joerg Sonnenberger]

On Thu, Jan 07, 2021 at 12:04:06PM -0500, Josef 'Jeff' Sipek wrote:
> On Tue, Jan 05, 2021 at 19:33:36 +0100, Joerg Sonnenberger wrote:
> > On Tue, Jan 05, 2021 at 04:38:20PM +0100, Raphaël Gomès wrote:
> > > I've opened a very much draft plan page [1] to try to list all the things 
> > > we
> > > want to do in that version and try to figure out an efficient new format.
> > 
> > "No support for hash version"
> > 
> > I don't think that points really matters. The plan for the hash
> > migration allows them in theory to coexist fully on the revlog layer and
> > the main problems for mixing them are on the changeset/manifest layer
> > anyway. That is, any migration strategy will IMO rewrite all revlogs to
> > the newer hash anyway and only keep a secondary index for changesets and
> > maybe manifests.
> 
> At the same time, I think it is sensible (and very useful when looking an a
> revlog without repo-level info) for revlogs to identify which hash they
> contain.  Either in some sort of revlog header or in each entry (if hash can
> vary between entries).

I plan the replacement hash to be tagged, so yes, they can be
individually distinguished. 

> 
> > "No support for sidedata"
> > 
> > My big design level concern is that revlog ATM is optimized for fast
> > integer indexing and append-only storage.
> 
> This is an interesting point.  What *are* the most common revlog operations?
> It probably varies between repos, but I suspect that they are mostly reads
> rather than writes.  As a consequence, a good revlog format would optimize
> for the common case (without making the less common cases completely suck).

The problem is that anything that needs inplace writes is a lot more
difficult to get right for on-disk consistency and for concurrent
read-access. Normal revision data does not change, by design. That's
quite different from any unversioned metadata. This can include
signatures for example, it could include obsolescence data etc.
Separating mutable and immutable data is a natural design choice.

> hg already makes use of CBOR, so it'd be reasonable to use here - either for
> the whole entry or just for parts of it.  For example, CBOR's interegers are
> encoded as 1 byte type, followed by 0, 1, 2, 4, or 8 byte integer.  Smaller
> values use less space.  For example, values less than 2^32 use 1-5 bytes.

Needing a separate index from the index for efficient access would
defeat the point of revlog being an index format in first place...

Joerg
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D9689: comments: fix typos

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9689

AFFECTED FILES
  mercurial/repoview.py
  mercurial/revlogutils/sidedata.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/sidedata.py 
b/mercurial/revlogutils/sidedata.py
--- a/mercurial/revlogutils/sidedata.py
+++ b/mercurial/revlogutils/sidedata.py
@@ -13,8 +13,8 @@
 The current implementation is experimental and subject to changes. Do not rely
 on it in production.
 
-Sidedata are stored in the revlog itself, withing the revision rawtext. They
-are inserted, removed from it using the flagprocessors mechanism. The following
+Sidedata are stored in the revlog itself, within the revision rawtext. They
+are inserted and removed from it using the flagprocessors mechanism. The 
following
 format is currently used::
 
 initial header:
@@ -27,7 +27,7 @@
 normal raw text:
 
 
-This is a simple and effective format. It should be enought to experiment with
+This is a simple and effective format. It should be enough to experiment with
 the concept.
 """
 
diff --git a/mercurial/repoview.py b/mercurial/repoview.py
--- a/mercurial/repoview.py
+++ b/mercurial/repoview.py
@@ -159,7 +159,7 @@
 This filter out any mutable changeset and any public changeset that may be
 impacted by something happening to a mutable revision.
 
-This is achieved by filtered everything with a revision number egal or
+This is achieved by filtered everything with a revision number equal or
 higher than the first mutable changeset is filtered."""
 assert not repo.changelog.filteredrevs
 cl = repo.changelog



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D9688: nodemap: match comment to actual code

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  REV_OFFSET constant is 2, not 10.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9688

AFFECTED FILES
  mercurial/revlogutils/nodemap.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/nodemap.py b/mercurial/revlogutils/nodemap.py
--- a/mercurial/revlogutils/nodemap.py
+++ b/mercurial/revlogutils/nodemap.py
@@ -391,7 +391,7 @@
 #
 #  * value >=  0 -> index of sub-block
 #  * value == -1 -> no value
-#  * value <  -1 -> a revision value: rev = -(value+10)
+#  * value <  -1 -> a revision value: rev = -(value+2)
 #
 # The implementation focus on simplicity, not on performance. A Rust
 # implementation should provide a efficient version of the same binary



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: New revlog format, plan page

[Joerg Sonnenberger]

On Thu, Jan 07, 2021 at 01:05:34AM +, Johannes Totz wrote:
> Have we ever addressed the file duplication on hg-mv? .hg/store/data/ will
> end up with lots of duplicated data. That has always been my biggest gripe.

That's the unified revlog issue.

Joerg
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D9687: contrib: py3 compat for perfnodemap

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9687

AFFECTED FILES
  contrib/perf.py

CHANGE DETAILS

diff --git a/contrib/perf.py b/contrib/perf.py
--- a/contrib/perf.py
+++ b/contrib/perf.py
@@ -1627,12 +1627,12 @@
 mercurial.revlog._prereadsize = 2 ** 24  # disable lazy parser in old hg
 
 unfi = repo.unfiltered()
-clearcaches = opts['clear_caches']
+clearcaches = opts[b'clear_caches']
 # find the filecache func directly
 # This avoid polluting the benchmark with the filecache logic
 makecl = unfi.__class__.changelog.func
 if not opts[b'rev']:
-raise error.Abort('use --rev to specify revisions to look up')
+raise error.Abort(b'use --rev to specify revisions to look up')
 revs = scmutil.revrange(repo, opts[b'rev'])
 cl = repo.changelog
 nodes = [cl.node(r) for r in revs]



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: New revlog format, plan page

[Joerg Sonnenberger]

On Tue, Jan 05, 2021 at 04:38:20PM +0100, Raphaël Gomès wrote:
> I've opened a very much draft plan page [1] to try to list all the things we
> want to do in that version and try to figure out an efficient new format.

"No support for hash version"

I don't think that points really matters. The plan for the hash
migration allows them in theory to coexist fully on the revlog layer and
the main problems for mixing them are on the changeset/manifest layer
anyway. That is, any migration strategy will IMO rewrite all revlogs to
the newer hash anyway and only keep a secondary index for changesets and
maybe manifests.

"No support for sidedata"

My big design level concern is that revlog ATM is optimized for fast
integer indexing and append-only storage. At least for some sidedata
use cases I have, that is an ill fit.

"No support for unified revlog"

IMO this should be the driving feature. The biggest issue for me is that
it creates two challenges that didn't exist so far:
(1) Inter-file patches and how they interact with the wire protocol
(2) Identical revisions stored in different places.

"No support for larger files"

Supporting large revlog files is sensible and having a store for
design-challenged file systems might be necessary. Microsoft, I'm
looking at you. Otherwise the concern is space use in the revlog file
and RAM use during operations. I don't think the latter is as big an
issue now as it was 15 years ago, but the former is real. But it might
be a good point in time to just go for 64bit offsets by default...

Joerg
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: Any way to get Mercurial version from inside a Python hook?

[Joerg Sonnenberger]

On Fri, Dec 25, 2020 at 08:04:14PM +0100, Ansis Māliņš wrote:
> I want to obtain the current Mercurial version from inside a Python hook. I
> could just shell execute hg --version, but I thought I'd ask around for a
> cheaper, more direct way. Is there any global function or constant that I
> could use? If it matters, I'm on Mercurial 4.0 (Debian 9).

mercurial.__version__.version?

Joerg
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D9663: largefiles: redo heads interception

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  The largefiles extension replaces the "heads" wire command and tries to
  redirect all uses towards the custom "lheads" wire command. As seen in
  issue6384, this doesn't currently work for ssh. Instead of hooking into
  the _callstream interface, properly register the command for the peer
  instance and monkeypatch the executor to do the redirection. This works
  transparently for both all kinds of peers and both for the batch and
  non-batch case.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9663

AFFECTED FILES
  hgext/largefiles/__init__.py
  hgext/largefiles/proto.py

CHANGE DETAILS

diff --git a/hgext/largefiles/proto.py b/hgext/largefiles/proto.py
--- a/hgext/largefiles/proto.py
+++ b/hgext/largefiles/proto.py
@@ -33,10 +33,6 @@
 
 eh = exthelper.exthelper()
 
-# these will all be replaced by largefiles.uisetup
-ssholdcallstream = None
-httpoldcallstream = None
-
 
 def putlfile(repo, proto, sha):
 """Server command for putting a largefile into a repository's local store
@@ -106,7 +102,27 @@
 
 
 def wirereposetup(ui, repo):
+orig_commandexecutor = repo.commandexecutor
+
 class lfileswirerepository(repo.__class__):
+def commandexecutor(self):
+executor = orig_commandexecutor()
+if self.capable(b'largefiles'):
+orig_callcommand = executor.callcommand
+
+class lfscommandexecutor(executor.__class__):
+def callcommand(self, command, args):
+if command == b'heads':
+command = b'lheads'
+return orig_callcommand(command, args)
+
+executor.__class__ = lfscommandexecutor
+return executor
+
+@wireprotov1peer.batchable
+def lheads(self):
+return self.heads.batchable(self)
+
 def putlfile(self, sha, fd):
 # unfortunately, httprepository._callpush tries to convert its
 # input file-like into a bundle before sending it, so we can't use
@@ -200,22 +216,3 @@
 return wireprototypes.ooberror(LARGEFILES_REQUIRED_MSG)
 
 return orig(repo, proto)
-
-
-def sshrepocallstream(self, cmd, **args):
-if cmd == b'heads' and self.capable(b'largefiles'):
-cmd = b'lheads'
-if cmd == b'batch' and self.capable(b'largefiles'):
-args['cmds'] = args[r'cmds'].replace(b'heads ', b'lheads ')
-return ssholdcallstream(self, cmd, **args)
-
-
-headsre = re.compile(br'(^|;)heads\b')
-
-
-def httprepocallstream(self, cmd, **args):
-if cmd == b'heads' and self.capable(b'largefiles'):
-cmd = b'lheads'
-if cmd == b'batch' and self.capable(b'largefiles'):
-args['cmds'] = headsre.sub(b'lheads', args['cmds'])
-return httpoldcallstream(self, cmd, **args)
diff --git a/hgext/largefiles/__init__.py b/hgext/largefiles/__init__.py
--- a/hgext/largefiles/__init__.py
+++ b/hgext/largefiles/__init__.py
@@ -190,13 +190,6 @@
 )
 # TODO also wrap wireproto.commandsv2 once heads is implemented there.
 
-# can't do this in reposetup because it needs to have happened before
-# wirerepo.__init__ is called
-proto.ssholdcallstream = sshpeer.sshv1peer._callstream
-proto.httpoldcallstream = httppeer.httppeer._callstream
-sshpeer.sshv1peer._callstream = proto.sshrepocallstream
-httppeer.httppeer._callstream = proto.httprepocallstream
-
 # override some extensions' stuff as well
 for name, module in extensions.extensions():
 if name == b'rebase':



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D9662: pycompat: fix typos

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9662

AFFECTED FILES
  mercurial/pycompat.py

CHANGE DETAILS

diff --git a/mercurial/pycompat.py b/mercurial/pycompat.py
--- a/mercurial/pycompat.py
+++ b/mercurial/pycompat.py
@@ -335,7 +335,7 @@
 def strkwargs(dic):
 """
 Converts the keys of a python dictonary to str i.e. unicodes so that
-they can be passed as keyword arguments as dictonaries with bytes keys
+they can be passed as keyword arguments as dictionaries with bytes keys
 can't be passed as keyword arguments to functions on Python 3.
 """
 dic = {k.decode('latin-1'): v for k, v in dic.items()}
@@ -343,7 +343,7 @@
 
 def byteskwargs(dic):
 """
-Converts keys of python dictonaries to bytes as they were converted to
+Converts keys of python dictionaries to bytes as they were converted to
 str to pass that dictonary as a keyword argument on Python 3.
 """
 dic = {k.encode('latin-1'): v for k, v in dic.items()}



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D9661: statichttprepo: explicitly convert error message to str

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  For Python 2.7, the implicit conversion of the HTTPError instance to
  str was good enough. For Python 3.x, this fails later when hitting the
  str to bytes conversion logic.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9661

AFFECTED FILES
  mercurial/statichttprepo.py

CHANGE DETAILS

diff --git a/mercurial/statichttprepo.py b/mercurial/statichttprepo.py
--- a/mercurial/statichttprepo.py
+++ b/mercurial/statichttprepo.py
@@ -61,7 +61,7 @@
 code = f.code
 except urlerr.httperror as inst:
 num = inst.code == 404 and errno.ENOENT or None
-raise IOError(num, inst)
+raise IOError(num, str(inst))
 except urlerr.urlerror as inst:
 raise IOError(None, inst.reason)
 



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D9660: worker: restrict use of worker procesess to the main thread

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is a test workaround for bz6460

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9660

AFFECTED FILES
  mercurial/worker.py

CHANGE DETAILS

diff --git a/mercurial/worker.py b/mercurial/worker.py
--- a/mercurial/worker.py
+++ b/mercurial/worker.py
@@ -67,6 +67,9 @@
 
 if pycompat.ispy3:
 
+def ismainthread():
+return threading.current_thread() == threading.main_thread()
+
 class _blockingreader(object):
 def __init__(self, wrapped):
 self._wrapped = wrapped
@@ -100,6 +103,9 @@
 
 else:
 
+def ismainthread():
+return isinstance(threading.current_thread(), threading._MainThread)
+
 def _blockingreader(wrapped):
 return wrapped
 
@@ -154,7 +160,7 @@
 a thread-based worker. Should be disabled for CPU heavy tasks that don't
 release the GIL.
 """
-enabled = ui.configbool(b'worker', b'enabled')
+enabled = ui.configbool(b'worker', b'enabled') and ismainthread()
 if enabled and worthwhile(ui, costperarg, len(args), 
threadsafe=threadsafe):
 return _platformworker(ui, func, staticargs, args, hasretval)
 return func(*staticargs + (args,))



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D9636: bundle: add option to avoid checking further delta candidates [POC]

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: indygreg.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  Especially when applying a clone bundle and the follow-up pull, the
  deltas will be nearly optimal. Add a test config flag to skip any
  attempts at finding a better delta base. Test base for how this
  interacts with parallel delta compression in addgroup.
  
  Test with a bundle of the HG repo: -2%, non-noticable size difference
  Test with a bundle of the NetBSD src repo: -18%, +10% manifest size

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9636

AFFECTED FILES
  mercurial/changegroup.py
  mercurial/configitems.py
  mercurial/filelog.py
  mercurial/manifest.py
  mercurial/revlog.py
  mercurial/revlogutils/deltas.py
  mercurial/unionrepo.py

CHANGE DETAILS

diff --git a/mercurial/unionrepo.py b/mercurial/unionrepo.py
--- a/mercurial/unionrepo.py
+++ b/mercurial/unionrepo.py
@@ -131,6 +131,7 @@
 addrevisioncb=None,
 duplicaterevisioncb=None,
 maybemissingparents=False,
+trustdeltas=False,
 ):
 raise NotImplementedError
 
diff --git a/mercurial/revlogutils/deltas.py b/mercurial/revlogutils/deltas.py
--- a/mercurial/revlogutils/deltas.py
+++ b/mercurial/revlogutils/deltas.py
@@ -1032,7 +1032,7 @@
 snapshotdepth,
 )
 
-def finddeltainfo(self, revinfo, fh):
+def finddeltainfo(self, revinfo, fh, trustdelta=False):
 """Find an acceptable delta against a candidate revision
 
 revinfo: information about the revision (instance of _revisioninfo)
@@ -1076,8 +1076,12 @@
 if candidatedelta is not None:
 if isgooddeltainfo(self.revlog, candidatedelta, revinfo):
 nominateddeltas.append(candidatedelta)
+if trustdelta and cachedelta[0] != -1:
+break
 if nominateddeltas:
 deltainfo = min(nominateddeltas, key=lambda x: x.deltalen)
+if trustdelta and cachedelta[0] != -1:
+break
 if deltainfo is not None:
 candidaterevs = groups.send(deltainfo.base)
 else:
diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -2244,6 +2244,7 @@
 ifh,
 dfh,
 alwayscache=False,
+trustdelta=False,
 deltacomputer=None,
 ):
 """internal function to add revisions to the log
@@ -2297,7 +2298,9 @@
 
 revinfo = _revisioninfo(node, p1, p2, btext, textlen, cachedelta, 
flags)
 
-deltainfo = deltacomputer.finddeltainfo(revinfo, fh)
+deltainfo = deltacomputer.finddeltainfo(
+revinfo, fh, trustdelta=trustdelta
+)
 
 e = (
 offset_type(offset, flags),
@@ -2367,6 +2370,7 @@
 transaction,
 addrevisioncb=None,
 duplicaterevisioncb=None,
+trustdeltas=False,
 ):
 """
 add a delta group
@@ -2467,6 +2471,7 @@
 dfh,
 alwayscache=bool(addrevisioncb),
 deltacomputer=deltacomputer,
+trustdelta=trustdeltas,
 )
 
 if addrevisioncb:
diff --git a/mercurial/manifest.py b/mercurial/manifest.py
--- a/mercurial/manifest.py
+++ b/mercurial/manifest.py
@@ -1838,6 +1838,7 @@
 transaction,
 addrevisioncb=None,
 duplicaterevisioncb=None,
+trustdeltas=False,
 ):
 return self._revlog.addgroup(
 deltas,
@@ -1845,6 +1846,7 @@
 transaction,
 addrevisioncb=addrevisioncb,
 duplicaterevisioncb=duplicaterevisioncb,
+trustdeltas=trustdeltas,
 )
 
 def rawsize(self, rev):
diff --git a/mercurial/filelog.py b/mercurial/filelog.py
--- a/mercurial/filelog.py
+++ b/mercurial/filelog.py
@@ -141,6 +141,7 @@
 addrevisioncb=None,
 duplicaterevisioncb=None,
 maybemissingparents=False,
+trustdeltas=False,
 ):
 if maybemissingparents:
 raise error.Abort(
@@ -156,6 +157,7 @@
 transaction,
 addrevisioncb=addrevisioncb,
 duplicaterevisioncb=duplicaterevisioncb,
+trustdeltas=trustdeltas,
 )
 
 def getstrippoint(self, minlink):
diff --git a/mercurial/configitems.py b/mercurial/configitems.py
--- a/mercurial/configitems.py
+++ b/mercurial/configitems.py
@@ -1109,6 +1109,11 @@
 )
 coreconfigitem(
 b'experimental',
+b'trust-bundle-deltas',
+default=False,
+)
+coreconfigitem(
+b'experimental',
 b'update.atomic-file',
 default=False,
 )
diff --git a/mercurial/changegroup.py b/mercurial/changegroup.py
--- a/mercurial/changegroup.py
+++ b/mercurial/changegroup.py
@@ -257,7 +257,7 @@
 # be empty 

D9631: branchmap: avoid ancestor computations for absent non-continous branches

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  The branchhead computation is one of the more heavy operations for
  bigger repositories as it has to scan all changesets and potentially
  involves the expensive computation of the ancestor sets. Redo the
  computation to handle the common cases directly and use tighter
  conditions for when the ancestor scan is necessary. Most importantly,
  avoid it completely if the non-continous branches are processed in one
  update as seen in the initial computation after a clone.
  
  For the Mercurial repository, it gives a small 2-3% performance boost.
  For the NetBSD test repository, it cuts the time in half.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9631

AFFECTED FILES
  mercurial/branchmap.py

CHANGE DETAILS

diff --git a/mercurial/branchmap.py b/mercurial/branchmap.py
--- a/mercurial/branchmap.py
+++ b/mercurial/branchmap.py
@@ -443,33 +443,65 @@
 if closesbranch:
 self._closednodes.add(cl.node(r))
 
-# fetch current topological heads to speed up filtering
-topoheads = set(cl.headrevs())
-
 # new tip revision which we found after iterating items from new
 # branches
 ntiprev = self.tiprev
 
-# if older branchheads are reachable from new ones, they aren't
-# really branchheads. Note checking parents is insufficient:
-# 1 (branch a) -> 2 (branch b) -> 3 (branch a)
+# Delay fetching the topological heads until they are needed.
+# A repository without non-continous branches can skip this part.
+topoheads = None
+
+# If a changeset is visible, its parents must be visible too, so
+# use the faster unfiltered parent accessor.
+parentrevs = repo.unfiltered().changelog.parentrevs
+
 for branch, newheadrevs in pycompat.iteritems(newbranches):
+# The set of branchheads is the union of the existing branchheads
+# with the heads of new revisions of that branch, but without
+# existing branchheads that are ancestors of new revisions.
+# The latter condition is necessary for non-continous branches,
+# i.e. 1 (branch a) -> 2 (branch b) -> 3 (branch a).
+#
+# The newrev loop processes all new revisions in order and updates
+# the branchheads for the simple case of continous branches.
+# The sorting ensures that parents are processed first and the root
+# of a potential non-continous branch is seen first.
+# It followes that all revisions that are not a child of the branch
+# are candidates for such branches and therefore kept on the
+# uncertain set. The exception is a local branch root with no
+# pre-existing branchheads. This is the initial start of a branch
+# and safe.
+#
+# If the newrev loop left any uncertain candidates for potential
+# non-continous branches around, further checks are necessary.
+# If all the remaining pre-existing branchheads (i.e. those without
+# a child in the new revision set) are still topological heads,
+# they are automatically also branchheads. Otherwise a full
+# ancestor check is necessary to filter out obsoleted branchheads.
+
 bheads = self._entries.setdefault(branch, [])
 bheadset = {cl.rev(node) for node in bheads}
-
-# This have been tested True on all internal usage of this 
function.
-# run it again in case of doubt
-# assert not (set(bheadrevs) & set(newheadrevs))
-bheadset.update(newheadrevs)
+uncertain = set()
+for newrev in sorted(newheadrevs):
+parents = [p for p in parentrevs(newrev) if p != nullrev]
+gotit = False
+for p in parents:
+if p in bheadset:
+bheadset.remove(p)
+gotit = True
+elif getbranchinfo(p)[0] == branch:
+gotit = True
+if not gotit and bheadset:
+uncertain.add(newrev)
+bheadset.add(newrev)
 
-# This prunes out two kinds of heads - heads that are superseded by
-# a head in newheadrevs, and newheadrevs that are not heads because
-# an existing head is their descendant.
-uncertain = bheadset - topoheads
 if uncertain:
-floorrev = min(uncertain)
-ancestors = set(cl.ancestors(newheadrevs, floorrev))
-bheadset -= ancestors
+if topoheads is None:
+topoheads = set(cl.headrevs())
+if bheadset - topoheads:

D9627: cext: shut-up sign compare warnings

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9627

AFFECTED FILES
  mercurial/cext/revlog.c

CHANGE DETAILS

diff --git a/mercurial/cext/revlog.c b/mercurial/cext/revlog.c
--- a/mercurial/cext/revlog.c
+++ b/mercurial/cext/revlog.c
@@ -2612,7 +2612,7 @@
"data does not support buffer interface");
return -1;
}
-   if (self->nodelen < 20 || self->nodelen > sizeof(nullid)) {
+   if (self->nodelen < 20 || self->nodelen > (Py_ssize_t)sizeof(nullid)) {
PyErr_SetString(PyExc_RuntimeError, "unsupported node size");
return -1;
}



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D9626: branchmap: micro-optimize branchinfo

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  changelogrevision() is supposed to be used if not all data of
  changelog.read is used. This is the case here as only the extra field is
  used. This also improves extensibility as at least hgext.git doesn't
  implement changelog.read.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9626

AFFECTED FILES
  mercurial/changelog.py

CHANGE DETAILS

diff --git a/mercurial/changelog.py b/mercurial/changelog.py
--- a/mercurial/changelog.py
+++ b/mercurial/changelog.py
@@ -601,7 +601,7 @@
 
 This function exists because creating a changectx object
 just to access this is costly."""
-extra = self.read(rev)[5]
+extra = self.changelogrevision(rev).extra
 return encoding.tolocal(extra.get(b"branch")), b'close' in extra
 
 def _nodeduplicatecallback(self, transaction, node):



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[RFC] Interaction between strip and caches

[Joerg Sonnenberger]

Hello all,
while looking at the revbranchcache, I noticed that it is doing quite an
expensive probalistic invalidation dance. It is essentially looking up
the revision in the changelog again and compares the first 32bit to see
if they (still) match. Other caches are doing cheaper checks like
remembering the head revision and node and checking it again to match.
The goal is in all cases to detect one of two cases:

(1) Repository additions by a hg instance without support for the cache.
(2) Repository removals by strip without update support specific to the
cache in use.

The first part is generally handled reasonable well and cheap. Keep
track of the number of revisions and process to all missing changesets
is something code has to support anyway. The real difficult problem is
the second part. I would like us to adopt a more explicit way of dealing
with this and opt-in support via a repository requirement. Given that
the strip command has finally become part of core, it looks like a good
time to do this now.

The first option is to require strip to nuke all caches that it can't
update. This is easy to implement and works reliable by nature with all
existing caches. It is also the more blunt option.

The second option is to keep a journal of strips. This can be a single
monotonically increasing counter and every cache just reads the counter
and rebuilds itself. Alternatively it could be a full journal that lists
the revisions and associated nodes removed. This requires changes to
existing caches but has the advantage that strip can be replayed by the
cache logic to avoid a full rebuild.

Joerg
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D9603: branchmap: refactor revbranchmap and use it as topicmap [PoC]

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9603

AFFECTED FILES
  mercurial/branchmap.py
  mercurial/cacheutil.py
  mercurial/changegroup.py
  mercurial/changelog.py
  mercurial/commit.py
  mercurial/exchangev2.py
  mercurial/interfaces/repository.py
  mercurial/localrepo.py
  tests/test-acl.t
  tests/test-clone-uncompressed.t
  tests/test-clone.t
  tests/test-clonebundles.t
  tests/test-debugcommands.t
  tests/test-fncache.t
  tests/test-hardlinks.t
  tests/test-http-proxy.t
  tests/test-http.t
  tests/test-inherit-mode.t
  tests/test-keyword.t
  tests/test-mq-symlinks.t
  tests/test-push-http.t
  tests/test-rebase-conflicts.t
  tests/test-remote-hidden.t
  tests/test-server-view.t
  tests/test-share.t
  tests/test-ssh.t
  tests/test-stream-bundle-v2.t
  tests/test-tags.t
  tests/test-treemanifest.t

CHANGE DETAILS

diff --git a/tests/test-treemanifest.t b/tests/test-treemanifest.t
--- a/tests/test-treemanifest.t
+++ b/tests/test-treemanifest.t
@@ -792,7 +792,7 @@
   $ hg clone --config experimental.changegroup3=True --stream -U \
   >   http://localhost:$HGPORT1 stream-clone-basicstore
   streaming all changes
-  21 files to transfer, * of data (glob)
+  23 files to transfer, * of data (glob)
   transferred * in * seconds (*) (glob)
   $ hg -R stream-clone-basicstore verify
   checking changesets
@@ -806,7 +806,7 @@
   $ hg clone --config experimental.changegroup3=True --stream -U \
   >   http://localhost:$HGPORT2 stream-clone-encodedstore
   streaming all changes
-  21 files to transfer, * of data (glob)
+  23 files to transfer, * of data (glob)
   transferred * in * seconds (*) (glob)
   $ hg -R stream-clone-encodedstore verify
   checking changesets
@@ -820,7 +820,7 @@
   $ hg clone --config experimental.changegroup3=True --stream -U \
   >   http://localhost:$HGPORT stream-clone-fncachestore
   streaming all changes
-  22 files to transfer, * of data (glob)
+  24 files to transfer, * of data (glob)
   transferred * in * seconds (*) (glob)
   $ hg -R stream-clone-fncachestore verify
   checking changesets
diff --git a/tests/test-tags.t b/tests/test-tags.t
--- a/tests/test-tags.t
+++ b/tests/test-tags.t
@@ -724,6 +724,8 @@
   hgtagsfnodes1
   rbc-names-v1
   rbc-revs-v1
+  topic-names-v1
+  topic-revs-v1
 
 Cache should contain the head only, even though other nodes have tags data
 
@@ -749,6 +751,8 @@
   rbc-names-v1
   rbc-revs-v1
   tags2-visible
+  topic-names-v1
+  topic-revs-v1
 
   $ f --size --hexdump tagsclient/.hg/cache/hgtagsfnodes1
   tagsclient/.hg/cache/hgtagsfnodes1: size=96
diff --git a/tests/test-stream-bundle-v2.t b/tests/test-stream-bundle-v2.t
--- a/tests/test-stream-bundle-v2.t
+++ b/tests/test-stream-bundle-v2.t
@@ -46,7 +46,7 @@
   $ hg bundle -a --type="none-v2;stream=v2" bundle.hg
   $ hg debugbundle bundle.hg
   Stream params: {}
-  stream2 -- {bytecount: 1693, filecount: 11, requirements: 
dotencode%2Cfncache%2Cgeneraldelta%2Crevlogv1%2Csparserevlog%2Cstore} 
(mandatory: True)
+  stream2 -- {bytecount: 1740, filecount: 13, requirements: 
dotencode%2Cfncache%2Cgeneraldelta%2Crevlogv1%2Csparserevlog%2Cstore} 
(mandatory: True)
   $ hg debugbundle --spec bundle.hg
   
none-v2;stream=v2;requirements%3Ddotencode%2Cfncache%2Cgeneraldelta%2Crevlogv1%2Csparserevlog%2Cstore
 
@@ -71,7 +71,7 @@
   bundle2-input-bundle: with-transaction
   bundle2-input-part: "stream2" (params: 3 mandatory) supported
   applying stream bundle
-  11 files to transfer, 1.65 KB of data
+  13 files to transfer, 1.70 KB of data
   starting 4 threads for background file closing (?)
   starting 4 threads for background file closing (?)
   adding [s] data/A.i (66 bytes)
@@ -85,8 +85,10 @@
   adding [c] branch2-served (94 bytes)
   adding [c] rbc-names-v1 (7 bytes)
   adding [c] rbc-revs-v1 (40 bytes)
-  transferred 1.65 KB in \d\.\d seconds \(.*/sec\) (re)
-  bundle2-input-part: total payload size 1840
+  adding [c] topic-names-v1 (7 bytes)
+  adding [c] topic-revs-v1 (40 bytes)
+  transferred 1.70 KB in \d\.\d seconds \(.*/sec\) (re)
+  bundle2-input-part: total payload size 1920
   bundle2-input-bundle: 1 parts total
   updating the branch cache
   finished applying clone bundle
@@ -127,7 +129,7 @@
   bundle2-input-bundle: with-transaction
   bundle2-input-part: "stream2" (params: 3 mandatory) supported
   applying stream bundle
-  11 files to transfer, 1.65 KB of data
+  13 files to transfer, 1.70 KB of data
   starting 4 threads for background file closing (?)
   starting 4 threads for background file closing (?)
   adding [s] data/A.i (66 bytes)
@@ -141,8 +143,10 @@
   adding [c] branch2-served (94 bytes)
   adding [c] rbc-names-v1 (7 bytes)
   adding [c] rbc-revs-v1 (40 bytes)
-  transferred 1.65 KB in *.* seconds (*/sec) (glob)
-  bundle2-input-part: total payload size 1840
+  adding [c] topic-names-v1 (7 bytes)
+  adding 

D9598: share: propery copy cache files when cloning from a share

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  If a is shared to b and b cloned to c, the old code would look directly
  under b/.hg for the cache directory and not use the cachevfs layer to
  pick the members from a/.hg/cache. Adjust variable names and comments to
  reflect that the function is used for more than just the branchmap
  cache.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9598

AFFECTED FILES
  mercurial/hg.py
  tests/test-share.t

CHANGE DETAILS

diff --git a/tests/test-share.t b/tests/test-share.t
--- a/tests/test-share.t
+++ b/tests/test-share.t
@@ -56,6 +56,17 @@
   rbc-revs-v1
   tags2-visible
 
+Cloning a shared repo should pick up the full cache dir on the other hand.
+
+  $ hg clone . ../repo2-clone
+  updating to branch default
+  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  $ ls -1 ../repo2-clone/.hg/cache
+  branch2-served
+  rbc-names-v1
+  rbc-revs-v1
+  tags2-visible
+
 Some sed versions appends newline, some don't, and some just fails
 
   $ cat .hg/sharedpath; echo
diff --git a/mercurial/hg.py b/mercurial/hg.py
--- a/mercurial/hg.py
+++ b/mercurial/hg.py
@@ -590,16 +590,15 @@
 return srcpeer, peer(ui, peeropts, dest)
 
 
-# Recomputing branch cache might be slow on big repos,
-# so just copy it
+# Recomputing caches is often slow on big repos, so copy them.
 def _copycache(srcrepo, dstcachedir, fname):
 """copy a cache from srcrepo to destcachedir (if it exists)"""
-srcbranchcache = srcrepo.vfs.join(b'cache/%s' % fname)
-dstbranchcache = os.path.join(dstcachedir, fname)
-if os.path.exists(srcbranchcache):
+srcfname = srcrepo.cachevfs.join(fname)
+dstfname = os.path.join(dstcachedir, fname)
+if os.path.exists(srcfname):
 if not os.path.exists(dstcachedir):
 os.mkdir(dstcachedir)
-util.copyfile(srcbranchcache, dstbranchcache)
+util.copyfile(srcfname, dstfname)
 
 
 def clone(



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D9597: tests: workaround for a flacky test

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9597

AFFECTED FILES
  tests/test-wireproto-exchangev2-shallow.t

CHANGE DETAILS

diff --git a/tests/test-wireproto-exchangev2-shallow.t 
b/tests/test-wireproto-exchangev2-shallow.t
--- a/tests/test-wireproto-exchangev2-shallow.t
+++ b/tests/test-wireproto-exchangev2-shallow.t
@@ -98,13 +98,14 @@
   received frame(size=9; request=1; stream=2; streamflags=stream-begin; 
type=stream-settings; flags=eos)
   received frame(size=11; request=1; stream=2; streamflags=encoded; 
type=command-response; flags=continuation)
   received frame(size=1170; request=1; stream=2; streamflags=encoded; 
type=command-response; flags=continuation)
-  received frame(size=0; request=1; stream=2; streamflags=; 
type=command-response; flags=eos)
+  received frame(size=0; request=1; stream=2; streamflags=; 
type=command-response; flags=eos) (?)
   add changeset 3390ef850073
   add changeset b709380892b1
   add changeset 47fe012ab237
   add changeset 97765fc3cd62
   add changeset dc666cf9ecf3
   add changeset 93a8bd067ed2
+  received frame(size=0; request=1; stream=2; streamflags=; 
type=command-response; flags=eos) (?)
   checking for updated bookmarks
   sending 1 commands
   sending command manifestdata: {
@@ -402,11 +403,12 @@
   received frame(size=9; request=1; stream=2; streamflags=stream-begin; 
type=stream-settings; flags=eos)
   received frame(size=11; request=1; stream=2; streamflags=encoded; 
type=command-response; flags=continuation)
   received frame(size=783; request=1; stream=2; streamflags=encoded; 
type=command-response; flags=continuation)
-  received frame(size=0; request=1; stream=2; streamflags=; 
type=command-response; flags=eos)
+  received frame(size=0; request=1; stream=2; streamflags=; 
type=command-response; flags=eos) (?)
   add changeset 3390ef850073
   add changeset b709380892b1
   add changeset 47fe012ab237
   add changeset 97765fc3cd62
+  received frame(size=0; request=1; stream=2; streamflags=; 
type=command-response; flags=eos) (?)
   checking for updated bookmarks
   sending 1 commands
   sending command manifestdata: {
@@ -515,9 +517,10 @@
   received frame(size=9; request=1; stream=2; streamflags=stream-begin; 
type=stream-settings; flags=eos)
   received frame(size=11; request=1; stream=2; streamflags=encoded; 
type=command-response; flags=continuation)
   received frame(size=400; request=1; stream=2; streamflags=encoded; 
type=command-response; flags=continuation)
-  received frame(size=0; request=1; stream=2; streamflags=; 
type=command-response; flags=eos)
+  received frame(size=0; request=1; stream=2; streamflags=; 
type=command-response; flags=eos) (?)
   add changeset dc666cf9ecf3
   add changeset 93a8bd067ed2
+  received frame(size=0; request=1; stream=2; streamflags=; 
type=command-response; flags=eos) (?)
   checking for updated bookmarks
   sending 1 commands
   sending command manifestdata: {



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel