D7834: nodemap: have some python code writing a nodemap in persistent binary form

[martinvonz (Martin von Zweigbergk)]

martinvonz added inline comments.

INLINE COMMENTS

> marmoute wrote in nodemap.py:47-49
> The Python code is not meant to be use in production, especially not the 
> parsing one (because it bring no advantage).
> 
> So I want the API to be named after the actual semantic, not after this 
> annecdotical python code.

The actual semantic of this code (the Python code) is to serialize the data. To 
me, "persist" implies writing it to disk, which is not what this code does. I 
think it's just misleading to call it that even if the Rust code will actually 
be writing directly to disk.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7834/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7834

To: marmoute, #hg-reviewers, martinvonz
Cc: martinvonz, mjpieters, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D8076: worker: Manually buffer reads from pickle stream

[heftig (Jan Alexander Steffens)]

heftig added a comment.


  I went for the wrapper overriding `read` after failing to find a way to ask 
`BufferedReader` whether data remains in the buffer. Even `peek` will trigger a 
blocking read when the buffer is empty.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D8076/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D8076

To: heftig, #hg-reviewers, yuja
Cc: yuja, mjpieters, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D7894: nodemap: introduce an option to use mmap to read the nodemap mapping

[marmoute (Pierre-Yves David)]

marmoute added a comment.
marmoute updated this revision to Diff 19906.


  rebase to latest default

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D7894?vs=19846=19906

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7894/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7894

AFFECTED FILES
  mercurial/configitems.py
  mercurial/debugcommands.py
  mercurial/localrepo.py
  mercurial/revlogutils/nodemap.py
  tests/test-persistent-nodemap.t

CHANGE DETAILS

diff --git a/tests/test-persistent-nodemap.t b/tests/test-persistent-nodemap.t
--- a/tests/test-persistent-nodemap.t
+++ b/tests/test-persistent-nodemap.t
@@ -84,3 +84,37 @@
   $ hg debugnodemap --check
   revision in index:   5002
   revision in nodemap: 5002
+
+Test code path without mmap
+---
+
+  $ echo bar > bar
+  $ hg add bar
+  $ hg ci -m 'bar' --config experimental.exp-persistent-nodemap.mmap=no
+
+  $ hg debugnodemap --check --config 
experimental.exp-persistent-nodemap.mmap=yes
+  revision in index:   5003
+  revision in nodemap: 5003
+  $ hg debugnodemap --check --config 
experimental.exp-persistent-nodemap.mmap=no
+  revision in index:   5003
+  revision in nodemap: 5003
+
+
+#if pure
+  $ hg debugnodemap --metadata
+  uid:  (glob)
+  tip-rev: 5002
+  data-length: 123328
+  data-unused: 384
+  $ f --sha256 .hg/store/00changelog-*.nd --size
+  .hg/store/00changelog-.nd: size=123328, 
sha256=10d26e9776b6596af0f89143a54eba8cc581e929c38242a02a7b0760698c6c70 (glob)
+
+#else
+  $ hg debugnodemap --metadata
+  uid:  (glob)
+  tip-rev: 5002
+  data-length: 122944
+  data-unused: 0
+  $ f --sha256 .hg/store/00changelog-*.nd --size
+  .hg/store/00changelog-.nd: size=122944, 
sha256=755976b22b64ab680401b45395953504e64e7fa8c31ac570f58dee21e15f9bc0 (glob)
+#endif
diff --git a/mercurial/revlogutils/nodemap.py b/mercurial/revlogutils/nodemap.py
--- a/mercurial/revlogutils/nodemap.py
+++ b/mercurial/revlogutils/nodemap.py
@@ -8,6 +8,7 @@
 
 from __future__ import absolute_import
 
+import errno
 import os
 import re
 import struct
@@ -46,10 +47,17 @@
 
 filename = _rawdata_filepath(revlog, docket)
 data = revlog.opener.tryread(filename)
+try:
+with revlog.opener(filename) as fd:
+if revlog.opener.options.get(b"exp-persistent-nodemap.mmap"):
+data = fd.read(data_length)
+else:
+data = util.buffer(util.mmapread(fd, data_length))
+except OSError as e:
+if e.errno != errno.ENOENT:
+raise
 if len(data) < data_length:
 return None
-elif len(data) > data_length:
-data = data[:data_length]
 return docket, data
 
 
@@ -81,6 +89,8 @@
 
 can_incremental = util.safehasattr(revlog.index, 
"nodemap_data_incremental")
 ondisk_docket = revlog._nodemap_docket
+feed_data = util.safehasattr(revlog.index, "update_nodemap_data")
+use_mmap = revlog.opener.options.get("exp-persistent-nodemap.mmap")
 
 data = None
 # first attemp an incremental update of the data
@@ -97,12 +107,18 @@
 datafile = _rawdata_filepath(revlog, target_docket)
 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
 # store vfs
+new_length = target_docket.data_length + len(data)
 with revlog.opener(datafile, b'r+') as fd:
 fd.seek(target_docket.data_length)
 fd.write(data)
-fd.seek(0)
-new_data = fd.read(target_docket.data_length + len(data))
-target_docket.data_length += len(data)
+if feed_data:
+if use_mmap:
+fd.seek(0)
+new_data = fd.read(new_length)
+else:
+fd.flush()
+new_data = util.buffer(util.mmapread(fd, new_length))
+target_docket.data_length = new_length
 target_docket.data_unused += data_changed_count
 
 if data is None:
@@ -115,9 +131,14 @@
 data = persistent_data(revlog.index)
 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
 # store vfs
-new_data = data
-with revlog.opener(datafile, b'w') as fd:
+with revlog.opener(datafile, b'w+') as fd:
 fd.write(data)
+if feed_data:
+if use_mmap:
+new_data = data
+else:
+fd.flush()
+new_data = util.buffer(util.mmapread(fd, len(data)))
 target_docket.data_length = len(data)
 target_docket.tip_rev = revlog.tiprev()
 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
@@ -125,7 +146,7 @@
 with revlog.opener(revlog.nodemap_file, b'w', atomictemp=True) as fp:
 fp.write(target_docket.serialize())
 

D7888: nodemap: track the maximum revision tracked in the nodemap

[marmoute (Pierre-Yves David)]

marmoute added a comment.
marmoute updated this revision to Diff 19900.


  rebase to latest default

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D7888?vs=19840=19900

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7888/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7888

AFFECTED FILES
  mercurial/debugcommands.py
  mercurial/pure/parsers.py
  mercurial/revlog.py
  mercurial/revlogutils/nodemap.py
  tests/test-persistent-nodemap.t

CHANGE DETAILS

diff --git a/tests/test-persistent-nodemap.t b/tests/test-persistent-nodemap.t
--- a/tests/test-persistent-nodemap.t
+++ b/tests/test-persistent-nodemap.t
@@ -14,8 +14,9 @@
   $ hg debugbuilddag .+5000
   $ hg debugnodemap --metadata
   uid:  (glob)
+  tip-rev: 5000
   $ f --size .hg/store/00changelog.n
-  .hg/store/00changelog.n: size=18
+  .hg/store/00changelog.n: size=26
   $ f --sha256 .hg/store/00changelog-*.nd
   .hg/store/00changelog-.nd: 
sha256=b961925120e1c9bc345c199b2cc442abc477029fdece37ef9d99cbe59c0558b7 (glob)
   $ hg debugnodemap --dump-new | f --sha256 --size
@@ -51,8 +52,9 @@
   $ hg ci -m 'foo'
   $ hg debugnodemap --metadata
   uid:  (glob)
+  tip-rev: 5001
   $ f --size .hg/store/00changelog.n
-  .hg/store/00changelog.n: size=18
+  .hg/store/00changelog.n: size=26
 
 (The pure code use the debug code that perform incremental update, the C code 
reencode from scratch)
 
diff --git a/mercurial/revlogutils/nodemap.py b/mercurial/revlogutils/nodemap.py
--- a/mercurial/revlogutils/nodemap.py
+++ b/mercurial/revlogutils/nodemap.py
@@ -36,9 +36,11 @@
 if version != ONDISK_VERSION:
 return None
 offset += S_VERSION.size
-(uid_size,) = S_HEADER.unpack(pdata[offset : offset + S_HEADER.size])
+headers = S_HEADER.unpack(pdata[offset : offset + S_HEADER.size])
+uid_size, tip_rev = headers
 offset += S_HEADER.size
 docket = NodeMapDocket(pdata[offset : offset + uid_size])
+docket.tip_rev = tip_rev
 
 filename = _rawdata_filepath(revlog, docket)
 return docket, revlog.opener.tryread(filename)
@@ -94,6 +96,7 @@
 # store vfs
 with revlog.opener(datafile, b'w') as fd:
 fd.write(data)
+target_docket.tip_rev = revlog.tiprev()
 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
 # store vfs
 with revlog.opener(revlog.nodemap_file, b'w', atomictemp=True) as fp:
@@ -142,7 +145,7 @@
 ONDISK_VERSION = 0
 
 S_VERSION = struct.Struct(">B")
-S_HEADER = struct.Struct(">B")
+S_HEADER = struct.Struct(">BQ")
 
 ID_SIZE = 8
 
@@ -164,15 +167,19 @@
 if uid is None:
 uid = _make_uid()
 self.uid = uid
+self.tip_rev = None
 
 def copy(self):
-return NodeMapDocket(uid=self.uid)
+new = NodeMapDocket(uid=self.uid)
+new.tip_rev = self.tip_rev
+return new
 
 def serialize(self):
 """return serialized bytes for a docket using the passed uid"""
 data = []
 data.append(S_VERSION.pack(ONDISK_VERSION))
-data.append(S_HEADER.pack(len(self.uid)))
+headers = (len(self.uid), self.tip_rev)
+data.append(S_HEADER.pack(*headers))
 data.append(self.uid)
 return b''.join(data)
 
diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -639,7 +639,7 @@
 if use_nodemap:
 nodemap_data = nodemaputil.persisted_data(self)
 if nodemap_data is not None:
-index.update_nodemap_data(nodemap_data[1])
+index.update_nodemap_data(*nodemap_data)
 except (ValueError, IndexError):
 raise error.RevlogError(
 _(b"index %s is corrupted") % self.indexfile
diff --git a/mercurial/pure/parsers.py b/mercurial/pure/parsers.py
--- a/mercurial/pure/parsers.py
+++ b/mercurial/pure/parsers.py
@@ -170,15 +170,15 @@
 self._nm_root = self._nm_max_idx = self._nm_rev = None
 return data
 
-def update_nodemap_data(self, nm_data):
-"""provide full blokc of persisted binary data for a nodemap
+def update_nodemap_data(self, docket, nm_data):
+"""provide full block of persisted binary data for a nodemap
 
 The data are expected to come from disk. See `nodemap_data_all` for a
 produceur of such data."""
 if nm_data is not None:
 self._nm_root, self._nm_max_idx = nodemaputil.parse_data(nm_data)
 if self._nm_root:
-self._nm_rev = len(self) - 1
+self._nm_rev = docket.tip_rev
 else:
 self._nm_root = self._nm_max_idx = self._nm_rev = None
 
diff --git a/mercurial/debugcommands.py b/mercurial/debugcommands.py
--- a/mercurial/debugcommands.py
+++ b/mercurial/debugcommands.py
@@ -2136,6 +2136,7 @@
 if nm_data is not None:
 docket, data = nm_data
 

D7889: nodemap: track the total and unused amount of data in the rawdata file

[marmoute (Pierre-Yves David)]

marmoute added a comment.
marmoute updated this revision to Diff 19901.


  rebase to latest default

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D7889?vs=19841=19901

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7889/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7889

AFFECTED FILES
  mercurial/debugcommands.py
  mercurial/pure/parsers.py
  mercurial/revlogutils/nodemap.py
  tests/test-persistent-nodemap.t

CHANGE DETAILS

diff --git a/tests/test-persistent-nodemap.t b/tests/test-persistent-nodemap.t
--- a/tests/test-persistent-nodemap.t
+++ b/tests/test-persistent-nodemap.t
@@ -15,8 +15,10 @@
   $ hg debugnodemap --metadata
   uid:  (glob)
   tip-rev: 5000
+  data-length: 122880
+  data-unused: 0
   $ f --size .hg/store/00changelog.n
-  .hg/store/00changelog.n: size=26
+  .hg/store/00changelog.n: size=42
   $ f --sha256 .hg/store/00changelog-*.nd
   .hg/store/00changelog-.nd: 
sha256=b961925120e1c9bc345c199b2cc442abc477029fdece37ef9d99cbe59c0558b7 (glob)
   $ hg debugnodemap --dump-new | f --sha256 --size
@@ -50,11 +52,22 @@
   $ echo foo > foo
   $ hg add foo
   $ hg ci -m 'foo'
+
+#if pure
   $ hg debugnodemap --metadata
   uid:  (glob)
   tip-rev: 5001
+  data-length: 123072
+  data-unused: 192
+#else
+  $ hg debugnodemap --metadata
+  uid:  (glob)
+  tip-rev: 5001
+  data-length: 122880
+  data-unused: 0
+#endif
   $ f --size .hg/store/00changelog.n
-  .hg/store/00changelog.n: size=26
+  .hg/store/00changelog.n: size=42
 
 (The pure code use the debug code that perform incremental update, the C code 
reencode from scratch)
 
diff --git a/mercurial/revlogutils/nodemap.py b/mercurial/revlogutils/nodemap.py
--- a/mercurial/revlogutils/nodemap.py
+++ b/mercurial/revlogutils/nodemap.py
@@ -37,10 +37,12 @@
 return None
 offset += S_VERSION.size
 headers = S_HEADER.unpack(pdata[offset : offset + S_HEADER.size])
-uid_size, tip_rev = headers
+uid_size, tip_rev, data_length, data_unused = headers
 offset += S_HEADER.size
 docket = NodeMapDocket(pdata[offset : offset + uid_size])
 docket.tip_rev = tip_rev
+docket.data_length = data_length
+docket.data_unused = data_unused
 
 filename = _rawdata_filepath(revlog, docket)
 return docket, revlog.opener.tryread(filename)
@@ -78,12 +80,14 @@
 # first attemp an incremental update of the data
 if can_incremental and ondisk_docket is not None:
 target_docket = revlog._nodemap_docket.copy()
-data = revlog.index.nodemap_data_incremental()
+data_changed_count, data = revlog.index.nodemap_data_incremental()
 datafile = _rawdata_filepath(revlog, target_docket)
 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
 # store vfs
 with revlog.opener(datafile, b'a') as fd:
 fd.write(data)
+target_docket.data_length += len(data)
+target_docket.data_unused += data_changed_count
 else:
 # otherwise fallback to a full new export
 target_docket = NodeMapDocket()
@@ -96,6 +100,7 @@
 # store vfs
 with revlog.opener(datafile, b'w') as fd:
 fd.write(data)
+target_docket.data_length = len(data)
 target_docket.tip_rev = revlog.tiprev()
 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
 # store vfs
@@ -143,9 +148,8 @@
 
 # version 0 is experimental, no BC garantee, do no use outside of tests.
 ONDISK_VERSION = 0
-
 S_VERSION = struct.Struct(">B")
-S_HEADER = struct.Struct(">BQ")
+S_HEADER = struct.Struct(">BQQQ")
 
 ID_SIZE = 8
 
@@ -168,17 +172,26 @@
 uid = _make_uid()
 self.uid = uid
 self.tip_rev = None
+self.data_length = None
+self.data_unused = 0
 
 def copy(self):
 new = NodeMapDocket(uid=self.uid)
 new.tip_rev = self.tip_rev
+new.data_length = self.data_length
+new.data_unused = self.data_unused
 return new
 
 def serialize(self):
 """return serialized bytes for a docket using the passed uid"""
 data = []
 data.append(S_VERSION.pack(ONDISK_VERSION))
-headers = (len(self.uid), self.tip_rev)
+headers = (
+len(self.uid),
+self.tip_rev,
+self.data_length,
+self.data_unused,
+)
 data.append(S_HEADER.pack(*headers))
 data.append(self.uid)
 return b''.join(data)
@@ -236,8 +249,11 @@
 def update_persistent_data(index, root, max_idx, last_rev):
 """return the incremental update for persistent nodemap from a given index
 """
-trie = _update_trie(index, root, last_rev)
-return _persist_trie(trie, existing_idx=max_idx)
+changed_block, trie = _update_trie(index, root, last_rev)
+return (
+changed_block * S_BLOCK.size,
+_persist_trie(trie, existing_idx=max_idx),
+)
 
 
 

D7893: nodemap: update the index with the newly written data (when appropriate)

[marmoute (Pierre-Yves David)]

marmoute added a comment.
marmoute updated this revision to Diff 19905.


  rebase to latest default

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D7893?vs=19845=19905

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7893/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7893

AFFECTED FILES
  mercurial/revlogutils/nodemap.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/nodemap.py b/mercurial/revlogutils/nodemap.py
--- a/mercurial/revlogutils/nodemap.py
+++ b/mercurial/revlogutils/nodemap.py
@@ -100,6 +100,8 @@
 with revlog.opener(datafile, b'r+') as fd:
 fd.seek(target_docket.data_length)
 fd.write(data)
+fd.seek(0)
+new_data = fd.read(target_docket.data_length + len(data))
 target_docket.data_length += len(data)
 target_docket.data_unused += data_changed_count
 
@@ -113,6 +115,7 @@
 data = persistent_data(revlog.index)
 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
 # store vfs
+new_data = data
 with revlog.opener(datafile, b'w') as fd:
 fd.write(data)
 target_docket.data_length = len(data)
@@ -122,6 +125,9 @@
 with revlog.opener(revlog.nodemap_file, b'w', atomictemp=True) as fp:
 fp.write(target_docket.serialize())
 revlog._nodemap_docket = target_docket
+if util.safehasattr(revlog.index, "update_nodemap_data"):
+revlog.index.update_nodemap_data(target_docket, new_data)
+
 # EXP-TODO: if the transaction abort, we should remove the new data and
 # reinstall the old one.
 



To: marmoute, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D7892: nodemap: never read more than the expected data amount

[marmoute (Pierre-Yves David)]

marmoute added a comment.
marmoute updated this revision to Diff 19904.


  rebase to latest default

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D7892?vs=19844=19904

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7892/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7892

AFFECTED FILES
  mercurial/revlogutils/nodemap.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/nodemap.py b/mercurial/revlogutils/nodemap.py
--- a/mercurial/revlogutils/nodemap.py
+++ b/mercurial/revlogutils/nodemap.py
@@ -45,7 +45,12 @@
 docket.data_unused = data_unused
 
 filename = _rawdata_filepath(revlog, docket)
-return docket, revlog.opener.tryread(filename)
+data = revlog.opener.tryread(filename)
+if len(data) < data_length:
+return None
+elif len(data) > data_length:
+data = data[:data_length]
+return docket, data
 
 
 def setup_persistent_nodemap(tr, revlog):



To: marmoute, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D7890: nodemap: double check the source docket when doing incremental update

[marmoute (Pierre-Yves David)]

marmoute added a comment.
marmoute updated this revision to Diff 19902.


  rebase to latest default

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D7890?vs=19842=19902

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7890/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7890

AFFECTED FILES
  mercurial/pure/parsers.py
  mercurial/revlogutils/nodemap.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/nodemap.py b/mercurial/revlogutils/nodemap.py
--- a/mercurial/revlogutils/nodemap.py
+++ b/mercurial/revlogutils/nodemap.py
@@ -77,18 +77,27 @@
 can_incremental = util.safehasattr(revlog.index, 
"nodemap_data_incremental")
 ondisk_docket = revlog._nodemap_docket
 
+data = None
 # first attemp an incremental update of the data
 if can_incremental and ondisk_docket is not None:
 target_docket = revlog._nodemap_docket.copy()
-data_changed_count, data = revlog.index.nodemap_data_incremental()
-datafile = _rawdata_filepath(revlog, target_docket)
-# EXP-TODO: if this is a cache, this should use a cache vfs, not a
-# store vfs
-with revlog.opener(datafile, b'a') as fd:
-fd.write(data)
-target_docket.data_length += len(data)
-target_docket.data_unused += data_changed_count
-else:
+(
+src_docket,
+data_changed_count,
+data,
+) = revlog.index.nodemap_data_incremental()
+if src_docket != target_docket:
+data = None
+else:
+datafile = _rawdata_filepath(revlog, target_docket)
+# EXP-TODO: if this is a cache, this should use a cache vfs, not a
+# store vfs
+with revlog.opener(datafile, b'a') as fd:
+fd.write(data)
+target_docket.data_length += len(data)
+target_docket.data_unused += data_changed_count
+
+if data is None:
 # otherwise fallback to a full new export
 target_docket = NodeMapDocket()
 datafile = _rawdata_filepath(revlog, target_docket)
@@ -182,6 +191,20 @@
 new.data_unused = self.data_unused
 return new
 
+def __cmp__(self, other):
+if self.uid < other.uid:
+return -1
+if self.uid > other.uid:
+return 1
+elif self.data_length < other.data_length:
+return -1
+elif self.data_length > other.data_length:
+return 1
+return 0
+
+def __eq__(self, other):
+return self.uid == other.uid and self.data_length == other.data_length
+
 def serialize(self):
 """return serialized bytes for a docket using the passed uid"""
 data = []
diff --git a/mercurial/pure/parsers.py b/mercurial/pure/parsers.py
--- a/mercurial/pure/parsers.py
+++ b/mercurial/pure/parsers.py
@@ -164,11 +164,13 @@
 """
 if self._nm_root is None:
 return None
+docket = self._nm_docket
 changed, data = nodemaputil.update_persistent_data(
-self, self._nm_root, self._nm_max_idx, self._nm_rev
+self, self._nm_root, self._nm_max_idx, self._nm_docket.tip_rev
 )
-self._nm_root = self._nm_max_idx = self._nm_rev = None
-return changed, data
+
+self._nm_root = self._nm_max_idx = self._nm_docket = None
+return docket, changed, data
 
 def update_nodemap_data(self, docket, nm_data):
 """provide full block of persisted binary data for a nodemap
@@ -178,9 +180,9 @@
 if nm_data is not None:
 self._nm_root, self._nm_max_idx = nodemaputil.parse_data(nm_data)
 if self._nm_root:
-self._nm_rev = docket.tip_rev
+self._nm_docket = docket
 else:
-self._nm_root = self._nm_max_idx = self._nm_rev = None
+self._nm_root = self._nm_max_idx = self._nm_docket = None
 
 
 class InlinedIndexObject(BaseIndexObject):



To: marmoute, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D7891: nodemap: write new data from the expected current data length

[marmoute (Pierre-Yves David)]

marmoute added a comment.
marmoute updated this revision to Diff 19903.


  rebase to latest default

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D7891?vs=19843=19903

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7891/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7891

AFFECTED FILES
  mercurial/revlogutils/nodemap.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/nodemap.py b/mercurial/revlogutils/nodemap.py
--- a/mercurial/revlogutils/nodemap.py
+++ b/mercurial/revlogutils/nodemap.py
@@ -92,7 +92,8 @@
 datafile = _rawdata_filepath(revlog, target_docket)
 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
 # store vfs
-with revlog.opener(datafile, b'a') as fd:
+with revlog.opener(datafile, b'r+') as fd:
+fd.seek(target_docket.data_length)
 fd.write(data)
 target_docket.data_length += len(data)
 target_docket.data_unused += data_changed_count



To: marmoute, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D7885: nodemap: keep track of the docket for loaded data

[marmoute (Pierre-Yves David)]

marmoute added a comment.
marmoute updated this revision to Diff 19897.


  rebase to latest default

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D7885?vs=19838=19897

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7885/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7885

AFFECTED FILES
  mercurial/debugcommands.py
  mercurial/revlog.py
  mercurial/revlogutils/nodemap.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/nodemap.py b/mercurial/revlogutils/nodemap.py
--- a/mercurial/revlogutils/nodemap.py
+++ b/mercurial/revlogutils/nodemap.py
@@ -41,7 +41,7 @@
 docket = NodeMapDocket(pdata[offset : offset + uid_size])
 
 filename = _rawdata_filepath(revlog, docket)
-return revlog.opener.tryread(filename)
+return docket, revlog.opener.tryread(filename)
 
 
 def setup_persistent_nodemap(tr, revlog):
@@ -93,6 +93,7 @@
 # store vfs
 with revlog.opener(revlog.nodemap_file, b'w', atomictemp=True) as fp:
 fp.write(target_docket.serialize())
+revlog._nodemap_docket = target_docket
 # EXP-TODO: if the transaction abort, we should remove the new data and
 # reinstall the old one.
 
diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -455,6 +455,7 @@
 self._maxchainlen = None
 self._deltabothparents = True
 self.index = None
+self._nodemap_docket = None
 # Mapping of partial identifiers to full nodes.
 self._pcache = {}
 # Mapping of revision integer to full node.
@@ -544,6 +545,9 @@
 indexdata = b''
 self._initempty = True
 try:
+nodemap_data = nodemaputil.persisted_data(self)
+if nodemap_data is not None:
+self._nodemap_docket = nodemap_data[0]
 with self._indexfp() as f:
 if (
 mmapindexthreshold is not None
@@ -635,7 +639,7 @@
 if use_nodemap:
 nodemap_data = nodemaputil.persisted_data(self)
 if nodemap_data is not None:
-index.update_nodemap_data(nodemap_data)
+index.update_nodemap_data(nodemap_data[1])
 except (ValueError, IndexError):
 raise error.RevlogError(
 _(b"index %s is corrupted") % self.indexfile
diff --git a/mercurial/debugcommands.py b/mercurial/debugcommands.py
--- a/mercurial/debugcommands.py
+++ b/mercurial/debugcommands.py
@@ -2112,13 +2112,17 @@
 elif opts['dump_disk']:
 unfi = repo.unfiltered()
 cl = unfi.changelog
-data = nodemap.persisted_data(cl)
-ui.write(data)
+nm_data = nodemap.persisted_data(cl)
+if nm_data is not None:
+docket, data = nm_data
+ui.write(data)
 elif opts['check']:
 unfi = repo.unfiltered()
 cl = unfi.changelog
-data = nodemap.persisted_data(cl)
-return nodemap.check_data(ui, cl.index, data)
+nm_data = nodemap.persisted_data(cl)
+if nm_data is not None:
+docket, data = nm_data
+return nodemap.check_data(ui, cl.index, data)
 
 
 @command(



To: marmoute, indygreg, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D7887: nodemap: add a flag to dump the details of the docket

[marmoute (Pierre-Yves David)]

marmoute added a comment.
marmoute updated this revision to Diff 19899.


  rebase to latest default

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D7887?vs=19768=19899

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7887/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7887

AFFECTED FILES
  mercurial/debugcommands.py
  tests/test-completion.t
  tests/test-persistent-nodemap.t

CHANGE DETAILS

diff --git a/tests/test-persistent-nodemap.t b/tests/test-persistent-nodemap.t
--- a/tests/test-persistent-nodemap.t
+++ b/tests/test-persistent-nodemap.t
@@ -12,6 +12,8 @@
   > persistent-nodemap=yes
   > EOF
   $ hg debugbuilddag .+5000
+  $ hg debugnodemap --metadata
+  uid:  (glob)
   $ f --size .hg/store/00changelog.n
   .hg/store/00changelog.n: size=18
   $ f --sha256 .hg/store/00changelog-*.nd
@@ -47,6 +49,8 @@
   $ echo foo > foo
   $ hg add foo
   $ hg ci -m 'foo'
+  $ hg debugnodemap --metadata
+  uid:  (glob)
   $ f --size .hg/store/00changelog.n
   .hg/store/00changelog.n: size=18
 
diff --git a/tests/test-completion.t b/tests/test-completion.t
--- a/tests/test-completion.t
+++ b/tests/test-completion.t
@@ -290,7 +290,7 @@
   debugmanifestfulltextcache: clear, add
   debugmergestate: 
   debugnamecomplete: 
-  debugnodemap: dump-new, dump-disk, check
+  debugnodemap: dump-new, dump-disk, check, metadata
   debugobsolete: flags, record-parents, rev, exclusive, index, delete, date, 
user, template
   debugp1copies: rev
   debugp2copies: rev
diff --git a/mercurial/debugcommands.py b/mercurial/debugcommands.py
--- a/mercurial/debugcommands.py
+++ b/mercurial/debugcommands.py
@@ -2099,6 +2099,12 @@
 False,
 _(b'check that the data on disk data are correct.'),
 ),
+(
+b'',
+b'metadata',
+False,
+_(b'display the on disk meta data for the nodemap'),
+),
 ],
 )
 def debugnodemap(ui, repo, **opts):
@@ -2123,6 +2129,13 @@
 if nm_data is not None:
 docket, data = nm_data
 return nodemap.check_data(ui, cl.index, data)
+elif opts['metadata']:
+unfi = repo.unfiltered()
+cl = unfi.changelog
+nm_data = nodemap.persisted_data(cl)
+if nm_data is not None:
+docket, data = nm_data
+ui.write((b"uid: %s\n") % docket.uid)
 
 
 @command(



To: marmoute, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D7840: nodemap: add a (python) index class for persistent nodemap testing

[marmoute (Pierre-Yves David)]

marmoute added a comment.
marmoute updated this revision to Diff 19888.


  rebase to latest default

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D7840?vs=19756=19888

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7840/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7840

AFFECTED FILES
  mercurial/configitems.py
  mercurial/localrepo.py
  mercurial/pure/parsers.py
  mercurial/revlog.py
  tests/test-persistent-nodemap.t

CHANGE DETAILS

diff --git a/tests/test-persistent-nodemap.t b/tests/test-persistent-nodemap.t
--- a/tests/test-persistent-nodemap.t
+++ b/tests/test-persistent-nodemap.t
@@ -8,6 +8,8 @@
   $ cat << EOF >> .hg/hgrc
   > [experimental]
   > exp-persistent-nodemap=yes
+  > [devel]
+  > persistent-nodemap=yes
   > EOF
   $ hg debugbuilddag .+5000
   $ f --size .hg/store/00changelog.n
diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -352,6 +352,21 @@
 return p
 
 
+NodemapRevlogIO = None
+
+if util.safehasattr(parsers, 'parse_index_devel_nodemap'):
+
+class NodemapRevlogIO(revlogio):
+"""A debug oriented IO class that return a PersistentNodeMapIndexObject
+
+The PersistentNodeMapIndexObject object is meant to test the 
persistent nodemap feature.
+"""
+
+def parseindex(self, data, inline):
+index, cache = parsers.parse_index_devel_nodemap(data, inline)
+return index, cache
+
+
 class rustrevlogio(revlogio):
 def parseindex(self, data, inline):
 index, cache = super(rustrevlogio, self).parseindex(data, inline)
@@ -596,9 +611,17 @@
 
 self._storedeltachains = True
 
+devel_nodemap = (
+self.nodemap_file
+and opts.get(b'devel-force-nodemap', False)
+and NodemapRevlogIO is not None
+)
+
 self._io = revlogio()
 if self.version == REVLOGV0:
 self._io = revlogoldio()
+elif devel_nodemap:
+self._io = NodemapRevlogIO()
 elif rustrevlog is not None and self.opener.options.get(b'rust.index'):
 self._io = rustrevlogio()
 try:
diff --git a/mercurial/pure/parsers.py b/mercurial/pure/parsers.py
--- a/mercurial/pure/parsers.py
+++ b/mercurial/pure/parsers.py
@@ -141,6 +141,15 @@
 self._extra = self._extra[: i - self._lgt]
 
 
+class PersistentNodeMapIndexObject(IndexObject):
+"""a Debug oriented class to test persistent nodemap
+
+We need a simple python object to test API and higher level behavior. See
+the Rust implementation for  more serious usage. This should be used only
+through the dedicated `devel.persistent-nodemap` config.
+"""
+
+
 class InlinedIndexObject(BaseIndexObject):
 def __init__(self, data, inline=0):
 self._data = data
@@ -188,6 +197,12 @@
 return InlinedIndexObject(data, inline), (0, data)
 
 
+def parse_index_devel_nodemap(data, inline):
+"""like parse_index2, but alway return a PersistentNodeMapIndexObject
+"""
+return PersistentNodeMapIndexObject(data), None
+
+
 def parse_dirstate(dmap, copymap, st):
 parents = [st[:20], st[20:40]]
 # dereference fields so they will be local in loop
diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -933,6 +933,8 @@
 options[b'rust.index'] = True
 if ui.configbool(b'experimental', b'exp-persistent-nodemap'):
 options[b'exp-persistent-nodemap'] = True
+if ui.configbool(b'devel', b'persistent-nodemap'):
+options[b'devel-force-nodemap'] = True
 
 return options
 
diff --git a/mercurial/configitems.py b/mercurial/configitems.py
--- a/mercurial/configitems.py
+++ b/mercurial/configitems.py
@@ -406,6 +406,9 @@
 b'devel', b'legacy.exchange', default=list,
 )
 coreconfigitem(
+b'devel', b'persistent-nodemap', default=False,
+)
+coreconfigitem(
 b'devel', b'servercafile', default=b'',
 )
 coreconfigitem(



To: marmoute, indygreg, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D7845: nodemap: add basic checking of the on disk nodemap content

[marmoute (Pierre-Yves David)]

marmoute added a comment.
marmoute updated this revision to Diff 19893.


  rebase to latest default

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D7845?vs=19834=19893

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7845/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7845

AFFECTED FILES
  mercurial/debugcommands.py
  mercurial/revlogutils/nodemap.py
  tests/test-completion.t
  tests/test-persistent-nodemap.t

CHANGE DETAILS

diff --git a/tests/test-persistent-nodemap.t b/tests/test-persistent-nodemap.t
--- a/tests/test-persistent-nodemap.t
+++ b/tests/test-persistent-nodemap.t
@@ -36,6 +36,9 @@
   00d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ||
   00e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ||
   00f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ||
+  $ hg debugnodemap --check
+  revision in index:   5001
+  revision in nodemap: 5001
 
 add a new commit
 
@@ -48,3 +51,6 @@
   .hg/store/00changelog.n: size=18
   $ f --sha256 .hg/store/00changelog-*.nd --size
   .hg/store/00changelog-.nd: size=122880, 
sha256=bfafebd751c4f6d116a76a37a1dee2a251747affe7efbcc4f4842ccc746d4db9 (glob)
+  $ hg debugnodemap --check
+  revision in index:   5002
+  revision in nodemap: 5002
diff --git a/tests/test-completion.t b/tests/test-completion.t
--- a/tests/test-completion.t
+++ b/tests/test-completion.t
@@ -290,7 +290,7 @@
   debugmanifestfulltextcache: clear, add
   debugmergestate: 
   debugnamecomplete: 
-  debugnodemap: dump-new, dump-disk
+  debugnodemap: dump-new, dump-disk, check
   debugobsolete: flags, record-parents, rev, exclusive, index, delete, date, 
user, template
   debugp1copies: rev
   debugp2copies: rev
diff --git a/mercurial/revlogutils/nodemap.py b/mercurial/revlogutils/nodemap.py
--- a/mercurial/revlogutils/nodemap.py
+++ b/mercurial/revlogutils/nodemap.py
@@ -337,3 +337,37 @@
 else:
 b[idx] = _transform_rev(v)
 return block
+
+
+# debug utility
+
+
+def check_data(ui, index, data):
+"""verify that the provided nodemap data are valid for the given idex"""
+ret = 0
+ui.status((b"revision in index:   %d\n") % len(index))
+root = parse_data(data)
+all_revs = set(_all_revisions(root))
+ui.status((b"revision in nodemap: %d\n") % len(all_revs))
+for r in range(len(index)):
+if r not in all_revs:
+msg = b"  revision missing from nodemap: %d\n" % r
+ui.write_err(msg)
+ret = 1
+else:
+all_revs.remove(r)
+if all_revs:
+for r in sorted(all_revs):
+msg = b"  extra revision in  nodemap: %d\n" % r
+ui.write_err(msg)
+ret = 1
+return ret
+
+
+def _all_revisions(root):
+"""return all revisions stored in a Trie"""
+for block in _walk_trie(root):
+for v in block:
+if v is None or isinstance(v, Block):
+continue
+yield v
diff --git a/mercurial/debugcommands.py b/mercurial/debugcommands.py
--- a/mercurial/debugcommands.py
+++ b/mercurial/debugcommands.py
@@ -2093,6 +2093,12 @@
 _(b'write a (new) persistent binary nodemap on stdin'),
 ),
 (b'', b'dump-disk', False, _(b'dump on-disk data on stdin')),
+(
+b'',
+b'check',
+False,
+_(b'check that the data on disk data are correct.'),
+),
 ],
 )
 def debugnodemap(ui, repo, **opts):
@@ -2108,6 +2114,11 @@
 cl = unfi.changelog
 data = nodemap.persisted_data(cl)
 ui.write(data)
+elif opts['check']:
+unfi = repo.unfiltered()
+cl = unfi.changelog
+data = nodemap.persisted_data(cl)
+return nodemap.check_data(ui, cl.index, data)
 
 
 @command(



To: marmoute, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D7886: nodemap: introduce append-only incremental update of the persisten data

[marmoute (Pierre-Yves David)]

marmoute added a comment.
marmoute updated this revision to Diff 19898.


  rebase to latest default

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D7886?vs=19839=19898

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7886/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7886

AFFECTED FILES
  mercurial/pure/parsers.py
  mercurial/revlogutils/nodemap.py
  tests/test-persistent-nodemap.t

CHANGE DETAILS

diff --git a/tests/test-persistent-nodemap.t b/tests/test-persistent-nodemap.t
--- a/tests/test-persistent-nodemap.t
+++ b/tests/test-persistent-nodemap.t
@@ -49,8 +49,19 @@
   $ hg ci -m 'foo'
   $ f --size .hg/store/00changelog.n
   .hg/store/00changelog.n: size=18
+
+(The pure code use the debug code that perform incremental update, the C code 
reencode from scratch)
+
+#if pure
+  $ f --sha256 .hg/store/00changelog-*.nd --size
+  .hg/store/00changelog-.nd: size=123072, 
sha256=136472751566c8198ff09e306a7d2f9bd18bd32298d614752b73da4d6df23340 (glob)
+
+#else
   $ f --sha256 .hg/store/00changelog-*.nd --size
   .hg/store/00changelog-.nd: size=122880, 
sha256=bfafebd751c4f6d116a76a37a1dee2a251747affe7efbcc4f4842ccc746d4db9 (glob)
+
+#endif
+
   $ hg debugnodemap --check
   revision in index:   5002
   revision in nodemap: 5002
diff --git a/mercurial/revlogutils/nodemap.py b/mercurial/revlogutils/nodemap.py
--- a/mercurial/revlogutils/nodemap.py
+++ b/mercurial/revlogutils/nodemap.py
@@ -69,12 +69,41 @@
 if revlog.nodemap_file is None:
 msg = "calling persist nodemap on a revlog without the feature enableb"
 raise error.ProgrammingError(msg)
-if util.safehasattr(revlog.index, "nodemap_data_all"):
-data = revlog.index.nodemap_data_all()
+
+can_incremental = util.safehasattr(revlog.index, 
"nodemap_data_incremental")
+ondisk_docket = revlog._nodemap_docket
+
+# first attemp an incremental update of the data
+if can_incremental and ondisk_docket is not None:
+target_docket = revlog._nodemap_docket.copy()
+data = revlog.index.nodemap_data_incremental()
+datafile = _rawdata_filepath(revlog, target_docket)
+# EXP-TODO: if this is a cache, this should use a cache vfs, not a
+# store vfs
+with revlog.opener(datafile, b'a') as fd:
+fd.write(data)
 else:
-data = persistent_data(revlog.index)
-target_docket = NodeMapDocket()
-datafile = _rawdata_filepath(revlog, target_docket)
+# otherwise fallback to a full new export
+target_docket = NodeMapDocket()
+datafile = _rawdata_filepath(revlog, target_docket)
+if util.safehasattr(revlog.index, "nodemap_data_all"):
+data = revlog.index.nodemap_data_all()
+else:
+data = persistent_data(revlog.index)
+# EXP-TODO: if this is a cache, this should use a cache vfs, not a
+# store vfs
+with revlog.opener(datafile, b'w') as fd:
+fd.write(data)
+# EXP-TODO: if this is a cache, this should use a cache vfs, not a
+# store vfs
+with revlog.opener(revlog.nodemap_file, b'w', atomictemp=True) as fp:
+fp.write(target_docket.serialize())
+revlog._nodemap_docket = target_docket
+# EXP-TODO: if the transaction abort, we should remove the new data and
+# reinstall the old one.
+
+# search for old index file in all cases, some older process might have
+# left one behind.
 olds = _other_rawdata_filepath(revlog, target_docket)
 if olds:
 realvfs = getattr(revlog, '_realopener', revlog.opener)
@@ -85,17 +114,6 @@
 
 callback_id = b"revlog-cleanup-nodemap-%s" % revlog.nodemap_file
 tr.addpostclose(callback_id, cleanup)
-# EXP-TODO: if this is a cache, this should use a cache vfs, not a
-# store vfs
-with revlog.opener(datafile, b'w') as fd:
-fd.write(data)
-# EXP-TODO: if this is a cache, this should use a cache vfs, not a
-# store vfs
-with revlog.opener(revlog.nodemap_file, b'w', atomictemp=True) as fp:
-fp.write(target_docket.serialize())
-revlog._nodemap_docket = target_docket
-# EXP-TODO: if the transaction abort, we should remove the new data and
-# reinstall the old one.
 
 
 ### Nodemap docket file
@@ -208,6 +226,13 @@
 return _persist_trie(trie)
 
 
+def update_persistent_data(index, root, max_idx, last_rev):
+"""return the incremental update for persistent nodemap from a given index
+"""
+trie = _update_trie(index, root, last_rev)
+return _persist_trie(trie, existing_idx=max_idx)
+
+
 S_BLOCK = struct.Struct(">" + ("l" * 16))
 
 NO_ENTRY = -1
@@ -260,6 +285,14 @@
 return root
 
 
+def _update_trie(index, root, last_rev):
+"""consume"""
+for rev in range(last_rev + 1, len(index)):
+hex = nodemod.hex(index[rev][7])
+_insert_into_block(index, 0, root, rev, hex)
+return root
+
+
 def 

D7884: nodemap: introduce an explicit class/object for the docket

[marmoute (Pierre-Yves David)]

marmoute added a comment.
marmoute updated this revision to Diff 19896.


  rebase to latest default

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D7884?vs=19837=19896

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7884/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7884

AFFECTED FILES
  mercurial/revlogutils/nodemap.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/nodemap.py b/mercurial/revlogutils/nodemap.py
--- a/mercurial/revlogutils/nodemap.py
+++ b/mercurial/revlogutils/nodemap.py
@@ -36,11 +36,11 @@
 if version != ONDISK_VERSION:
 return None
 offset += S_VERSION.size
-(uuid_size,) = S_HEADER.unpack(pdata[offset : offset + S_HEADER.size])
+(uid_size,) = S_HEADER.unpack(pdata[offset : offset + S_HEADER.size])
 offset += S_HEADER.size
-uid = pdata[offset : offset + uuid_size]
+docket = NodeMapDocket(pdata[offset : offset + uid_size])
 
-filename = _rawdata_filepath(revlog, uid)
+filename = _rawdata_filepath(revlog, docket)
 return revlog.opener.tryread(filename)
 
 
@@ -73,9 +73,9 @@
 data = revlog.index.nodemap_data_all()
 else:
 data = persistent_data(revlog.index)
-uid = _make_uid()
-datafile = _rawdata_filepath(revlog, uid)
-olds = _other_rawdata_filepath(revlog, uid)
+target_docket = NodeMapDocket()
+datafile = _rawdata_filepath(revlog, target_docket)
+olds = _other_rawdata_filepath(revlog, target_docket)
 if olds:
 realvfs = getattr(revlog, '_realopener', revlog.opener)
 
@@ -92,7 +92,7 @@
 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
 # store vfs
 with revlog.opener(revlog.nodemap_file, b'w', atomictemp=True) as fp:
-fp.write(_serialize_docket(uid))
+fp.write(target_docket.serialize())
 # EXP-TODO: if the transaction abort, we should remove the new data and
 # reinstall the old one.
 
@@ -135,25 +135,39 @@
 return nodemod.hex(os.urandom(ID_SIZE))
 
 
-def _serialize_docket(uid):
-"""return serialized bytes for a docket using the passed uid"""
-data = []
-data.append(S_VERSION.pack(ONDISK_VERSION))
-data.append(S_HEADER.pack(len(uid)))
-data.append(uid)
-return b''.join(data)
+class NodeMapDocket(object):
+"""metadata associated with persistent nodemap data
+
+The persistent data may come from disk or be on their way to disk.
+"""
+
+def __init__(self, uid=None):
+if uid is None:
+uid = _make_uid()
+self.uid = uid
+
+def copy(self):
+return NodeMapDocket(uid=self.uid)
+
+def serialize(self):
+"""return serialized bytes for a docket using the passed uid"""
+data = []
+data.append(S_VERSION.pack(ONDISK_VERSION))
+data.append(S_HEADER.pack(len(self.uid)))
+data.append(self.uid)
+return b''.join(data)
 
 
-def _rawdata_filepath(revlog, uid):
+def _rawdata_filepath(revlog, docket):
 """The (vfs relative) nodemap's rawdata file for a given uid"""
 prefix = revlog.nodemap_file[:-2]
-return b"%s-%s.nd" % (prefix, uid)
+return b"%s-%s.nd" % (prefix, docket.uid)
 
 
-def _other_rawdata_filepath(revlog, uid):
+def _other_rawdata_filepath(revlog, docket):
 prefix = revlog.nodemap_file[:-2]
 pattern = re.compile(b"(^|/)%s-[0-9a-f]+\.nd$" % prefix)
-new_file_path = _rawdata_filepath(revlog, uid)
+new_file_path = _rawdata_filepath(revlog, docket)
 new_file_name = revlog.opener.basename(new_file_path)
 dirpath = revlog.opener.dirname(new_file_path)
 others = []



To: marmoute, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D7835: nodemap: write nodemap data on disk

[marmoute (Pierre-Yves David)]

marmoute added a comment.
marmoute updated this revision to Diff 19883.


  rebase to latest default

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D7835?vs=19825=19883

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7835/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7835

AFFECTED FILES
  mercurial/changelog.py
  mercurial/configitems.py
  mercurial/localrepo.py
  mercurial/revlog.py
  mercurial/revlogutils/nodemap.py
  tests/test-persistent-nodemap.t

CHANGE DETAILS

diff --git a/tests/test-persistent-nodemap.t b/tests/test-persistent-nodemap.t
--- a/tests/test-persistent-nodemap.t
+++ b/tests/test-persistent-nodemap.t
@@ -5,8 +5,14 @@
 
   $ hg init test-repo
   $ cd test-repo
+  $ cat << EOF >> .hg/hgrc
+  > [experimental]
+  > exp-persistent-nodemap=yes
+  > EOF
   $ hg debugbuilddag .+5000
-  $ hg debugnodemap --dump | f --sha256 --bytes=256 --hexdump --size
+  $ hg debugnodemap --dump | f --sha256 --size
+  size=122880, 
sha256=b961925120e1c9bc345c199b2cc442abc477029fdece37ef9d99cbe59c0558b7
+  $ f --sha256 --bytes=256 --hexdump --size < .hg/store/00changelog.n
   size=122880, 
sha256=b961925120e1c9bc345c199b2cc442abc477029fdece37ef9d99cbe59c0558b7
   : ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ||
   0010: ff ff ff ff ff ff ff ff ff ff fa c2 ff ff ff ff ||
diff --git a/mercurial/revlogutils/nodemap.py b/mercurial/revlogutils/nodemap.py
--- a/mercurial/revlogutils/nodemap.py
+++ b/mercurial/revlogutils/nodemap.py
@@ -22,6 +22,39 @@
 raise error.RevlogError(b'unknown node: %s' % x)
 
 
+def setup_persistent_nodemap(tr, revlog):
+"""Install whatever is needed transaction side to persist a nodemap on disk
+
+(only actually persist the nodemap if this is relevant for this revlog)
+"""
+if revlog.nodemap_file is None:
+return  # we do not use persistent_nodemap on this revlog
+callback_id = b"revlog-persistent-nodemap-%s" % revlog.nodemap_file
+if tr.hasfinalize(callback_id):
+return  # no need to register again
+tr.addfinalize(callback_id, lambda tr: _persist_nodemap(tr, revlog))
+
+
+def _persist_nodemap(tr, revlog):
+"""Write nodemap data on disk for a given revlog
+"""
+if getattr(revlog, 'filteredrevs', ()):
+raise error.ProgrammingError(
+"cannot persist nodemap of a filtered changelog"
+)
+if revlog.nodemap_file is None:
+msg = "calling persist nodemap on a revlog without the feature enableb"
+raise error.ProgrammingError(msg)
+data = persistent_data(revlog.index)
+# EXP-TODO: if this is a cache, this should use a cache vfs, not a
+# store vfs
+with revlog.opener(revlog.nodemap_file, b'w') as f:
+f.write(data)
+# EXP-TODO: if the transaction abort, we should remove the new data and
+# reinstall the old one. (This will be simpler when the file format get a
+# bit more advanced)
+
+
 ### Nodemap Trie
 #
 # This is a simple reference implementation to compute and persist a nodemap
diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -407,6 +407,7 @@
 mmaplargeindex=False,
 censorable=False,
 upperboundcomp=None,
+persistentnodemap=False,
 ):
 """
 create a revlog object
@@ -418,6 +419,10 @@
 self.upperboundcomp = upperboundcomp
 self.indexfile = indexfile
 self.datafile = datafile or (indexfile[:-2] + b".d")
+self.nodemap_file = None
+if persistentnodemap:
+self.nodemap_file = indexfile[:-2] + b".n"
+
 self.opener = opener
 #  When True, indexfile is opened with checkambig=True at writing, to
 #  avoid file stat ambiguity.
@@ -2286,6 +2291,7 @@
 ifh.write(data[0])
 ifh.write(data[1])
 self._enforceinlinesize(transaction, ifh)
+nodemaputil.setup_persistent_nodemap(transaction, self)
 
 def addgroup(self, deltas, linkmapper, transaction, addrevisioncb=None):
 """
diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -931,6 +931,8 @@
 
 if ui.configbool(b'experimental', b'rust.index'):
 options[b'rust.index'] = True
+if ui.configbool(b'experimental', b'exp-persistent-nodemap'):
+options[b'exp-persistent-nodemap'] = True
 
 return options
 
diff --git a/mercurial/configitems.py b/mercurial/configitems.py
--- a/mercurial/configitems.py
+++ b/mercurial/configitems.py
@@ -660,6 +660,9 @@
 b'experimental', b'rust.index', default=False,
 )
 coreconfigitem(
+b'experimental', b'exp-persistent-nodemap', default=False,
+)
+coreconfigitem(
 b'experimental', b'server.filesdata.recommended-batch-size', default=5,
 )
 coreconfigitem(
diff --git a/mercurial/changelog.py b/mercurial/changelog.py
--- a/mercurial/changelog.py

D7883: nodemap: keep track of the ondisk id of nodemap blocks

[marmoute (Pierre-Yves David)]

marmoute added a comment.
marmoute updated this revision to Diff 19895.


  rebase to latest default

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D7883?vs=19836=19895

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7883/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7883

AFFECTED FILES
  mercurial/revlogutils/nodemap.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/nodemap.py b/mercurial/revlogutils/nodemap.py
--- a/mercurial/revlogutils/nodemap.py
+++ b/mercurial/revlogutils/nodemap.py
@@ -221,6 +221,11 @@
 
 contains up to 16 entry indexed from 0 to 15"""
 
+def __init__(self):
+super(Block, self).__init__()
+# If this block exist on disk, here is its ID
+self.ondisk_id = None
+
 def __iter__(self):
 return iter(self.get(i) for i in range(16))
 
@@ -323,8 +328,8 @@
 new_blocks = []
 for i in range(0, len(data), S_BLOCK.size):
 block = Block()
-ondisk_id = len(block_map)
-block_map[ondisk_id] = block
+block.ondisk_id = len(block_map)
+block_map[block.ondisk_id] = block
 block_data = data[i : i + S_BLOCK.size]
 values = S_BLOCK.unpack(block_data)
 new_blocks.append((block, values))



To: marmoute, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D7846: nodemap: all check that revision and nodes match in the nodemap

[marmoute (Pierre-Yves David)]

marmoute added a comment.
marmoute updated this revision to Diff 19894.


  rebase to latest default

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D7846?vs=19835=19894

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7846/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7846

AFFECTED FILES
  mercurial/revlogutils/nodemap.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/nodemap.py b/mercurial/revlogutils/nodemap.py
--- a/mercurial/revlogutils/nodemap.py
+++ b/mercurial/revlogutils/nodemap.py
@@ -356,6 +356,19 @@
 ret = 1
 else:
 all_revs.remove(r)
+nm_rev = _find_node(root, nodemod.hex(index[r][7]))
+if nm_rev is None:
+msg = b"  revision node does not match any entries: %d\n" % r
+ui.write_err(msg)
+ret = 1
+elif nm_rev != r:
+msg = (
+b"  revision node does not match the expected revision: "
+b"%d != %d\n" % (r, nm_rev)
+)
+ui.write_err(msg)
+ret = 1
+
 if all_revs:
 for r in sorted(all_revs):
 msg = b"  extra revision in  nodemap: %d\n" % r
@@ -371,3 +384,11 @@
 if v is None or isinstance(v, Block):
 continue
 yield v
+
+
+def _find_node(block, node):
+"""find the revision associated with a given node"""
+entry = block.get(_to_int(node[0:1]))
+if isinstance(entry, dict):
+return _find_node(entry, node[1:])
+return entry



To: marmoute, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D7844: nodemap: code to parse the persistent binary nodemap data

[marmoute (Pierre-Yves David)]

marmoute added a comment.
marmoute updated this revision to Diff 19892.


  rebase to latest default

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D7844?vs=19833=19892

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7844/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7844

AFFECTED FILES
  mercurial/revlogutils/nodemap.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/nodemap.py b/mercurial/revlogutils/nodemap.py
--- a/mercurial/revlogutils/nodemap.py
+++ b/mercurial/revlogutils/nodemap.py
@@ -310,3 +310,30 @@
 return block_map[id(item)]
 else:
 return _transform_rev(item)
+
+
+def parse_data(data):
+"""parse parse nodemap data into a nodemap Trie"""
+if (len(data) % S_BLOCK.size) != 0:
+msg = "nodemap data size is not a multiple of block size (%d): %d"
+raise error.Abort(msg % (S_BLOCK.size, len(data)))
+if not data:
+return Block()
+block_map = {}
+new_blocks = []
+for i in range(0, len(data), S_BLOCK.size):
+block = Block()
+ondisk_id = len(block_map)
+block_map[ondisk_id] = block
+block_data = data[i : i + S_BLOCK.size]
+values = S_BLOCK.unpack(block_data)
+new_blocks.append((block, values))
+for b, values in new_blocks:
+for idx, v in enumerate(values):
+if v == NO_ENTRY:
+continue
+elif v >= 0:
+b[idx] = block_map[v]
+else:
+b[idx] = _transform_rev(v)
+return block



To: marmoute, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D7843: nodemap: move the iteratio inside the Block object

[marmoute (Pierre-Yves David)]

marmoute added a comment.
marmoute updated this revision to Diff 19891.


  rebase to latest default

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D7843?vs=19832=19891

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7843/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7843

AFFECTED FILES
  mercurial/revlogutils/nodemap.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/nodemap.py b/mercurial/revlogutils/nodemap.py
--- a/mercurial/revlogutils/nodemap.py
+++ b/mercurial/revlogutils/nodemap.py
@@ -221,7 +221,8 @@
 
 contains up to 16 entry indexed from 0 to 15"""
 
-pass
+def __iter__(self):
+return iter(self.get(i) for i in range(16))
 
 
 def _build_trie(index):
@@ -297,7 +298,7 @@
 Children block are assumed to be already persisted and present in
 block_map.
 """
-data = tuple(_to_value(block_node.get(i), block_map) for i in range(16))
+data = tuple(_to_value(v, block_map) for v in block_node)
 return S_BLOCK.pack(*data)
 
 



To: marmoute, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D7834: nodemap: have some python code writing a nodemap in persistent binary form

[marmoute (Pierre-Yves David)]

marmoute added a comment.
marmoute updated this revision to Diff 19882.


  rebase to latest default

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D7834?vs=19824=19882

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7834/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7834

AFFECTED FILES
  mercurial/debugcommands.py
  mercurial/revlogutils/nodemap.py
  tests/test-completion.t
  tests/test-help.t
  tests/test-persistent-nodemap.t

CHANGE DETAILS

diff --git a/tests/test-persistent-nodemap.t b/tests/test-persistent-nodemap.t
new file mode 100644
--- /dev/null
+++ b/tests/test-persistent-nodemap.t
@@ -0,0 +1,26 @@
+===
+Test the persistent on-disk nodemap
+===
+
+
+  $ hg init test-repo
+  $ cd test-repo
+  $ hg debugbuilddag .+5000
+  $ hg debugnodemap --dump | f --sha256 --bytes=256 --hexdump --size
+  size=122880, 
sha256=b961925120e1c9bc345c199b2cc442abc477029fdece37ef9d99cbe59c0558b7
+  : ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ||
+  0010: ff ff ff ff ff ff ff ff ff ff fa c2 ff ff ff ff ||
+  0020: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ||
+  0030: ff ff ff ff ff ff ed b3 ff ff ff ff ff ff ff ff ||
+  0040: ff ff ff ff ff ff ee 34 00 00 00 00 ff ff ff ff |...4|
+  0050: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ||
+  0060: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ||
+  0070: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ||
+  0080: ff ff ff ff ff ff f8 50 ff ff ff ff ff ff ff ff |...P|
+  0090: ff ff ff ff ff ff ff ff ff ff ec c7 ff ff ff ff ||
+  00a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ||
+  00b0: ff ff ff ff ff ff fa be ff ff f2 fc ff ff ff ff ||
+  00c0: ff ff ff ff ff ff ef ea ff ff ff ff ff ff f9 17 ||
+  00d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ||
+  00e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ||
+  00f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ||
diff --git a/tests/test-help.t b/tests/test-help.t
--- a/tests/test-help.t
+++ b/tests/test-help.t
@@ -1017,6 +1017,7 @@
  print merge state
debugnamecomplete
  complete "names" - tags, open branch names, bookmark names
+   debugnodemap  write and inspect on disk nodemap
debugobsolete
  create arbitrary obsolete marker
debugoptADV   (no help text available)
diff --git a/tests/test-completion.t b/tests/test-completion.t
--- a/tests/test-completion.t
+++ b/tests/test-completion.t
@@ -107,6 +107,7 @@
   debugmanifestfulltextcache
   debugmergestate
   debugnamecomplete
+  debugnodemap
   debugobsolete
   debugp1copies
   debugp2copies
@@ -289,6 +290,7 @@
   debugmanifestfulltextcache: clear, add
   debugmergestate: 
   debugnamecomplete: 
+  debugnodemap: dump
   debugobsolete: flags, record-parents, rev, exclusive, index, delete, date, 
user, template
   debugp1copies: rev
   debugp2copies: rev
diff --git a/mercurial/revlogutils/nodemap.py b/mercurial/revlogutils/nodemap.py
--- a/mercurial/revlogutils/nodemap.py
+++ b/mercurial/revlogutils/nodemap.py
@@ -7,9 +7,156 @@
 # GNU General Public License version 2 or any later version.
 
 from __future__ import absolute_import
-from .. import error
+
+import struct
+
+from .. import (
+error,
+node as nodemod,
+pycompat,
+)
 
 
 class NodeMap(dict):
 def __missing__(self, x):
 raise error.RevlogError(b'unknown node: %s' % x)
+
+
+### Nodemap Trie
+#
+# This is a simple reference implementation to compute and persist a nodemap
+# trie. This reference implementation is write only. The python version of this
+# is not expected to be actually used, since it wont provide performance
+# improvement over existing non-persistent C implementation.
+#
+# The nodemap is persisted as Trie using 4bits-address/16-entries block. each
+# revision can be adressed using its node shortest prefix.
+#
+# The trie is stored as a sequence of block. Each block contains 16 entries
+# (signed 64bit integer, big endian). Each entry can be one of the following:
+#
+#  * value >=  0 -> index of sub-block
+#  * value == -1 -> no value
+#  * value <  -1 -> a revision value: rev = -(value+10)
+#
+# The implementation focus on simplicity, not on performance. A Rust
+# implementation should provide a efficient version of the same binary
+# persistence. This reference python implementation is never meant to be
+# extensively use in production.
+
+
+def persistent_data(index):
+"""return the persistent binary form for a nodemap for a given index
+"""
+trie = _build_trie(index)
+return _persist_trie(trie)
+
+
+S_BLOCK = struct.Struct(">" + ("l" * 16))
+
+NO_ENTRY = -1
+# rev 0 need to be -2 because 0 is 

D7842: nodemap: use an explicit "Block" object in the reference implementation

[marmoute (Pierre-Yves David)]

marmoute added a comment.
marmoute updated this revision to Diff 19890.


  rebase to latest default

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D7842?vs=19831=19890

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7842/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7842

AFFECTED FILES
  mercurial/revlogutils/nodemap.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/nodemap.py b/mercurial/revlogutils/nodemap.py
--- a/mercurial/revlogutils/nodemap.py
+++ b/mercurial/revlogutils/nodemap.py
@@ -216,6 +216,14 @@
 return int(hex_digit, 16)
 
 
+class Block(dict):
+"""represent a block of the Trie
+
+contains up to 16 entry indexed from 0 to 15"""
+
+pass
+
+
 def _build_trie(index):
 """build a nodemap trie
 
@@ -224,7 +232,7 @@
 Each block is a dictionary with keys in `[0, 15]`. Values are either
 another block or a revision number.
 """
-root = {}
+root = Block()
 for rev in range(len(index)):
 hex = nodemod.hex(index[rev][7])
 _insert_into_block(index, 0, root, rev, hex)
@@ -253,7 +261,7 @@
 # vertices to fit both entry.
 other_hex = nodemod.hex(index[entry][7])
 other_rev = entry
-new = {}
+new = Block()
 block[hex_digit] = new
 _insert_into_block(index, level + 1, new, other_rev, other_hex)
 _insert_into_block(index, level + 1, new, current_rev, current_hex)



To: marmoute, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D7841: nodemap: add a optional `nodemap_add_full` method on indexes

[marmoute (Pierre-Yves David)]

marmoute added a comment.
marmoute updated this revision to Diff 19889.


  rebase to latest default

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D7841?vs=19830=19889

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7841/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7841

AFFECTED FILES
  mercurial/pure/parsers.py
  mercurial/revlogutils/nodemap.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/nodemap.py b/mercurial/revlogutils/nodemap.py
--- a/mercurial/revlogutils/nodemap.py
+++ b/mercurial/revlogutils/nodemap.py
@@ -15,7 +15,7 @@
 from .. import (
 error,
 node as nodemod,
-pycompat,
+util,
 )
 
 
@@ -69,7 +69,10 @@
 if revlog.nodemap_file is None:
 msg = "calling persist nodemap on a revlog without the feature enableb"
 raise error.ProgrammingError(msg)
-data = persistent_data(revlog.index)
+if util.safehasattr(revlog.index, "nodemap_data_all"):
+data = revlog.index.nodemap_data_all()
+else:
+data = persistent_data(revlog.index)
 uid = _make_uid()
 datafile = _rawdata_filepath(revlog, uid)
 olds = _other_rawdata_filepath(revlog, uid)
diff --git a/mercurial/pure/parsers.py b/mercurial/pure/parsers.py
--- a/mercurial/pure/parsers.py
+++ b/mercurial/pure/parsers.py
@@ -149,6 +149,13 @@
 through the dedicated `devel.persistent-nodemap` config.
 """
 
+def nodemap_data_all(self):
+"""Return bytes containing a full serialization of a nodemap
+
+The nodemap should be valid for the full set of revisions in the
+index."""
+return nodemaputil.persistent_data(self)
+
 
 class InlinedIndexObject(BaseIndexObject):
 def __init__(self, data, inline=0):



To: marmoute, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D7839: nodemap: delete older raw data file when creating a new ones

[marmoute (Pierre-Yves David)]

marmoute added a comment.
marmoute updated this revision to Diff 19887.


  rebase to latest default

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D7839?vs=19829=19887

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7839/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7839

AFFECTED FILES
  mercurial/revlogutils/nodemap.py
  tests/test-persistent-nodemap.t

CHANGE DETAILS

diff --git a/tests/test-persistent-nodemap.t b/tests/test-persistent-nodemap.t
--- a/tests/test-persistent-nodemap.t
+++ b/tests/test-persistent-nodemap.t
@@ -12,6 +12,8 @@
   $ hg debugbuilddag .+5000
   $ f --size .hg/store/00changelog.n
   .hg/store/00changelog.n: size=18
+  $ f --sha256 .hg/store/00changelog-*.nd
+  .hg/store/00changelog-.nd: 
sha256=b961925120e1c9bc345c199b2cc442abc477029fdece37ef9d99cbe59c0558b7 (glob)
   $ hg debugnodemap --dump-new | f --sha256 --size
   size=122880, 
sha256=b961925120e1c9bc345c199b2cc442abc477029fdece37ef9d99cbe59c0558b7
   $ hg debugnodemap --dump-disk | f --sha256 --bytes=256 --hexdump --size
@@ -32,3 +34,15 @@
   00d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ||
   00e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ||
   00f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ||
+
+add a new commit
+
+  $ hg up
+  0 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  $ echo foo > foo
+  $ hg add foo
+  $ hg ci -m 'foo'
+  $ f --size .hg/store/00changelog.n
+  .hg/store/00changelog.n: size=18
+  $ f --sha256 .hg/store/00changelog-*.nd --size
+  .hg/store/00changelog-.nd: size=122880, 
sha256=bfafebd751c4f6d116a76a37a1dee2a251747affe7efbcc4f4842ccc746d4db9 (glob)
diff --git a/mercurial/revlogutils/nodemap.py b/mercurial/revlogutils/nodemap.py
--- a/mercurial/revlogutils/nodemap.py
+++ b/mercurial/revlogutils/nodemap.py
@@ -9,6 +9,7 @@
 from __future__ import absolute_import
 
 import os
+import re
 import struct
 
 from .. import (
@@ -71,6 +72,16 @@
 data = persistent_data(revlog.index)
 uid = _make_uid()
 datafile = _rawdata_filepath(revlog, uid)
+olds = _other_rawdata_filepath(revlog, uid)
+if olds:
+realvfs = getattr(revlog, '_realopener', revlog.opener)
+
+def cleanup(tr):
+for oldfile in olds:
+realvfs.tryunlink(oldfile)
+
+callback_id = b"revlog-cleanup-nodemap-%s" % revlog.nodemap_file
+tr.addpostclose(callback_id, cleanup)
 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
 # store vfs
 with revlog.opener(datafile, b'w') as fd:
@@ -136,6 +147,19 @@
 return b"%s-%s.nd" % (prefix, uid)
 
 
+def _other_rawdata_filepath(revlog, uid):
+prefix = revlog.nodemap_file[:-2]
+pattern = re.compile(b"(^|/)%s-[0-9a-f]+\.nd$" % prefix)
+new_file_path = _rawdata_filepath(revlog, uid)
+new_file_name = revlog.opener.basename(new_file_path)
+dirpath = revlog.opener.dirname(new_file_path)
+others = []
+for f in revlog.opener.listdir(dirpath):
+if pattern.match(f) and f != new_file_name:
+others.append(f)
+return others
+
+
 ### Nodemap Trie
 #
 # This is a simple reference implementation to compute and persist a nodemap



To: marmoute, #hg-reviewers
Cc: martinvonz, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D7836: nodemap: add a function to read the data from disk

[marmoute (Pierre-Yves David)]

marmoute added a comment.
marmoute updated this revision to Diff 19884.


  rebase to latest default

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D7836?vs=19826=19884

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7836/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7836

AFFECTED FILES
  mercurial/debugcommands.py
  mercurial/revlogutils/nodemap.py
  tests/test-completion.t
  tests/test-persistent-nodemap.t

CHANGE DETAILS

diff --git a/tests/test-persistent-nodemap.t b/tests/test-persistent-nodemap.t
--- a/tests/test-persistent-nodemap.t
+++ b/tests/test-persistent-nodemap.t
@@ -10,9 +10,9 @@
   > exp-persistent-nodemap=yes
   > EOF
   $ hg debugbuilddag .+5000
-  $ hg debugnodemap --dump | f --sha256 --size
+  $ hg debugnodemap --dump-new | f --sha256 --size
   size=122880, 
sha256=b961925120e1c9bc345c199b2cc442abc477029fdece37ef9d99cbe59c0558b7
-  $ f --sha256 --bytes=256 --hexdump --size < .hg/store/00changelog.n
+  $ hg debugnodemap --dump-disk | f --sha256 --bytes=256 --hexdump --size
   size=122880, 
sha256=b961925120e1c9bc345c199b2cc442abc477029fdece37ef9d99cbe59c0558b7
   : ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ||
   0010: ff ff ff ff ff ff ff ff ff ff fa c2 ff ff ff ff ||
diff --git a/tests/test-completion.t b/tests/test-completion.t
--- a/tests/test-completion.t
+++ b/tests/test-completion.t
@@ -290,7 +290,7 @@
   debugmanifestfulltextcache: clear, add
   debugmergestate: 
   debugnamecomplete: 
-  debugnodemap: dump
+  debugnodemap: dump-new, dump-disk
   debugobsolete: flags, record-parents, rev, exclusive, index, delete, date, 
user, template
   debugp1copies: rev
   debugp2copies: rev
diff --git a/mercurial/revlogutils/nodemap.py b/mercurial/revlogutils/nodemap.py
--- a/mercurial/revlogutils/nodemap.py
+++ b/mercurial/revlogutils/nodemap.py
@@ -22,6 +22,13 @@
 raise error.RevlogError(b'unknown node: %s' % x)
 
 
+def persisted_data(revlog):
+"""read the nodemap for a revlog from disk"""
+if revlog.nodemap_file is None:
+return None
+return revlog.opener.tryread(revlog.nodemap_file)
+
+
 def setup_persistent_nodemap(tr, revlog):
 """Install whatever is needed transaction side to persist a nodemap on disk
 
diff --git a/mercurial/debugcommands.py b/mercurial/debugcommands.py
--- a/mercurial/debugcommands.py
+++ b/mercurial/debugcommands.py
@@ -2085,16 +2085,29 @@
 
 @command(
 b'debugnodemap',
-[(b'', b'dump', False, _(b'write persistent binary nodemap on stdin'))],
+[
+(
+b'',
+b'dump-new',
+False,
+_(b'write a (new) persistent binary nodemap on stdin'),
+),
+(b'', b'dump-disk', False, _(b'dump on-disk data on stdin')),
+],
 )
 def debugnodemap(ui, repo, **opts):
 """write and inspect on disk nodemap
 """
-if opts['dump']:
+if opts['dump_new']:
 unfi = repo.unfiltered()
 cl = unfi.changelog
 data = nodemap.persistent_data(cl.index)
 ui.write(data)
+elif opts['dump_disk']:
+unfi = repo.unfiltered()
+cl = unfi.changelog
+data = nodemap.persisted_data(cl)
+ui.write(data)
 
 
 @command(



To: marmoute, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D7838: nodemap: use an intermediate "docket" file to carry small metadata

[marmoute (Pierre-Yves David)]

marmoute added a comment.
marmoute updated this revision to Diff 19886.


  rebase to latest default

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D7838?vs=19828=19886

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7838/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7838

AFFECTED FILES
  mercurial/revlogutils/nodemap.py
  tests/test-persistent-nodemap.t

CHANGE DETAILS

diff --git a/tests/test-persistent-nodemap.t b/tests/test-persistent-nodemap.t
--- a/tests/test-persistent-nodemap.t
+++ b/tests/test-persistent-nodemap.t
@@ -10,6 +10,8 @@
   > exp-persistent-nodemap=yes
   > EOF
   $ hg debugbuilddag .+5000
+  $ f --size .hg/store/00changelog.n
+  .hg/store/00changelog.n: size=18
   $ hg debugnodemap --dump-new | f --sha256 --size
   size=122880, 
sha256=b961925120e1c9bc345c199b2cc442abc477029fdece37ef9d99cbe59c0558b7
   $ hg debugnodemap --dump-disk | f --sha256 --bytes=256 --hexdump --size
diff --git a/mercurial/revlogutils/nodemap.py b/mercurial/revlogutils/nodemap.py
--- a/mercurial/revlogutils/nodemap.py
+++ b/mercurial/revlogutils/nodemap.py
@@ -8,6 +8,7 @@
 
 from __future__ import absolute_import
 
+import os
 import struct
 
 from .. import (
@@ -26,7 +27,20 @@
 """read the nodemap for a revlog from disk"""
 if revlog.nodemap_file is None:
 return None
-return revlog.opener.tryread(revlog.nodemap_file)
+pdata = revlog.opener.tryread(revlog.nodemap_file)
+if not pdata:
+return None
+offset = 0
+(version,) = S_VERSION.unpack(pdata[offset : offset + S_VERSION.size])
+if version != ONDISK_VERSION:
+return None
+offset += S_VERSION.size
+(uuid_size,) = S_HEADER.unpack(pdata[offset : offset + S_HEADER.size])
+offset += S_HEADER.size
+uid = pdata[offset : offset + uuid_size]
+
+filename = _rawdata_filepath(revlog, uid)
+return revlog.opener.tryread(filename)
 
 
 def setup_persistent_nodemap(tr, revlog):
@@ -55,13 +69,71 @@
 msg = "calling persist nodemap on a revlog without the feature enableb"
 raise error.ProgrammingError(msg)
 data = persistent_data(revlog.index)
+uid = _make_uid()
+datafile = _rawdata_filepath(revlog, uid)
+# EXP-TODO: if this is a cache, this should use a cache vfs, not a
+# store vfs
+with revlog.opener(datafile, b'w') as fd:
+fd.write(data)
 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
 # store vfs
-with revlog.opener(revlog.nodemap_file, b'w') as f:
-f.write(data)
+with revlog.opener(revlog.nodemap_file, b'w', atomictemp=True) as fp:
+fp.write(_serialize_docket(uid))
 # EXP-TODO: if the transaction abort, we should remove the new data and
-# reinstall the old one. (This will be simpler when the file format get a
-# bit more advanced)
+# reinstall the old one.
+
+
+### Nodemap docket file
+#
+# The nodemap data are stored on disk using 2 files:
+#
+# * a raw data files containing a persistent nodemap
+#   (see `Nodemap Trie` section)
+#
+# * a small "docket" file containing medatadata
+#
+# While the nodemap data can be multiple tens of megabytes, the "docket" is
+# small, it is easy to update it automatically or to duplicated its content
+# during a transaction.
+#
+# Multiple raw data can exist at the same time (The currently valid one and a
+# new one beind used by an in progress transaction). To accomodate this, the
+# filename hosting the raw data has a variable parts. The exact filename is
+# specified inside the "docket" file.
+#
+# The docket file contains information to find, qualify and validate the raw
+# data. Its content is currently very light, but it will expand as the on disk
+# nodemap gains the necessary features to be used in production.
+
+# version 0 is experimental, no BC garantee, do no use outside of tests.
+ONDISK_VERSION = 0
+
+S_VERSION = struct.Struct(">B")
+S_HEADER = struct.Struct(">B")
+
+ID_SIZE = 8
+
+
+def _make_uid():
+"""return a new unique identifier.
+
+The identifier is random and composed of ascii characters."""
+return nodemod.hex(os.urandom(ID_SIZE))
+
+
+def _serialize_docket(uid):
+"""return serialized bytes for a docket using the passed uid"""
+data = []
+data.append(S_VERSION.pack(ONDISK_VERSION))
+data.append(S_HEADER.pack(len(uid)))
+data.append(uid)
+return b''.join(data)
+
+
+def _rawdata_filepath(revlog, uid):
+"""The (vfs relative) nodemap's rawdata file for a given uid"""
+prefix = revlog.nodemap_file[:-2]
+return b"%s-%s.nd" % (prefix, uid)
 
 
 ### Nodemap Trie



To: marmoute, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D7837: nodemap: only use persistent nodemap for non-inlined revlog

[marmoute (Pierre-Yves David)]

marmoute added a comment.
marmoute updated this revision to Diff 19885.


  rebase to latest default

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D7837?vs=19827=19885

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7837/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7837

AFFECTED FILES
  mercurial/revlog.py
  mercurial/revlogutils/nodemap.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/nodemap.py b/mercurial/revlogutils/nodemap.py
--- a/mercurial/revlogutils/nodemap.py
+++ b/mercurial/revlogutils/nodemap.py
@@ -34,6 +34,8 @@
 
 (only actually persist the nodemap if this is relevant for this revlog)
 """
+if revlog._inline:
+return  # inlined revlog are too small for this to be relevant
 if revlog.nodemap_file is None:
 return  # we do not use persistent_nodemap on this revlog
 callback_id = b"revlog-persistent-nodemap-%s" % revlog.nodemap_file
diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -1965,6 +1965,7 @@
 # manager
 
 tr.replace(self.indexfile, trindex * self._io.size)
+nodemaputil.setup_persistent_nodemap(tr, self)
 self._chunkclear()
 
 def _nodeduplicatecallback(self, transaction, node):



To: marmoute, indygreg, #hg-reviewers, martinvonz
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D8076: worker: Manually buffer reads from pickle stream

[heftig (Jan Alexander Steffens)]

heftig created this revision.
Herald added subscribers: mercurial-devel, mjpieters.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  My previous fix (D8051 , which added 
Python's built-in buffering to the pickle
  stream) has the problem that the selector will ignore the buffer. When 
multiple
  pickled objects are read from the pipe into the buffer at once, only one 
object
  will be loaded.
  
  This can repeat until the buffer is full and delays the processing of 
completed
  items until the worker exits, at which point the pipe is always considered
  readable and all remaining items are processed.
  
  This changeset reverts D8051 , removing 
the buffer again. Instead, on Python 3
  only, we use a wrapper to modify the "read" provided to the Unpickler to 
behave
  more like a buffered read. We never read more bytes from the pipe than the
  Unpickler requests, so the selector behaves as expected.
  
  Also add a test case for "pickle data was truncated" issue.
  
  https://phab.mercurial-scm.org/D8051#119193

REPOSITORY
  rHG Mercurial

BRANCH
  stable

REVISION DETAIL
  https://phab.mercurial-scm.org/D8076

AFFECTED FILES
  mercurial/worker.py
  tests/test-worker.t

CHANGE DETAILS

diff --git a/tests/test-worker.t b/tests/test-worker.t
--- a/tests/test-worker.t
+++ b/tests/test-worker.t
@@ -131,4 +131,35 @@
   abort: known exception
   [255]
 
+Do not crash on partially read result
+
+  $ cat > $TESTTMP/detecttruncated.py < from __future__ import absolute_import
+  > import os
+  > import sys
+  > import time
+  > sys.unraisablehook = lambda x: None
+  > oldwrite = os.write
+  > def splitwrite(fd, string):
+  > ret = oldwrite(fd, string[:9])
+  > if ret == 9:
+  > time.sleep(0.1)
+  > ret += oldwrite(fd, string[9:])
+  > return ret
+  > os.write = splitwrite
+  > EOF
+
+  $ hg --config "extensions.t=$abspath" --config worker.numcpus=8 --config \
+  > "extensions.d=$TESTTMP/detecttruncated.py" test 10.0
+  start
+  run
+  run
+  run
+  run
+  run
+  run
+  run
+  run
+  done
+
 #endif
diff --git a/mercurial/worker.py b/mercurial/worker.py
--- a/mercurial/worker.py
+++ b/mercurial/worker.py
@@ -65,6 +65,41 @@
 return min(max(countcpus(), 4), 32)
 
 
+if pycompat.ispy3:
+
+class _blockingreader(object):
+def __init__(self, wrapped):
+self._wrapped = wrapped
+
+def __getattr__(self, attr):
+return getattr(self._wrapped, attr)
+
+# issue multiple reads until size is fulfilled
+def read(self, size=-1):
+if size < 0:
+return self._wrapped.readall()
+
+buf = bytearray(size)
+view = memoryview(buf)
+pos = 0
+
+while pos < size:
+ret = self._wrapped.readinto(view[pos:])
+if not ret:
+break
+pos += ret
+
+del view
+del buf[pos:]
+return buf
+
+
+else:
+
+def _blockingreader(wrapped):
+return wrapped
+
+
 if pycompat.isposix or pycompat.iswindows:
 _STARTUP_COST = 0.01
 # The Windows worker is thread based. If tasks are CPU bound, threads
@@ -226,7 +261,7 @@
 selector = selectors.DefaultSelector()
 for rfd, wfd in pipes:
 os.close(wfd)
-selector.register(os.fdopen(rfd, 'rb'), selectors.EVENT_READ)
+selector.register(os.fdopen(rfd, 'rb', 0), selectors.EVENT_READ)
 
 def cleanup():
 signal.signal(signal.SIGINT, oldhandler)
@@ -240,7 +275,7 @@
 while openpipes > 0:
 for key, events in selector.select():
 try:
-res = util.pickle.load(key.fileobj)
+res = util.pickle.load(_blockingreader(key.fileobj))
 if hasretval and res[0]:
 retval.update(res[1])
 else:



To: heftig, #hg-reviewers
Cc: mjpieters, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D8075: config: also respect HGRCSKIPREPO in the zeroconf extension

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D8075

AFFECTED FILES
  hgext/zeroconf/__init__.py
  tests/test-hgrc.t

CHANGE DETAILS

diff --git a/tests/test-hgrc.t b/tests/test-hgrc.t
--- a/tests/test-hgrc.t
+++ b/tests/test-hgrc.t
@@ -299,3 +299,11 @@
   $ killdaemons.py
   $ cat access.log
   $ cat errors.log
+
+Check that zeroconf respect HGRCSKIPREPO=1
+
+  $ hg paths --config extensions.zeroconf=
+  hg: parse error at $TESTTMP/.hg/hgrc:3: [broken
+  [255]
+  $ HGRCSKIPREPO=1 hg paths --config extensions.zeroconf=
+  foo = $TESTTMP/bar
diff --git a/hgext/zeroconf/__init__.py b/hgext/zeroconf/__init__.py
--- a/hgext/zeroconf/__init__.py
+++ b/hgext/zeroconf/__init__.py
@@ -35,6 +35,7 @@
 extensions,
 hg,
 pycompat,
+rcutil,
 ui as uimod,
 )
 from mercurial.hgweb import server as servermod
@@ -144,7 +145,8 @@
 prefix = app.ui.config(b"web", b"prefix", b"").strip(b'/') + b'/'
 for repo, path in repos:
 u = app.ui.copy()
-u.readconfig(os.path.join(path, b'.hg', b'hgrc'))
+if rcutil.use_repo_hgrc():
+u.readconfig(os.path.join(path, b'.hg', b'hgrc'))
 name = os.path.basename(repo)
 path = (prefix + repo).strip(b'/')
 desc = u.config(b'web', b'description')



To: marmoute, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D8074: config: also respect HGRCSKIPREPO in hgwebdir_mod

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D8074

AFFECTED FILES
  mercurial/hgweb/hgwebdir_mod.py
  tests/test-hgrc.t

CHANGE DETAILS

diff --git a/tests/test-hgrc.t b/tests/test-hgrc.t
--- a/tests/test-hgrc.t
+++ b/tests/test-hgrc.t
@@ -281,3 +281,21 @@
   $ HGRCSKIPREPO=1 hg path
   foo = $TESTTMP/bar
 
+Check that hgweb respect HGRCSKIPREPO=1
+
+  $ hg serve -n test -p $HGPORT -d --pid-file=hg.pid -A access.log -E 
errors.log
+  hg: parse error at $TESTTMP/.hg/hgrc:3: [broken
+  [255]
+  $ test -f hg.pid && (cat hg.pid >> $DAEMON_PIDS)
+  [1]
+  $ killdaemons.py
+  $ test -f access.log && cat access.log
+  [1]
+  $ test -f errors.log && cat errors.log
+  [1]
+
+  $ HGRCSKIPREPO=1 hg serve -n test -p $HGPORT -d --pid-file=hg.pid -A 
access.log -E errors.log
+  $ cat hg.pid >> $DAEMON_PIDS
+  $ killdaemons.py
+  $ cat access.log
+  $ cat errors.log
diff --git a/mercurial/hgweb/hgwebdir_mod.py b/mercurial/hgweb/hgwebdir_mod.py
--- a/mercurial/hgweb/hgwebdir_mod.py
+++ b/mercurial/hgweb/hgwebdir_mod.py
@@ -35,6 +35,7 @@
 pathutil,
 profiling,
 pycompat,
+rcutil,
 registrar,
 scmutil,
 templater,
@@ -192,11 +193,12 @@
 continue
 
 u = ui.copy()
-try:
-u.readconfig(os.path.join(path, b'.hg', b'hgrc'))
-except Exception as e:
-u.warn(_(b'error reading %s/.hg/hgrc: %s\n') % (path, e))
-continue
+if rcutil.use_repo_hgrc():
+try:
+u.readconfig(os.path.join(path, b'.hg', b'hgrc'))
+except Exception as e:
+u.warn(_(b'error reading %s/.hg/hgrc: %s\n') % (path, e))
+continue
 
 def get(section, name, default=uimod._unset):
 return u.config(section, name, default, untrusted=True)



To: marmoute, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D8072: config: add a function in `rcutil` to abstract HGRCSKIPREPO

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  We wil need to respect this environment variable in more place.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D8072

AFFECTED FILES
  mercurial/localrepo.py
  mercurial/rcutil.py

CHANGE DETAILS

diff --git a/mercurial/rcutil.py b/mercurial/rcutil.py
--- a/mercurial/rcutil.py
+++ b/mercurial/rcutil.py
@@ -112,3 +112,8 @@
 intended to be set before starting a pager.
 '''
 return {b'LESS': b'FRX', b'LV': b'-c'}
+
+
+def use_repo_hgrc():
+"""True if repositories `.hg/hgrc` config should be read"""
+return b'HGRCSKIPREPO' not in encoding.environ
diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -53,6 +53,7 @@
 phases,
 pushkey,
 pycompat,
+rcutil,
 repoview,
 revset,
 revsetlang,
@@ -676,7 +677,7 @@
 configs are loaded. For example, an extension may wish to pull in
 configs from alternate files or sources.
 """
-if b'HGRCSKIPREPO' in encoding.environ:
+if not rcutil.use_repo_hgrc():
 return False
 try:
 ui.readconfig(hgvfs.join(b'hgrc'), root=wdirvfs.base)



To: marmoute, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D8073: config: also respect HGRCSKIPREPO in `dispatch._getlocal`

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  For some reason, we are also reading the local config in that function.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D8073

AFFECTED FILES
  mercurial/dispatch.py
  tests/test-hgrc.t

CHANGE DETAILS

diff --git a/tests/test-hgrc.t b/tests/test-hgrc.t
--- a/tests/test-hgrc.t
+++ b/tests/test-hgrc.t
@@ -271,3 +271,13 @@
   $ HGRCSKIPREPO=1 hg path
   foo = $TESTTMP/bar
 
+  $ cat >> .hg/hgrc < [broken
+  > EOF
+
+  $ hg path
+  hg: parse error at $TESTTMP/.hg/hgrc:3: [broken
+  [255]
+  $ HGRCSKIPREPO=1 hg path
+  foo = $TESTTMP/bar
+
diff --git a/mercurial/dispatch.py b/mercurial/dispatch.py
--- a/mercurial/dispatch.py
+++ b/mercurial/dispatch.py
@@ -37,6 +37,7 @@
 hook,
 profiling,
 pycompat,
+rcutil,
 registrar,
 scmutil,
 ui as uimod,
@@ -902,17 +903,20 @@
 _(b"error getting current working directory: %s")
 % encoding.strtolocal(e.strerror)
 )
+
 path = cmdutil.findrepo(wd) or b""
 if not path:
 lui = ui
 else:
 lui = ui.copy()
-lui.readconfig(os.path.join(path, b".hg", b"hgrc"), path)
+if rcutil.use_repo_hgrc():
+lui.readconfig(os.path.join(path, b".hg", b"hgrc"), path)
 
 if rpath:
 path = lui.expandpath(rpath)
 lui = ui.copy()
-lui.readconfig(os.path.join(path, b".hg", b"hgrc"), path)
+if rcutil.use_repo_hgrc():
+lui.readconfig(os.path.join(path, b".hg", b"hgrc"), path)
 
 return path, lui
 



To: marmoute, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D8039: chg: force-set LC_CTYPE on server start to actual value from the environment

[spectral (Kyle Lippincott)]

spectral updated this revision to Diff 19876.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D8039?vs=19676=19876

BRANCH
  default

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D8039/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D8039

AFFECTED FILES
  contrib/chg/chg.c
  mercurial/chgserver.py
  tests/test-chg.t

CHANGE DETAILS

diff --git a/tests/test-chg.t b/tests/test-chg.t
--- a/tests/test-chg.t
+++ b/tests/test-chg.t
@@ -332,8 +332,8 @@
   /MM/DD HH:MM:SS (PID)> log -R cached
   /MM/DD HH:MM:SS (PID)> loaded repo into cache: $TESTTMP/cached (in  ...s)
 
-Test that chg works even when python "coerces" the locale (py3.7+, which is 
done
-by default if none of LC_ALL, LC_CTYPE, or LANG are set in the environment)
+Test that chg works (sets to the user's actual LC_CTYPE) even when python
+"coerces" the locale (py3.7+)
 
   $ cat > $TESTTMP/debugenv.py < from mercurial import encoding
@@ -347,9 +347,22 @@
   > if v is not None:
   > ui.write(b'%s=%s\n' % (k, encoding.environ[k]))
   > EOF
+(hg keeps python's modified LC_CTYPE, chg doesn't)
+  $ (unset LC_ALL; unset LANG; LC_CTYPE= "$CHGHG" \
+  >--config extensions.debugenv=$TESTTMP/debugenv.py debugenv)
+  LC_CTYPE=C.UTF-8 (py37 !)
+  LC_CTYPE= (no-py37 !)
+  $ (unset LC_ALL; unset LANG; LC_CTYPE= chg \
+  >--config extensions.debugenv=$TESTTMP/debugenv.py debugenv)
+  LC_CTYPE=
+  $ (unset LC_ALL; unset LANG; LC_CTYPE=unsupported_value chg \
+  >--config extensions.debugenv=$TESTTMP/debugenv.py debugenv)
+  LC_CTYPE=unsupported_value
+  $ (unset LC_ALL; unset LANG; LC_CTYPE= chg \
+  >--config extensions.debugenv=$TESTTMP/debugenv.py debugenv)
+  LC_CTYPE=
   $ LANG= LC_ALL= LC_CTYPE= chg \
   >--config extensions.debugenv=$TESTTMP/debugenv.py debugenv
   LC_ALL=
-  LC_CTYPE=C.UTF-8 (py37 !)
-  LC_CTYPE= (no-py37 !)
+  LC_CTYPE=
   LANG=
diff --git a/mercurial/chgserver.py b/mercurial/chgserver.py
--- a/mercurial/chgserver.py
+++ b/mercurial/chgserver.py
@@ -550,40 +550,6 @@
 raise ValueError(b'unexpected value in setenv request')
 self.ui.log(b'chgserver', b'setenv: %r\n', sorted(newenv.keys()))
 
-# Python3 has some logic to "coerce" the C locale to a UTF-8 capable
-# one, and it sets LC_CTYPE in the environment to C.UTF-8 if none of
-# 'LC_CTYPE', 'LC_ALL' or 'LANG' are set (to any value). This can be
-# disabled with PYTHONCOERCECLOCALE=0 in the environment.
-#
-# When fromui is called via _inithashstate, python has already set
-# this, so that's in the environment right when we start up the hg
-# process. Then chg will call us and tell us to set the environment to
-# the one it has; this might NOT have LC_CTYPE, so we'll need to
-# carry-forward the LC_CTYPE that was coerced in these situations.
-#
-# If this is not handled, we will fail config+env validation and fail
-# to start chg. If this is just ignored instead of carried forward, we
-# may have different behavior between chg and non-chg.
-if pycompat.ispy3:
-# Rename for wordwrapping purposes
-oldenv = encoding.environ
-if not any(
-e.get(b'PYTHONCOERCECLOCALE') == b'0' for e in [oldenv, newenv]
-):
-keys = [b'LC_CTYPE', b'LC_ALL', b'LANG']
-old_keys = [k for k, v in oldenv.items() if k in keys and v]
-new_keys = [k for k, v in newenv.items() if k in keys and v]
-# If the user's environment (from chg) doesn't have ANY of the
-# keys that python looks for, and the environment (from
-# initialization) has ONLY LC_CTYPE and it's set to C.UTF-8,
-# carry it forward.
-if (
-not new_keys
-and old_keys == [b'LC_CTYPE']
-and oldenv[b'LC_CTYPE'] == b'C.UTF-8'
-):
-newenv[b'LC_CTYPE'] = oldenv[b'LC_CTYPE']
-
 encoding.environ.clear()
 encoding.environ.update(newenv)
 
@@ -730,6 +696,16 @@
 # environ cleaner.
 if b'CHGINTERNALMARK' in encoding.environ:
 del encoding.environ[b'CHGINTERNALMARK']
+# Python3.7+ "coerces" the LC_CTYPE environment variable to a UTF-8 one if
+# it thinks the current value is "C". This breaks the hash computation and
+# causes chg to restart loop.
+if b'CHGORIG_LC_CTYPE' in encoding.environ:
+encoding.environ[b'LC_CTYPE'] = encoding.environ[b'CHGORIG_LC_CTYPE']
+del encoding.environ[b'CHGORIG_LC_CTYPE']
+elif b'CHG_CLEAR_LC_CTYPE' in encoding.environ:
+if b'LC_CTYPE' in encoding.environ:
+del encoding.environ[b'LC_CTYPE']
+del encoding.environ[b'CHG_CLEAR_LC_CTYPE']
 
 if repo:
 # one chgserver can serve multiple repos. drop repo 

D8023: chg: read CHGORIG_ values from env and handle these appropriately

[spectral (Kyle Lippincott)]

spectral updated this revision to Diff 19875.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D8023?vs=19652=19875

BRANCH
  default

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D8023/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D8023

AFFECTED FILES
  mercurial/chgserver.py
  tests/test-chg.t

CHANGE DETAILS

diff --git a/tests/test-chg.t b/tests/test-chg.t
--- a/tests/test-chg.t
+++ b/tests/test-chg.t
@@ -332,8 +332,7 @@
   /MM/DD HH:MM:SS (PID)> log -R cached
   /MM/DD HH:MM:SS (PID)> loaded repo into cache: $TESTTMP/cached (in  ...s)
 
-Test that chg works even when python "coerces" the locale (py3.7+, which is 
done
-by default if none of LC_ALL, LC_CTYPE, or LANG are set in the environment)
+Test that chg works even when python "coerces" the locale (py3.7+)
 
   $ cat > $TESTTMP/debugenv.py < from mercurial import encoding
@@ -345,11 +344,31 @@
   > for k in [b'LC_ALL', b'LC_CTYPE', b'LANG']:
   > v = encoding.environ.get(k)
   > if v is not None:
-  > ui.write(b'%s=%s\n' % (k, encoding.environ[k]))
+  > ui.write(b'debugenv: %s=%s\n' % (k, encoding.environ[k]))
   > EOF
-  $ LANG= LC_ALL= LC_CTYPE= chg \
-  >--config extensions.debugenv=$TESTTMP/debugenv.py debugenv
-  LC_ALL=
-  LC_CTYPE=C.UTF-8 (py37 !)
-  LC_CTYPE= (no-py37 !)
-  LANG=
+  $ CHGDEBUG= LANG= LC_ALL= LC_CTYPE= chg \
+  >--config extensions.debugenv=$TESTTMP/debugenv.py debugenv 2>&1 \
+  >| egrep 'debugenv|start'
+  chg: debug: * start cmdserver at * (glob)
+  debugenv: LC_ALL=
+  debugenv: LC_CTYPE=C.UTF-8 (py37 !)
+  debugenv: LC_CTYPE= (no-py37 !)
+  debugenv: LANG=
+(Should not trigger a command server restart)
+  $ CHGDEBUG= LANG= LC_ALL= LC_CTYPE= chg \
+  >--config extensions.debugenv=$TESTTMP/debugenv.py debugenv 2>&1 \
+  >| egrep 'debugenv|start'
+  debugenv: LC_ALL=
+  debugenv: LC_CTYPE=C.UTF-8 (py37 !)
+  debugenv: LC_CTYPE= (no-py37 !)
+  debugenv: LANG=
+(Should trigger a command server restart since the LC_CTYPE=C value should be
+used, not the coerced one)
+  $ CHGDEBUG= LANG= LC_ALL= LC_CTYPE=C chg \
+  >--config extensions.debugenv=$TESTTMP/debugenv.py debugenv 2>&1 \
+  >| egrep 'debugenv|start'
+  chg: debug: * start cmdserver at * (glob)
+  debugenv: LC_ALL=
+  debugenv: LC_CTYPE=C.UTF-8 (py37 !)
+  debugenv: LC_CTYPE=C (no-py37 !)
+  debugenv: LANG=
diff --git a/mercurial/chgserver.py b/mercurial/chgserver.py
--- a/mercurial/chgserver.py
+++ b/mercurial/chgserver.py
@@ -129,12 +129,37 @@
 ignored = {b'HG'}
 else:
 ignored = set()
-envitems = [
-(k, v)
+envitems = {
+k: v
 for k, v in pycompat.iteritems(encoding.environ)
 if _envre.match(k) and k not in ignored
-]
-envhash = _hashlist(sorted(envitems))
+}
+# Py3 may modify environment variables before `hg serve` starts. When chg
+# connects to us, it passes in its environment (without the modifications),
+# and we'll generate a different config hash.
+#
+# To handle this situation, chg will pass the variables through using
+# CHGORIG_=<0 if not in env, 1 if in env>, and we use
+# *that* value only when calculating the hash. Examples:
+#   CHGORIG_LC_CTYPE=0   <-- not in chg's env, delete for hash
+#   CHGORIG_LC_CTYPE=1   <-- in chg's env, but empty. Set to empty for 
hash.
+#   CHGORIG_LC_CTYPE=1UTF-8  <-- set to UTF-8 for hash.
+#
+# This does NOT cause us to lose Py3's modifications to these variables (we
+# aren't copying this into our environment), so chg and hg should continue
+# to behave the same - both will operate with the modifications Py3 made.
+for k, v in pycompat.iteritems(encoding.environ):
+if k.startswith(b'CHGORIG_'):
+origname = k[len(b'CHGORIG_') :]
+if not _envre.match(origname) or origname in ignored:
+continue
+
+if v.startswith(b'0') and origname in envitems:
+del envitems[origname]
+else:
+envitems[origname] = v[1:]
+
+envhash = _hashlist(sorted(pycompat.iteritems(envitems)))
 return sectionhash[:6] + envhash[:6]
 
 
@@ -550,39 +575,18 @@
 raise ValueError(b'unexpected value in setenv request')
 self.ui.log(b'chgserver', b'setenv: %r\n', sorted(newenv.keys()))
 
-# Python3 has some logic to "coerce" the C locale to a UTF-8 capable
-# one, and it sets LC_CTYPE in the environment to C.UTF-8 if none of
-# 'LC_CTYPE', 'LC_ALL' or 'LANG' are set (to any value). This can be
-# disabled with PYTHONCOERCECLOCALE=0 in the environment.
-#
-# When fromui is called via _inithashstate, python has already set
-# this, so that's in the environment right when we start up the hg
-# process. Then chg will call us and tell us to set the environment to
-# the one it has; this might 

D8022: chg: pass copies of some envvars so we can detect py37+ modifications

[spectral (Kyle Lippincott)]

spectral updated this revision to Diff 19874.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D8022?vs=19656=19874

BRANCH
  default

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D8022/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D8022

AFFECTED FILES
  contrib/chg/chg.c

CHANGE DETAILS

diff --git a/contrib/chg/chg.c b/contrib/chg/chg.c
--- a/contrib/chg/chg.c
+++ b/contrib/chg/chg.c
@@ -201,6 +201,36 @@
return hgcmd;
 }
 
+static void putsidechannelenv(const char *envname)
+{
+   char *namebuf, *valbuf;
+   const char *existing_val = getenv(envname);
+   size_t existing_val_len = 0, namebuf_len = 0, valbuf_len = 0;
+   size_t envname_len = strlen(envname);
+
+   if (existing_val != NULL) {
+   existing_val_len = strlen(existing_val);
+   }
+
+   namebuf_len = envname_len + 9;
+   namebuf = mallocx(namebuf_len);
+   memcpy(namebuf, "CHGORIG_", 8);
+   memcpy(namebuf + 8, envname, envname_len + 1);
+
+   valbuf_len = existing_val_len + 2;
+   valbuf = mallocx(valbuf_len);
+   valbuf[0] = (existing_val == NULL) ? '0' : '1';
+   if (existing_val != NULL && existing_val_len > 0) {
+   memcpy(valbuf + 1, existing_val, existing_val_len);
+   }
+   valbuf[valbuf_len - 1] = '\0';
+   if (setenv(namebuf, valbuf, 1) != 0) {
+   abortmsgerrno("failed to setenv for stored vars");
+   }
+   free(valbuf);
+   free(namebuf);
+}
+
 static void execcmdserver(const struct cmdserveropts *opts)
 {
const char *hgcmd = gethgcmd();
@@ -226,6 +256,14 @@
}
argv[argsize - 1] = NULL;
 
+   char *coerce = getenv("PYTHONCOERCECLOCALE");
+   if (coerce == NULL || coerce[0] != '0') {
+   /* Py3 modifies the environment on execution, we need a side
+* channel to get an unmodified environment to `hg serve` for
+* some variables */
+   putsidechannelenv("LC_CTYPE");
+   }
+
if (putenv("CHGINTERNALMARK=") != 0)
abortmsgerrno("failed to putenv");
if (execvp(hgcmd, (char **)argv) < 0)



To: spectral, #hg-reviewers
Cc: yuja, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D8039: chg: force-set LC_CTYPE on server start to actual value from the environment

[spectral (Kyle Lippincott)]

spectral added a comment.


  In D8039#118719 , @yuja wrote:
  
  >>   What do you think about this approach:
  >>   1. The server detects that LC_TYPE is coerced.
  >>   2. When handling the "validate" command, the server sends back 
"invalidate this server, and fallback to original hg" response.
  >>   This makes chg/non-chg behave consistently with some startup overhead in 
mis-configured environment. The chg client can potentially print a warning to 
remind the user to fix their environment.
  >
  > That could be, but if we do want to make chg/hg behavior consistent, maybe 
we
  > can adjust the hash computation logic?
  >
  > 1. client sends CHG_ORIG_LC_CTYPE or CHG_UNSET_LC_CTYPE when spawning server
  > 2. they're kept in environ dict, but override LC_CTYPE while computing 
hash, and excluded from the hash
  > 3. client does not send these variables over setenv command, but passes the 
validation because `{CHG_ORIG_LC_CTYPE: x, LC_CTYPE: y} == {LC_CTYPE: x}`.
  
  I think that's almost what I was doing in the original stack of three commits 
ending in D8023 , though I used a 
different encoding than two separate environment variables (one for clear, one 
for set), I have no strong preference between the two encoding methods. There's 
still a bit of a issue with the setenv command: it'll clear the environment, 
and replace it with the one that came from chg. So we need to somehow keep it 
from clearing/modifying CHG_ORIG_LC_CTYPE and LC_CTYPE, which isn't difficult 
(D8023  already does this, just with a 
more complicated mechanism).  With a small change to D8023 
 I think D8021 
 could be dropped, and D8022 
 could be changed to only do this in 
execcmdserver.  Done.
  
  > If we had a sha1 logic in C, we could compute the env hash at client side.
  
  I think sha1dc has been added to the repo, so we do have a sha1 
implementation in C. :)  Computing the hash on the client avoids the restart 
loop, but doesn't avoid the behavior difference - chg will still call setenv 
and replace the modified version with the original one.
  
  > Python 3 can't be trusted.
  
  Yeah :(
  
  Sorry about the delay responding here, I was sick with the flu.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D8039/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D8039

To: spectral, #hg-reviewers
Cc: quark, yuja, mjpieters, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D8071: copies: avoid filtering by short-circuit dirstate-only copies earlier

[martinvonz (Martin von Zweigbergk)]

martinvonz created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  The call to `y.ancestor(x)` triggered repo filtering, which we'd like
  to avoid in the simple `hg status --copies` case.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D8071

AFFECTED FILES
  mercurial/copies.py
  tests/test-repo-filters-tiptoe.t

CHANGE DETAILS

diff --git a/tests/test-repo-filters-tiptoe.t b/tests/test-repo-filters-tiptoe.t
--- a/tests/test-repo-filters-tiptoe.t
+++ b/tests/test-repo-filters-tiptoe.t
@@ -68,7 +68,6 @@
   ! b
 
   $ hg status --copies
-  debug.filters: computing revision filter for "visible"
   M c
   A d
   R a
diff --git a/mercurial/copies.py b/mercurial/copies.py
--- a/mercurial/copies.py
+++ b/mercurial/copies.py
@@ -403,13 +403,15 @@
 )
 if x == y or not x or not y:
 return {}
+if y.rev() is None and x == y.p1():
+if debug:
+repo.ui.debug(b'debug.copies: search mode: dirstate\n')
+# short-circuit to avoid issues with merge states
+return _dirstatecopies(repo, match)
 a = y.ancestor(x)
 if a == x:
 if debug:
 repo.ui.debug(b'debug.copies: search mode: forward\n')
-if y.rev() is None and x == y.p1():
-# short-circuit to avoid issues with merge states
-return _dirstatecopies(repo, match)
 copies = _forwardcopies(x, y, match=match)
 elif a == y:
 if debug:



To: martinvonz, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D8070: tests: add test showing that repo filter is calculated for `hg st --copies`

[martinvonz (Martin von Zweigbergk)]

martinvonz created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D8070

AFFECTED FILES
  tests/test-repo-filters-tiptoe.t

CHANGE DETAILS

diff --git a/tests/test-repo-filters-tiptoe.t b/tests/test-repo-filters-tiptoe.t
--- a/tests/test-repo-filters-tiptoe.t
+++ b/tests/test-repo-filters-tiptoe.t
@@ -67,6 +67,13 @@
   R a
   ! b
 
+  $ hg status --copies
+  debug.filters: computing revision filter for "visible"
+  M c
+  A d
+  R a
+  ! b
+
 Getting data about the working copy parent
 
   $ hg log -r '.' -T "{node}\n{date}\n"



To: martinvonz, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D7827: rebase: don't use rebased node as dirstate p2 (BC)

[martinvonz (Martin von Zweigbergk)]

martinvonz updated this revision to Diff 19871.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D7827?vs=19861=19871

BRANCH
  default

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7827/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7827

AFFECTED FILES
  hgext/rebase.py
  relnotes/next
  tests/test-rebase-abort.t
  tests/test-rebase-collapse.t
  tests/test-rebase-conflicts.t
  tests/test-rebase-interruptions.t
  tests/test-rebase-obsolete.t
  tests/test-rebase-parameters.t
  tests/test-rebase-transaction.t

CHANGE DETAILS

diff --git a/tests/test-rebase-transaction.t b/tests/test-rebase-transaction.t
--- a/tests/test-rebase-transaction.t
+++ b/tests/test-rebase-transaction.t
@@ -114,7 +114,7 @@
   |
   | @  4: Z
   | |
-  @ |  3: C
+  % |  3: C
   | |
   | o  2: Y
   | |
@@ -123,9 +123,9 @@
   o  0: A
   
   $ hg st
-  M C
   M conflict
   A B
+  A C
   ? conflict.orig
   $ echo resolved > conflict
   $ hg resolve -m
diff --git a/tests/test-rebase-parameters.t b/tests/test-rebase-parameters.t
--- a/tests/test-rebase-parameters.t
+++ b/tests/test-rebase-parameters.t
@@ -481,11 +481,9 @@
   $ hg summary
   parent: 1:56daeba07f4b 
c2
-  parent: 2:e4e3f3546619 tip
-   c2b
   branch: default
-  commit: 1 modified, 1 unresolved (merge)
-  update: (current)
+  commit: 1 unresolved (clean)
+  update: 1 new changesets, 2 branch heads (merge)
   phases: 3 draft
   rebase: 0 rebased, 1 remaining (rebase --continue)
 
diff --git a/tests/test-rebase-obsolete.t b/tests/test-rebase-obsolete.t
--- a/tests/test-rebase-obsolete.t
+++ b/tests/test-rebase-obsolete.t
@@ -1795,19 +1795,15 @@
   $ hg log -G
   @  2:b18e25de2cf5 D
   |
-  | @  1:2ec65233581b B (pruned using prune)
-  |/
   o  0:426bada5c675 A
   
   $ hg summary
   parent: 2:b18e25de2cf5 tip
D
-  parent: 1:2ec65233581b  (obsolete)
-   B
   branch: default
-  commit: 2 modified, 1 unknown, 1 unresolved (merge)
+  commit: 1 modified, 1 added, 1 unknown, 1 unresolved
   update: (current)
-  phases: 3 draft
+  phases: 2 draft
   rebase: 0 rebased, 2 remaining (rebase --continue)
 
   $ hg rebase --abort
diff --git a/tests/test-rebase-interruptions.t 
b/tests/test-rebase-interruptions.t
--- a/tests/test-rebase-interruptions.t
+++ b/tests/test-rebase-interruptions.t
@@ -294,7 +294,7 @@
   $ hg tglogp
   @  7: 401ccec5e39f secret 'C'
   |
-  | @  6: a0b2430ebfb8 secret 'F'
+  | o  6: a0b2430ebfb8 secret 'F'
   | |
   o |  5: 45396c49d53b public 'B'
   | |
@@ -345,7 +345,7 @@
   $ hg tglogp
   @  7: 401ccec5e39f secret 'C'
   |
-  | @  6: a0b2430ebfb8 secret 'F'
+  | o  6: a0b2430ebfb8 secret 'F'
   | |
   o |  5: 45396c49d53b public 'B'
   | |
@@ -395,7 +395,7 @@
   $ hg tglogp
   @  7: 401ccec5e39f secret 'C'
   |
-  | @  6: a0b2430ebfb8 secret 'F'
+  | o  6: a0b2430ebfb8 secret 'F'
   | |
   o |  5: 45396c49d53b public 'B'
   | |
diff --git a/tests/test-rebase-conflicts.t b/tests/test-rebase-conflicts.t
--- a/tests/test-rebase-conflicts.t
+++ b/tests/test-rebase-conflicts.t
@@ -456,15 +456,14 @@
   warning: conflicts while merging conflict! (edit, then use 'hg resolve 
--mark')
   unresolved conflicts (see hg resolve, then hg rebase --continue)
   [1]
-The current parents are not 7 and 8 even though that's what we're merging
   $ hg tglog
   @  8:draft 'E'
   |
-  | o  7:draft 'D'
+  | @  7:draft 'D'
   |/
   o  6:draft 'C'
   |
-  | @5:draft 'F'
+  | %5:draft 'F'
   | |\
   | | o  4:draft 'E'
   | | |
diff --git a/tests/test-rebase-collapse.t b/tests/test-rebase-collapse.t
--- a/tests/test-rebase-collapse.t
+++ b/tests/test-rebase-collapse.t
@@ -712,7 +712,7 @@
   |
   | @  2: 82b8abf9c185 'D'
   | |
-  @ |  1: f899f3910ce7 'B'
+  % |  1: f899f3910ce7 'B'
   |/
   o  0: 4a2df7238c3b 'A'
   
@@ -736,7 +736,7 @@
   unresolved conflicts (see hg resolve, then hg rebase --continue)
   [1]
   $ hg tglog
-  @  3: 63668d570d21 'C'
+  %  3: 63668d570d21 'C'
   |
   | @  2: 82b8abf9c185 'D'
   | |
diff --git a/tests/test-rebase-abort.t b/tests/test-rebase-abort.t
--- a/tests/test-rebase-abort.t
+++ b/tests/test-rebase-abort.t
@@ -236,7 +236,7 @@
   [1]
 
   $ hg tglog
-  @  4:draft 'C1'
+  %  4:draft 'C1'
   |
   o  3:draft 'B bis'
   |
diff --git a/relnotes/next b/relnotes/next
--- a/relnotes/next
+++ b/relnotes/next
@@ -18,6 +18,11 @@
 
 == Backwards Compatibility Changes ==
 
+ * When `hg rebase` pauses for merge conflict resolution, the working
+   copy will no longer have the rebased node as a second parent. You
+   can use the new `conflictparents()` revset for finding the other
+   parent during a conflict.
+
 
 == Internal API Changes ==
 
diff --git a/hgext/rebase.py b/hgext/rebase.py
--- a/hgext/rebase.py
+++ b/hgext/rebase.py
@@ -618,6 +618,7 @@
 repo,
 rev,
 p1,
+p2,
 base,
 self.collapsef,
 dest,
@@ -642,10 +643,6 @@
  

D8041: revset: add a revset for parents in merge state

[martinvonz (Martin von Zweigbergk)]

martinvonz added inline comments.
martinvonz marked an inline comment as done.

INLINE COMMENTS

> martinvonz wrote in next:3
> Those both sound like good ideas. I'll send an update later today.

I've renamed them `conflictlocal()` and `conflictother()`. Hopefully that's 
clear enough and not too long.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D8041/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D8041

To: martinvonz, #hg-reviewers
Cc: marmoute, yuja, pulkit, mjpieters, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D8043: graphlog: use '%' for other context in merge conflict

[martinvonz (Martin von Zweigbergk)]

martinvonz updated this revision to Diff 19870.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D8043?vs=19723=19870

BRANCH
  default

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D8043/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D8043

AFFECTED FILES
  hgext/beautifygraph.py
  mercurial/hgweb/webutil.py
  mercurial/logcmdutil.py
  mercurial/templatekw.py
  relnotes/next
  tests/test-backout.t
  tests/test-graft-interrupted.t
  tests/test-rebase-collapse.t
  tests/test-strip.t
  tests/test-update-branches.t

CHANGE DETAILS

diff --git a/tests/test-update-branches.t b/tests/test-update-branches.t
--- a/tests/test-update-branches.t
+++ b/tests/test-update-branches.t
@@ -254,7 +254,7 @@
   |
   @  4:d047485b3896 0:60829823a42a  b1
   |
-  | o  3:6efa171f091b 1:0786582aa4b1
+  | %  3:6efa171f091b 1:0786582aa4b1
   | |
   | | o  2:bd10386d478c
   | |/
diff --git a/tests/test-strip.t b/tests/test-strip.t
--- a/tests/test-strip.t
+++ b/tests/test-strip.t
@@ -598,7 +598,7 @@
   |  date:Thu Jan 01 00:00:00 1970 +
   |  summary: b
   |
-  o  changeset:   0:9ab35a2d17cb
+  %  changeset:   0:9ab35a2d17cb
  user:test
  date:Thu Jan 01 00:00:00 1970 +
  summary: a
diff --git a/tests/test-rebase-collapse.t b/tests/test-rebase-collapse.t
--- a/tests/test-rebase-collapse.t
+++ b/tests/test-rebase-collapse.t
@@ -762,7 +762,7 @@
   abort: edit failed: false exited with status 1
   [255]
   $ hg tglog
-  o  3: 63668d570d21 'C'
+  %  3: 63668d570d21 'C'
   |
   | @  2: 82b8abf9c185 'D'
   | |
diff --git a/tests/test-graft-interrupted.t b/tests/test-graft-interrupted.t
--- a/tests/test-graft-interrupted.t
+++ b/tests/test-graft-interrupted.t
@@ -431,7 +431,7 @@
   $ hg log -GT "{rev}:{node|short} {desc}"
   @  6:6ec71c037d94 added x
   |
-  | o  5:36b793615f78 added foo to c
+  | %  5:36b793615f78 added foo to c
   | |
   | | o  4:863a25e1a9ea added x
   | |/
@@ -622,7 +622,7 @@
   $ hg log -GT "{rev}:{node|short} {desc}\n"
   @  4:2aa9ad1006ff B in file a
   |
-  | o  3:09e253b87e17 A in file a
+  | %  3:09e253b87e17 A in file a
   | |
   | o  2:d36c0562f908 c
   | |
@@ -669,7 +669,7 @@
   $ hg log -GT "{rev}:{node|short} {desc}\n"
   @  4:2aa9ad1006ff B in file a
   |
-  | o  3:09e253b87e17 A in file a
+  | %  3:09e253b87e17 A in file a
   | |
   | o  2:d36c0562f908 c
   | |
@@ -712,7 +712,7 @@
   $ hg log -GT "{rev}:{node|short} {desc}\n"
   @  4:2aa9ad1006ff B in file a
   |
-  | o  3:09e253b87e17 A in file a
+  | %  3:09e253b87e17 A in file a
   | |
   | o  2:d36c0562f908 c
   | |
diff --git a/tests/test-backout.t b/tests/test-backout.t
--- a/tests/test-backout.t
+++ b/tests/test-backout.t
@@ -103,7 +103,7 @@
   |  date:Thu Jan 01 00:00:02 1970 +
   |  summary: grapes
   |
-  o  changeset:   1:22cb4f70d813
+  %  changeset:   1:22cb4f70d813
   |  user:test
   |  date:Thu Jan 01 00:00:01 1970 +
   |  summary: chair
@@ -748,7 +748,7 @@
   |  date:Thu Jan 01 00:00:00 1970 +
   |  summary: capital three
   |
-  o  changeset:   0:a30dd8addae3
+  %  changeset:   0:a30dd8addae3
  user:test
  date:Thu Jan 01 00:00:00 1970 +
  summary: initial
diff --git a/relnotes/next b/relnotes/next
--- a/relnotes/next
+++ b/relnotes/next
@@ -1,5 +1,10 @@
 == New Features ==
 
+ * `hg log` now defaults to using an '%' symbol for commits involved
+in unresolved merge conflicts. That includes unresolved conflicts
+caused by e.g. `hg update --merge` and `hg graft`. '@' still takes
+precedence, so what used to be marked '@' still is.
+
  * New `conflictlocal()` and `conflictother()` revsets returns the
commits that are being merged, when there are conflicts. Also works
for conflicts caused by e.g. `hg graft`.
diff --git a/mercurial/templatekw.py b/mercurial/templatekw.py
--- a/mercurial/templatekw.py
+++ b/mercurial/templatekw.py
@@ -396,26 +396,38 @@
 return templateutil.compatfileslist(context, mapping, b'file', ctx.files())
 
 
-@templatekeyword(b'graphnode', requires={b'repo', b'ctx'})
+@templatekeyword(b'graphnode', requires={b'repo', b'ctx', b'cache'})
 def showgraphnode(context, mapping):
 """String. The character representing the changeset node in an ASCII
 revision graph."""
 repo = context.resource(mapping, b'repo')
 ctx = context.resource(mapping, b'ctx')
-return getgraphnode(repo, ctx)
+cache = context.resource(mapping, b'cache')
+return getgraphnode(repo, ctx, cache)
 
 
-def getgraphnode(repo, ctx):
-return getgraphnodecurrent(repo, ctx) or getgraphnodesymbol(ctx)
+def getgraphnode(repo, ctx, cache):
+return getgraphnodecurrent(repo, ctx, cache) or getgraphnodesymbol(ctx)
 
 
-def getgraphnodecurrent(repo, ctx):
+def getgraphnodecurrent(repo, ctx, cache):
 wpnodes = repo.dirstate.parents()
 if wpnodes[1] == nullid:
 wpnodes = wpnodes[:1]
 if 

D8041: revset: add a revset for parents in merge state

[martinvonz (Martin von Zweigbergk)]

martinvonz updated this revision to Diff 19869.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D8041?vs=19722=19869

BRANCH
  default

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D8041/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D8041

AFFECTED FILES
  mercurial/revset.py
  relnotes/next
  tests/test-merge4.t

CHANGE DETAILS

diff --git a/tests/test-merge4.t b/tests/test-merge4.t
--- a/tests/test-merge4.t
+++ b/tests/test-merge4.t
@@ -23,3 +23,37 @@
   abort: cannot commit merge with missing files
   [255]
 
+
+Test conflict*() revsets
+
+# Bad usage
+  $ hg log -r 'conflictlocal(foo)'
+  hg: parse error: conflictlocal takes no arguments
+  [255]
+  $ hg log -r 'conflictother(foo)'
+  hg: parse error: conflictother takes no arguments
+  [255]
+  $ hg co -C .
+  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
+# No merge parents when not merging
+  $ hg log -r 'conflictlocal() + conflictother()'
+# No merge parents when there is no conflict
+  $ hg merge 1
+  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  (branch merge, don't forget to commit)
+  $ hg log -r 'conflictlocal() + conflictother()'
+  $ hg co -C .
+  0 files updated, 0 files merged, 1 files removed, 0 files unresolved
+  $ echo conflict > b
+  $ hg ci -Aqm 'conflicting change to b'
+  $ hg merge 1
+  merging b
+  warning: conflicts while merging b! (edit, then use 'hg resolve --mark')
+  0 files updated, 0 files merged, 0 files removed, 1 files unresolved
+  use 'hg resolve' to retry unresolved file merges or 'hg merge --abort' to 
abandon
+  [1]
+# Shows merge parents when there is a conflict
+  $ hg log -r 'conflictlocal()' -T '{rev} {desc}\n'
+  3 conflicting change to b
+  $ hg log -r 'conflictother()' -T '{rev} {desc}\n'
+  1 commit #1
diff --git a/relnotes/next b/relnotes/next
--- a/relnotes/next
+++ b/relnotes/next
@@ -1,5 +1,9 @@
 == New Features ==
 
+ * New `conflictlocal()` and `conflictother()` revsets returns the
+   commits that are being merged, when there are conflicts. Also works
+   for conflicts caused by e.g. `hg graft`.
+
 
 == New Experimental Features ==
 
diff --git a/mercurial/revset.py b/mercurial/revset.py
--- a/mercurial/revset.py
+++ b/mercurial/revset.py
@@ -769,6 +769,38 @@
 return subset
 
 
+@predicate(b'conflictlocal()', safe=True)
+def conflictlocal(repo, subset, x):
+"""The local side of the merge, if currently in an unresolved merge.
+
+"merge" here includes merge conflicts from e.g. 'hg rebase' or 'hg graft'.
+"""
+getargs(x, 0, 0, _(b"conflictlocal takes no arguments"))
+from . import merge
+
+mergestate = merge.mergestate.read(repo)
+if mergestate.active() and repo.changelog.hasnode(mergestate.local):
+return subset & {repo.changelog.rev(mergestate.local)}
+
+return baseset()
+
+
+@predicate(b'conflictother()', safe=True)
+def conflictother(repo, subset, x):
+"""The other side of the merge, if currently in an unresolved merge.
+
+"merge" here includes merge conflicts from e.g. 'hg rebase' or 'hg graft'.
+"""
+getargs(x, 0, 0, _(b"conflictother takes no arguments"))
+from . import merge
+
+mergestate = merge.mergestate.read(repo)
+if mergestate.active() and repo.changelog.hasnode(mergestate.other):
+return subset & {repo.changelog.rev(mergestate.other)}
+
+return baseset()
+
+
 @predicate(b'contains(pattern)', weight=100)
 def contains(repo, subset, x):
 """The revision's manifest contains a file matching pattern (but might not



To: martinvonz, #hg-reviewers
Cc: marmoute, yuja, pulkit, mjpieters, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D7932: debugstripbackups: introduce command to interact with strip backups

[pulkit (Pulkit Goyal)]

pulkit edited the summary of this revision.
pulkit retitled this revision from "[RFC]debugbackups: introduce command to 
interact with strip backups" to "debugstripbackups: introduce command to 
interact with strip backups".
pulkit updated this revision to Diff 19868.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D7932?vs=19436=19868

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7932/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7932

AFFECTED FILES
  mercurial/debugcommands.py
  tests/test-debugstripbackup.t

CHANGE DETAILS

diff --git a/tests/test-debugstripbackup.t b/tests/test-debugstripbackup.t
new file mode 100644
--- /dev/null
+++ b/tests/test-debugstripbackup.t
@@ -0,0 +1,39 @@
+  $ cat >> $HGRCPATH << EOF
+  > [extensions]
+  > strip=
+  > EOF
+
+Setup repo
+
+  $ hg init repo
+  $ cd repo
+
+Test backups list and recover
+
+  $ hg debugstripbackups
+  no backup changesets found
+
+  $ mkcommit() {
+  >echo "$1" > "$1"
+  >hg add "$1"
+  >hg ci -l $1
+  > }
+  $ mkcommit a
+  $ mkcommit b
+  $ hg strip .
+  0 files updated, 0 files merged, 1 files removed, 0 files unresolved
+  saved backup bundle to 
$TESTTMP/repo/.hg/strip-backup/d2ae7f538514-2953539b-backup.hg (glob)
+  $ hg debugstripbackups
+  Recover changesets using: hg debugstripbackups --recover 
+  
+  Available backup changesets:
+  * (glob)
+  d2ae7f538514 b
+
+  $ hg debugstripbackups --recover d2ae7f538514
+  Unbundling d2ae7f538514
+  adding changesets
+  adding manifests
+  adding file changes
+  added 1 changesets with 1 changes to 1 files
+  new changesets d2ae7f538514 (1 drafts)
diff --git a/mercurial/debugcommands.py b/mercurial/debugcommands.py
--- a/mercurial/debugcommands.py
+++ b/mercurial/debugcommands.py
@@ -11,6 +11,7 @@
 import collections
 import difflib
 import errno
+import glob
 import operator
 import os
 import platform
@@ -39,6 +40,7 @@
 )
 from . import (
 bundle2,
+bundlerepo,
 changegroup,
 cmdutil,
 color,
@@ -130,6 +132,7 @@
 gen.apply(repo)
 
 
+
 @command(
 b'debugbuilddag',
 [
@@ -3361,6 +3364,131 @@
 
 
 @command(
+"debugstripbackups",
+[
+(
+"",
+"recover",
+"",
+"brings the specified changeset back into the repository",
+)
+]
++ cmdutil.logopts,
+_("hg debugstripbackups [--recover HASH]"),
+)
+def debugstripbackups(ui, repo, *pats, **opts):
+"""lists the changesets available in backup bundles
+
+Without any arguments, this command prints a list of the changesets in each
+backup bundle.
+
+--recover takes a changeset hash and unbundles the first bundle that
+contains that hash, which puts that changeset back in your repository.
+
+--verbose will print the entire commit message and the bundle path for that
+backup.
+"""
+backups = filter(os.path.isfile, glob.glob(repo.vfs.join("strip-backup") + 
"/*.hg"))
+backups.sort(key=lambda x: os.path.getmtime(x), reverse=True)
+
+opts["bundle"] = ""
+opts["force"] = None
+limit = logcmdutil.getlimit(opts)
+
+def display(other, chlist, displayer):
+if opts.get("newest_first"):
+chlist.reverse()
+count = 0
+for n in chlist:
+if limit is not None and count >= limit:
+break
+parents = [True for p in other.changelog.parents(n) if p != nullid]
+if opts.get("no_merges") and len(parents) == 2:
+continue
+count += 1
+displayer.show(other[n])
+
+recovernode = opts.get("recover")
+if recovernode:
+if scmutil.isrevsymbol(repo, recovernode):
+ui.warn(_("%s already exists in the repo\n") % recovernode)
+return
+elif backups:
+msg = _(
+"Recover changesets using: hg debugstripbackups --recover "
+"\n\nAvailable backup changesets:"
+)
+ui.status(msg, label="status.removed")
+else:
+ui.status(_("no backup changesets found\n"))
+return
+
+for backup in backups:
+# Much of this is copied from the hg incoming logic
+source = ui.expandpath(os.path.relpath(backup, encoding.getcwd()))
+source, branches = hg.parseurl(source, opts.get("branch"))
+try:
+other = hg.peer(repo, opts, source)
+except error.LookupError as ex:
+msg = _("\nwarning: unable to open bundle %s") % source
+hint = _("\n(missing parent rev %s)\n") % short(ex.name)
+ui.warn(msg, hint=hint)
+continue
+revs, checkout = hg.addbranchrevs(
+repo, other, branches, opts.get("rev")
+)
+
+if revs:
+revs = [other.lookup(rev) for rev in revs]
+
+quiet = ui.quiet
+try:
+ui.quiet = True
+other, chlist, cleanupfn = bundlerepo.getremotechanges(
+ui, 

D8051: worker: Use buffered input from the pickle stream

[yuja (Yuya Nishihara)]

yuja added a comment.


  >   for rfd, wfd in pipes:
  >   os.close(wfd)
  >
  > - selector.register(os.fdopen(rfd, 'rb', 0), selectors.EVENT_READ)
  >
  > +selector.register(os.fdopen(rfd, 'rb'), selectors.EVENT_READ)
  
  Using buffered I/O can cause a deadlock (until the worker process exits.)
  The master process expects EVENT_READ will be asserted (i.e. level-triggered)
  if there are more than one readable items, but buffered file won't
  since almost all readable items will be moved to its internal buffer.
  
import time
from mercurial import (
ui as uimod,
worker,
)

def some_work(n):
# send back many items at once
for i in range(10):
yield (n, i)
# and don't close() the pipe for a while
time.sleep(10)

ui = uimod.ui()
ui.setconfig(b'worker', b'numcpus', b'2')
gen = worker._posixworker(ui, some_work, staticargs=(), args=[0, 1],
  hasretval=False)
for x in gen:
ui.write(b'%r\n' % (x,))
ui.flush()

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D8051/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D8051

To: heftig, #hg-reviewers
Cc: yuja, sheehan, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: D8051: worker: Use buffered input from the pickle stream

[Yuya Nishihara]

>  for rfd, wfd in pipes:
>  os.close(wfd)
> -selector.register(os.fdopen(rfd, 'rb', 0), selectors.EVENT_READ)
> +selector.register(os.fdopen(rfd, 'rb'), selectors.EVENT_READ)

Using buffered I/O can cause a deadlock (until the worker process exits.)
The master process expects EVENT_READ will be asserted (i.e. level-triggered)
if there are more than one readable items, but buffered file won't
since almost all readable items will be moved to its internal buffer.

```
import time
from mercurial import (
ui as uimod,
worker,
)

def some_work(n):
# send back many items at once
for i in range(10):
yield (n, i)
# and don't close() the pipe for a while
time.sleep(10)

ui = uimod.ui()
ui.setconfig(b'worker', b'numcpus', b'2')
gen = worker._posixworker(ui, some_work, staticargs=(), args=[0, 1],
  hasretval=False)
for x in gen:
ui.write(b'%r\n' % (x,))
ui.flush()
```
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D7730: rebase: make sure pruning does not confuse rebase (issue6180)

[pulkit (Pulkit Goyal)]

pulkit added a comment.


  In D7730#117268 , @marmoute 
wrote:
  
  > I think is would be simpler and sfare to prevent unrelated operation during 
rebase. If the user cannot prune here we won't have to deal with it. This woudl 
also apply to other operation that can alter the repository, like another 
rebase, amend or a pull. Starting using a unified and safe approach seems to 
provide more benefit with less chance of UI inconsistency.
  
  I agree. We should disallow prune if an unfinished operation exists.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7730/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7730

To: khanchi97, martinvonz, #hg-reviewers
Cc: marmoute, pulkit, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel