date:20220127

D12096: stream-clone: allow to change persistent-nodemap format during stream clone

2022-01-27 Thread marmoute (Pierre-Yves David)

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  Persistent nodemap affect the store format. However it is fairly isolated and
  fast to general locally. So not making it a fixed part of the stream clone is
  useful.
  
  This allow clients without persistent-nodemap support (default for client
  without Rust enabled, or simply older client). So it make it possible to 
enable
  persistent nodemap on client, where it can provide a massive boost. without 
too
  much consequence.
  
  To do so, we stop using it in the advertisement requirements for streaming and
  let the client add/remove the necessary file depending of its configuration.
  
  We still send the files as it seems like a small save to not regenerate them.
  In addition, the way we match them will overlap with the 
changelog-v2/revlog-v2
  so we can't simply skip the associated patterns.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D12096

AFFECTED FILES
  mercurial/requirements.py
  mercurial/revlogutils/nodemap.py
  mercurial/streamclone.py
  tests/test-bundle.t
  tests/test-clone-stream-format.t
  tests/test-clone-stream.t
  tests/test-clonebundles.t
  tests/test-debugcommands.t
  tests/test-stream-bundle-v2.t

CHANGE DETAILS

diff --git a/tests/test-stream-bundle-v2.t b/tests/test-stream-bundle-v2.t
--- a/tests/test-stream-bundle-v2.t
+++ b/tests/test-stream-bundle-v2.t
@@ -47,11 +47,11 @@
   Stream params: {}
   stream2 -- {bytecount: 1693, filecount: 11, requirements: 
generaldelta%2Crevlogv1%2Csparserevlog} (mandatory: True) (no-zstd !)
   stream2 -- {bytecount: 1693, filecount: 11, requirements: 
generaldelta%2Crevlog-compression-zstd%2Crevlogv1%2Csparserevlog} (mandatory: 
True) (zstd no-rust !)
-  stream2 -- {bytecount: 1693, filecount: 11, requirements: 
generaldelta%2Cpersistent-nodemap%2Crevlog-compression-zstd%2Crevlogv1%2Csparserevlog}
 (mandatory: True) (rust !)
+  stream2 -- {bytecount: 1693, filecount: 11, requirements: 
generaldelta%2Crevlog-compression-zstd%2Crevlogv1%2Csparserevlog} (mandatory: 
True) (rust !)
   $ hg debugbundle --spec bundle.hg
   none-v2;stream=v2;requirements%3Dgeneraldelta%2Crevlogv1%2Csparserevlog 
(no-zstd !)
   
none-v2;stream=v2;requirements%3Dgeneraldelta%2Crevlog-compression-zstd%2Crevlogv1%2Csparserevlog
 (zstd no-rust !)
-  
none-v2;stream=v2;requirements%3Dgeneraldelta%2Cpersistent-nodemap%2Crevlog-compression-zstd%2Crevlogv1%2Csparserevlog
 (rust !)
+  
none-v2;stream=v2;requirements%3Dgeneraldelta%2Crevlog-compression-zstd%2Crevlogv1%2Csparserevlog
 (rust !)
 
 Test that we can apply the bundle as a stream clone bundle
 
diff --git a/tests/test-debugcommands.t b/tests/test-debugcommands.t
--- a/tests/test-debugcommands.t
+++ b/tests/test-debugcommands.t
@@ -657,8 +657,8 @@
   devel-peer-request:   pairs: 81 bytes
   sending hello command
   sending between command
-  remote: 487
-  remote: capabilities: batch branchmap $USUAL_BUNDLE2_CAPS$ changegroupsubset 
getbundle known lookup protocaps pushkey 
streamreqs=generaldelta,persistent-nodemap,revlog-compression-zstd,revlogv1,sparserevlog
 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
+  remote: 468
+  remote: capabilities: batch branchmap $USUAL_BUNDLE2_CAPS$ changegroupsubset 
getbundle known lookup protocaps pushkey 
streamreqs=generaldelta,revlog-compression-zstd,revlogv1,sparserevlog 
unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
   remote: 1
   devel-peer-request: protocaps
   devel-peer-request:   caps: * bytes (glob)
diff --git a/tests/test-clonebundles.t b/tests/test-clonebundles.t
--- a/tests/test-clonebundles.t
+++ b/tests/test-clonebundles.t
@@ -281,7 +281,7 @@
   writing 613 bytes for 4 files
   bundle requirements: generaldelta, revlogv1, sparserevlog (no-rust no-zstd !)
   bundle requirements: generaldelta, revlog-compression-zstd, revlogv1, 
sparserevlog (no-rust zstd !)
-  bundle requirements: generaldelta, persistent-nodemap, 
revlog-compression-zstd, revlogv1, sparserevlog (rust !)
+  bundle requirements: generaldelta, revlog-compression-zstd, revlogv1, 
sparserevlog (rust !)
 
 No bundle spec should work
 
diff --git a/tests/test-clone-stream.t b/tests/test-clone-stream.t
--- a/tests/test-clone-stream.t
+++ b/tests/test-clone-stream.t
@@ -338,23 +338,23 @@
 #endif
 #if zstd rust no-dirstate-v2
   $ f --size --hex --bytes 256 body
-  body: size=116331
+  body: size=116310
   : 04 6e 6f 6e 65 48 47 32 30 00 00 00 00 00 00 00 |.noneHG20...|
-  0010: 91 07 53 54 52 45 41 4d 32 00 00 00 00 03 00 09 |..STREAM2...|
-  0020: 06 09 04 0c 55 62 79 74 65 63 6f 75 6e 74 31 30 |Ubytecount10|
+  0010: 7c 07 53 54 52 45 41 4d 32 00 00 00 00 03 00 09 ||.STREAM2...|
+  0020: 06 09 04 0c 40 62 79 74 65 63 6f 75 6e 74 31 30 |@bytecount10|
   0030: 31 32 37 36 66 69 6c 65 63 6f 75 6e 74 31 30 39 |1276filecount109|
   0040: 33 72 65 71 75 69 72 65 6d 65 6e 74 73 67 65 6e |3requirementsgen|

D12094: debugbuilddag: add a flag to allow running it from a non-empty repository

2022-01-27 Thread marmoute (Pierre-Yves David)

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  Allow that by default seems "dangerous", but having a flag to make it possible
  will be useful to help building some repository incrementally. The newly 
introduced support is basic, but already useful.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D12094

AFFECTED FILES
  mercurial/debugcommands.py
  tests/test-completion.t

CHANGE DETAILS

diff --git a/tests/test-completion.t b/tests/test-completion.t
--- a/tests/test-completion.t
+++ b/tests/test-completion.t
@@ -272,7 +272,7 @@
   debugantivirusrunning: 
   debugapplystreamclonebundle: 
   debugbackupbundle: recover, patch, git, limit, no-merges, stat, graph, 
style, template
-  debugbuilddag: mergeable-file, overwritten-file, new-file
+  debugbuilddag: mergeable-file, overwritten-file, new-file, from-existing
   debugbundle: all, part-type, spec
   debugcapabilities: 
   debugchangedfiles: compute
diff --git a/mercurial/debugcommands.py b/mercurial/debugcommands.py
--- a/mercurial/debugcommands.py
+++ b/mercurial/debugcommands.py
@@ -178,6 +178,12 @@
 _(b'add single file all revs overwrite'),
 ),
 (b'n', b'new-file', None, _(b'add new file at each rev')),
+(
+b'',
+b'from-existing',
+None,
+_(b'continue from a non-empty repository'),
+),
 ],
 _(b'[OPTION]... [TEXT]'),
 )
@@ -188,6 +194,7 @@
 mergeable_file=False,
 overwritten_file=False,
 new_file=False,
+from_existing=False,
 ):
 """builds a repo with a given DAG from scratch in the current empty repo
 
@@ -226,7 +233,7 @@
 text = ui.fin.read()
 
 cl = repo.changelog
-if len(cl) > 0:
+if len(cl) > 0 and not from_existing:
 raise error.Abort(_(b'repository is not empty'))
 
 # determine number of revs in DAG



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D12095: stream-clone: add 5000 changesets to test-clone-stream-format

2022-01-27 Thread marmoute (Pierre-Yves David)

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This make sure the changelog is not inlined. We needs this to test coming
  behavior change around persistent-nodemap.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D12095

AFFECTED FILES
  tests/test-clone-stream-format.t

CHANGE DETAILS

diff --git a/tests/test-clone-stream-format.t b/tests/test-clone-stream-format.t
--- a/tests/test-clone-stream-format.t
+++ b/tests/test-clone-stream-format.t
@@ -79,6 +79,12 @@
   adding undo.i
   adding undo.n
   adding undo.py
+  $ hg debugbuilddag .+5000 --from-existing
+  $ ls -1 .hg/store/00changelog*
+  .hg/store/00changelog-*.nd (glob)
+  .hg/store/00changelog.d
+  .hg/store/00changelog.i
+  .hg/store/00changelog.n
   $ cd ..
 
 
@@ -90,16 +96,16 @@
   adding changesets
   adding manifests
   adding file changes
-  added 3 changesets with 1088 changes to 1088 files
-  new changesets 96ee1d7354c4:5223b5e3265f
+  added 5004 changesets with 1088 changes to 1088 files (+1 heads)
+  new changesets 96ee1d7354c4:06ddac466af5
   updating to branch default
-  1088 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  0 files updated, 0 files merged, 0 files removed, 0 files unresolved
   $ hg verify -R server-no-store
   checking changesets
   checking manifests
   crosschecking files in changesets and manifests
   checking files
-  checked 3 changesets with 1088 changes to 1088 files
+  checked 5004 changesets with 1088 changes to 1088 files
   $ hg -R server serve -p $HGPORT -d --pid-file=hg-1.pid --error errors-1.txt
   $ cat hg-1.pid > $DAEMON_PIDS
   $ hg -R server-no-store serve -p $HGPORT2 -d --pid-file=hg-2.pid --error 
errors-2.txt
@@ -118,7 +124,7 @@
   checking manifests
   crosschecking files in changesets and manifests
   checking files
-  checked 3 changesets with 1088 changes to 1088 files
+  checked 5004 changesets with 1088 changes to 1088 files
   $ hg debugrequires -R clone-remove-store | grep store
   [1]
 
@@ -132,7 +138,7 @@
   checking manifests
   crosschecking files in changesets and manifests
   checking files
-  checked 3 changesets with 1088 changes to 1088 files
+  checked 5004 changesets with 1088 changes to 1088 files
   $ hg debugrequires -R clone-add-store | grep store
   store
 
@@ -149,16 +155,16 @@
   adding changesets
   adding manifests
   adding file changes
-  added 3 changesets with 1088 changes to 1088 files
-  new changesets 96ee1d7354c4:5223b5e3265f
+  added 5004 changesets with 1088 changes to 1088 files (+1 heads)
+  new changesets 96ee1d7354c4:06ddac466af5
   updating to branch default
-  1088 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  0 files updated, 0 files merged, 0 files removed, 0 files unresolved
   $ hg verify -R server-no-fncache
   checking changesets
   checking manifests
   crosschecking files in changesets and manifests
   checking files
-  checked 3 changesets with 1088 changes to 1088 files
+  checked 5004 changesets with 1088 changes to 1088 files
   $ hg -R server serve -p $HGPORT -d --pid-file=hg-1.pid --error errors-1.txt
   $ cat hg-1.pid > $DAEMON_PIDS
   $ hg -R server-no-fncache serve -p $HGPORT2 -d --pid-file=hg-2.pid --error 
errors-2.txt
@@ -177,7 +183,7 @@
   checking manifests
   crosschecking files in changesets and manifests
   checking files
-  checked 3 changesets with 1088 changes to 1088 files
+  checked 5004 changesets with 1088 changes to 1088 files
   $ hg debugrequires -R clone-remove-fncache | grep fncache
   [1]
 
@@ -191,7 +197,7 @@
   checking manifests
   crosschecking files in changesets and manifests
   checking files
-  checked 3 changesets with 1088 changes to 1088 files
+  checked 5004 changesets with 1088 changes to 1088 files
   $ hg debugrequires -R clone-add-fncache | grep fncache
   fncache
 
@@ -209,16 +215,16 @@
   adding changesets
   adding manifests
   adding file changes
-  added 3 changesets with 1088 changes to 1088 files
-  new changesets 96ee1d7354c4:5223b5e3265f
+  added 5004 changesets with 1088 changes to 1088 files (+1 heads)
+  new changesets 96ee1d7354c4:06ddac466af5
   updating to branch default
-  1088 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  0 files updated, 0 files merged, 0 files removed, 0 files unresolved
   $ hg verify -R server-no-dotencode
   checking changesets
   checking manifests
   crosschecking files in changesets and manifests
   checking files
-  checked 3 changesets with 1088 changes to 1088 files
+  checked 5004 changesets with 1088 changes to 1088 files
   $ hg -R server serve -p $HGPORT -d --pid-file=hg-1.pid --error errors-1.txt
   $ cat hg-1.pid > $DAEMON_PIDS
   $ hg -R server-no-dotencode serve -p $HGPORT2 -d --pid-file=hg-2.pid --error 
errors-2.txt
@@ -237,7 +243,7 @@
   checking manifests
   crosschecking files in changesets and manifests
   checking files
-  checked 3

D12093: persistent-nodemap: properly delete all nodemap files on downgrade

2022-01-27 Thread marmoute (Pierre-Yves David)

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  Previously, we were only deleting the docket file…

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D12093

AFFECTED FILES
  mercurial/revlogutils/nodemap.py
  tests/test-persistent-nodemap.t

CHANGE DETAILS

diff --git a/tests/test-persistent-nodemap.t b/tests/test-persistent-nodemap.t
--- a/tests/test-persistent-nodemap.t
+++ b/tests/test-persistent-nodemap.t
@@ -809,8 +809,6 @@
 - manifest
   
   $ ls -1 .hg/store/ | egrep '00(changelog|manifest)(\.n|-.*\.nd)'
-  00changelog-*.nd (glob)
-  00manifest-*.nd (glob)
   undo.backup.00changelog.n
   undo.backup.00manifest.n
   $ hg debugnodemap --metadata
diff --git a/mercurial/revlogutils/nodemap.py b/mercurial/revlogutils/nodemap.py
--- a/mercurial/revlogutils/nodemap.py
+++ b/mercurial/revlogutils/nodemap.py
@@ -144,10 +144,12 @@
 
 def delete_nodemap(tr, repo, revlog):
 """Delete nodemap data on disk for a given revlog"""
-if revlog._nodemap_file is None:
-msg = "calling persist nodemap on a revlog without the feature enabled"
-raise error.ProgrammingError(msg)
-repo.svfs.tryunlink(revlog._nodemap_file)
+prefix = revlog.radix
+pattern = re.compile(br"(^|/)%s(-[0-9a-f]+\.nd|\.n(\.a)?)$" % prefix)
+dirpath = revlog.opener.dirname(revlog._indexfile)
+for f in revlog.opener.listdir(dirpath):
+if pattern.match(f):
+repo.svfs.tryunlink(f)
 
 
 def persist_nodemap(tr, revlog, pending=False, force=False):



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: hgweb: access to `rawenv` (or `$REQUEST_URI`) within a template

2022-01-27 Thread Norman Gray


Greetings.

I've just realised that, although I planned to subscribe to the general 
mercurial list, I seem (rather stupidly) to have accidentally subscribed to the 
developers' list instead, where my how-to-use question might be seen as out of 
scope.  Apologies for that: would the question indeed be better in scope on the 
mercurial@ list?

Best wishes,

Norman


-- 
Norman Gray  :  https://nxg.me.uk
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D12092: interfaces: add missing caches kwarg of localrepo.updatecaches()

2022-01-27 Thread av6 (Anton Shestakov)

av6 created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  Added to localrepo: in 1337bfaa88ca 
, 
added to the interface: now.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D12092

AFFECTED FILES
  mercurial/interfaces/repository.py

CHANGE DETAILS

diff --git a/mercurial/interfaces/repository.py 
b/mercurial/interfaces/repository.py
--- a/mercurial/interfaces/repository.py
+++ b/mercurial/interfaces/repository.py
@@ -1787,7 +1787,7 @@
 DANGEROUS.
 """
 
-def updatecaches(tr=None, full=False):
+def updatecaches(tr=None, full=False, caches=None):
 """Warm repo caches."""
 
 def invalidatecaches():



To: av6, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D12091: stream-clone: filter possible missing requirements using all supported one

2022-01-27 Thread marmoute (Pierre-Yves David)

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  The `supportedformat` requirements is missing some important requirements and 
it
  seems better to filter out with all requirements we know, not just an
  "arbitrary" subset.
  
  The `supportedformat` set is lacking some important requirements (for example
  `revlog-compression-zstd`). This is getting fixed on default (for Mercurial 
6.1)
  
  However, fixing that in 6.1 means the stream requirements sent over the wire
  will contains more items. And if we don't apply this fix on older version, 
they
  might end up complaining about lacking support for feature they actually 
support
  for years.
  
  This patch does not fix the deeper problem (advertised stream requirement
  lacking some of them), but focus on the trivial part : Lets use the full set 
of
  supported requirement for looking for unsupported ones.
  
  This patch should be simple to backport to older version of Mercurial and
  packager should be encouraged to do so.

REPOSITORY
  rHG Mercurial

BRANCH
  stable

REVISION DETAIL
  https://phab.mercurial-scm.org/D12091

AFFECTED FILES
  mercurial/streamclone.py
  tests/test-http-bundle1.t
  tests/test-http.t

CHANGE DETAILS

diff --git a/tests/test-http.t b/tests/test-http.t
--- a/tests/test-http.t
+++ b/tests/test-http.t
@@ -59,8 +59,10 @@
 
   $ cat > $TESTTMP/removesupportedformat.py << EOF
   > from mercurial import localrepo
-  > def extsetup(ui):
-  > localrepo.localrepository.supportedformats.remove(b'generaldelta')
+  > def reposetup(ui, repo):
+  > local = repo.local()
+  > if local is not None:
+  > local.supported.remove(b'generaldelta')
   > EOF
 
   $ hg clone --config extensions.rsf=$TESTTMP/removesupportedformat.py 
--stream http://localhost:$HGPORT/ copy3
diff --git a/tests/test-http-bundle1.t b/tests/test-http-bundle1.t
--- a/tests/test-http-bundle1.t
+++ b/tests/test-http-bundle1.t
@@ -70,8 +70,10 @@
 
   $ cat > $TESTTMP/removesupportedformat.py << EOF
   > from mercurial import localrepo
-  > def extsetup(ui):
-  > localrepo.localrepository.supportedformats.remove(b'generaldelta')
+  > def reposetup(ui, repo):
+  > local = repo.local()
+  > if local is not None:
+  > local.supported.remove(b'generaldelta')
   > EOF
 
   $ hg clone --config extensions.rsf=$TESTTMP/removesupportedformat.py 
--stream http://localhost:$HGPORT/ copy3
diff --git a/mercurial/streamclone.py b/mercurial/streamclone.py
--- a/mercurial/streamclone.py
+++ b/mercurial/streamclone.py
@@ -104,7 +104,7 @@
 
 streamreqs = set(streamreqs.split(b','))
 # Server requires something we don't support. Bail.
-missingreqs = streamreqs - repo.supportedformats
+missingreqs = streamreqs - repo.supported
 if missingreqs:
 pullop.repo.ui.warn(
 _(
@@ -481,7 +481,7 @@
 )
 
 filecount, bytecount, requirements = readbundle1header(fp)
-missingreqs = requirements - repo.supportedformats
+missingreqs = requirements - repo.supported
 if missingreqs:
 raise error.Abort(
 _(b'unable to apply stream clone: unsupported format: %s')



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D12096: stream-clone: allow to change persistent-nodemap format during stream clone

D12094: debugbuilddag: add a flag to allow running it from a non-empty repository

D12095: stream-clone: add 5000 changesets to test-clone-stream-format

D12093: persistent-nodemap: properly delete all nodemap files on downgrade

Re: hgweb: access to `rawenv` (or `$REQUEST_URI`) within a template

D12092: interfaces: add missing caches kwarg of localrepo.updatecaches()

D12091: stream-clone: filter possible missing requirements using all supported one

7 matches

Site Navigation

Mail list logo

Footer information