D987: copies: add a config to limit the number of candidates to check in heuristics

[stash (Stanislau Hlebik)]

stash added a comment.


  @pulkit sorry for being late with the comments.
  I agree with @yuja  - falling back is not an option, because it's probably 
going to be slow.
  As for these two options
  
  > - do not try to find copy from the first n candidates, and show the message 
saying copy tracing is disabled due to too many candidates,
  > - or show the message only when copy cannot be detected.
  
  Do I understand correctly that
  
  > copy tracing is disabled
  
  means that it's disabled only for this particular file and not for all files?
  
  If yes, then I think that either option is fine.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D987

To: pulkit, #hg-reviewers, yuja
Cc: stash, yuja, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D358: copytrace: move fb extension to core under flag experimental.fastcopytrace

[stash (Stanislau Hlebik)]

stash added a comment.


  Quick heads-up: one of our bootcampers works on the task that does the 
following: if rebase/merge/graft happens only on the local stack (i.e. base 
commit is draft), then we fallback to the normal copytrace, because we are sure 
it's going to be fast enough

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D358

To: pulkit, #hg-reviewers
Cc: quark, stash, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH V2] selectivepull: fix tests

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1496247075 25200
#  Wed May 31 09:11:15 2017 -0700
# Node ID 8728c97e46330e67bcf5237686d3078c9eb83d03
# Parent  900f1c70436bd7a266f64a6db592af602f8138a5
selectivepull: fix tests

There was a change upstream bce5ebe728594150db39a8e1744b217bb62862c0 that
added a warning. --force makes this warning go away.

diff --git a/tests/test-selective-pull.t b/tests/test-selective-pull.t
--- a/tests/test-selective-pull.t
+++ b/tests/test-selective-pull.t
@@ -100,7 +100,7 @@
   $ hg clone -q ssh://user@dummy/remoterepo secondremoterepo
   $ cd secondremoterepo
   $ hg up -q 0238718db2b1
-  $ hg book master
+  $ hg book master --force
   $ cd ..
 
 Add second remote repo path in localrepo
@@ -119,7 +119,7 @@
 
 Move bookmark in second remote, pull and make sure it doesn't move in local 
repo
   $ cd ../secondremoterepo
-  $ hg book secondbook
+  $ hg book secondbook --force
   $ echo aaa >> a
   $ hg commit -m 'Move bookmark in second remote'
   $ cd ../localrepo
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH remotenames-ext] selectivepull: fix tests

[Stanislau Hlebik]

Excerpts from Sean Farley's message of 2017-05-30 17:23:11 -0700:
> Stanislau Hlebik <st...@fb.com> writes:
> 
> > # HG changeset patch
> > # User Stanislau Hlebik <st...@fb.com>
> > # Date 1495731588 25200
> > #  Thu May 25 09:59:48 2017 -0700
> > # Node ID 78c8966c02759755235a6566e0c557b82ff9cb0a
> > # Parent  900f1c70436bd7a266f64a6db592af602f8138a5
> > selectivepull: fix tests
> 
> Can you provide a bit more of what's going on? Is this due to Aguie's
> change about unambiguous matching?

Yes, that's because of this change. I'll send V2 with update message
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 5 of 7] copies: remove msrc and mdst parameters

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1496062645 25200
#  Mon May 29 05:57:25 2017 -0700
# Node ID e460d9165b77c5333a0288bd423175ad9ac4519a
# Parent  cfdf3dc92e2c9415474cef9b004a002e9b0bc7ff
copies: remove msrc and mdst parameters

This function already has lots of parameters. And we can get manifests from
contexts. So let's get msrc and mdst parameters from srcctx and dstctx.

diff --git a/mercurial/copies.py b/mercurial/copies.py
--- a/mercurial/copies.py
+++ b/mercurial/copies.py
@@ -414,10 +414,10 @@
   baselabel='topological common ancestor')
 
 for f in u1u:
-_checkcopies(c1, c2, f, m1, m2, base, tca, dirtyc1, limit, data1)
+_checkcopies(c1, c2, f, base, tca, dirtyc1, limit, data1)
 
 for f in u2u:
-_checkcopies(c2, c1, f, m2, m1, base, tca, dirtyc2, limit, data2)
+_checkcopies(c2, c1, f, base, tca, dirtyc2, limit, data2)
 
 copy = dict(data1['copy'].items() + data2['copy'].items())
 fullcopy = dict(data1['fullcopy'].items() + data2['fullcopy'].items())
@@ -462,8 +462,8 @@
  'incompletediverge': bothincompletediverge
 }
 for f in bothnew:
-_checkcopies(c1, c2, f, m1, m2, base, tca, dirtyc1, limit, both1)
-_checkcopies(c2, c1, f, m2, m1, base, tca, dirtyc2, limit, both2)
+_checkcopies(c1, c2, f, base, tca, dirtyc1, limit, both1)
+_checkcopies(c2, c1, f, base, tca, dirtyc2, limit, both2)
 if dirtyc1:
 # incomplete copies may only be found on the "dirty" side for bothnew
 assert not both2['incomplete']
@@ -598,16 +598,13 @@
 except StopIteration:
 return False
 
-def _checkcopies(srcctx, dstctx, f, msrc, mdst, base, tca, remotebase,
- limit, data):
+def _checkcopies(srcctx, dstctx, f, base, tca, remotebase, limit, data):
 """
 check possible copies of f from msrc to mdst
 
 srcctx = starting context for f in msrc
 dstctx = destination context for f in mdst
 f = the filename to check (as in msrc)
-msrc = the source manifest
-mdst = the destination manifest
 base = the changectx used as a merge base
 tca = topological common ancestor for graft-like scenarios
 remotebase = True if base is outside tca::srcctx, False otherwise
@@ -620,6 +617,8 @@
 once it "goes behind a certain revision".
 """
 
+msrc = srcctx.manifest()
+mdst = dstctx.manifest()
 mb = base.manifest()
 mta = tca.manifest()
 # Might be true if this call is about finding backward renames,
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 3 of 7] copies: rename ctx to srcctx

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1496062577 25200
#  Mon May 29 05:56:17 2017 -0700
# Node ID cff8c67aae48c7ee7c07b0878e15fb5aa54aed8c
# Parent  2cf8c7e99a33dce384b49d3fd0fc4c9512ee94b5
copies: rename ctx to srcctx

In the next diff we'll pass new dstctx parameter. Let's rename ctx to srcctx in
this patch.

diff --git a/mercurial/copies.py b/mercurial/copies.py
--- a/mercurial/copies.py
+++ b/mercurial/copies.py
@@ -598,17 +598,17 @@
 except StopIteration:
 return False
 
-def _checkcopies(ctx, f, msrc, mdst, base, tca, remotebase, limit, data):
+def _checkcopies(srcctx, f, msrc, mdst, base, tca, remotebase, limit, data):
 """
 check possible copies of f from msrc to mdst
 
-ctx = starting context for f in srcm
+srcctx = starting context for f in msrc
 f = the filename to check (as in msrc)
 msrc = the source manifest
 mdst = the destination manifest
 base = the changectx used as a merge base
 tca = topological common ancestor for graft-like scenarios
-remotebase = True if base is outside tca::ctx, False otherwise
+remotebase = True if base is outside tca::srcctx, False otherwise
 limit = the rev number to not search beyond
 data = dictionary of dictionary to store copy data. (see mergecopies)
 
@@ -630,7 +630,7 @@
 # the base) this is more complicated as we must detect a divergence.
 # We use 'backwards = False' in that case.
 backwards = not remotebase and base != tca and f in mb
-getfctx = _makegetfctx(ctx)
+getfctx = _makegetfctx(srcctx)
 
 if msrc[f] == mb.get(f) and not remotebase:
 # Nothing to merge
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 7 of 7] copies: introduce getdstfctx

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1496063173 25200
#  Mon May 29 06:06:13 2017 -0700
# Node ID 337ac332b0a1260583c82a3bf346a304a9245ff1
# Parent  216a6a72c032ab10e8cc8dc6f916593508dc0a45
copies: introduce getdstfctx

Previously `c2` may had an incorrect linkrev because getsrcfctx set wrong
_descendantrev. getsrcfctx() sets descendant rev equals to srcctx.rev() (see
_makegetfctx()), but for `c2` descendant rev should be dstctx. While we were
lucky it didn't broke copytracing it made it significantly slower in some
cases. Besides it broke some external extensions, for example remotefilelog.

diff --git a/mercurial/copies.py b/mercurial/copies.py
--- a/mercurial/copies.py
+++ b/mercurial/copies.py
@@ -632,6 +632,7 @@
 # We use 'backwards = False' in that case.
 backwards = not remotebase and base != tca and f in mb
 getsrcfctx = _makegetfctx(srcctx)
+getdstfctx = _makegetfctx(dstctx)
 
 if msrc[f] == mb.get(f) and not remotebase:
 # Nothing to merge
@@ -658,7 +659,7 @@
 continue # no match, keep looking
 if mdst[of] == mb.get(of):
 return # no merge needed, quit early
-c2 = getsrcfctx(of, mdst[of])
+c2 = getdstfctx(of, mdst[of])
 # c2 might be a plain new file on added on destination side that is
 # unrelated to the droids we are looking for.
 cr = _related(oc, c2, tca.rev())
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 4 of 7] copies: add dstctx parameter

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1496062623 25200
#  Mon May 29 05:57:03 2017 -0700
# Node ID cfdf3dc92e2c9415474cef9b004a002e9b0bc7ff
# Parent  cff8c67aae48c7ee7c07b0878e15fb5aa54aed8c
copies: add dstctx parameter

Add parameter with destination context

diff --git a/mercurial/copies.py b/mercurial/copies.py
--- a/mercurial/copies.py
+++ b/mercurial/copies.py
@@ -414,10 +414,10 @@
   baselabel='topological common ancestor')
 
 for f in u1u:
-_checkcopies(c1, f, m1, m2, base, tca, dirtyc1, limit, data1)
+_checkcopies(c1, c2, f, m1, m2, base, tca, dirtyc1, limit, data1)
 
 for f in u2u:
-_checkcopies(c2, f, m2, m1, base, tca, dirtyc2, limit, data2)
+_checkcopies(c2, c1, f, m2, m1, base, tca, dirtyc2, limit, data2)
 
 copy = dict(data1['copy'].items() + data2['copy'].items())
 fullcopy = dict(data1['fullcopy'].items() + data2['fullcopy'].items())
@@ -462,8 +462,8 @@
  'incompletediverge': bothincompletediverge
 }
 for f in bothnew:
-_checkcopies(c1, f, m1, m2, base, tca, dirtyc1, limit, both1)
-_checkcopies(c2, f, m2, m1, base, tca, dirtyc2, limit, both2)
+_checkcopies(c1, c2, f, m1, m2, base, tca, dirtyc1, limit, both1)
+_checkcopies(c2, c1, f, m2, m1, base, tca, dirtyc2, limit, both2)
 if dirtyc1:
 # incomplete copies may only be found on the "dirty" side for bothnew
 assert not both2['incomplete']
@@ -598,11 +598,13 @@
 except StopIteration:
 return False
 
-def _checkcopies(srcctx, f, msrc, mdst, base, tca, remotebase, limit, data):
+def _checkcopies(srcctx, dstctx, f, msrc, mdst, base, tca, remotebase,
+ limit, data):
 """
 check possible copies of f from msrc to mdst
 
 srcctx = starting context for f in msrc
+dstctx = destination context for f in mdst
 f = the filename to check (as in msrc)
 msrc = the source manifest
 mdst = the destination manifest
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 1 of 7] copies: rename m1 to msrc

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1496062335 25200
#  Mon May 29 05:52:15 2017 -0700
# Node ID 75f0d8342109877a0dd620df3f566d0f080a9138
# Parent  c2b7fb580794ccbe21fa8c47f493eff6e9431fee
copies: rename m1 to msrc

Small refactoring that renames `m1` parameter name to a more clearer name
`msrc`.

diff --git a/mercurial/copies.py b/mercurial/copies.py
--- a/mercurial/copies.py
+++ b/mercurial/copies.py
@@ -598,13 +598,13 @@
 except StopIteration:
 return False
 
-def _checkcopies(ctx, f, m1, m2, base, tca, remotebase, limit, data):
+def _checkcopies(ctx, f, msrc, m2, base, tca, remotebase, limit, data):
 """
-check possible copies of f from m1 to m2
+check possible copies of f from msrc to m2
 
-ctx = starting context for f in m1
-f = the filename to check (as in m1)
-m1 = the source manifest
+ctx = starting context for f in srcm
+f = the filename to check (as in msrc)
+msrc = the source manifest
 m2 = the destination manifest
 base = the changectx used as a merge base
 tca = topological common ancestor for graft-like scenarios
@@ -632,13 +632,13 @@
 backwards = not remotebase and base != tca and f in mb
 getfctx = _makegetfctx(ctx)
 
-if m1[f] == mb.get(f) and not remotebase:
+if msrc[f] == mb.get(f) and not remotebase:
 # Nothing to merge
 return
 
 of = None
 seen = {f}
-for oc in getfctx(f, m1[f]).ancestors():
+for oc in getfctx(f, msrc[f]).ancestors():
 ocr = oc.linkrev()
 of = oc.path()
 if of in seen:
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 7] copies: rename m2 to mdst

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1496062335 25200
#  Mon May 29 05:52:15 2017 -0700
# Node ID 2cf8c7e99a33dce384b49d3fd0fc4c9512ee94b5
# Parent  75f0d8342109877a0dd620df3f566d0f080a9138
copies: rename m2 to mdst

Small refactoring to rename m2 to more clearer mdst.

diff --git a/mercurial/copies.py b/mercurial/copies.py
--- a/mercurial/copies.py
+++ b/mercurial/copies.py
@@ -598,14 +598,14 @@
 except StopIteration:
 return False
 
-def _checkcopies(ctx, f, msrc, m2, base, tca, remotebase, limit, data):
+def _checkcopies(ctx, f, msrc, mdst, base, tca, remotebase, limit, data):
 """
-check possible copies of f from msrc to m2
+check possible copies of f from msrc to mdst
 
 ctx = starting context for f in srcm
 f = the filename to check (as in msrc)
 msrc = the source manifest
-m2 = the destination manifest
+mdst = the destination manifest
 base = the changectx used as a merge base
 tca = topological common ancestor for graft-like scenarios
 remotebase = True if base is outside tca::ctx, False otherwise
@@ -653,11 +653,11 @@
 data['fullcopy'][of] = f # grafting backwards through renames
 else:
 data['fullcopy'][f] = of
-if of not in m2:
+if of not in mdst:
 continue # no match, keep looking
-if m2[of] == mb.get(of):
+if mdst[of] == mb.get(of):
 return # no merge needed, quit early
-c2 = getfctx(of, m2[of])
+c2 = getfctx(of, mdst[of])
 # c2 might be a plain new file on added on destination side that is
 # unrelated to the droids we are looking for.
 cr = _related(oc, c2, tca.rev())
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH remotenames-ext] selectivepull: fix tests

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1495731588 25200
#  Thu May 25 09:59:48 2017 -0700
# Node ID 78c8966c02759755235a6566e0c557b82ff9cb0a
# Parent  900f1c70436bd7a266f64a6db592af602f8138a5
selectivepull: fix tests

diff --git a/tests/test-selective-pull.t b/tests/test-selective-pull.t
--- a/tests/test-selective-pull.t
+++ b/tests/test-selective-pull.t
@@ -100,7 +100,7 @@
   $ hg clone -q ssh://user@dummy/remoterepo secondremoterepo
   $ cd secondremoterepo
   $ hg up -q 0238718db2b1
-  $ hg book master
+  $ hg book master --force
   $ cd ..
 
 Add second remote repo path in localrepo
@@ -119,7 +119,7 @@
 
 Move bookmark in second remote, pull and make sure it doesn't move in local 
repo
   $ cd ../secondremoterepo
-  $ hg book secondbook
+  $ hg book secondbook --force
   $ echo aaa >> a
   $ hg commit -m 'Move bookmark in second remote'
   $ cd ../localrepo
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH V2] filemerge: store error messages in module variables

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1495190863 25200
#  Fri May 19 03:47:43 2017 -0700
# Node ID 48651bcde6516d2bc075ce694a3d8b2fc0084e7b
# Parent  8a87bfc5bebbbe0ac996ac8e047a029eb931af45
filemerge: store error messages in module variables

Copytracing may be disabled because it's too slow (see
experimental.disablecopytrace config option). In that case user may get errors
like 'local changed FILE which other deleted'. It would be nice to give user a
hint to rerun command with `--config experimental.disablecopytrace=False`. To
make it possible let's extract error message to variables so that extension may
overwrite them.

diff --git a/mercurial/filemerge.py b/mercurial/filemerge.py
--- a/mercurial/filemerge.py
+++ b/mercurial/filemerge.py
@@ -49,6 +49,17 @@
 mergeonly = 'mergeonly'  # just the full merge, no premerge
 fullmerge = 'fullmerge'  # both premerge and merge
 
+_localchangedotherdeletedmsg = _(
+"local%(l)s changed %(fd)s which other%(o)s deleted\n"
+"use (c)hanged version, (d)elete, or leave (u)nresolved?"
+"$$  $$  $$ ")
+
+_otherchangedlocaldeletedmsg = _(
+"other%(o)s changed %(fd)s which local%(l)s deleted\n"
+"use (c)hanged version, leave (d)eleted, or "
+"leave (u)nresolved?"
+"$$  $$  $$ ")
+
 class absentfilectx(object):
 """Represents a file that's ostensibly in a context but is actually not
 present in it.
@@ -250,16 +261,11 @@
 try:
 if fco.isabsent():
 index = ui.promptchoice(
-_("local%(l)s changed %(fd)s which other%(o)s deleted\n"
-  "use (c)hanged version, (d)elete, or leave (u)nresolved?"
-  "$$  $$  $$ ") % prompts, 2)
+_localchangedotherdeletedmsg % prompts, 2)
 choice = ['local', 'other', 'unresolved'][index]
 elif fcd.isabsent():
 index = ui.promptchoice(
-_("other%(o)s changed %(fd)s which local%(l)s deleted\n"
-  "use (c)hanged version, leave (d)eleted, or "
-  "leave (u)nresolved?"
-  "$$  $$  $$ ") % prompts, 2)
+_otherchangedlocaldeletedmsg % prompts, 2)
 choice = ['other', 'local', 'unresolved'][index]
 else:
 index = ui.promptchoice(
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH] filemerge: store error messages in module variables

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1495125989 25200
#  Thu May 18 09:46:29 2017 -0700
# Node ID a1bf790daa1dcaf534c766b961951947d69938ae
# Parent  8a87bfc5bebbbe0ac996ac8e047a029eb931af45
filemerge: store error messages in module variables

Copytracing may be disabled because it's too slow (see
experimental.disablecopytrace config option). In that case user may get errors
like 'local changed FILE which other deleted'. It would be nice to give user a
hint to rerun command with `--config experimental.disablecopytrace=False`. To
make it possible let's extract error message to variables so that extension may
overwrite them.

diff --git a/mercurial/filemerge.py b/mercurial/filemerge.py
--- a/mercurial/filemerge.py
+++ b/mercurial/filemerge.py
@@ -49,6 +49,17 @@
 mergeonly = 'mergeonly'  # just the full merge, no premerge
 fullmerge = 'fullmerge'  # both premerge and merge
 
+_localchangedotherdeletedmsg = (
+"local%(l)s changed %(fd)s which other%(o)s deleted\n"
+"use (c)hanged version, (d)elete, or leave (u)nresolved?"
+"$$  $$  $$ ")
+
+_otherchangedlocaldeletedmsg = (
+"other%(o)s changed %(fd)s which local%(l)s deleted\n"
+"use (c)hanged version, leave (d)eleted, or "
+"leave (u)nresolved?"
+"$$  $$  $$ ")
+
 class absentfilectx(object):
 """Represents a file that's ostensibly in a context but is actually not
 present in it.
@@ -250,16 +261,11 @@
 try:
 if fco.isabsent():
 index = ui.promptchoice(
-_("local%(l)s changed %(fd)s which other%(o)s deleted\n"
-  "use (c)hanged version, (d)elete, or leave (u)nresolved?"
-  "$$  $$  $$ ") % prompts, 2)
+_(_localchangedotherdeletedmsg) % prompts, 2)
 choice = ['local', 'other', 'unresolved'][index]
 elif fcd.isabsent():
 index = ui.promptchoice(
-_("other%(o)s changed %(fd)s which local%(l)s deleted\n"
-  "use (c)hanged version, leave (d)eleted, or "
-  "leave (u)nresolved?"
-  "$$  $$  $$ ") % prompts, 2)
+_(_otherchangedlocaldeletedmsg) % prompts, 2)
 choice = ['other', 'local', 'unresolved'][index]
 else:
 index = ui.promptchoice(
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 1 of 2 remotenames-ext] remotenames: get rid of useless lookup

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1487690067 28800
#  Tue Feb 21 07:14:27 2017 -0800
# Node ID cf04161615a48d42a2bb6181060ae9be8ceaa467
# Parent  4f17d4ae821e7654b3c134002537155b28f7e146
remotenames: get rid of useless lookup

Lookup is useless since `bookmarks` dict already contains nodes for remote
bookmarks

diff --git a/remotenames.py b/remotenames.py
--- a/remotenames.py
+++ b/remotenames.py
@@ -158,8 +158,8 @@
 if args[0]:
 heads = args[0]
 args = args[1:]
-for bookmark in bookmarks:
-heads.append(remote.lookup(remotebookmarks[bookmark]))
+for node in bookmarks.values():
+heads.append(bin(node))
 kwargs['bookmarks'] = bookmarks
 kwargs['heads'] = heads
 else:
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 2 remotenames-ext] remotenames: selectivepull, add _listremotebookmarks

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1487690190 28800
#  Tue Feb 21 07:16:30 2017 -0800
# Node ID f49560facf4802449ab15aaf9b12287754619990
# Parent  cf04161615a48d42a2bb6181060ae9be8ceaa467
remotenames: selectivepull, add _listremotebookmarks

Adding a separate function to fetch only selected bookmarks from the server
allows extensions to wrap it. For example, infinitepush extensions
(https://bitbucket.org/facebook/hg-experimental/src/
98384e8d1db6534f7e5df08080f8ebf57688d380/infinitepush/?at=default) can wrap
this function and return bookmarks that are not present on hg server but
stored in a separate storage.

diff --git a/remotenames.py b/remotenames.py
--- a/remotenames.py
+++ b/remotenames.py
@@ -85,19 +85,18 @@
 def _isselectivepull(ui):
 return ui.configbool('remotenames', 'selectivepull', False)
 
-def _getselectivepulldefaultbookmarks(ui, remotebookmarks):
+def _getselectivepulldefaultbookmarks(ui):
 default_books = ui.configlist('remotenames', 'selectivepulldefault')
 if not default_books:
 raise error.Abort(_('no default bookmarks specified for 
selectivepull'))
+return default_books
 
+def _listremotebookmarks(remote, bookmarks):
+remotebookmarks = remote.listkeys('bookmarks')
 result = {}
-for default_book in default_books:
-if default_book in remotebookmarks:
-result[default_book] = remotebookmarks[default_book]
-
-if not default_books:
-raise error.Abort(
-_('default bookmarks %s are not found on remote') % default_books)
+for book in bookmarks:
+if book in remotebookmarks:
+result[book] = remotebookmarks[book]
 return result
 
 def _trypullremotebookmark(mayberemotebookmark, repo, ui):
@@ -122,7 +121,6 @@
 ui.warn(_('`%s` found remotely\n') % mayberemotebookmark)
 
 def expull(orig, repo, remote, *args, **kwargs):
-remotebookmarks = remote.listkeys('bookmarks')
 if _isselectivepull(repo.ui):
 # if selectivepull is enabled then we don't save all of the remote
 # bookmarks in remotenames file. Instead we save only bookmarks that
@@ -135,23 +133,21 @@
 # Selectivepull is helpful when server has too many remote bookmarks
 # because it may slow down clients.
 path = activepath(repo.ui, remote)
-bookmarks = {}
+remotebookmarkslist = []
 if repo.vfs.exists(_selectivepullenabledfile):
 # 'selectivepullenabled' file is used for transition between
 # non-selectivepull repo to selectivepull repo. It is used as
 # indicator to whether "non-interesting" bookmarks were removed
 # from remotenames file.
-for bookmark in readbookmarknames(repo, path):
-if bookmark in remotebookmarks:
-bookmarks[bookmark] = remotebookmarks[bookmark]
-if not bookmarks:
-bookmarks = _getselectivepulldefaultbookmarks(repo.ui,
-  remotebookmarks)
+remotebookmarkslist = list(readbookmarknames(repo, path))
+if not remotebookmarkslist:
+remotebookmarkslist = _getselectivepulldefaultbookmarks(repo.ui)
 
 if kwargs.get('bookmarks'):
-for bookmark in kwargs['bookmarks']:
-bookmarks[bookmark] = remotebookmarks[bookmark]
+remotebookmarkslist.extend(kwargs['bookmarks'])
+bookmarks = _listremotebookmarks(remote, remotebookmarkslist)
 else:
+bookmarks = _listremotebookmarks(remote, remotebookmarkslist)
 heads = kwargs.get('heads') or []
 # heads may be passed as positional args
 if len(args) > 0:
@@ -163,7 +159,7 @@
 kwargs['bookmarks'] = bookmarks
 kwargs['heads'] = heads
 else:
-bookmarks = remotebookmarks
+bookmarks = remote.listkeys('bookmarks')
 
 res = orig(repo, remote, *args, **kwargs)
 pullremotenames(repo, remote, bookmarks)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH] repoview: separate cache hash computation from cache reading

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1487584447 28800
#  Mon Feb 20 01:54:07 2017 -0800
# Node ID cb1151ce14f7ec9e2ca991bc8bd590020ec4b061
# Parent  693a5bb478543a986808264e586073a3ceedc38f
repoview: separate cache hash computation from cache reading

This change will make it easier for extensions to use another cache hash.

diff --git a/mercurial/repoview.py b/mercurial/repoview.py
--- a/mercurial/repoview.py
+++ b/mercurial/repoview.py
@@ -139,15 +139,13 @@
 if wlock:
 wlock.release()
 
-def tryreadcache(repo, hideable):
-"""read a cache if the cache exists and is valid, otherwise returns 
None."""
+def _readhiddencache(repo, cachefilename, newhash):
 hidden = fh = None
 try:
 if repo.vfs.exists(cachefile):
 fh = repo.vfs.open(cachefile, 'rb')
 version, = struct.unpack(">H", fh.read(2))
 oldhash = fh.read(20)
-newhash = cachehash(repo, hideable)
 if (cacheversion, oldhash) == (version, newhash):
 # cache is valid, so we can start reading the hidden revs
 data = fh.read()
@@ -165,6 +163,11 @@
 if fh:
 fh.close()
 
+def tryreadcache(repo, hideable):
+"""read a cache if the cache exists and is valid, otherwise returns 
None."""
+newhash = cachehash(repo, hideable)
+return _readhiddencache(repo, cachefile, newhash)
+
 def computehidden(repo):
 """compute the set of hidden revision to filter
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: RFC: bitmap storage for precursors and phases

[Stanislau Hlebik]

Excerpts from Augie Fackler's message of 2017-02-19 21:06:53 -0500:
> On Fri, Feb 17, 2017 at 09:59:48PM +0000, Stanislau Hlebik wrote:
> > Excerpts from Bryan O'Sullivan's message of 2017-02-17 13:29:58 -0800:
> > > On Fri, Feb 17, 2017 at 10:30 AM, Jun Wu <qu...@fb.com> wrote:
> > >
> > > > Excerpts from Stanislau Hlebik's message of 2017-02-17 11:24:34 +:
> > > > > As I said before we will load all non-public revs in one set and all
> > > >
> > > > The problem is, loading a Python set from disk is O(size-of-the-set).
> > > >
> > > > Bitmap's loading cost should be basically 0 (with mmap). I think that's 
> > > > why
> > > > we want bitmap at the first place. There are other choices like packfile
> > > > index, hash tables, but bitmap is the simplest and most efficient.
> > > >
> > >
> > > Hey folks,
> > >
> > > I haven't yet seen mention of some considerations that seem very important
> > > in driving the decision-making, so I'd appreciate it if someone could fill
> > > me in.
> > >
> > > Firstly, what's our current understanding of the sizes and compositions of
> > > these sets of numbers? In theory, we have a lot of data from practical
> > > application at Facebook, but nobody's brought it up.
> >
> > I assume that both sets (set for nonpublic commits and set for
> > obsstore) are going to be very small comparing to the repo size. I
> > expect both sets < 1% of the repo size. And the sets is going to be
> > sparse.
> 
> I replied elsewhere in the thread, but in my clone of hg it's on the
> order of 25-30% of the history, so assuming it's going to be very
> sparse is probably unwise.

In that case it's better to use bitmaps. But to do it we need to get rid
of filteredrevs iteration in scmutil.filteredhash() function.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: RFC: bitmap storage for precursors and phases

[Stanislau Hlebik]

Excerpts from Jun Wu's message of 2017-02-17 19:14:12 -0800:
> Excerpts from Bryan O'Sullivan's message of 2017-02-17 13:29:58 -0800:
> > On Fri, Feb 17, 2017 at 10:30 AM, Jun Wu  wrote:
> > 
> > > Excerpts from Stanislau Hlebik's message of 2017-02-17 11:24:34 +:
> > > > As I said before we will load all non-public revs in one set and all
> > >
> > > The problem is, loading a Python set from disk is O(size-of-the-set).
> > >
> > > Bitmap's loading cost should be basically 0 (with mmap). I think that's 
> > > why
> > > we want bitmap at the first place. There are other choices like packfile
> > > index, hash tables, but bitmap is the simplest and most efficient.
> > >
> > 
> > Hey folks,
> > 
> > I haven't yet seen mention of some considerations that seem very important
> > in driving the decision-making, so I'd appreciate it if someone could fill
> > me in.
> > 
> > Firstly, what's our current understanding of the sizes and compositions of
> > these sets of numbers? In theory, we have a lot of data from practical
> > application at Facebook, but nobody's brought it up.
> > 
> > To clarify why this matters, if an obsstore contains say 100 entries, then
> > a very naive implementation should be fine. If it's related to something
> > else, such as the size of, amount of activity in, or local age of a repo,
> > then maybe we have a different set of decisions to make.
> 
> First, I agree that if N is small, O(N) or O(log(N)), or O(1), do not matter
> much in practice. I tend to always reason about future-proof solutions,
> which may or may not be a good habit - I'll adjust in the future.
> 
> I think there are multiple topics being discussed:
> 
>   1. How to solve the overhead loading hiddenrevs with minimal changes?
>   2. Why is the bitmap format interesting (future use-cases)?
> 
> For 1,
> 
>   I once tried to make one bit of revlog flag as the source of truth of
>   "hidden", which looks a nice solution. But it got rejected because revlog
>   has to be append-only. Moving the flag bits out formed the bitmap idea
>   naturally.
> 
>   Bitmap fits here but it does not mean we have to use it to solve this
>   particular problem. For example, stateful chg will get us some easy wins,
>   and see the problem analysis and proposed solution below.
> 
>   Note that we actually have a simple caching format for the "hiddenrevs".
>   See repoview.py, computehidden.
> 
>   But what's wrong with "computehidden" is the choice of the cache key - it
>   calls "hideablerevs", which takes 0.5 seconds in obsolete.getrevs(repo,
>   'obsolete'), which reads the giant obsstore (in hg-committed) and do some
>   complex computation. The result is then hashed and used as a cache key of
>   "computehidden".
> 
>   So, if we do not want to know the "obsolete()" revisions, but just want to
>   know what's hidden. Changing the cache key would be a very straightforward
>   and effective solution which does not require a new file format.

Hm, changing the cache key to smth like phaseroots + obsstore +
changelog mtime + size  may be an easy and good win. That's a good idea.
Although it will still get invalidated after each write.

> 
>   That'll help commands like "hg st", "hg log -T foo", etc. But it won't
>   help complex "log"s which do require the "obsolete()" revisions. To speed
>   up the latter, we can probably just copy the caching infrastructure from
>   repoview.py.
> 
> For 2,
> 
>   len(filteredrevs) in my hg-committed is 2155. It's small. I tend to think
>   about solutions that scale longer and are not too complex to build. That
>   may or may not be a good habit.
>   
>   That said, I agree if the set is small and perf, then whatever easier to
>   implement makes sense.
>   
>   I have some crazy ideas about the future where a giant bitmap is
>   practically useful. For example, source controlling "visible heads" and
>   the obsstore, then adding some C code to quickly fill a bitmap (requires
>   "set" and "setancestors/descendants"), then we can show the user whenever
>   their repo used to look like back in any time by using the bitmap filter.
>   That'll help investigate hard issues, and implement "hg undo".
> 
> > Secondly, what operations are being performed on these sets today, and do
> > we know if those operations are going to change?
> >
> > For instance, if a may be extended via new elements added to its end, then
> > an mmap-based implementation immediately becomes awkward. If a set is
> 
> Not that bad if we preallocate space.
> 
> > traversed or we perform operations like union or intersection, then a
> > bitmap might make sense if the set is very small and dense, but not if it's
> > big and sparse.
> 
> Not necessarily true if we use an advanced variant, like the "roaringbitmap"
> mentioned by you last year [1].
> 
> [1]: 
> https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-September/087864.html
> 
> > (Jun's idea that mmap is effectively free is also not borne out in
> > 

Re: RFC: bitmap storage for precursors and phases

[Stanislau Hlebik]

Excerpts from Bryan O'Sullivan's message of 2017-02-17 13:29:58 -0800:
> On Fri, Feb 17, 2017 at 10:30 AM, Jun Wu  wrote:
> 
> > Excerpts from Stanislau Hlebik's message of 2017-02-17 11:24:34 +:
> > > As I said before we will load all non-public revs in one set and all
> >
> > The problem is, loading a Python set from disk is O(size-of-the-set).
> >
> > Bitmap's loading cost should be basically 0 (with mmap). I think that's why
> > we want bitmap at the first place. There are other choices like packfile
> > index, hash tables, but bitmap is the simplest and most efficient.
> >
> 
> Hey folks,
> 
> I haven't yet seen mention of some considerations that seem very important
> in driving the decision-making, so I'd appreciate it if someone could fill
> me in.
> 
> Firstly, what's our current understanding of the sizes and compositions of
> these sets of numbers? In theory, we have a lot of data from practical
> application at Facebook, but nobody's brought it up.

I assume that both sets (set for nonpublic commits and set for
obsstore) are going to be very small comparing to the repo size. I
expect both sets < 1% of the repo size. And the sets is going to be
sparse.

> 
> To clarify why this matters, if an obsstore contains say 100 entries, then
> a very naive implementation should be fine. If it's related to something
> else, such as the size of, amount of activity in, or local age of a repo,
> then maybe we have a different set of decisions to make.
> 
> Secondly, what operations are being performed on these sets today, and do
> we know if those operations are going to change?
> 
> For instance, if a may be extended via new elements added to its end, then
> an mmap-based implementation immediately becomes awkward. If a set is
> traversed or we perform operations like union or intersection, then a
> bitmap might make sense if the set is very small and dense, but not if it's
> big and sparse.

We need three operations: iteration over the whole
set, testing if an element belongs to the set and adding new elements.
New elements can be added to the end of the set.
Not sure if more operations will be added, probably not.

Agree, we can't use bitmaps for iteration.
But potentially we can clean all of the places where iteration is used.
Most of them are pretty straight-forward except for one tricky case in
scmutil.filteredhash() function.

> 
> (Jun's idea that mmap is effectively free is also not borne out in
> practice, but that's a second-order issue here.)
> 
> So tell us what the needed API is, what the characteristics of the data
> are, and that'll help understand how to make a good decision.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: RFC: bitmap storage for precursors and phases

[Stanislau Hlebik]

Excerpts from Jun Wu's message of 2017-02-17 10:30:45 -0800:
> Excerpts from Stanislau Hlebik's message of 2017-02-17 11:24:34 +:
> > As I said before we will load all non-public revs in one set and all
> 
> The problem is, loading a Python set from disk is O(size-of-the-set).
> 
> Bitmap's loading cost should be basically 0 (with mmap). I think that's why
> we want bitmap at the first place. There are other choices like packfile
> index, hash tables, but bitmap is the simplest and most efficient.

Agree, but Python set is going to be small and loading it is also fast.

> 
> > precursor revs in another set. So `ispublic()` and `isprecursor()`
> > checks are very fast, it's just a set lookup. Same with update - it's
> > just an insertion in the set.
> 
> If cache invalidation happens for common write operations and the cache
> needs to be rebuilt from scratch, then the solution is not better than
> stateful chg - the chg way is easier (no need to serialize data), and safer
> (not vulnerable to cache file corruption).
> 
> To make the bitmap cache really useful (solving problems that chg cannot not
> solve), we need to incrementally update the cache, and avoid rebuilding it
> from scratch, at least for common commands like "rebase", "amend", "commit".
> 
> > > Some code can be changed - "scmutil.filteredhash" seems to be one user
> > > that iterates "filteredrevs". But what it needs is only a hash - it
> > > could hash something else, like the mtime, size etc.
> > 
> > Bookmarks, changelog, obsstore and tags can affect filtered set. 
> > For filtered repo we'll need to use size + mtime of bookmarks,
> > changelog, obsstore, tags and maybe even smth else. That maybe
> > error-prone.
> 
> The "filteredhash" seems to be only used by tags and branchmap cache. I
> think we can actually get rid of it by only caching unfiltered versions and
> do the filtering later. It also seems wrong to have tags and filteredrevs
> mutually depend on each other.

If we can get rid of `filteredhash` then we can probably proceed with
bitmaps. Although using bitmaps will still require bigger code changes
than using serialized sets.

> 
> > > Bitmaps could also be smarter - like maintaining the min and max revisions
> > > so it does not need to be exactly len(repo).
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: RFC: bitmap storage for precursors and phases

[Stanislau Hlebik]

Excerpts from Stanislau Hlebik's message of 2017-02-17 11:24:34 +:
> Excerpts from Jun Wu's message of 2017-02-16 13:42:46 -0800:
> > Excerpts from Stanislau Hlebik's message of 2017-02-16 19:39:07 +:
> > > Excerpts from Stanislau Hlebik's message of 2017-02-14 09:29:25 +:
> > > > Excerpts from Sean Farley's message of 2017-02-13 18:30:25 -0800:
> > > > > Jun Wu  writes:
> > > > > 
> > > > > > Excerpts from Sean Farley's message of 2017-02-13 17:04:35 -0800:
> > > > > >> I was thinking about a more high-level approach (please feel free 
> > > > > >> to
> > > > > >> pick apart):
> > > > > >> 
> > > > > >> r = repo.filtered("bitmap1")
> > > > > >> r2 = r.filtered("bitmap2")
> > > > > >> 
> > > > > >> So that r2 would be an intersection of bitmap1 and bitmap2 (haven't
> > > > > >> thought about a union nor the inverse).
> > > > > >
> > > > > > That does not conflict with my comments. It could be implemented as 
> > > > > > nested
> > > > > > filters, or flatten the bitmap by doing an "or" operation.
> > > > > 
> > > > > Righto. Just wanted to bring it up early before things are set in 
> > > > > stone.
> > > > 
> > > > Current `repoview.filtered()` implementation can apply only one filter. 
> > > > I
> > > > think it will be error-prone to change it, won't it?
> > > 
> > > It seems that it's better to use sorted lists instead of bitmaps. In a
> > > couple of places it is expected that repo.filteredrevs supports
> > > iteration. But iteration over a bitmap is very slow. Instead we can
> > > store list of non-public revs and list of precursor revs and load them
> > > in a set.
> > > 
> > > It will be slow for the case where repo has lots of draft commits.
> > > In this case it's probably better to disable this feature completely.
> > 
> > I think it's fine to have slow iteration, as long as the iteration operation
> > is uncommon. The point is, the most common operation should be fast - that
> > is, ishidden(rev) should be fast. Stateful chg will help the read-only case,
> > the bitmap + mmap would ideally make write (like rebase) cases fast
> .
> As I said before we will load all non-public revs in one set and all
> precursor revs in another set. So `ispublic()` and `isprecursor()`
> checks are very fast, it's just a set lookup. Same with update - it's
> just an insertion in the set.
> 
> > 
> > Some code can be changed - "scmutil.filteredhash" seems to be one user that
> > iterates "filteredrevs". But what it needs is only a hash - it could hash
> > something else, like the mtime, size etc.
> 
> Bookmarks, changelog, obsstore and tags can affect filtered set. 
> For filtered repo we'll need to use size + mtime of bookmarks,
> changelog, obsstore, tags and maybe even smth else. That maybe
> error-prone.

This is implementation of two caches (nonpublic + precursor) using
serialized sorted lists and sets
https://bitbucket.org/stashlebik/hg/commits/99879579ac2848a2567810b677d8344150a7b319?at=hiddenbitmaps_lists

> > 
> > Bitmaps could also be smarter - like maintaining the min and max revisions
> > so it does not need to be exactly len(repo).
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: RFC: bitmap storage for precursors and phases

[Stanislau Hlebik]

Excerpts from Jun Wu's message of 2017-02-16 13:42:46 -0800:
> Excerpts from Stanislau Hlebik's message of 2017-02-16 19:39:07 +:
> > Excerpts from Stanislau Hlebik's message of 2017-02-14 09:29:25 +:
> > > Excerpts from Sean Farley's message of 2017-02-13 18:30:25 -0800:
> > > > Jun Wu  writes:
> > > > 
> > > > > Excerpts from Sean Farley's message of 2017-02-13 17:04:35 -0800:
> > > > >> I was thinking about a more high-level approach (please feel free to
> > > > >> pick apart):
> > > > >> 
> > > > >> r = repo.filtered("bitmap1")
> > > > >> r2 = r.filtered("bitmap2")
> > > > >> 
> > > > >> So that r2 would be an intersection of bitmap1 and bitmap2 (haven't
> > > > >> thought about a union nor the inverse).
> > > > >
> > > > > That does not conflict with my comments. It could be implemented as 
> > > > > nested
> > > > > filters, or flatten the bitmap by doing an "or" operation.
> > > > 
> > > > Righto. Just wanted to bring it up early before things are set in stone.
> > > 
> > > Current `repoview.filtered()` implementation can apply only one filter. I
> > > think it will be error-prone to change it, won't it?
> > 
> > It seems that it's better to use sorted lists instead of bitmaps. In a
> > couple of places it is expected that repo.filteredrevs supports
> > iteration. But iteration over a bitmap is very slow. Instead we can
> > store list of non-public revs and list of precursor revs and load them
> > in a set.
> > 
> > It will be slow for the case where repo has lots of draft commits.
> > In this case it's probably better to disable this feature completely.
> 
> I think it's fine to have slow iteration, as long as the iteration operation
> is uncommon. The point is, the most common operation should be fast - that
> is, ishidden(rev) should be fast. Stateful chg will help the read-only case,
> the bitmap + mmap would ideally make write (like rebase) cases fast
.
As I said before we will load all non-public revs in one set and all
precursor revs in another set. So `ispublic()` and `isprecursor()`
checks are very fast, it's just a set lookup. Same with update - it's
just an insertion in the set.

> 
> Some code can be changed - "scmutil.filteredhash" seems to be one user that
> iterates "filteredrevs". But what it needs is only a hash - it could hash
> something else, like the mtime, size etc.

Bookmarks, changelog, obsstore and tags can affect filtered set. 
For filtered repo we'll need to use size + mtime of bookmarks,
changelog, obsstore, tags and maybe even smth else. That maybe
error-prone.
> 
> Bitmaps could also be smarter - like maintaining the min and max revisions
> so it does not need to be exactly len(repo).
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: RFC: bitmap storage for precursors and phases

[Stanislau Hlebik]

Excerpts from Stanislau Hlebik's message of 2017-02-14 09:29:25 +:
> Excerpts from Sean Farley's message of 2017-02-13 18:30:25 -0800:
> > Jun Wu  writes:
> > 
> > > Excerpts from Sean Farley's message of 2017-02-13 17:04:35 -0800:
> > >> I was thinking about a more high-level approach (please feel free to
> > >> pick apart):
> > >> 
> > >> r = repo.filtered("bitmap1")
> > >> r2 = r.filtered("bitmap2")
> > >> 
> > >> So that r2 would be an intersection of bitmap1 and bitmap2 (haven't
> > >> thought about a union nor the inverse).
> > >
> > > That does not conflict with my comments. It could be implemented as nested
> > > filters, or flatten the bitmap by doing an "or" operation.
> > 
> > Righto. Just wanted to bring it up early before things are set in stone.
> 
> Current `repoview.filtered()` implementation can apply only one filter. I
> think it will be error-prone to change it, won't it?

It seems that it's better to use sorted lists instead of bitmaps. In a
couple of places it is expected that repo.filteredrevs supports
iteration. But iteration over a bitmap is very slow. Instead we can
store list of non-public revs and list of precursor revs and load them
in a set.

It will be slow for the case where repo has lots of draft commits.
In this case it's probably better to disable this feature completely.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: RFC: bitmap storage for precursors and phases

[Stanislau Hlebik]

Excerpts from Gregory Szorc's message of 2017-02-13 20:28:37 -0700:
> 
> > On Feb 10, 2017, at 10:33, Stanislau Hlebik <st...@fb.com> wrote:
> > 
> > Last year Durham sent a proposal for bitmap storage for computehidden() 
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.mercurial-2Dscm.org_pipermail_mercurial-2Ddevel_2016-2DSeptember_087860.html=DwIFAg=5VD0RTtNlTh3ycd41b3MUw=1EQ58Dmb5uX1qHujcsT1Mg=ee96J8IDM8uhZeONo6skPSXItouNwkeRGNxHeCaXiXM=biY_gtRGCl25baAE_Nlw8cl4YUefBM0Xb7L0kuGklRA=
> >  .
> >  
> > It got a few useful comments, two most important comments:
> > Use bitmaps for lower-level data structures, for example, bitmap for public 
> > commits and bitmap for commits that are affected by obsolescence markers
> > Add cache validation checks
> >  
> > This is a new RFC that addresses these issues.
> > In the code, there is a cache only for precursors. Later I'm planning to 
> > add cache for public commits.
> > Cache validation key was added. It can be any key. For precursorcache I've 
> > chosen obsstore file size and mtime and cache validation key. For 
> > phasecache we can use hash of sorted phaseroots file since this file is 
> > usually small. 
> 
> Until there is a better supported mechanism for storing "draft pushes" in an 
> external store (Facebook's inifinitepush extension may fulfill this), 
> Mozilla's "Try" and code review repos need to scale to >100,000 heads / phase 
> roots. So a hash of phaseroots may not scale. (Of course, one solution to 
> this problem would be inverting phases storage to be head based instead of 
> root based. That doesn't solve problem if you have 100k public heads and 100k 
> draft heads. Scaling is hard.)

Then I will use the same cache key as for 'obsstore' - mtime + size.
BTW infinitepush is ready and opensourced so you can give it a try: 
https://bitbucket.org/facebook/hg-experimental/src/567c1477eebf43ff5fe7412c650ec49aa3e44a3c/infinitepush/?at=default

> 
> >  
> > In a few places, I decided to delete cache entirely:
> > Caches are blown up if we receive obsmarkers via bundle2 or 
> > exchange._pullobsolete(). This is not necessary since cache won't be valid 
> > on the next run anyway because obsstore size and/or mtime was changed. We 
> > can change it.
> > Obsstore can store markers for node that are not present in the repo. Since 
> > bitmap is basically a dict from rev to boolean it can't store markers for 
> > revisions that are not in the repo. It doesn't cause issues unless missing 
> > node is received from bundle or during pull. In this case precursorcache 
> > doesn't recognize it as obsoleted. To fix it cache is deleted during 
> > changegroup.apply() if we receive a node that is in obsstore.successors 
> > dict.
> > Similar thing in bundlerepo - new obsoleted nodes may have been added, 
> > let's just not use precursorcache in bundlerepo.
> >  
> >  
> > The code is here: 
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__bitbucket.org_stashlebik_hg_commits_branch_hiddenbitmaps=DwIFAg=5VD0RTtNlTh3ycd41b3MUw=1EQ58Dmb5uX1qHujcsT1Mg=ee96J8IDM8uhZeONo6skPSXItouNwkeRGNxHeCaXiXM=dp9GiK3_WKq2zB8mVuMy2ksPF5tnGV_1T_OSAp6jD8o=
> >  
> > It's not super clean yet, but all tests pass (except maybe for commit/code 
> > checks).
> > Please look and let me know what you think!
> >  
> >  
> > Thanks,
> > Stas
> > ___
> > Mercurial-devel mailing list
> > Mercurial-devel@mercurial-scm.org
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.mercurial-2Dscm.org_mailman_listinfo_mercurial-2Ddevel=DwIFAg=5VD0RTtNlTh3ycd41b3MUw=1EQ58Dmb5uX1qHujcsT1Mg=ee96J8IDM8uhZeONo6skPSXItouNwkeRGNxHeCaXiXM=mqvxWBluu-CxyLDxFLW49s61VhbzU_NiqHHCSu6NVxw=
> >  
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: RFC: bitmap storage for precursors and phases

[Stanislau Hlebik]

Excerpts from Sean Farley's message of 2017-02-13 18:30:25 -0800:
> Jun Wu  writes:
> 
> > Excerpts from Sean Farley's message of 2017-02-13 17:04:35 -0800:
> >> I was thinking about a more high-level approach (please feel free to
> >> pick apart):
> >> 
> >> r = repo.filtered("bitmap1")
> >> r2 = r.filtered("bitmap2")
> >> 
> >> So that r2 would be an intersection of bitmap1 and bitmap2 (haven't
> >> thought about a union nor the inverse).
> >
> > That does not conflict with my comments. It could be implemented as nested
> > filters, or flatten the bitmap by doing an "or" operation.
> 
> Righto. Just wanted to bring it up early before things are set in stone.

Current `repoview.filtered()` implementation can apply only one filter. I
think it will be error-prone to change it, won't it?
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 2] localrepo: avoid unnecessary sorting

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1486981916 28800
#  Mon Feb 13 02:31:56 2017 -0800
# Node ID 30fd6051a7e441aac16b5ba7b4a19aa3731576f0
# Parent  8bc5ae6cf51408dbd1b789555196f031bfef19d4
localrepo: avoid unnecessary sorting

headrevs output already sorted, no need to sort it again.

diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -1854,7 +1854,7 @@
 def heads(self, start=None):
 if start is None:
 cl = self.changelog
-headrevs = sorted(cl.headrevs(), reverse=True)
+headrevs = reversed(cl.headrevs())
 return [cl.node(rev) for rev in headrevs]
 
 heads = self.changelog.heads(start)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 1 of 2] localrepo: cache self.changelog in local variable

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1486981578 28800
#  Mon Feb 13 02:26:18 2017 -0800
# Node ID 8bc5ae6cf51408dbd1b789555196f031bfef19d4
# Parent  a0e3d808690d57d1c9dff840e0b8ee099526397b
localrepo: cache self.changelog in local variable

Repeated self.changelog lookups can incur overhead. Let's cache it in the
separate variable.

diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -1853,8 +1853,9 @@
 
 def heads(self, start=None):
 if start is None:
-headrevs = sorted(self.changelog.headrevs(), reverse=True)
-return [self.changelog.node(rev) for rev in headrevs]
+cl = self.changelog
+headrevs = sorted(cl.headrevs(), reverse=True)
+return [cl.node(rev) for rev in headrevs]
 
 heads = self.changelog.heads(start)
 # sort the output in rev descending order
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 6 of 6 remotenames-ext] selectivepull: support list of default bookmarks

[Stanislau Hlebik]

Excerpts from Gregory Szorc's message of 2017-02-11 10:54:43 -0800:
> 
> > On Jan 30, 2017, at 07:56, Stanislau Hlebik <st...@fb.com> wrote:
> > 
> > # HG changeset patch
> > # User Stanislau Hlebik <st...@fb.com>
> > # Date 1485791649 28800
> > #  Mon Jan 30 07:54:09 2017 -0800
> > # Node ID 227796849698292c76c70e874179de52b7b688d6
> > # Parent  a96117003c763be640a975d8128068d2bd3527c0
> > selectivepull: support list of default bookmarks
> > 
> > The addition is that we now support list of default bookmarks, instead of a
> > single default bookmark (the previous behavior).
> 
> Cool series!
> 
> [paths] entries have sub-options to control push behavior, including which 
> rev is pushed by default 
> (https://urldefense.proofpoint.com/v2/url?u=https-3A__www.mercurial-2Dscm.org_repo_hg_file_a95fc01aaffe_mercurial_help_config.txt-23l1348=DwIFAg=5VD0RTtNlTh3ycd41b3MUw=1EQ58Dmb5uX1qHujcsT1Mg=m80cJkPdWL1VwONVzbgynAt-AQewDC6TBiaCt4FE9EE=HPqaL5aBO4IeOlcb_2VjalCA7VhXXAmjqYHNWbaFsBY=
>  ).
> 
> So if we wanted to implement selective pull in core, the mechanism to support 
> defining the config is there...

There is one problem with that. We want to update pullrev whenever we
pull one more bookmark. For example by default `hg pull` will pull only `@`
bookmark, but after user does `hg pull -B release-bookmark` next `hg pull`
will pull `@` and `release-bookmark`. I think there isn't a way to update
config from inside hg, is there?

> 
> > 
> > diff --git a/remotenames.py b/remotenames.py
> > --- a/remotenames.py
> > +++ b/remotenames.py
> > @@ -81,14 +81,19 @@
> > return ui.configbool('remotenames', 'selectivepull', False)
> > 
> > def _getselectivepulldefaultbookmarks(ui, remotebookmarks):
> > -default_book = ui.config('remotenames', 'selectivepulldefault')
> > -if not default_book:
> > -raise error.Abort(_('no default bookmark specified for 
> > selectivepull'))
> > -if default_book in remotebookmarks:
> > -return {default_book: remotebookmarks[default_book]}
> > -else:
> > +default_books = ui.configlist('remotenames', 'selectivepulldefault')
> > +if not default_books:
> > +raise error.Abort(_('no default bookmarks specified for 
> > selectivepull'))
> > +
> > +result = {}
> > +for default_book in default_books:
> > +if default_book in remotebookmarks:
> > +result[default_book] = remotebookmarks[default_book]
> > +
> > +if not default_books:
> > raise error.Abort(
> > -_('default bookmark %s is not found on remote') % default_book)
> > +_('default bookmarks %s are not found on remote') % 
> > default_books)
> > +return result
> > 
> > def _trypullremotebookmark(mayberemotebookmark, repo, ui):
> > ui.warn(_('`%s` not found: assuming it is a remote bookmark '
> > diff --git a/tests/test-selective-pull.t b/tests/test-selective-pull.t
> > --- a/tests/test-selective-pull.t
> > +++ b/tests/test-selective-pull.t
> > @@ -227,3 +227,14 @@
> >   $ hg pull -q
> >   $ hg book --remote
> >  default/master2:0238718db2b1
> > +
> > +Set two bookmarks in selectivepulldefault, make sure both of them were 
> > pulled
> > +  $ cat >> .hg/hgrc << EOF
> > +  > [remotenames]
> > +  > selectivepulldefault=master,thirdbook
> > +  > EOF
> > +  $ rm .hg/selectivepullenabled
> > +  $ hg pull -q
> > +  $ hg book --remote
> > + default/master2:0238718db2b1
> > + default/thirdbook 0:1449e7934ec1
> > ___
> > Mercurial-devel mailing list
> > Mercurial-devel@mercurial-scm.org
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.mercurial-2Dscm.org_mailman_listinfo_mercurial-2Ddevel=DwIFAg=5VD0RTtNlTh3ycd41b3MUw=1EQ58Dmb5uX1qHujcsT1Mg=m80cJkPdWL1VwONVzbgynAt-AQewDC6TBiaCt4FE9EE=SwBFnDsS8d4zYF2z3lzuaDpSruRPoom8wzTyogP_JVQ=
> >  
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: RFC: bitmap storage for precursors and phases

[Stanislau Hlebik]

Excerpts from Jun Wu's message of 2017-02-10 11:04:24 -0800:
> In general, I think this is a good direction. Some random thoughts:
> 
>   - general purposed
> 
> I think the bitmap is not always a cache, so it should only have
> operations like set/unset/readfromdisk/writetodisk. Practically, I won't
> couple cache invalidation with the bitmap implementation.
> 
> In additional, I'll try to avoid using Python-only types in the
> interface. So once we decide to rewrite part of the implementation in
> native C, we won't have trouble.

I can decouple them into bitmap and bitmapcache. bitmap then need to
have an generic header. bitmapcache will store invalidation key there.

> 
> See "revset" below for a possibility that bitmap is used as a non-set.
> 
>   - revset
> 
> This is a possibility that probably won't happen any time soon.
> 
> The revset currently uses Python set for maintaining its state. For huge
> sets, Python sets may not be a good option. And various operations could
> benefit from an always-topologically-sorted set, which is the bitmap.
> 
>   - mmap
> 
> My intuition is that bitmaps fit better with mmap which can reduce the
> reading loading cost. I think "vfs.mmapread" could be a thing, and
> various places can benefit from it - Gabor recently showed interest in
> loading revlog data by mmap, I had patches that uses mmap to read revlog
> index.

Bitmap files are going to be small and reading them all in memory should
be fast. But if it'll ever become a bottleneck we can add mmap read of
bitmap files.

> 
> In additional, not directly related to this series, I'm a big fan of
> single direction data flow. But the current code base does not seem to do a
> good job in this area. As we are adding more caching layers to the code
> base, it'd be nice if we have some tiny framework maintaining the dependency
> of all kinds of data, to be able to understand the data flow easily, and
> just to be more confident about loading orders. I think people more
> experienced on architecture may want to share some ideas here.
> 
> Excerpts from Stanislau Hlebik's message of 2017-02-10 17:33:28 +:
> > Last year Durham sent a proposal for bitmap storage for computehidden() 
> > https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-September/087860.html
> >  .
> > 
> > It got a few useful comments, two most important comments:
> > 
> >   1.  Use bitmaps for lower-level data structures, for example, bitmap for 
> > public commits and bitmap for commits that are affected by obsolescence 
> > markers
> >   2.  Add cache validation checks
> > 
> > This is a new RFC that addresses these issues.
> > 
> >   1.  In the code, there is a cache only for precursors. Later I'm planning 
> > to add cache for public commits.
> >   2.  Cache validation key was added. It can be any key. For precursorcache 
> > I've chosen obsstore file size and mtime and cache validation key. For 
> > phasecache we can use hash of sorted phaseroots file since this file is 
> > usually small.
> > 
> > 
> > In a few places, I decided to delete cache entirely:
> > 
> >   1.  Caches are blown up if we receive obsmarkers via bundle2 or 
> > exchange._pullobsolete(). This is not necessary since cache won't be valid 
> > on the next run anyway because obsstore size and/or mtime was changed. We 
> > can change it.
> >   2.  Obsstore can store markers for node that are not present in the repo. 
> > Since bitmap is basically a dict from rev to boolean it can't store markers 
> > for revisions that are not in the repo. It doesn't cause issues unless 
> > missing node is received from bundle or during pull. In this case 
> > precursorcache doesn't recognize it as obsoleted. To fix it cache is 
> > deleted during changegroup.apply() if we receive a node that is in 
> > obsstore.successors dict.
> >   3.  Similar thing in bundlerepo - new obsoleted nodes may have been 
> > added, let's just not use precursorcache in bundlerepo.
> > 
> > 
> > The code is here: 
> > https://bitbucket.org/stashlebik/hg/commits/branch/hiddenbitmaps 
> > It's not super clean yet, but all tests pass (except maybe for commit/code 
> > checks).
> > Please look and let me know what you think!
> > 
> > 
> > Thanks,
> > Stas
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

RFC: bitmap storage for precursors and phases

[Stanislau Hlebik]

Last year Durham sent a proposal for bitmap storage for computehidden() 
https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-September/087860.html.

It got a few useful comments, two most important comments:

  1.  Use bitmaps for lower-level data structures, for example, bitmap for 
public commits and bitmap for commits that are affected by obsolescence markers
  2.  Add cache validation checks

This is a new RFC that addresses these issues.

  1.  In the code, there is a cache only for precursors. Later I'm planning to 
add cache for public commits.
  2.  Cache validation key was added. It can be any key. For precursorcache 
I've chosen obsstore file size and mtime and cache validation key. For 
phasecache we can use hash of sorted phaseroots file since this file is usually 
small.


In a few places, I decided to delete cache entirely:

  1.  Caches are blown up if we receive obsmarkers via bundle2 or 
exchange._pullobsolete(). This is not necessary since cache won't be valid on 
the next run anyway because obsstore size and/or mtime was changed. We can 
change it.
  2.  Obsstore can store markers for node that are not present in the repo. 
Since bitmap is basically a dict from rev to boolean it can't store markers for 
revisions that are not in the repo. It doesn't cause issues unless missing node 
is received from bundle or during pull. In this case precursorcache doesn't 
recognize it as obsoleted. To fix it cache is deleted during 
changegroup.apply() if we receive a node that is in obsstore.successors dict.
  3.  Similar thing in bundlerepo - new obsoleted nodes may have been added, 
let's just not use precursorcache in bundlerepo.


The code is here: 
https://bitbucket.org/stashlebik/hg/commits/branch/hiddenbitmaps
It's not super clean yet, but all tests pass (except maybe for commit/code 
checks).
Please look and let me know what you think!


Thanks,
Stas
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH V2] localrepo: avoid unnecessary conversion from node to rev

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1486457478 28800
#  Tue Feb 07 00:51:18 2017 -0800
# Node ID 8614d5d15d6fe47c3bcfdf79823f68f18c7741ae
# Parent  1f51b4658f21bbb797e922d155c1046eddccf91d
localrepo: avoid unnecessary conversion from node to rev

changelog.heads() first calls headrevs then converts them to nodes.
localrepo.heads() then sorts them using self.changelog.rev function and makes
useless conversion back to revs. Instead let's call changelog.headrevs() from
localrepo.heads(), sort the output and then convert to nodes. Because headrevs
does not support start parameter this optimization only works if start is None.

diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -1852,6 +1852,10 @@
   listsubrepos)
 
 def heads(self, start=None):
+if start is None:
+headrevs = reversed(self.changelog.headrevs())
+return [self.changelog.node(rev) for rev in headrevs]
+
 heads = self.changelog.heads(start)
 # sort the output in rev descending order
 return sorted(heads, key=self.changelog.rev, reverse=True)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 4 of 6 remotenames-ext] tests: write to local hgrc

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1485791649 28800
#  Mon Jan 30 07:54:09 2017 -0800
# Node ID 25d873eac976b58ac99dabb3f0e62814bb2aa2f8
# Parent  3e17b130293b0c65918041ad4e51a32a909e4f48
tests: write to local hgrc

A future test will need to run without selective pull enabled, so let’s not
set the global config

diff --git a/tests/test-selective-pull.t b/tests/test-selective-pull.t
--- a/tests/test-selective-pull.t
+++ b/tests/test-selective-pull.t
@@ -29,7 +29,7 @@
  default/master0:1449e7934ec1
 
 Set up selective pull
-  $ cat >> $HGRCPATH << EOF
+  $ cat >> .hg/hgrc << EOF
   > [remotenames]
   > selectivepull=True
   > selectivepulldefault=master
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 3 of 6 remotenames-ext] selectivepull: add function to get default bookmarks

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1485791649 28800
#  Mon Jan 30 07:54:09 2017 -0800
# Node ID 3e17b130293b0c65918041ad4e51a32a909e4f48
# Parent  a7786a178d5cb9abed981c76e47588fffede7511
selectivepull: add function to get default bookmarks

Small refactoring: move logic to get default bookmarks to separate function

diff --git a/remotenames.py b/remotenames.py
--- a/remotenames.py
+++ b/remotenames.py
@@ -77,6 +77,16 @@
 def _isselectivepull(ui):
 return ui.configbool('remotenames', 'selectivepull', False)
 
+def _getselectivepulldefaultbookmarks(ui, remotebookmarks):
+default_book = ui.config('remotenames', 'selectivepulldefault')
+if not default_book:
+raise error.Abort(_('no default bookmark specified for selectivepull'))
+if default_book in remotebookmarks:
+return {default_book: remotebookmarks[default_book]}
+else:
+raise error.Abort(
+_('default bookmark %s is not found on remote') % default_book)
+
 def _trypullremotebookmark(mayberemotebookmark, repo, ui):
 ui.warn(_('`%s` not found: assuming it is a remote bookmark '
   'and trying to pull it\n') % mayberemotebookmark)
@@ -117,16 +127,8 @@
 if bookmark in remotebookmarks:
 bookmarks[bookmark] = remotebookmarks[bookmark]
 if not bookmarks:
-default_book = repo.ui.config('remotenames', 
'selectivepulldefault')
-if not default_book:
-raise error.Abort(
-_('no default bookmark specified for selectivepull'))
-if default_book in remotebookmarks:
-bookmarks = {default_book: remotebookmarks[default_book]}
-else:
-raise error.Abort(
-_('default bookmark %s is not found on remote') %
-default_book)
+bookmarks = _getselectivepulldefaultbookmarks(repo.ui,
+  remotebookmarks)
 
 if kwargs.get('bookmarks'):
 for bookmark in kwargs['bookmarks']:
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 6 remotenames-ext] selectivepull: fix passing `heads` argument multiple times

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1485791608 28800
#  Mon Jan 30 07:53:28 2017 -0800
# Node ID a7786a178d5cb9abed981c76e47588fffede7511
# Parent  8b66569e49ce6382670497aa4e96c5e81c224439
selectivepull: fix passing `heads` argument multiple times

Usually `heads` are passed using kwargs but sometimes they are passed using
args (for example, during clone). In this case we set in kwargs and args
and orig call fails. This patch fixes it.

Also change the tests to cover this behaviour.

diff --git a/remotenames.py b/remotenames.py
--- a/remotenames.py
+++ b/remotenames.py
@@ -133,6 +133,11 @@
 bookmarks[bookmark] = remotebookmarks[bookmark]
 else:
 heads = kwargs.get('heads') or []
+# heads may be passed as positional args
+if len(args) > 0:
+if args[0]:
+heads = args[0]
+args = args[1:]
 for bookmark in bookmarks:
 heads.append(remote.lookup(remotebookmarks[bookmark]))
 kwargs['bookmarks'] = bookmarks
diff --git a/tests/test-selective-pull.t b/tests/test-selective-pull.t
--- a/tests/test-selective-pull.t
+++ b/tests/test-selective-pull.t
@@ -97,7 +97,7 @@
 
 Create second remote
   $ cd ..
-  $ hg clone -q remoterepo secondremoterepo
+  $ hg clone -q ssh://user@dummy/remoterepo secondremoterepo
   $ cd secondremoterepo
   $ hg up -q 0238718db2b1
   $ hg book master
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 1 of 6 remotenames-ext] selectivepull: add comments

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1485515845 28800
#  Fri Jan 27 03:17:25 2017 -0800
# Node ID 8b66569e49ce6382670497aa4e96c5e81c224439
# Parent  18f8e0f8ba54270bf158734c781327581cf43634
selectivepull: add comments

Explain why selectivepull is useful

diff --git a/remotenames.py b/remotenames.py
--- a/remotenames.py
+++ b/remotenames.py
@@ -101,6 +101,16 @@
 def expull(orig, repo, remote, *args, **kwargs):
 remotebookmarks = remote.listkeys('bookmarks')
 if _isselectivepull(repo.ui):
+# if selectivepull is enabled then we don't save all of the remote
+# bookmarks in remotenames file. Instead we save only bookmarks that
+# are "interesting" to a user. Moreover, "hg pull" without parameters
+# pulls only "interesting" bookmarks. There is a config option to
+# set default "interesting" bookmarks
+# (see _getselectivepulldefaultbookmarks).
+# Then bookmark is considered "interesting" if user did
+# "hg update REMOTE_BOOK_NAME" or "hg pull -B REMOTE_BOOK_NAME".
+# Selectivepull is helpful when server has too many remote bookmarks
+# because it may slow down clients.
 path = activepath(repo.ui, remote)
 bookmarks = {}
 for bookmark in readbookmarknames(repo, path):
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 6 of 6 remotenames-ext] selectivepull: support list of default bookmarks

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1485791649 28800
#  Mon Jan 30 07:54:09 2017 -0800
# Node ID 227796849698292c76c70e874179de52b7b688d6
# Parent  a96117003c763be640a975d8128068d2bd3527c0
selectivepull: support list of default bookmarks

The addition is that we now support list of default bookmarks, instead of a
single default bookmark (the previous behavior).

diff --git a/remotenames.py b/remotenames.py
--- a/remotenames.py
+++ b/remotenames.py
@@ -81,14 +81,19 @@
 return ui.configbool('remotenames', 'selectivepull', False)
 
 def _getselectivepulldefaultbookmarks(ui, remotebookmarks):
-default_book = ui.config('remotenames', 'selectivepulldefault')
-if not default_book:
-raise error.Abort(_('no default bookmark specified for selectivepull'))
-if default_book in remotebookmarks:
-return {default_book: remotebookmarks[default_book]}
-else:
+default_books = ui.configlist('remotenames', 'selectivepulldefault')
+if not default_books:
+raise error.Abort(_('no default bookmarks specified for 
selectivepull'))
+
+result = {}
+for default_book in default_books:
+if default_book in remotebookmarks:
+result[default_book] = remotebookmarks[default_book]
+
+if not default_books:
 raise error.Abort(
-_('default bookmark %s is not found on remote') % default_book)
+_('default bookmarks %s are not found on remote') % default_books)
+return result
 
 def _trypullremotebookmark(mayberemotebookmark, repo, ui):
 ui.warn(_('`%s` not found: assuming it is a remote bookmark '
diff --git a/tests/test-selective-pull.t b/tests/test-selective-pull.t
--- a/tests/test-selective-pull.t
+++ b/tests/test-selective-pull.t
@@ -227,3 +227,14 @@
   $ hg pull -q
   $ hg book --remote
  default/master2:0238718db2b1
+
+Set two bookmarks in selectivepulldefault, make sure both of them were pulled
+  $ cat >> .hg/hgrc << EOF
+  > [remotenames]
+  > selectivepulldefault=master,thirdbook
+  > EOF
+  $ rm .hg/selectivepullenabled
+  $ hg pull -q
+  $ hg book --remote
+ default/master2:0238718db2b1
+ default/thirdbook 0:1449e7934ec1
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH remotenames-ext] remotenames: lazily read remotenames file

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1483717320 28800
#  Fri Jan 06 07:42:00 2017 -0800
# Node ID 043090278718e0b82f6b3d8f7258d56080209d40
# Parent  c65e41aef8dc18f29f8652f9d71a9c0d028fd95d
remotenames: lazily read remotenames file

remotenames file is read every time even if it's not needed. That wastes up
to 100 ms on every call. Let's read it lazily instead.

diff --git a/remotenames.py b/remotenames.py
--- a/remotenames.py
+++ b/remotenames.py
@@ -282,7 +282,7 @@
 """Read-only dict-like Class to lazily resolve remotename entries
 
 We are doing that because remotenames startup was slow.
-We read the remotenames file once to figure out the potential entries
+We lazily read the remotenames file once to figure out the potential 
entries
 and store them in self.potentialentries. Then when asked to resolve an
 entry, if it is not in self.potentialentries, then it isn't there, if it
 is in self.potentialentries we resolve it and store the result in
@@ -293,10 +293,11 @@
 self.potentialentries = {}
 self._kind = kind # bookmarks or branches
 self._repo = repo
-self._load()
+self.loaded = False
 
 def _load(self):
 """Read the remotenames file, store entries matching selected kind"""
+self.loaded = True
 repo = self._repo
 alias_default = repo.ui.configbool('remotenames', 'alias.default')
 for node, nametype, remote, rname in readremotenames(repo):
@@ -328,6 +329,8 @@
 return [binnode]
 
 def __getitem__(self, key):
+if not self.loaded:
+self._load()
 val = self._fetchandcache(key)
 if val is not None:
 return val
@@ -345,6 +348,8 @@
 return None
 
 def keys(self):
+if not self.loaded:
+self._load()
 for u in self.potentialentries.keys():
 self._fetchandcache(u)
 return self.cache.keys()
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH evolve-ext] tests: fix tests to reflect hg core changes

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1483608997 28800
#  Thu Jan 05 01:36:37 2017 -0800
# Node ID 98b6678ee09c76aa471b119939935a9b67b1ff77
# Parent  ba9fabaca91b0897cffb7048d81193e44d440948
tests: fix tests to reflect hg core changes

In f05ede08dcf7d13794ccc9abb53877a50bf2b58b in core hg there were changes to
changeset_printer.

diff --git a/tests/test-obsolete.t b/tests/test-obsolete.t
--- a/tests/test-obsolete.t
+++ b/tests/test-obsolete.t
@@ -580,6 +580,7 @@
   parent:  10:2033b4e49474
   user:test
   date:Thu Jan 01 00:00:00 1970 +
+  trouble: bumped
   summary: add obsol_d'''
   
   $ hg push ../other-new/
@@ -609,6 +610,7 @@
   |/   parent:  10:2033b4e49474
   |user:test
   |date:Thu Jan 01 00:00:00 1970 +
+  |trouble: bumped
   |summary: add obsol_d'''
   |
   | o  changeset:   11:9468a5f5d8b2
@@ -672,6 +674,7 @@
   parent:  10:2033b4e49474
   user:test
   date:Thu Jan 01 00:00:00 1970 +
+  trouble: bumped, divergent
   summary: add obsol_d'''
   
   changeset:   16:50f11e5e3a63
@@ -679,6 +682,7 @@
   parent:  11:9468a5f5d8b2
   user:test
   date:Thu Jan 01 00:00:00 1970 +
+  trouble: divergent
   summary: add obsolet_conflicting_d
   
 
@@ -716,12 +720,14 @@
   |  parent:  2:4538525df7e2
   |  user:test
   |  date:Thu Jan 01 00:00:00 1970 +
+  |  trouble: unstable
   |  summary: add obsol_d''
   |
   | o  changeset:   16:50f11e5e3a63
   | |  parent:  11:9468a5f5d8b2
   | |  user:test
   | |  date:Thu Jan 01 00:00:00 1970 +
+  | |  trouble: divergent
   | |  summary: add obsolet_conflicting_d
   | |
   | | o  changeset:   15:705ab2a6b72e
@@ -745,6 +751,7 @@
   | | |/   parent:  10:2033b4e49474
   | | |user:test
   | | |date:Thu Jan 01 00:00:00 1970 +
+  | | |trouble: bumped, divergent
   | | |summary: add obsol_d'''
   | | |
   | o |  changeset:   11:9468a5f5d8b2
diff --git a/tests/test-stabilize-conflict.t b/tests/test-stabilize-conflict.t
--- a/tests/test-stabilize-conflict.t
+++ b/tests/test-stabilize-conflict.t
@@ -145,6 +145,7 @@
   | o  changeset:   5:71c18f70c34f
   | |  user:test
   | |  date:Thu Jan 01 00:00:00 1970 +
+  | |  trouble: unstable
   | |  summary: babar count up to fifteen
   | |
   | x  changeset:   4:5977072d13c5
@@ -235,6 +236,7 @@
   | o  changeset:   8:1836b91c6c1d
   | |  user:test
   | |  date:Thu Jan 01 00:00:00 1970 +
+  | |  trouble: unstable
   | |  summary: babar count up to fifteen
   | |
   | x  changeset:   7:e04690b09bc6
diff --git a/tests/test-stabilize-order.t b/tests/test-stabilize-order.t
--- a/tests/test-stabilize-order.t
+++ b/tests/test-stabilize-order.t
@@ -209,11 +209,13 @@
   | |  parent:  12:2256dae6521f
   | |  user:test
   | |  date:Thu Jan 01 00:00:00 1970 +
+  | |  trouble: unstable
   | |  summary: secondambiguous
   | |
   | | o  changeset:   13:bdc003b6eec2
   | |/   user:test
   | |date:Thu Jan 01 00:00:00 1970 +
+  | |trouble: unstable
   | |summary: firstambiguous
   | |
   | x  changeset:   12:2256dae6521f
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] bookmarks: introduce binary encoding

[Stanislau Hlebik]

Excerpts from Stanislau Hlebik's message of 2016-12-20 13:49:37 +:
> Excerpts from Pierre-Yves David's message of 2016-12-17 08:39:18 +0100:
> > 
> > On 12/09/2016 12:16 PM, Stanislau Hlebik wrote:
> > > # HG changeset patch
> > > # User Stanislau Hlebik <st...@fb.com>
> > > # Date 1481281951 28800
> > > #  Fri Dec 09 03:12:31 2016 -0800
> > > # Node ID 001ceadc2bc36699bdf816370899a27203bf1818
> > > # Parent  9e29d4e4e08b5996adda49cdd0b497d89e2b16ee
> > > bookmarks: introduce binary encoding
> > >
> > > Bookmarks binary encoding will be used for `bookmarks` bundle2 part.
> > > The format is: <4 bytes - bookmark size, big-endian>
> > ><1 byte - 0 if node is empty 1 otherwise><20 bytes node>
> > > ValueError maybe thrown if input is incorrect.
> > >
> > > diff --git a/mercurial/bookmarks.py b/mercurial/bookmarks.py
> > > --- a/mercurial/bookmarks.py
> > > +++ b/mercurial/bookmarks.py
> > > @@ -8,7 +8,9 @@
> > >  from __future__ import absolute_import
> > >
> > >  import errno
> > > +import io
> > >  import os
> > > +import struct
> > >
> > >  from .i18n import _
> > >  from .node import (
> > > @@ -23,6 +25,72 @@
> > >  util,
> > >  )
> > >
> > > +_NONEMPTYNODE = struct.pack('?', False)
> > > +_EMPTYNODE = struct.pack('?', True)
> > 
> > Our constant are still lower case ;-)
> > 
> > > +def _unpackbookmarksize(stream):
> > > +"""Returns 0 if stream is empty.
> > > +"""
> > > +
> > > +intstruct = struct.Struct('>i')
> > > +expectedsize = intstruct.size
> > > +encodedbookmarksize = stream.read(expectedsize)
> > > +if not encodedbookmarksize:
> > > +return 0
> > 
> > [small nits]
> > 
> > What does this "stream" empty case means?
> > 
> > If this is an error we should probably just error.
> > 
> > If this is an end condition, we could make that more explicit by 
> > returning None.
> >
> 
> It's an end condition, I'll change to None and update comment

Oh, and I won't be able to use changegroup.readexactly here because it
will throw exception if stream is empty

> 
> > > +if len(encodedbookmarksize) != expectedsize:
> > > +raise ValueError(
> > > +_('cannot decode bookmark size: '
> > > +  'expected size: %d, actual size: %d') %
> > > +(expectedsize, len(encodedbookmarksize)))
> > 
> > Check "changegroup.readexactly" it does this check for you.
> >
> 
> Thanks for the pointer. I noticed that it's used in couple of files
> already so it's not specific to changegroup. I think it makes sense to
> move it from changegroup.py to other file (maybe util.py?) What do you
> think?
> 
> > > +return intstruct.unpack(encodedbookmarksize)[0]
> > > +
> > > +def encodebookmarks(bookmarks):
> > > +"""Encodes bookmark to node mapping to the byte string.
> > > +
> > > +Format: <4 bytes - bookmark size>
> > > +<1 byte - 0 if node is empty 1 otherwise><20 bytes node>
> > > +Node may be 20 byte binary string or empty
> > > +"""
> > > +
> > > +intstruct = struct.Struct('>i')
> > > +for bookmark, node in sorted(bookmarks.iteritems()):
> > > +encodedbookmark = encoding.fromlocal(bookmark)
> > > +yield intstruct.pack(len(encodedbookmark))
> > > +yield encodedbookmark
> > > +if node:
> > > +if len(node) != 20:
> > > +raise ValueError(_('node must be 20 or bytes long'))
> > 
> > Is there case where the content  of the node can be wrong ? if not, I 
> > would probably just use an assert.
> 
> Will fix
> 
> > 
> > > +yield _NONEMPTYNODE
> > > +yield node
> > > +else:
> > > +yield _EMPTYNODE
> > > +
> > > +def decodebookmarks(buf):
> > > +"""Decodes buffer into bookmark to node mapping.
> > > +
> > > +Node is either 20 bytes or empty.
> > > +"""
> > > +
> > > +stream = io.BytesIO(buf)
> > > +bookmarks = {}
> > > +bookmarksize = _unpackbookmarksize(stream

Re: [PATCH] bookmarks: introduce binary encoding

[Stanislau Hlebik]

Excerpts from Pierre-Yves David's message of 2016-12-17 08:39:18 +0100:
> 
> On 12/09/2016 12:16 PM, Stanislau Hlebik wrote:
> > # HG changeset patch
> > # User Stanislau Hlebik <st...@fb.com>
> > # Date 1481281951 28800
> > #  Fri Dec 09 03:12:31 2016 -0800
> > # Node ID 001ceadc2bc36699bdf816370899a27203bf1818
> > # Parent  9e29d4e4e08b5996adda49cdd0b497d89e2b16ee
> > bookmarks: introduce binary encoding
> >
> > Bookmarks binary encoding will be used for `bookmarks` bundle2 part.
> > The format is: <4 bytes - bookmark size, big-endian>
> ><1 byte - 0 if node is empty 1 otherwise><20 bytes node>
> > ValueError maybe thrown if input is incorrect.
> >
> > diff --git a/mercurial/bookmarks.py b/mercurial/bookmarks.py
> > --- a/mercurial/bookmarks.py
> > +++ b/mercurial/bookmarks.py
> > @@ -8,7 +8,9 @@
> >  from __future__ import absolute_import
> >
> >  import errno
> > +import io
> >  import os
> > +import struct
> >
> >  from .i18n import _
> >  from .node import (
> > @@ -23,6 +25,72 @@
> >  util,
> >  )
> >
> > +_NONEMPTYNODE = struct.pack('?', False)
> > +_EMPTYNODE = struct.pack('?', True)
> 
> Our constant are still lower case ;-)
> 
> > +def _unpackbookmarksize(stream):
> > +"""Returns 0 if stream is empty.
> > +"""
> > +
> > +intstruct = struct.Struct('>i')
> > +expectedsize = intstruct.size
> > +encodedbookmarksize = stream.read(expectedsize)
> > +if not encodedbookmarksize:
> > +return 0
> 
> [small nits]
> 
> What does this "stream" empty case means?
> 
> If this is an error we should probably just error.
> 
> If this is an end condition, we could make that more explicit by 
> returning None.
>

It's an end condition, I'll change to None and update comment

> > +if len(encodedbookmarksize) != expectedsize:
> > +raise ValueError(
> > +_('cannot decode bookmark size: '
> > +  'expected size: %d, actual size: %d') %
> > +(expectedsize, len(encodedbookmarksize)))
> 
> Check "changegroup.readexactly" it does this check for you.
>

Thanks for the pointer. I noticed that it's used in couple of files
already so it's not specific to changegroup. I think it makes sense to
move it from changegroup.py to other file (maybe util.py?) What do you
think?

> > +return intstruct.unpack(encodedbookmarksize)[0]
> > +
> > +def encodebookmarks(bookmarks):
> > +"""Encodes bookmark to node mapping to the byte string.
> > +
> > +Format: <4 bytes - bookmark size>
> > +<1 byte - 0 if node is empty 1 otherwise><20 bytes node>
> > +Node may be 20 byte binary string or empty
> > +"""
> > +
> > +intstruct = struct.Struct('>i')
> > +for bookmark, node in sorted(bookmarks.iteritems()):
> > +encodedbookmark = encoding.fromlocal(bookmark)
> > +yield intstruct.pack(len(encodedbookmark))
> > +yield encodedbookmark
> > +if node:
> > +if len(node) != 20:
> > +raise ValueError(_('node must be 20 or bytes long'))
> 
> Is there case where the content  of the node can be wrong ? if not, I 
> would probably just use an assert.

Will fix

> 
> > +yield _NONEMPTYNODE
> > +yield node
> > +else:
> > +yield _EMPTYNODE
> > +
> > +def decodebookmarks(buf):
> > +"""Decodes buffer into bookmark to node mapping.
> > +
> > +Node is either 20 bytes or empty.
> > +"""
> > +
> > +stream = io.BytesIO(buf)
> > +bookmarks = {}
> > +bookmarksize = _unpackbookmarksize(stream)
> > +boolstruct = struct.Struct('?')
> > +while bookmarksize:
> > +bookmark = stream.read(bookmarksize)
> > +if len(bookmark) != bookmarksize:
> > +raise ValueError(
> > +_('cannot decode bookmark: expected size: %d, '
> > +'actual size: %d') % (bookmarksize, len(bookmark)))
> 
> CF previous comment about changegroup.readexactly.
> 
> > +bookmark = encoding.tolocal(bookmark)
> > +packedemptynodeflag = stream.read(boolstruct.size)
> > +emptynode = boolstruct.unpack(packedemptynodeflag)[0]
> > +node = ''
> > +if not emptynode:
> > +node = stream.read(20)
> 
> lalala, changegroup.readexactly.
> 
> > +bookmarks[bookmark] = node
> > +bookmarksize = _unpackbookmarksize(stream)
> > +return bookmarks
> > +
> >  def _getbkfile(repo):
> >  """Hook so that extensions that mess with the store can hook bm 
> > storage.
> 
> Cheers,
> 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 2 of 2] bookmarks: make bookmarks.comparebookmarks accept binary nodes (API)

[Stanislau Hlebik]

Excerpts from Pierre-Yves David's message of 2016-12-17 08:33:20 +0100:
> 
> On 12/09/2016 12:31 PM, Stanislau Hlebik wrote:
> > # HG changeset patch
> > # User Stanislau Hlebik <st...@fb.com>
> > # Date 1481282546 28800
> > #  Fri Dec 09 03:22:26 2016 -0800
> > # Node ID df861963a18c00d72362b415a77a62d2c18660bf
> > # Parent  08ab8f9d0abcbd1b2405ecb0a8670d212866af1f
> > bookmarks: make bookmarks.comparebookmarks accept binary nodes (API)
> >
> > Binary bookmark format should be used internally. It doesn't make sense to 
> > have
> > optional parameters `srchex` and `dsthex`. This patch removes them. It will
> > also be useful for `bookmarks` bundle2 part because unnecessary conversions
> > between hex and bin nodes will be avoided.
> 
> Great, I've put a second acceptance stamp on this changeset and it 
> should show as public shortly.
> 
> There is a couple thing in this patch that highlight the need for extra 
> work on that topic to remove all list key call from the client/server 
> exchange. I'm not sure if we should have something special to track 
> these, but see my comment below.
> 
> > diff --git a/mercurial/bookmarks.py b/mercurial/bookmarks.py
> > --- a/mercurial/bookmarks.py
> > +++ b/mercurial/bookmarks.py
> > @@ -391,8 +391,7 @@
> >  finally:
> >  lockmod.release(tr, l, w)
> >
> > -def comparebookmarks(repo, srcmarks, dstmarks,
> > - srchex=None, dsthex=None, targets=None):
> > +def comparebookmarks(repo, srcmarks, dstmarks, targets=None):
> >  '''Compare bookmarks between srcmarks and dstmarks
> >
> >  This returns tuple "(addsrc, adddst, advsrc, advdst, diverge,
> > @@ -415,19 +414,9 @@
> >  Changeset IDs of tuples in "addsrc", "adddst", "differ" or
> >   "invalid" list may be unknown for repo.
> >
> > -This function expects that "srcmarks" and "dstmarks" return
> > -changeset ID in 40 hexadecimal digit string for specified
> > -bookmark. If not so (e.g. bmstore "repo._bookmarks" returning
> > -binary value), "srchex" or "dsthex" should be specified to convert
> > -into such form.
> > -
> >  If "targets" is specified, only bookmarks listed in it are
> >  examined.
> >  '''
> > -if not srchex:
> > -srchex = lambda x: x
> > -if not dsthex:
> > -dsthex = lambda x: x
> >
> >  if targets:
> >  bset = set(targets)
> > @@ -449,14 +438,14 @@
> >  for b in sorted(bset):
> >  if b not in srcmarks:
> >  if b in dstmarks:
> > -adddst((b, None, dsthex(dstmarks[b])))
> > +adddst((b, None, dstmarks[b]))
> >  else:
> >  invalid((b, None, None))
> >  elif b not in dstmarks:
> > -addsrc((b, srchex(srcmarks[b]), None))
> > +addsrc((b, srcmarks[b], None))
> >  else:
> > -scid = srchex(srcmarks[b])
> > -dcid = dsthex(dstmarks[b])
> > +scid = srcmarks[b]
> > +dcid = dstmarks[b]
> >  if scid == dcid:
> >  same((b, scid, dcid))
> >  elif scid in repo and dcid in repo:
> > @@ -507,11 +496,17 @@
> >
> >  return None
> >
> > +def unhexlifybookmarks(marks):
> > +binremotemarks = {}
> > +for name, node in marks.items():
> > +binremotemarks[name] = bin(node)
> > +return binremotemarks
> > +
> 
> This function looked suspicious and was the start of my quest to know 
> more. My thinking seeing this was, "wait", if we moved internal to 
> binary why do we still needs it?
> 
> Keep scrolling for the result of that investigation.
> 
> >  def updatefromremote(ui, repo, remotemarks, path, trfunc, explicit=()):
> >  ui.debug("checking for updated bookmarks\n")
> >  localmarks = repo._bookmarks
> >  (addsrc, adddst, advsrc, advdst, diverge, differ, invalid, same
> > - ) = comparebookmarks(repo, remotemarks, localmarks, dsthex=hex)
> > +) = comparebookmarks(repo, remotemarks, localmarks)
> >
> >  status = ui.status
> >  warn = ui.warn
> > @@ -522,15 +517,15 @@
> >  changed = []
> >  for b, scid, dcid in addsrc:
> >  if scid in repo: # add remote bookmarks for changes we already have
> > -changed.append((b, b

[PATCH] cg1packer: fix `compressed` method

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1481738036 28800
#  Wed Dec 14 09:53:56 2016 -0800
# Node ID f511d71c272bf87ea263cdcda3045e26fd5f8dfe
# Parent  4cdc738f8246c748dc1639754d0e7ced97d15e23
cg1packer: fix `compressed` method

`cg1packer.compressed()` returns True even if `self._type` is 'UN'. This patch
fixes it.

diff --git a/mercurial/changegroup.py b/mercurial/changegroup.py
--- a/mercurial/changegroup.py
+++ b/mercurial/changegroup.py
@@ -154,7 +154,7 @@
 # These methods (compressed, read, seek, tell) all appear to only
 # be used by bundlerepo, but it's a little hard to tell.
 def compressed(self):
-return self._type is not None
+return self._type is not None and self._type != 'UN'
 def read(self, l):
 return self._stream.read(l)
 def seek(self, pos):
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 2] bookmarks: make bookmarks.comparebookmarks accept binary nodes (API)

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1481282546 28800
#  Fri Dec 09 03:22:26 2016 -0800
# Node ID df861963a18c00d72362b415a77a62d2c18660bf
# Parent  08ab8f9d0abcbd1b2405ecb0a8670d212866af1f
bookmarks: make bookmarks.comparebookmarks accept binary nodes (API)

Binary bookmark format should be used internally. It doesn't make sense to have
optional parameters `srchex` and `dsthex`. This patch removes them. It will
also be useful for `bookmarks` bundle2 part because unnecessary conversions
between hex and bin nodes will be avoided.

diff --git a/mercurial/bookmarks.py b/mercurial/bookmarks.py
--- a/mercurial/bookmarks.py
+++ b/mercurial/bookmarks.py
@@ -391,8 +391,7 @@
 finally:
 lockmod.release(tr, l, w)
 
-def comparebookmarks(repo, srcmarks, dstmarks,
- srchex=None, dsthex=None, targets=None):
+def comparebookmarks(repo, srcmarks, dstmarks, targets=None):
 '''Compare bookmarks between srcmarks and dstmarks
 
 This returns tuple "(addsrc, adddst, advsrc, advdst, diverge,
@@ -415,19 +414,9 @@
 Changeset IDs of tuples in "addsrc", "adddst", "differ" or
  "invalid" list may be unknown for repo.
 
-This function expects that "srcmarks" and "dstmarks" return
-changeset ID in 40 hexadecimal digit string for specified
-bookmark. If not so (e.g. bmstore "repo._bookmarks" returning
-binary value), "srchex" or "dsthex" should be specified to convert
-into such form.
-
 If "targets" is specified, only bookmarks listed in it are
 examined.
 '''
-if not srchex:
-srchex = lambda x: x
-if not dsthex:
-dsthex = lambda x: x
 
 if targets:
 bset = set(targets)
@@ -449,14 +438,14 @@
 for b in sorted(bset):
 if b not in srcmarks:
 if b in dstmarks:
-adddst((b, None, dsthex(dstmarks[b])))
+adddst((b, None, dstmarks[b]))
 else:
 invalid((b, None, None))
 elif b not in dstmarks:
-addsrc((b, srchex(srcmarks[b]), None))
+addsrc((b, srcmarks[b], None))
 else:
-scid = srchex(srcmarks[b])
-dcid = dsthex(dstmarks[b])
+scid = srcmarks[b]
+dcid = dstmarks[b]
 if scid == dcid:
 same((b, scid, dcid))
 elif scid in repo and dcid in repo:
@@ -507,11 +496,17 @@
 
 return None
 
+def unhexlifybookmarks(marks):
+binremotemarks = {}
+for name, node in marks.items():
+binremotemarks[name] = bin(node)
+return binremotemarks
+
 def updatefromremote(ui, repo, remotemarks, path, trfunc, explicit=()):
 ui.debug("checking for updated bookmarks\n")
 localmarks = repo._bookmarks
 (addsrc, adddst, advsrc, advdst, diverge, differ, invalid, same
- ) = comparebookmarks(repo, remotemarks, localmarks, dsthex=hex)
+) = comparebookmarks(repo, remotemarks, localmarks)
 
 status = ui.status
 warn = ui.warn
@@ -522,15 +517,15 @@
 changed = []
 for b, scid, dcid in addsrc:
 if scid in repo: # add remote bookmarks for changes we already have
-changed.append((b, bin(scid), status,
+changed.append((b, scid, status,
 _("adding remote bookmark %s\n") % (b)))
 elif b in explicit:
 explicit.remove(b)
 ui.warn(_("remote bookmark %s points to locally missing %s\n")
-% (b, scid[:12]))
+% (b, hex(scid)[:12]))
 
 for b, scid, dcid in advsrc:
-changed.append((b, bin(scid), status,
+changed.append((b, scid, status,
 _("updating bookmark %s\n") % (b)))
 # remove normal movement from explicit set
 explicit.difference_update(d[0] for d in changed)
@@ -538,13 +533,12 @@
 for b, scid, dcid in diverge:
 if b in explicit:
 explicit.discard(b)
-changed.append((b, bin(scid), status,
+changed.append((b, scid, status,
 _("importing bookmark %s\n") % (b)))
 else:
-snode = bin(scid)
-db = _diverge(ui, b, path, localmarks, snode)
+db = _diverge(ui, b, path, localmarks, scid)
 if db:
-changed.append((db, snode, warn,
+changed.append((db, scid, warn,
 _("divergent bookmark %s stored as %s\n") %
 (b, db)))
 else:
@@ -553,13 +547,13 @@
 for b, scid, dcid in adddst + advdst:
 if b in explicit:
 explicit.discard(b)
-changed.append((b, bin(scid), status,
+changed.append((b, scid, status,
 _("importing bookmark %s\n&

[PATCH] bookmarks: introduce binary encoding

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1481281951 28800
#  Fri Dec 09 03:12:31 2016 -0800
# Node ID 001ceadc2bc36699bdf816370899a27203bf1818
# Parent  9e29d4e4e08b5996adda49cdd0b497d89e2b16ee
bookmarks: introduce binary encoding

Bookmarks binary encoding will be used for `bookmarks` bundle2 part.
The format is: <4 bytes - bookmark size, big-endian>
   <1 byte - 0 if node is empty 1 otherwise><20 bytes node>
ValueError maybe thrown if input is incorrect.

diff --git a/mercurial/bookmarks.py b/mercurial/bookmarks.py
--- a/mercurial/bookmarks.py
+++ b/mercurial/bookmarks.py
@@ -8,7 +8,9 @@
 from __future__ import absolute_import
 
 import errno
+import io
 import os
+import struct
 
 from .i18n import _
 from .node import (
@@ -23,6 +25,72 @@
 util,
 )
 
+_NONEMPTYNODE = struct.pack('?', False)
+_EMPTYNODE = struct.pack('?', True)
+
+def _unpackbookmarksize(stream):
+"""Returns 0 if stream is empty.
+"""
+
+intstruct = struct.Struct('>i')
+expectedsize = intstruct.size
+encodedbookmarksize = stream.read(expectedsize)
+if not encodedbookmarksize:
+return 0
+if len(encodedbookmarksize) != expectedsize:
+raise ValueError(
+_('cannot decode bookmark size: '
+  'expected size: %d, actual size: %d') %
+(expectedsize, len(encodedbookmarksize)))
+return intstruct.unpack(encodedbookmarksize)[0]
+
+def encodebookmarks(bookmarks):
+"""Encodes bookmark to node mapping to the byte string.
+
+Format: <4 bytes - bookmark size>
+<1 byte - 0 if node is empty 1 otherwise><20 bytes node>
+Node may be 20 byte binary string or empty
+"""
+
+intstruct = struct.Struct('>i')
+for bookmark, node in sorted(bookmarks.iteritems()):
+encodedbookmark = encoding.fromlocal(bookmark)
+yield intstruct.pack(len(encodedbookmark))
+yield encodedbookmark
+if node:
+if len(node) != 20:
+raise ValueError(_('node must be 20 or bytes long'))
+yield _NONEMPTYNODE
+yield node
+else:
+yield _EMPTYNODE
+
+def decodebookmarks(buf):
+"""Decodes buffer into bookmark to node mapping.
+
+Node is either 20 bytes or empty.
+"""
+
+stream = io.BytesIO(buf)
+bookmarks = {}
+bookmarksize = _unpackbookmarksize(stream)
+boolstruct = struct.Struct('?')
+while bookmarksize:
+bookmark = stream.read(bookmarksize)
+if len(bookmark) != bookmarksize:
+raise ValueError(
+_('cannot decode bookmark: expected size: %d, '
+'actual size: %d') % (bookmarksize, len(bookmark)))
+bookmark = encoding.tolocal(bookmark)
+packedemptynodeflag = stream.read(boolstruct.size)
+emptynode = boolstruct.unpack(packedemptynodeflag)[0]
+node = ''
+if not emptynode:
+node = stream.read(20)
+bookmarks[bookmark] = node
+bookmarksize = _unpackbookmarksize(stream)
+return bookmarks
+
 def _getbkfile(repo):
 """Hook so that extensions that mess with the store can hook bm storage.
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 8 of 8 V12] help: add documentation about bookmark part

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1479807361 28800
#  Tue Nov 22 01:36:01 2016 -0800
# Node ID 261bc4a0670a47a8fbe8d633f19f92542be4790c
# Parent  dc34527b28959ef96a4e7b23430ad467b5fbf85e
help: add documentation about bookmark part

diff --git a/mercurial/help/internals/bundles.txt 
b/mercurial/help/internals/bundles.txt
--- a/mercurial/help/internals/bundles.txt
+++ b/mercurial/help/internals/bundles.txt
@@ -92,3 +92,32 @@
 ``HGS1UN`` support was added as an experimental feature in version 3.6
 (released November 2015) as part of the initial offering of the *clone
 bundles* feature.
+
+Bundle2 parts
+=
+
+Bundle2 may contain many different pieces of information. These pieces are
+called parts.
+
+Bookmarks part
+--
+
+This part contains information about bookmarks. Part consists of many entries.
+Each entry describes one bookmark. Entry format:
+
+4 bytes
+  bookmark size
+1 byte
+  boolean. True if node is empty, False otherwise
+20 bytes (optional)
+  node. Present only if previous field is True
+
+Modes:
+
+1. 'ignore' - do not apply any changes to the repo, just decode the passed
+bookmarks. Will be used to list bookmarks in remote repo.
+2. 'diverge' - apply bookmark changes to the repo. Create divergent bookmarks 
if
+there is a non-fastforward move. Will be used during pull.
+3. 'apply' - apply bookmark changes to the repo. Overwrite current bookmark 
node
+if there is a non-fastforward move. Will be used during push.
+
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 4 of 8 V12] bundle2: add `bookmarks` part handler

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1479807361 28800
#  Tue Nov 22 01:36:01 2016 -0800
# Node ID 49d598a6db4b0fa63b7a30b55899caa0fa1f9c99
# Parent  81854771326941912a0dfd80f4f0c9d0191eea75
bundle2: add `bookmarks` part handler

Applies bookmarks part to the local repo. `processbookmarksmode` determines
how remote bookmarks are handled. They are either ignored ('ignore' mode),
diverged ('diverge' mode) or applied ('apply' mode).

diff --git a/mercurial/bundle2.py b/mercurial/bundle2.py
--- a/mercurial/bundle2.py
+++ b/mercurial/bundle2.py
@@ -155,6 +155,7 @@
 
 from .i18n import _
 from . import (
+bookmarks as bookmod,
 changegroup,
 error,
 obsolete,
@@ -287,13 +288,19 @@
 * a way to construct a bundle response when applicable.
 """
 
-def __init__(self, repo, transactiongetter, captureoutput=True):
+def __init__(self, repo, transactiongetter, captureoutput=True,
+ behavior=None):
+"""
+`behavior` is a dictionary that is passed to part handlers to tweak
+their behaviour
+"""
 self.repo = repo
 self.ui = repo.ui
 self.records = unbundlerecords()
 self.gettransaction = transactiongetter
 self.reply = None
 self.captureoutput = captureoutput
+self.behavior = behavior or {}
 
 class TransactionUnavailable(RuntimeError):
 pass
@@ -1616,3 +1623,35 @@
 
 cache.write()
 op.ui.debug('applied %i hgtags fnodes cache entries\n' % count)
+
+@parthandler('bookmarks')
+def handlebookmarks(op, inpart):
+"""Processes bookmarks part.
+
+`processbookmarksmode` determines how remote bookmarks are handled. They 
are
+either ignored ('ignore' mode), diverged ('diverge' mode) or applied
+('apply' mode). 'ignore' mode is used to get bookmarks and process them
+later, 'diverge' mode is used to process bookmarks during pull, 'apply'
+mode is used during push.
+"""
+
+bookmarks = bookmod.decodebookmarks(inpart.read())
+processbookmarksmode = op.behavior.get('processbookmarksmode', 'ignore')
+if processbookmarksmode == 'apply':
+for bookmark, node in bookmarks.items():
+if node:
+op.repo._bookmarks[bookmark] = node
+else:
+try:
+del op.repo._bookmarks[bookmark]
+except KeyError:
+# ignore if bookmark does not exist
+pass
+op.repo._bookmarks.recordchange(op.gettransaction())
+elif processbookmarksmode == 'diverge':
+remotepath = op.behavior.get('remotepath', '')
+explicitbookmarks = op.behavior.get('explicitbookmarks', ())
+bookmod.updatefromremote(op.ui, op.repo, bookmarks,
+ remotepath, op.gettransaction,
+ explicit=explicitbookmarks)
+op.records.add('bookmarks', bookmarks)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 8 V12] bookmarks: rename `compare()` to `comparebookmarks()` (API)

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1479807211 28800
#  Tue Nov 22 01:33:31 2016 -0800
# Node ID 0fd97a0c043a3f5c1d4ba050435d86e09dbd0f54
# Parent  d184b91890ae3cbd21f656847b6b2728892b2425
bookmarks: rename `compare()` to `comparebookmarks()` (API)

Next commit will remove optional parameters from `compare()` function.
Let's rename `compare()` to `comparebookmarks()` to avoid ambiguity from
callers from external extensions.

diff --git a/mercurial/bookmarks.py b/mercurial/bookmarks.py
--- a/mercurial/bookmarks.py
+++ b/mercurial/bookmarks.py
@@ -391,8 +391,8 @@
 finally:
 lockmod.release(tr, l, w)
 
-def compare(repo, srcmarks, dstmarks,
-srchex=None, dsthex=None, targets=None):
+def comparebookmarks(repo, srcmarks, dstmarks,
+ srchex=None, dsthex=None, targets=None):
 '''Compare bookmarks between srcmarks and dstmarks
 
 This returns tuple "(addsrc, adddst, advsrc, advdst, diverge,
@@ -511,7 +511,7 @@
 ui.debug("checking for updated bookmarks\n")
 localmarks = repo._bookmarks
 (addsrc, adddst, advsrc, advdst, diverge, differ, invalid, same
- ) = compare(repo, remotemarks, localmarks, dsthex=hex)
+ ) = comparebookmarks(repo, remotemarks, localmarks, dsthex=hex)
 
 status = ui.status
 warn = ui.warn
@@ -573,8 +573,8 @@
 '''
 ui.status(_("searching for changed bookmarks\n"))
 
-r = compare(repo, other.listkeys('bookmarks'), repo._bookmarks,
-dsthex=hex)
+r = comparebookmarks(repo, other.listkeys('bookmarks'), repo._bookmarks,
+ dsthex=hex)
 addsrc, adddst, advsrc, advdst, diverge, differ, invalid, same = r
 
 incomings = []
@@ -615,8 +615,8 @@
 '''
 ui.status(_("searching for changed bookmarks\n"))
 
-r = compare(repo, repo._bookmarks, other.listkeys('bookmarks'),
-srchex=hex)
+r = comparebookmarks(repo, repo._bookmarks, other.listkeys('bookmarks'),
+ srchex=hex)
 addsrc, adddst, advsrc, advdst, diverge, differ, invalid, same = r
 
 outgoings = []
@@ -660,8 +660,8 @@
 
 This returns "(# of incoming, # of outgoing)" tuple.
 '''
-r = compare(repo, other.listkeys('bookmarks'), repo._bookmarks,
-dsthex=hex)
+r = comparebookmarks(repo, other.listkeys('bookmarks'), repo._bookmarks,
+ dsthex=hex)
 addsrc, adddst, advsrc, advdst, diverge, differ, invalid, same = r
 return (len(addsrc), len(adddst))
 
diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -603,7 +603,8 @@
 explicit = set([repo._bookmarks.expandname(bookmark)
 for bookmark in pushop.bookmarks])
 
-comp = bookmod.compare(repo, repo._bookmarks, remotebookmark, srchex=hex)
+comp = bookmod.comparebookmarks(repo, repo._bookmarks,
+remotebookmark, srchex=hex)
 addsrc, adddst, advsrc, advdst, diverge, differ, invalid, same = comp
 for b, scid, dcid in advsrc:
 if b in explicit:
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 5 of 8 V12] exchange: getbundle `bookmarks` part generator

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1479807361 28800
#  Tue Nov 22 01:36:01 2016 -0800
# Node ID b08e4f9ed873bb9b6bd8e170401a150f9b094bfd
# Parent  49d598a6db4b0fa63b7a30b55899caa0fa1f9c99
exchange: getbundle `bookmarks` part generator

This generator will be used during pull operation.

diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -1675,6 +1675,21 @@
 if chunks:
 bundler.newpart('hgtagsfnodes', data=''.join(chunks))
 
+@getbundle2partsgenerator('bookmarks')
+def _getbundlebookmarkspart(bundler, repo, source, bundlecaps=None,
+b2caps=None, heads=None, common=None,
+**kwargs):
+if not kwargs.get('bookmarks'):
+return
+if 'bookmarks' not in b2caps:
+raise ValueError(
+_('bookmarks are requested but client is not capable '
+  'of receiving it'))
+
+bookmarks = _getbookmarks(repo, **kwargs)
+encodedbookmarks = bookmod.encodebookmarks(bookmarks)
+bundler.newpart('bookmarks', data=encodedbookmarks)
+
 def _getbookmarks(repo, **kwargs):
 """Returns bookmark to node mapping.
 
diff --git a/mercurial/wireproto.py b/mercurial/wireproto.py
--- a/mercurial/wireproto.py
+++ b/mercurial/wireproto.py
@@ -212,7 +212,9 @@
  'bundlecaps': 'scsv',
  'listkeys': 'csv',
  'cg': 'boolean',
- 'cbattempted': 'boolean'}
+ 'cbattempted': 'boolean',
+ 'bookmarks': 'boolean',
+}
 
 # client side
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 7 of 8 V12] bundle2: advertise bookmark capability

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1479807361 28800
#  Tue Nov 22 01:36:01 2016 -0800
# Node ID dc34527b28959ef96a4e7b23430ad467b5fbf85e
# Parent  be39a83d5cb8c60e9b9a2a1b58b4b0dde833c77b
bundle2: advertise bookmark capability

diff --git a/mercurial/bundle2.py b/mercurial/bundle2.py
--- a/mercurial/bundle2.py
+++ b/mercurial/bundle2.py
@@ -1265,6 +1265,7 @@
 'digests': tuple(sorted(util.DIGESTS.keys())),
 'remote-changegroup': ('http', 'https'),
 'hgtagsfnodes': (),
+'bookmarks': (),
}
 
 def getrepocaps(repo, allowpushback=False):
diff --git a/tests/test-acl.t b/tests/test-acl.t
--- a/tests/test-acl.t
+++ b/tests/test-acl.t
@@ -92,13 +92,13 @@
   f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd
   911600dab2ae7a9baff75958b84fe606851ce955
   bundle2-output-bundle: "HG20", 4 parts total
-  bundle2-output-part: "replycaps" 155 bytes payload
+  bundle2-output-part: "replycaps" 165 bytes payload
   bundle2-output-part: "check:heads" streamed payload
   bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload
   bundle2-output-part: "pushkey" (params: 4 mandatory) empty payload
   bundle2-input-bundle: with-transaction
   bundle2-input-part: "replycaps" supported
-  bundle2-input-part: total payload size 155
+  bundle2-input-part: total payload size 165
   bundle2-input-part: "check:heads" supported
   bundle2-input-part: total payload size 20
   bundle2-input-part: "changegroup" (params: 1 mandatory) supported
@@ -155,13 +155,13 @@
   f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd
   911600dab2ae7a9baff75958b84fe606851ce955
   bundle2-output-bundle: "HG20", 4 parts total
-  bundle2-output-part: "replycaps" 155 bytes payload
+  bundle2-output-part: "replycaps" 165 bytes payload
   bundle2-output-part: "check:heads" streamed payload
   bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload
   bundle2-output-part: "pushkey" (params: 4 mandatory) empty payload
   bundle2-input-bundle: with-transaction
   bundle2-input-part: "replycaps" supported
-  bundle2-input-part: total payload size 155
+  bundle2-input-part: total payload size 165
   bundle2-input-part: "check:heads" supported
   bundle2-input-part: total payload size 20
   bundle2-input-part: "changegroup" (params: 1 mandatory) supported
@@ -221,13 +221,13 @@
   f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd
   911600dab2ae7a9baff75958b84fe606851ce955
   bundle2-output-bundle: "HG20", 4 parts total
-  bundle2-output-part: "replycaps" 155 bytes payload
+  bundle2-output-part: "replycaps" 165 bytes payload
   bundle2-output-part: "check:heads" streamed payload
   bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload
   bundle2-output-part: "pushkey" (params: 4 mandatory) empty payload
   bundle2-input-bundle: with-transaction
   bundle2-input-part: "replycaps" supported
-  bundle2-input-part: total payload size 155
+  bundle2-input-part: total payload size 165
   bundle2-input-part: "check:heads" supported
   bundle2-input-part: total payload size 20
   bundle2-input-part: "changegroup" (params: 1 mandatory) supported
@@ -297,13 +297,13 @@
   f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd
   911600dab2ae7a9baff75958b84fe606851ce955
   bundle2-output-bundle: "HG20", 4 parts total
-  bundle2-output-part: "replycaps" 155 bytes payload
+  bundle2-output-part: "replycaps" 165 bytes payload
   bundle2-output-part: "check:heads" streamed payload
   bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload
   bundle2-output-part: "pushkey" (params: 4 mandatory) empty payload
   bundle2-input-bundle: with-transaction
   bundle2-input-part: "replycaps" supported
-  bundle2-input-part: total payload size 155
+  bundle2-input-part: total payload size 165
   bundle2-input-part: "check:heads" supported
   bundle2-input-part: total payload size 20
   bundle2-input-part: "changegroup" (params: 1 mandatory) supported
@@ -362,13 +362,13 @@
   f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd
   911600dab2ae7a9baff75958b84fe606851ce955
   bundle2-output-bundle: "HG20", 4 parts total
-  bundle2-output-part: "replycaps" 155 bytes payload
+  bundle2-output-part: "replycaps" 165 bytes payload
   bundle2-output-part: "check:heads" streamed payload
   bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload
   bundle2-output-part: "pushkey" (params: 4 mandatory) empty payload
   bundle2-input-bundle: with-transaction
   bundle2-input-part: "replycaps" supported
-  bundle2-input-par

[PATCH 1 of 8 V12] bookmarks: introduce binary encoding

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1479807211 28800
#  Tue Nov 22 01:33:31 2016 -0800
# Node ID d184b91890ae3cbd21f656847b6b2728892b2425
# Parent  9e29d4e4e08b5996adda49cdd0b497d89e2b16ee
bookmarks: introduce binary encoding

Bookmarks binary encoding will be used for `bookmarks` bundle2 part.
The format is: <4 bytes - bookmark size, big-endian>
   <1 byte - 0 if node is empty 1 otherwise><20 bytes node>
BookmarksEncodeError and BookmarksDecodeError maybe thrown if input is
incorrect.

diff --git a/mercurial/bookmarks.py b/mercurial/bookmarks.py
--- a/mercurial/bookmarks.py
+++ b/mercurial/bookmarks.py
@@ -8,7 +8,9 @@
 from __future__ import absolute_import
 
 import errno
+import io
 import os
+import struct
 
 from .i18n import _
 from .node import (
@@ -23,6 +25,72 @@
 util,
 )
 
+_NONEMPTYNODE = struct.pack('?', False)
+_EMPTYNODE = struct.pack('?', True)
+
+def _unpackbookmarksize(stream):
+"""Returns 0 if stream is empty.
+"""
+
+intstruct = struct.Struct('>i')
+expectedsize = intstruct.size
+encodedbookmarksize = stream.read(expectedsize)
+if not encodedbookmarksize:
+return 0
+if len(encodedbookmarksize) != expectedsize:
+raise ValueError(
+_('cannot decode bookmark size: '
+  'expected size: %d, actual size: %d') %
+(expectedsize, len(encodedbookmarksize)))
+return intstruct.unpack(encodedbookmarksize)[0]
+
+def encodebookmarks(bookmarks):
+"""Encodes bookmark to node mapping to the byte string.
+
+Format: <4 bytes - bookmark size>
+<1 byte - 0 if node is empty 1 otherwise><20 bytes node>
+Node may be 20 byte binary string or empty
+"""
+
+intstruct = struct.Struct('>i')
+for bookmark, node in sorted(bookmarks.iteritems()):
+encodedbookmark = encoding.fromlocal(bookmark)
+yield intstruct.pack(len(encodedbookmark))
+yield encodedbookmark
+if node:
+if len(node) != 20:
+raise ValueError(_('node must be 20 or bytes long'))
+yield _NONEMPTYNODE
+yield node
+else:
+yield _EMPTYNODE
+
+def decodebookmarks(buf):
+"""Decodes buffer into bookmark to node mapping.
+
+Node is either 20 bytes or empty.
+"""
+
+stream = io.BytesIO(buf)
+bookmarks = {}
+bookmarksize = _unpackbookmarksize(stream)
+boolstruct = struct.Struct('?')
+while bookmarksize:
+bookmark = stream.read(bookmarksize)
+if len(bookmark) != bookmarksize:
+raise ValueError(
+_('cannot decode bookmark: expected size: %d, '
+'actual size: %d') % (bookmarksize, len(bookmark)))
+bookmark = encoding.tolocal(bookmark)
+packedemptynodeflag = stream.read(boolstruct.size)
+emptynode = boolstruct.unpack(packedemptynodeflag)[0]
+node = ''
+if not emptynode:
+node = stream.read(20)
+bookmarks[bookmark] = node
+bookmarksize = _unpackbookmarksize(stream)
+return bookmarks
+
 def _getbkfile(repo):
 """Hook so that extensions that mess with the store can hook bm storage.
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 7 of 8 V11] bundle2: advertise bookmark capability

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1479807361 28800
#  Tue Nov 22 01:36:01 2016 -0800
# Node ID 729a7c3e2a60c5740c1629abfba7726933d018b7
# Parent  df763dd7f611acbc9fcb4ad9c3fe4706b60bde6e
bundle2: advertise bookmark capability

diff --git a/mercurial/bundle2.py b/mercurial/bundle2.py
--- a/mercurial/bundle2.py
+++ b/mercurial/bundle2.py
@@ -1265,6 +1265,7 @@
 'digests': tuple(sorted(util.DIGESTS.keys())),
 'remote-changegroup': ('http', 'https'),
 'hgtagsfnodes': (),
+'bookmarks': (),
}
 
 def getrepocaps(repo, allowpushback=False):
diff --git a/tests/test-acl.t b/tests/test-acl.t
--- a/tests/test-acl.t
+++ b/tests/test-acl.t
@@ -92,13 +92,13 @@
   f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd
   911600dab2ae7a9baff75958b84fe606851ce955
   bundle2-output-bundle: "HG20", 4 parts total
-  bundle2-output-part: "replycaps" 155 bytes payload
+  bundle2-output-part: "replycaps" 165 bytes payload
   bundle2-output-part: "check:heads" streamed payload
   bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload
   bundle2-output-part: "pushkey" (params: 4 mandatory) empty payload
   bundle2-input-bundle: with-transaction
   bundle2-input-part: "replycaps" supported
-  bundle2-input-part: total payload size 155
+  bundle2-input-part: total payload size 165
   bundle2-input-part: "check:heads" supported
   bundle2-input-part: total payload size 20
   bundle2-input-part: "changegroup" (params: 1 mandatory) supported
@@ -155,13 +155,13 @@
   f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd
   911600dab2ae7a9baff75958b84fe606851ce955
   bundle2-output-bundle: "HG20", 4 parts total
-  bundle2-output-part: "replycaps" 155 bytes payload
+  bundle2-output-part: "replycaps" 165 bytes payload
   bundle2-output-part: "check:heads" streamed payload
   bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload
   bundle2-output-part: "pushkey" (params: 4 mandatory) empty payload
   bundle2-input-bundle: with-transaction
   bundle2-input-part: "replycaps" supported
-  bundle2-input-part: total payload size 155
+  bundle2-input-part: total payload size 165
   bundle2-input-part: "check:heads" supported
   bundle2-input-part: total payload size 20
   bundle2-input-part: "changegroup" (params: 1 mandatory) supported
@@ -221,13 +221,13 @@
   f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd
   911600dab2ae7a9baff75958b84fe606851ce955
   bundle2-output-bundle: "HG20", 4 parts total
-  bundle2-output-part: "replycaps" 155 bytes payload
+  bundle2-output-part: "replycaps" 165 bytes payload
   bundle2-output-part: "check:heads" streamed payload
   bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload
   bundle2-output-part: "pushkey" (params: 4 mandatory) empty payload
   bundle2-input-bundle: with-transaction
   bundle2-input-part: "replycaps" supported
-  bundle2-input-part: total payload size 155
+  bundle2-input-part: total payload size 165
   bundle2-input-part: "check:heads" supported
   bundle2-input-part: total payload size 20
   bundle2-input-part: "changegroup" (params: 1 mandatory) supported
@@ -297,13 +297,13 @@
   f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd
   911600dab2ae7a9baff75958b84fe606851ce955
   bundle2-output-bundle: "HG20", 4 parts total
-  bundle2-output-part: "replycaps" 155 bytes payload
+  bundle2-output-part: "replycaps" 165 bytes payload
   bundle2-output-part: "check:heads" streamed payload
   bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload
   bundle2-output-part: "pushkey" (params: 4 mandatory) empty payload
   bundle2-input-bundle: with-transaction
   bundle2-input-part: "replycaps" supported
-  bundle2-input-part: total payload size 155
+  bundle2-input-part: total payload size 165
   bundle2-input-part: "check:heads" supported
   bundle2-input-part: total payload size 20
   bundle2-input-part: "changegroup" (params: 1 mandatory) supported
@@ -362,13 +362,13 @@
   f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd
   911600dab2ae7a9baff75958b84fe606851ce955
   bundle2-output-bundle: "HG20", 4 parts total
-  bundle2-output-part: "replycaps" 155 bytes payload
+  bundle2-output-part: "replycaps" 165 bytes payload
   bundle2-output-part: "check:heads" streamed payload
   bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload
   bundle2-output-part: "pushkey" (params: 4 mandatory) empty payload
   bundle2-input-bundle: with-transaction
   bundle2-input-part: "replycaps" supported
-  bundle2-input-par

[PATCH 6 of 8 V11] pull: use `bookmarks` bundle2 part

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1479807361 28800
#  Tue Nov 22 01:36:01 2016 -0800
# Node ID df763dd7f611acbc9fcb4ad9c3fe4706b60bde6e
# Parent  ce7bdfedd1b36290a981228f630daabc9fc09cfd
pull: use `bookmarks` bundle2 part

Apply changes from `bookmarks` part.

diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -1330,9 +1330,13 @@
 kwargs['cg'] = pullop.fetch
 if 'listkeys' in pullop.remotebundle2caps:
 kwargs['listkeys'] = ['phases']
-if pullop.remotebookmarks is None:
-# make sure to always includes bookmark data when migrating
-# `hg incoming --bundle` to using this function.
+
+if pullop.remotebookmarks is None:
+# make sure to always includes bookmark data when migrating
+# `hg incoming --bundle` to using this function.
+if 'bookmarks' in pullop.remotebundle2caps:
+kwargs['bookmarks'] = True
+elif 'listkeys' in pullop.remotebundle2caps:
 kwargs['listkeys'].append('bookmarks')
 
 # If this is a full pull / clone and the server supports the clone bundles
@@ -1360,10 +1364,23 @@
 _pullbundle2extraprepare(pullop, kwargs)
 bundle = pullop.remote.getbundle('pull', **kwargs)
 try:
-op = bundle2.processbundle(pullop.repo, bundle, pullop.gettransaction)
+bundleopbehavior = {
+'processbookmarksmode': 'diverge',
+'explicitbookmarks': pullop.explicitbookmarks,
+'remotepath': pullop.remote.url(),
+}
+bundleop = bundle2.bundleoperation(pullop.repo, pullop.gettransaction,
+   behavior=bundleopbehavior)
+op = bundle2.processbundle(pullop.repo, bundle,
+   pullop.gettransaction, op=bundleop)
 except error.BundleValueError as exc:
 raise error.Abort(_('missing support for %s') % exc)
 
+# `bookmarks` part was in bundle and it was applied to the repo. No need to
+# apply bookmarks one more time
+if 'bookmarks' in kwargs and kwargs['bookmarks']:
+pullop.stepsdone.add('bookmarks')
+
 if pullop.fetch:
 results = [cg['return'] for cg in op.records['changegroup']]
 pullop.cgresult = changegroup.combineresults(results)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 8 of 8 V11] help: add documentation about bookmark part

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1479807361 28800
#  Tue Nov 22 01:36:01 2016 -0800
# Node ID 4e82e7ed1c1eb0eddb2fe7409a812c27be133b94
# Parent  729a7c3e2a60c5740c1629abfba7726933d018b7
help: add documentation about bookmark part

diff --git a/mercurial/help/internals/bundles.txt 
b/mercurial/help/internals/bundles.txt
--- a/mercurial/help/internals/bundles.txt
+++ b/mercurial/help/internals/bundles.txt
@@ -92,3 +92,32 @@
 ``HGS1UN`` support was added as an experimental feature in version 3.6
 (released November 2015) as part of the initial offering of the *clone
 bundles* feature.
+
+Bundle2 parts
+=
+
+Bundle2 may contain many different pieces of information. These pieces are
+called parts.
+
+Bookmarks part
+--
+
+This part contains information about bookmarks. Part consists of many entries.
+Each entry describes one bookmark. Entry format:
+
+4 bytes
+  bookmark size
+1 byte
+  boolean. True if node is empty, False otherwise
+20 bytes (optional)
+  node. Present only if previous field is True
+
+Modes:
+
+1. 'ignore' - do not apply any changes to the repo, just decode the passed
+bookmarks. Will be used to list bookmarks in remote repo.
+2. 'diverge' - apply bookmark changes to the repo. Create divergent bookmarks 
if
+there is a non-fastforward move. Will be used during pull.
+3. 'apply' - apply bookmark changes to the repo. Overwrite current bookmark 
node
+if there is a non-fastforward move. Will be used during push.
+
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 8 V11] bookmarks: rename `compare()` to `comparebookmarks()` (API)

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1479807211 28800
#  Tue Nov 22 01:33:31 2016 -0800
# Node ID 20e2f13b7a1083f8d44a7a9e554eb3d5735b80f2
# Parent  8efced91de4d48478555d91c53e89cd1b4e3b78d
bookmarks: rename `compare()` to `comparebookmarks()` (API)

Next commit will remove optional parameters from `compare()` function.
Let's rename `compare()` to `comparebookmarks()` to avoid ambiguity from
callers from external extensions.

diff --git a/mercurial/bookmarks.py b/mercurial/bookmarks.py
--- a/mercurial/bookmarks.py
+++ b/mercurial/bookmarks.py
@@ -391,8 +391,8 @@
 finally:
 lockmod.release(tr, l, w)
 
-def compare(repo, srcmarks, dstmarks,
-srchex=None, dsthex=None, targets=None):
+def comparebookmarks(repo, srcmarks, dstmarks,
+ srchex=None, dsthex=None, targets=None):
 '''Compare bookmarks between srcmarks and dstmarks
 
 This returns tuple "(addsrc, adddst, advsrc, advdst, diverge,
@@ -511,7 +511,7 @@
 ui.debug("checking for updated bookmarks\n")
 localmarks = repo._bookmarks
 (addsrc, adddst, advsrc, advdst, diverge, differ, invalid, same
- ) = compare(repo, remotemarks, localmarks, dsthex=hex)
+ ) = comparebookmarks(repo, remotemarks, localmarks, dsthex=hex)
 
 status = ui.status
 warn = ui.warn
@@ -573,8 +573,8 @@
 '''
 ui.status(_("searching for changed bookmarks\n"))
 
-r = compare(repo, other.listkeys('bookmarks'), repo._bookmarks,
-dsthex=hex)
+r = comparebookmarks(repo, other.listkeys('bookmarks'), repo._bookmarks,
+ dsthex=hex)
 addsrc, adddst, advsrc, advdst, diverge, differ, invalid, same = r
 
 incomings = []
@@ -615,8 +615,8 @@
 '''
 ui.status(_("searching for changed bookmarks\n"))
 
-r = compare(repo, repo._bookmarks, other.listkeys('bookmarks'),
-srchex=hex)
+r = comparebookmarks(repo, repo._bookmarks, other.listkeys('bookmarks'),
+ srchex=hex)
 addsrc, adddst, advsrc, advdst, diverge, differ, invalid, same = r
 
 outgoings = []
@@ -660,8 +660,8 @@
 
 This returns "(# of incoming, # of outgoing)" tuple.
 '''
-r = compare(repo, other.listkeys('bookmarks'), repo._bookmarks,
-dsthex=hex)
+r = comparebookmarks(repo, other.listkeys('bookmarks'), repo._bookmarks,
+ dsthex=hex)
 addsrc, adddst, advsrc, advdst, diverge, differ, invalid, same = r
 return (len(addsrc), len(adddst))
 
diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -603,7 +603,8 @@
 explicit = set([repo._bookmarks.expandname(bookmark)
 for bookmark in pushop.bookmarks])
 
-comp = bookmod.compare(repo, repo._bookmarks, remotebookmark, srchex=hex)
+comp = bookmod.comparebookmarks(repo, repo._bookmarks,
+remotebookmark, srchex=hex)
 addsrc, adddst, advsrc, advdst, diverge, differ, invalid, same = comp
 for b, scid, dcid in advsrc:
 if b in explicit:
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 3 of 8 V11] bookmarks: make bookmarks.compare accept binary nodes (API)

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1479807361 28800
#  Tue Nov 22 01:36:01 2016 -0800
# Node ID 36d83269edc51168908c9b1160e22282ab6c5c04
# Parent  20e2f13b7a1083f8d44a7a9e554eb3d5735b80f2
bookmarks: make bookmarks.compare accept binary nodes (API)

New `bookmarks` bundle2 part will contain byte nodes, and since bookmarks are
stored in binary format, it doesn't make sense to convert them from binary to
hex and then back to binary again

diff --git a/mercurial/bookmarks.py b/mercurial/bookmarks.py
--- a/mercurial/bookmarks.py
+++ b/mercurial/bookmarks.py
@@ -391,8 +391,7 @@
 finally:
 lockmod.release(tr, l, w)
 
-def comparebookmarks(repo, srcmarks, dstmarks,
- srchex=None, dsthex=None, targets=None):
+def comparebookmarks(repo, srcmarks, dstmarks, targets=None):
 '''Compare bookmarks between srcmarks and dstmarks
 
 This returns tuple "(addsrc, adddst, advsrc, advdst, diverge,
@@ -415,19 +414,9 @@
 Changeset IDs of tuples in "addsrc", "adddst", "differ" or
  "invalid" list may be unknown for repo.
 
-This function expects that "srcmarks" and "dstmarks" return
-changeset ID in 40 hexadecimal digit string for specified
-bookmark. If not so (e.g. bmstore "repo._bookmarks" returning
-binary value), "srchex" or "dsthex" should be specified to convert
-into such form.
-
 If "targets" is specified, only bookmarks listed in it are
 examined.
 '''
-if not srchex:
-srchex = lambda x: x
-if not dsthex:
-dsthex = lambda x: x
 
 if targets:
 bset = set(targets)
@@ -449,14 +438,14 @@
 for b in sorted(bset):
 if b not in srcmarks:
 if b in dstmarks:
-adddst((b, None, dsthex(dstmarks[b])))
+adddst((b, None, dstmarks[b]))
 else:
 invalid((b, None, None))
 elif b not in dstmarks:
-addsrc((b, srchex(srcmarks[b]), None))
+addsrc((b, srcmarks[b], None))
 else:
-scid = srchex(srcmarks[b])
-dcid = dsthex(dstmarks[b])
+scid = srcmarks[b]
+dcid = dstmarks[b]
 if scid == dcid:
 same((b, scid, dcid))
 elif scid in repo and dcid in repo:
@@ -507,11 +496,17 @@
 
 return None
 
+def hexifybookmarks(marks):
+binremotemarks = {}
+for name, node in marks.items():
+binremotemarks[name] = bin(node)
+return binremotemarks
+
 def updatefromremote(ui, repo, remotemarks, path, trfunc, explicit=()):
 ui.debug("checking for updated bookmarks\n")
 localmarks = repo._bookmarks
 (addsrc, adddst, advsrc, advdst, diverge, differ, invalid, same
- ) = comparebookmarks(repo, remotemarks, localmarks, dsthex=hex)
+) = comparebookmarks(repo, remotemarks, localmarks)
 
 status = ui.status
 warn = ui.warn
@@ -522,15 +517,15 @@
 changed = []
 for b, scid, dcid in addsrc:
 if scid in repo: # add remote bookmarks for changes we already have
-changed.append((b, bin(scid), status,
+changed.append((b, scid, status,
 _("adding remote bookmark %s\n") % (b)))
 elif b in explicit:
 explicit.remove(b)
 ui.warn(_("remote bookmark %s points to locally missing %s\n")
-% (b, scid[:12]))
+% (b, hex(scid)[:12]))
 
 for b, scid, dcid in advsrc:
-changed.append((b, bin(scid), status,
+changed.append((b, scid, status,
 _("updating bookmark %s\n") % (b)))
 # remove normal movement from explicit set
 explicit.difference_update(d[0] for d in changed)
@@ -538,13 +533,12 @@
 for b, scid, dcid in diverge:
 if b in explicit:
 explicit.discard(b)
-changed.append((b, bin(scid), status,
+changed.append((b, scid, status,
 _("importing bookmark %s\n") % (b)))
 else:
-snode = bin(scid)
-db = _diverge(ui, b, path, localmarks, snode)
+db = _diverge(ui, b, path, localmarks, scid)
 if db:
-changed.append((db, snode, warn,
+changed.append((db, scid, warn,
 _("divergent bookmark %s stored as %s\n") %
 (b, db)))
 else:
@@ -553,13 +547,13 @@
 for b, scid, dcid in adddst + advdst:
 if b in explicit:
 explicit.discard(b)
-changed.append((b, bin(scid), status,
+changed.append((b, scid, status,
 _("importing bookmark %s\n") % (b)))
 for b, scid, dcid in differ:
 if b in explicit:
 explicit.remov

[PATCH 4 of 8 V11] bundle2: add `bookmarks` part handler

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1479807361 28800
#  Tue Nov 22 01:36:01 2016 -0800
# Node ID 1572af8f4545b11c8147170cb421c23547d918fc
# Parent  36d83269edc51168908c9b1160e22282ab6c5c04
bundle2: add `bookmarks` part handler

Applies bookmarks part to the local repo. `processbookmarksmode` determines
how remote bookmarks are handled. They are either ignored ('ignore' mode),
diverged ('diverge' mode) or applied ('apply' mode).

diff --git a/mercurial/bundle2.py b/mercurial/bundle2.py
--- a/mercurial/bundle2.py
+++ b/mercurial/bundle2.py
@@ -155,6 +155,7 @@
 
 from .i18n import _
 from . import (
+bookmarks as bookmod,
 changegroup,
 error,
 obsolete,
@@ -287,13 +288,19 @@
 * a way to construct a bundle response when applicable.
 """
 
-def __init__(self, repo, transactiongetter, captureoutput=True):
+def __init__(self, repo, transactiongetter, captureoutput=True,
+ behavior=None):
+"""
+`behavior` is a dictionary that is passed to part handlers to tweak
+their behaviour
+"""
 self.repo = repo
 self.ui = repo.ui
 self.records = unbundlerecords()
 self.gettransaction = transactiongetter
 self.reply = None
 self.captureoutput = captureoutput
+self.behavior = behavior or {}
 
 class TransactionUnavailable(RuntimeError):
 pass
@@ -1616,3 +1623,35 @@
 
 cache.write()
 op.ui.debug('applied %i hgtags fnodes cache entries\n' % count)
+
+@parthandler('bookmarks')
+def handlebookmarks(op, inpart):
+"""Processes bookmarks part.
+
+`processbookmarksmode` determines how remote bookmarks are handled. They 
are
+either ignored ('ignore' mode), diverged ('diverge' mode) or applied
+('apply' mode). 'ignore' mode is used to get bookmarks and process them
+later, 'diverge' mode is used to process bookmarks during pull, 'apply'
+mode is used during push.
+"""
+
+bookmarks = bookmod.decodebookmarks(inpart.read())
+processbookmarksmode = op.behavior.get('processbookmarksmode', 'ignore')
+if processbookmarksmode == 'apply':
+for bookmark, node in bookmarks.items():
+if node:
+op.repo._bookmarks[bookmark] = node
+else:
+try:
+del op.repo._bookmarks[bookmark]
+except KeyError:
+# ignore if bookmark does not exist
+pass
+op.repo._bookmarks.recordchange(op.gettransaction())
+elif processbookmarksmode == 'diverge':
+remotepath = op.behavior.get('remotepath', '')
+explicitbookmarks = op.behavior.get('explicitbookmarks', ())
+bookmod.updatefromremote(op.ui, op.repo, bookmarks,
+ remotepath, op.gettransaction,
+ explicit=explicitbookmarks)
+op.records.add('bookmarks', bookmarks)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 5 of 8 V11] exchange: getbundle `bookmarks` part generator

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1479807361 28800
#  Tue Nov 22 01:36:01 2016 -0800
# Node ID ce7bdfedd1b36290a981228f630daabc9fc09cfd
# Parent  1572af8f4545b11c8147170cb421c23547d918fc
exchange: getbundle `bookmarks` part generator

This generator will be used during pull operation.

diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -1675,6 +1675,21 @@
 if chunks:
 bundler.newpart('hgtagsfnodes', data=''.join(chunks))
 
+@getbundle2partsgenerator('bookmarks')
+def _getbundlebookmarkspart(bundler, repo, source, bundlecaps=None,
+b2caps=None, heads=None, common=None,
+**kwargs):
+if not kwargs.get('bookmarks'):
+return
+if 'bookmarks' not in b2caps:
+raise ValueError(
+_('bookmarks are requested but client is not capable '
+  'of receiving it'))
+
+bookmarks = _getbookmarks(repo, **kwargs)
+encodedbookmarks = bookmod.encodebookmarks(bookmarks)
+bundler.newpart('bookmarks', data=encodedbookmarks)
+
 def _getbookmarks(repo, **kwargs):
 """Returns bookmark to node mapping.
 
diff --git a/mercurial/wireproto.py b/mercurial/wireproto.py
--- a/mercurial/wireproto.py
+++ b/mercurial/wireproto.py
@@ -228,7 +228,9 @@
  'bundlecaps': 'scsv',
  'listkeys': 'csv',
  'cg': 'boolean',
- 'cbattempted': 'boolean'}
+ 'cbattempted': 'boolean',
+ 'bookmarks': 'boolean',
+}
 
 # client side
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 1 of 8 V11] bookmarks: introduce binary encoding

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1479807211 28800
#  Tue Nov 22 01:33:31 2016 -0800
# Node ID 8efced91de4d48478555d91c53e89cd1b4e3b78d
# Parent  259296eb59e77951572872fb2f542c6dbff8bc74
bookmarks: introduce binary encoding

Bookmarks binary encoding will be used for `bookmarks` bundle2 part.
The format is: <4 bytes - bookmark size, big-endian>
   <1 byte - 0 if node is empty 1 otherwise><20 bytes node>
BookmarksEncodeError and BookmarksDecodeError maybe thrown if input is
incorrect.

diff --git a/mercurial/bookmarks.py b/mercurial/bookmarks.py
--- a/mercurial/bookmarks.py
+++ b/mercurial/bookmarks.py
@@ -8,7 +8,9 @@
 from __future__ import absolute_import
 
 import errno
+import io
 import os
+import struct
 
 from .i18n import _
 from .node import (
@@ -23,6 +25,72 @@
 util,
 )
 
+_NONEMPTYNODE = struct.pack('?', False)
+_EMPTYNODE = struct.pack('?', True)
+
+def _unpackbookmarksize(stream):
+"""Returns 0 if stream is empty.
+"""
+
+intstruct = struct.Struct('>i')
+expectedsize = intstruct.size
+encodedbookmarksize = stream.read(expectedsize)
+if not encodedbookmarksize:
+return 0
+if len(encodedbookmarksize) != expectedsize:
+raise ValueError(
+_('cannot decode bookmark size: '
+  'expected size: %d, actual size: %d') %
+(expectedsize, len(encodedbookmarksize)))
+return intstruct.unpack(encodedbookmarksize)[0]
+
+def encodebookmarks(bookmarks):
+"""Encodes bookmark to node mapping to the byte string.
+
+Format: <4 bytes - bookmark size>
+<1 byte - 0 if node is empty 1 otherwise><20 bytes node>
+Node may be 20 byte binary string or empty
+"""
+
+intstruct = struct.Struct('>i')
+for bookmark, node in sorted(bookmarks.iteritems()):
+encodedbookmark = encoding.fromlocal(bookmark)
+yield intstruct.pack(len(encodedbookmark))
+yield encodedbookmark
+if node:
+if len(node) != 20:
+raise ValueError(_('node must be 20 or bytes long'))
+yield _NONEMPTYNODE
+yield node
+else:
+yield _EMPTYNODE
+
+def decodebookmarks(buf):
+"""Decodes buffer into bookmark to node mapping.
+
+Node is either 20 bytes or empty.
+"""
+
+stream = io.BytesIO(buf)
+bookmarks = {}
+bookmarksize = _unpackbookmarksize(stream)
+boolstruct = struct.Struct('?')
+while bookmarksize:
+bookmark = stream.read(bookmarksize)
+if len(bookmark) != bookmarksize:
+raise ValueError(
+_('cannot decode bookmark: expected size: %d, '
+'actual size: %d') % (bookmarksize, len(bookmark)))
+bookmark = encoding.tolocal(bookmark)
+packedemptynodeflag = stream.read(boolstruct.size)
+emptynode = boolstruct.unpack(packedemptynodeflag)[0]
+node = ''
+if not emptynode:
+node = stream.read(20)
+bookmarks[bookmark] = node
+bookmarksize = _unpackbookmarksize(stream)
+return bookmarks
+
 def _getbkfile(repo):
 """Hook so that extensions that mess with the store can hook bm storage.
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 08 of 10 V10] pull: use `bookmarks` bundle2 part

[Stanislau Hlebik]

From: Gregory Szorc <gregory.sz...@gmail.com>
Date: Tuesday, November 22, 2016 at 3:48 AM
To: Stanislau Hlebik <st...@fb.com>
Cc: mercurial-devel <mercurial-devel@mercurial-scm.org>, Pierre-Yves David 
<pierre-yves.da...@ens-lyon.org>
Subject: Re: [PATCH 08 of 10 V10] pull: use `bookmarks` bundle2 part

On Sun, Nov 20, 2016 at 4:14 AM, Stanislau Hlebik 
<st...@fb.com<mailto:st...@fb.com>> wrote:
# HG changeset patch
# User Stanislau Hlebik <st...@fb.com<mailto:st...@fb.com>>
# Date 1479373181 28800
#  Thu Nov 17 00:59:41 2016 -0800
# Node ID 2ac3e9d5983f18f94a1df84317d1d2f1bd9b88b8
# Parent  5af41d2c5226c36d5a1f999ff3d99d8694ae68b9
pull: use `bookmarks` bundle2 part

Apply changes from `bookmarks` part.

diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -1335,9 +1335,13 @@
 kwargs['cg'] = pullop.fetch
 if 'listkeys' in pullop.remotebundle2caps:
 kwargs['listkeys'] = ['phases']
-if pullop.remotebookmarks is None:
-# make sure to always includes bookmark data when migrating
-# `hg incoming --bundle` to using this function.
+
+if pullop.remotebookmarks is None:
+# make sure to always includes bookmark data when migrating
+# `hg incoming --bundle` to using this function.
+if 'bookmarks' in pullop.remotebundle2caps:
+kwargs['bookmarks'] = True
+elif 'listkeys' in pullop.remotebundle2caps:

This is already inside an `if 'listkeys' in pullop.remotebundle2caps:` block, 
so this can simply be "else:".

No, it’s not inside `if 'listkeys' in pullop.remotebundle2caps:`, it’s in 
different block so I have to use elif

 kwargs['listkeys'].append('bookmarks')

 # If this is a full pull / clone and the server supports the clone bundles
@@ -1365,10 +1369,23 @@
 _pullbundle2extraprepare(pullop, kwargs)
 bundle = pullop.remote.getbundle('pull', **kwargs)
 try:
-op = bundle2.processbundle(pullop.repo, bundle, pullop.gettransaction)
+bundleopbehavior = {
+'processbookmarksmode': 'diverge',
+'explicitbookmarks': pullop.explicitbookmarks,
+'remotepath': pullop.remote.url(),
+}
+op = bundle2.bundleoperation(pullop.repo, pullop.gettransaction,
+ behavior=bundleopbehavior)
+op = bundle2.processbundle(pullop.repo, bundle,
+   pullop.gettransaction, op=op)

The reuse of "op" here reads a bit weird. Can you please rename one of the 
variables?

Ok, I’ll rename first op to bundleop.

 except error.BundleValueError as exc:
 raise error.Abort(_('missing support for %s') % exc)

+# `bookmarks` part was in bundle and it was applied to the repo. No need to
+# apply bookmarks one more time
+if 'bookmarks' in kwargs and kwargs['bookmarks']:
+pullop.stepsdone.add('bookmarks')
+

This feels a bit weird to me because we're assuming that sending the request 
for bookmarks means that we received a bookmarks part. If you look at similar 
code, you'll find that we update pullops.stepsdone in the part handler for a 
bundle2 part. And I guess the reason we don't do things that way for bookmarks 
is because we're processing bookmarks immediately as a @parthandler (from the 
previous patch) and that doesn't have an "op" argument to update. This raises 
another concern, which is that the previous patch reorders /may/ reorder the 
application of bookmarks. Before, bookmarks were in a listkeys and we processed 
them *after* phases. Now, it appears that we may apply bookmarks *before* 
phases, as the @parthandler for listkeys defers their application. I'm not sure 
if this matters. But it is definitely something Pierre-Yves should take a look 
at.

I think changing of order should not affect the end result, but let’s wait for 
Pierre-Yves.
Meanwhile I’ll send new series without first 3 patches (they were queued) and 
with your comments fixed


 if pullop.fetch:
 results = [cg['return'] for cg in op.records['changegroup']]
 pullop.cgresult = changegroup.combineresults(results)

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 03 of 10 V10] exchange: add `_getbookmarks()` function

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1479373181 28800
#  Thu Nov 17 00:59:41 2016 -0800
# Node ID 4e782ccf84f8fee96963f09439f7da7202c37775
# Parent  54b24eb2eac7c181f9fd15423fd05218a6a11186
exchange: add `_getbookmarks()` function

This function will be used to generate bookmarks bundle2 part.
It is a separate function in order to make it easy to overwrite it
in extensions. Passing `kwargs` to the function makes it easy to
add new parameters in extensions.

diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -1666,6 +1666,17 @@
 if chunks:
 bundler.newpart('hgtagsfnodes', data=''.join(chunks))
 
+def _getbookmarks(repo, **kwargs):
+"""Returns bookmark to node mapping.
+
+This function is primarily used to generate `bookmarks` bundle2 part.
+It is a separate function in order to make it easy to wrap it
+in extensions. Passing `kwargs` to the function makes it easy to
+add new parameters in extensions.
+"""
+
+return dict(bookmod.listbinbookmarks(repo))
+
 def check_heads(repo, their_heads, context):
 """check if the heads of a repo have been modified
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 04 of 10 V10] bookmarks: introduce binary encoding

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1479373181 28800
#  Thu Nov 17 00:59:41 2016 -0800
# Node ID bd590f83eb640f4464ba5465f4e10677e348e83c
# Parent  4e782ccf84f8fee96963f09439f7da7202c37775
bookmarks: introduce binary encoding

Bookmarks binary encoding will be used for `bookmarks` bundle2 part.
The format is: <4 bytes - bookmark size, big-endian>
   <1 byte - 0 if node is empty 1 otherwise><20 bytes node>
BookmarksEncodeError and BookmarksDecodeError maybe thrown if input is
incorrect.

diff --git a/mercurial/bookmarks.py b/mercurial/bookmarks.py
--- a/mercurial/bookmarks.py
+++ b/mercurial/bookmarks.py
@@ -8,7 +8,9 @@
 from __future__ import absolute_import
 
 import errno
+import io
 import os
+import struct
 
 from .i18n import _
 from .node import (
@@ -23,6 +25,71 @@
 util,
 )
 
+_NONEMPTYNODE = struct.pack('?', False)
+_EMPTYNODE = struct.pack('?', True)
+
+def _unpackbookmarksize(stream):
+"""Returns 0 if stream is empty.
+"""
+
+intstruct = struct.Struct('>i')
+expectedsize = intstruct.size
+encodedbookmarksize = stream.read(expectedsize)
+if not encodedbookmarksize:
+return 0
+if len(encodedbookmarksize) != expectedsize:
+raise ValueError(
+_('cannot decode bookmark size: '
+  'expected size: %d, actual size: %d') %
+(expectedsize, len(encodedbookmarksize)))
+return intstruct.unpack(encodedbookmarksize)[0]
+
+def encodebookmarks(bookmarks):
+"""Encodes bookmark to node mapping to the byte string.
+
+Format: <4 bytes - bookmark size>
+<1 byte - 0 if node is empty 1 otherwise><20 bytes node>
+Node may be 20 byte binary string or empty
+"""
+
+intstruct = struct.Struct('>i')
+for bookmark, node in sorted(bookmarks.iteritems()):
+yield intstruct.pack(len(bookmark))
+yield encoding.fromlocal(bookmark)
+if node:
+if len(node) != 20:
+raise ValueError(_('node must be 20 or bytes long'))
+yield _NONEMPTYNODE
+yield node
+else:
+yield _EMPTYNODE
+
+def decodebookmarks(buf):
+"""Decodes buffer into bookmark to node mapping.
+
+Node is either 20 bytes or empty.
+"""
+
+stream = io.BytesIO(buf)
+bookmarks = {}
+bookmarksize = _unpackbookmarksize(stream)
+boolstruct = struct.Struct('?')
+while bookmarksize:
+bookmark = stream.read(bookmarksize)
+if len(bookmark) != bookmarksize:
+raise ValueError(
+_('cannot decode bookmark: expected size: %d, '
+'actual size: %d') % (bookmarksize, len(bookmark)))
+bookmark = encoding.tolocal(bookmark)
+packedemptynodeflag = stream.read(boolstruct.size)
+emptynode = boolstruct.unpack(packedemptynodeflag)[0]
+node = ''
+if not emptynode:
+node = stream.read(20)
+bookmarks[bookmark] = node
+bookmarksize = _unpackbookmarksize(stream)
+return bookmarks
+
 def _getbkfile(repo):
 """Hook so that extensions that mess with the store can hook bm storage.
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 08 of 10 V10] pull: use `bookmarks` bundle2 part

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1479373181 28800
#  Thu Nov 17 00:59:41 2016 -0800
# Node ID 2ac3e9d5983f18f94a1df84317d1d2f1bd9b88b8
# Parent  5af41d2c5226c36d5a1f999ff3d99d8694ae68b9
pull: use `bookmarks` bundle2 part

Apply changes from `bookmarks` part.

diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -1335,9 +1335,13 @@
 kwargs['cg'] = pullop.fetch
 if 'listkeys' in pullop.remotebundle2caps:
 kwargs['listkeys'] = ['phases']
-if pullop.remotebookmarks is None:
-# make sure to always includes bookmark data when migrating
-# `hg incoming --bundle` to using this function.
+
+if pullop.remotebookmarks is None:
+# make sure to always includes bookmark data when migrating
+# `hg incoming --bundle` to using this function.
+if 'bookmarks' in pullop.remotebundle2caps:
+kwargs['bookmarks'] = True
+elif 'listkeys' in pullop.remotebundle2caps:
 kwargs['listkeys'].append('bookmarks')
 
 # If this is a full pull / clone and the server supports the clone bundles
@@ -1365,10 +1369,23 @@
 _pullbundle2extraprepare(pullop, kwargs)
 bundle = pullop.remote.getbundle('pull', **kwargs)
 try:
-op = bundle2.processbundle(pullop.repo, bundle, pullop.gettransaction)
+bundleopbehavior = {
+'processbookmarksmode': 'diverge',
+'explicitbookmarks': pullop.explicitbookmarks,
+'remotepath': pullop.remote.url(),
+}
+op = bundle2.bundleoperation(pullop.repo, pullop.gettransaction,
+ behavior=bundleopbehavior)
+op = bundle2.processbundle(pullop.repo, bundle,
+   pullop.gettransaction, op=op)
 except error.BundleValueError as exc:
 raise error.Abort(_('missing support for %s') % exc)
 
+# `bookmarks` part was in bundle and it was applied to the repo. No need to
+# apply bookmarks one more time
+if 'bookmarks' in kwargs and kwargs['bookmarks']:
+pullop.stepsdone.add('bookmarks')
+
 if pullop.fetch:
 results = [cg['return'] for cg in op.records['changegroup']]
 pullop.cgresult = changegroup.combineresults(results)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 07 of 10 V10] exchange: getbundle `bookmarks` part generator

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1479373181 28800
#  Thu Nov 17 00:59:41 2016 -0800
# Node ID 5af41d2c5226c36d5a1f999ff3d99d8694ae68b9
# Parent  866281dae2407308c19c7c3109bb5501b940ee67
exchange: getbundle `bookmarks` part generator

This generator will be used during pull operation.

diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -1680,6 +1680,21 @@
 if chunks:
 bundler.newpart('hgtagsfnodes', data=''.join(chunks))
 
+@getbundle2partsgenerator('bookmarks')
+def _getbundlebookmarkspart(bundler, repo, source, bundlecaps=None,
+b2caps=None, heads=None, common=None,
+**kwargs):
+if not kwargs.get('bookmarks'):
+return
+if 'bookmarks' not in b2caps:
+raise ValueError(
+_('bookmarks are requested but client is not capable '
+  'of receiving it'))
+
+bookmarks = _getbookmarks(repo, **kwargs)
+encodedbookmarks = bookmod.encodebookmarks(bookmarks)
+bundler.newpart('bookmarks', data=encodedbookmarks)
+
 def _getbookmarks(repo, **kwargs):
 """Returns bookmark to node mapping.
 
diff --git a/mercurial/wireproto.py b/mercurial/wireproto.py
--- a/mercurial/wireproto.py
+++ b/mercurial/wireproto.py
@@ -228,7 +228,9 @@
  'bundlecaps': 'scsv',
  'listkeys': 'csv',
  'cg': 'boolean',
- 'cbattempted': 'boolean'}
+ 'cbattempted': 'boolean',
+ 'bookmarks': 'boolean',
+}
 
 # client side
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 10 of 10 V10] help: add documentation about bookmark part

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1479373181 28800
#  Thu Nov 17 00:59:41 2016 -0800
# Node ID b46026585f5961e520c21ea336baa7a6f68a3d85
# Parent  adbd3b69427059264d78fa8f29fa6eabcc38bf5f
help: add documentation about bookmark part

diff --git a/mercurial/help/internals/bundles.txt 
b/mercurial/help/internals/bundles.txt
--- a/mercurial/help/internals/bundles.txt
+++ b/mercurial/help/internals/bundles.txt
@@ -92,3 +92,32 @@
 ``HGS1UN`` support was added as an experimental feature in version 3.6
 (released November 2015) as part of the initial offering of the *clone
 bundles* feature.
+
+Bundle2 parts
+=
+
+Bundle2 may contain many different pieces of information. These pieces are
+called parts.
+
+Bookmarks part
+--
+
+This part contains information about bookmarks. Part consists of many entries.
+Each entry describes one bookmark. Entry format:
+
+4 bytes
+  bookmark size
+1 byte
+  boolean. True if node is empty, False otherwise
+20 bytes (optional)
+  node. Present only if previous field is True
+
+Modes:
+
+1. 'ignore' - do not apply any changes to the repo, just decode the passed
+bookmarks. Will be used to list bookmarks in remote repo.
+2. 'diverge' - apply bookmark changes to the repo. Create divergent bookmarks 
if
+there is a non-fastforward move. Will be used during pull.
+3. 'apply' - apply bookmark changes to the repo. Overwrite current bookmark 
node
+if there is a non-fastforward move. Will be used during push.
+
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 06 of 10 V10] bundle2: add `bookmarks` part handler

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1479643450 28800
#  Sun Nov 20 04:04:10 2016 -0800
# Node ID 866281dae2407308c19c7c3109bb5501b940ee67
# Parent  57d7f92db34461da87850e26d831d2d235282356
bundle2: add `bookmarks` part handler

Applies bookmarks part to the local repo. `processbookmarksmode` determines
how remote bookmarks are handled. They are either ignored ('ignore' mode),
diverged ('diverge' mode) or applied ('apply' mode).

diff --git a/mercurial/bundle2.py b/mercurial/bundle2.py
--- a/mercurial/bundle2.py
+++ b/mercurial/bundle2.py
@@ -155,6 +155,7 @@
 
 from .i18n import _
 from . import (
+bookmarks as bookmod,
 changegroup,
 error,
 obsolete,
@@ -287,13 +288,19 @@
 * a way to construct a bundle response when applicable.
 """
 
-def __init__(self, repo, transactiongetter, captureoutput=True):
+def __init__(self, repo, transactiongetter, captureoutput=True,
+ behavior=None):
+"""
+`behavior` is a dictionary that is passed to part handlers to tweak
+their behaviour
+"""
 self.repo = repo
 self.ui = repo.ui
 self.records = unbundlerecords()
 self.gettransaction = transactiongetter
 self.reply = None
 self.captureoutput = captureoutput
+self.behavior = behavior or {}
 
 class TransactionUnavailable(RuntimeError):
 pass
@@ -1616,3 +1623,36 @@
 
 cache.write()
 op.ui.debug('applied %i hgtags fnodes cache entries\n' % count)
+
+@parthandler('bookmarks')
+def handlebookmarks(op, inpart):
+"""Processes bookmarks part.
+
+`processbookmarksmode` determines how remote bookmarks are handled. They 
are
+either ignored ('ignore' mode), diverged ('diverge' mode) or applied
+('apply' mode). 'ignore' mode is used to get bookmarks and process them
+later, 'diverge' mode is used to process bookmarks during pull, 'apply'
+mode is used during push.
+"""
+
+bookmarks = {}
+bookmarks = bookmod.decodebookmarks(inpart.read())
+processbookmarksmode = op.behavior.get('processbookmarksmode', 'ignore')
+if processbookmarksmode == 'apply':
+for bookmark, node in bookmarks.items():
+if node:
+op.repo._bookmarks[bookmark] = node
+else:
+try:
+del op.repo._bookmarks[bookmark]
+except KeyError:
+# ignore if bookmark does not exist
+pass
+op.repo._bookmarks.recordchange(op.gettransaction())
+elif processbookmarksmode == 'diverge':
+remotepath = op.behavior.get('remotepath', '')
+explicitbookmarks = op.behavior.get('explicitbookmarks', ())
+bookmod.updatefromremote(op.ui, op.repo, bookmarks,
+ remotepath, op.gettransaction,
+ explicit=explicitbookmarks)
+op.records.add('bookmarks', bookmarks)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 05 of 10 V10] bookmarks: make bookmarks.compare accept binary nodes

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1479373181 28800
#  Thu Nov 17 00:59:41 2016 -0800
# Node ID 57d7f92db34461da87850e26d831d2d235282356
# Parent  bd590f83eb640f4464ba5465f4e10677e348e83c
bookmarks: make bookmarks.compare accept binary nodes

New `bookmarks` bundle2 part will contain byte nodes, and since bookmarks are
stored in binary format, it doesn't make sense to convert them from binary to
hex and then back to binary again

diff --git a/mercurial/bookmarks.py b/mercurial/bookmarks.py
--- a/mercurial/bookmarks.py
+++ b/mercurial/bookmarks.py
@@ -390,8 +390,7 @@
 finally:
 lockmod.release(tr, l, w)
 
-def compare(repo, srcmarks, dstmarks,
-srchex=None, dsthex=None, targets=None):
+def compare(repo, srcmarks, dstmarks, targets=None):
 '''Compare bookmarks between srcmarks and dstmarks
 
 This returns tuple "(addsrc, adddst, advsrc, advdst, diverge,
@@ -414,19 +413,9 @@
 Changeset IDs of tuples in "addsrc", "adddst", "differ" or
  "invalid" list may be unknown for repo.
 
-This function expects that "srcmarks" and "dstmarks" return
-changeset ID in 40 hexadecimal digit string for specified
-bookmark. If not so (e.g. bmstore "repo._bookmarks" returning
-binary value), "srchex" or "dsthex" should be specified to convert
-into such form.
-
 If "targets" is specified, only bookmarks listed in it are
 examined.
 '''
-if not srchex:
-srchex = lambda x: x
-if not dsthex:
-dsthex = lambda x: x
 
 if targets:
 bset = set(targets)
@@ -448,14 +437,14 @@
 for b in sorted(bset):
 if b not in srcmarks:
 if b in dstmarks:
-adddst((b, None, dsthex(dstmarks[b])))
+adddst((b, None, dstmarks[b]))
 else:
 invalid((b, None, None))
 elif b not in dstmarks:
-addsrc((b, srchex(srcmarks[b]), None))
+addsrc((b, srcmarks[b], None))
 else:
-scid = srchex(srcmarks[b])
-dcid = dsthex(dstmarks[b])
+scid = srcmarks[b]
+dcid = dstmarks[b]
 if scid == dcid:
 same((b, scid, dcid))
 elif scid in repo and dcid in repo:
@@ -506,11 +495,17 @@
 
 return None
 
+def hexifybookmarks(marks):
+binremotemarks = {}
+for name, node in marks.items():
+binremotemarks[name] = bin(node)
+return binremotemarks
+
 def updatefromremote(ui, repo, remotemarks, path, trfunc, explicit=()):
 ui.debug("checking for updated bookmarks\n")
 localmarks = repo._bookmarks
 (addsrc, adddst, advsrc, advdst, diverge, differ, invalid, same
- ) = compare(repo, remotemarks, localmarks, dsthex=hex)
+) = compare(repo, remotemarks, localmarks)
 
 status = ui.status
 warn = ui.warn
@@ -521,15 +516,15 @@
 changed = []
 for b, scid, dcid in addsrc:
 if scid in repo: # add remote bookmarks for changes we already have
-changed.append((b, bin(scid), status,
+changed.append((b, scid, status,
 _("adding remote bookmark %s\n") % (b)))
 elif b in explicit:
 explicit.remove(b)
 ui.warn(_("remote bookmark %s points to locally missing %s\n")
-% (b, scid[:12]))
+% (b, hex(scid)[:12]))
 
 for b, scid, dcid in advsrc:
-changed.append((b, bin(scid), status,
+changed.append((b, scid, status,
 _("updating bookmark %s\n") % (b)))
 # remove normal movement from explicit set
 explicit.difference_update(d[0] for d in changed)
@@ -537,13 +532,12 @@
 for b, scid, dcid in diverge:
 if b in explicit:
 explicit.discard(b)
-changed.append((b, bin(scid), status,
+changed.append((b, scid, status,
 _("importing bookmark %s\n") % (b)))
 else:
-snode = bin(scid)
-db = _diverge(ui, b, path, localmarks, snode)
+db = _diverge(ui, b, path, localmarks, scid)
 if db:
-changed.append((db, snode, warn,
+changed.append((db, scid, warn,
 _("divergent bookmark %s stored as %s\n") %
 (b, db)))
 else:
@@ -552,13 +546,13 @@
 for b, scid, dcid in adddst + advdst:
 if b in explicit:
 explicit.discard(b)
-changed.append((b, bin(scid), status,
+changed.append((b, scid, status,
 _("importing bookmark %s\n") % (b)))
 for b, scid, dcid in differ:
 if b in explicit:
 explicit.remove(b)
 ui.warn(_("remote 

[PATCH 02 of 10 V10] bookmarks: use listbinbookmarks() in listbookmarks()

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1479373181 28800
#  Thu Nov 17 00:59:41 2016 -0800
# Node ID 54b24eb2eac7c181f9fd15423fd05218a6a11186
# Parent  a3159f73e59868f8ae0993cc91277142aaa10536
bookmarks: use listbinbookmarks() in listbookmarks()

diff --git a/mercurial/bookmarks.py b/mercurial/bookmarks.py
--- a/mercurial/bookmarks.py
+++ b/mercurial/bookmarks.py
@@ -296,16 +296,9 @@
 yield k, v
 
 def listbookmarks(repo):
-# We may try to list bookmarks on a repo type that does not
-# support it (e.g., statichttprepository).
-marks = getattr(repo, '_bookmarks', {})
-
 d = {}
-hasnode = repo.changelog.hasnode
-for k, v in marks.iteritems():
-# don't expose local divergent bookmarks
-if hasnode(v) and ('@' not in k or k.endswith('@')):
-d[k] = hex(v)
+for book, node in listbinbookmarks(repo):
+d[book] = hex(node)
 return d
 
 def pushbookmark(repo, key, old, new):
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 02 of 10 V9] bookmarks: use listbinbookmarks() in listbookmarks()

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1479032114 28800
#  Sun Nov 13 02:15:14 2016 -0800
# Branch stable
# Node ID 031419529f6594c3f23fc179d67c8de9ef63472f
# Parent  ffa31155703b725b20d67f379bea4de7c56379fe
bookmarks: use listbinbookmarks() in listbookmarks()

diff --git a/mercurial/bookmarks.py b/mercurial/bookmarks.py
--- a/mercurial/bookmarks.py
+++ b/mercurial/bookmarks.py
@@ -296,16 +296,9 @@
 yield k, v
 
 def listbookmarks(repo):
-# We may try to list bookmarks on a repo type that does not
-# support it (e.g., statichttprepository).
-marks = getattr(repo, '_bookmarks', {})
-
 d = {}
-hasnode = repo.changelog.hasnode
-for k, v in marks.iteritems():
-# don't expose local divergent bookmarks
-if hasnode(v) and ('@' not in k or k.endswith('@')):
-d[k] = hex(v)
+for book, node in listbinbookmarks(repo):
+d[book] = hex(node)
 return d
 
 def pushbookmark(repo, key, old, new):
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 03 of 10 V9] exchange: add `_getbookmarks()` function

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1479032456 28800
#  Sun Nov 13 02:20:56 2016 -0800
# Branch stable
# Node ID 0917ce41b232a5fc2a6e503cbad15879e037a2f9
# Parent  031419529f6594c3f23fc179d67c8de9ef63472f
exchange: add `_getbookmarks()` function

This function will be used to generate bookmarks bundle2 part.
It is a separate function in order to make it easy to overwrite it
in extensions. Passing `kwargs` to the function makes it easy to
add new parameters in extensions.

diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -1666,6 +1666,17 @@
 if chunks:
 bundler.newpart('hgtagsfnodes', data=''.join(chunks))
 
+def _getbookmarks(repo, **kwargs):
+"""Returns bookmark to node mapping.
+
+This function is primarily used to generate `bookmarks` bundle2 part.
+It is a separate function in order to make it easy to wrap it
+in extensions. Passing `kwargs` to the function makes it easy to
+add new parameters in extensions.
+"""
+
+return dict(bookmod.listbinbookmarks(repo))
+
 def check_heads(repo, their_heads, context):
 """check if the heads of a repo have been modified
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 05 of 10 V9] bookmarks: make bookmarks.compare accept binary nodes

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1479032456 28800
#  Sun Nov 13 02:20:56 2016 -0800
# Branch stable
# Node ID 5cdf21805bb97ba99199c8779e3ad91137061c07
# Parent  8b6ab3b8df3aae90ac72d7fa4603e7d5b4ab55e9
bookmarks: make bookmarks.compare accept binary nodes

New `bookmarks` bundle2 part will contain byte nodes, and since bookmarks are
stored in binary format, it doesn't make sense to convert them from binary to
hex and then back to binary again

diff --git a/mercurial/bookmarks.py b/mercurial/bookmarks.py
--- a/mercurial/bookmarks.py
+++ b/mercurial/bookmarks.py
@@ -390,8 +390,7 @@
 finally:
 lockmod.release(tr, l, w)
 
-def compare(repo, srcmarks, dstmarks,
-srchex=None, dsthex=None, targets=None):
+def compare(repo, srcmarks, dstmarks, targets=None):
 '''Compare bookmarks between srcmarks and dstmarks
 
 This returns tuple "(addsrc, adddst, advsrc, advdst, diverge,
@@ -414,19 +413,9 @@
 Changeset IDs of tuples in "addsrc", "adddst", "differ" or
  "invalid" list may be unknown for repo.
 
-This function expects that "srcmarks" and "dstmarks" return
-changeset ID in 40 hexadecimal digit string for specified
-bookmark. If not so (e.g. bmstore "repo._bookmarks" returning
-binary value), "srchex" or "dsthex" should be specified to convert
-into such form.
-
 If "targets" is specified, only bookmarks listed in it are
 examined.
 '''
-if not srchex:
-srchex = lambda x: x
-if not dsthex:
-dsthex = lambda x: x
 
 if targets:
 bset = set(targets)
@@ -448,14 +437,14 @@
 for b in sorted(bset):
 if b not in srcmarks:
 if b in dstmarks:
-adddst((b, None, dsthex(dstmarks[b])))
+adddst((b, None, dstmarks[b]))
 else:
 invalid((b, None, None))
 elif b not in dstmarks:
-addsrc((b, srchex(srcmarks[b]), None))
+addsrc((b, srcmarks[b], None))
 else:
-scid = srchex(srcmarks[b])
-dcid = dsthex(dstmarks[b])
+scid = srcmarks[b]
+dcid = dstmarks[b]
 if scid == dcid:
 same((b, scid, dcid))
 elif scid in repo and dcid in repo:
@@ -506,11 +495,17 @@
 
 return None
 
+def hexifybookmarks(marks):
+binremotemarks = {}
+for name, node in marks.items():
+binremotemarks[name] = bin(node)
+return binremotemarks
+
 def updatefromremote(ui, repo, remotemarks, path, trfunc, explicit=()):
 ui.debug("checking for updated bookmarks\n")
 localmarks = repo._bookmarks
 (addsrc, adddst, advsrc, advdst, diverge, differ, invalid, same
- ) = compare(repo, remotemarks, localmarks, dsthex=hex)
+) = compare(repo, remotemarks, localmarks)
 
 status = ui.status
 warn = ui.warn
@@ -521,15 +516,15 @@
 changed = []
 for b, scid, dcid in addsrc:
 if scid in repo: # add remote bookmarks for changes we already have
-changed.append((b, bin(scid), status,
+changed.append((b, scid, status,
 _("adding remote bookmark %s\n") % (b)))
 elif b in explicit:
 explicit.remove(b)
 ui.warn(_("remote bookmark %s points to locally missing %s\n")
-% (b, scid[:12]))
+% (b, hex(scid)[:12]))
 
 for b, scid, dcid in advsrc:
-changed.append((b, bin(scid), status,
+changed.append((b, scid, status,
 _("updating bookmark %s\n") % (b)))
 # remove normal movement from explicit set
 explicit.difference_update(d[0] for d in changed)
@@ -537,13 +532,12 @@
 for b, scid, dcid in diverge:
 if b in explicit:
 explicit.discard(b)
-changed.append((b, bin(scid), status,
+changed.append((b, scid, status,
 _("importing bookmark %s\n") % (b)))
 else:
-snode = bin(scid)
-db = _diverge(ui, b, path, localmarks, snode)
+db = _diverge(ui, b, path, localmarks, scid)
 if db:
-changed.append((db, snode, warn,
+changed.append((db, scid, warn,
 _("divergent bookmark %s stored as %s\n") %
 (b, db)))
 else:
@@ -552,13 +546,13 @@
 for b, scid, dcid in adddst + advdst:
 if b in explicit:
 explicit.discard(b)
-changed.append((b, bin(scid), status,
+changed.append((b, scid, status,
 _("importing bookmark %s\n") % (b)))
 for b, scid, dcid in differ:
 if b in explicit:
 explicit.remove(b)
 ui.warn(_("

[PATCH 07 of 10 V9] exchange: getbundle `bookmarks` part generator

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1479032456 28800
#  Sun Nov 13 02:20:56 2016 -0800
# Branch stable
# Node ID 606bb4a7fb818f24d52e764828ba0d0a7921f999
# Parent  bf21586f26e5a41f7d8bf342d4b4c16d71dbc6d2
exchange: getbundle `bookmarks` part generator

This generator will be used during pull operation.

diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -1680,6 +1680,21 @@
 if chunks:
 bundler.newpart('hgtagsfnodes', data=''.join(chunks))
 
+@getbundle2partsgenerator('bookmarks')
+def _getbundlebookmarkspart(bundler, repo, source, bundlecaps=None,
+b2caps=None, heads=None, common=None,
+**kwargs):
+if not kwargs.get('bookmarks'):
+return
+if 'bookmarks' not in b2caps:
+raise ValueError(
+_('bookmarks are requested but client is not capable '
+  'of receiving it'))
+
+bookmarks = _getbookmarks(repo, **kwargs)
+encodedbookmarks = bookmod.encodebookmarks(bookmarks)
+bundler.newpart('bookmarks', data=encodedbookmarks)
+
 def _getbookmarks(repo, **kwargs):
 """Returns bookmark to node mapping.
 
diff --git a/mercurial/wireproto.py b/mercurial/wireproto.py
--- a/mercurial/wireproto.py
+++ b/mercurial/wireproto.py
@@ -228,7 +228,9 @@
  'bundlecaps': 'scsv',
  'listkeys': 'csv',
  'cg': 'boolean',
- 'cbattempted': 'boolean'}
+ 'cbattempted': 'boolean',
+ 'bookmarks': 'boolean',
+}
 
 # client side
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 08 of 10 V9] pull: use `bookmarks` bundle2 part

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1479032456 28800
#  Sun Nov 13 02:20:56 2016 -0800
# Branch stable
# Node ID 35096015b14e891b5d6a12af036163875bc60460
# Parent  606bb4a7fb818f24d52e764828ba0d0a7921f999
pull: use `bookmarks` bundle2 part

Apply changes from `bookmarks` part.

diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -1335,9 +1335,13 @@
 kwargs['cg'] = pullop.fetch
 if 'listkeys' in pullop.remotebundle2caps:
 kwargs['listkeys'] = ['phases']
-if pullop.remotebookmarks is None:
-# make sure to always includes bookmark data when migrating
-# `hg incoming --bundle` to using this function.
+
+if pullop.remotebookmarks is None:
+# make sure to always includes bookmark data when migrating
+# `hg incoming --bundle` to using this function.
+if 'bookmarks' in pullop.remotebundle2caps:
+kwargs['bookmarks'] = True
+elif 'listkeys' in pullop.remotebundle2caps:
 kwargs['listkeys'].append('bookmarks')
 
 # If this is a full pull / clone and the server supports the clone bundles
@@ -1365,10 +1369,23 @@
 _pullbundle2extraprepare(pullop, kwargs)
 bundle = pullop.remote.getbundle('pull', **kwargs)
 try:
-op = bundle2.processbundle(pullop.repo, bundle, pullop.gettransaction)
+bundleopinput = {
+'processbookmarksmode': 'diverge',
+'explicitbookmarks': pullop.explicitbookmarks,
+'remotepath': pullop.remote.url(),
+}
+op = bundle2.bundleoperation(pullop.repo, pullop.gettransaction,
+ input=bundleopinput)
+op = bundle2.processbundle(pullop.repo, bundle,
+   pullop.gettransaction, op=op)
 except error.BundleValueError as exc:
 raise error.Abort(_('missing support for %s') % exc)
 
+# `bookmarks` part was in bundle and it was applied to the repo. No need to
+# apply bookmarks one more time
+if 'bookmarks' in kwargs and kwargs['bookmarks']:
+pullop.stepsdone.add('bookmarks')
+
 if pullop.fetch:
 results = [cg['return'] for cg in op.records['changegroup']]
 pullop.cgresult = changegroup.combineresults(results)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 10 of 10 V9] help: add documentation about bookmark part

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1479032456 28800
#  Sun Nov 13 02:20:56 2016 -0800
# Branch stable
# Node ID f221666dfbb6e055219634aa18c7460b0bcf24a4
# Parent  a261e5b0c5a402c5f084bb8fcf53866f2f900687
help: add documentation about bookmark part

diff --git a/mercurial/help/internals/bundles.txt 
b/mercurial/help/internals/bundles.txt
--- a/mercurial/help/internals/bundles.txt
+++ b/mercurial/help/internals/bundles.txt
@@ -92,3 +92,32 @@
 ``HGS1UN`` support was added as an experimental feature in version 3.6
 (released November 2015) as part of the initial offering of the *clone
 bundles* feature.
+
+Bundle2 parts
+=
+
+Bundle2 may contain many different pieces of information. These pieces are
+called parts.
+
+Bookmarks part
+--
+
+This part contains information about bookmarks. Part consists of many entries.
+Each entry describes one bookmark. Entry format:
+
+4 bytes
+  bookmark size
+1 byte
+  boolean. True if node is empty, False otherwise
+20 bytes (optional)
+  node. Present only if previous field is True
+
+Modes:
+
+1. 'ignore' - do not apply any changes to the repo, just decode the passed
+bookmarks. Will be used to list bookmarks in remote repo.
+2. 'diverge' - apply bookmark changes to the repo. Create divergent bookmarks 
if
+there is a non-fastforward move. Will be used during pull.
+3. 'apply' - apply bookmark changes to the repo. Overwrite current bookmark 
node
+if there is a non-fastforward move. Will be used during push.
+
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 01 of 10 V9] bookmarks: introduce listbinbookmarks()

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1479032036 28800
#  Sun Nov 13 02:13:56 2016 -0800
# Branch stable
# Node ID ffa31155703b725b20d67f379bea4de7c56379fe
# Parent  b9f7b0c10027764cee77f9c6d61877fcffea837f
bookmarks: introduce listbinbookmarks()

`bookmarks` bundle2 part will work with binary nodes. To avoid unnecessary
conversions between binary and hex nodes let's add `listbinbookmarks()` that
returns binary nodes. For now this function is a copy-paste of
listbookmarks(). In the next patch this copy-paste will be removed.

diff --git a/mercurial/bookmarks.py b/mercurial/bookmarks.py
--- a/mercurial/bookmarks.py
+++ b/mercurial/bookmarks.py
@@ -284,6 +284,17 @@
 lockmod.release(tr, lock)
 return update
 
+def listbinbookmarks(repo):
+# We may try to list bookmarks on a repo type that does not
+# support it (e.g., statichttprepository).
+marks = getattr(repo, '_bookmarks', {})
+
+hasnode = repo.changelog.hasnode
+for k, v in marks.iteritems():
+# don't expose local divergent bookmarks
+if hasnode(v) and ('@' not in k or k.endswith('@')):
+yield k, v
+
 def listbookmarks(repo):
 # We may try to list bookmarks on a repo type that does not
 # support it (e.g., statichttprepository).
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 09 of 10 V8] bundle2: advertise bookmark capability

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1478980606 28800
#  Sat Nov 12 11:56:46 2016 -0800
# Branch stable
# Node ID dd7caf9f799c4c8e012afca214683159645cc931
# Parent  66787435b18cf4050ba50097b4d6725f09233b74
bundle2: advertise bookmark capability

diff --git a/mercurial/bundle2.py b/mercurial/bundle2.py
--- a/mercurial/bundle2.py
+++ b/mercurial/bundle2.py
@@ -1271,6 +1271,7 @@
 'digests': tuple(sorted(util.DIGESTS.keys())),
 'remote-changegroup': ('http', 'https'),
 'hgtagsfnodes': (),
+'bookmarks': (),
}
 
 def getrepocaps(repo, allowpushback=False):
diff --git a/tests/test-acl.t b/tests/test-acl.t
--- a/tests/test-acl.t
+++ b/tests/test-acl.t
@@ -92,13 +92,13 @@
   f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd
   911600dab2ae7a9baff75958b84fe606851ce955
   bundle2-output-bundle: "HG20", 4 parts total
-  bundle2-output-part: "replycaps" 155 bytes payload
+  bundle2-output-part: "replycaps" 165 bytes payload
   bundle2-output-part: "check:heads" streamed payload
   bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload
   bundle2-output-part: "pushkey" (params: 4 mandatory) empty payload
   bundle2-input-bundle: with-transaction
   bundle2-input-part: "replycaps" supported
-  bundle2-input-part: total payload size 155
+  bundle2-input-part: total payload size 165
   bundle2-input-part: "check:heads" supported
   bundle2-input-part: total payload size 20
   bundle2-input-part: "changegroup" (params: 1 mandatory) supported
@@ -155,13 +155,13 @@
   f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd
   911600dab2ae7a9baff75958b84fe606851ce955
   bundle2-output-bundle: "HG20", 4 parts total
-  bundle2-output-part: "replycaps" 155 bytes payload
+  bundle2-output-part: "replycaps" 165 bytes payload
   bundle2-output-part: "check:heads" streamed payload
   bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload
   bundle2-output-part: "pushkey" (params: 4 mandatory) empty payload
   bundle2-input-bundle: with-transaction
   bundle2-input-part: "replycaps" supported
-  bundle2-input-part: total payload size 155
+  bundle2-input-part: total payload size 165
   bundle2-input-part: "check:heads" supported
   bundle2-input-part: total payload size 20
   bundle2-input-part: "changegroup" (params: 1 mandatory) supported
@@ -221,13 +221,13 @@
   f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd
   911600dab2ae7a9baff75958b84fe606851ce955
   bundle2-output-bundle: "HG20", 4 parts total
-  bundle2-output-part: "replycaps" 155 bytes payload
+  bundle2-output-part: "replycaps" 165 bytes payload
   bundle2-output-part: "check:heads" streamed payload
   bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload
   bundle2-output-part: "pushkey" (params: 4 mandatory) empty payload
   bundle2-input-bundle: with-transaction
   bundle2-input-part: "replycaps" supported
-  bundle2-input-part: total payload size 155
+  bundle2-input-part: total payload size 165
   bundle2-input-part: "check:heads" supported
   bundle2-input-part: total payload size 20
   bundle2-input-part: "changegroup" (params: 1 mandatory) supported
@@ -297,13 +297,13 @@
   f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd
   911600dab2ae7a9baff75958b84fe606851ce955
   bundle2-output-bundle: "HG20", 4 parts total
-  bundle2-output-part: "replycaps" 155 bytes payload
+  bundle2-output-part: "replycaps" 165 bytes payload
   bundle2-output-part: "check:heads" streamed payload
   bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload
   bundle2-output-part: "pushkey" (params: 4 mandatory) empty payload
   bundle2-input-bundle: with-transaction
   bundle2-input-part: "replycaps" supported
-  bundle2-input-part: total payload size 155
+  bundle2-input-part: total payload size 165
   bundle2-input-part: "check:heads" supported
   bundle2-input-part: total payload size 20
   bundle2-input-part: "changegroup" (params: 1 mandatory) supported
@@ -362,13 +362,13 @@
   f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd
   911600dab2ae7a9baff75958b84fe606851ce955
   bundle2-output-bundle: "HG20", 4 parts total
-  bundle2-output-part: "replycaps" 155 bytes payload
+  bundle2-output-part: "replycaps" 165 bytes payload
   bundle2-output-part: "check:heads" streamed payload
   bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload
   bundle2-output-part: "pushkey" (params: 4 mandatory) empty payload
   bundle2-input-bundle: with-transaction
   bundle2-input-part: "replycaps" supported
-  bundle

[PATCH 10 of 10 V8] help: add documentation about bookmark part

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1478980606 28800
#  Sat Nov 12 11:56:46 2016 -0800
# Branch stable
# Node ID d1bfcb4a1594060275db040e6534fd262c5a43ec
# Parent  dd7caf9f799c4c8e012afca214683159645cc931
help: add documentation about bookmark part

diff --git a/mercurial/help/internals/bundles.txt 
b/mercurial/help/internals/bundles.txt
--- a/mercurial/help/internals/bundles.txt
+++ b/mercurial/help/internals/bundles.txt
@@ -92,3 +92,32 @@
 ``HGS1UN`` support was added as an experimental feature in version 3.6
 (released November 2015) as part of the initial offering of the *clone
 bundles* feature.
+
+Bundle2 parts
+=
+
+Bundle2 may contain many different pieces of information. These pieces are
+called parts.
+
+Bookmarks part
+--
+
+This part contains information about bookmarks. Part consists of many entries.
+Each entry describes one bookmark. Entry format:
+
+4 bytes
+  bookmark size
+1 byte
+  boolean. True if node is empty, False otherwise
+20 bytes (optional)
+  node. Present only if previous field is True
+
+Modes:
+
+1. 'ignore' - do not apply any changes to the repo, just decode the passed
+bookmarks. Will be used to list bookmarks in remote repo.
+2. 'diverge' - apply bookmark changes to the repo. Create divergent bookmarks 
if
+there is a non-fastforward move. Will be used during pull.
+3. 'apply' - apply bookmark changes to the repo. Overwrite current bookmark 
node
+if there is a non-fastforward move. Will be used during push.
+
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 06 of 10 V8] bundle2: add `bookmarks` part handler

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1478980606 28800
#  Sat Nov 12 11:56:46 2016 -0800
# Branch stable
# Node ID f8a24164d9190f457ff5cd5089b8879c71ec801f
# Parent  745156544137985fe5c6b31fe99232e84d596ec4
bundle2: add `bookmarks` part handler

Applies bookmarks part to the local repo. `processbookmarksmode` determines
how remote bookmarks are handled. They are either ignored ('ignore' mode),
diverged ('diverge' mode) or applied ('apply' mode).

diff --git a/mercurial/bundle2.py b/mercurial/bundle2.py
--- a/mercurial/bundle2.py
+++ b/mercurial/bundle2.py
@@ -155,6 +155,7 @@
 
 from .i18n import _
 from . import (
+bookmarks as bookmod,
 changegroup,
 error,
 obsolete,
@@ -287,13 +288,21 @@
 * a way to construct a bundle response when applicable.
 """
 
-def __init__(self, repo, transactiongetter, captureoutput=True):
+def __init__(self, repo, transactiongetter, captureoutput=True,
+ input=None):
+"""
+`input` is a dictionary that is passed to part handlers to tweak
+their behaviour
+"""
 self.repo = repo
 self.ui = repo.ui
 self.records = unbundlerecords()
 self.gettransaction = transactiongetter
 self.reply = None
 self.captureoutput = captureoutput
+if input is None:
+input = {}
+self.input = input
 
 class TransactionUnavailable(RuntimeError):
 pass
@@ -1624,3 +1633,32 @@
 
 cache.write()
 op.ui.debug('applied %i hgtags fnodes cache entries\n' % count)
+
+@parthandler('bookmarks')
+def handlebookmarks(op, inpart):
+"""Processes bookmarks part.
+
+`processbookmarksmode` determines how remote bookmarks are handled. They 
are
+either ignored ('ignore' mode), diverged ('diverge' mode) or applied
+('apply' mode). 'ignore' mode is used to get bookmarks and process them
+later, 'diverge' mode is used to process bookmarks during pull, 'apply'
+mode is used during push.
+"""
+
+bookmarks = {}
+bookmarks = bookmod.decodebookmarks(inpart.read())
+processbookmarksmode = op.input.get('processbookmarksmode', 'ignore')
+if processbookmarksmode == 'apply':
+for bookmark, node in bookmarks.items():
+if node:
+op.repo._bookmarks[bookmark] = node
+else:
+del op.repo._bookmarks[bookmark]
+op.repo._bookmarks.recordchange(op.gettransaction())
+elif processbookmarksmode == 'diverge':
+remotepath = op.input.get('remotepath', '')
+explicitbookmarks = op.input.get('explicitbookmarks', ())
+bookmod.updatefromremote(op.ui, op.repo, bookmarks,
+ remotepath, op.gettransaction,
+ explicit=explicitbookmarks)
+op.records.add('bookmarks', bookmarks)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 07 of 10 V8] exchange: getbundle `bookmarks` part generator

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1478980606 28800
#  Sat Nov 12 11:56:46 2016 -0800
# Branch stable
# Node ID 7bd99f5871d67df64e60174f2c349bc088a584f6
# Parent  f8a24164d9190f457ff5cd5089b8879c71ec801f
exchange: getbundle `bookmarks` part generator

This generator will be used during pull operation.

diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -1680,6 +1680,21 @@
 if chunks:
 bundler.newpart('hgtagsfnodes', data=''.join(chunks))
 
+@getbundle2partsgenerator('bookmarks')
+def _getbundlebookmarkspart(bundler, repo, source, bundlecaps=None,
+b2caps=None, heads=None, common=None,
+**kwargs):
+if not kwargs.get('bookmarks'):
+return
+if 'bookmarks' not in b2caps:
+raise ValueError(
+_('bookmarks are requested but client is not capable '
+  'of receiving it'))
+
+bookmarks = _getbookmarks(repo, **kwargs)
+encodedbookmarks = bookmod.encodebookmarks(bookmarks)
+bundler.newpart('bookmarks', data=encodedbookmarks)
+
 def _getbookmarks(repo, **kwargs):
 """Returns bookmark to node mapping.
 
diff --git a/mercurial/wireproto.py b/mercurial/wireproto.py
--- a/mercurial/wireproto.py
+++ b/mercurial/wireproto.py
@@ -228,7 +228,9 @@
  'bundlecaps': 'scsv',
  'listkeys': 'csv',
  'cg': 'boolean',
- 'cbattempted': 'boolean'}
+ 'cbattempted': 'boolean',
+ 'bookmarks': 'boolean',
+}
 
 # client side
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 08 of 10 V8] pull: use `bookmarks` bundle2 part

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1478980606 28800
#  Sat Nov 12 11:56:46 2016 -0800
# Branch stable
# Node ID 66787435b18cf4050ba50097b4d6725f09233b74
# Parent  7bd99f5871d67df64e60174f2c349bc088a584f6
pull: use `bookmarks` bundle2 part

Apply changes from `bookmarks` part.

diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -1335,9 +1335,13 @@
 kwargs['cg'] = pullop.fetch
 if 'listkeys' in pullop.remotebundle2caps:
 kwargs['listkeys'] = ['phases']
-if pullop.remotebookmarks is None:
-# make sure to always includes bookmark data when migrating
-# `hg incoming --bundle` to using this function.
+
+if pullop.remotebookmarks is None:
+# make sure to always includes bookmark data when migrating
+# `hg incoming --bundle` to using this function.
+if 'bookmarks' in pullop.remotebundle2caps:
+kwargs['bookmarks'] = True
+elif 'listkeys' in pullop.remotebundle2caps:
 kwargs['listkeys'].append('bookmarks')
 
 # If this is a full pull / clone and the server supports the clone bundles
@@ -1365,10 +1369,23 @@
 _pullbundle2extraprepare(pullop, kwargs)
 bundle = pullop.remote.getbundle('pull', **kwargs)
 try:
-op = bundle2.processbundle(pullop.repo, bundle, pullop.gettransaction)
+bundleopinput = {
+'processbookmarksmode': 'diverge',
+'explicitbookmarks': pullop.explicitbookmarks,
+'remotepath': pullop.remote.url(),
+}
+op = bundle2.bundleoperation(pullop.repo, pullop.gettransaction,
+ input=bundleopinput)
+op = bundle2.processbundle(pullop.repo, bundle,
+   pullop.gettransaction, op=op)
 except error.BundleValueError as exc:
 raise error.Abort(_('missing support for %s') % exc)
 
+# `bookmarks` part was in bundle and it was applied to the repo. No need to
+# apply bookmarks one more time
+if 'bookmarks' in kwargs and kwargs['bookmarks']:
+pullop.stepsdone.add('bookmarks')
+
 if pullop.fetch:
 results = [cg['return'] for cg in op.records['changegroup']]
 pullop.cgresult = changegroup.combineresults(results)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 05 of 10 V8] bookmarks: make bookmarks.compare accept binary nodes

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1478980606 28800
#  Sat Nov 12 11:56:46 2016 -0800
# Branch stable
# Node ID 745156544137985fe5c6b31fe99232e84d596ec4
# Parent  767dc51625be5f1ca62fe15cfa17f96dcd1b7cec
bookmarks: make bookmarks.compare accept binary nodes

New `bookmarks` bundle2 part will contain byte nodes, and since bookmarks are
stored in binary format, it doesn't make sense to convert them from binary to
hex and then back to binary again

diff --git a/mercurial/bookmarks.py b/mercurial/bookmarks.py
--- a/mercurial/bookmarks.py
+++ b/mercurial/bookmarks.py
@@ -390,8 +390,7 @@
 finally:
 lockmod.release(tr, l, w)
 
-def compare(repo, srcmarks, dstmarks,
-srchex=None, dsthex=None, targets=None):
+def compare(repo, srcmarks, dstmarks, targets=None):
 '''Compare bookmarks between srcmarks and dstmarks
 
 This returns tuple "(addsrc, adddst, advsrc, advdst, diverge,
@@ -414,19 +413,9 @@
 Changeset IDs of tuples in "addsrc", "adddst", "differ" or
  "invalid" list may be unknown for repo.
 
-This function expects that "srcmarks" and "dstmarks" return
-changeset ID in 40 hexadecimal digit string for specified
-bookmark. If not so (e.g. bmstore "repo._bookmarks" returning
-binary value), "srchex" or "dsthex" should be specified to convert
-into such form.
-
 If "targets" is specified, only bookmarks listed in it are
 examined.
 '''
-if not srchex:
-srchex = lambda x: x
-if not dsthex:
-dsthex = lambda x: x
 
 if targets:
 bset = set(targets)
@@ -448,14 +437,14 @@
 for b in sorted(bset):
 if b not in srcmarks:
 if b in dstmarks:
-adddst((b, None, dsthex(dstmarks[b])))
+adddst((b, None, dstmarks[b]))
 else:
 invalid((b, None, None))
 elif b not in dstmarks:
-addsrc((b, srchex(srcmarks[b]), None))
+addsrc((b, srcmarks[b], None))
 else:
-scid = srchex(srcmarks[b])
-dcid = dsthex(dstmarks[b])
+scid = srcmarks[b]
+dcid = dstmarks[b]
 if scid == dcid:
 same((b, scid, dcid))
 elif scid in repo and dcid in repo:
@@ -506,11 +495,17 @@
 
 return None
 
+def hexifybookmarks(marks):
+binremotemarks = {}
+for name, node in marks.items():
+binremotemarks[name] = bin(node)
+return binremotemarks
+
 def updatefromremote(ui, repo, remotemarks, path, trfunc, explicit=()):
 ui.debug("checking for updated bookmarks\n")
 localmarks = repo._bookmarks
 (addsrc, adddst, advsrc, advdst, diverge, differ, invalid, same
- ) = compare(repo, remotemarks, localmarks, dsthex=hex)
+) = compare(repo, remotemarks, localmarks)
 
 status = ui.status
 warn = ui.warn
@@ -521,15 +516,15 @@
 changed = []
 for b, scid, dcid in addsrc:
 if scid in repo: # add remote bookmarks for changes we already have
-changed.append((b, bin(scid), status,
+changed.append((b, scid, status,
 _("adding remote bookmark %s\n") % (b)))
 elif b in explicit:
 explicit.remove(b)
 ui.warn(_("remote bookmark %s points to locally missing %s\n")
-% (b, scid[:12]))
+% (b, hex(scid)[:12]))
 
 for b, scid, dcid in advsrc:
-changed.append((b, bin(scid), status,
+changed.append((b, scid, status,
 _("updating bookmark %s\n") % (b)))
 # remove normal movement from explicit set
 explicit.difference_update(d[0] for d in changed)
@@ -537,13 +532,12 @@
 for b, scid, dcid in diverge:
 if b in explicit:
 explicit.discard(b)
-changed.append((b, bin(scid), status,
+changed.append((b, scid, status,
 _("importing bookmark %s\n") % (b)))
 else:
-snode = bin(scid)
-db = _diverge(ui, b, path, localmarks, snode)
+db = _diverge(ui, b, path, localmarks, scid)
 if db:
-changed.append((db, snode, warn,
+changed.append((db, scid, warn,
 _("divergent bookmark %s stored as %s\n") %
 (b, db)))
 else:
@@ -552,13 +546,13 @@
 for b, scid, dcid in adddst + advdst:
 if b in explicit:
 explicit.discard(b)
-changed.append((b, bin(scid), status,
+changed.append((b, scid, status,
 _("importing bookmark %s\n") % (b)))
 for b, scid, dcid in differ:
 if b in explicit:
 explicit.remove(b)
 ui.warn(_("

[PATCH 04 of 10 V8] bookmarks: introduce binary encoding

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1478980606 28800
#  Sat Nov 12 11:56:46 2016 -0800
# Branch stable
# Node ID 767dc51625be5f1ca62fe15cfa17f96dcd1b7cec
# Parent  cf754de559217f5965d1101c4591689d9924bddc
bookmarks: introduce binary encoding

Bookmarks binary encoding will be used for `bookmarks` bundle2 part.
The format is: <4 bytes - bookmark size, big-endian>
   <1 byte - 0 if node is empty 1 otherwise><20 bytes node>
BookmarksEncodeError and BookmarksDecodeError maybe thrown if input is
incorrect.

diff --git a/mercurial/bookmarks.py b/mercurial/bookmarks.py
--- a/mercurial/bookmarks.py
+++ b/mercurial/bookmarks.py
@@ -8,7 +8,9 @@
 from __future__ import absolute_import
 
 import errno
+import io
 import os
+import struct
 
 from .i18n import _
 from .node import (
@@ -23,6 +25,71 @@
 util,
 )
 
+_NONEMPTYNODE = struct.pack('?', False)
+_EMPTYNODE = struct.pack('?', True)
+
+def _unpackbookmarksize(stream):
+"""Returns 0 if stream is empty.
+"""
+
+intstruct = struct.Struct('>i')
+expectedsize = intstruct.size
+encodedbookmarksize = stream.read(expectedsize)
+if not encodedbookmarksize:
+return 0
+if len(encodedbookmarksize) != expectedsize:
+raise ValueError(
+_('cannot decode bookmark size: '
+  'expected size: %d, actual size: %d') %
+(expectedsize, len(encodedbookmarksize)))
+return intstruct.unpack(encodedbookmarksize)[0]
+
+def encodebookmarks(bookmarks):
+"""Encodes bookmark to node mapping to the byte string.
+
+Format: <4 bytes - bookmark size>
+<1 byte - 0 if node is empty 1 otherwise><20 bytes node>
+Node may be 20 byte binary string or empty
+"""
+
+intstruct = struct.Struct('>i')
+for bookmark, node in sorted(bookmarks.iteritems()):
+yield intstruct.pack(len(bookmark))
+yield encoding.fromlocal(bookmark)
+if node:
+if len(node) != 20:
+raise ValueError(_('node must be 20 or bytes long'))
+yield _NONEMPTYNODE
+yield node
+else:
+yield _EMPTYNODE
+
+def decodebookmarks(buf):
+"""Decodes buffer into bookmark to node mapping.
+
+Node is either 20 bytes or empty.
+"""
+
+stream = io.BytesIO(buf)
+bookmarks = {}
+bookmarksize = _unpackbookmarksize(stream)
+boolstruct = struct.Struct('?')
+while bookmarksize:
+bookmark = stream.read(bookmarksize)
+if len(bookmark) != bookmarksize:
+raise ValueError(
+_('cannot decode bookmark: expected size: %d, '
+'actual size: %d') % (bookmarksize, len(bookmark)))
+bookmark = encoding.tolocal(bookmark)
+packedemptynodeflag = stream.read(boolstruct.size)
+emptynode = boolstruct.unpack(packedemptynodeflag)[0]
+node = ''
+if not emptynode:
+node = stream.read(20)
+bookmarks[bookmark] = node
+bookmarksize = _unpackbookmarksize(stream)
+return bookmarks
+
 def _getbkfile(repo):
 """Hook so that extensions that mess with the store can hook bm storage.
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 03 of 10 V8] exchange: add `_getbookmarks()` function

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1478980606 28800
#  Sat Nov 12 11:56:46 2016 -0800
# Branch stable
# Node ID cf754de559217f5965d1101c4591689d9924bddc
# Parent  13b3e16c68d303a523550980d7739bb676420851
exchange: add `_getbookmarks()` function

This function will be used to generate bookmarks bundle2 part.
It is a separate function in order to make it easy to overwrite it
in extensions. Passing `kwargs` to the function makes it easy to
add new parameters in extensions.

diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -1666,6 +1666,17 @@
 if chunks:
 bundler.newpart('hgtagsfnodes', data=''.join(chunks))
 
+def _getbookmarks(repo, **kwargs):
+"""Returns bookmark to node mapping.
+
+This function is primarily used to generate `bookmarks` bundle2 part.
+It is a separate function in order to make it easy to wrap it
+in extensions. Passing `kwargs` to the function makes it easy to
+add new parameters in extensions.
+"""
+
+return dict(bookmod.listbookmarks(repo))
+
 def check_heads(repo, their_heads, context):
 """check if the heads of a repo have been modified
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 02 of 10 V8] bookmarks: introduce listbookmarks()

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1478980589 28800
#  Sat Nov 12 11:56:29 2016 -0800
# Branch stable
# Node ID 13b3e16c68d303a523550980d7739bb676420851
# Parent  fe9e78883230a875ae7acf6c7a5b324c1d9016f5
bookmarks: introduce listbookmarks()

`bookmarks` bundle2 part will work with binary nodes.
To avoid unnecessary conversions between binary and hex nodes
let's add `listbookmarks()` that returns binary nodes.

diff --git a/mercurial/bookmarks.py b/mercurial/bookmarks.py
--- a/mercurial/bookmarks.py
+++ b/mercurial/bookmarks.py
@@ -284,17 +284,21 @@
 lockmod.release(tr, lock)
 return update
 
-def listhexbookmarks(repo):
+def listbookmarks(repo):
 # We may try to list bookmarks on a repo type that does not
 # support it (e.g., statichttprepository).
 marks = getattr(repo, '_bookmarks', {})
 
-d = {}
 hasnode = repo.changelog.hasnode
 for k, v in marks.iteritems():
 # don't expose local divergent bookmarks
 if hasnode(v) and ('@' not in k or k.endswith('@')):
-d[k] = hex(v)
+yield k, v
+
+def listhexbookmarks(repo):
+d = {}
+for book, node in listbookmarks(repo):
+d[book] = hex(node)
 return d
 
 def pushbookmark(repo, key, old, new):
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 6 of 9 V6] bundle2: add `bookmarks` part handler

[Stanislau Hlebik]

BTW, how do you think it should look like?
Do you want to introduce another type of categories in unbundlerecords or just 
rewrite categories whenever a new bookmark category is added?

On 11/11/16, 11:50 AM, "Stanislau Hlebik" <st...@fb.com> wrote:

I think we can but it would require a bit of rewriting in unbundlerecords.

On 11/10/16, 5:39 PM, "Pierre-Yves David" <pierre-yves.da...@ens-lyon.org> 
wrote:



On 11/02/2016 10:56 AM, Stanislau Hlebik wrote:
>
>
> On 10/14/16, 2:28 AM, "Pierre-Yves David" 
<pierre-yves.da...@ens-lyon.org> wrote:
>
> What is your plan regarding hooks execution here?
>
> I planned to add it when I’m going to work on push
>
> How do you handle cases where the part exist multiple time in the 
bundle ?
>
> Is there a problem with multiple parts? `op.records.add('bookmarks', 
bookmarks)` appends bookmarks from each part so it should work fine.

Yes but it adds them as multiple items instead of a unified 
dictionnary, 
this seems strange. Could we have them as a single dict instead?

-- 
Pierre-Yves David




___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 5 of 9 V6] bookmarks: add srchex param to updatefromremote

[Stanislau Hlebik]

That should be fine

On 11/11/16, 11:59 AM, "Pierre-Yves David" <pierre-yves.da...@ens-lyon.org> 
wrote:



On 11/11/2016 11:29 AM, Stanislau Hlebik wrote:
> Not sure I fully understand your idea.
> Do you suggest to rewrite bookmarks.compare and 
bookmarks.updatefromremote to accept only binary nodes?

Yes, it would like it would make more sense.

If I understand your series correctly, we otherwise get in a situation 
where a caller with the binary data has to convert them to hex and the 
function internal have to convert thing again because internal storage 
is node. If everybody were talking node the situation would be simpler.

Does this make sense ?

-- 
Pierre-Yves David


___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 2 of 8 V7] bookmarks: introduce binary encoding

[Stanislau Hlebik]

We can use 32 bytes, but only 20 bytes will be used.
What’s the point then?

On 11/9/16, 5:22 PM, "Augie Fackler" <r...@durin42.com> wrote:

On Wed, Nov 02, 2016 at 07:06:44AM -0700, Stanislau Hlebik wrote:
> # HG changeset patch
    > # User Stanislau Hlebik <st...@fb.com>
> # Date 1478016027 25200
> #  Tue Nov 01 09:00:27 2016 -0700
> # Branch stable
> # Node ID f3da841e3d47ccbb0be3892b521607c09fdeab13
> # Parent  a56a624a8a42b93ec980d2c284756a38719dffe6
> bookmarks: introduce binary encoding
>
> Bookmarks binary encoding will be used for `bookmarks` bundle2 part.
> The format is: <4 bytes - bookmark size, big-endian>
><1 byte - 0 if node is empty 1 otherwise><20 bytes node>

Enough other formats have 32 bytes of space for nodes (eg revlog) that
I suspect we should do 32 bytes here too. Thoughts?

> BookmarksEncodeError and BookmarksDecodeError maybe thrown if input is
> incorrect.
>
> diff --git a/mercurial/bookmarks.py b/mercurial/bookmarks.py
> --- a/mercurial/bookmarks.py
> +++ b/mercurial/bookmarks.py
> @@ -7,8 +7,10 @@
>
>  from __future__ import absolute_import
>
> +import StringIO
>  import errno
>  import os
> +import struct
>
>  from .i18n import _
>  from .node import (
> @@ -23,6 +25,70 @@
>  util,
>  )
>
> +_NONEMPTYNODE = struct.pack('?', False)
> +_EMPTYNODE = struct.pack('?', True)
> +
> +def _unpackbookmarksize(stream):
> +"""Returns 0 if stream is empty.
> +"""
> +
> +expectedsize = struct.calcsize('>i')
> +encodedbookmarksize = stream.read(expectedsize)
> +if len(encodedbookmarksize) == 0:
> +return 0
> +if len(encodedbookmarksize) != expectedsize:
> +raise ValueError(
> +_('cannot decode bookmark size: '
> +  'expected size: %d, actual size: %d') %
> +(expectedsize, len(encodedbookmarksize)))
> +return struct.unpack('>i', encodedbookmarksize)[0]
> +
> +def encodebookmarks(bookmarks):
> +"""Encodes bookmark to node mapping to the byte string.
> +
> +Format: <4 bytes - bookmark size>
> +<1 byte - 0 if node is empty 1 otherwise><20 bytes node>
> +Node may be 20 byte binary string, 40 byte hex string or empty
> +"""
> +
> +for bookmark, node in bookmarks.iteritems():
> +yield struct.pack('>i', (len(bookmark)))
> +yield encoding.fromlocal(bookmark)
> +if node:
> +if len(node) != 20 and len(node) != 40:
> +raise ValueError(_('node must be 20 or bytes long'))
> +if len(node) == 40:
> +node = bin(node)
> +yield _NONEMPTYNODE
> +yield node
> +else:
> +yield _EMPTYNODE
> +
> +def decodebookmarks(buf):
> +"""Decodes buffer into bookmark to node mapping.
> +
> +Node is either 20 bytes or empty.
> +"""
> +
> +stream = StringIO.StringIO(buf)
> +bookmarks = {}
> +bookmarksize = _unpackbookmarksize(stream)
> +while bookmarksize:
> +bookmark = stream.read(bookmarksize)
> +if len(bookmark) != bookmarksize:
> +raise ValueError(
> +_('cannot decode bookmark: expected size: %d, '
> +'actual size: %d') % (bookmarksize, len(bookmark)))
> +bookmark = encoding.tolocal(bookmark)
> +packedemptynodeflag = stream.read(struct.calcsize('?'))
> +emptynode = struct.unpack('?', packedemptynodeflag)[0]
> +node = ''
> +if not emptynode:
> +node = stream.read(20)
> +bookmarks[bookmark] = node
> +bookmarksize = _unpackbookmarksize(stream)
> +return bookmarks
> +
>  def _getbkfile(repo):
>  """Hook so that extensions that mess with the store can hook bm 
storage.
>
> ___
> Mercurial-devel mailing list
> Mercurial-devel@mercurial-scm.org
> 
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.mercurial-2Dscm.org_mailman_listinfo_mercurial-2Ddevel=DQIBAg=5VD0RTtNlTh3ycd41b3MUw=1EQ58Dmb5uX1qHujcsT1Mg=XfOL5bchNzFkHn3jvJndGgHchtnR08c9-WuY56aaUhw=xzxL1iYhrBKwmRk52RxWVbj-I3f4dfTiKl4dd9tjkOQ=
 


___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 8 of 8 V7] help: add documentation about bookmark part

[Stanislau Hlebik]

No, it doesn’t fix it yet, because this series only uses bookmarks part only 
for pull.
I’ll add support for bookmarks part for push in the next patches and it will 
fix the issue.

On 11/9/16, 5:27 PM, "Augie Fackler" <r...@durin42.com> wrote:

On Wed, Nov 02, 2016 at 07:06:50AM -0700, Stanislau Hlebik wrote:
> # HG changeset patch
    > # User Stanislau Hlebik <st...@fb.com>
> # Date 1478095405 25200
> #  Wed Nov 02 07:03:25 2016 -0700
> # Branch stable
> # Node ID 5ed1eda9bf3e92dd6037028a0202690dc845efa2
> # Parent  c2a9f675fac55522bb954ea81c29fa0158106214
> help: add documentation about bookmark part

Overall, I like where this has gone. I've got a question earlier in
the series, but this looks very close to done. Thanks!

Can I interest you in checking

https://urldefense.proofpoint.com/v2/url?u=https-3A__hg.durin42.com_hg-2Dwip_rev_5dc4e70f2807=DQIBAg=5VD0RTtNlTh3ycd41b3MUw=1EQ58Dmb5uX1qHujcsT1Mg=Gxpj-1Zv83d4eD2d-m1AIHJdewxh_0UB6udkd5Z39R4=cJJveAtT-wtKgQzoYyhdGbKEbIEUqxpQ9p_LUrQhw2g=
  to see if it fixes a
long-standing bug around exchange of long bookmark names?

>
> diff --git a/mercurial/help/internals/bundles.txt 
b/mercurial/help/internals/bundles.txt
> --- a/mercurial/help/internals/bundles.txt
> +++ b/mercurial/help/internals/bundles.txt
> @@ -92,3 +92,32 @@
>  ``HGS1UN`` support was added as an experimental feature in version 3.6
>  (released November 2015) as part of the initial offering of the *clone
>  bundles* feature.
> +
> +Bundle2 parts
> +=
> +
> +Bundle2 may contain many different pieces of information. These pieces 
are
> +called parts.
> +
> +Bookmarks part
> +--
> +
> +This part contains information about bookmarks. Part consists of many 
entries.
> +Each entry describes one bookmark. Entry format:
> +
> +4 bytes
> +  bookmark size
> +1 byte
> +  boolean. True if node is empty, False otherwise
> +20 bytes (optional)
> +  node. Present only if previous field is True
> +
> +Modes:
> +
> +1. 'ignore' - do not apply any changes to the repo, just decode the 
passed
> +bookmarks. Will be used to list bookmarks in remote repo.
> +2. 'diverge' - apply bookmark changes to the repo. Create divergent 
bookmarks if
> +there is a non-fastforward move. Will be used during pull.
> +3. 'apply' - apply bookmark changes to the repo. Overwrite current 
bookmark node
> +if there is a non-fastforward move. Will be used during push.
> +
> ___
> Mercurial-devel mailing list
> Mercurial-devel@mercurial-scm.org
> 
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.mercurial-2Dscm.org_mailman_listinfo_mercurial-2Ddevel=DQIBAg=5VD0RTtNlTh3ycd41b3MUw=1EQ58Dmb5uX1qHujcsT1Mg=Gxpj-1Zv83d4eD2d-m1AIHJdewxh_0UB6udkd5Z39R4=ON8rcjQ6bpGyVMtgJG4pD8JkizxtLHZX3eD2QZx8iOo=
 


___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 1 of 8 V7] exchange: add `_getbookmarks()` function

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1478016027 25200
#  Tue Nov 01 09:00:27 2016 -0700
# Branch stable
# Node ID a56a624a8a42b93ec980d2c284756a38719dffe6
# Parent  b9f7b0c10027764cee77f9c6d61877fcffea837f
exchange: add `_getbookmarks()` function

This function will be used to generate bookmarks bundle2 part.
It is a separate function in order to make it easy to overwrite it
in extensions. Passing `kwargs` to the function makes it easy to
add new parameters in extensions.

diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -1666,6 +1666,17 @@
 if chunks:
 bundler.newpart('hgtagsfnodes', data=''.join(chunks))
 
+def _getbookmarks(repo, **kwargs):
+"""Returns list of bookmarks.
+
+This function is primarily used to generate `bookmarks` bundle2 part.
+It is a separate function in order to make it easy to wrap it
+in extensions. Passing `kwargs` to the function makes it easy to
+add new parameters in extensions.
+"""
+
+return bookmod.listbookmarks(repo)
+
 def check_heads(repo, their_heads, context):
 """check if the heads of a repo have been modified
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 7 of 8 V7] bundle2: advertise bookmark capability

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1478088362 25200
#  Wed Nov 02 05:06:02 2016 -0700
# Branch stable
# Node ID c2a9f675fac55522bb954ea81c29fa0158106214
# Parent  e2122d93aeb4da4a39c0c257e74414ac348a89f1
bundle2: advertise bookmark capability

diff --git a/mercurial/bundle2.py b/mercurial/bundle2.py
--- a/mercurial/bundle2.py
+++ b/mercurial/bundle2.py
@@ -1272,6 +1272,7 @@
 'digests': tuple(sorted(util.DIGESTS.keys())),
 'remote-changegroup': ('http', 'https'),
 'hgtagsfnodes': (),
+'bookmarks': (),
}
 
 def getrepocaps(repo, allowpushback=False):
diff --git a/tests/test-acl.t b/tests/test-acl.t
--- a/tests/test-acl.t
+++ b/tests/test-acl.t
@@ -92,13 +92,13 @@
   f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd
   911600dab2ae7a9baff75958b84fe606851ce955
   bundle2-output-bundle: "HG20", 4 parts total
-  bundle2-output-part: "replycaps" 155 bytes payload
+  bundle2-output-part: "replycaps" 165 bytes payload
   bundle2-output-part: "check:heads" streamed payload
   bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload
   bundle2-output-part: "pushkey" (params: 4 mandatory) empty payload
   bundle2-input-bundle: with-transaction
   bundle2-input-part: "replycaps" supported
-  bundle2-input-part: total payload size 155
+  bundle2-input-part: total payload size 165
   bundle2-input-part: "check:heads" supported
   bundle2-input-part: total payload size 20
   bundle2-input-part: "changegroup" (params: 1 mandatory) supported
@@ -155,13 +155,13 @@
   f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd
   911600dab2ae7a9baff75958b84fe606851ce955
   bundle2-output-bundle: "HG20", 4 parts total
-  bundle2-output-part: "replycaps" 155 bytes payload
+  bundle2-output-part: "replycaps" 165 bytes payload
   bundle2-output-part: "check:heads" streamed payload
   bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload
   bundle2-output-part: "pushkey" (params: 4 mandatory) empty payload
   bundle2-input-bundle: with-transaction
   bundle2-input-part: "replycaps" supported
-  bundle2-input-part: total payload size 155
+  bundle2-input-part: total payload size 165
   bundle2-input-part: "check:heads" supported
   bundle2-input-part: total payload size 20
   bundle2-input-part: "changegroup" (params: 1 mandatory) supported
@@ -221,13 +221,13 @@
   f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd
   911600dab2ae7a9baff75958b84fe606851ce955
   bundle2-output-bundle: "HG20", 4 parts total
-  bundle2-output-part: "replycaps" 155 bytes payload
+  bundle2-output-part: "replycaps" 165 bytes payload
   bundle2-output-part: "check:heads" streamed payload
   bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload
   bundle2-output-part: "pushkey" (params: 4 mandatory) empty payload
   bundle2-input-bundle: with-transaction
   bundle2-input-part: "replycaps" supported
-  bundle2-input-part: total payload size 155
+  bundle2-input-part: total payload size 165
   bundle2-input-part: "check:heads" supported
   bundle2-input-part: total payload size 20
   bundle2-input-part: "changegroup" (params: 1 mandatory) supported
@@ -297,13 +297,13 @@
   f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd
   911600dab2ae7a9baff75958b84fe606851ce955
   bundle2-output-bundle: "HG20", 4 parts total
-  bundle2-output-part: "replycaps" 155 bytes payload
+  bundle2-output-part: "replycaps" 165 bytes payload
   bundle2-output-part: "check:heads" streamed payload
   bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload
   bundle2-output-part: "pushkey" (params: 4 mandatory) empty payload
   bundle2-input-bundle: with-transaction
   bundle2-input-part: "replycaps" supported
-  bundle2-input-part: total payload size 155
+  bundle2-input-part: total payload size 165
   bundle2-input-part: "check:heads" supported
   bundle2-input-part: total payload size 20
   bundle2-input-part: "changegroup" (params: 1 mandatory) supported
@@ -362,13 +362,13 @@
   f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd
   911600dab2ae7a9baff75958b84fe606851ce955
   bundle2-output-bundle: "HG20", 4 parts total
-  bundle2-output-part: "replycaps" 155 bytes payload
+  bundle2-output-part: "replycaps" 165 bytes payload
   bundle2-output-part: "check:heads" streamed payload
   bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload
   bundle2-output-part: "pushkey" (params: 4 mandatory) empty payload
   bundle2-input-bundle: with-transaction
   bundle2-input-part: "replycaps" supported
-  bundle

[PATCH 6 of 8 V7] pull: use `bookmarks` bundle2 part

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1478086459 25200
#  Wed Nov 02 04:34:19 2016 -0700
# Branch stable
# Node ID e2122d93aeb4da4a39c0c257e74414ac348a89f1
# Parent  12466c729bb6783fbef11a6e148c648809b4f592
pull: use `bookmarks` bundle2 part

Apply changes from `bookmarks` part.

diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -1322,9 +1322,13 @@
 kwargs['cg'] = pullop.fetch
 if 'listkeys' in pullop.remotebundle2caps:
 kwargs['listkeys'] = ['phases']
-if pullop.remotebookmarks is None:
-# make sure to always includes bookmark data when migrating
-# `hg incoming --bundle` to using this function.
+
+if pullop.remotebookmarks is None:
+# make sure to always includes bookmark data when migrating
+# `hg incoming --bundle` to using this function.
+if 'bookmarks' in pullop.remotebundle2caps:
+kwargs['bookmarks'] = True
+elif 'listkeys' in pullop.remotebundle2caps:
 kwargs['listkeys'].append('bookmarks')
 
 # If this is a full pull / clone and the server supports the clone bundles
@@ -1352,10 +1356,23 @@
 _pullbundle2extraprepare(pullop, kwargs)
 bundle = pullop.remote.getbundle('pull', **kwargs)
 try:
-op = bundle2.processbundle(pullop.repo, bundle, pullop.gettransaction)
+bundleopinput = {
+'processbookmarksmode': 'diverge',
+'explicitbookmarks': pullop.explicitbookmarks,
+'remotepath': pullop.remote.url(),
+}
+op = bundle2.bundleoperation(pullop.repo, pullop.gettransaction,
+ input=bundleopinput)
+op = bundle2.processbundle(pullop.repo, bundle,
+   pullop.gettransaction, op=op)
 except error.BundleValueError as exc:
 raise error.Abort(_('missing support for %s') % exc)
 
+# `bookmarks` part was in bundle and it was applied to the repo. No need to
+# apply bookmarks one more time
+if 'bookmarks' in kwargs and kwargs['bookmarks']:
+pullop.stepsdone.add('bookmarks')
+
 if pullop.fetch:
 results = [cg['return'] for cg in op.records['changegroup']]
 pullop.cgresult = changegroup.combineresults(results)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 5 of 8 V7] exchange: getbundle `bookmarks` part generator

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1478016027 25200
#  Tue Nov 01 09:00:27 2016 -0700
# Branch stable
# Node ID 12466c729bb6783fbef11a6e148c648809b4f592
# Parent  2f89680108366512c1a6345abec5cebdb85170ac
exchange: getbundle `bookmarks` part generator

This generator will be used during pull operation.

diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -1666,6 +1666,21 @@
 if chunks:
 bundler.newpart('hgtagsfnodes', data=''.join(chunks))
 
+@getbundle2partsgenerator('bookmarks')
+def _getbundlebookmarkspart(bundler, repo, source, bundlecaps=None,
+b2caps=None, heads=None, common=None,
+**kwargs):
+if not kwargs.get('bookmarks'):
+return
+if 'bookmarks' not in b2caps:
+raise ValueError(
+_('bookmarks are requested but client is not capable '
+  'of receiving it'))
+
+bookmarks = _getbookmarks(repo, **kwargs)
+encodedbookmarks = bookmod.encodebookmarks(bookmarks)
+bundler.newpart('bookmarks', data=encodedbookmarks)
+
 def _getbookmarks(repo, **kwargs):
 """Returns list of bookmarks.
 
diff --git a/mercurial/wireproto.py b/mercurial/wireproto.py
--- a/mercurial/wireproto.py
+++ b/mercurial/wireproto.py
@@ -228,7 +228,9 @@
  'bundlecaps': 'scsv',
  'listkeys': 'csv',
  'cg': 'boolean',
- 'cbattempted': 'boolean'}
+ 'cbattempted': 'boolean',
+ 'bookmarks': 'boolean',
+}
 
 # client side
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 8 V7] bookmarks: introduce binary encoding

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1478016027 25200
#  Tue Nov 01 09:00:27 2016 -0700
# Branch stable
# Node ID f3da841e3d47ccbb0be3892b521607c09fdeab13
# Parent  a56a624a8a42b93ec980d2c284756a38719dffe6
bookmarks: introduce binary encoding

Bookmarks binary encoding will be used for `bookmarks` bundle2 part.
The format is: <4 bytes - bookmark size, big-endian>
   <1 byte - 0 if node is empty 1 otherwise><20 bytes node>
BookmarksEncodeError and BookmarksDecodeError maybe thrown if input is
incorrect.

diff --git a/mercurial/bookmarks.py b/mercurial/bookmarks.py
--- a/mercurial/bookmarks.py
+++ b/mercurial/bookmarks.py
@@ -7,8 +7,10 @@
 
 from __future__ import absolute_import
 
+import StringIO
 import errno
 import os
+import struct
 
 from .i18n import _
 from .node import (
@@ -23,6 +25,70 @@
 util,
 )
 
+_NONEMPTYNODE = struct.pack('?', False)
+_EMPTYNODE = struct.pack('?', True)
+
+def _unpackbookmarksize(stream):
+"""Returns 0 if stream is empty.
+"""
+
+expectedsize = struct.calcsize('>i')
+encodedbookmarksize = stream.read(expectedsize)
+if len(encodedbookmarksize) == 0:
+return 0
+if len(encodedbookmarksize) != expectedsize:
+raise ValueError(
+_('cannot decode bookmark size: '
+  'expected size: %d, actual size: %d') %
+(expectedsize, len(encodedbookmarksize)))
+return struct.unpack('>i', encodedbookmarksize)[0]
+
+def encodebookmarks(bookmarks):
+"""Encodes bookmark to node mapping to the byte string.
+
+Format: <4 bytes - bookmark size>
+<1 byte - 0 if node is empty 1 otherwise><20 bytes node>
+Node may be 20 byte binary string, 40 byte hex string or empty
+"""
+
+for bookmark, node in bookmarks.iteritems():
+yield struct.pack('>i', (len(bookmark)))
+yield encoding.fromlocal(bookmark)
+if node:
+if len(node) != 20 and len(node) != 40:
+raise ValueError(_('node must be 20 or bytes long'))
+if len(node) == 40:
+node = bin(node)
+yield _NONEMPTYNODE
+yield node
+else:
+yield _EMPTYNODE
+
+def decodebookmarks(buf):
+"""Decodes buffer into bookmark to node mapping.
+
+Node is either 20 bytes or empty.
+"""
+
+stream = StringIO.StringIO(buf)
+bookmarks = {}
+bookmarksize = _unpackbookmarksize(stream)
+while bookmarksize:
+bookmark = stream.read(bookmarksize)
+if len(bookmark) != bookmarksize:
+raise ValueError(
+_('cannot decode bookmark: expected size: %d, '
+'actual size: %d') % (bookmarksize, len(bookmark)))
+bookmark = encoding.tolocal(bookmark)
+packedemptynodeflag = stream.read(struct.calcsize('?'))
+emptynode = struct.unpack('?', packedemptynodeflag)[0]
+node = ''
+if not emptynode:
+node = stream.read(20)
+bookmarks[bookmark] = node
+bookmarksize = _unpackbookmarksize(stream)
+return bookmarks
+
 def _getbkfile(repo):
 """Hook so that extensions that mess with the store can hook bm storage.
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 3 of 8 V7] bookmarks: add srchex param to updatefromremote

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1478016027 25200
#  Tue Nov 01 09:00:27 2016 -0700
# Branch stable
# Node ID d752daa2b736f0336a18760940104c59d6bd1a5c
# Parent  f3da841e3d47ccbb0be3892b521607c09fdeab13
bookmarks: add srchex param to updatefromremote

diff --git a/mercurial/bookmarks.py b/mercurial/bookmarks.py
--- a/mercurial/bookmarks.py
+++ b/mercurial/bookmarks.py
@@ -501,11 +501,12 @@
 
 return None
 
-def updatefromremote(ui, repo, remotemarks, path, trfunc, explicit=()):
+def updatefromremote(ui, repo, remotemarks, path, trfunc, explicit=(),
+ srchex=None):
 ui.debug("checking for updated bookmarks\n")
 localmarks = repo._bookmarks
 (addsrc, adddst, advsrc, advdst, diverge, differ, invalid, same
- ) = compare(repo, remotemarks, localmarks, dsthex=hex)
+ ) = compare(repo, remotemarks, localmarks, srchex=srchex, dsthex=hex)
 
 status = ui.status
 warn = ui.warn
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 4 of 8 V7] bundle2: add `bookmarks` part handler

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1478016027 25200
#  Tue Nov 01 09:00:27 2016 -0700
# Branch stable
# Node ID 2f89680108366512c1a6345abec5cebdb85170ac
# Parent  d752daa2b736f0336a18760940104c59d6bd1a5c
bundle2: add `bookmarks` part handler

Applies bookmarks part to the local repo. `processbookmarksmode` determines
how remote bookmarks are handled. They are either ignored ('ignore' mode),
diverged ('diverge' mode) or applied ('apply' mode).

diff --git a/mercurial/bundle2.py b/mercurial/bundle2.py
--- a/mercurial/bundle2.py
+++ b/mercurial/bundle2.py
@@ -154,7 +154,9 @@
 import sys
 
 from .i18n import _
+from .node import hex
 from . import (
+bookmarks as bookmod,
 changegroup,
 error,
 obsolete,
@@ -287,13 +289,21 @@
 * a way to construct a bundle response when applicable.
 """
 
-def __init__(self, repo, transactiongetter, captureoutput=True):
+def __init__(self, repo, transactiongetter, captureoutput=True,
+ input=None):
+"""
+`input` is a dictionary that is passed to part handlers to tweak
+their behaviour
+"""
 self.repo = repo
 self.ui = repo.ui
 self.records = unbundlerecords()
 self.gettransaction = transactiongetter
 self.reply = None
 self.captureoutput = captureoutput
+if input is None:
+input = {}
+self.input = input
 
 class TransactionUnavailable(RuntimeError):
 pass
@@ -1624,3 +1634,33 @@
 
 cache.write()
 op.ui.debug('applied %i hgtags fnodes cache entries\n' % count)
+
+@parthandler('bookmarks')
+def handlebookmarks(op, inpart):
+"""Processes bookmarks part.
+
+`processbookmarksmode` determines how remote bookmarks are handled. They 
are
+either ignored ('ignore' mode), diverged ('diverge' mode) or applied
+('apply' mode). 'ignore' mode is used to get bookmarks and process them
+later, 'diverge' mode is used to process bookmarks during pull, 'apply'
+mode is used during push.
+"""
+
+bookmarks = {}
+bookmarks = bookmod.decodebookmarks(inpart.read())
+processbookmarksmode = op.input.get('processbookmarksmode', 'ignore')
+if processbookmarksmode == 'apply':
+for bookmark, node in bookmarks.items():
+if node:
+op.repo._bookmarks[bookmark] = node
+else:
+del op.repo._bookmarks[bookmark]
+op.repo._bookmarks.recordchange(op.gettransaction())
+elif processbookmarksmode == 'diverge':
+remotepath = op.input.get('remotepath', '')
+explicitbookmarks = op.input.get('explicitbookmarks', ())
+bookmod.updatefromremote(op.ui, op.repo, bookmarks,
+ remotepath, op.gettransaction,
+ explicit=explicitbookmarks,
+ srchex=hex)
+op.records.add('bookmarks', bookmarks)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 5 of 9 V6] bookmarks: add srchex param to updatefromremote

[Stanislau Hlebik]

I’m using srchex in bookmarks part handler to convert from bin nodes to hex 
nodes, because updatefromremote expect hex nodes.

On 10/14/16, 2:28 AM, "Pierre-Yves David" <pierre-yves.da...@ens-lyon.org> 
wrote:



On 10/11/2016 06:25 PM, Stanislau Hlebik wrote:
> # HG changeset patch
    > # User Stanislau Hlebik <st...@fb.com>
> # Date 1476197429 25200
> #  Tue Oct 11 07:50:29 2016 -0700
> # Node ID f781756b8de11a6f3e7dd5fd6354e9778defd8c3
> # Parent  718ed86a3698631077a087efaf668d70513056f5
> bookmarks: add srchex param to updatefromremote

I do not understand what the purpose and effect of this changeset is. 
Can you elaborate a little ?

>
> diff --git a/mercurial/bookmarks.py b/mercurial/bookmarks.py
> --- a/mercurial/bookmarks.py
> +++ b/mercurial/bookmarks.py
> @@ -508,11 +508,12 @@
>
>  return None
>
> -def updatefromremote(ui, repo, remotemarks, path, trfunc, explicit=()):
> +def updatefromremote(ui, repo, remotemarks, path, trfunc, explicit=(),
> + srchex=None):
>  ui.debug("checking for updated bookmarks\n")
>  localmarks = repo._bookmarks
>  (addsrc, adddst, advsrc, advdst, diverge, differ, invalid, same
> - ) = compare(repo, remotemarks, localmarks, dsthex=hex)
> + ) = compare(repo, remotemarks, localmarks, srchex=srchex, 
dsthex=hex)
>
>  status = ui.status
>  warn = ui.warn
> ___
> Mercurial-devel mailing list
> Mercurial-devel@mercurial-scm.org
> 
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.mercurial-2Dscm.org_mailman_listinfo_mercurial-2Ddevel=DQICaQ=5VD0RTtNlTh3ycd41b3MUw=1EQ58Dmb5uX1qHujcsT1Mg=8-TOccCZw4oFT0da5cQ-wA28AAvE6qKjZzPGBL8t-1k=V5DeajSgRhOwuad7CFu_QgJzUJxWLtqkhpa1BOyyA-k=
 
>

-- 
Pierre-Yves David


___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 7 of 9 V6] exchange: getbundle `bookmarks` part generator

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1476195835 25200
#  Tue Oct 11 07:23:55 2016 -0700
# Node ID 48e5db9c751c1eca19019bbc24de43f7cb3368e7
# Parent  b9e71247c1b68ce1fac7dd69cf26d106f88f9372
exchange: getbundle `bookmarks` part generator

This generator will be used during pull operation.

diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -1672,6 +1672,21 @@
 if chunks:
 bundler.newpart('hgtagsfnodes', data=''.join(chunks))
 
+@getbundle2partsgenerator('bookmarks')
+def _getbundlebookmarkspart(bundler, repo, source, bundlecaps=None,
+b2caps=None, heads=None, common=None,
+**kwargs):
+if not kwargs.get('bookmarks'):
+return
+if 'bookmarks' not in b2caps:
+raise ValueError(
+_('bookmarks are requested but client is not capable '
+  'of receiving it'))
+
+bookmarks = _getbookmarks(repo, **kwargs)
+encodedbookmarks = bookmod.encodebookmarks(bookmarks)
+bundler.newpart('bookmarks', data=encodedbookmarks)
+
 def _getbookmarks(repo, **kwargs):
 """Returns list of bookmarks.
 
diff --git a/mercurial/wireproto.py b/mercurial/wireproto.py
--- a/mercurial/wireproto.py
+++ b/mercurial/wireproto.py
@@ -220,7 +220,8 @@
  'bundlecaps': 'scsv',
  'listkeys': 'csv',
  'cg': 'boolean',
- 'cbattempted': 'boolean'}
+ 'cbattempted': 'boolean',
+ 'bookmarks': 'boolean'}
 
 # client side
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 9 V6] bookmarks: introduce BookmarksEncodeError

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1476195835 25200
#  Tue Oct 11 07:23:55 2016 -0700
# Node ID 91ae660eb2faf74cb96ee7ee5bdcc1f2507f9cf7
# Parent  55e997127023d7208488c593adb933a1bfb23312
bookmarks: introduce BookmarksEncodeError

Binary encoding for bookmarks will be added in next diffs.
This error will be thrown if there are problems with bookmarks encoding.

diff --git a/mercurial/error.py b/mercurial/error.py
--- a/mercurial/error.py
+++ b/mercurial/error.py
@@ -243,3 +243,7 @@
 
 class CorruptedState(Exception):
 """error raised when a command is not able to read its state from file"""
+
+class BookmarksEncodeError(Exception):
+"""Raised if bookmarks can't be encoded
+"""
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 6 of 9 V6] bundle2: add `bookmarks` part handler

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1476197535 25200
#  Tue Oct 11 07:52:15 2016 -0700
# Node ID b9e71247c1b68ce1fac7dd69cf26d106f88f9372
# Parent  f781756b8de11a6f3e7dd5fd6354e9778defd8c3
bundle2: add `bookmarks` part handler

Applies bookmarks part to the local repo. `processbookmarksmode` determines
how remote bookmarks are handled. They are either ignored ('ignore' mode),
diverged ('diverge' mode) or applied ('apply' mode).

diff --git a/mercurial/bundle2.py b/mercurial/bundle2.py
--- a/mercurial/bundle2.py
+++ b/mercurial/bundle2.py
@@ -154,7 +154,9 @@
 import sys
 
 from .i18n import _
+from .node import hex
 from . import (
+bookmarks as bookmod,
 changegroup,
 error,
 obsolete,
@@ -287,13 +289,18 @@
 * a way to construct a bundle response when applicable.
 """
 
-def __init__(self, repo, transactiongetter, captureoutput=True):
+def __init__(self, repo, transactiongetter, captureoutput=True,
+ processbookmarksmode='ignore', explicitbookmarks=(),
+ remotepath=''):
 self.repo = repo
 self.ui = repo.ui
 self.records = unbundlerecords()
 self.gettransaction = transactiongetter
 self.reply = None
 self.captureoutput = captureoutput
+self.processbookmarksmode = processbookmarksmode
+self.explicitbookmarks = explicitbookmarks
+self.remotepath = remotepath
 
 class TransactionUnavailable(RuntimeError):
 pass
@@ -1620,3 +1627,28 @@
 
 cache.write()
 op.ui.debug('applied %i hgtags fnodes cache entries\n' % count)
+
+@parthandler('bookmarks')
+def handlebookmarks(op, inpart):
+"""Applies bookmarks to the local repo.
+
+`processbookmarksmode` determines how remote bookmarks are handled. They 
are
+either ignored ('ignore' mode), diverged ('diverge' mode) or applied
+('apply' mode).
+"""
+
+bookmarks = {}
+bookmarks = bookmod.decodebookmarks(inpart.read())
+if op.processbookmarksmode == 'apply':
+for bookmark, node in bookmarks.items():
+if node:
+op.repo._bookmarks[bookmark] = node
+else:
+del op.repo._bookmarks[bookmark]
+op.repo._bookmarks.recordchange(op.gettransaction())
+elif op.processbookmarksmode == 'diverge':
+bookmod.updatefromremote(op.ui, op.repo, bookmarks,
+ op.remotepath, op.gettransaction,
+ explicit=op.explicitbookmarks,
+ srchex=hex)
+op.records.add('bookmarks', bookmarks)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 5 of 9 V6] bookmarks: add srchex param to updatefromremote

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1476197429 25200
#  Tue Oct 11 07:50:29 2016 -0700
# Node ID f781756b8de11a6f3e7dd5fd6354e9778defd8c3
# Parent  718ed86a3698631077a087efaf668d70513056f5
bookmarks: add srchex param to updatefromremote

diff --git a/mercurial/bookmarks.py b/mercurial/bookmarks.py
--- a/mercurial/bookmarks.py
+++ b/mercurial/bookmarks.py
@@ -508,11 +508,12 @@
 
 return None
 
-def updatefromremote(ui, repo, remotemarks, path, trfunc, explicit=()):
+def updatefromremote(ui, repo, remotemarks, path, trfunc, explicit=(),
+ srchex=None):
 ui.debug("checking for updated bookmarks\n")
 localmarks = repo._bookmarks
 (addsrc, adddst, advsrc, advdst, diverge, differ, invalid, same
- ) = compare(repo, remotemarks, localmarks, dsthex=hex)
+ ) = compare(repo, remotemarks, localmarks, srchex=srchex, dsthex=hex)
 
 status = ui.status
 warn = ui.warn
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 4 of 9 V6] bookmarks: introduce binary encoding

[Stanislau Hlebik]

# HG changeset patch
# User Stanislau Hlebik <st...@fb.com>
# Date 1476195835 25200
#  Tue Oct 11 07:23:55 2016 -0700
# Node ID 718ed86a3698631077a087efaf668d70513056f5
# Parent  6f5a3300a5457c92eb09170a30c98328ebe3bcce
bookmarks: introduce binary encoding

Bookmarks binary encoding will be used for `bookmarks` bundle2 part.
The format is: <4 bytes - bookmark size, big-endian>
   <1 byte - 0 if node is empty 1 otherwise><20 bytes node>
BookmarksEncodeError and BookmarksDecodeError maybe thrown if input is
incorrect.

diff --git a/mercurial/bookmarks.py b/mercurial/bookmarks.py
--- a/mercurial/bookmarks.py
+++ b/mercurial/bookmarks.py
@@ -7,8 +7,10 @@
 
 from __future__ import absolute_import
 
+import StringIO
 import errno
 import os
+import struct
 
 from .i18n import _
 from .node import (
@@ -23,6 +25,77 @@
 util,
 )
 
+_NONEMPTYNODE = chr(1)
+_EMPTYNODE = chr(0)
+
+def _packbookmarksize(size):
+return struct.pack('>i', size)
+
+def _unpackbookmarksize(stream):
+"""Returns 0 if stream is empty.
+"""
+
+expectedsize = struct.calcsize('>i')
+encodedbookmarksize = stream.read(expectedsize)
+if len(encodedbookmarksize) == 0:
+return 0
+if len(encodedbookmarksize) != expectedsize:
+raise error.BookmarksDecodeError(
+_('cannot decode bookmark size: '
+  'expected size: %d, actual size: %d') %
+(expectedsize, len(encodedbookmarksize)))
+return struct.unpack('>i', encodedbookmarksize)[0]
+
+def encodebookmarks(bookmarks):
+"""Encodes bookmark to node mapping to the byte string.
+
+Format: <4 bytes - bookmark size, big-endian>
+<1 byte - 0 if node is empty 1 otherwise><20 bytes node>
+Node may be 20 byte binary string, 40 byte hex string or empty
+"""
+
+parts = []
+for bookmark, node in bookmarks.iteritems():
+encodedbookmarksize = _packbookmarksize(len(bookmark))
+parts.append(encodedbookmarksize)
+bookmark = encoding.fromlocal(bookmark)
+parts.append(bookmark)
+if node:
+if len(node) != 20 and len(node) != 40:
+raise error.BookmarksEncodeError()
+if len(node) == 40:
+node = bin(node)
+parts.append(_NONEMPTYNODE)
+parts.append(node)
+else:
+parts.append(_EMPTYNODE)
+return ''.join(parts)
+
+def decodebookmarks(buf):
+"""Decodes buffer into bookmark to node mapping.
+
+Node is either 20 bytes or empty.
+"""
+
+stream = StringIO.StringIO(buf)
+bookmarks = {}
+while True:
+bookmarksize = _unpackbookmarksize(stream)
+if not bookmarksize:
+break
+bookmark = stream.read(bookmarksize)
+if len(bookmark) != bookmarksize:
+raise error.BookmarksDecodeError(
+'cannot decode bookmark: expected size: %d, '
+'actual size: %d' % (bookmarksize, len(bookmark)))
+bookmark = encoding.tolocal(bookmark)
+emptynode = stream.read(1)
+node = ''
+if emptynode != _EMPTYNODE:
+node = stream.read(20)
+bookmarks[bookmark] = node
+return bookmarks
+
 def _getbkfile(repo):
 """Hook so that extensions that mess with the store can hook bm storage.
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel