[PATCH] lfs: rename {lfsattrs} to {pointer}

[Matt Harbison]

# HG changeset patch
# User Matt Harbison 
# Date 1516407650 18000
#  Fri Jan 19 19:20:50 2018 -0500
# Node ID 9179400df335ad78c59f165cea811471905ec51a
# Parent  069df0b952e803b3efb3c59c423522a4800ecde4
lfs: rename {lfsattrs} to {pointer}

This seems more descriptive.

diff --git a/hgext/lfs/__init__.py b/hgext/lfs/__init__.py
--- a/hgext/lfs/__init__.py
+++ b/hgext/lfs/__init__.py
@@ -315,7 +315,7 @@
 pointers = wrapper.pointersfromctx(ctx) # {path: pointer}
 files = sorted(pointers.keys())
 
-def lfsattrs(v):
+def pointer(v):
 # In the file spec, version is first and the other keys are sorted.
 sortkeyfunc = lambda x: (x[0] != 'version', x)
 items = sorted(pointers[v].iteritems(), key=sortkeyfunc)
@@ -324,7 +324,7 @@
 makemap = lambda v: {
 'file': v,
 'oid': pointers[v].oid(),
-'lfsattrs': templatekw.hybriddict(lfsattrs(v)),
+'pointer': templatekw.hybriddict(pointer(v)),
 }
 
 # TODO: make the separator ', '?
diff --git a/tests/test-lfs.t b/tests/test-lfs.t
--- a/tests/test-lfs.t
+++ b/tests/test-lfs.t
@@ -865,16 +865,16 @@
   oid sha256:5bb8341bee63b3649f222b2215bde37322bea075a30575aa685d8f8d21c77024
   size 29
   x-is-binary 0
-  $ hg --cwd convert_lfs log -r 0 -T "{lfs_files % '{lfsattrs % 
'{key}={value}\n'}'}"
+  $ hg --cwd convert_lfs log -r 0 -T "{lfs_files % '{pointer % 
'{key}={value}\n'}'}"
   version=https://git-lfs.github.com/spec/v1
   oid=sha256:5bb8341bee63b3649f222b2215bde37322bea075a30575aa685d8f8d21c77024
   size=29
   x-is-binary=0
   $ hg --cwd convert_lfs log -r 0 \
-  >-T '{lfs_files % "{get(lfsattrs, "oid")}\n"}{lfs_files % 
"{lfsattrs.oid}\n"}'
+  >-T '{lfs_files % "{get(pointer, "oid")}\n"}{lfs_files % 
"{pointer.oid}\n"}'
   sha256:5bb8341bee63b3649f222b2215bde37322bea075a30575aa685d8f8d21c77024
   sha256:5bb8341bee63b3649f222b2215bde37322bea075a30575aa685d8f8d21c77024
-  $ hg --cwd convert_lfs log -r 0 -T '{lfs_files % "{lfsattrs}\n"}'
+  $ hg --cwd convert_lfs log -r 0 -T '{lfs_files % "{pointer}\n"}'
   version=https://git-lfs.github.com/spec/v1 
oid=sha256:5bb8341bee63b3649f222b2215bde37322bea075a30575aa685d8f8d21c77024 
size=29 x-is-binary=0
   $ hg --cwd convert_lfs \
   > log -r 'all()' -T '{rev}: {lfs_files % "{file}: {oid}\n"}'
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 2 of 2] templatekw: add a {negrev} keyword

[Yuya Nishihara]

On Fri, 19 Jan 2018 14:32:34 -0500, Augie Fackler wrote:
> On Wed, Jan 17, 2018 at 10:14:30PM -0500, Jordi Gutiérrez Hermoso wrote:
> > # HG changeset patch
> > # User Jordi Gutiérrez Hermoso 
> > # Date 1516243120 18000
> > #  Wed Jan 17 21:38:40 2018 -0500
> > # Node ID cbf1d676a938e78d40cd3504dd916f787bcb47ee
> > # Parent  701f8a9defdc09bb63f2596e2fc426f2e78da313
> > templatekw: add a {negrev} keyword
> 
> This is a really interesting idea. It mostly has driven me crazy that
> negative revnums work, but this sort of provides a reason for their
> existence I guess.
> 
> That said, I'm too wary of locking this in on the last day before a
> freeze, so let's plan to discuss this after the freeze sometime in
> early February? Maybe set a calendar reminder to rebase && resend this
> patch then.

Can you rename the keyword to something saying that isn't actually a "rev"?
Negative integers aren't always usable where revision number is expected,
e.g. rev(n), and -1 means either null or tip depending on context.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 1 of 3] lfs: always exclude '.hg*' text files

[Jun Wu]

Excerpts from Yuya Nishihara's message of 2018-01-15 21:56:49 +0900:
> Seems fine. Queued, thanks.
> 
> CC-ed Jun since I'm not pretty sure if this is considered a nice feature.

It is considered as a nice feature. For example:

  - If the LFS server hostname expires, it's possible to "revive" dead links
without rewriting hashes.
  - It's possible to convert old normal large files to LFS, to save server's
disk space, without triggering integrity check errors client-side.

IIRC, mpm said lfs has a better design than largefiles. I *guess* the hash
preserving behavior is one of the things that make it better.

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH V2] lfs: add the '{lfsattrs}' template keyword to '{lfs_files}'

[Yuya Nishihara]

On Sat, 20 Jan 2018 01:34:49 -0500, Matt Harbison wrote:
> On Fri, 19 Jan 2018 07:21:30 -0500, Yuya Nishihara  wrote:
> 
> > On Fri, 19 Jan 2018 00:05:42 -0500, Matt Harbison wrote:
> >> # HG changeset patch
> >> # User Matt Harbison 
> >> # Date 1515967224 18000
> >> #  Sun Jan 14 17:00:24 2018 -0500
> >> # Node ID fccf09e44f5124abf18ae898fab553ea6d91e951
> >> # Parent  45b678bf3a787085d56fad5bee494e0c160aa120
> >> lfs: add the '{lfsattrs}' template keyword to '{lfs_files}'
> >
> > Queued updated version, thanks.
> >
> >> I liked {pointer} better, but couldn't make it work with the  
> >> singular/plural
> >> forms.
> >
> > I think {pointer} is okay here since its singular form is ({key},  
> > {value}).
> 
> OK, I'll rename after the freeze.

I think it's okay to rename that before cutting rc. Can you send a patch?

> >> @@ -303,6 +304,8 @@
> >>  # when writing a bundle via "hg bundle" command, upload related  
> >> LFS blobs
> >>  wrapfunction(bundle2, 'writenewbundle', wrapper.writenewbundle)
> >>
> >> +templatekw.defaulttempl['lfsattr'] = '{key}={value}'
> >
> > This isn't needed. Dropped.
> 
> Does that mean the envvar entry isn't needed?  Somehow, the output I was  
> getting was all of the keys run together, and that made me think of  
> envvar.  (Unfortunately, I didn't commit the code, so I can't go back to  
> see what exactly was wrong.)

showdict|list() requires defaulttempl entry to support old-style list
templates. hybriddict|list() doesn't.

> I wonder if this strategy is wrong in general.  Regular templates have  
> {files}, {file_adds}, {file_copies}, {file_copies_switch}, {file_dels},  
> {file_mods}, and {files(PATTERN)}.  It would be silly to copy all of  
> that.  But filters seem only text oriented.  Could a hypothetical  
> {lfs_files()}
> filter all of these to lfs only,

A unary template function can acts as a filter.

> and tack {pointer} and {oid} on to each
> entry somehow?

Alternatively, we could add pointer and oid to files entry only when it
is backed by lfs.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] lfs: expand the user facing documentation

[Yuya Nishihara]

On Fri, 19 Jan 2018 21:36:50 -0500, Matt Harbison wrote:
> # HG changeset patch
> # User Matt Harbison 
> # Date 1516415371 18000
> #  Fri Jan 19 21:29:31 2018 -0500
> # Node ID c54242f095daa75f24ba30fc91990a6c93f786ec
> # Parent  9179400df335ad78c59f165cea811471905ec51a
> lfs: expand the user facing documentation

Queued, thanks.

> -The extension reads its configuration from a versioned ``.hglfs``
> -configuration file found in the root of the working directory. The
> -``.hglfs`` file uses the same syntax as all other Mercurial
> -configuration files. It uses a single section, ``[track]``.
> +This extension allows large files to be tracked outside of the normal
> +repository storage and stored on a centralized server, similar to the
> +``largefiles`` extension.  The ``git-lfs`` protocol is used when
> +communicating with the server, so existing git infrastructure can be
> +harnessed.  Even though the files are stored ouside of the repository,

s/ouside/outside/.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH V2] lfs: add the '{lfsattrs}' template keyword to '{lfs_files}'

[Matt Harbison]

On Fri, 19 Jan 2018 07:21:30 -0500, Yuya Nishihara  wrote:


On Fri, 19 Jan 2018 00:05:42 -0500, Matt Harbison wrote:

# HG changeset patch
# User Matt Harbison 
# Date 1515967224 18000
#  Sun Jan 14 17:00:24 2018 -0500
# Node ID fccf09e44f5124abf18ae898fab553ea6d91e951
# Parent  45b678bf3a787085d56fad5bee494e0c160aa120
lfs: add the '{lfsattrs}' template keyword to '{lfs_files}'


Queued updated version, thanks.

I liked {pointer} better, but couldn't make it work with the  
singular/plural

forms.


I think {pointer} is okay here since its singular form is ({key},  
{value}).


OK, I'll rename after the freeze.


@@ -303,6 +304,8 @@
 # when writing a bundle via "hg bundle" command, upload related  
LFS blobs

 wrapfunction(bundle2, 'writenewbundle', wrapper.writenewbundle)

+templatekw.defaulttempl['lfsattr'] = '{key}={value}'


This isn't needed. Dropped.


Does that mean the envvar entry isn't needed?  Somehow, the output I was  
getting was all of the keys run together, and that made me think of  
envvar.  (Unfortunately, I didn't commit the code, so I can't go back to  
see what exactly was wrong.)


I wonder if this strategy is wrong in general.  Regular templates have  
{files}, {file_adds}, {file_copies}, {file_copies_switch}, {file_dels},  
{file_mods}, and {files(PATTERN)}.  It would be silly to copy all of  
that.  But filters seem only text oriented.  Could a hypothetical  
{lfs_files()}
filter all of these to lfs only, and tack {pointer} and {oid} on to each  
entry somehow?

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

mercurial@35752: new changeset

[Mercurial Commits]

New changeset in mercurial:

https://www.mercurial-scm.org/repo/hg/rev/047581ddb6ce
changeset:   35752:047581ddb6ce
bookmark:@
tag: tip
user:Siddharth Agarwal 
date:Fri Jan 19 14:25:09 2018 -0800
summary: sshserver: add a couple of tests for argument parsing

-- 
Repository URL: https://www.mercurial-scm.org/repo/hg
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH] lfs: expand the user facing documentation

[Matt Harbison]

# HG changeset patch
# User Matt Harbison 
# Date 1516415371 18000
#  Fri Jan 19 21:29:31 2018 -0500
# Node ID c54242f095daa75f24ba30fc91990a6c93f786ec
# Parent  9179400df335ad78c59f165cea811471905ec51a
lfs: expand the user facing documentation

diff --git a/hgext/lfs/__init__.py b/hgext/lfs/__init__.py
--- a/hgext/lfs/__init__.py
+++ b/hgext/lfs/__init__.py
@@ -7,29 +7,76 @@
 
 """lfs - large file support (EXPERIMENTAL)
 
-The extension reads its configuration from a versioned ``.hglfs``
-configuration file found in the root of the working directory. The
-``.hglfs`` file uses the same syntax as all other Mercurial
-configuration files. It uses a single section, ``[track]``.
+This extension allows large files to be tracked outside of the normal
+repository storage and stored on a centralized server, similar to the
+``largefiles`` extension.  The ``git-lfs`` protocol is used when
+communicating with the server, so existing git infrastructure can be
+harnessed.  Even though the files are stored ouside of the repository,
+they are still integrity checked in the same manner as normal files.
 
-The ``[track]`` section specifies which files are stored as LFS (or
-not). Each line is keyed by a file pattern, with a predicate value.
-The first file pattern match is used, so put more specific patterns
-first.  The available predicates are ``all()``, ``none()``, and
-``size()``. See "hg help filesets.size" for the latter.
+The files stored outside of the repository are downloaded on demand,
+which reduces the time to clone, and possibly the local disk usage.
+This changes fundamental workflows in a DVCS, so careful thought
+should be given before deploying it.  :hg:`convert` can be used to
+convert LFS repositories to normal repositories that no longer
+require this extension, and do so without changing the commit hashes.
+This allows the extension to be disabled if the centralized workflow
+becomes burdensome.  However, the pre and post convert clones will
+not be able to communicate with each other unless the extension is
+enabled on both.
+
+To start a new repository, or add new LFS files, just create and add
+an ``.hglfs`` file as described below.  Because the file is tracked in
+the repository, all clones will use the same selection policy.  During
+subsequent commits, Mercurial will consult this file to determine if
+an added or modified file should be stored externally.  The type of
+storage depends on the characteristics of the file at each commit.  A
+file that is near a size threshold may switch back and forth between
+LFS and normal storage, as needed.
+
+Alternately, both normal repositories and largefile controlled
+repositories can be converted to LFS by using :hg:`convert` and the
+``lfs.track`` config option described below.  The ``.hglfs`` file
+should then be created and added, to control subsequent LFS selection.
+The hashes are also unchanged in this case.  The LFS and non-LFS
+repositories can be distinguished because the LFS repository will
+abort any command if this extension is disabled.
 
-Example versioned ``.hglfs`` file::
+Committed LFS files are held locally, until the repository is pushed.
+Prior to pushing the normal repository data, the LFS files that are
+tracked by the outgoing commits are automatically uploaded to the
+configured central server.  No LFS files are transferred on
+:hg:`pull` or :hg:`clone`.  Instead, the files are downloaded on
+demand as they need to be read, if a cached copy cannot be found
+locally.  Both committing and downloading an LFS file will link the
+file to a usercache, to speed up future access.  See the `usercache`
+config setting described below.
+
+.hglfs::
+
+The extension reads its configuration from a versioned ``.hglfs``
+configuration file found in the root of the working directory. The
+``.hglfs`` file uses the same syntax as all other Mercurial
+configuration files. It uses a single section, ``[track]``.
 
-  [track]
-  # No Makefile or python file, anywhere, will be LFS
-  **Makefile = none()
-  **.py = none()
+The ``[track]`` section specifies which files are stored as LFS (or
+not). Each line is keyed by a file pattern, with a predicate value.
+The first file pattern match is used, so put more specific patterns
+first.  The available predicates are ``all()``, ``none()``, and
+``size()``. See "hg help filesets.size" for the latter.
+
+Example versioned ``.hglfs`` file::
 
-  **.zip = all()
-  **.exe = size(">1MB")
+  [track]
+  # No Makefile or python file, anywhere, will be LFS
+  **Makefile = none()
+  **.py = none()
 
-  # Catchall for everything not matched above
-  ** = size(">10MB")
+  **.zip = all()
+  **.exe = size(">1MB")
+
+  # Catchall for everything not matched above
+  ** = size(">10MB")
 
 Configs::
 
@@ -41,7 +88,7 @@
 #   local filesystem, usually for testing
 # if unset, lfs will prompt setting this when it must use 

Re: [PATCH 14 of 14 V3] streamclone: also stream caches to the client

[Augie Fackler]

On Sat, Jan 20, 2018 at 12:47:19AM +0100, Boris Feld wrote:
> # HG changeset patch
> # User Boris Feld 
> # Date 1516233012 -3600
> #  Thu Jan 18 00:50:12 2018 +0100
> # Node ID d44178c3fd9576e6ca6ab6e92b9823f117141079
> # Parent  9c2889f5050bea9d33be5f501eaf1ecf2d9617d3
> # EXP-Topic b2-stream
> # Available At https://bitbucket.org/octobus/mercurial-devel/
> #  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
> d44178c3fd95
> streamclone: also stream caches to the client

queued with lots of frustration - please budget time better in the
future so that big/important series aren't showing up the day of the
freeze.

> diff --git a/mercurial/streamclone.py b/mercurial/streamclone.py
> --- a/mercurial/streamclone.py
> +++ b/mercurial/streamclone.py

[...]

> @@ -507,10 +527,11 @@ def generatev2(repo):
>  """Emit content for version 2 of a streaming clone.
>
>  the data stream consists the following entries:
> -1) A varint containing the length of the filename
> -2) A varint containing the length of file data
> -3) N bytes containing the filename (the internal, store-agnostic form)
> -4) N bytes containing the file data
> +1) A char representing the file destination (eg: store or cache)

Please send a followup before 4.5 goes final that documents *here*
what these characters are. Don't make people read the code to figure
out the specifics of the format.

> +2) A varint containing the length of the filename
> +3) A varint containing the length of file data
> +4) N bytes containing the filename (the internal, store-agnostic form)
> +5) N bytes containing the file data
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 02 of 14 V3] util: implement varint functions

[Augie Fackler]

On Sat, Jan 20, 2018 at 12:47:07AM +0100, Boris Feld wrote:
> # HG changeset patch
> # User Gregory Szorc 
> # Date 1516398755 -3600
> #  Fri Jan 19 22:52:35 2018 +0100
> # Node ID 5dbc3c53c923b8d11b5efcaf0f415b3d8c8c5180
> # Parent  15f7795f96a5f9acb3ed2e640fcec82f3ccd6f53
> # EXP-Topic b2-stream
> # Available At https://bitbucket.org/octobus/mercurial-devel/
> #  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
> 5dbc3c53c923
> util: implement varint functions
>

also candidates for netutil imo, or maybe protoutil (readexactly also
fits in the same bucket)

> This will be useful in an incoming version-2 of the stream format.
>
> diff --git a/mercurial/util.py b/mercurial/util.py
> --- a/mercurial/util.py
> +++ b/mercurial/util.py
> @@ -3874,3 +3874,73 @@ def readexactly(stream, n):
> " (got %d bytes, expected %d)")
>% (len(s), n))
>  return s
> +
> +def uvarintencode(value):
> +"""Encode an unsigned integer value to a varint.
> +
> +A varint is a variable length integer of 1 or more bytes. Each byte
> +except the last has the most significant bit set. The lower 7 bits of
> +each byte store the 2's complement representation, least significant 
> group
> +first.
> +
> +>>> uvarintencode(0)
> +'\\x00'
> +>>> uvarintencode(1)
> +'\\x01'
> +>>> uvarintencode(127)
> +'\\x7f'
> +>>> uvarintencode(1337)
> +'\\xb9\\n'
> +>>> uvarintencode(65536)
> +'\\x80\\x80\\x04'
> +>>> uvarintencode(-1)
> +Traceback (most recent call last):
> +...
> +ProgrammingError: negative value for uvarint: -1
> +"""
> +if value < 0:
> +raise error.ProgrammingError('negative value for uvarint: %d'
> + % value)
> +bits = value & 0x7f
> +value >>= 7
> +bytes = []
> +while value:
> +bytes.append(pycompat.bytechr(0x80 | bits))
> +bits = value & 0x7f
> +value >>= 7
> +bytes.append(pycompat.bytechr(bits))
> +
> +return ''.join(bytes)
> +
> +def uvarintdecodestream(fh):
> +"""Decode an unsigned variable length integer from a stream.
> +
> +The passed argument is anything that has a ``.read(N)`` method.
> +
> +>>> try:
> +... from StringIO import StringIO as BytesIO
> +... except ImportError:
> +... from io import BytesIO
> +>>> uvarintdecodestream(BytesIO(b'\\x00'))
> +0
> +>>> uvarintdecodestream(BytesIO(b'\\x01'))
> +1
> +>>> uvarintdecodestream(BytesIO(b'\\x7f'))
> +127
> +>>> uvarintdecodestream(BytesIO(b'\\xb9\\n'))
> +1337
> +>>> uvarintdecodestream(BytesIO(b'\\x80\\x80\\x04'))
> +65536
> +>>> uvarintdecodestream(BytesIO(b'\\x80'))
> +Traceback (most recent call last):
> +...
> +Abort: stream ended unexpectedly (got 0 bytes, expected 1)
> +"""
> +result = 0
> +shift = 0
> +while True:
> +byte = ord(readexactly(fh, 1))
> +result |= ((byte & 0x7f) << shift)
> +if not (byte & 0x80):
> +return result
> +shift += 7
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 01 of 14 V3] util: move 'readexactly' in the util module

[Augie Fackler]

On Sat, Jan 20, 2018 at 12:47:06AM +0100, Boris Feld wrote:
> # HG changeset patch
> # User Boris Feld 
> # Date 1516391495 -3600
> #  Fri Jan 19 20:51:35 2018 +0100
> # Node ID 15f7795f96a5f9acb3ed2e640fcec82f3ccd6f53
> # Parent  de32acb24949c0e3633de373d1c6c8c814faa804
> # EXP-Topic b2-stream
> # Available At https://bitbucket.org/octobus/mercurial-devel/
> #  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
> 15f7795f96a5
> util: move 'readexactly' in the util module

Comment for 4.6 cycle: util is a dumpster and is a bad place for
this. Please plan to begin the 4.6 cycle by moving this to a
netutil.py or similar so we can get util.py on a diet.

>
> This function is used in multiple place, having it in util would be better.
> (existing caller will be migrated in another series)
>
> diff --git a/mercurial/changegroup.py b/mercurial/changegroup.py
> --- a/mercurial/changegroup.py
> +++ b/mercurial/changegroup.py
> @@ -32,14 +32,7 @@ from . import (
>  _CHANGEGROUPV2_DELTA_HEADER = "20s20s20s20s20s"
>  _CHANGEGROUPV3_DELTA_HEADER = ">20s20s20s20s20sH"
>
> -def readexactly(stream, n):
> -'''read n bytes from stream.read and abort if less was available'''
> -s = stream.read(n)
> -if len(s) < n:
> -raise error.Abort(_("stream ended unexpectedly"
> -   " (got %d bytes, expected %d)")
> -  % (len(s), n))
> -return s
> +readexactly = util.readexactly
>
>  def getchunk(stream):
>  """return the next chunk from stream as a string"""
> diff --git a/mercurial/util.py b/mercurial/util.py
> --- a/mercurial/util.py
> +++ b/mercurial/util.py
> @@ -3865,3 +3865,12 @@ def safename(f, tag, ctx, others=None):
>  fn = '%s~%s~%s' % (f, tag, n)
>  if fn not in ctx and fn not in others:
>  return fn
> +
> +def readexactly(stream, n):
> +'''read n bytes from stream.read and abort if less was available'''
> +s = stream.read(n)
> +if len(s) < n:
> +raise error.Abort(_("stream ended unexpectedly"
> +   " (got %d bytes, expected %d)")
> +  % (len(s), n))
> +return s
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] lfs: separate a debug message from the subsequent abort message

[Augie Fackler]

On Fri, Jan 19, 2018 at 07:14:18PM -0500, Matt Harbison wrote:
> # HG changeset patch
> # User Matt Harbison 
> # Date 1516407191 18000
> #  Fri Jan 19 19:13:11 2018 -0500
> # Node ID 069df0b952e803b3efb3c59c423522a4800ecde4
> # Parent  047581ddb6cefdbdeb50e314b3cc68f7611bddc7
> lfs: separate a debug message from the subsequent abort message

queued, thanks
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH] lfs: separate a debug message from the subsequent abort message

[Matt Harbison]

# HG changeset patch
# User Matt Harbison 
# Date 1516407191 18000
#  Fri Jan 19 19:13:11 2018 -0500
# Node ID 069df0b952e803b3efb3c59c423522a4800ecde4
# Parent  047581ddb6cefdbdeb50e314b3cc68f7611bddc7
lfs: separate a debug message from the subsequent abort message

diff --git a/hgext/lfs/blobstore.py b/hgext/lfs/blobstore.py
--- a/hgext/lfs/blobstore.py
+++ b/hgext/lfs/blobstore.py
@@ -313,7 +313,7 @@
 self.ui.debug('lfs %s response: %s' % (action, response))
 except util.urlerr.httperror as ex:
 if self.ui.debugflag:
-self.ui.debug('%s: %s' % (oid, ex.read()))
+self.ui.debug('%s: %s\n' % (oid, ex.read()))
 raise LfsRemoteError(_('HTTP error: %s (oid=%s, action=%s)')
  % (ex, oid, action))
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 13 of 14 V3] caches: make 'cachetocopy' available in scmutil

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1516207609 -3600
#  Wed Jan 17 17:46:49 2018 +0100
# Node ID 9c2889f5050bea9d33be5f501eaf1ecf2d9617d3
# Parent  789f14aef1ca0b39e61a47d8989c18cdc6015b53
# EXP-Topic b2-stream
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
9c2889f5050b
caches: make 'cachetocopy' available in scmutil

For more code to use this information, we need it to be more publicly available.

diff --git a/mercurial/cacheutil.py b/mercurial/cacheutil.py
new file mode 100644
--- /dev/null
+++ b/mercurial/cacheutil.py
@@ -0,0 +1,21 @@
+# scmutil.py - Mercurial core utility functions
+#
+#  Copyright Matt Mackall  and other
+#
+# This software may be used and distributed according to the terms of the
+# GNU General Public License version 2 or any later version.
+from __future__ import absolute_import
+
+from . import repoview
+
+def cachetocopy(srcrepo):
+"""return the list of cache file valuable to copy during a clone"""
+# In local clones we're copying all nodes, not just served
+# ones. Therefore copy all branch caches over.
+cachefiles = ['branch2']
+cachefiles += ['branch2-%s' % f for f in repoview.filtertable]
+cachefiles += ['rbc-names-v1', 'rbc-revs-v1']
+cachefiles += ['tags2']
+cachefiles += ['tags2-%s' % f for f in repoview.filtertable]
+cachefiles += ['hgtagsfnodes1']
+return cachefiles
diff --git a/mercurial/hg.py b/mercurial/hg.py
--- a/mercurial/hg.py
+++ b/mercurial/hg.py
@@ -21,6 +21,7 @@ from .node import (
 from . import (
 bookmarks,
 bundlerepo,
+cacheutil,
 cmdutil,
 destutil,
 discovery,
@@ -34,7 +35,6 @@ from . import (
 merge as mergemod,
 node,
 phases,
-repoview,
 scmutil,
 sshpeer,
 statichttprepo,
@@ -459,18 +459,6 @@ def _copycache(srcrepo, dstcachedir, fna
 os.mkdir(dstcachedir)
 util.copyfile(srcbranchcache, dstbranchcache)
 
-def _cachetocopy(srcrepo):
-"""return the list of cache file valuable to copy during a clone"""
-# In local clones we're copying all nodes, not just served
-# ones. Therefore copy all branch caches over.
-cachefiles = ['branch2']
-cachefiles += ['branch2-%s' % f for f in repoview.filtertable]
-cachefiles += ['rbc-names-v1', 'rbc-revs-v1']
-cachefiles += ['tags2']
-cachefiles += ['tags2-%s' % f for f in repoview.filtertable]
-cachefiles += ['hgtagsfnodes1']
-return cachefiles
-
 def clone(ui, peeropts, source, dest=None, pull=False, rev=None,
   update=True, stream=False, branch=None, shareopts=None):
 """Make a copy of an existing repository.
@@ -629,7 +617,7 @@ def clone(ui, peeropts, source, dest=Non
 util.copyfile(srcbookmarks, dstbookmarks)
 
 dstcachedir = os.path.join(destpath, 'cache')
-for cache in _cachetocopy(srcrepo):
+for cache in cacheutil.cachetocopy(srcrepo):
 _copycache(srcrepo, dstcachedir, cache)
 
 # we need to re-init the repo after manually copying the data
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 10 of 14 V3] streamclone: add support for bundle2 based stream clone

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1516203704 -3600
#  Wed Jan 17 16:41:44 2018 +0100
# Node ID 022b04512e28b248c61e034fefaf36d68a6a9f50
# Parent  1bbb83b1ccd5ff07b4d9863f6fa1472bd2ffb34a
# EXP-Topic b2-stream
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
022b04512e28
streamclone: add support for bundle2 based stream clone

The feature put to use the various bits introduced previously. If the server
supports it, the client will request its stream clone through bundle2 instead of
the legacy 'stream_out' commands. The bundle2 version use the better 'v2'
version of stream bundles.

The 'v2' format is not finalized yet. Now that there are some code running it,
we can start working on it again.

Performance numbers are available at the end of this series.

diff --git a/mercurial/bundle2.py b/mercurial/bundle2.py
--- a/mercurial/bundle2.py
+++ b/mercurial/bundle2.py
@@ -1487,6 +1487,7 @@ capabilities = {'HG20': (),
 'remote-changegroup': ('http', 'https'),
 'hgtagsfnodes': (),
 'phases': ('heads',),
+'stream': ('v2',),
}
 
 def getrepocaps(repo, allowpushback=False):
@@ -1507,6 +1508,8 @@ def getrepocaps(repo, allowpushback=Fals
 caps['checkheads'] = ('related',)
 if 'phases' in repo.ui.configlist('devel', 'legacy.exchange'):
 caps.pop('phases')
+if not repo.ui.configbool('experimental', 'bundle2.stream'):
+caps.pop('stream')
 return caps
 
 def bundle2caps(remote):
diff --git a/mercurial/configitems.py b/mercurial/configitems.py
--- a/mercurial/configitems.py
+++ b/mercurial/configitems.py
@@ -431,6 +431,9 @@ coreconfigitem('experimental', 'bundle2-
 coreconfigitem('experimental', 'bundle2.pushback',
 default=False,
 )
+coreconfigitem('experimental', 'bundle2.stream',
+default=False,
+)
 coreconfigitem('experimental', 'bundle2lazylocking',
 default=False,
 )
diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -1455,13 +1455,18 @@ def _pullbundle2(pullop):
 
 # At the moment we don't do stream clones over bundle2. If that is
 # implemented then here's where the check for that will go.
-streaming = False
+streaming = streamclone.canperformstreamclone(pullop, bundle2=True)[0]
 
 # declare pull perimeters
 kwargs['common'] = pullop.common
 kwargs['heads'] = pullop.heads or pullop.rheads
 
-if True:
+if streaming:
+kwargs['cg'] = False
+kwargs['stream'] = True
+pullop.stepsdone.add('changegroup')
+
+else:
 # pulling changegroup
 pullop.stepsdone.add('changegroup')
 
diff --git a/tests/test-clone-uncompressed.t b/tests/test-clone-uncompressed.t
--- a/tests/test-clone-uncompressed.t
+++ b/tests/test-clone-uncompressed.t
@@ -1,5 +1,14 @@
 #require serve
 
+#testcases stream-legacy stream-bundle2
+
+#if stream-bundle2
+  $ cat << EOF >> $HGRCPATH
+  > [experimental]
+  > bundle2.stream = yes
+  > EOF
+#endif
+
 Initialize repository
 the status call is to check for issue5130
 
@@ -18,24 +27,41 @@ the status call is to check for issue513
 
 Basic clone
 
+#if stream-legacy
   $ hg clone --stream -U http://localhost:$HGPORT clone1
   streaming all changes
   1027 files to transfer, 96.3 KB of data
   transferred 96.3 KB in * seconds (*/sec) (glob)
   searching for changes
   no changes found
+#endif
+#if stream-bundle2
+  $ hg clone --stream -U http://localhost:$HGPORT clone1
+  streaming all changes
+  1027 files to transfer, 96.3 KB of data
+  transferred 96.3 KB in * seconds (* */sec) (glob)
+#endif
 
 --uncompressed is an alias to --stream
 
+#if stream-legacy
   $ hg clone --uncompressed -U http://localhost:$HGPORT clone1-uncompressed
   streaming all changes
   1027 files to transfer, 96.3 KB of data
   transferred 96.3 KB in * seconds (*/sec) (glob)
   searching for changes
   no changes found
+#endif
+#if stream-bundle2
+  $ hg clone --uncompressed -U http://localhost:$HGPORT clone1-uncompressed
+  streaming all changes
+  1027 files to transfer, 96.3 KB of data
+  transferred 96.3 KB in * seconds (* */sec) (glob)
+#endif
 
 Clone with background file closing enabled
 
+#if stream-legacy
   $ hg --debug --config worker.backgroundclose=true --config 
worker.backgroundcloseminfilecount=1 clone --stream -U http://localhost:$HGPORT 
clone-background | grep -v adding
   using http://localhost:$HGPORT/
   sending capabilities command
@@ -57,6 +83,28 @@ Clone with background file closing enabl
   bundle2-input-part: total payload size 24
   bundle2-input-bundle: 1 parts total
   checking for updated bookmarks
+#endif
+#if stream-bundle2
+  $ hg --debug --config worker.backgroundclose=true --config 
worker.backgroundcloseminfilecount=1 clone --stream -U http://localhost:$HGPORT 
clone-background | grep -v adding
+  using 

[PATCH 14 of 14 V3] streamclone: also stream caches to the client

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1516233012 -3600
#  Thu Jan 18 00:50:12 2018 +0100
# Node ID d44178c3fd9576e6ca6ab6e92b9823f117141079
# Parent  9c2889f5050bea9d33be5f501eaf1ecf2d9617d3
# EXP-Topic b2-stream
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
d44178c3fd95
streamclone: also stream caches to the client

When stream clone is used over bundle2, relevant cache files are also streamed.
This is expected to be a massive performance win for clone since no important
cache will have to be recomputed.

Some performance numbers:

(All times are wall-clock times in seconds, 2 attempts per case.)

# Mozilla-Central

## Clone over ssh over lan
V1 streaming: 234.3 239.6
V2 streaming: 248.4 243.7

## Clone over ssh over Internet
V1 streaming: 175.5 110.9
V2 streaming: 109.1 111.0

## Clone over HTTP over lan
V1 streaming: 105.3 105.6
V2 streaming: 112.7 111.4

## Clone over HTTP over internet
V1 streaming: 105.6 114.6
V2 streaming: 226.7 225.9

## Hg tags
V1 streaming (no cache): 1.084 1.071
V2 streaming (cache):0.312 0.325

## Hg branches
V1 streaming (no cache):   14.047 14.148
V2 streaming (with cache):  0.312  0.333

# Pypy

## Clone over ssh over internet
V1 streaming: 29.4 30.1
V2 streaming: 31.2 30.1

## Clone over http over internet
V1 streaming: 29.7 29.7
V2 streaming: 75.2 72.9

(since ssh and lan are not affected, there seems to be an issue with how we
read/write the http stream on connection with latency, unrelated to the format)

## Hg tags
V1 streaming (no cache):   1.752 1.664
V2 streaming (with cache): 0.274 0.260

## Hg branches
V1 streaming (no cache):   4.469 4.728
V2 streaming (with cache): 0.318 0.321

# Private repository:
* 500K revision revisions
* 11K topological heads
* 28K branch heads

## hg tags
no cache:   1543.332
with cache:4.900

## hg branches
no cache:   91.828
with cache:  2.955

diff --git a/mercurial/streamclone.py b/mercurial/streamclone.py
--- a/mercurial/streamclone.py
+++ b/mercurial/streamclone.py
@@ -11,10 +11,12 @@ import contextlib
 import os
 import struct
 import tempfile
+import warnings
 
 from .i18n import _
 from . import (
 branchmap,
+cacheutil,
 error,
 phases,
 store,
@@ -435,6 +437,10 @@ class streamcloneapplier(object):
 _fileappend = 0 # append only file
 _filefull = 1   # full snapshot file
 
+# Source of the file
+_srcstore = 's' # store (svfs)
+_srccache = 'c' # cache (cache)
+
 # This is it's own function so extensions can override it.
 def _walkstreamfullstorefiles(repo):
 """list snapshot file from the store"""
@@ -443,12 +449,12 @@ def _walkstreamfullstorefiles(repo):
 fnames.append('phaseroots')
 return fnames
 
-def _filterfull(entry, copy, vfs):
+def _filterfull(entry, copy, vfsmap):
 """actually copy the snapshot files"""
-name, ftype, data = entry
+src, name, ftype, data = entry
 if ftype != _filefull:
 return entry
-return (name, ftype, copy(vfs.join(name)))
+return (src, name, ftype, copy(vfsmap[src].join(name)))
 
 @contextlib.contextmanager
 def maketempcopies():
@@ -466,19 +472,33 @@ def maketempcopies():
 for tmp in files:
 util.tryunlink(tmp)
 
+def _makemap(repo):
+"""make a (src -> vfs) map for the repo"""
+vfsmap = {
+_srcstore: repo.svfs,
+_srccache: repo.cachevfs,
+}
+# we keep repo.vfs out of the on purpose, ther are too many danger there
+# (eg: .hg/hgrc)
+assert repo.vfs not in vfsmap.values()
+
+return vfsmap
+
 def _emit(repo, entries, totalfilesize):
 """actually emit the stream bundle"""
-vfs = repo.svfs
+vfsmap = _makemap(repo)
 progress = repo.ui.progress
 progress(_('bundle'), 0, total=totalfilesize, unit=_('bytes'))
 with maketempcopies() as copy:
 try:
 # copy is delayed until we are in the try
-entries = [_filterfull(e, copy, vfs) for e in entries]
+entries = [_filterfull(e, copy, vfsmap) for e in entries]
 yield None # this release the lock on the repository
 seen = 0
 
-for name, ftype, data in entries:
+for src, name, ftype, data in entries:
+vfs = vfsmap[src]
+yield src
 yield util.uvarintencode(len(name))
 if ftype == _fileappend:
 fp = vfs(name)
@@ -507,10 +527,11 @@ def generatev2(repo):
 """Emit content for version 2 of a streaming clone.
 
 the data stream consists the following entries:
-1) A varint containing the length of the filename
-2) A varint containing the length of file data
-3) N bytes containing the filename (the internal, store-agnostic form)
-4) N bytes containing the file data
+1) A char representing the file destination (eg: store or cache)
+2) A varint containing the length of the 

[PATCH 12 of 14 V3] streamclone: add support for cloning non append-only file

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1516233002 -3600
#  Thu Jan 18 00:50:02 2018 +0100
# Node ID 789f14aef1ca0b39e61a47d8989c18cdc6015b53
# Parent  d8a918033dcfd3dcbac8635616cc1b6c12078b18
# EXP-Topic b2-stream
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
789f14aef1ca
streamclone: add support for cloning non append-only file

The phaseroots are stored in a non append-only file in the repository. We
include them in the stream too. Since they are not append-only, we have to
keep a copy around while we hold the lock to be able to stream them later.

Since phase get exchanged within the stream we can skip requesting them
independently.

As a side effect, this will fixes issue5648 once the feature is enabled by
default.

diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -1465,6 +1465,7 @@ def _pullbundle2(pullop):
 kwargs['cg'] = False
 kwargs['stream'] = True
 pullop.stepsdone.add('changegroup')
+pullop.stepsdone.add('phases')
 
 else:
 # pulling changegroup
@@ -1472,15 +1473,15 @@ def _pullbundle2(pullop):
 
 kwargs['cg'] = pullop.fetch
 
-legacyphase = 'phases' in ui.configlist('devel', 'legacy.exchange')
-hasbinaryphase = 'heads' in pullop.remotebundle2caps.get('phases', ())
-if (not legacyphase and hasbinaryphase):
-kwargs['phases'] = True
-pullop.stepsdone.add('phases')
+legacyphase = 'phases' in ui.configlist('devel', 'legacy.exchange')
+hasbinaryphase = 'heads' in pullop.remotebundle2caps.get('phases', ())
+if (not legacyphase and hasbinaryphase):
+kwargs['phases'] = True
+pullop.stepsdone.add('phases')
 
-if 'listkeys' in pullop.remotebundle2caps:
-if 'phases' not in pullop.stepsdone:
-kwargs['listkeys'] = ['phases']
+if 'listkeys' in pullop.remotebundle2caps:
+if 'phases' not in pullop.stepsdone:
+kwargs['listkeys'] = ['phases']
 
 bookmarksrequested = False
 legacybookmark = 'bookmarks' in ui.configlist('devel', 'legacy.exchange')
diff --git a/mercurial/streamclone.py b/mercurial/streamclone.py
--- a/mercurial/streamclone.py
+++ b/mercurial/streamclone.py
@@ -7,7 +7,10 @@
 
 from __future__ import absolute_import
 
+import contextlib
+import os
 import struct
+import tempfile
 
 from .i18n import _
 from . import (
@@ -428,32 +431,77 @@ class streamcloneapplier(object):
 def apply(self, repo):
 return applybundlev1(repo, self._fh)
 
+# type of file to stream
+_fileappend = 0 # append only file
+_filefull = 1   # full snapshot file
+
+# This is it's own function so extensions can override it.
+def _walkstreamfullstorefiles(repo):
+"""list snapshot file from the store"""
+fnames = []
+if not repo.publishing():
+fnames.append('phaseroots')
+return fnames
+
+def _filterfull(entry, copy, vfs):
+"""actually copy the snapshot files"""
+name, ftype, data = entry
+if ftype != _filefull:
+return entry
+return (name, ftype, copy(vfs.join(name)))
+
+@contextlib.contextmanager
+def maketempcopies():
+"""return a function to temporary copy file"""
+files = []
+try:
+def copy(src):
+fd, dst = tempfile.mkstemp()
+os.close(fd)
+files.append(dst)
+util.copyfiles(src, dst, hardlink=True)
+return dst
+yield copy
+finally:
+for tmp in files:
+util.tryunlink(tmp)
+
 def _emit(repo, entries, totalfilesize):
 """actually emit the stream bundle"""
+vfs = repo.svfs
 progress = repo.ui.progress
 progress(_('bundle'), 0, total=totalfilesize, unit=_('bytes'))
-vfs = repo.svfs
-try:
-seen = 0
-for name, size in entries:
-yield util.uvarintencode(len(name))
-fp = vfs(name)
-try:
-yield util.uvarintencode(size)
-yield name
-if size <= 65536:
-chunks = (fp.read(size),)
-else:
-chunks = util.filechunkiter(fp, limit=size)
-for chunk in chunks:
-seen += len(chunk)
-progress(_('bundle'), seen, total=totalfilesize,
- unit=_('bytes'))
-yield chunk
-finally:
-fp.close()
-finally:
-progress(_('bundle'), None)
+with maketempcopies() as copy:
+try:
+# copy is delayed until we are in the try
+entries = [_filterfull(e, copy, vfs) for e in entries]
+yield None # this release the lock on the repository
+seen = 0
+
+for name, ftype, data in entries:
+yield util.uvarintencode(len(name))
+ 

[PATCH 11 of 14 V3] streamclone: tests phase exchange during stream clone

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1516238924 -3600
#  Thu Jan 18 02:28:44 2018 +0100
# Node ID d8a918033dcfd3dcbac8635616cc1b6c12078b18
# Parent  022b04512e28b248c61e034fefaf36d68a6a9f50
# EXP-Topic b2-stream
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
d8a918033dcf
streamclone: tests phase exchange during stream clone

We add a test dedicated to phases. As reported in issue 5648 stream from a non 
publishing
server is currently broken (does not preserve the phase). We'll fix it with 'v2'
support in the next changesets.

diff --git a/tests/test-clone-uncompressed.t b/tests/test-clone-uncompressed.t
--- a/tests/test-clone-uncompressed.t
+++ b/tests/test-clone-uncompressed.t
@@ -262,3 +262,71 @@ clone it
 #endif
   $ hg -R with-bookmarks bookmarks
  some-bookmark 1:c17445101a72
+
+Stream repository with phases
+-
+
+Clone as publishing
+
+  $ hg -R server phase -r 'all()'
+  0: draft
+  1: draft
+
+#if stream-legacy
+  $ hg clone --stream http://localhost:$HGPORT phase-publish
+  streaming all changes
+  1027 files to transfer, 96.3 KB of data
+  transferred 96.3 KB in * seconds (*) (glob)
+  searching for changes
+  no changes found
+  updating to branch default
+  1025 files updated, 0 files merged, 0 files removed, 0 files unresolved
+#endif
+#if stream-bundle2
+  $ hg clone --stream http://localhost:$HGPORT phase-publish
+  streaming all changes
+  1027 files to transfer, 96.3 KB of data
+  transferred 96.3 KB in * seconds (* */sec) (glob)
+  updating to branch default
+  1025 files updated, 0 files merged, 0 files removed, 0 files unresolved
+#endif
+  $ hg -R phase-publish phase -r 'all()'
+  0: public
+  1: public
+
+Clone as non publishing
+
+  $ cat << EOF >> server/.hg/hgrc
+  > [phases]
+  > publish = False
+  > EOF
+  $ killdaemons.py
+  $ hg -R server serve -p $HGPORT -d --pid-file=hg.pid
+  $ cat hg.pid >> $DAEMON_PIDS
+
+#if stream-legacy
+  $ hg clone --stream http://localhost:$HGPORT phase-no-publish
+  streaming all changes
+  1027 files to transfer, 96.3 KB of data
+  transferred 96.3 KB in * seconds (*) (glob)
+  searching for changes
+  no changes found
+  updating to branch default
+  1025 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  $ hg -R phase-no-publish phase -r 'all()'
+  0: public
+  1: public
+#endif
+#if stream-bundle2
+  $ hg clone --stream http://localhost:$HGPORT phase-no-publish
+  streaming all changes
+  1027 files to transfer, 96.3 KB of data
+  transferred 96.3 KB in * seconds (* */sec) (glob)
+  updating to branch default
+  1025 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  $ hg -R phase-no-publish phase -r 'all()'
+  0: public
+  1: public
+#endif
+
+  $ killdaemons.py
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 04 of 14 V3] streamclone: rework canperformstreamclone

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1516232727 -3600
#  Thu Jan 18 00:45:27 2018 +0100
# Node ID 615f8f725f24c2c541b85a006bfa4be0915bfcf3
# Parent  542df1a9814ff6e7e688c59528e3d8bad82f8c11
# EXP-Topic b2-stream
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
615f8f725f24
streamclone: rework canperformstreamclone

There is code about bundle2 laying around in `canperformstreamclone` but not
put to any uses. As we discovered with the previous patch, streambundle 'v1'
won't work on bundle2 because they are readline based. So we jump to 'v2' as
the first expected supported version.

diff --git a/mercurial/streamclone.py b/mercurial/streamclone.py
--- a/mercurial/streamclone.py
+++ b/mercurial/streamclone.py
@@ -18,12 +18,11 @@ from . import (
 util,
 )
 
-def canperformstreamclone(pullop, bailifbundle2supported=False):
+def canperformstreamclone(pullop, bundle2=False):
 """Whether it is possible to perform a streaming clone as part of pull.
 
-``bailifbundle2supported`` will cause the function to return False if
-bundle2 stream clones are supported. It should only be called by the
-legacy stream clone code path.
+``bundle2`` will cause the function to consider stream clone through
+bundle2 and only through bundle2.
 
 Returns a tuple of (supported, requirements). ``supported`` is True if
 streaming clone is supported and False otherwise. ``requirements`` is
@@ -35,18 +34,18 @@ def canperformstreamclone(pullop, bailif
 
 bundle2supported = False
 if pullop.canusebundle2:
-if 'v1' in pullop.remotebundle2caps.get('stream', []):
+if 'v2' in pullop.remotebundle2caps.get('stream', []):
 bundle2supported = True
 # else
 # Server doesn't support bundle2 stream clone or doesn't support
 # the versions we support. Fall back and possibly allow legacy.
 
 # Ensures legacy code path uses available bundle2.
-if bailifbundle2supported and bundle2supported:
+if bundle2supported and not bundle2:
 return False, None
 # Ensures bundle2 doesn't try to do a stream clone if it isn't supported.
-#elif not bailifbundle2supported and not bundle2supported:
-#return False, None
+elif bundle2 and not bundle2supported:
+return False, None
 
 # Streaming clone only works on empty repositories.
 if len(repo):
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 06 of 14 V3] bundle2: add support for a 'stream' parameter to 'getbundle'

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1516203383 -3600
#  Wed Jan 17 16:36:23 2018 +0100
# Node ID 752abe0317e37feb6d837b2b17e0d63fa10fd63d
# Parent  b9cc543a1208750e10125905dd09af32854ee285
# EXP-Topic b2-stream
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
752abe0317e3
bundle2: add support for a 'stream' parameter to 'getbundle'

This parameter can be used to request a stream bundle.

diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -1747,6 +1747,19 @@ def getbundlechunks(repo, source, heads=
 
 return bundler.getchunks()
 
+@getbundle2partsgenerator('stream')
+def _getbundlestream(bundler, repo, source, bundlecaps=None,
+ b2caps=None, heads=None, common=None, **kwargs):
+if not kwargs.get('stream', False):
+return
+filecount, bytecount, it = streamclone.generatev2(repo)
+requirements = ' '.join(repo.requirements)
+part = bundler.newpart('stream', data=it)
+part.addparam('bytecount', '%d' % bytecount, mandatory=True)
+part.addparam('filecount', '%d' % filecount, mandatory=True)
+part.addparam('requirements', requirements, mandatory=True)
+part.addparam('version', 'v2', mandatory=True)
+
 @getbundle2partsgenerator('changegroup')
 def _getbundlechangegrouppart(bundler, repo, source, bundlecaps=None,
   b2caps=None, heads=None, common=None, **kwargs):
diff --git a/mercurial/wireproto.py b/mercurial/wireproto.py
--- a/mercurial/wireproto.py
+++ b/mercurial/wireproto.py
@@ -212,7 +212,9 @@ gboptsmap = {'heads':  'nodes',
  'bundlecaps': 'scsv',
  'listkeys': 'csv',
  'cg': 'boolean',
- 'cbattempted': 'boolean'}
+ 'cbattempted': 'boolean',
+ 'stream': 'boolean',
+}
 
 # client side
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 05 of 14 V3] bundle2: add a 'stream' part handler for stream cloning

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1516203322 -3600
#  Wed Jan 17 16:35:22 2018 +0100
# Node ID b9cc543a1208750e10125905dd09af32854ee285
# Parent  615f8f725f24c2c541b85a006bfa4be0915bfcf3
# EXP-Topic b2-stream
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
b9cc543a1208
bundle2: add a 'stream' part handler for stream cloning

The part contains the necessary arguments and payload to handle a stream bundle
v2. It will be put to use in later changesets.

diff --git a/mercurial/bundle2.py b/mercurial/bundle2.py
--- a/mercurial/bundle2.py
+++ b/mercurial/bundle2.py
@@ -164,6 +164,7 @@ from . import (
 phases,
 pushkey,
 pycompat,
+streamclone,
 tags,
 url,
 util,
@@ -2114,3 +2115,30 @@ def bundle2getvars(op, part):
 key = "USERVAR_" + key
 hookargs[key] = value
 op.addhookargs(hookargs)
+
+@parthandler('stream', ('requirements', 'filecount', 'bytecount', 'version'))
+def handlestreambundle(op, part):
+
+version = part.params['version']
+if version != 'v2':
+raise error.Abort(_('unknown stream bundle version %s') % version)
+requirements = part.params['requirements'].split()
+filecount = int(part.params['filecount'])
+bytecount = int(part.params['bytecount'])
+
+repo = op.repo
+if len(repo):
+msg = _('cannot apply stream clone to non empty repository')
+raise error.Abort(msg)
+
+repo.ui.debug('applying stream bundle\n')
+streamclone.applybundlev2(repo, part, filecount, bytecount,
+  requirements)
+
+# new requirements = old non-format requirements +
+#new format-related remote requirements
+# requirements from the streamed-in repository
+repo.requirements = set(requirements) | (
+repo.requirements - repo.supportedformats)
+repo._applyopenerreqs()
+repo._writerequirements()
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 07 of 14 V3] clone: allow bundle2's stream clone with 'server.disablefullbundle'

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1516203512 -3600
#  Wed Jan 17 16:38:32 2018 +0100
# Node ID e0d5763061cc551ea34f748631fc9985836885a3
# Parent  752abe0317e37feb6d837b2b17e0d63fa10fd63d
# EXP-Topic b2-stream
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
e0d5763061cc
clone: allow bundle2's stream clone with 'server.disablefullbundle'

The previous check was a bit too strict and would not recognize a get bundle
not requesting changegroup.

diff --git a/mercurial/wireproto.py b/mercurial/wireproto.py
--- a/mercurial/wireproto.py
+++ b/mercurial/wireproto.py
@@ -855,10 +855,11 @@ def getbundle(repo, proto, others):
 if repo.ui.configbool('server', 'disablefullbundle'):
 # Check to see if this is a full clone.
 clheads = set(repo.changelog.heads())
+changegroup = opts.get('cg', True)
 heads = set(opts.get('heads', set()))
 common = set(opts.get('common', set()))
 common.discard(nullid)
-if not common and clheads == heads:
+if changegroup and not common and clheads == heads:
 raise error.Abort(
 _('server has pull-based clones disabled'),
 hint=_('remove --pull if specified or upgrade Mercurial'))
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 08 of 14 V3] pull: reorganize bundle2 argument bundling

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1516203125 -3600
#  Wed Jan 17 16:32:05 2018 +0100
# Node ID 6c54ed31dd5dbc8ba7de011517ce9c595787ad7d
# Parent  e0d5763061cc551ea34f748631fc9985836885a3
# EXP-Topic b2-stream
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
6c54ed31dd5d
pull: reorganize bundle2 argument bundling

We are about to add the ability to use stream bundle with bundle2. Before doing
so, we need to gather some code that will not be used in the bundle2 case. There
is no behavior change within this changeset.

diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -1450,24 +1450,32 @@ def _pullbundle2(pullop):
 For now, the only supported data are changegroup."""
 kwargs = {'bundlecaps': caps20to10(pullop.repo)}
 
+# make ui easier to access
+ui = pullop.repo.ui
+
 # At the moment we don't do stream clones over bundle2. If that is
 # implemented then here's where the check for that will go.
 streaming = False
 
+# declare pull perimeters
+kwargs['common'] = pullop.common
+kwargs['heads'] = pullop.heads or pullop.rheads
+
 # pulling changegroup
 pullop.stepsdone.add('changegroup')
 
-kwargs['common'] = pullop.common
-kwargs['heads'] = pullop.heads or pullop.rheads
 kwargs['cg'] = pullop.fetch
 
-ui = pullop.repo.ui
 legacyphase = 'phases' in ui.configlist('devel', 'legacy.exchange')
 hasbinaryphase = 'heads' in pullop.remotebundle2caps.get('phases', ())
 if (not legacyphase and hasbinaryphase):
 kwargs['phases'] = True
 pullop.stepsdone.add('phases')
 
+if 'listkeys' in pullop.remotebundle2caps:
+if 'phases' not in pullop.stepsdone:
+kwargs['listkeys'] = ['phases']
+
 bookmarksrequested = False
 legacybookmark = 'bookmarks' in ui.configlist('devel', 'legacy.exchange')
 hasbinarybook = 'bookmarks' in pullop.remotebundle2caps
@@ -1482,8 +1490,6 @@ def _pullbundle2(pullop):
 bookmarksrequested = True
 
 if 'listkeys' in pullop.remotebundle2caps:
-if 'phases' not in pullop.stepsdone:
-kwargs['listkeys'] = ['phases']
 if 'request-bookmarks' not in pullop.stepsdone:
 # make sure to always includes bookmark data when migrating
 # `hg incoming --bundle` to using this function.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 09 of 14 V3] pull: preindent some code

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1516194826 -3600
#  Wed Jan 17 14:13:46 2018 +0100
# Node ID 1bbb83b1ccd5ff07b4d9863f6fa1472bd2ffb34a
# Parent  6c54ed31dd5dbc8ba7de011517ce9c595787ad7d
# EXP-Topic b2-stream
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
1bbb83b1ccd5
pull: preindent some code

Next changesets will add support for using stream cloning with bundle2. We
introduce indentation change first for clarity.

diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -1461,10 +1461,11 @@ def _pullbundle2(pullop):
 kwargs['common'] = pullop.common
 kwargs['heads'] = pullop.heads or pullop.rheads
 
-# pulling changegroup
-pullop.stepsdone.add('changegroup')
+if True:
+# pulling changegroup
+pullop.stepsdone.add('changegroup')
 
-kwargs['cg'] = pullop.fetch
+kwargs['cg'] = pullop.fetch
 
 legacyphase = 'phases' in ui.configlist('devel', 'legacy.exchange')
 hasbinaryphase = 'heads' in pullop.remotebundle2caps.get('phases', ())
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 01 of 14 V3] util: move 'readexactly' in the util module

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1516391495 -3600
#  Fri Jan 19 20:51:35 2018 +0100
# Node ID 15f7795f96a5f9acb3ed2e640fcec82f3ccd6f53
# Parent  de32acb24949c0e3633de373d1c6c8c814faa804
# EXP-Topic b2-stream
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
15f7795f96a5
util: move 'readexactly' in the util module

This function is used in multiple place, having it in util would be better.
(existing caller will be migrated in another series)

diff --git a/mercurial/changegroup.py b/mercurial/changegroup.py
--- a/mercurial/changegroup.py
+++ b/mercurial/changegroup.py
@@ -32,14 +32,7 @@ from . import (
 _CHANGEGROUPV2_DELTA_HEADER = "20s20s20s20s20s"
 _CHANGEGROUPV3_DELTA_HEADER = ">20s20s20s20s20sH"
 
-def readexactly(stream, n):
-'''read n bytes from stream.read and abort if less was available'''
-s = stream.read(n)
-if len(s) < n:
-raise error.Abort(_("stream ended unexpectedly"
-   " (got %d bytes, expected %d)")
-  % (len(s), n))
-return s
+readexactly = util.readexactly
 
 def getchunk(stream):
 """return the next chunk from stream as a string"""
diff --git a/mercurial/util.py b/mercurial/util.py
--- a/mercurial/util.py
+++ b/mercurial/util.py
@@ -3865,3 +3865,12 @@ def safename(f, tag, ctx, others=None):
 fn = '%s~%s~%s' % (f, tag, n)
 if fn not in ctx and fn not in others:
 return fn
+
+def readexactly(stream, n):
+'''read n bytes from stream.read and abort if less was available'''
+s = stream.read(n)
+if len(s) < n:
+raise error.Abort(_("stream ended unexpectedly"
+   " (got %d bytes, expected %d)")
+  % (len(s), n))
+return s
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 03 of 14 V3] streamclone: define first iteration of version 2 of stream format

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1516232936 -3600
#  Thu Jan 18 00:48:56 2018 +0100
# Node ID 542df1a9814ff6e7e688c59528e3d8bad82f8c11
# Parent  5dbc3c53c923b8d11b5efcaf0f415b3d8c8c5180
# EXP-Topic b2-stream
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
542df1a9814f
streamclone: define first iteration of version 2 of stream format

(This patch is based on a first draft from Gregory Szorc, with deeper rework)

Version 1 of the stream clone format was invented many years ago and suffers
from a few deficiencies:

1) Filenames are stored in store-encoded (on filesystem) form rather than in
   their internal form. This makes future compatibility with new store
   filename encodings more difficult.
2) File entry "headers" consist of a newline of the file name followed by the
   string file size. Converting strings to integers is avoidable overhead. We
   can't store filenames with newlines (manifests have this limitation as
   well, so it isn't a major concern). But the big concern here is the
   necessity for readline(). Scanning for newlines means reading ahead and
   that means extra buffer allocations and slicing (in Python) and this makes
   performance suffer.
3) Filenames aren't compressed optimally. Filenames should be compressed well
   since there is a lot of repeated data. However, since they are scattered
   all over the stream (with revlog data in between), they typically fall
   outside the window size of the compressor and don't compress.
4) It can only exchange stored based content, being able to exchange caches
   too would be nice.
5) It is limited to a stream-based protocol and isn't suitable for an on-disk
   format for general repository reading because the offset of individual file
   entries requires scanning the entire file to find file records.

As part of enabling streaming clones to work in bundle2, #2 proved to have a
significant negative impact on performance. Since bundle2 provides the
opportunity to start fresh, Gregory Szorc figured he would take the
opportunity to invent a new streaming clone data format.

The new format devised in this series addresses #1, #2, and #4. It punts on #3
because it was complex without yielding a significant gain and on #5 because
devising a new store format that "packs" multiple revlogs into a single
"packed revlog" is massive scope bloat. However, this v2 format might be
suitable for streaming into a "packed revlog" with minimal processing. If it
works, great. If not, we can always invent stream format when it is needed.

This patch only introduces the bases of the format. We'll get it usable through
bundle2 first, then we'll extend the format in future patches to bring it to its
full potential (especially #4).

diff --git a/mercurial/streamclone.py b/mercurial/streamclone.py
--- a/mercurial/streamclone.py
+++ b/mercurial/streamclone.py
@@ -428,3 +428,115 @@ class streamcloneapplier(object):
 
 def apply(self, repo):
 return applybundlev1(repo, self._fh)
+
+def _emit(repo, entries, totalfilesize):
+"""actually emit the stream bundle"""
+progress = repo.ui.progress
+progress(_('bundle'), 0, total=totalfilesize, unit=_('bytes'))
+vfs = repo.svfs
+try:
+seen = 0
+for name, size in entries:
+yield util.uvarintencode(len(name))
+fp = vfs(name)
+try:
+yield util.uvarintencode(size)
+yield name
+if size <= 65536:
+chunks = (fp.read(size),)
+else:
+chunks = util.filechunkiter(fp, limit=size)
+for chunk in chunks:
+seen += len(chunk)
+progress(_('bundle'), seen, total=totalfilesize,
+ unit=_('bytes'))
+yield chunk
+finally:
+fp.close()
+finally:
+progress(_('bundle'), None)
+
+def generatev2(repo):
+"""Emit content for version 2 of a streaming clone.
+
+the data stream consists the following entries:
+1) A varint containing the length of the filename
+2) A varint containing the length of file data
+3) N bytes containing the filename (the internal, store-agnostic form)
+4) N bytes containing the file data
+
+Returns a 3-tuple of (file count, file size, data iterator).
+"""
+
+with repo.lock():
+
+entries = []
+totalfilesize = 0
+
+repo.ui.debug('scanning\n')
+for name, ename, size in _walkstreamfiles(repo):
+if size:
+entries.append((name, size))
+totalfilesize += size
+
+chunks = _emit(repo, entries, totalfilesize)
+
+return len(entries), totalfilesize, chunks
+
+def consumev2(repo, fp, filecount, filesize):
+"""Apply the contents from a version 2 streaming clone.
+
+Data 

[PATCH 02 of 14 V3] util: implement varint functions

[Boris Feld]

# HG changeset patch
# User Gregory Szorc 
# Date 1516398755 -3600
#  Fri Jan 19 22:52:35 2018 +0100
# Node ID 5dbc3c53c923b8d11b5efcaf0f415b3d8c8c5180
# Parent  15f7795f96a5f9acb3ed2e640fcec82f3ccd6f53
# EXP-Topic b2-stream
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
5dbc3c53c923
util: implement varint functions

This will be useful in an incoming version-2 of the stream format.

diff --git a/mercurial/util.py b/mercurial/util.py
--- a/mercurial/util.py
+++ b/mercurial/util.py
@@ -3874,3 +3874,73 @@ def readexactly(stream, n):
" (got %d bytes, expected %d)")
   % (len(s), n))
 return s
+
+def uvarintencode(value):
+"""Encode an unsigned integer value to a varint.
+
+A varint is a variable length integer of 1 or more bytes. Each byte
+except the last has the most significant bit set. The lower 7 bits of
+each byte store the 2's complement representation, least significant group
+first.
+
+>>> uvarintencode(0)
+'\\x00'
+>>> uvarintencode(1)
+'\\x01'
+>>> uvarintencode(127)
+'\\x7f'
+>>> uvarintencode(1337)
+'\\xb9\\n'
+>>> uvarintencode(65536)
+'\\x80\\x80\\x04'
+>>> uvarintencode(-1)
+Traceback (most recent call last):
+...
+ProgrammingError: negative value for uvarint: -1
+"""
+if value < 0:
+raise error.ProgrammingError('negative value for uvarint: %d'
+ % value)
+bits = value & 0x7f
+value >>= 7
+bytes = []
+while value:
+bytes.append(pycompat.bytechr(0x80 | bits))
+bits = value & 0x7f
+value >>= 7
+bytes.append(pycompat.bytechr(bits))
+
+return ''.join(bytes)
+
+def uvarintdecodestream(fh):
+"""Decode an unsigned variable length integer from a stream.
+
+The passed argument is anything that has a ``.read(N)`` method.
+
+>>> try:
+... from StringIO import StringIO as BytesIO
+... except ImportError:
+... from io import BytesIO
+>>> uvarintdecodestream(BytesIO(b'\\x00'))
+0
+>>> uvarintdecodestream(BytesIO(b'\\x01'))
+1
+>>> uvarintdecodestream(BytesIO(b'\\x7f'))
+127
+>>> uvarintdecodestream(BytesIO(b'\\xb9\\n'))
+1337
+>>> uvarintdecodestream(BytesIO(b'\\x80\\x80\\x04'))
+65536
+>>> uvarintdecodestream(BytesIO(b'\\x80'))
+Traceback (most recent call last):
+...
+Abort: stream ended unexpectedly (got 0 bytes, expected 1)
+"""
+result = 0
+shift = 0
+while True:
+byte = ord(readexactly(fh, 1))
+result |= ((byte & 0x7f) << shift)
+if not (byte & 0x80):
+return result
+shift += 7
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] sshserver: add a couple of tests for argument parsing

[Augie Fackler]

> On Jan 19, 2018, at 17:25, Siddharth Agarwal  wrote:
> 
> # HG changeset patch
> # User Siddharth Agarwal 
> # Date 1516400709 28800
> #  Fri Jan 19 14:25:09 2018 -0800
> # Node ID 897f09345e370f08c019e7025d51e3e7ff3832fb
> # Parent  6d65cef5b038ff4c141c0bbc1a2e366e45c016a2
> sshserver: add a couple of tests for argument parsing
> 
> I noticed that we didn't have any unit tests covering wire protocol argument
> parsing.

Nice catch. I'd have taken this even during the freeze, queued, thanks.

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH] sshserver: add a couple of tests for argument parsing

[Siddharth Agarwal]

# HG changeset patch
# User Siddharth Agarwal 
# Date 1516400709 28800
#  Fri Jan 19 14:25:09 2018 -0800
# Node ID 897f09345e370f08c019e7025d51e3e7ff3832fb
# Parent  6d65cef5b038ff4c141c0bbc1a2e366e45c016a2
sshserver: add a couple of tests for argument parsing

I noticed that we didn't have any unit tests covering wire protocol argument
parsing.

diff --git a/tests/test-sshserver.py b/tests/test-sshserver.py
new file mode 100644
--- /dev/null
+++ b/tests/test-sshserver.py
@@ -0,0 +1,44 @@
+from __future__ import absolute_import, print_function
+
+import io
+import unittest
+
+import silenttestrunner
+
+from mercurial import (
+sshserver,
+wireproto,
+)
+
+class SSHServerGetArgsTests(unittest.TestCase):
+def testparseknown(self):
+tests = [
+('* 0\nnodes 0\n', ['', {}]),
+('* 0\nnodes 40\n\n',
+ ['', {}]),
+]
+for input, expected in tests:
+self.assertparse('known', input, expected)
+
+def assertparse(self, cmd, input, expected):
+server = mockserver(input)
+_func, spec = wireproto.commands[cmd]
+self.assertEqual(server.getargs(spec), expected)
+
+def mockserver(inbytes):
+ui = mockui(inbytes)
+repo = mockrepo(ui)
+return sshserver.sshserver(ui, repo)
+
+class mockrepo(object):
+def __init__(self, ui):
+self.ui = ui
+
+class mockui(object):
+def __init__(self, inbytes):
+self.fin = io.BytesIO(inbytes)
+self.fout = io.BytesIO()
+self.ferr = io.BytesIO()
+
+if __name__ == '__main__':
+silenttestrunner.main(__name__)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

mercurial@35751: 5 new changesets

[Mercurial Commits]

5 new changesets in mercurial:

https://www.mercurial-scm.org/repo/hg/rev/de32acb24949
changeset:   35747:de32acb24949
user:Boris Feld 
date:Wed Jan 17 16:01:06 2018 +0100
summary: stream: add a test showing we also clone bookmarks

https://www.mercurial-scm.org/repo/hg/rev/963a611b2f39
changeset:   35748:963a611b2f39
user:Martin von Zweigbergk 
date:Fri Jan 19 11:35:55 2018 -0800
summary: scmutil: 0-pad transaction report callback category

https://www.mercurial-scm.org/repo/hg/rev/3a3b59bbe7ce
changeset:   35749:3a3b59bbe7ce
user:Martin von Zweigbergk 
date:Fri Jan 19 12:33:03 2018 -0800
summary: localrepo: run cache-warming transaction callback before report 
callback

https://www.mercurial-scm.org/repo/hg/rev/a39a9df7ecca
changeset:   35750:a39a9df7ecca
user:Joerg Sonnenberger 
date:Fri Jan 12 10:59:58 2018 +0100
summary: wireproto: split streamres into legacy and modern case

https://www.mercurial-scm.org/repo/hg/rev/6d65cef5b038
changeset:   35751:6d65cef5b038
bookmark:@
tag: tip
parent:  35750:a39a9df7ecca
parent:  35541:87676e8ee056
user:Augie Fackler 
date:Fri Jan 19 16:28:11 2018 -0500
summary: merge with stable

-- 
Repository URL: https://www.mercurial-scm.org/repo/hg
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 04 of 15] streamclone: define first iteration of version 2 of stream format

[Augie Fackler]

> On Jan 19, 2018, at 16:35, Boris Feld  wrote:
> 
> On Fri, 2018-01-19 at 15:54 -0500, Augie Fackler wrote:
>> On Fri, Jan 19, 2018 at 09:08:48PM +0100, Boris Feld wrote:
>>> # HG changeset patch
>>> # User Boris Feld 
>>> # Date 1516232936 -3600
>>> #  Thu Jan 18 00:48:56 2018 +0100
>>> # Node ID 4ee91fb55e208e8b139595ce9c2cae25aa9c54ea
>>> # Parent  b80a8e39ac9bf984c25a666bd7f6c47d876d26af
>>> # EXP-Topic b2-stream
>>> # Available At https://bitbucket.org/octobus/mercurial-devel/
>>> #  hg pull https://bitbucket.org/octobus/mercurial-deve
>>> l/ -r 4ee91fb55e20
>>> streamclone: define first iteration of version 2 of stream format
>> 
>> This is a good start of a series, but:
>> 
>> 1) patch 3 is begging for doctests on the varint scheme
>> 
> 
> We are about to follow up with them.

Just hold them for 4.6 please.

> 
>> 2) This patch should probably include help/internals/ documentation
>> on
>>   the format, rather than only encoding it in a docstring
>> 
> 
> We can follow up with that.
> 
>> 3) Patches that claim to be a big performance win, but don't include
>>   any concrete testing numbers.
> 
> The performance win is about recomputing cache. We are getting number
> as we speak, but performance issue related to branchmap and tags have
> been well documented in the past.

You're mostly missing the point: the reason for this significant body of new 
functionality was performance wins. I'd expect patches that come in under a 
performance banner to explain, including with benchmarks why they're a win. In 
this case, that includes not only explaining that we avoid a 25 minute cache 
hit (which is great), but also some testing that demonstrates that the new 
format is at least *not worse* than what was there before. bundle2 doesn't have 
a spotless performance record, so I get reflexively dubious when a new format 
inside bundle2 claims to be a win. :)

> On our repository with the most heads, it took 25 minutes to recompute
> the tags cache and 1 minute for the branch cache. And the laptop wasn't
> doing much else.
> 
>> 
>> I think at this point we're going to admit defeat in the name of
>> getting an RC release done before I go to bed tonight. Performance
>> information I'd like to see in any v3 of this series:
>> 
>> 1) comparison between the new streaming clone and the existing one on
>> small repos
>> 
>> 2) comparison on a medium repo with few branches (the hg repo could
>> be good for this)
>> 
>> 3) comparison on a large repo with many heads (might need to use
>> contrib/synthrepo to make something for this?)
>> 
>> 4) comparison on a large repo with many named branches (pypy?)
>> 
>> 5) comparison on mozilla-central
> 
> Those characteristics shouldn't impact the performance of the stream
> clone.

It's more about relative overheads and things. My gut is that involving bundle2 
adds measurable overhead to the process (though I could be wrong!), so I'd like 
a decent spectrum of repo types to try and help figure out how much the cache 
recomputation is a win or not. Does that make sense?

Anyway, it's too late for this cycle, as I'm about out of time in my day to 
make a release. Sorry. :/
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 08 of 15] clone: allow bundle2's stream clone with 'server.disablefullbundle'

[Boris Feld]

On Fri, 2018-01-19 at 15:27 -0500, Augie Fackler wrote:
> On Fri, Jan 19, 2018 at 09:08:52PM +0100, Boris Feld wrote:
> > # HG changeset patch
> > # User Boris Feld 
> > # Date 1516203512 -3600
> > #  Wed Jan 17 16:38:32 2018 +0100
> > # Node ID 77a0634011b5bc89472a134c5ea2b5623f6ca273
> > # Parent  b11f4652647e791727e14c94a0ccb7c0282c5a29
> > # EXP-Topic b2-stream
> > # Available At https://bitbucket.org/octobus/mercurial-devel/
> > #  hg pull https://bitbucket.org/octobus/mercurial-deve
> > l/ -r 77a0634011b5
> > clone: allow bundle2's stream clone with 'server.disablefullbundle'
> 
> That is, prior to this patch server.disablefullbundle also banned
> streaming clones even if streaming clones were enabled? Did I get
> that
> right?
> 

This is only affecting the v2 stream. v1 works fine without this patch.
V2 (via bundle2) needs this patch.

> > 
> > The previous check was a bit too strict and would not recognize a
> > get bundle
> > not requesting changegroup.
> > 
> > diff --git a/mercurial/wireproto.py b/mercurial/wireproto.py
> > --- a/mercurial/wireproto.py
> > +++ b/mercurial/wireproto.py
> > @@ -855,10 +855,11 @@ def getbundle(repo, proto, others):
> >  if repo.ui.configbool('server', 'disablefullbundle'):
> >  # Check to see if this is a full clone.
> >  clheads = set(repo.changelog.heads())
> > +changegroup = opts.get('cg', True)
> >  heads = set(opts.get('heads', set()))
> >  common = set(opts.get('common', set()))
> >  common.discard(nullid)
> > -if not common and clheads == heads:
> > +if changegroup and not common and clheads == heads:
> >  raise error.Abort(
> >  _('server has pull-based clones disabled'),
> >  hint=_('remove --pull if specified or upgrade
> > Mercurial'))
> > ___
> > Mercurial-devel mailing list
> > Mercurial-devel@mercurial-scm.org
> > https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 04 of 15] streamclone: define first iteration of version 2 of stream format

[Boris Feld]

On Fri, 2018-01-19 at 15:54 -0500, Augie Fackler wrote:
> On Fri, Jan 19, 2018 at 09:08:48PM +0100, Boris Feld wrote:
> > # HG changeset patch
> > # User Boris Feld 
> > # Date 1516232936 -3600
> > #  Thu Jan 18 00:48:56 2018 +0100
> > # Node ID 4ee91fb55e208e8b139595ce9c2cae25aa9c54ea
> > # Parent  b80a8e39ac9bf984c25a666bd7f6c47d876d26af
> > # EXP-Topic b2-stream
> > # Available At https://bitbucket.org/octobus/mercurial-devel/
> > #  hg pull https://bitbucket.org/octobus/mercurial-deve
> > l/ -r 4ee91fb55e20
> > streamclone: define first iteration of version 2 of stream format
> 
> This is a good start of a series, but:
> 
> 1) patch 3 is begging for doctests on the varint scheme
> 

We are about to follow up with them.

> 2) This patch should probably include help/internals/ documentation
> on
>the format, rather than only encoding it in a docstring
> 

We can follow up with that.

> 3) Patches that claim to be a big performance win, but don't include
>any concrete testing numbers.

The performance win is about recomputing cache. We are getting number
as we speak, but performance issue related to branchmap and tags have
been well documented in the past.

On our repository with the most heads, it took 25 minutes to recompute
the tags cache and 1 minute for the branch cache. And the laptop wasn't
doing much else.

> 
> I think at this point we're going to admit defeat in the name of
> getting an RC release done before I go to bed tonight. Performance
> information I'd like to see in any v3 of this series:
> 
> 1) comparison between the new streaming clone and the existing one on
> small repos
> 
> 2) comparison on a medium repo with few branches (the hg repo could
> be good for this)
> 
> 3) comparison on a large repo with many heads (might need to use
> contrib/synthrepo to make something for this?)
> 
> 4) comparison on a large repo with many named branches (pypy?)
> 
> 5) comparison on mozilla-central

Those characteristics shouldn't impact the performance of the stream
clone.

> 
> I'm gathering that this is important work for you all for an
> important
> client or set of clients, and I'm sorry we're not going to manage it
> today. In the future, you could potentially sidestep some of this
> frustration by giving people a heads up earlier than a
> performance-critical 14 patch series with a new wireproto format.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Freeze in effect for hg 4.5

[Augie Fackler]

I'll cut the 4.5-rc release sometime today, tomorrow at the latest.

Please try and focus on bug fixes until we get 4.5 out. If you want to get work 
done on complicated stuff (new protocols etc), we can definitely figure that 
out, so don't hesitate to ask.

Thanks!
Augie
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1862: wireproto: split streamres into legacy and modern case

[joerg.sonnenberger (Joerg Sonnenberger)]

This revision was automatically updated to reflect the committed changes.
Closed by commit rHGa39a9df7ecca: wireproto: split streamres into legacy and 
modern case (authored by joerg.sonnenberger, committed by ).

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1862?vs=4933=4959

REVISION DETAIL
  https://phab.mercurial-scm.org/D1862

AFFECTED FILES
  hgext/largefiles/proto.py
  mercurial/hgweb/protocol.py
  mercurial/sshserver.py
  mercurial/wireproto.py

CHANGE DETAILS

diff --git a/mercurial/wireproto.py b/mercurial/wireproto.py
--- a/mercurial/wireproto.py
+++ b/mercurial/wireproto.py
@@ -522,15 +522,26 @@
 
 Accepts a generator containing chunks of data to be sent to the client.
 
-``v1compressible`` indicates whether this data can be compressed to
-"version 1" clients (technically: HTTP peers using
-application/mercurial-0.1 media type). This flag should NOT be used on
-new commands because new clients should support a more modern compression
-mechanism.
+``prefer_uncompressed`` indicates that the data is expected to be
+uncompressable and that the stream should therefore use the ``none``
+engine.
 """
-def __init__(self, gen=None, v1compressible=False):
+def __init__(self, gen=None, prefer_uncompressed=False):
 self.gen = gen
-self.v1compressible = v1compressible
+self.prefer_uncompressed = prefer_uncompressed
+
+class streamres_legacy(object):
+"""wireproto reply: uncompressed binary stream
+
+The call was successful and the result is a stream.
+
+Accepts a generator containing chunks of data to be sent to the client.
+
+Like ``streamres``, but sends an uncompressed data for "version 1" clients
+using the application/mercurial-0.1 media type.
+"""
+def __init__(self, gen=None):
+self.gen = gen
 
 class pushres(object):
 """wireproto reply: success with simple integer return
@@ -802,7 +813,7 @@
   missingheads=repo.heads())
 cg = changegroupmod.makechangegroup(repo, outgoing, '01', 'serve')
 gen = iter(lambda: cg.read(32768), '')
-return streamres(gen=gen, v1compressible=True)
+return streamres(gen=gen)
 
 @wireprotocommand('changegroupsubset', 'bases heads')
 def changegroupsubset(repo, proto, bases, heads):
@@ -812,7 +823,7 @@
   missingheads=heads)
 cg = changegroupmod.makechangegroup(repo, outgoing, '01', 'serve')
 gen = iter(lambda: cg.read(32768), '')
-return streamres(gen=gen, v1compressible=True)
+return streamres(gen=gen)
 
 @wireprotocommand('debugwireargs', 'one two *')
 def debugwireargs(repo, proto, one, two, others):
@@ -877,8 +888,8 @@
 advargs.append(('hint', exc.hint))
 bundler.addpart(bundle2.bundlepart('error:abort',
manargs, advargs))
-return streamres(gen=bundler.getchunks(), v1compressible=True)
-return streamres(gen=chunks, v1compressible=True)
+return streamres(gen=bundler.getchunks())
+return streamres(gen=chunks)
 
 @wireprotocommand('heads')
 def heads(repo, proto):
@@ -955,7 +966,7 @@
 capability with a value representing the version and flags of the repo
 it is serving. Client checks to see if it understands the format.
 '''
-return streamres(streamclone.generatev1wireproto(repo))
+return streamres_legacy(streamclone.generatev1wireproto(repo))
 
 @wireprotocommand('unbundle', 'heads')
 def unbundle(repo, proto, heads):
@@ -990,7 +1001,7 @@
 if util.safehasattr(r, 'addpart'):
 # The return looks streamable, we are in the bundle2 case and
 # should return a stream.
-return streamres(gen=r.getchunks())
+return streamres_legacy(gen=r.getchunks())
 return pushres(r)
 
 finally:
@@ -1054,4 +1065,4 @@
manargs, advargs))
 except error.PushRaced as exc:
 bundler.newpart('error:pushraced', [('message', str(exc))])
-return streamres(gen=bundler.getchunks())
+return streamres_legacy(gen=bundler.getchunks())
diff --git a/mercurial/sshserver.py b/mercurial/sshserver.py
--- a/mercurial/sshserver.py
+++ b/mercurial/sshserver.py
@@ -105,6 +105,7 @@
 handlers = {
 str: sendresponse,
 wireproto.streamres: sendstream,
+wireproto.streamres_legacy: sendstream,
 wireproto.pushres: sendpushresponse,
 wireproto.pusherr: sendpusherror,
 wireproto.ooberror: sendooberror,
diff --git a/mercurial/hgweb/protocol.py b/mercurial/hgweb/protocol.py
--- a/mercurial/hgweb/protocol.py
+++ b/mercurial/hgweb/protocol.py
@@ -102,25 +102,20 @@
 urlreq.quote(self.req.env.get('REMOTE_HOST', '')),
 urlreq.quote(self.req.env.get('REMOTE_USER', '')))
 
-def responsetype(self, v1compressible=False):
+def 

D1918: localrepo: run cache-warming transaction callback before report callback

[martinvonz (Martin von Zweigbergk)]

This revision was automatically updated to reflect the committed changes.
Closed by commit rHG3a3b59bbe7ce: localrepo: run cache-warming transaction 
callback before report callback (authored by martinvonz, committed by ).

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1918?vs=4956=4958

REVISION DETAIL
  https://phab.mercurial-scm.org/D1918

AFFECTED FILES
  mercurial/localrepo.py
  tests/test-obsolete-changeset-exchange.t

CHANGE DETAILS

diff --git a/tests/test-obsolete-changeset-exchange.t 
b/tests/test-obsolete-changeset-exchange.t
--- a/tests/test-obsolete-changeset-exchange.t
+++ b/tests/test-obsolete-changeset-exchange.t
@@ -171,6 +171,6 @@
   bundle2-input-part: total payload size 24
   bundle2-input-bundle: 2 parts total
   checking for updated bookmarks
+  updating the branch cache
   new changesets bec0734cd68e
-  updating the branch cache
   (run 'hg heads' to see heads, 'hg merge' to merge)
diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -1322,7 +1322,11 @@
   **pycompat.strkwargs(hookargs))
 reporef()._afterlock(hookfunc)
 tr.addfinalize('txnclose-hook', txnclosehook)
-tr.addpostclose('warms-cache', self._buildcacheupdater(tr))
+# Include a leading "-" to make it happen before the transaction 
summary
+# reports registered via scmutil.registersummarycallback() whose names
+# are 00-txnreport etc. That way, the caches will be warm when the
+# callbacks run.
+tr.addpostclose('-warm-cache', self._buildcacheupdater(tr))
 def txnaborthook(tr2):
 """To be run if transaction is aborted
 """



To: martinvonz, #hg-reviewers, durin42
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1917: scmutil: 0-pad transaction report callback category

[martinvonz (Martin von Zweigbergk)]

This revision was automatically updated to reflect the committed changes.
Closed by commit rHG963a611b2f39: scmutil: 0-pad transaction report callback 
category (authored by martinvonz, committed by ).

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1917?vs=4955=4957

REVISION DETAIL
  https://phab.mercurial-scm.org/D1917

AFFECTED FILES
  mercurial/scmutil.py

CHANGE DETAILS

diff --git a/mercurial/scmutil.py b/mercurial/scmutil.py
--- a/mercurial/scmutil.py
+++ b/mercurial/scmutil.py
@@ -1247,7 +1247,7 @@
 if filtername:
 repo = repo.filtered(filtername)
 func(repo, tr)
-newcat = '%2i-txnreport' % len(categories)
+newcat = '%02i-txnreport' % len(categories)
 otr.addpostclose(newcat, wrapped)
 categories.append(newcat)
 return wrapped



To: martinvonz, #hg-reviewers, durin42
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

mercurial@35746: 22 new changesets

[Mercurial Commits]

22 new changesets in mercurial:

https://www.mercurial-scm.org/repo/hg/rev/2a7e777c9eed
changeset:   35725:2a7e777c9eed
user:Boris Feld 
date:Wed Jan 17 16:52:13 2018 +0100
summary: write: add the possibility to pass keyword argument from batchget 
to vfs

https://www.mercurial-scm.org/repo/hg/rev/45b678bf3a78
changeset:   35726:45b678bf3a78
user:Boris Feld 
date:Wed Jan 17 17:07:55 2018 +0100
summary: atomicupdate: add an experimental option to use atomictemp when 
updating

https://www.mercurial-scm.org/repo/hg/rev/05c70675e5b9
changeset:   35727:05c70675e5b9
user:Pulkit Goyal <7895pul...@gmail.com>
date:Fri Jan 19 14:10:18 2018 +0530
summary: blackbox: don't unpack the list while passing into str.join()

https://www.mercurial-scm.org/repo/hg/rev/22a877215ea1
changeset:   35728:22a877215ea1
user:Paul Morelle 
date:Fri Jan 19 08:35:22 2018 +0100
summary: debugdeltachain: cleanup the double call to _slicechunk

https://www.mercurial-scm.org/repo/hg/rev/7415cc923613
changeset:   35729:7415cc923613
user:Matt Harbison 
date:Fri Jan 19 00:18:45 2018 -0500
summary: test-blackbox: stabilize for Windows

https://www.mercurial-scm.org/repo/hg/rev/05d415790761
changeset:   35730:05d415790761
user:Boris Feld 
date:Thu Jan 18 16:47:14 2018 +0100
summary: debugdownload: read repository hgrc if there is one

https://www.mercurial-scm.org/repo/hg/rev/f58245b9e3ea
changeset:   35731:f58245b9e3ea
user:Matt Harbison 
date:Sun Jan 14 17:00:24 2018 -0500
summary: lfs: add the '{lfsattrs}' template keyword to '{lfs_files}'

https://www.mercurial-scm.org/repo/hg/rev/10e62d5efa73
changeset:   35732:10e62d5efa73
user:Matt Harbison 
date:Thu Jan 18 15:11:34 2018 -0500
summary: lfs: default to not using workers for upload/download

https://www.mercurial-scm.org/repo/hg/rev/3d48ae1aaa5e
changeset:   35733:3d48ae1aaa5e
user:Matt Harbison 
date:Thu Jan 18 15:59:21 2018 -0500
summary: lfs: default the User-Agent header for blob transfers to 'git-lfs'

https://www.mercurial-scm.org/repo/hg/rev/b4e1d0654736
changeset:   35734:b4e1d0654736
user:Matt Harbison 
date:Thu Jan 18 18:04:56 2018 -0500
summary: lfs: dump the full response on httperror in debug mode

https://www.mercurial-scm.org/repo/hg/rev/693e3bcae19e
changeset:   35735:693e3bcae19e
user:Matt Harbison 
date:Thu Jan 18 21:18:10 2018 -0500
summary: lfs: defer registering the pre-push hook until blobs are committed

https://www.mercurial-scm.org/repo/hg/rev/29f57ce416ed
changeset:   35736:29f57ce416ed
user:Yuya Nishihara 
date:Fri Jan 19 21:39:11 2018 +0900
summary: localrepo: micro-optimize __len__() to bypass repoview

https://www.mercurial-scm.org/repo/hg/rev/d99b07bc69fb
changeset:   35737:d99b07bc69fb
user:Paul Morelle 
date:Sun Jan 14 14:36:22 2018 +0100
summary: revlog: refactor out _finddeltainfo from _addrevision

https://www.mercurial-scm.org/repo/hg/rev/f90f6fd130c1
changeset:   35738:f90f6fd130c1
user:Paul Morelle 
date:Sun Jan 14 21:28:12 2018 +0100
summary: revlog: group delta computation methods under _deltacomputer object

https://www.mercurial-scm.org/repo/hg/rev/9eb5c400f488
changeset:   35739:9eb5c400f488
user:Yuya Nishihara 
date:Sun Jan 14 13:28:20 2018 +0900
summary: fileset: move import of match module to top

https://www.mercurial-scm.org/repo/hg/rev/06a757b9e334
changeset:   35740:06a757b9e334
user:Yuya Nishihara 
date:Sun Jan 14 13:33:56 2018 +0900
summary: minifileset: unify handling of symbol and string patterns

https://www.mercurial-scm.org/repo/hg/rev/73432eee0ac4
changeset:   35741:73432eee0ac4
user:Yuya Nishihara 
date:Sun Jan 14 13:29:15 2018 +0900
summary: fileset: add kind:pat operator

https://www.mercurial-scm.org/repo/hg/rev/7a1806e0daea
changeset:   35742:7a1806e0daea
user:Hollis Blanchard 
date:Thu Jan 18 13:33:21 2018 -0800
summary: sparse: --include 'dir1/dir2' should not include 'dir1/*'

https://www.mercurial-scm.org/repo/hg/rev/3c2a6246fd63
changeset:   35743:3c2a6246fd63
user:Yuya Nishihara 
date:Tue Jan 16 21:46:17 2018 +0900
summary: log: fix typo in comment about _matchfiles()

https://www.mercurial-scm.org/repo/hg/rev/8685192a8733
changeset:   35744:8685192a8733
user:Yuya Nishihara 
date:Tue Jan 16 23:50:01 2018 +0900
summary:

D1856: * wireproto: support for pullbundles

[durin42 (Augie Fackler)]

durin42 added a comment.


  I like where this is headed, but we'll have to plan to land this early in the 
next cycle, as I'm too chicken to land a feature with protocol implications on 
the last day of work before we freeze

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1856

To: joerg.sonnenberger, #hg-reviewers, indygreg
Cc: indygreg, durin42, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 04 of 15] streamclone: define first iteration of version 2 of stream format

[Augie Fackler]

On Fri, Jan 19, 2018 at 09:08:48PM +0100, Boris Feld wrote:
> # HG changeset patch
> # User Boris Feld 
> # Date 1516232936 -3600
> #  Thu Jan 18 00:48:56 2018 +0100
> # Node ID 4ee91fb55e208e8b139595ce9c2cae25aa9c54ea
> # Parent  b80a8e39ac9bf984c25a666bd7f6c47d876d26af
> # EXP-Topic b2-stream
> # Available At https://bitbucket.org/octobus/mercurial-devel/
> #  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
> 4ee91fb55e20
> streamclone: define first iteration of version 2 of stream format

This is a good start of a series, but:

1) patch 3 is begging for doctests on the varint scheme

2) This patch should probably include help/internals/ documentation on
   the format, rather than only encoding it in a docstring

3) Patches that claim to be a big performance win, but don't include
   any concrete testing numbers.

I think at this point we're going to admit defeat in the name of
getting an RC release done before I go to bed tonight. Performance
information I'd like to see in any v3 of this series:

1) comparison between the new streaming clone and the existing one on small 
repos

2) comparison on a medium repo with few branches (the hg repo could be good for 
this)

3) comparison on a large repo with many heads (might need to use 
contrib/synthrepo to make something for this?)

4) comparison on a large repo with many named branches (pypy?)

5) comparison on mozilla-central

I'm gathering that this is important work for you all for an important
client or set of clients, and I'm sorry we're not going to manage it
today. In the future, you could potentially sidestep some of this
frustration by giving people a heads up earlier than a
performance-critical 14 patch series with a new wireproto format.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1918: localrepo: run cache-warming transaction callback before report callback

[martinvonz (Martin von Zweigbergk)]

martinvonz created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  See in-code comment for details.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1918

AFFECTED FILES
  mercurial/localrepo.py
  tests/test-obsolete-changeset-exchange.t

CHANGE DETAILS

diff --git a/tests/test-obsolete-changeset-exchange.t 
b/tests/test-obsolete-changeset-exchange.t
--- a/tests/test-obsolete-changeset-exchange.t
+++ b/tests/test-obsolete-changeset-exchange.t
@@ -171,6 +171,6 @@
   bundle2-input-part: total payload size 24
   bundle2-input-bundle: 2 parts total
   checking for updated bookmarks
+  updating the branch cache
   new changesets bec0734cd68e
-  updating the branch cache
   (run 'hg heads' to see heads, 'hg merge' to merge)
diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -1320,7 +1320,11 @@
   **pycompat.strkwargs(hookargs))
 reporef()._afterlock(hookfunc)
 tr.addfinalize('txnclose-hook', txnclosehook)
-tr.addpostclose('warms-cache', self._buildcacheupdater(tr))
+# Include a leading "-" to make it happen before the transaction 
summary
+# reports registered via scmutil.registersummarycallback() whose names
+# are 00-txnreport etc. That way, the caches will be warm when the
+# callbacks run.
+tr.addpostclose('-warm-cache', self._buildcacheupdater(tr))
 def txnaborthook(tr2):
 """To be run if transaction is aborted
 """



To: martinvonz, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1917: scmutil: 0-pad transaction report callback category

[martinvonz (Martin von Zweigbergk)]

martinvonz created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  Before this patch, the transaction name was '%2i-txnreport', which
  means the first one would be ' 0-txnreport'. It seems more intuitive
  for sorting purposes (the callbacks are called in lexicographical
  order) to make it 0-padded.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1917

AFFECTED FILES
  mercurial/scmutil.py

CHANGE DETAILS

diff --git a/mercurial/scmutil.py b/mercurial/scmutil.py
--- a/mercurial/scmutil.py
+++ b/mercurial/scmutil.py
@@ -1247,7 +1247,7 @@
 if filtername:
 repo = repo.filtered(filtername)
 func(repo, tr)
-newcat = '%2i-txnreport' % len(categories)
+newcat = '%02i-txnreport' % len(categories)
 otr.addpostclose(newcat, wrapped)
 categories.append(newcat)
 return wrapped



To: martinvonz, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 15 of 15] streamclone: also stream caches to the client

[Augie Fackler]

On Fri, Jan 19, 2018 at 09:08:59PM +0100, Boris Feld wrote:
> # HG changeset patch
> # User Boris Feld 
> # Date 1516233012 -3600
> #  Thu Jan 18 00:50:12 2018 +0100
> # Node ID cc93d342d0a692565edc6a1c8cf8acdea36a0980
> # Parent  fec6950ccabdd6d93484732b341ae06697954890
> # EXP-Topic b2-stream
> # Available At https://bitbucket.org/octobus/mercurial-devel/
> #  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
> cc93d342d0a6
> streamclone: also stream caches to the client
>
> When stream clone is used over bundle2, relevant cache files are also 
> streamed.
> This is expected to be a massive performance win for clone since no important
> cache will have to be recomputed.

Some numbers here would be nice.

>
> diff --git a/mercurial/streamclone.py b/mercurial/streamclone.py
> --- a/mercurial/streamclone.py
> +++ b/mercurial/streamclone.py
> @@ -11,10 +11,12 @@ import contextlib
>  import os
>  import struct
>  import tempfile
> +import warnings
>
>  from .i18n import _
>  from . import (
>  branchmap,
> +cacheutil,
>  error,
>  phases,
>  store,
> @@ -435,6 +437,10 @@ class streamcloneapplier(object):
>  _fileappend = 0 # append only file
>  _filefull = 1   # full snapshot file
>
> +# Source of the file
> +_srcstore = 's' # store (svfs)
> +_srccache = 'c' # cache (cache)
> +
>  # This is it's own function so extensions can override it.
>  def _walkstreamfullstorefiles(repo):
>  """list snapshot file from the store"""
> @@ -443,12 +449,12 @@ def _walkstreamfullstorefiles(repo):
>  fnames.append('phaseroots')
>  return fnames
>
> -def _filterfull(entry, copy, vfs):
> +def _filterfull(entry, copy, vfsmap):
>  """actually copy the snapshot files"""
> -name, ftype, data = entry
> +src, name, ftype, data = entry
>  if ftype != _filefull:
>  return entry
> -return (name, ftype, copy(vfs.join(name)))
> +return (src, name, ftype, copy(vfsmap[src].join(name)))
>
>  @contextlib.contextmanager
>  def maketempcopies():
> @@ -466,19 +472,33 @@ def maketempcopies():
>  for tmp in files:
>  util.tryunlink(tmp)
>
> +def _makemap(repo):
> +"""make a (src -> vfs) map for the repo"""
> +vfsmap = {
> +_srcstore: repo.svfs,
> +_srccache: repo.cachevfs,
> +}
> +# we keep repo.vfs out of the on purpose, ther are too many danger there
> +# (eg: .hg/hgrc)
> +assert repo.vfs not in vfsmap.values()
> +
> +return vfsmap
> +
>  def _emit(repo, entries, totalfilesize):
>  """actually emit the stream bundle"""
> -vfs = repo.svfs
> +vfsmap = _makemap(repo)
>  progress = repo.ui.progress
>  progress(_('bundle'), 0, total=totalfilesize, unit=_('bytes'))
>  with maketempcopies() as copy:
>  try:
>  # copy is delayed until we are in the try
> -entries = [_filterfull(e, copy, vfs) for e in entries]
> +entries = [_filterfull(e, copy, vfsmap) for e in entries]
>  yield None # this release the lock on the repository
>  seen = 0
>
> -for name, ftype, data in entries:
> +for src, name, ftype, data in entries:
> +vfs = vfsmap[src]
> +yield src
>  yield util.uvarintencode(len(name))
>  if ftype == _fileappend:
>  fp = vfs(name)
> @@ -507,10 +527,11 @@ def generatev2(repo):
>  """Emit content for version 2 of a streaming clone.
>
>  the data stream consists the following entries:
> -1) A varint containing the length of the filename
> -2) A varint containing the length of file data
> -3) N bytes containing the filename (the internal, store-agnostic form)
> -4) N bytes containing the file data
> +1) A char representing the file destination (eg: store or cache)
> +2) A varint containing the length of the filename
> +3) A varint containing the length of file data
> +4) N bytes containing the filename (the internal, store-agnostic form)
> +5) N bytes containing the file data
>
>  Returns a 3-tuple of (file count, file size, data iterator).
>  """
> @@ -523,12 +544,16 @@ def generatev2(repo):
>  repo.ui.debug('scanning\n')
>  for name, ename, size in _walkstreamfiles(repo):
>  if size:
> -entries.append((name, _fileappend, size))
> +entries.append((_srcstore, name, _fileappend, size))
>  totalfilesize += size
>  for name in _walkstreamfullstorefiles(repo):
>  if repo.svfs.exists(name):
>  totalfilesize += repo.svfs.lstat(name).st_size
> -entries.append((name, _filefull, None))
> +entries.append((_srcstore, name, _filefull, None))
> +for name in cacheutil.cachetocopy(repo):
> +if repo.cachevfs.exists(name):
> +totalfilesize += 

Re: [PATCH 08 of 15] clone: allow bundle2's stream clone with 'server.disablefullbundle'

[Augie Fackler]

On Fri, Jan 19, 2018 at 09:08:52PM +0100, Boris Feld wrote:
> # HG changeset patch
> # User Boris Feld 
> # Date 1516203512 -3600
> #  Wed Jan 17 16:38:32 2018 +0100
> # Node ID 77a0634011b5bc89472a134c5ea2b5623f6ca273
> # Parent  b11f4652647e791727e14c94a0ccb7c0282c5a29
> # EXP-Topic b2-stream
> # Available At https://bitbucket.org/octobus/mercurial-devel/
> #  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
> 77a0634011b5
> clone: allow bundle2's stream clone with 'server.disablefullbundle'

That is, prior to this patch server.disablefullbundle also banned
streaming clones even if streaming clones were enabled? Did I get that
right?

>
> The previous check was a bit too strict and would not recognize a get bundle
> not requesting changegroup.
>
> diff --git a/mercurial/wireproto.py b/mercurial/wireproto.py
> --- a/mercurial/wireproto.py
> +++ b/mercurial/wireproto.py
> @@ -855,10 +855,11 @@ def getbundle(repo, proto, others):
>  if repo.ui.configbool('server', 'disablefullbundle'):
>  # Check to see if this is a full clone.
>  clheads = set(repo.changelog.heads())
> +changegroup = opts.get('cg', True)
>  heads = set(opts.get('heads', set()))
>  common = set(opts.get('common', set()))
>  common.discard(nullid)
> -if not common and clheads == heads:
> +if changegroup and not common and clheads == heads:
>  raise error.Abort(
>  _('server has pull-based clones disabled'),
>  hint=_('remove --pull if specified or upgrade 
> Mercurial'))
> ___
> Mercurial-devel mailing list
> Mercurial-devel@mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

mercurial@35724: 25 new changesets

[Mercurial Commits]

25 new changesets in mercurial:

https://www.mercurial-scm.org/repo/hg/rev/a71316bfac87
changeset:   35700:a71316bfac87
user:Augie Fackler 
date:Wed Jan 17 17:59:12 2018 -0500
summary: python3: whitelist two more passing tests

https://www.mercurial-scm.org/repo/hg/rev/c5d220a621e7
changeset:   35701:c5d220a621e7
user:Phil Cohen 
date:Wed Dec 27 17:38:28 2017 -0600
summary: rebase: don't run IMM if running rebase in a transaction

https://www.mercurial-scm.org/repo/hg/rev/c0439e11af16
changeset:   35702:c0439e11af16
user:Phil Cohen 
date:Thu Jan 04 21:36:58 2018 -0800
summary: filemerge: fix backing up an in-memory file to a custom location

https://www.mercurial-scm.org/repo/hg/rev/9a50ffd15b25
changeset:   35703:9a50ffd15b25
user:Phil Cohen 
date:Thu Jan 04 21:37:03 2018 -0800
summary: filemerge: only write in-memory backup during premerge

https://www.mercurial-scm.org/repo/hg/rev/41ef02ba329b
changeset:   35704:41ef02ba329b
user:Pulkit Goyal <7895pul...@gmail.com>
date:Mon Jan 08 19:41:47 2018 +0530
summary: merge: add `--abort` flag which can abort the merge

https://www.mercurial-scm.org/repo/hg/rev/8cdb671dbd0b
changeset:   35705:8cdb671dbd0b
user:Gregory Szorc 
date:Mon Jan 15 15:20:02 2018 -0800
summary: wireproto: drop support for reader interface from streamres (API)

https://www.mercurial-scm.org/repo/hg/rev/5748f404dad3
changeset:   35706:5748f404dad3
user:Martin von Zweigbergk 
date:Sun Jan 14 23:37:06 2018 -0800
summary: repair: drop unnecessary phase cache invalidation

https://www.mercurial-scm.org/repo/hg/rev/54074a82e050
changeset:   35707:54074a82e050
user:Martin von Zweigbergk 
date:Sun Jan 14 14:39:17 2018 -0800
summary: repair: filter out unknown revisions from phasecache within 
transaction

https://www.mercurial-scm.org/repo/hg/rev/03e921942163
changeset:   35708:03e921942163
user:Martin von Zweigbergk 
date:Wed Jan 10 14:00:23 2018 -0800
summary: transaction: register summary callbacks only at start of 
transaction (BC)

https://www.mercurial-scm.org/repo/hg/rev/1a09dad8b85a
changeset:   35709:1a09dad8b85a
user:Martin von Zweigbergk 
date:Sun Jan 14 23:59:17 2018 -0800
summary: evolution: report new unstable changesets

https://www.mercurial-scm.org/repo/hg/rev/5cd60b0587a8
changeset:   35710:5cd60b0587a8
user:Martin von Zweigbergk 
date:Sun Jan 14 00:02:40 2018 -0800
summary: evolution: make reporting of new unstable changesets optional

https://www.mercurial-scm.org/repo/hg/rev/35a0f6f31eef
changeset:   35711:35a0f6f31eef
user:Boris Feld 
date:Tue Jan 16 14:08:54 2018 +0100
summary: update: display the obsfate of hidden revision we update to

https://www.mercurial-scm.org/repo/hg/rev/a1a5c3842b6f
changeset:   35712:a1a5c3842b6f
user:Boris Feld 
date:Tue Jan 16 14:28:57 2018 +0100
summary: bookmarks: display the obsfate of hidden revision we create a 
bookmark on

https://www.mercurial-scm.org/repo/hg/rev/7ffbd911dbc9
changeset:   35713:7ffbd911dbc9
user:Pulkit Goyal <7895pul...@gmail.com>
date:Thu Jan 18 19:40:17 2018 +0530
summary: merge: use public interface ms.localctx instead of ms._local

https://www.mercurial-scm.org/repo/hg/rev/113281667205
changeset:   35714:113281667205
user:Gregory Szorc 
date:Mon Dec 18 20:44:59 2017 -0800
summary: githelp: vendor Facebook authored extension

https://www.mercurial-scm.org/repo/hg/rev/8dbd000f7de9
changeset:   35715:8dbd000f7de9
user:Gregory Szorc 
date:Mon Dec 18 20:51:20 2017 -0800
summary: githelp: improve help for `git add`

https://www.mercurial-scm.org/repo/hg/rev/05b8adf38c55
changeset:   35716:05b8adf38c55
user:Gregory Szorc 
date:Fri Dec 22 18:38:29 2017 -0700
summary: githelp: recommend `hg import` for `git am`

https://www.mercurial-scm.org/repo/hg/rev/5edfead8cc95
changeset:   35717:5edfead8cc95
user:Gregory Szorc 
date:Mon Dec 18 20:56:01 2017 -0800
summary: githelp: remove reference to tweakdefaults

https://www.mercurial-scm.org/repo/hg/rev/a10a0d5561a9
changeset:   35718:a10a0d5561a9
user:Gregory Szorc 
date:Mon Dec 18 20:58:00 2017 -0800
summary: githelp: replace suggestion of `hg record`

https://www.mercurial-scm.org/repo/hg/rev/a4cd8f527a7f
changeset:   35719:a4cd8f527a7f
user:Gregory Szorc 
date:Mon Dec 18 21:02:49 2017 -0800

D1886: commands: replace map() with list comprehension

[durin42 (Augie Fackler)]

durin42 updated this revision to Diff 4947.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1886?vs=4876=4947

REVISION DETAIL
  https://phab.mercurial-scm.org/D1886

AFFECTED FILES
  mercurial/commands.py

CHANGE DETAILS

diff --git a/mercurial/commands.py b/mercurial/commands.py
--- a/mercurial/commands.py
+++ b/mercurial/commands.py
@@ -1219,7 +1219,7 @@
 raise error.Abort(_("--base is incompatible with specifying "
"a destination"))
 common = [repo.lookup(rev) for rev in base]
-heads = map(repo.lookup, revs) if revs else None
+heads = [repo.lookup(r) for r in revs] if revs else None
 outgoing = discovery.outgoing(repo, common, heads)
 else:
 dest = ui.expandpath(dest or 'default-push', dest or 'default')



To: durin42, #hg-reviewers, pulkit
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1899: localrepo: pass transaction kwargs as strings, not bytes

[durin42 (Augie Fackler)]

durin42 updated this revision to Diff 4951.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1899?vs=4889=4951

REVISION DETAIL
  https://phab.mercurial-scm.org/D1899

AFFECTED FILES
  mercurial/localrepo.py

CHANGE DETAILS

diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -1327,7 +1327,7 @@
 """To be run if transaction is aborted
 """
 reporef().hook('txnabort', throw=False, txnname=desc,
-   **tr2.hookargs)
+   **pycompat.strkwargs(tr2.hookargs))
 tr.addabort('txnabort-hook', txnaborthook)
 # avoid eager cache invalidation. in-memory data should be identical
 # to stored data if transaction has no error.



To: durin42, #hg-reviewers, pulkit
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1907: tests: fix a missed b prefix in a test extension in test-strip.t

[durin42 (Augie Fackler)]

durin42 updated this revision to Diff 4953.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1907?vs=4912=4953

REVISION DETAIL
  https://phab.mercurial-scm.org/D1907

AFFECTED FILES
  tests/test-strip.t

CHANGE DETAILS

diff --git a/tests/test-strip.t b/tests/test-strip.t
--- a/tests/test-strip.t
+++ b/tests/test-strip.t
@@ -899,7 +899,7 @@
   > transaction = orig(repo, desc, *args, **kwargs)
   > # warm up the phase cache
   > list(repo.revs(b"not public()"))
-  > if desc != 'strip':
+  > if desc != b'strip':
   >  transaction.addpostclose(b"phase invalidation test", test)
   > return transaction
   > def extsetup(ui):



To: durin42, #hg-reviewers, pulkit
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1898: localrepo: consistently use native str when __dict__ is involved

[durin42 (Augie Fackler)]

durin42 updated this revision to Diff 4950.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1898?vs=4888=4950

REVISION DETAIL
  https://phab.mercurial-scm.org/D1898

AFFECTED FILES
  mercurial/localrepo.py

CHANGE DETAILS

diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -1569,7 +1569,8 @@
 def _refreshfilecachestats(self, tr):
 """Reload stats of cached files so that they are flagged as valid"""
 for k, ce in self._filecache.items():
-if k == 'dirstate' or k not in self.__dict__:
+k = pycompat.sysstr(k)
+if k == r'dirstate' or k not in self.__dict__:
 continue
 ce.refresh()
 



To: durin42, #hg-reviewers, pulkit
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1909: cmdutil: add a kludge to make bytes repr() the same on 2 and 3

[durin42 (Augie Fackler)]

durin42 updated this revision to Diff 4954.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1909?vs=4914=4954

REVISION DETAIL
  https://phab.mercurial-scm.org/D1909

AFFECTED FILES
  mercurial/cmdutil.py

CHANGE DETAILS

diff --git a/mercurial/cmdutil.py b/mercurial/cmdutil.py
--- a/mercurial/cmdutil.py
+++ b/mercurial/cmdutil.py
@@ -2077,6 +2077,19 @@
 
 return changeset_templater(ui, repo, spec, match, opts, buffered)
 
+class _regrettablereprbytes(bytes):
+"""Bytes subclass that makes the repr the same on Python 3 as Python 2.
+
+This is a huge hack.
+"""
+def __repr__(self):
+return repr(pycompat.sysstr(self))
+
+def _maybebytestr(v):
+if pycompat.ispy3 and isinstance(v, bytes):
+return _regrettablereprbytes(v)
+return v
+
 def showmarker(fm, marker, index=None):
 """utility function to display obsolescence marker in a readable way
 
@@ -2095,7 +2108,8 @@
 fm.write('date', '(%s) ', fm.formatdate(marker.date()))
 meta = marker.metadata().copy()
 meta.pop('date', None)
-fm.write('metadata', '{%s}', fm.formatdict(meta, fmt='%r: %r', sep=', '))
+smeta = {_maybebytestr(k): _maybebytestr(v) for k, v in meta.iteritems()}
+fm.write('metadata', '{%s}', fm.formatdict(smeta, fmt='%r: %r', sep=', '))
 fm.plain('\n')
 
 def finddate(ui, repo, date):



To: durin42, #hg-reviewers, yuja
Cc: yuja, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1906: revlog: correct type in check to verify rawtext is immutable

[durin42 (Augie Fackler)]

durin42 updated this revision to Diff 4952.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1906?vs=4911=4952

REVISION DETAIL
  https://phab.mercurial-scm.org/D1906

AFFECTED FILES
  mercurial/revlog.py

CHANGE DETAILS

diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -2099,7 +2099,7 @@
 if alwayscache and rawtext is None:
 rawtext = deltacomputer._buildtext(revinfo, fh)
 
-if type(rawtext) == str: # only accept immutable objects
+if type(rawtext) == bytes: # only accept immutable objects
 self._cache = (node, curr, rawtext)
 self._chainbasecache[curr] = chainbase
 return node



To: durin42, indygreg, #hg-reviewers, pulkit
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1885: commands: rewrite legacy ternary operator hack using modern syntax

[durin42 (Augie Fackler)]

durin42 updated this revision to Diff 4946.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1885?vs=4875=4946

REVISION DETAIL
  https://phab.mercurial-scm.org/D1885

AFFECTED FILES
  mercurial/commands.py

CHANGE DETAILS

diff --git a/mercurial/commands.py b/mercurial/commands.py
--- a/mercurial/commands.py
+++ b/mercurial/commands.py
@@ -1219,7 +1219,7 @@
 raise error.Abort(_("--base is incompatible with specifying "
"a destination"))
 common = [repo.lookup(rev) for rev in base]
-heads = revs and map(repo.lookup, revs) or None
+heads = map(repo.lookup, revs) if revs else None
 outgoing = discovery.outgoing(repo, common, heads)
 else:
 dest = ui.expandpath(dest or 'default-push', dest or 'default')



To: durin42, #hg-reviewers, pulkit
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1893: context: use native string when peeking in __dict__

[durin42 (Augie Fackler)]

durin42 updated this revision to Diff 4949.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1893?vs=4883=4949

REVISION DETAIL
  https://phab.mercurial-scm.org/D1893

AFFECTED FILES
  mercurial/context.py

CHANGE DETAILS

diff --git a/mercurial/context.py b/mercurial/context.py
--- a/mercurial/context.py
+++ b/mercurial/context.py
@@ -1051,7 +1051,7 @@
 # renamed filectx won't have a filelog yet, so set it
 # from the cache to save time
 for p in pl:
-if not '_filelog' in p.__dict__:
+if not r'_filelog' in p.__dict__:
 p._filelog = getlog(p.path())
 
 return pl



To: durin42, #hg-reviewers, pulkit
Cc: pulkit, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1889: tests: bytestring-ify all the adhoc extensions in test-strip.t

[durin42 (Augie Fackler)]

durin42 updated this revision to Diff 4948.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1889?vs=4879=4948

REVISION DETAIL
  https://phab.mercurial-scm.org/D1889

AFFECTED FILES
  tests/test-strip.t

CHANGE DETAILS

diff --git a/tests/test-strip.t b/tests/test-strip.t
--- a/tests/test-strip.t
+++ b/tests/test-strip.t
@@ -893,17 +893,17 @@
   > def test(transaction):
   > # observe cache inconsistency
   > try:
-  > [repo.changelog.node(r) for r in repo.revs("not public()")]
+  > [repo.changelog.node(r) for r in repo.revs(b"not public()")]
   > except IndexError:
-  > repo.ui.status("Index error!\n")
+  > repo.ui.status(b"Index error!\n")
   > transaction = orig(repo, desc, *args, **kwargs)
   > # warm up the phase cache
-  > list(repo.revs("not public()"))
+  > list(repo.revs(b"not public()"))
   > if desc != 'strip':
-  >  transaction.addpostclose("phase invalidation test", test)
+  >  transaction.addpostclose(b"phase invalidation test", test)
   > return transaction
   > def extsetup(ui):
-  > extensions.wrapfunction(localrepo.localrepository, "transaction",
+  > extensions.wrapfunction(localrepo.localrepository, b"transaction",
   > transactioncallback)
   > EOF
   $ hg up -C 2
@@ -930,9 +930,9 @@
   > class crashstriprepo(repo.__class__):
   > def transaction(self, desc, *args, **kwargs):
   > tr = super(crashstriprepo, self).transaction(desc, *args, 
**kwargs)
-  > if desc == 'strip':
-  > def crash(tra): raise error.Abort('boom')
-  > tr.addpostclose('crash', crash)
+  > if desc == b'strip':
+  > def crash(tra): raise error.Abort(b'boom')
+  > tr.addpostclose(b'crash', crash)
   > return tr
   > repo.__class__ = crashstriprepo
   > EOF
@@ -1175,16 +1175,16 @@
   > from mercurial import commands, registrar, repair
   > cmdtable = {}
   > command = registrar.command(cmdtable)
-  > @command('testdelayedstrip')
+  > @command(b'testdelayedstrip')
   > def testdelayedstrip(ui, repo):
   > def getnodes(expr):
   > return [repo.changelog.node(r) for r in repo.revs(expr)]
   > with repo.wlock():
   > with repo.lock():
-  > with repo.transaction('delayedstrip'):
-  > repair.delayedstrip(ui, repo, getnodes('B+I+Z+D+E'), 'J')
-  > repair.delayedstrip(ui, repo, getnodes('G+H+Z'), 'I')
-  > commands.commit(ui, repo, message='J', date='0 0')
+  > with repo.transaction(b'delayedstrip'):
+  > repair.delayedstrip(ui, repo, getnodes(b'B+I+Z+D+E'), b'J')
+  > repair.delayedstrip(ui, repo, getnodes(b'G+H+Z'), b'I')
+  > commands.commit(ui, repo, message=b'J', date=b'0 0')
   > EOF
   $ hg testdelayedstrip --config extensions.t=$TESTTMP/delayedstrip.py
   warning: orphaned descendants detected, not stripping 08ebfeb61bac, 
112478962961, 7fb047a69f22
@@ -1225,20 +1225,21 @@
   > from mercurial import registrar, scmutil
   > cmdtable = {}
   > command = registrar.command(cmdtable)
-  > @command('testnodescleanup')
+  > @command(b'testnodescleanup')
   > def testnodescleanup(ui, repo):
   > def nodes(expr):
   > return [repo.changelog.node(r) for r in repo.revs(expr)]
   > def node(expr):
   > return nodes(expr)[0]
   > with repo.wlock():
   > with repo.lock():
-  > with repo.transaction('delayedstrip'):
-  > mapping = {node('F'): [node('F2')],
-  >node('D'): [node('D2')],
-  >node('G'): [node('G2')]}
-  > scmutil.cleanupnodes(repo, mapping, 'replace')
-  > scmutil.cleanupnodes(repo, nodes('((B::)+I+Z)-D2'), 
'replace')
+  > with repo.transaction(b'delayedstrip'):
+  > mapping = {node(b'F'): [node(b'F2')],
+  >node(b'D'): [node(b'D2')],
+  >node(b'G'): [node(b'G2')]}
+  > scmutil.cleanupnodes(repo, mapping, b'replace')
+  > scmutil.cleanupnodes(repo, nodes(b'((B::)+I+Z)-D2'),
+  >  b'replace')
   > EOF
   $ hg testnodescleanup --config extensions.t=$TESTTMP/scmutilcleanup.py
   warning: orphaned descendants detected, not stripping 112478962961, 
1fc8102cda62, 26805aba1e60



To: durin42, #hg-reviewers, pulkit
Cc: pulkit, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 12 of 15] streamclone: tests phase exchange during stream clone

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1516238924 -3600
#  Thu Jan 18 02:28:44 2018 +0100
# Node ID 243565f94076531d0223abb4434864dda2c995d8
# Parent  e7a1641a31fbdb2f1e0960bc0090e2c7705a8d1c
# EXP-Topic b2-stream
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
243565f94076
streamclone: tests phase exchange during stream clone

We add a test dedicated to phases. As reported in issue 5648 stream from a non 
publishing
server is currently broken (does not preserve the phase). We'll fix it with 'v2'
support in the next changesets.

diff --git a/tests/test-clone-uncompressed.t b/tests/test-clone-uncompressed.t
--- a/tests/test-clone-uncompressed.t
+++ b/tests/test-clone-uncompressed.t
@@ -262,3 +262,71 @@ clone it
 #endif
   $ hg -R with-bookmarks bookmarks
  some-bookmark 1:c17445101a72
+
+Stream repository with phases
+-
+
+Clone as publishing
+
+  $ hg -R server phase -r 'all()'
+  0: draft
+  1: draft
+
+#if stream-legacy
+  $ hg clone --stream http://localhost:$HGPORT phase-publish
+  streaming all changes
+  1027 files to transfer, 96.3 KB of data
+  transferred 96.3 KB in * seconds (*) (glob)
+  searching for changes
+  no changes found
+  updating to branch default
+  1025 files updated, 0 files merged, 0 files removed, 0 files unresolved
+#endif
+#if stream-bundle2
+  $ hg clone --stream http://localhost:$HGPORT phase-publish
+  streaming all changes
+  1027 files to transfer, 96.3 KB of data
+  transferred 96.3 KB in * seconds (* */sec) (glob)
+  updating to branch default
+  1025 files updated, 0 files merged, 0 files removed, 0 files unresolved
+#endif
+  $ hg -R phase-publish phase -r 'all()'
+  0: public
+  1: public
+
+Clone as non publishing
+
+  $ cat << EOF >> server/.hg/hgrc
+  > [phases]
+  > publish = False
+  > EOF
+  $ killdaemons.py
+  $ hg -R server serve -p $HGPORT -d --pid-file=hg.pid
+  $ cat hg.pid >> $DAEMON_PIDS
+
+#if stream-legacy
+  $ hg clone --stream http://localhost:$HGPORT phase-no-publish
+  streaming all changes
+  1027 files to transfer, 96.3 KB of data
+  transferred 96.3 KB in * seconds (*) (glob)
+  searching for changes
+  no changes found
+  updating to branch default
+  1025 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  $ hg -R phase-no-publish phase -r 'all()'
+  0: public
+  1: public
+#endif
+#if stream-bundle2
+  $ hg clone --stream http://localhost:$HGPORT phase-no-publish
+  streaming all changes
+  1027 files to transfer, 96.3 KB of data
+  transferred 96.3 KB in * seconds (* */sec) (glob)
+  updating to branch default
+  1025 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  $ hg -R phase-no-publish phase -r 'all()'
+  0: public
+  1: public
+#endif
+
+  $ killdaemons.py
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 05 of 15] streamclone: rework canperformstreamclone

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1516232727 -3600
#  Thu Jan 18 00:45:27 2018 +0100
# Node ID 2ef371994c9c4525a525c95586691d43b04ecd33
# Parent  4ee91fb55e208e8b139595ce9c2cae25aa9c54ea
# EXP-Topic b2-stream
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
2ef371994c9c
streamclone: rework canperformstreamclone

There is code about bundle2 laying around in `canperformstreamclone` but not
put to any uses. As we discovered with the previous patch, streambundle 'v1'
won't work on bundle2 because they are readline based. So we jump to 'v2' as
the first expected supported version.

diff --git a/mercurial/streamclone.py b/mercurial/streamclone.py
--- a/mercurial/streamclone.py
+++ b/mercurial/streamclone.py
@@ -18,12 +18,11 @@ from . import (
 util,
 )
 
-def canperformstreamclone(pullop, bailifbundle2supported=False):
+def canperformstreamclone(pullop, bundle2=False):
 """Whether it is possible to perform a streaming clone as part of pull.
 
-``bailifbundle2supported`` will cause the function to return False if
-bundle2 stream clones are supported. It should only be called by the
-legacy stream clone code path.
+``bundle2`` will cause the function to consider stream clone through
+bundle2 and only through bundle2.
 
 Returns a tuple of (supported, requirements). ``supported`` is True if
 streaming clone is supported and False otherwise. ``requirements`` is
@@ -35,18 +34,18 @@ def canperformstreamclone(pullop, bailif
 
 bundle2supported = False
 if pullop.canusebundle2:
-if 'v1' in pullop.remotebundle2caps.get('stream', []):
+if 'v2' in pullop.remotebundle2caps.get('stream', []):
 bundle2supported = True
 # else
 # Server doesn't support bundle2 stream clone or doesn't support
 # the versions we support. Fall back and possibly allow legacy.
 
 # Ensures legacy code path uses available bundle2.
-if bailifbundle2supported and bundle2supported:
+if bundle2supported and not bundle2:
 return False, None
 # Ensures bundle2 doesn't try to do a stream clone if it isn't supported.
-#elif not bailifbundle2supported and not bundle2supported:
-#return False, None
+elif bundle2 and not bundle2supported:
+return False, None
 
 # Streaming clone only works on empty repositories.
 if len(repo):
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 04 of 15] streamclone: define first iteration of version 2 of stream format

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1516232936 -3600
#  Thu Jan 18 00:48:56 2018 +0100
# Node ID 4ee91fb55e208e8b139595ce9c2cae25aa9c54ea
# Parent  b80a8e39ac9bf984c25a666bd7f6c47d876d26af
# EXP-Topic b2-stream
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
4ee91fb55e20
streamclone: define first iteration of version 2 of stream format

(This patch is based on a first draft from Gregory Szorc, with deeper rework)

Version 1 of the stream clone format was invented many years ago and suffers
from a few deficiencies:

1) Filenames are stored in store-encoded (on filesystem) form rather than in
   their internal form. This makes future compatibility with new store
   filename encodings more difficult.
2) File entry "headers" consist of a newline of the file name followed by the
   string file size. Converting strings to integers is avoidable overhead. We
   can't store filenames with newlines (manifests have this limitation as
   well, so it isn't a major concern). But the big concern here is the
   necessity for readline(). Scanning for newlines means reading ahead and
   that means extra buffer allocations and slicing (in Python) and this makes
   performance suffer.
3) Filenames aren't compressed optimally. Filenames should be compressed well
   since there is a lot of repeated data. However, since they are scattered
   all over the stream (with revlog data in between), they typically fall
   outside the window size of the compressor and don't compress.
4) It can only exchange stored based content, being able to exchange caches
   too would be nice.
5) It is limited to a stream-based protocol and isn't suitable for an on-disk
   format for general repository reading because the offset of individual file
   entries requires scanning the entire file to find file records.

As part of enabling streaming clones to work in bundle2, #2 proved to have a
significant negative impact on performance. Since bundle2 provides the
opportunity to start fresh, Gregory Szorc figured he would take the
opportunity to invent a new streaming clone data format.

The new format devised in this series addresses #1, #2, and #4. It punts on #3
because it was complex without yielding a significant gain and on #5 because
devising a new store format that "packs" multiple revlogs into a single
"packed revlog" is massive scope bloat. However, this v2 format might be
suitable for streaming into a "packed revlog" with minimal processing. If it
works, great. If not, we can always invent stream format when it is needed.

This patch only introduces the bases of the format. We'll get it usable through
bundle2 first, then we'll extend the format in future patches to bring it to its
full potential (especially #4).

diff --git a/mercurial/streamclone.py b/mercurial/streamclone.py
--- a/mercurial/streamclone.py
+++ b/mercurial/streamclone.py
@@ -428,3 +428,115 @@ class streamcloneapplier(object):
 
 def apply(self, repo):
 return applybundlev1(repo, self._fh)
+
+def _emit(repo, entries, totalfilesize):
+"""actually emit the stream bundle"""
+progress = repo.ui.progress
+progress(_('bundle'), 0, total=totalfilesize, unit=_('bytes'))
+vfs = repo.svfs
+try:
+seen = 0
+for name, size in entries:
+yield util.uvarintencode(len(name))
+fp = vfs(name)
+try:
+yield util.uvarintencode(size)
+yield name
+if size <= 65536:
+chunks = (fp.read(size),)
+else:
+chunks = util.filechunkiter(fp, limit=size)
+for chunk in chunks:
+seen += len(chunk)
+progress(_('bundle'), seen, total=totalfilesize,
+ unit=_('bytes'))
+yield chunk
+finally:
+fp.close()
+finally:
+progress(_('bundle'), None)
+
+def generatev2(repo):
+"""Emit content for version 2 of a streaming clone.
+
+the data stream consists the following entries:
+1) A varint containing the length of the filename
+2) A varint containing the length of file data
+3) N bytes containing the filename (the internal, store-agnostic form)
+4) N bytes containing the file data
+
+Returns a 3-tuple of (file count, file size, data iterator).
+"""
+
+with repo.lock():
+
+entries = []
+totalfilesize = 0
+
+repo.ui.debug('scanning\n')
+for name, ename, size in _walkstreamfiles(repo):
+if size:
+entries.append((name, size))
+totalfilesize += size
+
+chunks = _emit(repo, entries, totalfilesize)
+
+return len(entries), totalfilesize, chunks
+
+def consumev2(repo, fp, filecount, filesize):
+"""Apply the contents from a version 2 streaming clone.
+
+Data 

[PATCH 15 of 15] streamclone: also stream caches to the client

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1516233012 -3600
#  Thu Jan 18 00:50:12 2018 +0100
# Node ID cc93d342d0a692565edc6a1c8cf8acdea36a0980
# Parent  fec6950ccabdd6d93484732b341ae06697954890
# EXP-Topic b2-stream
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
cc93d342d0a6
streamclone: also stream caches to the client

When stream clone is used over bundle2, relevant cache files are also streamed.
This is expected to be a massive performance win for clone since no important
cache will have to be recomputed.

diff --git a/mercurial/streamclone.py b/mercurial/streamclone.py
--- a/mercurial/streamclone.py
+++ b/mercurial/streamclone.py
@@ -11,10 +11,12 @@ import contextlib
 import os
 import struct
 import tempfile
+import warnings
 
 from .i18n import _
 from . import (
 branchmap,
+cacheutil,
 error,
 phases,
 store,
@@ -435,6 +437,10 @@ class streamcloneapplier(object):
 _fileappend = 0 # append only file
 _filefull = 1   # full snapshot file
 
+# Source of the file
+_srcstore = 's' # store (svfs)
+_srccache = 'c' # cache (cache)
+
 # This is it's own function so extensions can override it.
 def _walkstreamfullstorefiles(repo):
 """list snapshot file from the store"""
@@ -443,12 +449,12 @@ def _walkstreamfullstorefiles(repo):
 fnames.append('phaseroots')
 return fnames
 
-def _filterfull(entry, copy, vfs):
+def _filterfull(entry, copy, vfsmap):
 """actually copy the snapshot files"""
-name, ftype, data = entry
+src, name, ftype, data = entry
 if ftype != _filefull:
 return entry
-return (name, ftype, copy(vfs.join(name)))
+return (src, name, ftype, copy(vfsmap[src].join(name)))
 
 @contextlib.contextmanager
 def maketempcopies():
@@ -466,19 +472,33 @@ def maketempcopies():
 for tmp in files:
 util.tryunlink(tmp)
 
+def _makemap(repo):
+"""make a (src -> vfs) map for the repo"""
+vfsmap = {
+_srcstore: repo.svfs,
+_srccache: repo.cachevfs,
+}
+# we keep repo.vfs out of the on purpose, ther are too many danger there
+# (eg: .hg/hgrc)
+assert repo.vfs not in vfsmap.values()
+
+return vfsmap
+
 def _emit(repo, entries, totalfilesize):
 """actually emit the stream bundle"""
-vfs = repo.svfs
+vfsmap = _makemap(repo)
 progress = repo.ui.progress
 progress(_('bundle'), 0, total=totalfilesize, unit=_('bytes'))
 with maketempcopies() as copy:
 try:
 # copy is delayed until we are in the try
-entries = [_filterfull(e, copy, vfs) for e in entries]
+entries = [_filterfull(e, copy, vfsmap) for e in entries]
 yield None # this release the lock on the repository
 seen = 0
 
-for name, ftype, data in entries:
+for src, name, ftype, data in entries:
+vfs = vfsmap[src]
+yield src
 yield util.uvarintencode(len(name))
 if ftype == _fileappend:
 fp = vfs(name)
@@ -507,10 +527,11 @@ def generatev2(repo):
 """Emit content for version 2 of a streaming clone.
 
 the data stream consists the following entries:
-1) A varint containing the length of the filename
-2) A varint containing the length of file data
-3) N bytes containing the filename (the internal, store-agnostic form)
-4) N bytes containing the file data
+1) A char representing the file destination (eg: store or cache)
+2) A varint containing the length of the filename
+3) A varint containing the length of file data
+4) N bytes containing the filename (the internal, store-agnostic form)
+5) N bytes containing the file data
 
 Returns a 3-tuple of (file count, file size, data iterator).
 """
@@ -523,12 +544,16 @@ def generatev2(repo):
 repo.ui.debug('scanning\n')
 for name, ename, size in _walkstreamfiles(repo):
 if size:
-entries.append((name, _fileappend, size))
+entries.append((_srcstore, name, _fileappend, size))
 totalfilesize += size
 for name in _walkstreamfullstorefiles(repo):
 if repo.svfs.exists(name):
 totalfilesize += repo.svfs.lstat(name).st_size
-entries.append((name, _filefull, None))
+entries.append((_srcstore, name, _filefull, None))
+for name in cacheutil.cachetocopy(repo):
+if repo.cachevfs.exists(name):
+totalfilesize += repo.cachevfs.lstat(name).st_size
+entries.append((_srccache, name, _filefull, None))
 
 chunks = _emit(repo, entries, totalfilesize)
 first = next(chunks)
@@ -536,6 +561,16 @@ def generatev2(repo):
 
 return len(entries), totalfilesize, chunks
 
+@contextlib.contextmanager
+def nested(*ctxs):
+with 

[PATCH 07 of 15] bundle2: add support for a 'stream' parameter to 'getbundle'

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1516203383 -3600
#  Wed Jan 17 16:36:23 2018 +0100
# Node ID b11f4652647e791727e14c94a0ccb7c0282c5a29
# Parent  a4d9afe3a419720acb7d7d06fdfe11f8f6234b95
# EXP-Topic b2-stream
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
b11f4652647e
bundle2: add support for a 'stream' parameter to 'getbundle'

This parameter can be used to request a stream bundle.

diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -1747,6 +1747,19 @@ def getbundlechunks(repo, source, heads=
 
 return bundler.getchunks()
 
+@getbundle2partsgenerator('stream')
+def _getbundlestream(bundler, repo, source, bundlecaps=None,
+ b2caps=None, heads=None, common=None, **kwargs):
+if not kwargs.get('stream', False):
+return
+filecount, bytecount, it = streamclone.generatev2(repo)
+requirements = ' '.join(repo.requirements)
+part = bundler.newpart('stream', data=it)
+part.addparam('bytecount', '%d' % bytecount, mandatory=True)
+part.addparam('filecount', '%d' % filecount, mandatory=True)
+part.addparam('requirements', requirements, mandatory=True)
+part.addparam('version', 'v2', mandatory=True)
+
 @getbundle2partsgenerator('changegroup')
 def _getbundlechangegrouppart(bundler, repo, source, bundlecaps=None,
   b2caps=None, heads=None, common=None, **kwargs):
diff --git a/mercurial/wireproto.py b/mercurial/wireproto.py
--- a/mercurial/wireproto.py
+++ b/mercurial/wireproto.py
@@ -212,7 +212,9 @@ gboptsmap = {'heads':  'nodes',
  'bundlecaps': 'scsv',
  'listkeys': 'csv',
  'cg': 'boolean',
- 'cbattempted': 'boolean'}
+ 'cbattempted': 'boolean',
+ 'stream': 'boolean',
+}
 
 # client side
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 14 of 15] caches: make 'cachetocopy' available in scmutil

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1516207609 -3600
#  Wed Jan 17 17:46:49 2018 +0100
# Node ID fec6950ccabdd6d93484732b341ae06697954890
# Parent  fe0252dde2b121fb05353bb9f1bf1fdf83e8c9a7
# EXP-Topic b2-stream
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
fec6950ccabd
caches: make 'cachetocopy' available in scmutil

For more code to use this information, we need it to be more publicly available.

diff --git a/mercurial/cacheutil.py b/mercurial/cacheutil.py
new file mode 100644
--- /dev/null
+++ b/mercurial/cacheutil.py
@@ -0,0 +1,21 @@
+# scmutil.py - Mercurial core utility functions
+#
+#  Copyright Matt Mackall  and other
+#
+# This software may be used and distributed according to the terms of the
+# GNU General Public License version 2 or any later version.
+from __future__ import absolute_import
+
+from . import repoview
+
+def cachetocopy(srcrepo):
+"""return the list of cache file valuable to copy during a clone"""
+# In local clones we're copying all nodes, not just served
+# ones. Therefore copy all branch caches over.
+cachefiles = ['branch2']
+cachefiles += ['branch2-%s' % f for f in repoview.filtertable]
+cachefiles += ['rbc-names-v1', 'rbc-revs-v1']
+cachefiles += ['tags2']
+cachefiles += ['tags2-%s' % f for f in repoview.filtertable]
+cachefiles += ['hgtagsfnodes1']
+return cachefiles
diff --git a/mercurial/hg.py b/mercurial/hg.py
--- a/mercurial/hg.py
+++ b/mercurial/hg.py
@@ -21,6 +21,7 @@ from .node import (
 from . import (
 bookmarks,
 bundlerepo,
+cacheutil,
 cmdutil,
 destutil,
 discovery,
@@ -34,7 +35,6 @@ from . import (
 merge as mergemod,
 node,
 phases,
-repoview,
 scmutil,
 sshpeer,
 statichttprepo,
@@ -459,18 +459,6 @@ def _copycache(srcrepo, dstcachedir, fna
 os.mkdir(dstcachedir)
 util.copyfile(srcbranchcache, dstbranchcache)
 
-def _cachetocopy(srcrepo):
-"""return the list of cache file valuable to copy during a clone"""
-# In local clones we're copying all nodes, not just served
-# ones. Therefore copy all branch caches over.
-cachefiles = ['branch2']
-cachefiles += ['branch2-%s' % f for f in repoview.filtertable]
-cachefiles += ['rbc-names-v1', 'rbc-revs-v1']
-cachefiles += ['tags2']
-cachefiles += ['tags2-%s' % f for f in repoview.filtertable]
-cachefiles += ['hgtagsfnodes1']
-return cachefiles
-
 def clone(ui, peeropts, source, dest=None, pull=False, rev=None,
   update=True, stream=False, branch=None, shareopts=None):
 """Make a copy of an existing repository.
@@ -629,7 +617,7 @@ def clone(ui, peeropts, source, dest=Non
 util.copyfile(srcbookmarks, dstbookmarks)
 
 dstcachedir = os.path.join(destpath, 'cache')
-for cache in _cachetocopy(srcrepo):
+for cache in cacheutil.cachetocopy(srcrepo):
 _copycache(srcrepo, dstcachedir, cache)
 
 # we need to re-init the repo after manually copying the data
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 11 of 15] streamclone: add support for bundle2 based stream clone

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1516203704 -3600
#  Wed Jan 17 16:41:44 2018 +0100
# Node ID e7a1641a31fbdb2f1e0960bc0090e2c7705a8d1c
# Parent  36876df4b7cf74f276da656a9da1fe8e47597e15
# EXP-Topic b2-stream
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
e7a1641a31fb
streamclone: add support for bundle2 based stream clone

The feature put to use the various bits introduced previously. If the server
supports it, the client will request its stream clone through bundle2 instead of
the legacy 'stream_out' commands. The bundle2 version use the better 'v2'
version of stream bundles.

The 'v2' format is not finalized yet. Now that there are some code running it,
we can start working on it again.

diff --git a/mercurial/bundle2.py b/mercurial/bundle2.py
--- a/mercurial/bundle2.py
+++ b/mercurial/bundle2.py
@@ -1487,6 +1487,7 @@ capabilities = {'HG20': (),
 'remote-changegroup': ('http', 'https'),
 'hgtagsfnodes': (),
 'phases': ('heads',),
+'stream': ('v2',),
}
 
 def getrepocaps(repo, allowpushback=False):
@@ -1507,6 +1508,8 @@ def getrepocaps(repo, allowpushback=Fals
 caps['checkheads'] = ('related',)
 if 'phases' in repo.ui.configlist('devel', 'legacy.exchange'):
 caps.pop('phases')
+if not repo.ui.configbool('experimental', 'bundle2.stream'):
+caps.pop('stream')
 return caps
 
 def bundle2caps(remote):
diff --git a/mercurial/configitems.py b/mercurial/configitems.py
--- a/mercurial/configitems.py
+++ b/mercurial/configitems.py
@@ -431,6 +431,9 @@ coreconfigitem('experimental', 'bundle2-
 coreconfigitem('experimental', 'bundle2.pushback',
 default=False,
 )
+coreconfigitem('experimental', 'bundle2.stream',
+default=False,
+)
 coreconfigitem('experimental', 'bundle2lazylocking',
 default=False,
 )
diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -1455,13 +1455,18 @@ def _pullbundle2(pullop):
 
 # At the moment we don't do stream clones over bundle2. If that is
 # implemented then here's where the check for that will go.
-streaming = False
+streaming = streamclone.canperformstreamclone(pullop, bundle2=True)[0]
 
 # declare pull perimeters
 kwargs['common'] = pullop.common
 kwargs['heads'] = pullop.heads or pullop.rheads
 
-if True:
+if streaming:
+kwargs['cg'] = False
+kwargs['stream'] = True
+pullop.stepsdone.add('changegroup')
+
+else:
 # pulling changegroup
 pullop.stepsdone.add('changegroup')
 
diff --git a/tests/test-clone-uncompressed.t b/tests/test-clone-uncompressed.t
--- a/tests/test-clone-uncompressed.t
+++ b/tests/test-clone-uncompressed.t
@@ -1,5 +1,14 @@
 #require serve
 
+#testcases stream-legacy stream-bundle2
+
+#if stream-bundle2
+  $ cat << EOF >> $HGRCPATH
+  > [experimental]
+  > bundle2.stream = yes
+  > EOF
+#endif
+
 Initialize repository
 the status call is to check for issue5130
 
@@ -18,24 +27,41 @@ the status call is to check for issue513
 
 Basic clone
 
+#if stream-legacy
   $ hg clone --stream -U http://localhost:$HGPORT clone1
   streaming all changes
   1027 files to transfer, 96.3 KB of data
   transferred 96.3 KB in * seconds (*/sec) (glob)
   searching for changes
   no changes found
+#endif
+#if stream-bundle2
+  $ hg clone --stream -U http://localhost:$HGPORT clone1
+  streaming all changes
+  1027 files to transfer, 96.3 KB of data
+  transferred 96.3 KB in * seconds (* */sec) (glob)
+#endif
 
 --uncompressed is an alias to --stream
 
+#if stream-legacy
   $ hg clone --uncompressed -U http://localhost:$HGPORT clone1-uncompressed
   streaming all changes
   1027 files to transfer, 96.3 KB of data
   transferred 96.3 KB in * seconds (*/sec) (glob)
   searching for changes
   no changes found
+#endif
+#if stream-bundle2
+  $ hg clone --uncompressed -U http://localhost:$HGPORT clone1-uncompressed
+  streaming all changes
+  1027 files to transfer, 96.3 KB of data
+  transferred 96.3 KB in * seconds (* */sec) (glob)
+#endif
 
 Clone with background file closing enabled
 
+#if stream-legacy
   $ hg --debug --config worker.backgroundclose=true --config 
worker.backgroundcloseminfilecount=1 clone --stream -U http://localhost:$HGPORT 
clone-background | grep -v adding
   using http://localhost:$HGPORT/
   sending capabilities command
@@ -57,6 +83,28 @@ Clone with background file closing enabl
   bundle2-input-part: total payload size 24
   bundle2-input-bundle: 1 parts total
   checking for updated bookmarks
+#endif
+#if stream-bundle2
+  $ hg --debug --config worker.backgroundclose=true --config 
worker.backgroundcloseminfilecount=1 clone --stream -U http://localhost:$HGPORT 
clone-background | grep -v adding
+  using http://localhost:$HGPORT/
+  sending capabilities command
+  

[PATCH 09 of 15] pull: reorganize bundle2 argument bundling

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1516203125 -3600
#  Wed Jan 17 16:32:05 2018 +0100
# Node ID 0db72f8d443d7ca27d57e650aa44b5d74ef965c2
# Parent  77a0634011b5bc89472a134c5ea2b5623f6ca273
# EXP-Topic b2-stream
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
0db72f8d443d
pull: reorganize bundle2 argument bundling

We are about to add the ability to use stream bundle with bundle2. Before doing
so, we need to gather some code that will not be used in the bundle2 case. There
is no behavior change within this changeset.

diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -1450,24 +1450,32 @@ def _pullbundle2(pullop):
 For now, the only supported data are changegroup."""
 kwargs = {'bundlecaps': caps20to10(pullop.repo)}
 
+# make ui easier to access
+ui = pullop.repo.ui
+
 # At the moment we don't do stream clones over bundle2. If that is
 # implemented then here's where the check for that will go.
 streaming = False
 
+# declare pull perimeters
+kwargs['common'] = pullop.common
+kwargs['heads'] = pullop.heads or pullop.rheads
+
 # pulling changegroup
 pullop.stepsdone.add('changegroup')
 
-kwargs['common'] = pullop.common
-kwargs['heads'] = pullop.heads or pullop.rheads
 kwargs['cg'] = pullop.fetch
 
-ui = pullop.repo.ui
 legacyphase = 'phases' in ui.configlist('devel', 'legacy.exchange')
 hasbinaryphase = 'heads' in pullop.remotebundle2caps.get('phases', ())
 if (not legacyphase and hasbinaryphase):
 kwargs['phases'] = True
 pullop.stepsdone.add('phases')
 
+if 'listkeys' in pullop.remotebundle2caps:
+if 'phases' not in pullop.stepsdone:
+kwargs['listkeys'] = ['phases']
+
 bookmarksrequested = False
 legacybookmark = 'bookmarks' in ui.configlist('devel', 'legacy.exchange')
 hasbinarybook = 'bookmarks' in pullop.remotebundle2caps
@@ -1482,8 +1490,6 @@ def _pullbundle2(pullop):
 bookmarksrequested = True
 
 if 'listkeys' in pullop.remotebundle2caps:
-if 'phases' not in pullop.stepsdone:
-kwargs['listkeys'] = ['phases']
 if 'request-bookmarks' not in pullop.stepsdone:
 # make sure to always includes bookmark data when migrating
 # `hg incoming --bundle` to using this function.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 10 of 15] pull: preindent some code

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1516194826 -3600
#  Wed Jan 17 14:13:46 2018 +0100
# Node ID 36876df4b7cf74f276da656a9da1fe8e47597e15
# Parent  0db72f8d443d7ca27d57e650aa44b5d74ef965c2
# EXP-Topic b2-stream
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
36876df4b7cf
pull: preindent some code

Next changesets will add support for using stream cloning with bundle2. We
introduce indentation change first for clarity.

diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -1461,10 +1461,11 @@ def _pullbundle2(pullop):
 kwargs['common'] = pullop.common
 kwargs['heads'] = pullop.heads or pullop.rheads
 
-# pulling changegroup
-pullop.stepsdone.add('changegroup')
+if True:
+# pulling changegroup
+pullop.stepsdone.add('changegroup')
 
-kwargs['cg'] = pullop.fetch
+kwargs['cg'] = pullop.fetch
 
 legacyphase = 'phases' in ui.configlist('devel', 'legacy.exchange')
 hasbinaryphase = 'heads' in pullop.remotebundle2caps.get('phases', ())
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 13 of 15] streamclone: add support for cloning non append-only file

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1516233002 -3600
#  Thu Jan 18 00:50:02 2018 +0100
# Node ID fe0252dde2b121fb05353bb9f1bf1fdf83e8c9a7
# Parent  243565f94076531d0223abb4434864dda2c995d8
# EXP-Topic b2-stream
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
fe0252dde2b1
streamclone: add support for cloning non append-only file

The phaseroots are stored in a non append-only file in the repository. We
include them in the stream too. Since they are not append-only, we have to
keep a copy around while we hold the lock to be able to stream them later.

Since phase get exchanged within the stream we can skip requesting them
independently.

As a side effect, this will fixes issue5648 once the feature is enabled by
default.

diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -1465,6 +1465,7 @@ def _pullbundle2(pullop):
 kwargs['cg'] = False
 kwargs['stream'] = True
 pullop.stepsdone.add('changegroup')
+pullop.stepsdone.add('phases')
 
 else:
 # pulling changegroup
@@ -1472,15 +1473,15 @@ def _pullbundle2(pullop):
 
 kwargs['cg'] = pullop.fetch
 
-legacyphase = 'phases' in ui.configlist('devel', 'legacy.exchange')
-hasbinaryphase = 'heads' in pullop.remotebundle2caps.get('phases', ())
-if (not legacyphase and hasbinaryphase):
-kwargs['phases'] = True
-pullop.stepsdone.add('phases')
+legacyphase = 'phases' in ui.configlist('devel', 'legacy.exchange')
+hasbinaryphase = 'heads' in pullop.remotebundle2caps.get('phases', ())
+if (not legacyphase and hasbinaryphase):
+kwargs['phases'] = True
+pullop.stepsdone.add('phases')
 
-if 'listkeys' in pullop.remotebundle2caps:
-if 'phases' not in pullop.stepsdone:
-kwargs['listkeys'] = ['phases']
+if 'listkeys' in pullop.remotebundle2caps:
+if 'phases' not in pullop.stepsdone:
+kwargs['listkeys'] = ['phases']
 
 bookmarksrequested = False
 legacybookmark = 'bookmarks' in ui.configlist('devel', 'legacy.exchange')
diff --git a/mercurial/streamclone.py b/mercurial/streamclone.py
--- a/mercurial/streamclone.py
+++ b/mercurial/streamclone.py
@@ -7,7 +7,10 @@
 
 from __future__ import absolute_import
 
+import contextlib
+import os
 import struct
+import tempfile
 
 from .i18n import _
 from . import (
@@ -428,32 +431,77 @@ class streamcloneapplier(object):
 def apply(self, repo):
 return applybundlev1(repo, self._fh)
 
+# type of file to stream
+_fileappend = 0 # append only file
+_filefull = 1   # full snapshot file
+
+# This is it's own function so extensions can override it.
+def _walkstreamfullstorefiles(repo):
+"""list snapshot file from the store"""
+fnames = []
+if not repo.publishing():
+fnames.append('phaseroots')
+return fnames
+
+def _filterfull(entry, copy, vfs):
+"""actually copy the snapshot files"""
+name, ftype, data = entry
+if ftype != _filefull:
+return entry
+return (name, ftype, copy(vfs.join(name)))
+
+@contextlib.contextmanager
+def maketempcopies():
+"""return a function to temporary copy file"""
+files = []
+try:
+def copy(src):
+fd, dst = tempfile.mkstemp()
+os.close(fd)
+files.append(dst)
+util.copyfiles(src, dst, hardlink=True)
+return dst
+yield copy
+finally:
+for tmp in files:
+util.tryunlink(tmp)
+
 def _emit(repo, entries, totalfilesize):
 """actually emit the stream bundle"""
+vfs = repo.svfs
 progress = repo.ui.progress
 progress(_('bundle'), 0, total=totalfilesize, unit=_('bytes'))
-vfs = repo.svfs
-try:
-seen = 0
-for name, size in entries:
-yield util.uvarintencode(len(name))
-fp = vfs(name)
-try:
-yield util.uvarintencode(size)
-yield name
-if size <= 65536:
-chunks = (fp.read(size),)
-else:
-chunks = util.filechunkiter(fp, limit=size)
-for chunk in chunks:
-seen += len(chunk)
-progress(_('bundle'), seen, total=totalfilesize,
- unit=_('bytes'))
-yield chunk
-finally:
-fp.close()
-finally:
-progress(_('bundle'), None)
+with maketempcopies() as copy:
+try:
+# copy is delayed until we are in the try
+entries = [_filterfull(e, copy, vfs) for e in entries]
+yield None # this release the lock on the repository
+seen = 0
+
+for name, ftype, data in entries:
+yield util.uvarintencode(len(name))
+ 

[PATCH 06 of 15] bundle2: add a 'stream' part handler for stream cloning

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1516203322 -3600
#  Wed Jan 17 16:35:22 2018 +0100
# Node ID a4d9afe3a419720acb7d7d06fdfe11f8f6234b95
# Parent  2ef371994c9c4525a525c95586691d43b04ecd33
# EXP-Topic b2-stream
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
a4d9afe3a419
bundle2: add a 'stream' part handler for stream cloning

The part contains the necessary arguments and payload to handle a stream bundle
v2. It will be put to use in later changesets.

diff --git a/mercurial/bundle2.py b/mercurial/bundle2.py
--- a/mercurial/bundle2.py
+++ b/mercurial/bundle2.py
@@ -164,6 +164,7 @@ from . import (
 phases,
 pushkey,
 pycompat,
+streamclone,
 tags,
 url,
 util,
@@ -2114,3 +2115,30 @@ def bundle2getvars(op, part):
 key = "USERVAR_" + key
 hookargs[key] = value
 op.addhookargs(hookargs)
+
+@parthandler('stream', ('requirements', 'filecount', 'bytecount', 'version'))
+def handlestreambundle(op, part):
+
+version = part.params['version']
+if version != 'v2':
+raise error.Abort(_('unknown stream bundle version %s') % version)
+requirements = part.params['requirements'].split()
+filecount = int(part.params['filecount'])
+bytecount = int(part.params['bytecount'])
+
+repo = op.repo
+if len(repo):
+msg = _('cannot apply stream clone to non empty repository')
+raise error.Abort(msg)
+
+repo.ui.debug('applying stream bundle\n')
+streamclone.applybundlev2(repo, part, filecount, bytecount,
+  requirements)
+
+# new requirements = old non-format requirements +
+#new format-related remote requirements
+# requirements from the streamed-in repository
+repo.requirements = set(requirements) | (
+repo.requirements - repo.supportedformats)
+repo._applyopenerreqs()
+repo._writerequirements()
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 08 of 15] clone: allow bundle2's stream clone with 'server.disablefullbundle'

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1516203512 -3600
#  Wed Jan 17 16:38:32 2018 +0100
# Node ID 77a0634011b5bc89472a134c5ea2b5623f6ca273
# Parent  b11f4652647e791727e14c94a0ccb7c0282c5a29
# EXP-Topic b2-stream
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
77a0634011b5
clone: allow bundle2's stream clone with 'server.disablefullbundle'

The previous check was a bit too strict and would not recognize a get bundle
not requesting changegroup.

diff --git a/mercurial/wireproto.py b/mercurial/wireproto.py
--- a/mercurial/wireproto.py
+++ b/mercurial/wireproto.py
@@ -855,10 +855,11 @@ def getbundle(repo, proto, others):
 if repo.ui.configbool('server', 'disablefullbundle'):
 # Check to see if this is a full clone.
 clheads = set(repo.changelog.heads())
+changegroup = opts.get('cg', True)
 heads = set(opts.get('heads', set()))
 common = set(opts.get('common', set()))
 common.discard(nullid)
-if not common and clheads == heads:
+if changegroup and not common and clheads == heads:
 raise error.Abort(
 _('server has pull-based clones disabled'),
 hint=_('remove --pull if specified or upgrade Mercurial'))
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 03 of 15] util: implement varint functions

[Boris Feld]

# HG changeset patch
# User Gregory Szorc 
# Date 1474779566 25200
#  Sat Sep 24 21:59:26 2016 -0700
# Node ID b80a8e39ac9bf984c25a666bd7f6c47d876d26af
# Parent  f6baa2844ea380da1a63832a35b1b83007c181a3
# EXP-Topic b2-stream
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
b80a8e39ac9b
util: implement varint functions

This will be useful in an incoming version-2 of the stream format.

diff --git a/mercurial/util.py b/mercurial/util.py
--- a/mercurial/util.py
+++ b/mercurial/util.py
@@ -3874,3 +3874,36 @@ def readexactly(stream, n):
" (got %d bytes, expected %d)")
   % (len(s), n))
 return s
+
+def uvarintencode(value):
+"""Encode an unsigned integer value to a varint.
+
+A varint is a variable length integer of 1 or more bytes. Each byte
+except the last has the most significant bit set. The lower 7 bits of
+each byte store the 2's complement representation, least significant group
+first.
+"""
+bits = value & 0x7f
+value >>= 7
+bytes = []
+while value:
+bytes.append(pycompat.bytechr(0x80 | bits))
+bits = value & 0x7f
+value >>= 7
+bytes.append(pycompat.bytechr(bits))
+
+return ''.join(bytes)
+
+def uvarintdecodestream(fh):
+"""Decode an unsigned variable length integer from a stream.
+
+The passed argument is anything that has a ``.read(N)`` method.
+"""
+result = 0
+shift = 0
+while True:
+byte = ord(readexactly(fh, 1))
+result |= ((byte & 0x7f) << shift)
+if not (byte & 0x80):
+return result
+shift += 7
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 01 of 15] stream: add a test showing we also clone bookmarks

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1516201266 -3600
#  Wed Jan 17 16:01:06 2018 +0100
# Node ID bb08b77c4988817b0ff2d25e705b455169868c7e
# Parent  853bf7d9080462e737f584ff682dd18e17717187
# EXP-Topic b2-stream
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
bb08b77c4988
stream: add a test showing we also clone bookmarks

Bookmarks are not stored in `.hg/store`. We need to make sure they are cloned
with `--stream`.

diff --git a/tests/test-clone-uncompressed.t b/tests/test-clone-uncompressed.t
--- a/tests/test-clone-uncompressed.t
+++ b/tests/test-clone-uncompressed.t
@@ -171,3 +171,28 @@ actually serving file content
   $ wait
   $ hg -R clone id
   
+  $ cd ..
+
+Stream repository with bookmarks
+
+
+(revert introduction of secret changeset)
+
+  $ hg -R server phase --draft 'secret()'
+
+add a bookmark
+
+  $ hg -R server bookmark -r tip some-bookmark
+
+clone it
+
+  $ hg clone --stream http://localhost:$HGPORT with-bookmarks
+  streaming all changes
+  1027 files to transfer, 96.3 KB of data
+  transferred 96.3 KB in * seconds (*) (glob)
+  searching for changes
+  no changes found
+  updating to branch default
+  1025 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  $ hg -R with-bookmarks bookmarks
+ some-bookmark 1:c17445101a72
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 02 of 15] util: move 'readexactly' in the util module

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1516391495 -3600
#  Fri Jan 19 20:51:35 2018 +0100
# Node ID f6baa2844ea380da1a63832a35b1b83007c181a3
# Parent  bb08b77c4988817b0ff2d25e705b455169868c7e
# EXP-Topic b2-stream
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
f6baa2844ea3
util: move 'readexactly' in the util module

This function is used in multiple place, having it in util would be better.
(existing caller will be migrated in another series)

diff --git a/mercurial/changegroup.py b/mercurial/changegroup.py
--- a/mercurial/changegroup.py
+++ b/mercurial/changegroup.py
@@ -32,14 +32,7 @@ from . import (
 _CHANGEGROUPV2_DELTA_HEADER = "20s20s20s20s20s"
 _CHANGEGROUPV3_DELTA_HEADER = ">20s20s20s20s20sH"
 
-def readexactly(stream, n):
-'''read n bytes from stream.read and abort if less was available'''
-s = stream.read(n)
-if len(s) < n:
-raise error.Abort(_("stream ended unexpectedly"
-   " (got %d bytes, expected %d)")
-  % (len(s), n))
-return s
+readexactly = util.readexactly
 
 def getchunk(stream):
 """return the next chunk from stream as a string"""
diff --git a/mercurial/util.py b/mercurial/util.py
--- a/mercurial/util.py
+++ b/mercurial/util.py
@@ -3865,3 +3865,12 @@ def safename(f, tag, ctx, others=None):
 fn = '%s~%s~%s' % (f, tag, n)
 if fn not in ctx and fn not in others:
 return fn
+
+def readexactly(stream, n):
+'''read n bytes from stream.read and abort if less was available'''
+s = stream.read(n)
+if len(s) < n:
+raise error.Abort(_("stream ended unexpectedly"
+   " (got %d bytes, expected %d)")
+  % (len(s), n))
+return s
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1913: branch: allow changing branch name to existing name if possible

[pulkit (Pulkit Goyal)]

This revision was automatically updated to reflect the committed changes.
Closed by commit rHGe5b6ba786d83: branch: allow changing branch name to 
existing name if possible (authored by pulkit, committed by ).

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1913?vs=4936=4945

REVISION DETAIL
  https://phab.mercurial-scm.org/D1913

AFFECTED FILES
  mercurial/cmdutil.py
  mercurial/commands.py
  tests/test-branch-change.t

CHANGE DETAILS

diff --git a/tests/test-branch-change.t b/tests/test-branch-change.t
--- a/tests/test-branch-change.t
+++ b/tests/test-branch-change.t
@@ -267,15 +267,49 @@
   $ hg branch
   stable
 
-Changing to same branch name does not work
+Changing to same branch is no-op
 
   $ hg branch -r 19::21 stable
-  abort: a branch of the same name already exists
-  [255]
+  changed branch on 0 changesets
+
+Changing branch name to existing branch name if the branch of parent of root of
+revs is same as the new branch name
+
+  $ hg branch -r 20::21 bugfix
+  changed branch on 2 changesets
+  $ hg glog
+  o  25:714defe1cf34 Added d
+  |  bugfix ()
+  o  24:98394def28fc Added c
+  |  bugfix ()
+  | @  23:6a5ddbcfb870 added bar
+  | |  stable (b1)
+  | o  22:baedc6e98a67 Added e
+  |/   stable ()
+  o  19:fd45b986b109 Added b
+  |  stable ()
+  o  18:204d2769eca2 Added a
+ stable ()
+
+  $ hg branch -r 24:25 stable
+  changed branch on 2 changesets
+  $ hg glog
+  o  27:4ec342341562 Added d
+  |  stable ()
+  o  26:83f48859c2de Added c
+  |  stable ()
+  | @  23:6a5ddbcfb870 added bar
+  | |  stable (b1)
+  | o  22:baedc6e98a67 Added e
+  |/   stable ()
+  o  19:fd45b986b109 Added b
+  |  stable ()
+  o  18:204d2769eca2 Added a
+ stable ()
 
 Testing on merge
 
-  $ hg merge -r 20
+  $ hg merge -r 26
   1 files updated, 0 files merged, 0 files removed, 0 files unresolved
   (branch merge, don't forget to commit)
 
@@ -289,8 +323,8 @@
 
 Changing branch on public changeset
 
-  $ hg phase -r 21 -p
-  $ hg branch -r 21 def
+  $ hg phase -r 27 -p
+  $ hg branch -r 27 def
   abort: cannot change branch of public changesets
   (see 'hg help phases' for details)
   [255]
diff --git a/mercurial/commands.py b/mercurial/commands.py
--- a/mercurial/commands.py
+++ b/mercurial/commands.py
@@ -1055,11 +1055,6 @@
 
 scmutil.checknewlabel(repo, label, 'branch')
 if revs:
-# XXX: we should allow setting name to existing branch if the
-# branch of root of the revs is same as the new branch name
-if label in repo.branchmap():
-raise error.Abort(_('a branch of the same'
-' name already exists'))
 return cmdutil.changebranch(ui, repo, revs, label)
 
 if not opts.get('force') and label in repo.branchmap():
diff --git a/mercurial/cmdutil.py b/mercurial/cmdutil.py
--- a/mercurial/cmdutil.py
+++ b/mercurial/cmdutil.py
@@ -727,6 +727,11 @@
 if len(roots) > 1:
 raise error.Abort(_("cannot change branch of non-linear 
revisions"))
 rewriteutil.precheck(repo, revs, 'change branch of')
+
+root = repo[roots.first()]
+if not root.p1().branch() == label and label in repo.branchmap():
+raise error.Abort(_("a branch of the same name already exists"))
+
 if repo.revs('merge() and %ld', revs):
 raise error.Abort(_("cannot change branch of a merge commit"))
 if repo.revs('obsolete() and %ld', revs):



To: pulkit, #hg-reviewers, durin42
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1074: branch: add a --rev flag to change branch name of given revisions

[pulkit (Pulkit Goyal)]

This revision was automatically updated to reflect the committed changes.
Closed by commit rHG3bd8ab4c80a5: branch: add a --rev flag to change branch 
name of given revisions (authored by pulkit, committed by ).

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1074?vs=4935=4944

REVISION DETAIL
  https://phab.mercurial-scm.org/D1074

AFFECTED FILES
  mercurial/cmdutil.py
  mercurial/commands.py
  tests/test-branch-change.t
  tests/test-completion.t

CHANGE DETAILS

diff --git a/tests/test-completion.t b/tests/test-completion.t
--- a/tests/test-completion.t
+++ b/tests/test-completion.t
@@ -242,7 +242,7 @@
   backout: merge, commit, no-commit, parent, rev, edit, tool, include, 
exclude, message, logfile, date, user
   bisect: reset, good, bad, skip, extend, command, noupdate
   bookmarks: force, rev, delete, rename, inactive, template
-  branch: force, clean
+  branch: force, clean, rev
   branches: active, closed, template
   bundle: force, rev, branch, base, all, type, ssh, remotecmd, insecure
   cat: output, rev, decode, include, exclude, template
diff --git a/tests/test-branch-change.t b/tests/test-branch-change.t
new file mode 100644
--- /dev/null
+++ b/tests/test-branch-change.t
@@ -0,0 +1,296 @@
+Testing changing branch on commits
+==
+
+Setup
+
+  $ cat >> $HGRCPATH << EOF
+  > [alias]
+  > glog = log -G -T "{rev}:{node|short} {desc}\n{branch} ({bookmarks})"
+  > [experimental]
+  > evolution = createmarkers
+  > [extensions]
+  > rebase=
+  > EOF
+
+  $ hg init repo
+  $ cd repo
+  $ for ch in a b c d e; do echo foo >> $ch; hg ci -Aqm "Added "$ch; done
+  $ hg glog
+  @  4:aa98ab95a928 Added e
+  |  default ()
+  o  3:62615734edd5 Added d
+  |  default ()
+  o  2:28ad74487de9 Added c
+  |  default ()
+  o  1:29becc82797a Added b
+  |  default ()
+  o  0:18d04c59bb5d Added a
+ default ()
+
+  $ hg branches
+  default4:aa98ab95a928
+
+Try without passing a new branch name
+
+  $ hg branch -r .
+  abort: no branch name specified for the revisions
+  [255]
+
+Setting an invalid branch name
+
+  $ hg branch -r . a:b
+  abort: ':' cannot be used in a name
+  [255]
+  $ hg branch -r . tip
+  abort: the name 'tip' is reserved
+  [255]
+  $ hg branch -r . 1234
+  abort: cannot use an integer as a name
+  [255]
+
+Change on non-linear set of commits
+
+  $ hg branch -r 2 -r 4 foo
+  abort: cannot change branch of non-linear revisions
+  [255]
+
+Change in middle of the stack (linear commits)
+
+  $ hg branch -r 1::3 foo
+  abort: cannot change branch of changeset with children
+  [255]
+
+Change with dirty working directory
+
+  $ echo bar > a
+  $ hg branch -r . foo
+  abort: uncommitted changes
+  [255]
+
+  $ hg revert --all
+  reverting a
+
+Change on empty revision set
+
+  $ hg branch -r 'draft() - all()' foo
+  abort: empty revision set
+  [255]
+
+Changing branch on linear set of commits from head
+
+Without obsmarkers
+
+  $ hg branch -r 3:4 foo --config experimental.evolution=!
+  changed branch on 2 changesets
+  saved backup bundle to 
$TESTTMP/repo/.hg/strip-backup/62615734edd5-e86bd13a-branch-change.hg (glob)
+  $ hg glog
+  @  4:3938acfb5c0f Added e
+  |  foo ()
+  o  3:9435da006bdc Added d
+  |  foo ()
+  o  2:28ad74487de9 Added c
+  |  default ()
+  o  1:29becc82797a Added b
+  |  default ()
+  o  0:18d04c59bb5d Added a
+ default ()
+
+  $ hg branches
+  foo4:3938acfb5c0f
+  default2:28ad74487de9 (inactive)
+
+With obsmarkers
+
+  $ hg branch -r 3::4 bar
+  changed branch on 2 changesets
+  $ hg glog
+  @  6:7c1991464886 Added e
+  |  bar ()
+  o  5:1ea05e93925f Added d
+  |  bar ()
+  o  2:28ad74487de9 Added c
+  |  default ()
+  o  1:29becc82797a Added b
+  |  default ()
+  o  0:18d04c59bb5d Added a
+ default ()
+
+  $ hg branches
+  bar6:7c1991464886
+  default2:28ad74487de9 (inactive)
+
+Change branch name to an existing branch
+
+  $ hg branch -r . default
+  abort: a branch of the same name already exists
+  [255]
+
+Changing on a branch head which is not topological head
+
+  $ hg branch -r 2 stable
+  abort: cannot change branch of changeset with children
+  [255]
+
+Enabling the allowunstable config and trying to change branch on a branch head
+which is not a topological head
+
+  $ echo "[experimental]" >> .hg/hgrc
+  $ echo "evolution.allowunstable=yes" >> .hg/hgrc
+  $ hg branch -r 2 foo
+  changed branch on 1 changesets
+  2 new orphan changesets
+
+Changing branch of an obsoleted changeset
+
+  $ hg branch -r 4 foobar
+  abort: hidden revision '4' was rewritten as: 7c1991464886!
+  (use --hidden to access hidden revisions)
+  [255]
+
+  $ hg branch -r 4 --hidden foobar
+  abort: cannot change branch of a obsolete changeset
+  [255]
+
+Make sure bookmark movement is correct
+
+  $ hg bookmark b1
+  $ hg glog -r '.^::'
+  @  6:7c1991464886 Added e
+  |  bar (b1)
+  *  

D1074: branch: add a --rev flag to change branch name of given revisions

[durin42 (Augie Fackler)]

durin42 added a comment.


  Accepting this since it seems generally liked and is marked experimental. We 
can always rip it out if it turns out to be a mistake.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1074

To: pulkit, #hg-reviewers, dlax, ryanmce, lothiraldan, durin42
Cc: martinvonz, lothiraldan, ryanmce, dlax, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] templater: fix crash by empty group expression

[Augie Fackler]

On Thu, Jan 18, 2018 at 09:07:58PM +0900, Yuya Nishihara wrote:
> # HG changeset patch
> # User Yuya Nishihara 
> # Date 1516114201 -32400
> #  Tue Jan 16 23:50:01 2018 +0900
> # Node ID 927c55b5ae4ebb16b01da9a2ac60539ecc6bb234
> # Parent  604c08ad12c4c3a2fdf182cdf90d5d4acaa25c9e
> templater: fix crash by empty group expression

queued, thanks
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] log: fix typo in comment about _matchfiles()

[Augie Fackler]

On Thu, Jan 18, 2018 at 09:07:43PM +0900, Yuya Nishihara wrote:
> # HG changeset patch
> # User Yuya Nishihara 
> # Date 1516106777 -32400
> #  Tue Jan 16 21:46:17 2018 +0900
> # Node ID 604c08ad12c4c3a2fdf182cdf90d5d4acaa25c9e
> # Parent  a62b08f6626bd0f838b2c77a2f1a48068852be1f
> log: fix typo in comment about _matchfiles()

queued, thanks
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] sparse: --include 'dir1/dir2' should not include 'dir1/*'

[Augie Fackler]

On Thu, Jan 18, 2018 at 01:39:29PM -0800, Hollis Blanchard wrote:
> # HG changeset patch
> # User Hollis Blanchard 
> # Date 1516311201 28800
> #  Thu Jan 18 13:33:21 2018 -0800
> # Node ID 0056cfcec61857cee0fa330865bd08ca1e704046
> # Parent  659dfbd852e28656fad791dce4f2a22646888ba8
> sparse: --include 'dir1/dir2' should not include 'dir1/*'
>
> In 2015 there was a workaround added 
> (f39bace2d6cad32907c0d7961b3c0dbd64a1b7ad)
> to sparse in the hg-experimental repo. That workaround:
> a) no longer seems to be needed, since its testcase passes even with the code
>removed, and
> b) caused a new problem: --include 'dir1/dir2' ended up including dir1/*
>too. (--include 'glob:dir1/dir2' is a user-level workaround.)
>
> Remove the offending code, and add a testcase for situation B.

Nice. Queued, with tests modified to use $TESTDIR/list-tree.py instead
of the non-portable tree(1). Thanks!

>
> diff --git a/mercurial/sparse.py b/mercurial/sparse.py
> --- a/mercurial/sparse.py
> +++ b/mercurial/sparse.py
> @@ -294,24 +294,9 @@ def matcher(repo, revs=None, includetemp
>  includes, excludes, profiles = patternsforrev(repo, rev)
>
>  if includes or excludes:
> -# Explicitly include subdirectories of includes so
> -# status will walk them down to the actual include.
> -subdirs = set()
> -for include in includes:
> -# TODO consider using posix path functions here so 
> Windows
> -# \ directory separators don't come into play.
> -dirname = os.path.dirname(include)
> -# basename is used to avoid issues with absolute
> -# paths (which on Windows can include the drive).
> -while os.path.basename(dirname):
> -subdirs.add(dirname)
> -dirname = os.path.dirname(dirname)
> -
>  matcher = matchmod.match(repo.root, '', [],
>   include=includes, exclude=excludes,
>   default='relpath')
> -if subdirs:
> -matcher = forceincludematcher(matcher, subdirs)
>  matchers.append(matcher)
>  except IOError:
>  pass
> diff --git a/tests/test-sparse.t b/tests/test-sparse.t
> --- a/tests/test-sparse.t
> +++ b/tests/test-sparse.t
> @@ -284,6 +284,31 @@ Test status on a file in a subdir
>$ hg status
>? dir1/dir2/file
>
> +Mix files and subdirectories, both "glob:" and unprefixed
> +
> +  $ hg debugsparse --reset
> +  $ touch dir1/notshown
> +  $ hg commit -A dir1/notshown -m "notshown"
> +  $ hg debugsparse --include 'dir1/dir2'
> +  $ tree
> +  .
> +  |-- dir1
> +  |   `-- dir2
> +  |   `-- file
> +  `-- hide.orig
> +
> +  2 directories, 2 files
> +  $ hg debugsparse --delete 'dir1/dir2'
> +  $ hg debugsparse --include 'glob:dir1/dir2'
> +  $ tree
> +  .
> +  |-- dir1
> +  |   `-- dir2
> +  |   `-- file
> +  `-- hide.orig
> +
> +  2 directories, 2 files
> +
>  Test that add -s adds dirs to sparse profile
>
>$ hg debugsparse --reset
> ___
> Mercurial-devel mailing list
> Mercurial-devel@mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 3 of 3] fileset: add kind:pat operator

[Augie Fackler]

On Thu, Jan 18, 2018 at 09:08:38PM +0900, Yuya Nishihara wrote:
> # HG changeset patch
> # User Yuya Nishihara 
> # Date 1515904155 -32400
> #  Sun Jan 14 13:29:15 2018 +0900
> # Node ID 77ef795761a1dcc59c3098f9dec308d01b1d7846
> # Parent  d3fce96625e610085c2335f9446b726c74326108
> fileset: add kind:pat operator

queued, thanks
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 2 of 2 V2] revlog: group delta computation methods under _deltacomputer object

[Augie Fackler]

On Fri, Jan 19, 2018 at 09:16:41AM +0100, Paul Morelle wrote:
> # HG changeset patch
> # User Paul Morelle 
> # Date 1515961692 -3600
> #  Sun Jan 14 21:28:12 2018 +0100
> # Node ID 82018742fcabfd0e26aa6f34bf773b6f25d27985
> # Parent  32bc4595737c2211dfcbf53cb499a366b9986dfd
> # EXP-Topic refactor-revlog
> # Available At https://bitbucket.org/octobus/mercurial-devel/
> #  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
> 82018742fcab
> revlog: group delta computation methods under _deltacomputer object
>
> Extracting these methods from revlog will allow changing the implementation of
> the deltacomputer, by providing this interface:
>   __init__(self, revlog) - constructor that initialize the object from a given
>revlog
>   buildtext(self, revinfo, fh) - builds the fulltext version of a revision 
> from
>  a _revisioninfo object and the file handle to
>  the .d (or .i for inline mode) file.
>   finddeltainfo(self, revinfo, fh) - find a revision in the revlog against
>  which it is acceptable to build a delta,
>  and build the corresponding _deltainfo.
>
> It should now be easier to write an experimental feature that would replace
> _deltacomputer by another object, for example one that would know how to
> parallelize the delta computation in order to quicken the storage of multiple
> revisions.

Aha! That is a fantastic justification, and gives me some excited
ideas about doing some Rust things in this area. Queued with
enthusiasm, many thanks.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 2 of 2] templatekw: add a {negrev} keyword

[Augie Fackler]

On Wed, Jan 17, 2018 at 10:14:30PM -0500, Jordi Gutiérrez Hermoso wrote:
> # HG changeset patch
> # User Jordi Gutiérrez Hermoso 
> # Date 1516243120 18000
> #  Wed Jan 17 21:38:40 2018 -0500
> # Node ID cbf1d676a938e78d40cd3504dd916f787bcb47ee
> # Parent  701f8a9defdc09bb63f2596e2fc426f2e78da313
> templatekw: add a {negrev} keyword

This is a really interesting idea. It mostly has driven me crazy that
negative revnums work, but this sort of provides a reason for their
existence I guess.

That said, I'm too wary of locking this in on the last day before a
freeze, so let's plan to discuss this after the freeze sometime in
early February? Maybe set a calendar reminder to rebase && resend this
patch then.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] localrepo: micro-optimize __len__() to bypass repoview

[Augie Fackler]

On Fri, Jan 19, 2018 at 10:14:05PM +0900, Yuya Nishihara wrote:
> # HG changeset patch
> # User Yuya Nishihara 
> # Date 1516365551 -32400
> #  Fri Jan 19 21:39:11 2018 +0900
> # Node ID 03870a74aaca105aa0b0f1c5edd68754098be5bc
> # Parent  f58245b9e3ea6e8945a81b951110cde155819345
> localrepo: micro-optimize __len__() to bypass repoview

queued, thanks
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

mercurial@35699: 14 new changesets

[Mercurial Commits]

14 new changesets in mercurial:

https://www.mercurial-scm.org/repo/hg/rev/b25fa5da4ca2
changeset:   35686:b25fa5da4ca2
user:Yuya Nishihara 
date:Wed Jan 03 15:46:15 2018 +0900
summary: log: resolve --follow thoroughly in getlogrevs()

https://www.mercurial-scm.org/repo/hg/rev/67893a516272
changeset:   35687:67893a516272
user:Yuya Nishihara 
date:Wed Jan 03 15:58:59 2018 +0900
summary: log: follow file history across copies even with -rREV (BC) 
(issue4959)

https://www.mercurial-scm.org/repo/hg/rev/84d0e99c063a
changeset:   35688:84d0e99c063a
user:Yuya Nishihara 
date:Thu Jan 04 14:37:15 2018 +0900
summary: log: replace "not pats" with matcher attribute for consistency

https://www.mercurial-scm.org/repo/hg/rev/5fe6f946f111
changeset:   35689:5fe6f946f111
user:Yuya Nishihara 
date:Thu Jan 04 15:20:46 2018 +0900
summary: log: allow matchfn to be non-null even if both --patch/--stat are 
off

https://www.mercurial-scm.org/repo/hg/rev/3e394e0558d7
changeset:   35690:3e394e0558d7
user:Yuya Nishihara 
date:Thu Jan 04 14:20:58 2018 +0900
summary: log: build follow-log filematcher at once

https://www.mercurial-scm.org/repo/hg/rev/735f47b41521
changeset:   35691:735f47b41521
user:Yuya Nishihara 
date:Sat Jan 13 15:07:37 2018 +0900
summary: fileset: make it robust for bad function calls

https://www.mercurial-scm.org/repo/hg/rev/a62b08f6626b
changeset:   35692:a62b08f6626b
user:Yuya Nishihara 
date:Sat Jan 13 15:13:29 2018 +0900
summary: fileset: do not crash by unary negate operation

https://www.mercurial-scm.org/repo/hg/rev/1880a0bdfc5e
changeset:   35693:1880a0bdfc5e
user:Jordi Gutiérrez Hermoso 
date:Wed Jan 17 22:12:10 2018 -0500
summary: test-convert-svn-move: sort svn checkout output

https://www.mercurial-scm.org/repo/hg/rev/8a23082f4d93
changeset:   35694:8a23082f4d93
user:Matt Harbison 
date:Wed Jan 17 20:54:05 2018 -0500
summary: lfs: correct documentation typo

https://www.mercurial-scm.org/repo/hg/rev/dd672e3d059f
changeset:   35695:dd672e3d059f
user:Matt Harbison 
date:Wed Jan 17 21:44:32 2018 -0500
summary: lfs: raise an error if the server sends an unsolicited oid

https://www.mercurial-scm.org/repo/hg/rev/925107e37619
changeset:   35696:925107e37619
user:Boris Feld 
date:Fri Jan 12 10:57:29 2018 +
summary: http: add a debug version of the push test

https://www.mercurial-scm.org/repo/hg/rev/5a7906ed78d4
changeset:   35697:5a7906ed78d4
user:Boris Feld 
date:Fri Jan 12 10:14:20 2018 +
summary: httppeer: move url opening in its own method

https://www.mercurial-scm.org/repo/hg/rev/0c4b23ccf1a5
changeset:   35698:0c4b23ccf1a5
user:Boris Feld 
date:Fri Jan 12 10:41:03 2018 +
summary: httppeer: add support for tracing all http request made by the peer

https://www.mercurial-scm.org/repo/hg/rev/f7ef49e44d7c
changeset:   35699:f7ef49e44d7c
bookmark:@
tag: tip
user:Boris Feld 
date:Fri Jan 12 11:52:57 2018 +
summary: sshpeer: add support for request tracing

-- 
Repository URL: https://www.mercurial-scm.org/repo/hg
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 3 of 3] lfs: dump the full response on httperror in debug mode

[Matt Harbison]

> On Jan 19, 2018, at 10:14 AM, Yuya Nishihara  wrote:
> 
>> On Thu, 18 Jan 2018 18:51:43 -0500, Matt Harbison wrote:
>> # HG changeset patch
>> # User Matt Harbison 
>> # Date 1516316696 18000
>> #  Thu Jan 18 18:04:56 2018 -0500
>> # Node ID ace830f0f944ae48b482e224fa8e8f557f092741
>> # Parent  b25b315be2f43cefdb364e52aa614ee3dcddefed
>> lfs: dump the full response on httperror in debug mode
> 
> Also queued, thanks.
> 
>> diff --git a/hgext/lfs/blobstore.py b/hgext/lfs/blobstore.py
>> --- a/hgext/lfs/blobstore.py
>> +++ b/hgext/lfs/blobstore.py
>> @@ -312,6 +312,8 @@ class _gitlfsremote(object):
>> if response:
>> self.ui.debug('lfs %s response: %s' % (action, response))
>> except util.urlerr.httperror as ex:
>> +if self.ui.debugflag:
>> +self.ui.write('%s: %s' % (oid, ex.read()))
> 
> s/ui.write/ui.debug/ so outputs are labeled.
> 
> The format string is unchanged. I have no idea if ex.read() is terminated by
> '\n'.

Looking at the example again, it’s not. I’ll follow up unless you can amend it.

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 3 of 3] lfs: dump the full response on httperror in debug mode

[Yuya Nishihara]

On Thu, 18 Jan 2018 18:51:43 -0500, Matt Harbison wrote:
> # HG changeset patch
> # User Matt Harbison 
> # Date 1516316696 18000
> #  Thu Jan 18 18:04:56 2018 -0500
> # Node ID ace830f0f944ae48b482e224fa8e8f557f092741
> # Parent  b25b315be2f43cefdb364e52aa614ee3dcddefed
> lfs: dump the full response on httperror in debug mode

Also queued, thanks.

> diff --git a/hgext/lfs/blobstore.py b/hgext/lfs/blobstore.py
> --- a/hgext/lfs/blobstore.py
> +++ b/hgext/lfs/blobstore.py
> @@ -312,6 +312,8 @@ class _gitlfsremote(object):
>  if response:
>  self.ui.debug('lfs %s response: %s' % (action, response))
>  except util.urlerr.httperror as ex:
> +if self.ui.debugflag:
> +self.ui.write('%s: %s' % (oid, ex.read()))

s/ui.write/ui.debug/ so outputs are labeled.

The format string is unchanged. I have no idea if ex.read() is terminated by
'\n'.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] lfs: defer registering the pre-push hook until blobs are committed

[Yuya Nishihara]

On Thu, 18 Jan 2018 21:21:02 -0500, Matt Harbison wrote:
> # HG changeset patch
> # User Matt Harbison 
> # Date 1516328290 18000
> #  Thu Jan 18 21:18:10 2018 -0500
> # Node ID 4e88280e794b1caafabf903a6afadbe0bd4402c0
> # Parent  45b678bf3a787085d56fad5bee494e0c160aa120
> lfs: defer registering the pre-push hook until blobs are committed

Queued, thanks.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 2 of 4] revbranchcache: add a bundle2 handler for a rbc part

[Yuya Nishihara]

On Thu, 18 Jan 2018 16:38:45 +0100, Boris Feld wrote:
> # HG changeset patch
> # User Boris Feld 
> # Date 1516282142 -3600
> #  Thu Jan 18 14:29:02 2018 +0100
> # Node ID 9c1ad82226a2e1fc7ca69550a46a7f7329c6b579
> # Parent  4ad1a1054450063cc9aa19ab2037722c64877eb7
> # EXP-Topic wire-rbc
> # Available At https://bitbucket.org/octobus/mercurial-devel/
> #  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
> 9c1ad82226a2
> revbranchcache: add a bundle2 handler for a rbc part

> diff --git a/mercurial/bundle2.py b/mercurial/bundle2.py
> --- a/mercurial/bundle2.py
> +++ b/mercurial/bundle2.py
> @@ -2101,6 +2101,40 @@ def handlehgtagsfnodes(op, inpart):
>  cache.write()
>  op.ui.debug('applied %i hgtags fnodes cache entries\n' % count)
>  
> +rbcstruct = struct.Struct('>III')
> +
> +@parthandler('cache:rev-branch-cache')
> +def handlerbc(op, inpart):
> +"""receive a rev-branch-cache payload and update the local cache
> +
> +The payload is a series of data related to each branch
> +
> +1) branch name length
> +2) number of open heads
> +3) number of closed heads
> +4) open heads nodes
> +5) closed heads nodes
> +"""
> +total = 0
> +rawheader = inpart.read(rbcstruct.size)
> +cache = op.repo.revbranchcache()
> +cl = op.repo.unfiltered().changelog
> +while rawheader:
> +header = rbcstruct.unpack(rawheader)
> +total += header[1] + header[2]
> +branch = inpart.read(header[0])

We'll probably need to convert branch name between utf-8 and local encoding.

> +for x in xrange(header[1]):
> +node = inpart.read(20)
> +rev = cl.rev(node)
> +cache.setdata(branch, rev, node, False)
> +for x in xrange(header[2]):
> +node = inpart.read(20)
> +rev = cl.rev(node)
> +cache.setdata(branch, rev, node, True)
> +rawheader = inpart.read(rbcstruct.size)

> +if total and 'branchinfo' in vars(cache):
> +del cache.branchinfo

No idea why we have to undo 'self.branchinfo = self._branchinfo'. Can you
add a comment?
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 4 of 4] revbranchcache: advertise and use 'rbc' exchange capability

[Yuya Nishihara]

On Thu, 18 Jan 2018 16:38:47 +0100, Boris Feld wrote:
> # HG changeset patch
> # User Boris Feld 
> # Date 1516284622 -3600
> #  Thu Jan 18 15:10:22 2018 +0100
> # Node ID 70277121f723577a57008607ef0536d23af8df2d
> # Parent  544af0806c0556e6281b38214466d5acd623b5f3
> # EXP-Topic wire-rbc
> # Available At https://bitbucket.org/octobus/mercurial-devel/
> #  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
> 70277121f723
> revbranchcache: advertise and use 'rbc' exchange capability
> 
> The feature is now advertised and use.
> 
> Updating the branchmap cache can be very expensive (up to minutes on large
> repository) and fetching revision branch data is about 80% of that. Exchanging
> the rev branch cache over the wire really help to recover from branch map
> invalidation.

Correct me if I get it wrong. This is fast mainly because we no longer need
to uncompress changelog data to retrieve branch names.

It would be nice if some benchmark result were included in the commit message.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 02 of 14] util: implement varint functions

[Yuya Nishihara]

On Thu, 18 Jan 2018 12:21:28 +0100, Boris Feld wrote:
> # HG changeset patch
> # User Gregory Szorc 
> # Date 1474779566 25200
> #  Sat Sep 24 21:59:26 2016 -0700
> # Node ID 1262be5f656bb0597b59a7dd2139b81922829fae
> # Parent  939c242897c47ede8cda0b0a0149e15b74803402
> # EXP-Topic b2-stream
> # Available At https://bitbucket.org/octobus/mercurial-devel/
> #  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
> 1262be5f656b
> util: implement varint functions

(I stopped here. This series will need to be reviewed by someone having more
expertise, and I don't think I can finish reviewing today.)

> +def uvarintencode(value):
> +"""Encode an unsigned integer value to a varint.
> +
> +A varint is a variable length integer of 1 or more bytes. Each byte
> +except the last has the most significant bit set. The lower 7 bits of
> +each byte store the 2's complement representation, least significant 
> group
> +first.
> +"""
> +bits = value & 0x7f
> +value >>= 7
> +bytes = []
> +while value:
> +bytes.append(chr(0x80 | bits))
> +bits = value & 0x7f
> +value >>= 7
> +bytes.append(chr(bits))

Nit: use pycompat.bytechr()

> +return ''.join(bytes)
> +
> +def uvarintdecodestream(fh):
> +"""Decode an unsigned variable length integer from a stream.
> +
> +The passed argument is anything that has a ``.read(N)`` method.
> +"""
> +result = 0
> +shift = 0
> +while True:
> +byte = ord(fh.read(1))

Need to test EOF?

Alternatively, we could split this function into
 a) read bytes while MSB is set
 b) decode bytes into integer (in reverse order)

> +result |= ((byte & 0x7f) << shift)
> +if not (byte & 0x80):
> +return result
> +shift += 7
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1074: branch: add a --rev flag to change branch name of given revisions

[pulkit (Pulkit Goyal)]

pulkit updated this revision to Diff 4935.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1074?vs=4934=4935

REVISION DETAIL
  https://phab.mercurial-scm.org/D1074

AFFECTED FILES
  mercurial/cmdutil.py
  mercurial/commands.py
  tests/test-branch-change.t
  tests/test-completion.t

CHANGE DETAILS

diff --git a/tests/test-completion.t b/tests/test-completion.t
--- a/tests/test-completion.t
+++ b/tests/test-completion.t
@@ -242,7 +242,7 @@
   backout: merge, commit, no-commit, parent, rev, edit, tool, include, 
exclude, message, logfile, date, user
   bisect: reset, good, bad, skip, extend, command, noupdate
   bookmarks: force, rev, delete, rename, inactive, template
-  branch: force, clean
+  branch: force, clean, rev
   branches: active, closed, template
   bundle: force, rev, branch, base, all, type, ssh, remotecmd, insecure
   cat: output, rev, decode, include, exclude, template
diff --git a/tests/test-branch-change.t b/tests/test-branch-change.t
new file mode 100644
--- /dev/null
+++ b/tests/test-branch-change.t
@@ -0,0 +1,296 @@
+Testing changing branch on commits
+==
+
+Setup
+
+  $ cat >> $HGRCPATH << EOF
+  > [alias]
+  > glog = log -G -T "{rev}:{node|short} {desc}\n{branch} ({bookmarks})"
+  > [experimental]
+  > evolution = createmarkers
+  > [extensions]
+  > rebase=
+  > EOF
+
+  $ hg init repo
+  $ cd repo
+  $ for ch in a b c d e; do echo foo >> $ch; hg ci -Aqm "Added "$ch; done
+  $ hg glog
+  @  4:aa98ab95a928 Added e
+  |  default ()
+  o  3:62615734edd5 Added d
+  |  default ()
+  o  2:28ad74487de9 Added c
+  |  default ()
+  o  1:29becc82797a Added b
+  |  default ()
+  o  0:18d04c59bb5d Added a
+ default ()
+
+  $ hg branches
+  default4:aa98ab95a928
+
+Try without passing a new branch name
+
+  $ hg branch -r .
+  abort: no branch name specified for the revisions
+  [255]
+
+Setting an invalid branch name
+
+  $ hg branch -r . a:b
+  abort: ':' cannot be used in a name
+  [255]
+  $ hg branch -r . tip
+  abort: the name 'tip' is reserved
+  [255]
+  $ hg branch -r . 1234
+  abort: cannot use an integer as a name
+  [255]
+
+Change on non-linear set of commits
+
+  $ hg branch -r 2 -r 4 foo
+  abort: cannot change branch of non-linear revisions
+  [255]
+
+Change in middle of the stack (linear commits)
+
+  $ hg branch -r 1::3 foo
+  abort: cannot change branch of changeset with children
+  [255]
+
+Change with dirty working directory
+
+  $ echo bar > a
+  $ hg branch -r . foo
+  abort: uncommitted changes
+  [255]
+
+  $ hg revert --all
+  reverting a
+
+Change on empty revision set
+
+  $ hg branch -r 'draft() - all()' foo
+  abort: empty revision set
+  [255]
+
+Changing branch on linear set of commits from head
+
+Without obsmarkers
+
+  $ hg branch -r 3:4 foo --config experimental.evolution=!
+  changed branch on 2 changesets
+  saved backup bundle to 
$TESTTMP/repo/.hg/strip-backup/62615734edd5-e86bd13a-branch-change.hg (glob)
+  $ hg glog
+  @  4:3938acfb5c0f Added e
+  |  foo ()
+  o  3:9435da006bdc Added d
+  |  foo ()
+  o  2:28ad74487de9 Added c
+  |  default ()
+  o  1:29becc82797a Added b
+  |  default ()
+  o  0:18d04c59bb5d Added a
+ default ()
+
+  $ hg branches
+  foo4:3938acfb5c0f
+  default2:28ad74487de9 (inactive)
+
+With obsmarkers
+
+  $ hg branch -r 3::4 bar
+  changed branch on 2 changesets
+  $ hg glog
+  @  6:7c1991464886 Added e
+  |  bar ()
+  o  5:1ea05e93925f Added d
+  |  bar ()
+  o  2:28ad74487de9 Added c
+  |  default ()
+  o  1:29becc82797a Added b
+  |  default ()
+  o  0:18d04c59bb5d Added a
+ default ()
+
+  $ hg branches
+  bar6:7c1991464886
+  default2:28ad74487de9 (inactive)
+
+Change branch name to an existing branch
+
+  $ hg branch -r . default
+  abort: a branch of the same name already exists
+  [255]
+
+Changing on a branch head which is not topological head
+
+  $ hg branch -r 2 stable
+  abort: cannot change branch of changeset with children
+  [255]
+
+Enabling the allowunstable config and trying to change branch on a branch head
+which is not a topological head
+
+  $ echo "[experimental]" >> .hg/hgrc
+  $ echo "evolution.allowunstable=yes" >> .hg/hgrc
+  $ hg branch -r 2 foo
+  changed branch on 1 changesets
+  2 new orphan changesets
+
+Changing branch of an obsoleted changeset
+
+  $ hg branch -r 4 foobar
+  abort: hidden revision '4' was rewritten as: 7c1991464886!
+  (use --hidden to access hidden revisions)
+  [255]
+
+  $ hg branch -r 4 --hidden foobar
+  abort: cannot change branch of a obsolete changeset
+  [255]
+
+Make sure bookmark movement is correct
+
+  $ hg bookmark b1
+  $ hg glog -r '.^::'
+  @  6:7c1991464886 Added e
+  |  bar (b1)
+  *  5:1ea05e93925f Added d
+  |  bar ()
+  ~
+
+  $ hg branch -r '(.^)::' wat --debug
+  changing branch of '1ea05e93925f806d875a2163f9b76764be644636' from 'bar' to 
'wat'
+  

D1913: branch: allow changing branch name to existing name if possible

[pulkit (Pulkit Goyal)]

pulkit created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  With the functionality added in previous patch we can change branches of a
  revision but not everytime even if it's possible to do so. For example cosider
  the following case:
  
  o 3  added a (foo)
  o 2  added b (foo)
  o 1  added c (bar)
  o 0  added d (bar)
  
  Here if I want to change the branch of rev 2,3 to bar, it was not possible and
  it will say, "a branch with same name exists".
  
  This patch allows us to change branch of 2,3 to bar. The underlying logic for
  changing branch finds the changesets from the revs passed which have no 
parents
  in revs. We only support revsets which have only one such root, so to support
  this we check whether the parent of the root has the same name as that of the
  new name and if so, we can use the new name to change branches.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1913

AFFECTED FILES
  mercurial/cmdutil.py
  mercurial/commands.py
  tests/test-branch-change.t

CHANGE DETAILS

diff --git a/tests/test-branch-change.t b/tests/test-branch-change.t
--- a/tests/test-branch-change.t
+++ b/tests/test-branch-change.t
@@ -267,15 +267,49 @@
   $ hg branch
   stable
 
-Changing to same branch name does not work
+Changing to same branch is no-op
 
   $ hg branch -r 19::21 stable
-  abort: a branch of the same name already exists
-  [255]
+  changed branch on 0 changesets
+
+Changing branch name to existing branch name if the branch of parent of root of
+revs is same as the new branch name
+
+  $ hg branch -r 20::21 bugfix
+  changed branch on 2 changesets
+  $ hg glog
+  o  25:714defe1cf34 Added d
+  |  bugfix ()
+  o  24:98394def28fc Added c
+  |  bugfix ()
+  | @  23:6a5ddbcfb870 added bar
+  | |  stable (b1)
+  | o  22:baedc6e98a67 Added e
+  |/   stable ()
+  o  19:fd45b986b109 Added b
+  |  stable ()
+  o  18:204d2769eca2 Added a
+ stable ()
+
+  $ hg branch -r 24:25 stable
+  changed branch on 2 changesets
+  $ hg glog
+  o  27:4ec342341562 Added d
+  |  stable ()
+  o  26:83f48859c2de Added c
+  |  stable ()
+  | @  23:6a5ddbcfb870 added bar
+  | |  stable (b1)
+  | o  22:baedc6e98a67 Added e
+  |/   stable ()
+  o  19:fd45b986b109 Added b
+  |  stable ()
+  o  18:204d2769eca2 Added a
+ stable ()
 
 Testing on merge
 
-  $ hg merge -r 20
+  $ hg merge -r 26
   1 files updated, 0 files merged, 0 files removed, 0 files unresolved
   (branch merge, don't forget to commit)
 
@@ -289,8 +323,8 @@
 
 Changing branch on public changeset
 
-  $ hg phase -r 21 -p
-  $ hg branch -r 21 def
+  $ hg phase -r 27 -p
+  $ hg branch -r 27 def
   abort: cannot change branch of public changesets
   (see 'hg help phases' for details)
   [255]
diff --git a/mercurial/commands.py b/mercurial/commands.py
--- a/mercurial/commands.py
+++ b/mercurial/commands.py
@@ -1055,11 +1055,6 @@
 
 scmutil.checknewlabel(repo, label, 'branch')
 if revs:
-# XXX: we should allow setting name to existing branch if the
-# branch of root of the revs is same as the new branch name
-if label in repo.branchmap():
-raise error.Abort(_('a branch of the same'
-' name already exists'))
 return cmdutil.changebranch(ui, repo, revs, label)
 
 if not opts.get('force') and label in repo.branchmap():
diff --git a/mercurial/cmdutil.py b/mercurial/cmdutil.py
--- a/mercurial/cmdutil.py
+++ b/mercurial/cmdutil.py
@@ -727,6 +727,11 @@
 if len(roots) > 1:
 raise error.Abort(_("cannot change branch of non-linear 
revisions"))
 rewriteutil.precheck(repo, revs, 'change branch of')
+
+root = repo[roots.first()]
+if not root.p1().branch() == label and label in repo.branchmap():
+raise error.Abort(_("a branch of the same name already exists"))
+
 if repo.revs('merge() and %ld', revs):
 raise error.Abort(_("cannot change branch of a merge commit"))
 if repo.revs('obsolete() and %ld', revs):



To: pulkit, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1074: branch: add a --rev flag to change branch name of given revisions

[pulkit (Pulkit Goyal)]

pulkit added inline comments.

INLINE COMMENTS

> cmdutil.py:739
> +if repo.revs('(%ld::head()) - %ld', revs, revs):
> +raise error.Abort(_("cannot change branch in middle of a stack"))
> +

After some more testing, I found this is not correct and raise the error even 
in cases when it should not. For example:

o  3

| o  2 |
| /|
|

o  1
o  0

Here if I want to change the branch of 1::2, this will raise error however this 
should be completely fine if we have allowunstable enabled. So in next version, 
I am going to add a test for allowunstable and if that's enabled, allow to 
change branch of 1::2.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1074

To: pulkit, #hg-reviewers, dlax, ryanmce, lothiraldan
Cc: martinvonz, lothiraldan, ryanmce, dlax, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1909: cmdutil: add a kludge to make bytes repr() the same on 2 and 3

[yuja (Yuya Nishihara)]

yuja accepted this revision.
yuja added a comment.
This revision is now accepted and ready to land.


  This looks ugly, but seems okay. Since `sysstr()` decodes bytes as latin-1,
  8-bit characters will be escaped by `\x`, not by `\u`.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1909

To: durin42, #hg-reviewers, yuja
Cc: yuja, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH] localrepo: micro-optimize __len__() to bypass repoview

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1516365551 -32400
#  Fri Jan 19 21:39:11 2018 +0900
# Node ID 03870a74aaca105aa0b0f1c5edd68754098be5bc
# Parent  f58245b9e3ea6e8945a81b951110cde155819345
localrepo: micro-optimize __len__() to bypass repoview

Since len(changelog) isn't overridden, we don't have to validate a cache of
unfiltered changelog.

  $ python -m timeit -n 1 \
   -s 'from mercurial import hg, ui; repo = hg.repository(ui.ui());' \
   'len(repo)'
  orig) 1 loops, best of 3: 32.1 usec per loop
  new)  1 loops, best of 3: 1.79 usec per loop

Spotted by Jordi Gutiérrez Hermoso.

diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -762,7 +762,9 @@ class localrepository(object):
 __bool__ = __nonzero__
 
 def __len__(self):
-return len(self.changelog)
+# no need to pay the cost of repoview.changelog
+unfi = self.unfiltered()
+return len(unfi.changelog)
 
 def __iter__(self):
 return iter(self.changelog)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 2 of 2] templatekw: add a {negrev} keyword

[Yuya Nishihara]

On Thu, 18 Jan 2018 10:35:09 -0500, Jordi Gutiérrez Hermoso wrote:
> On Thu, 2018-01-18 at 21:05 +0900, Yuya Nishihara wrote:
> > On Wed, 17 Jan 2018 22:14:30 -0500, Jordi Gutiérrez Hermoso wrote:
> > > # HG changeset patch
> > > # User Jordi Gutiérrez Hermoso 
> > > # Date 1516243120 18000
> > > #  Wed Jan 17 21:38:40 2018 -0500
> > > # Node ID cbf1d676a938e78d40cd3504dd916f787bcb47ee
> > > # Parent  701f8a9defdc09bb63f2596e2fc426f2e78da313
> > > templatekw: add a {negrev} keyword
> > > 
> > > Revision numbers are getting much maligned for two reasons: they
> > > are
> > > too long in large repos and users get confused by their local-only
> > > nature. It just occurred to me that negative revision numbers avoid
> > > both of those problems. Since negative revision numbers change
> > > whenever the repo changes, it's much more obvious that they are a
> > > local-only convenience. Additionally, for the recent commits that
> > > we
> > > usually care about the most, negative revision numbers are always
> > > near
> > > zero.
> > > 
> > > This commit adds a negrev templatekw to more easily expose negative
> > > revision numbers. It's not easy to reliably produce this output
> > > with
> > > existing keywords due to hidden commits or without a significant
> > > slowdown due to computing a complicated template expression.
> > 
> > Isn't it "{rev - revset("tip")|stringify - 1}" ? Well, the stringify
> > looks strange, but not terrible.
> 
> That's incorrect if tip is hidden.

Good point. Perhaps we'll need to update the help or change the behavior
of negative integer revisions. Currently "help revisions" says "sequential
offsets from the tip, with -1 denoting the tip."

> And much noticeably slower.

What use case do you suppose?

It's slow to run complex template, but in this case, it's just 34.5us per
revision. I don't think it's noticeable while reading log in pager.

  % hg log -T '{rev}\n' --time > /dev/null
  time: real 1.390 secs (user 1.390+0.000 sys 0.000+0.000)
  % hg log -T '{rev - revset("tip")|stringify - 1}\n' --time > /dev/null
  time: real 2.620 secs (user 2.610+0.000 sys 0.010+0.000)

'{desc}' is much slower by the way.

  % hg log -T '{rev} {desc|firstline}\n' --time > /dev/null
  time: real 3.740 secs (user 3.710+0.000 sys 0.010+0.000)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 3 of 4 V3] largefiles: add support for 'largefiles://' url scheme

[Yuya Nishihara]

On Thu, 18 Jan 2018 17:14:18 +0100, Boris Feld wrote:
> `On Tue, 2018-01-09 at 21:08 +0900, Yuya Nishihara wrote:
> > On Mon, 08 Jan 2018 22:16:47 +0100, Boris Feld wrote:
> > > # HG changeset patch
> > > # User Boris Feld 
> > > # Date 1513861077 -3600
> > > #  Thu Dec 21 13:57:57 2017 +0100
> > > # Node ID 18db7b5d796d31fbced9d41dd50a61d0e62e6fcb
> > > # Parent  a503a19221d6c6113ad1e3add9eb084be3177daf
> > > # EXP-Topic largefile-url
> > > # Available At https://bitbucket.org/octobus/mercurial-devel/
> > > #  hg pull https://bitbucket.org/octobus/mercurial-deve
> > > l/ -r 18db7b5d796d
> > > largefiles: add support for 'largefiles://' url scheme
> > > +_lfscheme = 'largefile://'
> > > +def openlargefile(orig, ui, url_, data=''):
> > 
> > Nit: s/data=''/data=None/ according to the signature of url.open().
> > 
> > > +if url_.startswith(_lfscheme):
> > > +if data:
> > > +msg = "cannot use data on a 'largefile://' url"
> > > +raise error.ProgrammingError(msg)
> > 
> > If the error can be triggered by user (by e.g. setting crafted repo
> > path),
> > it shouldn't be a ProgrammingError.
> 
> How could the user trigger that?

No idea. If there's no url.open(userspecifiedurl, data) call, ProgrammingError
is okay.

> > > +def getlfile(ui, hash):
> > > +return util.chunkbuffer(openstore(ui=ui)._get(hash))
> > 
> > AttributeError would be raised if the default path isn't remote. Can
> > you
> > send a follow up?
> 
> What attribute error? Could you be a bit more specific?


--- tests/test-url-download.t
+++ tests/test-url-download.t.err
@@ -68,4 +68,46 @@
   $ cd ..
 
   $ hg clone client client2
   updating to branch default
   0 files updated, 0 files merged, 0 files removed, 0 files unresolved
   $ hg -R client2 debugdownload 
"largefile://a57b57b39ee4dc3da1e03526596007f480ecdbe8"
+  ** unknown exception encountered, please report by visiting
+  ** https://mercurial-scm.org/wiki/BugTracker
+  ** Python 2.7.14+ (default, Dec  5 2017, 15:17:02) [GCC 7.2.0]
+  ** Mercurial Distributed SCM (version 4.4.2+690-a62b08f6626b)
+  ** Extensions loaded: largefiles
+  Traceback (most recent call last):
+File "hg", line 41, in 
+  dispatch.run()
+File "mercurial/dispatch.py", line 88, in run
+  status = (dispatch(req) or 0) & 255
+File "mercurial/dispatch.py", line 183, in dispatch
+  ret = _runcatch(req)
+File "mercurial/dispatch.py", line 324, in _runcatch
+  return _callcatch(ui, _runcatchfunc)
+File "mercurial/dispatch.py", line 332, in _callcatch
+  return scmutil.callcatch(ui, func)
+File "mercurial/scmutil.py", line 154, in callcatch
+  return func()
+File "mercurial/dispatch.py", line 314, in _runcatchfunc
+  return _dispatch(req)
+File "mercurial/dispatch.py", line 918, in _dispatch
+  cmdpats, cmdoptions)
+File "mercurial/dispatch.py", line 673, in runcommand
+  ret = _runcommand(ui, options, cmd, d)
+File "mercurial/dispatch.py", line 926, in _runcommand
+  return cmdfunc()
+File "mercurial/dispatch.py", line 915, in 
+  d = lambda: util.checksignature(func)(ui, *args, **strcmdopt)
+File "mercurial/util.py", line 1195, in check
+  return func(*args, **kwargs)
+File "mercurial/debugcommands.py", line 804, in debugdownload
+  fh = urlmod.open(ui, url, output)
+File "mercurial/extensions.py", line 344, in closure
+  return func(*(args + a), **kw)
+File "hgext/largefiles/overrides.py", line 1490, in openlargefile
+  return storefactory.getlfile(ui, lfid)
+File "hgext/largefiles/storefactory.py", line 86, in getlfile
+  return util.chunkbuffer(openstore(ui=ui)._get(hash))
+  AttributeError: 'localstore' object has no attribute '_get'
+  [1]
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH V2] lfs: add the '{lfsattrs}' template keyword to '{lfs_files}'

[Yuya Nishihara]

On Fri, 19 Jan 2018 00:05:42 -0500, Matt Harbison wrote:
> # HG changeset patch
> # User Matt Harbison 
> # Date 1515967224 18000
> #  Sun Jan 14 17:00:24 2018 -0500
> # Node ID fccf09e44f5124abf18ae898fab553ea6d91e951
> # Parent  45b678bf3a787085d56fad5bee494e0c160aa120
> lfs: add the '{lfsattrs}' template keyword to '{lfs_files}'

Queued updated version, thanks.

> I liked {pointer} better, but couldn't make it work with the singular/plural
> forms.

I think {pointer} is okay here since its singular form is ({key}, {value}).

> @@ -303,6 +304,8 @@
>  # when writing a bundle via "hg bundle" command, upload related LFS blobs
>  wrapfunction(bundle2, 'writenewbundle', wrapper.writenewbundle)
>  
> +templatekw.defaulttempl['lfsattr'] = '{key}={value}'

This isn't needed. Dropped.

> @@ -311,9 +314,16 @@
>  pointers = wrapper.pointersfromctx(ctx) # {path: pointer}
>  files = sorted(pointers.keys())
>  
> +def lfsattrs(v):
> +# In the file spec, version is first and the other keys are sorted.
> +sortkeyfunc = lambda x: (x[0] != 'version', x)
> +items = sorted(pointers[v].iteritems(), key=sortkeyfunc)
> +return util.sortdict(kv for kv in items)

s/kv for kv in//

> diff --git a/tests/test-lfs.t b/tests/test-lfs.t
> --- a/tests/test-lfs.t
> +++ b/tests/test-lfs.t
> @@ -865,6 +865,17 @@
>oid sha256:5bb8341bee63b3649f222b2215bde37322bea075a30575aa685d8f8d21c77024
>size 29
>x-is-binary 0
> +  $ hg --cwd convert_lfs log -r 0 -T "{lfs_files % '{lfsattrs % 
> \"{lfsattr}\n\"}'}"

This example just applies defaulttempl['lfsattr']. Inlined '{key}={value}'.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] debugdeltachain: cleanup the double call to _slicechunk

[Yuya Nishihara]

On Fri, 19 Jan 2018 08:48:35 +0100, Paul Morelle wrote:
> # HG changeset patch
> # User Paul Morelle 
> # Date 1516347322 -3600
> #  Fri Jan 19 08:35:22 2018 +0100
> # Node ID ce3b9d44cbf7de6fa0faad42442797cee9b337c8
> # Parent  45b678bf3a787085d56fad5bee494e0c160aa120
> # EXP-Topic debug-sparse-read
> # Available At https://bitbucket.org/octobus/mercurial-devel/
> #  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
> ce3b9d44cbf7
> debugdeltachain: cleanup the double call to _slicechunk

Queued, thanks.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] blackbox: don't unpack the list while passing into str.join()

[Yuya Nishihara]

On Fri, 19 Jan 2018 14:24:47 +0530, Pulkit Goyal wrote:
> # HG changeset patch
> # User Pulkit Goyal <7895pul...@gmail.com>
> # Date 1516351218 -19800
> #  Fri Jan 19 14:10:18 2018 +0530
> # Node ID d8c9a1f2aa9c588434ffd4e28c4719d3c4ca5e51
> # Parent  91a74c6c3e88634a9ac6bf9f4161c253fe1c4331
> blackbox: don't unpack the list while passing into str.join()

Queued, thanks.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] test-blackbox: stabilize for Windows

[Yuya Nishihara]

On Fri, 19 Jan 2018 00:41:10 -0500, Matt Harbison wrote:
> # HG changeset patch
> # User Matt Harbison 
> # Date 1516339125 18000
> #  Fri Jan 19 00:18:45 2018 -0500
> # Node ID b02eb22589468c72dc49d83d272caa909f01fe65
> # Parent  fccf09e44f5124abf18ae898fab553ea6d91e951
> test-blackbox: stabilize for Windows

Queued, thanks.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] debugdownload: read repository hgrc if there is one

[Yuya Nishihara]

On Thu, 18 Jan 2018 17:10:23 +0100, Boris Feld wrote:
> # HG changeset patch
> # User Boris Feld 
> # Date 1516290434 -3600
> #  Thu Jan 18 16:47:14 2018 +0100
> # Node ID 197a41c16b737d38c0d1bc9f1608073a9da7acc1
> # Parent  9c575c22dcf4305db2160dc45350ba8e40cf5e60
> # EXP-Topic lfurl
> # Available At https://bitbucket.org/octobus/mercurial-devel/
> #  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
> 197a41c16b73
> debugdownload: read repository hgrc if there is one

Queued, thanks.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH] blackbox: don't unpack the list while passing into str.join()

[Pulkit Goyal]

# HG changeset patch
# User Pulkit Goyal <7895pul...@gmail.com>
# Date 1516351218 -19800
#  Fri Jan 19 14:10:18 2018 +0530
# Node ID d8c9a1f2aa9c588434ffd4e28c4719d3c4ca5e51
# Parent  91a74c6c3e88634a9ac6bf9f4161c253fe1c4331
blackbox: don't unpack the list while passing into str.join()

The current state may result in error TypeError.
Caught using evolve-tests.

diff --git a/hgext/blackbox.py b/hgext/blackbox.py
--- a/hgext/blackbox.py
+++ b/hgext/blackbox.py
@@ -133,7 +133,7 @@ def wrapui(ui):
 def debug(self, *msg, **opts):
 super(blackboxui, self).debug(*msg, **opts)
 if self.debugflag:
-self.log('debug', '%s', ''.join(*msg))
+self.log('debug', '%s', ''.join(msg))
 
 def log(self, event, *msg, **opts):
 global lastui
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 2 V2] revlog: group delta computation methods under _deltacomputer object

[Paul Morelle]

# HG changeset patch
# User Paul Morelle 
# Date 1515961692 -3600
#  Sun Jan 14 21:28:12 2018 +0100
# Node ID 82018742fcabfd0e26aa6f34bf773b6f25d27985
# Parent  32bc4595737c2211dfcbf53cb499a366b9986dfd
# EXP-Topic refactor-revlog
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
82018742fcab
revlog: group delta computation methods under _deltacomputer object

Extracting these methods from revlog will allow changing the implementation of
the deltacomputer, by providing this interface:
  __init__(self, revlog) - constructor that initialize the object from a given
   revlog
  buildtext(self, revinfo, fh) - builds the fulltext version of a revision from
 a _revisioninfo object and the file handle to
 the .d (or .i for inline mode) file.
  finddeltainfo(self, revinfo, fh) - find a revision in the revlog against
 which it is acceptable to build a delta,
 and build the corresponding _deltainfo.

It should now be easier to write an experimental feature that would replace
_deltacomputer by another object, for example one that would know how to
parallelize the delta computation in order to quicken the storage of multiple
revisions.

diff -r 32bc4595737c -r 82018742fcab mercurial/revlog.py
--- a/mercurial/revlog.py   Sun Jan 14 14:36:22 2018 +0100
+++ b/mercurial/revlog.py   Sun Jan 14 21:28:12 2018 +0100
@@ -264,6 +264,155 @@
 chainlen = attr.ib()
 compresseddeltalen = attr.ib()
 
+class _deltacomputer(object):
+def __init__(self, revlog):
+self.revlog = revlog
+
+def _getcandidaterevs(self, p1, p2, cachedelta):
+"""
+Provides revisions that present an interest to be diffed against,
+grouped by level of easiness.
+"""
+revlog = self.revlog
+curr = len(revlog)
+prev = curr - 1
+p1r, p2r = revlog.rev(p1), revlog.rev(p2)
+
+# should we try to build a delta?
+if prev != nullrev and revlog.storedeltachains:
+tested = set()
+# This condition is true most of the time when processing
+# changegroup data into a generaldelta repo. The only time it
+# isn't true is if this is the first revision in a delta chain
+# or if ``format.generaldelta=true`` disabled ``lazydeltabase``.
+if cachedelta and revlog._generaldelta and revlog._lazydeltabase:
+# Assume what we received from the server is a good choice
+# build delta will reuse the cache
+yield (cachedelta[0],)
+tested.add(cachedelta[0])
+
+if revlog._generaldelta:
+# exclude already lazy tested base if any
+parents = [p for p in (p1r, p2r)
+   if p != nullrev and p not in tested]
+if parents and not revlog._aggressivemergedeltas:
+# Pick whichever parent is closer to us (to minimize the
+# chance of having to build a fulltext).
+parents = [max(parents)]
+tested.update(parents)
+yield parents
+
+if prev not in tested:
+# other approach failed try against prev to hopefully save us a
+# fulltext.
+yield (prev,)
+
+def buildtext(self, revinfo, fh):
+"""Builds a fulltext version of a revision
+
+revinfo: _revisioninfo instance that contains all needed info
+fh:  file handle to either the .i or the .d revlog file,
+ depending on whether it is inlined or not
+"""
+btext = revinfo.btext
+if btext[0] is not None:
+return btext[0]
+
+revlog = self.revlog
+cachedelta = revinfo.cachedelta
+flags = revinfo.flags
+node = revinfo.node
+
+baserev = cachedelta[0]
+delta = cachedelta[1]
+# special case deltas which replace entire base; no need to decode
+# base revision. this neatly avoids censored bases, which throw when
+# they're decoded.
+hlen = struct.calcsize(">lll")
+if delta[:hlen] == mdiff.replacediffheader(revlog.rawsize(baserev),
+   len(delta) - hlen):
+btext[0] = delta[hlen:]
+else:
+basetext = revlog.revision(baserev, _df=fh, raw=True)
+btext[0] = mdiff.patch(basetext, delta)
+
+try:
+res = revlog._processflags(btext[0], flags, 'read', raw=True)
+btext[0], validatehash = res
+if validatehash:
+revlog.checkhash(btext[0], node, p1=revinfo.p1, p2=revinfo.p2)
+if flags & REVIDX_ISCENSORED:
+