date:20180716

D3958: Allow to run setup.py with python 3 without a mercurial checkout

2018-07-16 Thread indygreg (Gregory Szorc)

This revision was automatically updated to reflect the committed changes.
Closed by commit rHGd5b31ae9cc0a: Allow to run setup.py with python 3 without a 
mercurial checkout (authored by indygreg, committed by ).

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D3958?vs=9614=9615

REVISION DETAIL
  https://phab.mercurial-scm.org/D3958

AFFECTED FILES
  setup.py

CHANGE DETAILS

diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -74,7 +74,7 @@
 badpython = True
 
 # Allow Python 3 from source checkouts.
-if os.path.isdir('.hg'):
+if os.path.isdir('.hg') or 'HGPYTHON3' in os.environ:
 badpython = False
 
 if badpython:
@@ -369,7 +369,7 @@
 from mercurial import __version__
 version = __version__.version
 except ImportError:
-version = 'unknown'
+version = b'unknown'
 finally:
 if oldpolicy is None:
 del os.environ['HGMODULEPOLICY']



To: glandium, indygreg, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D3958: Allow to run setup.py with python 3 without a mercurial checkout

2018-07-16 Thread indygreg (Gregory Szorc)

indygreg accepted this revision.
indygreg added a comment.
This revision is now accepted and ready to land.


  I'm OK with this undocumented hack. And there are cases where we may want to 
test setup.py outside the context of a checkout as well. The important thing is 
end-users not running Mercurial with Python 3 unknowingly.
  
  The only thing I could bikeshed here is the naming. History tells us that 
someone will find this hack and e.g. post it to stackoverflow without the 
context that Mercurial is unsupported on Python 3. Perhaps a name like 
`HGALLOWUNSUPPORTEDPYTHON3` would be better?
  
  I'll queue this for now. We can change the name easily enough if others want 
that.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D3958

To: glandium, indygreg, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D3958: Allow to run setup.py with python 3 without a mercurial checkout

2018-07-16 Thread glandium (Mike Hommey)

glandium created this revision.
glandium added a reviewer: indygreg.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  Some people may want to test mercurial in a python 3 environment through e.g. 
pip, in which case setup.py doesn't run in a mercurial checkout, so the hack in 
setup.py to allow python 3 cannot be overcome.
  
  This change allows a manual override with the HGPYTHON3 environment variable.
  
  Additionally, when for some reason the version is unknown (for crazy people 
like me, who have a git checkout of the mercurial repo), the version variable 
ends up being an unicode string, which fails the `isinstance(version, bytes)` 
assertion. So fix that at the same time.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D3958

AFFECTED FILES
  setup.py

CHANGE DETAILS

diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -74,7 +74,7 @@
 badpython = True
 
 # Allow Python 3 from source checkouts.
-if os.path.isdir('.hg'):
+if os.path.isdir('.hg') or 'HGPYTHON3' in os.environ:
 badpython = False
 
 if badpython:
@@ -369,7 +369,7 @@
 from mercurial import __version__
 version = __version__.version
 except ImportError:
-version = 'unknown'
+version = b'unknown'
 finally:
 if oldpolicy is None:
 del os.environ['HGMODULEPOLICY']



To: glandium, indygreg, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

mercurial@38704: 27 new changesets

2018-07-16 Thread Mercurial Commits

27 new changesets in mercurial:

https://www.mercurial-scm.org/repo/hg/rev/7e4a856a4f05
changeset:   38678:7e4a856a4f05
user:Joerg Sonnenberger 
date:Thu Jul 12 15:29:03 2018 +0200
summary: pullbundle: fix handling of gzip bundlespecs

https://www.mercurial-scm.org/repo/hg/rev/b3d0c97a0820
changeset:   38679:b3d0c97a0820
user:Sushil khanchi 
date:Sat Jul 14 08:59:42 2018 +0530
summary: rebase: in --confirm option just abort if hit a conflict

https://www.mercurial-scm.org/repo/hg/rev/70448ec995c2
changeset:   38680:70448ec995c2
user:Sushil khanchi 
date:Sat Jul 14 23:00:22 2018 +0530
summary: rebase: remove unnecessary confirm block

https://www.mercurial-scm.org/repo/hg/rev/8eeed92475d5
changeset:   38681:8eeed92475d5
user:Sushil khanchi 
date:Sun Jul 15 15:48:18 2018 +0530
summary: rebase: remove unused variable "release" and an extra blank line

https://www.mercurial-scm.org/repo/hg/rev/88efb7d6bcb6
changeset:   38682:88efb7d6bcb6
user:Yuya Nishihara 
date:Sat Jul 14 13:19:18 2018 +0900
summary: phases: remove excessive optimization from newheads() (issue5939)

https://www.mercurial-scm.org/repo/hg/rev/ddd2165438cd
changeset:   38683:ddd2165438cd
user:Yuya Nishihara 
date:Sat Jul 14 13:21:49 2018 +0900
summary: phases: micro-optimize newheads() to not create context objects

https://www.mercurial-scm.org/repo/hg/rev/07b551a4df44
changeset:   38684:07b551a4df44
user:Yuya Nishihara 
date:Sun Jun 10 20:02:53 2018 +0900
summary: fileset: add helpers to make predicatematcher and nevermatcher

https://www.mercurial-scm.org/repo/hg/rev/80466fd85ac9
changeset:   38685:80466fd85ac9
user:Yuya Nishihara 
date:Sat Jun 09 18:11:49 2018 +0900
summary: fileset: rewrite andset() to not use mctx.narrow()

https://www.mercurial-scm.org/repo/hg/rev/131aae58a316
changeset:   38686:131aae58a316
user:Yuya Nishihara 
date:Sun Jun 10 22:19:56 2018 +0900
summary: fileset: add "tracked()" to explicitly select files in the revision

https://www.mercurial-scm.org/repo/hg/rev/1500cbe22d53
changeset:   38687:1500cbe22d53
user:Yuya Nishihara 
date:Sun Jun 10 20:58:10 2018 +0900
summary: fileset: parse argument of size() by predicate function

https://www.mercurial-scm.org/repo/hg/rev/2570dca0f21c
changeset:   38688:2570dca0f21c
user:Yuya Nishihara 
date:Tue Jul 10 23:49:48 2018 +0900
summary: highlight: use matcher API to test if file should be pygmentized

https://www.mercurial-scm.org/repo/hg/rev/ff5b6fca1082
changeset:   38689:ff5b6fca1082
user:Yuya Nishihara 
date:Sat Jun 09 19:55:10 2018 +0900
summary: fileset: rewrite predicates to return matcher not closed to subset 
(API) (BC)

https://www.mercurial-scm.org/repo/hg/rev/5d9749c598f0
changeset:   38690:5d9749c598f0
user:Yuya Nishihara 
date:Sat Jun 09 22:31:51 2018 +0900
summary: fileset: remove callexisting flag and mctx.existing() (API)

https://www.mercurial-scm.org/repo/hg/rev/370ff3e34160
changeset:   38691:370ff3e34160
user:Yuya Nishihara 
date:Sat Jun 09 22:35:21 2018 +0900
summary: fileset: remove subset and unused filtering functions from matchctx

https://www.mercurial-scm.org/repo/hg/rev/3700564c63fe
changeset:   38692:3700564c63fe
user:Yuya Nishihara 
date:Sat Jun 09 22:38:29 2018 +0900
summary: fileset: remove fullmatchctx class

https://www.mercurial-scm.org/repo/hg/rev/9b6a708f2263
changeset:   38693:9b6a708f2263
user:Boris Feld 
date:Wed Jul 11 10:46:15 2018 +0200
summary: perf: document the perfmanifest command

https://www.mercurial-scm.org/repo/hg/rev/55101513ed94
changeset:   38694:55101513ed94
user:Boris Feld 
date:Fri Jun 22 11:02:42 2018 +0100
summary: perf: add a 'perf.all-timing' option to display more than best time

https://www.mercurial-scm.org/repo/hg/rev/2cdb82e8fb44
changeset:   38695:2cdb82e8fb44
user:Boris Feld 
date:Sat Jul 14 02:09:47 2018 +0200
summary: perffncachewrite: load fncache after lock is acquired

https://www.mercurial-scm.org/repo/hg/rev/89d93dd1a222
changeset:   38696:89d93dd1a222
user:Boris Feld 
date:Sat Jul 14 02:10:43 2018 +0200
summary: store: assert the fncache have been loaded if dirty

https://www.mercurial-scm.org/repo/hg/rev/3c569172848d
changeset:   38697:3c569172848d
user:Boris Feld 
date:Fri Jun 01 12:10:34 2018 +0200
summary: statprof: small if cleanup

https://www.mercurial-scm.org/repo/hg/rev/6437e89b64d5
changeset:   38698:6437e89b64d5
user:Yuya Nishihara 
date:Sun Jul 08 19:41:00 2018 +0900
summary: py3: open file in binary mode in test-bundle2-remote-changegroup.t

https://www.mercurial-scm.org/repo/hg/rev/72c086b1af12
changeset:   38699:72c086b1af12
user:Yuya Nishihara 
date:Sun Jul

Re: [PATCH 1 of 5] sparse-revlog: new requirement enabled with format.sparse-revlog

2018-07-16 Thread Gregory Szorc

On Mon, Jul 16, 2018 at 11:50 AM, Boris Feld  wrote:

> # HG changeset patch
> # User Paul Morelle 
> # Date 1528143798 -7200
> #  Mon Jun 04 22:23:18 2018 +0200
> # Node ID 6ed641517622f53dd51193c3ae3930c66fc18078
> # Parent  cc76692f401d0063cd5dfaf46f0fca1563a85fae
> # EXP-Topic write-for-sparse-read
> # Available At https://bitbucket.org/octobus/mercurial-devel/
> #  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r
> 6ed641517622
> sparse-revlog: new requirement enabled with format.sparse-revlog
>

Queued this series, thanks.

I did flip the order of parts 1 and 2 when landing so all the sparse
commits are grouped.

I'm reasonably convinced this code won't have a negative impact on repos
that haven't opted into the feature. So it feels safe to land just before
feature freeze.


>
> The meaning of the new 'sparse-revlog' requirement is that the revlogs are
> allowed to contain wider delta chains with larger holes between the
> interesting
> chunks. These sparse delta chains should be read in several chunks to
> avoid a
> potential explosion of memory usage.
>
> Former version won't know how to read a delta chain in several chunks. They
> would keep reading them in a single read, and therefore would be subject
> to the
> potential memory explosion. Hence this new requirement: only versions
> having
> support of sparse-revlog reading should be allowed to read such a revlog.
>
> Implementation of this new algorithm and tools to enable or disable the
> requirement will follow in the next changesets.
>
> diff --git a/mercurial/configitems.py b/mercurial/configitems.py
> --- a/mercurial/configitems.py
> +++ b/mercurial/configitems.py
> @@ -652,6 +652,9 @@ coreconfigitem('format', 'maxchainlen',
>  coreconfigitem('format', 'obsstore-version',
>  default=None,
>  )
> +coreconfigitem('format', 'sparse-revlog',
> +default=False,
> +)
>  coreconfigitem('format', 'usefncache',
>  default=True,
>  )
> diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
> --- a/mercurial/localrepo.py
> +++ b/mercurial/localrepo.py
> @@ -354,6 +354,15 @@ class locallegacypeer(localpeer):
>  # clients.
>  REVLOGV2_REQUIREMENT = 'exp-revlogv2.0'
>
> +# A repository with the sparserevlog feature will have delta chains that
> +# can spread over a larger span. Sparse reading cuts these large spans
> into
> +# pieces, so that each piece isn't too big.
> +# Without the sparserevlog capability, reading from the repository could
> use
> +# huge amounts of memory, because the whole span would be read at once,
> +# including all the intermediate revisions that aren't pertinent for the
> chain.
> +# This is why once a repository has enabled sparse-read, it becomes
> required.
> +SPARSEREVLOG_REQUIREMENT = 'sparserevlog'
> +
>  # Functions receiving (ui, features) that extensions can register to
> impact
>  # the ability to load repositories with custom requirements. Only
>  # functions defined in loaded extensions are called.
> @@ -376,6 +385,7 @@ class localrepository(object):
>  'generaldelta',
>  'treemanifest',
>  REVLOGV2_REQUIREMENT,
> +SPARSEREVLOG_REQUIREMENT,
>  }
>  _basesupported = supportedformats | {
>  'store',
> @@ -678,6 +688,8 @@ class localrepository(object):
>  self.svfs.options['with-sparse-read'] = withsparseread
>  self.svfs.options['sparse-read-density-threshold'] =
> srdensitythres
>  self.svfs.options['sparse-read-min-gap-size'] = srmingapsize
> +sparserevlog = SPARSEREVLOG_REQUIREMENT in self.requirements
> +self.svfs.options['sparse-revlog'] = sparserevlog
>
>  for r in self.requirements:
>  if r.startswith('exp-compression-'):
> @@ -2370,6 +2382,9 @@ def newreporequirements(repo):
>  requirements.add('generaldelta')
>  if ui.configbool('experimental', 'treemanifest'):
>  requirements.add('treemanifest')
> +# experimental config: format.sparse-revlog
> +if ui.configbool('format', 'sparse-revlog'):
> +requirements.add(SPARSEREVLOG_REQUIREMENT)
>
>  revlogv2 = ui.config('experimental', 'revlogv2')
>  if revlogv2 == 'enable-unstable-format-and-corrupt-my-data':
> diff --git a/mercurial/revlog.py b/mercurial/revlog.py
> --- a/mercurial/revlog.py
> +++ b/mercurial/revlog.py
> @@ -895,6 +895,7 @@ class revlog(object):
>  self._compengine = 'zlib'
>  self._maxdeltachainspan = -1
>  self._withsparseread = False
> +self._sparserevlog = False
>  self._srdensitythreshold = 0.50
>  self._srmingapsize = 262144
>
> @@ -923,7 +924,10 @@ class revlog(object):
>  self._maxdeltachainspan = opts['maxdeltachainspan']
>  if mmaplargeindex and 'mmapindexthreshold' in opts:
>  mmapindexthreshold = opts['mmapindexthreshold']
> -self._withsparseread = bool(opts.get('with-sparse-read',
> False))
> +self._sparserevlog =

D3957: patchbomb: python 3 really wants those email addresses in unicode

2018-07-16 Thread durin42 (Augie Fackler)

durin42 created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  At this point, test-patchbomb-tls.t almost passes on Python 3, but I'm
  not really sure what the issue is: we just get a `abort: Connection
  unexpectedly closed` that doesn't make sense to me.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D3957

AFFECTED FILES
  hgext/patchbomb.py

CHANGE DETAILS

diff --git a/hgext/patchbomb.py b/hgext/patchbomb.py
--- a/hgext/patchbomb.py
+++ b/hgext/patchbomb.py
@@ -821,6 +821,8 @@
 generator = emailgen.Generator(_filestrtobytesfacade(fp),
mangle_from_=False)
 generator.flatten(m, 0)
-sendmail(sender_addr, to + bcc + cc, fp.getvalue())
+alldests = to + bcc + cc
+alldests = [pycompat.strurl(d) for d in alldests]
+sendmail(sender_addr, alldests, fp.getvalue())
 
 progress.complete()



To: durin42, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D3956: mail: stop using the smtplib.SSLFakeFile and use socket.socket.makefile

2018-07-16 Thread durin42 (Augie Fackler)

durin42 created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  They're equivalent, and the latter is what Python 3.3 says to use in
  the release notes. Turns out it works on Python 2 as well.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D3956

AFFECTED FILES
  mercurial/mail.py

CHANGE DETAILS

diff --git a/mercurial/mail.py b/mercurial/mail.py
--- a/mercurial/mail.py
+++ b/mercurial/mail.py
@@ -79,7 +79,7 @@
 self.keyfile, self.certfile,
 ui=self._ui,
 serverhostname=self._host)
-self.file = smtplib.SSLFakeFile(new_socket)
+self.file = new_socket.makefile()
 return new_socket
 
 def _pyhastls():



To: durin42, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D3955: mail: modernize check for Python-with-TLS

2018-07-16 Thread durin42 (Augie Fackler)

durin42 created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  We used to be going indirectly through the socket module, but now we
  just check for the ssl module.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D3955

AFFECTED FILES
  mercurial/mail.py

CHANGE DETAILS

diff --git a/mercurial/mail.py b/mercurial/mail.py
--- a/mercurial/mail.py
+++ b/mercurial/mail.py
@@ -82,14 +82,23 @@
 self.file = smtplib.SSLFakeFile(new_socket)
 return new_socket
 
+def _pyhastls():
+"""Returns true iff Python has TLS support, false otherwise."""
+try:
+import ssl
+getattr(ssl, 'HAS_TLS', False)
+return True
+except ImportError:
+return False
+
 def _smtp(ui):
 '''build an smtp connection and return a function to send mail'''
 local_hostname = ui.config('smtp', 'local_hostname')
 tls = ui.config('smtp', 'tls')
 # backward compatible: when tls = true, we use starttls.
 starttls = tls == 'starttls' or stringutil.parsebool(tls)
 smtps = tls == 'smtps'
-if (starttls or smtps) and not util.safehasattr(socket, 'ssl'):
+if (starttls or smtps) and not _pyhastls():
 raise error.Abort(_("can't use TLS: Python SSL support not installed"))
 mailhost = ui.config('smtp', 'host')
 if not mailhost:



To: durin42, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D3954: mail: cope with Py3 unicode antics on email addresses

2018-07-16 Thread durin42 (Augie Fackler)

durin42 created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D3954

AFFECTED FILES
  mercurial/mail.py

CHANGE DETAILS

diff --git a/mercurial/mail.py b/mercurial/mail.py
--- a/mercurial/mail.py
+++ b/mercurial/mail.py
@@ -299,7 +299,7 @@
 def _addressencode(ui, name, addr, charsets=None):
 name = headencode(ui, name, charsets)
 try:
-acc, dom = addr.split('@')
+acc, dom = addr.split(r'@')
 acc = acc.encode('ascii')
 dom = dom.decode(encoding.encoding).encode('idna')
 addr = '%s@%s' % (acc, dom)
@@ -311,27 +311,31 @@
 addr = addr.encode('ascii')
 except UnicodeDecodeError:
 raise error.Abort(_('invalid local address: %s') % addr)
-return email.utils.formataddr((name, addr))
+return pycompat.bytesurl(
+email.utils.formataddr((name, addr.decode('ascii'
 
 def addressencode(ui, address, charsets=None, display=False):
 '''Turns address into RFC-2047 compliant header.'''
 if display or not address:
 return address or ''
-name, addr = email.utils.parseaddr(address)
+name, addr = email.utils.parseaddr(pycompat.strurl(address))
 return _addressencode(ui, name, addr, charsets)
 
 def addrlistencode(ui, addrs, charsets=None, display=False):
 '''Turns a list of addresses into a list of RFC-2047 compliant headers.
 A single element of input list may contain multiple addresses, but output
 always has one address per item'''
+for a in addrs:
+assert isinstance(a, bytes), (r'%r unexpectedly not a bytestr' % a)
 if display:
 return [a.strip() for a in addrs if a.strip()]
 
 result = []
-for name, addr in email.utils.getaddresses(addrs):
+for name, addr in email.utils.getaddresses(
+[pycompat.strurl(a) for a in addrs]):
 if name or addr:
 result.append(_addressencode(ui, name, addr, charsets))
-return result
+return [pycompat.bytesurl(r) for r in result]
 
 def mimeencode(ui, s, charsets=None, display=False):
 '''creates mime text object, encodes it if needed, and sets



To: durin42, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D3953: mail: fix _encode to be more correct on Python 3

2018-07-16 Thread durin42 (Augie Fackler)

durin42 created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  This code appears to be on the wrong side of the law in Python 2, at
  least some of the time. In Python 3, it's definitely wrong in places,
  but fortunately that's easy to fix.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D3953

AFFECTED FILES
  mercurial/mail.py

CHANGE DETAILS

diff --git a/mercurial/mail.py b/mercurial/mail.py
--- a/mercurial/mail.py
+++ b/mercurial/mail.py
@@ -252,18 +252,35 @@
 order. Tries both encoding and fallbackencoding for input. Only as
 last resort send as is in fake ascii.
 Caveat: Do not use for mail parts containing patches!'''
+sendcharsets = charsets or _charsets(ui)
+if not isinstance(s, bytes):
+# We have unicode data, which we need to try and encode to
+# some reasonable-ish encoding. Try the encodings the user
+# wants, and fall back to garbage-in-ascii.
+for ocs in sendcharsets:
+try:
+return s.encode(pycompat.sysstr(ocs)), ocs
+except UnicodeEncodeError:
+pass
+except LookupError:
+ui.warn(_('ignoring invalid sendcharset: %s\n') % ocs)
+else:
+# Everything failed, ascii-armor what we've got and send it.
+return s.encode('ascii', 'backslashreplace')
+# We have a bytes of unknown encoding. We'll try and guess a valid
+# encoding, falling back to pretending we had ascii even though we
+# know that's wrong.
 try:
 s.decode('ascii')
 except UnicodeDecodeError:
-sendcharsets = charsets or _charsets(ui)
 for ics in (encoding.encoding, encoding.fallbackencoding):
 try:
 u = s.decode(ics)
 except UnicodeDecodeError:
 continue
 for ocs in sendcharsets:
 try:
-return u.encode(ocs), ocs
+return u.encode(pycompat.sysstr(ocs)), ocs
 except UnicodeEncodeError:
 pass
 except LookupError:



To: durin42, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D3951: patchbomb: work around email module really wanting to write unicode data

2018-07-16 Thread durin42 (Augie Fackler)

durin42 created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D3951

AFFECTED FILES
  hgext/patchbomb.py

CHANGE DETAILS

diff --git a/hgext/patchbomb.py b/hgext/patchbomb.py
--- a/hgext/patchbomb.py
+++ b/hgext/patchbomb.py
@@ -139,6 +139,14 @@
 default=None,
 )
 
+class _filestrtobytesfacade(object):
+def __init__(self, ui):
+self._ui = ui
+def write(self, data):
+if not isinstance(data, bytes):
+data = pycompat.bytesurl(data)
+return self._ui.write(data)
+
 # Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' 
for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
@@ -783,7 +791,8 @@
 if opts.get('test'):
 ui.status(_('displaying '), subj, ' ...\n')
 ui.pager('email')
-generator = emailgen.Generator(ui, mangle_from_=False)
+generator = emailgen.Generator(_filestrtobytesfacade(ui),
+   mangle_from_=False)
 try:
 generator.flatten(m, 0)
 ui.write('\n')
@@ -799,7 +808,8 @@
 # Exim does not remove the Bcc field
 del m['Bcc']
 fp = stringio()
-generator = emailgen.Generator(fp, mangle_from_=False)
+generator = emailgen.Generator(_filestrtobytesfacade(fp),
+   mangle_from_=False)
 generator.flatten(m, 0)
 sendmail(sender_addr, to + bcc + cc, fp.getvalue())
 



To: durin42, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D3952: patchbomb: ensure all headers and values given to email mod are native strings

2018-07-16 Thread durin42 (Augie Fackler)

durin42 created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  This lets test-patch-bookmark.t only fail with some harmless header
  output changes on Python 3, so I think patchbomb might be basically
  useful on Python 3 now.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D3952

AFFECTED FILES
  hgext/patchbomb.py

CHANGE DETAILS

diff --git a/hgext/patchbomb.py b/hgext/patchbomb.py
--- a/hgext/patchbomb.py
+++ b/hgext/patchbomb.py
@@ -788,6 +788,16 @@
 m['Bcc'] = ', '.join(bcc)
 if replyto:
 m['Reply-To'] = ', '.join(replyto)
+# Fix up all headers to be native strings.
+# TODO(durin42): this should probably be cleaned up above in the 
future.
+if pycompat.ispy3:
+for hdr, val in list(m.items()):
+if isinstance(hdr, bytes):
+del m[hdr]
+hdr = pycompat.strurl(hdr)
+if isinstance(val, bytes):
+val = pycompat.strurl(val)
+m[hdr] = val
 if opts.get('test'):
 ui.status(_('displaying '), subj, ' ...\n')
 ui.pager('email')



To: durin42, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1945: tests: refactor common bundle2 capabilities

2018-07-16 Thread joerg.sonnenberger (Joerg Sonnenberger)

joerg.sonnenberger updated this revision to Diff 9606.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1945?vs=5012=9606

REVISION DETAIL
  https://phab.mercurial-scm.org/D1945

AFFECTED FILES
  tests/common-pattern.py
  tests/test-http-bad-server.t

CHANGE DETAILS

diff --git a/tests/test-http-bad-server.t b/tests/test-http-bad-server.t
--- a/tests/test-http-bad-server.t
+++ b/tests/test-http-bad-server.t
@@ -120,7 +120,7 @@
   write(41) -> Content-Type: application/mercurial-0.1\r\n
   write(21) -> Content-Length: 436\r\n
   write(2) -> \r\n
-  write(436) -> batch branchmap 
bundle2=HG20%0Abookmarks%0Achangegroup%3D01%2C02%0Adigests%3Dmd5%2Csha1%2Csha512%0Aerror%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgtagsfnodes%0Alistkeys%0Apushkey%0Aremote-changegroup%3Dhttp%2Chttps%0Arev-branch-cache
 changegroupsubset compression=none getbundle httpheader=1024 
httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey 
streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
+  write(436) -> batch branchmap $USUAL_BUNDLE2_CAPS_NO_PHASES$ 
changegroupsubset compression=none getbundle httpheader=1024 
httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey 
streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
   readline(4? from 65537) -> (26) GET /?cmd=batch HTTP/1.1\r\n (glob)
   readline(1? from -1) -> (1?) Accept-Encoding* (glob)
   read limit reached; closing socket
@@ -161,7 +161,7 @@
   write(41) -> Content-Type: application/mercurial-0.1\r\n
   write(21) -> Content-Length: 436\r\n
   write(2) -> \r\n
-  write(436) -> batch branchmap 
bundle2=HG20%0Abookmarks%0Achangegroup%3D01%2C02%0Adigests%3Dmd5%2Csha1%2Csha512%0Aerror%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgtagsfnodes%0Alistkeys%0Apushkey%0Aremote-changegroup%3Dhttp%2Chttps%0Arev-branch-cache
 changegroupsubset compression=none getbundle httpheader=1024 
httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey 
streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
+  write(436) -> batch branchmap $USUAL_BUNDLE2_CAPS_NO_PHASES$ 
changegroupsubset compression=none getbundle httpheader=1024 
httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey 
streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
   readline(13? from 65537) -> (26) GET /?cmd=batch HTTP/1.1\r\n (glob)
   readline(1?? from -1) -> (27) Accept-Encoding: identity\r\n (glob)
   readline(8? from -1) -> (29) vary: X-HgArg-1,X-HgProto-1\r\n (glob)
@@ -218,7 +218,7 @@
   write(41) -> Content-Type: application/mercurial-0.1\r\n
   write(21) -> Content-Length: 449\r\n
   write(2) -> \r\n
-  write(449) -> batch branchmap 
bundle2=HG20%0Abookmarks%0Achangegroup%3D01%2C02%0Adigests%3Dmd5%2Csha1%2Csha512%0Aerror%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgtagsfnodes%0Alistkeys%0Apushkey%0Aremote-changegroup%3Dhttp%2Chttps%0Arev-branch-cache
 changegroupsubset compression=none getbundle httpheader=1024 
httpmediatype=0.1rx,0.1tx,0.2tx httppostargs known lookup pushkey 
streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
+  write(449) -> batch branchmap $USUAL_BUNDLE2_CAPS_NO_PHASES$ 
changegroupsubset compression=none getbundle httpheader=1024 
httpmediatype=0.1rx,0.1tx,0.2tx httppostargs known lookup pushkey 
streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
   readline(1?? from 65537) -> (27) POST /?cmd=batch HTTP/1.1\r\n (glob)
   readline(1?? from -1) -> (27) Accept-Encoding: identity\r\n (glob)
   readline(1?? from -1) -> (41) content-type: application/mercurial-0.1\r\n 
(glob)
@@ -329,7 +329,7 @@
   write(41 from 41) -> (577) Content-Type: application/mercurial-0.1\r\n
   write(21 from 21) -> (556) Content-Length: 436\r\n
   write(2 from 2) -> (554) \r\n
-  write(436 from 436) -> (118) batch branchmap 
bundle2=HG20%0Abookmarks%0Achangegroup%3D01%2C02%0Adigests%3Dmd5%2Csha1%2Csha512%0Aerror%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgtagsfnodes%0Alistkeys%0Apushkey%0Aremote-changegroup%3Dhttp%2Chttps%0Arev-branch-cache
 changegroupsubset compression=none getbundle httpheader=1024 
httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey 
streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
+  write(436 from 436) -> (118) batch branchmap $USUAL_BUNDLE2_CAPS_NO_PHASES$ 
changegroupsubset compression=none getbundle httpheader=1024 
httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey 
streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
   readline(65537) -> (26) GET /?cmd=batch HTTP/1.1\r\n
   readline(-1) -> (27) Accept-Encoding: identity\r\n
   readline(-1) -> (29) vary: X-HgArg-1,X-HgProto-1\r\n
@@ -377,7 +377,7 @@
   write(41 from 41) -> (642) Content-Type: application/mercurial-0.1\r\n
   write(21 from 21) -> (621) Content-Length: 436\r\n
   write(2 from 2) -> (619) \r\n
-  write(436 from 436) -> (183) batch branchmap

D3949: shelve: improve help text for --patch and --stat

2018-07-16 Thread hooper (Danny Hooper)

This revision was automatically updated to reflect the committed changes.
Closed by commit rHGabcf500d527c: shelve: improve help text for --patch and 
--stat (authored by hooper, committed by ).

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D3949?vs=9602=9604

REVISION DETAIL
  https://phab.mercurial-scm.org/D3949

AFFECTED FILES
  hgext/shelve.py
  tests/test-shelve.t

CHANGE DETAILS

diff --git a/tests/test-shelve.t b/tests/test-shelve.t
--- a/tests/test-shelve.t
+++ b/tests/test-shelve.t
@@ -68,9 +68,12 @@
-l --listlist current shelves
-m --message TEXTuse text as shelve message
-n --name NAME   use the given name for the shelved commit
-   -p --patch   show patch
+   -p --patch   output patches for changes (provide the names of 
the
+shelved changes as positional arguments)
-i --interactive interactive mode, only works while creating a 
shelve
-  --statoutput diffstat-style summary of changes
+  --statoutput diffstat-style summary of changes (provide
+the names of the shelved changes as positional
+arguments)
-I --include PATTERN [+] include names matching the given patterns
-X --exclude PATTERN [+] exclude names matching the given patterns
   --mq  operate on patch repository
diff --git a/hgext/shelve.py b/hgext/shelve.py
--- a/hgext/shelve.py
+++ b/hgext/shelve.py
@@ -1011,11 +1011,14 @@
   ('n', 'name', '',
_('use the given name for the shelved commit'), _('NAME')),
   ('p', 'patch', None,
-   _('show patch')),
+   _('output patches for changes (provide the names of the shelved '
+ 'changes as positional arguments)')),
   ('i', 'interactive', None,
_('interactive mode, only works while creating a shelve')),
   ('', 'stat', None,
-   _('output diffstat-style summary of changes'))] + cmdutil.walkopts,
+   _('output diffstat-style summary of changes (provide the names of '
+ 'the shelved changes as positional arguments)')
+   )] + cmdutil.walkopts,
  _('hg shelve [OPTION]... [FILE]...'))
 def shelvecmd(ui, repo, *pats, **opts):
 '''save and set aside changes from the working directory



To: hooper, #hg-reviewers, pulkit
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D3950: shelve: pick the most recent shelve if none specified for --patch/--stat

2018-07-16 Thread hooper (Danny Hooper)

This revision was automatically updated to reflect the committed changes.
Closed by commit rHG905b66681004: shelve: pick the most recent shelve if none 
specified for --patch/--stat (authored by hooper, committed by ).

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D3950?vs=9603=9605

REVISION DETAIL
  https://phab.mercurial-scm.org/D3950

AFFECTED FILES
  hgext/shelve.py
  tests/test-shelve.t

CHANGE DETAILS

diff --git a/tests/test-shelve.t b/tests/test-shelve.t
--- a/tests/test-shelve.t
+++ b/tests/test-shelve.t
@@ -1057,8 +1057,33 @@
   $ hg shelve --patch default nonexistentshelf
   abort: cannot find shelf nonexistentshelf
   [255]
+
+when the user asks for a patch, we assume they want the most recent shelve if
+they don't provide a shelve name
+
   $ hg shelve --patch
-  abort: --patch expects at least one shelf
+  default-01  (*)* changes to: create conflict (glob)
+  
+  diff --git a/shelf-patch-b b/shelf-patch-b
+  new file mode 100644
+  --- /dev/null
+  +++ b/shelf-patch-b
+  @@ -0,0 +1,1 @@
+  +patch b
+
+  $ cd ..
+
+you shouldn't be able to ask for the patch/stats of the most recent shelve if
+there are no shelves
+
+  $ hg init noshelves
+  $ cd noshelves
+
+  $ hg shelve --patch
+  abort: there are no shelves to show
+  [255]
+  $ hg shelve --stat
+  abort: there are no shelves to show
   [255]
 
   $ cd ..
diff --git a/hgext/shelve.py b/hgext/shelve.py
--- a/hgext/shelve.py
+++ b/hgext/shelve.py
@@ -594,10 +594,15 @@
 for chunk, label in patch.diffstatui(difflines, width=width):
 ui.write(chunk, label=label)
 
-def patchcmds(ui, repo, pats, opts, subcommand):
+def patchcmds(ui, repo, pats, opts):
 """subcommand that displays shelves"""
 if len(pats) == 0:
-raise error.Abort(_("--%s expects at least one shelf") % subcommand)
+shelves = listshelves(repo)
+if not shelves:
+raise error.Abort(_("there are no shelves to show"))
+mtime, name = shelves[0]
+sname = util.split(name)[1]
+pats = [sname]
 
 for shelfname in pats:
 if not shelvedfile(repo, shelfname, patchextension).exists():
@@ -1082,10 +1087,8 @@
 return deletecmd(ui, repo, pats)
 elif checkopt('list'):
 return listcmd(ui, repo, pats, opts)
-elif checkopt('patch'):
-return patchcmds(ui, repo, pats, opts, subcommand='patch')
-elif checkopt('stat'):
-return patchcmds(ui, repo, pats, opts, subcommand='stat')
+elif checkopt('patch') or checkopt('stat'):
+return patchcmds(ui, repo, pats, opts)
 else:
 return createcmd(ui, repo, pats, opts)
 



To: hooper, #hg-reviewers, pulkit
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 1 of 2] revset-benchmark: use a generic revset to test `heads(commonancestors())`

2018-07-16 Thread Pulkit Goyal

On Mon, Jul 16, 2018 at 10:07 PM Boris Feld  wrote:

> # HG changeset patch
> # User Boris Feld 
> # Date 1531752215 -7200
> #  Mon Jul 16 16:43:35 2018 +0200
> # Node ID d401e0d99b2c8e18bda7e007c2cd7599afbb365c
> # Parent  3c569172848d329a81b1420d7cd8a5549b48f25f
> # EXP-Topic revset-bench
> # Available At https://bitbucket.org/octobus/mercurial-devel/
> #  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r
> d401e0d99b2c
> revset-benchmark: use a generic revset to test `heads(commonancestors())`
>
> This allow to benchmark revset performance in other repositories than just
> the mercurial one.
>

This one is already queued. Many thanks!
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 1 of 3] obsolete: clarify users in markerusers() never contain None

2018-07-16 Thread Pulkit Goyal

On Mon, Jul 16, 2018 at 4:19 PM Yuya Nishihara  wrote:

> # HG changeset patch
> # User Yuya Nishihara 
> # Date 1531646560 -32400
> #  Sun Jul 15 18:22:40 2018 +0900
> # Node ID 4625a49972b58bb56bed7e618c71ef1742940a5c
> # Parent  f749aaad0d2f52e3da2a8e80f44b798c0dd85f00
> obsolete: clarify users in markerusers() never contain None
>

I don't know who did, but this series is already queued. Many thanks!
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 1 of 4] hook: disable the shell to native command translation by default

2018-07-16 Thread Matt Harbison

# HG changeset patch
# User Matt Harbison 
# Date 1531712769 14400
#  Sun Jul 15 23:46:09 2018 -0400
# Node ID 211dc3185a392d27964fc729c41b7265a98043f6
# Parent  3c569172848d329a81b1420d7cd8a5549b48f25f
hook: disable the shell to native command translation by default

There are other things I want to add like ~ expansion and translating single to
double quotes for cmd.exe.  So off by default is safer.

I'm having second thoughts about the name, but I don't have any better ideas.

diff --git a/mercurial/help/config.txt b/mercurial/help/config.txt
--- a/mercurial/help/config.txt
+++ b/mercurial/help/config.txt
@@ -891,16 +891,17 @@ be ``$HG_HOOKTYPE=incoming`` and ``$HG_H
 
 .. container:: windows
 
-  Some basic Unix syntax is supported for portability, including ``$VAR``
+  Some basic Unix syntax can be enabled for portability, including ``$VAR``
   and ``${VAR}`` style variables.  To use a literal ``$``, it must be
-  escaped with a back slash or inside of a strong quote.  This can be
-  disabled by adding a prefix of ``tonative.`` to the hook name on a new
-  line, and setting it to ``False``.  For example::
+  escaped with a back slash or inside of a strong quote.
+
+  This feature is enabled by adding a prefix of ``tonative.`` to the hook
+  name on a new line, and setting it to ``True``.  For example::
 
 [hooks]
 incoming.autobuild = /my/build/hook
-# disable translation to cmd.exe syntax for autobuild hook
-tonative.incoming.autobuild = False
+# enable translation to cmd.exe syntax for autobuild hook
+tonative.incoming.autobuild = True
 
 ``changegroup``
   Run after a changegroup has been added via push, pull or unbundle.  The ID of
diff --git a/mercurial/hook.py b/mercurial/hook.py
--- a/mercurial/hook.py
+++ b/mercurial/hook.py
@@ -139,7 +139,7 @@ def _exthook(ui, repo, htype, name, cmd,
 v = stringutil.pprint(v)
 env['HG_' + k.upper()] = v
 
-if ui.configbool('hooks', 'tonative.%s' % name, pycompat.iswindows):
+if ui.configbool('hooks', 'tonative.%s' % name, False):
 ui.note(_('converting hook "%s" to native\n') % name)
 cmd = procutil.shelltonative(cmd, env)
 
diff --git a/tests/test-bookmarks-pushpull.t b/tests/test-bookmarks-pushpull.t
--- a/tests/test-bookmarks-pushpull.t
+++ b/tests/test-bookmarks-pushpull.t
@@ -194,7 +194,6 @@ delete a remote bookmark
   bundle2-input: part header size: 0
   bundle2-input: end of bundle2 stream
   bundle2-input-bundle: 3 parts total
-  converting hook "txnclose-bookmark.test" to native (windows !)
   running hook txnclose-bookmark.test: sh $TESTTMP/hook.sh
   test-hook-bookmark: W:   -> 
   bundle2-output-bundle: "HG20", 1 parts total
@@ -309,7 +308,6 @@ delete a remote bookmark
   bundle2-input: part header size: 0
   bundle2-input: end of bundle2 stream
   bundle2-input-bundle: 3 parts total
-  converting hook "txnclose-bookmark.test" to native (windows !)
   running hook txnclose-bookmark.test: sh $TESTTMP/hook.sh
   test-hook-bookmark: W:   -> 
   bundle2-output-bundle: "HG20", 0 parts total
diff --git a/tests/test-commit-amend.t b/tests/test-commit-amend.t
--- a/tests/test-commit-amend.t
+++ b/tests/test-commit-amend.t
@@ -205,7 +205,6 @@ at first, test saving last-message.txt
   a
   committing manifest
   committing changelog
-  converting hook "pretxncommit.test-saving-last-message" to native (windows !)
   running hook pretxncommit.test-saving-last-message: false
   transaction abort!
   rollback completed
@@ -231,7 +230,6 @@ at first, test saving last-message.txt
   a
   committing manifest
   committing changelog
-  converting hook "pretxncommit.test-saving-last-message" to native (windows !)
   running hook pretxncommit.test-saving-last-message: false
   transaction abort!
   rollback completed
diff --git a/tests/test-histedit-fold.t b/tests/test-histedit-fold.t
--- a/tests/test-histedit-fold.t
+++ b/tests/test-histedit-fold.t
@@ -478,7 +478,8 @@ This is an excuse to test hook with hist
   1:199b6bb90248 b
   0:6c795aa153cb a
 
-  $ hg histedit 6c795aa153cb --config hooks.commit='echo commit $HG_NODE' 
--commands - 2>&1 << EOF | fixbundle
+  $ hg histedit 6c795aa153cb --config hooks.commit='echo commit $HG_NODE' 
--config hooks.tonative.commit=True \
+  > --commands - 2>&1 << EOF | fixbundle
   > pick 199b6bb90248 b
   > fold a1a953ffb4b0 c
   > pick 6c795aa153cb a
@@ -494,8 +495,8 @@ Test unix -> windows style variable subs
   $ cat > $TESTTMP/tmp.hgrc <<'EOF'
   > [hooks]
   > pre-add = echo no variables
-  > tonative.pre-add = False
   > post-add = echo ran $HG_ARGS, literal \$non-var, 'also $non-var', 
$HG_RESULT
+  > tonative.post-add = True
   > EOF
 
 TODO: Windows should output double quotes around "also $non-var"
diff --git a/tests/test-keyword.t b/tests/test-keyword.t
--- a/tests/test-keyword.t
+++ b/tests/test-keyword.t
@@ -183,7 +183,6 @@ Commit with several checks

[PATCH 2 of 4] hook: only print the note about native cmd translation if it actually changes

2018-07-16 Thread Matt Harbison

# HG changeset patch
# User Matt Harbison 
# Date 1531713103 14400
#  Sun Jul 15 23:51:43 2018 -0400
# Node ID 48070b1bc067ac31924c598686fb0822c6ff9409
# Parent  211dc3185a392d27964fc729c41b7265a98043f6
hook: only print the note about native cmd translation if it actually changes

This makes it so that it will never occur on a non Windows platform.

diff --git a/mercurial/hook.py b/mercurial/hook.py
--- a/mercurial/hook.py
+++ b/mercurial/hook.py
@@ -140,8 +140,10 @@ def _exthook(ui, repo, htype, name, cmd,
 env['HG_' + k.upper()] = v
 
 if ui.configbool('hooks', 'tonative.%s' % name, False):
-ui.note(_('converting hook "%s" to native\n') % name)
+oldcmd = cmd
 cmd = procutil.shelltonative(cmd, env)
+if cmd != oldcmd:
+ui.note(_('converting hook "%s" to native\n') % name)
 
 ui.note(_("running hook %s: %s\n") % (name, cmd))
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 4 of 4] windows: expand '~/' and '~\' to %USERPROFILE% when translating to cmd.exe

2018-07-16 Thread Matt Harbison

# HG changeset patch
# User Matt Harbison 
# Date 1531715553 14400
#  Mon Jul 16 00:32:33 2018 -0400
# Node ID 54611420fcf7868ee195e9fa3070efc0d57e9757
# Parent  e71e478d5e49768357287a2f181e8a2b23213239
windows: expand '~/' and '~\' to %USERPROFILE% when translating to cmd.exe

It's convenient to be able to reference hooks in a portable location on any
platform.

diff --git a/mercurial/help/config.txt b/mercurial/help/config.txt
--- a/mercurial/help/config.txt
+++ b/mercurial/help/config.txt
@@ -892,9 +892,11 @@ be ``$HG_HOOKTYPE=incoming`` and ``$HG_H
 .. container:: windows
 
   Some basic Unix syntax can be enabled for portability, including ``$VAR``
-  and ``${VAR}`` style variables.  To use a literal ``$``, it must be
-  escaped with a back slash or inside of a strong quote.  Strong quotes
-  will be replaced by double quotes after processing.
+  and ``${VAR}`` style variables.  A ``~`` followed by ``\`` or ``/`` will
+  be expanded to ``%USERPROFILE%`` to simulate a subset of tilde expansion
+  on Unix.  To use a literal ``$`` or ``~``, it must be escaped with a back
+  slash or inside of a strong quote.  Strong quotes will be replaced by
+  double quotes after processing.
 
   This feature is enabled by adding a prefix of ``tonative.`` to the hook
   name on a new line, and setting it to ``True``.  For example::
diff --git a/mercurial/windows.py b/mercurial/windows.py
--- a/mercurial/windows.py
+++ b/mercurial/windows.py
@@ -276,8 +276,11 @@ def shelltocmdexe(path, env):
 >>> # No double substitution
 >>> shelltocmdexe(b"$var1 %var1%", {b'var1': b'%var2%', b'var2': b'boom'})
 '%var1% %var1%'
+>>> # Tilde expansion
+>>> shelltocmdexe(b"~/dir ~\dir2 ~tmpfile \~/", {})
+'%USERPROFILE%/dir %USERPROFILE%\\dir2 ~tmpfile ~/'
 """
-if not any(c in path for c in b"$'"):
+if not any(c in path for c in b"$'~"):
 return path
 
 varchars = pycompat.sysbytes(string.ascii_letters + string.digits) + b'_-'
@@ -344,9 +347,13 @@ def shelltocmdexe(path, env):
 
 if c != '':
 index -= 1
-elif c == b'\\' and index + 1 < pathlen and path[index + 1] == b'$':
-# Skip '\', but only if it is escaping $
-res += b'$'
+elif (c == b'~' and index + 1 < pathlen
+  and path[index + 1] in (b'\\', b'/')):
+res += "%USERPROFILE%"
+elif (c == b'\\' and index + 1 < pathlen
+  and path[index + 1] in (b'$', b'~')):
+# Skip '\', but only if it is escaping $ or ~
+res += path[index + 1]
 index += 1
 else:
 res += c
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 3 of 4] windows: replace single quote with double quote when translating to cmd.exe

2018-07-16 Thread Matt Harbison

# HG changeset patch
# User Matt Harbison 
# Date 1531713519 14400
#  Sun Jul 15 23:58:39 2018 -0400
# Node ID e71e478d5e49768357287a2f181e8a2b23213239
# Parent  48070b1bc067ac31924c598686fb0822c6ff9409
windows: replace single quote with double quote when translating to cmd.exe

Since cmd.exe doesn't understand single quotes, single quotes to prevent $var
expansion is basically unusable without this.  Single quote isn't allowed in a
path name, so it seems unlikely to come up otherwise.

diff --git a/mercurial/help/config.txt b/mercurial/help/config.txt
--- a/mercurial/help/config.txt
+++ b/mercurial/help/config.txt
@@ -893,7 +893,8 @@ be ``$HG_HOOKTYPE=incoming`` and ``$HG_H
 
   Some basic Unix syntax can be enabled for portability, including ``$VAR``
   and ``${VAR}`` style variables.  To use a literal ``$``, it must be
-  escaped with a back slash or inside of a strong quote.
+  escaped with a back slash or inside of a strong quote.  Strong quotes
+  will be replaced by double quotes after processing.
 
   This feature is enabled by adding a prefix of ``tonative.`` to the hook
   name on a new line, and setting it to ``True``.  For example::
diff --git a/mercurial/windows.py b/mercurial/windows.py
--- a/mercurial/windows.py
+++ b/mercurial/windows.py
@@ -268,7 +268,7 @@ def shelltocmdexe(path, env):
 'cmd %var1% %var2% %var3% $missing ${missing} %missing%'
 >>> # Single quote prevents expansion, as does \$ escaping
 >>> shelltocmdexe(b"cmd '$var1 ${var2} %var3%' \$var1 \${var2} \\", e)
-"cmd '$var1 ${var2} %var3%' $var1 ${var2} \\"
+'cmd "$var1 ${var2} %var3%" $var1 ${var2} \\'
 >>> # $$ is not special. %% is not special either, but can be the end and
 >>> # start of consecutive variables
 >>> shelltocmdexe(b"cmd $$ %% %var1%%var2%", e)
@@ -277,7 +277,7 @@ def shelltocmdexe(path, env):
 >>> shelltocmdexe(b"$var1 %var1%", {b'var1': b'%var2%', b'var2': b'boom'})
 '%var1% %var1%'
 """
-if b'$' not in path:
+if not any(c in path for c in b"$'"):
 return path
 
 varchars = pycompat.sysbytes(string.ascii_letters + string.digits) + b'_-'
@@ -292,7 +292,7 @@ def shelltocmdexe(path, env):
 pathlen = len(path)
 try:
 index = path.index(b'\'')
-res += b'\'' + path[:index + 1]
+res += b'"' + path[:index] + b'"'
 except ValueError:
 res += c + path
 index = pathlen - 1
diff --git a/tests/test-histedit-fold.t b/tests/test-histedit-fold.t
--- a/tests/test-histedit-fold.t
+++ b/tests/test-histedit-fold.t
@@ -499,16 +499,15 @@ Test unix -> windows style variable subs
   > tonative.post-add = True
   > EOF
 
-TODO: Windows should output double quotes around "also $non-var"
   $ echo "foo" > amended.txt
   $ HGRCPATH=$TESTTMP/tmp.hgrc hg add -v amended.txt
   running hook pre-add: echo no variables
   no variables
   adding amended.txt
   converting hook "post-add" to native (windows !)
-  running hook post-add: echo ran %HG_ARGS%, literal $non-var, 'also 
$non-var', %HG_RESULT% (windows !)
+  running hook post-add: echo ran %HG_ARGS%, literal $non-var, "also 
$non-var", %HG_RESULT% (windows !)
   running hook post-add: echo ran $HG_ARGS, literal \$non-var, 'also 
$non-var', $HG_RESULT (no-windows !)
-  ran add -v amended.txt, literal $non-var, 'also $non-var', 0 (windows !)
+  ran add -v amended.txt, literal $non-var, "also $non-var", 0 (windows !)
   ran add -v amended.txt, literal $non-var, also $non-var, 0 (no-windows !)
   $ hg ci -q --config extensions.largefiles= --amend -I amended.txt
   The fsmonitor extension is incompatible with the largefiles extension and 
has been disabled. (fsmonitor !)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D3949: shelve: improve help text for --patch and --stat

2018-07-16 Thread hooper (Danny Hooper)

hooper created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  It's not currently obvious why "hg shelve -p" fails, since -p doesn't take an 
argument.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D3949

AFFECTED FILES
  hgext/shelve.py
  tests/test-shelve.t

CHANGE DETAILS

diff --git a/tests/test-shelve.t b/tests/test-shelve.t
--- a/tests/test-shelve.t
+++ b/tests/test-shelve.t
@@ -68,9 +68,12 @@
-l --listlist current shelves
-m --message TEXTuse text as shelve message
-n --name NAME   use the given name for the shelved commit
-   -p --patch   show patch
+   -p --patch   output patches for changes (provide the names of 
the
+shelved changes as positional arguments)
-i --interactive interactive mode, only works while creating a 
shelve
-  --statoutput diffstat-style summary of changes
+  --statoutput diffstat-style summary of changes (provide
+the names of the shelved changes as positional
+arguments)
-I --include PATTERN [+] include names matching the given patterns
-X --exclude PATTERN [+] exclude names matching the given patterns
   --mq  operate on patch repository
diff --git a/hgext/shelve.py b/hgext/shelve.py
--- a/hgext/shelve.py
+++ b/hgext/shelve.py
@@ -1011,11 +1011,14 @@
   ('n', 'name', '',
_('use the given name for the shelved commit'), _('NAME')),
   ('p', 'patch', None,
-   _('show patch')),
+   _('output patches for changes (provide the names of the shelved '
+ 'changes as positional arguments)')),
   ('i', 'interactive', None,
_('interactive mode, only works while creating a shelve')),
   ('', 'stat', None,
-   _('output diffstat-style summary of changes'))] + cmdutil.walkopts,
+   _('output diffstat-style summary of changes (provide the names of '
+ 'the shelved changes as positional arguments)')
+   )] + cmdutil.walkopts,
  _('hg shelve [OPTION]... [FILE]...'))
 def shelvecmd(ui, repo, *pats, **opts):
 '''save and set aside changes from the working directory



To: hooper, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D3937: ssh: avoid reading beyond the end of stream when using compression

2018-07-16 Thread joerg.sonnenberger (Joerg Sonnenberger)

This revision was automatically updated to reflect the committed changes.
Closed by commit rHG27391d74aaa2: ssh: avoid reading beyond the end of stream 
when using compression (authored by joerg.sonnenberger, committed by ).

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D3937?vs=9576=9601

REVISION DETAIL
  https://phab.mercurial-scm.org/D3937

AFFECTED FILES
  mercurial/sshpeer.py
  mercurial/util.py

CHANGE DETAILS

diff --git a/mercurial/util.py b/mercurial/util.py
--- a/mercurial/util.py
+++ b/mercurial/util.py
@@ -322,6 +322,11 @@
 self._fillbuffer()
 return self._frombuffer(size)
 
+def unbufferedread(self, size):
+if not self._eof and self._lenbuf == 0:
+self._fillbuffer(max(size, _chunksize))
+return self._frombuffer(min(self._lenbuf, size))
+
 def readline(self, *args, **kwargs):
 if 1 < len(self._buffer):
 # this should not happen because both read and readline end with a
@@ -363,9 +368,9 @@
 self._lenbuf = 0
 return data
 
-def _fillbuffer(self):
+def _fillbuffer(self, size=_chunksize):
 """read data to the buffer"""
-data = os.read(self._input.fileno(), _chunksize)
+data = os.read(self._input.fileno(), size)
 if not data:
 self._eof = True
 else:
@@ -3302,6 +3307,104 @@
 """
 raise NotImplementedError()
 
+class _CompressedStreamReader(object):
+def __init__(self, fh):
+if safehasattr(fh, 'unbufferedread'):
+self._reader = fh.unbufferedread
+else:
+self._reader = fh.read
+self._pending = []
+self._pos = 0
+self._eof = False
+
+def _decompress(self, chunk):
+raise NotImplementedError()
+
+def read(self, l):
+buf = []
+while True:
+while self._pending:
+if len(self._pending[0]) > l + self._pos:
+newbuf = self._pending[0]
+buf.append(newbuf[self._pos:self._pos + l])
+self._pos += l
+return ''.join(buf)
+
+newbuf = self._pending.pop(0)
+if self._pos:
+buf.append(newbuf[self._pos:])
+l -= len(newbuf) - self._pos
+else:
+buf.append(newbuf)
+l -= len(newbuf)
+self._pos = 0
+
+if self._eof:
+return ''.join(buf)
+chunk = self._reader(65536)
+self._decompress(chunk)
+
+class _GzipCompressedStreamReader(_CompressedStreamReader):
+def __init__(self, fh):
+super(_GzipCompressedStreamReader, self).__init__(fh)
+self._decompobj = zlib.decompressobj()
+def _decompress(self, chunk):
+newbuf = self._decompobj.decompress(chunk)
+if newbuf:
+self._pending.append(newbuf)
+d = self._decompobj.copy()
+try:
+d.decompress('x')
+d.flush()
+if d.unused_data == 'x':
+self._eof = True
+except zlib.error:
+pass
+
+class _BZ2CompressedStreamReader(_CompressedStreamReader):
+def __init__(self, fh):
+super(_BZ2CompressedStreamReader, self).__init__(fh)
+self._decompobj = bz2.BZ2Decompressor()
+def _decompress(self, chunk):
+newbuf = self._decompobj.decompress(chunk)
+if newbuf:
+self._pending.append(newbuf)
+try:
+while True:
+newbuf = self._decompobj.decompress('')
+if newbuf:
+self._pending.append(newbuf)
+else:
+break
+except EOFError:
+self._eof = True
+
+class _TruncatedBZ2CompressedStreamReader(_BZ2CompressedStreamReader):
+def __init__(self, fh):
+super(_TruncatedBZ2CompressedStreamReader, self).__init__(fh)
+newbuf = self._decompobj.decompress('BZ')
+if newbuf:
+self._pending.append(newbuf)
+
+class _ZstdCompressedStreamReader(_CompressedStreamReader):
+def __init__(self, fh, zstd):
+super(_ZstdCompressedStreamReader, self).__init__(fh)
+self._zstd = zstd
+self._decompobj = zstd.ZstdDecompressor().decompressobj()
+def _decompress(self, chunk):
+newbuf = self._decompobj.decompress(chunk)
+if newbuf:
+self._pending.append(newbuf)
+try:
+while True:
+newbuf = self._decompobj.decompress('')
+if newbuf:
+self._pending.append(newbuf)
+else:
+break
+except self._zstd.ZstdError:
+self._eof = True
+
 class _zlibengine(compressionengine):
 def name(self):
 return 'zlib'
@@ -3335,15 +3438,7 @@
 yield z.flush()
 
 def decompressorreader(self, fh):
-def

[PATCH experimental] manifest: persist the manifestfulltext cache

2018-07-16 Thread Martijn Pieters

# HG changeset patch
# User Martijn Pieters 
# Date 1531769057 -3600
#  Mon Jul 16 20:24:17 2018 +0100
# Node ID 6b76d41520cded0655a6214fdd515014a23b6b9c
# Parent  2a227782e75426a3e4408273a9a4eebe79dfdcea
# EXP-Topic persistentmanifestcache
manifest: persist the manifestfulltext cache

Reconstructing the manifest from the revlog takes time, so much so that there
already is a LRU cache to avoid having to load a manifest multiple times.

This patch persists that LRU cache in the .hg/cache directory, so we can re-use
this cache across hg commands. Commit benchmark (run on Macos 10.13 on a
2017-model Macbook Pro with Core i7 2.9GHz and flash drive), testing without
and with patch run 5 times, baseline is r2a227782e754:

* committing to an existing file, against the mozilla-central repository.
  Baseline real time average 1.9692, with patch 1.3786.

A new debugcommand "hg debugmanifestfulltextcache" lets you inspect the cache,
clear it, or add specific manifest nodeids to it. When calling
repo.updatecaches(), the manifest(s) for the working copy parents are added to
the cache.

diff -r 2a227782e754 -r 6b76d41520cd mercurial/debugcommands.py
--- a/mercurial/debugcommands.pyWed Jul 11 16:44:33 2018 -0700
+++ b/mercurial/debugcommands.pyMon Jul 16 20:24:17 2018 +0100
@@ -1446,6 +1446,53 @@
 
 return held
 
+@command('debugmanifestfulltextcache', [
+('', 'clear', False, _('clear the cache')),
+('a', 'add', '', _('add the given manifest node to the cache'),
+ _('NODE'))
+], '')
+def debugmanifestfulltextcache(ui, repo, add=None, **opts):
+"""show, clear or amend the contents of the manifest fulltext cache"""
+with repo.lock():
+r = repo.manifestlog._revlog
+try:
+cache = r._fulltextcache
+except AttributeError:
+ui.warn(_(
+"Current revlog implementation doesn't appear to have a "
+'manifest fulltext cache\n'))
+return
+
+if opts.get(r'clear'):
+cache.clear()
+
+if add:
+try:
+manifest = repo.manifestlog[r.lookup(add)]
+except error.LookupError as e:
+raise error.Abort(e, hint="Check your manifest node id")
+manifest.read()  # stores revisision in cache too
+
+if not len(cache):
+ui.write(_('Cache empty'))
+else:
+ui.write(
+_('Cache contains %d manifest entries, in order of most to '
+  'least recent:\n') % (len(cache),))
+totalsize = 0
+for nodeid in cache:
+# Use cache.get to not update the LRU order
+data = cache.get(nodeid)
+size = len(data)
+totalsize += size + 24   # 20 bytes nodeid, 4 bytes size
+ui.write(_('id: %s, size %s\n') % (
+hex(nodeid), util.bytecount(size)))
+ondisk = cache._opener.stat('manifestfulltextcache').st_size
+ui.write(
+_('Total cache data size %s, on-disk %s\n') % (
+util.bytecount(totalsize), util.bytecount(ondisk))
+)
+
 @command('debugmergestate', [], '')
 def debugmergestate(ui, repo, *args):
 """print merge state
diff -r 2a227782e754 -r 6b76d41520cd mercurial/localrepo.py
--- a/mercurial/localrepo.pyWed Jul 11 16:44:33 2018 -0700
+++ b/mercurial/localrepo.pyMon Jul 16 20:24:17 2018 +0100
@@ -1599,6 +1599,10 @@
 rbc.branchinfo(r)
 rbc.write()
 
+# ensure the working copy parents are in the manifestfulltextcache
+for ctx in self['.'].parents():
+ctx.manifest()  # accessing the manifest is enough
+
 def invalidatecaches(self):
 
 if '_tagscache' in vars(self):
diff -r 2a227782e754 -r 6b76d41520cd mercurial/manifest.py
--- a/mercurial/manifest.py Wed Jul 11 16:44:33 2018 -0700
+++ b/mercurial/manifest.py Mon Jul 16 20:24:17 2018 +0100
@@ -10,6 +10,7 @@
 import heapq
 import itertools
 import struct
+import weakref
 
 from .i18n import _
 from .node import (
@@ -1136,6 +1137,114 @@
 for subtree in subm.walksubtrees(matcher=matcher):
 yield subtree
 
+class manifestfulltextcache(util.lrucachedict):
+"""File-backed LRU cache for the manifest cache
+
+File consists of entries, up to EOF:
+
+- 20 bytes node, 4 bytes length,  manifest data
+
+These are written in reverse cache order (oldest to newest).
+
+"""
+def __init__(self, max):
+super(manifestfulltextcache, self).__init__(max)
+self._dirty = False
+self._read = False
+self._opener = None
+
+def read(self):
+if self._read or self._opener is None:
+return
+
+try:
+with self._opener('manifestfulltextcache') as fp:
+set = super(manifestfulltextcache, self).__setitem__
+#

D3948: curses: use "text" interface if TERM=dumb

2018-07-16 Thread spectral (Kyle Lippincott)

spectral created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D3948

AFFECTED FILES
  mercurial/ui.py
  tests/test-commit-interactive-curses.t

CHANGE DETAILS

diff --git a/tests/test-commit-interactive-curses.t 
b/tests/test-commit-interactive-curses.t
--- a/tests/test-commit-interactive-curses.t
+++ b/tests/test-commit-interactive-curses.t
@@ -368,6 +368,16 @@
   $ chunkselectorinterface
   curses
 
+If TERM=dumb, we use text, even if the config says curses
+  $ chunkselectorinterface
+  curses
+  $ TERM=dumb chunkselectorinterface
+  text
+(Something is keeping TERM=dumb in the environment unless I do this, it's not
+scoped to just that previous command like in many shells)
+  $ TERM=xterm chunkselectorinterface
+  curses
+
 It is possible to override the default interface with a feature specific
 interface
   $ cp $HGRCPATH.pretest $HGRCPATH
diff --git a/mercurial/ui.py b/mercurial/ui.py
--- a/mercurial/ui.py
+++ b/mercurial/ui.py
@@ -1214,7 +1214,7 @@
 "Feature %s does not handle all default interfaces" %
 feature)
 
-if self.plain():
+if self.plain() or encoding.environ.get('TERM') == 'dumb':
 return "text"
 
 # Default interface for all the features



To: spectral, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 4 of 5] debugformat: add information about sparse-revlog

2018-07-16 Thread Boris Feld

# HG changeset patch
# User Paul Morelle 
# Date 1529584381 -7200
#  Thu Jun 21 14:33:01 2018 +0200
# Node ID 35a23148fd05821bc620673e8217cbdb400f4412
# Parent  c34ef3def14e04dca76c43667766496a99636b44
# EXP-Topic write-for-sparse-read
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
35a23148fd05
debugformat: add information about sparse-revlog

Show information about sparse-revlog in debugformat, just like other
requirements.

diff --git a/mercurial/upgrade.py b/mercurial/upgrade.py
--- a/mercurial/upgrade.py
+++ b/mercurial/upgrade.py
@@ -259,6 +259,28 @@ class generaldelta(requirementformatvari
'faster')
 
 @registerformatvariant
+class sparserevlog(requirementformatvariant):
+name = 'sparserevlog'
+
+_requirement = localrepo.SPARSEREVLOG_REQUIREMENT
+
+default = False
+
+description = _('in order to limit disk reading and memory usage on older '
+'version, the span of a delta chain from its root to its '
+'end is limited, whatever the relevant data in this span. '
+'This can severly limit Mercurial ability to build good '
+'chain of delta resulting is much more storage space being 
'
+'taken and limit reusability of on disk delta during '
+'exchange.'
+   )
+
+upgrademessage = _('Revlog supports delta chain with more unused data '
+   'between payload. These gaps will be skipped at read '
+   'time. This allows for better delta chains, making a '
+   'better compression and faster exchange with server.')
+
+@registerformatvariant
 class removecldeltachain(formatvariant):
 name = 'plain-cl-delta'
 
diff --git a/tests/test-upgrade-repo.t b/tests/test-upgrade-repo.t
--- a/tests/test-upgrade-repo.t
+++ b/tests/test-upgrade-repo.t
@@ -56,6 +56,7 @@ An upgrade of a repository created with 
   fncache:yes
   dotencode:  yes
   generaldelta:   yes
+  sparserevlog:no
   plain-cl-delta: yes
   compression:zlib
   $ hg debugformat --verbose
@@ -63,6 +64,7 @@ An upgrade of a repository created with 
   fncache:yesyes yes
   dotencode:  yesyes yes
   generaldelta:   yesyes yes
+  sparserevlog:no no  no
   plain-cl-delta: yesyes yes
   compression:zlib   zlibzlib
   $ hg debugformat --verbose --config format.usegfncache=no
@@ -70,6 +72,7 @@ An upgrade of a repository created with 
   fncache:yesyes yes
   dotencode:  yesyes yes
   generaldelta:   yesyes yes
+  sparserevlog:no no  no
   plain-cl-delta: yesyes yes
   compression:zlib   zlibzlib
   $ hg debugformat --verbose --config format.usegfncache=no --color=debug
@@ -77,6 +80,7 @@ An upgrade of a repository created with 
   [formatvariant.name.uptodate|fncache:   ][formatvariant.repo.uptodate| 
yes][formatvariant.config.default|yes][formatvariant.default| yes]
   [formatvariant.name.uptodate|dotencode: ][formatvariant.repo.uptodate| 
yes][formatvariant.config.default|yes][formatvariant.default| yes]
   [formatvariant.name.uptodate|generaldelta:  ][formatvariant.repo.uptodate| 
yes][formatvariant.config.default|yes][formatvariant.default| yes]
+  [formatvariant.name.uptodate|sparserevlog:  ][formatvariant.repo.uptodate|  
no][formatvariant.config.default| no][formatvariant.default|  no]
   [formatvariant.name.uptodate|plain-cl-delta:][formatvariant.repo.uptodate| 
yes][formatvariant.config.default|yes][formatvariant.default| yes]
   [formatvariant.name.uptodate|compression:   ][formatvariant.repo.uptodate| 
zlib][formatvariant.config.default|   zlib][formatvariant.default|zlib]
   $ hg debugformat -Tjson
@@ -100,6 +104,12 @@ An upgrade of a repository created with 
 "repo": true
},
{
+"config": false,
+"default": false,
+"name": "sparserevlog",
+"repo": false
+   },
+   {
 "config": true,
 "default": true,
 "name": "plain-cl-delta",
@@ -119,6 +129,9 @@ An upgrade of a repository created with 
   requirements
  preserved: dotencode, fncache, generaldelta, revlogv1, store
   
+  sparserevlog
+ Revlog supports delta chain with more unused data between payload. These 
gaps will be skipped at read time. This allows for better delta chains, making 
a better compression and faster exchange with server.
+  
   additional optimizations are available by specifying "--optimize ":
   
   redeltaparent
@@ -143,6 +156,9 @@ An upgrade of a repository created with 
   requirements
  preserved: dotencode, fncache, generaldelta, revlogv1, store
   
+  sparserevlog
+ Revlog supports delta chain with more unused data between payload. These 
gaps will be skipped at read time. This allows for better delta

[PATCH 3 of 5] sparse-revlog: implement algorithm to write sparse delta chains (issue5480)

2018-07-16 Thread Boris Feld

# HG changeset patch
# User Paul Morelle 
# Date 1528179575 -7200
#  Tue Jun 05 08:19:35 2018 +0200
# Node ID c34ef3def14e04dca76c43667766496a99636b44
# Parent  5ae60e5a705ef273d316e33df401e4c44a4c482a
# EXP-Topic write-for-sparse-read
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
c34ef3def14e
sparse-revlog: implement algorithm to write sparse delta chains (issue5480)

The classic behavior of revlog._isgooddeltainfo is to consider the span size
of the whole delta chain, and limit it to 4 * textlen.
Once sparse-revlog writing is allowed (and enforced with a requirement),
revlog._isgooddeltainfo considers the span of the largest chunk as the
distance used in the verification, instead of using the span of the whole
delta chain.

In order to compute the span of the largest chunk, we need to slice into
chunks a chain with the new revision at the top of the revlog, and take the
maximal span of these chunks. The sparse read density is a parameter to the
slicing, as it will stop when the global read density reaches this threshold.
For instance, a density of 50% means that 2 of 4 read bytes are actually used
for the reconstruction of the revision (the others are part of other chains).

This allows a new revision to be potentially stored with a diff against
another revision anywhere in the history, instead of forcing it in the last 4
* textlen. The result is a much better compression on repositories that have
many concurrent branches. Here are a comparison between using deltas from
current upstream (aggressive-merge-deltas on by default) and deltas from a
sparse-revlog

Comparison of `.hg/store/` size:

mercurial (6.74% merges):
before: 46,831,873 bytes
after:  46,795,992 bytes (no relevant change)
pypy (8.30% merges):
before:333,524,651 bytes
after: 308,417,511 bytes -8%
netbeans (34.21% merges):
before:  1,141,847,554 bytes
after:   1,131,093,161 bytes -1%
mozilla-central (4.84% merges):
before:  2,344,248,850 bytes
after:   2,328,459,258 bytes -1%
large-private-repo-A (merge 19.73%)
before: 41,510,550,163 bytes
after:   8,121,763,428 bytes -80%
large-private-repo-B (23.77%)
before: 58,702,221,709 bytes
after:   8,351,588,828 bytes -76%

Comparison of `00manifest.d` size:

mercurial (6.74% merges):
before:  6,143,044 bytes
after:   6,107,163 bytes
pypy (8.30% merges):
before: 52,941,780 bytes
after:  27,834,082 bytes -48%
netbeans (34.21% merges):
before:130,088,982 bytes
after: 119,337,636 bytes -10%
mozilla-central (4.84% merges):
before:215,096,339 bytes
after: 199,496,863 bytes -8%
large-private-repo-A (merge 19.73%)
before: 33,725,285,081 bytes
after: 390,302,545 bytes -99%
large-private-repo-B (23.77%)
before: 49,457,701,645 bytes
after:   1,366,752,187 bytes -97%


The better delta chains provide a performance boost in relevant repositories:

pypy, bundling 1000 revisions:
before: 1.670s
after:  1.149s -31%

Unbundling got a bit slower. probably because the sparse algorithm is still
pure
python.

pypy, unbundling 1000 revisions:
before: 4.062s
after:  4.507s +10%

Performance of bundle/unbundle in repository with few concurrent branches (eg:
mercurial) are unaffected.

No significant differences have been noticed then timing `hg push` and `hg
pull` locally. More state timings are being gathered.

Same as for aggressive-merge-delta, better delta comes with longer delta
chains. Longer chains have a performance impact. For example. The length of
the chain needed to get the manifest of pypy's tip moves from 82 item to 1929
items. This moves the restore time from 3.88ms to 11.3ms.

Delta chain length is an independent issue that affects repository without
this changes. It will be dealt with independently.

No significant differences have been observed on repositories where
`sparse-revlog` have not much effect (mercurial, unity, netbeans). On pypy,
small differences have been observed on some operation affected by delta chain
building and retrieval.


pypy, perfmanifest
before: 0.006162s
after:  0.017899s +190%

pypy, commit:
before: 0.382
after:  0.376 -1%

pypy, status:
before: 0.157
after:  0.168 +7%

More comprehensive and stable timing comparisons are in progress.

diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -216,6 +216,9 @@ class _testrevlog(object):
 def length(self, rev):
 return self.end(rev) - self.start(rev)
 
+def __len__(self):
+return len(self._data)
+
 def _trimchunk(revlog, revs, startidx, endidx=None):
 """returns

[PATCH 1 of 5] sparse-revlog: new requirement enabled with format.sparse-revlog

2018-07-16 Thread Boris Feld

# HG changeset patch
# User Paul Morelle 
# Date 1528143798 -7200
#  Mon Jun 04 22:23:18 2018 +0200
# Node ID 6ed641517622f53dd51193c3ae3930c66fc18078
# Parent  cc76692f401d0063cd5dfaf46f0fca1563a85fae
# EXP-Topic write-for-sparse-read
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
6ed641517622
sparse-revlog: new requirement enabled with format.sparse-revlog

The meaning of the new 'sparse-revlog' requirement is that the revlogs are
allowed to contain wider delta chains with larger holes between the interesting
chunks. These sparse delta chains should be read in several chunks to avoid a
potential explosion of memory usage.

Former version won't know how to read a delta chain in several chunks. They
would keep reading them in a single read, and therefore would be subject to the
potential memory explosion. Hence this new requirement: only versions having
support of sparse-revlog reading should be allowed to read such a revlog.

Implementation of this new algorithm and tools to enable or disable the
requirement will follow in the next changesets.

diff --git a/mercurial/configitems.py b/mercurial/configitems.py
--- a/mercurial/configitems.py
+++ b/mercurial/configitems.py
@@ -652,6 +652,9 @@ coreconfigitem('format', 'maxchainlen',
 coreconfigitem('format', 'obsstore-version',
 default=None,
 )
+coreconfigitem('format', 'sparse-revlog',
+default=False,
+)
 coreconfigitem('format', 'usefncache',
 default=True,
 )
diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -354,6 +354,15 @@ class locallegacypeer(localpeer):
 # clients.
 REVLOGV2_REQUIREMENT = 'exp-revlogv2.0'
 
+# A repository with the sparserevlog feature will have delta chains that
+# can spread over a larger span. Sparse reading cuts these large spans into
+# pieces, so that each piece isn't too big.
+# Without the sparserevlog capability, reading from the repository could use
+# huge amounts of memory, because the whole span would be read at once,
+# including all the intermediate revisions that aren't pertinent for the chain.
+# This is why once a repository has enabled sparse-read, it becomes required.
+SPARSEREVLOG_REQUIREMENT = 'sparserevlog'
+
 # Functions receiving (ui, features) that extensions can register to impact
 # the ability to load repositories with custom requirements. Only
 # functions defined in loaded extensions are called.
@@ -376,6 +385,7 @@ class localrepository(object):
 'generaldelta',
 'treemanifest',
 REVLOGV2_REQUIREMENT,
+SPARSEREVLOG_REQUIREMENT,
 }
 _basesupported = supportedformats | {
 'store',
@@ -678,6 +688,8 @@ class localrepository(object):
 self.svfs.options['with-sparse-read'] = withsparseread
 self.svfs.options['sparse-read-density-threshold'] = srdensitythres
 self.svfs.options['sparse-read-min-gap-size'] = srmingapsize
+sparserevlog = SPARSEREVLOG_REQUIREMENT in self.requirements
+self.svfs.options['sparse-revlog'] = sparserevlog
 
 for r in self.requirements:
 if r.startswith('exp-compression-'):
@@ -2370,6 +2382,9 @@ def newreporequirements(repo):
 requirements.add('generaldelta')
 if ui.configbool('experimental', 'treemanifest'):
 requirements.add('treemanifest')
+# experimental config: format.sparse-revlog
+if ui.configbool('format', 'sparse-revlog'):
+requirements.add(SPARSEREVLOG_REQUIREMENT)
 
 revlogv2 = ui.config('experimental', 'revlogv2')
 if revlogv2 == 'enable-unstable-format-and-corrupt-my-data':
diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -895,6 +895,7 @@ class revlog(object):
 self._compengine = 'zlib'
 self._maxdeltachainspan = -1
 self._withsparseread = False
+self._sparserevlog = False
 self._srdensitythreshold = 0.50
 self._srmingapsize = 262144
 
@@ -923,7 +924,10 @@ class revlog(object):
 self._maxdeltachainspan = opts['maxdeltachainspan']
 if mmaplargeindex and 'mmapindexthreshold' in opts:
 mmapindexthreshold = opts['mmapindexthreshold']
-self._withsparseread = bool(opts.get('with-sparse-read', False))
+self._sparserevlog = bool(opts.get('sparse-revlog', False))
+withsparseread = bool(opts.get('with-sparse-read', False))
+# sparse-revlog forces sparse-read
+self._withsparseread = self._sparserevlog or withsparseread
 if 'sparse-read-density-threshold' in opts:
 self._srdensitythreshold = 
opts['sparse-read-density-threshold']
 if 'sparse-read-min-gap-size' in opts:
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org

[PATCH 2 of 5] revlog: extract `deltainfo.distance` for future conditional redefinition

2018-07-16 Thread Boris Feld

# HG changeset patch
# User Paul Morelle 
# Date 1528107120 -7200
#  Mon Jun 04 12:12:00 2018 +0200
# Node ID 5ae60e5a705ef273d316e33df401e4c44a4c482a
# Parent  6ed641517622f53dd51193c3ae3930c66fc18078
# EXP-Topic write-for-sparse-read
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
5ae60e5a705e
revlog: extract `deltainfo.distance` for future conditional redefinition

This commit exist to make the next one clearer.

diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -2393,13 +2393,14 @@ class revlog(object):
 #   deltas we need to apply -- bounding it limits the amount of CPU
 #   we consume.
 
+distance = deltainfo.distance
 textlen = revinfo.textlen
 defaultmax = textlen * 4
 maxdist = self._maxdeltachainspan
 if not maxdist:
-maxdist = deltainfo.distance # ensure the conditional pass
+maxdist = distance # ensure the conditional pass
 maxdist = max(maxdist, defaultmax)
-if (deltainfo.distance > maxdist or deltainfo.deltalen > textlen or
+if (distance > maxdist or deltainfo.deltalen > textlen or
 deltainfo.compresseddeltalen > textlen * 2 or
 (self._maxchainlen and deltainfo.chainlen > self._maxchainlen)):
 return False
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 5 of 5] upgrade: enable adding or removing sparse-revlog requirement

2018-07-16 Thread Boris Feld

# HG changeset patch
# User Paul Morelle 
# Date 1529597155 -7200
#  Thu Jun 21 18:05:55 2018 +0200
# Node ID 238a33e6424ec36cd95a254722af5602d8f981f6
# Parent  35a23148fd05821bc620673e8217cbdb400f4412
# EXP-Topic write-for-sparse-read
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
238a33e6424e
upgrade: enable adding or removing sparse-revlog requirement

diff --git a/mercurial/upgrade.py b/mercurial/upgrade.py
--- a/mercurial/upgrade.py
+++ b/mercurial/upgrade.py
@@ -61,7 +61,9 @@ def supportremovedrequirements(repo):
 the dropped requirement must appear in the returned set for the upgrade
 to be allowed.
 """
-return set()
+return {
+localrepo.SPARSEREVLOG_REQUIREMENT,
+}
 
 def supporteddestrequirements(repo):
 """Obtain requirements that upgrade supports in the destination.
@@ -77,6 +79,7 @@ def supporteddestrequirements(repo):
 'generaldelta',
 'revlogv1',
 'store',
+localrepo.SPARSEREVLOG_REQUIREMENT,
 }
 
 def allowednewrequirements(repo):
@@ -93,6 +96,7 @@ def allowednewrequirements(repo):
 'dotencode',
 'fncache',
 'generaldelta',
+localrepo.SPARSEREVLOG_REQUIREMENT,
 }
 
 def preservedrequirements(repo):
diff --git a/tests/test-upgrade-repo.t b/tests/test-upgrade-repo.t
--- a/tests/test-upgrade-repo.t
+++ b/tests/test-upgrade-repo.t
@@ -129,9 +129,6 @@ An upgrade of a repository created with 
   requirements
  preserved: dotencode, fncache, generaldelta, revlogv1, store
   
-  sparserevlog
- Revlog supports delta chain with more unused data between payload. These 
gaps will be skipped at read time. This allows for better delta chains, making 
a better compression and faster exchange with server.
-  
   additional optimizations are available by specifying "--optimize ":
   
   redeltaparent
@@ -156,9 +153,6 @@ An upgrade of a repository created with 
   requirements
  preserved: dotencode, fncache, generaldelta, revlogv1, store
   
-  sparserevlog
- Revlog supports delta chain with more unused data between payload. These 
gaps will be skipped at read time. This allows for better delta chains, making 
a better compression and faster exchange with server.
-  
   redeltaparent
  deltas within internal storage will choose a new base revision if needed
   
@@ -241,9 +235,6 @@ Various sub-optimal detections work
   generaldelta
  repository storage will be able to create optimal deltas; new repository 
data will be smaller and read times should decrease; interacting with other 
repositories using this storage model should require less network and CPU 
resources, making "hg push" and "hg pull" faster
   
-  sparserevlog
- Revlog supports delta chain with more unused data between payload. These 
gaps will be skipped at read time. This allows for better delta chains, making 
a better compression and faster exchange with server.
-  
   additional optimizations are available by specifying "--optimize ":
   
   redeltaparent
@@ -286,9 +277,6 @@ Various sub-optimal detections work
   generaldelta
  repository storage will be able to create optimal deltas; new repository 
data will be smaller and read times should decrease; interacting with other 
repositories using this storage model should require less network and CPU 
resources, making "hg push" and "hg pull" faster
   
-  sparserevlog
- Revlog supports delta chain with more unused data between payload. These 
gaps will be skipped at read time. This allows for better delta chains, making 
a better compression and faster exchange with server.
-  
   additional optimizations are available by specifying "--optimize ":
   
   redeltaparent
@@ -315,9 +303,6 @@ Upgrading a repository that is already m
   requirements
  preserved: dotencode, fncache, generaldelta, revlogv1, store
   
-  sparserevlog
- Revlog supports delta chain with more unused data between payload. These 
gaps will be skipped at read time. This allows for better delta chains, making 
a better compression and faster exchange with server.
-  
   beginning upgrade...
   repository locked and read-only
   creating temporary repository to stage migrated data: 
$TESTTMP/modern/.hg/upgrade.* (glob)
@@ -355,9 +340,6 @@ Upgrading a repository to generaldelta w
   generaldelta
  repository storage will be able to create optimal deltas; new repository 
data will be smaller and read times should decrease; interacting with other 
repositories using this storage model should require less network and CPU 
resources, making "hg push" and "hg pull" faster
   
-  sparserevlog
- Revlog supports delta chain with more unused data between payload. These 
gaps will be skipped at read time. This allows for better delta chains, making 
a better compression and faster exchange with server.
-  
   beginning upgrade...
   repository locked and read-only
   creating temporary

Re: [PATCH] debug: move extensions debug behind a dedicated flag

2018-07-16 Thread Martin von Zweigbergk via Mercurial-devel

On Mon, Jul 16, 2018 at 10:07 AM Boris FELD  wrote:

> On 16/07/2018 18:48, Martin von Zweigbergk via Mercurial-devel wrote:
>
>
>
> On Mon, Jul 16, 2018, 09:24 Boris Feld  wrote:
>
>> # HG changeset patch
>> # User Boris Feld 
>> # Date 1531756950 -7200
>> #  Mon Jul 16 18:02:30 2018 +0200
>> # Node ID ae4c03e746f6837692762fe488991b64008efd3e
>> # Parent  3c569172848d329a81b1420d7cd8a5549b48f25f
>> # EXP-Topic extdebug
>> # Available At https://bitbucket.org/octobus/mercurial-devel/
>> #  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r
>> ae4c03e746f6
>> debug: move extensions debug behind a dedicated flag
>>
>> Since b86664c81833, we process the `--debug` flag earlier. This is overall
>> good and useful, but has at least one negative side effect.
>>
>> Previously the debug message we report when trying to import extensions
>> were
>> issued before we processed the `--debug` flag. Now they happen after.
>>
>> Before:
>>
>>   $ ./hg id --debug
>>   21f507b8de2f9c1606e9aeb5ec7d2a6dedb7a4a7 tip
>>
>> After:
>>
>>   $ ./hg id --debug  ☿ (revset-bench)
>>   could not import hgext.evolve (No module named evolve): trying
>> hgext3rd.evolve
>>   could not import hgext.mercurial_keyring (No module named
>> mercurial_keyring): trying hgext3rd.mercurial_keyring
>>   could not import hgext3rd.mercurial_keyring (No module named
>> mercurial_keyring): trying mercurial_keyring
>>   could not import hgext.hggit (No module named hggit): trying
>> hgext3rd.hggit
>>   could not import hgext3rd.hggit (No module named hggit): trying hggit
>>   21f507b8de2f9c1606e9aeb5ec7d2a6dedb7a4a7 tip
>>
>
> Isn't it about time we add a --full-node option? The --debug flag has no
> backwards compatibility guarantee.
>
>
> The --full-node is covered by templating. So I don't think we use anything
> special regarding `hg id` here.
>
> The issue here is not about backward compatibility, but about usability.
> The extra debug lines are very verbose and gets in the ways (especially
> with traceback). I think they should be removed. `hg id` was only used an
> example of a simple command with a small output.
>

I see. I think `hg id --debug` is actually somewhat frequently (there are
at least 30 instances in our test cases), so I thought you wanted to
preserve that behavior.

I agree that the extension-loading debug printouts are a bit excessive, so
I don't mind hiding them behind a config option.


>
>
> Cheers,
>
> --
> Boris Feld
>
>
>
> ___
> Mercurial-devel mailing 
> listMercurial-devel@mercurial-scm.orghttps://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
>
>
>
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] debug: move extensions debug behind a dedicated flag

2018-07-16 Thread Boris FELD


On 16/07/2018 18:48, Martin von Zweigbergk via Mercurial-devel wrote:



On Mon, Jul 16, 2018, 09:24 Boris Feld > wrote:


# HG changeset patch
# User Boris Feld mailto:boris.f...@octobus.net>>
# Date 1531756950 -7200
#      Mon Jul 16 18:02:30 2018 +0200
# Node ID ae4c03e746f6837692762fe488991b64008efd3e
# Parent  3c569172848d329a81b1420d7cd8a5549b48f25f
# EXP-Topic extdebug
# Available At https://bitbucket.org/octobus/mercurial-devel/
#              hg pull
https://bitbucket.org/octobus/mercurial-devel/ -r ae4c03e746f6
debug: move extensions debug behind a dedicated flag

Since b86664c81833, we process the `--debug` flag earlier. This is
overall
good and useful, but has at least one negative side effect.

Previously the debug message we report when trying to import
extensions were
issued before we processed the `--debug` flag. Now they happen after.

Before:

  $ ./hg id --debug
  21f507b8de2f9c1606e9aeb5ec7d2a6dedb7a4a7 tip

After:

  $ ./hg id --debug                          ☿ (revset-bench)
  could not import hgext.evolve (No module named evolve): trying
hgext3rd.evolve
  could not import hgext.mercurial_keyring (No module named
mercurial_keyring): trying hgext3rd.mercurial_keyring
  could not import hgext3rd.mercurial_keyring (No module named
mercurial_keyring): trying mercurial_keyring
  could not import hgext.hggit (No module named hggit): trying
hgext3rd.hggit
  could not import hgext3rd.hggit (No module named hggit): trying
hggit
  21f507b8de2f9c1606e9aeb5ec7d2a6dedb7a4a7 tip


Isn't it about time we add a --full-node option? The --debug flag has 
no backwards compatibility guarantee.




The --full-node is covered by templating. So I don't think we use 
anything special regarding `hg id` here.


The issue here is not about backward compatibility, but about usability. 
The extra debug lines are very verbose and gets in the ways (especially 
with traceback). I think they should be removed. `hg id` was only used 
an example of a simple command with a small output.


Cheers,

--
Boris Feld



___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel


___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] debug: move extensions debug behind a dedicated flag

2018-07-16 Thread Martin von Zweigbergk via Mercurial-devel

On Mon, Jul 16, 2018, 09:24 Boris Feld  wrote:

> # HG changeset patch
> # User Boris Feld 
> # Date 1531756950 -7200
> #  Mon Jul 16 18:02:30 2018 +0200
> # Node ID ae4c03e746f6837692762fe488991b64008efd3e
> # Parent  3c569172848d329a81b1420d7cd8a5549b48f25f
> # EXP-Topic extdebug
> # Available At https://bitbucket.org/octobus/mercurial-devel/
> #  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r
> ae4c03e746f6
> debug: move extensions debug behind a dedicated flag
>
> Since b86664c81833, we process the `--debug` flag earlier. This is overall
> good and useful, but has at least one negative side effect.
>
> Previously the debug message we report when trying to import extensions
> were
> issued before we processed the `--debug` flag. Now they happen after.
>
> Before:
>
>   $ ./hg id --debug
>   21f507b8de2f9c1606e9aeb5ec7d2a6dedb7a4a7 tip
>
> After:
>
>   $ ./hg id --debug  ☿ (revset-bench)
>   could not import hgext.evolve (No module named evolve): trying
> hgext3rd.evolve
>   could not import hgext.mercurial_keyring (No module named
> mercurial_keyring): trying hgext3rd.mercurial_keyring
>   could not import hgext3rd.mercurial_keyring (No module named
> mercurial_keyring): trying mercurial_keyring
>   could not import hgext.hggit (No module named hggit): trying
> hgext3rd.hggit
>   could not import hgext3rd.hggit (No module named hggit): trying hggit
>   21f507b8de2f9c1606e9aeb5ec7d2a6dedb7a4a7 tip
>

Isn't it about time we add a --full-node option? The --debug flag has no
backwards compatibility guarantee.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 2] revset: add larger test for heads(ancestors(…))

2018-07-16 Thread Boris Feld

# HG changeset patch
# User Boris Feld 
# Date 1531752392 -7200
#  Mon Jul 16 16:46:32 2018 +0200
# Node ID 9967063d91c275f21634c36e809d0e9982896d0c
# Parent  d401e0d99b2c8e18bda7e007c2cd7599afbb365c
# EXP-Topic revset-bench
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
9967063d91c2
revset: add larger test for heads(ancestors(…))

It is important to not regress on this benchmark so we move it into the "base"
file. And we add another benchmark with more than two revisions.

diff --git a/contrib/all-revsets.txt b/contrib/all-revsets.txt
--- a/contrib/all-revsets.txt
+++ b/contrib/all-revsets.txt
@@ -138,3 +138,4 @@ secret()
 
 # test finding common ancestors
 heads(commonancestors(last(head(), 2)))
+heads(commonancestors(head()))
diff --git a/contrib/base-revsets.txt b/contrib/base-revsets.txt
--- a/contrib/base-revsets.txt
+++ b/contrib/base-revsets.txt
@@ -46,3 +46,4 @@ parents(2)
 (2::) - (2)
 # The one below is used by rebase
 (children(ancestor(tip~5, tip)) and ::(tip~5))::
+heads(commonancestors(last(head(), 2)))
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 1 of 2] revset-benchmark: use a generic revset to test `heads(commonancestors())`

2018-07-16 Thread Boris Feld

# HG changeset patch
# User Boris Feld 
# Date 1531752215 -7200
#  Mon Jul 16 16:43:35 2018 +0200
# Node ID d401e0d99b2c8e18bda7e007c2cd7599afbb365c
# Parent  3c569172848d329a81b1420d7cd8a5549b48f25f
# EXP-Topic revset-bench
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
d401e0d99b2c
revset-benchmark: use a generic revset to test `heads(commonancestors())`

This allow to benchmark revset performance in other repositories than just
the mercurial one.

diff --git a/contrib/all-revsets.txt b/contrib/all-revsets.txt
--- a/contrib/all-revsets.txt
+++ b/contrib/all-revsets.txt
@@ -137,4 +137,4 @@ draft()
 secret()
 
 # test finding common ancestors
-heads(commonancestors(46879 + ae7f))
+heads(commonancestors(last(head(), 2)))
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 2] revlog: reintroduce `revlog.descendant` as deprecated

2018-07-16 Thread Boris Feld

# HG changeset patch
# User Boris Feld 
# Date 1531750963 -7200
#  Mon Jul 16 16:22:43 2018 +0200
# Node ID 52aa3f0c736e48779c3f0ef4ee54545dd73576b7
# Parent  94a6a0ef74f6e52cbb26e3461df2f155ab873039
# EXP-Topic deprecate-descendant
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
52aa3f0c736e
revlog: reintroduce `revlog.descendant` as deprecated

Reintroduce `revlog.descendant` to help extensions authors update their
extensions in order to use the new API.

diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -1666,6 +1666,11 @@ class revlog(object):
 a, b = self.rev(a), self.rev(b)
 return self.isancestorrev(a, b)
 
+def descendant(self, a, b):
+msg = (b'revlog.descendant is deprecated, use revlog.isancestorrev')
+self._repo.ui.deprecwarn(msg, b'4.7')
+return self.isancestorrev(a, b)
+
 def isancestorrev(self, a, b):
 """return True if revision a is an ancestor of revision b
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 1 of 2] context: reintroduce `ctx.descendant` as deprecated

2018-07-16 Thread Boris Feld

# HG changeset patch
# User Boris Feld 
# Date 1531750872 -7200
#  Mon Jul 16 16:21:12 2018 +0200
# Node ID 94a6a0ef74f6e52cbb26e3461df2f155ab873039
# Parent  3c569172848d329a81b1420d7cd8a5549b48f25f
# EXP-Topic deprecate-descendant
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
94a6a0ef74f6
context: reintroduce `ctx.descendant` as deprecated

Reintroduce `ctx.descendant` to help extensions authors update their
extensions in order to use the new API.

diff --git a/mercurial/context.py b/mercurial/context.py
--- a/mercurial/context.py
+++ b/mercurial/context.py
@@ -589,6 +589,12 @@ class changectx(basectx):
 short(n) for n in sorted(cahs) if n != anc))
 return changectx(self._repo, anc)
 
+def descendant(self, other):
+msg = (b'ctx.descendant(other) is deprecated, '
+   'use ctx.isancestorof(other)')
+self._repo.ui.deprecwarn(msg, b'4.7')
+return self.isancestorof(other)
+
 def isancestorof(self, other):
 """True if this changeset is an ancestor of other"""
 return self._repo.changelog.isancestorrev(self._rev, other._rev)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH] debug: move extensions debug behind a dedicated flag

2018-07-16 Thread Boris Feld

# HG changeset patch
# User Boris Feld 
# Date 1531756950 -7200
#  Mon Jul 16 18:02:30 2018 +0200
# Node ID ae4c03e746f6837692762fe488991b64008efd3e
# Parent  3c569172848d329a81b1420d7cd8a5549b48f25f
# EXP-Topic extdebug
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
ae4c03e746f6
debug: move extensions debug behind a dedicated flag

Since b86664c81833, we process the `--debug` flag earlier. This is overall
good and useful, but has at least one negative side effect.

Previously the debug message we report when trying to import extensions were
issued before we processed the `--debug` flag. Now they happen after.

Before:

  $ ./hg id --debug
  21f507b8de2f9c1606e9aeb5ec7d2a6dedb7a4a7 tip

After:

  $ ./hg id --debug  ☿ (revset-bench)
  could not import hgext.evolve (No module named evolve): trying hgext3rd.evolve
  could not import hgext.mercurial_keyring (No module named mercurial_keyring): 
trying hgext3rd.mercurial_keyring
  could not import hgext3rd.mercurial_keyring (No module named 
mercurial_keyring): trying mercurial_keyring
  could not import hgext.hggit (No module named hggit): trying hgext3rd.hggit
  could not import hgext3rd.hggit (No module named hggit): trying hggit
  21f507b8de2f9c1606e9aeb5ec7d2a6dedb7a4a7 tip

(This get worse if --traceback is used).

To work around this, we move this extensions related debug message behind a
new flag 'devel.debug.extensions' and restore the previous output.

I'm not fully happy about using the 'devel' section for a flag that can be
used by legitimate users to debug extensions issues. However, it fits well
next to other `devel.devel.*` options and is mostly used by extensions author
anyway.

We might move it to another, more appropriate section in the future (using
alias).

diff --git a/mercurial/configitems.py b/mercurial/configitems.py
--- a/mercurial/configitems.py
+++ b/mercurial/configitems.py
@@ -372,6 +372,9 @@ coreconfigitem('devel', 'user.obsmarker'
 coreconfigitem('devel', 'warn-config-unknown',
 default=None,
 )
+coreconfigitem('devel', 'debug.extensions',
+default=False,
+)
 coreconfigitem('devel', 'debug.peer-request',
 default=False,
 )
diff --git a/mercurial/extensions.py b/mercurial/extensions.py
--- a/mercurial/extensions.py
+++ b/mercurial/extensions.py
@@ -123,10 +123,11 @@ def _importext(name, path=None, reportfu
 def _reportimporterror(ui, err, failed, next):
 # note: this ui.debug happens before --debug is processed,
 #   Use --config ui.debug=1 to see them.
-ui.debug('could not import %s (%s): trying %s\n'
- % (failed, stringutil.forcebytestr(err), next))
-if ui.debugflag:
-ui.traceback()
+if ui.configbool('devel', 'debug.extensions'):
+ui.debug('could not import %s (%s): trying %s\n'
+ % (failed, stringutil.forcebytestr(err), next))
+if ui.debugflag:
+ui.traceback()
 
 def _rejectunicode(name, xs):
 if isinstance(xs, (list, set, tuple)):
diff --git a/tests/test-bad-extension.t b/tests/test-bad-extension.t
--- a/tests/test-bad-extension.t
+++ b/tests/test-bad-extension.t
@@ -72,10 +72,9 @@ names of extensions failed to load can b
   $ hg --config extensions.badexts=showbadexts.py showbadexts 2>&1 | grep 
'^BADEXTS'
   BADEXTS: badext badext2
 
-show traceback for ImportError of hgext.name if debug is set
-(note that --debug option isn't applied yet when loading extensions)
+show traceback for ImportError of hgext.name if devel.debug.extensions is set
 
-  $ (hg help help --traceback --config ui.debug=yes 2>&1) \
+  $ (hg help help --traceback --debug --config devel.debug.extensions=yes 
2>&1) \
   > | grep -v '^ ' \
   > | egrep 'extension..[^p]|^Exception|Traceback|ImportError|not import'
   *** failed to import extension badext from $TESTTMP/badext.py: bit bucket 
overflow
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

mercurial@38677: 18 new changesets

2018-07-16 Thread Mercurial Commits

18 new changesets in mercurial:

https://www.mercurial-scm.org/repo/hg/rev/b07b691d2667
changeset:   38660:b07b691d2667
user:Sune Foldager 
date:Wed Jul 11 16:11:33 2018 +0200
summary: tests: add diff color trailing whitespace test

https://www.mercurial-scm.org/repo/hg/rev/8ac0c9cd4c48
changeset:   38661:8ac0c9cd4c48
user:Martijn Pieters 
date:Wed Jul 11 14:28:13 2018 +0100
summary: fncache: avoid loading the filename cache when not actually 
modifying it

https://www.mercurial-scm.org/repo/hg/rev/ad2aa4e85047
changeset:   38662:ad2aa4e85047
user:Kyle Lippincott 
date:Wed Jul 11 13:02:06 2018 -0700
summary: removeemptydirs: add test for `hg split` inside a disappearing 
directory

https://www.mercurial-scm.org/repo/hg/rev/93d9690ff2b0
changeset:   38663:93d9690ff2b0
user:Martin von Zweigbergk 
date:Wed Jul 11 16:27:40 2018 -0700
summary: revlog: move comment about commonancestorsheads where it's used

https://www.mercurial-scm.org/repo/hg/rev/160da69ba1bf
changeset:   38664:160da69ba1bf
user:Martin von Zweigbergk 
date:Wed Jul 11 16:21:41 2018 -0700
summary: revlog: replace descendant(b, a) by isdescendantrev(a, b) (API)

https://www.mercurial-scm.org/repo/hg/rev/6afa928033bd
changeset:   38665:6afa928033bd
user:Martin von Zweigbergk 
date:Wed Jul 11 16:37:30 2018 -0700
summary: revlog: make isdescendantrev(a, b) check if a < b

https://www.mercurial-scm.org/repo/hg/rev/a06b2b032557
changeset:   38666:a06b2b032557
user:Martin von Zweigbergk 
date:Wed Jul 11 16:29:23 2018 -0700
summary: revlog: introduce a isancestorrev() and use it in rebase

https://www.mercurial-scm.org/repo/hg/rev/572dff5c946e
changeset:   38667:572dff5c946e
user:Sushil khanchi 
date:Sat Jun 30 12:42:49 2018 +0530
summary: rebase: add --confirm option

https://www.mercurial-scm.org/repo/hg/rev/21846c94e605
changeset:   38668:21846c94e605
user:Martin von Zweigbergk 
date:Thu Jul 12 08:14:21 2018 -0700
summary: revlog: delete isdescendantrev() in favor of isancestorrev()

https://www.mercurial-scm.org/repo/hg/rev/b5891bf8ab13
changeset:   38669:b5891bf8ab13
user:Martin von Zweigbergk 
date:Thu Jul 12 09:25:02 2018 -0700
summary: copies: delete now-unnecessary check for "a == b" before 
"a.descendant(b)"

https://www.mercurial-scm.org/repo/hg/rev/fbec9c0b32d3
changeset:   38670:fbec9c0b32d3
user:Martin von Zweigbergk 
date:Thu Jul 12 08:22:28 2018 -0700
summary: context: rename descendant() to isancestorof()

https://www.mercurial-scm.org/repo/hg/rev/99ed6e2f6606
changeset:   38671:99ed6e2f6606
user:Martin von Zweigbergk 
date:Wed Jul 11 15:03:39 2018 -0700
summary: rebase: reduce scope of a variable

https://www.mercurial-scm.org/repo/hg/rev/0f8599afb92f
changeset:   38672:0f8599afb92f
user:Martin von Zweigbergk 
date:Wed Jul 11 15:01:47 2018 -0700
summary: rebase: avoid converting from nodes to revnums twice

https://www.mercurial-scm.org/repo/hg/rev/20a30bb8f276
changeset:   38673:20a30bb8f276
user:Martin von Zweigbergk 
date:Wed Jul 11 15:12:01 2018 -0700
summary: rebase: use revnums (not nodes) for set of extinct revisions

https://www.mercurial-scm.org/repo/hg/rev/a50482254b0a
changeset:   38674:a50482254b0a
user:Sushil khanchi 
date:Thu Jul 12 22:23:51 2018 +0530
summary: rebase: make sure we don't loose the return code in --confirm 
option

https://www.mercurial-scm.org/repo/hg/rev/35b3f686157a
changeset:   38675:35b3f686157a
user:Sushil khanchi 
date:Thu Jul 12 22:35:54 2018 +0530
summary: rebase: correct misleading message in --confirm option

https://www.mercurial-scm.org/repo/hg/rev/3b072388ca78
changeset:   38676:3b072388ca78
user:Gregory Szorc 
date:Fri Jul 13 14:20:12 2018 -0700
summary: scmutil: rewrite docstring for filecache

https://www.mercurial-scm.org/repo/hg/rev/2a227782e754
changeset:   38677:2a227782e754
bookmark:@
tag: tip
user:Gregory Szorc 
date:Wed Jul 11 16:44:33 2018 -0700
summary: tests: add test demonstrating phase loss when cloning (issue5939)

-- 
Repository URL: https://www.mercurial-scm.org/repo/hg
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 2 V2] revset: special case commonancestors(none()) to be empty set

2018-07-16 Thread Yuya Nishihara

# HG changeset patch
# User Yuya Nishihara 
# Date 1531404449 -32400
#  Thu Jul 12 23:07:29 2018 +0900
# Node ID f749aaad0d2f52e3da2a8e80f44b798c0dd85f00
# Parent  ca00702fa0d3619554b2d17fb321fd107186c7b3
revset: special case commonancestors(none()) to be empty set

This matches the behavior of ancestor(none()).

From an implementation perspective, ancestor() and commonancestors() are
intersection, and ancestors() is union, so it would make some sense that
commonancestors(none()) returned all revisions. However, ancestor(none())
isn't implemented as such, which breaks ancestor(x) == max(commonancestors(x)).

From a user perspective, ancestors of nothing is nothing whichever type
of operation the ancestor predicate does.

diff --git a/mercurial/revset.py b/mercurial/revset.py
--- a/mercurial/revset.py
+++ b/mercurial/revset.py
@@ -632,7 +632,10 @@ def commonancestors(repo, subset, x):
 
 """
 # only wants the heads of the set passed in
-for r in heads(repo, fullreposet(repo), x, anyorder):
+h = heads(repo, fullreposet(repo), x, anyorder)
+if not h:
+return baseset()
+for r in h:
 subset &= dagop.revancestors(repo, baseset([r]))
 
 return subset
diff --git a/tests/test-revset.t b/tests/test-revset.t
--- a/tests/test-revset.t
+++ b/tests/test-revset.t
@@ -1063,6 +1063,12 @@ test common ancestors
   8
   9
 
+test ancestor variants of empty revision
+
+  $ log 'ancestor(none())'
+  $ log 'ancestors(none())'
+  $ log 'commonancestors(none())'
+
 test ancestors with depth limit
 
  (depth=0 selects the node itself)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 3 of 3] obsolete: explode if metadata contains invalid UTF-8 sequence (API)

2018-07-16 Thread Yuya Nishihara

# HG changeset patch
# User Yuya Nishihara 
# Date 1531647137 -32400
#  Sun Jul 15 18:32:17 2018 +0900
# Node ID d113293cbe05e5026bf0359ace3790ea2f3d2f95
# Parent  f34a53a6a9a0df0458669d5ce6a55a4af85e9cb6
obsolete: explode if metadata contains invalid UTF-8 sequence (API)

The current metadata API can be a source of bugs since it forces callers to
process encoding conversion by themselves. So let's make it reject bad data
as a last ditch. I assume there's no metadata field which is supposed to store
arbitrary BLOB like transplant_source.

diff --git a/mercurial/obsolete.py b/mercurial/obsolete.py
--- a/mercurial/obsolete.py
+++ b/mercurial/obsolete.py
@@ -80,6 +80,7 @@ from . import (
 obsutil,
 phases,
 policy,
+pycompat,
 util,
 )
 from .utils import dateutil
@@ -600,6 +601,16 @@ class obsstore(object):
 raise ValueError(_('in-marker cycle with %s') % node.hex(prec))
 
 metadata = tuple(sorted(metadata.iteritems()))
+for k, v in metadata:
+try:
+# might be better to reject non-ASCII keys
+k.decode('utf-8')
+v.decode('utf-8')
+except UnicodeDecodeError:
+raise error.ProgrammingError(
+'obsstore metadata must be valid UTF-8 sequence '
+'(key = %r, value = %r)'
+% (pycompat.bytestr(k), pycompat.bytestr(v)))
 
 marker = (bytes(prec), tuple(succs), int(flag), metadata, date, 
parents)
 return bool(self.add(transaction, [marker]))
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 3] obsolete: store user name and note in UTF-8 (issue5754) (BC)

2018-07-16 Thread Yuya Nishihara

# HG changeset patch
# User Yuya Nishihara 
# Date 1531646697 -32400
#  Sun Jul 15 18:24:57 2018 +0900
# Node ID f34a53a6a9a0df0458669d5ce6a55a4af85e9cb6
# Parent  4625a49972b58bb56bed7e618c71ef1742940a5c
obsolete: store user name and note in UTF-8 (issue5754) (BC)

Before, user names were stored in local encoding and transferred across
repositories, which made it impossible to restore non-ASCII user names on
different platforms. This patch fixes new markers to be encoded in UTF-8
and decoded back to local encoding when displaying. Existing markers are
unfixable so they may result in mojibake.

I don't like the API that requires metadata dict to be UTF-8 encoded, which
is a source of bugs, but there's no abstraction layer to process the encoding
thingy efficiently. So we apply the same rule as extras dict to obsstore
metadata.

diff --git a/mercurial/cmdutil.py b/mercurial/cmdutil.py
--- a/mercurial/cmdutil.py
+++ b/mercurial/cmdutil.py
@@ -2555,7 +2555,7 @@ def amend(ui, repo, old, extra, pats, op
 mapping = {old.node(): (newid,)}
 obsmetadata = None
 if opts.get('note'):
-obsmetadata = {'note': opts['note']}
+obsmetadata = {'note': encoding.fromlocal(opts['note'])}
 scmutil.cleanupnodes(repo, mapping, 'amend', metadata=obsmetadata,
  fixphase=True, targetphase=commitphase)
 
diff --git a/mercurial/debugcommands.py b/mercurial/debugcommands.py
--- a/mercurial/debugcommands.py
+++ b/mercurial/debugcommands.py
@@ -1619,7 +1619,7 @@ def debugobsolete(ui, repo, precursor=No
 if opts['rev']:
 raise error.Abort('cannot select revision when creating marker')
 metadata = {}
-metadata['user'] = opts['user'] or ui.username()
+metadata['user'] = encoding.fromlocal(opts['user'] or ui.username())
 succs = tuple(parsenodeid(succ) for succ in successors)
 l = repo.lock()
 try:
diff --git a/mercurial/obsolete.py b/mercurial/obsolete.py
--- a/mercurial/obsolete.py
+++ b/mercurial/obsolete.py
@@ -74,6 +74,7 @@ import struct
 
 from .i18n import _
 from . import (
+encoding,
 error,
 node,
 obsutil,
@@ -526,7 +527,7 @@ class obsstore(object):
 # prec:nodeid, predecessors changesets
 # succs:   tuple of nodeid, successor changesets (0-N length)
 # flag:integer, flag field carrying modifier for the markers (see doc)
-# meta:binary blob, encoded metadata dictionary
+# meta:binary blob in UTF-8, encoded metadata dictionary
 # date:(float, int) tuple, date of marker creation
 # parents: (tuple of nodeid) or None, parents of predecessors
 #  None is used when no data has been recorded
@@ -950,7 +951,8 @@ def createmarkers(repo, relations, flag=
  must be an iterable of (, (, ...)[,{metadata}])
 tuple. `old` and `news` are changectx. metadata is an optional dictionary
 containing metadata for this marker only. It is merged with the global
-metadata specified through the `metadata` argument of this function,
+metadata specified through the `metadata` argument of this function.
+Any string values in metadata must be UTF-8 bytes.
 
 Trying to obsolete a public changeset will raise an exception.
 
@@ -964,11 +966,8 @@ def createmarkers(repo, relations, flag=
 if metadata is None:
 metadata = {}
 if 'user' not in metadata:
-develuser = repo.ui.config('devel', 'user.obsmarker')
-if develuser:
-metadata['user'] = develuser
-else:
-metadata['user'] = repo.ui.username()
+luser = repo.ui.config('devel', 'user.obsmarker') or repo.ui.username()
+metadata['user'] = encoding.fromlocal(luser)
 
 # Operation metadata handling
 useoperation = repo.ui.configbool('experimental',
diff --git a/mercurial/obsutil.py b/mercurial/obsutil.py
--- a/mercurial/obsutil.py
+++ b/mercurial/obsutil.py
@@ -12,6 +12,7 @@ import re
 from .i18n import _
 from . import (
 diffutil,
+encoding,
 node as nodemod,
 phases,
 util,
@@ -822,7 +823,8 @@ def markersusers(markers):
 """ Returns a sorted list of markers users without duplicates
 """
 markersmeta = [dict(m[3]) for m in markers]
-users = set(meta['user'] for meta in markersmeta if meta.get('user'))
+users = set(encoding.tolocal(meta['user']) for meta in markersmeta
+if meta.get('user'))
 
 return sorted(users)
 
diff --git a/tests/test-obsmarker-template.t b/tests/test-obsmarker-template.t
--- a/tests/test-obsmarker-template.t
+++ b/tests/test-obsmarker-template.t
@@ -2581,3 +2581,61 @@ Check other fatelog implementations
  date:Thu Jan 01 00:00:00 1970 +
  summary: ROOT
   
+
+Test metadata encoding (issue5754)
+==
+
+  $ hg init $TESTTMP/metadata-encoding
+  $ cd $TESTTMP/metadata-encoding
+  $ cat <<'EOF' >> .hg/hgrc
+  > [extensions]
+  > amend =
+  >

[PATCH 1 of 2 V2] revset: clarify heads() order doesn't matter while computing common ancestors

2018-07-16 Thread Yuya Nishihara

# HG changeset patch
# User Yuya Nishihara 
# Date 1531231313 -32400
#  Tue Jul 10 23:01:53 2018 +0900
# Node ID ca00702fa0d3619554b2d17fb321fd107186c7b3
# Parent  3c569172848d329a81b1420d7cd8a5549b48f25f
revset: clarify heads() order doesn't matter while computing common ancestors

Follows up 5460926352ee and 52f19a840543.

diff --git a/mercurial/revset.py b/mercurial/revset.py
--- a/mercurial/revset.py
+++ b/mercurial/revset.py
@@ -616,7 +616,7 @@ def _commonancestorheads(repo, subset, x
 
 # These greatest common ancestors are the same ones that the consesus bid
 # merge will find.
-h = heads(repo, fullreposet(repo), x, defineorder)
+h = heads(repo, fullreposet(repo), x, anyorder)
 
 ancs = repo.changelog._commonancestorsheads(*list(h))
 return subset & baseset(ancs)
@@ -632,7 +632,7 @@ def commonancestors(repo, subset, x):
 
 """
 # only wants the heads of the set passed in
-for r in heads(repo, fullreposet(repo), x, defineorder):
+for r in heads(repo, fullreposet(repo), x, anyorder):
 subset &= dagop.revancestors(repo, baseset([r]))
 
 return subset
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 3 of 7] dispatch: rework the serve --stdio safe argument checks

2018-07-16 Thread Yuya Nishihara

On Sun, 15 Jul 2018 12:33:48 +0200, Boris FELD wrote:
> The properties that are important for us are the following and we will 
> explain why a bit further:
>      - We want to activate it as soon as possible (ssh: dispatch level, 
> http: wsgi server),
>      - Controlled by the client,
>      - Security managed by core.
> 
> Yuya would like to move the implementation outside of core dispatch, for 
> security and maintenance concerns. Either, into custom ssh script, or at 
> the protocol level. Neither of these two locations fits our needs (and 
> constraints).
> 
> So we need to find a way to do this early extra validation of parameter 
> in a way that is good enough in terms of security and maintenance.
> 
> # Longer explanation
> 
> ## Why does it has to be set up so early?
> 
> A significant part of the slowness we observe happen very early, during 
> the initial setup (extensions setup, various data loading, custom 
> extensions code, etc...). So the earlier the extra debug/profile is 
> setup, the more useful data we gather.

I don't disagree with that, but I just doubt if that would be worth making
the dispatcher hack complicated and adding more variations of ssh entry
arguments. Honestly, I'm thinking that the use of "hg serve" as an entry
point was a design mistake which we can't fix, so I don't think it's good
idea to extend the syntax of the entry point.

> Doing this at the protocol level would be too late, as a lot of the 
> important time-consuming activities have already happened.
> 
> ## Why control it from the client instead of being a custom ssh script?
> 
> Some of the people we work with have a high level of security (eg: we 
> don't have a full shell access on their server and we only have access 
> to a subset of their repositories).
> 
> This means multiples things:
> 
> * We still want some security management, using `hg debugserve` would 
> allow arbitrary code to be executed, something we still want to prevent.
> 
> * Deploying a new ssh script is something complex that takes time. 
> Admins have to look at the new script and deploy it. It can take 
> multiple days, making us unable to collect these valuable data (in time).
> 
> * If we, mercurial core developers, are afraid of getting the validation 
> logic in core wrong, I don't think we should expect people not familiar 
> with Mercurial to be able to get that right. So moving the 
> responsibility to validate input to admins seems dangerous.
> 
> * Having an example script in `contrib/` would not help much as people 
> usually already have some custom script.

Why? If there's already a custom script installed, IIUC, only thing the
sysadmin will have to do is to replace 'hg serve' with
/path/to/contrib/hg-ssh-script, one time. After that, you can inject
commands understood by that script via sys.argv or stdin.

If the server has a robust pre-validation logic of 'hg serve' command,
or hg-ssh is used for example, argv injection wouldn't work anyway unless
the validation rule is relaxed or the existing script is adjusted to allow
extra command arguments.

> * Controlling option from the client is critical. When investigating 
> something, we try various options to look at the issue from different 
> angles (profile cpu vs time). Having to redeploy a new script every time 
> we want to change an option break our ability to investigate quickly as 
> it often requires synchronization between different teams.
> 
> ## Why do you want this into core
> 
> * Make it much simpler for us to use it at more place (including to help 
> random people on the list and bug tracker). Having to deploy a custom 
> Mercurial binaries on the server is a strong barrier.
> 
> * Given how valuable this is to us, we expect it to be valuable for 
> others. In fact server-side debug message is something various users 
> asked for in the past.

Well, server-side "debug" message could be handled at protocol level
unlike profiling of the early uisetup() thingy. That said, I'm slightly
interested in how ssh issues are debugged in various production environments
as there might be a better approach.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 1 of 3] obsolete: clarify users in markerusers() never contain None

2018-07-16 Thread Yuya Nishihara

# HG changeset patch
# User Yuya Nishihara 
# Date 1531646560 -32400
#  Sun Jul 15 18:22:40 2018 +0900
# Node ID 4625a49972b58bb56bed7e618c71ef1742940a5c
# Parent  f749aaad0d2f52e3da2a8e80f44b798c0dd85f00
obsolete: clarify users in markerusers() never contain None

diff --git a/mercurial/obsutil.py b/mercurial/obsutil.py
--- a/mercurial/obsutil.py
+++ b/mercurial/obsutil.py
@@ -822,7 +822,7 @@ def markersusers(markers):
 """ Returns a sorted list of markers users without duplicates
 """
 markersmeta = [dict(m[3]) for m in markers]
-users = set(meta.get('user') for meta in markersmeta if meta.get('user'))
+users = set(meta['user'] for meta in markersmeta if meta.get('user'))
 
 return sorted(users)
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 1 of 2] statprof: cleanup string construction

2018-07-16 Thread Yuya Nishihara

On Mon, 16 Jul 2018 10:59:57 +0200, Boris Feld wrote:
> # HG changeset patch
> # User Boris Feld 
> # Date 1527845766 -7200
> #  Fri Jun 01 11:36:06 2018 +0200
> # Node ID eabea66333aa934d8a8f0fc68633a72623a0864b
> # Parent  3700564c63fee7b26ca948e75dee2ea631b1dd4e
> # EXP-Topic cleanupstat
> # Available At https://bitbucket.org/octobus/mercurial-devel/
> #  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
> eabea66333aa
> statprof: cleanup string construction
> 
> Use string substitutions instead of string additions.
> 
> diff --git a/mercurial/statprof.py b/mercurial/statprof.py
> --- a/mercurial/statprof.py
> +++ b/mercurial/statprof.py
> @@ -356,7 +356,7 @@ def save_data(path):
>  stack = sample.stack
>  sites = ['\1'.join([s.path, str(s.lineno), s.function])
>   for s in stack]
> -file.write(time + '\0' + '\0'.join(sites) + '\n')
> +file.write("%s\05%s\n" % (time, '\0'.join(sites)))

\05 ?
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 2 of 2] statprof: small if cleanup

2018-07-16 Thread Yuya Nishihara

On Mon, 16 Jul 2018 10:59:58 +0200, Boris Feld wrote:
> # HG changeset patch
> # User Boris Feld 
> # Date 1527847834 -7200
> #  Fri Jun 01 12:10:34 2018 +0200
> # Node ID e97a69d13f38cbb4389fdc8a7555eab0753c5af9
> # Parent  eabea66333aa934d8a8f0fc68633a72623a0864b
> # EXP-Topic cleanupstat
> # Available At https://bitbucket.org/octobus/mercurial-devel/
> #  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
> e97a69d13f38
> statprof: small if cleanup

Queued this, thanks.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] perf: document the perfmanifest command

2018-07-16 Thread Yuya Nishihara

On Mon, 16 Jul 2018 11:08:00 +0200, Boris Feld wrote:
> # HG changeset patch
> # User Boris Feld 
> # Date 1531298775 -7200
> #  Wed Jul 11 10:46:15 2018 +0200
> # Node ID 24d42facbae0722990b7088d25e21dbe4cba4b92
> # Parent  3700564c63fee7b26ca948e75dee2ea631b1dd4e
> # EXP-Topic doc-perf
> # Available At https://bitbucket.org/octobus/mercurial-devel/
> #  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
> 24d42facbae0
> perf: document the perfmanifest command

Queued, thanks.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH] perf: document the perfmanifest command

2018-07-16 Thread Boris Feld

# HG changeset patch
# User Boris Feld 
# Date 1531298775 -7200
#  Wed Jul 11 10:46:15 2018 +0200
# Node ID 24d42facbae0722990b7088d25e21dbe4cba4b92
# Parent  3700564c63fee7b26ca948e75dee2ea631b1dd4e
# EXP-Topic doc-perf
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
24d42facbae0
perf: document the perfmanifest command

diff --git a/contrib/perf.py b/contrib/perf.py
--- a/contrib/perf.py
+++ b/contrib/perf.py
@@ -774,6 +774,10 @@ def perfphases(ui, repo, **opts):
 
 @command('perfmanifest', [], 'REV')
 def perfmanifest(ui, repo, rev, **opts):
+"""benchmark the time to read a manifest from disk and return a usable
+dict-like object
+
+Manifest caches are cleared before retrieval."""
 timer, fm = gettimer(ui, opts)
 ctx = scmutil.revsingle(repo, rev, rev)
 t = ctx.manifestnode()
diff --git a/tests/test-contrib-perf.t b/tests/test-contrib-perf.t
--- a/tests/test-contrib-perf.t
+++ b/tests/test-contrib-perf.t
@@ -88,7 +88,8 @@ perfstatus
perflookup(no help text available)
perflrucachedict
  (no help text available)
-   perfmanifest  (no help text available)
+   perfmanifest  benchmark the time to read a manifest from disk and return a
+ usable
perfmergecalculate
  (no help text available)
perfmoonwalk  benchmark walking the changelog backwards
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 2] statprof: small if cleanup

2018-07-16 Thread Boris Feld

# HG changeset patch
# User Boris Feld 
# Date 1527847834 -7200
#  Fri Jun 01 12:10:34 2018 +0200
# Node ID e97a69d13f38cbb4389fdc8a7555eab0753c5af9
# Parent  eabea66333aa934d8a8f0fc68633a72623a0864b
# EXP-Topic cleanupstat
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
e97a69d13f38
statprof: small if cleanup

Explicitly testing for None to avoid comparison bugs.

diff --git a/mercurial/statprof.py b/mercurial/statprof.py
--- a/mercurial/statprof.py
+++ b/mercurial/statprof.py
@@ -454,7 +454,8 @@ class DisplayFormats:
 
 def display(fp=None, format=3, data=None, **kwargs):
 '''Print statistics, either to stdout or the given file object.'''
-data = data or state
+if data is None:
+data = state
 
 if fp is None:
 import sys
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH] perf: add a 'perf.all-timing' option to display more than best time

2018-07-16 Thread Boris Feld

# HG changeset patch
# User Boris Feld 
# Date 1529661762 -3600
#  Fri Jun 22 11:02:42 2018 +0100
# Node ID b1591371c449492acb95e2fe071340ecc673e3bc
# Parent  3700564c63fee7b26ca948e75dee2ea631b1dd4e
# EXP-Topic perf-details
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
b1591371c449
perf: add a 'perf.all-timing' option to display more than best time

Minimal time is a useful information, but it is useful to have a wider view on
the performance picture.

diff --git a/contrib/perf.py b/contrib/perf.py
--- a/contrib/perf.py
+++ b/contrib/perf.py
@@ -178,6 +178,9 @@ try:
 configitem('perf', 'parentscount',
 default=mercurial.configitems.dynamicdefault,
 )
+configitem('perf', 'all-timing',
+default=mercurial.configitems.dynamicdefault,
+)
 except (ImportError, AttributeError):
 pass
 
@@ -247,12 +250,15 @@ def gettimer(ui, opts=None):
 # experimental config: perf.stub
 if ui.configbool("perf", "stub", False):
 return functools.partial(stub_timer, fm), fm
-return functools.partial(_timer, fm), fm
+
+# experimental config: perf.all-timing
+displayall = ui.configbool("perf", "all-timing", False)
+return functools.partial(_timer, fm, displayall=displayall), fm
 
 def stub_timer(fm, func, title=None):
 func()
 
-def _timer(fm, func, title=None):
+def _timer(fm, func, title=None, displayall=False):
 gc.collect()
 results = []
 begin = util.timer()
@@ -277,14 +283,27 @@ def _timer(fm, func, title=None):
 fm.write('title', '! %s\n', title)
 if r:
 fm.write('result', '! result: %s\n', r)
-m = min(results)
-fm.plain('!')
-fm.write('wall', ' wall %f', m[0])
-fm.write('comb', ' comb %f', m[1] + m[2])
-fm.write('user', ' user %f', m[1])
-fm.write('sys',  ' sys %f', m[2])
-fm.write('count',  ' (best of %d)', count)
-fm.plain('\n')
+def display(role, entry):
+prefix = ''
+if role != 'best':
+prefix = '%s.' % role
+fm.plain('!')
+fm.write(prefix + 'wall', ' wall %f', entry[0])
+fm.write(prefix + 'comb', ' comb %f', entry[1] + entry[2])
+fm.write(prefix + 'user', ' user %f', entry[1])
+fm.write(prefix + 'sys',  ' sys %f', entry[2])
+fm.write(prefix + 'count',  ' (%s of %d)', role, count)
+fm.plain('\n')
+results.sort()
+min_val = results[0]
+display('best', min_val)
+if displayall:
+max_val = results[-1]
+display('max', max_val)
+avg = tuple([sum(x) / count for x in zip(*results)])
+display('avg', avg)
+median = results[len(results)//2]
+display('median', median)
 
 # utilities for historical portability
 
diff --git a/tests/test-contrib-perf.t b/tests/test-contrib-perf.t
--- a/tests/test-contrib-perf.t
+++ b/tests/test-contrib-perf.t
@@ -175,7 +175,24 @@ perfstatus
   $ hg perfwalk
   $ hg perfparents
 
+test actual output
+--
+
+normal output:
+
+  $ hg perfheads --config perf.stub=no
+  ! wall * comb * user * sys * (best of *) (glob)
+
+detailed output:
+
+  $ hg perfheads --config perf.all-timing=yes --config perf.stub=no
+  ! wall * comb * user * sys * (best of *) (glob)
+  ! wall * comb * user * sys * (max of *) (glob)
+  ! wall * comb * user * sys * (avg of *) (glob)
+  ! wall * comb * user * sys * (median of *) (glob)
+
 Check perf.py for historical portability
+
 
   $ cd "$TESTDIR/.."
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 1 of 2] perffncachewrite: load fncache after lock is acquired

2018-07-16 Thread Boris Feld

# HG changeset patch
# User Boris Feld 
# Date 1531526987 -7200
#  Sat Jul 14 02:09:47 2018 +0200
# Node ID fa319c92bed5fc495d62122697ee32a8fe2be8c4
# Parent  cc76692f401d0063cd5dfaf46f0fca1563a85fae
# EXP-Topic perffncachewrite
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
fa319c92bed5
perffncachewrite: load fncache after lock is acquired

Without this patch, running perffncachewrite on a repository destroy its
fncache.

Lock Acquisition drops various caches, including the fncache one. Then writing
of an non-loaded fncache result into an empty one.

diff --git a/contrib/perf.py b/contrib/perf.py
--- a/contrib/perf.py
+++ b/contrib/perf.py
@@ -957,9 +957,10 @@ def perffncacheload(ui, repo, **opts):
 def perffncachewrite(ui, repo, **opts):
 timer, fm = gettimer(ui, opts)
 s = repo.store
+lock = repo.lock()
 s.fncache._load()
-lock = repo.lock()
 tr = repo.transaction('perffncachewrite')
+tr.addbackup('fncache')
 def d():
 s.fncache._dirty = True
 s.fncache.write(tr)
diff --git a/tests/test-contrib-perf.t b/tests/test-contrib-perf.t
--- a/tests/test-contrib-perf.t
+++ b/tests/test-contrib-perf.t
@@ -145,7 +145,11 @@ perfstatus
 #if repofncache
   $ hg perffncacheencode
   $ hg perffncacheload
+  $ hg debugrebuildfncache
+  fncache already up to date
   $ hg perffncachewrite
+  $ hg debugrebuildfncache
+  fncache already up to date
 #endif
   $ hg perfheads
   $ hg perfindex
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 9 of 9] fileset: remove fullmatchctx class

2018-07-16 Thread Martin von Zweigbergk via Mercurial-devel

On Tue, Jul 10, 2018 at 8:08 AM Yuya Nishihara  wrote:

> # HG changeset patch
> # User Yuya Nishihara 
> # Date 1528551509 -32400
> #  Sat Jun 09 22:38:29 2018 +0900
> # Node ID 89e05df138ac4e056bfd2c748b2f4554df1359b3
> # Parent  87bb964e745e47fc897f56864b66e9a8e986301d
> fileset: remove fullmatchctx class
>

Nice! Queued, thanks.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 2 of 2] phases: micro-optimize newheads() to not create context objects

2018-07-16 Thread Martin von Zweigbergk via Mercurial-devel

On Fri, Jul 13, 2018 at 10:43 PM Yuya Nishihara  wrote:

> # HG changeset patch
> # User Yuya Nishihara 
> # Date 1531542109 -32400
> #  Sat Jul 14 13:21:49 2018 +0900
> # Node ID 503d14253b22603b1f76c80dd6b89b6ebcfe5e06
> # Parent  c8f181c48ae26247478aea82c8d2ab2f886831f9
> phases: micro-optimize newheads() to not create context objects

Queued these, thanks.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

54 matches

Mail list logo