Re: [PATCH] py3: update test output

[Pulkit Goyal]

On Mon, Oct 17, 2016 at 4:14 AM, Gregory Szorc  wrote:
> On Sun, Oct 16, 2016 at 3:03 PM, Pulkit Goyal <7895pul...@gmail.com> wrote:
>>
>> # HG changeset patch
>> # User Pulkit Goyal <7895pul...@gmail.com>
>> # Date 1476557196 -19800
>> #  Sun Oct 16 00:16:36 2016 +0530
>> # Node ID d49781829eafd0ee4917c7792aaa8987170ffe78
>> # Parent  c1134c39ff3ad961af17a4130623f87e0a42d392
>> py3: update test output
>>
>> A lot of patches have been pushed related to porting. This patch updates
>> both
>> our py3 tests.
>>
>> diff -r c1134c39ff3a -r d49781829eaf tests/test-check-py3-commands.t
>> --- a/tests/test-check-py3-commands.t   Sun Oct 16 10:38:52 2016 -0700
>> +++ b/tests/test-check-py3-commands.t   Sun Oct 16 00:16:36 2016 +0530
>> @@ -9,6 +9,6 @@
>>>   $PYTHON3 `which hg` $cmd 2>&1 2>&1 | tail -1
>>> done
>>version
>> -  TypeError: str expected, not bytes
>> +  ImportError:
>> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
>> symbol: Py_InitModule4_64
>>debuginstall
>> -  TypeError: str expected, not bytes
>> +  ImportError:
>> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
>> symbol: Py_InitModule4_64
>> diff -r c1134c39ff3a -r d49781829eaf tests/test-check-py3-compat.t
>> --- a/tests/test-check-py3-compat.t Sun Oct 16 10:38:52 2016 -0700
>> +++ b/tests/test-check-py3-compat.t Sun Oct 16 00:16:36 2016 +0530
>> @@ -16,14 +16,164 @@
>>$ hg files 'set:(**.py) - grep(pygments)' | sed 's|\\|/|g' \
>>> | xargs $PYTHON3 contrib/check-py3-compat.py \
>>> | sed 's/[0-9][0-9]*)$/*)/'
>> -  hgext/convert/transport.py: error importing:  No module
>> named 'svn.client' (error at transport.py:*)
>> -  hgext/fsmonitor/pywatchman/capabilities.py: error importing:
>>  No module named 'pybser' (error at __init__.py:*)
>> -  hgext/fsmonitor/pywatchman/pybser.py: error importing:  No
>> module named 'pybser' (error at __init__.py:*)
>> -  hgext/fsmonitor/watchmanclient.py: error importing:  No
>> module named 'pybser' (error at __init__.py:*)
>> -  hgext/mq.py: error importing:  __import__() argument 1 must
>> be str, not bytes (error at extensions.py:*)
>> -  mercurial/scmwindows.py: error importing:  No module named
>> 'winreg' (error at scmwindows.py:*)
>> +  hgext/acl.py: error importing: 
>> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
>> symbol: Py_InitModule4_64 (error at util.py:*)
>> +  hgext/automv.py: error importing: 
>> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
>> symbol: Py_InitModule4_64 (error at util.py:*)
>> +  hgext/blackbox.py: error importing: 
>> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
>> symbol: Py_InitModule4_64 (error at util.py:*)
>> +  hgext/bugzilla.py: error importing: 
>> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
>> symbol: Py_InitModule4_64 (error at util.py:*)
>> +  hgext/censor.py: error importing: 
>> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
>> symbol: Py_InitModule4_64 (error at util.py:*)
>> +  hgext/chgserver.py: error importing: 
>> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
>> symbol: Py_InitModule4_64 (error at util.py:*)
>> +  hgext/children.py: error importing: 
>> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
>> symbol: Py_InitModule4_64 (error at util.py:*)
>> +  hgext/churn.py: error importing: 
>> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
>> symbol: Py_InitModule4_64 (error at util.py:*)
>> +  hgext/clonebundles.py: error importing: 
>> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
>> symbol: Py_InitModule4_64 (error at util.py:*)
>> +  hgext/color.py: error importing: 
>> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
>> symbol: Py_InitModule4_64 (error at util.py:*)
>> +  hgext/convert/bzr.py: error importing: 
>> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
>> symbol: Py_InitModule4_64 (error at util.py:*)
>> +  hgext/convert/common.py: error importing: 
>> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
>> symbol: Py_InitModule4_64 (error at util.py:*)
>> +  hgext/convert/convcmd.py: error importing: 
>> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
>> symbol: Py_InitModule4_64 (error at util.py:*)
>> +  hgext/convert/cvs.py: error importing: 
>> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
>> symbol: Py_InitModule4_64 (error at util.py:*)
>> +  hgext/convert/cvsps.py: error importing: 
>> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
>> symbol: Py_InitModule4_64 (error at util.py:*)
>> +  hgext/convert/darcs.py: error importing: 
>> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
>> symbol: Py_InitModule4_64 (error at util.py:*)
>> +  hgext/convert/filemap.py: error 

Re: [PATCH] py3: update test output

[Gregory Szorc]

On Sun, Oct 16, 2016 at 3:03 PM, Pulkit Goyal <7895pul...@gmail.com> wrote:

> # HG changeset patch
> # User Pulkit Goyal <7895pul...@gmail.com>
> # Date 1476557196 -19800
> #  Sun Oct 16 00:16:36 2016 +0530
> # Node ID d49781829eafd0ee4917c7792aaa8987170ffe78
> # Parent  c1134c39ff3ad961af17a4130623f87e0a42d392
> py3: update test output
>
> A lot of patches have been pushed related to porting. This patch updates
> both
> our py3 tests.
>
> diff -r c1134c39ff3a -r d49781829eaf tests/test-check-py3-commands.t
> --- a/tests/test-check-py3-commands.t   Sun Oct 16 10:38:52 2016 -0700
> +++ b/tests/test-check-py3-commands.t   Sun Oct 16 00:16:36 2016 +0530
> @@ -9,6 +9,6 @@
>>   $PYTHON3 `which hg` $cmd 2>&1 2>&1 | tail -1
>> done
>version
> -  TypeError: str expected, not bytes
> +  ImportError: /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so:
> undefined symbol: Py_InitModule4_64
>debuginstall
> -  TypeError: str expected, not bytes
> +  ImportError: /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so:
> undefined symbol: Py_InitModule4_64
> diff -r c1134c39ff3a -r d49781829eaf tests/test-check-py3-compat.t
> --- a/tests/test-check-py3-compat.t Sun Oct 16 10:38:52 2016 -0700
> +++ b/tests/test-check-py3-compat.t Sun Oct 16 00:16:36 2016 +0530
> @@ -16,14 +16,164 @@
>$ hg files 'set:(**.py) - grep(pygments)' | sed 's|\\|/|g' \
>> | xargs $PYTHON3 contrib/check-py3-compat.py \
>> | sed 's/[0-9][0-9]*)$/*)/'
> -  hgext/convert/transport.py: error importing:  No module
> named 'svn.client' (error at transport.py:*)
> -  hgext/fsmonitor/pywatchman/capabilities.py: error importing:
>  No module named 'pybser' (error at __init__.py:*)
> -  hgext/fsmonitor/pywatchman/pybser.py: error importing: 
> No module named 'pybser' (error at __init__.py:*)
> -  hgext/fsmonitor/watchmanclient.py: error importing:  No
> module named 'pybser' (error at __init__.py:*)
> -  hgext/mq.py: error importing:  __import__() argument 1 must
> be str, not bytes (error at extensions.py:*)
> -  mercurial/scmwindows.py: error importing:  No module named
> 'winreg' (error at scmwindows.py:*)
> +  hgext/acl.py: error importing: 
> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
> symbol: Py_InitModule4_64 (error at util.py:*)
> +  hgext/automv.py: error importing: 
> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
> symbol: Py_InitModule4_64 (error at util.py:*)
> +  hgext/blackbox.py: error importing: 
> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
> symbol: Py_InitModule4_64 (error at util.py:*)
> +  hgext/bugzilla.py: error importing: 
> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
> symbol: Py_InitModule4_64 (error at util.py:*)
> +  hgext/censor.py: error importing: 
> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
> symbol: Py_InitModule4_64 (error at util.py:*)
> +  hgext/chgserver.py: error importing: 
> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
> symbol: Py_InitModule4_64 (error at util.py:*)
> +  hgext/children.py: error importing: 
> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
> symbol: Py_InitModule4_64 (error at util.py:*)
> +  hgext/churn.py: error importing: 
> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
> symbol: Py_InitModule4_64 (error at util.py:*)
> +  hgext/clonebundles.py: error importing: 
> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
> symbol: Py_InitModule4_64 (error at util.py:*)
> +  hgext/color.py: error importing: 
> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
> symbol: Py_InitModule4_64 (error at util.py:*)
> +  hgext/convert/bzr.py: error importing: 
> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
> symbol: Py_InitModule4_64 (error at util.py:*)
> +  hgext/convert/common.py: error importing: 
> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
> symbol: Py_InitModule4_64 (error at util.py:*)
> +  hgext/convert/convcmd.py: error importing: 
> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
> symbol: Py_InitModule4_64 (error at util.py:*)
> +  hgext/convert/cvs.py: error importing: 
> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
> symbol: Py_InitModule4_64 (error at util.py:*)
> +  hgext/convert/cvsps.py: error importing: 
> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
> symbol: Py_InitModule4_64 (error at util.py:*)
> +  hgext/convert/darcs.py: error importing: 
> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
> symbol: Py_InitModule4_64 (error at util.py:*)
> +  hgext/convert/filemap.py: error importing: 
> /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined
> symbol: Py_InitModule4_64 (error at util.py:*)
> +  hgext/convert/git.py: error importing: 
> 

[PATCH] py3: update test output

[Pulkit Goyal]

# HG changeset patch
# User Pulkit Goyal <7895pul...@gmail.com>
# Date 1476557196 -19800
#  Sun Oct 16 00:16:36 2016 +0530
# Node ID d49781829eafd0ee4917c7792aaa8987170ffe78
# Parent  c1134c39ff3ad961af17a4130623f87e0a42d392
py3: update test output

A lot of patches have been pushed related to porting. This patch updates both
our py3 tests.

diff -r c1134c39ff3a -r d49781829eaf tests/test-check-py3-commands.t
--- a/tests/test-check-py3-commands.t   Sun Oct 16 10:38:52 2016 -0700
+++ b/tests/test-check-py3-commands.t   Sun Oct 16 00:16:36 2016 +0530
@@ -9,6 +9,6 @@
   >   $PYTHON3 `which hg` $cmd 2>&1 2>&1 | tail -1
   > done
   version
-  TypeError: str expected, not bytes
+  ImportError: /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: 
undefined symbol: Py_InitModule4_64
   debuginstall
-  TypeError: str expected, not bytes
+  ImportError: /tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: 
undefined symbol: Py_InitModule4_64
diff -r c1134c39ff3a -r d49781829eaf tests/test-check-py3-compat.t
--- a/tests/test-check-py3-compat.t Sun Oct 16 10:38:52 2016 -0700
+++ b/tests/test-check-py3-compat.t Sun Oct 16 00:16:36 2016 +0530
@@ -16,14 +16,164 @@
   $ hg files 'set:(**.py) - grep(pygments)' | sed 's|\\|/|g' \
   > | xargs $PYTHON3 contrib/check-py3-compat.py \
   > | sed 's/[0-9][0-9]*)$/*)/'
-  hgext/convert/transport.py: error importing:  No module named 
'svn.client' (error at transport.py:*)
-  hgext/fsmonitor/pywatchman/capabilities.py: error importing:  
No module named 'pybser' (error at __init__.py:*)
-  hgext/fsmonitor/pywatchman/pybser.py: error importing:  No 
module named 'pybser' (error at __init__.py:*)
-  hgext/fsmonitor/watchmanclient.py: error importing:  No module 
named 'pybser' (error at __init__.py:*)
-  hgext/mq.py: error importing:  __import__() argument 1 must be 
str, not bytes (error at extensions.py:*)
-  mercurial/scmwindows.py: error importing:  No module named 
'winreg' (error at scmwindows.py:*)
+  hgext/acl.py: error importing:  
/tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined symbol: 
Py_InitModule4_64 (error at util.py:*)
+  hgext/automv.py: error importing:  
/tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined symbol: 
Py_InitModule4_64 (error at util.py:*)
+  hgext/blackbox.py: error importing:  
/tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined symbol: 
Py_InitModule4_64 (error at util.py:*)
+  hgext/bugzilla.py: error importing:  
/tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined symbol: 
Py_InitModule4_64 (error at util.py:*)
+  hgext/censor.py: error importing:  
/tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined symbol: 
Py_InitModule4_64 (error at util.py:*)
+  hgext/chgserver.py: error importing:  
/tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined symbol: 
Py_InitModule4_64 (error at util.py:*)
+  hgext/children.py: error importing:  
/tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined symbol: 
Py_InitModule4_64 (error at util.py:*)
+  hgext/churn.py: error importing:  
/tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined symbol: 
Py_InitModule4_64 (error at util.py:*)
+  hgext/clonebundles.py: error importing:  
/tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined symbol: 
Py_InitModule4_64 (error at util.py:*)
+  hgext/color.py: error importing:  
/tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined symbol: 
Py_InitModule4_64 (error at util.py:*)
+  hgext/convert/bzr.py: error importing:  
/tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined symbol: 
Py_InitModule4_64 (error at util.py:*)
+  hgext/convert/common.py: error importing:  
/tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined symbol: 
Py_InitModule4_64 (error at util.py:*)
+  hgext/convert/convcmd.py: error importing:  
/tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined symbol: 
Py_InitModule4_64 (error at util.py:*)
+  hgext/convert/cvs.py: error importing:  
/tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined symbol: 
Py_InitModule4_64 (error at util.py:*)
+  hgext/convert/cvsps.py: error importing:  
/tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined symbol: 
Py_InitModule4_64 (error at util.py:*)
+  hgext/convert/darcs.py: error importing:  
/tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined symbol: 
Py_InitModule4_64 (error at util.py:*)
+  hgext/convert/filemap.py: error importing:  
/tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined symbol: 
Py_InitModule4_64 (error at util.py:*)
+  hgext/convert/git.py: error importing:  
/tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: undefined symbol: 
Py_InitModule4_64 (error at util.py:*)
+  hgext/convert/gnuarch.py: error importing:  
/tmp/hgtests.fqjMQo/install/lib/python/mercurial/osutil.so: 

Re: [PATCH 2 of 2 V3] py3: use namedtuple._replace to produce new tokens

[Martijn Pieters]

On 16 Oct 2016, at 15:30, Pierre-Yves David  
wrote:
> 
>> py3: use namedtuple._replace to produce new tokens
> 
> We seems to be using a private function of some stdlib type?
> Can you elaborate on why this is a good move?

It is not private. This is one of the exceptions to the Python style guide; the 
namedtuple methods all have a single leading underscore to avoid clashing with 
the field names. If they didn't, you wouldn't be able to name a field `replace`.

See the 
https://docs.python.org/3/library/collections.html#collections.namedtuple 
documentation:

> In addition to the methods inherited from tuples, named tuples support three 
> additional methods and two attributes. To prevent conflicts with field names, 
> the method and attribute names start with an underscore.

and 
https://docs.python.org/3/library/collections.html#collections.somenamedtuple._replace
 for the specific method:

> Return a new instance of the named tuple replacing specified fields with new 
> values

--
Martijn


signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 3 of 4] perf: cache revlog file handle during perfrevlog

[Gregory Szorc]

# HG changeset patch
# User Gregory Szorc 
# Date 1476649617 25200
#  Sun Oct 16 13:26:57 2016 -0700
# Node ID 9e2f957b05ac5c76595280a6084ba01d7b369a05
# Parent  ee7144d686de700d60cb2a78adac3aab43f2a71d
perf: cache revlog file handle during perfrevlog

The aim of perfrevlog is to measure how fast revlog revisions
can be resolved, not I/O. Since we now have a mechanism to cache
an open file handle on a revlog instance, let's use it.

diff --git a/contrib/perf.py b/contrib/perf.py
--- a/contrib/perf.py
+++ b/contrib/perf.py
@@ -788,18 +788,19 @@ def perfrevlog(ui, repo, file_=None, sta
 startrev = 0
 endrev = _len(r)
 dist = opts['dist']
 
 if reverse:
 startrev, endrev = endrev, startrev
 dist = -1 * dist
 
-for x in xrange(startrev, endrev, dist):
-r.revision(r.node(x))
+with r.cachefilehandle():
+for x in xrange(startrev, endrev, dist):
+r.revision(r.node(x))
 
 timer(d)
 fm.end()
 
 @command('perfrevlogrevision', revlogopts + formatteropts +
  [('', 'cache', False, 'use caches instead of clearing')],
  '-c|-m|FILE REV')
 def perfrevlogrevision(ui, repo, file_, rev=None, cache=None, **opts):
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 4] changegroup: reuse revlog file handle when generating group

[Gregory Szorc]

# HG changeset patch
# User Gregory Szorc 
# Date 1476648175 25200
#  Sun Oct 16 13:02:55 2016 -0700
# Node ID ee7144d686de700d60cb2a78adac3aab43f2a71d
# Parent  735adf139447190fe52c65ec6e9b4e5e0c81d6aa
changegroup: reuse revlog file handle when generating group

Previously, every time we needed new data from a revlog during
changegroup generation, a new file handle would be opened,
seeked, read, and closed.

After this patch, we use the just-added context manager on the
revlog class to cache and reuse a file handle for the duration
of the changegroup operation.

When generating a v2 bundle on the mozilla-unified repo, this change
causes the following changes to system call counts:

FunctionBefore After Delta
read576,062   576,085   +23
open274,939   269,167-5,772
fstat   808,762   797,216   -11,546
write   952,449   952,473   +24
lseek   536,450   536,452+2
close   272,314   266,542-5,772
lstat20,18520,185 0

It's worth noting that this repo has 265,773 revlog files (.i + .d)
but only 941 of them are non-inline. That means of the non-inline
revlogs, we're preventing multiple redundant opens per revlog on
average. (Most of the prevented opens are on changelog and manifest.)

On my Linux machine, this change appears to show no improvement in
wall time. However, fewer system calls is fewer system calls. And
when I/O is involved, I think aiming for 0 system calls is worthwhile.

diff --git a/mercurial/changegroup.py b/mercurial/changegroup.py
--- a/mercurial/changegroup.py
+++ b/mercurial/changegroup.py
@@ -560,23 +560,25 @@ class cg1packer(object):
 
 # add the parent of the first rev
 p = revlog.parentrevs(revs[0])[0]
 revs.insert(0, p)
 
 # build deltas
 total = len(revs) - 1
 msgbundling = _('bundling')
-for r in xrange(len(revs) - 1):
-if units is not None:
-self._progress(msgbundling, r + 1, unit=units, total=total)
-prev, curr = revs[r], revs[r + 1]
-linknode = lookup(revlog.node(curr))
-for c in self.revchunk(revlog, curr, prev, linknode):
-yield c
+
+with revlog.cachefilehandle():
+for r in xrange(len(revs) - 1):
+if units is not None:
+self._progress(msgbundling, r + 1, unit=units, total=total)
+prev, curr = revs[r], revs[r + 1]
+linknode = lookup(revlog.node(curr))
+for c in self.revchunk(revlog, curr, prev, linknode):
+yield c
 
 if units is not None:
 self._progress(msgbundling, None)
 yield self.close()
 
 # filter any nodes that claim to be part of the known set
 def prune(self, revlog, missing, commonrevs):
 rr, rl = revlog.rev, revlog.linkrev
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 4 of 4] changegroup: increase write buffer size to 128k

[Gregory Szorc]

# HG changeset patch
# User Gregory Szorc 
# Date 1476650123 25200
#  Sun Oct 16 13:35:23 2016 -0700
# Node ID 4582f12754622ae049afefee05176ef107d99a7e
# Parent  9e2f957b05ac5c76595280a6084ba01d7b369a05
changegroup: increase write buffer size to 128k

By default, Python defers to the operating system for choosing the
default buffer size on opened files. On my Linux machine, the default
is 4k, which is really small for 2016.

This patch bumps the write buffer size when writing
changegroups/bundles to 128k. This matches the 128k read buffer
we already use on revlogs.

It's worth noting that this only impacts when writing to an explicit
file (such as during `hg bundle`). Buffers when writing to bundle
files via the repo vfs or to a temporary file are not impacted.

When producing a none-v2 bundle file of the mozilla-unified repository,
this change caused the number of write() system calls to drop from
952,449 to 29,788. After this change, the most frequent system
calls are fstat(), read(), lseek(), and open(). There were
2,523,672 system calls after this patch (so a net decrease of
~950k is statistically significant).

This change shows no performance change on my system. But I have a
high-end system with a fast SSD. It is quite possible this change
will have a significant impact on network file systems, where
extra network round trips due to excessive I/O system calls could
introduce significant latency.

diff --git a/mercurial/changegroup.py b/mercurial/changegroup.py
--- a/mercurial/changegroup.py
+++ b/mercurial/changegroup.py
@@ -88,17 +88,19 @@ def writechunks(ui, chunks, filename, vf
 """
 fh = None
 cleanup = None
 try:
 if filename:
 if vfs:
 fh = vfs.open(filename, "wb")
 else:
-fh = open(filename, "wb")
+# Increase default buffer size because default is usually
+# small (4k is common on Linux).
+fh = open(filename, "wb", 131072)
 else:
 fd, filename = tempfile.mkstemp(prefix="hg-bundle-", suffix=".hg")
 fh = os.fdopen(fd, "wb")
 cleanup = filename
 for c in chunks:
 fh.write(c)
 cleanup = None
 return filename
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 1 of 4] revlog: add a context manager to allow file handle reuse

[Gregory Szorc]

# HG changeset patch
# User Gregory Szorc 
# Date 1476649322 25200
#  Sun Oct 16 13:22:02 2016 -0700
# Node ID 735adf139447190fe52c65ec6e9b4e5e0c81d6aa
# Parent  757d25d2bbe63fc560e92b6bb25fbfbf09b09342
revlog: add a context manager to allow file handle reuse

Currently, read-only operations traversing revlogs must open a new
file handle whenever they need uncached data from the underlying
revlog. This can add overhead to operations such as changegroup
generation.

The revlog APIs have a mechanism for reusing a file descriptor
for I/O. This was added by me a while ago as a means to speed up
revlog.addgroup(). At that time, I didn't do any work around
reusing file handles for read operations.

This patch introduces a context manager to cache an open file handle
on a revlog. When the context manager is active, revlog reads will
be routed to the opened file handle instead of opening a single
use file handle.

There is definitely room to improve the API. We could probably
even refactor the write file descriptor caching to use a context
manager - possibly the same one! However, this is a bit of work
since the write file handle can be swapped out if a revlog
transitions from inline to non-inline in the course of adding
revisions.

diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -9,16 +9,17 @@
 
 This provides efficient delta storage with O(1) retrieve and append
 and O(changes) merge between branches.
 """
 
 from __future__ import absolute_import
 
 import collections
+import contextlib
 import errno
 import hashlib
 import os
 import struct
 import zlib
 
 # import stuff from node for others to import from revlog
 from .node import (
@@ -309,16 +310,18 @@ class revlog(object):
 self.index, nodemap, self._chunkcache = d
 if nodemap is not None:
 self.nodemap = self._nodecache = nodemap
 if not self._chunkcache:
 self._chunkclear()
 # revnum -> (chain-length, sum-delta-length)
 self._chaininfocache = {}
 
+self._readfh = None
+
 def tip(self):
 return self.node(len(self.index) - 2)
 def __contains__(self, rev):
 return 0 <= rev < len(self)
 def __len__(self):
 return len(self.index) - 1
 def __iter__(self):
 return iter(xrange(len(self)))
@@ -1013,27 +1016,57 @@ class revlog(object):
 """
 o, d = self._chunkcache
 # try to add to existing cache
 if o + len(d) == offset and len(d) + len(data) < _chunksize:
 self._chunkcache = o, d + data
 else:
 self._chunkcache = offset, data
 
+@contextlib.contextmanager
+def cachefilehandle(self):
+"""Maintain a persistent file handle during operations.
+
+When this context manager is active, a file descriptor will be reused
+for all read operations, ensuring the underlying revlog file isn't
+reopened multiple times.
+"""
+if self._readfh:
+raise error.Abort('cachefilehandle already active')
+
+# Inline revlogs have data chunks cached at open time. If we opened
+# a file handle it wouldn't be read.
+if self._inline:
+yield
+return
+
+try:
+self._readfh = self.opener(self.datafile)
+yield
+finally:
+self._readfh.close()
+self._readfh = None
+
 def _loadchunk(self, offset, length, df=None):
 """Load a segment of raw data from the revlog.
 
 Accepts an absolute offset, length to read, and an optional existing
 file handle to read from.
 
 If an existing file handle is passed, it will be seeked and the
 original seek position will NOT be restored.
 
 Returns a str or buffer of raw byte data.
 """
+# Use cached file handle from context manager automatically.
+# self._readfh is always opened in read-only mode. df may be opened
+# in append mode. The context manager should only be active during
+# read operations. So this mismatch is OK.
+df = df or self._readfh
+
 if df is not None:
 closehandle = False
 else:
 if self._inline:
 df = self.opener(self.indexfile)
 else:
 df = self.opener(self.datafile)
 closehandle = True
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: history at diff blocks level

[Jun Wu]

Excerpts from Denis Laxalde's message of 2016-10-03 16:38:17 +0200:
> Hi all,
> 
> I've been recently thinking about adding some support in Mercurial to
> query repository history based on changes within a given line range in a
> file. I think that would be useful in at least two commands:
> 
> * log, to restrict history to revisions that modify specified part(s) of
> file(s) and only display the diff around specified line range and,
> 
> * annotate, to window the annotate result and maybe consider walking
> file revision from tip to base to reduce computation costs.
> 
> (The "log" part is more interesting, I think.)

I've been thinking about this as well for "fastannotate --deleted". Although
"annotate" is generally easier than "log" in this case: slicing the
annotated lines seems to be enough.

>  From UI point of view, the basic idea is to specify a (file name, line
> range) pair and the simplest solution I could find is something like:
> 
>hg log/annotate --line-range fromline,toline FILE
> 
> but this does not work well with several files. (Perhaps something like
> hg log FILE:fromline:toline would be better.) I also thought about a

+1 for "FILE:fromline:toline". It is intuitive and makes sense. A new
boolean flag (like "--line-ranges") that enables the syntax explicitly
may be necessary. The flag can avoid conflicts with existing matcher syntax,
and make it clear that some commands like "add" do not support line ranges.

> "changes(filename, fromline, toline)" revset (or an extension of the
> existing "modifies()" revset), but it seems to me that this would not
> fit well for both log and annotate. Suggestions welcome.
> 
> 
>  From the technical point of view, my idea is to rely on
> mdiff.allblocks(, ) (used
> in both annotate and log, when --patch option is given) to:
> 
> 1. filter blocks depending on whether they are relevant to specified
> line range (e.g., for the log command there's some "!" block), and,
> 
> 2. track the evolution of the line range across revisions (namely, given
> the line range at rev2, find the line range at rev1 in the example above).
>
> I have something working concerning this "low level" aspect, but I'm
> somehow getting stuck when it comes to plug things into the log command
> call. Namely, I need to pass the "line range" information from one
> revision to another during iterations of the loop on revisions in
> commands.log() [1] and pass this information down to the mdiff.unidiff()
> call [2] which would then give me back another line range to push up to
> the outer loop on revisions. Given the complexity of the call chain, I
> actually think this is not a very good idea... So the best idea I could
> come up with is to filter revisions beforehand (as would a revset do)
> but this would imply keeping track of files' line ranges per revision
> (to avoid processing diff blocks twice when --patch option is specified
> in particular). All in all, it's not clear to me which "tool" I may use
> to achieve this (I thought about using the "filematcher" built along
> with "revs" in commands.log(), but not really sure it's a good idea).
> Maybe I just need a data structure that does not exist yet?
> I'd appreciate any pointer to move forward.

I think "changeset_printer" and "diffordiffstat" are worth considering.
"diffordiffstat" is currently stateless. A possible direction is to add a
new stateful "diffordiffstat" that tracks the line ranges.

If revisions are filtered before-hand, the state could be passed to the new
"diffordiffstat" function to avoid unnecessary calculation.

It seems to me that high level diff function like "mdiff.unidiff" could take
an extra parameter "difffunc" which defaults to "allblocks". Then we can
have a "filteredallblocks" passed to "unidiff".

> 
> (I'll be at the sprint, so this can also be a working topic if some
> people are interested.)
> 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 1 of 2 V2] exchange: refactor APIs to obtain bundle data (API)

[Pierre-Yves David]

On 10/16/2016 08:14 PM, Gregory Szorc wrote:

# HG changeset patch
# User Gregory Szorc 
# Date 1476639532 25200
#  Sun Oct 16 10:38:52 2016 -0700
# Node ID 3f18f7464e651128a5f8d9c9312805adbc22f547
# Parent  757d25d2bbe63fc560e92b6bb25fbfbf09b09342
exchange: refactor APIs to obtain bundle data (API)


That first one is pushed, thanks,

--
Pierre-Yves David
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 2 of 2 v3] evolve: lock the working copy early in next and prev (issue5244)

[Pierre-Yves David]

On 10/16/2016 06:53 PM, Simon Farnsworth wrote:

# HG changeset patch
# User Simon Farnsworth 
# Date 1476636773 25200
#  Sun Oct 16 09:52:53 2016 -0700
# Node ID 778468237ecf195c4eb1d87bc4f7b5d30e538c63
# Parent  ee284d7c5faa5d9f17853f51c17883844b985c58
evolve: lock the working copy early in next and prev (issue5244)

Both next and prev depend on a consistent working copy, but were waiting to
take the lock until they were ready to alter the working copy.

Take the lock before reading the working copy state, and do not release it
until we're definitely not going to change the working copy.


These are pushed on stable, thanks (the fake-editor hack is a bit ugly, 
but lets move forward).



diff --git a/hgext/evolve.py b/hgext/evolve.py
--- a/hgext/evolve.py
+++ b/hgext/evolve.py
@@ -2213,10 +2213,13 @@
 """update to parent revision

 Displays the summary line of the destination for clarity."""
-if True:
+wlock = None
+dryrunopt = opts['dry_run']
+if not dryrunopt:
+wlock = repo.wlock()
+try:
 wkctx = repo[None]
 wparents = wkctx.parents()
-dryrunopt = opts['dry_run']
 if len(wparents) != 1:
 raise error.Abort('merge in progress')
 if not opts['merge']:
@@ -2246,7 +2249,6 @@
 ret = hg.update(repo, p.rev())
 if not ret:
 tr = lock = None
-wlock = repo.wlock()
 try:
 lock = repo.lock()
 tr = repo.transaction('previous')
@@ -2257,7 +2259,8 @@
 bmdeactivate(repo)
 tr.close()
 finally:
-lockmod.release(tr, lock, wlock)
+lockmod.release(tr, lock)
+
 displayer.show(p)
 return 0
 else:
@@ -2265,6 +2268,8 @@
 displayer.show(p)
 ui.warn(_('multiple parents, explicitly update to one\n'))
 return 1
+finally:
+lockmod.release(wlock)

 @command('^next',
  [('B', 'move-bookmark', False,
@@ -2282,10 +2287,13 @@

 Displays the summary line of the destination for clarity.
 """
-if True:
+wlock = None
+dryrunopt = opts['dry_run']
+if not dryrunopt:
+wlock = repo.wlock()
+try:
 wkctx = repo[None]
 wparents = wkctx.parents()
-dryrunopt = opts['dry_run']
 if len(wparents) != 1:
 raise error.Abort('merge in progress')
 if not opts['merge']:
@@ -2315,7 +2323,6 @@
 ret = hg.update(repo, c.rev())
 if not ret:
 lock = tr = None
-wlock = repo.wlock()
 try:
 lock = repo.lock()
 tr = repo.transaction('next')
@@ -2326,7 +2333,7 @@
 bmdeactivate(repo)
 tr.close()
 finally:
-lockmod.release(tr, lock, wlock)
+lockmod.release(tr, lock)
 displayer.show(c)
 result = 0
 elif children:
@@ -2368,6 +2375,8 @@
 return result
 return 1
 return result
+finally:
+lockmod.release(wlock)

 def _reachablefrombookmark(repo, revs, bookmarks):
 """filter revisions and bookmarks reachable from the given bookmark
diff --git a/tests/fake-editor.sh b/tests/fake-editor.sh
new file mode 100755
--- /dev/null
+++ b/tests/fake-editor.sh
@@ -0,0 +1,3 @@
+#!/bin/sh
+sleep 5
+echo "new desc" >> $1
diff --git a/tests/test-prev-next.t b/tests/test-prev-next.t
--- a/tests/test-prev-next.t
+++ b/tests/test-prev-next.t
@@ -206,3 +206,34 @@
   move:[5] added d
   atop:[6] added b (3)
   working directory is now at 47ea25be8aea
+
+prev and next should lock properly against other commands
+
+  $ hg init repo
+  $ cd repo
+  $ HGEDITOR=${TESTDIR}/fake-editor.sh
+  $ echo hi > foo
+  $ hg ci -Am 'one'
+  adding foo
+  $ echo bye > foo
+  $ hg ci -Am 'two'
+
+  $ hg amend --edit &
+  $ sleep 1
+  $ hg prev
+  waiting for lock on working directory of $TESTTMP/repo held by process '*' 
on host '*' (glob)
+  got lock after [4-6] seconds (re)
+  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  [0] one
+  $ wait
+
+  $ hg amend --edit &
+  $ sleep 1
+  $ hg next --evolve
+  waiting for lock on working directory of $TESTTMP/repo held by process '*' 
on host '*' (glob)
+  1 new unstable changesets
+  got lock after [4-6] seconds (re)
+  move:[2] two
+  atop:[3] one
+  working directory now at a7d885c75614
+  $ wait
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel



--
Pierre-Yves David
___
Mercurial-devel mailing list

[PATCH 2 of 2 V2] wireproto: compress data from a generator

[Gregory Szorc]

# HG changeset patch
# User Gregory Szorc 
# Date 1476641421 25200
#  Sun Oct 16 11:10:21 2016 -0700
# Node ID 23457f8910d43bd637b945ced2f963ab71255844
# Parent  3f18f7464e651128a5f8d9c9312805adbc22f547
wireproto: compress data from a generator

Currently, the "getbundle" wire protocol command obtains a generator of
data, converts it to a util.chunkbuffer, then converts it back to a
generator via the protocol's groupchunks() implementation. For the SSH
protocol, groupchunks() simply reads 4kb chunks then write()s the
data to a file descriptor. For the HTTP protocol, groupchunks() reads
32kb chunks, feeds those into a zlib compressor, emits compressed data
as it is available, and that is sent to the WSGI layer, where it is
likely turned into HTTP chunked transfer chunks as is or further
buffered and turned into a larger chunk.

For both the SSH and HTTP protocols, there is inefficiency from using
util.chunkbuffer.

For SSH, emitting consistent 4kb chunks sounds nice. However, the file
descriptor it is writing to is almost certainly buffered. That means
that a Python .write() probably doesn't translate into exactly what is
written to the I/O layer.

For HTTP, we're going through an intermediate layer to zlib compress
data. So all util.chunkbuffer is doing is ensuring that the chunks we
feed into the zlib compressor are of uniform size. This means more CPU
time in Python buffering and emitting chunks in util.chunkbuffer but
fewer function calls to zlib.

This patch introduces and implements a new wire protocol abstract
method: compresschunks(). It is like groupchunks() except it operates
on a generator instead of something with a .read(). The SSH
implementation simply proxies chunks. The HTTP implementation uses
zlib compression.

To avoid duplicate code, the HTTP groupchunks() has been reimplemented
in terms of compresschunks().

To prove this all works, the "getbundle" wire protocol command has been
switched to compresschunks(). This removes the util.chunkbuffer from
that command. Now, data essentially streams straight from the
changegroup emitter to the wire, possibly through a zlib compressor.
Generators all the way, baby.

There were slim to no performance changes on the server as measured
with the mozilla-central repository. This is likely because CPU
time is dominated by reading revlogs, producing the changegroup, and
zlib compressing the output stream. Still, this brings us a little
closer to our ideal of using generators everywhere.

diff --git a/mercurial/hgweb/protocol.py b/mercurial/hgweb/protocol.py
--- a/mercurial/hgweb/protocol.py
+++ b/mercurial/hgweb/protocol.py
@@ -68,31 +68,40 @@ class webproto(wireproto.abstractserverp
 fp.write(s)
 def redirect(self):
 self.oldio = self.ui.fout, self.ui.ferr
 self.ui.ferr = self.ui.fout = stringio()
 def restore(self):
 val = self.ui.fout.getvalue()
 self.ui.ferr, self.ui.fout = self.oldio
 return val
+
 def groupchunks(self, fh):
+def getchunks():
+while True:
+chunk = fh.read(32768)
+if not chunk:
+break
+yield chunk
+
+return self.compresschunks(getchunks())
+
+def compresschunks(self, chunks):
 # Don't allow untrusted settings because disabling compression or
 # setting a very high compression level could lead to flooding
 # the server's network or CPU.
 z = zlib.compressobj(self.ui.configint('server', 'zliblevel', -1))
-while True:
-chunk = fh.read(32768)
-if not chunk:
-break
+for chunk in chunks:
 data = z.compress(chunk)
 # Not all calls to compress() emit data. It is cheaper to inspect
 # that here than to send it via the generator.
 if data:
 yield data
 yield z.flush()
+
 def _client(self):
 return 'remote:%s:%s:%s' % (
 self.req.env.get('wsgi.url_scheme') or 'http',
 urlreq.quote(self.req.env.get('REMOTE_HOST', '')),
 urlreq.quote(self.req.env.get('REMOTE_USER', '')))
 
 def iscmd(cmd):
 return cmd in wireproto.commands
diff --git a/mercurial/sshserver.py b/mercurial/sshserver.py
--- a/mercurial/sshserver.py
+++ b/mercurial/sshserver.py
@@ -66,16 +66,20 @@ class sshserver(wireproto.abstractserver
 count = int(self.fin.readline())
 
 def redirect(self):
 pass
 
 def groupchunks(self, fh):
 return iter(lambda: fh.read(4096), '')
 
+def compresschunks(self, chunks):
+for chunk in chunks:
+yield chunk
+
 def sendresponse(self, v):
 self.fout.write("%d\n" % len(v))
 self.fout.write(v)
 self.fout.flush()
 
 def sendstream(self, source):
 write = self.fout.write
 for chunk in source.gen:
diff --git a/mercurial/wireproto.py 

[PATCH 1 of 2 V2] exchange: refactor APIs to obtain bundle data (API)

[Gregory Szorc]

# HG changeset patch
# User Gregory Szorc 
# Date 1476639532 25200
#  Sun Oct 16 10:38:52 2016 -0700
# Node ID 3f18f7464e651128a5f8d9c9312805adbc22f547
# Parent  757d25d2bbe63fc560e92b6bb25fbfbf09b09342
exchange: refactor APIs to obtain bundle data (API)

Currently, exchange.getbundle() returns either a cg1unpacker or a
util.chunkbuffer (in the case of bundle2). This is kinda OK, as
both expose a .read() to consumers. However, localpeer.getbundle()
has code inferring what the response type is based on arguments and
converts the util.chunkbuffer returned in the bundle2 case to a
bundle2.unbundle20 instance. This is a sign that the API for
exchange.getbundle() is not ideal because it doesn't consistently
return an "unbundler" instance.

In addition, unbundlers mask the fact that there is an underlying
generator of changegroup data. In both cg1 and bundle2, this generator
is being fed into a util.chunkbuffer so it can be re-exposed as a
file object.

util.chunkbuffer is a nice abstraction. However, it should only be
used "at the edges." This is because keeping data as a generator is
more efficient than converting it to a chunkbuffer, especially if we
convert that chunkbuffer back to a generator (as is the case in some
code paths currently).

This patch refactors exchange.getbundle() into
exchange.getbundlechunks(). The new API returns an iterator of chunks
instead of a file-like object.

Callers of exchange.getbundle() have been updated to use the new API.

There is a minor change of behavior in test-getbundle.t. This is
because `hg debuggetbundle` isn't defining bundlecaps. As a result,
a cg1 data stream and unpacker is being produced. This is getting fed
into a new bundle20 instance via bundle2.writebundle(), which uses
a backchannel mechanism between changegroup generation to add the
"nbchanges" part parameter. I never liked this backchannel mechanism
and I plan to remove it someday. `hg bundle` still produces the
"nbchanges" part parameter, so there should be no user-visible
change of behavior. I consider this "regression" a bug in
`hg debuggetbundle`. And that bug is captured by an existing
"TODO" in the code to use bundle2 capabilities.

diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -1527,43 +1527,37 @@ def getbundle2partsgenerator(stepname, i
 return func
 return dec
 
 def bundle2requested(bundlecaps):
 if bundlecaps is not None:
 return any(cap.startswith('HG2') for cap in bundlecaps)
 return False
 
-def getbundle(repo, source, heads=None, common=None, bundlecaps=None,
-  **kwargs):
-"""return a full bundle (with potentially multiple kind of parts)
+def getbundlechunks(repo, source, heads=None, common=None, bundlecaps=None,
+**kwargs):
+"""Return chunks constituting a bundle's raw data.
 
 Could be a bundle HG10 or a bundle HG20 depending on bundlecaps
-passed. For now, the bundle can contain only changegroup, but this will
-changes when more part type will be available for bundle2.
+passed.
 
-This is different from changegroup.getchangegroup that only returns an HG10
-changegroup bundle. They may eventually get reunited in the future when we
-have a clearer idea of the API we what to query different data.
-
-The implementation is at a very early stage and will get massive rework
-when the API of bundle is refined.
+Returns an iterator over raw chunks (of varying sizes).
 """
 usebundle2 = bundle2requested(bundlecaps)
 # bundle10 case
 if not usebundle2:
 if bundlecaps and not kwargs.get('cg', True):
 raise ValueError(_('request for bundle10 must include 
changegroup'))
 
 if kwargs:
 raise ValueError(_('unsupported getbundle arguments: %s')
  % ', '.join(sorted(kwargs.keys(
 outgoing = _computeoutgoing(repo, heads, common)
-return changegroup.getchangegroup(repo, source, outgoing,
-  bundlecaps=bundlecaps)
+bundler = changegroup.getbundler('01', repo, bundlecaps)
+return changegroup.getsubsetraw(repo, outgoing, bundler, source)
 
 # bundle20 case
 b2caps = {}
 for bcaps in bundlecaps:
 if bcaps.startswith('bundle2='):
 blob = urlreq.unquote(bcaps[len('bundle2='):])
 b2caps.update(bundle2.decodecaps(blob))
 bundler = bundle2.bundle20(repo.ui, b2caps)
@@ -1571,17 +1565,17 @@ def getbundle(repo, source, heads=None, 
 kwargs['heads'] = heads
 kwargs['common'] = common
 
 for name in getbundle2partsorder:
 func = getbundle2partsmapping[name]
 func(bundler, repo, source, bundlecaps=bundlecaps, b2caps=b2caps,
  **kwargs)
 
-return util.chunkbuffer(bundler.getchunks())
+return bundler.getchunks()
 
 @getbundle2partsgenerator('changegroup')
 def 

Re: history at diff blocks level

[Gregory Szorc]

On Mon, Oct 3, 2016 at 7:38 AM, Denis Laxalde 
wrote:

> Hi all,
>
> I've been recently thinking about adding some support in Mercurial to
> query repository history based on changes within a given line range in a
> file. I think that would be useful in at least two commands:
>
> * log, to restrict history to revisions that modify specified part(s) of
> file(s) and only display the diff around specified line range and,
>
> * annotate, to window the annotate result and maybe consider walking
> file revision from tip to base to reduce computation costs.
>
> (The "log" part is more interesting, I think.)
>
>
> From UI point of view, the basic idea is to specify a (file name, line
> range) pair and the simplest solution I could find is something like:
>
>   hg log/annotate --line-range fromline,toline FILE
>
> but this does not work well with several files. (Perhaps something like
> hg log FILE:fromline:toline would be better.) I also thought about a
> "changes(filename, fromline, toline)" revset (or an extension of the
> existing "modifies()" revset), but it seems to me that this would not
> fit well for both log and annotate. Suggestions welcome.
>
>
> From the technical point of view, my idea is to rely on
> mdiff.allblocks(, ) (used
> in both annotate and log, when --patch option is given) to:
>
> 1. filter blocks depending on whether they are relevant to specified
> line range (e.g., for the log command there's some "!" block), and,
>
> 2. track the evolution of the line range across revisions (namely, given
> the line range at rev2, find the line range at rev1 in the example above).
>
> I have something working concerning this "low level" aspect, but I'm
> somehow getting stuck when it comes to plug things into the log command
> call. Namely, I need to pass the "line range" information from one
> revision to another during iterations of the loop on revisions in
> commands.log() [1] and pass this information down to the mdiff.unidiff()
> call [2] which would then give me back another line range to push up to
> the outer loop on revisions. Given the complexity of the call chain, I
> actually think this is not a very good idea... So the best idea I could
> come up with is to filter revisions beforehand (as would a revset do)
> but this would imply keeping track of files' line ranges per revision
> (to avoid processing diff blocks twice when --patch option is specified
> in particular). All in all, it's not clear to me which "tool" I may use
> to achieve this (I thought about using the "filematcher" built along
> with "revs" in commands.log(), but not really sure it's a good idea).
> Maybe I just need a data structure that does not exist yet?
> I'd appreciate any pointer to move forward.
>
> (I'll be at the sprint, so this can also be a working topic if some
> people are interested.)
>
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
>

I can't comment on the specifics of how to specify the lines in the command
line interface. But an hgweb UI that allowed you to select multiple lines
and filter log and annotate results accordingly would be a *very* useful
feature to many at Mozilla. As I said at the sprint, implementing this
should easily fall under scope of the MOSS grant.

Also, once you limit output to specific lines, a new output "view" showing
each distinct revision of requested lines (as opposed to having to perform
N commands or HTML page views to see each revision) would be a significant
win as well.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] osutil: implement listdir for linux

[Gregory Szorc]

On Fri, Oct 14, 2016 at 12:02 AM, Maciej Fijalkowski 
wrote:

> # HG changeset patch
> # User Maciej Fijalkowski 
> # Date 1476428549 -7200
> #  Fri Oct 14 09:02:29 2016 +0200
> # Node ID 4e80a66124279e235dec7ae8f58c0a14b0b137bf
> # Parent  c770219dc4c253d7cd82519ce3c74438bb2829d3
> osutil: implement listdir for linux
>
> diff --git a/mercurial/pure/osutil.py b/mercurial/pure/osutil.py
> --- a/mercurial/pure/osutil.py
> +++ b/mercurial/pure/osutil.py
> @@ -66,13 +66,20 @@
>  return result
>
>  ffi = None
> -if modulepolicy not in policynocffi and sys.platform == 'darwin':
> +if modulepolicy not in policynocffi and (
> +sys.platform == 'darwin' or sys.platform.startswith('linux')):
>  try:
>  from _osutil_cffi import ffi, lib
>  except ImportError:
>  if modulepolicy == 'cffi': # strict cffi import
>  raise
>
> +class stat_res(object):
> +def __init__(self, st_mode, st_mtime, st_size):
> +self.st_mode = st_mode
> +self.st_mtime = st_mtime
> +self.st_size = st_size
> +
>  if sys.platform == 'darwin' and ffi is not None:
>  listdir_batch_size = 4096
>  # tweakable number, only affects performance, which chunks
> @@ -88,12 +95,6 @@
>  attrkinds[lib.VFIFO] = statmod.S_IFIFO
>  attrkinds[lib.VSOCK] = statmod.S_IFSOCK
>
> -class stat_res(object):
> -def __init__(self, st_mode, st_mtime, st_size):
> -self.st_mode = st_mode
> -self.st_mtime = st_mtime
> -self.st_size = st_size
> -
>  tv_sec_ofs = ffi.offsetof("struct timespec", "tv_sec")
>  buf = ffi.new("char[]", listdir_batch_size)
>
> @@ -117,7 +118,7 @@
>  tp = attrkinds[cur.obj_type]
>  if name == "." or name == "..":
>  continue
> -if skip == name and tp == statmod.S_ISDIR:
> +if skip == name and tp == statmod.S_IFDIR:
>  return []
>  if stat:
>  mtime = cur.mtime.tv_sec
> @@ -146,12 +147,74 @@
>  try:
>  ret = listdirinternal(dfd, req, stat, skip)
>  finally:
> +lib.close(dfd)
> +return ret
> +
> +elif sys.platform.startswith('linux') and ffi is not None:
> +
> +_known_common_file_types = {
> +lib.DT_DIR:  statmod.S_IFDIR,
> +lib.DT_CHR:  statmod.S_IFCHR,
> +lib.DT_BLK:  statmod.S_IFBLK,
> +lib.DT_REG:  statmod.S_IFREG,
> +lib.DT_FIFO: statmod.S_IFIFO,
> +lib.DT_LNK:  statmod.S_IFLNK,
> +lib.DT_SOCK: statmod.S_IFSOCK,
> +}
> +
> +def listdirinternal(dir, dirfd, stat, skip):
> +buf = ffi.new("struct stat *")
> +ret = []
> +while True:
> +ffi.errno = 0
> +dirent = lib.readdir(dir)
> +if not dirent:
> +break
> +dname = dirent.d_name
> +if dname[0] == "." and (dname[1] == "\x00" or
> +(dname[1] == "." and dname[2] == "\x00")):
> +continue
> +#
> +dtype = dirent.d_type
> +if stat or dtype not in _known_common_file_types:
> +if lib.fstatat(dirfd, dirent.d_name, buf,
> +   lib.AT_SYMLINK_NOFOLLOW) != 0:
> +raise OSError(ffi.errno, os.strerror(ffi.errno))
> +tp = statmod.S_IFMT(buf.st_mode)
> +else:
> +tp = _known_common_file_types[dtype]
> +#
> +name = ffi.string(dirent.d_name)
> +if skip == name and tp == statmod.S_IFDIR:
> +return []
> +if stat:
> +ret.append((name, tp, stat_res(
> +   st_mode  = buf.st_mode,
> +   st_mtime = buf.st_mtim.tv_sec,
> +   st_size  = buf.st_size)))
> +else:
> +ret.append((name, tp))
> +
> +if ffi.errno != 0:
> +raise OSError(ffi.errno, os.strerror(ffi.errno))
> +return ret
> +
> +def listdir(path, stat=False, skip=None):
> +dfd = os.open(path, os.O_RDONLY)
> +dir = lib.fdopendir(dfd)
> +if dir == ffi.NULL:
>  try:
> -lib.close(dfd)
> +os.close(dfd)
>  except BaseException:
> -pass # we ignore all the errors from closing, not
> -# much we can do about that
> +pass
> +raise OSError(ffi.errno, os.strerror(ffi.errno))
> +
> +try:
> +ret = listdirinternal(dir, dfd, stat, skip)
> +finally:
> +lib.closedir(dir)
>  return ret
> +
>  else:
>  listdir = listdirpure
>
> diff --git a/setup.py b/setup.py
> --- a/setup.py
> +++ b/setup.py
> @@ -323,7 +323,7 @@
>  exts = [setup_mpatch_cffi.ffi.distutils_extension(),
>  

[PATCH 2 of 2 v3] evolve: lock the working copy early in next and prev (issue5244)

[Simon Farnsworth]

# HG changeset patch
# User Simon Farnsworth 
# Date 1476636773 25200
#  Sun Oct 16 09:52:53 2016 -0700
# Node ID 778468237ecf195c4eb1d87bc4f7b5d30e538c63
# Parent  ee284d7c5faa5d9f17853f51c17883844b985c58
evolve: lock the working copy early in next and prev (issue5244)

Both next and prev depend on a consistent working copy, but were waiting to
take the lock until they were ready to alter the working copy.

Take the lock before reading the working copy state, and do not release it
until we're definitely not going to change the working copy.

diff --git a/hgext/evolve.py b/hgext/evolve.py
--- a/hgext/evolve.py
+++ b/hgext/evolve.py
@@ -2213,10 +2213,13 @@
 """update to parent revision
 
 Displays the summary line of the destination for clarity."""
-if True:
+wlock = None
+dryrunopt = opts['dry_run']
+if not dryrunopt:
+wlock = repo.wlock()
+try:
 wkctx = repo[None]
 wparents = wkctx.parents()
-dryrunopt = opts['dry_run']
 if len(wparents) != 1:
 raise error.Abort('merge in progress')
 if not opts['merge']:
@@ -2246,7 +2249,6 @@
 ret = hg.update(repo, p.rev())
 if not ret:
 tr = lock = None
-wlock = repo.wlock()
 try:
 lock = repo.lock()
 tr = repo.transaction('previous')
@@ -2257,7 +2259,8 @@
 bmdeactivate(repo)
 tr.close()
 finally:
-lockmod.release(tr, lock, wlock)
+lockmod.release(tr, lock)
+
 displayer.show(p)
 return 0
 else:
@@ -2265,6 +2268,8 @@
 displayer.show(p)
 ui.warn(_('multiple parents, explicitly update to one\n'))
 return 1
+finally:
+lockmod.release(wlock)
 
 @command('^next',
  [('B', 'move-bookmark', False,
@@ -2282,10 +2287,13 @@
 
 Displays the summary line of the destination for clarity.
 """
-if True:
+wlock = None
+dryrunopt = opts['dry_run']
+if not dryrunopt:
+wlock = repo.wlock()
+try:
 wkctx = repo[None]
 wparents = wkctx.parents()
-dryrunopt = opts['dry_run']
 if len(wparents) != 1:
 raise error.Abort('merge in progress')
 if not opts['merge']:
@@ -2315,7 +2323,6 @@
 ret = hg.update(repo, c.rev())
 if not ret:
 lock = tr = None
-wlock = repo.wlock()
 try:
 lock = repo.lock()
 tr = repo.transaction('next')
@@ -2326,7 +2333,7 @@
 bmdeactivate(repo)
 tr.close()
 finally:
-lockmod.release(tr, lock, wlock)
+lockmod.release(tr, lock)
 displayer.show(c)
 result = 0
 elif children:
@@ -2368,6 +2375,8 @@
 return result
 return 1
 return result
+finally:
+lockmod.release(wlock)
 
 def _reachablefrombookmark(repo, revs, bookmarks):
 """filter revisions and bookmarks reachable from the given bookmark
diff --git a/tests/fake-editor.sh b/tests/fake-editor.sh
new file mode 100755
--- /dev/null
+++ b/tests/fake-editor.sh
@@ -0,0 +1,3 @@
+#!/bin/sh
+sleep 5
+echo "new desc" >> $1
diff --git a/tests/test-prev-next.t b/tests/test-prev-next.t
--- a/tests/test-prev-next.t
+++ b/tests/test-prev-next.t
@@ -206,3 +206,34 @@
   move:[5] added d
   atop:[6] added b (3)
   working directory is now at 47ea25be8aea
+
+prev and next should lock properly against other commands
+
+  $ hg init repo
+  $ cd repo
+  $ HGEDITOR=${TESTDIR}/fake-editor.sh
+  $ echo hi > foo
+  $ hg ci -Am 'one'
+  adding foo
+  $ echo bye > foo
+  $ hg ci -Am 'two'
+
+  $ hg amend --edit &
+  $ sleep 1
+  $ hg prev
+  waiting for lock on working directory of $TESTTMP/repo held by process '*' 
on host '*' (glob)
+  got lock after [4-6] seconds (re)
+  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  [0] one
+  $ wait
+
+  $ hg amend --edit &
+  $ sleep 1
+  $ hg next --evolve
+  waiting for lock on working directory of $TESTTMP/repo held by process '*' 
on host '*' (glob)
+  1 new unstable changesets
+  got lock after [4-6] seconds (re)
+  move:[2] two
+  atop:[3] one
+  working directory now at a7d885c75614
+  $ wait
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 1 of 2 v3] evolve: indent cmdnext and cmdprev ready for locking change (issue5244)

[Simon Farnsworth]

# HG changeset patch
# User Simon Farnsworth 
# Date 1475939661 25200
#  Sat Oct 08 08:14:21 2016 -0700
# Node ID ee284d7c5faa5d9f17853f51c17883844b985c58
# Parent  5383671ef612a1764bbbed13a7ef2d339d0a9c2d
evolve: indent cmdnext and cmdprev ready for locking change (issue5244)

The locking change I'm about to introduce forces an indentation shift. Do
the indentation change with no code change now, to make the next change
easier to review

diff --git a/hgext/evolve.py b/hgext/evolve.py
--- a/hgext/evolve.py
+++ b/hgext/evolve.py
@@ -2213,57 +2213,58 @@
 """update to parent revision
 
 Displays the summary line of the destination for clarity."""
-wkctx = repo[None]
-wparents = wkctx.parents()
-dryrunopt = opts['dry_run']
-if len(wparents) != 1:
-raise error.Abort('merge in progress')
-if not opts['merge']:
-try:
-cmdutil.bailifchanged(repo)
-except error.Abort as exc:
-exc.hint = _('do you want --merge?')
-raise
-
-parents = wparents[0].parents()
-topic = getattr(repo, 'currenttopic', '')
-if topic and not opts.get("no_topic", False):
-parents = [ctx for ctx in parents if ctx.topic() == topic]
-displayer = cmdutil.show_changeset(ui, repo, {'template': shorttemplate})
-if not parents:
-ui.warn(_('no parent in topic "%s"\n') % topic)
-ui.warn(_('(do you want --no-topic)\n'))
-elif len(parents) == 1:
-p = parents[0]
-bm = bmactive(repo)
-shouldmove = opts.get('move_bookmark') and bm is not None
-if dryrunopt:
-ui.write(('hg update %s;\n' % p.rev()))
-if shouldmove:
-ui.write(('hg bookmark %s -r %s;\n' % (bm, p.rev(
+if True:
+wkctx = repo[None]
+wparents = wkctx.parents()
+dryrunopt = opts['dry_run']
+if len(wparents) != 1:
+raise error.Abort('merge in progress')
+if not opts['merge']:
+try:
+cmdutil.bailifchanged(repo)
+except error.Abort as exc:
+exc.hint = _('do you want --merge?')
+raise
+
+parents = wparents[0].parents()
+topic = getattr(repo, 'currenttopic', '')
+if topic and not opts.get("no_topic", False):
+parents = [ctx for ctx in parents if ctx.topic() == topic]
+displayer = cmdutil.show_changeset(ui, repo, {'template': 
shorttemplate})
+if not parents:
+ui.warn(_('no parent in topic "%s"\n') % topic)
+ui.warn(_('(do you want --no-topic)\n'))
+elif len(parents) == 1:
+p = parents[0]
+bm = bmactive(repo)
+shouldmove = opts.get('move_bookmark') and bm is not None
+if dryrunopt:
+ui.write(('hg update %s;\n' % p.rev()))
+if shouldmove:
+ui.write(('hg bookmark %s -r %s;\n' % (bm, p.rev(
+else:
+ret = hg.update(repo, p.rev())
+if not ret:
+tr = lock = None
+wlock = repo.wlock()
+try:
+lock = repo.lock()
+tr = repo.transaction('previous')
+if shouldmove:
+repo._bookmarks[bm] = p.node()
+repo._bookmarks.recordchange(tr)
+else:
+bmdeactivate(repo)
+tr.close()
+finally:
+lockmod.release(tr, lock, wlock)
+displayer.show(p)
+return 0
 else:
-ret = hg.update(repo, p.rev())
-if not ret:
-tr = lock = None
-wlock = repo.wlock()
-try:
-lock = repo.lock()
-tr = repo.transaction('previous')
-if shouldmove:
-repo._bookmarks[bm] = p.node()
-repo._bookmarks.recordchange(tr)
-else:
-bmdeactivate(repo)
-tr.close()
-finally:
-lockmod.release(tr, lock, wlock)
-displayer.show(p)
-return 0
-else:
-for p in parents:
-displayer.show(p)
-ui.warn(_('multiple parents, explicitly update to one\n'))
-return 1
+for p in parents:
+displayer.show(p)
+ui.warn(_('multiple parents, explicitly update to one\n'))
+return 1
 
 @command('^next',
  [('B', 'move-bookmark', False,
@@ -2281,91 +2282,92 @@
 
 Displays the summary line of the destination for clarity.
 """
-wkctx = repo[None]
-wparents = wkctx.parents()
-dryrunopt = opts['dry_run']
-if len(wparents) != 1:
-raise error.Abort('merge in progress')
-

Re: [PATCH 1 of 2] evolve: indent cmdnext and cmdprev ready for locking change (issue5244)

[Simon Farnsworth]

On 16/10/2016 15:46, Pierre-Yves David wrote:



On 10/16/2016 04:25 PM, Simon Farnsworth wrote:

# HG changeset patch
# User Simon Farnsworth 
# Date 1475939661 25200
#  Sat Oct 08 08:14:21 2016 -0700
# Node ID ee284d7c5faa5d9f17853f51c17883844b985c58
# Parent  5383671ef612a1764bbbed13a7ef2d339d0a9c2d
evolve: indent cmdnext and cmdprev ready for locking change (issue5244)


Looks like an aborted V2. Are you aware of the --confirm flag ?

(and the associated patchbomb.confirm=True config)


I wasn't. My .hgrc now contains the associated config.
--
Simon Farnsworth
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

RE: [PATCH 05 of 12 v2] copies: make _checkcopies handle simple renames in a rotated DAG

[Gábor STEFANIK]

>


--
This message, including its attachments, is confidential. For more information 
please read NNG's email policy here:
http://www.nng.com/emailpolicy/
By responding to this email you accept the email policy.


-Original Message-
> From: Mercurial-devel [mailto:mercurial-devel-boun...@mercurial-scm.org]
> On Behalf Of Gábor Stefanik
> Sent: Sunday, October 16, 2016 5:16 PM
> To: mercurial-devel@mercurial-scm.org
> Subject: [PATCH 05 of 12 v2] copies: make _checkcopies handle simple
> renames in a rotated DAG
>
> # HG changeset patch
> # User Gábor Stefanik  # Date 1476317034 -7200
> #  Thu Oct 13 02:03:54 2016 +0200
> # Node ID 48dba8d1f7fab137418bfbb833c3096969eec934
> # Parent  1f91343556e7cef7a71edf512b02f7e0f09d5e9b
> copies: make _checkcopies handle simple renames in a rotated DAG
>
> This introduces a distinction between "merge base" and "topological
> common ancestor". During a regular merge, these two are identical. Graft,
> however, performs a merge in a rotated DAG, where the merge common
> ancestor will not be a common ancestor at all in the original DAG.

That was meant to read "merge base will not be".

>
> To correctly find copies in case of a graft, we need to take both the merge
> base and the topological CA into account, and track any renames between
> them in reverse. Fortunately we can detect this in advance, see comment in
> the code about "backwards".
>
> This patch only supports finding non-divergent renames contained entirely
> between the merge base and the topological CA. Further patches are coming
> to support more complex cases.
>
> (Pierre-Yves David was involved in the cleanup of this patch.)
>
> diff -r 1f91343556e7 -r 48dba8d1f7fa mercurial/copies.py
> --- a/mercurial/copies.pyThu Oct 13 02:03:49 2016 +0200
> +++ b/mercurial/copies.pyThu Oct 13 02:03:54 2016 +0200
> @@ -374,10 +374,10 @@
>  bothnew = sorted(addedinm1 & addedinm2)
>
>  for f in u1u:
> -_checkcopies(c1, f, m1, m2, base, limit, data1)
> +_checkcopies(c1, f, m1, m2, base, tca, limit, data1)
>
>  for f in u2u:
> -_checkcopies(c2, f, m2, m1, base, limit, data2)
> +_checkcopies(c2, f, m2, m1, base, tca, limit, data2)
>
>  copy = dict(data1['copy'].items() + data2['copy'].items())
>  fullcopy = dict(data1['fullcopy'].items() + data2['fullcopy'].items()) 
> @@ -
> 405,8 +405,8 @@
>  'diverge': bothdiverge,
> }
>  for f in bothnew:
> -_checkcopies(c1, f, m1, m2, base, limit, bothdata)
> -_checkcopies(c2, f, m2, m1, base, limit, bothdata)
> +_checkcopies(c1, f, m1, m2, base, tca, limit, bothdata)
> +_checkcopies(c2, f, m2, m1, base, tca, limit, bothdata)
>  for of, fl in bothdiverge.items():
>  if len(fl) == 2 and fl[0] == fl[1]:
>  copy[fl[0]] = of # not actually divergent, just matching renames 
> @@ -
> 521,7 +521,7 @@
>  except StopIteration:
>  return False
>
> -def _checkcopies(ctx, f, m1, m2, base, limit, data):
> +def _checkcopies(ctx, f, m1, m2, base, tca, limit, data):
>  """
>  check possible copies of f from m1 to m2
>
> @@ -530,6 +530,7 @@
>  m1 = the source manifest
>  m2 = the destination manifest
>  base = the changectx used as a merge base
> +tca = topological common ancestor for graft-like scenarios
>  limit = the rev number to not search beyond
>  data = dictionary of dictionary to store copy data. (see mergecopies)
>
> @@ -540,6 +541,17 @@
>  """
>
>  mb = base.manifest()
> +# Might be true if this call is about finding backward renames,
> +# This happens in the case of grafts because the DAG is then rotated.
> +# If the file exists in both the base and the source, we are not looking
> +# for a rename on the source side, but on the part of the DAG that is
> +# traversed backwards.
> +#
> +# In the case there is both backward and forward renames (before and
> after
> +# the base) this is more complicated as we must detect a divergence. This
> +# is currently broken and hopefully some later code update will make that
> +# work (we use 'backwards = False' in that case)
> +backwards = base != tca and f in mb
>  getfctx = _makegetfctx(ctx)
>
>  of = None
> @@ -554,7 +566,11 @@
>  continue
>  seen.add(of)
>
> -data['fullcopy'][f] = of # remember for dir rename detection
> +# remember for dir rename detection
> +if backwards:
> +data['fullcopy'][of] = f # grafting backwards through renames
> +else:
> +data['fullcopy'][f] = of
>  if of not in m2:
>  continue # no match, keep looking
>  if m2[of] == mb.get(of):
> @@ -562,9 +578,11 @@
>  c2 = getfctx(of, m2[of])
>  # c2 might be a plain new file on added on destination side that is
>

Re: [PATCH 4 of 4] mergecopies: rename 'ca' to 'base'

[Yuya Nishihara]

On Sun, 16 Oct 2016 16:47:23 +0200, Pierre-Yves David wrote:
> # HG changeset patch
> # User Pierre-Yves David 
> # Date 1476315014 -7200
> #  Thu Oct 13 01:30:14 2016 +0200
> # Node ID b8bb58f87a94f5b78beaa1db710c5ef69967dd4c
> # Parent  6f0dc570c4383ad42c572158af200d2142bbef4f
> # EXP-Topic checkcopies
> mergecopies: rename 'ca' to 'base'

LGTM. Queued the series, thanks.

> diff --git a/mercurial/copies.py b/mercurial/copies.py
> --- a/mercurial/copies.py
> +++ b/mercurial/copies.py
> @@ -285,14 +285,14 @@ def _makegetfctx(ctx):
>  return fctx
>  return util.lrucachefunc(makectx)
>  
> -def mergecopies(repo, c1, c2, ca):
> +def mergecopies(repo, c1, c2, base):
>  """
>  Find moves and copies between context c1 and c2 that are relevant
> -for merging.
> +for merging. 'base' will be used as the merge base.
>  
>  Returns four dicts: "copy", "movewithdir", "diverge", and
>  "renamedelete".
> -
> +`

Removed this garbage character. I wiped my screen physically, but it never
disappeared.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 12 of 12 v2] update: enable copy tracing for backwards and non-linear updates

[Gábor Stefanik]

# HG changeset patch
# User Gábor Stefanik 
# Date 1472155346 -7200
#  Thu Aug 25 22:02:26 2016 +0200
# Node ID 066dc1a5c8f4c9dfb582543bff5df8bbeba9cc31
# Parent  a3a36d782f4d4252adb9069eb3714f946881
update: enable copy tracing for backwards and non-linear updates

As a followup to the issue4028 series, this fixes a variant of the issue
that can occur when updating with uncommited local changes.

diff -r a3a36d782f4d -r 066dc1a5c8f4 mercurial/merge.py
--- a/mercurial/merge.pyTue Oct 11 04:39:47 2016 +0200
+++ b/mercurial/merge.pyThu Aug 25 22:02:26 2016 +0200
@@ -1537,15 +1537,16 @@
 pas = [p1]
 
 # deprecated config: merge.followcopies
-followcopies = False
+followcopies = repo.ui.configbool('merge', 'followcopies', True)
 if overwrite:
 pas = [wc]
+followcopies = False
 elif pas == [p2]: # backwards
-pas = [wc.p1()]
-elif not branchmerge and not wc.dirty(missing=True):
-pass
-elif pas[0] and repo.ui.configbool('merge', 'followcopies', True):
-followcopies = True
+pas = [p1]
+elif not pas[0]:
+followcopies = False
+if not branchmerge and not wc.dirty(missing=True):
+followcopies = False
 
 ### calculate phase
 actionbyfile, diverge, renamedelete = calculateupdates(
diff -r a3a36d782f4d -r 066dc1a5c8f4 tests/test-merge-local.t
--- a/tests/test-merge-local.t  Tue Oct 11 04:39:47 2016 +0200
+++ b/tests/test-merge-local.t  Thu Aug 25 22:02:26 2016 +0200
@@ -66,7 +66,7 @@
   merging zzz1_merge_ok
   merging zzz2_merge_bad
   warning: conflicts while merging zzz2_merge_bad! (edit, then use 'hg resolve 
--mark')
-  2 files updated, 1 files merged, 3 files removed, 1 files unresolved
+  2 files updated, 1 files merged, 2 files removed, 1 files unresolved
   use 'hg resolve' to retry unresolved file merges
   [1]
 
@@ -104,7 +104,7 @@
   merging zzz1_merge_ok
   merging zzz2_merge_bad
   warning: conflicts while merging zzz2_merge_bad! (edit, then use 'hg resolve 
--mark')
-  2 files updated, 1 files merged, 3 files removed, 1 files unresolved
+  2 files updated, 1 files merged, 2 files removed, 1 files unresolved
   use 'hg resolve' to retry unresolved file merges
   [1]
 
diff -r a3a36d782f4d -r 066dc1a5c8f4 tests/test-mq-subrepo.t
--- a/tests/test-mq-subrepo.t   Tue Oct 11 04:39:47 2016 +0200
+++ b/tests/test-mq-subrepo.t   Thu Aug 25 22:02:26 2016 +0200
@@ -304,6 +304,7 @@
   record this change to '.hgsub'? [Ynesfdaq?] y
   
   warning: subrepo spec file '.hgsub' not found
+  warning: subrepo spec file '.hgsub' not found
   abort: uncommitted changes in subrepository 'sub'
   [255]
   % update substate when adding .hgsub w/clean updated subrepo
@@ -319,6 +320,7 @@
   record this change to '.hgsub'? [Ynesfdaq?] y
   
   warning: subrepo spec file '.hgsub' not found
+  warning: subrepo spec file '.hgsub' not found
   path sub
source   sub
revision b2fdb12cd82b021c3b7053d67802e77b6eeaee31
diff -r a3a36d782f4d -r 066dc1a5c8f4 tests/test-up-local-change.t
--- a/tests/test-up-local-change.t  Tue Oct 11 04:39:47 2016 +0200
+++ b/tests/test-up-local-change.t  Thu Aug 25 22:02:26 2016 +0200
@@ -67,6 +67,10 @@
   summary: 2
   
   $ hg --debug up 0
+  starting 4 threads for background file closing (?)
+searching for copies back to rev 0
+unmatched files in local (from topological common ancestor):
+ b
   resolving manifests
branchmerge: False, force: False, partial: False
ancestor: 1e71731e6fbb, local: 1e71731e6fbb+, remote: c19d34741b0a
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 05 of 12 v2] copies: make _checkcopies handle simple renames in a rotated DAG

[Gábor Stefanik]

# HG changeset patch
# User Gábor Stefanik 
# Date 1476317034 -7200
#  Thu Oct 13 02:03:54 2016 +0200
# Node ID 48dba8d1f7fab137418bfbb833c3096969eec934
# Parent  1f91343556e7cef7a71edf512b02f7e0f09d5e9b
copies: make _checkcopies handle simple renames in a rotated DAG

This introduces a distinction between "merge base" and
"topological common ancestor". During a regular merge, these two are
identical. Graft, however, performs a merge in a rotated DAG, where the
merge common ancestor will not be a common ancestor at all in the
original DAG.

To correctly find copies in case of a graft, we need to take both the
merge base and the topological CA into account, and track any renames
between them in reverse. Fortunately we can detect this in advance,
see comment in the code about "backwards".

This patch only supports finding non-divergent renames contained entirely
between the merge base and the topological CA. Further patches are coming
to support more complex cases.

(Pierre-Yves David was involved in the cleanup of this patch.)

diff -r 1f91343556e7 -r 48dba8d1f7fa mercurial/copies.py
--- a/mercurial/copies.py   Thu Oct 13 02:03:49 2016 +0200
+++ b/mercurial/copies.py   Thu Oct 13 02:03:54 2016 +0200
@@ -374,10 +374,10 @@
 bothnew = sorted(addedinm1 & addedinm2)
 
 for f in u1u:
-_checkcopies(c1, f, m1, m2, base, limit, data1)
+_checkcopies(c1, f, m1, m2, base, tca, limit, data1)
 
 for f in u2u:
-_checkcopies(c2, f, m2, m1, base, limit, data2)
+_checkcopies(c2, f, m2, m1, base, tca, limit, data2)
 
 copy = dict(data1['copy'].items() + data2['copy'].items())
 fullcopy = dict(data1['fullcopy'].items() + data2['fullcopy'].items())
@@ -405,8 +405,8 @@
 'diverge': bothdiverge,
}
 for f in bothnew:
-_checkcopies(c1, f, m1, m2, base, limit, bothdata)
-_checkcopies(c2, f, m2, m1, base, limit, bothdata)
+_checkcopies(c1, f, m1, m2, base, tca, limit, bothdata)
+_checkcopies(c2, f, m2, m1, base, tca, limit, bothdata)
 for of, fl in bothdiverge.items():
 if len(fl) == 2 and fl[0] == fl[1]:
 copy[fl[0]] = of # not actually divergent, just matching renames
@@ -521,7 +521,7 @@
 except StopIteration:
 return False
 
-def _checkcopies(ctx, f, m1, m2, base, limit, data):
+def _checkcopies(ctx, f, m1, m2, base, tca, limit, data):
 """
 check possible copies of f from m1 to m2
 
@@ -530,6 +530,7 @@
 m1 = the source manifest
 m2 = the destination manifest
 base = the changectx used as a merge base
+tca = topological common ancestor for graft-like scenarios
 limit = the rev number to not search beyond
 data = dictionary of dictionary to store copy data. (see mergecopies)
 
@@ -540,6 +541,17 @@
 """
 
 mb = base.manifest()
+# Might be true if this call is about finding backward renames,
+# This happens in the case of grafts because the DAG is then rotated.
+# If the file exists in both the base and the source, we are not looking
+# for a rename on the source side, but on the part of the DAG that is
+# traversed backwards.
+#
+# In the case there is both backward and forward renames (before and after
+# the base) this is more complicated as we must detect a divergence. This
+# is currently broken and hopefully some later code update will make that
+# work (we use 'backwards = False' in that case)
+backwards = base != tca and f in mb
 getfctx = _makegetfctx(ctx)
 
 of = None
@@ -554,7 +566,11 @@
 continue
 seen.add(of)
 
-data['fullcopy'][f] = of # remember for dir rename detection
+# remember for dir rename detection
+if backwards:
+data['fullcopy'][of] = f # grafting backwards through renames
+else:
+data['fullcopy'][f] = of
 if of not in m2:
 continue # no match, keep looking
 if m2[of] == mb.get(of):
@@ -562,9 +578,11 @@
 c2 = getfctx(of, m2[of])
 # c2 might be a plain new file on added on destination side that is
 # unrelated to the droids we are looking for.
-cr = _related(oc, c2, base.rev())
+cr = _related(oc, c2, tca.rev())
 if cr and (of == f or of == c2.path()): # non-divergent
-if of in mb:
+if backwards:
+data['copy'][of] = f
+elif of in mb:
 data['copy'][f] = of
 return
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

mercurial@30163: new changeset

[Mercurial Commits]

New changeset in mercurial:

http://selenic.com/repo/hg//rev/f5607b6253da
changeset:   30163:f5607b6253da
bookmark:@
tag: tip
user:Gregory Szorc 
date:Thu Oct 13 21:42:11 2016 +0200
summary: pathencode: use assert() for PyBytes_Check()

-- 
Repository URL: http://selenic.com/repo/hg/
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 11 of 12 v2] copies: make _checkcopies handle copy sequences spanning the TCA (issue4028)

[Gábor Stefanik]

# HG changeset patch
# User Gábor Stefanik 
# Date 1476153587 -7200
#  Tue Oct 11 04:39:47 2016 +0200
# Node ID a3a36d782f4d4252adb9069eb3714f946881
# Parent  2bf601f8e1d3569fdef9ab9f3fefa354acce8669
copies: make _checkcopies handle copy sequences spanning the TCA (issue4028)

When working in a rotated DAG (for a graftlike merge), there can be files
that are renamed both between the base and the topological CA, and between
the TCA and the endpoint farther from the base. Such renames span the TCA
(and thus need both passes of _checkcopies to be fully detected), but may
not necessarily be divergent.

Make _checkcopies return "incomplete copies" and "incomplete divergences"
in this case, and let mergecopies recombine them once data from both passes
of _checkcopies is available.

With this patch, all known cases involving renames and grafts pass.

(Developed together with Pierre-Yves David)

diff -r 2bf601f8e1d3 -r a3a36d782f4d mercurial/copies.py
--- a/mercurial/copies.py   Tue Oct 11 04:25:59 2016 +0200
+++ b/mercurial/copies.py   Tue Oct 11 04:39:47 2016 +0200
@@ -611,6 +611,7 @@
 """
 
 mb = base.manifest()
+mta = tca.manifest()
 # Might be true if this call is about finding backward renames,
 # This happens in the case of grafts because the DAG is then rotated.
 # If the file exists in both the base and the source, we are not looking
@@ -665,8 +666,17 @@
 break
 return
 
-if of in mb:
-data['diverge'].setdefault(of, []).append(f)
+if of in mta:
+if backwards or remotebase:
+data['incomplete'][of] = f
+else:
+for sf in seen:
+if sf in mb:
+if tca == base:
+data['diverge'].setdefault(sf, []).append(f)
+else:
+data['incompletediverge'][sf] = [of, f]
+return
 
 def duplicatecopies(repo, rev, fromrev, skiprev=None):
 '''reproduce copies from fromrev to rev in the dirstate
diff -r 2bf601f8e1d3 -r a3a36d782f4d tests/test-graft.t
--- a/tests/test-graft.tTue Oct 11 04:25:59 2016 +0200
+++ b/tests/test-graft.tTue Oct 11 04:39:47 2016 +0200
@@ -957,15 +957,6 @@
   grafting 2:f58c7e2b28fa "C0"
   merging f1a and f1b to f1a
   merging f5a
-  warning: conflicts while merging f5a! (edit, then use 'hg resolve --mark')
-  abort: unresolved conflicts, can't continue
-  (use 'hg resolve' and 'hg graft --continue')
-  [255]
-  $ hg resolve f5a -t ':other' # XXX work around failure
-  (no more unresolved files)
-  continue: hg graft --continue
-  $ hg graft --continue # XXX work around failure
-  grafting 2:f58c7e2b28fa "C0"
   warning: can't find ancestor for 'f5a' copied from 'f5b'!
   $ hg status --change .
   M f1a
@@ -975,7 +966,7 @@
   $ hg cat f1a
   c1c
   $ hg cat f1b
-  f1b: no such file in rev 43e4b415492d
+  f1b: no such file in rev c9763722f9bd
   [1]
 
 Test the cases A.0 (f4x) and A.6 (f3x)
@@ -997,23 +988,23 @@
   $ hg mv f5a f5b
   $ hg ci -qAm "E0"
   $ hg log -G
-  @  changeset:   6:ebba59d1fb02
+  @  changeset:   6:6bd1736cab86
   |  tag: tip
   |  parent:  0:11f7a1b56675
   |  user:test
   |  date:Thu Jan 01 00:00:00 1970 +
   |  summary: E0
   |
-  | o  changeset:   5:4f4ba7a6e606
+  | o  changeset:   5:560daee679da
   | |  user:test
   | |  date:Thu Jan 01 00:00:00 1970 +
   | |  summary: D1
   | |
-  | o  changeset:   4:43e4b415492d
+  | o  changeset:   4:c9763722f9bd
   |/   parent:  0:11f7a1b56675
   |user:test
   |date:Thu Jan 01 00:00:00 1970 +
-  |summary: C0
+  |summary: C1
   |
   | o  changeset:   3:b69f5839d2d9
   | |  user:test
@@ -1041,34 +1032,24 @@
 
   $ HGEDITOR="echo C2 >" hg graft -r 'desc("C0")' --edit
   grafting 2:f58c7e2b28fa "C0"
-  other [graft] changed f1b which local [local] deleted
-  use (c)hanged version, leave (d)eleted, or leave (u)nresolved? u
+  merging f1e and f1b to f1e
   merging f2a and f2c to f2c
   merging f5b and f5a to f5a
-  abort: unresolved conflicts, can't continue
-  (use 'hg resolve' and 'hg graft --continue')
-  [255]
-  $ hg resolve f1b -t ':other' # XXX work around failure
-  (no more unresolved files)
-  continue: hg graft --continue
-  $ hg graft --continue # XXX work around failure
-  grafting 2:f58c7e2b28fa "C0"
-  grafting 4:43e4b415492d "C0"
-  merging f1e and f1a to f1e
-  merging f2c
-  warning: can't find ancestor for 'f2c' copied from 'f2a'!
 
 Test the cases A.1 (f4x) and A.7 (f3x).
 
   $ HGEDITOR="echo D2 >" hg graft -r 'desc("D0")' --edit
   grafting 3:b69f5839d2d9 "D0"
+  note: possible conflict - f3b was renamed multiple times to:
+   f3e
+   f3d
   merging f4e and f4a to f4e
   warning: can't find ancestor for 'f3d' copied from 'f3b'!
 
 Check the results of the grafts tested
 
   $ hg log -CGv --patch --git
-  @  changeset:   

[PATCH 03 of 12 v2] copies: detect graft-like merges

[Gábor Stefanik]

# HG changeset patch
# User Gábor Stefanik 
# Date 1476316053 -7200
#  Thu Oct 13 01:47:33 2016 +0200
# Node ID 981a622717343796f65b5858f4d4b89f81cb437a
# Parent  7f9a6f30730074c869e5cf9b77c53929b06c4fb3
copies: detect graft-like merges

Right now, nothing changes as a result of this, but we want to handle
grafts differently from ordinary merges later.

(Series developed together with Pierre-Yves David)

diff -r 7f9a6f307300 -r 981a62271734 mercurial/copies.py
--- a/mercurial/copies.py   Wed Oct 12 12:41:28 2016 +0200
+++ b/mercurial/copies.py   Thu Oct 13 01:47:33 2016 +0200
@@ -321,6 +321,23 @@
 if repo.ui.configbool('experimental', 'disablecopytrace'):
 return {}, {}, {}, {}
 
+# In certain scenarios (e.g. graft, update or rebase), base can be
+# overridden We still need to know a real common ancestor in this case We
+# can't just compute _c1.ancestor(_c2) and compare it to ca, because there
+# can be multiple common ancestors, e.g. in case of bidmerge.  Because our
+# caller may not know if the revision passed in lieu of the CA is a genuine
+# common ancestor or not without explicitly checking it, it's better to
+# determine that here.
+#
+# base.descendant(wc) and base.descendant(base) are False, work around that
+_c1 = c1.p1() if c1.rev() is None else c1
+_c2 = c2.p1() if c2.rev() is None else c2
+# an endpoint is "dirty" if it isn't a descendant of the merge base
+# if we have a dirty endpoint, we need to trigger graft logic, and also
+# keep track of which endpoint is dirty
+dirtyc1 = not (base == _c1 or base.descendant(_c1))
+dirtyc2 = not (base== _c2 or base.descendant(_c2))
+graft = dirtyc1 or dirtyc2
 limit = _findlimit(repo, c1.rev(), c2.rev())
 if limit is None:
 # no common ancestor, no copies
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 01 of 12 v2] checkcopies: add a sanity check against false-positive copies

[Gábor Stefanik]

# HG changeset patch
# User Gábor Stefanik 
# Date 1476300825 -7200
#  Wed Oct 12 21:33:45 2016 +0200
# Node ID e8e15a687df328f37b2d5e97e67f1a2071b35134
# Parent  bfb99ef3749757161bf439cc0ec678de46e24672
checkcopies: add a sanity check against false-positive copies

When grafting a copy backwards through a rename, a copy is wrongly detected,
which causes the graft to be applied inappropriately, in a destructive way.
Make sure that the old file name really exists in the common ancestor,
and bail out if it doesn't.

This fixes the aggravated case of bug 5343, although the basic issue
(failure to duplicate the copy information) still occurs.

diff -r bfb99ef37497 -r e8e15a687df3 mercurial/copies.py
--- a/mercurial/copies.py   Thu Oct 13 01:30:14 2016 +0200
+++ b/mercurial/copies.py   Wed Oct 12 21:33:45 2016 +0200
@@ -543,7 +543,8 @@
 # unrelated to the droids we are looking for.
 cr = _related(oc, c2, base.rev())
 if cr and (of == f or of == c2.path()): # non-divergent
-data['copy'][f] = of
+if of in mb:
+data['copy'][f] = of
 return
 
 if of in mb:
diff -r bfb99ef37497 -r e8e15a687df3 tests/test-graft.t
--- a/tests/test-graft.tThu Oct 13 01:30:14 2016 +0200
+++ b/tests/test-graft.tWed Oct 12 21:33:45 2016 +0200
@@ -427,8 +427,8 @@
   $ hg graft 3 --log -u foo
   grafting 3:4c60f11aa304 "3"
   warning: can't find ancestor for 'c' copied from 'b'!
-  $ hg log --template '{rev} {parents} {desc}\n' -r tip
-  14 1:5d205f8b35b6  3
+  $ hg log --template '{rev}:{node|short} {parents} {desc}\n' -r tip
+  14:0c921c65ef1e 1:5d205f8b35b6  3
   (grafted from 4c60f11aa304a54ae1c199feb94e7fc771e51ed8)
 
 Resolve conflicted graft
@@ -620,7 +620,7 @@
   date:Thu Jan 01 00:00:00 1970 +
   summary: 2
   
-  changeset:   14:f64defefacee
+  changeset:   14:0c921c65ef1e
   parent:  1:5d205f8b35b6
   user:foo
   date:Thu Jan 01 00:00:00 1970 +
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 02 of 12 v2] tests: introduce tests for grafting through renames

[Gábor Stefanik]

# HG changeset patch
# User Gábor Stefanik 
# Date 147626 -7200
#  Wed Oct 12 12:41:28 2016 +0200
# Node ID 7f9a6f30730074c869e5cf9b77c53929b06c4fb3
# Parent  e8e15a687df328f37b2d5e97e67f1a2071b35134
tests: introduce tests for grafting through renames

These cover all currently known cases of renames being grafted,
or changes being grafted through renames.

Right now, most of these cases are broken. Later patches in this series
will make them behave correctly.

The testcases heavily rely on each other, which would make it very difficult
to separate them and add them one-by-one for each case fixed by a patch.
Separating them should perhaps be a 4.1 task, if it doesn't slow down
the tests too much.

(Developed together with Pierre-Yves David)

diff -r e8e15a687df3 -r 7f9a6f307300 tests/test-graft.t
--- a/tests/test-graft.tWed Oct 12 21:33:45 2016 +0200
+++ b/tests/test-graft.tWed Oct 12 12:41:28 2016 +0200
@@ -842,3 +842,461 @@
   |/
   o  0
   
+Graft from behind a move or rename
+==
+
+NOTE: This is affected by issue5343, and will need updating when it's fixed
+
+Possible cases during a regular graft (when ca is between cta and c2):
+
+name | c1<-cta | cta<->ca | ca->c2
+A.0  | |  |
+A.1  |X|  |
+A.2  | | X|
+A.3  | |  |   X
+A.4  |X| X|
+A.5  |X|  |   X
+A.6  | | X|   X
+A.7  |X| X|   X
+
+A.0 is trivial, and doesn't need copy tracking.
+For A.1, a forward rename is recorded in the c1 pass, to be followed later.
+In A.2, the rename is recorded in the c2 pass and followed backwards.
+A.3 is recorded in the c2 pass as a forward rename to be duplicated on target.
+In A.4, both passes of checkcopies record incomplete renames, which are
+then joined in mergecopies to record a rename to be followed.
+In A.5 and A.7, the c1 pass records an incomplete rename, while the c2 pass
+records an incomplete divergence. The incomplete rename is then joined to the
+appropriate side of the incomplete divergence, and the result is recorded as a
+divergence. The code doesn't distinguish at all between these two cases, since
+the end result of them is the same: an incomplete divergence joined with an
+incomplete rename into a divergence.
+Finally, A.6 records a divergence entirely in the c2 pass.
+
+A.4 has a degenerate case a<-b<-a->a, where checkcopies isn't needed at all.
+A.5 has a special case a<-b<-b->a, which is treated like a<-b->a in a merge.
+A.6 has a special case a<-a<-b->a. Here, checkcopies will find a spurious
+incomplete divergence, which is in fact complete. This is handled later in
+mergecopies.
+A.7 has 4 special cases: a<-b<-a->b (the "ping-pong" case), a<-b<-c->b,
+a<-b<-a->c and a<-b<-c->a. Of these, only the "ping-pong" case is interesting,
+the others are fairly trivial (a<-b<-c->b and a<-b<-a->c proceed like the base
+case, a<-b<-c->a is treated the same as a<-b<-b->a).
+
+f5a therefore tests the "ping-pong" rename case, where a file is renamed to the
+same name on both branches, then the rename is backed out on one branch, and
+the backout is grafted to the other branch. This creates a challenging rename
+sequence of a<-b<-a->b in the graft target, topological CA, graft CA and graft
+source, respectively. Since rename detection will run on the c1 side for such a
+sequence (as for technical reasons, we split the c1 and c2 sides not at the
+graft CA, but rather at the topological CA), it will pick up a false rename,
+and cause a spurious merge conflict. This false rename is always exactly the
+reverse of the true rename that would be detected on the c2 side, so we can
+correct for it by detecting this condition and reversing as necessary.
+
+First, set up the repository with commits to be grafted
+
+  $ hg init ../graftmove
+  $ cd ../graftmove
+  $ echo c1a > f1a
+  $ echo c2a > f2a
+  $ echo c3a > f3a
+  $ echo c4a > f4a
+  $ echo c5a > f5a
+  $ hg ci -qAm A0
+  $ hg mv f1a f1b
+  $ hg mv f3a f3b
+  $ hg mv f5a f5b
+  $ hg ci -qAm B0
+  $ echo c1c > f1b
+  $ hg mv f2a f2c
+  $ hg mv f5b f5a
+  $ echo c5c > f5a
+  $ hg ci -qAm C0
+  $ hg mv f3b f3d
+  $ echo c4d > f4a
+  $ hg ci -qAm D0
+  $ hg log -G
+  @  changeset:   3:b69f5839d2d9
+  |  tag: tip
+  |  user:test
+  |  date:Thu Jan 01 00:00:00 1970 +
+  |  summary: D0
+  |
+  o  changeset:   2:f58c7e2b28fa
+  |  user:test
+  |  date:Thu Jan 01 00:00:00 1970 +
+  |  summary: C0
+  |
+  o  changeset:   1:3d7bba921b5d
+  |  user:test
+  |  date:Thu Jan 01 00:00:00 1970 +
+  |  summary: B0
+  |
+  o  changeset:   0:11f7a1b56675
+ user:test
+ date:Thu Jan 01 00:00:00 1970 +
+ summary: A0
+  
+
+Test the cases A.2 (f1x), A.3 (f2x) and a special case of A.6 (f5x) where the
+two renames actually converge to the same name (thus no actual 

[PATCH 10 of 12 v2] checkcopies: add logic to handle remotebase

[Gábor Stefanik]

# HG changeset patch
# User Gábor Stefanik 
# Date 1476152759 -7200
#  Tue Oct 11 04:25:59 2016 +0200
# Node ID 2bf601f8e1d3569fdef9ab9f3fefa354acce8669
# Parent  9fc636d9950feb76af774f881ea179037231e697
checkcopies: add logic to handle remotebase

As the two _checkcopies passes' ranges are separated by tca, not base,
only one of the two passes will actually encounter the base.
Pass "remotebase" to the other pass to let it know not to expect passing
over the base. This is required for handling a few unusual rename cases.

diff -r 9fc636d9950f -r 2bf601f8e1d3 mercurial/copies.py
--- a/mercurial/copies.py   Tue Oct 04 12:51:54 2016 +0200
+++ b/mercurial/copies.py   Tue Oct 11 04:25:59 2016 +0200
@@ -413,10 +413,10 @@
   baselabel='topological common ancestor')
 
 for f in u1u:
-_checkcopies(c1, f, m1, m2, base, tca, limit, data1)
+_checkcopies(c1, f, m1, m2, base, tca, dirtyc1, limit, data1)
 
 for f in u2u:
-_checkcopies(c2, f, m2, m1, base, tca, limit, data2)
+_checkcopies(c2, f, m2, m1, base, tca, dirtyc2, limit, data2)
 
 copy = dict(data1['copy'].items() + data2['copy'].items())
 fullcopy = dict(data1['fullcopy'].items() + data2['fullcopy'].items())
@@ -460,8 +460,8 @@
  'incompletediverge': bothincompletediverge
 }
 for f in bothnew:
-_checkcopies(c1, f, m1, m2, base, tca, limit, both1)
-_checkcopies(c2, f, m2, m1, base, tca, limit, both2)
+_checkcopies(c1, f, m1, m2, base, tca, dirtyc1, limit, both1)
+_checkcopies(c2, f, m2, m1, base, tca, dirtyc2, limit, both2)
 if dirtyc1:
 assert both2['incomplete'] == {}
 remainder = _combinecopies({}, both1['incomplete'], copy, bothdiverge,
@@ -590,7 +590,7 @@
 except StopIteration:
 return False
 
-def _checkcopies(ctx, f, m1, m2, base, tca, limit, data):
+def _checkcopies(ctx, f, m1, m2, base, tca, remotebase, limit, data):
 """
 check possible copies of f from m1 to m2
 
@@ -600,6 +600,7 @@
 m2 = the destination manifest
 base = the changectx used as a merge base
 tca = topological common ancestor for graft-like scenarios
+remotebase = True if base is outside tca::ctx, False otherwise
 limit = the rev number to not search beyond
 data = dictionary of dictionary to store copy data. (see mergecopies)
 
@@ -619,7 +620,7 @@
 # In the case there is both backward and forward renames (before and after
 # the base) this is more complicated as we must detect a divergence.
 # We use 'backwards = False' in that case.
-backwards = base != tca and f in mb
+backwards = not remotebase and base != tca and f in mb
 getfctx = _makegetfctx(ctx)
 
 of = None
@@ -652,6 +653,10 @@
 data['copy'][of] = f
 elif of in mb:
 data['copy'][f] = of
+elif remotebase: # special case: a <- b <- a -> b "ping-pong" 
rename
+data['copy'][of] = f
+del data['fullcopy'][f]
+data['fullcopy'][of] = f
 else: # divergence w.r.t. graft CA on one side of topological CA
 for sf in seen:
 if sf in mb:
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 07 of 12 v2] mergecopies: invoke _computenonoverlap for both base and tca during merges

[Gábor Stefanik]

# HG changeset patch
# User Gábor Stefanik 
# Date 1476317983 -7200
#  Thu Oct 13 02:19:43 2016 +0200
# Node ID e61355cb337ab5328e4cf5d0f153821600c53e72
# Parent  72b4f2514efd9decffb1115c0f7b8e8f004bdf44
mergecopies: invoke _computenonoverlap for both base and tca during merges

The algorithm of _checkcopies can only walk backwards in the DAG, never
forward. Because of this, the two _checkcopies patches need to run from
their respective endpoints to the TCA to cover the entire subgraph where
the merge is being performed. However, detection of files new in both
endpoints, as well as directory rename detection, need to run with respect
to the merge base, so we need lists of new files both from the TCA's and
the merge base's viewpoint to correctly detect renames in a graft-like
merge scenario.

(Series reworked by Pierre-Yves David)

diff -r 72b4f2514efd -r e61355cb337a mercurial/copies.py
--- a/mercurial/copies.py   Thu Oct 13 17:21:49 2016 +0200
+++ b/mercurial/copies.py   Thu Oct 13 02:19:43 2016 +0200
@@ -373,9 +373,21 @@
 # find interesting file sets from manifests
 addedinm1 = m1.filesnotin(mb)
 addedinm2 = m2.filesnotin(mb)
-u1r, u2r = _computenonoverlap(repo, c1, c2, addedinm1, addedinm2)
-u1u, u2u = u1r, u2r
 bothnew = sorted(addedinm1 & addedinm2)
+if tca == base:
+# unmatched file from base
+u1r, u2r = _computenonoverlap(repo, c1, c2, addedinm1, addedinm2)
+u1u, u2u = u1r, u2r
+else:
+# unmatched file from base (DAG rotation in the graft case)
+u1r, u2r = _computenonoverlap(repo, c1, c2, addedinm1, addedinm2,
+  baselabel='base')
+# unmatched file from topological common ancestors (no DAG rotation)
+# need to recompute this for directory move handling when grafting
+mta = tca.manifest()
+u1u, u2u = _computenonoverlap(repo, c1, c2, m1.filesnotin(mta),
+m2.filesnotin(mta),
+  baselabel='topological common ancestor')
 
 for f in u1u:
 _checkcopies(c1, f, m1, m2, base, tca, limit, data1)
diff -r 72b4f2514efd -r e61355cb337a tests/test-graft.t
--- a/tests/test-graft.tThu Oct 13 17:21:49 2016 +0200
+++ b/tests/test-graft.tThu Oct 13 02:19:43 2016 +0200
@@ -179,6 +179,11 @@
   committing changelog
   grafting 5:97f8bfe72746 "5"
 searching for copies back to rev 1
+unmatched files in other (from topological common ancestor):
+ c
+all copies found (* = to merge, ! = divergent, % = renamed and deleted):
+ src: 'c' -> dst: 'b' *
+checking for directory renames
   resolving manifests
branchmerge: True, force: True, partial: False
ancestor: 4c60f11aa304, local: 6b9e5368ca4e+, remote: 97f8bfe72746
@@ -193,6 +198,11 @@
   scanning for duplicate grafts
   grafting 4:9c233e8e184d "4"
 searching for copies back to rev 1
+unmatched files in other (from topological common ancestor):
+ c
+all copies found (* = to merge, ! = divergent, % = renamed and deleted):
+ src: 'c' -> dst: 'b' *
+checking for directory renames
   resolving manifests
branchmerge: True, force: True, partial: False
ancestor: 4c60f11aa304, local: 1905859650ec+, remote: 9c233e8e184d
@@ -945,6 +955,7 @@
   $ hg up -q 'desc("A0")'
   $ HGEDITOR="echo C1 >" hg graft -r 'desc("C0")' --edit
   grafting 2:f58c7e2b28fa "C0"
+  merging f1a and f1b to f1a
   merging f5a
   warning: conflicts while merging f5a! (edit, then use 'hg resolve --mark')
   abort: unresolved conflicts, can't continue
@@ -957,14 +968,15 @@
   grafting 2:f58c7e2b28fa "C0"
   warning: can't find ancestor for 'f5a' copied from 'f5b'!
   $ hg status --change .
+  M f1a
   M f5a
-  A f1b
   A f2c
   R f2a
   $ hg cat f1a
-  c1a
+  c1c
   $ hg cat f1b
-  c1c
+  f1b: no such file in rev 43e4b415492d
+  [1]
 
 Test the cases A.0 (f4x) and A.6 (f3x)
 
@@ -989,12 +1001,12 @@
   |  date:Thu Jan 01 00:00:00 1970 +
   |  summary: E0
   |
-  | o  changeset:   5:573bb6b4b56d
+  | o  changeset:   5:4f4ba7a6e606
   | |  user:test
   | |  date:Thu Jan 01 00:00:00 1970 +
   | |  summary: D1
   | |
-  | o  changeset:   4:af23416e619b
+  | o  changeset:   4:43e4b415492d
   |/   parent:  0:11f7a1b56675
   |user:test
   |date:Thu Jan 01 00:00:00 1970 +
@@ -1029,6 +1041,7 @@
   other [graft] changed f1b which local [local] deleted
   use (c)hanged version, leave (d)eleted, or leave (u)nresolved? u
   merging f2a and f2c to f2c
+  merging f5b and f5a to f5a
   abort: unresolved conflicts, can't continue
   (use 'hg resolve' and 'hg graft --continue')
   [255]
@@ -1037,7 +1050,8 @@
   continue: hg graft --continue
   $ hg graft --continue # XXX work around failure
   grafting 2:f58c7e2b28fa "C0"
-  grafting 4:af23416e619b "C0"
+  grafting 4:43e4b415492d "C0"
+  merging f1e and f1a to f1e
   

[PATCH 08 of 12 v2] checkcopies: handle divergences contained entirely in tca::ctx

[Gábor Stefanik]

# HG changeset patch
# User Gábor Stefanik 
# Date 1476266043 -7200
#  Wed Oct 12 11:54:03 2016 +0200
# Node ID 73b55839aca7019d88cc9885e43244fb1e6d68ab
# Parent  e61355cb337ab5328e4cf5d0f153821600c53e72
checkcopies: handle divergences contained entirely in tca::ctx

During a graftlike merge, _checkcopies runs from ctx to tca, possibly
passing over the merge base. If there is a rename both before and after
the base, then we're actually dealing with divergent renames.
If there is no rename on the other side of tca, then the divergence is
contained entirely in the range of one _checkcopies invocation, and
should be detected "in the loop" without having to rely on the other
_checkcopies pass.

diff -r e61355cb337a -r 73b55839aca7 mercurial/copies.py
--- a/mercurial/copies.py   Thu Oct 13 02:19:43 2016 +0200
+++ b/mercurial/copies.py   Wed Oct 12 11:54:03 2016 +0200
@@ -564,9 +564,8 @@
 # traversed backwards.
 #
 # In the case there is both backward and forward renames (before and after
-# the base) this is more complicated as we must detect a divergence. This
-# is currently broken and hopefully some later code update will make that
-# work (we use 'backwards = False' in that case)
+# the base) this is more complicated as we must detect a divergence.
+# We use 'backwards = False' in that case.
 backwards = base != tca and f in mb
 getfctx = _makegetfctx(ctx)
 
@@ -600,6 +599,12 @@
 data['copy'][of] = f
 elif of in mb:
 data['copy'][f] = of
+else: # divergence w.r.t. graft CA on one side of topological CA
+for sf in seen:
+if sf in mb:
+assert sf not in data['diverge']
+data['diverge'][sf] = [f, of]
+break
 return
 
 if of in mb:
diff -r e61355cb337a -r 73b55839aca7 tests/test-graft.t
--- a/tests/test-graft.tThu Oct 13 02:19:43 2016 +0200
+++ b/tests/test-graft.tWed Oct 12 11:54:03 2016 +0200
@@ -982,6 +982,9 @@
 
   $ HGEDITOR="echo D1 >" hg graft -r 'desc("D0")' --edit
   grafting 3:b69f5839d2d9 "D0"
+  note: possible conflict - f3b was renamed multiple times to:
+   f3d
+   f3a
   warning: can't find ancestor for 'f3d' copied from 'f3b'!
 
 Set up the repository for some further tests
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 09 of 12 v2] mergecopies: add logic to process incomplete data

[Gábor Stefanik]

# HG changeset patch
# User Gábor Stefanik 
# Date 1475578314 -7200
#  Tue Oct 04 12:51:54 2016 +0200
# Node ID 9fc636d9950feb76af774f881ea179037231e697
# Parent  73b55839aca7019d88cc9885e43244fb1e6d68ab
mergecopies: add logic to process incomplete data

We first combine incomplete copies on the two sides of the topological CA
into complete copies.
Any leftover incomplete copies are then combined with the incomplete
divergences to reconstruct divergences spanning over the topological CA.
Finally we promote any divergences falsely flagged as incomplete to full
divergences.

Right now, there is nothing generating incomplete copy/divergence data,
so this code does nothing. Changes to _checkcopies to populate these
dicts are coming later in this series.

diff -r 73b55839aca7 -r 9fc636d9950f mercurial/copies.py
--- a/mercurial/copies.py   Wed Oct 12 11:54:03 2016 +0200
+++ b/mercurial/copies.py   Tue Oct 04 12:51:54 2016 +0200
@@ -289,6 +289,22 @@
 return fctx
 return util.lrucachefunc(makectx)
 
+def _combinecopies(copyfrom, copyto, finalcopy, diverge, incompletediverge):
+"""combine partial copy paths"""
+remainder = {}
+for f in copyfrom:
+if f in copyto:
+finalcopy[copyto[f]] = copyfrom[f]
+del copyto[f]
+for f in incompletediverge:
+assert f not in diverge
+ic = incompletediverge[f]
+if ic[0] in copyto:
+diverge[f] = [copyto[ic[0]], ic[1]]
+else:
+remainder[f] = ic
+return remainder
+
 def mergecopies(repo, c1, c2, base):
 """
 Find moves and copies between context c1 and c2 that are relevant
@@ -360,14 +376,21 @@
 # - diverge = record all diverges in this dict
 # - copy = record all non-divergent copies in this dict
 # - fullcopy = record all copies in this dict
+# - incomplete = record non-divergent partial copies here
+# - incompletediverge = record divergent partial copies here
 diverge = {} # divergence data is shared
+incompletediverge  = {}
 data1 = {'copy': {},
  'fullcopy': {},
+ 'incomplete': {},
  'diverge': diverge,
+ 'incompletediverge': incompletediverge,
 }
 data2 = {'copy': {},
  'fullcopy': {},
+ 'incomplete': {},
  'diverge': diverge,
+ 'incompletediverge': incompletediverge,
 }
 
 # find interesting file sets from manifests
@@ -398,6 +421,13 @@
 copy = dict(data1['copy'].items() + data2['copy'].items())
 fullcopy = dict(data1['fullcopy'].items() + data2['fullcopy'].items())
 
+if dirtyc1:
+_combinecopies(data2['incomplete'], data1['incomplete'], copy, diverge,
+   incompletediverge)
+else:
+_combinecopies(data1['incomplete'], data2['incomplete'], copy, diverge,
+   incompletediverge)
+
 renamedelete = {}
 renamedeleteset = set()
 divergeset = set()
@@ -416,13 +446,36 @@
 repo.ui.debug("  unmatched files new in both:\n   %s\n"
   % "\n   ".join(bothnew))
 bothdiverge = {}
-bothdata = {'copy': {},
-'fullcopy': {},
-'diverge': bothdiverge,
-   }
+bothincompletediverge = {}
+both1 = {'copy': {},
+ 'fullcopy': {},
+ 'incomplete': {},
+ 'diverge': bothdiverge,
+ 'incompletediverge': bothincompletediverge
+}
+both2 = {'copy': {},
+ 'fullcopy': {},
+ 'incomplete': {},
+ 'diverge': bothdiverge,
+ 'incompletediverge': bothincompletediverge
+}
 for f in bothnew:
-_checkcopies(c1, f, m1, m2, base, tca, limit, bothdata)
-_checkcopies(c2, f, m2, m1, base, tca, limit, bothdata)
+_checkcopies(c1, f, m1, m2, base, tca, limit, both1)
+_checkcopies(c2, f, m2, m1, base, tca, limit, both2)
+if dirtyc1:
+assert both2['incomplete'] == {}
+remainder = _combinecopies({}, both1['incomplete'], copy, bothdiverge,
+   bothincompletediverge)
+else:
+assert both1['incomplete'] == {}
+remainder = _combinecopies({}, both2['incomplete'], copy, bothdiverge,
+   bothincompletediverge)
+for f in remainder:
+assert f not in bothdiverge
+ic = remainder[f]
+if ic[0] in (m1 if dirtyc1 else m2):
+# backed-out rename on one side, but watch out for deleted files
+bothdiverge[f] = ic
 for of, fl in bothdiverge.items():
 if len(fl) == 2 and fl[0] == fl[1]:
 copy[fl[0]] = of # not actually divergent, just matching renames
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 04 of 12 v2] copies: compute a suitable TCA if base turns out to be unsuitable

[Gábor Stefanik]

# HG changeset patch
# User Gábor Stefanik 
# Date 1476317029 -7200
#  Thu Oct 13 02:03:49 2016 +0200
# Node ID 1f91343556e7cef7a71edf512b02f7e0f09d5e9b
# Parent  981a622717343796f65b5858f4d4b89f81cb437a
copies: compute a suitable TCA if base turns out to be unsuitable


This will be used later in an update to _checkcopies.

(Pierre-Yves David was involved in the cleanup of this patch.)

diff -r 981a62271734 -r 1f91343556e7 mercurial/copies.py
--- a/mercurial/copies.py   Thu Oct 13 01:47:33 2016 +0200
+++ b/mercurial/copies.py   Thu Oct 13 02:03:49 2016 +0200
@@ -338,6 +338,10 @@
 dirtyc1 = not (base == _c1 or base.descendant(_c1))
 dirtyc2 = not (base== _c2 or base.descendant(_c2))
 graft = dirtyc1 or dirtyc2
+tca = base
+if graft:
+tca = _c1.ancestor(_c2)
+
 limit = _findlimit(repo, c1.rev(), c2.rev())
 if limit is None:
 # no common ancestor, no copies
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 06 of 12 v2] copies: make it possible to distinguish betwen _computenonoverlap invocations

[Gábor Stefanik]

# HG changeset patch
# User Pierre-Yves David 
# Date 1476372109 -7200
#  Thu Oct 13 17:21:49 2016 +0200
# Node ID 72b4f2514efd9decffb1115c0f7b8e8f004bdf44
# Parent  48dba8d1f7fab137418bfbb833c3096969eec934
copies: make it possible to distinguish betwen _computenonoverlap invocations

_computenonoverlap needs to be invoked twice during a graft, and debugging
messages should be distinguishable between the two invocations

diff -r 48dba8d1f7fa -r 72b4f2514efd mercurial/copies.py
--- a/mercurial/copies.py   Thu Oct 13 02:03:54 2016 +0200
+++ b/mercurial/copies.py   Thu Oct 13 17:21:49 2016 +0200
@@ -231,23 +231,27 @@
 return _chain(x, y, _backwardrenames(x, a),
   _forwardcopies(a, y, match=match))
 
-def _computenonoverlap(repo, c1, c2, addedinm1, addedinm2):
+def _computenonoverlap(repo, c1, c2, addedinm1, addedinm2, baselabel=''):
 """Computes, based on addedinm1 and addedinm2, the files exclusive to c1
 and c2. This is its own function so extensions can easily wrap this call
 to see what files mergecopies is about to process.
 
 Even though c1 and c2 are not used in this function, they are useful in
 other extensions for being able to read the file nodes of the changed 
files.
+
+"baselabel" can be passed to help distinguish the multiple computations
+done in the graft case.
 """
 u1 = sorted(addedinm1 - addedinm2)
 u2 = sorted(addedinm2 - addedinm1)
 
+header = "  unmatched files in %s"
+if baselabel:
+header += ' (from %s)' % baselabel
 if u1:
-repo.ui.debug("  unmatched files in local:\n   %s\n"
-  % "\n   ".join(u1))
+repo.ui.debug("%s:\n   %s\n" % (header % 'local', "\n   ".join(u1)))
 if u2:
-repo.ui.debug("  unmatched files in other:\n   %s\n"
-  % "\n   ".join(u2))
+repo.ui.debug("%s:\n   %s\n" % (header % 'other', "\n   ".join(u2)))
 return u1, u2
 
 def _makegetfctx(ctx):
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 2 of 2 V3] py3: use namedtuple._replace to produce new tokens

[Pierre-Yves David]

On 10/16/2016 04:48 PM, Yuya Nishihara wrote:

On Sun, 16 Oct 2016 16:30:04 +0200, Pierre-Yves David wrote:

On 10/14/2016 06:55 PM, Martijn Pieters wrote:

# HG changeset patch
# User Martijn Pieters 
# Date 1476347257 -3600
#  Thu Oct 13 09:27:37 2016 +0100
# Node ID d20dd7db86044bdca79825499b913840d726d841
# Parent  9031460519503abe5dc430c8ece29d198121cd65
py3: use namedtuple._replace to produce new tokens


We seems to be using a private function of some stdlib type?
Can you elaborate on why this is a good move?


It is a public function. All public functions of namedtuple start with '_'
to avoid conflicts with field names.


Ha, much confusion.


--
Pierre-Yves David
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 2 of 2 V3] py3: use namedtuple._replace to produce new tokens

[Yuya Nishihara]

On Sun, 16 Oct 2016 16:30:04 +0200, Pierre-Yves David wrote:
> On 10/14/2016 06:55 PM, Martijn Pieters wrote:
> > # HG changeset patch
> > # User Martijn Pieters 
> > # Date 1476347257 -3600
> > #  Thu Oct 13 09:27:37 2016 +0100
> > # Node ID d20dd7db86044bdca79825499b913840d726d841
> > # Parent  9031460519503abe5dc430c8ece29d198121cd65
> > py3: use namedtuple._replace to produce new tokens
> 
> We seems to be using a private function of some stdlib type?
> Can you elaborate on why this is a good move?

It is a public function. All public functions of namedtuple start with '_'
to avoid conflicts with field names.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 3 of 4] copies: move variable document from checkcopies to mergecopies

[Pierre-Yves David]

# HG changeset patch
# User Pierre-Yves David 
# Date 1476314793 -7200
#  Thu Oct 13 01:26:33 2016 +0200
# Node ID 6f0dc570c4383ad42c572158af200d2142bbef4f
# Parent  823b3ea33d263ae3179cbbcf39e0ddfc7a938ce5
# EXP-Topic checkcopies
copies: move variable document from checkcopies to mergecopies

It appears that 'mergecopies' is the function consuming these data so we move
the documentation there.

diff --git a/mercurial/copies.py b/mercurial/copies.py
--- a/mercurial/copies.py
+++ b/mercurial/copies.py
@@ -331,7 +331,10 @@ def mergecopies(repo, c1, c2, ca):
 m2 = c2.manifest()
 ma = ca.manifest()
 
-# see _checkcopies documentation below for these dicts
+# gather data from _checkcopies:
+# - diverge = record all diverges in this dict
+# - copy = record all non-divergent copies in this dict
+# - fullcopy = record all copies in this dict
 diverge = {} # divergence data is shared
 data1 = {'copy': {},
  'fullcopy': {},
@@ -507,10 +510,7 @@ def _checkcopies(ctx, f, m1, m2, base, l
 m2 = the destination manifest
 base = the changectx used as a merge base
 limit = the rev number to not search beyond
-data = dictionary of dictionary to store copy data. The keys are:
-- diverge = record all diverges in this dict
-- copy = record all non-divergent copies in this dict
-- fullcopy = record all copies in this dict
+data = dictionary of dictionary to store copy data. (see mergecopies)
 
 note: limit is only an optimization, and there is no guarantee that
 irrelevant revisions will not be limited
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 4 of 4] mergecopies: rename 'ca' to 'base'

[Pierre-Yves David]

# HG changeset patch
# User Pierre-Yves David 
# Date 1476315014 -7200
#  Thu Oct 13 01:30:14 2016 +0200
# Node ID b8bb58f87a94f5b78beaa1db710c5ef69967dd4c
# Parent  6f0dc570c4383ad42c572158af200d2142bbef4f
# EXP-Topic checkcopies
mergecopies: rename 'ca' to 'base'

This variable was named after the common ancestor. It is actually the merge
base that might differ from the common ancestor in the graft case. We rename the
variable before a larger refactoring to clarify the situation. Similar rename
was also applied to 'checkcopies' in a prior changeset.

diff --git a/mercurial/copies.py b/mercurial/copies.py
--- a/mercurial/copies.py
+++ b/mercurial/copies.py
@@ -285,14 +285,14 @@ def _makegetfctx(ctx):
 return fctx
 return util.lrucachefunc(makectx)
 
-def mergecopies(repo, c1, c2, ca):
+def mergecopies(repo, c1, c2, base):
 """
 Find moves and copies between context c1 and c2 that are relevant
-for merging.
+for merging. 'base' will be used as the merge base.
 
 Returns four dicts: "copy", "movewithdir", "diverge", and
 "renamedelete".
-
+`
 "copy" is a mapping from destination name -> source name,
 where source is in c1 and destination is in c2 or vice-versa.
 
@@ -329,7 +329,7 @@ def mergecopies(repo, c1, c2, ca):
 
 m1 = c1.manifest()
 m2 = c2.manifest()
-ma = ca.manifest()
+mb = base.manifest()
 
 # gather data from _checkcopies:
 # - diverge = record all diverges in this dict
@@ -346,17 +346,17 @@ def mergecopies(repo, c1, c2, ca):
 }
 
 # find interesting file sets from manifests
-addedinm1 = m1.filesnotin(ma)
-addedinm2 = m2.filesnotin(ma)
+addedinm1 = m1.filesnotin(mb)
+addedinm2 = m2.filesnotin(mb)
 u1r, u2r = _computenonoverlap(repo, c1, c2, addedinm1, addedinm2)
 u1u, u2u = u1r, u2r
 bothnew = sorted(addedinm1 & addedinm2)
 
 for f in u1u:
-_checkcopies(c1, f, m1, m2, ca, limit, data1)
+_checkcopies(c1, f, m1, m2, base, limit, data1)
 
 for f in u2u:
-_checkcopies(c2, f, m2, m1, ca, limit, data2)
+_checkcopies(c2, f, m2, m1, base, limit, data2)
 
 copy = dict(data1['copy'].items() + data2['copy'].items())
 fullcopy = dict(data1['fullcopy'].items() + data2['fullcopy'].items())
@@ -384,8 +384,8 @@ def mergecopies(repo, c1, c2, ca):
 'diverge': bothdiverge,
}
 for f in bothnew:
-_checkcopies(c1, f, m1, m2, ca, limit, bothdata)
-_checkcopies(c2, f, m2, m1, ca, limit, bothdata)
+_checkcopies(c1, f, m1, m2, base, limit, bothdata)
+_checkcopies(c2, f, m2, m1, base, limit, bothdata)
 for of, fl in bothdiverge.items():
 if len(fl) == 2 and fl[0] == fl[1]:
 copy[fl[0]] = of # not actually divergent, just matching renames
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 4] checkcopies: pass data as a dictionary of dictionaries

[Pierre-Yves David]

# HG changeset patch
# User Pierre-Yves David 
# Date 1476145302 -7200
#  Tue Oct 11 02:21:42 2016 +0200
# Node ID 823b3ea33d263ae3179cbbcf39e0ddfc7a938ce5
# Parent  ea295754eb9a275b61a17e60aa4384dea2544d86
# EXP-Topic checkcopies
checkcopies: pass data as a dictionary of dictionaries

more are coming

diff --git a/mercurial/copies.py b/mercurial/copies.py
--- a/mercurial/copies.py
+++ b/mercurial/copies.py
@@ -332,9 +332,15 @@ def mergecopies(repo, c1, c2, ca):
 ma = ca.manifest()
 
 # see _checkcopies documentation below for these dicts
-copy1, copy2 = {}, {}
-fullcopy1, fullcopy2 = {}, {}
-diverge = {}
+diverge = {} # divergence data is shared
+data1 = {'copy': {},
+ 'fullcopy': {},
+ 'diverge': diverge,
+}
+data2 = {'copy': {},
+ 'fullcopy': {},
+ 'diverge': diverge,
+}
 
 # find interesting file sets from manifests
 addedinm1 = m1.filesnotin(ma)
@@ -344,13 +350,13 @@ def mergecopies(repo, c1, c2, ca):
 bothnew = sorted(addedinm1 & addedinm2)
 
 for f in u1u:
-_checkcopies(c1, f, m1, m2, ca, limit, diverge, copy1, fullcopy1)
+_checkcopies(c1, f, m1, m2, ca, limit, data1)
 
 for f in u2u:
-_checkcopies(c2, f, m2, m1, ca, limit, diverge, copy2, fullcopy2)
+_checkcopies(c2, f, m2, m1, ca, limit, data2)
 
-copy = dict(copy1.items() + copy2.items())
-fullcopy = dict(fullcopy1.items() + fullcopy2.items())
+copy = dict(data1['copy'].items() + data2['copy'].items())
+fullcopy = dict(data1['fullcopy'].items() + data2['fullcopy'].items())
 
 renamedelete = {}
 renamedeleteset = set()
@@ -369,10 +375,14 @@ def mergecopies(repo, c1, c2, ca):
 if bothnew:
 repo.ui.debug("  unmatched files new in both:\n   %s\n"
   % "\n   ".join(bothnew))
-bothdiverge, _copy, _fullcopy = {}, {}, {}
+bothdiverge = {}
+bothdata = {'copy': {},
+'fullcopy': {},
+'diverge': bothdiverge,
+   }
 for f in bothnew:
-_checkcopies(c1, f, m1, m2, ca, limit, bothdiverge, _copy, _fullcopy)
-_checkcopies(c2, f, m2, m1, ca, limit, bothdiverge, _copy, _fullcopy)
+_checkcopies(c1, f, m1, m2, ca, limit, bothdata)
+_checkcopies(c2, f, m2, m1, ca, limit, bothdata)
 for of, fl in bothdiverge.items():
 if len(fl) == 2 and fl[0] == fl[1]:
 copy[fl[0]] = of # not actually divergent, just matching renames
@@ -487,7 +497,7 @@ def _related(f1, f2, limit):
 except StopIteration:
 return False
 
-def _checkcopies(ctx, f, m1, m2, base, limit, diverge, copy, fullcopy):
+def _checkcopies(ctx, f, m1, m2, base, limit, data):
 """
 check possible copies of f from m1 to m2
 
@@ -497,9 +507,10 @@ def _checkcopies(ctx, f, m1, m2, base, l
 m2 = the destination manifest
 base = the changectx used as a merge base
 limit = the rev number to not search beyond
-diverge = record all diverges in this dict
-copy = record all non-divergent copies in this dict
-fullcopy = record all copies in this dict
+data = dictionary of dictionary to store copy data. The keys are:
+- diverge = record all diverges in this dict
+- copy = record all non-divergent copies in this dict
+- fullcopy = record all copies in this dict
 
 note: limit is only an optimization, and there is no guarantee that
 irrelevant revisions will not be limited
@@ -522,7 +533,7 @@ def _checkcopies(ctx, f, m1, m2, base, l
 continue
 seen.add(of)
 
-fullcopy[f] = of # remember for dir rename detection
+data['fullcopy'][f] = of # remember for dir rename detection
 if of not in m2:
 continue # no match, keep looking
 if m2[of] == mb.get(of):
@@ -532,11 +543,11 @@ def _checkcopies(ctx, f, m1, m2, base, l
 # unrelated to the droids we are looking for.
 cr = _related(oc, c2, base.rev())
 if cr and (of == f or of == c2.path()): # non-divergent
-copy[f] = of
+data['copy'][f] = of
 return
 
 if of in mb:
-diverge.setdefault(of, []).append(f)
+data['diverge'].setdefault(of, []).append(f)
 
 def duplicatecopies(repo, rev, fromrev, skiprev=None):
 '''reproduce copies from fromrev to rev in the dirstate
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 1 of 2] evolve: indent cmdnext and cmdprev ready for locking change (issue5244)

[Pierre-Yves David]

On 10/16/2016 04:25 PM, Simon Farnsworth wrote:

# HG changeset patch
# User Simon Farnsworth 
# Date 1475939661 25200
#  Sat Oct 08 08:14:21 2016 -0700
# Node ID ee284d7c5faa5d9f17853f51c17883844b985c58
# Parent  5383671ef612a1764bbbed13a7ef2d339d0a9c2d
evolve: indent cmdnext and cmdprev ready for locking change (issue5244)


Looks like an aborted V2. Are you aware of the --confirm flag ?

(and the associated patchbomb.confirm=True config)

--
Pierre-Yves David
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 04 of 16] mergecopies: rename 'ca' to 'base'

[Pierre-Yves David]

On 10/16/2016 04:32 PM, Gábor Stefanik wrote:

# HG changeset patch
# User Pierre-Yves David 
# Date 1476315014 -7200
#  Thu Oct 13 01:30:14 2016 +0200
# Node ID bfb99ef3749757161bf439cc0ec678de46e24672
# Parent  92dfc856c6cc25366eb8a7fb9b59d992ad5efc36
mergecopies: rename 'ca' to 'base'

This variable was named after the common ancestor. It is actually the merge
base that might differ from the common ancestor in the graft case. We rename the
variable before a larger refactoring to clarify the situation. Similar rename
was also applied to 'checkcopies' in a prior changeset.


Hum I though I already got these one in, but I did not. Let me emails 
these first four myself for clarity.


Cheers,

--
Pierre-Yves David
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

RE: [PATCH 07 of 16] copies: detect graft-like merges penis

[Gábor STEFANIK]

>


--
This message, including its attachments, is confidential. For more information 
please read NNG's email policy here:
http://www.nng.com/emailpolicy/
By responding to this email you accept the email policy.


-Original Message-
> From: Mercurial-devel [mailto:mercurial-devel-boun...@mercurial-scm.org]
> On Behalf Of Gábor Stefanik
> Sent: Sunday, October 16, 2016 4:30 PM
> To: mercurial-devel@mercurial-scm.org
> Subject: [PATCH 07 of 16] copies: detect graft-like merges penis
>
> # HG changeset patch
> # User Gábor Stefanik  # Date 1476316053 -7200
> #  Thu Oct 13 01:47:33 2016 +0200
> # Node ID b3631f615bc2fff5cbce4cde7373dde052197a0e
> # Parent  7f9a6f30730074c869e5cf9b77c53929b06c4fb3
> copies: detect graft-like merges penis

Sorry! That was a random change to make histedit generate a new version. Please 
disregard,
new version coming soon.

>
> Right now, nothing changes as a result of this, but we want to handle grafts
> differently from ordinary merges later.
>
> (Series developed together with Pierre-Yves David)
>
> diff -r 7f9a6f307300 -r b3631f615bc2 mercurial/copies.py
> --- a/mercurial/copies.pyWed Oct 12 12:41:28 2016 +0200
> +++ b/mercurial/copies.pyThu Oct 13 01:47:33 2016 +0200
> @@ -321,6 +321,23 @@
>  if repo.ui.configbool('experimental', 'disablecopytrace'):
>  return {}, {}, {}, {}
>
> +# In certain scenarios (e.g. graft, update or rebase), base can be
> +# overridden We still need to know a real common ancestor in this case
> We
> +# can't just compute _c1.ancestor(_c2) and compare it to ca, because
> there
> +# can be multiple common ancestors, e.g. in case of bidmerge.  Because
> our
> +# caller may not know if the revision passed in lieu of the CA is a 
> genuine
> +# common ancestor or not without explicitly checking it, it's better to
> +# determine that here.
> +#
> +# base.descendant(wc) and base.descendant(base) are False, work
> around that
> +_c1 = c1.p1() if c1.rev() is None else c1
> +_c2 = c2.p1() if c2.rev() is None else c2
> +# an endpoint is "dirty" if it isn't a descendant of the merge base
> +# if we have a dirty endpoint, we need to trigger graft logic, and also
> +# keep track of which endpoint is dirty
> +dirtyc1 = not (base == _c1 or base.descendant(_c1))
> +dirtyc2 = not (base== _c2 or base.descendant(_c2))
> +graft = dirtyc1 or dirtyc2
>  limit = _findlimit(repo, c1.rev(), c2.rev())
>  if limit is None:
>  # no common ancestor, no copies
> ___
> Mercurial-devel mailing list
> Mercurial-devel@mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 11 of 16] mergecopies: invoke _computenonoverlap for both base and tca during merges

[Gábor Stefanik]

# HG changeset patch
# User Gábor Stefanik 
# Date 1476317983 -7200
#  Thu Oct 13 02:19:43 2016 +0200
# Node ID e61355cb337ab5328e4cf5d0f153821600c53e72
# Parent  72b4f2514efd9decffb1115c0f7b8e8f004bdf44
mergecopies: invoke _computenonoverlap for both base and tca during merges

The algorithm of _checkcopies can only walk backwards in the DAG, never
forward. Because of this, the two _checkcopies patches need to run from
their respective endpoints to the TCA to cover the entire subgraph where
the merge is being performed. However, detection of files new in both
endpoints, as well as directory rename detection, need to run with respect
to the merge base, so we need lists of new files both from the TCA's and
the merge base's viewpoint to correctly detect renames in a graft-like
merge scenario.

(Series reworked by Pierre-Yves David)

diff -r 72b4f2514efd -r e61355cb337a mercurial/copies.py
--- a/mercurial/copies.py   Thu Oct 13 17:21:49 2016 +0200
+++ b/mercurial/copies.py   Thu Oct 13 02:19:43 2016 +0200
@@ -373,9 +373,21 @@
 # find interesting file sets from manifests
 addedinm1 = m1.filesnotin(mb)
 addedinm2 = m2.filesnotin(mb)
-u1r, u2r = _computenonoverlap(repo, c1, c2, addedinm1, addedinm2)
-u1u, u2u = u1r, u2r
 bothnew = sorted(addedinm1 & addedinm2)
+if tca == base:
+# unmatched file from base
+u1r, u2r = _computenonoverlap(repo, c1, c2, addedinm1, addedinm2)
+u1u, u2u = u1r, u2r
+else:
+# unmatched file from base (DAG rotation in the graft case)
+u1r, u2r = _computenonoverlap(repo, c1, c2, addedinm1, addedinm2,
+  baselabel='base')
+# unmatched file from topological common ancestors (no DAG rotation)
+# need to recompute this for directory move handling when grafting
+mta = tca.manifest()
+u1u, u2u = _computenonoverlap(repo, c1, c2, m1.filesnotin(mta),
+m2.filesnotin(mta),
+  baselabel='topological common ancestor')
 
 for f in u1u:
 _checkcopies(c1, f, m1, m2, base, tca, limit, data1)
diff -r 72b4f2514efd -r e61355cb337a tests/test-graft.t
--- a/tests/test-graft.tThu Oct 13 17:21:49 2016 +0200
+++ b/tests/test-graft.tThu Oct 13 02:19:43 2016 +0200
@@ -179,6 +179,11 @@
   committing changelog
   grafting 5:97f8bfe72746 "5"
 searching for copies back to rev 1
+unmatched files in other (from topological common ancestor):
+ c
+all copies found (* = to merge, ! = divergent, % = renamed and deleted):
+ src: 'c' -> dst: 'b' *
+checking for directory renames
   resolving manifests
branchmerge: True, force: True, partial: False
ancestor: 4c60f11aa304, local: 6b9e5368ca4e+, remote: 97f8bfe72746
@@ -193,6 +198,11 @@
   scanning for duplicate grafts
   grafting 4:9c233e8e184d "4"
 searching for copies back to rev 1
+unmatched files in other (from topological common ancestor):
+ c
+all copies found (* = to merge, ! = divergent, % = renamed and deleted):
+ src: 'c' -> dst: 'b' *
+checking for directory renames
   resolving manifests
branchmerge: True, force: True, partial: False
ancestor: 4c60f11aa304, local: 1905859650ec+, remote: 9c233e8e184d
@@ -945,6 +955,7 @@
   $ hg up -q 'desc("A0")'
   $ HGEDITOR="echo C1 >" hg graft -r 'desc("C0")' --edit
   grafting 2:f58c7e2b28fa "C0"
+  merging f1a and f1b to f1a
   merging f5a
   warning: conflicts while merging f5a! (edit, then use 'hg resolve --mark')
   abort: unresolved conflicts, can't continue
@@ -957,14 +968,15 @@
   grafting 2:f58c7e2b28fa "C0"
   warning: can't find ancestor for 'f5a' copied from 'f5b'!
   $ hg status --change .
+  M f1a
   M f5a
-  A f1b
   A f2c
   R f2a
   $ hg cat f1a
-  c1a
+  c1c
   $ hg cat f1b
-  c1c
+  f1b: no such file in rev 43e4b415492d
+  [1]
 
 Test the cases A.0 (f4x) and A.6 (f3x)
 
@@ -989,12 +1001,12 @@
   |  date:Thu Jan 01 00:00:00 1970 +
   |  summary: E0
   |
-  | o  changeset:   5:573bb6b4b56d
+  | o  changeset:   5:4f4ba7a6e606
   | |  user:test
   | |  date:Thu Jan 01 00:00:00 1970 +
   | |  summary: D1
   | |
-  | o  changeset:   4:af23416e619b
+  | o  changeset:   4:43e4b415492d
   |/   parent:  0:11f7a1b56675
   |user:test
   |date:Thu Jan 01 00:00:00 1970 +
@@ -1029,6 +1041,7 @@
   other [graft] changed f1b which local [local] deleted
   use (c)hanged version, leave (d)eleted, or leave (u)nresolved? u
   merging f2a and f2c to f2c
+  merging f5b and f5a to f5a
   abort: unresolved conflicts, can't continue
   (use 'hg resolve' and 'hg graft --continue')
   [255]
@@ -1037,7 +1050,8 @@
   continue: hg graft --continue
   $ hg graft --continue # XXX work around failure
   grafting 2:f58c7e2b28fa "C0"
-  grafting 4:af23416e619b "C0"
+  grafting 4:43e4b415492d "C0"
+  merging f1e and f1a to f1e
   

[PATCH 13 of 16] mergecopies: add logic to process incomplete data

[Gábor Stefanik]

# HG changeset patch
# User Gábor Stefanik 
# Date 1475578314 -7200
#  Tue Oct 04 12:51:54 2016 +0200
# Node ID 9fc636d9950feb76af774f881ea179037231e697
# Parent  73b55839aca7019d88cc9885e43244fb1e6d68ab
mergecopies: add logic to process incomplete data

We first combine incomplete copies on the two sides of the topological CA
into complete copies.
Any leftover incomplete copies are then combined with the incomplete
divergences to reconstruct divergences spanning over the topological CA.
Finally we promote any divergences falsely flagged as incomplete to full
divergences.

Right now, there is nothing generating incomplete copy/divergence data,
so this code does nothing. Changes to _checkcopies to populate these
dicts are coming later in this series.

diff -r 73b55839aca7 -r 9fc636d9950f mercurial/copies.py
--- a/mercurial/copies.py   Wed Oct 12 11:54:03 2016 +0200
+++ b/mercurial/copies.py   Tue Oct 04 12:51:54 2016 +0200
@@ -289,6 +289,22 @@
 return fctx
 return util.lrucachefunc(makectx)
 
+def _combinecopies(copyfrom, copyto, finalcopy, diverge, incompletediverge):
+"""combine partial copy paths"""
+remainder = {}
+for f in copyfrom:
+if f in copyto:
+finalcopy[copyto[f]] = copyfrom[f]
+del copyto[f]
+for f in incompletediverge:
+assert f not in diverge
+ic = incompletediverge[f]
+if ic[0] in copyto:
+diverge[f] = [copyto[ic[0]], ic[1]]
+else:
+remainder[f] = ic
+return remainder
+
 def mergecopies(repo, c1, c2, base):
 """
 Find moves and copies between context c1 and c2 that are relevant
@@ -360,14 +376,21 @@
 # - diverge = record all diverges in this dict
 # - copy = record all non-divergent copies in this dict
 # - fullcopy = record all copies in this dict
+# - incomplete = record non-divergent partial copies here
+# - incompletediverge = record divergent partial copies here
 diverge = {} # divergence data is shared
+incompletediverge  = {}
 data1 = {'copy': {},
  'fullcopy': {},
+ 'incomplete': {},
  'diverge': diverge,
+ 'incompletediverge': incompletediverge,
 }
 data2 = {'copy': {},
  'fullcopy': {},
+ 'incomplete': {},
  'diverge': diverge,
+ 'incompletediverge': incompletediverge,
 }
 
 # find interesting file sets from manifests
@@ -398,6 +421,13 @@
 copy = dict(data1['copy'].items() + data2['copy'].items())
 fullcopy = dict(data1['fullcopy'].items() + data2['fullcopy'].items())
 
+if dirtyc1:
+_combinecopies(data2['incomplete'], data1['incomplete'], copy, diverge,
+   incompletediverge)
+else:
+_combinecopies(data1['incomplete'], data2['incomplete'], copy, diverge,
+   incompletediverge)
+
 renamedelete = {}
 renamedeleteset = set()
 divergeset = set()
@@ -416,13 +446,36 @@
 repo.ui.debug("  unmatched files new in both:\n   %s\n"
   % "\n   ".join(bothnew))
 bothdiverge = {}
-bothdata = {'copy': {},
-'fullcopy': {},
-'diverge': bothdiverge,
-   }
+bothincompletediverge = {}
+both1 = {'copy': {},
+ 'fullcopy': {},
+ 'incomplete': {},
+ 'diverge': bothdiverge,
+ 'incompletediverge': bothincompletediverge
+}
+both2 = {'copy': {},
+ 'fullcopy': {},
+ 'incomplete': {},
+ 'diverge': bothdiverge,
+ 'incompletediverge': bothincompletediverge
+}
 for f in bothnew:
-_checkcopies(c1, f, m1, m2, base, tca, limit, bothdata)
-_checkcopies(c2, f, m2, m1, base, tca, limit, bothdata)
+_checkcopies(c1, f, m1, m2, base, tca, limit, both1)
+_checkcopies(c2, f, m2, m1, base, tca, limit, both2)
+if dirtyc1:
+assert both2['incomplete'] == {}
+remainder = _combinecopies({}, both1['incomplete'], copy, bothdiverge,
+   bothincompletediverge)
+else:
+assert both1['incomplete'] == {}
+remainder = _combinecopies({}, both2['incomplete'], copy, bothdiverge,
+   bothincompletediverge)
+for f in remainder:
+assert f not in bothdiverge
+ic = remainder[f]
+if ic[0] in (m1 if dirtyc1 else m2):
+# backed-out rename on one side, but watch out for deleted files
+bothdiverge[f] = ic
 for of, fl in bothdiverge.items():
 if len(fl) == 2 and fl[0] == fl[1]:
 copy[fl[0]] = of # not actually divergent, just matching renames
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 08 of 16] copies: compute a suitable TCA if base turns out to be unsuitable

[Gábor Stefanik]

# HG changeset patch
# User Gábor Stefanik 
# Date 1476317029 -7200
#  Thu Oct 13 02:03:49 2016 +0200
# Node ID 1f91343556e7cef7a71edf512b02f7e0f09d5e9b
# Parent  981a622717343796f65b5858f4d4b89f81cb437a
copies: compute a suitable TCA if base turns out to be unsuitable


This will be used later in an update to _checkcopies.

(Pierre-Yves David was involved in the cleanup of this patch.)

diff -r 981a62271734 -r 1f91343556e7 mercurial/copies.py
--- a/mercurial/copies.py   Thu Oct 13 01:47:33 2016 +0200
+++ b/mercurial/copies.py   Thu Oct 13 02:03:49 2016 +0200
@@ -338,6 +338,10 @@
 dirtyc1 = not (base == _c1 or base.descendant(_c1))
 dirtyc2 = not (base== _c2 or base.descendant(_c2))
 graft = dirtyc1 or dirtyc2
+tca = base
+if graft:
+tca = _c1.ancestor(_c2)
+
 limit = _findlimit(repo, c1.rev(), c2.rev())
 if limit is None:
 # no common ancestor, no copies
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 06 of 16] tests: introduce tests for grafting through renames

[Gábor Stefanik]

# HG changeset patch
# User Gábor Stefanik 
# Date 147626 -7200
#  Wed Oct 12 12:41:28 2016 +0200
# Node ID 7f9a6f30730074c869e5cf9b77c53929b06c4fb3
# Parent  e8e15a687df328f37b2d5e97e67f1a2071b35134
tests: introduce tests for grafting through renames

These cover all currently known cases of renames being grafted,
or changes being grafted through renames.

Right now, most of these cases are broken. Later patches in this series
will make them behave correctly.

The testcases heavily rely on each other, which would make it very difficult
to separate them and add them one-by-one for each case fixed by a patch.
Separating them should perhaps be a 4.1 task, if it doesn't slow down
the tests too much.

(Developed together with Pierre-Yves David)

diff -r e8e15a687df3 -r 7f9a6f307300 tests/test-graft.t
--- a/tests/test-graft.tWed Oct 12 21:33:45 2016 +0200
+++ b/tests/test-graft.tWed Oct 12 12:41:28 2016 +0200
@@ -842,3 +842,461 @@
   |/
   o  0
   
+Graft from behind a move or rename
+==
+
+NOTE: This is affected by issue5343, and will need updating when it's fixed
+
+Possible cases during a regular graft (when ca is between cta and c2):
+
+name | c1<-cta | cta<->ca | ca->c2
+A.0  | |  |
+A.1  |X|  |
+A.2  | | X|
+A.3  | |  |   X
+A.4  |X| X|
+A.5  |X|  |   X
+A.6  | | X|   X
+A.7  |X| X|   X
+
+A.0 is trivial, and doesn't need copy tracking.
+For A.1, a forward rename is recorded in the c1 pass, to be followed later.
+In A.2, the rename is recorded in the c2 pass and followed backwards.
+A.3 is recorded in the c2 pass as a forward rename to be duplicated on target.
+In A.4, both passes of checkcopies record incomplete renames, which are
+then joined in mergecopies to record a rename to be followed.
+In A.5 and A.7, the c1 pass records an incomplete rename, while the c2 pass
+records an incomplete divergence. The incomplete rename is then joined to the
+appropriate side of the incomplete divergence, and the result is recorded as a
+divergence. The code doesn't distinguish at all between these two cases, since
+the end result of them is the same: an incomplete divergence joined with an
+incomplete rename into a divergence.
+Finally, A.6 records a divergence entirely in the c2 pass.
+
+A.4 has a degenerate case a<-b<-a->a, where checkcopies isn't needed at all.
+A.5 has a special case a<-b<-b->a, which is treated like a<-b->a in a merge.
+A.6 has a special case a<-a<-b->a. Here, checkcopies will find a spurious
+incomplete divergence, which is in fact complete. This is handled later in
+mergecopies.
+A.7 has 4 special cases: a<-b<-a->b (the "ping-pong" case), a<-b<-c->b,
+a<-b<-a->c and a<-b<-c->a. Of these, only the "ping-pong" case is interesting,
+the others are fairly trivial (a<-b<-c->b and a<-b<-a->c proceed like the base
+case, a<-b<-c->a is treated the same as a<-b<-b->a).
+
+f5a therefore tests the "ping-pong" rename case, where a file is renamed to the
+same name on both branches, then the rename is backed out on one branch, and
+the backout is grafted to the other branch. This creates a challenging rename
+sequence of a<-b<-a->b in the graft target, topological CA, graft CA and graft
+source, respectively. Since rename detection will run on the c1 side for such a
+sequence (as for technical reasons, we split the c1 and c2 sides not at the
+graft CA, but rather at the topological CA), it will pick up a false rename,
+and cause a spurious merge conflict. This false rename is always exactly the
+reverse of the true rename that would be detected on the c2 side, so we can
+correct for it by detecting this condition and reversing as necessary.
+
+First, set up the repository with commits to be grafted
+
+  $ hg init ../graftmove
+  $ cd ../graftmove
+  $ echo c1a > f1a
+  $ echo c2a > f2a
+  $ echo c3a > f3a
+  $ echo c4a > f4a
+  $ echo c5a > f5a
+  $ hg ci -qAm A0
+  $ hg mv f1a f1b
+  $ hg mv f3a f3b
+  $ hg mv f5a f5b
+  $ hg ci -qAm B0
+  $ echo c1c > f1b
+  $ hg mv f2a f2c
+  $ hg mv f5b f5a
+  $ echo c5c > f5a
+  $ hg ci -qAm C0
+  $ hg mv f3b f3d
+  $ echo c4d > f4a
+  $ hg ci -qAm D0
+  $ hg log -G
+  @  changeset:   3:b69f5839d2d9
+  |  tag: tip
+  |  user:test
+  |  date:Thu Jan 01 00:00:00 1970 +
+  |  summary: D0
+  |
+  o  changeset:   2:f58c7e2b28fa
+  |  user:test
+  |  date:Thu Jan 01 00:00:00 1970 +
+  |  summary: C0
+  |
+  o  changeset:   1:3d7bba921b5d
+  |  user:test
+  |  date:Thu Jan 01 00:00:00 1970 +
+  |  summary: B0
+  |
+  o  changeset:   0:11f7a1b56675
+ user:test
+ date:Thu Jan 01 00:00:00 1970 +
+ summary: A0
+  
+
+Test the cases A.2 (f1x), A.3 (f2x) and a special case of A.6 (f5x) where the
+two renames actually converge to the same name (thus no actual 

[PATCH 02 of 16] checkcopies: pass data as a dictionary of dictionaries

[Gábor Stefanik]

# HG changeset patch
# User Pierre-Yves David 
# Date 1476145302 -7200
#  Tue Oct 11 02:21:42 2016 +0200
# Node ID 52fcb1d6cf58c18d63c6e63f3bc1d39194271707
# Parent  c2b7ab2954aecc3030a4c401390031513712d360
checkcopies: pass data as a dictionary of dictionaries

New dictionaries will be introduced in later parts of this patch series.
Make the prototype of _checkcopies more resilient to such changes.

diff -r c2b7ab2954ae -r 52fcb1d6cf58 mercurial/copies.py
--- a/mercurial/copies.py   Tue Oct 11 02:15:23 2016 +0200
+++ b/mercurial/copies.py   Tue Oct 11 02:21:42 2016 +0200
@@ -332,9 +332,15 @@
 ma = ca.manifest()
 
 # see _checkcopies documentation below for these dicts
-copy1, copy2 = {}, {}
-fullcopy1, fullcopy2 = {}, {}
-diverge = {}
+diverge = {} # divergence data is shared
+data1 = {'copy': {},
+ 'fullcopy': {},
+ 'diverge': diverge,
+}
+data2 = {'copy': {},
+ 'fullcopy': {},
+ 'diverge': diverge,
+}
 
 # find interesting file sets from manifests
 addedinm1 = m1.filesnotin(ma)
@@ -344,13 +350,13 @@
 bothnew = sorted(addedinm1 & addedinm2)
 
 for f in u1u:
-_checkcopies(c1, f, m1, m2, ca, limit, diverge, copy1, fullcopy1)
+_checkcopies(c1, f, m1, m2, ca, limit, data1)
 
 for f in u2u:
-_checkcopies(c2, f, m2, m1, ca, limit, diverge, copy2, fullcopy2)
+_checkcopies(c2, f, m2, m1, ca, limit, data2)
 
-copy = dict(copy1.items() + copy2.items())
-fullcopy = dict(fullcopy1.items() + fullcopy2.items())
+copy = dict(data1['copy'].items() + data2['copy'].items())
+fullcopy = dict(data1['fullcopy'].items() + data2['fullcopy'].items())
 
 renamedelete = {}
 renamedeleteset = set()
@@ -369,10 +375,14 @@
 if bothnew:
 repo.ui.debug("  unmatched files new in both:\n   %s\n"
   % "\n   ".join(bothnew))
-bothdiverge, _copy, _fullcopy = {}, {}, {}
+bothdiverge = {}
+bothdata = {'copy': {},
+'fullcopy': {},
+'diverge': bothdiverge,
+   }
 for f in bothnew:
-_checkcopies(c1, f, m1, m2, ca, limit, bothdiverge, _copy, _fullcopy)
-_checkcopies(c2, f, m2, m1, ca, limit, bothdiverge, _copy, _fullcopy)
+_checkcopies(c1, f, m1, m2, ca, limit, bothdata)
+_checkcopies(c2, f, m2, m1, ca, limit, bothdata)
 for of, fl in bothdiverge.items():
 if len(fl) == 2 and fl[0] == fl[1]:
 copy[fl[0]] = of # not actually divergent, just matching renames
@@ -487,7 +497,7 @@
 except StopIteration:
 return False
 
-def _checkcopies(ctx, f, m1, m2, base, limit, diverge, copy, fullcopy):
+def _checkcopies(ctx, f, m1, m2, base, limit, data):
 """
 check possible copies of f from m1 to m2
 
@@ -497,9 +507,10 @@
 m2 = the destination manifest
 base = the changectx used as a merge base
 limit = the rev number to not search beyond
-diverge = record all diverges in this dict
-copy = record all non-divergent copies in this dict
-fullcopy = record all copies in this dict
+data = dictionary of dictionary to store copy data. The keys are:
+- diverge = record all diverges in this dict
+- copy = record all non-divergent copies in this dict
+- fullcopy = record all copies in this dict
 
 note: limit is only an optimization, and there is no guarantee that
 irrelevant revisions will not be limited
@@ -522,7 +533,7 @@
 continue
 seen.add(of)
 
-fullcopy[f] = of # remember for dir rename detection
+data['fullcopy'][f] = of # remember for dir rename detection
 if of not in m2:
 continue # no match, keep looking
 if m2[of] == mb.get(of):
@@ -532,11 +543,11 @@
 # unrelated to the droids we are looking for.
 cr = _related(oc, c2, base.rev())
 if cr and (of == f or of == c2.path()): # non-divergent
-copy[f] = of
+data['copy'][f] = of
 return
 
 if of in mb:
-diverge.setdefault(of, []).append(f)
+data['diverge'].setdefault(of, []).append(f)
 
 def duplicatecopies(repo, rev, fromrev, skiprev=None):
 '''reproduce copies from fromrev to rev in the dirstate
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 05 of 16] checkcopies: add a sanity check against false-positive copies

[Gábor Stefanik]

# HG changeset patch
# User Gábor Stefanik 
# Date 1476300825 -7200
#  Wed Oct 12 21:33:45 2016 +0200
# Node ID e8e15a687df328f37b2d5e97e67f1a2071b35134
# Parent  bfb99ef3749757161bf439cc0ec678de46e24672
checkcopies: add a sanity check against false-positive copies

When grafting a copy backwards through a rename, a copy is wrongly detected,
which causes the graft to be applied inappropriately, in a destructive way.
Make sure that the old file name really exists in the common ancestor,
and bail out if it doesn't.

This fixes the aggravated case of bug 5343, although the basic issue
(failure to duplicate the copy information) still occurs.

diff -r bfb99ef37497 -r e8e15a687df3 mercurial/copies.py
--- a/mercurial/copies.py   Thu Oct 13 01:30:14 2016 +0200
+++ b/mercurial/copies.py   Wed Oct 12 21:33:45 2016 +0200
@@ -543,7 +543,8 @@
 # unrelated to the droids we are looking for.
 cr = _related(oc, c2, base.rev())
 if cr and (of == f or of == c2.path()): # non-divergent
-data['copy'][f] = of
+if of in mb:
+data['copy'][f] = of
 return
 
 if of in mb:
diff -r bfb99ef37497 -r e8e15a687df3 tests/test-graft.t
--- a/tests/test-graft.tThu Oct 13 01:30:14 2016 +0200
+++ b/tests/test-graft.tWed Oct 12 21:33:45 2016 +0200
@@ -427,8 +427,8 @@
   $ hg graft 3 --log -u foo
   grafting 3:4c60f11aa304 "3"
   warning: can't find ancestor for 'c' copied from 'b'!
-  $ hg log --template '{rev} {parents} {desc}\n' -r tip
-  14 1:5d205f8b35b6  3
+  $ hg log --template '{rev}:{node|short} {parents} {desc}\n' -r tip
+  14:0c921c65ef1e 1:5d205f8b35b6  3
   (grafted from 4c60f11aa304a54ae1c199feb94e7fc771e51ed8)
 
 Resolve conflicted graft
@@ -620,7 +620,7 @@
   date:Thu Jan 01 00:00:00 1970 +
   summary: 2
   
-  changeset:   14:f64defefacee
+  changeset:   14:0c921c65ef1e
   parent:  1:5d205f8b35b6
   user:foo
   date:Thu Jan 01 00:00:00 1970 +
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 03 of 16] copies: move variable document from checkcopies to mergecopies

[Gábor Stefanik]

# HG changeset patch
# User Pierre-Yves David 
# Date 1476314793 -7200
#  Thu Oct 13 01:26:33 2016 +0200
# Node ID 92dfc856c6cc25366eb8a7fb9b59d992ad5efc36
# Parent  52fcb1d6cf58c18d63c6e63f3bc1d39194271707
copies: move variable document from checkcopies to mergecopies

It appears that 'mergecopies' is the function consuming these data so we move
the documentation there.

diff -r 52fcb1d6cf58 -r 92dfc856c6cc mercurial/copies.py
--- a/mercurial/copies.py   Tue Oct 11 02:21:42 2016 +0200
+++ b/mercurial/copies.py   Thu Oct 13 01:26:33 2016 +0200
@@ -331,7 +331,10 @@
 m2 = c2.manifest()
 ma = ca.manifest()
 
-# see _checkcopies documentation below for these dicts
+# gather data from _checkcopies:
+# - diverge = record all diverges in this dict
+# - copy = record all non-divergent copies in this dict
+# - fullcopy = record all copies in this dict
 diverge = {} # divergence data is shared
 data1 = {'copy': {},
  'fullcopy': {},
@@ -507,10 +510,7 @@
 m2 = the destination manifest
 base = the changectx used as a merge base
 limit = the rev number to not search beyond
-data = dictionary of dictionary to store copy data. The keys are:
-- diverge = record all diverges in this dict
-- copy = record all non-divergent copies in this dict
-- fullcopy = record all copies in this dict
+data = dictionary of dictionary to store copy data. (see mergecopies)
 
 note: limit is only an optimization, and there is no guarantee that
 irrelevant revisions will not be limited
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 16 of 16] update: enable copy tracing for backwards and non-linear updates

[Gábor Stefanik]

# HG changeset patch
# User Gábor Stefanik 
# Date 1472155346 -7200
#  Thu Aug 25 22:02:26 2016 +0200
# Node ID 066dc1a5c8f4c9dfb582543bff5df8bbeba9cc31
# Parent  a3a36d782f4d4252adb9069eb3714f946881
update: enable copy tracing for backwards and non-linear updates

As a followup to the issue4028 series, this fixes a variant of the issue
that can occur when updating with uncommited local changes.

diff -r a3a36d782f4d -r 066dc1a5c8f4 mercurial/merge.py
--- a/mercurial/merge.pyTue Oct 11 04:39:47 2016 +0200
+++ b/mercurial/merge.pyThu Aug 25 22:02:26 2016 +0200
@@ -1537,15 +1537,16 @@
 pas = [p1]
 
 # deprecated config: merge.followcopies
-followcopies = False
+followcopies = repo.ui.configbool('merge', 'followcopies', True)
 if overwrite:
 pas = [wc]
+followcopies = False
 elif pas == [p2]: # backwards
-pas = [wc.p1()]
-elif not branchmerge and not wc.dirty(missing=True):
-pass
-elif pas[0] and repo.ui.configbool('merge', 'followcopies', True):
-followcopies = True
+pas = [p1]
+elif not pas[0]:
+followcopies = False
+if not branchmerge and not wc.dirty(missing=True):
+followcopies = False
 
 ### calculate phase
 actionbyfile, diverge, renamedelete = calculateupdates(
diff -r a3a36d782f4d -r 066dc1a5c8f4 tests/test-merge-local.t
--- a/tests/test-merge-local.t  Tue Oct 11 04:39:47 2016 +0200
+++ b/tests/test-merge-local.t  Thu Aug 25 22:02:26 2016 +0200
@@ -66,7 +66,7 @@
   merging zzz1_merge_ok
   merging zzz2_merge_bad
   warning: conflicts while merging zzz2_merge_bad! (edit, then use 'hg resolve 
--mark')
-  2 files updated, 1 files merged, 3 files removed, 1 files unresolved
+  2 files updated, 1 files merged, 2 files removed, 1 files unresolved
   use 'hg resolve' to retry unresolved file merges
   [1]
 
@@ -104,7 +104,7 @@
   merging zzz1_merge_ok
   merging zzz2_merge_bad
   warning: conflicts while merging zzz2_merge_bad! (edit, then use 'hg resolve 
--mark')
-  2 files updated, 1 files merged, 3 files removed, 1 files unresolved
+  2 files updated, 1 files merged, 2 files removed, 1 files unresolved
   use 'hg resolve' to retry unresolved file merges
   [1]
 
diff -r a3a36d782f4d -r 066dc1a5c8f4 tests/test-mq-subrepo.t
--- a/tests/test-mq-subrepo.t   Tue Oct 11 04:39:47 2016 +0200
+++ b/tests/test-mq-subrepo.t   Thu Aug 25 22:02:26 2016 +0200
@@ -304,6 +304,7 @@
   record this change to '.hgsub'? [Ynesfdaq?] y
   
   warning: subrepo spec file '.hgsub' not found
+  warning: subrepo spec file '.hgsub' not found
   abort: uncommitted changes in subrepository 'sub'
   [255]
   % update substate when adding .hgsub w/clean updated subrepo
@@ -319,6 +320,7 @@
   record this change to '.hgsub'? [Ynesfdaq?] y
   
   warning: subrepo spec file '.hgsub' not found
+  warning: subrepo spec file '.hgsub' not found
   path sub
source   sub
revision b2fdb12cd82b021c3b7053d67802e77b6eeaee31
diff -r a3a36d782f4d -r 066dc1a5c8f4 tests/test-up-local-change.t
--- a/tests/test-up-local-change.t  Tue Oct 11 04:39:47 2016 +0200
+++ b/tests/test-up-local-change.t  Thu Aug 25 22:02:26 2016 +0200
@@ -67,6 +67,10 @@
   summary: 2
   
   $ hg --debug up 0
+  starting 4 threads for background file closing (?)
+searching for copies back to rev 0
+unmatched files in local (from topological common ancestor):
+ b
   resolving manifests
branchmerge: False, force: False, partial: False
ancestor: 1e71731e6fbb, local: 1e71731e6fbb+, remote: c19d34741b0a
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 12 of 16] checkcopies: handle divergences contained entirely in tca::ctx

[Gábor Stefanik]

# HG changeset patch
# User Gábor Stefanik 
# Date 1476266043 -7200
#  Wed Oct 12 11:54:03 2016 +0200
# Node ID 73b55839aca7019d88cc9885e43244fb1e6d68ab
# Parent  e61355cb337ab5328e4cf5d0f153821600c53e72
checkcopies: handle divergences contained entirely in tca::ctx

During a graftlike merge, _checkcopies runs from ctx to tca, possibly
passing over the merge base. If there is a rename both before and after
the base, then we're actually dealing with divergent renames.
If there is no rename on the other side of tca, then the divergence is
contained entirely in the range of one _checkcopies invocation, and
should be detected "in the loop" without having to rely on the other
_checkcopies pass.

diff -r e61355cb337a -r 73b55839aca7 mercurial/copies.py
--- a/mercurial/copies.py   Thu Oct 13 02:19:43 2016 +0200
+++ b/mercurial/copies.py   Wed Oct 12 11:54:03 2016 +0200
@@ -564,9 +564,8 @@
 # traversed backwards.
 #
 # In the case there is both backward and forward renames (before and after
-# the base) this is more complicated as we must detect a divergence. This
-# is currently broken and hopefully some later code update will make that
-# work (we use 'backwards = False' in that case)
+# the base) this is more complicated as we must detect a divergence.
+# We use 'backwards = False' in that case.
 backwards = base != tca and f in mb
 getfctx = _makegetfctx(ctx)
 
@@ -600,6 +599,12 @@
 data['copy'][of] = f
 elif of in mb:
 data['copy'][f] = of
+else: # divergence w.r.t. graft CA on one side of topological CA
+for sf in seen:
+if sf in mb:
+assert sf not in data['diverge']
+data['diverge'][sf] = [f, of]
+break
 return
 
 if of in mb:
diff -r e61355cb337a -r 73b55839aca7 tests/test-graft.t
--- a/tests/test-graft.tThu Oct 13 02:19:43 2016 +0200
+++ b/tests/test-graft.tWed Oct 12 11:54:03 2016 +0200
@@ -982,6 +982,9 @@
 
   $ HGEDITOR="echo D1 >" hg graft -r 'desc("D0")' --edit
   grafting 3:b69f5839d2d9 "D0"
+  note: possible conflict - f3b was renamed multiple times to:
+   f3d
+   f3a
   warning: can't find ancestor for 'f3d' copied from 'f3b'!
 
 Set up the repository for some further tests
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 01 of 16] checkcopies: move 'movewithdir' initialisation right before its usage

[Gábor Stefanik]

# HG changeset patch
# User Pierre-Yves David 
# Date 1476144923 -7200
#  Tue Oct 11 02:15:23 2016 +0200
# Node ID c2b7ab2954aecc3030a4c401390031513712d360
# Parent  5cb830801855dbb63e98b948e355bc995d295bf3
checkcopies: move 'movewithdir' initialisation right before its usage

The 'movewithdir' had a lot of related logic all around the 'mergecopies'.
However it is actually never containing anything until the very last loop in
that function. We move the (simplified) variable definition there for clarity

diff -r 5cb830801855 -r c2b7ab2954ae mercurial/copies.py
--- a/mercurial/copies.py   Wed Oct 12 12:22:18 2016 +0200
+++ b/mercurial/copies.py   Tue Oct 11 02:15:23 2016 +0200
@@ -333,7 +333,6 @@
 
 # see _checkcopies documentation below for these dicts
 copy1, copy2 = {}, {}
-movewithdir1, movewithdir2 = {}, {}
 fullcopy1, fullcopy2 = {}, {}
 diverge = {}
 
@@ -351,7 +350,6 @@
 _checkcopies(c2, f, m2, m1, ca, limit, diverge, copy2, fullcopy2)
 
 copy = dict(copy1.items() + copy2.items())
-movewithdir = dict(movewithdir1.items() + movewithdir2.items())
 fullcopy = dict(fullcopy1.items() + fullcopy2.items())
 
 renamedelete = {}
@@ -395,7 +393,7 @@
 del divergeset
 
 if not fullcopy:
-return copy, movewithdir, diverge, renamedelete
+return copy, {}, diverge, renamedelete
 
 repo.ui.debug("  checking for directory renames\n")
 
@@ -433,12 +431,13 @@
 del d1, d2, invalid
 
 if not dirmove:
-return copy, movewithdir, diverge, renamedelete
+return copy, {}, diverge, renamedelete
 
 for d in dirmove:
 repo.ui.debug("   discovered dir src: '%s' -> dst: '%s'\n" %
   (d, dirmove[d]))
 
+movewithdir = {}
 # check unaccounted nonoverlapping files against directory moves
 for f in u1r + u2r:
 if f not in fullcopy:
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 02 of 16] checkcopies: pass data as a dictionary of dictionaries

[Gábor Stefanik]

# HG changeset patch
# User Pierre-Yves David 
# Date 1476145302 -7200
#  Tue Oct 11 02:21:42 2016 +0200
# Node ID 52fcb1d6cf58c18d63c6e63f3bc1d39194271707
# Parent  c2b7ab2954aecc3030a4c401390031513712d360
checkcopies: pass data as a dictionary of dictionaries

New dictionaries will be introduced in later parts of this patch series.
Make the prototype of _checkcopies more resilient to such changes.

diff -r c2b7ab2954ae -r 52fcb1d6cf58 mercurial/copies.py
--- a/mercurial/copies.py   Tue Oct 11 02:15:23 2016 +0200
+++ b/mercurial/copies.py   Tue Oct 11 02:21:42 2016 +0200
@@ -332,9 +332,15 @@
 ma = ca.manifest()
 
 # see _checkcopies documentation below for these dicts
-copy1, copy2 = {}, {}
-fullcopy1, fullcopy2 = {}, {}
-diverge = {}
+diverge = {} # divergence data is shared
+data1 = {'copy': {},
+ 'fullcopy': {},
+ 'diverge': diverge,
+}
+data2 = {'copy': {},
+ 'fullcopy': {},
+ 'diverge': diverge,
+}
 
 # find interesting file sets from manifests
 addedinm1 = m1.filesnotin(ma)
@@ -344,13 +350,13 @@
 bothnew = sorted(addedinm1 & addedinm2)
 
 for f in u1u:
-_checkcopies(c1, f, m1, m2, ca, limit, diverge, copy1, fullcopy1)
+_checkcopies(c1, f, m1, m2, ca, limit, data1)
 
 for f in u2u:
-_checkcopies(c2, f, m2, m1, ca, limit, diverge, copy2, fullcopy2)
+_checkcopies(c2, f, m2, m1, ca, limit, data2)
 
-copy = dict(copy1.items() + copy2.items())
-fullcopy = dict(fullcopy1.items() + fullcopy2.items())
+copy = dict(data1['copy'].items() + data2['copy'].items())
+fullcopy = dict(data1['fullcopy'].items() + data2['fullcopy'].items())
 
 renamedelete = {}
 renamedeleteset = set()
@@ -369,10 +375,14 @@
 if bothnew:
 repo.ui.debug("  unmatched files new in both:\n   %s\n"
   % "\n   ".join(bothnew))
-bothdiverge, _copy, _fullcopy = {}, {}, {}
+bothdiverge = {}
+bothdata = {'copy': {},
+'fullcopy': {},
+'diverge': bothdiverge,
+   }
 for f in bothnew:
-_checkcopies(c1, f, m1, m2, ca, limit, bothdiverge, _copy, _fullcopy)
-_checkcopies(c2, f, m2, m1, ca, limit, bothdiverge, _copy, _fullcopy)
+_checkcopies(c1, f, m1, m2, ca, limit, bothdata)
+_checkcopies(c2, f, m2, m1, ca, limit, bothdata)
 for of, fl in bothdiverge.items():
 if len(fl) == 2 and fl[0] == fl[1]:
 copy[fl[0]] = of # not actually divergent, just matching renames
@@ -487,7 +497,7 @@
 except StopIteration:
 return False
 
-def _checkcopies(ctx, f, m1, m2, base, limit, diverge, copy, fullcopy):
+def _checkcopies(ctx, f, m1, m2, base, limit, data):
 """
 check possible copies of f from m1 to m2
 
@@ -497,9 +507,10 @@
 m2 = the destination manifest
 base = the changectx used as a merge base
 limit = the rev number to not search beyond
-diverge = record all diverges in this dict
-copy = record all non-divergent copies in this dict
-fullcopy = record all copies in this dict
+data = dictionary of dictionary to store copy data. The keys are:
+- diverge = record all diverges in this dict
+- copy = record all non-divergent copies in this dict
+- fullcopy = record all copies in this dict
 
 note: limit is only an optimization, and there is no guarantee that
 irrelevant revisions will not be limited
@@ -522,7 +533,7 @@
 continue
 seen.add(of)
 
-fullcopy[f] = of # remember for dir rename detection
+data['fullcopy'][f] = of # remember for dir rename detection
 if of not in m2:
 continue # no match, keep looking
 if m2[of] == mb.get(of):
@@ -532,11 +543,11 @@
 # unrelated to the droids we are looking for.
 cr = _related(oc, c2, base.rev())
 if cr and (of == f or of == c2.path()): # non-divergent
-copy[f] = of
+data['copy'][f] = of
 return
 
 if of in mb:
-diverge.setdefault(of, []).append(f)
+data['diverge'].setdefault(of, []).append(f)
 
 def duplicatecopies(repo, rev, fromrev, skiprev=None):
 '''reproduce copies from fromrev to rev in the dirstate
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 10 of 16] copies: make it possible to distinguish betwen _computenonoverlap invocations

[Gábor Stefanik]

# HG changeset patch
# User Pierre-Yves David 
# Date 1476372109 -7200
#  Thu Oct 13 17:21:49 2016 +0200
# Node ID 4bb6c6ac3db22c3c4f947dd69496d895ad083efd
# Parent  1420cffe78a3ef2d0a2d3f125ce780754745ead8
copies: make it possible to distinguish betwen _computenonoverlap invocations

_computenonoverlap needs to be invoked twice during a graft, and debugging
messages should be distinguishable between the two invocations

diff -r 1420cffe78a3 -r 4bb6c6ac3db2 mercurial/copies.py
--- a/mercurial/copies.py   Thu Oct 13 02:03:54 2016 +0200
+++ b/mercurial/copies.py   Thu Oct 13 17:21:49 2016 +0200
@@ -231,23 +231,27 @@
 return _chain(x, y, _backwardrenames(x, a),
   _forwardcopies(a, y, match=match))
 
-def _computenonoverlap(repo, c1, c2, addedinm1, addedinm2):
+def _computenonoverlap(repo, c1, c2, addedinm1, addedinm2, baselabel=''):
 """Computes, based on addedinm1 and addedinm2, the files exclusive to c1
 and c2. This is its own function so extensions can easily wrap this call
 to see what files mergecopies is about to process.
 
 Even though c1 and c2 are not used in this function, they are useful in
 other extensions for being able to read the file nodes of the changed 
files.
+
+"baselabel" can be passed to help distinguish the multiple computations
+done in the graft case.
 """
 u1 = sorted(addedinm1 - addedinm2)
 u2 = sorted(addedinm2 - addedinm1)
 
+header = "  unmatched files in %s"
+if baselabel:
+header += ' (from %s)' % baselabel
 if u1:
-repo.ui.debug("  unmatched files in local:\n   %s\n"
-  % "\n   ".join(u1))
+repo.ui.debug("%s:\n   %s\n" % (header % 'local', "\n   ".join(u1)))
 if u2:
-repo.ui.debug("  unmatched files in other:\n   %s\n"
-  % "\n   ".join(u2))
+repo.ui.debug("%s:\n   %s\n" % (header % 'other', "\n   ".join(u2)))
 return u1, u2
 
 def _makegetfctx(ctx):
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 2 of 2 V3] py3: use namedtuple._replace to produce new tokens

[Pierre-Yves David]

On 10/14/2016 06:55 PM, Martijn Pieters wrote:

# HG changeset patch
# User Martijn Pieters 
# Date 1476347257 -3600
#  Thu Oct 13 09:27:37 2016 +0100
# Node ID d20dd7db86044bdca79825499b913840d726d841
# Parent  9031460519503abe5dc430c8ece29d198121cd65
py3: use namedtuple._replace to produce new tokens


We seems to be using a private function of some stdlib type?
Can you elaborate on why this is a good move?


diff --git a/mercurial/__init__.py b/mercurial/__init__.py
--- a/mercurial/__init__.py
+++ b/mercurial/__init__.py
@@ -233,9 +233,7 @@
 """
 st = tokens[j]
 if st.type == token.STRING and st.string.startswith(("'", '"')):
-rt = tokenize.TokenInfo(st.type, 'u%s' % st.string,
-st.start, st.end, st.line)
-tokens[j] = rt
+tokens[j] = st._replace(string='u%s' % st.string)

 for i, t in enumerate(tokens):
 # Convert most string literals to byte literals. String literals
@@ -266,8 +264,7 @@
 continue

 # String literal. Prefix to make a b'' string.
-yield tokenize.TokenInfo(t.type, 'b%s' % s, t.start, t.end,
-  t.line)
+yield t._replace(string='b%s' % t.string)
 continue

 # Insert compatibility imports at "from __future__ import" line.
@@ -287,10 +284,8 @@
 for u in tokenize.tokenize(io.BytesIO(l).readline):
 if u.type in (tokenize.ENCODING, token.ENDMARKER):
 continue
-yield tokenize.TokenInfo(u.type, u.string,
- (r, c + u.start[1]),
- (r, c + u.end[1]),
- '')
+yield u._replace(
+start=(r, c + u.start[1]), end=(r, c + u.end[1]))
 continue

 # This looks like a function call.
@@ -322,8 +317,7 @@
 # It changes iteritems to items as iteritems is not
 # present in Python 3 world.
 elif fn == 'iteritems':
-yield tokenize.TokenInfo(t.type, 'items',
- t.start, t.end, t.line)
+yield t._replace(string='items')
 continue

 # Emit unmodified token.


--
Pierre-Yves David
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 13 of 16] mergecopies: add logic to process incomplete data

[Gábor Stefanik]

# HG changeset patch
# User Gábor Stefanik 
# Date 1475578314 -7200
#  Tue Oct 04 12:51:54 2016 +0200
# Node ID a1c80b2f6896c7c77c660d15344f2c50845aee84
# Parent  905ceb1509480f912ece7c5622e151f80581782f
mergecopies: add logic to process incomplete data

We first combine incomplete copies on the two sides of the topological CA
into complete copies.
Any leftover incomplete copies are then combined with the incomplete
divergences to reconstruct divergences spanning over the topological CA.
Finally we promote any divergences falsely flagged as incomplete to full
divergences.

Right now, there is nothing generating incomplete copy/divergence data,
so this code does nothing. Changes to _checkcopies to populate these
dicts are coming later in this series.

diff -r 905ceb150948 -r a1c80b2f6896 mercurial/copies.py
--- a/mercurial/copies.py   Wed Oct 12 11:54:03 2016 +0200
+++ b/mercurial/copies.py   Tue Oct 04 12:51:54 2016 +0200
@@ -289,6 +289,22 @@
 return fctx
 return util.lrucachefunc(makectx)
 
+def _combinecopies(copyfrom, copyto, finalcopy, diverge, incompletediverge):
+"""combine partial copy paths"""
+remainder = {}
+for f in copyfrom:
+if f in copyto:
+finalcopy[copyto[f]] = copyfrom[f]
+del copyto[f]
+for f in incompletediverge:
+assert f not in diverge
+ic = incompletediverge[f]
+if ic[0] in copyto:
+diverge[f] = [copyto[ic[0]], ic[1]]
+else:
+remainder[f] = ic
+return remainder
+
 def mergecopies(repo, c1, c2, base):
 """
 Find moves and copies between context c1 and c2 that are relevant
@@ -360,14 +376,21 @@
 # - diverge = record all diverges in this dict
 # - copy = record all non-divergent copies in this dict
 # - fullcopy = record all copies in this dict
+# - incomplete = record non-divergent partial copies here
+# - incompletediverge = record divergent partial copies here
 diverge = {} # divergence data is shared
+incompletediverge  = {}
 data1 = {'copy': {},
  'fullcopy': {},
+ 'incomplete': {},
  'diverge': diverge,
+ 'incompletediverge': incompletediverge,
 }
 data2 = {'copy': {},
  'fullcopy': {},
+ 'incomplete': {},
  'diverge': diverge,
+ 'incompletediverge': incompletediverge,
 }
 
 # find interesting file sets from manifests
@@ -398,6 +421,13 @@
 copy = dict(data1['copy'].items() + data2['copy'].items())
 fullcopy = dict(data1['fullcopy'].items() + data2['fullcopy'].items())
 
+if dirtyc1:
+_combinecopies(data2['incomplete'], data1['incomplete'], copy, diverge,
+   incompletediverge)
+else:
+_combinecopies(data1['incomplete'], data2['incomplete'], copy, diverge,
+   incompletediverge)
+
 renamedelete = {}
 renamedeleteset = set()
 divergeset = set()
@@ -416,13 +446,36 @@
 repo.ui.debug("  unmatched files new in both:\n   %s\n"
   % "\n   ".join(bothnew))
 bothdiverge = {}
-bothdata = {'copy': {},
-'fullcopy': {},
-'diverge': bothdiverge,
-   }
+bothincompletediverge = {}
+both1 = {'copy': {},
+ 'fullcopy': {},
+ 'incomplete': {},
+ 'diverge': bothdiverge,
+ 'incompletediverge': bothincompletediverge
+}
+both2 = {'copy': {},
+ 'fullcopy': {},
+ 'incomplete': {},
+ 'diverge': bothdiverge,
+ 'incompletediverge': bothincompletediverge
+}
 for f in bothnew:
-_checkcopies(c1, f, m1, m2, base, tca, limit, bothdata)
-_checkcopies(c2, f, m2, m1, base, tca, limit, bothdata)
+_checkcopies(c1, f, m1, m2, base, tca, limit, both1)
+_checkcopies(c2, f, m2, m1, base, tca, limit, both2)
+if dirtyc1:
+assert both2['incomplete'] == {}
+remainder = _combinecopies({}, both1['incomplete'], copy, bothdiverge,
+   bothincompletediverge)
+else:
+assert both1['incomplete'] == {}
+remainder = _combinecopies({}, both2['incomplete'], copy, bothdiverge,
+   bothincompletediverge)
+for f in remainder:
+assert f not in bothdiverge
+ic = remainder[f]
+if ic[0] in (m1 if dirtyc1 else m2):
+# backed-out rename on one side, but watch out for deleted files
+bothdiverge[f] = ic
 for of, fl in bothdiverge.items():
 if len(fl) == 2 and fl[0] == fl[1]:
 copy[fl[0]] = of # not actually divergent, just matching renames
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 09 of 16] copies: make _checkcopies handle simple renames in a rotated DAG

[Gábor Stefanik]

# HG changeset patch
# User Gábor Stefanik 
# Date 1476317034 -7200
#  Thu Oct 13 02:03:54 2016 +0200
# Node ID 1420cffe78a3ef2d0a2d3f125ce780754745ead8
# Parent  2284ef0bd7df9b83f10f91f79f74f5b44b6ffde0
copies: make _checkcopies handle simple renames in a rotated DAG

This introduces a distinction between "merge base" and
"topological common ancestor". During a regular merge, these two are
identical. Graft, however, performs a merge in a rotated DAG, where the
merge common ancestor will not be a common ancestor at all in the
original DAG.

To correctly find copies in case of a graft, we need to take both the
merge base and the topological CA into account, and track any renames
between them in reverse. Fortunately we can detect this in advance,
see comment in the code about "backwards".

This patch only supports finding non-divergent renames contained entirely
between the merge base and the topological CA. Further patches are coming
to support more complex cases.

(Pierre-Yves David was involved in the cleanup of this patch.)

diff -r 2284ef0bd7df -r 1420cffe78a3 mercurial/copies.py
--- a/mercurial/copies.py   Thu Oct 13 02:03:49 2016 +0200
+++ b/mercurial/copies.py   Thu Oct 13 02:03:54 2016 +0200
@@ -374,10 +374,10 @@
 bothnew = sorted(addedinm1 & addedinm2)
 
 for f in u1u:
-_checkcopies(c1, f, m1, m2, base, limit, data1)
+_checkcopies(c1, f, m1, m2, base, tca, limit, data1)
 
 for f in u2u:
-_checkcopies(c2, f, m2, m1, base, limit, data2)
+_checkcopies(c2, f, m2, m1, base, tca, limit, data2)
 
 copy = dict(data1['copy'].items() + data2['copy'].items())
 fullcopy = dict(data1['fullcopy'].items() + data2['fullcopy'].items())
@@ -405,8 +405,8 @@
 'diverge': bothdiverge,
}
 for f in bothnew:
-_checkcopies(c1, f, m1, m2, base, limit, bothdata)
-_checkcopies(c2, f, m2, m1, base, limit, bothdata)
+_checkcopies(c1, f, m1, m2, base, tca, limit, bothdata)
+_checkcopies(c2, f, m2, m1, base, tca, limit, bothdata)
 for of, fl in bothdiverge.items():
 if len(fl) == 2 and fl[0] == fl[1]:
 copy[fl[0]] = of # not actually divergent, just matching renames
@@ -521,7 +521,7 @@
 except StopIteration:
 return False
 
-def _checkcopies(ctx, f, m1, m2, base, limit, data):
+def _checkcopies(ctx, f, m1, m2, base, tca, limit, data):
 """
 check possible copies of f from m1 to m2
 
@@ -530,6 +530,7 @@
 m1 = the source manifest
 m2 = the destination manifest
 base = the changectx used as a merge base
+tca = topological common ancestor for graft-like scenarios
 limit = the rev number to not search beyond
 data = dictionary of dictionary to store copy data. (see mergecopies)
 
@@ -540,6 +541,17 @@
 """
 
 mb = base.manifest()
+# Might be true if this call is about finding backward renames,
+# This happens in the case of grafts because the DAG is then rotated.
+# If the file exists in both the base and the source, we are not looking
+# for a rename on the source side, but on the part of the DAG that is
+# traversed backwards.
+#
+# In the case there is both backward and forward renames (before and after
+# the base) this is more complicated as we must detect a divergence. This
+# is currently broken and hopefully some later code update will make that
+# work (we use 'backwards = False' in that case)
+backwards = base != tca and f in mb
 getfctx = _makegetfctx(ctx)
 
 of = None
@@ -554,7 +566,11 @@
 continue
 seen.add(of)
 
-data['fullcopy'][f] = of # remember for dir rename detection
+# remember for dir rename detection
+if backwards:
+data['fullcopy'][of] = f # grafting backwards through renames
+else:
+data['fullcopy'][f] = of
 if of not in m2:
 continue # no match, keep looking
 if m2[of] == mb.get(of):
@@ -562,9 +578,11 @@
 c2 = getfctx(of, m2[of])
 # c2 might be a plain new file on added on destination side that is
 # unrelated to the droids we are looking for.
-cr = _related(oc, c2, base.rev())
+cr = _related(oc, c2, tca.rev())
 if cr and (of == f or of == c2.path()): # non-divergent
-if of in mb:
+if backwards:
+data['copy'][of] = f
+elif of in mb:
 data['copy'][f] = of
 return
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 08 of 16] copies: compute a suitable TCA if base turns out to be unsuitable

[Gábor Stefanik]

# HG changeset patch
# User Gábor Stefanik 
# Date 1476317029 -7200
#  Thu Oct 13 02:03:49 2016 +0200
# Node ID 2284ef0bd7df9b83f10f91f79f74f5b44b6ffde0
# Parent  b3631f615bc2fff5cbce4cde7373dde052197a0e
copies: compute a suitable TCA if base turns out to be unsuitable


This will be used later in an update to _checkcopies.

(Pierre-Yves David was involved in the cleanup of this patch.)

diff -r b3631f615bc2 -r 2284ef0bd7df mercurial/copies.py
--- a/mercurial/copies.py   Thu Oct 13 01:47:33 2016 +0200
+++ b/mercurial/copies.py   Thu Oct 13 02:03:49 2016 +0200
@@ -338,6 +338,10 @@
 dirtyc1 = not (base == _c1 or base.descendant(_c1))
 dirtyc2 = not (base== _c2 or base.descendant(_c2))
 graft = dirtyc1 or dirtyc2
+tca = base
+if graft:
+tca = _c1.ancestor(_c2)
+
 limit = _findlimit(repo, c1.rev(), c2.rev())
 if limit is None:
 # no common ancestor, no copies
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 07 of 16] copies: detect graft-like merges penis

[Gábor Stefanik]

# HG changeset patch
# User Gábor Stefanik 
# Date 1476316053 -7200
#  Thu Oct 13 01:47:33 2016 +0200
# Node ID b3631f615bc2fff5cbce4cde7373dde052197a0e
# Parent  7f9a6f30730074c869e5cf9b77c53929b06c4fb3
copies: detect graft-like merges penis

Right now, nothing changes as a result of this, but we want to handle
grafts differently from ordinary merges later.

(Series developed together with Pierre-Yves David)

diff -r 7f9a6f307300 -r b3631f615bc2 mercurial/copies.py
--- a/mercurial/copies.py   Wed Oct 12 12:41:28 2016 +0200
+++ b/mercurial/copies.py   Thu Oct 13 01:47:33 2016 +0200
@@ -321,6 +321,23 @@
 if repo.ui.configbool('experimental', 'disablecopytrace'):
 return {}, {}, {}, {}
 
+# In certain scenarios (e.g. graft, update or rebase), base can be
+# overridden We still need to know a real common ancestor in this case We
+# can't just compute _c1.ancestor(_c2) and compare it to ca, because there
+# can be multiple common ancestors, e.g. in case of bidmerge.  Because our
+# caller may not know if the revision passed in lieu of the CA is a genuine
+# common ancestor or not without explicitly checking it, it's better to
+# determine that here.
+#
+# base.descendant(wc) and base.descendant(base) are False, work around that
+_c1 = c1.p1() if c1.rev() is None else c1
+_c2 = c2.p1() if c2.rev() is None else c2
+# an endpoint is "dirty" if it isn't a descendant of the merge base
+# if we have a dirty endpoint, we need to trigger graft logic, and also
+# keep track of which endpoint is dirty
+dirtyc1 = not (base == _c1 or base.descendant(_c1))
+dirtyc2 = not (base== _c2 or base.descendant(_c2))
+graft = dirtyc1 or dirtyc2
 limit = _findlimit(repo, c1.rev(), c2.rev())
 if limit is None:
 # no common ancestor, no copies
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 01 of 16] checkcopies: move 'movewithdir' initialisation right before its usage

[Gábor Stefanik]

# HG changeset patch
# User Pierre-Yves David 
# Date 1476144923 -7200
#  Tue Oct 11 02:15:23 2016 +0200
# Node ID c2b7ab2954aecc3030a4c401390031513712d360
# Parent  5cb830801855dbb63e98b948e355bc995d295bf3
checkcopies: move 'movewithdir' initialisation right before its usage

The 'movewithdir' had a lot of related logic all around the 'mergecopies'.
However it is actually never containing anything until the very last loop in
that function. We move the (simplified) variable definition there for clarity

diff -r 5cb830801855 -r c2b7ab2954ae mercurial/copies.py
--- a/mercurial/copies.py   Wed Oct 12 12:22:18 2016 +0200
+++ b/mercurial/copies.py   Tue Oct 11 02:15:23 2016 +0200
@@ -333,7 +333,6 @@
 
 # see _checkcopies documentation below for these dicts
 copy1, copy2 = {}, {}
-movewithdir1, movewithdir2 = {}, {}
 fullcopy1, fullcopy2 = {}, {}
 diverge = {}
 
@@ -351,7 +350,6 @@
 _checkcopies(c2, f, m2, m1, ca, limit, diverge, copy2, fullcopy2)
 
 copy = dict(copy1.items() + copy2.items())
-movewithdir = dict(movewithdir1.items() + movewithdir2.items())
 fullcopy = dict(fullcopy1.items() + fullcopy2.items())
 
 renamedelete = {}
@@ -395,7 +393,7 @@
 del divergeset
 
 if not fullcopy:
-return copy, movewithdir, diverge, renamedelete
+return copy, {}, diverge, renamedelete
 
 repo.ui.debug("  checking for directory renames\n")
 
@@ -433,12 +431,13 @@
 del d1, d2, invalid
 
 if not dirmove:
-return copy, movewithdir, diverge, renamedelete
+return copy, {}, diverge, renamedelete
 
 for d in dirmove:
 repo.ui.debug("   discovered dir src: '%s' -> dst: '%s'\n" %
   (d, dirmove[d]))
 
+movewithdir = {}
 # check unaccounted nonoverlapping files against directory moves
 for f in u1r + u2r:
 if f not in fullcopy:
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 2 v2] evolve: lock the working copy early in next and prev (issue5244)

[Simon Farnsworth]

# HG changeset patch
# User Simon Farnsworth 
# Date 1476627823 25200
#  Sun Oct 16 07:23:43 2016 -0700
# Node ID 99c560f7d07d3d08b83f5f1802fc329d67e5de2e
# Parent  ee284d7c5faa5d9f17853f51c17883844b985c58
evolve: lock the working copy early in next and prev (issue5244)

Both next and prev depend on a consistent working copy, but were waiting to
take the lock until they were ready to alter the working copy.

Take the lock before reading the working copy state, and do not release it
until we're definitely not going to change the working copy.

diff --git a/hgext/evolve.py b/hgext/evolve.py
--- a/hgext/evolve.py
+++ b/hgext/evolve.py
@@ -2213,10 +2213,13 @@
 """update to parent revision
 
 Displays the summary line of the destination for clarity."""
-if True:
+wlock = None
+dryrunopt = opts['dry_run']
+if not dryrunopt:
+wlock = repo.wlock()
+try:
 wkctx = repo[None]
 wparents = wkctx.parents()
-dryrunopt = opts['dry_run']
 if len(wparents) != 1:
 raise error.Abort('merge in progress')
 if not opts['merge']:
@@ -2246,7 +2249,6 @@
 ret = hg.update(repo, p.rev())
 if not ret:
 tr = lock = None
-wlock = repo.wlock()
 try:
 lock = repo.lock()
 tr = repo.transaction('previous')
@@ -2258,6 +2260,8 @@
 tr.close()
 finally:
 lockmod.release(tr, lock, wlock)
+wlock = None
+
 displayer.show(p)
 return 0
 else:
@@ -2265,6 +2269,8 @@
 displayer.show(p)
 ui.warn(_('multiple parents, explicitly update to one\n'))
 return 1
+finally:
+lockmod.release(wlock)
 
 @command('^next',
  [('B', 'move-bookmark', False,
@@ -2282,10 +2288,13 @@
 
 Displays the summary line of the destination for clarity.
 """
-if True:
+wlock = None
+dryrunopt = opts['dry_run']
+if not dryrunopt:
+wlock = repo.wlock()
+try:
 wkctx = repo[None]
 wparents = wkctx.parents()
-dryrunopt = opts['dry_run']
 if len(wparents) != 1:
 raise error.Abort('merge in progress')
 if not opts['merge']:
@@ -2315,7 +2324,6 @@
 ret = hg.update(repo, c.rev())
 if not ret:
 lock = tr = None
-wlock = repo.wlock()
 try:
 lock = repo.lock()
 tr = repo.transaction('next')
@@ -2327,6 +2335,7 @@
 tr.close()
 finally:
 lockmod.release(tr, lock, wlock)
+wlock = None
 displayer.show(c)
 result = 0
 elif children:
@@ -2368,6 +2377,8 @@
 return result
 return 1
 return result
+finally:
+lockmod.release(wlock)
 
 def _reachablefrombookmark(repo, revs, bookmarks):
 """filter revisions and bookmarks reachable from the given bookmark
diff --git a/tests/fake-editor.sh b/tests/fake-editor.sh
new file mode 100755
--- /dev/null
+++ b/tests/fake-editor.sh
@@ -0,0 +1,3 @@
+#!/bin/sh
+sleep 5
+echo "new desc" >> $1
diff --git a/tests/test-prev-next.t b/tests/test-prev-next.t
--- a/tests/test-prev-next.t
+++ b/tests/test-prev-next.t
@@ -206,3 +206,34 @@
   move:[5] added d
   atop:[6] added b (3)
   working directory is now at 47ea25be8aea
+
+prev and next should lock properly against other commands
+
+  $ hg init repo
+  $ cd repo
+  $ HGEDITOR=${TESTDIR}/fake-editor.sh
+  $ echo hi > foo
+  $ hg ci -Am 'one'
+  adding foo
+  $ echo bye > foo
+  $ hg ci -Am 'two'
+
+  $ hg amend --edit &
+  $ sleep 1
+  $ hg prev
+  waiting for lock on working directory of $TESTTMP/repo held by process '*' 
on host '*' (glob)
+  got lock after [4-6] seconds (re)
+  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  [0] one
+  $ wait
+
+  $ hg amend --edit &
+  $ sleep 1
+  $ hg next --evolve
+  waiting for lock on working directory of $TESTTMP/repo held by process '*' 
on host '*' (glob)
+  1 new unstable changesets
+  got lock after [4-6] seconds (re)
+  move:[2] two
+  atop:[3] one
+  working directory now at a7d885c75614
+  $ wait
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 1 of 2 v2] evolve: indent cmdnext and cmdprev ready for locking change (issue5244)

[Simon Farnsworth]

# HG changeset patch
# User Simon Farnsworth 
# Date 1475939661 25200
#  Sat Oct 08 08:14:21 2016 -0700
# Node ID ee284d7c5faa5d9f17853f51c17883844b985c58
# Parent  5383671ef612a1764bbbed13a7ef2d339d0a9c2d
evolve: indent cmdnext and cmdprev ready for locking change (issue5244)

The locking change I'm about to introduce forces an indentation shift. Do
the indentation change with no code change now, to make the next change
easier to review

diff --git a/hgext/evolve.py b/hgext/evolve.py
--- a/hgext/evolve.py
+++ b/hgext/evolve.py
@@ -2213,57 +2213,58 @@
 """update to parent revision
 
 Displays the summary line of the destination for clarity."""
-wkctx = repo[None]
-wparents = wkctx.parents()
-dryrunopt = opts['dry_run']
-if len(wparents) != 1:
-raise error.Abort('merge in progress')
-if not opts['merge']:
-try:
-cmdutil.bailifchanged(repo)
-except error.Abort as exc:
-exc.hint = _('do you want --merge?')
-raise
-
-parents = wparents[0].parents()
-topic = getattr(repo, 'currenttopic', '')
-if topic and not opts.get("no_topic", False):
-parents = [ctx for ctx in parents if ctx.topic() == topic]
-displayer = cmdutil.show_changeset(ui, repo, {'template': shorttemplate})
-if not parents:
-ui.warn(_('no parent in topic "%s"\n') % topic)
-ui.warn(_('(do you want --no-topic)\n'))
-elif len(parents) == 1:
-p = parents[0]
-bm = bmactive(repo)
-shouldmove = opts.get('move_bookmark') and bm is not None
-if dryrunopt:
-ui.write(('hg update %s;\n' % p.rev()))
-if shouldmove:
-ui.write(('hg bookmark %s -r %s;\n' % (bm, p.rev(
+if True:
+wkctx = repo[None]
+wparents = wkctx.parents()
+dryrunopt = opts['dry_run']
+if len(wparents) != 1:
+raise error.Abort('merge in progress')
+if not opts['merge']:
+try:
+cmdutil.bailifchanged(repo)
+except error.Abort as exc:
+exc.hint = _('do you want --merge?')
+raise
+
+parents = wparents[0].parents()
+topic = getattr(repo, 'currenttopic', '')
+if topic and not opts.get("no_topic", False):
+parents = [ctx for ctx in parents if ctx.topic() == topic]
+displayer = cmdutil.show_changeset(ui, repo, {'template': 
shorttemplate})
+if not parents:
+ui.warn(_('no parent in topic "%s"\n') % topic)
+ui.warn(_('(do you want --no-topic)\n'))
+elif len(parents) == 1:
+p = parents[0]
+bm = bmactive(repo)
+shouldmove = opts.get('move_bookmark') and bm is not None
+if dryrunopt:
+ui.write(('hg update %s;\n' % p.rev()))
+if shouldmove:
+ui.write(('hg bookmark %s -r %s;\n' % (bm, p.rev(
+else:
+ret = hg.update(repo, p.rev())
+if not ret:
+tr = lock = None
+wlock = repo.wlock()
+try:
+lock = repo.lock()
+tr = repo.transaction('previous')
+if shouldmove:
+repo._bookmarks[bm] = p.node()
+repo._bookmarks.recordchange(tr)
+else:
+bmdeactivate(repo)
+tr.close()
+finally:
+lockmod.release(tr, lock, wlock)
+displayer.show(p)
+return 0
 else:
-ret = hg.update(repo, p.rev())
-if not ret:
-tr = lock = None
-wlock = repo.wlock()
-try:
-lock = repo.lock()
-tr = repo.transaction('previous')
-if shouldmove:
-repo._bookmarks[bm] = p.node()
-repo._bookmarks.recordchange(tr)
-else:
-bmdeactivate(repo)
-tr.close()
-finally:
-lockmod.release(tr, lock, wlock)
-displayer.show(p)
-return 0
-else:
-for p in parents:
-displayer.show(p)
-ui.warn(_('multiple parents, explicitly update to one\n'))
-return 1
+for p in parents:
+displayer.show(p)
+ui.warn(_('multiple parents, explicitly update to one\n'))
+return 1
 
 @command('^next',
  [('B', 'move-bookmark', False,
@@ -2281,91 +2282,92 @@
 
 Displays the summary line of the destination for clarity.
 """
-wkctx = repo[None]
-wparents = wkctx.parents()
-dryrunopt = opts['dry_run']
-if len(wparents) != 1:
-raise error.Abort('merge in progress')
-

Re: [PATCH 2 of 2] evolve: lock the working copy early in next and prev (issue5244)

[Simon Farnsworth]

On 16/10/2016 12:43, Pierre-Yves David wrote:



On 10/08/2016 05:18 PM, Simon Farnsworth wrote:

# HG changeset patch
# User Simon Farnsworth 
# Date 1475939634 25200
#  Sat Oct 08 08:13:54 2016 -0700
# Node ID 8a0c9c0158b3e9574a4571af3dce9978844b825d
# Parent  ee284d7c5faa5d9f17853f51c17883844b985c58
evolve: lock the working copy early in next and prev (issue5244)

Both next and prev depend on a consistent working copy, but were
waiting to
take the lock until they were ready to alter the working copy.

Take the lock before reading the working copy state, and do not
release it
until we're definitely not going to change the working copy.


The fix approach seems good, but the implementation is a bit strangeto
me. See inline.

I can remove the early unlocks, but the goal is to unlock early when we 
no longer need the locks.




diff --git a/hgext/evolve.py b/hgext/evolve.py
--- a/hgext/evolve.py
+++ b/hgext/evolve.py
@@ -2213,10 +2213,13 @@
 """update to parent revision

 Displays the summary line of the destination for clarity."""
-if True:
+wlock = None
+dryrunopt = opts['dry_run']
+if not dryrunopt:
+wlock = repo.wlock()
+try:
 wkctx = repo[None]
 wparents = wkctx.parents()
-dryrunopt = opts['dry_run']
 if len(wparents) != 1:
 raise error.Abort('merge in progress')
 if not opts['merge']:
@@ -2246,7 +2249,6 @@
 ret = hg.update(repo, p.rev())
 if not ret:
 tr = lock = None
-wlock = repo.wlock()
 try:
 lock = repo.lock()
 tr = repo.transaction('previous')
@@ -2258,13 +2260,22 @@
 tr.close()
 finally:
 lockmod.release(tr, lock, wlock)
+wlock = None
+else:
+lockmod.release(wlock)
+wlock = None
+
 displayer.show(p)
 return 0
 else:
+lockmod.release(wlock)
+wlock = None



Why are we issuing multiple lockmode.release call instead of just using
the final finally clause ?

Each lockmod.release() happens at a point where we no longer need the 
lock for correctness (and in the old code, didn't have the lock at all).


The aim is to minimise the time we're holding the lock for. I can easily 
redo this with wlock held for the entire duration of the command if 
you'd prefer that version?



 for p in parents:
 displayer.show(p)
 ui.warn(_('multiple parents, explicitly update to one\n'))
 return 1
+finally:
+lockmod.release(wlock)

 @command('^next',
  [('B', 'move-bookmark', False,
@@ -2282,10 +2293,13 @@

 Displays the summary line of the destination for clarity.
 """
-if True:
+wlock = None
+dryrunopt = opts['dry_run']
+if not dryrunopt:
+wlock = repo.wlock()
+try:
 wkctx = repo[None]
 wparents = wkctx.parents()
-dryrunopt = opts['dry_run']
 if len(wparents) != 1:
 raise error.Abort('merge in progress')
 if not opts['merge']:
@@ -2315,7 +2329,6 @@
 ret = hg.update(repo, c.rev())
 if not ret:
 lock = tr = None
-wlock = repo.wlock()
 try:
 lock = repo.lock()
 tr = repo.transaction('next')
@@ -2327,15 +2340,23 @@
 tr.close()
 finally:
 lockmod.release(tr, lock, wlock)
+wlock = None
+else:
+lockmod.release(wlock)
+wlock = None
 displayer.show(c)
 result = 0
 elif children:
+lockmod.release(wlock)
+wlock = None
 ui.warn(_("ambigious next changeset:\n"))
 for c in children:
 displayer.show(c)
 ui.warn(_('explicitly update to one of them\n'))
 result = 1
 else:
+lockmod.release(wlock)
+wlock = None
 aspchildren = _aspiringchildren(repo, [repo['.'].rev()])
 if topic:
 filtered.extend(repo[c] for c in children
@@ -2368,6 +2389,8 @@
 return result
 return 1
 return result
+finally:
+lockmod.release(wlock)

 def _reachablefrombookmark(repo, revs, bookmarks):
 """filter revisions and bookmarks reachable from the given bookmark
diff --git a/tests/fake-editor.sh b/tests/fake-editor.sh
new file mode 100755
--- /dev/null
+++ b/tests/fake-editor.sh
@@ -0,0 +1,3 @@
+#!/bin/sh
+sleep 5
+echo "new desc" >> $1
diff --git a/tests/test-prev-next.t b/tests/test-prev-next.t
--- a/tests/test-prev-next.t
+++ 

Re: [PATCH 3 of 3 v2] repos: introduce '-R readonly:PATH' for doing local operations "read only"

[Yuya Nishihara]

On Fri, 14 Oct 2016 03:17:30 +0200, Mads Kiilerich wrote:
> # HG changeset patch
> # User Mads Kiilerich 
> # Date 1476402795 -7200
> #  Fri Oct 14 01:53:15 2016 +0200
> # Node ID 491da143c2f310da732c2e1005e7f8394134a51d
> # Parent  24a7ccfb932b134da24e58817991943c8bbd63fa
> repos: introduce '-R readonly:PATH' for doing local operations "read only"

> The existing "API" for repository types could use some cleanup - it requires
> modules with special undefined duck typing. This patch seems to do what is
> needed.

I don't know which is better, but you can use a class to provide the API.
That's what schemes.py is doing.

> +# created, not thrown yet
> +readonlyexception = IOError(errno.EACCES, _('this is a "readonly" 
> repository'))
> +
> +# this seems to be a necessary part of the repository type API
> +islocal = localrepo.islocal
> +
> +class readonlyvfs(scmutil.vfs):
> +"""A VFS that only can be called with read modes - writing will fail with
> +an IO error as if the user didn't have write access"""
> +
> +def __call__(self, path, mode='r', *args, **kw):
> +if mode not in ('r', 'rb'):
> +raise readonlyexception
> +return super(readonlyvfs, self).__call__(path, mode, *args, **kw)
> +
> +class readonlyrepo(localrepo.localrepository):
> +"""A repository that is local but read only, as if the user didn't have
> +file system write access."""
> +
> +def __init__(self, baseui, path=None, create=False):
> +# we know the "scheme" for path is "readonly" but do not want to 
> extend
> +# the file/bundle hack in the "url" parser - just strip it here
> +assert path.startswith('readonly:'), path
> +path = path[len('readonly:'):]
> +
> +super(readonlyrepo, self).__init__(baseui, path=path, create=False)
> +
> +assert self.vfs.__class__ is scmutil.vfs
> +self.vfs.__class__ = readonlyvfs
> +assert self.wvfs.__class__ is scmutil.vfs
> +self.wvfs.__class__ = readonlyvfs

Don't we need to wrap svfs as well?

> +def lock(self, wait=True):
> +raise readonlyexception
> +
> +def wlock(self, wait=True):
> +raise readonlyexception

Some operations (e.g status) "try" to take a lock to update dirstate or cache
files. In which case, LockUnavailable should be raised.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] match: adding non-recursive directory matching

[Pierre-Yves David]

On 10/08/2016 06:58 PM, Rodrigo Damazio Bovendorp via Mercurial-devel wrote:

# HG changeset patch
# User Rodrigo Damazio Bovendorp 
# Date 1475944120 25200
#  Sat Oct 08 09:28:40 2016 -0700
# Node ID 545efe5a72efdce925a6a3fd3774b350c90b5c55
# Parent  dbcef8918bbdd8a64d9f79a37bcfa284a26f3a39
match: adding non-recursive directory matching

This allows one to match all files in a directory, without matching anything in 
subdirectories.
It's implemented almost identically to path:, except for the regex termination, 
which doesn't
allow more than one / after the directory name.

diff --git a/mercurial/match.py b/mercurial/match.py
--- a/mercurial/match.py
+++ b/mercurial/match.py
@@ -105,6 +105,9 @@
 'glob:' - a glob relative to cwd
 're:' - a regular expression
 'path:' - a path relative to repository root
+'files:' - a path relative to repository root, which is matched
+ non-recursively (files inside the directory will 
match,
+ but subdirectories and files in them won't


The feature seems useful and we should have it.

The current behavior is a bit strange to me. because we have directory 
being implicitly recursed of just 1 level (directory content). Could we 
have a xxx: where path is never recursed for anything. Listing a 
directory content would be an explicite 'xxx:my/directory/path/*'


We could use 'exact' or 'norecursion' for xxx.


 'relglob:' - an unrooted glob (*.c matches C files in all dirs)
 'relpath:' - a path relative to cwd
 'relre:' - a regexp that needn't match the start of a name
@@ -286,7 +289,7 @@
 for kind, pat in [_patsplit(p, default) for p in patterns]:
 if kind in ('glob', 'relpath'):
 pat = pathutil.canonpath(root, cwd, pat, auditor)
-elif kind in ('relglob', 'path'):
+elif kind in ('relglob', 'path', 'files'):
 pat = util.normpath(pat)
 elif kind in ('listfile', 'listfile0'):
 try:
@@ -447,7 +450,8 @@
 if ':' in pattern:
 kind, pat = pattern.split(':', 1)
 if kind in ('re', 'glob', 'path', 'relglob', 'relpath', 'relre',
-'listfile', 'listfile0', 'set', 'include', 'subinclude'):
+'listfile', 'listfile0', 'set', 'include', 'subinclude',
+'files'):
 return kind, pat
 return default, pattern

@@ -540,6 +544,19 @@
 if pat == '.':
 return ''
 return '^' + util.re.escape(pat) + '(?:/|$)'
+if kind == 'files':
+# Match one of:
+# For pat = 'some/dir':
+# some/dir
+# some/dir/
+# some/dir/filename
+# For pat = '' or pat = '.':
+# filename
+if pat == '.':
+escaped = ''
+else:
+escaped = util.re.escape(pat)
+return '^' + escaped + '(?:^|/|$)[^/]*$'
 if kind == 'relglob':
 return '(?:|.*/)' + _globre(pat) + globsuffix
 if kind == 'relpath':
@@ -628,7 +645,7 @@
 break
 root.append(p)
 r.append('/'.join(root) or '.')
-elif kind in ('relpath', 'path'):
+elif kind in ('relpath', 'path', 'files'):
 r.append(pat or '.')
 else: # relglob, re, relre
 r.append('.')
diff --git a/tests/test-locate.t b/tests/test-locate.t
--- a/tests/test-locate.t
+++ b/tests/test-locate.t
@@ -52,6 +52,12 @@
   t/b
   t/e.h
   t/x
+  $ hg locate files:
+  b
+  t.h
+  $ hg locate files:.
+  b
+  t.h
   $ hg locate -r 0 a
   a
   $ hg locate -r 0 NONEXISTENT
@@ -119,6 +125,13 @@
   ../t/e.h (glob)
   ../t/x (glob)

+  $ hg files files:
+  ../b (glob)
+  ../t.h (glob)
+  $ hg files files:.
+  ../b (glob)
+  ../t.h (glob)
+
   $ hg locate b
   ../b (glob)
   ../t/b (glob)
diff --git a/tests/test-walk.t b/tests/test-walk.t
--- a/tests/test-walk.t
+++ b/tests/test-walk.t
@@ -112,6 +112,8 @@
   f  beans/navy  ../beans/navy
   f  beans/pinto ../beans/pinto
   f  beans/turtle../beans/turtle
+  $ hg debugwalk -I 'files:mammals'
+  f  mammals/skunk  skunk
   $ hg debugwalk .
   f  mammals/Procyonidae/cacomistle  Procyonidae/cacomistle
   f  mammals/Procyonidae/coatimundi  Procyonidae/coatimundi
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel



--
Pierre-Yves David
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 1 of 3 v2] cmdutil: satisfy expections in dirstateguard.__del__, even if __init__ fails

[Pierre-Yves David]

On 10/13/2016 02:39 AM, Mads Kiilerich wrote:

# HG changeset patch
# User Mads Kiilerich 
# Date 1476317981 -7200
#  Thu Oct 13 02:19:41 2016 +0200
# Node ID 8db96226f748ff80003c0964bc4ece3b6c3e6d12
# Parent  b85fa6bf298be07804a74d8fdec0d19fdbc6d740
cmdutil: satisfy expections in dirstateguard.__del__, even if __init__ fails


That first one is obviously correct and pushed.

Thanks.

--
Pierre-Yves David
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 2 of 2 v2] util: increase filechunkiter size to 128k

[Yuya Nishihara]

On Fri, 14 Oct 2016 03:08:25 +0200, Mads Kiilerich wrote:
> # HG changeset patch
> # User Mads Kiilerich 
> # Date 1476402795 -7200
> #  Fri Oct 14 01:53:15 2016 +0200
> # Node ID 8909ce1593659bce0563cc66ca3fa34525daed65
> # Parent  b7889580507c3d1d40e61904c7a2c2aba335c6cd
> util: increase filechunkiter size to 128k

Queued the series, thanks.

> util.filechunkiter has been using a chunk size of 64k for more than 10 years,
> also in years where Moore's law still was a law. It is probably ok to bump it
> now and perhaps get a slight win in some cases.
> 
> Also, largefiles have been using 128k for a long time. Specifying that size
> multiple times (or forgetting to do it) seems a bit stupid. Decreasing it to
> 64k also seems unfortunate.
> 
> Thus, we will set the default chunksize to 128k and use the default 
> everywhere.

Perhaps there would be no practical difference between 64k and 128k. So
it seems okay to always use 128k.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 2 of 2 v2] util: increase filechunkiter size to 128k

[Pierre-Yves David]

On 10/14/2016 03:08 AM, Mads Kiilerich wrote:

# HG changeset patch
# User Mads Kiilerich 
# Date 1476402795 -7200
#  Fri Oct 14 01:53:15 2016 +0200
# Node ID 8909ce1593659bce0563cc66ca3fa34525daed65
# Parent  b7889580507c3d1d40e61904c7a2c2aba335c6cd
util: increase filechunkiter size to 128k


Pushed, thanks.

Cheers,

--
Pierre-Yves David
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 2 of 3] cmdutil: add support for evolution "troubles" display in changeset_printer

[Pierre-Yves David]

On 10/15/2016 02:36 PM, Yuya Nishihara wrote:

On Mon, 10 Oct 2016 14:33:18 +0200, Denis Laxalde wrote:

# HG changeset patch
# User Denis Laxalde 
# Date 1476094018 -7200
#  Mon Oct 10 12:06:58 2016 +0200
# Node ID d9f7776c40b8c82bf438322f2442d99c4e116161
# Parent  6c916ce602f5c92c5a5a4de954629670b8c7ca8c
# EXP-Topic evolve-ui
cmdutil: add support for evolution "troubles" display in changeset_printer

Add a "troubles" line in changeset header along with a couple of labels on
"log.changeset" line to indicate whether a changeset is troubled or not and
which kind trouble occurs.

Extract a _changesetlabels function to be reused in summary command.

diff --git a/mercurial/cmdutil.py b/mercurial/cmdutil.py
--- a/mercurial/cmdutil.py
+++ b/mercurial/cmdutil.py
@@ -1204,6 +1204,14 @@ def diffordiffstat(ui, repo, diffopts, n
 sub.diff(ui, diffopts, tempnode2, submatch, changes=changes,
  stat=stat, fp=fp, prefix=prefix)

+def _changesetlabels(ctx):
+labels = ['log.changeset', 'changeset.%s' % ctx.phasestr()]
+if ctx.troubled():
+labels.append('changeset.troubled')
+for trouble in ctx.troubles():
+labels.append('trouble.%s' % trouble)
+return ' '.join(labels)
+
 class changeset_printer(object):
 '''show changeset information when templating not requested.'''

@@ -1264,7 +1272,7 @@ class changeset_printer(object):

 # i18n: column positioning for "hg log"
 self.ui.write(_("changeset:   %d:%s\n") % revnode,
-  label='log.changeset changeset.%s' % ctx.phasestr())
+  label=_changesetlabels(ctx))

 # branches are shown first before any other names due to backwards
 # compatibility
@@ -1309,6 +1317,10 @@ class changeset_printer(object):
 self.ui.write(_("date:%s\n") % date,
   label='log.date')

+if ctx.troubled():
+self.ui.write(_("troubles:%s\n") % ', '.join(ctx.troubles()),
+  label='ui.note log.troubles')


The change looks good, but I have no idea if we can expose "troubles",
"troubled", etc. as a public API. I have a vague memory someone saying
"trouble" sounds trange to native speakers.


There have been discussion about the name of the trouble them-self too.

My personal opinion about this is:

- concerns have been expressed for years without actual progress made, 
so I think we should take this change at the beginning of the 4.1 cycle. 
This should give time to adjust naming if necessary.


- As long as Evolution and its bits are flagged as experimental we have 
some room to make change there.



And we'll need to update templates/map-cmdline.default to reflect this change.


+1, we could have some special mode that use the default cmdline 
template instead of the dedicated code and run the tests for it in a 
buildbot.


--
Pierre-Yves David
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 1 of 3] templatekw: use a "?" to represent troubled changesets in graph

[Pierre-Yves David]

On 10/15/2016 02:25 PM, Yuya Nishihara wrote:

On Mon, 10 Oct 2016 14:33:17 +0200, Denis Laxalde wrote:

# HG changeset patch
# User Denis Laxalde 
# Date 1475933106 -7200
#  Sat Oct 08 15:25:06 2016 +0200
# Node ID 6c916ce602f5c92c5a5a4de954629670b8c7ca8c
# Parent  dbcef8918bbdd8a64d9f79a37bcfa284a26f3a39
# EXP-Topic evolve-ui
templatekw: use a "?" to represent troubled changesets in graph

diff --git a/mercurial/templatekw.py b/mercurial/templatekw.py
--- a/mercurial/templatekw.py
+++ b/mercurial/templatekw.py
@@ -394,6 +394,8 @@ def showgraphnode(repo, ctx, **args):
 return '@'
 elif ctx.obsolete():
 return 'x'
+elif ctx.troubled():
+return '?'
 elif ctx.closesbranch():
 return '_'


Is this change an RFC or already discussed at the sprint?


This is an RFC, we had just a quick chat about it at the sprint.
Having a dedicated "char" for troubled changeset make senses, but I'm 
not a fan of using "?". We should probably do a small brainstorming of 
possible char at the beginning of the 4.1 cycle.


Cheers,

--
Pierre-Yves David
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 1 of 2] eol: do not wait on lack when writing cache

[Pierre-Yves David]

On 10/13/2016 05:39 PM, Pierre-Yves David wrote:

On 10/13/2016 05:34 PM, Yuya Nishihara wrote:

On Thu, 13 Oct 2016 17:28:02 +0200, Mads Kiilerich wrote:

On 10/13/2016 05:07 PM, Pierre-Yves David wrote:

On 10/13/2016 03:21 PM, Mads Kiilerich wrote:

On 10/13/2016 01:53 PM, Pierre-Yves David wrote:

# HG changeset patch
# User Pierre-Yves David 
# Date 1476359131 -7200
#  Thu Oct 13 13:45:31 2016 +0200
# Node ID 88cc944830d0c1895e527d6ca13687f1d5e1c785
# Parent  747e546c561fbf34d07cd30013eaf42b0190bb3b
eol: do not wait on lack when writing cache

The cache writing process is properly catching and handling the case
where the
lock is unavailable. However, it fails to specify the lock can failed
to be
acquired when requesting it. This is now fixed.


Hmm.

*If* the user has write access to the repo and *could* lock the repo,
then it seems reasonable that it waits for the lock and does the right
thing. It would be unfortunate to bail out early and happily
continue to
expose the less optimal state that read only users might have to deal
with.


The change introduced by this changeset make the cache in line with
how most of other cache works in Mercurial.


A part of the problem here might be that it is unclear to me what
happens with wait=False. I don't remember the details: will it continue
without locking or will it raise? It would be nice to get that clarified
in the localrepo docstrings.


LockHeld would be raised. So we'll need to catch LockError if wait=False.


And we actually only catch LockUnavailable, lets drop patch 1 for now
(patch 2 is still valid)


So looking more into this it appears that:

- This function is always run for every reposetup(), (irk)

- If the cache seems up to date the function end early, (okay)

- If the content within the locking scope is not run, the repository 
have an inconsistent state and can misbehave (irk)


My current strategy is to slowly back away from this code and leave it 
in its current state.


Cheers,

--
Pierre-Yves David
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 2 of 2] changegroup: skip delta when the underlying revlog do not use them

[Pierre-Yves David]

On 10/14/2016 09:01 AM, Gregory Szorc wrote:

Cool. I was going to author this patch when I got back home!

This patch will result in CPU regression for old clients having to re-deltify. 
It would be nice to have numbers for that. I'm optimistic it is roughly the 
same as the server gains and it won't be significant enough to not take the 
patch. We also don't have a perf* command to measure changegroup application 
for a single component IIRC. So getting data isn't trivial :/


As you just said, the redeltifying are probably around the same cost 
than the one we save with this patch (6 seconds for a full Mozilla 
clone). The extra cost seems reasonable to me in that case.


Cheers,

--
Pierre-Yves David
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 2 of 2] evolve: lock the working copy early in next and prev (issue5244)

[Pierre-Yves David]

On 10/08/2016 05:18 PM, Simon Farnsworth wrote:

# HG changeset patch
# User Simon Farnsworth 
# Date 1475939634 25200
#  Sat Oct 08 08:13:54 2016 -0700
# Node ID 8a0c9c0158b3e9574a4571af3dce9978844b825d
# Parent  ee284d7c5faa5d9f17853f51c17883844b985c58
evolve: lock the working copy early in next and prev (issue5244)

Both next and prev depend on a consistent working copy, but were waiting to
take the lock until they were ready to alter the working copy.

Take the lock before reading the working copy state, and do not release it
until we're definitely not going to change the working copy.


The fix approach seems good, but the implementation is a bit strangeto 
me. See inline.




diff --git a/hgext/evolve.py b/hgext/evolve.py
--- a/hgext/evolve.py
+++ b/hgext/evolve.py
@@ -2213,10 +2213,13 @@
 """update to parent revision

 Displays the summary line of the destination for clarity."""
-if True:
+wlock = None
+dryrunopt = opts['dry_run']
+if not dryrunopt:
+wlock = repo.wlock()
+try:
 wkctx = repo[None]
 wparents = wkctx.parents()
-dryrunopt = opts['dry_run']
 if len(wparents) != 1:
 raise error.Abort('merge in progress')
 if not opts['merge']:
@@ -2246,7 +2249,6 @@
 ret = hg.update(repo, p.rev())
 if not ret:
 tr = lock = None
-wlock = repo.wlock()
 try:
 lock = repo.lock()
 tr = repo.transaction('previous')
@@ -2258,13 +2260,22 @@
 tr.close()
 finally:
 lockmod.release(tr, lock, wlock)
+wlock = None
+else:
+lockmod.release(wlock)
+wlock = None
+
 displayer.show(p)
 return 0
 else:
+lockmod.release(wlock)
+wlock = None



Why are we issuing multiple lockmode.release call instead of just using 
the final finally clause ?



 for p in parents:
 displayer.show(p)
 ui.warn(_('multiple parents, explicitly update to one\n'))
 return 1
+finally:
+lockmod.release(wlock)

 @command('^next',
  [('B', 'move-bookmark', False,
@@ -2282,10 +2293,13 @@

 Displays the summary line of the destination for clarity.
 """
-if True:
+wlock = None
+dryrunopt = opts['dry_run']
+if not dryrunopt:
+wlock = repo.wlock()
+try:
 wkctx = repo[None]
 wparents = wkctx.parents()
-dryrunopt = opts['dry_run']
 if len(wparents) != 1:
 raise error.Abort('merge in progress')
 if not opts['merge']:
@@ -2315,7 +2329,6 @@
 ret = hg.update(repo, c.rev())
 if not ret:
 lock = tr = None
-wlock = repo.wlock()
 try:
 lock = repo.lock()
 tr = repo.transaction('next')
@@ -2327,15 +2340,23 @@
 tr.close()
 finally:
 lockmod.release(tr, lock, wlock)
+wlock = None
+else:
+lockmod.release(wlock)
+wlock = None
 displayer.show(c)
 result = 0
 elif children:
+lockmod.release(wlock)
+wlock = None
 ui.warn(_("ambigious next changeset:\n"))
 for c in children:
 displayer.show(c)
 ui.warn(_('explicitly update to one of them\n'))
 result = 1
 else:
+lockmod.release(wlock)
+wlock = None
 aspchildren = _aspiringchildren(repo, [repo['.'].rev()])
 if topic:
 filtered.extend(repo[c] for c in children
@@ -2368,6 +2389,8 @@
 return result
 return 1
 return result
+finally:
+lockmod.release(wlock)

 def _reachablefrombookmark(repo, revs, bookmarks):
 """filter revisions and bookmarks reachable from the given bookmark
diff --git a/tests/fake-editor.sh b/tests/fake-editor.sh
new file mode 100755
--- /dev/null
+++ b/tests/fake-editor.sh
@@ -0,0 +1,3 @@
+#!/bin/sh
+sleep 5
+echo "new desc" >> $1
diff --git a/tests/test-prev-next.t b/tests/test-prev-next.t
--- a/tests/test-prev-next.t
+++ b/tests/test-prev-next.t
@@ -206,3 +206,34 @@
   move:[5] added d
   atop:[6] added b (3)
   working directory is now at 47ea25be8aea
+
+prev and next should lock properly against other commands
+
+  $ hg init repo
+  $ cd repo
+  $ HGEDITOR=${TESTDIR}/fake-editor.sh
+  $ echo hi > foo
+  $ hg ci -Am 'one'
+  adding foo
+  $ echo bye > foo
+  $ hg ci -Am 'two'
+
+  $ hg amend --edit &
+  $ sleep 1
+  $ hg prev
+  waiting for lock on working directory of $TESTTMP/repo 

Re: [PATCH 3 of 3] color: debugcolor should emit the user-defined colors

[Yuya Nishihara]

On Sat, 15 Oct 2016 15:07:08 -0700, Danek Duvall wrote:
> Yuya Nishihara wrote:
> > On Thu, 13 Oct 2016 13:27:35 -0700, danek.duv...@oracle.com wrote:
> > > # HG changeset patch
> > > # User Danek Duvall 
> > > # Date 1476389401 25200
> > > #  Thu Oct 13 13:10:01 2016 -0700
> > > # Node ID c59334b806a8f10c80c2ea196eba5f4b1b86b229
> > > # Parent  333e875ce30da3760c8655e303d9bea554efe7e4
> > > color: debugcolor should emit the user-defined colors
> > > 
> > > This also fixes a long-standing bug that reversed the sense of the color
> > > name and the label used to print it, which was never relevant before.
> > > 
> > > diff --git a/hgext/color.py b/hgext/color.py
> > > --- a/hgext/color.py
> > > +++ b/hgext/color.py
> > > @@ -524,10 +524,16 @@ def debugcolor(ui, repo, **opts):
> > >  _styles = {}
> > >  for effect in _effects.keys():
> > >  _styles[effect] = effect
> > > +if _terminfo_params:
> > > +for k, v in ui.configitems('color'):
> > > +if k.startswith('color.'):
> > > +_styles[k] = k[6:]
> > > +elif k.startswith('terminfo.'):
> > > +_styles[k] = k[9:]
> > >  ui.write(('color mode: %s\n') % ui._colormode)
> > >  ui.write(_('available colors:\n'))
> > > -for label, colors in _styles.items():
> > > -ui.write(('%s\n') % colors, label=label)
> > > +for colorname, label in _styles.items():
> > > +ui.write(('%s\n') % colorname, label=label)
> > 
> > I was surprised to know that a label can be a valid effect name.
> 
> This bit confused me for a while, and maybe I didn't make things better in
> that regard.  There's "label" in the sense of the keyword argument to
> ui.write(), and "label" in the sense of the word that we're printing out
> here, labelling the color.  I used the latter sense in my comment, but the
> code uses the former sense.

Maybe we can refactor it to not modify _styles table. _styles is a
label-to-effects map and we build new _styles map in which label=colorname,
so colorname == label (in ui.label() sense), label (variable) == effect,
and colorui.write() can take label=label|effect. That made me a bit confused
while reviewing this patch.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 1 of 9 V6] exchange: add `_getbookmarks()` function

[Pierre-Yves David]

On 10/11/2016 06:25 PM, Stanislau Hlebik wrote:

# HG changeset patch
# User Stanislau Hlebik 
# Date 1476195835 25200
#  Tue Oct 11 07:23:55 2016 -0700
# Node ID 55e997127023d7208488c593adb933a1bfb23312
# Parent  b85fa6bf298be07804a74d8fdec0d19fdbc6d740
exchange: add `_getbookmarks()` function


I was thinking about the various in-flight series and it made me 
realized that the various important point about this bookmarks series 
are spread in various emails and might be a bit hard to follow.


Since I still had them in my head I've made a summary of the important 
point that needs to be addressed or discussed. Since the 4.0 feature 
freeze is Tuesday, this will probably not reach conclusion this cycle.


- Do not go through pushkey to list the bookmarks
  (using a function from mercurial.bookmarks seems the way to go),

- either drop the dedicated  exception classes (BookmarksEncodeError and 
BookmarksDecodeError) or make it clear why we need them,


- write internal documentations about the new parts and the process 
around them in mercurial/helps/internals/,


- double check the binary encoding with Greg. I personnaly think we 
should drop the optional 'nodeid' and include one in all case,


- introduce a new attributes on the unbundle2 to pass "inputs" around 
(instead of our previous approach of using dedicated attributes for each)


(there is also a small number of various questions and comments inline 
but they seems less important)


Cheers,

--
Pierre-Yves David
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] checkhash: merge subfunctions

[Pierre-Yves David]

On 10/08/2016 03:46 PM, Remi Chaintron wrote:

# HG changeset patch
# User Remi Chaintron 
# Date 1475932840 25200
#  Sat Oct 08 06:20:40 2016 -0700
# Node ID a7f89275e96e0e8fe594f99120034d3345130dcc
# Parent  3741a8f86e88702595c29f8ed824a28da0cfa961
checkhash: merge subfunctions


The fact that a function called 'checkhash' was in charge of returning 
the text.


Remi and I chatted about this and other things related to LFS and as a 
result Remi will be poking at an approach based on flags.


--
Pierre-Yves David
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] bundle2: only emit compressed chunks if they have data

[Pierre-Yves David]

On 10/16/2016 02:19 AM, Gregory Szorc wrote:

# HG changeset patch
# User Gregory Szorc 
# Date 1476576653 25200
#  Sat Oct 15 17:10:53 2016 -0700
# Node ID adac553f5fced14a7393e11c71e642eabbc82073
# Parent  5cb830801855dbb63e98b948e355bc995d295bf3
bundle2: only emit compressed chunks if they have data


pushed, thanks.

--
Pierre-Yves David
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] color: add some documentation for custom terminfo codes

[Pierre-Yves David]

On 10/16/2016 12:08 AM, danek.duv...@oracle.com wrote:

# HG changeset patch
# User Danek Duvall 
# Date 1476568874 25200
#  Sat Oct 15 15:01:14 2016 -0700
# Node ID 42f5a2db9e7f9d66058a7d77bc8c0c263272a8de
# Parent  c59334b806a8f10c80c2ea196eba5f4b1b86b229
color: add some documentation for custom terminfo codes


Pushed, thanks.

--
Pierre-Yves David
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel