Re: How to name template filter to remove duplicates from list?

[Yuya Nishihara]

On Fri, 16 Sep 2022 03:38:42 +0200, Manuel Jacob wrote:
> The Unix tool uniq removes only adjacent duplicates, which I found 
> surprising at first. I wouldn’t find "uniq" a good name for any variant 
> of the filter, since removing only adjacent duplicates might surprise 
> users not knowing the Unix tool, and removing all duplicates might 
> surprise users knowing the Unix tool.
> 
> The name "removeduplicates" is descriptive but quite long. It could take 
> an argument to choose whether all or only adjacent duplicates should be 
> removed. Examples:
> 
> "{'a\na\nb\na'|splitlines|removeduplicates}\n"
> -> "a b"  
> 
> "{removeduplicates('a\na\nb\na'|splitlines, onlyadjacent=False)}\n"
> -> "a b"  
> 
> "{removeduplicates('a\na\nb\na'|splitlines, onlyadjacent=True)}\n"
> -> "a b a"  

"uniq"/"unique" sounds good to me. Itertools of Rust provides unique()
for onlyadjacent=False, and dedup() for onlyadjacent=True.

https://docs.rs/itertools/latest/itertools/trait.Itertools.html#method.unique

I think the uniq command only uniquify adjacents with the assumption that
the input is sorted. The command could be used to deduplicate repeated lines
from unsorted text, but I've never used uniq for that use case.

I don't know if there's any use case of onlyadjacent=True in hg template.
___
Mercurial-devel mailing list
Mercurial-devel@lists.mercurial-scm.org
https://lists.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 1 of 2 stable] worker: do not suppress EINTR

[Yuya Nishihara]

On Wed, 25 May 2022 02:30:29 +0200, Manuel Jacob wrote:
> # HG changeset patch
> # User Manuel Jacob 
> # Date 1653433864 -7200
> #  Wed May 25 01:11:04 2022 +0200
> # Branch stable
> # Node ID d058898bdd462b03c5bff38ad40d1f855ea51c23
> # Parent  477b5145e1a02715f846ce017b460858a58e03b1
> # EXP-Topic worker-pickle-load-EINTR
> worker: do not suppress EINTR
> 
> Before this change, when IOError with errno EINTR was raised during
> pickle.load(), the error was suppressed and loading from other file 
> descriptors
> was continued.
> 
> On Python 3, system calls failing with EINTR are retried (PEP 475). Therefore,
> the removal of this code should not make any difference.
> 
> On Python 2, this is not generally the case. CPickle has no handling of EINTR.
> In one place it misinterprets it as EOF. In another place, it will raise
> IOError. However, this could happen in the middle of the stream. In this case,
> if pickle tries to load from the stream later, it will behave wrongly (usually
> it will raise an error, but loading of incorrect data or interpreter crashes
> are thinkable).

Do we want this (and the partial write fix) on stable?

The problem could occur if the payload were quite large, but I think it's
unlikely. Since writer issues a single os.write() in the original code,
reader wouldn't be interrupted in the middle of pickle message in general.

We don't have to think about Python 2 on default, so the patches will get
simplified.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 1 of 4] worker: explain why pickle reading stream has to be unbuffered

[Yuya Nishihara]

On Sun, 22 May 2022 12:03:02 +0900, Yuya Nishihara wrote:
> On Sun, 22 May 2022 02:37:11 +0200, Manuel Jacob wrote:
> > # HG changeset patch
> > # User Manuel Jacob 
> > # Date 1653164539 -7200
> > #  Sat May 21 22:22:19 2022 +0200
> > # Node ID 5dfd26ed9c4eeb51bbf910ca7450bdb65f06af81
> > # Parent  a17ffde1e71b1c9de5899ae8d5078e67c54ef1d3
> > # EXP-Topic worker-improvements
> > worker: explain why pickle reading stream has to be unbuffered  
> 
> Queued the series,

Oops.

Alphare, can you handle this series?
I don't know how to deal with emailed patches these days.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH stable] worker: avoid potential partial write of pickled data

[Yuya Nishihara]

On Sun, 22 May 2022 04:34:58 +0200, Manuel Jacob wrote:
> # HG changeset patch
> # User Manuel Jacob 
> # Date 1653184234 -7200
> #  Sun May 22 03:50:34 2022 +0200
> # Branch stable
> # Node ID beebf9c4b8ed6257c8f8bfeb5e9fcae6f54268d7
> # Parent  477b5145e1a02715f846ce017b460858a58e03b1
> # EXP-Topic worker-pickle-fix_partial_write
> worker: avoid potential partial write of pickled data
> 
> Previously, the code wrote the pickled data using os.write(). However,
> os.write() can write less bytes than passed to it. To trigger the problem, the
> pickled data had to be larger than 2147479552 bytes on my system.
> 
> Instead, open a file object and pass it to pickle.dump(). This also has the
> advantage that it doesn’t buffer the whole pickled data in memory.
> 
> Note that the opened file must be buffered because pickle doesn’t support
> unbuffered streams because unbuffered streams’ write() method might write less
> bytes than passed to it (like os.write()) but pickle.dump() relies on that all
> bytes are written (see https://github.com/python/cpython/issues/93050).
> 
> diff --git a/mercurial/worker.py b/mercurial/worker.py
> --- a/mercurial/worker.py
> +++ b/mercurial/worker.py
> @@ -255,8 +255,10 @@
>  os.close(r)
>  os.close(w)
>  os.close(rfd)
> +wf = os.fdopen(wfd, 'wb')
>  for result in func(*(staticargs + (pargs,))):
> -os.write(wfd, util.pickle.dumps(result))
> +util.pickle.dump(result, wf)
> +wf.flush()

It's probably better to write "with os.fdopen(wfd, 'wb') as wf:" to clarify
that the wf and wfd are closed there.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 1 of 4] worker: explain why pickle reading stream has to be unbuffered

[Yuya Nishihara]

On Sun, 22 May 2022 02:37:11 +0200, Manuel Jacob wrote:
> # HG changeset patch
> # User Manuel Jacob 
> # Date 1653164539 -7200
> #  Sat May 21 22:22:19 2022 +0200
> # Node ID 5dfd26ed9c4eeb51bbf910ca7450bdb65f06af81
> # Parent  a17ffde1e71b1c9de5899ae8d5078e67c54ef1d3
> # EXP-Topic worker-improvements
> worker: explain why pickle reading stream has to be unbuffered

Queued the series, thanks. I didn't know that memoryview can be
a context manager.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH stable] hghave: make black version regex work with newer versions of black

[Yuya Nishihara]

On Sun, 22 May 2022 02:43:05 +0200, Joerg Sonnenberger wrote:
> Am Sun, May 22, 2022 at 02:13:03AM +0200 schrieb Manuel Jacob:
> > # HG changeset patch
> > # User Manuel Jacob 
> > # Date 1653176900 -7200
> > #  Sun May 22 01:48:20 2022 +0200
> > # Branch stable
> > # Node ID 29f2716c5c54c7e0f7aa6d91979893f5d2078862
> > # Parent  477b5145e1a02715f846ce017b460858a58e03b1
> > # EXP-Topic black_version_regex
> > hghave: make black version regex work with newer versions of black

This works for me, but I hesitated to queue since test-check-format.t starts
detecting formatting errors with

  % black --version
  black, 22.3.0 (compiled: no)

> > Black commit 117891878e5be4d6b771ae5de299e51b679cea27 (included in black >=
> > 21.11b0) dropped the string "version " from the output of "black 
> > --version". To
> > make the regex work with newer black versions, make matching of "version "
> > optional.  
> 
> I had a patch like this locally, but newer black versions insist on
> incompatible output and that's where I stopped. There is also the issue
> that the regex itself seems wrong, e.g. the unescaped "." in the [].

[.] should be okay. "." has no special meaning in character set.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH STABLE] tags: fix typo in fast path detection of fnode resolution (issue6673)

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1648545349 -32400
#  Tue Mar 29 18:15:49 2022 +0900
# Branch stable
# Node ID 1da3259a1a38ed0ba6607c0830bb1aec461ec784
# Parent  9bb700223f000be88912382adee80b8a7afe5fcb
tags: fix typo in fast path detection of fnode resolution (issue6673)

If I understand it, mctx.readfast() is unreliable here if p1/p2 .hgtags
nodes differ, and tags on that branch would be randomly discarded
depending on which parent were picked.

The test case added by this patch would fail only on zstd-compressed
repository. I didn't try hard to stabilize the failure case.

diff --git a/mercurial/tags.py b/mercurial/tags.py
--- a/mercurial/tags.py
+++ b/mercurial/tags.py
@@ -808,7 +808,7 @@ class hgtagsfnodescache(object):
 # There is some no-merge changeset where p1 is null and p2 is set
 # Processing them as merge is just slower, but still gives a good
 # result.
-p2node = cl.node(p1rev)
+p2node = cl.node(p2rev)
 p2fnode = self.getfnode(p2node, computemissing=False)
 if p1fnode != p2fnode:
 # we cannot rely on readfast because we don't know against what
diff --git a/tests/test-tags.t b/tests/test-tags.t
--- a/tests/test-tags.t
+++ b/tests/test-tags.t
@@ -933,3 +933,58 @@ Avoid writing logs on trying to delete a
   a8a82d372bb35b42ff736e74f07c23bcd99c371f a
   a8a82d372bb35b42ff736e74f07c23bcd99c371f a
    a
+
+  $ cd ..
+
+.hgtags fnode should be properly resolved at merge revision (issue6673)
+
+  $ hg init issue6673
+  $ cd issue6673
+
+  $ touch a
+  $ hg ci -qAm a
+  $ hg branch -q stable
+  $ hg ci -m branch
+
+  $ hg up -q default
+  $ hg merge -q stable
+  $ hg ci -m merge
+
+ add tag to stable branch:
+
+  $ hg up -q stable
+  $ echo a >> a
+  $ hg ci -m a
+  $ hg tag whatever
+  $ hg log -GT'{rev} {tags}\n'
+  @  4 tip
+  |
+  o  3 whatever
+  |
+  | o  2
+  |/|
+  o |  1
+  |/
+  o  0
+  
+
+ merge tagged stable into default:
+
+  $ hg up -q default
+  $ hg merge -q  stable
+  $ hg ci -m merge
+  $ hg log -GT'{rev} {tags}\n'
+  @5 tip
+  |\
+  | o  4
+  | |
+  | o  3 whatever
+  | |
+  o |  2
+  |\|
+  | o  1
+  |/
+  o  0
+  
+
+  $ cd ..

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH] chgserver: remove Python 2 file descriptor logic

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1646357326 -32400
#  Fri Mar 04 10:28:46 2022 +0900
# Node ID 3a9729bead90ef7caaf4d25138680c83a888be78
# Parent  46b3ecfb16e2781ede9752d972dc22f0e1dfea87
chgserver: remove Python 2 file descriptor logic

Follows up 0bb28b7736bc "chgserver: remove Python 2 support code."

On Python 2, we had to close newfp prior to restoring the original file
description since "delete newfp" would otherwise close the file descriptor
shared with the long-lived fp:

  in attachio():
newfp = os.fdopen(fp.fileno(), mode, bufsize)
  in _restoreio():
newfp.close()  # temporarily close newfp.fileno() (= fp.fileno())
os.dup2(fd, fp.fileno())  # reopen fp.fileno() with original fd

On the other hand, we shouldn't call newfp.close() on Python 3 since
any function calls are proxied to the underlying file object by
procutil.LineBufferedWrapper.

diff --git a/mercurial/chgserver.py b/mercurial/chgserver.py
--- a/mercurial/chgserver.py
+++ b/mercurial/chgserver.py
@@ -438,14 +438,8 @@ class chgcmdserver(commandserver.server)
 nullfd = os.open(os.devnull, os.O_WRONLY)
 ui = self.ui
 for (ch, fp, fd), (cn, fn, mode) in zip(self._oldios, _iochannels):
-newfp = getattr(ui, fn)
-# On Python 3, newfp is just a wrapper around fp even if newfp is
-# not fp, so deleting newfp is safe.
-if newfp is not fp:
-newfp.close()
-# restore original fd: fp is open again
 try:
-if newfp is fp and 'w' in mode:
+if 'w' in mode:
 # Discard buffered data which couldn't be flushed because
 # of EPIPE. The data should belong to the current session
 # and should never persist.

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 1 of 2] cext: backout e9ca736f5b52 "remove Python 2 file handling code"

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1646268190 -32400
#  Thu Mar 03 09:43:10 2022 +0900
# Node ID 499733de460faf0d8cee6ea5e22bd05cec2fc93c
# Parent  7b068abe4aa2d1848cf91c2c203b68aa59feaaf7
cext: backout e9ca736f5b52 "remove Python 2 file handling code"

It's if"n"def.

diff --git a/mercurial/cext/osutil.c b/mercurial/cext/osutil.c
--- a/mercurial/cext/osutil.c
+++ b/mercurial/cext/osutil.c
@@ -1176,7 +1176,9 @@ static PyObject *posixfile(PyObject *sel
char fpmode[4];
int fppos = 0;
int plus;
+#ifndef IS_PY3K
FILE *fp;
+#endif
 
if (!PyArg_ParseTupleAndKeywords(args, kwds, "et|yi:posixfile",
 kwlist,
@@ -1248,6 +1250,7 @@ static PyObject *posixfile(PyObject *sel
PyErr_SetFromErrnoWithFilename(PyExc_IOError, name);
goto bail;
}
+#ifndef IS_PY3K
fp = _fdopen(fd, fpmode);
if (fp == NULL) {
_close(fd);
@@ -1262,6 +1265,11 @@ static PyObject *posixfile(PyObject *sel
}
 
PyFile_SetBufSize(file_obj, bufsize);
+#else
+   file_obj = PyFile_FromFd(fd, name, mode, bufsize, NULL, NULL, NULL, 1);
+   if (file_obj == NULL)
+   goto bail;
+#endif
 bail:
PyMem_Free(name);
return file_obj;

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 2] cext: really remove Python 2 file handling code

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1646268321 -32400
#  Thu Mar 03 09:45:21 2022 +0900
# Node ID 2ef3b7d30cc1df92f52bc5ef04b8e549db971095
# Parent  499733de460faf0d8cee6ea5e22bd05cec2fc93c
cext: really remove Python 2 file handling code

Disclaimer: This is _WIN32 code and I have no machine to test.

diff --git a/mercurial/cext/osutil.c b/mercurial/cext/osutil.c
--- a/mercurial/cext/osutil.c
+++ b/mercurial/cext/osutil.c
@@ -1176,9 +1176,6 @@ static PyObject *posixfile(PyObject *sel
char fpmode[4];
int fppos = 0;
int plus;
-#ifndef IS_PY3K
-   FILE *fp;
-#endif
 
if (!PyArg_ParseTupleAndKeywords(args, kwds, "et|yi:posixfile",
 kwlist,
@@ -1250,26 +1247,9 @@ static PyObject *posixfile(PyObject *sel
PyErr_SetFromErrnoWithFilename(PyExc_IOError, name);
goto bail;
}
-#ifndef IS_PY3K
-   fp = _fdopen(fd, fpmode);
-   if (fp == NULL) {
-   _close(fd);
-   PyErr_SetFromErrnoWithFilename(PyExc_IOError, name);
-   goto bail;
-   }
-
-   file_obj = PyFile_FromFile(fp, name, mode, fclose);
-   if (file_obj == NULL) {
-   fclose(fp);
-   goto bail;
-   }
-
-   PyFile_SetBufSize(file_obj, bufsize);
-#else
file_obj = PyFile_FromFd(fd, name, mode, bufsize, NULL, NULL, NULL, 1);
if (file_obj == NULL)
goto bail;
-#endif
 bail:
PyMem_Free(name);
return file_obj;

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH stable, v2] doc: inspect.getargspec has been removed in Python 3.11

[Yuya Nishihara]

On Mon, 07 Feb 2022 15:56:04 +0100, Mads Kiilerich wrote:
> # HG changeset patch
> # User Mads Kiilerich 
> # Date 1644245153 -3600
> #  Mon Feb 07 15:45:53 2022 +0100
> # Branch stable
> # Node ID 937db8e65dd5614cc4595cced61f2e21026d3be3
> # Parent  01fde63b4eded708802bfd0d0d4cb4ecc5ff6e1c
> doc: inspect.getargspec has been removed in Python 3.11

Queued for stable, thanks.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH stable] doc: inspect.getargspec has been removed in Python 3.11

[Yuya Nishihara]

On Sat, 05 Feb 2022 14:23:33 +0100, Mads Kiilerich wrote:
> # HG changeset patch
> # User Mads Kiilerich 
> # Date 1644066813 -3600
> #  Sat Feb 05 14:13:33 2022 +0100
> # Branch stable
> # Node ID e3c50cc1facafbd13120b7693155add1af9f94ed
> # Parent  01fde63b4eded708802bfd0d0d4cb4ecc5ff6e1c
> doc: inspect.getargspec has been removed in Python 3.11
> 
> Fix problem left over from 646002338365.
> 
> Reported for Fedora on https://bugzilla.redhat.com/show_bug.cgi?id=2022252#c2 
> .
> 
> diff --git a/doc/hgmanpage.py b/doc/hgmanpage.py
> --- a/doc/hgmanpage.py
> +++ b/doc/hgmanpage.py
> @@ -177,7 +177,11 @@ class Translator(nodes.NodeVisitor):
>  nodes.NodeVisitor.__init__(self, document)
>  self.settings = settings = document.settings
>  lcode = settings.language_code
> -arglen = len(inspect.getargspec(languages.get_language)[0])
> +try:
> +getfullargspec = inspect.getfullargspec
> +except:  # Python 2
   ^^
check-code complains about naked except clause.

I think it's okay to drop support for old docutils. cdda48c93676 says this
hack is for Docutils < 0.8.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH stable] cext: fix Python 3.11 compatibility - Py_SIZE is not an lvalue (issue6610)

[Yuya Nishihara]

On Thu, 18 Nov 2021 16:49:28 +0100, Mads Kiilerich wrote:
> # HG changeset patch
> # User Mads Kiilerich 
> # Date 1637235097 -3600
> #  Thu Nov 18 12:31:37 2021 +0100
> # Branch stable
> # Node ID 9c41494c5dd2fa8816bc7b70e76b28c4d3941af0
> # Parent  6e576e4665f43101ef9bb98b0f5234f85c5db2ea
> cext: fix Python 3.11 compatibility - Py_SIZE is not an lvalue (issue6610)

Queued for stable, thanks.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] hg: remove reserved identifiers

[Yuya Nishihara]

On Thu, 21 Oct 2021 11:21:33 +0200, David Demelier wrote:
> # HG changeset patch
> # User David Demelier 
> # Date 1634808081 -7200
> #  Thu Oct 21 11:21:21 2021 +0200
> # Node ID 84d470049d323b8137cd6bbc68a521612fdebad9
> # Parent  f7fd629ffb98d56e1023092e1210c4523f301f9b
> hg: remove reserved identifiers

Queued, thanks.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 1 of 2] dirstate: fix parse_dirstate() to error out if NULL entry created

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1632383057 -32400
#  Thu Sep 23 16:44:17 2021 +0900
# Node ID 13efd8fdc55fe3ed8ab195fff0c52a249d4dcc73
# Parent  1b2ee68e85f93c0726db9f2f739bc6768db1f92f
dirstate: fix parse_dirstate() to error out if NULL entry created

Since 83f0e93ec34b "dirstate-item: move the C implementation to the same
logic", dirstate_item_from_v1_data() gets more likely to return NULL, and
the fuzzer crashes because of that.

diff --git a/mercurial/cext/parsers.c b/mercurial/cext/parsers.c
--- a/mercurial/cext/parsers.c
+++ b/mercurial/cext/parsers.c
@@ -770,6 +770,8 @@ static PyObject *parse_dirstate(PyObject
 
entry = (PyObject *)dirstate_item_from_v1_data(state, mode,
   size, mtime);
+   if (!entry)
+   goto quit;
cpos = memchr(cur, 0, flen);
if (cpos) {
fname = PyBytes_FromStringAndSize(cur, cpos - cur);

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 2] dirstate: fix leak of entry object in dirstate_item_from_v1_data()

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1632383555 -32400
#  Thu Sep 23 16:52:35 2021 +0900
# Node ID bc82b526a41b47adfe8c49639b0664bcaa4b26b8
# Parent  13efd8fdc55fe3ed8ab195fff0c52a249d4dcc73
dirstate: fix leak of entry object in dirstate_item_from_v1_data()

diff --git a/mercurial/cext/parsers.c b/mercurial/cext/parsers.c
--- a/mercurial/cext/parsers.c
+++ b/mercurial/cext/parsers.c
@@ -352,6 +352,7 @@ dirstate_item_from_v1_data(char state, i
PyErr_Format(PyExc_RuntimeError,
 "unknown state: `%c` (%d, %d, %d)", state, mode,
 size, mtime, NULL);
+   Py_DECREF(t);
return NULL;
}
 

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] largefiles: properly pass kwargs into url.open

[Yuya Nishihara]

On Thu, 04 Feb 2021 17:06:09 -0500, Jordi Gutiérrez Hermoso wrote:
> # HG changeset patch
> # User Jordi Gutiérrez Hermoso 
> # Date 1612475986 18000
> #  Thu Feb 04 16:59:46 2021 -0500
> # Node ID fee215d5eb63abf93c50de4355fbd46123f50cba
> # Parent  95b276283b671cd835a2a0918f4297eb2baab425
> largefiles: properly pass kwargs into url.open

Queued for stable, thanks.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH STABLE] procutil: extend gui test to detect wayland session (issue6479)

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1612436205 -32400
#  Thu Feb 04 19:56:45 2021 +0900
# Branch stable
# Node ID 6cc6b8610cec85dc624f4bb6568c2fd16ac018d1
# Parent  14feaa5792ed116ec8cc8469fc3de976fb4dd8f5
procutil: extend gui test to detect wayland session (issue6479)

diff --git a/mercurial/utils/procutil.py b/mercurial/utils/procutil.py
--- a/mercurial/utils/procutil.py
+++ b/mercurial/utils/procutil.py
@@ -546,7 +546,11 @@ def _gui():
 # pure build; use a safe default
 return True
 else:
-return pycompat.iswindows or encoding.environ.get(b"DISPLAY")
+return (
+pycompat.iswindows
+or encoding.environ.get(b"DISPLAY")
+or encoding.environ.get(b"WAYLAND_DISPLAY")
+)
 
 
 def gui():

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH STABLE] log: fix handling of root (or empty) path provided by matcher (issue6478)

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1612264817 -32400
#  Tue Feb 02 20:20:17 2021 +0900
# Branch stable
# Node ID 14feaa5792ed116ec8cc8469fc3de976fb4dd8f5
# Parent  0e2e7300f4302b02412b0b734717697049494c4c
log: fix handling of root (or empty) path provided by matcher (issue6478)

Since 27d6956d386b "match: use '' instead of '.' for root directory",
'.' should be translated to ''. We can't blame repo.file() about this because
an empty string is invalid as a file path, but I found at least two callers
(_makematcher() and revset.filelog()) would crash because of this path[0].

So let's make repo.file() accept an empty string. path[0] == b'/' wouldn't
work on Python 3 anyways.

diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -1135,7 +1135,7 @@ class revlogfilestorage(object):
 """File storage when using revlogs."""
 
 def file(self, path):
-if path[0] == b'/':
+if path.startswith(b'/'):
 path = path[1:]
 
 return filelog.filelog(self.svfs, path)
@@ -1146,7 +1146,7 @@ class revlognarrowfilestorage(object):
 """File storage when using revlogs and narrow files."""
 
 def file(self, path):
-if path[0] == b'/':
+if path.startswith(b'/'):
 path = path[1:]
 
 return filelog.narrowfilelog(self.svfs, path, self._storenarrowmatch)
diff --git a/mercurial/logcmdutil.py b/mercurial/logcmdutil.py
--- a/mercurial/logcmdutil.py
+++ b/mercurial/logcmdutil.py
@@ -845,7 +845,7 @@ def _makematcher(repo, revs, wopts):
 # slowpath; otherwise, we can turn off the slowpath
 if slowpath:
 for path in match.files():
-if path == b'.' or path in repo.store:
+if not path or path in repo.store:
 break
 else:
 slowpath = False
diff --git a/tests/test-log.t b/tests/test-log.t
--- a/tests/test-log.t
+++ b/tests/test-log.t
@@ -102,6 +102,41 @@ log on directory
   summary: c
   
 
+log empty path (or repo root) of slow path shouldn't crash (issue6478)
+
+  $ hg log -ql1 '' inexistent
+  4:7e4639b4691b
+  $ hg log -ql1 . inexistent
+  4:7e4639b4691b
+  $ hg log -ql1 "`pwd`" inexistent
+  4:7e4639b4691b
+
+  $ hg log -ql1 '' e
+  4:7e4639b4691b
+  $ hg log -ql1 . e
+  4:7e4639b4691b
+  $ hg log -ql1 "`pwd`" e
+  4:7e4639b4691b
+
+log -f empty path (or repo root) shouldn't crash
+
+  $ hg log -qfl1 '' inexistent
+  abort: cannot follow file not in parent revision: "inexistent"
+  [255]
+  $ hg log -qfl1 . inexistent
+  abort: cannot follow file not in parent revision: "inexistent"
+  [255]
+  $ hg log -qfl1 "`pwd`" inexistent
+  abort: cannot follow file not in parent revision: "inexistent"
+  [255]
+
+  $ hg log -qfl1 '' e
+  4:7e4639b4691b
+  $ hg log -qfl1 . e
+  4:7e4639b4691b
+  $ hg log -qfl1 "`pwd`" e
+  4:7e4639b4691b
+
 -X, with explicit path
 
   $ hg log a -X a

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH STABLE] log: update documentation about --follow with/without --rev (issue6459)

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1608793088 -32400
#  Thu Dec 24 15:58:08 2020 +0900
# Branch stable
# Node ID 6ad7f9653fef0cba2d3a6858c4ac83aeb50f577f
# Parent  734d051d0efb8122e1680895b214b1d143895273
log: update documentation about --follow with/without --rev (issue6459)

diff --git a/mercurial/commands.py b/mercurial/commands.py
--- a/mercurial/commands.py
+++ b/mercurial/commands.py
@@ -4386,7 +4386,7 @@ def locate(ui, repo, *pats, **opts):
 b'r',
 b'rev',
 [],
-_(b'show the specified revision or revset'),
+_(b'revisions to select or follow from'),
 _(b'REV'),
 ),
 (
@@ -4448,13 +4448,13 @@ def log(ui, repo, *pats, **opts):
 project.
 
 If no revision range is specified, the default is ``tip:0`` unless
---follow is set, in which case the working directory parent is
-used as the starting revision.
+--follow is set.
 
 File history is shown without following rename or copy history of
 files. Use -f/--follow with a filename to follow history across
 renames and copies. --follow without a filename will only show
-ancestors of the starting revision.
+ancestors of the starting revisions. The starting revisions can be
+specified by -r/--rev, which default to the working directory parent.
 
 By default this command prints revision number and changeset id,
 tags, non-trivial parents, user, date and time, and a summary for

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 4 of 4 V2] procutil: assign pseudo file object if sys.stdout/stderr is missing

[Yuya Nishihara]

On Mon, 21 Dec 2020 08:15:29 +0100, Pierre-Yves David wrote:
> On 12/19/20 3:49 AM, Yuya Nishihara wrote:
> > # HG changeset patch
> > # User Yuya Nishihara 
> > # Date 1608289751 -32400
> > #  Fri Dec 18 20:09:11 2020 +0900
> > # Node ID 09beb9a133f168111fbef4a729f1c0cc2bfb87ce
> > # Parent  ca07a5705bf78401bf7c910b9be9a6c5a16ceb64
> > procutil: assign pseudo file object if sys.stdout/stderr is missing
> > 
> > This basically simulates the Python 2 behavior. If libc stdio were used,
> > these file objects would be available and raise EBADF. There is subtle
> > difference between py2 and py3, but I think py3 behavior (i.e. exit 255)
> > is more correct.
> 
> Does this means you are considering moving to the py3 ways once we drop 
> py2 support ?

No for availability of procutil.stdin/out/err. Setting them to None would
be source of problems.

I just meant exiting with EBADF (and code 255) is more correct if stdout fd
is closed. I don't know why it doesn't on Python 2.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 2] chg: format code by clang-format version 11.0.1-+rc1-1

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1608347244 -32400
#  Sat Dec 19 12:07:24 2020 +0900
# Node ID babd57945ad10283f753b25d15ab593811959a70
# Parent  5db45dfd6206490f8668559838cc4d4ca9ac
chg: format code by clang-format version 11.0.1-+rc1-1

test-check-clang-format.t started failing on my Debian sid environment, and
new style looks slightly better. So let's bump the required clang-format
version to 11.

diff --git a/contrib/chg/chg.c b/contrib/chg/chg.c
--- a/contrib/chg/chg.c
+++ b/contrib/chg/chg.c
@@ -451,9 +451,10 @@ static int runinstructions(struct cmdser
  */
 static int isunsupported(int argc, const char *argv[])
 {
-   enum { SERVE = 1,
-  DAEMON = 2,
-  SERVEDAEMON = SERVE | DAEMON,
+   enum {
+   SERVE = 1,
+   DAEMON = 2,
+   SERVEDAEMON = SERVE | DAEMON,
};
unsigned int state = 0;
int i;
diff --git a/contrib/chg/hgclient.c b/contrib/chg/hgclient.c
--- a/contrib/chg/hgclient.c
+++ b/contrib/chg/hgclient.c
@@ -26,15 +26,16 @@
 #include "procutil.h"
 #include "util.h"
 
-enum { CAP_GETENCODING = 0x0001,
-   CAP_RUNCOMMAND = 0x0002,
-   /* cHg extension: */
-   CAP_ATTACHIO = 0x0100,
-   CAP_CHDIR = 0x0200,
-   CAP_SETENV = 0x0800,
-   CAP_SETUMASK2 = 0x1000,
-   CAP_VALIDATE = 0x2000,
-   CAP_SETPROCNAME = 0x4000,
+enum {
+   CAP_GETENCODING = 0x0001,
+   CAP_RUNCOMMAND = 0x0002,
+   /* cHg extension: */
+   CAP_ATTACHIO = 0x0100,
+   CAP_CHDIR = 0x0200,
+   CAP_SETENV = 0x0800,
+   CAP_SETUMASK2 = 0x1000,
+   CAP_VALIDATE = 0x2000,
+   CAP_SETPROCNAME = 0x4000,
 };
 
 typedef struct {
diff --git a/tests/hghave.py b/tests/hghave.py
--- a/tests/hghave.py
+++ b/tests/hghave.py
@@ -592,8 +592,8 @@ def has_pylint():
 @check("clang-format", "clang-format C code formatter")
 def has_clang_format():
 m = matchoutput('clang-format --version', br'clang-format version (\d+)')
-# style changed somewhere between 4.x and 6.x
-return m and int(m.group(1)) >= 6
+# style changed somewhere between 10.x and 11.x
+return m and int(m.group(1)) >= 11
 
 
 @check("jshint", "JSHint static code analysis tool")

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 1 of 2] test-check-interface: do not expect bare "python" executable exists

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1608346918 -32400
#  Sat Dec 19 12:01:58 2020 +0900
# Node ID 5db45dfd6206490f8668559838cc4d4ca9ac
# Parent  09beb9a133f168111fbef4a729f1c0cc2bfb87ce
test-check-interface: do not expect bare "python" executable exists

The test would crash if python executable had version suffix.

diff --git a/tests/test-check-interfaces.py b/tests/test-check-interfaces.py
--- a/tests/test-check-interfaces.py
+++ b/tests/test-check-interfaces.py
@@ -12,7 +12,7 @@ import sys
 
 # Only run if tests are run in a repo
 if subprocess.call(
-['python', '%s/hghave' % os.environ['TESTDIR'], 'test-repo']
+[sys.executable, '%s/hghave' % os.environ['TESTDIR'], 'test-repo']
 ):
 sys.exit(80)
 

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: D9059: procutil: assign stdio objects if they are None

[Yuya Nishihara]

As Pulkit pointed out, I and Pulkit made another version in parallel.

https://patchwork.mercurial-scm.org/patch/47937/
https://patchwork.mercurial-scm.org/patch/47938/
https://patchwork.mercurial-scm.org/patch/47939/

`sys.std*` are left unmodified. `procutil.std*` are fixed up to raise EBADF.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 4 of 4 V2] procutil: assign pseudo file object if sys.stdout/stderr is missing

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1608289751 -32400
#  Fri Dec 18 20:09:11 2020 +0900
# Node ID 09beb9a133f168111fbef4a729f1c0cc2bfb87ce
# Parent  ca07a5705bf78401bf7c910b9be9a6c5a16ceb64
procutil: assign pseudo file object if sys.stdout/stderr is missing

This basically simulates the Python 2 behavior. If libc stdio were used,
these file objects would be available and raise EBADF. There is subtle
difference between py2 and py3, but I think py3 behavior (i.e. exit 255)
is more correct.

"if" conditions are adjust so that they look similar to
dispatch.initstdio().

diff --git a/mercurial/utils/procutil.py b/mercurial/utils/procutil.py
--- a/mercurial/utils/procutil.py
+++ b/mercurial/utils/procutil.py
@@ -131,17 +131,25 @@ def _make_write_all(stream):
 
 
 if pycompat.ispy3:
-# Python 3 implements its own I/O streams.
+# Python 3 implements its own I/O streams. Unlike stdio of C library,
+# sys.stdin/stdout/stderr may be None if underlying fd is closed.
+
 # TODO: .buffer might not exist if std streams were replaced; we'll need
 # a silly wrapper to make a bytes stream backed by a unicode one.
 
-# sys.stdin can be None
-if sys.stdin:
+if sys.stdin is None:
+stdin = BadFile()
+else:
 stdin = sys.stdin.buffer
+if sys.stdout is None:
+stdout = BadFile()
 else:
-stdin = BadFile()
-stdout = _make_write_all(sys.stdout.buffer)
-stderr = _make_write_all(sys.stderr.buffer)
+stdout = _make_write_all(sys.stdout.buffer)
+if sys.stderr is None:
+stderr = BadFile()
+else:
+stderr = _make_write_all(sys.stderr.buffer)
+
 if pycompat.iswindows:
 # Work around Windows bugs.
 stdout = platform.winstdout(stdout)
diff --git a/tests/test-basic.t b/tests/test-basic.t
--- a/tests/test-basic.t
+++ b/tests/test-basic.t
@@ -49,6 +49,31 @@ Writes to stdio succeed and fail appropr
   [255]
 #endif
 
+On Python 3, stdio may be None:
+
+  $ hg debuguiprompt --config ui.interactive=true 0<&-
+   abort: Bad file descriptor
+  [255]
+  $ hg version -q 0<&-
+  Mercurial Distributed SCM * (glob)
+
+#if py3
+  $ hg version -q 1>&-
+  abort: Bad file descriptor
+  [255]
+#else
+  $ hg version -q 1>&-
+#endif
+  $ hg unknown -q 1>&-
+  hg: unknown command 'unknown'
+  (did you mean debugknown?)
+  [255]
+
+  $ hg version -q 2>&-
+  Mercurial Distributed SCM * (glob)
+  $ hg unknown -q 2>&-
+  [255]
+
   $ hg commit -m test
 
 This command is ancient:

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 3 of 4 V2] procutil: introduce pseudo file object that just raises EBADF

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1608343818 -32400
#  Sat Dec 19 11:10:18 2020 +0900
# Node ID ca07a5705bf78401bf7c910b9be9a6c5a16ceb64
# Parent  a5fa4ea662d52a6eccb538e79ca36b2a4745a9b8
procutil: introduce pseudo file object that just raises EBADF

This should be safer than closing underlying fd as the fd may be reused.

On Python 2, closed sys.stdin could be redirected to a random file having
fd=0, but we'd be better not copying this behavior.

Only readinto() and write() are implemented according to the following ABC
table. fileno() is not implemented since fd=0/1/2 may be assigned later
to other files.

https://docs.python.org/3/library/io.html#class-hierarchy

diff --git a/mercurial/utils/procutil.py b/mercurial/utils/procutil.py
--- a/mercurial/utils/procutil.py
+++ b/mercurial/utils/procutil.py
@@ -50,6 +50,16 @@ def isatty(fp):
 return False
 
 
+class BadFile(io.RawIOBase):
+"""Dummy file object to simulate closed stdio behavior"""
+
+def readinto(self, b):
+raise IOError(errno.EBADF, 'Bad file descriptor')
+
+def write(self, b):
+raise IOError(errno.EBADF, 'Bad file descriptor')
+
+
 class LineBufferedWrapper(object):
 def __init__(self, orig):
 self.orig = orig
@@ -129,8 +139,7 @@ if pycompat.ispy3:
 if sys.stdin:
 stdin = sys.stdin.buffer
 else:
-stdin = open(os.devnull, 'rb')
-os.close(stdin.fileno())
+stdin = BadFile()
 stdout = _make_write_all(sys.stdout.buffer)
 stderr = _make_write_all(sys.stderr.buffer)
 if pycompat.iswindows:

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 4 V2] dispatch: gate against missing stdout/stderr

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1608290094 -32400
#  Fri Dec 18 20:14:54 2020 +0900
# Node ID a5fa4ea662d52a6eccb538e79ca36b2a4745a9b8
# Parent  a05f8740ce1faeda3f6299c3cbf961be48652a06
dispatch: gate against missing stdout/stderr

We do need procutil.stdin/stdout/stderr, but we don't care much for sys.std*.
Let's leave them be None as it is the Python 3 way.

diff --git a/mercurial/dispatch.py b/mercurial/dispatch.py
--- a/mercurial/dispatch.py
+++ b/mercurial/dispatch.py
@@ -166,26 +166,34 @@ if pycompat.ispy3:
 # "just work," here we change the sys.* streams to disable line ending
 # normalization, ensuring compatibility with our ui type.
 
-# write_through is new in Python 3.7.
-kwargs = {
-"newline": "\n",
-"line_buffering": sys.stdout.line_buffering,
-}
-if util.safehasattr(sys.stdout, "write_through"):
-kwargs["write_through"] = sys.stdout.write_through
-sys.stdout = io.TextIOWrapper(
-sys.stdout.buffer, sys.stdout.encoding, sys.stdout.errors, **kwargs
-)
+if sys.stdout is not None:
+# write_through is new in Python 3.7.
+kwargs = {
+"newline": "\n",
+"line_buffering": sys.stdout.line_buffering,
+}
+if util.safehasattr(sys.stdout, "write_through"):
+kwargs["write_through"] = sys.stdout.write_through
+sys.stdout = io.TextIOWrapper(
+sys.stdout.buffer,
+sys.stdout.encoding,
+sys.stdout.errors,
+**kwargs
+)
 
-kwargs = {
-"newline": "\n",
-"line_buffering": sys.stderr.line_buffering,
-}
-if util.safehasattr(sys.stderr, "write_through"):
-kwargs["write_through"] = sys.stderr.write_through
-sys.stderr = io.TextIOWrapper(
-sys.stderr.buffer, sys.stderr.encoding, sys.stderr.errors, **kwargs
-)
+if sys.stderr is not None:
+kwargs = {
+"newline": "\n",
+"line_buffering": sys.stderr.line_buffering,
+}
+if util.safehasattr(sys.stderr, "write_through"):
+kwargs["write_through"] = sys.stderr.write_through
+sys.stderr = io.TextIOWrapper(
+sys.stderr.buffer,
+sys.stderr.encoding,
+sys.stderr.errors,
+**kwargs
+)
 
 if sys.stdin is not None:
 # No write_through on read-only stream.
@@ -200,6 +208,8 @@ if pycompat.ispy3:
 
 def _silencestdio():
 for fp in (sys.stdout, sys.stderr):
+if fp is None:
+continue
 # Check if the file is okay
 try:
 fp.flush()

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 1 of 4 V2] dispatch: remove stale comment about fdopen()-ed stdio

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1608291311 -32400
#  Fri Dec 18 20:35:11 2020 +0900
# Node ID a05f8740ce1faeda3f6299c3cbf961be48652a06
# Parent  d109dda4a3e7bb1c6fa7de98b6872ddb80ec3827
dispatch: remove stale comment about fdopen()-ed stdio

On Python 3, stdout is just wrapped by LineBufferedWrapper.

diff --git a/mercurial/dispatch.py b/mercurial/dispatch.py
--- a/mercurial/dispatch.py
+++ b/mercurial/dispatch.py
@@ -207,9 +207,7 @@ if pycompat.ispy3:
 except IOError:
 pass
 # Otherwise mark it as closed to silence "Exception ignored in"
-# message emitted by the interpreter finalizer. Be careful to
-# not close procutil.stdout, which may be a fdopen-ed file object
-# and its close() actually closes the underlying file descriptor.
+# message emitted by the interpreter finalizer.
 try:
 fp.close()
 except IOError:

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 3 of 3] dispatch: remove stale comment about fdopen()-ed stdio

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1608291311 -32400
#  Fri Dec 18 20:35:11 2020 +0900
# Node ID 567d926ac2b6abfdcd0367540921dfccf8709c71
# Parent  bd4e513e64c974c38963b60fc14d4bbce6d574fe
dispatch: remove stale comment about fdopen()-ed stdio

On Python 3, stdout is just wrapped by LineBufferedWrapper.

diff --git a/mercurial/dispatch.py b/mercurial/dispatch.py
--- a/mercurial/dispatch.py
+++ b/mercurial/dispatch.py
@@ -217,9 +217,7 @@ if pycompat.ispy3:
 except IOError:
 pass
 # Otherwise mark it as closed to silence "Exception ignored in"
-# message emitted by the interpreter finalizer. Be careful to
-# not close procutil.stdout, which may be a fdopen-ed file object
-# and its close() actually closes the underlying file descriptor.
+# message emitted by the interpreter finalizer.
 try:
 fp.close()
 except IOError:

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 3] procutil: create dummy file object if sys.stdout/stderr is missing

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1608289751 -32400
#  Fri Dec 18 20:09:11 2020 +0900
# Node ID bd4e513e64c974c38963b60fc14d4bbce6d574fe
# Parent  47ce0f83ffad4fd66e4f76ea54733756398a98ba
procutil: create dummy file object if sys.stdout/stderr is missing

This basically simulates the Python 2 behavior. If libc stdio were used,
these file objects would be available and raise EBADF. There is subtle
difference between py2 and py3, but I think py3 behavior (i.e. exit 255)
is more correct.

"if" conditions are adjust so that they look similar to dispatch.initstdio().

diff --git a/mercurial/utils/procutil.py b/mercurial/utils/procutil.py
--- a/mercurial/utils/procutil.py
+++ b/mercurial/utils/procutil.py
@@ -120,19 +120,36 @@ def _make_write_all(stream):
 return WriteAllWrapper(stream)
 
 
+def _make_bad_file(mode):
+"""Creates a file which will raise EBADF for any I/O operation
+
+This simulates the Python 2 behavior of closed stdio.
+"""
+f = open(os.devnull, mode)
+os.close(f.fileno())
+return f
+
+
 if pycompat.ispy3:
-# Python 3 implements its own I/O streams.
+# Python 3 implements its own I/O streams. Unlike stdio of C library,
+# sys.stdin/stdout/stderr may be None if underlying fd is closed.
+
 # TODO: .buffer might not exist if std streams were replaced; we'll need
 # a silly wrapper to make a bytes stream backed by a unicode one.
 
-# sys.stdin can be None
-if sys.stdin:
+if sys.stdin is None:
+stdin = _make_bad_file('rb')
+else:
 stdin = sys.stdin.buffer
+if sys.stdout is None:
+stdout = _make_bad_file('wb')
 else:
-stdin = open(os.devnull, 'rb')
-os.close(stdin.fileno())
-stdout = _make_write_all(sys.stdout.buffer)
-stderr = _make_write_all(sys.stderr.buffer)
+stdout = _make_write_all(sys.stdout.buffer)
+if sys.stderr is None:
+stderr = _make_bad_file('wb')
+else:
+stderr = _make_write_all(sys.stderr.buffer)
+
 if pycompat.iswindows:
 # Work around Windows bugs.
 stdout = platform.winstdout(stdout)
diff --git a/tests/test-basic.t b/tests/test-basic.t
--- a/tests/test-basic.t
+++ b/tests/test-basic.t
@@ -49,6 +49,26 @@ Writes to stdio succeed and fail appropr
   [255]
 #endif
 
+On Python 3, stdio may be None:
+
+  $ hg version -q 0<&-
+  Mercurial Distributed SCM * (glob)
+#if py3
+  $ hg version -q 1>&-
+  abort: Bad file descriptor
+  [255]
+#else
+  $ hg version -q 1>&-
+#endif
+  $ hg version -q 2>&-
+  Mercurial Distributed SCM * (glob)
+  $ hg unknown -q 1>&-
+  hg: unknown command 'unknown'
+  (did you mean debugknown?)
+  [255]
+  $ hg unknown -q 2>&-
+  [255]
+
   $ hg commit -m test
 
 This command is ancient:

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 1 of 3] dispatch: gate against missing stdout/stderr

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1608290094 -32400
#  Fri Dec 18 20:14:54 2020 +0900
# Node ID 47ce0f83ffad4fd66e4f76ea54733756398a98ba
# Parent  d109dda4a3e7bb1c6fa7de98b6872ddb80ec3827
dispatch: gate against missing stdout/stderr

diff --git a/mercurial/dispatch.py b/mercurial/dispatch.py
--- a/mercurial/dispatch.py
+++ b/mercurial/dispatch.py
@@ -166,26 +166,34 @@ if pycompat.ispy3:
 # "just work," here we change the sys.* streams to disable line ending
 # normalization, ensuring compatibility with our ui type.
 
-# write_through is new in Python 3.7.
-kwargs = {
-"newline": "\n",
-"line_buffering": sys.stdout.line_buffering,
-}
-if util.safehasattr(sys.stdout, "write_through"):
-kwargs["write_through"] = sys.stdout.write_through
-sys.stdout = io.TextIOWrapper(
-sys.stdout.buffer, sys.stdout.encoding, sys.stdout.errors, **kwargs
-)
+if sys.stdout is not None:
+# write_through is new in Python 3.7.
+kwargs = {
+"newline": "\n",
+"line_buffering": sys.stdout.line_buffering,
+}
+if util.safehasattr(sys.stdout, "write_through"):
+kwargs["write_through"] = sys.stdout.write_through
+sys.stdout = io.TextIOWrapper(
+sys.stdout.buffer,
+sys.stdout.encoding,
+sys.stdout.errors,
+**kwargs
+)
 
-kwargs = {
-"newline": "\n",
-"line_buffering": sys.stderr.line_buffering,
-}
-if util.safehasattr(sys.stderr, "write_through"):
-kwargs["write_through"] = sys.stderr.write_through
-sys.stderr = io.TextIOWrapper(
-sys.stderr.buffer, sys.stderr.encoding, sys.stderr.errors, **kwargs
-)
+if sys.stderr is not None:
+kwargs = {
+"newline": "\n",
+"line_buffering": sys.stderr.line_buffering,
+}
+if util.safehasattr(sys.stderr, "write_through"):
+kwargs["write_through"] = sys.stderr.write_through
+sys.stderr = io.TextIOWrapper(
+sys.stderr.buffer,
+sys.stderr.encoding,
+sys.stderr.errors,
+**kwargs
+)
 
 if sys.stdin is not None:
 # No write_through on read-only stream.
@@ -200,6 +208,8 @@ if pycompat.ispy3:
 
 def _silencestdio():
 for fp in (sys.stdout, sys.stderr):
+if fp is None:
+continue
 # Check if the file is okay
 try:
 fp.flush()

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 2 of 3 chg] procutil: use os.devnull if sys.stdout/err is None

[Yuya Nishihara]

On Fri, 18 Dec 2020 14:16:20 +0530, Pulkit Goyal wrote:
> # HG changeset patch
> # User Pulkit Goyal <7895pul...@gmail.com>
> # Date 1607427603 -19800
> #  Tue Dec 08 17:10:03 2020 +0530
> # Node ID 9f0555329584b3e91e74e0606d66d16dd7a4c82f
> # Parent  e40f52691dac94d70a251b65806a4d1f4d2b5f19
> # EXP-Topic chg-test
> procutil: use os.devnull if sys.stdout/err is None
> 
> According to Yuya, it's better to make `procutil.stdin/err/out` always
> available.
> 
> This is a bandaid until someone suggests something better.
> 
> diff -r e40f52691dac -r 9f0555329584 mercurial/utils/procutil.py
> --- a/mercurial/utils/procutil.py Fri Apr 03 20:30:36 2020 +0530
> +++ b/mercurial/utils/procutil.py Tue Dec 08 17:10:03 2020 +0530
> @@ -131,8 +131,16 @@
>  else:
>  stdin = open(os.devnull, 'rb')
>  os.close(stdin.fileno())
> -stdout = _make_write_all(sys.stdout.buffer)
> -stderr = _make_write_all(sys.stderr.buffer)
> +if sys.stdout:
> +stdout = _make_write_all(sys.stdout.buffer)
> +else:
> +stdout = open(os.devnull, 'rb')
> +os.close(stdout.fileno())
> +if sys.stderr:
> +stderr = _make_write_all(sys.stderr.buffer)
> +else:
> +stderr = open(os.devnull, 'rb')

stdout/stderr should theoretically be 'wb'. I'll send a reworked version.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 1 of 3 chg] contrib: run python3+chg tests too in heptapod CI

[Yuya Nishihara]

On Fri, 18 Dec 2020 14:16:19 +0530, Pulkit Goyal wrote:
> # HG changeset patch
> # User Pulkit Goyal <7895pul...@gmail.com>
> # Date 1607767078 -19800
> #  Sat Dec 12 15:27:58 2020 +0530
> # Node ID 5179a4bd4fc35890cbfd962e0a8960a2f74720a1
> # Parent  e4f6dae01b3bc94b0198722522a13c914ab848cd
> # EXP-Topic chg-test
> contrib: run python3+chg tests too in heptapod CI

Queued 1 and 3, thanks.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 2 of 2 chg-tests-fix V3] procutil: don't assign stdin to None, use os.devnull instead

[Yuya Nishihara]

On Thu, 10 Dec 2020 14:41:45 +0530, Pulkit Goyal wrote:
> # HG changeset patch
> # User Pulkit Goyal <7895pul...@gmail.com>
> # Date 1607589226 -19800
> #  Thu Dec 10 14:03:46 2020 +0530
> # Node ID 9123fc028ed21e8aa39a353ea6ec9e102cf3dad0
> # Parent  96c66ca3f189d522dd53c736919e7343c29e6116
> # EXP-Topic chg-test
> procutil: don't assign stdin to None, use os.devnull instead

Also queued, thanks.

> -stdin = sys.stdin.buffer if sys.stdin else sys.stdin
> +if sys.stdin:
> +stdin = sys.stdin.buffer
> +else:
> +stdin = open(os.devnull, 'rb')
> +os.close(stdin.fileno())

Let's see if there is a better option applicable to all of stdin/out/err.
If not, we'll have to do the same for stdout/err.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 1 of 2 chg-tests-fix V3] dispatch: move IOError handling and flushing of streams to `dispatch()`

[Yuya Nishihara]

On Thu, 10 Dec 2020 14:41:44 +0530, Pulkit Goyal wrote:
> # HG changeset patch
> # User Pulkit Goyal <7895pul...@gmail.com>
> # Date 1607588516 -19800
> #  Thu Dec 10 13:51:56 2020 +0530
> # Node ID 96c66ca3f189d522dd53c736919e7343c29e6116
> # Parent  6de68a9dd7302f077edd73ea6d639d725ecf89b9
> # EXP-Topic chg-test
> dispatch: move IOError handling and flushing of streams to `dispatch()`

Queued, thanks.

> -def closestdio(ui, err):
> +def flushstdio(ui, err):

Renamed to _flushstdio as the function interface is very specific to
dispatch().
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 1 of 5 chg-tests-fix V2] procutils: don't try to get `.buffer` if sys.stdin is None

[Yuya Nishihara]

On Tue, 08 Dec 2020 19:12:20 +0530, Pulkit Goyal wrote:
> # HG changeset patch
> # User Pulkit Goyal <7895pul...@gmail.com>
> # Date 1606897517 -19800
> #  Wed Dec 02 13:55:17 2020 +0530
> # Node ID 13496bf926221a034c890135954fdc72c895f6d7
> # Parent  8b20d469a0c819c2ade8635c25e9fcf0af553796
> # EXP-Topic chg-test
> procutils: don't try to get `.buffer` if sys.stdin is None

> diff -r 8b20d469a0c8 -r 13496bf92622 mercurial/utils/procutil.py
> --- a/mercurial/utils/procutil.py Fri Apr 03 20:30:36 2020 +0530
> +++ b/mercurial/utils/procutil.py Wed Dec 02 13:55:17 2020 +0530
> @@ -124,7 +124,9 @@
>  # Python 3 implements its own I/O streams.
>  # TODO: .buffer might not exist if std streams were replaced; we'll need
>  # a silly wrapper to make a bytes stream backed by a unicode one.
> -stdin = sys.stdin.buffer
> +
> +# sys.stdin can be None
> +stdin = sys.stdin.buffer if sys.stdin else os.fdopen(0, 'rb')

Oops, os.fdopen() fails if fd=0 isn't available, so we'll need other
workaround. Sorry.

  % python3 -c 'import os; os.fdopen(0, "rb")' 0<&-
  Traceback (most recent call last):
File "", line 1, in 
File "/usr/lib/python3.9/os.py", line 1023, in fdopen
  return io.open(fd, *args, **kwargs)
  OSError: [Errno 9] Bad file descriptor

We can simulate Py2 behavior by opening /dev/null and closing the underlying
fd, but that might be a bit hacky.

  f = open(os.devnull, 'rb')
  os.close(f.fileno())

Any thoughts?

For stdout/stderr, we do need something. Adding null check to procutil
and dispatch isn't enough.

  % python3 hg version  1<&-
  ...
  Traceback (most recent call last):
File "mercurial/dispatch.py", line 434, in _runcatchfunc
  return _dispatch(req)
File "mercurial/dispatch.py", line 1039, in _dispatch
  extensions.loadall(lui)
File "mercurial/extensions.py", line 310, in loadall
  ui.warn(
File "mercurial/ui.py", line 1766, in warn
  self._writemsg(self._fmsgerr, type=b'warning', *msg, **opts)
File "mercurial/ui.py", line 1241, in _writemsg
  _writemsgwith(self._write, dest, *args, **opts)
File "mercurial/ui.py", line 2381, in _writemsgwith
  write(dest, *args, **opts)
File "mercurial/ui.py", line 1184, in _write
  self._writenobuf(dest, *args, **opts)
File "mercurial/ui.py", line 1196, in _writenobuf
  self._fout.flush()
  AttributeError: 'NoneType' object has no attribute 'flush'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
File "mercurial/dispatch.py", line 453, in _callcatch
  return scmutil.callcatch(ui, func)
File "mercurial/scmutil.py", line 155, in callcatch
  return func()
File "mercurial/dispatch.py", line 436, in _runcatchfunc
  ui.flush()
File "mercurial/ui.py", line 1253, in flush
  self._fout.flush()
  AttributeError: 'NoneType' object has no attribute 'flush'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
File "mercurial/dispatch.py", line 268, in dispatch
  ret = _runcatch(req) or 0
File "mercurial/dispatch.py", line 444, in _runcatch
  return _callcatch(ui, _runcatchfunc)
File "mercurial/dispatch.py", line 494, in _callcatch
  if not handlecommandexception(ui):
File "mercurial/dispatch.py", line 1346, in handlecommandexception
  ui.warn(warning)
File "mercurial/ui.py", line 1766, in warn
  self._writemsg(self._fmsgerr, type=b'warning', *msg, **opts)
File "mercurial/ui.py", line 1241, in _writemsg
  _writemsgwith(self._write, dest, *args, **opts)
File "mercurial/ui.py", line 2381, in _writemsgwith
  write(dest, *args, **opts)
File "mercurial/ui.py", line 1184, in _write
  self._writenobuf(dest, *args, **opts)
File "mercurial/ui.py", line 1196, in _writenobuf
  self._fout.flush()
  AttributeError: 'NoneType' object has no attribute 'flush'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
File "hg", line 59, in 
  dispatch.run()
File "mercurial/dispatch.py", line 115, in run
  status = dispatch(req)
File "mercurial/dispatch.py", line 291, in dispatch
  req.ui.flush()  # record blocked times
File "mercurial/ui.py", line 1253, in flush
  self._fout.flush()
  AttributeError: 'NoneType' object has no attribute 'flush'
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 1 of 4 chg-tests-fix] procutils: don't try to get `.buffer` if sys.stdin is None

[Yuya Nishihara]

On Tue, 8 Dec 2020 19:57:13 +0900, Yuya Nishihara wrote:
> On Tue, 08 Dec 2020 12:58:21 +0530, Pulkit Goyal wrote:
> > # HG changeset patch
> > # User Pulkit Goyal <7895pul...@gmail.com>
> > # Date 1606897517 -19800
> > #  Wed Dec 02 13:55:17 2020 +0530
> > # Node ID 6bab1270ddc9b7097a58de39afacb305f3364ea7
> > # Parent  8b20d469a0c819c2ade8635c25e9fcf0af553796
> > # EXP-Topic chg-test
> > procutils: don't try to get `.buffer` if sys.stdin is None
> 
> > diff -r 8b20d469a0c8 -r 6bab1270ddc9 mercurial/utils/procutil.py
> > --- a/mercurial/utils/procutil.py   Fri Apr 03 20:30:36 2020 +0530
> > +++ b/mercurial/utils/procutil.py   Wed Dec 02 13:55:17 2020 +0530
> > @@ -124,7 +124,9 @@
> >  # Python 3 implements its own I/O streams.
> >  # TODO: .buffer might not exist if std streams were replaced; we'll 
> > need
> >  # a silly wrapper to make a bytes stream backed by a unicode one.
> > -stdin = sys.stdin.buffer
> > +
> > +# sys.stdin can be None
> > +stdin = sys.stdin.buffer if sys.stdin else sys.stdin
> 
> It's probably better to make procutil.stdin/out/err always available.
> 
>   stdin = open(os.devnull, 'rb')

or maybe fdopen(0, 'rb'), which should be close to Python 2 behavior.

  % python2 -c 'import sys; print(sys.stdin.fileno()); sys.stdin.read()' 0<&-
  0
  Traceback (most recent call last):
File "", line 1, in 
  IOError: [Errno 9] Bad file descriptor
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 1 of 4 chg-tests-fix] procutils: don't try to get `.buffer` if sys.stdin is None

[Yuya Nishihara]

On Tue, 08 Dec 2020 12:58:21 +0530, Pulkit Goyal wrote:
> # HG changeset patch
> # User Pulkit Goyal <7895pul...@gmail.com>
> # Date 1606897517 -19800
> #  Wed Dec 02 13:55:17 2020 +0530
> # Node ID 6bab1270ddc9b7097a58de39afacb305f3364ea7
> # Parent  8b20d469a0c819c2ade8635c25e9fcf0af553796
> # EXP-Topic chg-test
> procutils: don't try to get `.buffer` if sys.stdin is None

> diff -r 8b20d469a0c8 -r 6bab1270ddc9 mercurial/utils/procutil.py
> --- a/mercurial/utils/procutil.py Fri Apr 03 20:30:36 2020 +0530
> +++ b/mercurial/utils/procutil.py Wed Dec 02 13:55:17 2020 +0530
> @@ -124,7 +124,9 @@
>  # Python 3 implements its own I/O streams.
>  # TODO: .buffer might not exist if std streams were replaced; we'll need
>  # a silly wrapper to make a bytes stream backed by a unicode one.
> -stdin = sys.stdin.buffer
> +
> +# sys.stdin can be None
> +stdin = sys.stdin.buffer if sys.stdin else sys.stdin

It's probably better to make procutil.stdin/out/err always available.

  stdin = open(os.devnull, 'rb')

I don't think we'll want to deal with None stdin/out/err everywhere stdio
is involved. That's insane.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 4 of 4 chg-tests-fix] commandserver: handle IOError related to flushing of streams

[Yuya Nishihara]

On Tue, 08 Dec 2020 12:58:24 +0530, Pulkit Goyal wrote:
> # HG changeset patch
> # User Pulkit Goyal <7895pul...@gmail.com>
> # Date 1606996129 -19800
> #  Thu Dec 03 17:18:49 2020 +0530
> # Node ID acdb053888ab7564b6b7fc702f94334ef854861c
> # Parent  263a5b17b4cad2e793817bd30944dcfcebc88a69
> # EXP-Topic chg-test
> commandserver: handle IOError related to flushing of streams
> 
> After dispatch, without chg we have handling of flushing of streams and
> exception handling related to it. The exception handling part is important
> because there can be exceptions when flushing fout or ferr.
> 
> One such case is in `test-basic.t` which was failing on python3+chg without 
> this
> patch as this handling was missing from chg.
> 
> Failure can be seen at
> https://foss.heptapod.net/octobus/mercurial-devel/-/jobs/128399
> 
> Honestly I am not sure which one of `chgserver.py` or `commandserver.py` the
> change should go in.
> 
> Differential Revision: https://phab.mercurial-scm.org/D9517
> 
> diff -r 263a5b17b4ca -r acdb053888ab mercurial/commandserver.py
> --- a/mercurial/commandserver.py  Wed Dec 02 14:27:45 2020 +0530
> +++ b/mercurial/commandserver.py  Thu Dec 03 17:18:49 2020 +0530
> @@ -355,7 +355,18 @@
>  )
>  
>  try:
> -ret = self._dispatchcommand(req) & 255
> +err = None
> +try:
> +status = self._dispatchcommand(req)
> +except error.StdioError as e:
> +status = -1
> +err = e
> +
> +retval = dispatch.closestdio(req.ui, err)

Catching StdioError here seems fine, but the function name is misleading.
stdio should never be closed here.

> --- a/mercurial/dispatch.py   Wed Dec 02 14:27:45 2020 +0530
> +++ b/mercurial/dispatch.py   Thu Dec 03 17:18:49 2020 +0530
> @@ -104,6 +104,35 @@
>  raise exc
>  
>  
> +def closestdio(ui, err):
> +status = None
> +# In all cases we try to flush stdio streams.
> +if util.safehasattr(ui, b'fout'):
> +assert ui is not None  # help pytype
> +assert ui.fout is not None  # help pytype
> +try:
> +ui.fout.flush()
> +except IOError as e:
> +err = e
> +status = -1
> +
> +if util.safehasattr(ui, b'ferr'):
> +assert ui is not None  # help pytype
> +assert ui.ferr is not None  # help pytype
> +try:
> +if err is not None and err.errno != errno.EPIPE:
> +ui.ferr.write(
> +b'abort: %s\n' % encoding.strtolocal(err.strerror)
> +)
> +ui.ferr.flush()
> +# There's not much we can do about an I/O error here. So (possibly)
> +# change the status code and move on.
> +except IOError:
> +status = -1
> +
> +return status

Maybe we can instead move this to dispatch.dispatch() since both hg cli and
commandserver need to translate StdioError to status -1.

def dispatch(...):
try:
... original dispatch()
except StdioError ...:

.. do flushing stuff ...
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 1 of 2] bugzilla: fix reporting of exceptions with py3

[Yuya Nishihara]

On Mon, 07 Dec 2020 22:30:04 +0100, Mads Kiilerich wrote:
> # HG changeset patch
> # User Mads Kiilerich 
> # Date 1607032141 -3600
> #  Thu Dec 03 22:49:01 2020 +0100
> # Branch stable
> # Node ID a141d1da33433998590be226b6788fb53e8c90e0
> # Parent  dadbd01f939379a2eea76eab313868fcde48138a
> bugzilla: fix reporting of exceptions with py3

Queued for stable, thanks.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH] test-extension: flush diagnostic message to stabilize chg output

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1607339556 -32400
#  Mon Dec 07 20:12:36 2020 +0900
# Node ID e19bab808d0b3491121113509601f990fe5988f1
# Parent  1bf2b44c40078c4e17479a1625c57370dc7750ec
test-extension: flush diagnostic message to stabilize chg output

Since chg server may create new file object for the attached stdout,
procutil.stdout is not ui.fout and the buffered procutil.stdout data wouldn't
be flushed at all. That's why test-extension.t passes without modification
on Python 2.

diff --git a/tests/test-extension.t b/tests/test-extension.t
--- a/tests/test-extension.t
+++ b/tests/test-extension.t
@@ -154,7 +154,10 @@ Check that extensions are loaded in phas
   > from mercurial import exthelper
   > from mercurial.utils import procutil
   > 
-  > write = procutil.stdout.write
+  > def write(msg):
+  > procutil.stdout.write(msg)
+  > procutil.stdout.flush()
+  > 
   > name = os.path.basename(__file__).rsplit('.', 1)[0]
   > bytesname = name.encode('utf-8')
   > write(b"1) %s imported\n" % bytesname)
@@ -194,6 +197,9 @@ Check that extensions are loaded in phas
 
 Check normal command's load order of extensions and registration of functions
 
+ On chg server, extension should be first set up by the server. Then
+ object-level setup should follow in the worker process.
+
   $ hg log -r "foo() and bar()" -q
   1) foo imported
   1) bar imported
@@ -209,6 +215,18 @@ Check normal command's load order of ext
   4) bar uipopulate
   5) foo reposetup
   5) bar reposetup
+  4) foo uipopulate (chg !)
+  4) bar uipopulate (chg !)
+  4) foo uipopulate (chg !)
+  4) bar uipopulate (chg !)
+  4) foo uipopulate (chg !)
+  4) bar uipopulate (chg !)
+  4) foo uipopulate (chg !)
+  4) bar uipopulate (chg !)
+  4) foo uipopulate (chg !)
+  4) bar uipopulate (chg !)
+  5) foo reposetup (chg !)
+  5) bar reposetup (chg !)
   0:c24b9ac61126
 
 Check hgweb's load order of extensions and registration of functions

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH STABLE] ui: remove excessive strtolocal() from debuguigetpass

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1606988543 -32400
#  Thu Dec 03 18:42:23 2020 +0900
# Branch stable
# Node ID 074a20823c4be85d4b98a76c2ba961ff640cc80e
# Parent  27c23c8f14da48a24a0896b588721c0139e92560
ui: remove excessive strtolocal() from debuguigetpass

ui.getpass() returns Optional[bytes], and strtolocal(bytes) would crash.
Follows up 07b0a687c01a "ui: ensure `getpass()` returns bytes."

diff --git a/mercurial/debugcommands.py b/mercurial/debugcommands.py
--- a/mercurial/debugcommands.py
+++ b/mercurial/debugcommands.py
@@ -3787,9 +3787,7 @@ def debugtemplate(ui, repo, tmpl, **opts
 def debuguigetpass(ui, prompt=b''):
 """show prompt to type password"""
 r = ui.getpass(prompt)
-if r is not None:
-r = encoding.strtolocal(r)
-else:
+if r is None:
 r = b""
 ui.writenoi18n(b'response: %s\n' % r)
 

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 4 of 4] log: do not accept string-matcher pattern as -u/-b/-B parameter

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1606821744 -32400
#  Tue Dec 01 20:22:24 2020 +0900
# Node ID 8e8feec5114933aaa5224ff33acfa1bc9a52625c
# Parent  773cf7f8899449979131792e82a49fc6bc0d25ec
log: do not accept string-matcher pattern as -u/-b/-B parameter

I'm pretty sure this is a bug introduced after we've switched the filtering
backend to revset matcher.

diff --git a/mercurial/logcmdutil.py b/mercurial/logcmdutil.py
--- a/mercurial/logcmdutil.py
+++ b/mercurial/logcmdutil.py
@@ -898,13 +898,13 @@ def _makenofollowfilematcher(repo, pats,
 def _makerevset(repo, wopts, slowpath):
 """Return a revset string built from log options and file patterns"""
 opts = {
-b'branch': [repo.lookupbranch(b) for b in wopts.branches],
+b'branch': [b'literal:' + repo.lookupbranch(b) for b in 
wopts.branches],
 b'date': wopts.date,
 b'keyword': wopts.keywords,
 b'no_merges': wopts.no_merges,
 b'only_merges': wopts.only_merges,
 b'prune': wopts.prune_ancestors,
-b'user': wopts.users,
+b'user': [b'literal:' + v for v in wopts.users],
 }
 
 if wopts.filter_revisions_by_pats and slowpath:
diff --git a/mercurial/scmutil.py b/mercurial/scmutil.py
--- a/mercurial/scmutil.py
+++ b/mercurial/scmutil.py
@@ -2310,6 +2310,7 @@ def bookmarkrevs(repo, mark):
 def format_bookmark_revspec(mark):
 """Build a revset expression to select revisions reachable by a given
 bookmark"""
+mark = b'literal:' + mark
 return revsetlang.formatspec(
 b"ancestors(bookmark(%s)) - "
 b"ancestors(head() and not bookmark(%s)) - "
diff --git a/tests/test-glog-beautifygraph.t b/tests/test-glog-beautifygraph.t
--- a/tests/test-glog-beautifygraph.t
+++ b/tests/test-glog-beautifygraph.t
@@ -1588,19 +1588,19 @@ glog always reorders nodes which explain
 (list
   (func
 (symbol 'user')
-(string 'test'))
+(string 'literal:test'))
   (func
 (symbol 'user')
-(string 'not-a-user'
+(string 'literal:not-a-user'
   ,
 ,
->,
+>,
   ,
->>>
+>>>
   $ testlog -b not-a-branch
   abort: unknown revision 'not-a-branch'
   abort: unknown revision 'not-a-branch'
@@ -1611,28 +1611,28 @@ glog always reorders nodes which explain
 (list
   (func
 (symbol 'branch')
-(string 'default'))
+(string 'literal:default'))
   (or
 (list
   (func
 (symbol 'branch')
-(string 'branch'))
+(string 'literal:branch'))
   (func
 (symbol 'branch')
-(string 'branch'))
+(string 'literal:branch'))
   ,
 ,
->,
+>,
   ,
-  >,
+  >,
 ,
-  >>>>
+  >>>>
   $ testlog -k expand -k merge
   []
   (or
diff --git a/tests/test-glog.t b/tests/test-glog.t
--- a/tests/test-glog.t
+++ b/tests/test-glog.t
@@ -1438,19 +1438,19 @@ glog always reorders nodes which explain
 (list
   (func
 (symbol 'user')
-(string 'test'))
+(string 'literal:test'))
   (func
 (symbol 'user')
-(string 'not-a-user'
+(string 'literal:not-a-user'
   ,
 ,
->,
+>,
   ,
->>>
+>>>
   $ testlog -b not-a-branch
   abort: unknown revision 'not-a-branch'
   abort: unknown revision 'not-a-branch'
@@ -1461,28 +1461,28 @@ glog always reorders nodes which explain
 (list
   (func
 (symbol 'branch')
-(string 'default'))
+(string 'literal:default'))
   (or
 (list
   (func
 (symbol 'branch')
-(string 'branch'))
+(string 'literal:branch'))
   (func
 (symbol 'branch')
-(string 'branch'))
+(string 'literal:branch'))
   ,
 ,
->,
+>,
   ,
-  >,
+  >,
 ,
-  >>>>
+  >>>>
   $ testlog -k expand -k merge
   []
   (or
diff --git a/tests/test-log-bookmark.t b/tests/test-log-bookmark.t
--- a/tests/test-log-bookmark.t
+++ b/tests/test-log-bookmark.t
@@ -190,3 +190,9 @@ Unknown bookmark:
   $ hg log -B unknown
   abort: bookmark 'unknown' does not exist
   [255]
+
+Shouldn't accept string-matcher syntax:
+
+  $ hg log -B 're:.*'
+  abort: bookmark 're:.*' does not exist
+  [255]
diff --git a/tests/test-log.t b/tests/test-log.t
--- a/tests/test-log.t
+++ b/tests/test-log.t
@@ -1378,6 +1378,14 @@ are specified (issue5100):
   1 k1
   0 k0
 
+ log -b/-u/-k shouldn't accept string-matcher syntax:
+
+  $ hg log -b 're:.*'
+  abort: unknown revision 're:.*'
+  [255]
+  $ hg log -k 're:.*'
+  $ hg log -u 're:.*'
+
  log FILE in

[PATCH 2 of 4] scmutil: extract function that builds revset expr to select bookmark branch

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1606818203 -32400
#  Tue Dec 01 19:23:23 2020 +0900
# Node ID 3b3f53b5b57f32b10c6e5c8aacffab002fd793ea
# Parent  4439db4d98bc5d5d05b709d0886cb81b1347516e
scmutil: extract function that builds revset expr to select bookmark branch

This is needed to process "log -B" option properly. "log" options have to
be translated to a revset expression, not to an evaluated set.

diff --git a/mercurial/scmutil.py b/mercurial/scmutil.py
--- a/mercurial/scmutil.py
+++ b/mercurial/scmutil.py
@@ -2304,7 +2304,13 @@ def bookmarkrevs(repo, mark):
 
 If the bookmarked revision isn't a head, an empty set will be returned.
 """
-return repo.revs(
+return repo.revs(format_bookmark_revspec(mark))
+
+
+def format_bookmark_revspec(mark):
+"""Build a revset expression to select revisions reachable by a given
+bookmark"""
+return revsetlang.formatspec(
 b"ancestors(bookmark(%s)) - "
 b"ancestors(head() and not bookmark(%s)) - "
 b"ancestors(bookmark() and not bookmark(%s))",

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 3 of 4] log: do not override other filtering and sorting options by --bookmark

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1606818756 -32400
#  Tue Dec 01 19:32:36 2020 +0900
# Node ID 773cf7f8899449979131792e82a49fc6bc0d25ec
# Parent  3b3f53b5b57f32b10c6e5c8aacffab002fd793ea
log: do not override other filtering and sorting options by --bookmark

This basically reimplements 0aa118f18d4b 'log: add bookmark option to
"hg log"'. Before, any other filtering options but --rev were ignored.
-G didn't work either since the ordering constraint wasn't enforced.

diff --git a/mercurial/commands.py b/mercurial/commands.py
--- a/mercurial/commands.py
+++ b/mercurial/commands.py
@@ -4636,12 +4636,6 @@ def log(ui, repo, *pats, **opts):
 # then filter the result by logcmdutil._makerevset() and --limit
 revs, differ = logcmdutil.getlinerangerevs(repo, revs, opts)
 
-if opts.get(b'bookmark'):
-cmdutil.check_at_most_one_arg(opts, b'rev', b'bookmark')
-bookmarks = opts.get(b'bookmark')
-bookmark = bookmarks[0]
-revs, differ = logcmdutil.get_bookmark_revs(repo, bookmark, walk_opts)
-
 getcopies = None
 if opts.get(b'copies'):
 endrev = None
diff --git a/mercurial/logcmdutil.py b/mercurial/logcmdutil.py
--- a/mercurial/logcmdutil.py
+++ b/mercurial/logcmdutil.py
@@ -691,6 +691,7 @@ class walkopts(object):
 revspec = attr.ib()  # type: List[bytes]
 
 # miscellaneous queries to filter revisions (see "hg help log" for details)
+bookmarks = attr.ib(default=attr.Factory(list))  # type: List[bytes]
 branches = attr.ib(default=attr.Factory(list))  # type: List[bytes]
 date = attr.ib(default=None)  # type: Optional[bytes]
 keywords = attr.ib(default=attr.Factory(list))  # type: List[bytes]
@@ -746,6 +747,7 @@ def parseopts(ui, pats, opts):
 pats=pats,
 opts=opts,
 revspec=opts.get(b'rev', []),
+bookmarks=opts.get(b'bookmark', []),
 # branch and only_branch are really aliases and must be handled at
 # the same time
 branches=opts.get(b'branch', []) + opts.get(b'only_branch', []),
@@ -937,6 +939,14 @@ def _makerevset(repo, wopts, slowpath):
 val = [revsetlang.formatspec(revop, v) for v in val]
 expr.append(revsetlang.formatspec(listop, val))
 
+if wopts.bookmarks:
+expr.append(
+revsetlang.formatspec(
+b'%lr',
+[scmutil.format_bookmark_revspec(v) for v in wopts.bookmarks],
+)
+)
+
 if expr:
 expr = b'(' + b' and '.join(expr) + b')'
 else:
@@ -1023,26 +1033,6 @@ def getrevs(repo, wopts):
 return revs, differ
 
 
-def get_bookmark_revs(repo, bookmark, walk_opts):
-# type: (Any, bookmark, walk_opts) -> Tuple[smartset.abstractsmartset, 
Optional[changesetdiffer]]
-"""Return (revs, differ) where revs is a smartset
-
-differ is a changesetdiffer with pre-configured file matcher.
-"""
-revs, filematcher = makewalker(repo, walk_opts)
-if not revs:
-return revs, None
-differ = changesetdiffer()
-differ._makefilematcher = filematcher
-
-if bookmark:
-if bookmark not in repo._bookmarks:
-raise error.Abort(_(b"bookmark '%s' not found") % bookmark)
-revs = scmutil.bookmarkrevs(repo, bookmark)
-
-return revs, differ
-
-
 def _parselinerangeopt(repo, opts):
 """Parse --line-range log option and return a list of tuples (filename,
 (fromline, toline)).
diff --git a/tests/test-log-bookmark.t b/tests/test-log-bookmark.t
--- a/tests/test-log-bookmark.t
+++ b/tests/test-log-bookmark.t
@@ -125,3 +125,68 @@ Check the log of topic X, topic Y, and d
   date:Thu Jan 01 00:00:00 1970 +
   summary: Add foo in 'default'
   
+
+Set up multiple bookmarked heads:
+
+  $ hg bookmark merged-head
+  $ hg up 1
+  0 files updated, 0 files merged, 1 files removed, 0 files unresolved
+  (leaving bookmark merged-head)
+  $ echo "Z" > z.txt
+  $ hg ci -Am 'Add Z'
+  adding z.txt
+  $ hg bookmark topic-Z
+
+  $ hg log -GT '{rev}: {branch}, {bookmarks}\n'
+  @  5: sebhtml, topic-Z
+  |
+  | o  4: default, merged-head
+  |/|
+  | o3: default,
+  | |\
+  | | o  2: sebhtml, sebhtml/2-topic-Y
+  | |/
+  o |  1: sebhtml, sebhtml/1-topic-X
+  |/
+  o  0: default,
+  
+
+Multiple revisions under bookmarked head:
+
+  $ hg log -GT '{rev}: {branch}, {bookmarks}\n' -B merged-head
+  o4: default, merged-head
+  |\
+  | ~
+  o3: default,
+  |\
+  ~ ~
+
+Follows multiple bookmarks:
+
+  $ hg log -GT '{rev}: {branch}, {bookmarks}\n' -B merged-head -B topic-Z
+  @  5: sebhtml, topic-Z
+  |
+  ~
+  o4: default, merged-head
+  |\
+  | ~
+  o3: default,
+  |\
+  ~ ~
+
+Filter by bookmark and branch:
+
+  $ hg log -GT '{rev}: {branch}, {bookmarks}\n' -B merged-head -B topic-Z -b 
default
+  o4: default, merged-head
+  |\
+  | ~
+  o3: default,
+  |\
+  ~ ~
+
+
+Unk

[PATCH 1 of 4] scmutil: document that bookmarkrevs() ignores non-head bookmark branch

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1606819561 -32400
#  Tue Dec 01 19:46:01 2020 +0900
# Node ID 4439db4d98bc5d5d05b709d0886cb81b1347516e
# Parent  d42809b6b10ff91cf6fcad5914e77c6974195e78
scmutil: document that bookmarkrevs() ignores non-head bookmark branch

"- ancestors(head() and not bookmark(%s))" excludes the bookmarked branch
itself if bookmark(%s) is not a head. I'm a bit surprised by this behavior
while writing "log -B" tests, so let's document it.

diff --git a/mercurial/scmutil.py b/mercurial/scmutil.py
--- a/mercurial/scmutil.py
+++ b/mercurial/scmutil.py
@@ -2300,8 +2300,9 @@ def _getrevsfromsymbols(repo, symbols):
 
 
 def bookmarkrevs(repo, mark):
-"""
-Select revisions reachable by a given bookmark
+"""Select revisions reachable by a given bookmark
+
+If the bookmarked revision isn't a head, an empty set will be returned.
 """
 return repo.revs(
 b"ancestors(bookmark(%s)) - "

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH STABLE] diff: do not concatenate immutable bytes while building a/b bodies (issue6445)

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1606529754 -32400
#  Sat Nov 28 11:15:54 2020 +0900
# Branch stable
# Node ID 1e93d856abc0ecb7f537af5c77e0d75ffe93db08
# Parent  81da6feb5000140adb8972607415855953415233
diff: do not concatenate immutable bytes while building a/b bodies (issue6445)

Use bytearray instead. I don't know what's changed since Python 2, but bytes
concatenation is 100x slow on Python 3.

  % python2.7 -m timeit -s "s = b''" "for i in range(1): s += b'line'"
  1000 loops, best of 3: 321 usec per loop

  % python3.9 -m timeit -s "s = b''" "for i in range(1): s += b'line'"
  5 loops, best of 5: 39.2 msec per loop

Benchmark using tailwind.css (measuring the fast path, a is empty):

  % HGRCPATH=/dev/null python2.7 ./hg log -R /tmp/issue6445 -p --time \
  --color=always --config diff.word-diff=true >/dev/null
  (prev) time: real 1.580 secs (user 1.560+0.000 sys 0.020+0.000)
  (this) time: real 1.610 secs (user 1.570+0.000 sys 0.030+0.000)

  % HGRCPATH=/dev/null python3.9 ./hg log -R /tmp/issue6445 -p --time \
  --color=always --config diff.word-diff=true >/dev/null
  (prev) time: real 114.500 secs (user 114.460+0.000 sys 0.030+0.000)
  (this) time: real 2.180 secs (user 2.140+0.000 sys 0.040+0.000)

Benchmark using random tabular text data (not the fast path):

  % dd if=/dev/urandom bs=1k count=1000 | hexdump -v -e '16/1 "%3u," "\n"' > ttf
  % hg ci -ma
  % dd if=/dev/urandom bs=1k count=1000 | hexdump -v -e '16/1 "%3u," "\n"' > ttf
  % hg ci -mb

  % HGRCPATH=/dev/null python2.7 ./hg log -R /tmp/issue6445 -p --time \
  --color=always --config diff.word-diff=true >/dev/null
  (prev) time: real 3.240 secs (user 3.040+0.000 sys 0.200+0.000
  (this) time: real 3.230 secs (user 3.070+0.000 sys 0.160+0.000)

  % HGRCPATH=/dev/null python3.9 ./hg log -R /tmp/issue6445 -p --time \
  --color=always --config diff.word-diff=true >/dev/null
  (prev) time: real 44.130 secs (user 43.850+0.000 sys 0.270+0.000)
  (this) time: real 4.170 secs (user 3.850+0.000 sys 0.310+0.000)

diff --git a/mercurial/patch.py b/mercurial/patch.py
--- a/mercurial/patch.py
+++ b/mercurial/patch.py
@@ -2731,8 +2731,8 @@ def diffsinglehunk(hunklines):
 def diffsinglehunkinline(hunklines):
 """yield tokens for a list of lines in a single hunk, with inline colors"""
 # prepare deleted, and inserted content
-a = b''
-b = b''
+a = bytearray()
+b = bytearray()
 for line in hunklines:
 if line[0:1] == b'-':
 a += line[1:]
@@ -2746,8 +2746,8 @@ def diffsinglehunkinline(hunklines):
 yield t
 return
 # re-split the content into words
-al = wordsplitter.findall(a)
-bl = wordsplitter.findall(b)
+al = wordsplitter.findall(bytes(a))
+bl = wordsplitter.findall(bytes(b))
 # re-arrange the words to lines since the diff algorithm is line-based
 aln = [s if s == b'\n' else s + b'\n' for s in al]
 bln = [s if s == b'\n' else s + b'\n' for s in bl]

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] heptapod-ci: hosting base image on registry.heptapod.net

[Yuya Nishihara]

On Sat, 21 Nov 2020 19:17:35 +0100, Pierre-Yves David wrote:
> # HG changeset patch
> # User Georges Racinet 
> # Date 1605902798 -3600
> #  Fri Nov 20 21:06:38 2020 +0100
> # Node ID bc8e611b22348e8f800c7840ba99e448f87ae9b4
> # Parent  f4a218331ff4e871248625d0116d9985dc8566ca
> # EXP-Topic heptapod-ci-registry
> # Available At https://foss.heptapod.net/octobus/mercurial-devel/
> #  hg pull https://foss.heptapod.net/octobus/mercurial-devel/ -r 
> bc8e611b2234
> heptapod-ci: hosting base image on registry.heptapod.net

Queued, thanks.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] context: small update to ctx.status doc

[Yuya Nishihara]

On Fri, 20 Nov 2020 11:12:12 +0100, Pierre-Yves David wrote:
> # HG changeset patch
> # User Pierre-Yves David 
> # Date 1605854229 -3600
> #  Fri Nov 20 07:37:09 2020 +0100
> # Node ID 54286f31bbf24cab63eb02e6ab1e4d95b413ed29
> # Parent  d68618954adef9c2cbec868c1af0e01f67288cb8
> # EXP-Topic status-doc
> # Available At https://foss.heptapod.net/octobus/mercurial-devel/
> #  hg pull https://foss.heptapod.net/octobus/mercurial-devel/ -r 
> 54286f31bbf2
> context: small update to ctx.status doc

Queued, thanks.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH] chgserver: backport py3 buffered I/O workarounds from procutil

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1605608948 -32400
#  Tue Nov 17 19:29:08 2020 +0900
# Node ID 0748354881c942727aef46945d721bc51d213e9d
# Parent  d68618954adef9c2cbec868c1af0e01f67288cb8
chgserver: backport py3 buffered I/O workarounds from procutil

I've recently switched to new machine and I found chg's stdout is fully
buffered.

Even though chg server is a daemon process, it inherits the environment
where the chg client originally forked the server. This means the server's
stdout might have been wrapped by LineBufferedWrapper. That's why we need
to do wrap/unwrap in both ways.

The "if" condition in _restoreio() looks weird, but I'm not willing to
clean things up because stdio behavior is fundamentally different between
py2 and py3, and py2 support will be dropped anyway.

diff --git a/mercurial/chgserver.py b/mercurial/chgserver.py
--- a/mercurial/chgserver.py
+++ b/mercurial/chgserver.py
@@ -409,14 +409,23 @@ class chgcmdserver(commandserver.server)
 # be unbuffered no matter if it is a tty or not.
 if fn == b'ferr':
 newfp = fp
+elif pycompat.ispy3:
+# On Python 3, the standard library doesn't offer line-buffered
+# binary streams, so wrap/unwrap it.
+if fp.isatty():
+newfp = procutil.make_line_buffered(fp)
+else:
+newfp = procutil.unwrap_line_buffered(fp)
 else:
-# make it line buffered explicitly because the default is
-# decided on first write(), where fout could be a pager.
+# Python 2 uses the I/O streams provided by the C library, so
+# make it line-buffered explicitly. Otherwise the default would
+# be decided on first write(), where fout could be a pager.
 if fp.isatty():
 bufsize = 1  # line buffered
 else:
 bufsize = -1  # system default
 newfp = os.fdopen(fp.fileno(), mode, bufsize)
+if newfp is not fp:
 setattr(ui, fn, newfp)
 setattr(self, cn, newfp)
 
@@ -440,13 +449,16 @@ class chgcmdserver(commandserver.server)
 ui = self.ui
 for (ch, fp, fd), (cn, fn, mode) in zip(self._oldios, _iochannels):
 newfp = getattr(ui, fn)
-# close newfp while it's associated with client; otherwise it
-# would be closed when newfp is deleted
-if newfp is not fp:
+# On Python 2, newfp and fp may be separate file objects associated
+# with the same fd, so we must close newfp while it's associated
+# with the client. Otherwise the new associated fd would be closed
+# when newfp gets deleted. On Python 3, newfp is just a wrapper
+# around fp even if newfp is not fp, so deleting newfp is safe.
+if not (pycompat.ispy3 or newfp is fp):
 newfp.close()
 # restore original fd: fp is open again
 try:
-if newfp is fp and 'w' in mode:
+if (pycompat.ispy3 or newfp is fp) and 'w' in mode:
 # Discard buffered data which couldn't be flushed because
 # of EPIPE. The data should belong to the current session
 # and should never persist.
diff --git a/mercurial/utils/procutil.py b/mercurial/utils/procutil.py
--- a/mercurial/utils/procutil.py
+++ b/mercurial/utils/procutil.py
@@ -80,6 +80,13 @@ def make_line_buffered(stream):
 return LineBufferedWrapper(stream)
 
 
+def unwrap_line_buffered(stream):
+if isinstance(stream, LineBufferedWrapper):
+assert not isinstance(stream.orig, LineBufferedWrapper)
+return stream.orig
+return stream
+
+
 class WriteAllWrapper(object):
 def __init__(self, orig):
 self.orig = orig

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: How to use the templater without color (even when ui has color)

[Yuya Nishihara]

On Fri, 13 Nov 2020 09:20:24 -0800, Martin von Zweigbergk wrote:
> I recently added support for some templated output in `hg split` (
> https://phab.mercurial-scm.org/D9255). I didn't realize when I wrote the
> patch that it was producing the templated output for use in a commit
> message template. The effect was that color codes ended up in the commit
> message template. How should I fix it? The following seems to fix it, but
> maybe there's a cleaner way?
> 
> ```
> diff --git a/hgext/split.py b/hgext/split.py
> --- a/hgext/split.py
> +++ b/hgext/split.py
> @@ -142,9 +142,13 @@ def dosplit(ui, repo, tr, ctx, opts):
>  header = _(
>  b'HG: Splitting %s. So far it has been split into:\n'
>  ) % short(ctx.node())
> -for c in committed:
> -summary = cmdutil.format_changeset_summary(ui, c, b'split')
> -header += _(b'HG: - %s\n') % summary
> +from mercurial import color
> +with ui.configoverride({(b'ui', b'color'): b"no"}, b'split'):
> +color.setup(ui)
> +for c in committed:
> +summary = cmdutil.format_changeset_summary(ui, c,
> b'split')
> +header += _(b'HG: - %s\n') % summary
> +color.setup(ui)

If we don't want to use separate templates for colored/plain outputs, we'll
need to disable the label() function. I think the easiest option is to
remove label() calls by alias:

  --config 'templatealias.label(l, x)=x'

Maybe local aliases can be inserted at formatter.maketemplater().
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 3 of 4 STABLE] chg: do not close dir fd while iterating

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1604369750 -32400
#  Tue Nov 03 11:15:50 2020 +0900
# Branch stable
# Node ID 9534de20358f656be05e0633a3a2ba59932326f3
# Parent  731ea8fa1f110b2856826db5ed78fc1178dc1306
chg: do not close dir fd while iterating

It works so long as the dp is the last entry, but readdir(dp) would fail
with EBADF. Let's not do that and close the dir fd explicitly.

diff --git a/contrib/chg/chg.c b/contrib/chg/chg.c
--- a/contrib/chg/chg.c
+++ b/contrib/chg/chg.c
@@ -288,7 +288,7 @@ static void execcmdserver(const struct c
 de->d_name);
continue;
}
-   if (fd_value > STDERR_FILENO) {
+   if (fd_value > STDERR_FILENO && fd_value != dirfd(dp)) {
debugmsg("closing fd %ld", fd_value);
int res = close(fd_value);
if (res) {
@@ -298,6 +298,7 @@ static void execcmdserver(const struct c
}
}
}
+   closedir(dp);
}
 
if (putenv("CHGINTERNALMARK=") != 0)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 4 STABLE] chg: show debug message for each fd to be closed

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1604369545 -32400
#  Tue Nov 03 11:12:25 2020 +0900
# Branch stable
# Node ID 731ea8fa1f110b2856826db5ed78fc1178dc1306
# Parent  41aaf960dc00d85712d412a4aba81e742aa7b69e
chg: show debug message for each fd to be closed

It helps debugging. The number of file descriptors should be small in most
cases, so the console output wouldn't get bloated even with CHG_DEBUG=1.

diff --git a/contrib/chg/chg.c b/contrib/chg/chg.c
--- a/contrib/chg/chg.c
+++ b/contrib/chg/chg.c
@@ -289,6 +289,7 @@ static void execcmdserver(const struct c
continue;
}
if (fd_value > STDERR_FILENO) {
+   debugmsg("closing fd %ld", fd_value);
int res = close(fd_value);
if (res) {
debugmsg("tried to close fd %ld: %d "
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 1 of 4 STABLE] chg: apply clang-format

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1604369175 -32400
#  Tue Nov 03 11:06:15 2020 +0900
# Branch stable
# Node ID 41aaf960dc00d85712d412a4aba81e742aa7b69e
# Parent  8711dc13474ce3d6db8c21389a994867331e348d
chg: apply clang-format

diff --git a/contrib/chg/chg.c b/contrib/chg/chg.c
--- a/contrib/chg/chg.c
+++ b/contrib/chg/chg.c
@@ -283,13 +283,17 @@ static void execcmdserver(const struct c
continue;
}
if (errno == ERANGE) {
-   debugmsg("tried to parse %s, but range error 
occurred", de->d_name);
+   debugmsg("tried to parse %s, but range error "
+"occurred",
+de->d_name);
continue;
}
if (fd_value > STDERR_FILENO) {
int res = close(fd_value);
if (res) {
-   debugmsg("tried to close fd %ld: %d 
(errno: %d)", fd_value, res, errno);
+   debugmsg("tried to close fd %ld: %d "
+"(errno: %d)",
+fd_value, res, errno);
}
}
}
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 4 of 4 STABLE] chg: reset errno prior to calling strtol()

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1604370261 -32400
#  Tue Nov 03 11:24:21 2020 +0900
# Branch stable
# Node ID 81da6feb5000140adb8972607415855953415233
# Parent  9534de20358f656be05e0633a3a2ba59932326f3
chg: reset errno prior to calling strtol()

Otherwise we can't figure out if the last strtol() invocation failed or
not.

diff --git a/contrib/chg/chg.c b/contrib/chg/chg.c
--- a/contrib/chg/chg.c
+++ b/contrib/chg/chg.c
@@ -276,6 +276,7 @@ static void execcmdserver(const struct c
debugmsg("closing files based on /proc contents");
struct dirent *de;
while ((de = readdir(dp))) {
+   errno = 0;
char *end;
long fd_value = strtol(de->d_name, , 10);
if (end == de->d_name) {
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 2 STABLE] url: do not continue HTTP authentication with user=None (issue6425)

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1603452816 -32400
#  Fri Oct 23 20:33:36 2020 +0900
# Branch stable
# Node ID be11c752c0fe3c4755f2567a302123bda3e236af
# Parent  15e5d72561363a7e83ea9a7c70d85db760bc57e8
url: do not continue HTTP authentication with user=None (issue6425)

I initially thought this is a py3-compat bug of passwordmgr._writedebug(),
but actually returning (None, str) pair is wrong at all. HTTP authentication
would continue with user="None" in that case.

Since registering a password of user=None should also be wrong, this patch
simply adds early return.

diff --git a/mercurial/url.py b/mercurial/url.py
--- a/mercurial/url.py
+++ b/mercurial/url.py
@@ -96,6 +96,13 @@ class passwordmgr(object):
 if not passwd:
 passwd = self.ui.getpass()
 
+# As of Python 3.8, the default implementation of
+# AbstractBasicAuthHandler.retry_http_basic_auth() assumes the user
+# is set if pw is not None. This means (None, str) is not a valid
+# return type of find_user_password().
+if user is None:
+return None, None
+
 self.passwddb.add_password(realm, authuri, user, passwd)
 self._writedebug(user, passwd)
 return (pycompat.strurl(user), pycompat.strurl(passwd))
diff --git a/tests/test-http.t b/tests/test-http.t
--- a/tests/test-http.t
+++ b/tests/test-http.t
@@ -192,6 +192,34 @@ test http authentication
   $ hg id http://localhost:$HGPORT2/
   abort: http authorization required for http://localhost:$HGPORT2/
   [255]
+  $ hg id --config ui.interactive=true --debug http://localhost:$HGPORT2/
+  using http://localhost:$HGPORT2/
+  sending capabilities command
+  http authorization required for http://localhost:$HGPORT2/
+  realm: mercurial
+  user: abort: response expected
+  [255]
+  $ cat <<'EOF' | hg id --config ui.interactive=true --config ui.nontty=true 
--debug http://localhost:$HGPORT2/
+  > 
+  > EOF
+  using http://localhost:$HGPORT2/
+  sending capabilities command
+  http authorization required for http://localhost:$HGPORT2/
+  realm: mercurial
+  user: 
+  password: abort: response expected
+  [255]
+  $ cat <<'EOF' | hg id --config ui.interactive=true --config ui.nontty=true 
--debug http://localhost:$HGPORT2/
+  > 
+  > 
+  > EOF
+  using http://localhost:$HGPORT2/
+  sending capabilities command
+  http authorization required for http://localhost:$HGPORT2/
+  realm: mercurial
+  user: 
+  password: abort: authorization failed
+  [255]
   $ hg id --config ui.interactive=true --config extensions.getpass=get_pass.py 
http://user@localhost:$HGPORT2/
   http authorization required for http://localhost:$HGPORT2/
   realm: mercurial
@@ -360,6 +388,9 @@ test http authentication
   "GET /?cmd=capabilities HTTP/1.1" 401 -
   "GET /?cmd=capabilities HTTP/1.1" 401 -
   "GET /?cmd=capabilities HTTP/1.1" 401 -
+  "GET /?cmd=capabilities HTTP/1.1" 401 -
+  "GET /?cmd=capabilities HTTP/1.1" 401 -
+  "GET /?cmd=capabilities HTTP/1.1" 401 -
   "GET /?cmd=capabilities HTTP/1.1" 200 -
   "GET /?cmd=lookup HTTP/1.1" 200 - x-hgarg-1:key=tip x-hgproto-1:0.1 0.2 
comp=$USUAL_COMPRESSIONS$ partial-pull
   "GET /?cmd=listkeys HTTP/1.1" 200 - x-hgarg-1:namespace=namespaces 
x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 1 of 2 STABLE] ui: fix echo back of ui.prompt() to not concatenate None as bytes

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1603451417 -32400
#  Fri Oct 23 20:10:17 2020 +0900
# Branch stable
# Node ID 15e5d72561363a7e83ea9a7c70d85db760bc57e8
# Parent  b713e4cae2d76e5930925c32d3ae2ae1b5a2601d
ui: fix echo back of ui.prompt() to not concatenate None as bytes

Spotted while writing tests for the issue6425. The default value may be
None.

diff --git a/mercurial/ui.py b/mercurial/ui.py
--- a/mercurial/ui.py
+++ b/mercurial/ui.py
@@ -1659,7 +1659,9 @@ class ui(object):
 if not r:
 r = default
 if self.configbool(b'ui', b'promptecho'):
-self._writemsg(self._fmsgout, r, b"\n", type=b'promptecho')
+self._writemsg(
+self._fmsgout, r or b'', b"\n", type=b'promptecho'
+)
 return r
 except EOFError:
 raise error.ResponseExpected()
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 1 of 2] revset: rename diff(pattern) to diffcontains(pattern)

[Yuya Nishihara]

On Sat, 17 Oct 2020 13:56:03 +0900, Yuya Nishihara wrote:
> # HG changeset patch
> # User Yuya Nishihara 
> # Date 1602907204 -32400
> #  Sat Oct 17 13:00:04 2020 +0900
> # Node ID ff0d9965fc9db08b95ad90523f47fa0cc5f5118e
> # Parent  94f681b84c7016289763937cde633cb1d8c831b1
> revset: rename diff(pattern) to diffcontains(pattern)

These patches are intended to be in 5.6.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH] help: update command synopsis to clarify "cp --forget" only takes destinations

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1602939441 -32400
#  Sat Oct 17 21:57:21 2020 +0900
# Node ID 83d7de825c015dfa43de228513ca8a75676a8286
# Parent  da20256a34d4c5804d7210b6c4f83767907508e6
help: update command synopsis to clarify "cp --forget" only takes destinations

I'm a bit confused while reading 03690079d7dd, which says "a destination
file", but the code loops over matched files.

diff --git a/mercurial/commands.py b/mercurial/commands.py
--- a/mercurial/commands.py
+++ b/mercurial/commands.py
@@ -2343,7 +2343,7 @@ def continuecmd(ui, repo, **opts):
 ]
 + walkopts
 + dryrunopts,
-_(b'[OPTION]... SOURCE... DEST'),
+_(b'[OPTION]... (SOURCE... DEST | --forget DEST...)'),
 helpcategory=command.CATEGORY_FILE_CONTENTS,
 )
 def copy(ui, repo, *pats, **opts):
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 1 of 2] revset: rename diff(pattern) to diffcontains(pattern)

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1602907204 -32400
#  Sat Oct 17 13:00:04 2020 +0900
# Node ID ff0d9965fc9db08b95ad90523f47fa0cc5f5118e
# Parent  94f681b84c7016289763937cde633cb1d8c831b1
revset: rename diff(pattern) to diffcontains(pattern)

Suggested by Augie, and I think it's better name.

diff --git a/mercurial/revset.py b/mercurial/revset.py
--- a/mercurial/revset.py
+++ b/mercurial/revset.py
@@ -994,24 +994,27 @@ def destination(repo, subset, x):
 )
 
 
-@predicate(b'diff(pattern)', weight=110)
-def diff(repo, subset, x):
+@predicate(b'diffcontains(pattern)', weight=110)
+def diffcontains(repo, subset, x):
 """Search revision differences for when the pattern was added or removed.
 
 The pattern may be a substring literal or a regular expression. See
 :hg:`help revisions.patterns`.
 """
-args = getargsdict(x, b'diff', b'pattern')
+args = getargsdict(x, b'diffcontains', b'pattern')
 if b'pattern' not in args:
-# i18n: "diff" is a keyword
-raise error.ParseError(_(b'diff takes at least 1 argument'))
-
-pattern = getstring(args[b'pattern'], _(b'diff requires a string pattern'))
+# i18n: "diffcontains" is a keyword
+raise error.ParseError(_(b'diffcontains takes at least 1 argument'))
+
+pattern = getstring(
+args[b'pattern'], _(b'diffcontains requires a string pattern')
+)
 regexp = stringutil.substringregexp(pattern, re.M)
 
 # TODO: add support for file pattern and --follow. For example,
-# diff(pattern[, set]) where set may be file(pattern) or follow(pattern),
-# and we'll eventually add a support for narrowing files by revset?
+# diffcontains(pattern[, set]) where set may be file(pattern) or
+# follow(pattern), and we'll eventually add a support for narrowing
+# files by revset?
 fmatch = matchmod.always()
 
 def makefilematcher(ctx):
@@ -1030,7 +1033,7 @@ def diff(repo, subset, x):
 found = True
 return found
 
-return subset.filter(testdiff, condrepr=(b'', pattern))
+return subset.filter(testdiff, condrepr=(b'', pattern))
 
 
 @predicate(b'contentdivergent()', safe=True)
diff --git a/tests/test-grep.t b/tests/test-grep.t
--- a/tests/test-grep.t
+++ b/tests/test-grep.t
@@ -23,13 +23,13 @@ pattern error
 
 invalid revset syntax
 
-  $ hg log -r 'diff()'
-  hg: parse error: diff takes at least 1 argument
+  $ hg log -r 'diffcontains()'
+  hg: parse error: diffcontains takes at least 1 argument
   [255]
-  $ hg log -r 'diff(:)'
-  hg: parse error: diff requires a string pattern
+  $ hg log -r 'diffcontains(:)'
+  hg: parse error: diffcontains requires a string pattern
   [255]
-  $ hg log -r 'diff("re:**test**")'
+  $ hg log -r 'diffcontains("re:**test**")'
   hg: parse error: invalid regular expression: nothing to repeat* (glob)
   [255]
 
@@ -567,13 +567,13 @@ Test wdir
 
 revset predicate for "grep --diff"
 
-  $ hg log -qr 'diff("re:^bl...$")'
+  $ hg log -qr 'diffcontains("re:^bl...$")'
   0:203191eb5e21
-  $ hg log -qr 'diff("orange")'
+  $ hg log -qr 'diffcontains("orange")'
   1:7c585a21e0d1
   2:11bd8bc8d653
   3:e0116d3829f8
-  $ hg log -qr '2:0 & diff("orange")'
+  $ hg log -qr '2:0 & diffcontains("orange")'
   2:11bd8bc8d653
   1:7c585a21e0d1
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 2] relnotes: add diffcontains() to new features list

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1602907578 -32400
#  Sat Oct 17 13:06:18 2020 +0900
# Node ID da20256a34d4c5804d7210b6c4f83767907508e6
# Parent  ff0d9965fc9db08b95ad90523f47fa0cc5f5118e
relnotes: add diffcontains() to new features list

diff --git a/relnotes/next b/relnotes/next
--- a/relnotes/next
+++ b/relnotes/next
@@ -5,6 +5,9 @@
source whether or not the source still exists (but the source must
exist in the parent revision).
 
+ * New revset predicate `diffcontains(pattern)` for filtering revisions
+   in the same way as `hg grep --diff pattern`.
+
 
 == New Experimental Features ==
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 2 of 4] dispatch: load shared source repository config in share-safe mode

[Yuya Nishihara]

On Fri, 16 Oct 2020 18:54:38 +0530, Pulkit Goyal wrote:
> On Fri, Oct 16, 2020 at 6:48 PM Yuya Nishihara  wrote:
> >
> > On Fri, 16 Oct 2020 18:08:04 +0530, Pulkit Goyal wrote:
> > > On Fri, Oct 16, 2020 at 4:27 PM Yuya Nishihara  wrote:
> > > >
> > > > On Fri, 16 Oct 2020 13:26:32 +0530, Pulkit Goyal wrote:
> > > > > # HG changeset patch
> > > > > # User Pulkit Goyal <7895pul...@gmail.com>
> > > > > # Date 1600435358 -19800
> > > > > #  Fri Sep 18 18:52:38 2020 +0530
> > > > > # Node ID 03bb891ad7fffc564b5af73f8887c41f4eb0a7f3
> > > > > # Parent  90e0e681dc5b94b1636417b7f5da75762a88572f
> > > > > # EXP-Topic share-safe
> > > > > dispatch: load shared source repository config in share-safe mode
> > > >
> > > > > +def _readsharedsourceconfig(ui, path):
> > > > > +"""if the current repository is shared one, this tries to read
> > > > > +.hg/hgrc of shared source if we are in share-safe mode
> > > > > +
> > > > > +Config read is loaded into the ui object passed
> > > > > +
> > > > > +This should be called before reading .hg/hgrc or the main repo
> > > > > +as that overrides config set in shared source"""
> > > > > +try:
> > > > > +with open(os.path.join(path, b".hg", b"requires"), "rb") as 
> > > > > fp:
> > > > > +requirements = set(fp.read().splitlines())
> > > > > +if not (
> > > > > +requirementsmod.SHARESAFE_REQUIREMENT in requirements
> > > > > +and requirementsmod.SHARED_REQUIREMENT in 
> > > > > requirements
> > > > > +):
> > > > > +return
> > > > > +hgvfs = vfs.vfs(os.path.join(path, b".hg"))
> > > > > +sharedvfs = localrepo._getsharedvfs(hgvfs, requirements)
> > > > > +ui.readconfig(sharedvfs.join(b"hgrc"), path)
> > > >
> > > > Maybe relative paths need to be resolved against shared wdir path, not 
> > > > the
> > > > path of the current repo.
> > >
> > > Here, path of the current repo = shared wdir path.
> > >
> > > Let's say we have a repo `main` and we share it as `sharedrepo`. This
> > > patch implements reading of `.hg/hgrc` of main repository when hg is
> > > run in `sharedrepo`.
> >
> > I meant paths in main_repo_root/.hg/hgrc should be resolved relative to
> > main_repo_root, not the current repo root.
> 
> Isn't we are doing that by first getting the sharedvfs (which returns
> main_repo_root/.hg) and join with that. Or you mean
> `localrepo._getsharedvfs()` is not doing the correct thing?

The path of the config file is correct, but its data are resolved against
the "path" of the current repo by ui.readconfig(..., root=path). Since the
config file is placed inside the main repo, relative paths in that config
file should be resolved as if the file were read in the main repo.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 3 of 4] config: add a .hg/hgrc-not-shared which won't be shared in share-safe mode

[Yuya Nishihara]

On Fri, 16 Oct 2020 18:12:06 +0530, Pulkit Goyal wrote:
> On Fri, Oct 16, 2020 at 4:27 PM Yuya Nishihara  wrote:
> > On Fri, 16 Oct 2020 13:26:33 +0530, Pulkit Goyal wrote:
> > > # HG changeset patch
> > > # User Pulkit Goyal <7895pul...@gmail.com>
> > > # Date 1593596699 -19800
> > > #  Wed Jul 01 15:14:59 2020 +0530
> > > # Node ID aea9b1e5192225f9d88773b3c3039dd307b3ad89
> > > # Parent  03bb891ad7fffc564b5af73f8887c41f4eb0a7f3
> > > # EXP-Topic share-safe
> > > config: add a .hg/hgrc-not-shared which won't be shared in share-safe mode
> > >
> > > Previous patches add a safe mode for sharing repositories which involve 
> > > sharing
> > > of source requirements and config files.
> > >
> > > In certain situations we might need to add a config to source repository 
> > > which
> > > we does not want to share. For this, we add a `.hg/hgrc-not-shared` which 
> > > won't
> > > be shared.
> >
> > Can you elaborate the "certain situations" more?
> 
> Things like usernames, paths, wdir specific extension like sparse.

I fell it's fine to share usernames and paths across repositories, which can
be overridden if needed.

For sparse, I don't know the current implementation, but I think it should be
gated by repo requirements. Extension will be loaded, but turned off if
requirements don't match.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 2 of 4] dispatch: load shared source repository config in share-safe mode

[Yuya Nishihara]

On Fri, 16 Oct 2020 18:08:04 +0530, Pulkit Goyal wrote:
> On Fri, Oct 16, 2020 at 4:27 PM Yuya Nishihara  wrote:
> >
> > On Fri, 16 Oct 2020 13:26:32 +0530, Pulkit Goyal wrote:
> > > # HG changeset patch
> > > # User Pulkit Goyal <7895pul...@gmail.com>
> > > # Date 1600435358 -19800
> > > #  Fri Sep 18 18:52:38 2020 +0530
> > > # Node ID 03bb891ad7fffc564b5af73f8887c41f4eb0a7f3
> > > # Parent  90e0e681dc5b94b1636417b7f5da75762a88572f
> > > # EXP-Topic share-safe
> > > dispatch: load shared source repository config in share-safe mode
> >
> > > +def _readsharedsourceconfig(ui, path):
> > > +"""if the current repository is shared one, this tries to read
> > > +.hg/hgrc of shared source if we are in share-safe mode
> > > +
> > > +Config read is loaded into the ui object passed
> > > +
> > > +This should be called before reading .hg/hgrc or the main repo
> > > +as that overrides config set in shared source"""
> > > +try:
> > > +with open(os.path.join(path, b".hg", b"requires"), "rb") as fp:
> > > +requirements = set(fp.read().splitlines())
> > > +if not (
> > > +requirementsmod.SHARESAFE_REQUIREMENT in requirements
> > > +and requirementsmod.SHARED_REQUIREMENT in requirements
> > > +):
> > > +return
> > > +hgvfs = vfs.vfs(os.path.join(path, b".hg"))
> > > +sharedvfs = localrepo._getsharedvfs(hgvfs, requirements)
> > > +ui.readconfig(sharedvfs.join(b"hgrc"), path)
> >
> > Maybe relative paths need to be resolved against shared wdir path, not the
> > path of the current repo.
> 
> Here, path of the current repo = shared wdir path.
> 
> Let's say we have a repo `main` and we share it as `sharedrepo`. This
> patch implements reading of `.hg/hgrc` of main repository when hg is
> run in `sharedrepo`.

I meant paths in main_repo_root/.hg/hgrc should be resolved relative to
main_repo_root, not the current repo root.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 7 of 7] revset: add diff(pattern) predicate for "grep --diff"

[Yuya Nishihara]

On Thu, 15 Oct 2020 17:42:48 +0530, Pulkit Goyal wrote:
> On Wed, Oct 14, 2020 at 8:00 PM Yuya Nishihara  wrote:
> > # HG changeset patch
> > # User Yuya Nishihara 
> > # Date 1599556584 -32400
> > #  Tue Sep 08 18:16:24 2020 +0900
> > # Node ID 86c34dc393abb0cd636ec03a2e81a26f4b0694d5
> > # Parent  c71d4b4018af62936283b28ce4ed70cc9ea33840
> > revset: add diff(pattern) predicate for "grep --diff"
> 
> I am not sure whether release-notes script will pick this up. It will
> be nice if you followup with an entry in `relnotes/next` about it.

Sure.

Btw, which name do you prefer?

 a. diff(pattern)
 b. diffcontains(pattern), by augie
 c. ...
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 3 of 4] config: add a .hg/hgrc-not-shared which won't be shared in share-safe mode

[Yuya Nishihara]

On Fri, 16 Oct 2020 13:26:33 +0530, Pulkit Goyal wrote:
> # HG changeset patch
> # User Pulkit Goyal <7895pul...@gmail.com>
> # Date 1593596699 -19800
> #  Wed Jul 01 15:14:59 2020 +0530
> # Node ID aea9b1e5192225f9d88773b3c3039dd307b3ad89
> # Parent  03bb891ad7fffc564b5af73f8887c41f4eb0a7f3
> # EXP-Topic share-safe
> config: add a .hg/hgrc-not-shared which won't be shared in share-safe mode
> 
> Previous patches add a safe mode for sharing repositories which involve 
> sharing
> of source requirements and config files.
> 
> In certain situations we might need to add a config to source repository which
> we does not want to share. For this, we add a `.hg/hgrc-not-shared` which 
> won't
> be shared.

Can you elaborate the "certain situations" more?

I feel it would be tedious job to sort out shared/unshared configs manually.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 2 of 4] dispatch: load shared source repository config in share-safe mode

[Yuya Nishihara]

On Fri, 16 Oct 2020 13:26:32 +0530, Pulkit Goyal wrote:
> # HG changeset patch
> # User Pulkit Goyal <7895pul...@gmail.com>
> # Date 1600435358 -19800
> #  Fri Sep 18 18:52:38 2020 +0530
> # Node ID 03bb891ad7fffc564b5af73f8887c41f4eb0a7f3
> # Parent  90e0e681dc5b94b1636417b7f5da75762a88572f
> # EXP-Topic share-safe
> dispatch: load shared source repository config in share-safe mode

> +def _readsharedsourceconfig(ui, path):
> +"""if the current repository is shared one, this tries to read
> +.hg/hgrc of shared source if we are in share-safe mode
> +
> +Config read is loaded into the ui object passed
> +
> +This should be called before reading .hg/hgrc or the main repo
> +as that overrides config set in shared source"""
> +try:
> +with open(os.path.join(path, b".hg", b"requires"), "rb") as fp:
> +requirements = set(fp.read().splitlines())
> +if not (
> +requirementsmod.SHARESAFE_REQUIREMENT in requirements
> +and requirementsmod.SHARED_REQUIREMENT in requirements
> +):
> +return
> +hgvfs = vfs.vfs(os.path.join(path, b".hg"))
> +sharedvfs = localrepo._getsharedvfs(hgvfs, requirements)
> +ui.readconfig(sharedvfs.join(b"hgrc"), path)

Maybe relative paths need to be resolved against shared wdir path, not the
path of the current repo.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 7 of 7] revset: add diff(pattern) predicate for "grep --diff"

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1599556584 -32400
#  Tue Sep 08 18:16:24 2020 +0900
# Node ID 86c34dc393abb0cd636ec03a2e81a26f4b0694d5
# Parent  c71d4b4018af62936283b28ce4ed70cc9ea33840
revset: add diff(pattern) predicate for "grep --diff"

I find this is useful in GUI log viewer since the tool only needs to support
"log -rREV" command.

This is basic implementation. Windowed search is not implemented since it
wouldn't work pretty well with the smartset API. And filename matcher is
not supported because the syntax isn't determined. My idea is to add handling
of diff(pattern, file(..)) and diff(pattern, follow(..)), which will then be
evolved to a full revset+matcher combinator support:

  x & diff(pattern, y & z)
=
y & z builds (revs(y) & revs(z), matcher(y) & matcher(z))
pair, and narrows the search space of diff()
  
  diff() returns matched (revs, matcher) pair
  
  revs and matcher will be combined respectively by &-operator, and the matcher
  will optionally be used to filter "hg log -p" output

The predicate name "diff()" wouldn't be great, but grep() is already used.
Another options I can think of are "grepdiff()" and "containsdiff()".
Naming suggestions are welcome.

diff --git a/mercurial/revset.py b/mercurial/revset.py
--- a/mercurial/revset.py
+++ b/mercurial/revset.py
@@ -17,6 +17,7 @@ from . import (
 diffutil,
 encoding,
 error,
+grep as grepmod,
 hbisect,
 match as matchmod,
 node,
@@ -993,6 +994,45 @@ def destination(repo, subset, x):
 )
 
 
+@predicate(b'diff(pattern)', weight=110)
+def diff(repo, subset, x):
+"""Search revision differences for when the pattern was added or removed.
+
+The pattern may be a substring literal or a regular expression. See
+:hg:`help revisions.patterns`.
+"""
+args = getargsdict(x, b'diff', b'pattern')
+if b'pattern' not in args:
+# i18n: "diff" is a keyword
+raise error.ParseError(_(b'diff takes at least 1 argument'))
+
+pattern = getstring(args[b'pattern'], _(b'diff requires a string pattern'))
+regexp = stringutil.substringregexp(pattern, re.M)
+
+# TODO: add support for file pattern and --follow. For example,
+# diff(pattern[, set]) where set may be file(pattern) or follow(pattern),
+# and we'll eventually add a support for narrowing files by revset?
+fmatch = matchmod.always()
+
+def makefilematcher(ctx):
+return fmatch
+
+# TODO: search in a windowed way
+searcher = grepmod.grepsearcher(repo.ui, repo, regexp, diff=True)
+
+def testdiff(rev):
+# consume the generator to discard revfiles/matches cache
+found = False
+for fn, ctx, pstates, states in searcher.searchfiles(
+baseset([rev]), makefilematcher
+):
+if next(grepmod.difflinestates(pstates, states), None):
+found = True
+return found
+
+return subset.filter(testdiff, condrepr=(b'', pattern))
+
+
 @predicate(b'contentdivergent()', safe=True)
 def contentdivergent(repo, subset, x):
 """
diff --git a/tests/test-grep.t b/tests/test-grep.t
--- a/tests/test-grep.t
+++ b/tests/test-grep.t
@@ -21,6 +21,18 @@ pattern error
   grep: invalid match pattern: nothing to repeat* (glob)
   [1]
 
+invalid revset syntax
+
+  $ hg log -r 'diff()'
+  hg: parse error: diff takes at least 1 argument
+  [255]
+  $ hg log -r 'diff(:)'
+  hg: parse error: diff requires a string pattern
+  [255]
+  $ hg log -r 'diff("re:**test**")'
+  hg: parse error: invalid regular expression: nothing to repeat* (glob)
+  [255]
+
 simple
 
   $ hg grep -r tip:0 '.*'
@@ -553,6 +565,18 @@ Test wdir
   color:2:-:orange
   color:1:+:orange
 
+revset predicate for "grep --diff"
+
+  $ hg log -qr 'diff("re:^bl...$")'
+  0:203191eb5e21
+  $ hg log -qr 'diff("orange")'
+  1:7c585a21e0d1
+  2:11bd8bc8d653
+  3:e0116d3829f8
+  $ hg log -qr '2:0 & diff("orange")'
+  2:11bd8bc8d653
+  1:7c585a21e0d1
+
 test substring match: '^' should only match at the beginning
 
   $ hg grep -r tip:0 '^.' --config extensions.color= --color debug
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 6 of 7] stringutil: add function to compile stringmatcher pattern into regexp

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1601898039 -32400
#  Mon Oct 05 20:40:39 2020 +0900
# Node ID c71d4b4018af62936283b28ce4ed70cc9ea33840
# Parent  18bb913c45acf3f8d1fcbc90e22853f8e5f8e767
stringutil: add function to compile stringmatcher pattern into regexp

Prepares for adding a revset predicate for "grep --diff". The grep logic
needs a regexp object instead of a match function.

diff --git a/mercurial/utils/stringutil.py b/mercurial/utils/stringutil.py
--- a/mercurial/utils/stringutil.py
+++ b/mercurial/utils/stringutil.py
@@ -376,6 +376,58 @@ def stringmatcher(pattern, casesensitive
 raise error.ProgrammingError(b'unhandled pattern kind: %s' % kind)
 
 
+def substringregexp(pattern, flags=0):
+"""Build a regexp object from a string pattern possibly starting with
+'re:' or 'literal:' prefix.
+
+helper for tests:
+>>> def test(pattern, *tests):
+... regexp = substringregexp(pattern)
+... return [bool(regexp.search(t)) for t in tests]
+>>> def itest(pattern, *tests):
+... regexp = substringregexp(pattern, remod.I)
+... return [bool(regexp.search(t)) for t in tests]
+
+substring matching (no prefix):
+>>> test(b'bcde', b'abc', b'def', b'abcdefg')
+[False, False, True]
+
+substring pattern should be escaped:
+>>> substringregexp(b'.bc').pattern
+'.bc'
+>>> test(b'.bc', b'abc', b'def', b'abcdefg')
+[False, False, False]
+
+regex matching ('re:' prefix)
+>>> test(b're:a.+b', b'nomatch', b'fooadef', b'fooadefbar')
+[False, False, True]
+
+force substring matches ('literal:' prefix)
+>>> test(b'literal:re:foobar', b'foobar', b're:foobar')
+[False, True]
+
+case insensitive literal matches
+>>> itest(b'BCDE', b'abc', b'def', b'abcdefg')
+[False, False, True]
+
+case insensitive regex matches
+>>> itest(b're:A.+b', b'nomatch', b'fooadef', b'fooadefBar')
+[False, False, True]
+"""
+kind, pattern = _splitpattern(pattern)
+if kind == b're':
+try:
+return remod.compile(pattern, flags)
+except remod.error as e:
+raise error.ParseError(
+_(b'invalid regular expression: %s') % forcebytestr(e)
+)
+elif kind == b'literal':
+return remod.compile(remod.escape(pattern), flags)
+
+raise error.ProgrammingError(b'unhandled pattern kind: %s' % kind)
+
+
 def shortuser(user):
 """Return a short representation of a user name or email address."""
 f = user.find(b'@')
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 5 of 7] py3: fix stringmatcher() to byte-stringify exception message

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1602681048 -32400
#  Wed Oct 14 22:10:48 2020 +0900
# Node ID 18bb913c45acf3f8d1fcbc90e22853f8e5f8e767
# Parent  b6c30689b9fe9e5f53bceb7913c71ae98109c804
py3: fix stringmatcher() to byte-stringify exception message

Spotted while writing regexp variant of stringmatcher().

diff --git a/mercurial/utils/stringutil.py b/mercurial/utils/stringutil.py
--- a/mercurial/utils/stringutil.py
+++ b/mercurial/utils/stringutil.py
@@ -361,7 +361,9 @@ def stringmatcher(pattern, casesensitive
 flags = remod.I
 regex = remod.compile(pattern, flags)
 except remod.error as e:
-raise error.ParseError(_(b'invalid regular expression: %s') % e)
+raise error.ParseError(
+_(b'invalid regular expression: %s') % forcebytestr(e)
+)
 return kind, pattern, regex.search
 elif kind == b'literal':
 if casesensitive:
diff --git a/tests/test-revset.t b/tests/test-revset.t
--- a/tests/test-revset.t
+++ b/tests/test-revset.t
@@ -1448,6 +1448,9 @@ test author
 (string '('))
   hg: parse error: invalid match pattern: (unbalanced parenthesis|missing 
\),.*) (re)
   [255]
+  $ log 'desc("re:(")'
+  hg: parse error: invalid regular expression: (unbalanced parenthesis|missing 
\),.*) (re)
+  [255]
   $ try 'grep("\bissue\d+")'
   (func
 (symbol 'grep')
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 4 of 7] stringutil: extract helper function that splits stringmatcher() pattern

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1601898814 -32400
#  Mon Oct 05 20:53:34 2020 +0900
# Node ID b6c30689b9fe9e5f53bceb7913c71ae98109c804
# Parent  5b3013b8e40a903e7d31196c1e00877120339242
stringutil: extract helper function that splits stringmatcher() pattern

diff --git a/mercurial/utils/stringutil.py b/mercurial/utils/stringutil.py
--- a/mercurial/utils/stringutil.py
+++ b/mercurial/utils/stringutil.py
@@ -307,6 +307,14 @@ def binary(s):
 return bool(s and b'\0' in s)
 
 
+def _splitpattern(pattern):
+if pattern.startswith(b're:'):
+return b're', pattern[3:]
+elif pattern.startswith(b'literal:'):
+return b'literal', pattern[8:]
+return b'literal', pattern
+
+
 def stringmatcher(pattern, casesensitive=True):
 """
 accepts a string, possibly starting with 're:' or 'literal:' prefix.
@@ -345,8 +353,8 @@ def stringmatcher(pattern, casesensitive
 >>> itest(b'ABCDEFG', b'abc', b'def', b'abcdefg')
 ('literal', 'ABCDEFG', [False, False, True])
 """
-if pattern.startswith(b're:'):
-pattern = pattern[3:]
+kind, pattern = _splitpattern(pattern)
+if kind == b're':
 try:
 flags = 0
 if not casesensitive:
@@ -354,16 +362,16 @@ def stringmatcher(pattern, casesensitive
 regex = remod.compile(pattern, flags)
 except remod.error as e:
 raise error.ParseError(_(b'invalid regular expression: %s') % e)
-return b're', pattern, regex.search
-elif pattern.startswith(b'literal:'):
-pattern = pattern[8:]
+return kind, pattern, regex.search
+elif kind == b'literal':
+if casesensitive:
+match = pattern.__eq__
+else:
+ipat = encoding.lower(pattern)
+match = lambda s: ipat == encoding.lower(s)
+return kind, pattern, match
 
-match = pattern.__eq__
-
-if not casesensitive:
-ipat = encoding.lower(pattern)
-match = lambda s: ipat == encoding.lower(s)
-return b'literal', pattern, match
+raise error.ProgrammingError(b'unhandled pattern kind: %s' % kind)
 
 
 def shortuser(user):
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 3 of 7] grep: extract main search loop as searcher method

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1599639458 -32400
#  Wed Sep 09 17:17:38 2020 +0900
# Node ID 5b3013b8e40a903e7d31196c1e00877120339242
# Parent  09a598f9d9790918f47f66b6ce5e51f198510289
grep: extract main search loop as searcher method

Still displayer part is in commands.grep(), the core grep logic is now
reusable. I'll revisit the displayer stuff later since it will be another
long series.

diff --git a/mercurial/commands.py b/mercurial/commands.py
--- a/mercurial/commands.py
+++ b/mercurial/commands.py
@@ -3403,8 +3403,6 @@ def grep(ui, repo, pattern, *pats, **opt
 )
 
 getfile = searcher._getfile
-matches = searcher._matches
-copies = searcher._copies
 
 uipathfn = scmutil.getuipathfn(repo)
 
@@ -3514,8 +3512,6 @@ def grep(ui, repo, pattern, *pats, **opt
 fm.data(matched=False)
 fm.end()
 
-skip = searcher._skip
-revfiles = searcher._revfiles
 found = False
 
 wopts = logcmdutil.walkopts(
@@ -3532,29 +3528,11 @@ def grep(ui, repo, pattern, *pats, **opt
 
 ui.pager(b'grep')
 fm = ui.formatter(b'grep', opts)
-for ctx in scmutil.walkchangerevs(
-repo, revs, makefilematcher, searcher._prep
-):
-rev = ctx.rev()
-parent = ctx.p1().rev()
-for fn in sorted(revfiles.get(rev, [])):
-states = matches[rev][fn]
-copy = copies.get(rev, {}).get(fn)
-if fn in skip:
-if copy:
-skip.add(copy)
-continue
-pstates = matches.get(parent, {}).get(copy or fn, [])
-if pstates or states:
-r = display(fm, fn, ctx, pstates, states)
-found = found or r
-if r and not diff and not all_files:
-searcher.skipfile(fn, rev)
-del revfiles[rev]
-# We will keep the matches dict for the duration of the window
-# clear the matches dict once the window is over
-if not revfiles:
-matches.clear()
+for fn, ctx, pstates, states in searcher.searchfiles(revs, 
makefilematcher):
+r = display(fm, fn, ctx, pstates, states)
+found = found or r
+if r and not diff and not all_files:
+searcher.skipfile(fn, ctx.rev())
 fm.end()
 
 return not found
diff --git a/mercurial/grep.py b/mercurial/grep.py
--- a/mercurial/grep.py
+++ b/mercurial/grep.py
@@ -115,6 +115,34 @@ class grepsearcher(object):
 if copy:
 self._skip.add(copy)
 
+def searchfiles(self, revs, makefilematcher):
+"""Walk files and revisions to yield (fn, ctx, pstates, states)
+matches
+
+states is a list of linestate objects. pstates may be empty unless
+diff is True.
+"""
+for ctx in scmutil.walkchangerevs(
+self._repo, revs, makefilematcher, self._prep
+):
+rev = ctx.rev()
+parent = ctx.p1().rev()
+for fn in sorted(self._revfiles.get(rev, [])):
+states = self._matches[rev][fn]
+copy = self._copies.get(rev, {}).get(fn)
+if fn in self._skip:
+if copy:
+self._skip.add(copy)
+continue
+pstates = self._matches.get(parent, {}).get(copy or fn, [])
+if pstates or states:
+yield fn, ctx, pstates, states
+del self._revfiles[rev]
+# We will keep the matches dict for the duration of the window
+# clear the matches dict once the window is over
+if not self._revfiles:
+self._matches.clear()
+
 def _grepbody(self, fn, rev, body):
 self._matches[rev].setdefault(fn, [])
 m = self._matches[rev][fn]
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 7] scmutil: move walkchangerevs() from cmdutil

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1601785077 -32400
#  Sun Oct 04 13:17:57 2020 +0900
# Node ID 09a598f9d9790918f47f66b6ce5e51f198510289
# Parent  d36d703c291e45670245384440993ff70ad17da4
scmutil: move walkchangerevs() from cmdutil

It's no longer a command-level function, but a pure helper to walk revisions
in a windowed way. This change will help eliminate reverse dependency of
revset.py -> grep.py -> cmdutil.py in future patches.

diff --git a/hgext/churn.py b/hgext/churn.py
--- a/hgext/churn.py
+++ b/hgext/churn.py
@@ -23,6 +23,7 @@ from mercurial import (
 patch,
 pycompat,
 registrar,
+scmutil,
 )
 
 cmdtable = {}
@@ -98,7 +99,7 @@ def countrate(ui, repo, amap, *pats, **o
 exclude_pats=opts[b'exclude'],
 )
 revs, makefilematcher = logcmdutil.makewalker(repo, wopts)
-for ctx in cmdutil.walkchangerevs(repo, revs, makefilematcher, prep):
+for ctx in scmutil.walkchangerevs(repo, revs, makefilematcher, prep):
 continue
 
 progress.complete()
diff --git a/mercurial/cmdutil.py b/mercurial/cmdutil.py
--- a/mercurial/cmdutil.py
+++ b/mercurial/cmdutil.py
@@ -2240,55 +2240,6 @@ def finddate(ui, repo, date):
 return b'%d' % rev
 
 
-def increasingwindows(windowsize=8, sizelimit=512):
-while True:
-yield windowsize
-if windowsize < sizelimit:
-windowsize *= 2
-
-
-def walkchangerevs(repo, revs, makefilematcher, prepare):
-'''Iterate over files and the revs in a "windowed" way.
-
-Callers most commonly need to iterate backwards over the history
-in which they are interested. Doing so has awful (quadratic-looking)
-performance, so we use iterators in a "windowed" way.
-
-We walk a window of revisions in the desired order.  Within the
-window, we first walk forwards to gather data, then in the desired
-order (usually backwards) to display it.
-
-This function returns an iterator yielding contexts. Before
-yielding each context, the iterator will first call the prepare
-function on each context in the window in forward order.'''
-
-if not revs:
-return []
-change = repo.__getitem__
-
-def iterate():
-it = iter(revs)
-stopiteration = False
-for windowsize in increasingwindows():
-nrevs = []
-for i in pycompat.xrange(windowsize):
-rev = next(it, None)
-if rev is None:
-stopiteration = True
-break
-nrevs.append(rev)
-for rev in sorted(nrevs):
-ctx = change(rev)
-prepare(ctx, makefilematcher(ctx))
-for rev in nrevs:
-yield change(rev)
-
-if stopiteration:
-break
-
-return iterate()
-
-
 def add(ui, repo, match, prefix, uipathfn, explicitonly, **opts):
 bad = []
 
diff --git a/mercurial/commands.py b/mercurial/commands.py
--- a/mercurial/commands.py
+++ b/mercurial/commands.py
@@ -3532,7 +3532,7 @@ def grep(ui, repo, pattern, *pats, **opt
 
 ui.pager(b'grep')
 fm = ui.formatter(b'grep', opts)
-for ctx in cmdutil.walkchangerevs(
+for ctx in scmutil.walkchangerevs(
 repo, revs, makefilematcher, searcher._prep
 ):
 rev = ctx.rev()
diff --git a/mercurial/scmutil.py b/mercurial/scmutil.py
--- a/mercurial/scmutil.py
+++ b/mercurial/scmutil.py
@@ -760,6 +760,55 @@ def revrange(repo, specs, localalias=Non
 return repo.anyrevs(allspecs, user=True, localalias=localalias)
 
 
+def increasingwindows(windowsize=8, sizelimit=512):
+while True:
+yield windowsize
+if windowsize < sizelimit:
+windowsize *= 2
+
+
+def walkchangerevs(repo, revs, makefilematcher, prepare):
+'''Iterate over files and the revs in a "windowed" way.
+
+Callers most commonly need to iterate backwards over the history
+in which they are interested. Doing so has awful (quadratic-looking)
+performance, so we use iterators in a "windowed" way.
+
+We walk a window of revisions in the desired order.  Within the
+window, we first walk forwards to gather data, then in the desired
+order (usually backwards) to display it.
+
+This function returns an iterator yielding contexts. Before
+yielding each context, the iterator will first call the prepare
+function on each context in the window in forward order.'''
+
+if not revs:
+return []
+change = repo.__getitem__
+
+def iterate():
+it = iter(revs)
+stopiteration = False
+for windowsize in increasingwindows():
+nrevs = []
+for i in pycompat.xrange(windowsize):
+rev = next(it, None)
+if rev is None:
+stopiteration = True
+break
+nrevs.append(rev)
+for rev in sorted(nrevs):
+  

[PATCH 1 of 7] grep: extract public function to register file to be skipped

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1599638684 -32400
#  Wed Sep 09 17:04:44 2020 +0900
# Node ID d36d703c291e45670245384440993ff70ad17da4
# Parent  0428978bca22dc1173577ca6247d588182805b78
grep: extract public function to register file to be skipped

The main grep loop will be extracted to a searcher method, but this skipping
condition depends on the result of display() function.

diff --git a/mercurial/commands.py b/mercurial/commands.py
--- a/mercurial/commands.py
+++ b/mercurial/commands.py
@@ -3549,9 +3549,7 @@ def grep(ui, repo, pattern, *pats, **opt
 r = display(fm, fn, ctx, pstates, states)
 found = found or r
 if r and not diff and not all_files:
-skip.add(fn)
-if copy:
-skip.add(copy)
+searcher.skipfile(fn, rev)
 del revfiles[rev]
 # We will keep the matches dict for the duration of the window
 # clear the matches dict once the window is over
diff --git a/mercurial/grep.py b/mercurial/grep.py
--- a/mercurial/grep.py
+++ b/mercurial/grep.py
@@ -107,6 +107,14 @@ class grepsearcher(object):
 self._skip = set()
 self._revfiles = {}
 
+def skipfile(self, fn, rev):
+"""Exclude the given file (and the copy at the specified revision)
+from future search"""
+copy = self._copies.get(rev, {}).get(fn)
+self._skip.add(fn)
+if copy:
+self._skip.add(copy)
+
 def _grepbody(self, fn, rev, body):
 self._matches[rev].setdefault(fn, [])
 m = self._matches[rev][fn]
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH V2] mergestate: add `allextras()` to get all extras

[Yuya Nishihara]

On Wed, 14 Oct 2020 13:47:17 +0530, Pulkit Goyal wrote:
> # HG changeset patch
> # User Pulkit Goyal <7895pul...@gmail.com>
> # Date 1602313984 -19800
> #  Sat Oct 10 12:43:04 2020 +0530
> # Node ID 4884022d4ad9b918f863946f6557fac47c664af7
> # Parent  04de8a1ec08f380fda794dda3b3f2ed1ccfd442b
> # EXP-Topic merge-newnode-final
> mergestate: add `allextras()` to get all extras

Queued, thanks.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH V2] mergestate: document `o` merge record state in _mergestate_base docs

[Yuya Nishihara]

On Wed, 14 Oct 2020 13:40:43 +0530, Pulkit Goyal wrote:
> # HG changeset patch
> # User Pulkit Goyal <7895pul...@gmail.com>
> # Date 1601991694 -19800
> #  Tue Oct 06 19:11:34 2020 +0530
> # Node ID c3ffbf50856b954e3a8be968b9a93000b03b1843
> # Parent  04de8a1ec08f380fda794dda3b3f2ed1ccfd442b
> mergestate: document `o` merge record state in _mergestate_base docs

Queued, thanks.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] mergestate: document `o` merge record state in _mergestate_base docs

[Yuya Nishihara]

On Tue, 13 Oct 2020 13:05:02 +0530, Pulkit Goyal wrote:
> # HG changeset patch
> # User Pulkit Goyal <7895pul...@gmail.com>
> # Date 1601991694 -19800
> #  Tue Oct 06 19:11:34 2020 +0530
> # Node ID 8af18289942a8e747ff76e28e80f3c9e2b437f18
> # Parent  d1759b2e18889eb2f58e9d348f45d76af047ba08
> mergestate: document `o` merge record state in _mergestate_base docs
> 
> _mergestate_base documentation serves as a nice documentation for mergestate.
> This also documents known merge records and known merge record states.
> 
> I missed adding `o` state to it when I introduced it. Let's add it now.
> 
> Differential Revision: https://phab.mercurial-scm.org/D9156
> 
> diff --git a/mercurial/mergestate.py b/mercurial/mergestate.py
> --- a/mercurial/mergestate.py
> +++ b/mercurial/mergestate.py
> @@ -160,6 +160,7 @@ class _mergestate_base(object):
>  r: resolved conflict
>  pu: unresolved path conflict (file conflicts with directory)
>  pr: resolved path conflict
> +o: file was merged in favor of other parent of merge

Maybe better to say it is " (deprecated)"?
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 2 of 5] commit: refactor salvage calculation to a different function

[Yuya Nishihara]

On Tue, 13 Oct 2020 13:03:28 +0530, Pulkit Goyal wrote:
> # HG changeset patch
> # User Pulkit Goyal <7895pul...@gmail.com>
> # Date 1602314615 -19800
> #  Sat Oct 10 12:53:35 2020 +0530
> # Node ID d3df36bcf4a885038d5df6302f18008945ce48e7
> # Parent  ca34efec098e04ef39341ef181c038f4a4522148
> # EXP-Topic merge-newnode-final
> commit: refactor salvage calculation to a different function

Queued 2-5, thanks.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 1 of 5] mergestate: make filename argument optional in _mergestate_base.extras()

[Yuya Nishihara]

On Tue, 13 Oct 2020 13:03:27 +0530, Pulkit Goyal wrote:
> # HG changeset patch
> # User Pulkit Goyal <7895pul...@gmail.com>
> # Date 1602313984 -19800
> #  Sat Oct 10 12:43:04 2020 +0530
> # Node ID ca34efec098e04ef39341ef181c038f4a4522148
> # Parent  64a9423450efb39d7f1bc5b6b2cca1efafb1870e
> # EXP-Topic merge-newnode-final
> mergestate: make filename argument optional in _mergestate_base.extras()

> diff --git a/mercurial/mergestate.py b/mercurial/mergestate.py
> --- a/mercurial/mergestate.py
> +++ b/mercurial/mergestate.py
> @@ -305,7 +305,12 @@ class _mergestate_base(object):
>  ):
>  yield f
>  
> -def extras(self, filename):
> +def extras(self, filename=None):
> +""" return extras stored with the mergestate
> +
> +if filename is passed, extras for that file is only returned """
> +if filename is None:
> +return self._stateextras
>  return self._stateextras[filename]

How about adding a separate function? I don't like this kind of overloads
because different return types typically mean the function will serve
differently.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: Default behavior of `hg absorb` with respect to empty changesets

[Yuya Nishihara]

On Sun, 11 Oct 2020 05:25:01 +0200, Manuel Jacob wrote:
> In 1ca0047fd7e1, I changed the default behavior of absorb to preserve 
> empty changesets that were already empty. Changesets that become empty 
> continue to be dropped by default (later and independently, I added a 
> configuration to change that behavior).
> 
> My reasoning was that no commands create empty changesets by default, so 
> it must be a deliberate choice of the user to create the empty 
> changesets and absorb shouldn’t mess with this. The presupposition is 
> true for the most part, however I forgot about one exception: `hg 
> uncommit` creates an empty changeset if everything is selected to be 
> uncommitted. That breaks a probably not too uncommon workflow where one 
> wants to absorb a changeset into its predecessors: `hg update X`, `hg 
> uncommit --all`, `hg absorb`. Before hg 5.5, `hg absorb` dropped the 
> empty changeset (that was in this case created by `hg uncommit`), but 
> starting with hg 5.5, the empty changeset would stay (more correctly, it 
> gets re-created). If X was meant to be a temporary changeset, my patch 
> makes an extra manual step (to prune that changeset) necessary.
> 
> Ideally, we would finish `hg absorb --from` soon, replacing the above 
> workflow by an easier one that doesn’t have the problem.
> 
> I heard that some people don’t like the fact that `hg uncommit --all` 
> creates an empty changeset instead of pruning the changeset. Changing 
> that behavior would make the extra step unnecessary.

I personally don't care if "hg uncommit" would prune the empty changeset or
not, but changing the default would break some use case such as "hg uncommit"
everything + "hg amend -i" to re-select changes to be committed. And wrong
amend isn't easy to recover.

So I'm -1 to change the default "hg uncommit" behavior.

> Personally, I never want any changeset to get pruned automatically, no 
> matter whether it was already empty or not. Therefore, I implemented the 
> rewrite.empty-successor configuration that makes that behavior 
> configurable.

Maybe we can add a rewrite.empty-successor variant (new config option or
new value of rewrite.empty-successor) that only applies to a single p1
mutation (e.g. amend, uncommit, etc.), and set it to "keep" by default?
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 3 of 3] revlog: prevent recreating a tuple again and again for each rev

[Yuya Nishihara]

On Sun, 11 Oct 2020 21:51:37 +0200, Joerg Sonnenberger wrote:
> On Sun, Oct 11, 2020 at 11:32:34AM +0900, Yuya Nishihara wrote:
> > On Sat, 10 Oct 2020 14:17:24 +0530, Pulkit Goyal wrote:
> > > # HG changeset patch
> > > # User Pulkit Goyal <7895pul...@gmail.com>
> > > # Date 1602252446 -19800
> > > #  Fri Oct 09 19:37:26 2020 +0530
> > > # Node ID a96cd3eebf8eba04a351615728c9635e8941af10
> > > # Parent  911cea33820c98da3fa2e4891153e674600df1af
> > > # EXP-Topic sidedata-upgrade
> > > revlog: prevent recreating a tuple again and again for each rev
> > > 
> > > I am tracking a memory leak during upgrade which does revlog cloning. 
> > > Although
> > > this does not make much of difference but seemed like a nice change to me.
> > > 
> > > diff --git a/mercurial/revlog.py b/mercurial/revlog.py
> > > --- a/mercurial/revlog.py
> > > +++ b/mercurial/revlog.py
> > > @@ -2773,6 +2773,7 @@ class revlog(object):
> > >  sidedatachanges = sidedatacompanion is not None
> > >  deltacomputer = deltautil.deltacomputer(destrevlog)
> > >  index = self.index
> > > +defsidedataactions = (False, (), {})
> > 
> > I don't think this would matter for memory usage, but sidedataactions
> > can be assigned here if we really want to optimize it.
> 
> The difficult and dangerous part here is the dictionary as mutable
> field. That prevents the use of a single constant object.

Yep, creating defsidedataactions per loop is safer option if we don't
know the update dict isn't mutated. I just meant the defsidedataactions
reference can be eliminated.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 1 of 3] upgrade: improve documentation of matchrevlog()

[Yuya Nishihara]

On Sat, 10 Oct 2020 14:17:22 +0530, Pulkit Goyal wrote:
> # HG changeset patch
> # User Pulkit Goyal <7895pul...@gmail.com>
> # Date 1602158391 -19800
> #  Thu Oct 08 17:29:51 2020 +0530
> # Node ID 3fb44ab16f68b2f76a752e873cce80d45fec442f
> # Parent  64a9423450efb39d7f1bc5b6b2cca1efafb1870e
> # EXP-Topic sidedata-upgrade
> upgrade: improve documentation of matchrevlog()

Queued this, thanks.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 3 of 3] revlog: prevent recreating a tuple again and again for each rev

[Yuya Nishihara]

On Sat, 10 Oct 2020 14:17:24 +0530, Pulkit Goyal wrote:
> # HG changeset patch
> # User Pulkit Goyal <7895pul...@gmail.com>
> # Date 1602252446 -19800
> #  Fri Oct 09 19:37:26 2020 +0530
> # Node ID a96cd3eebf8eba04a351615728c9635e8941af10
> # Parent  911cea33820c98da3fa2e4891153e674600df1af
> # EXP-Topic sidedata-upgrade
> revlog: prevent recreating a tuple again and again for each rev
> 
> I am tracking a memory leak during upgrade which does revlog cloning. Although
> this does not make much of difference but seemed like a nice change to me.
> 
> diff --git a/mercurial/revlog.py b/mercurial/revlog.py
> --- a/mercurial/revlog.py
> +++ b/mercurial/revlog.py
> @@ -2773,6 +2773,7 @@ class revlog(object):
>  sidedatachanges = sidedatacompanion is not None
>  deltacomputer = deltautil.deltacomputer(destrevlog)
>  index = self.index
> +defsidedataactions = (False, (), {})

I don't think this would matter for memory usage, but sidedataactions
can be assigned here if we really want to optimize it.

  sidedataactions = (False, [], {})  # constant if not sidedatachanges
  for rev in self:
  ...
  if sidedatachanges:
  sidedataactions = ...
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH V2] revset: fix sorting key of wdir revision

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1602238750 -32400
#  Fri Oct 09 19:19:10 2020 +0900
# Node ID 1ffc832a5390d73e844ff7f6e4cc05b168030fe0
# Parent  17a12f53dd72d39ce62721a0a43b681d4b4b4bb8
revset: fix sorting key of wdir revision

It would go wrong on Python 2, and would crash on Python 3.

diff --git a/mercurial/revset.py b/mercurial/revset.py
--- a/mercurial/revset.py
+++ b/mercurial/revset.py
@@ -2287,13 +2287,13 @@ def roots(repo, subset, x):
 
 
 _sortkeyfuncs = {
-b'rev': lambda c: c.rev(),
+b'rev': scmutil.intrev,
 b'branch': lambda c: c.branch(),
 b'desc': lambda c: c.description(),
 b'user': lambda c: c.user(),
 b'author': lambda c: c.user(),
 b'date': lambda c: c.date()[0],
-b'node': lambda c: c.node(),
+b'node': scmutil.binnode,
 }
 
 
diff --git a/tests/test-revset.t b/tests/test-revset.t
--- a/tests/test-revset.t
+++ b/tests/test-revset.t
@@ -2899,6 +2899,29 @@ test sorting by multiple keys including 
   0 b12  m111 u112 111 10800
   2 b111 m11  u12  111 3600
 
+ sort including wdir (rev/-rev has fast path):
+
+  $ hg log -r 'sort(. + wdir(), rev)' -T '{rev}\n'
+  4
+  2147483647
+  $ hg log -r 'sort(. + wdir(), -rev)' -T '{rev}\n'
+  2147483647
+  4
+
+  $ hg log -r 'sort(. + wdir(), "branch rev")' -T '{rev}\n'
+  4
+  2147483647
+  $ hg log -r 'sort(. + wdir(), "branch -rev")' -T '{rev}\n'
+  2147483647
+  4
+
+  $ hg log -r 'sort(. + wdir(), node)' -T '{node}\n'
+  ec7c1c90b589ade8603d5fb619dc6c25173a723f
+  
+  $ hg log -r 'sort(. + wdir(), -node)' -T '{node}\n'
+  
+  ec7c1c90b589ade8603d5fb619dc6c25173a723f
+
  toposort prioritises graph branches
 
   $ hg up 2
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH] revset: fix sorting key of wdir revision

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1602238750 -32400
#  Fri Oct 09 19:19:10 2020 +0900
# Node ID dab8e69ca0b9c811966d798237908f89e7459777
# Parent  613fbba0138b6a48d93617baede3bdb71b425b31
revset: fix sorting key of wdir revision

It would go wrong on Python 2, and would crash on Python 3.

diff --git a/mercurial/revset.py b/mercurial/revset.py
--- a/mercurial/revset.py
+++ b/mercurial/revset.py
@@ -2287,13 +2287,13 @@ def roots(repo, subset, x):
 
 
 _sortkeyfuncs = {
-b'rev': lambda c: c.rev(),
+b'rev': lambda c: scmutil.intrev(c),
 b'branch': lambda c: c.branch(),
 b'desc': lambda c: c.description(),
 b'user': lambda c: c.user(),
 b'author': lambda c: c.user(),
 b'date': lambda c: c.date()[0],
-b'node': lambda c: c.node(),
+b'node': lambda c: scmutil.binnode(c),
 }
 
 
diff --git a/tests/test-revset.t b/tests/test-revset.t
--- a/tests/test-revset.t
+++ b/tests/test-revset.t
@@ -2899,6 +2899,29 @@ test sorting by multiple keys including 
   0 b12  m111 u112 111 10800
   2 b111 m11  u12  111 3600
 
+ sort including wdir (rev/-rev has fast path):
+
+  $ hg log -r 'sort(. + wdir(), rev)' -T '{rev}\n'
+  4
+  2147483647
+  $ hg log -r 'sort(. + wdir(), -rev)' -T '{rev}\n'
+  2147483647
+  4
+
+  $ hg log -r 'sort(. + wdir(), "branch rev")' -T '{rev}\n'
+  4
+  2147483647
+  $ hg log -r 'sort(. + wdir(), "branch -rev")' -T '{rev}\n'
+  2147483647
+  4
+
+  $ hg log -r 'sort(. + wdir(), node)' -T '{node}\n'
+  ec7c1c90b589ade8603d5fb619dc6c25173a723f
+  
+  $ hg log -r 'sort(. + wdir(), -node)' -T '{node}\n'
+  
+  ec7c1c90b589ade8603d5fb619dc6c25173a723f
+
  toposort prioritises graph branches
 
   $ hg up 2
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] revset: add a `node` key for sorting

[Yuya Nishihara]

On Fri, 09 Oct 2020 09:11:09 +0200, Pierre-Yves David wrote:
> # HG changeset patch
> # User Pierre-Yves David 
> # Date 1602166446 -7200
> #  Thu Oct 08 16:14:06 2020 +0200
> # Node ID 187528344585900e0915965a14baf968f175fbbd
> # Parent  66cb7ad3787ca10558ef747036e5417c4f352707
> # EXP-Topic node-sorting
> # Available At https://foss.heptapod.net/octobus/mercurial-devel/
> #  hg pull https://foss.heptapod.net/octobus/mercurial-devel/ -r 
> 187528344585
> revset: add a `node` key for sorting

Queued, thanks.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 6 of 6] grep: move prep() to grepsearcher class

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1599635906 -32400
#  Wed Sep 09 16:18:26 2020 +0900
# Node ID 6026fbf740f499daabc0c97ae9b698d66ec4960a
# Parent  3bc6728f3ab93a4b2831c706052bbd6fe7090b23
grep: move prep() to grepsearcher class

diff --git a/mercurial/commands.py b/mercurial/commands.py
--- a/mercurial/commands.py
+++ b/mercurial/commands.py
@@ -45,7 +45,6 @@ from . import (
 help,
 hg,
 logcmdutil,
-match as matchmod,
 merge as mergemod,
 mergestate as mergestatemod,
 narrowspec,
@@ -3371,6 +3370,7 @@ def grep(ui, repo, pattern, *pats, **opt
 """
 opts = pycompat.byteskwargs(opts)
 diff = opts.get(b'all') or opts.get(b'diff')
+follow = opts.get(b'follow')
 if diff and opts.get(b'all_files'):
 raise error.Abort(_(b'--diff and --all-files are mutually exclusive'))
 if opts.get(b'all_files') is None and not diff:
@@ -3398,7 +3398,9 @@ def grep(ui, repo, pattern, *pats, **opt
 if opts.get(b'print0'):
 sep = eol = b'\0'
 
-searcher = grepmod.grepsearcher(ui, repo, regexp)
+searcher = grepmod.grepsearcher(
+ui, repo, regexp, all_files=all_files, diff=diff, follow=follow
+)
 
 getfile = searcher._getfile
 matches = searcher._matches
@@ -3515,58 +3517,6 @@ def grep(ui, repo, pattern, *pats, **opt
 skip = searcher._skip
 revfiles = searcher._revfiles
 found = False
-follow = opts.get(b'follow')
-
-getrenamed = searcher._getrenamed
-
-def prep(ctx, fmatch):
-rev = ctx.rev()
-pctx = ctx.p1()
-matches.setdefault(rev, {})
-if diff:
-parent = pctx.rev()
-matches.setdefault(parent, {})
-files = revfiles.setdefault(rev, [])
-if rev is None:
-# in `hg grep pattern`, 2/3 of the time is spent is spent in
-# pathauditor checks without this in mozilla-central
-contextmanager = repo.wvfs.audit.cached
-else:
-contextmanager = util.nullcontextmanager
-with contextmanager():
-# TODO: maybe better to warn missing files?
-if all_files:
-fmatch = matchmod.badmatch(fmatch, lambda f, msg: None)
-filenames = ctx.matches(fmatch)
-else:
-filenames = (f for f in ctx.files() if fmatch(f))
-for fn in filenames:
-# fn might not exist in the revision (could be a file removed 
by
-# the revision). We could check `fn not in ctx` even when rev 
is
-# None, but it's less racy to protect againt that in readfile.
-if rev is not None and fn not in ctx:
-continue
-
-copy = None
-if follow:
-copy = getrenamed(fn, rev)
-if copy:
-copies.setdefault(rev, {})[fn] = copy
-if fn in skip:
-skip.add(copy)
-if fn in skip:
-continue
-files.append(fn)
-
-if fn not in matches[rev]:
-searcher._grepbody(fn, rev, searcher._readfile(ctx, fn))
-
-if diff:
-pfn = copy or fn
-if pfn not in matches[parent] and pfn in pctx:
-searcher._grepbody(
-pfn, parent, searcher._readfile(pctx, pfn)
-)
 
 wopts = logcmdutil.walkopts(
 pats=pats,
@@ -3582,7 +3532,9 @@ def grep(ui, repo, pattern, *pats, **opt
 
 ui.pager(b'grep')
 fm = ui.formatter(b'grep', opts)
-for ctx in cmdutil.walkchangerevs(repo, revs, makefilematcher, prep):
+for ctx in cmdutil.walkchangerevs(
+repo, revs, makefilematcher, searcher._prep
+):
 rev = ctx.rev()
 parent = ctx.p1().rev()
 for fn in sorted(revfiles.get(rev, [])):
diff --git a/mercurial/grep.py b/mercurial/grep.py
--- a/mercurial/grep.py
+++ b/mercurial/grep.py
@@ -14,6 +14,7 @@ from .i18n import _
 
 from . import (
 error,
+match as matchmod,
 pycompat,
 scmutil,
 util,
@@ -80,12 +81,23 @@ def difflinestates(a, b):
 
 
 class grepsearcher(object):
-"""Search files and revisions for lines matching the given pattern"""
+"""Search files and revisions for lines matching the given pattern
 
-def __init__(self, ui, repo, regexp):
+Options:
+- all_files to search unchanged files at that revision.
+- diff to search files in the parent revision so diffs can be generated.
+- follow to skip files across copies and renames.
+"""
+
+def __init__(
+self, ui, repo, regexp, all_files=False, diff=False, follow=False
+):
 self._ui = ui
 self._repo = repo
 self._regexp = regexp
+self._all_files = all_files
+  

[PATCH 5 of 6] grep: move readfile() to grepsearcher class

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1599635079 -32400
#  Wed Sep 09 16:04:39 2020 +0900
# Node ID 3bc6728f3ab93a4b2831c706052bbd6fe7090b23
# Parent  a19bef408f16ba119ae395379df62aba2586c42a
grep: move readfile() to grepsearcher class

diff --git a/mercurial/commands.py b/mercurial/commands.py
--- a/mercurial/commands.py
+++ b/mercurial/commands.py
@@ -3519,28 +3519,6 @@ def grep(ui, repo, pattern, *pats, **opt
 
 getrenamed = searcher._getrenamed
 
-def readfile(ctx, fn):
-rev = ctx.rev()
-if rev is None:
-fctx = ctx[fn]
-try:
-return fctx.data()
-except IOError as e:
-if e.errno != errno.ENOENT:
-raise
-else:
-flog = getfile(fn)
-fnode = ctx.filenode(fn)
-try:
-return flog.read(fnode)
-except error.CensoredNodeError:
-ui.warn(
-_(
-b'cannot search in censored file: 
%(filename)s:%(revnum)s\n'
-)
-% {b'filename': fn, b'revnum': pycompat.bytestr(rev),}
-)
-
 def prep(ctx, fmatch):
 rev = ctx.rev()
 pctx = ctx.p1()
@@ -3581,12 +3559,14 @@ def grep(ui, repo, pattern, *pats, **opt
 files.append(fn)
 
 if fn not in matches[rev]:
-searcher._grepbody(fn, rev, readfile(ctx, fn))
+searcher._grepbody(fn, rev, searcher._readfile(ctx, fn))
 
 if diff:
 pfn = copy or fn
 if pfn not in matches[parent] and pfn in pctx:
-searcher._grepbody(pfn, parent, readfile(pctx, pfn))
+searcher._grepbody(
+pfn, parent, searcher._readfile(pctx, pfn)
+)
 
 wopts = logcmdutil.walkopts(
 pats=pats,
diff --git a/mercurial/grep.py b/mercurial/grep.py
--- a/mercurial/grep.py
+++ b/mercurial/grep.py
@@ -8,8 +8,12 @@
 from __future__ import absolute_import
 
 import difflib
+import errno
+
+from .i18n import _
 
 from . import (
+error,
 pycompat,
 scmutil,
 util,
@@ -100,3 +104,26 @@ class grepsearcher(object):
 for lnum, cstart, cend, line in matchlines(body, self._regexp):
 s = linestate(line, lnum, cstart, cend)
 m.append(s)
+
+def _readfile(self, ctx, fn):
+rev = ctx.rev()
+if rev is None:
+fctx = ctx[fn]
+try:
+return fctx.data()
+except IOError as e:
+if e.errno != errno.ENOENT:
+raise
+else:
+flog = self._getfile(fn)
+fnode = ctx.filenode(fn)
+try:
+return flog.read(fnode)
+except error.CensoredNodeError:
+self._ui.warn(
+_(
+b'cannot search in censored file: '
+b'%(filename)s:%(revnum)s\n'
+)
+% {b'filename': fn, b'revnum': pycompat.bytestr(rev)}
+)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 4 of 6] grep: move getbody() to grepsearcher class

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1599634803 -32400
#  Wed Sep 09 16:00:03 2020 +0900
# Node ID a19bef408f16ba119ae395379df62aba2586c42a
# Parent  55130738756a9836d6d8a8da75e7bca808a327cb
grep: move getbody() to grepsearcher class

diff --git a/mercurial/commands.py b/mercurial/commands.py
--- a/mercurial/commands.py
+++ b/mercurial/commands.py
@@ -3404,16 +3404,6 @@ def grep(ui, repo, pattern, *pats, **opt
 matches = searcher._matches
 copies = searcher._copies
 
-def grepbody(fn, rev, body):
-matches[rev].setdefault(fn, [])
-m = matches[rev][fn]
-if body is None:
-return
-
-for lnum, cstart, cend, line in grepmod.matchlines(body, regexp):
-s = grepmod.linestate(line, lnum, cstart, cend)
-m.append(s)
-
 uipathfn = scmutil.getuipathfn(repo)
 
 def display(fm, fn, ctx, pstates, states):
@@ -3591,12 +3581,12 @@ def grep(ui, repo, pattern, *pats, **opt
 files.append(fn)
 
 if fn not in matches[rev]:
-grepbody(fn, rev, readfile(ctx, fn))
+searcher._grepbody(fn, rev, readfile(ctx, fn))
 
 if diff:
 pfn = copy or fn
 if pfn not in matches[parent] and pfn in pctx:
-grepbody(pfn, parent, readfile(pctx, pfn))
+searcher._grepbody(pfn, parent, readfile(pctx, pfn))
 
 wopts = logcmdutil.walkopts(
 pats=pats,
diff --git a/mercurial/grep.py b/mercurial/grep.py
--- a/mercurial/grep.py
+++ b/mercurial/grep.py
@@ -90,3 +90,13 @@ class grepsearcher(object):
 self._copies = {}
 self._skip = set()
 self._revfiles = {}
+
+def _grepbody(self, fn, rev, body):
+self._matches[rev].setdefault(fn, [])
+m = self._matches[rev][fn]
+if body is None:
+return
+
+for lnum, cstart, cend, line in matchlines(body, self._regexp):
+s = linestate(line, lnum, cstart, cend)
+m.append(s)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 3 of 6] grep: add stub class that maintains cache and states of grep operation

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1599634600 -32400
#  Wed Sep 09 15:56:40 2020 +0900
# Node ID 55130738756a9836d6d8a8da75e7bca808a327cb
# Parent  622430a7bd70e13c752a5e75102c3555c99f98d8
grep: add stub class that maintains cache and states of grep operation

Prepares for extracting stateful functions from commands.grep().

diff --git a/mercurial/commands.py b/mercurial/commands.py
--- a/mercurial/commands.py
+++ b/mercurial/commands.py
@@ -3398,9 +3398,11 @@ def grep(ui, repo, pattern, *pats, **opt
 if opts.get(b'print0'):
 sep = eol = b'\0'
 
-getfile = util.lrucachefunc(repo.file)
-matches = {}
-copies = {}
+searcher = grepmod.grepsearcher(ui, repo, regexp)
+
+getfile = searcher._getfile
+matches = searcher._matches
+copies = searcher._copies
 
 def grepbody(fn, rev, body):
 matches[rev].setdefault(fn, [])
@@ -3520,12 +3522,12 @@ def grep(ui, repo, pattern, *pats, **opt
 fm.data(matched=False)
 fm.end()
 
-skip = set()
-revfiles = {}
+skip = searcher._skip
+revfiles = searcher._revfiles
 found = False
 follow = opts.get(b'follow')
 
-getrenamed = scmutil.getrenamedfn(repo)
+getrenamed = searcher._getrenamed
 
 def readfile(ctx, fn):
 rev = ctx.rev()
diff --git a/mercurial/grep.py b/mercurial/grep.py
--- a/mercurial/grep.py
+++ b/mercurial/grep.py
@@ -9,7 +9,11 @@ from __future__ import absolute_import
 
 import difflib
 
-from . import pycompat
+from . import (
+pycompat,
+scmutil,
+util,
+)
 
 
 def matchlines(body, regexp):
@@ -69,3 +73,20 @@ def difflinestates(a, b):
 yield (b'-', a[i])
 for i in pycompat.xrange(blo, bhi):
 yield (b'+', b[i])
+
+
+class grepsearcher(object):
+"""Search files and revisions for lines matching the given pattern"""
+
+def __init__(self, ui, repo, regexp):
+self._ui = ui
+self._repo = repo
+self._regexp = regexp
+
+self._getfile = util.lrucachefunc(repo.file)
+self._getrenamed = scmutil.getrenamedfn(repo)
+
+self._matches = {}
+self._copies = {}
+self._skip = set()
+self._revfiles = {}
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 6] grep: move match and diff logic to new module

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1599632629 -32400
#  Wed Sep 09 15:23:49 2020 +0900
# Node ID 622430a7bd70e13c752a5e75102c3555c99f98d8
# Parent  44729154f2ab3152224b89bfc565a70f832ebc3a
grep: move match and diff logic to new module

commands.grep() has lots of functions and classes. Let's split it into
reusable components so we can leverage them to implement a revset predicate
for 'hg grep --diff'. I want to do 'hg log -r "diff(pattern)"'.

diff --git a/mercurial/commands.py b/mercurial/commands.py
--- a/mercurial/commands.py
+++ b/mercurial/commands.py
@@ -7,7 +7,6 @@
 
 from __future__ import absolute_import
 
-import difflib
 import errno
 import os
 import re
@@ -41,6 +40,7 @@ from . import (
 filemerge,
 formatter,
 graphmod,
+grep as grepmod,
 hbisect,
 help,
 hg,
@@ -3399,48 +3399,6 @@ def grep(ui, repo, pattern, *pats, **opt
 sep = eol = b'\0'
 
 getfile = util.lrucachefunc(repo.file)
-
-def matchlines(body, regexp):
-begin = 0
-linenum = 0
-while begin < len(body):
-match = regexp.search(body, begin)
-if not match:
-break
-mstart, mend = match.span()
-linenum += body.count(b'\n', begin, mstart) + 1
-lstart = body.rfind(b'\n', begin, mstart) + 1 or begin
-begin = body.find(b'\n', mend) + 1 or len(body) + 1
-lend = begin - 1
-yield linenum, mstart - lstart, mend - lstart, body[lstart:lend]
-
-class linestate(object):
-def __init__(self, line, linenum, colstart, colend):
-self.line = line
-self.linenum = linenum
-self.colstart = colstart
-self.colend = colend
-
-def __hash__(self):
-return hash(self.line)
-
-def __eq__(self, other):
-return self.line == other.line
-
-def findpos(self, regexp):
-"""Iterate all (start, end) indices of matches"""
-yield self.colstart, self.colend
-p = self.colend
-while p < len(self.line):
-m = regexp.search(self.line, p)
-if not m:
-break
-if m.end() == p:
-p += 1
-else:
-yield m.span()
-p = m.end()
-
 matches = {}
 copies = {}
 
@@ -3450,25 +3408,10 @@ def grep(ui, repo, pattern, *pats, **opt
 if body is None:
 return
 
-for lnum, cstart, cend, line in matchlines(body, regexp):
-s = linestate(line, lnum, cstart, cend)
+for lnum, cstart, cend, line in grepmod.matchlines(body, regexp):
+s = grepmod.linestate(line, lnum, cstart, cend)
 m.append(s)
 
-def difflinestates(a, b):
-sm = difflib.SequenceMatcher(None, a, b)
-for tag, alo, ahi, blo, bhi in sm.get_opcodes():
-if tag == 'insert':
-for i in pycompat.xrange(blo, bhi):
-yield (b'+', b[i])
-elif tag == 'delete':
-for i in pycompat.xrange(alo, ahi):
-yield (b'-', a[i])
-elif tag == 'replace':
-for i in pycompat.xrange(alo, ahi):
-yield (b'-', a[i])
-for i in pycompat.xrange(blo, bhi):
-yield (b'+', b[i])
-
 uipathfn = scmutil.getuipathfn(repo)
 
 def display(fm, fn, ctx, pstates, states):
@@ -3493,7 +3436,7 @@ def grep(ui, repo, pattern, *pats, **opt
 
 fieldnamemap = {b'linenumber': b'lineno'}
 if diff:
-iter = difflinestates(pstates, states)
+iter = grepmod.difflinestates(pstates, states)
 else:
 iter = [(b'', l) for l in states]
 for change, l in iter:
diff --git a/mercurial/grep.py b/mercurial/grep.py
new file mode 100644
--- /dev/null
+++ b/mercurial/grep.py
@@ -0,0 +1,71 @@
+# grep.py - logic for history walk and grep
+#
+# Copyright 2005-2007 Matt Mackall 
+#
+# This software may be used and distributed according to the terms of the
+# GNU General Public License version 2 or any later version.
+
+from __future__ import absolute_import
+
+import difflib
+
+from . import pycompat
+
+
+def matchlines(body, regexp):
+begin = 0
+linenum = 0
+while begin < len(body):
+match = regexp.search(body, begin)
+if not match:
+break
+mstart, mend = match.span()
+linenum += body.count(b'\n', begin, mstart) + 1
+lstart = body.rfind(b'\n', begin, mstart) + 1 or begin
+begin = body.find(b'\n', mend) + 1 or len(body) + 1
+lend = begin - 1
+yield linenum, mstart - lstart, mend - lstart, body[lstart:lend]
+
+
+class linestate(object):
+def __init__(self, line, linenum, colstart, colend):
+self.line = line
+self.linenum = linenum
+

[PATCH 1 of 6] grep: explicitly pass regexp to closure functions

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1599632246 -32400
#  Wed Sep 09 15:17:26 2020 +0900
# Node ID 44729154f2ab3152224b89bfc565a70f832ebc3a
# Parent  d1759b2e18889eb2f58e9d348f45d76af047ba08
grep: explicitly pass regexp to closure functions

These functions will be extracted to new module.

diff --git a/mercurial/commands.py b/mercurial/commands.py
--- a/mercurial/commands.py
+++ b/mercurial/commands.py
@@ -3400,7 +3400,7 @@ def grep(ui, repo, pattern, *pats, **opt
 
 getfile = util.lrucachefunc(repo.file)
 
-def matchlines(body):
+def matchlines(body, regexp):
 begin = 0
 linenum = 0
 while begin < len(body):
@@ -3427,7 +3427,7 @@ def grep(ui, repo, pattern, *pats, **opt
 def __eq__(self, other):
 return self.line == other.line
 
-def findpos(self):
+def findpos(self, regexp):
 """Iterate all (start, end) indices of matches"""
 yield self.colstart, self.colend
 p = self.colend
@@ -3450,7 +3450,7 @@ def grep(ui, repo, pattern, *pats, **opt
 if body is None:
 return
 
-for lnum, cstart, cend, line in matchlines(body):
+for lnum, cstart, cend, line in matchlines(body, regexp):
 s = linestate(line, lnum, cstart, cend)
 m.append(s)
 
@@ -3562,7 +3562,7 @@ def grep(ui, repo, pattern, *pats, **opt
 
 def displaymatches(fm, l):
 p = 0
-for s, e in l.findpos():
+for s, e in l.findpos(regexp):
 if p < s:
 fm.startitem()
 fm.write(b'text', b'%s', l.line[p:s])
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 6 of 6] churn: leverage logcmdutil to filter revisions by --date

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1600659377 -32400
#  Mon Sep 21 12:36:17 2020 +0900
# Node ID 8ecb3d07b901cb14a6303008f053840fa47847ac
# Parent  c02ae0d9e1126c182a09ead972fce7f4a6fedad9
churn: leverage logcmdutil to filter revisions by --date

diff --git a/hgext/churn.py b/hgext/churn.py
--- a/hgext/churn.py
+++ b/hgext/churn.py
@@ -24,7 +24,6 @@ from mercurial import (
 pycompat,
 registrar,
 )
-from mercurial.utils import dateutil
 
 cmdtable = {}
 command = registrar.command(cmdtable)
@@ -71,15 +70,9 @@ def countrate(ui, repo, amap, *pats, **o
 _(b'analyzing'), unit=_(b'revisions'), total=len(repo)
 )
 rate = {}
-df = False
-if opts.get(b'date'):
-df = dateutil.matchdate(opts[b'date'])
 
 def prep(ctx, fmatch):
 rev = ctx.rev()
-if df and not df(ctx.date()[0]):  # doesn't match date format
-return
-
 key = getkey(ctx).strip()
 key = amap.get(key, key)  # alias remap
 if opts.get(b'changesets'):
@@ -100,6 +93,7 @@ def countrate(ui, repo, amap, *pats, **o
 pats=pats,
 opts=opts,
 revspec=opts[b'rev'],
+date=opts[b'date'],
 include_pats=opts[b'include'],
 exclude_pats=opts[b'exclude'],
 )
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 5 of 6] cmdutil: remove remainder of old walkchangerevs() implementation

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1599731851 -32400
#  Thu Sep 10 18:57:31 2020 +0900
# Node ID c02ae0d9e1126c182a09ead972fce7f4a6fedad9
# Parent  d399d4ab37adc58b51e15c4f829f7e7b2a150ac6
cmdutil: remove remainder of old walkchangerevs() implementation

diff --git a/hgext/remotefilelog/__init__.py b/hgext/remotefilelog/__init__.py
--- a/hgext/remotefilelog/__init__.py
+++ b/hgext/remotefilelog/__init__.py
@@ -136,7 +136,6 @@ from mercurial.pycompat import open
 from mercurial import (
 changegroup,
 changelog,
-cmdutil,
 commands,
 configitems,
 context,
@@ -341,7 +340,6 @@ def uisetup(ui):
 extensions.wrapfunction(scmutil, b'getrenamedfn', getrenamedfn)
 extensions.wrapfunction(revset, b'filelog', filelogrevset)
 revset.symbols[b'filelog'] = revset.filelog
-extensions.wrapfunction(cmdutil, b'walkfilerevs', walkfilerevs)
 
 
 def cloneshallow(orig, ui, repo, *args, **opts):
@@ -782,40 +780,6 @@ def getrenamedfn(orig, repo, endrev=None
 return getrenamed
 
 
-def walkfilerevs(orig, repo, match, follow, revs, fncache):
-if not isenabled(repo):
-return orig(repo, match, follow, revs, fncache)
-
-# remotefilelog's can't be walked in rev order, so throw.
-# The caller will see the exception and walk the commit tree instead.
-if not follow:
-raise cmdutil.FileWalkError(b"Cannot walk via filelog")
-
-wanted = set()
-minrev, maxrev = min(revs), max(revs)
-
-pctx = repo[b'.']
-for filename in match.files():
-if filename not in pctx:
-raise error.Abort(
-_(b'cannot follow file not in parent revision: "%s"') % 
filename
-)
-fctx = pctx[filename]
-
-linkrev = fctx.linkrev()
-if linkrev >= minrev and linkrev <= maxrev:
-fncache.setdefault(linkrev, []).append(filename)
-wanted.add(linkrev)
-
-for ancestor in fctx.ancestors():
-linkrev = ancestor.linkrev()
-if linkrev >= minrev and linkrev <= maxrev:
-fncache.setdefault(linkrev, []).append(ancestor.path())
-wanted.add(linkrev)
-
-return wanted
-
-
 def filelogrevset(orig, repo, subset, x):
 """``filelog(pattern)``
 Changesets connected to the specified filelog.
diff --git a/mercurial/cmdutil.py b/mercurial/cmdutil.py
--- a/mercurial/cmdutil.py
+++ b/mercurial/cmdutil.py
@@ -16,7 +16,6 @@ from .i18n import _
 from .node import (
 hex,
 nullid,
-nullrev,
 short,
 )
 from .pycompat import (
@@ -49,7 +48,6 @@ from . import (
 revlog,
 rewriteutil,
 scmutil,
-smartset,
 state as statemod,
 subrepoutil,
 templatekw,
@@ -2249,185 +2247,6 @@ def increasingwindows(windowsize=8, size
 windowsize *= 2
 
 
-def _walkrevs(repo, opts):
-# Default --rev value depends on --follow but --follow behavior
-# depends on revisions resolved from --rev...
-follow = opts.get(b'follow') or opts.get(b'follow_first')
-revspec = opts.get(b'rev')
-if follow and revspec:
-revs = scmutil.revrange(repo, revspec)
-revs = repo.revs(b'reverse(::%ld)', revs)
-elif revspec:
-revs = scmutil.revrange(repo, revspec)
-elif follow and repo.dirstate.p1() == nullid:
-revs = smartset.baseset()
-elif follow:
-revs = repo.revs(b'reverse(:.)')
-else:
-revs = smartset.spanset(repo)
-revs.reverse()
-return revs
-
-
-class FileWalkError(Exception):
-pass
-
-
-def walkfilerevs(repo, match, follow, revs, fncache):
-'''Walks the file history for the matched files.
-
-Returns the changeset revs that are involved in the file history.
-
-Throws FileWalkError if the file history can't be walked using
-filelogs alone.
-'''
-wanted = set()
-copies = []
-minrev, maxrev = min(revs), max(revs)
-
-def filerevs(filelog, last):
-"""
-Only files, no patterns.  Check the history of each file.
-
-Examines filelog entries within minrev, maxrev linkrev range
-Returns an iterator yielding (linkrev, parentlinkrevs, copied)
-tuples in backwards order
-"""
-cl_count = len(repo)
-revs = []
-for j in pycompat.xrange(0, last + 1):
-linkrev = filelog.linkrev(j)
-if linkrev < minrev:
-continue
-# only yield rev for which we have the changelog, it can
-# happen while doing "hg log" during a pull or commit
-if linkrev >= cl_count:
-break
-
-parentlinkrevs = []
-for p in filelog.parentrevs(j):
-if p != nullrev:
-parentlinkrevs.append(filelog.linkrev(p))
-n = filelog.node(j)
-revs.append(
-(linkrev, parentlinkrevs, follow

[PATCH 4 of 6] cmdutil: rewrite walkchangerevs() by using logcmdutil functions

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1599728503 -32400
#  Thu Sep 10 18:01:43 2020 +0900
# Node ID d399d4ab37adc58b51e15c4f829f7e7b2a150ac6
# Parent  8eec1c384c992f9ce343a4245c2abe5f371b9bcd
cmdutil: rewrite walkchangerevs() by using logcmdutil functions

cmdutil.walkchangerevs() now takes (revs, makefilematcher) in place of
(match, opts), and only provides the "windowing" functionality. Unused
classes and functions will be removed by the next patch.

"hg grep --follow" (--all-files) is still broken since there is no logic
to follow copies while traversing changelog, but at least, it does follow
the DAG.

diff --git a/hgext/churn.py b/hgext/churn.py
--- a/hgext/churn.py
+++ b/hgext/churn.py
@@ -23,7 +23,6 @@ from mercurial import (
 patch,
 pycompat,
 registrar,
-scmutil,
 )
 from mercurial.utils import dateutil
 
@@ -76,8 +75,6 @@ def countrate(ui, repo, amap, *pats, **o
 if opts.get(b'date'):
 df = dateutil.matchdate(opts[b'date'])
 
-m = scmutil.match(repo[None], pats, opts)
-
 def prep(ctx, fmatch):
 rev = ctx.rev()
 if df and not df(ctx.date()[0]):  # doesn't match date format
@@ -99,7 +96,15 @@ def countrate(ui, repo, amap, *pats, **o
 
 progress.increment()
 
-for ctx in cmdutil.walkchangerevs(repo, m, opts, prep):
+wopts = logcmdutil.walkopts(
+pats=pats,
+opts=opts,
+revspec=opts[b'rev'],
+include_pats=opts[b'include'],
+exclude_pats=opts[b'exclude'],
+)
+revs, makefilematcher = logcmdutil.makewalker(repo, wopts)
+for ctx in cmdutil.walkchangerevs(repo, revs, makefilematcher, prep):
 continue
 
 progress.complete()
diff --git a/mercurial/cmdutil.py b/mercurial/cmdutil.py
--- a/mercurial/cmdutil.py
+++ b/mercurial/cmdutil.py
@@ -2428,8 +2428,8 @@ class _followfilter(object):
 return False
 
 
-def walkchangerevs(repo, match, opts, prepare):
-'''Iterate over files and the revs in which they changed.
+def walkchangerevs(repo, revs, makefilematcher, prepare):
+'''Iterate over files and the revs in a "windowed" way.
 
 Callers most commonly need to iterate backwards over the history
 in which they are interested. Doing so has awful (quadratic-looking)
@@ -2443,107 +2443,11 @@ def walkchangerevs(repo, match, opts, pr
 yielding each context, the iterator will first call the prepare
 function on each context in the window in forward order.'''
 
-allfiles = opts.get(b'all_files')
-follow = opts.get(b'follow') or opts.get(b'follow_first')
-revs = _walkrevs(repo, opts)
 if not revs:
 return []
-wanted = set()
-slowpath = match.anypats() or (not match.always() and opts.get(b'removed'))
-fncache = {}
 change = repo.__getitem__
 
-# First step is to fill wanted, the set of revisions that we want to yield.
-# When it does not induce extra cost, we also fill fncache for revisions in
-# wanted: a cache of filenames that were changed (ctx.files()) and that
-# match the file filtering conditions.
-
-if match.always() or allfiles:
-# No files, no patterns.  Display all revs.
-wanted = revs
-elif not slowpath:
-# We only have to read through the filelog to find wanted revisions
-
-try:
-wanted = walkfilerevs(repo, match, follow, revs, fncache)
-except FileWalkError:
-slowpath = True
-
-# We decided to fall back to the slowpath because at least one
-# of the paths was not a file. Check to see if at least one of them
-# existed in history, otherwise simply return
-for path in match.files():
-if path == b'.' or path in repo.store:
-break
-else:
-return []
-
-if slowpath:
-# We have to read the changelog to match filenames against
-# changed files
-
-if follow:
-raise error.Abort(
-_(b'can only follow copies/renames for explicit filenames')
-)
-
-# The slow path checks files modified in every changeset.
-# This is really slow on large repos, so compute the set lazily.
-class lazywantedset(object):
-def __init__(self):
-self.set = set()
-self.revs = set(revs)
-
-# No need to worry about locality here because it will be accessed
-# in the same order as the increasing window below.
-def __contains__(self, value):
-if value in self.set:
-return True
-elif not value in self.revs:
-return False
-else:
-self.revs.discard(value)
-ctx = change(value)
-if allfiles:
-matches = list(ctx.manifest().walk(match))
-else:
-

[PATCH 3 of 6] grep: filter target files by matcher

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1599725643 -32400
#  Thu Sep 10 17:14:03 2020 +0900
# Node ID 8eec1c384c992f9ce343a4245c2abe5f371b9bcd
# Parent  c0d5362379c3b13d6d644260d395bad169888924
grep: filter target files by matcher

Prepares for the migration to logcmdutil's logic, where cmdutil.walkchangerevs()
won't always build an exact set of paths to be scanned.

diff --git a/mercurial/commands.py b/mercurial/commands.py
--- a/mercurial/commands.py
+++ b/mercurial/commands.py
@@ -45,6 +45,7 @@ from . import (
 help,
 hg,
 logcmdutil,
+match as matchmod,
 merge as mergemod,
 mergestate as mergestatemod,
 narrowspec,
@@ -3621,8 +3622,13 @@ def grep(ui, repo, pattern, *pats, **opt
 else:
 contextmanager = util.nullcontextmanager
 with contextmanager():
-assert fmatch.isexact()
-for fn in fmatch.files():
+# TODO: maybe better to warn missing files?
+if all_files:
+fmatch = matchmod.badmatch(fmatch, lambda f, msg: None)
+filenames = ctx.matches(fmatch)
+else:
+filenames = (f for f in ctx.files() if fmatch(f))
+for fn in filenames:
 # fn might not exist in the revision (could be a file removed 
by
 # the revision). We could check `fn not in ctx` even when rev 
is
 # None, but it's less racy to protect againt that in readfile.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 1 of 6] grep: add option for logcmdutil.makewalker() to not filter revs by file pats

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1599730801 -32400
#  Thu Sep 10 18:40:01 2020 +0900
# Node ID 14815a48ad0acf0e4e4da658c4cca0bee459351d
# Parent  ede4a1bf14bdd101e0760db8b8c7e8d65865050e
grep: add option for logcmdutil.makewalker() to not filter revs by file pats

This is needed to implement "grep --all-files".

diff --git a/mercurial/logcmdutil.py b/mercurial/logcmdutil.py
--- a/mercurial/logcmdutil.py
+++ b/mercurial/logcmdutil.py
@@ -711,6 +711,10 @@ class walkopts(object):
 # include revisions where files were removed
 force_changelog_traversal = attr.ib(default=False)  # type: bool
 
+# filter revisions by file patterns, which should be disabled only if
+# you want to include revisions where files were unmodified
+filter_revisions_by_pats = attr.ib(default=True)  # type: bool
+
 # sort revisions prior to traversal: 'desc', 'topo', or None
 sort_revisions = attr.ib(default=None)  # type: Optional[bytes]
 
@@ -902,7 +906,7 @@ def _makerevset(repo, wopts, slowpath):
 b'user': wopts.users,
 }
 
-if slowpath:
+if wopts.filter_revisions_by_pats and slowpath:
 # See walkchangerevs() slow path.
 #
 # pats/include/exclude cannot be represented as separate
@@ -919,7 +923,7 @@ def _makerevset(repo, wopts, slowpath):
 for p in wopts.exclude_pats:
 matchargs.append(b'x:' + p)
 opts[b'_matchfiles'] = matchargs
-elif not wopts.follow:
+elif wopts.filter_revisions_by_pats and not wopts.follow:
 opts[b'_patslog'] = list(wopts.pats)
 
 expr = []
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 6] cmdutil: make walkchangerevs() call prepare with matcher instead of filenames

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1599722088 -32400
#  Thu Sep 10 16:14:48 2020 +0900
# Node ID c0d5362379c3b13d6d644260d395bad169888924
# Parent  14815a48ad0acf0e4e4da658c4cca0bee459351d
cmdutil: make walkchangerevs() call prepare with matcher instead of filenames

Prepares for migrating walkchangerevs() to logcmdutil's logic, which provides
matcher-based interface.

diff --git a/hgext/churn.py b/hgext/churn.py
--- a/hgext/churn.py
+++ b/hgext/churn.py
@@ -36,9 +36,8 @@ command = registrar.command(cmdtable)
 testedwith = b'ships-with-hg-core'
 
 
-def changedlines(ui, repo, ctx1, ctx2, fns):
+def changedlines(ui, repo, ctx1, ctx2, fmatch):
 added, removed = 0, 0
-fmatch = scmutil.matchfiles(repo, fns)
 diff = b''.join(patch.diff(repo, ctx1.node(), ctx2.node(), fmatch))
 for l in diff.split(b'\n'):
 if l.startswith(b"+") and not l.startswith(b"+++ "):
@@ -79,7 +78,7 @@ def countrate(ui, repo, amap, *pats, **o
 
 m = scmutil.match(repo[None], pats, opts)
 
-def prep(ctx, fns):
+def prep(ctx, fmatch):
 rev = ctx.rev()
 if df and not df(ctx.date()[0]):  # doesn't match date format
 return
@@ -95,7 +94,7 @@ def countrate(ui, repo, amap, *pats, **o
 return
 
 ctx1 = parents[0]
-lines = changedlines(ui, repo, ctx1, ctx, fns)
+lines = changedlines(ui, repo, ctx1, ctx, fmatch)
 rate[key] = [r + l for r, l in zip(rate.get(key, (0, 0)), lines)]
 
 progress.increment()
diff --git a/mercurial/cmdutil.py b/mercurial/cmdutil.py
--- a/mercurial/cmdutil.py
+++ b/mercurial/cmdutil.py
@@ -2574,7 +2574,7 @@ def walkchangerevs(repo, match, opts, pr
 yield f
 
 fns = fns_generator()
-prepare(ctx, fns)
+prepare(ctx, scmutil.matchfiles(repo, fns))
 for rev in nrevs:
 yield change(rev)
 
diff --git a/mercurial/commands.py b/mercurial/commands.py
--- a/mercurial/commands.py
+++ b/mercurial/commands.py
@@ -3606,7 +3606,7 @@ def grep(ui, repo, pattern, *pats, **opt
 % {b'filename': fn, b'revnum': pycompat.bytestr(rev),}
 )
 
-def prep(ctx, fns):
+def prep(ctx, fmatch):
 rev = ctx.rev()
 pctx = ctx.p1()
 matches.setdefault(rev, {})
@@ -3621,7 +3621,8 @@ def grep(ui, repo, pattern, *pats, **opt
 else:
 contextmanager = util.nullcontextmanager
 with contextmanager():
-for fn in fns:
+assert fmatch.isexact()
+for fn in fmatch.files():
 # fn might not exist in the revision (could be a file removed 
by
 # the revision). We could check `fn not in ctx` even when rev 
is
 # None, but it's less racy to protect againt that in readfile.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] test: try to unflaky test-profile.t

[Yuya Nishihara]

On Fri, 02 Oct 2020 13:57:40 +0200, Pierre-Yves David wrote:
> # HG changeset patch
> # User Pierre-Yves David 
> # Date 1601627362 -7200
> #  Fri Oct 02 10:29:22 2020 +0200
> # Node ID 39e1477aa4d24bb8933ca571c797a257d5e270b8
> # Parent  d5407b2e7689c0526ff24bb1434818b8aa9b4fc4
> # EXP-Topic flaky-profile
> # Available At https://foss.heptapod.net/octobus/mercurial-devel/
> #  hg pull https://foss.heptapod.net/octobus/mercurial-devel/ -r 
> 39e1477aa4d2
> test: try to unflaky test-profile.t

Queued, thanks.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] merge: check for conflicting actions irrespective of length of bids

[Yuya Nishihara]

On Fri, 02 Oct 2020 19:12:52 +0530, Pulkit Goyal wrote:
> # HG changeset patch
> # User Pulkit Goyal <7895pul...@gmail.com>
> # Date 1601644239 -19800
> #  Fri Oct 02 18:40:39 2020 +0530
> # Node ID 57c951dae51f16f774d4d3e0a41ae1c4b44ad38c
> # Parent  e8078af6af3004f7abe94bc6e832bd688df11432
> # EXP-Topic merge-newnode-final
> merge: check for conflicting actions irrespective of length of bids

Queued, thanks.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel