date:20170228

Re: [PATCH 1 of 2 V2] manifest: allow specifying the revlog filename

2017-02-28 Thread Martin von Zweigbergk via Mercurial-devel

On Tue, Feb 28, 2017 at 11:44 AM, Durham Goode  wrote:
> # HG changeset patch
> # User Durham Goode 
> # Date 1488304382 28800
> #  Tue Feb 28 09:53:02 2017 -0800
> # Node ID 0a5e6ad0a25c348384bb6c14a51057b3587e2857
> # Parent  88203f26ea57627cabd7cf9c4f7843661d6c43ae
> manifest: allow specifying the revlog filename
>
> Previously we had hardcoded the manifest filename to be 00manifest.i. In our
> external treemanifest extension, we want to allow writing a treemanifest side 
> by
> side with a flat manifest, so we need to be able to store the root revisions 
> at
> a different location (in our extension we use 00manifesttree.i).
>
> This patches moves the revlog name to a parameter so we can adjust it.
>
> diff --git a/mercurial/manifest.py b/mercurial/manifest.py
> --- a/mercurial/manifest.py
> +++ b/mercurial/manifest.py
> @@ -1132,7 +1132,13 @@ class manifestrevlog(revlog.revlog):
>  '''A revlog that stores manifest texts. This is responsible for caching 
> the
>  full-text manifest contents.
>  '''
> -def __init__(self, opener, dir='', dirlogcache=None):
> +def __init__(self, opener, dir='', dirlogcache=None,
> + indexfile='00manifest.i'):
> +"""Constructs a new manifest revlog
> +
> +`indexfile` - used by extensions to have two manifests at once, like
> +when transitioning between flatmanifeset and treemanifests.
> +"""
>  # During normal operations, we expect to deal with not more than four
>  # revs at a time (such as during commit --amend). When rebasing large
>  # stacks of commits, the number can go up, hence the config knob 
> below.
> @@ -1150,12 +1156,11 @@ class manifestrevlog(revlog.revlog):
>
>  self._fulltextcache = util.lrucachedict(cachesize)
>
> -indexfile = "00manifest.i"
>  if dir:
>  assert self._treeondisk, 'opts is %r' % opts
>  if not dir.endswith('/'):
>  dir = dir + '/'
> -indexfile = "meta/" + dir + "00manifest.i"
> +indexfile = "meta/" + dir + indexfile
>  self._dir = dir
>  # The dirlogcache is kept on the root manifest log
>  if dir:

What I meant was for indexfile==None to be the default. I think it's
easier to follow if indexfile being set means that that's the exact
name to use. It also gives more flexibility to put the subdirectory
files in a different place. The patch would then look as follows. I
think that means your extension will pass None where it would pass
"00manifest.i" with your version above, so I assume it's similarly
simple. What do you think?

diff -r 0bb3089fe735 mercurial/manifest.py
--- a/mercurial/manifest.py Sat Feb 25 14:09:55 2017 +0900
+++ b/mercurial/manifest.py Tue Feb 28 22:17:28 2017 -0800
@@ -1132,7 +1132,12 @@ class manifestrevlog(revlog.revlog):
 '''A revlog that stores manifest texts. This is responsible for caching the
 full-text manifest contents.
 '''
-def __init__(self, opener, dir='', dirlogcache=None):
+def __init__(self, opener, dir='', dirlogcache=None, indexfile=None):
+"""Constructs a new manifest revlog
+
+`indexfile` - used by extensions to have two manifests at once, like
+when transitioning between flatmanifeset and treemanifests.
+"""
 # During normal operations, we expect to deal with not more than four
 # revs at a time (such as during commit --amend). When rebasing large
 # stacks of commits, the number can go up, hence the config knob below.
@@ -1150,12 +1155,15 @@ class manifestrevlog(revlog.revlog):

 self._fulltextcache = util.lrucachedict(cachesize)

-indexfile = "00manifest.i"
 if dir:
 assert self._treeondisk, 'opts is %r' % opts
 if not dir.endswith('/'):
 dir = dir + '/'
-indexfile = "meta/" + dir + "00manifest.i"
+
+if indexfile is None:
+indexfile = "00manifest.i"
+if dir:
+indexfile = "meta/" + dir + "00manifest.i"
 self._dir = dir
 # The dirlogcache is kept on the root manifest log
 if dir:
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] internals: add some brief documentation about censor

2017-02-28 Thread Matt Harbison


On Mon, 27 Feb 2017 18:11:15 -0500, Augie Fackler  wrote:


# HG changeset patch
# User Augie Fackler 
# Date 1485220644 18000
#  Mon Jan 23 20:17:24 2017 -0500
# Node ID 065219350c9ce3c31a35d13fe97be0673e321e09
# Parent  3f8f53190d6afed0aca6c6527236edad28ce785c
internals: add some brief documentation about censor


This is needed to keep test-install.t happy:

diff --git a/contrib/wix/help.wxs b/contrib/wix/help.wxs
--- a/contrib/wix/help.wxs
+++ b/contrib/wix/help.wxs
@@ -38,6 +38,7 @@
 
   Guid="$(var.help.internals.guid)" Win64='$(var.IsX64)'>
 KeyPath="yes" />

+
 />
 />

 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] ui: remove superfluous indent in _write()

2017-02-28 Thread Martin von Zweigbergk via Mercurial-devel

On Fri, Feb 24, 2017 at 9:58 PM, Yuya Nishihara  wrote:
> # HG changeset patch
> # User Yuya Nishihara 
> # Date 1487999395 -32400
> #  Sat Feb 25 14:09:55 2017 +0900
> # Node ID 18c067150b3c2b55da2a72c2eec55c65de12523e
> # Parent  b4cb86ab4c719eb615a4308eafd8b1386a511eeb
> ui: remove superfluous indent in _write()

Queued, thanks.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 6 of 8 V2] hgweb: link to diff with linerange in filelog template

2017-02-28 Thread Gregory Szorc

On Sat, Feb 25, 2017 at 1:06 AM, Denis Laxalde  wrote:

> # HG changeset patch
> # User Denis Laxalde 
> # Date 1484838729 -3600
> #  Thu Jan 19 16:12:09 2017 +0100
> # Node ID 206b34cf11b768c4e74582ffb89e78feedbce7ef
> # Parent  7fbe70c91d65683ab3984f7c80edc85f5c02860c
> # Available At https://hg.logilab.org/users/dlaxalde/hg
> #  hg pull https://hg.logilab.org/users/dlaxalde/hg -r
> 206b34cf11b7
> # EXP-Topic linerange-log/hgweb-filelog
> hgweb: link to diff with linerange in filelog template
>
> When "linerange" query parameter is present on a filelog request, the
> "diff"
> links in revision table rows will also have this parameter.
>
> Not sure this is that good from a user experience point of view, maybe an
> additional "block diff" link would be better.
>
> (Still only the gitweb template is updated.)
>
> diff --git a/mercurial/templates/gitweb/map b/mercurial/templates/gitweb/
> map
> --- a/mercurial/templates/gitweb/map
> +++ b/mercurial/templates/gitweb/map
> @@ -293,7 +293,7 @@ filelogentry = '
>  
>  
>file |
> -  diff |
> +   "?")}linerange={linerange}')}">diff |
>

{linerange} needs escaped because ':' is a reserved character in URLs. It
should be encoded as %3A in the HTML.


>annotate
>{rename%filelogrename}
>  
> diff --git a/tests/test-hgweb-filelog.t b/tests/test-hgweb-filelog.t
> --- a/tests/test-hgweb-filelog.t
> +++ b/tests/test-hgweb-filelog.t
> @@ -726,7 +726,7 @@ before addition - error
>
>
>file |
> -  diff |
> +  diff |
>annotate
>
>
> @@ -742,7 +742,7 @@ before addition - error
>
>
>file |
> -  diff |
> +  diff |
>annotate
>
>
> @@ -826,7 +826,7 @@ before addition - error
>
>
>file |
> -  diff |
> +   href="/diff/46c1a66bd8fc/c?revcount=1=gitweb=1:2">diff
> |
>annotate
>
>
> ___
> Mercurial-devel mailing list
> Mercurial-devel@mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
>
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 2 of 8 V2] hgweb: handle a "linerange" request parameter in filelog command

2017-02-28 Thread Gregory Szorc

On Sat, Feb 25, 2017 at 1:05 AM, Denis Laxalde  wrote:

> # HG changeset patch
> # User Denis Laxalde 
> # Date 1484844060 -3600
> #  Thu Jan 19 17:41:00 2017 +0100
> # Node ID 43d99fe8dcb510540db19366fcac8159d4e26691
> # Parent  17c6195ce42f3cbb47cee53aeb4e6a4e5074878a
> # Available At https://hg.logilab.org/users/dlaxalde/hg
> #  hg pull https://hg.logilab.org/users/dlaxalde/hg -r
> 43d99fe8dcb5
> # EXP-Topic linerange-log/hgweb-filelog
> hgweb: handle a "linerange" request parameter in filelog command
>
> We now handle a "linerange" URL query parameter to filter filelog using
> a logic similar to followlines() revset.
> The URL syntax is: log//?linerange=:
> As a result, filelog entries only consists of revision changing specified
> line range.
>
> The linerange information is propagated to "more"/"less" navigation links
> but
> not to numeric navigation links as this would apparently require a
> dedicated
> "revnav" class.
>
> Only the gitweb template got updated for now.
>
> diff --git a/mercurial/hgweb/webcommands.py b/mercurial/hgweb/webcommands.
> py
> --- a/mercurial/hgweb/webcommands.py
> +++ b/mercurial/hgweb/webcommands.py
> @@ -28,6 +28,7 @@ from .common import (
>
>  from .. import (
>  archival,
> +context,
>  encoding,
>  error,
>  graphmod,
> @@ -969,6 +970,8 @@ def filelog(web, req, tmpl):
>  except ValueError:
>  pass
>
> +lrange = webutil.linerange(req)
> +
>  lessvars = copy.copy(tmpl.defaults['sessionvars'])
>  lessvars['revcount'] = max(revcount / 2, 1)
>  morevars = copy.copy(tmpl.defaults['sessionvars'])
> @@ -982,26 +985,48 @@ def filelog(web, req, tmpl):
>  repo = web.repo
>  revs = fctx.filelog().revs(start, end - 1)
>  entries = []
> -for i in revs:
> -iterfctx = fctx.filectx(i)
> -entries.append(dict(
> -parity=next(parity),
> -filerev=i,
> -file=f,
> -rename=webutil.renamelink(iterfctx),
> -**webutil.commonentry(repo, iterfctx)))
> -entries.reverse()
> +linerange = None
> +if lrange is not None:
> +linerange = webutil.formatlinerange(*lrange)
> +# deactivate numeric nav links when linerange is specified as this
> +# would required a dedicated "revnav" class
> +nav = None
> +ancestors = context.blockancestors(fctx, *lrange)
> +for i, (c, lr) in enumerate(ancestors, 1):
> +# follow renames accross filtered (not in range) revisions
> +path = c.path()
> +entries.append(dict(
> +parity=next(parity),
> +filerev=c.rev(),
> +file=path,
> +linerange=webutil.formatlinerange(*lr),
> +**webutil.commonentry(repo, c)))
> +if i == revcount:
> +break
> +lessvars['linerange'] = webutil.formatlinerange(*lrange)
> +morevars['linerange'] = lessvars['linerange']
> +else:
> +for i in revs:
> +iterfctx = fctx.filectx(i)
> +entries.append(dict(
> +parity=next(parity),
> +filerev=i,
> +file=f,
> +rename=webutil.renamelink(iterfctx),
> +**webutil.commonentry(repo, iterfctx)))
> +entries.reverse()
> +revnav = webutil.filerevnav(web.repo, fctx.path())
> +nav = revnav.gen(end - 1, revcount, count)
>
>  latestentry = entries[:1]
>
> -revnav = webutil.filerevnav(web.repo, fctx.path())
> -nav = revnav.gen(end - 1, revcount, count)
>  return tmpl("filelog",
>  file=f,
>  nav=nav,
>  symrev=webutil.symrevorshortnode(req, fctx),
>  entries=entries,
>  latestentry=latestentry,
> +linerange=linerange,
>  revcount=revcount,
>  morevars=morevars,
>  lessvars=lessvars,
> diff --git a/mercurial/hgweb/webutil.py b/mercurial/hgweb/webutil.py
> --- a/mercurial/hgweb/webutil.py
> +++ b/mercurial/hgweb/webutil.py
> @@ -18,6 +18,7 @@ from ..node import hex, nullid, short
>
>  from .common import (
>  ErrorResponse,
> +HTTP_BAD_REQUEST,
>  HTTP_NOT_FOUND,
>  paritygen,
>  )
> @@ -313,6 +314,25 @@ def filectx(repo, req):
>
>  return fctx
>
> +def linerange(req):
> +linerange = req.form.get('linerange')
> +if linerange is None:
> +return None
> +if len(linerange) > 1:
> +raise ErrorResponse(HTTP_BAD_REQUEST,
> +'redundant linerange parameter')
> +try:
> +try:
> +fromline, toline = map(int, linerange[0].split(':', 1))
> +except ValueError:
> +raise ValueError('invalid linerange parameter')
> +return util.processlinerange(fromline, toline)
> +except ValueError as exc:
> +

Re: [PATCH 04 of 11] hgweb: handle a "linerange" request parameter in filelog command

2017-02-28 Thread Gregory Szorc

On Thu, Feb 23, 2017 at 6:54 AM, Yuya Nishihara  wrote:

> On Wed, 22 Feb 2017 16:09:58 +0100, Denis Laxalde wrote:
> > # HG changeset patch
> > # User Denis Laxalde 
> > # Date 1484844060 -3600
> > #  Thu Jan 19 17:41:00 2017 +0100
> > # Node ID 4d83e067c9c6e0f5fb7b126a9739aa9dd8113af4
> > # Parent  0a725b74eeb2a8eca6d1d82ba8ec8c2d188e8684
> > # EXP-Topic linerange-log/hgweb-filelog
> > hgweb: handle a "linerange" request parameter in filelog command
> >
> > We now handle a "linerange" URL query parameter to filter filelog using
> > a logic similar to followlines() revset.
> > The URL syntax is: log//?linerange=-
>
> Nit: how about copying the : syntax from revset?
>

I agree with this suggestion (consistency is good) and see it has been
incorporated into the v2 series. However, I'd like to state for the record
that ':' is a reserved character in URLs and needs to be percent escaped.
So that's slightly annoying, especially for power users that may wish to
tweak URLs in their browser. Although, I'm pretty sure the URL parsers in
browsers will allow the literal ':' so it shouldn't be a big deal in
practice.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 1 of 5] transaction: introduce scope

2017-02-28 Thread Durham Goode

On 2/28/17 2:41 PM, Martin von Zweigbergk wrote:

On Tue, Feb 28, 2017 at 2:34 PM, Durham Goode wrote:

On 2/27/17 9:35 AM, Jun Wu wrote:

# HG changeset patch
# User Jun Wu
# Date 1488185788 28800
# Mon Feb 27 00:56:28 2017 -0800
# Node ID 5dac612ec9a87553fcd329100846bcb01bae9d80
# Parent b4cb86ab4c719eb615a4308eafd8b1386a511eeb
# Available At
https://urldefense.proofpoint.com/v2/url?u=https-3A__bitbucket.org_quark-2Dzju_hg-2Ddraft=DwIGaQ=5VD0RTtNlTh3ycd41b3MUw=nuarHzhP1wi1T9iURRCj1A=32KRbLZXwqv-6cN9kkgbRvm2CXLXQaqOiiVERyotxSQ=QQPzFUNWmG02Bz8xilA5LPRAfljxDn-s2-q1qSGuXps=
# hg pull
https://urldefense.proofpoint.com/v2/url?u=https-3A__bitbucket.org_quark-2Dzju_hg-2Ddraft=DwIGaQ=5VD0RTtNlTh3ycd41b3MUw=nuarHzhP1wi1T9iURRCj1A=32KRbLZXwqv-6cN9kkgbRvm2CXLXQaqOiiVERyotxSQ=QQPzFUNWmG02Bz8xilA5LPRAfljxDn-s2-q1qSGuXps=
-r 5dac612ec9a8
transaction: introduce scope

Previously, transaction tries to cover everything: bookmarks, dirstate,
phaseroots, and branch. It backs them up unconditionally.

That could be inefficient if we still back up the dirstate in a giant
repo,
where the transaction is only intended to change bookmarks.

This patch introduces "scope" to transactions, which is a set of things
that
the transaction should cover. For bookmark-only update, it could avoid
specifying the "dirstate" scope to be more efficient.

I was talking with Jun about this in person and a few things came up. I
proposed we just reenable hardlink support for transaction backups of
atomic-rewrite-files like the dirstate. That would remove the IO of copying
that I assumed was the problem. However, it turns out the dirstate doesn't
even use the tr.addbackup() mechanism that everything else uses. Instead,
it writes out the *current* contents of the inmemory dirstate and treats
that as the backup, which seems extra wrong since the current inmemory data
may not match the data on disk that we're about to overwrite, and extra
wrong since it means CPU time spent serializing the dirstate whenever a
transaction opens.

Seems like we should refactor the dirstate/dirstateguard business to use the
real transaction logic, then we could benefit from the hardlink backups
optimization.

I was also wondering why it was treated differently. Great if we can
clean that up.

Jun tried and it seems it will require some significant effort.

Not that such a refactor should block this series.

If what you describe above is done, would there still be a desire for
this series or would we undo this series once the dirstate-cleanup and
hard-linking is done?

If the only benefit from the scope work is for the dirstate, then yea we
could undo this series after the dirstate refactor is done.

There may be other benefits to having the concept of scope though. If we
moved the scope concept up to the lock level too (I've always said we
should combine lock and transactions into one concept...) it might allow
us to get rid of the repo.wlock() and repo.lock() differences, and the
user just specifies a scope they care about. If we take it farther, we
could identify scopes that are independent and allow multiple locks at
the same time. But that's very future work.

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 1 of 5] transaction: introduce scope

2017-02-28 Thread Jun Wu

Excerpts from Martin von Zweigbergk's message of 2017-02-28 14:41:47 -0800:
> > I was talking with Jun about this in person and a few things came up.  I
> > proposed we just reenable hardlink support for transaction backups of
> > atomic-rewrite-files like the dirstate. That would remove the IO of copying
> > that I assumed was the problem. However, it turns out the dirstate doesn't
> > even use the tr.addbackup() mechanism that everything else uses.  Instead,
> > it writes out the *current* contents of the inmemory dirstate and treats
> > that as the backup, which seems extra wrong since the current inmemory data
> > may not match the data on disk that we're about to overwrite, and extra
> > wrong since it means CPU time spent serializing the dirstate whenever a
> > transaction opens.
> >
> > Seems like we should refactor the dirstate/dirstateguard business to use the
> > real transaction logic, then we could benefit from the hardlink backups
> > optimization.
> 
> I was also wondering why it was treated differently. Great if we can
> clean that up.

Note: I had attempted to do it that way by the following change:

# HG changeset patch
# User Jun Wu 
# Date 1488183419 28800
#  Mon Feb 27 00:16:59 2017 -0800
# Node ID 71621760eaf6b99416e5f274a07a10b370c78691
# Parent  b4cb86ab4c719eb615a4308eafd8b1386a511eeb
# Available At https://bitbucket.org/quark-zju/hg-draft
#  hg pull https://bitbucket.org/quark-zju/hg-draft -r 
71621760eaf6
debug: try to use transaction for dirstate backup

diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -1083,6 +1083,5 @@ class localrepository(object):
 # discard all changes (including ones already written
 # out) in this transaction
-repo.dirstate.restorebackup(None, prefix='journal.')
-
+# repo.dirstate.restorebackup(None, prefix='journal.')
 repo.invalidate(clearfilecache=True)
 
@@ -1123,4 +1122,5 @@ class localrepository(object):
 # to stored data if transaction has no error.
 tr.addpostclose('refresh-filecachestats', 
self._refreshfilecachestats)
+tr.addbackup('dirstate')
 self._transref = weakref.ref(tr)
 return tr
@@ -1128,5 +1128,4 @@ class localrepository(object):
 def _journalfiles(self):
 return ((self.svfs, 'journal'),
-(self.vfs, 'journal.dirstate'),
 (self.vfs, 'journal.branch'),
 (self.vfs, 'journal.desc'),
@@ -1138,5 +1137,4 @@ class localrepository(object):
 
 def _writejournal(self, desc):
-self.dirstate.savebackup(None, prefix='journal.')
 self.vfs.write("journal.branch",
   encoding.fromlocal(self.dirstate.branch()))

However, *many* tests will fail. I can use some time to try again but that
won't guarantee a success.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 1 of 5] transaction: introduce scope

2017-02-28 Thread Martin von Zweigbergk via Mercurial-devel

On Tue, Feb 28, 2017 at 2:34 PM, Durham Goode  wrote:
>
>
> On 2/27/17 9:35 AM, Jun Wu wrote:
>>
>> # HG changeset patch
>> # User Jun Wu 
>> # Date 1488185788 28800
>> #  Mon Feb 27 00:56:28 2017 -0800
>> # Node ID 5dac612ec9a87553fcd329100846bcb01bae9d80
>> # Parent  b4cb86ab4c719eb615a4308eafd8b1386a511eeb
>> # Available At
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__bitbucket.org_quark-2Dzju_hg-2Ddraft=DwIGaQ=5VD0RTtNlTh3ycd41b3MUw=nuarHzhP1wi1T9iURRCj1A=32KRbLZXwqv-6cN9kkgbRvm2CXLXQaqOiiVERyotxSQ=QQPzFUNWmG02Bz8xilA5LPRAfljxDn-s2-q1qSGuXps=
>> #  hg pull
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__bitbucket.org_quark-2Dzju_hg-2Ddraft=DwIGaQ=5VD0RTtNlTh3ycd41b3MUw=nuarHzhP1wi1T9iURRCj1A=32KRbLZXwqv-6cN9kkgbRvm2CXLXQaqOiiVERyotxSQ=QQPzFUNWmG02Bz8xilA5LPRAfljxDn-s2-q1qSGuXps=
>> -r 5dac612ec9a8
>> transaction: introduce scope
>>
>> Previously, transaction tries to cover everything: bookmarks, dirstate,
>> phaseroots, and branch. It backs them up unconditionally.
>>
>> That could be inefficient if we still back up the dirstate in a giant
>> repo,
>> where the transaction is only intended to change bookmarks.
>>
>> This patch introduces "scope" to transactions, which is a set of things
>> that
>> the transaction should cover. For bookmark-only update, it could avoid
>> specifying the "dirstate" scope to be more efficient.
>
>
> I was talking with Jun about this in person and a few things came up.  I
> proposed we just reenable hardlink support for transaction backups of
> atomic-rewrite-files like the dirstate. That would remove the IO of copying
> that I assumed was the problem. However, it turns out the dirstate doesn't
> even use the tr.addbackup() mechanism that everything else uses.  Instead,
> it writes out the *current* contents of the inmemory dirstate and treats
> that as the backup, which seems extra wrong since the current inmemory data
> may not match the data on disk that we're about to overwrite, and extra
> wrong since it means CPU time spent serializing the dirstate whenever a
> transaction opens.
>
> Seems like we should refactor the dirstate/dirstateguard business to use the
> real transaction logic, then we could benefit from the hardlink backups
> optimization.

I was also wondering why it was treated differently. Great if we can
clean that up.

>
> Not that such a refactor should block this series.

If what you describe above is done, would there still be a desire for
this series or would we undo this series once the dirstate-cleanup and
hard-linking is done?

>
> ___
> Mercurial-devel mailing list
> Mercurial-devel@mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 1 of 5] transaction: introduce scope

2017-02-28 Thread Durham Goode

On 2/27/17 9:35 AM, Jun Wu wrote:

Previously, transaction tries to cover everything: bookmarks, dirstate,
phaseroots, and branch. It backs them up unconditionally.

That could be inefficient if we still back up the dirstate in a giant repo,
where the transaction is only intended to change bookmarks.

This patch introduces "scope" to transactions, which is a set of things that
the transaction should cover. For bookmark-only update, it could avoid
specifying the "dirstate" scope to be more efficient.

I was talking with Jun about this in person and a few things came up. I
proposed we just reenable hardlink support for transaction backups of
atomic-rewrite-files like the dirstate. That would remove the IO of
copying that I assumed was the problem. However, it turns out the
dirstate doesn't even use the tr.addbackup() mechanism that everything
else uses. Instead, it writes out the *current* contents of the
inmemory dirstate and treats that as the backup, which seems extra wrong
since the current inmemory data may not match the data on disk that
we're about to overwrite, and extra wrong since it means CPU time spent
serializing the dirstate whenever a transaction opens.

Seems like we should refactor the dirstate/dirstateguard business to use
the real transaction logic, then we could benefit from the hardlink
backups optimization.

Not that such a refactor should block this series.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] smartset: reorder initialization of baseset in more intuitive way

2017-02-28 Thread Martin von Zweigbergk via Mercurial-devel

On Sat, Feb 18, 2017 at 5:39 PM, Jun Wu  wrote:
> Looks good to me. Thanks for the clean-up!

Yep, looks good. Queued, thanks. And sorry about the delay.

>
> Excerpts from Yuya Nishihara's message of 2017-02-18 18:48:06 +0900:
>> # HG changeset patch
>> # User Yuya Nishihara 
>> # Date 1487407072 -32400
>> #  Sat Feb 18 17:37:52 2017 +0900
>> # Node ID 1ca3469fdd08c0d5d814a4bc359869bc157c7fc9
>> # Parent  3f4bdcfe63e9f1e357187fa728ae0da9d35b3973
>> smartset: reorder initialization of baseset in more intuitive way
>>
>> What we want to do is to assign either _set or _list per the given data
>> type.
>>
>> diff --git a/mercurial/smartset.py b/mercurial/smartset.py
>> --- a/mercurial/smartset.py
>> +++ b/mercurial/smartset.py
>> @@ -166,16 +166,14 @@ class baseset(abstractsmartset):
>>  """
>>  self._ascending = None
>>  self._istopo = istopo
>> -if not isinstance(data, list):
>> -if isinstance(data, set):
>> -self._set = data
>> -# set has no order we pick one for stability purpose
>> -self._ascending = True
>> -# converting set to list has a cost, do it lazily
>> -data = None
>> -else:
>> +if isinstance(data, set):
>> +# converting set to list has a cost, do it lazily
>> +self._set = data
>> +# set has no order we pick one for stability purpose
>> +self._ascending = True
>> +else:
>> +if not isinstance(data, list):
>>  data = list(data)
>> -if data is not None:
>>  self._list = data
>>  self._datarepr = datarepr
>>
> ___
> Mercurial-devel mailing list
> Mercurial-devel@mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 4 of 5] localrepo: do not write dirstate if it's not in transaction scope

2017-02-28 Thread Jun Wu

Excerpts from Martin von Zweigbergk's message of 2017-02-28 13:15:17 -0800:
> On Mon, Feb 27, 2017 at 9:35 AM, Jun Wu  wrote:
> > # HG changeset patch
> > # User Jun Wu 
> > # Date 1488189479 28800
> > #  Mon Feb 27 01:57:59 2017 -0800
> > # Node ID f88c37484782ae7fede3c7a2a27a5f5d9b12bc9a
> > # Parent  89793c9bfdc93896fd308ad9f4c400710ff2bde2
> > # Available At https://bitbucket.org/quark-zju/hg-draft 
> > #  hg pull https://bitbucket.org/quark-zju/hg-draft  -r 
> > f88c37484782
> > localrepo: do not write dirstate if it's not in transaction scope
> >
> > See the previous patch for why.
> >
> > diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
> > --- a/mercurial/localrepo.py
> > +++ b/mercurial/localrepo.py
> > @@ -1077,14 +1077,16 @@ class localrepository(object):
> >  repo = reporef()
> >  if success:
> > -# this should be explicitly invoked here, because
> > -# in-memory changes aren't written out at closing
> > -# transaction, if tr.addfilegenerator (via
> > -# dirstate.write or so) isn't invoked while
> > -# transaction running
> > -repo.dirstate.write(None)
> > +if 'dirstate' in tr.scope():
> > +# this should be explicitly invoked here, because
> > +# in-memory changes aren't written out at closing
> > +# transaction, if tr.addfilegenerator (via
> > +# dirstate.write or so) isn't invoked while
> > +# transaction running
> > +repo.dirstate.write(None)
> 
> Can we check that the dirstate has not been modified since the start
> of the transaction and fail if it has? Otherwise it seems like we may
> easily forgot to add e.g. "dirstate" to the scope one day when "hg
> bookmarks" starts modifying it. I suspect the answer is that that
> won't be easy, but how much work it be? The same thing of course
> applies to the journal files in the other patch.

That's not enough - in theory we also want to make sure dirstate becomes
immutable afterwards.

Originally I was thinking about making "backupdirstate" lazy - only backup
it when dirstate is being changed. That should be even better than what you
suggest. But aside from the implementation complexity, I don't feel very
comfortable with dirstate holding a reference to the transaction or repo
object. I'll try again, but if it turned out to be hairy, I'd still want
this simple approach for the perf win.

> 
> >  else:
> > -# discard all changes (including ones already written
> > -# out) in this transaction
> > -repo.dirstate.restorebackup(None, prefix='journal.')
> > +if 'dirstate' in tr.scope():
> > +# discard all changes (including ones already written
> > +# out) in this transaction
> > +repo.dirstate.restorebackup(None, prefix='journal.')
> >
> >  repo.invalidate(clearfilecache=True)
> > ___
> > Mercurial-devel mailing list
> > Mercurial-devel@mercurial-scm.org
> > https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 4 of 5] localrepo: do not write dirstate if it's not in transaction scope

2017-02-28 Thread Martin von Zweigbergk via Mercurial-devel

On Mon, Feb 27, 2017 at 9:35 AM, Jun Wu  wrote:
> # HG changeset patch
> # User Jun Wu 
> # Date 1488189479 28800
> #  Mon Feb 27 01:57:59 2017 -0800
> # Node ID f88c37484782ae7fede3c7a2a27a5f5d9b12bc9a
> # Parent  89793c9bfdc93896fd308ad9f4c400710ff2bde2
> # Available At https://bitbucket.org/quark-zju/hg-draft
> #  hg pull https://bitbucket.org/quark-zju/hg-draft -r 
> f88c37484782
> localrepo: do not write dirstate if it's not in transaction scope
>
> See the previous patch for why.
>
> diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
> --- a/mercurial/localrepo.py
> +++ b/mercurial/localrepo.py
> @@ -1077,14 +1077,16 @@ class localrepository(object):
>  repo = reporef()
>  if success:
> -# this should be explicitly invoked here, because
> -# in-memory changes aren't written out at closing
> -# transaction, if tr.addfilegenerator (via
> -# dirstate.write or so) isn't invoked while
> -# transaction running
> -repo.dirstate.write(None)
> +if 'dirstate' in tr.scope():
> +# this should be explicitly invoked here, because
> +# in-memory changes aren't written out at closing
> +# transaction, if tr.addfilegenerator (via
> +# dirstate.write or so) isn't invoked while
> +# transaction running
> +repo.dirstate.write(None)

Can we check that the dirstate has not been modified since the start
of the transaction and fail if it has? Otherwise it seems like we may
easily forgot to add e.g. "dirstate" to the scope one day when "hg
bookmarks" starts modifying it. I suspect the answer is that that
won't be easy, but how much work it be? The same thing of course
applies to the journal files in the other patch.

>  else:
> -# discard all changes (including ones already written
> -# out) in this transaction
> -repo.dirstate.restorebackup(None, prefix='journal.')
> +if 'dirstate' in tr.scope():
> +# discard all changes (including ones already written
> +# out) in this transaction
> +repo.dirstate.restorebackup(None, prefix='journal.')
>
>  repo.invalidate(clearfilecache=True)
> ___
> Mercurial-devel mailing list
> Mercurial-devel@mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 7 of 7] config: update the Windows example config file

2017-02-28 Thread Martin von Zweigbergk via Mercurial-devel

On Tue, Feb 28, 2017 at 11:25 AM, Pierre-Yves David
 wrote:
> # HG changeset patch
> # User Pierre-Yves David 
> # Date 1488309790 -3600
> #  Tue Feb 28 20:23:10 2017 +0100
> # Node ID b20bdc56362c3e4f676a47c71edf02107b01c507
> # Parent  9fffdf9b71de54fba36d3efd581aa660ab8076d1
> # EXP-Topic color
> config: update the Windows example config file

Queued with some minor copy editing done in flight. Thanks.

>
> We move from the color extensions to the 'ui.color' config.
>
> diff --git a/contrib/win32/mercurial.ini b/contrib/win32/mercurial.ini
> --- a/contrib/win32/mercurial.ini
> +++ b/contrib/win32/mercurial.ini
> @@ -19,6 +19,8 @@
>  editor = notepad
>  ; show changed files and be a bit more verbose if True
>  ; verbose = True
> +; colorize commands output
> +; color = auto
>
>  ; username data to appear in commits
>  ; it usually takes the form: Joe User 
> @@ -40,7 +42,6 @@ editor = notepad
>  ;bugzilla =
>  ;children =
>  ;churn =
> -;color =
>  ;convert =
>  ;eol =
>  ;extdiff =
> ___
> Mercurial-devel mailing list
> Mercurial-devel@mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 2 V2] treemanifest: make node reuse match flat manifest behavior

2017-02-28 Thread Durham Goode

# HG changeset patch
# User Durham Goode 
# Date 1488067881 28800
#  Sat Feb 25 16:11:21 2017 -0800
# Node ID fb45d40719699efa3139e3509775fd5442c1c36a
# Parent  0a5e6ad0a25c348384bb6c14a51057b3587e2857
treemanifest: make node reuse match flat manifest behavior

In a flat manifest, a node with the same content but different parents is still
considered a new node. In the current tree manifests however, if the content is
the same, we ignore the parents entirely and just reuse the existing node.

In our external treemanifest extension, we want to allow having one treemanifest
for every flat manifests, as a way of easeing the migration to treemanifests. To
make this possible, let's change the root node treemanifest behavior to match
the behavior for flat manifests, so we can have a 1:1 relationship.

While this sounds like a BC breakage, it's not actually a state users can
normally get in because: A) you can't make empty commits, and B) even if you try
to make an empty commit (by making a commit then amending it's changes away),
the higher level commit logic in localrepo.commitctx() forces the commit to use
the original p1 manifest node if no files were changed. So this would only
affect extensions and automation that reached passed the normal
localrepo.commit() logic straight into the manifest logic.

diff --git a/mercurial/manifest.py b/mercurial/manifest.py
--- a/mercurial/manifest.py
+++ b/mercurial/manifest.py
@@ -1229,7 +1229,7 @@ class manifestrevlog(revlog.revlog):
 def _addtree(self, m, transaction, link, m1, m2, readtree):
 # If the manifest is unchanged compared to one parent,
 # don't write a new revision
-if m.unmodifiedsince(m1) or m.unmodifiedsince(m2):
+if self._dir != '' and (m.unmodifiedsince(m1) or 
m.unmodifiedsince(m2)):
 return m.node()
 def writesubtree(subm, subp1, subp2):
 sublog = self.dirlog(subm.dir())
@@ -1237,13 +1237,17 @@ class manifestrevlog(revlog.revlog):
readtree=readtree)
 m.writesubtrees(m1, m2, writesubtree)
 text = m.dirtext(self._usemanifestv2)
-# Double-check whether contents are unchanged to one parent
-if text == m1.dirtext(self._usemanifestv2):
-n = m1.node()
-elif text == m2.dirtext(self._usemanifestv2):
-n = m2.node()
-else:
+n = None
+if self._dir != '':
+# Double-check whether contents are unchanged to one parent
+if text == m1.dirtext(self._usemanifestv2):
+n = m1.node()
+elif text == m2.dirtext(self._usemanifestv2):
+n = m2.node()
+
+if not n:
 n = self.addrevision(text, transaction, link, m1.node(), m2.node())
+
 # Save nodeid so parent manifest can calculate its nodeid
 m.setnode(n)
 return n
diff --git a/tests/test-treemanifest.t b/tests/test-treemanifest.t
--- a/tests/test-treemanifest.t
+++ b/tests/test-treemanifest.t
@@ -825,3 +825,13 @@ other branch
   added 1 changesets with 1 changes to 1 files (+1 heads)
   (run 'hg heads' to see heads, 'hg merge' to merge)
 
+Committing a empty commit does not duplicate root treemanifest
+  $ echo z >> z
+  $ hg commit -Aqm 'pre-empty commit'
+  $ hg rm z
+  $ hg commit --amend -m 'empty commit'
+  saved backup bundle to 
$TESTTMP/grafted-dir-repo-clone/.hg/strip-backup/cb99d5717cea-de37743b-amend-backup.hg
 (glob)
+  $ hg log -r 'tip + tip^' -T '{manifest}\n'
+  1:678d3574b88c
+  1:678d3574b88c
+  $ hg --config extensions.strip= strip -r . -q
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 1 of 2 V2] manifest: allow specifying the revlog filename

2017-02-28 Thread Durham Goode

# HG changeset patch
# User Durham Goode 
# Date 1488304382 28800
#  Tue Feb 28 09:53:02 2017 -0800
# Node ID 0a5e6ad0a25c348384bb6c14a51057b3587e2857
# Parent  88203f26ea57627cabd7cf9c4f7843661d6c43ae
manifest: allow specifying the revlog filename

Previously we had hardcoded the manifest filename to be 00manifest.i. In our
external treemanifest extension, we want to allow writing a treemanifest side by
side with a flat manifest, so we need to be able to store the root revisions at
a different location (in our extension we use 00manifesttree.i).

This patches moves the revlog name to a parameter so we can adjust it.

diff --git a/mercurial/manifest.py b/mercurial/manifest.py
--- a/mercurial/manifest.py
+++ b/mercurial/manifest.py
@@ -1132,7 +1132,13 @@ class manifestrevlog(revlog.revlog):
 '''A revlog that stores manifest texts. This is responsible for caching the
 full-text manifest contents.
 '''
-def __init__(self, opener, dir='', dirlogcache=None):
+def __init__(self, opener, dir='', dirlogcache=None,
+ indexfile='00manifest.i'):
+"""Constructs a new manifest revlog
+
+`indexfile` - used by extensions to have two manifests at once, like
+when transitioning between flatmanifeset and treemanifests.
+"""
 # During normal operations, we expect to deal with not more than four
 # revs at a time (such as during commit --amend). When rebasing large
 # stacks of commits, the number can go up, hence the config knob below.
@@ -1150,12 +1156,11 @@ class manifestrevlog(revlog.revlog):
 
 self._fulltextcache = util.lrucachedict(cachesize)
 
-indexfile = "00manifest.i"
 if dir:
 assert self._treeondisk, 'opts is %r' % opts
 if not dir.endswith('/'):
 dir = dir + '/'
-indexfile = "meta/" + dir + "00manifest.i"
+indexfile = "meta/" + dir + indexfile
 self._dir = dir
 # The dirlogcache is kept on the root manifest log
 if dir:
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 1 of 2] manifest: allow specifying the revlog filename

2017-02-28 Thread Durham Goode




On 2/28/17 9:53 AM, Martin von Zweigbergk wrote:

On Tue, Feb 28, 2017 at 9:49 AM, Durham Goode  wrote:



On 2/27/17 5:10 PM, Martin von Zweigbergk wrote:


+mercurial-devel again (I had initially thought my comment was not
about the patch, but it turns out it was)

On Sat, Feb 25, 2017 at 3:23 PM, Durham Goode  wrote:


On 2/24/17 10:00 PM, Martin von Zweigbergk wrote:



-list

On Fri, Feb 24, 2017, 14:44 Durham Goode > wrote:

# HG changeset patch
# User Durham Goode >
# Date 1487973639 28800
#  Fri Feb 24 14:00:39 2017 -0800
# Node ID ed987b24e755a4c61c006f7d98a4ff370229faad
# Parent  f01df5d2fe493376a67569756a9bc2ddffa5cd81
manifest: allow specifying the revlog filename

Previously we had hardcoded the manifest filename to be
00manifest.i. In our
external treemanifest extension, we want to allow writing a
treemanifest side by
side with a flat manifest, so we need to be able to store the root
revisions at
a different location (in our extension we use 00manifesttree.i).


Why not meta/00manifest.i? That would be more consistent. I let them
(flat and tree) share a revlog only to allow for transition between
them. But if you don't care about that, meta/00manifest.i sounds like
the obvious choice.




I'd be fine with that too, but I'm not sure how we'd go about that. I
guess
I could put an option on the opener that the manifest constructor checks
for, like 'separatetreeroot=True' and then the constructor would know to
put
it in meta/.  Seem reasonable?



Or you could keep your new indexfile argument to the constructor and
use that if it was set and otherwise use the current way of
calculating it. Your extension could then pass 'meta/00manifest.i' for
dir=='' and None for all others. That gives more flexibility if you
want to use some other scheme too. Also, I'd like a comment explaining
that the parameter is for extensions, so the next person doesn't
remove it.



I can add a comment.  I think putting directories in the indexfile string
might be a bit funky since I'm not sure if other parts of the code rely on
self.indexfile to be just a filename.


I don't follow. You patch touches a line that says

  indexfile = "meta/" + dir + "00manifest.i"

so it already (potentially) has a directory in it. Or do you mean
something else?


You're right, so that's not a big deal I guess.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

51 matches

Mail list logo