[PATCH 09 of 10] repair: copy non-revlog store files during upgrade

[Gregory Szorc]

# HG changeset patch
# User Gregory Szorc 
# Date 1478392019 25200
#  Sat Nov 05 17:26:59 2016 -0700
# Node ID 3d4dd237b705479f8c7475b821ae719893381fa8
# Parent  d2261c558ca9639fb81c182de15d75151cbad0f9
repair: copy non-revlog store files during upgrade

The store contains more than just revlogs. This patch teaches the
upgrade code to copy regular files as well.

As the test changes demonstrate, the phaseroots file is now copied.

diff --git a/mercurial/repair.py b/mercurial/repair.py
--- a/mercurial/repair.py
+++ b/mercurial/repair.py
@@ -10,6 +10,7 @@ from __future__ import absolute_import
 
 import errno
 import hashlib
+import stat
 import tempfile
 import time
 
@@ -622,6 +623,39 @@ def _copyrevlogs(ui, srcrepo, dstrepo, t
'across %d revlogs and %d revisions\n') % (
  dstsize, dstsize - srcsize, rlcount, revcount))
 
+def _upgradefilterstorefile(srcrepo, dstrepo, requirements, path, mode, st):
+"""Determine whether to copy a store file during upgrade.
+
+This function is called when migrating store files from ``srcrepo`` to
+``dstrepo`` as part of upgrading a repository. The function receives
+the ``requirements`` for ``dstrepo``, the relative ``path`` of the
+store file being examined, its ``ST_MODE`` file type, and a full
+stat data structure.
+
+Function should return ``True`` if the file is to be copied.
+"""
+# Skip revlogs.
+if path.endswith(('.i', '.d')):
+return False
+# Skip transaction related files.
+if path.startswith('undo'):
+return False
+# Only copy regular files.
+if mode != stat.S_IFREG:
+return False
+# Skip other skipped files.
+if path in ('lock', 'fncache'):
+return False
+
+return True
+
+def _upgradefinishdatamigration(ui, srcrepo, dstrepo, requirements):
+"""Hook point for extensions to perform additional actions during upgrade.
+
+This function is called after revlogs and store files have been copied but
+before the new store is swapped into the original location.
+"""
+
 def _upgraderepo(ui, srcrepo, dstrepo, requirements):
 """Do the low-level work of upgrading a repository.
 
@@ -641,7 +675,18 @@ def _upgraderepo(ui, srcrepo, dstrepo, r
 with dstrepo.transaction('upgrade') as tr:
 _copyrevlogs(ui, srcrepo, dstrepo, tr)
 
-# TODO copy non-revlog store files
+# Now copy other files in the store directory.
+for p, kind, st in srcrepo.store.vfs.readdir('', stat=True):
+if not _upgradefilterstorefile(srcrepo, dstrepo, requirements,
+   p, kind, st):
+continue
+
+srcrepo.ui.write(_('copying %s\n') % p)
+src = srcrepo.store.vfs.join(p)
+dst = dstrepo.store.vfs.join(p)
+util.copyfile(src, dst, copystat=True)
+
+_upgradefinishdatamigration(ui, srcrepo, dstrepo, requirements)
 
 ui.write(_('data fully migrated to temporary repository\n'))
 
diff --git a/tests/test-upgrade-repo.t b/tests/test-upgrade-repo.t
--- a/tests/test-upgrade-repo.t
+++ b/tests/test-upgrade-repo.t
@@ -149,6 +149,7 @@ Upgrading a repository to generaldelta w
   migrating manifests...
   migrating changelog...
   revlogs migration complete; wrote 917 bytes (delta 0 bytes) across 5 revlogs 
and 9 revisions
+  copying phaseroots
   data fully migrated to temporary repository
   starting in-place swap of repository data
   (clients may error or see inconsistent repository data until this operation 
completes)
@@ -182,6 +183,7 @@ store directory has files we expect
   00manifest.i
   data
   fncache
+  phaseroots
   undo
   undo.backupfiles
   undo.phaseroots
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 06 of 10] repair: print what upgrade will do

[Gregory Szorc]

# HG changeset patch
# User Gregory Szorc 
# Date 1478393040 25200
#  Sat Nov 05 17:44:00 2016 -0700
# Node ID 82799afa72be5bb540b2b07cd878f307622c4354
# Parent  b768004ef2db9c2e6dd267997e9e0c011f1b732a
repair: print what upgrade will do

The final part of pre-upgrade checks is validating what the upgrade
will do and printing a summary of this to the user. This patch
implements that.

Again, multiple functions are used to facilitate monkeypatching by
extensions.

diff --git a/mercurial/repair.py b/mercurial/repair.py
--- a/mercurial/repair.py
+++ b/mercurial/repair.py
@@ -480,6 +480,61 @@ def upgradereporequirements(repo):
 
 return createreqs
 
+def upgradevalidateactions(repo, sourcereqs, destreqs, actions):
+"""Validate (and modify accordingly) the list of actions to perform.
+
+Returns a set of actions that will be performed. The set is advisory
+only and is intended to drive presentation of what actions will be
+taken.
+
+Extensions can monkeypatch this to mutate the set of actions that
+will be performed.
+"""
+newactions = set(actions)
+
+for req in ('dotencode', 'fncache', 'generaldelta'):
+if req in actions and req not in destreqs:
+newactions.remove(req)
+
+return newactions
+
+def upgradesummarizeactions(repo, actions):
+"""Summarize actions that will be taken as part of upgrade.
+
+Receives a set of action names. Returns a list of strings that will be
+presented to user describing these actions and a set of actions processed
+by this function.
+"""
+l = []
+handled = set()
+
+if 'dotencode' in actions:
+l.append(_('dotencode repository layout will be used; the repository '
+   'will be able be more resilient to storing paths beginning '
+   'with periods or spaces'))
+handled.add('dotencode')
+
+if 'fncache' in actions:
+l.append(_('fncache repository layout will be used; the repository '
+   'will be more resilient storing long paths and special '
+   'filenames'))
+handled.add('fncache')
+
+if 'generaldelta' in actions:
+l.append(_('repository data will be re-encoded as "generaldelta"; '
+   'files inside the repository should be smaller, read times '
+   'should decrease, and interacting with other generaldelta '
+   'repositories should be faster'))
+handled.add('generaldelta')
+
+if 'removecldeltachain' in actions:
+l.append(_('delta chains will be removed from the changelog; reading '
+   'changelogs should be faster, changelogs may decrease '
+   'in size\n'))
+handled.add('removecldeltachain')
+
+return l, handled
+
 def upgraderepo(ui, repo, dryrun=False):
 """Upgrade a repository in place."""
 repo = repo.unfiltered()
@@ -507,3 +562,24 @@ def upgraderepo(ui, repo, dryrun=False):
 ui.write(_('adding repository requirements: %s\n') %
  ', '.join(sorted(newreqs - repo.requirements)))
 
+actions = upgradevalidateactions(repo, repo.requirements, newreqs, actions)
+
+changes, handledactions = upgradesummarizeactions(repo, actions)
+
+unhandled = actions - handledactions
+if unhandled:
+raise error.Abort(_('unable to upgrade repository; proposed upgrade '
+'action not handled: %s') %
+  ', '.join(sorted(unhandled)))
+
+if actions:
+ui.write(_('upgrading the repository will make the following '
+   'significant changes:\n\n'))
+for change in changes:
+ui.write('* %s\n' % change)
+else:
+ui.write(_('no notable upgrade changes identified\n'))
+
+if dryrun:
+ui.warn(_('dry run requested; not continuing with upgrade\n'))
+return 1
diff --git a/tests/test-upgrade-repo.t b/tests/test-upgrade-repo.t
--- a/tests/test-upgrade-repo.t
+++ b/tests/test-upgrade-repo.t
@@ -5,6 +5,18 @@
   
   checking repository requirements...
   preserving repository requirements: dotencode, fncache, generaldelta, 
revlogv1, store
+  no notable upgrade changes identified
+
+dry run works
+
+  $ hg debugupgraderepo --dry-run
+  no obvious deficiencies found in existing repository
+  
+  checking repository requirements...
+  preserving repository requirements: dotencode, fncache, generaldelta, 
revlogv1, store
+  no notable upgrade changes identified
+  dry run requested; not continuing with upgrade
+  [1]
 
 Various sub-optimal detections work
 
@@ -23,6 +35,11 @@ Various sub-optimal detections work
   checking repository requirements...
   preserving repository requirements: revlogv1, store
   adding repository requirements: dotencode, fncache, generaldelta
+  upgrading the repository will make the following significant changes:
+  
+  * dotencode repository layout will be used; the 

[PATCH 01 of 10] revlog: add clone method

[Gregory Szorc]

# HG changeset patch
# User Gregory Szorc 
# Date 1478405835 25200
#  Sat Nov 05 21:17:15 2016 -0700
# Node ID ebbd8d975e4bf59b2bdd44736fdf13222988d1a4
# Parent  f01367faa792635ad2f7a6b175ae3252292b5121
revlog: add clone method

Upcoming patches will introduce functionality for in-place
repository/store "upgrades." Copying the contents of a revlog
feels sufficiently low-level to warrant being in the revlog
class.

diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -1818,3 +1818,32 @@ class revlog(object):
 if not self._inline:
 res.append(self.datafile)
 return res
+
+def clone(self, tr, destrevlog, addrevisioncb=None):
+"""Copy the contents of this revlog to another revlog.
+
+The destination revlog will contain the same revisions and nodes.
+However, it may not be bit-for-bit identical due to e.g. delta encoding
+differences.
+"""
+if len(destrevlog):
+raise ValueError(_('destination revlog is not empty'))
+
+index = self.index
+for rev in self:
+entry = index[rev]
+
+# Some classes override linkrev to take filtered revs into
+# account. Use raw entry from index.
+linkrev = entry[4]
+p1 = index[entry[5]][7]
+p2 = index[entry[6]][7]
+node = entry[7]
+# FUTURE we could optionally allow reusing the delta to avoid
+# expensive recomputation.
+text = self.revision(rev)
+
+destrevlog.addrevision(text, tr, linkrev, p1, p2, node=node)
+
+if addrevisioncb:
+addrevisioncb(self, rev, node)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 07 of 10] repair: begin implementation of in-place upgrading

[Gregory Szorc]

# HG changeset patch
# User Gregory Szorc 
# Date 1478393266 25200
#  Sat Nov 05 17:47:46 2016 -0700
# Node ID 958bcf2577608bbb6d8ae078cde0ca451f3ab31a
# Parent  82799afa72be5bb540b2b07cd878f307622c4354
repair: begin implementation of in-place upgrading

Now that all the upgrade planning work is complete, we can start
doing the real work: actually upgrading a repository.

The main goal of this patch is to get the "framework" for running
in-place upgrade actions in place.

Rather than get too clever and low-level with regards to in-place
upgrades, our strategy for an in-place upgrade is to create a new,
temporary repository, copy data to it, then replace the old data
with the new. This allows us to reuse a lot of code in localrepo.py
around store interaction, which will eventually consume the bulk of
the upgrade code.

But we have to start small. This patch implements adding new
repository requirements. But it still sets up a temporary
repository and locks it and the source repo before performing the
requirements file swap. This means all the plumbing is in place
to implement store copying in subsequent patches.

diff --git a/mercurial/repair.py b/mercurial/repair.py
--- a/mercurial/repair.py
+++ b/mercurial/repair.py
@@ -10,6 +10,7 @@ from __future__ import absolute_import
 
 import errno
 import hashlib
+import tempfile
 
 from .i18n import _
 from .node import short
@@ -19,6 +20,7 @@ from . import (
 error,
 exchange,
 obsolete,
+scmutil,
 util,
 )
 
@@ -535,8 +537,41 @@ def upgradesummarizeactions(repo, action
 
 return l, handled
 
+def _upgraderepo(ui, srcrepo, dstrepo, requirements):
+"""Do the low-level work of upgrading a repository.
+
+The upgrade is effectively performed as a copy between a source
+repository and a temporary destination repository.
+
+The resource repository is intended to be read only for as long as
+possible so the upgrade can abort at any time without corrupting
+the source.
+"""
+assert srcrepo.currentwlock()
+assert dstrepo.currentwlock()
+
+# TODO copy store
+
+ui.write(_('starting in-place swap of repository data\n'))
+ui.warn(_('(clients may error or see inconsistent repository data until '
+  'this operation completes)\n'))
+
+backuppath = tempfile.mkdtemp(prefix='upgradebackup.', dir=srcrepo.path)
+backupvfs = scmutil.vfs(backuppath)
+ui.write(_('replaced files will be backed up at %s\n') %
+ backuppath)
+
+# We first write the requirements file. Any new requirements will lock
+# out legacy clients.
+ui.write(_('updating requirements in %s\n') % srcrepo.join('requires'))
+util.copyfile(srcrepo.join('requires'), backupvfs.join('requires'))
+scmutil.writerequires(srcrepo.vfs, requirements)
+
 def upgraderepo(ui, repo, dryrun=False):
 """Upgrade a repository in place."""
+# Avoid cycle: cmdutil -> repair -> localrepo -> cmdutil
+from . import localrepo
+
 repo = repo.unfiltered()
 deficiencies, actions = upgradefinddeficiencies(repo)
 
@@ -583,3 +618,27 @@ def upgraderepo(ui, repo, dryrun=False):
 if dryrun:
 ui.warn(_('dry run requested; not continuing with upgrade\n'))
 return 1
+
+ui.warn(_('starting repository upgrade\n'))
+
+# Now we're finally at the actual upgrade part.
+with repo.wlock():
+with repo.lock():
+ui.write(_('source repository locked and read-only\n'))
+# It is easier to create a new repo than to instantiate all the
+# components (like the store) separately.
+tmppath = tempfile.mkdtemp(prefix='upgrade.', dir=repo.path)
+try:
+ui.write(_('creating temporary repository to stage migrated '
+   'data: %s\n') % tmppath)
+dstrepo = localrepo.localrepository(repo.baseui,
+path=tmppath,
+create=True)
+
+with dstrepo.wlock():
+with dstrepo.lock():
+_upgraderepo(ui, repo, dstrepo, newreqs)
+
+finally:
+ui.write(_('removing temporary repository %s\n') % tmppath)
+repo.vfs.rmtree(tmppath, forcibly=True)
diff --git a/tests/test-upgrade-repo.t b/tests/test-upgrade-repo.t
--- a/tests/test-upgrade-repo.t
+++ b/tests/test-upgrade-repo.t
@@ -6,6 +6,14 @@
   checking repository requirements...
   preserving repository requirements: dotencode, fncache, generaldelta, 
revlogv1, store
   no notable upgrade changes identified
+  starting repository upgrade
+  source repository locked and read-only
+  creating temporary repository to stage migrated data: 
$TESTTMP/empty/.hg/upgrade.* (glob)
+  starting in-place swap of repository data
+  (clients may error or see inconsistent repository data until this operation 
completes)
+  replaced files will 

[PATCH 08 of 10] repair: migrate revlogs during upgrade

[Gregory Szorc]

# HG changeset patch
# User Gregory Szorc 
# Date 1478393405 25200
#  Sat Nov 05 17:50:05 2016 -0700
# Node ID d2261c558ca9639fb81c182de15d75151cbad0f9
# Parent  958bcf2577608bbb6d8ae078cde0ca451f3ab31a
repair: migrate revlogs during upgrade

Our next step for in-place upgrade is to migrate store data. Revlogs
are the biggest source of data within the store and a store is useless
without them, so we implement their migration first.

Our strategy for migrating revlogs is to walk the store and call
`revlog.copy()` on each revlog. There are some minor complications.

Because revlogs have different storage options (e.g. changelog has
generaldelta and delta chains disabled), we need to obtain the
correct class of revlog so inserted data is encoded properly for its
type.

Because manifests are converted after filelogs and because manifest
conversion can take a long time when large manifests are in play,
a naive progress bar for revlog count was misleading, as it effectively
got to 99% and froze there when processing the manifest. So, there is
a first pass to count revisions and use revisions in the progress bar.
The extra code is somewhat annoying. But this pass serves a secondary
useful purpose: ensuring we can open all revlogs that will be copied.
We don't want to spend several minutes copying revlogs only to
encounter a permissions error or some such later.

As part of this change, we also add swapping of the store directory
to the upgrade function. After revlogs are converted, we move the
old store into the backup directory then move the temporary repo's
store into the old store's location. On well-behaved systems, this
should be 2 atomic operations and the window of inconsistency show be
very narrow.

There are still a number of improvements that need to be made for
store copying...

diff --git a/mercurial/repair.py b/mercurial/repair.py
--- a/mercurial/repair.py
+++ b/mercurial/repair.py
@@ -11,15 +11,19 @@ from __future__ import absolute_import
 import errno
 import hashlib
 import tempfile
+import time
 
 from .i18n import _
 from .node import short
 from . import (
 bundle2,
 changegroup,
+changelog,
 error,
 exchange,
+manifest,
 obsolete,
+revlog,
 scmutil,
 util,
 )
@@ -537,6 +541,87 @@ def upgradesummarizeactions(repo, action
 
 return l, handled
 
+def _revlogfrompath(repo, path):
+"""Obtain a revlog from a repo path.
+
+An instance of the appropriate class is returned.
+"""
+if path == '00changelog.i':
+return changelog.changelog(repo.svfs)
+elif path.endswith('00manifest.i'):
+mandir = path[:-len('00manifest.i')]
+return manifest.manifestrevlog(repo.svfs, dir=mandir)
+else:
+# Filelogs don't do anything special with settings. So we can use a
+# vanilla revlog.
+return revlog.revlog(repo.svfs, path)
+
+def _copyrevlogs(ui, srcrepo, dstrepo, tr):
+"""Copy revlogs between 2 repos.
+
+Full decoding/encoding is performed on both ends, ensuring that revlog
+settings on the destination are honored.
+"""
+rlcount = 0
+revcount = 0
+srcsize = 0
+dstsize = 0
+
+# Perform a pass to collect metadata. This validates we can open all
+# source files and allows a unified progress bar to be displayed.
+for unencoded, encoded, size in srcrepo.store.walk():
+srcsize += size
+if unencoded.endswith('.d'):
+continue
+rl = _revlogfrompath(srcrepo, unencoded)
+rlcount += 1
+revcount += len(rl)
+
+ui.write(_('migrating %d revlogs containing %d revisions (%d bytes)\n') %
+ (rlcount, revcount, srcsize))
+
+if not rlcount:
+return
+
+# Used to keep track of progress.
+convertedcount = [0]
+def oncopiedrevision(rl, rev, node):
+convertedcount[0] += 1
+srcrepo.ui.progress(_('revisions'), convertedcount[0], total=revcount)
+
+# Do the actual copying.
+# FUTURE this operation can be farmed off to worker processes.
+seen = set()
+for unencoded, encoded, size in srcrepo.store.walk():
+if unencoded.endswith('.d'):
+continue
+
+ui.progress(_('revisions'), convertedcount[0], total=revcount)
+
+oldrl = _revlogfrompath(srcrepo, unencoded)
+newrl = _revlogfrompath(dstrepo, unencoded)
+
+if isinstance(oldrl, manifest.manifestrevlog) and 'm' not in seen:
+seen.add('m')
+ui.write(_('migrating manifests...\n'))
+elif isinstance(oldrl, changelog.changelog) and 'c' not in seen:
+seen.add('c')
+ui.write(_('migrating changelog...\n'))
+elif 'f' not in seen:
+seen.add('f')
+ui.write(_('migrating file histories...\n'))
+
+ui.note(_('copying %d revisions from %s\n') % (len(oldrl), unencoded))
+oldrl.clone(tr, newrl, addrevisioncb=oncopiedrevision)
+
+dstsize += 

[PATCH 03 of 10] commands: stub for debugupgraderepo command

[Gregory Szorc]

# HG changeset patch
# User Gregory Szorc 
# Date 1478391613 25200
#  Sat Nov 05 17:20:13 2016 -0700
# Node ID 9daec9c7adabe8c84cf2c01fc938e010ee4884d6
# Parent  ed3241d8b00e476818ff1aec3db0136bf960de35
commands: stub for debugupgraderepo command

Currently, if Mercurial introduces a new repository/store feature or
changes behavior of an existing feature, users must perform an
`hg clone` to create a new repository with hopefully the
correct/optimal settings. Unfortunately, even `hg clone` may not
give the correct results. For example, if you do a local `hg clone`,
you may get hardlinks to revlog files that inherit the old state.
If you `hg clone` from a remote or `hg clone --pull`, changegroup
application may bypass some optimization, such as converting to
generaldelta.

Optimizing a repository is harder than it seems and requires more
than a simple `hg` command invocation.

This patch starts the process of changing that. We introduce
`hg debugupgraderepo`, a command that performs an in-place upgrade
of a repository to use new, optimal features. The command is just
a stub right now. Features will be added in subsequent patches.

diff --git a/mercurial/commands.py b/mercurial/commands.py
--- a/mercurial/commands.py
+++ b/mercurial/commands.py
@@ -3747,6 +3747,17 @@ def debugtemplate(ui, repo, tmpl, **opts
 displayer.show(repo[r], **props)
 displayer.close()
 
+@command('debugupgraderepo', dryrunopts)
+def debugupgraderepo(ui, repo, **opts):
+"""upgrade a repository to use different features
+
+During the upgrade, the repository will be locked and no writes will be
+allowed.
+
+At times during the upgrade, the repository may not be readable.
+"""
+raise error.Abort(_('not yet implemented'))
+
 @command('debugwalk', walkopts, _('[OPTION]... [FILE]...'), inferrepo=True)
 def debugwalk(ui, repo, *pats, **opts):
 """show how files match on given patterns"""
diff --git a/tests/test-completion.t b/tests/test-completion.t
--- a/tests/test-completion.t
+++ b/tests/test-completion.t
@@ -109,6 +109,7 @@ Show debug commands if there are no othe
   debugsub
   debugsuccessorssets
   debugtemplate
+  debugupgraderepo
   debugwalk
   debugwireargs
 
@@ -274,6 +275,7 @@ Show all commands + options
   debugsub: rev
   debugsuccessorssets: 
   debugtemplate: rev, define
+  debugupgraderepo: dry-run
   debugwalk: include, exclude
   debugwireargs: three, four, five, ssh, remotecmd, insecure
   files: rev, print0, include, exclude, template, subrepos
diff --git a/tests/test-help.t b/tests/test-help.t
--- a/tests/test-help.t
+++ b/tests/test-help.t
@@ -917,6 +917,8 @@ Test list of internal help commands
  show set of successors for revision
debugtemplate
  parse and apply a template
+   debugupgraderepo
+ upgrade a repository to use different features
debugwalk show how files match on given patterns
debugwireargs
  (no help text available)
diff --git a/tests/test-upgrade-repo.t b/tests/test-upgrade-repo.t
new file mode 100644
--- /dev/null
+++ b/tests/test-upgrade-repo.t
@@ -0,0 +1,5 @@
+  $ hg init empty
+  $ cd empty
+  $ hg debugupgraderepo
+  abort: not yet implemented
+  [255]
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 10 of 10] repair: clean up stale lock file from store backup

[Gregory Szorc]

# HG changeset patch
# User Gregory Szorc 
# Date 1478392394 25200
#  Sat Nov 05 17:33:14 2016 -0700
# Node ID 0e130e8d2156d9f2523c711ef4fefe4ba33e6818
# Parent  3d4dd237b705479f8c7475b821ae719893381fa8
repair: clean up stale lock file from store backup

So inline comment for reasons.

diff --git a/mercurial/repair.py b/mercurial/repair.py
--- a/mercurial/repair.py
+++ b/mercurial/repair.py
@@ -716,6 +716,12 @@ def _upgraderepo(ui, srcrepo, dstrepo, r
 ui.write(_('store replacement complete; repository was inconsistent for '
'%0.1fs\n') % elapsed)
 
+# The lock file from the old store won't be removed because nothing has a
+# reference to its new location. So clean it up manually. Alternatively, we
+# could update srcrepo.svfs and other variables to point to the new
+# location. This is simpler.
+backupvfs.unlink('store/lock')
+
 def upgraderepo(ui, repo, dryrun=False):
 """Upgrade a repository in place."""
 # Avoid cycle: cmdutil -> repair -> localrepo -> cmdutil
diff --git a/tests/test-upgrade-repo.t b/tests/test-upgrade-repo.t
--- a/tests/test-upgrade-repo.t
+++ b/tests/test-upgrade-repo.t
@@ -209,7 +209,6 @@ old store should be backed up
   00manifest.i
   data
   fncache
-  lock
   phaseroots
   undo
   undo.backup.fncache
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 05 of 10] repair: obtain and validate requirements for upgraded repo

[Gregory Szorc]

# HG changeset patch
# User Gregory Szorc 
# Date 1478394961 25200
#  Sat Nov 05 18:16:01 2016 -0700
# Node ID b768004ef2db9c2e6dd267997e9e0c011f1b732a
# Parent  7518e68e2f8276e85fb68174b3055a9dd16c665d
repair: obtain and validate requirements for upgraded repo

Not all existing repositories can be upgraded. Not all requirements
are supported in new repositories. Some transitions between
repository requirements (notably removing a requirement) are not
supported.

This patch adds code for validating that the requirements transition
for repository upgrades is acceptable and aborts if it isn't.
Functionality is split into various functions to give extensions an
opportunity to monkeypatch.

diff --git a/mercurial/repair.py b/mercurial/repair.py
--- a/mercurial/repair.py
+++ b/mercurial/repair.py
@@ -401,6 +401,85 @@ def upgradefinddeficiencies(repo):
 
 return l, actions
 
+def upgradesupporteddestrequirements(repo):
+"""Obtain requirements that upgrade supports in the destination.
+
+Extensions should monkeypatch this to add their custom requirements.
+"""
+return set([
+'dotencode',
+'fncache',
+'generaldelta',
+'manifestv2',
+'revlogv1',
+'store',
+'treemanifest',
+])
+
+def upgraderequiredsourcerequirements(repo):
+"""Obtain requirements that must be present in the source repository."""
+return set([
+'revlogv1',
+'store',
+])
+
+def upgradeallowednewrequirements(repo):
+"""Obtain requirements that can be added to a repository.
+
+This is used to disallow proposed requirements from being added when
+they weren't present before.
+
+We use a whitelist of allowed requirement additions instead of a
+blacklist of known bad additions because the whitelist approach is
+safer and will prevent future, unknown requirements from accidentally
+being added.
+"""
+return set([
+'dotencode',
+'fncache',
+'generaldelta',
+])
+
+def upgradereporequirements(repo):
+"""Obtain and validate requirements for repository after upgrade.
+
+Should raise ``Abort`` if existing or new requirements aren't sufficient.
+"""
+# Avoid cycle: cmdutil -> repair -> localrepo -> cmdutil
+from . import localrepo
+
+existing = repo.requirements
+
+missingreqs = upgraderequiredsourcerequirements(repo) - existing
+if missingreqs:
+raise error.Abort(_('cannot upgrade repository; requirement '
+'missing: %s') % ', '.join(sorted(missingreqs)))
+
+createreqs = localrepo.newreporequirements(repo)
+
+# This might be overly restrictive. It is definitely safer to enforce this
+# as it prevents unwanted deletion of requirements.
+removedreqs = existing - createreqs
+if removedreqs:
+raise error.Abort(_('cannot upgrade repository; requirement would '
+'be removed: %s') % ', '.join(sorted(removedreqs)))
+
+unsupportedreqs = createreqs - upgradesupporteddestrequirements(repo)
+if unsupportedreqs:
+raise error.Abort(_('cannot upgrade repository; new requirement not '
+   'supported: %s') %
+  ', '.join(sorted(unsupportedreqs)))
+
+# Alternatively, we could silently remove the disallowed requirement
+# instead of aborting.
+noaddreqs = createreqs - existing - upgradeallowednewrequirements(repo)
+if noaddreqs:
+raise error.Abort(_('cannot upgrade repository; proposed new '
+'requirement cannot be added: %s') %
+  ', '.join(sorted(noaddreqs)))
+
+return createreqs
+
 def upgraderepo(ui, repo, dryrun=False):
 """Upgrade a repository in place."""
 repo = repo.unfiltered()
@@ -414,3 +493,17 @@ def upgraderepo(ui, repo, dryrun=False):
 ui.write('* %s\n' % d)
 else:
 ui.write(_('no obvious deficiencies found in existing repository\n'))
+
+ui.write(_('\n'))
+ui.write(_('checking repository requirements...\n'))
+
+newreqs = upgradereporequirements(repo)
+# We don't drop requirements.
+assert not len(repo.requirements - newreqs)
+
+ui.write(_('preserving repository requirements: %s\n') %
+ ', '.join(sorted(repo.requirements & newreqs)))
+if newreqs - repo.requirements:
+ui.write(_('adding repository requirements: %s\n') %
+ ', '.join(sorted(newreqs - repo.requirements)))
+
diff --git a/tests/test-upgrade-repo.t b/tests/test-upgrade-repo.t
--- a/tests/test-upgrade-repo.t
+++ b/tests/test-upgrade-repo.t
@@ -2,6 +2,9 @@
   $ cd empty
   $ hg debugupgraderepo
   no obvious deficiencies found in existing repository
+  
+  checking repository requirements...
+  preserving repository requirements: dotencode, fncache, generaldelta, 
revlogv1, store
 
 Various sub-optimal detections work
 
@@ -16,3 +19,57 @@ Various 

Re: [PATCH 2 of 8] py3: add os.fsdecode() as pycompat.fsdecode()

[Yuya Nishihara]

On Sun, 06 Nov 2016 04:46:19 +0530, Pulkit Goyal wrote:
> # HG changeset patch
> # User Pulkit Goyal <7895pul...@gmail.com>
> # Date 1478382160 -19800
> #  Sun Nov 06 03:12:40 2016 +0530
> # Node ID 972b0c0175fc402ffe8beac276b737b32ce0df0b
> # Parent  2d456de3a41ab961da8ae71dfefccdc9d7febe78
> py3: add os.fsdecode() as pycompat.fsdecode()
> 
> We need to use os.fsdecode() but this was not present in Python 2. So added
> the function in pycompat.py

Queued 1, 2 and 5 as 2 and 1+5, thanks. We can't split 1 and 5 since they
depend on each other.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 8 of 8] py3: have bytes version of sys.argv

[Yuya Nishihara]

On Sun, 06 Nov 2016 04:46:25 +0530, Pulkit Goyal wrote:
> # HG changeset patch
> # User Pulkit Goyal <7895pul...@gmail.com>
> # Date 1478387186 -19800
> #  Sun Nov 06 04:36:26 2016 +0530
> # Node ID b5fc4e71286dd4f6e4f38e0b9fb17f51f1e3
> # Parent  6eed3ee0df425da61d03bfe024dd082f3176ce5d
> py3: have bytes version of sys.argv
> 
> sys.argv returns unicodes on Python 3. We need a bytes version for us.
> There was also a python bug/feature request which wanted then to implement
> one. They rejected and it is quoted in one of the comments that we can use
> fsencode() to get a bytes version of sys.argv. Though not sure about its
> correctness.
> 
> Link to the comment: http://bugs.python.org/issue8776#msg217416
> 
> After this patch we will have pycompat.sysargv which will return us bytes
> version of sys.argv. If this patch goes in, i will like to make transformer
> rewrite sys.argv with pycompat.argv because there are lot of occurences.
> 
> diff -r 6eed3ee0df42 -r b5fc4e71286d mercurial/pycompat.py
> --- a/mercurial/pycompat.py   Sun Nov 06 04:17:19 2016 +0530
> +++ b/mercurial/pycompat.py   Sun Nov 06 04:36:26 2016 +0530
> @@ -41,6 +41,7 @@
>  osname = os.name.encode('ascii')
>  ospathsep = os.pathsep.encode('ascii')
>  ossep = os.sep.encode('ascii')
> +sysargv = list(map(os.fsencode, sys.argv))

Looks good to me. Can you add a comment why we can use os.fsencode() here
(and the weirdness of Python 3 on Unix.) We might need a Windows workaround
because the situation is slightly different, but we wouldn't want to care
for now.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH V2] help: show help for disabled extensions(issue5228)

[Pulkit Goyal]

# HG changeset patch
# User Pulkit Goyal <7895pul...@gmail.com>
# Date 1478395471 -19800
#  Sun Nov 06 06:54:31 2016 +0530
# Node ID 6d27c42a7e01adcc0eabf88c623390c85c4a5943
# Parent  b5fc4e71286dd4f6e4f38e0b9fb17f51f1e3
help: show help for disabled extensions(issue5228)

This patch does not exactly solve issue5228 but it results in a better
condition on this issue. For disabled extensions, we used to parse the
module and get the first occurences of docstring and then retrun the first
line of that as an introductory heading of extension. This is what we get
today.
This patch returns the whole docstring of the module as a help for extension,
which is more informative. There are some modules which don't have much
docstring at top level except the heading so those are unaffected by this
change. To follow the existing trend of showing commands either we have to
load the extension or have a very ugly parsing method which don't even assure
correctness.

diff -r b5fc4e71286d -r 6d27c42a7e01 mercurial/extensions.py
--- a/mercurial/extensions.py   Sun Nov 06 04:36:26 2016 +0530
+++ b/mercurial/extensions.py   Sun Nov 06 06:54:31 2016 +0530
@@ -426,7 +426,7 @@
 file.close()
 
 if doc: # extracting localized synopsis
-return gettext(doc).splitlines()[0]
+return gettext(doc)
 else:
 return _('(no help text available)')
 
@@ -448,7 +448,7 @@
 for name, path in paths.iteritems():
 doc = _disabledhelp(path)
 if doc:
-exts[name] = doc
+exts[name] = doc.splitlines()[0]
 
 return exts
 
diff -r b5fc4e71286d -r 6d27c42a7e01 tests/test-extension.t
--- a/tests/test-extension.tSun Nov 06 04:36:26 2016 +0530
+++ b/tests/test-extension.tSun Nov 06 06:54:31 2016 +0530
@@ -1045,6 +1045,61 @@
   $ hg help patchbomb
   patchbomb extension - command to send changesets as (a series of) patch 
emails
   
+  The series is started off with a "[PATCH 0 of N]" introduction, which
+  describes the series as a whole.
+  
+  Each patch email has a Subject line of "[PATCH M of N] ...", using the first
+  line of the changeset description as the subject text. The message contains
+  two or three body parts:
+  
+  - The changeset description.
+  - [Optional] The result of running diffstat on the patch.
+  - The patch itself, as generated by 'hg export'.
+  
+  Each message refers to the first in the series using the In-Reply-To and
+  References headers, so they will show up as a sequence in threaded mail and
+  news readers, and in mail archives.
+  
+  To configure other defaults, add a section like this to your configuration
+  file:
+  
+[email]
+from = My Name 
+to = recipient1, recipient2, ...
+cc = cc1, cc2, ...
+bcc = bcc1, bcc2, ...
+reply-to = address1, address2, ...
+  
+  Use "[patchbomb]" as configuration section name if you need to override 
global
+  "[email]" address settings.
+  
+  Then you can use the 'hg email' command to mail a series of changesets as a
+  patchbomb.
+  
+  You can also either configure the method option in the email section to be a
+  sendmail compatible mailer or fill out the [smtp] section so that the
+  patchbomb extension can automatically send patchbombs directly from the
+  commandline. See the [email] and [smtp] sections in hgrc(5) for details.
+  
+  By default, 'hg email' will prompt for a "To" or "CC" header if you do not
+  supply one via configuration or the command line.  You can override this to
+  never prompt by configuring an empty value:
+  
+[email]
+cc =
+  
+  You can control the default inclusion of an introduction message with the
+  "patchbomb.intro" configuration option. The configuration is always
+  overwritten by command line flags like --intro and --desc:
+  
+[patchbomb]
+intro=auto   # include introduction message if more than 1 patch (default)
+intro=never  # never include an introduction message
+intro=always # always include an introduction message
+  
+  You can set patchbomb to always ask for confirmation by setting
+  "patchbomb.confirm" to true.
+  
   (use 'hg help extensions' for information on enabling extensions)
 
 
diff -r b5fc4e71286d -r 6d27c42a7e01 tests/test-qrecord.t
--- a/tests/test-qrecord.t  Sun Nov 06 04:36:26 2016 +0530
+++ b/tests/test-qrecord.t  Sun Nov 06 06:54:31 2016 +0530
@@ -9,6 +9,9 @@
   record extension - commands to interactively select changes for
   commit/qrefresh (DEPRECATED)
   
+  The feature provided by this extension has been moved into core Mercurial as
+  'hg commit --interactive'.
+  
   (use 'hg help extensions' for information on enabling extensions)
 
 help qrecord (no record)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH STABLE V2] hgweb: cache fctx.parents() in annotate command (issue5414)

[Yuya Nishihara]

On Sat, 05 Nov 2016 09:41:14 -0700, Gregory Szorc wrote:
> # HG changeset patch
> # User Gregory Szorc 
> # Date 1478363887 25200
> #  Sat Nov 05 09:38:07 2016 -0700
> # Branch stable
> # Node ID 1258a74819e5c2b4904d11c71231e3b3f33a0aa1
> # Parent  e5cc44ea12de681d971fcbebb65a7fb71fd1c3c7
> hgweb: cache fctx.parents() in annotate command (issue5414)

Queued this, thanks.

> Looking at the code of basefilectx.parents(), there is room for
> further improvements. Notably, we still perform redundant calls to
> filelog.renamed() and basefilectx._parentfilectx(). And
> basefilectx.annotate() also makes similar calls, so there is potential
> for object reuse. However, introducing caches here are not appropriate
> for the stable branch.

Perhaps fctx.parents() can be property-cached, but we'll need to drop
uninteresting chains of parents in fctx.annotate().
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH] help: show help for disabled extensions(issue5228)

[Pulkit Goyal]

# HG changeset patch
# User Pulkit Goyal <7895pul...@gmail.com>
# Date 1478395471 -19800
#  Sun Nov 06 06:54:31 2016 +0530
# Node ID 691d3960ba61d07db8d34f5ca2c148ecb32f4f7c
# Parent  b5fc4e71286dd4f6e4f38e0b9fb17f51f1e3
help: show help for disabled extensions(issue5228)

This patch does not exactly solve issue5228 but it results in a better
condition on this issue. For disabled extensions, we used to parse the
module and get the first occurences of docstring and then retrun the first
line of that as an introductory heading of extension. This is what we get
today.
This patch returns the whole docstring of the module as a help for extension,
which is more informative. There are some modules which don't have much
docstring at top level except the heading so those are unaffected by this
change. To follow the existing trend of showing commands either we have to
load the extension or have a very ugly parsing method which don't even assure
correctness.

diff -r b5fc4e71286d -r 691d3960ba61 mercurial/extensions.py
--- a/mercurial/extensions.py   Sun Nov 06 04:36:26 2016 +0530
+++ b/mercurial/extensions.py   Sun Nov 06 06:54:31 2016 +0530
@@ -426,7 +426,7 @@
 file.close()
 
 if doc: # extracting localized synopsis
-return gettext(doc).splitlines()[0]
+return gettext(doc)
 else:
 return _('(no help text available)')
 
@@ -446,7 +446,7 @@
 
 exts = {}
 for name, path in paths.iteritems():
-doc = _disabledhelp(path)
+doc = _disabledhelp(path).splitlines()[0]
 if doc:
 exts[name] = doc
 
diff -r b5fc4e71286d -r 691d3960ba61 tests/test-extension.t
--- a/tests/test-extension.tSun Nov 06 04:36:26 2016 +0530
+++ b/tests/test-extension.tSun Nov 06 06:54:31 2016 +0530
@@ -1045,6 +1045,61 @@
   $ hg help patchbomb
   patchbomb extension - command to send changesets as (a series of) patch 
emails
   
+  The series is started off with a "[PATCH 0 of N]" introduction, which
+  describes the series as a whole.
+  
+  Each patch email has a Subject line of "[PATCH M of N] ...", using the first
+  line of the changeset description as the subject text. The message contains
+  two or three body parts:
+  
+  - The changeset description.
+  - [Optional] The result of running diffstat on the patch.
+  - The patch itself, as generated by 'hg export'.
+  
+  Each message refers to the first in the series using the In-Reply-To and
+  References headers, so they will show up as a sequence in threaded mail and
+  news readers, and in mail archives.
+  
+  To configure other defaults, add a section like this to your configuration
+  file:
+  
+[email]
+from = My Name 
+to = recipient1, recipient2, ...
+cc = cc1, cc2, ...
+bcc = bcc1, bcc2, ...
+reply-to = address1, address2, ...
+  
+  Use "[patchbomb]" as configuration section name if you need to override 
global
+  "[email]" address settings.
+  
+  Then you can use the 'hg email' command to mail a series of changesets as a
+  patchbomb.
+  
+  You can also either configure the method option in the email section to be a
+  sendmail compatible mailer or fill out the [smtp] section so that the
+  patchbomb extension can automatically send patchbombs directly from the
+  commandline. See the [email] and [smtp] sections in hgrc(5) for details.
+  
+  By default, 'hg email' will prompt for a "To" or "CC" header if you do not
+  supply one via configuration or the command line.  You can override this to
+  never prompt by configuring an empty value:
+  
+[email]
+cc =
+  
+  You can control the default inclusion of an introduction message with the
+  "patchbomb.intro" configuration option. The configuration is always
+  overwritten by command line flags like --intro and --desc:
+  
+[patchbomb]
+intro=auto   # include introduction message if more than 1 patch (default)
+intro=never  # never include an introduction message
+intro=always # always include an introduction message
+  
+  You can set patchbomb to always ask for confirmation by setting
+  "patchbomb.confirm" to true.
+  
   (use 'hg help extensions' for information on enabling extensions)
 
 
diff -r b5fc4e71286d -r 691d3960ba61 tests/test-qrecord.t
--- a/tests/test-qrecord.t  Sun Nov 06 04:36:26 2016 +0530
+++ b/tests/test-qrecord.t  Sun Nov 06 06:54:31 2016 +0530
@@ -9,6 +9,9 @@
   record extension - commands to interactively select changes for
   commit/qrefresh (DEPRECATED)
   
+  The feature provided by this extension has been moved into core Mercurial as
+  'hg commit --interactive'.
+  
   (use 'hg help extensions' for information on enabling extensions)
 
 help qrecord (no record)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 8 of 8] py3: have bytes version of sys.argv

[Pulkit Goyal]

# HG changeset patch
# User Pulkit Goyal <7895pul...@gmail.com>
# Date 1478387186 -19800
#  Sun Nov 06 04:36:26 2016 +0530
# Node ID b5fc4e71286dd4f6e4f38e0b9fb17f51f1e3
# Parent  6eed3ee0df425da61d03bfe024dd082f3176ce5d
py3: have bytes version of sys.argv

sys.argv returns unicodes on Python 3. We need a bytes version for us.
There was also a python bug/feature request which wanted then to implement
one. They rejected and it is quoted in one of the comments that we can use
fsencode() to get a bytes version of sys.argv. Though not sure about its
correctness.

Link to the comment: http://bugs.python.org/issue8776#msg217416

After this patch we will have pycompat.sysargv which will return us bytes
version of sys.argv. If this patch goes in, i will like to make transformer
rewrite sys.argv with pycompat.argv because there are lot of occurences.

diff -r 6eed3ee0df42 -r b5fc4e71286d mercurial/pycompat.py
--- a/mercurial/pycompat.py Sun Nov 06 04:17:19 2016 +0530
+++ b/mercurial/pycompat.py Sun Nov 06 04:36:26 2016 +0530
@@ -41,6 +41,7 @@
 osname = os.name.encode('ascii')
 ospathsep = os.pathsep.encode('ascii')
 ossep = os.sep.encode('ascii')
+sysargv = list(map(os.fsencode, sys.argv))
 
 def sysstr(s):
 """Return a keyword str to be passed to Python functions such as
@@ -89,6 +90,7 @@
 osname = os.name
 ospathsep = os.pathsep
 ossep = os.sep
+sysargv = sys.argv
 
 stringio = io.StringIO
 empty = _queue.Empty
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 8] py3: add os.fsdecode() as pycompat.fsdecode()

[Pulkit Goyal]

# HG changeset patch
# User Pulkit Goyal <7895pul...@gmail.com>
# Date 1478382160 -19800
#  Sun Nov 06 03:12:40 2016 +0530
# Node ID 972b0c0175fc402ffe8beac276b737b32ce0df0b
# Parent  2d456de3a41ab961da8ae71dfefccdc9d7febe78
py3: add os.fsdecode() as pycompat.fsdecode()

We need to use os.fsdecode() but this was not present in Python 2. So added
the function in pycompat.py

diff -r 2d456de3a41a -r 972b0c0175fc mercurial/pycompat.py
--- a/mercurial/pycompat.py Sun Nov 06 02:59:43 2016 +0530
+++ b/mercurial/pycompat.py Sun Nov 06 03:12:40 2016 +0530
@@ -36,6 +36,7 @@
 import functools
 import os
 fsencode = os.fsencode
+fsdecode = os.fsdecode
 
 def sysstr(s):
 """Return a keyword str to be passed to Python functions such as
@@ -76,6 +77,11 @@
 raise TypeError(
 "expect str, not %s" % type(filename).__name__)
 
+# In Python 2, fsdecode() have a very chances to recieve bytes. So it's
+# better not to touch Python 2 part as its already working fine.
+def fsdecode(filename):
+return filename
+
 stringio = io.StringIO
 empty = _queue.Empty
 queue = _queue.Queue
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 6 of 8] py3: use pycompat.ossep at certain places

[Pulkit Goyal]

# HG changeset patch
# User Pulkit Goyal <7895pul...@gmail.com>
# Date 1478385633 -19800
#  Sun Nov 06 04:10:33 2016 +0530
# Node ID d9461776be8ccc38def14ef34c3337099fda9b23
# Parent  0c892b7a70b56a27d485562b4191eb44da625f5b
py3: use pycompat.ossep at certain places

Certain instances of os.sep has been converted to pycompat.ossep where it was
sure to use bytes only. There are more such instances which needs some more
attention and will get surely.

diff -r 0c892b7a70b5 -r d9461776be8c mercurial/commands.py
--- a/mercurial/commands.py Sun Nov 06 03:57:34 2016 +0530
+++ b/mercurial/commands.py Sun Nov 06 04:10:33 2016 +0530
@@ -61,6 +61,7 @@
 phases,
 policy,
 pvec,
+pycompat,
 repair,
 revlog,
 revset,
@@ -3160,7 +3161,7 @@
 if os.path.isdir(spec):
 spec += '/'
 spec = spec[len(rootdir):]
-fixpaths = os.sep != '/'
+fixpaths = pycompat.ossep != '/'
 if fixpaths:
 spec = spec.replace(os.sep, '/')
 speclen = len(spec)
@@ -3755,7 +3756,7 @@
 if not items:
 return
 f = lambda fn: fn
-if ui.configbool('ui', 'slash') and os.sep != '/':
+if ui.configbool('ui', 'slash') and pycompat.ossep != '/':
 f = lambda fn: util.normpath(fn)
 fmt = 'f  %%-%ds  %%-%ds  %%s' % (
 max([len(abs) for abs in items]),
diff -r 0c892b7a70b5 -r d9461776be8c mercurial/dirstate.py
--- a/mercurial/dirstate.py Sun Nov 06 03:57:34 2016 +0530
+++ b/mercurial/dirstate.py Sun Nov 06 04:10:33 2016 +0530
@@ -21,6 +21,7 @@
 osutil,
 parsers,
 pathutil,
+pycompat,
 scmutil,
 util,
 )
@@ -215,7 +216,7 @@
 
 @propertycache
 def _slash(self):
-return self._ui.configbool('ui', 'slash') and os.sep != '/'
+return self._ui.configbool('ui', 'slash') and pycompat.ossep != '/'
 
 @propertycache
 def _checklink(self):
diff -r 0c892b7a70b5 -r d9461776be8c mercurial/pure/osutil.py
--- a/mercurial/pure/osutil.py  Sun Nov 06 03:57:34 2016 +0530
+++ b/mercurial/pure/osutil.py  Sun Nov 06 04:10:33 2016 +0530
@@ -14,7 +14,11 @@
 import stat as statmod
 import sys
 
-from . import policy
+from . import (
+policy,
+pycompat,
+)
+
 modulepolicy = policy.policy
 policynocffi = policy.policynocffi
 
@@ -51,8 +55,8 @@
 '''
 result = []
 prefix = path
-if not prefix.endswith(os.sep):
-prefix += os.sep
+if not prefix.endswith(pycompat.ossep):
+prefix += pycompat.ossep
 names = os.listdir(path)
 names.sort()
 for fn in names:
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 1 of 8] py3: make util.datapath a bytes variable

[Pulkit Goyal]

# HG changeset patch
# User Pulkit Goyal <7895pul...@gmail.com>
# Date 1478381383 -19800
#  Sun Nov 06 02:59:43 2016 +0530
# Node ID 2d456de3a41ab961da8ae71dfefccdc9d7febe78
# Parent  a19a17a662f669546e75d51ec3b381beb51e737c
py3: make util.datapath a bytes variable

In this patch we make util.datapath a bytes variables.

diff -r a19a17a662f6 -r 2d456de3a41a mercurial/util.py
--- a/mercurial/util.py Fri Nov 04 20:22:37 2016 -0700
+++ b/mercurial/util.py Sun Nov 06 02:59:43 2016 +0530
@@ -938,6 +938,9 @@
 else:
 datapath = os.path.dirname(__file__)
 
+if not isinstance(datapath, bytes):
+datapath = pycompat.fsencode(datapath)
+
 i18n.setdatapath(datapath)
 
 _hgexecutable = None
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 4 of 8] py3: have pycompat.ospathsep and pycompat.ossep

[Pulkit Goyal]

# HG changeset patch
# User Pulkit Goyal <7895pul...@gmail.com>
# Date 1478384084 -19800
#  Sun Nov 06 03:44:44 2016 +0530
# Node ID 95b462cda94a4cf9c399a3f3bae348923855542d
# Parent  20ee6f9c56233f7c43ce73d6d976ef6817fb0695
py3: have pycompat.ospathsep and pycompat.ossep

We needed bytes version of os.sep and os.pathsep in py3 as they return
unicodes.

diff -r 20ee6f9c5623 -r 95b462cda94a mercurial/pycompat.py
--- a/mercurial/pycompat.py Sun Nov 06 03:33:22 2016 +0530
+++ b/mercurial/pycompat.py Sun Nov 06 03:44:44 2016 +0530
@@ -39,6 +39,8 @@
 fsdecode = os.fsdecode
 #A bytes version of os.name.
 osname = os.name.encode('ascii')
+ospathsep = os.pathsep.encode('ascii')
+ossep = os.sep.encode('ascii')
 
 def sysstr(s):
 """Return a keyword str to be passed to Python functions such as
@@ -85,6 +87,8 @@
 return filename
 
 osname = os.name
+ospathsep = os.pathsep
+ossep = os.sep
 
 stringio = io.StringIO
 empty = _queue.Empty
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 3 of 8] py3: add a bytes version of os.name

[Pulkit Goyal]

# HG changeset patch
# User Pulkit Goyal <7895pul...@gmail.com>
# Date 1478383402 -19800
#  Sun Nov 06 03:33:22 2016 +0530
# Node ID 20ee6f9c56233f7c43ce73d6d976ef6817fb0695
# Parent  972b0c0175fc402ffe8beac276b737b32ce0df0b
py3: add a bytes version of os.name

os.name returns unicodes on py3. Most of our checks are like
os.name == 'nt'

Because of the transformer, on the right hand side we have b'nt'. The
condition will never satisfy even if os.name returns 'nt' as that will be an
unicode.
We either need to encode every occurence of os.name or have a
new variable which is much cleaner. Now we have pycompat.osname.
There are around 53 occurences of os.name in the codebase which needs to
be replaced by pycompat.osname to support Python 3.

diff -r 972b0c0175fc -r 20ee6f9c5623 mercurial/pycompat.py
--- a/mercurial/pycompat.py Sun Nov 06 03:12:40 2016 +0530
+++ b/mercurial/pycompat.py Sun Nov 06 03:33:22 2016 +0530
@@ -10,6 +10,7 @@
 
 from __future__ import absolute_import
 
+import os
 import sys
 
 ispy3 = (sys.version_info[0] >= 3)
@@ -34,9 +35,10 @@
 if ispy3:
 import builtins
 import functools
-import os
 fsencode = os.fsencode
 fsdecode = os.fsdecode
+#A bytes version of os.name.
+osname = os.name.encode('ascii')
 
 def sysstr(s):
 """Return a keyword str to be passed to Python functions such as
@@ -82,6 +84,8 @@
 def fsdecode(filename):
 return filename
 
+osname = os.name
+
 stringio = io.StringIO
 empty = _queue.Empty
 queue = _queue.Queue
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 2 of 2] hghave: check darcs version more strictly

[Pierre-Yves David]

On 11/05/2016 05:45 AM, Yuya Nishihara wrote:

# HG changeset patch
# User Yuya Nishihara 
# Date 1478319653 -32400
#  Sat Nov 05 13:20:53 2016 +0900
# Node ID 5aa8ce5ded5e10b4830975a2735e0143750f2f7c
# Parent  566a342697a2740f94fd85006f7dac0580bd473b
hghave: check darcs version more strictly

test-convert-darcs.t suddenly started failing on my Debian sid machine. The
reason was Darcs was upgraded from 2.12.0 to 2.12.4 so the original pattern
got to match the last two digits. Fix the pattern to match 2.2+.


This is pushed, thanks.

--
Pierre-Yves David
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 1 of 4] statprof: return state from stop()

[Pierre-Yves David]

On 11/05/2016 05:53 AM, Gregory Szorc wrote:

# HG changeset patch
# User Gregory Szorc 
# Date 1478316157 25200
#  Fri Nov 04 20:22:37 2016 -0700
# Node ID cffcf1865da6bdeb0a59de9ac00c563bbadf1a9b
# Parent  d06c049695e6ad3219e7479c65ce98a2f123e878
statprof: return state from stop()

I don't like global variables. Have stop() return the captured
state so callers can pass data to the display function.


Patch 1 is pushed thanks.

Cheers,

--
Pierre-Yves David
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 2] commands: introduce `hg display`

[Gregory Szorc]

# HG changeset patch
# User Gregory Szorc 
# Date 1478370119 25200
#  Sat Nov 05 11:21:59 2016 -0700
# Node ID 0ebc1e627383bdf017f509a60725c0b39d6d40d9
# Parent  4bce42e7330311b58ecd38a4acbcd2d2dd1351ba
commands: introduce `hg display`

Currently, Mercurial has a number of commands to show information. And,
there are features coming down the pipe that will introduce more
commands for showing information.

Currently, when introducing a new class of data or a view that we
wish to expose to the user, the strategy is to introduce a new command
or overload an existing command, sometimes both. For example, there is
a desire to formalize the wip/smartlog/underway/mine functionality that
many have devised. There is also a desire to introduce a "topics"
concept. In the current model, we'd need a new command for
wip/smartlog/etc (that behaves a lot like a pre-defined alias of `hg
log`). For topics, we'd likely overload `hg topic[s]` to both display
and manipulate topics.

Adding new commands for every pre-defined query doesn't scale well
and pollutes `hg help`. Overloading commands to perform read-only and
write operations is arguably an UX anti-pattern: while having all
functionality for a given concept in one command is nice, having a
single command doing multiple discrete operations is not. Furthermore,
a user may be surprised that a command they thought was read-only
actually changes something.

We discussed this at the Mercurial 4.0 Sprint in Paris and decided that
having a single command where we could hang pre-defined views of
various data would be a good idea. Having such a command would:

* Help prevent an explosion of new query-related commands
* Create a clear separation between read and write operations
  (mitigates footguns)
* Avoids overloading the meaning of commands that manipulate data
  (bookmark, tag, branch, etc) (while we can't take away the
  existing behavior for BC reasons, we now won't introduce this
  behavior on new commands)
* Allows users to discover informational views more easily by
  aggregating them in a single location
* Lowers the barrier to creating the new views (since the barrier
  to creating a top-level command is relatively high)

So, this commit introduces the `hg display` command. This command
accepts a positional argument of the "view" to show. New views
can be registered with a decorator. To prove it works, we implement
the "bookmarks" view, which shows a table of bookmarks and their
associated nodes.

We introduce a new style to hold everything used by `hg display`.

For our initial bookmarks view, the output varies from `hg bookmarks`:

* Padding is performed in the template itself as opposed to Python
* Revision integers are not shown
* shortest() is used to display a 5 character node by default (as
  opposed to static 12 characters)

I chose to implement the "bookmarks" view first because it is simple
and shouldn't invite too much bikeshedding that detracts from the
evaluation of `hg display` itself. But there is an important point
to consider: we now have 2 ways to show a list of bookmarks. I'm not
a fan of introducing multiple ways to do very similar things. So it
might be worth discussing how we wish to tackle this issue for
bookmarks, tags, branches, MQ series, etc.

I also made the choice of explicitly declaring the default display
template not part of the standard BC guarantees. History has shown
that we make mistakes and poor choices with output formatting but
can't fix these mistakes later because random tools are parsing
output and we don't want to break these tools. Optimizing for human
consumption is one of my goals for `hg display`. So, by not covering
the formatting as part of BC, the barrier to future change is much
lower and humans benefit.

At the aforementioned Sprint, we discussed and discarded various
alternatives to `hg display`.

We considered making `hg log ` perform this behavior. The main
reason we can't do this is because a positional argument to `hg log`
can be a file path and if there is a conflict between a path name and
a view name, behavior is ambiguous. We could have introduced
`hg log --view` or similar, but we felt that required too much typing
(we don't want to require a command flag to show a view) and wasn't
very discoverable. Furthermore, `hg log` is optimized for showing
changelog data and there are things that `hg display` could display
that aren't changelog centric.

For the command name, we would have preferred `hg show` because it is
shorter and not ambigious with any other core command. However, a
number of people have created `hg show` as effectively an alias to
`hg export`. And, some were concerned that Git users used to `git show`
being equivalent to `hg export` would be confused by a `hg show` doing
something different.

We also considered `hg view`, but that is already used by the "hgk"
extension.

"display" is an accurate description of what the command does. The
biggest concern with 

[Bug 5416] New: pad(), label(), and color don't interact well

[bugzilla]

https://bz.mercurial-scm.org/show_bug.cgi?id=5416

Bug ID: 5416
   Summary: pad(), label(), and color don't interact well
   Product: Mercurial
   Version: 4.0
  Hardware: All
OS: All
Status: UNCONFIRMED
  Severity: feature
  Priority: wish
 Component: templater
  Assignee: bugzi...@selenic.com
  Reporter: gregory.sz...@gmail.com
CC: mercurial-de...@selenic.com, y...@tcha.org

Say you want to create a template that uses pad() to create aligned columns.
Then say you want to add a label() around a column value so output can be
colorized. Each works on its own but when you combine them, things start to
fall apart.

If you do `pad(label())`, the expanded text from label() may contain
non-printable color escape sequences that count against the length of the
string and affect pad()'s behavior, leading to unexpected results.

If you do `label(pad())`, the labeling/coloring extends over the padding
character, which is likely a space. This could make output look weird due to
coloring of whitespace.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 1 of 6 RFC] manifest: introduce an accessor class for manifests

[Yuya Nishihara]

On Thu, 3 Nov 2016 15:27:37 -0700, Durham Goode wrote:
> # HG changeset patch
> # User Durham Goode 
> # Date 1478208817 25200
> #  Thu Nov 03 14:33:37 2016 -0700
> # Branch stable
> # Node ID 1788ee9e1df92ac94b9be84eac6d16e3bad903a9
> # Parent  b9f7b0c10027764cee77f9c6d61877fcffea837f
> manifest: introduce an accessor class for manifests
> 
> This introduces a revlogaccessor class which can be used to allow multiple
> objects hold an auto-invalidating reference to a revlog, without having to 
> hold
> a reference to the actual repo object. Future patches will switch 
> repo.manifest
> and repo.manifestlog to access the manifest through this accessor. This will 
> fix
> the circular reference caused by manifestlog and manifestctx holding a 
> reference
> to the repo
> 
> diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
> --- a/mercurial/localrepo.py
> +++ b/mercurial/localrepo.py
> @@ -514,6 +514,11 @@ class localrepository(object):
>  # manifest creation.
>  return manifest.manifest(self.svfs)
>  
> +@unfilteredpropertycache
> +def manifestaccessor(self):
> +return revlogaccessor('00manifest.i', self.svfs,
> +  self._constructmanifest)

Honestly I don't get why manifestlog and manifestctxs have to live longer
than the other repo properties. But suppose that is necessary, I agree we'll
need this kind of a wrapper. Maybe we can move the accessor to the manifestlog,
but still the accessor will have to be shared (and updated transparently) by
the manifestlog and its cachable manifestctxs.

> +def revlogaccessor(filename, opener, constructor):
> +"""Creates an accessor that provides cached and invalidated access to a
> +revlog, via instance.revlog. This is useful for letting multiple objects
> +hold a reference to the revlog, without having to hold a 
> possibly-circular
> +reference to the actual repository.  """
> +
> +# We have to use a runtime type here, because the only way to create a
> +# property is to put it on a class itself, and the property is 
> dynamically
> +# defined by the filename parameter.
> +class accessor(object):

Perhaps we could refactor filecache to avoid dynamically creating a class, but
that would be a minor issue of this RFC series.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH STABLE] hgweb: cache fctx.parents() in annotate command (issue5414)

[Yuya Nishihara]

On Fri, 04 Nov 2016 23:07:21 -0700, Gregory Szorc wrote:
> # HG changeset patch
> # User Gregory Szorc 
> # Date 1478325927 25200
> #  Fri Nov 04 23:05:27 2016 -0700
> # Branch stable
> # Node ID b0dc7dd87fc8b5efee6e6d8c0bcc6ddfa52d6df3
> # Parent  e5cc44ea12de681d971fcbebb65a7fb71fd1c3c7
> hgweb: cache fctx.parents() in annotate command (issue5414)

Looks good, but one nit.

> --- a/mercurial/hgweb/webcommands.py
> +++ b/mercurial/hgweb/webcommands.py
> @@ -861,12 +861,25 @@ def annotate(web, req, tmpl):
>  f = fctx.path()
>  parity = paritygen(web.stripecount)
>  
> +# parents() is called once per line and several lines likely belong to
> +# same revision. So it is worth caching.
> +# TODO there are still redundant operations within basefilectx.parents()
> +# that could also be cached.
> +parentscache = {}
>  def parents(f):
> -for p in f.parents():
> -yield {
> -"node": p.hex(),
> -"rev": p.rev(),
> -}
> +rev = f.rev()
> +if rev in parentscache:
> +for entry in parentscache[rev]:
> +yield entry
> +else:
> +parentscache[rev] = []
> +for p in f.parents():
> +entry = {
> +'node': p.hex(),
> +'rev': p.rev(),
> +}
> +parentscache[rev].append(entry)
> +yield entry

If the generator is stopped at the first yield, parentscache would be
incomplete.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH STABLE] hgweb: cache fctx.parents() in annotate command (issue5414)

[Gregory Szorc]

On Fri, Nov 4, 2016 at 11:07 PM, Gregory Szorc 
wrote:

> # HG changeset patch
> # User Gregory Szorc 
> # Date 1478325927 25200
> #  Fri Nov 04 23:05:27 2016 -0700
> # Branch stable
> # Node ID b0dc7dd87fc8b5efee6e6d8c0bcc6ddfa52d6df3
> # Parent  e5cc44ea12de681d971fcbebb65a7fb71fd1c3c7
> hgweb: cache fctx.parents() in annotate command (issue5414)
>
> 9c37df347485 introduced a call to fctx.parents() for each line in
> annotate output. This function call isn't cheap, as it requires
> linkrev adjustment.
>
> Since multiple lines in annotate output tend to belong to the same
> file revision, a cache of fctx.parents() lookups for each input
> should be effective in the common case. So we implement one.
>
> The effect of this change when requesting /annotate/96ca0ecdcfa/
> browser/locales/en-US/chrome/browser/downloads/downloads.dtd on
> the mozilla-aurora repo is significant:
>
> p1(9c37df347485)  5.5s
> 9c37df347485:66.3s
> this patch:  10.8s
>
> We're still slower than before. But only by ~2x instead of ~12x.
>
> On the tip revisions of layout/base/nsCSSFrameConstructor.cpp file in
> the mozilla-unified repo, time went from 12.5s to 14.5s and back to
> 12.5s. I'm not sure why the mozilla-aurora repo is so slow.
>
> Looking at the code of basefilectx.parents(), there is room for
> further improvements. Notably, we still perform redundant calls to
> filelog.renamed() and basefilectx._parentfilectx(). However,
> introducing a cache for them is not appropriate for the stable branch.
>

basefilectx.annotate() already performs some linkrev adjustment via
basefilectx.parents(). So I'm pretty sure the basefilectx.parents() calls
in hgweb are redundant with what annotate() is doing. I think there is room
for basefilectx.annotate() to return this data so hgweb doesn't have to
recompute it. But I've never really grokked linkrev adjustment or the
annotate code. If someone else wants to take a shot at this, please do!
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH STABLE] hgweb: cache fctx.parents() in annotate command (issue5414)

[Gregory Szorc]

# HG changeset patch
# User Gregory Szorc 
# Date 1478325927 25200
#  Fri Nov 04 23:05:27 2016 -0700
# Branch stable
# Node ID b0dc7dd87fc8b5efee6e6d8c0bcc6ddfa52d6df3
# Parent  e5cc44ea12de681d971fcbebb65a7fb71fd1c3c7
hgweb: cache fctx.parents() in annotate command (issue5414)

9c37df347485 introduced a call to fctx.parents() for each line in
annotate output. This function call isn't cheap, as it requires
linkrev adjustment.

Since multiple lines in annotate output tend to belong to the same
file revision, a cache of fctx.parents() lookups for each input
should be effective in the common case. So we implement one.

The effect of this change when requesting /annotate/96ca0ecdcfa/
browser/locales/en-US/chrome/browser/downloads/downloads.dtd on
the mozilla-aurora repo is significant:

p1(9c37df347485)  5.5s
9c37df347485:66.3s
this patch:  10.8s

We're still slower than before. But only by ~2x instead of ~12x.

On the tip revisions of layout/base/nsCSSFrameConstructor.cpp file in
the mozilla-unified repo, time went from 12.5s to 14.5s and back to
12.5s. I'm not sure why the mozilla-aurora repo is so slow.

Looking at the code of basefilectx.parents(), there is room for
further improvements. Notably, we still perform redundant calls to
filelog.renamed() and basefilectx._parentfilectx(). However,
introducing a cache for them is not appropriate for the stable branch.

diff --git a/mercurial/hgweb/webcommands.py b/mercurial/hgweb/webcommands.py
--- a/mercurial/hgweb/webcommands.py
+++ b/mercurial/hgweb/webcommands.py
@@ -861,12 +861,25 @@ def annotate(web, req, tmpl):
 f = fctx.path()
 parity = paritygen(web.stripecount)
 
+# parents() is called once per line and several lines likely belong to
+# same revision. So it is worth caching.
+# TODO there are still redundant operations within basefilectx.parents()
+# that could also be cached.
+parentscache = {}
 def parents(f):
-for p in f.parents():
-yield {
-"node": p.hex(),
-"rev": p.rev(),
-}
+rev = f.rev()
+if rev in parentscache:
+for entry in parentscache[rev]:
+yield entry
+else:
+parentscache[rev] = []
+for p in f.parents():
+entry = {
+'node': p.hex(),
+'rev': p.rev(),
+}
+parentscache[rev].append(entry)
+yield entry
 
 def annotate(**map):
 if util.binary(fctx.data()):
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel