Re: To use vcsh or not to use vcsh, that is the question

[Adam Spiers]

On 14 January 2017 at 17:51, Sean Whitton  wrote:
> On Sat, Jan 14, 2017 at 09:15:46AM -0700, Sean Whitton wrote:
> > Note that an updated plug-in has already been merged into Joey's branch
> > of myrepos -- this work is done!
>
> Some updates awaiting review here: 
> http://myrepos.branchable.com/todo/updates_to_stow_lib/

Wow cool, I had no idea!  Thanks so much for doing that! :-)
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: To use vcsh or not to use vcsh, that is the question

[Adam Spiers]

On 12 January 2017 at 04:20, Samuel Hodgkins 
wrote:

> So, I am a fan of keeping as much under version control as possible
> (Indeed, I recently reorganized my home directory to help with this) but
> I've hit an interesting dilemma.
>
> On the one hand, vcsh is a very powerful and useful utility and comes with
> handy commands and functionality and I currently use it for most
> repositories. However, it also both conceptually and practically more
> complex than plain git due to the use of fake-bare repositories, meaning
> you cannot just 'cd' and use git. For self-contained directories, this
> makes me wonder if the additional vcsh-provided commands are worth it.
>
> In comparison to vcsh, I also use myrepos which is excellent at managing a
> set of repositories, and although I haven't used it much it can also do
> commands. I think (but cannot confirm) that it can be extended with
> additional commands just like vcsh and has the benefit of also using normal
> git repositories, enabling easier usage from the command-line.
>
> While vcsh is the obvious (and only) option for some cases, such as
> dotfiles directly in ~
>

It's certainly not the only option.  Some of us are using GNU Stow (the
symlink farm manager) for this:

https://www.gnu.org/software/stow/

The first two links within the description give you more details about how
it works.  The second of those two pages gives details of the plugin I
wrote for mr (now myrepos) which integrates support for stowable
repositories.  This combination has worked well enough for me for several
years that I haven't actually bothered to upgrade my mr installation to the
latest myrepos.  However if others are interested in it then I'd be glad to
do that - I doubt it would take much work.

There are pros and cons to both approaches.  I remember having an amicable
conversation with vcsh's author about this at a FOSDEM conference a long
time ago.  My recommendation would be to try both vcsh and Stow (ideally
both in combination with myrepos) and see which you prefer.  I'll do my
best to answer any questions you have regarding Stow and the mr plugin I
wrote.

Hope this is of interest!
Adam

P.S. Declaration of interest: I maintain GNU Stow.
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: Quieting ``mr`` output

[Adam Spiers]

Thanks for sharing!

This reminds me of this very old thread

http://lists.madduck.net/pipermail/vcs-home/2012-February/000705.html

which unfortunately went nowhere due to Joey having concerns about
layering violations.

On 28 June 2015 at 14:34, Michael Henry vcs-h...@drmikehenry.com wrote:
 On 06/25/2015 01:49 PM, John Whitley wrote:
 Nice idea, Michael.  It’s possible to prepend text only if
 some command has output.

 I've never seen that particular trick with ``sed``, but I like
 it.  It's very compact and uses a standard tool instead of
 hundreds of lines of Python, which makes it a much better
 candidate for inclusion in ``mr`` or ``vcsh`` directly.  The
 only thing it lacks is a display of progress as the
 repositories are being checked.  For ``mr status`` using git
 repositories on my desktop with RAID SSDs, the difference isn't
 significant, as the entire operation completes in under half a
 second; but for commands like ``mr update``, I find it nice to
 see the activity as it progresses (especially on a slower link).
 In addition, some commands generate output even in the
 uninteresting cases.  For example, a default ``mr update``
 looks like this::

   mr update: /home/mike/.
   Already up-to-date.

   mr update: /home/mike/projects/ProjectOne
   Already up-to-date.

   [...]

   mr update: /home/mike/projects/ProjectN
   Already up-to-date.

 I'd like to be able to squelch this kind of output as well.  But
 if you're willing to forgo seeing the progress live as it
 happens, then such lines could be removed by additional ``sed``
 logic.

 For most users, having something like your ``sed``-based
 implementation built-in would be advantageous, though for myself
 I'd find it nice (but not absolutely necessary) if that feature
 could be bypassed to allow a display of progress as the
 overall operation proceeds.

 Thanks,
 Michael Henry

 ___
 vcs-home mailing list
 vcs-home@lists.madduck.net
 http://lists.madduck.net/listinfo/vcs-home
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: automating git-annex relationships

[Adam Spiers]

On 9 September 2014 14:24, martin f krafft madd...@madduck.net wrote:
 Has anyone come up with a smart way of auto-configuring
 relationships between hosts? git-annex keeps track of where files
 are, so theoretically, it should be able to auto-configure the
 remotes if I tell it that two remotes are directly in reach of each
 other?

This is how I do it:

http://lists.madduck.net/pipermail/vcs-home/2012-May/000806.html
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: Using subtree with vcsh and mr?

[Adam Spiers]

On 8 August 2014 15:34, Michael McCracken michael.mccrac...@gmail.com wrote:
 Hi, I've been happily using vcsh and mr for a while, to manage my emacs,
 zsh, git  bzr configs.

 I've hit a problem with integrating third-party code that works with e.g.
 emacs or zsh.
 I suspect I'm missing a straightforward way to solve this, but in searching
 through the vcsh issues on github, it seems like it wasn't solved (or no
 one updated the issue, #27 [1]).

 For example, useful stuff like zsh-git-prompt [2] just has a top-level repo
 with files in it, so if I directly add it as a main repo in vcsh, I'll just
 get those files dumped into $HOME. For that one, I 'solved' it by forking
 and creating the directory structure I wanted [3]. However, that's going to
 be a pain for anything that's more actively developed.

 For example, emacs-helm [4] is really useful, and it's actively developed,
 but look at all those files in the top level just flooding $HOME.

 Any tips?

I don't know how to do this with vcsh, but it works fine with my
approach using mr + GNU Stow:

  http://lists.gnu.org/archive/html/info-stow/2011-12/msg0.html

Incidentally, zsh-git-prompt looks like nice functionality, but it
annoys me every time I see zsh prompt code not developed using the
built-in prompt theme system which I introduced to zsh core back in
the 1990s.  Following the standardized system would have been so much
nicer for everyone; yet another case of NIH syndrome I guess :-(
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: Using subtree with vcsh and mr?

[Adam Spiers]

On 8 August 2014 17:05, Michael McCracken michael.mccrac...@gmail.com wrote:
 On Fri, Aug 8, 2014 at 8:56 AM, Richard Hartmann 
 richih.mailingl...@gmail.com wrote:
 On Fri, Aug 8, 2014 at 4:34 PM, Michael McCracken 
 michael.mccrac...@gmail.com wrote:
  I suspect I'm missing a straightforward way to solve this, but in searching
  through the vcsh issues on github, it seems like it wasn't solved (or no
  one updated the issue, #27 [1]).

 I talked to the authors of `git submodule` about this issue at the
 last CeBIT, but the very short version is that this is hard for
 submodule-internal reasons but that `git subtree` is working

 That sounds like good news. At the risk of sounding dense, is there any
 example of it working with mr  vcsh that I can learn from?

Why is git subtree or submodule required at all?  Can't vcsh just set
GIT_DIR to something other than $HOME?
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: How to start syncing two existing directories with git annex?

[Adam Spiers]

On 24 November 2013 20:03, Sean Hammond snh...@gmail.com wrote:
 Hey, I have a ~/Music directory on computer A, and a ~/Music directory on
 computer B. They contain mostly the same files (and with the same paths).
 But there might be some files on A but not B, or vice-versa. And there might
 be some files on both but different (e.g. different id3 tags).

 I want to use git annex assistant to sync the two dirs. Neither computer is
 big enough to hold two copies of the Music dir at once. Ideally, I'd prefer
 not to have to delete the Music dir from computer B, for example, and then
 let git annex sync it from A over to B again.

This is a similar scenario to

http://git-annex.branchable.com/tips/migrating_two_seperate_disconnected_directories_to_git_annex/
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: cdgit: cd relative to git workdir root

[Adam Spiers]

On Mon, Sep 16, 2013 at 09:01:14PM +0200, Thomas Koch wrote:
 Hi,
 
 I frequently have the need to cd back to the root of the current git workdir 
 and created to alias for this:
 
 git alias root = rev-parse --show-toplevel
 
 shell alias cdgit = cd $(git root)
 
 Maybe somebody likes this.

I do, and I was about to copy it into my own config when I discovered
that I'd already done this 4 years ago and totally forgotten about it :)

 Now it would be cool, to have an alias to cd into a specific subdir of the 
 git 
 worktree and have zsh (or bash) auto completion for this like
 
 git clone git://git.kernel.org/pub/scm/git/git.git
 cd git/gitweb/static/js/lib (just to be somewhere deep)
 cdgit contab/comtab (becomes: cdgit contrib/completion)
 
 Somebody with zsh/bash skills who already did this?

I hadn't done this before today, but it's easy in zsh:

  https://github.com/aspiers/git-config/commit/89df1d78

Of course, _gitcd needs to be somewhere on your $fpath so it can be
autoloaded.
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: --repos patch rebased against 1.07

[Adam Spiers]

Hi Joey,

I don't know how much time you are currently able to devote to mr
given your kickstarter-related activities (which totally rock BTW ;-),
but I would really like to revisit the below unresolved issues at some
point soon if possible, because they are blocking my goal of
out-of-the-box integration between mr and GNU Stow in a similar manner
to what has already been accomplished with vcsh.  I have been using my
lib/stow plugin for mr for about 10 months now and it works well, but
not without some kind of solution to the below problems.

Thanks,
Adam

On Thu, Dec 15, 2011 at 11:55 AM, Adam Spiers vcs-h...@adamspiers.org wrote:
 On Wed, Dec 14, 2011 at 6:14 PM, Joey Hess j...@kitenet.net wrote:
 Adam Spiers wrote:
 please could you cherry-pick this new version of the patch?

 Most people will not use it. Some people will use it when they could
 have simply used symlinks to point to the actual location of their
 repositories to provide short names for them.

 OK, so if I went to the effort of setting up these symlinks, how would
 I use them to accomplish the equivalent of

   mr -r zsh,emacs up

 in a single, easy-to-type command?

 It has the potential to affect any current user of mr, who could be
 confronted with a warning message about inability to slot a repo
 into this other namespace.

 That's a corner case which should happen very infrequently, but if you
 don't like it, I can easily move that warning to a new --check-names
 option or similar.

 It adds the complexity of a separate namespace, which feels
 strongly like a bad idea.

 Why?  The existing namespaces do not facilitate per-repository
 granularity (the repository space already exists, but individual
 members cannot be conveniently referenced) so AFAICS this is filling a
 gap in functionality.

 You seem to be using the idea as your sole hammer to knock down various
 nails that can be pulled in simpler ways, as seen in
 CAOkDyE-Tanqr8zsNMuXJiemYnb=vb-wbk1_cqrgadgqredv...@mail.gmail.com.

 [ For the benefit of anyone else following, he's referring to
   http://thread.gmane.org/gmane.comp.version-control.home-dir/463/focus=467 ]

 I still don't see any easier way of achieving what I want.  Perhaps
 your answer to my first question above will enlighten me.

 Stow's need for a flat namespace also seems to have a simpler approach
 of echo $MR_REPO | tr / _

 http://thread.gmane.org/gmane.comp.version-control.home-dir/356/focus=386
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: dvcs-autosync+git is broken by design

[Adam Spiers]

On Mon, Mar 5, 2012 at 12:18 AM, Chanoch (Ken) Bloom kbl...@gmail.com wrote:
 On Sun, 2012-03-04 at 20:08 +0100, Dieter Plaetinck wrote:
 On Sun, 4 Mar 2012 17:19:25 +
 Adam Spiers vcs-h...@adamspiers.org wrote:
  I have not yet used dvcs-autosync in anger;
 what do you mean?

 It's a stretch from a military idiom. To say that a soldier never fired
 a shot in anger means that he never fired it in actual combat. It did
 fire it in drills and training, but not in battle.

I didn't know that, thanks :)
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: dvcs-autosync+git is broken by design

[Adam Spiers]

On Mon, Mar 5, 2012 at 10:06 AM, Dieter Plaetinck die...@plaetinck.be wrote:
 On Sun, 4 Mar 2012 22:39:22 +
 Adam Spiers vcs-h...@adamspiers.org wrote:

 Sure - I don't mind changes to existing files being autosynced,
 but I want to manually add new files to the repo myself.

 Then you should change startupcmd = git add -A to git add -u

Consulting my .autosync, it seems I already did that.

 although that's vulnerable to race conditions:
 if you're updating files while 'git add -u' is running, it may abort.  The 
 wrapper script I showed should probably be expanded with '-u' and '-A' like 
 support.

Thanks for the heads-up.  I don't think this is likely to be an issue
with my anticipated usage patterns.
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: insteadOf vcsh://

[Adam Spiers]

On Sun, Mar 4, 2012 at 10:59 AM, Raf Czlonka r...@linuxstuff.pl wrote:
 On Sat, Mar 03, 2012 at 11:49:01PM GMT, Joey Hess wrote:
 It also makes mr update -j20 a lot faster, as well as not opening 20
 ssh connections to some single server, which sometimes causes ssh
 cracking detectors to ban an IP. :)

 Using Control* [0] in your ssh_config should remedy that :^)

Beware of ControlPersist - I am not yet 100% sure but I think
I have seen this bug when mr uses ssh with ControlPersist
enabled:

  https://bugzilla.mindrot.org/show_bug.cgi?id=1988

The higher the timeout value, the longer mr takes to complete.
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: insteadOf vcsh://

[Adam Spiers]

On Sun, Mar 4, 2012 at 12:28 PM, Raf Czlonka r...@linuxstuff.pl wrote:
 On Sun, Mar 04, 2012 at 12:03:34PM GMT, Adam Spiers wrote:
 On Sun, Mar 4, 2012 at 10:59 AM, Raf Czlonka r...@linuxstuff.pl wrote:
 
  Using Control* [0] in your ssh_config should remedy that :^)

 Beware of ControlPersist - I am not yet 100% sure but I think
 I have seen this bug when mr uses ssh with ControlPersist
 enabled:

   https://bugzilla.mindrot.org/show_bug.cgi?id=1988

 The higher the timeout value, the longer mr takes to complete.

 Yes, I'm aware of that. However, ControlMaster without ControlPersist is
 not very useful and the default to no for the latter doesn't isn't
 either IMHO - neither for automated tasks, not for interactive session.
 That's why I always set it to 0.

Now I'm puzzled.  Setting it to 0 will prevent the connection ever
timing out, right?  In which case, if you were affected by that bug,
mr would permanently hang.  So presumably you're not affected by it.

I thought I was seeing it, but I can't reproduce the hang now.  I'll
keep investigating ...
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: dvcs-autosync+git is broken by design

[Adam Spiers]

On Sun, Mar 4, 2012 at 4:35 PM, Dieter Plaetinck die...@plaetinck.be wrote:
 this seems to work okay so far in my 'notes' repo.

Thanks a lot for sharing.  I have not yet used dvcs-autosync in anger;
however I had already decided during testing that I never wanted it to
automatically 'git add' anything:

  addcmd = echo New untracked file %s

Am I right in thinking that in this case, there is no synchronisation
issue to worry about?
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: insteadOf vcsh://

[Adam Spiers]

Hi Thomas,

On Sat, Mar 3, 2012 at 6:14 AM, Thomas Koch tho...@koch.ro wrote:
 I just learned (via http://http://git.madduck.net ) about git's insteadOf
 config (man git-config , search insteadOf) and did:

 export COMMON=my.server.org:git/vcsh/
 git config --global url.${COMMON}.insteadOf vcsh://

 and changed in each vcsh repo the remote url to vcsh://$VCSHREPONAME

Then you may be interested in this post I made in December:

http://article.gmane.org/gmane.comp.version-control.home-dir/502

HTH,
Adam
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: insteadOf vcsh://

[Adam Spiers]

On Sat, Mar 3, 2012 at 5:36 PM, Yaroslav Halchenko li...@onerussian.com wrote:
 you might also like even more pushInsteadOf, so that you can clone using
 read-only git@ protocol but push over ssh

What are the advantages of that approach?
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: mr: breaking down complex subroutines

[Adam Spiers]

On Fri, Feb 17, 2012 at 4:19 PM, Tommy Stanton tommystan...@gmail.com wrote:
 On Fri, Feb 17, 2012 at 10:02 AM, Adam Spiers vcs-h...@adamspiers.org wrote:
 consequently to write patches for.  For example, loadconfig() has 221
 lines, and is 8 levels of indentation deep at certain points.  Would

 The code excerpt of loadconfig() being discussed can be glanced at here:
 http://git.kitenet.net/?p=mr.git;a=blob;f=mr;h=8408c61d5407d56301f081e000b01bf02d0a547f;hb=HEAD#l1204

Thanks Tommy.  The other big candidate for refactoring is action()
which is 164 lines long and 6 levels deep in places:

http://git.kitenet.net/?p=mr.git;a=blob;f=mr;h=8408c61d5407d56301f081e000b01bf02d0a547f;hb=HEAD#l684
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

first stab at zsh completion

[Adam Spiers]

On Mon, Dec 26, 2011 at 7:15 PM, Joey Hess j...@kitenet.net wrote:
 Adam Spiers wrote:
 there is a bigger cost - the risk of having a version of the completion
 rules which does not match the version of mr installed.

 This is, in practice, not a large problem, and can be dealt with by
 distribution integrators.

Agreed.

 There's also a converse argument.  Completion functions are not only
 coupled to the thing they are completing for, but also to the shell's
 completion API.  When the API changes, it's better to have completion
 functions within the shell's distribution, because the shell's
 developers can fix all completion functions to work with the new API
 in one go.

Actually, (somewhat contradicting my own argument) the zsh completion
API is extremely stable these days.  Back around 1999--2000, I *used*
to be something very loosely approaching a zsh guru by normal people's
standards (though nowhere near the likes of Peter Stephenson, Bart
Schaefer, Zefram etc.), having contributed patches for various bits
and pieces ... and now, over 10 years later, while making a first stab
at completion for mr (see below), I have not noticed *any* significant
changes in the API since then.

 Which is why I would certianly not like to bundle zsh completion
 functions with the programs they complete. You have to be a zsh guru to
 write them, they have changed a *lot* over the years (I don't recognize
 anything in the current dpkg completion that's left from the one I
 originally wrote),

Perhaps you are referring to the change from the ancient system based
on compctl one-liners, to the all-singing all-dancing 'compsys' which
is based on complex shell functions and various built-ins from the
zsh/zle module?  That was a huge, one-off change.

 and upstream is very responsive, to keep the completions
 up-to-date.

 I suspect that bash completion will head in a similar direction, as they
 get reworked to support dynamically loading completions on demand, per
 http://thread.gmane.org/gmane.comp.shells.bash.completion.devel/3375

Yes, I suspect bash's completion is still light years behind zsh's,
and may require major changes to catch up.

 With that said, putting a bash completion in mr now just means a little
 probable pain later on, so I'm not strongly opposed.

 The real difficulty in completing mr is that it accepts an arbitrary set
 of subcommands, even depending on what repository it's run in.

Actually, that's pretty easy to deal with, albeit with a performance
penalty.  My first draft version automatically detects the available
actions by parsing the output of `mr thisisaninvalidcommand`:

  https://github.com/aspiers/kitenet-mr/commit/zsh

Currently it caches the result within the shell, but if as you point
out the output varies depending on which repository you're in, it
might make sense to remove that caching layer.

 In practice, I just type abbreviated things like mr up and mr p
 instead of reaching for the tab key; happily mr will accept any
 nonambiguous abbreviations and can be taught others. :)

Yeah, it's not a huge win.  Having completion for the options is quite
nice though :)
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: inventory of files unavailable in local repository?

[Adam Spiers]

On Fri, Dec 23, 2011 at 5:13 PM, Joey Hess j...@kitenet.net wrote:
 Adam Spiers wrote:
 I don't get that error message.

 version?

3.20111211

 So, your git-annex version was, apparently, seeing --in= as in the
 remote named '', and doing the expensive query of git for that info..
 or something. Hmm, I vaguely think I might have fixed a bug like that at
 some point.

Maybe not released yet?
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: inventory of files unavailable in local repository?

[Adam Spiers]

On Fri, Dec 23, 2011 at 6:08 PM, Joey Hess j...@kitenet.net wrote:
 Adam Spiers wrote:
 On Fri, Dec 23, 2011 at 5:13 PM, Joey Hess j...@kitenet.net wrote:
  Adam Spiers wrote:
  I don't get that error message.
 
  version?

 3.20111211

  So, your git-annex version was, apparently, seeing --in= as in the
  remote named '', and doing the expensive query of git for that info..
  or something. Hmm, I vaguely think I might have fixed a bug like that at
  some point.

 Maybe not released yet?

 Aha.. no, what's really happening is you have a remote with a
 description of , and so it thinks that is the one you mean. I have
 added a special case to not match on such empty descriptions.

Ahah - thanks! :)
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

mr: trust errors cause Use of uninitialized value warnings

[Adam Spiers]

The trust error handling code expects five parameters:

my $trusterror = sub {
my $msg=shift;
my ($err, $file, $lineno, $url)=@_;

but it is only ever invoked with one, e.g.

if (! $trusted  /[[:cntrl:]]/) {
$trusterror-(illegal control character);
}

For me, this results in a mess when I have an untrusted file:

Use of uninitialized value $err in concatenation (.) or string at
/home/adam/bin/mr line 1295.
Use of uninitialized value $file in concatenation (.) or string at
/home/adam/bin/mr line 1295.
Use of uninitialized value $lineno in concatenation (.) or string
at /home/adam/bin/mr line 1295.
mr:  in untrusted  line
(To trust this file, list it in ~/.mrtrust.)

Ideally there would be a test suite to catch things like this.
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

inventory of files unavailable in local repository?

[Adam Spiers]

git annex find can provide an inventory of files whose content is
available in the local repository.  But is there a way of finding
which files are *not* present locally?  Checking for broken symlinks
and filtering for /.git/annex/objects is kinda ugly.
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: inventory of files unavailable in local repository?

[Adam Spiers]

On Thu, Dec 22, 2011 at 8:36 PM, Richard Hartmann
richih.mailingl...@gmail.com wrote:
 Did you try --not?

Ahah!  I think this works:

git annex find --not --in=

Thanks!
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: inventory of files unavailable in local repository?

[Adam Spiers]

On Thu, Dec 22, 2011 at 8:47 PM, Adam Spiers vcs-h...@adamspiers.org wrote:
 On Thu, Dec 22, 2011 at 8:36 PM, Richard Hartmann
 richih.mailingl...@gmail.com wrote:
 Did you try --not?

 Ahah!  I think this works:

    git annex find --not --in=

 Thanks!

Unfortunately this is cripplingly slow:

$ time symlinks -r . | egrep '^dangling: .+/\.git/annex/objects/' |
awk '{print $2}' /dev/null
symlinks -r .  0.15s user 0.37s system 97% cpu 0.536 total
egrep '^dangling: .+/\.git/annex/objects/'  0.01s user 0.00s system 1%
cpu 0.534 total
awk '{print $2}'  /dev/null  0.00s user 0.00s system 1% cpu 0.526 total

$ time git annex find --not --in= /dev/null
git annex find --not --in=  /dev/null  6.73s user 1.76s system 21%
cpu 39.483 total

Ouch!  Joey, is there an optimization that can be made for the local case here?
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: syncing non-git trees with git-annex

[Adam Spiers]

On Wed, Dec 14, 2011 at 12:53 PM, Richard Hartmann
richih.mailingl...@gmail.com wrote:
 I would use

  find -name \*.avi -exec git annex add {} \;

That's substantially less performant, because it forks a whole new
tree of git / git-annex processes per file.

If we're getting picky, we should also worry about spaces in
filenames:

  find -name \*.avi -print0 | xargs -0 git annex add

but it was only an example, and to be honest, I didn't even use the
xargs variant myself; I used zsh's recursive globbing:

  git annex add **/*.avi

but I thought that might confuse non-zsh users so I replaced it with
xargs in the example :-)

 but other than that, this seems fine.

Great, thanks!

 Depending on your level of OCD,
 it might make sense to throw away the initial repo once your data is
 clean and import everything in a clean one. That's what I do.

Interesting - any particular reason for doing that?

 PS: There are various tools for finding duplicates, but git-annex
 gives you this functionality for free, so..

Yes, this one is particularly good:

  http://sourceforge.net/projects/fastdup/

but using git-annex facilitates it in a much more distributed fashion.
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

--repos patch rebased against 1.07

[Adam Spiers]

Hi Joey,

I see you already cherry-picked that small refactoring patch from my
'for-joey' branch; thanks for that.

I also notice that the new --force caused some conflicts with my
--repos patch, so I have fixed those and made the new version
available in the same place:

https://github.com/aspiers/kitenet-mr/commit/for-joey

Bearing in mind the below:

On Tue, Dec 13, 2011 at 11:58 AM, Adam Spiers vcs-h...@adamspiers.org wrote:
 However, even ignoring all these reasons, the requirement for a
 namespace of short repository identifiers (which cannot contain the
 '/' character) is an unavoidable consequence of supporting integration
 between mr and GNU Stow, since Stow already requires it.  My --repos
 patch automatically builds this namespace by taking the basenames of
 each repository directory, and then carefully handling the minority of
 cases where these are non-unique with minimal disruption to the end
 user.

please could you cherry-pick this new version of the patch?

Thanks,
Adam
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: Is there any interest in a patchset for mr to manage which repos are being handled?

[Adam Spiers]

Hi Richard,

On Mon, Dec 12, 2011 at 12:28 AM, Richard Hartmann
richih.mailingl...@gmail.com wrote:
 Hi all,

 the subject line may be a bit convoluted, but it's just what I meant.

 To make a short story somewhat longer, my setup looks like this:

 % cat .mrconfig
 [DEFAULT]
 git_gc = git gc $@
 jobs = 5
 include = cat /usr/share/mr/vcsh
 include = cat ~/.config/mr/config.d/*
 % ls .config/mr/available.d
 foo bar baz
 % ls -l .config/mr/config.d
 foo - ../available.d/foo
 bar - ../available.d/bar
 %

 So I only have foo  bar activated on this machine, baz is not checked
 out. I may not want to have your ssh config on a semi-trusted machine,
 my mplayer config on a server, etc.

 I would like to handle the local repo set via mr and not by manually
 using ln  rm. Thus, I would like to patch mr to do this, but only if
 there is any interest and if Joey is OK with actually merging them.
 As I know several other people are using my directory layout, I
 suspect there is some interest, but it's always good to get direct
 feedback.

I've already solved this, but in a different way to what you are
proposing.  I'm using a layout which is superficially similar to
yours; effectively:

include = cat ~/.config/mr/groups.d/*

(I got the idea to use ~/.config/mr from you.)  However, the
similarity ends there - each included file contains a group of related
repositories, and I'm not using active-available symlinks to activate
them.  I did consider using this approach, but I use mr on about 6
machines, and with ~70 repositories I really didn't like the idea of
having to manually make ~420 decisions about which repositories to
enable on which machines.  So I decided I wanted to automate these
decisions wherever possible by putting as much intelligence as
possible in my mr config.

The solution I came up with was a combination of the following
elements ...

Firstly, I built a library of skip functions:

https://github.com/aspiers/mr-config/blob/master/lib/skippers

which lets me write things like:

[$HOME/.GIT/adamspiers.org/gnupg.sec]
skip = default_skipper || missing_exe gpg

The '.sec' suffix indicates that this repository contains
security-sensitive data (my GPG private key).  The default_skipper
notices this, and only allows checkout on machines I consider secure
(i.e. with a known hostname, behind a NAT firewall, and only
physically accessible by me).  The repository is also skipped unless
gpg is installed.

When I don't know in advance whether I'll need a repo on a particular
machine, I use the 'lazy' skipper:

skip = default_skipper || lazy

However, in the upstream mr, this is not fully implemented yet because
it does not prevent checkouts of lazy repositories:

http://thread.gmane.org/gmane.comp.version-control.home-dir/396/focus=398

To solve this, I knew mr would need a mechanism for referring to a
single repository, which in turn would require a new namespace for
repositories.  Your 'available.d' directory effectively provides this
namespace, but at the cost of being forced to split all repository
definitions into separate files *and* keep all those definitions in a
single directory (which by necessity would exclude the repository
containing those definitions).  So I decided on a different approach:

http://thread.gmane.org/gmane.comp.version-control.home-dir/356/focus=385

I added support for naming repositories via a special 'name' config
parameter and referring to them via a new --repos option:

https://github.com/aspiers/kitenet-mr/commit/183ec2c55

and coincidentally I sent a pull request for this yesterday:

http://thread.gmane.org/gmane.comp.version-control.home-dir/462

Then I patched lazy() so that checkouts only happen when --repos is used:

https://github.com/aspiers/kitenet-mr/commit/586f60

(No pull request for this yet, because it depends on the first patch.)

The end result is that lazy repositories will always get skipped
unless I specifically ask mr to check them out, e.g.

mr -r foo checkout

This is effectively equivalent to your proposal:

mr enable foo

except that it's more direct, since if you enable 'foo', surely you
would checkout 'foo' immediately after.  Then the only missing piece
is 'disable'.  Personally I don't need this (yet, at least).  But if
you really needed it, the lazy() skipper could easily be extended (or
a new skipper written) to perform an extra check:

test -d .mrdisabled

and then the disable action could be implemented via:

[DEFAULT]
enable  = rm .mrdisabled
disable = touch .mrdisabled

Would love to hear your thoughts on all this!
Adam
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

couple more patches for mr

[Adam Spiers]

Available from my 'for-joey' branch on github:

https://github.com/aspiers/kitenet-mr/commits/for-joey
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: New integration branch

[Adam Spiers]

On Mon, Dec 5, 2011 at 5:38 PM, Joey Hess j...@kitenet.net wrote:
 Adam Spiers wrote:
  9c87f2352214175de307efedb8fd93889a26afbc
         Can you give an example of when this is needed?

 I can't remember but I definitely saw it happen at least once :-/

 My worry is that, since that really shouldn't happen AFIACS, you
 were actually seeing a bug. Either that or it's a corner case I have not
 identified.

It's on my TODO list to try to reproduce; I'll let you know if I
manage to.

  602f26714254f3c65389b7665d15d1d5d0e227a9
         mr is quite typically (I know, not by you) run
         inside the repository. Which would leave the user
         in an apparently empty directory after mr update if
         an mr update deleted and remade the whole repository.
 
         I don't like that; I don't think things in mr should be
         deleting repositories in general; mr doesn't even delete
         a repo that has deleted = true, it only warns the user about it.

 Hmm, that's a fair point, although the only alternative is to change
 the contents of the directory rather than the directory itself -
 similarly to how `git checkout' does, for instance.  I'll see if I can
 get around to doing that.  Perhaps some of the effort could be reused
 for implementing download_diff (diff against the archive file).

 I think you could just use rsync :)

Yeah, that sounds worth trying.

         If a machine-parseable list of repositories is needed,
         I think it'd be better to have a perl function that emits
         it in one go.

 I don't see how that's possible without ignoring the `skip',
 `deleted', and `include' parameters.

 The include parameter is not a big problem, it's unlikely to require
 more than one shell process, which will be relatively fast.

 It's not clear to me what should be done about skip and deleted.
 skip in particular can behave in weird ways, when something like
 hours_since is used. To handle that all the skips would need to be
 tested, which would be less work than mr list but still verging on
 expensive.

 Depending on the application, it might be better to just dump all the
 defined repositories including skipped and deleted ones; if the consumer
 than runs mr in a skipped/deleted repo, mr will do something sane after
 all.

Skipper functions like hours_since could (and probably should) decide
not to skip if MR_ACTION is set to a read-only action such list -
arguably diff and status too, although that's a matter of personal
taste.

But maybe we should step back a bit.  Currently we know that a full
mr list is not particularly fast, but has anyone actually profiled
it to find out where most of the time is being spent?  If we're only
guessing then we might have it completely wrong ...
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

git annex map does not spot graph loops

[Adam Spiers]

I set up two git annex repos on the local machine which point to each
other and then run git annex map, it chews up a load of CPU,
presumably trying to traverse the cyclic repository graph without ever
noticing there's a loop:

% mkdir -p annex-test/{a,b}
% cd annex-test/a
% git init
Initialized empty Git repository in /home/adam/tmp/2/annex-test/a/.git/
% git annex init
init  ok
% cd ../b
% git init
Initialized empty Git repository in /home/adam/tmp/2/annex-test/b/.git/
% git annex init
init  ok
% git remote add a ../a
% cd ../a
% git remote add b ../b
% time git annex map  map.out 21
git annex map  map.out 21  25.67s user 4.38s system 93% cpu 32.238 total
% head -n10 map.out
map /home/adam/tmp/2/annex-test/a ok
map b ok
map a ok
map b ok
map a ok
map b ok
map a ok
map b ok
map a ok
map b ok
% dd if=map.dot bs=1k count=1
digraph map {
subgraph cluster_localhost {
label=localhost
style=filled
fillcolor=lightblue

/home/adam/tmp/2/annex-test/a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b/../a/../b1+0
records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.000182134 s, 5.6 MB/s
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

git annex --help doesn't show manual page

[Adam Spiers]

$ git annex  --help
No manual entry for git-annex

Is this issue related to the fact that I installed git-annex with cabal install?
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: New integration branch

[Adam Spiers]

On Sun, Dec 4, 2011 at 9:11 PM, Joey Hess j...@kitenet.net wrote:
 37617a63ec993b128f77a945a2020ec894c58eb1
        loadconfig already uses %loaded to avoid reloading the same
        config twice, so this extra check is not necessary, I think.

Ah yes, I missed that.  Still, for the cost of an extra line of code,
isn't it worth being explicit?  If it confused me, then presumably it
could confuse other developers in the future.

 a61c1450ff4b108af26e899a89a1d8ff49cab86c
        I picked the bugfix part.

        The warning message on missing chain files exposes an unclear
        thing in mr; it will try to chain to directories even when their
        repository has skip = true, which causes the warning to show
        up unexpectedly (ie, here). I think it needs to be changed to
        honor skip = true even if chain = true.

Ah OK, I'll look into that.

 b3b68137988e61be1a0f7d90caf05eabf7850f44
        I developed a different fix this morning that shows correct
        line numbers for both the mrconfig and the position in the
        include, it's in my tree.

Yep, I saw that - very nice :)

 135e0076c9a93cd0556b9b25ff355ad25546a78c
        This makes mr fetch do a git fetch, but nothing for
        the over DVCSes which can also do things like fetch, and
        no documentation of it

I can add documentation.  However, although ISTR that `hg pull' is
akin to `git fetch', I don't know how it's done for the other
decentralised VCS.

 9c87f2352214175de307efedb8fd93889a26afbc
        Can you give an example of when this is needed?

I can't remember but I definitely saw it happen at least once :-/

 602f26714254f3c65389b7665d15d1d5d0e227a9
        mr is quite typically (I know, not by you) run
        inside the repository. Which would leave the user
        in an apparently empty directory after mr update if
        an mr update deleted and remade the whole repository.

        I don't like that; I don't think things in mr should be
        deleting repositories in general; mr doesn't even delete
        a repo that has deleted = true, it only warns the user about it.

Hmm, that's a fair point, although the only alternative is to change
the contents of the directory rather than the directory itself -
similarly to how `git checkout' does, for instance.  I'll see if I can
get around to doing that.  Perhaps some of the effort could be reused
for implementing download_diff (diff against the archive file).

 650620d7b6661f9cc59b4adfb6a7d945240fe5c7
 f16e51cea8595afc92f3ab9230e3c5a44baed904
        I've held off on these plugins since I think they
        depend on 602f26714254f3c65389b7665d15d1d5d0e227a9

No, only the download plugin.  The stow plugin *never* writes to the
repository tree, it manages symlinks in an entirely different place
(typically $HOME).  Having said that, I just remembered that the stow
plugin depends on MR_NAME so that will have to wait too.

 cf3388f443b9d7afe6dc7d8a2159b45fb01ab4e4
        This is a slow way to make machine-parsable info available --
        the similar mr list takes 8 seconds here, since it has to run
        169 shells. That's ok when you're just running mr, but I would
        not like to use a command that depended on that information.

Sure, that's why I used a simple on-disk cache:

  
https://github.com/aspiers/kitenet-mr/commit/b60acb2e767b91ca6d406198d7eea1b3f73ad2bf

It works fine.  I could get more sophisticated and allow per-user
configuration of the cache invalidation strategy, e.g. so that it
would automatically rebuild the cache when ~/.mrconfig et al. are
changed, but manual rebuilds aren't a great hardship.  In fact I could
even rebuild the cache every time mr runs!

        If a machine-parseable list of repositories is needed,
        I think it'd be better to have a perl function that emits
        it in one go.

I don't see how that's possible without ignoring the `skip',
`deleted', and `include' parameters.

        (Also, the patch references a MR_NAME that is not present in my
        mr tree.)

Yeah :-(  I had to rebase about 20 times to separate out the patches
you are currently willing to consider, so mistakes were inevitable.

 4cd2b59d0c66d71316dfc1d411a3e3da439643bc
        I'm not quite sure of the point of this refactoring,

Legibility and modularity.  Longer functions makes unfamiliar code
harder to understand, and often give variables an unnecessarily wide
scope.  If I see this line of code within bootstrap():

my $tmpconfig = download($url);

then

  (a) it's immediately obvious that $url contains config which is
  being downloaded and stored in a temporary file, so there's no
  need for the existing comment Download the config file to a
  temporary location,

  (b) it makes it easy _and_optional_ for me to view how the download
  is implemented, and

  (c) it means that the scope of @curlargs and $curlstatus are clearly
  limited to the download, so I don't need to visually grep for
  them in 

variables for plug-ins

[Adam Spiers]

Hi Dieter,

On Thu, Dec 1, 2011 at 8:12 AM, Dieter Plaetinck die...@plaetinck.be wrote:
 On Thu, 1 Dec 2011 00:20:54 +
 Adam Spiers vcs-h...@adamspiers.org wrote:

       [shell-config]
       checkout = git clone ...
       stowable = true

 Is this the right way to go? that mr plugins expose themselves by
 introducing a new variable just to enable them?  I think vcsh was
 planning to do the same.  Why not have a 'type' variable or
 something which can be set to stow/vcsh/...

Thanks for the reply.  Just to clarify, the plugin is first enabled by
the standard mechanism:

include = cat $plugins_dir/stow

but this only has any noticeable effect on any repository when you add
the 'stowable = true' line to it.

However, it does not make sense to think of this as setting the 'type'
of the repository, because it's possible for a repository to utilise
multiple plug-ins at the same time.  For instance you could have a
repository which is stowable but is also retrieved via my 'download'
plugin.  So the only sensible definition of the repository 'type' is
whichever vcs_test returns true for it (and the stow plug-in does not
add a new vcs_test).

By the way, my patch to add support for ${action}_append makes it
easier for multiple plug-ins to cooperate:

https://github.com/aspiers/kitenet-mr/commit/append

since they can all extend actions in different ways without
accidentally overriding each other's behaviour.

One other point: currently this stow plug-in works by treating
'stowable' as a new action.  That's not great because it requires a
sub-invocation of mr.  A more efficient way would be to write in the
mrconfig file:

lib = mr_stowable=yes

and then have the stow plug-in do [ -n $mr_stowable ] tests.
However this is much uglier to read and write than stowable = true.

Therefore what I would like to propose is a new special parameter
called something like 'variables', which lets you declare new
non-action parameters which would then be efficiently accessible by
any shell code that wants to use them.  This would offer an easy way
to configure DEFAULT and per-repo behaviour for any given plug-in.

Plucking an example out of thin air, here's how you might configure
stow to use hard links for a particular repository instead of the
default of symlinks.  The stow plugin would contain something like:

[DEFAULT]
variables_append =
stowable
stow_links_type

and the user's ~/.mrconfig would contain:

[DEFAULT]
stow_links_type = soft

...

[repo]
stowable = true
stow_links_type = hard

Does that sound reasonable?
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: managing remotes collaboratively with mr

[Adam Spiers]

On Mon, May 2, 2011 at 6:36 PM, Joey Hess j...@kitenet.net wrote:
 micah anderson wrote:
  [DEFAULT]
  post_checkout = mr addremotes
  pre_update = mr addremotes

 Of course these run on all repos, like any mr command.
 So, what's really needed is:

 [DEFAULT]
 post_checkout = mr -d $MR_REPO addremotes
 pre_update = mr -d $MR_REPO addremotes

I know this is an old thread, but I thought it would be worth
sharing my solution, which is sort of like a hybrid between
Joey's and Yaroslav's suggestions:

https://github.com/aspiers/mr-config/blob/master/lib/git-add-remotes
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

new mr patches on their way (not quite ready yet though)

[Adam Spiers]

Hi Joey et al.,

I've been doing a lot of work on my stow plug-in for mr, and in the
process have not only resurrected GNU stow from the dead (last release
was 2002) but also ended up doing quite a bit of work on mr itself
too.  I've made a number of fixes to minor bugs, a few enhancements
(including the new --repos switch, for which I posted a preliminary
and somewhat broken patch to this list a week or 3 ago), and some
cosmetic changes.  I'm not quite done yet, but I thought it would be
prudent to give you guys a heads up anyway.  Hopefully most of the
patches are relatively uncontroversial, but I expect a few will
generate some discussion too.  Anyway, if you want a sneak preview
then you can see the patch series here:

  https://github.com/aspiers/kitenet-mr/commits/master

WARNING!  This branch *will* get rebased regularly, so I strongly
recommend that you do not attempt to merge from it at this point
otherwise you will almost certainly run into issues.  When I'm happy
that it's stabilised then I'll send another mail to the list.

Cheers,
Adam
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: mr: chaining to absolute paths

[Adam Spiers]

On Thu, Nov 3, 2011 at 5:46 PM, Joey Hess j...@kitenet.net wrote:
 Adam Spiers wrote:
     - Do you track your mrconfig files with version control?
 
  yes

 How do you do that?  Are they all in one repo?  How do you get each
 one into the right subdirectory of ~ ?

 They're checked out by mr as part of the repositories that provide the
 subdirectories they're in.

Thanks for the info, but I'm confused because that doesn't seem to
correspond exactly with the layout you gave earlier.  For example, you
said that you have a ~/doc/.mrconfig, but you didn't say that there
was a repository tracking ~/doc itself - only that ~/doc had various
document repositories inside it.

So is your ~/doc/.mrconfig tracked by a repository?  If so it sounds
like you have nested repositories, which git would presumably detect
as submodules unless you are using a symlink manager?

Thanks!  By the way there are good reasons for my curiosity ;-)  I'm
mulling over a similar layout but want to avoid nested repositories
and also prefer a symlink manager to detached git working directories.
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: various suggestions for mr

[Adam Spiers]

On Sun, Oct 30, 2011 at 1:26 PM, Adam Spiers vcs-h...@adamspiers.org wrote:
 On Sat, Oct 29, 2011 at 5:52 PM, Joey Hess j...@kitenet.net wrote:
 Adam Spiers wrote:
 However, the basename operation does not preserve the uniqueness
 property which $MR_REPO had, and that's why I say that we need an
 additional namespace.

 So pick an operation that does? tr / _ would do, for example.

 The other implicit requirement of this namespace was that it is
 easy to remember and type.  The rest of my previous email
 gives the context for this requirement.

OK, I've made a patch which fulfills this requirement pretty well.
Hopefully you'll find it reasonably clean and unintrusive:


https://github.com/aspiers/kitenet-mr/commit/b9a4e45aefe87c11ade1e4c4022e511f0d96d53c

With this patch, if you have .mrconfig files defining repositories:

[path/to/foo]
...

[path/to/bar]
...

then you can limit mr to only act on those via:

mr -r foo,bar $action

If there is a clash of directory names, then it can be resolved via a
new special parameter 'name':

[path/to/foo]
checkout = git clone ...
name = foo.git

[path/to/a/different/foo]
checkout = cvs checkout ...
name = foo.cvs

and then you can do:

mr -r foo.cvs update

etc.
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: mr: chaining to absolute paths

[Adam Spiers]

On Wed, Nov 02, 2011 at 05:02:13PM -0400, Joey Hess wrote:
 Adam Spiers wrote:
  I notice that chaining to absolute paths does not work, e.g.:
  Is this a feature or a bug?  I would have thought it would be useful
  to chain to absolute paths.
 
 Probably because nobody noticed since when you're in ~/foo/bar,
 ~/foo/bar/.mrconfig will be read anyway without chaining.

But ~/foo/bar is just a single repo, so if that's already your current
working directory, wouldn't you just end up invoking git (or whatever)
directly instead?

I'm beginning to suspect that the way I imagine using mr is
fundamentally different to everyone else's way.  Your previous point
about mr working best with locality of reference (i.e. each .mrconfig
being in a parent or near ancestor of the directories containing the
repos it manages) also contributed to this suspicion.  I can
understand how that makes for clean .mrconfig files with short
relative paths in the section headers, but I can't understand how you
could then version control all your .mrconfig files and share them
across computers.  And if you can't, then doesn't that discard a very
large part of the advantage of using mr in the first place?

I guess it would really help me if one or two people would be kind
enough to briefly describe the way they use mr, e.g.

  - How is your home directory structured, i.e. where do your mrconfig
files and repos live within it, and which mrconfig files point to
which repos?

  - How many mrconfig files and mr-managed repos do you have?

  - Do you track your mrconfig files with version control?

  - Do you frequently use the -d or -c options?

  - Do you usually cd to a particular directory before running mr, and
if so, why?

To me, it's important that a tool like mr can act on an arbitrary list
of repositories regardless of my current working directory.  In other
words, I'd rather choose which repositories mr should act on via the
parameters I'm invoking mr with, rather than by cd'ing to a particular
directory, running mr, then potentially having to repeat the cd + mr
steps multiple times.

 And there's rarely a reason to use an absolute path.

I can't see how to accomplish what I need without them.  Hopefully
answers to the above questions would help me realise what I'm missing.

 Fixed in git.

Great, thanks!
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: mr: chaining to absolute paths

[Adam Spiers]

On Thu, Nov 03, 2011 at 12:22:48PM -0400, Joey Hess wrote:
 Adam Spiers wrote:
  On Wed, Nov 02, 2011 at 05:02:13PM -0400, Joey Hess wrote:
   Adam Spiers wrote:
I notice that chaining to absolute paths does not work, e.g.:
Is this a feature or a bug?  I would have thought it would be useful
to chain to absolute paths.
   
   Probably because nobody noticed since when you're in ~/foo/bar,
   ~/foo/bar/.mrconfig will be read anyway without chaining.
 
 I probably meant to say ~/foo/.mrconfig fwiw.

Ah, OK.

[snipped]

  I guess it would really help me if one or two people would be kind
  enough to briefly describe the way they use mr, e.g.
  
- How is your home directory structured, i.e. where do your mrconfig
  files and repos live within it, and which mrconfig files point to
  which repos?
 
 Sure:

[snipped]

Thanks, this is really useful!

- Do you track your mrconfig files with version control?
 
 yes

How do you do that?  Are they all in one repo?  How do you get each
one into the right subdirectory of ~ ?

- Do you frequently use the -d or -c options?
 
 never
 
- Do you usually cd to a particular directory before running mr, and
  if so, why?
 
 I always run mr in the directory I want to affect. Sometimes this
 directory contains many repositories, sometimes only one. The point of
 mr is I don't need to care how many underlying repositories there are.
 If I run it in ~/src/d-i, I want to act on d-i; in
 ~/src/d-i/package/main-menu I'm only dealing with one package; in ~/src
 I want to act on all my source repos.

Very helpful and food for thought, thanks again.
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: various suggestions for mr

[Adam Spiers]

On Sat, Oct 29, 2011 at 5:52 PM, Joey Hess j...@kitenet.net wrote:
 Adam Spiers wrote:
 However, the basename operation does not preserve the uniqueness
 property which $MR_REPO had, and that's why I say that we need an
 additional namespace.

 So pick an operation that does? tr / _ would do, for example.

The other implicit requirement of this namespace was that it is
easy to remember and type.  The rest of my previous email
gives the context for this requirement.
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: how to refactor path duplication in .mrconfig section headers?

[Adam Spiers]

On Sat, Oct 29, 2011 at 2:11 PM, Richard Hartmann
richih.mailingl...@gmail.com wrote:
 On Sat, Oct 29, 2011 at 03:11, Adam Spiers vcs-h...@adamspiers.org wrote:
 I like that scheme with things like apache (sites-available
 vs. sites-enabled) although with mr config, the same effect can by
 automated via the special 'skip' parameter, which to me seems nicer
 than manually fiddling with symlinks.

 If you use skip, you will have to manually fiddle with mr to do the
 first checkout ;)

Why?  mr bootstrap http://... which then checks out the repo
containing all the mr config files, and then chain-loads them.  Each
repo subsequently loaded can then optionally have a skip parameter
which intelligently decides whether or not it should apply to this
machine (personally I do this by giving each machine a unique nickname
contained in ~/.nickname, and then building conditionals on that;
sometimes I'll also base it on the DNS domain name for repos which are
only relevant to a particular LAN).

 I still hope to sell Joey on the idea of incorporating link management
 in mr. A definite yes/no/onlyifyousubmitapatch would be nice, here.

I'd be willing to help if it's only if you submit a patch.

 If that doesn't work out, I will most likely extend vcsh to do this.
 Along with zsh completion to grab available repos from your local
 setup.

Yes, gotta have proper zsh completion ;-)
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: various suggestions for mr

[Adam Spiers]

On Fri, Oct 28, 2011 at 6:08 PM, Joey Hess j...@kitenet.net wrote:
 Having two namespaces for the same thing does not strike me as
 necessarily a good idea.

 But if you wanted to do that with mr, you could
 maybe take advantage of a little-known thing it does with determining the
 absolute path:

 joey@gnu:~mkdir namespace
 joey@gnu:~cd namespace
 joey@gnu:~/namespaceln -s ~/lib/sound
 joey@gnu:~/namespaceln -s ~/src/git-annex
 joey@gnu:~/namespacecd git-annex
 joey@gnu:~/namespace/git-annexmr update
 mr update: /home/joey/src/git-annex

I already did this; in fact I *had* to, in order to support GNU stow,
which requires the stow package namespace to be the list of
directories under a single stow directory.  If you look for
$STOW_PKG_PATH in the code I originally posted, you'll see:

STOW_DIR=$HOME/.cfg
...
MR_NAME=`basename $MR_REPO`
...
STOW_PKG_PATH=$STOW_DIR/$MR_NAME

and then post_{checkout,update} call the install() function which
does:

ensure_symlink_exists $STOW_PKG_PATH ${MR_REPO%/}

However, the basename operation does not preserve the uniqueness
property which $MR_REPO had, and that's why I say that we need an
additional namespace.  I can easily hack one, e.g.:

[$HOME/.my-repos/foo]
lib = MR_NAME=my-foo

[$HOME/.upstream-repos/foo]
lib = MR_NAME=your-foo

but that's ugly, and requires execution of shell code which would make
it difficult to implement a reverse lookup from package name to repo
path.

 The only problem with this approach is that it only work when inside the
 symlinked directory, so mr update in ~/namespace won't update the
 directories symlinked to there.

Right.  I feel like there's probably some elegant tweak we could make
to mr which would solve all this cleanly without much effort, but I'm
still trying to figure out what it is :-)

Let's examine the status quo.  Currently mr has:

  - a namespace for repositories (let's call it R) which is a subset
of the filesystem namespace, and values are specified in mr config
section headers,

  - a namespace for mr config files (let's call it C) which is a
different subset of the filesystem namespace from R, and

  - a 1:many mapping between C and R, i.e. each config file can
contain one or more repos.

I don't think there's anything wrong with this design - in fact that
1:many mapping is a nice configuration-time grouping mechanism, but
it's not quite enough to cater for some use cases.

For instance, sometimes it's just not acceptable to update all repos
at once, e.g. if you know that something is broken in repo X but you
really need the updates from repos Y and Z.

However mr cannot currently perform actions on repos at a finer
granularity than how they are grouped in config files.  So if you
wanted to update your 'zsh' and 'emacs' repos, you could only do that
if they were in a config file containing no other repos.  In general,
unless you restrict yourself to one config file per repo, there is no
way to get mr to operate on an arbitrary subset of R.

Similarly, if the 1:many mapping between C and R is used to logically
group repositories together (e.g. CLI, X11 and so on), there is no
way to get mr to operate on an arbitrary collection of groups.

Another issue is that whilst mr -c $config ls extracts mappings from
C to R, there's no way to extract the reverse mapping from R to C,
i.e.  what config file do I need to use in order to perform actions
on repo X?

In summary, maybe the tweak I'm looking for is simply to decouple
configuration-time grouping of repos from run-time selection of which
repos to act on:

[$HOME/.my-repos/emacs]
name = emacs

[$HOME/.my-repos/zsh]
name = my-zsh

[$HOME/.upstream-repos/zsh]
name = upstream-zsh

which would allow:

$ mr -r my-zsh,emacs

And then the reverse mapping from R to C could be extracted via:

[DEFAULT]
showconfig = echo $MR_CONFIG

(except at the moment, MR_CONFIG isn't changed via includes
which could cause a problem.)
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

various suggestions for mr

[Adam Spiers]

Hi all,

I've been tracking my dot files and related stuff since around 1999,
and was very excited to discover this mailing list two years ago.
Since then I've only been able to lurk, but finally have a bit of
spare time to participate.

Since 2005 I have been using my own (as yet unpublished) Perl suite
called cfgctl to manage my dot files, which serves a similar to
purpose to mr - applying operations such as checkout, update
etc. to multiple repositories using multiple VCS backends.  However
its shortcomings have increasingly irked me, and just when I was being
to resign myself to a complete rewrite, I noticed that mr had already
stolen most of my thunder :-)

The main differences between mr and cfgctl are:

  - cfgctl's repository meta-data is written in pure Perl (and then
require'd at run-time) rather than one or more .INI-style
.mrconfig files.  When I originally wrote it, needing
functionality akin to mr's skip parameter I naively thought that
this would give me maximum flexibility, and I mistakenly
underestimated the value of using a DSL.

  - cfgctl only has a fixup hook (i.e. post-checkout/update), and no
hooks for any other actions.

  - cfgctl is not nearly as extensible.  It doesn't support writing
arbitrary new actions either globally or per repo.

  - With hindsight cfgctl should have been made more shell-oriented in
general, because the kinds of tasks required for managing dot
files/repos, building/install packages etc. are much more easily
achieved in shell than in Perl.

  - Adding support for a new VCS backend is more work in cfgctl,
because it requires writing a new Perl subclass of Cfg::Pkg::Base.

  - cfgctl is harder to type than 'mr'.

So far mr is clearly winning :-)  However, cfgctl does have one or two
tricks up its sleeve:

  - Config modules / packages / repositories / whatever you want to
call them are indexed by name within a unique namespace, rather
than by directory path, and packages are grouped together into
sections.  This allows you to easily run any of the actions on:

  - all the packages (just like running mr from $HOME)

  - a single package, just by specifying its name without needing
to know where it lives, e.g. cfgctl --update zsh would
update just the zsh repository

  - a section (i.e. group of packages) just by specifying its name
(e.g. CLI or mail or Xorg) without needing to know where
anything lives, e.g. cfgctl --pull Xorg would update all
repos containing config relating to my Xorg (previously X11)
environment

  - any packages matching a regular expression e.g.
cfgctl --update /emacs/

  - cfgctl integrates out of the box with GNU stow.  Based on this
list's frequency of discussion on fake bare / detached git
repos, it seems that most people here have an aversion to the
symlink approach to managing dot files in $HOME.  This surprises
me as I have been using them very successfully for 12 years,
although I will leave that debate for another thread.

  - cfgctl's internals have a reasonable amount of pod / comments.
mr's code, whilst mostly self-explanatory at the high-level, does
have a few very long subroutines which I feel would gain clarity
by being refactored into some appropriately-named smaller subs.

All in all, I feel that mr has a better design than cfgctl, and has
greater longevity.  So last night I spent an hour or two doing a quick
proof of concept, to see whether I could extend mr to implement the
functionality I require, in particular the integration with GNU stow.
I'm pleased to say that so far it's looking very promising :-)
This is pretty much all that's needed:

- 8 - 8 - 8 - 8 - 8 -
[DEFAULT]
lib =
STOW_DIR=$HOME/.cfg
STOW_TARGET=$HOME
MR_NAME=`basename $MR_REPO`
#
set_stow_common_opts () {
STOW_PKG_PATH=$STOW_DIR/$MR_NAME
stow_common_opts=-t $STOW_TARGET -d $STOW_DIR
}
#
install () {
set_stow_common_opts
ensure_symlink_exists $STOW_PKG_PATH ${MR_REPO%/}
}
#
ensure_symlink_exists () {
[ $# = 2 ] || error CONFIG BUG: Usage: ensure_symlink_exists
SYMLINK TARGET
symlink=$1
required_target=$2
if [ -L $symlink ]; then
actual_target=`readlink $symlink`
if [ $actual_target = $required_target ]; then
return
else
error Symlink $symlink already points to
$actual_target, cannot point to $required_target; aborting.
fi
fi
[ -e $symlink ]  error Cannot create symlink $symlink -
already exists; aborting.
ln -s $required_target $symlink
}
#
stow () {
set_stow_common_opts
command stow $stow_common_opts $MR_NAME
}
restow () {
set_stow_common_opts
command stow -R -p $stow_common_opts $MR_NAME
 

Re: various suggestions for mr

[Adam Spiers]

On Fri, Oct 28, 2011 at 12:40 PM, Dieter Plaetinck die...@plaetinck.be wrote:
 Sorry that I go slightly off-topic but you mentioned Gnu Stow, I looked it up 
 and it seems very nice.
 i haven't run it yet, but (for those who don't want to read the long 
 description) from the description it seems like a simple and elegant tool,
 which you give a directory (i want symlinks here) and a bunch of package 
 directories (all files in here must be symlinked)
 and it will do the right thing on package additions and removals.

Yes, it is pretty nice (although the code base is very old-fashioned).

One thing it does which I guess other symlink managers wouldn't, is
called tree folding - i.e. it makes intelligent decisions about when
to symlink to a subdirectory vs. to individual files within that
subtree:

  http://www.gnu.org/software/stow/manual.html#SEC4

For example, imagine I have a stow package called 'foo' which wants to
install a file to ~/local/lib/perl/Acme/Foo.pm (by install, I mean
set up a symlink so that that path points to the Foo.pm inside the
stow package).  If ~/local doesn't already exist, and 'foo' is the
only stow package which installs anywhere under ~/local, then stow's
tree folding feature will ensure that ~/local is a symlink back to
$STOW_DIR/foo/local/lib/perl/Acme/Foo.pm.

However I might very well want to manually place other files inside
~/local which have nothing to do with stow, let alone the 'foo' stow
package.  It might make sense to have ~/local/lib/perl/Acme/ be a
symlink to $STOW_DIR/foo/local/lib/perl/Acme/, but symlinks higher up
the tree would be undesirable.  This is easily overcome by creating a
special stow package (which I call 'ANTIFOLD') which contains (empty)
dummy files called .no-stow-folding in those trees.  So I run a simple
little custom shell script:

$ unfold ~/local/lib/perl5

which creates an empty file

$STOW_DIR/ANTIFOLD/local/lib/perl5/.no-stow-folding

and (re)installs the 'ANTIFOLD' stow package, which causes stow to
automatically split the ~/local/lib/perl5 tree open so that

~/local/
~/local/lib/
~/local/lib/perl5/

are all normal directories, and ~/local/lib/perl5/Acme becomes the
symlink.  Problem solved!
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: various suggestions for mr

[Adam Spiers]

On Fri, Oct 28, 2011 at 2:43 PM, Dieter Plaetinck die...@plaetinck.be wrote:
 On Fri, 28 Oct 2011 14:34:16 +0100
 Adam Spiers vcs-h...@adamspiers.org wrote:

 On Fri, Oct 28, 2011 at 12:40 PM, Dieter Plaetinck
 die...@plaetinck.be wrote:
  Sorry that I go slightly off-topic but you mentioned Gnu Stow, I
  looked it up and it seems very nice. i haven't run it yet, but (for
  those who don't want to read the long description) from the
  description it seems like a simple and elegant tool, which you give
  a directory (i want symlinks here) and a bunch of package
  directories (all files in here must be symlinked) and it will do
  the right thing on package additions and removals.

 Yes, it is pretty nice (although the code base is very old-fashioned).

 One thing it does which I guess other symlink managers wouldn't, is
 called tree folding - i.e. it makes intelligent decisions about when
 to symlink to a subdirectory vs. to individual files within that
 subtree:

   http://www.gnu.org/software/stow/manual.html#SEC4

 IMHO this feature is just common sense and it's among the first things
 I think of when I'm thinking what would I want a symlink manager to do?,
 so I would expect that people who implement symlink managers either
 also do something like this, or at least list it as a todo.

 For example, imagine I have a stow package called 'foo' which wants to
 install a file to ~/local/lib/perl/Acme/Foo.pm (by install, I mean
 set up a symlink so that that path points to the Foo.pm inside the
 stow package).  If ~/local doesn't already exist, and 'foo' is the
 only stow package which installs anywhere under ~/local, then stow's
 tree folding feature will ensure that ~/local is a symlink back to
 $STOW_DIR/foo/local/lib/perl/Acme/Foo.pm.

 Don't you mean to $STOW_DIR/foo/local ?

Yes, sorry :)

 However I might very well want to manually place other files inside
 ~/local which have nothing to do with stow, let alone the 'foo' stow
 package.  It might make sense to have ~/local/lib/perl/Acme/ be a
 symlink to $STOW_DIR/foo/local/lib/perl/Acme/, but symlinks higher up
 the tree would be undesirable.  This is easily overcome by creating a
 special stow package (which I call 'ANTIFOLD')

 this seems a bit messy though. Once you go the way of having a tool 
 automagically
 manage all your symlinks, why not just have the discipline to put all your 
 files
 in appropriate packages? so that you never _need_ to create antifold 
 packages?
 what you're doing seems a bit like running into the opposite direction.

I don't understand what you mean; please could you elaborate?
As far as I'm aware, all my files are nicely separated into appropriate
packages, but that doesn't solve the problem.
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: various suggestions for mr

[Adam Spiers]

On Fri, Oct 28, 2011 at 3:26 PM, Dieter Plaetinck die...@plaetinck.be wrote:
 You wrote However I might very well want to manually place other
 files inside ~/local which have nothing to do with stow.

 Now you wrote As far as I'm aware, all my files are nicely
 separated into appropriate packages.

 That confuses me.  I was responding to your approach in which you
 describe how you make stow unfold by creating a dummy 'antifold'
 package which would allow you store files not managed by stow in
 your ~/local

Ah, I see!  Well, in an ideal world, I would have time to ensure that
every single file in my home directory is perfectly managed as a stow
package :-) But in practice is there anyone who really achieves that?
There are many cases where it just isn't worth the effort.  Perhaps
~/local was a bad example, because that path is suggestive of a
typical GNU-like installation sequence:

./configure --prefix=~/local
make install

in which case I agree it's not much more effort to do:

./configure --prefix=~/.stow/$pkgname
make install
# now install via stow

but there are plenty of other examples where it is not pragmatic to
spend the time ensuring every single file is controlled by stow, e.g.

./configure --prefix=~
make install# this puts some files in ~/doc

# days/weeks/months later:
emacs ~/doc/shopping-list.txt

Without a ~/doc/.no-stow-folding file present, suddenly you have
accidentally put your shopping list inside a third party piece of
software :-) And it's not worth having a stow package for a shopping
list which will probably be deleted tomorrow anyway.

Another approach would be to add an option to stow that disables tree
folding entirely; it shouldn't be hard to do that.  But personally I
find tree folding useful as long as it's managed well.
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

how to refactor path duplication in .mrconfig section headers?

[Adam Spiers]

If I have multiple repository paths all similar but spread across
different .mrconfig files, e.g.

in ~/.mrconfig

[.config/mr]
checkout = ...

in ~/.config/mr/config.d/CLI:

[$HOME/.git-repos/zsh]
...

[$HOME/.git-repos/mutt]
...

and in ~/.config/mr/config.d/GUI:

[$HOME/.git-repos/urxvt]
...

[$HOME/.git-repos/fonts]
...

then the DRY fascist inside me is irked by the duplication.  I'd much
rather write this in ~/.mrconfig:

[DEFAULT]
lib = MR_GIT=$HOME/.git-repos

[.config/mr]
checkout = ...

and then ~/.config/mr/config.d/CLI would have:

[$MR_GIT/zsh]
...

[$MR_GIT/mutt]
...

and in ~/.config/mr/config.d/GUI:

[$MR_GIT/urxvt]
...

[$MR_GIT/fonts]
...

However this doesn't work - I'm guessing that variable expansion in
the section headers is done in Perl, so is not influenced by the
shell-code in the DEFAULT lib?

So is there any other way of avoiding this duplication?

Thanks,
Adam
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: how to refactor path duplication in .mrconfig section headers?

[Adam Spiers]

On Fri, Oct 28, 2011 at 5:40 PM, Pieter Praet pie...@praet.org wrote:
 However this doesn't work - I'm guessing that variable expansion in
 the section headers is done in Perl, so is not influenced by the
 shell-code in the DEFAULT lib?

 So is there any other way of avoiding this duplication?

 This should do the trick:

  [DEFAULT]
  lib = export MR_GIT=$HOME/.git-repos
        ^^

That was the first thing I tried, but I'm pretty sure it didn't work,
hence my guess above.
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home

Re: how to refactor path duplication in .mrconfig section headers?

[Adam Spiers]

On Fri, Oct 28, 2011 at 8:03 PM, Richard Hartmann
richih.mailingl...@gmail.com wrote:
 On Fri, Oct 28, 2011 at 18:45, Joey Hess j...@kitenet.net wrote:
 mr can look much nicer if you take advantage of locality and chaining.
 By locality, I mean putting a mrconfig close to the directories it
 checks out, rather than in some standards-body controlled directory like
 ~/.config.

 You mean that ~/.git-repos contains the actual repos?

By you are you referring to me or Joey?  Yes, my intention
was that ~/.git-repos contains all git repos (or at least most).

 I.e. that every repo comes with its own .mrconfig?

If you mean, every repo *contains* its own .mrconfig, then
I don't see how that would work - wouldn't it create a chicken
and egg situation prior to first checkout?

 I used that approach at first, but
 I feel the symlinks (here they are ;) from ~/.config/mr/available.de
 to ~/.config/mr/config.d are more flexible.

I like that scheme with things like apache (sites-available
vs. sites-enabled) although with mr config, the same effect can by
automated via the special 'skip' parameter, which to me seems nicer
than manually fiddling with symlinks.

 Plus, you get central management of all mr configs.

Right on the money!  This is the real reason I want all my mr configs
in one place - I want them within a single git repo which I can
version control everywhere.  Then bootstrapping my entire environment
is as simple as:

[$HOME/.config/mr]
checkout = ...
chain = true

 That can be handy when you want to make a global update on a machine
 which does not have all your mr-controlled repos checked out.

If I understand that correctly, you are saying pretty much the same
thing as me.
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home