Re: Quieting ``mr`` output
Seems to me that mr -m status is what you seek. -- see shy jo ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: Spam on vcs-home wiki recentchanges page
Edward Betts wrote: The page http://vcs-home.branchable.com/recentchanges/ contains spam. Reverting it meant deleting recentchanges.mdwn. How do I get my change onto the wiki? There's a bug in ikiwiki where refresh does not notice a deleted page has exposed a page from the underlay. A rebuild has to be done to work around. I've done so in this case. Given the amount of spam on the recentchanges page, perhaps it should be locked? -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: [PATCH] Fix vcsh_trusted_checkout
Lorenzo Cappelletti wrote: Signed-off-by: Lorenzo Cappelletti lorenzo.cappelle...@gmail.com --- mr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mr b/mr index ddc0738..ecd1c02 100755 --- a/mr +++ b/mr @@ -2076,7 +2076,7 @@ bzr_trusted_checkout = bzr checkout|clone|branch|get $url $repo hg_trusted_checkout = hg clone $url $repo darcs_trusted_checkout = darcs get $url $repo git_bare_trusted_checkout = git clone --bare $url $repo -vcsh_trusted_checkout = vcsh run $MR_REPO git clone $url $repo +vcsh_trusted_checkout = vcsh clone $url $repo # fossil: messy to do veracity_trusted_checkout = vv clone $url $repo I don't really use vcsh, so my guess is that the former syntax was needed by an old version and it has been simplified to the new syntax? This change would break mrconfig files containing the old syntax. I don't know how common that is. It could support both syntaxes, by using vcsh_trusted_checkout_old to give the template for the old syntax. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: git-annex - documentation clarification - I've had data loss
Matthew Hannigan wrote: Anyway, time passes, and I decide to move all this content over to a different disk. I just used drag and drop from a gui (not sure whether windows or linux) and of course the select doesn't pick up the .git directory. Next I reformated the disk, and get a funny feeling that the copy went a little too fast ... Can I suggest that the following wording be changed from this: When you add a file to the annex and commit it, only a symlink to the annexed content is committed. The content itself is stored in git-annex's backend. to this: When you add a file to the annex, the content is moved to git-annex's backend (.git/annex/..) and replaced with a symlink. The commit only commits the symlink. I've made a similar wording change. Thanks for taking an unfortunate mistake and turning it constructive! The http://git-annex.branchable.com/how_it_works/ page might be improved the same way. It currently says The contents of large files are not stored in git, only the names of the files and some other metadata remain there. which is strictly true but might lead others to make the same mistaken assumption that I did. I think the next paragraph after that explains it well enough there. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: How to start syncing two existing directories with git annex?
Sean Hammond wrote: 1. The total number of files in ~/Annex, not including .git, on A and B is different: ls -R1 ~/Annex | wc -l 21830 ls -R1 ~/Annex | wc -l 21845 2. git-annex status shows untracked and modified files on both machines (different files on each machine). These seem likely to be related. Can you show the status? Are you using direct mode, or indirect mode? 3. On each machine, 7 files have been replaced with broken symlinks to files in .git/objects. This time it is the same files on both machines, so it looks as if git-annex might have lost these files from both machines. git-annex fsck finds these 7 and reports them as 'No known copies exist'. You run git annex log on some of these files to see the history of which repository they were in and how they moved around. For example: - 2013-11-27 10:01:14 Parallel_and_Concurrent_Programming_in_Haskell.epub | d322dff8-8b32-11e0-bbce-bb98bc1ede5b -- wren + 2013-11-25 17:40:59 Parallel_and_Concurrent_Programming_in_Haskell.epub | 7e88d964-437e-47be-885a-e158af656729 -- darkstar + 2013-11-25 17:13:34 Parallel_and_Concurrent_Programming_in_Haskell.epub | d322dff8-8b32-11e0-bbce-bb98bc1ede5b -- wren + 2013-11-25 17:13:34 Parallel_and_Concurrent_Programming_in_Haskell.epub | ----0001 -- web So, this would also let you see where the last copy resided, and when it was recorded as being dropped. Although git-annex should never drop the last copy of a file on its own in normal circumstances. 4. Even after running git gc --aggressive --prune and git-annex dropunused, the .git directories are massive: 23G and 2.5G, for just ~20,000 files. Are you looking at the sizes of the .git/objects directories, or the .git/annex/objects directories? (.git/annex/tmp is also a possible place where cruft could somehow accumulate) When you ran git annex dropunused, did it drop something? git annex unused should not find any unused files if you've just synced 2 directories, and never deleted any of the files yet. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: How to start syncing two existing directories with git annex?
Sean Hammond wrote: I think it worked. Git annex assistant did a lot of transferring, and when it was eventually done the number of files and size of the ~/Annex/Music dir is exactly the same on both machines, and the files look fine. The ~/Annex/.git dirs are quite big though: 640M and 1.5G. I ran git gc --aggressive --prune on them, and got them down to 180M and 894M. That still seems surprisingly large but I guess it's not big enough to cause a problem. It's possible that using `git annex unused` will find some old files in the larger of the two. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: status of sharebox-fs?
René Mayrhofer wrote: No, I punted on it. The inotify managed directory will behave differently/annoyingly when the user tries to modify files. This certianly doesn't perfectly cover every use case, but I feel it's an ok tradeoff, you can get used to that behavior. There is some event coalescing code towards making file changes less annoying in my current dvcs-autosync devel branch (on gitorious). However, due to time constraints, I haven't yet gotten around to implementing the corner cases that I wanted to for about 4 months (Hint: unpacking a big tar file is something I don't want to see being split into as many separate git commits as there are files, while I would like a commit for every change to a document I am currently editing - there's a trade-off in there when only passively monitoring the changes via inotify.) If you'd like to talk about a few of the issues, I hope to find time to do so (life will remain crazy for the next about 4 months). I've been playing with the inotify-triggered commits idea for a while and already bumped my nose at a few ugly cases that I might assist you in avoiding. Would absolutely be appreciated. I have been planning on only committing when it syncs, or something like that, rather than on every new file. It seems to make sense to commit every time an existing file is changed, but bundle new files. As far as I can tell, inotify has an unavoidable race when a new directory is created -- before the program gets a chance to add a watch on the directory, files or directories can be added to it, with no inotify events generated for them. There is a workaround, manually recursively scan each new directory after adding the watch. If you have other gotchas like these, do tell.. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: One git-annex repo refuses to see if remotes have files
Adam Spiers wrote: Looks to me like you got the order wrong - don't you need to commit and push *before* the git annex copy? Otherwise the meta-data for doc.2012.tar.gz isn't in the git-annex branch when the annex copy is done and it will just copy the previous contents of oldscm to metaarray, without the new file. Nope, once a file is git-annex added, the data is in git and it can be manipulated with all git-annex commands, even if the staged file has not yet been committed. zephyr$ git commit -a git push My guess is that this push is not pushing the git-annex branch. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: intercept file access to git-annex repos
Thomas Koch wrote: Hi, still about photos and git-annex: When my wife edits photos prior to uploading them, it would be fine if she wouldn't need to know about git-annex. Hacking git-annex support in every photo application is no option. But is there any linux kernel or LD_PRELOAD magic that could intercept access to a git-annex repo and make it look normal? (I just realize that we already talked about a fuse filesystem for git-annex.) http://git-annex.branchable.com/todo/automatic_bookkeeping_watch_command/ I have a proof of concept branch using inotify, however, to make it truely end user friendly with syncing is going to be a lot more work. I have been thinking about doing a Kickstarter to pay for 3 months of my time to seriously tackle this. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: more git-annex plumbing
Thomas Koch wrote: Hi, I have a few things that I'd like to do with git-annex but that are hard because git-annex does not expose internal plumbing commands. I intend to start a list of such commands here with possible use cases: - calculate checksum of a file - doesAnnexHasChecksum? - doesAnnexHasFile? Useful to deduplicate files from many old CD backups git annex find --format can be used to get at this to some extent. But I'd instead suggest to just add the files, let git-annex deduplicate the data, which it does automatically, and if desired dedup the filenames afterwards. - checksum to path - move file into annex You already have an annexed file somewhere on the disk and just want to move it in the local annex without copying it Sounds like the reinject command, I think? - find duplicates It might happen that a file is put into an annex in multiple places. One could try to find duplicate symlinks, but that's inefficient. As I understand, the git-annex data should allow a quick lookup of all symlinks pointing to a particular checkum. http://git-annex.branchable.com/tips/finding_duplicate_files/ -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: Photo gallery with git-annex?
Thomas Koch wrote: I'm thinking, how a simple web photo gallery setup could be possible with git- annex, a static gallery generator like lazygal (Debian packaged) and some scripting. But to make this a viable alternative to facebook for my wife, I'd additionally need (or like): - comments - non-public galleries with openid/facebook/twitter login - web based uploads? I think that integrating git-annex support into ikiwiki, and then folding in the contrib gallery plugin could be a good way to go. ikiwiki already has web-based uploads (drag-n-drop-to-browser even). Tho it would not cover your non-public case, unless you did something tricky with apache htaccess. Unfortunately I have not been able to get the authors of git-annex and ikiwiki to sit down together and integrate their software yet. -- see shy jo ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: Photo gallery with git-annex?
Paul Stodghill wrote: 1. Do something with symbolic links other than ignore them. Note that this is a potentially difficult security situation. One idea that has been floated is to keep the annexed files in a different repo, in an ikiwiki underlay. It would be easier to make ikiwiki skip security checks that prohibit symlinks that way. -- see shy jo ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: checking which files on a CD are not in a git-annex repo
Thomas Koch wrote: It'd be of course wonderful if I could tell git-annex directly to import all files of the disc. Duplicate files should symlink to the same file in the git- annex backend, shouldn't they? Yes. If you don't mind the overhead of copying all the files, simply copying the whole CD to a subdirectory and running git annex add will do the trick. Any duplicate files will coalesce when added. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: insteadOf vcsh://
Yaroslav Halchenko wrote: so I could use the same mr configuration across boxes even where I do not forward my ssh key, while leaving me an opportunity to push later on happen I need that (not to mention that why to bother with ssh tunneling for pulling if straight git: is available) It also makes mr update -j20 a lot faster, as well as not opening 20 ssh connections to some single server, which sometimes causes ssh cracking detectors to ban an IP. :) -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: mr: breaking down complex subroutines
Adam Spiers wrote: I find some parts of the mr script quite hard to follow, and consequently to write patches for. For example, loadconfig() has 221 lines, and is 8 levels of indentation deep at certain points. Would you accept extract subroutine refactoring patches which do not change the functionality in any way, but simply break down some of the longer subroutines into more easily understandable chunks? I'm always glad to accept good refactorings. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: Strange mr/vcsh error on Debian Squeeze
Sean Whitton wrote: Thank you for your reply, and sorry for not thinking a little more before e-mailing. Here is the output. I think that this may be a vcsh problem rather than a mr problem after looking at this. Looks to me like a mr version older than 1.09 being used with a mr include file for vcsh that's written for that version. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: Spaces in remotes and git annex
Klaus Ethgen wrote: I seems to be predestined to find strange problems. ;-) Today I cloned a remote git annexed repository that has a space in path. With git I have to use one backslash instead of three that are needed in scp. Now I tried to get a document from that repository and failed as git-annex seems to use scp and need all three backslashes. Is there any way to work around that problem? I've pushed a change that improves git-annex's support for weird urls like this: url = ssh://localhost/home/joey/tmp/foo bar url = localhost:tmp/foo bar Before, git-annex would seem to ignore the first of these (it actually thought it was a local directory!), and complained the second was a bad url, now both are supported and work in my tests. Maybe this will fix your problem. If not, what kind of repository is it, a git repository, or a special remote, and what does the url for it in .git/config look like? -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: git annex add silently ignores non-existent files
Richard Hartmann wrote: On Sat, Dec 24, 2011 at 15:44, Joey Hess j...@kitenet.net wrote: Since using git-ls-files is so convenient in most ways, all I can think to do about this is document it. --error-unmatch ? Produces a strange error message if run on files that are already in git: joey@gnu:~/src/other/gitgit ls-files --error-unmatch --others zlib.o zlib.c zlib.o error: pathspec 'zlib.c' did not match any file(s) known to git. Did you forget to 'git add'? -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: git-annex: Puzzling over bare repositories
Richard Hartmann wrote: On Thu, Dec 15, 2011 at 20:44, Richard Hartmann richih.mailingl...@gmail.com wrote: You need to run git annex merge before pushing and all will be well. This seems to be a _very_ common problem for new users. I know it's a message from git, not git-annex, but would there be any way to display a hint? As a follow-up, there are no hooks that could be used. Pity. To follow-up, I have added a new one, called tweak-fetch. Hopefully it will be accepted into git in due course; I already have a tweak-fetch branch of git-annex that can use the hook to avoid any need of manually running `git annex merge`. The hook also allows for other interesting possibilities.. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: [mr] bash-completion rules
Adam Spiers wrote: there is a bigger cost - the risk of having a version of the completion rules which does not match the version of mr installed. This is, in practice, not a large problem, and can be dealt with by distribution integrators. There's also a converse argument. Completion functions are not only coupled to the thing they are completing for, but also to the shell's completion API. When the API changes, it's better to have completion functions within the shell's distribution, because the shell's developers can fix all completion functions to work with the new API in one go. Which is why I would certianly not like to bundle zsh completion functions with the programs they complete. You have to be a zsh guru to write them, they have changed a *lot* over the years (I don't recognize anything in the current dpkg completion that's left from the one I originally wrote), and upstream is very responsive, to keep the completions up-to-date. I suspect that bash completion will head in a similar direction, as they get reworked to support dynamically loading completions on demand, per http://thread.gmane.org/gmane.comp.shells.bash.completion.devel/3375 With that said, putting a bash completion in mr now just means a little probable pain later on, so I'm not strongly opposed. The real difficulty in completing mr is that it accepts an arbitrary set of subcommands, even depending on what repository it's run in. In practice, I just type abbreviated things like mr up and mr p instead of reaching for the tab key; happily mr will accept any nonambiguous abbreviations and can be taught others. :) -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: git annex add silently ignores non-existent files
Adam Spiers wrote: git annex add this_file_does_not_exist does not result in a warning. This leads to confusing (lack of) behaviour in certain cases, e.g. generate_a_list_of_files_some_of_which_contain_spaces | xargs git annex add would silently fail to add the files containing spaces (because you'd need the -d or -0 switch to xargs to cope with this case). This can indeed be confusing; it's a result of git-annex using git-ls-files --others. joey@gnu:~/src/other/gitgit ls-files --others zlib.o fofofofoofof zlib.o Since using git-ls-files is so convenient in most ways, all I can think to do about this is document it. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: inventory of files unavailable in local repository?
Adam Spiers wrote: Furthermore, git annex whereis --not --in= lists all files, not just the ones which aren't locally available. joey@gnu:~/lib/soundgit annex whereis --not --in= git-annex: no remote specified If your mailer is eating trailing periods or something.. whereis --not --in=. works as expected here, only displaying anything for non-present files. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: inventory of files unavailable in local repository?
Adam Spiers wrote: On Fri, Dec 23, 2011 at 5:13 PM, Joey Hess j...@kitenet.net wrote: Adam Spiers wrote: I don't get that error message. version? 3.20111211 So, your git-annex version was, apparently, seeing --in= as in the remote named '', and doing the expensive query of git for that info.. or something. Hmm, I vaguely think I might have fixed a bug like that at some point. Maybe not released yet? Aha.. no, what's really happening is you have a remote with a description of , and so it thinks that is the one you mean. I have added a special case to not match on such empty descriptions. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: git-annex: can pull, but can’t push
Sean Whitton wrote: On my second machine, my laptop, I don’t seem to be able to push to the centralised repository: I am getting the error one gets when one hasn’t yet done a pull and done a merge, but I definitely have: | ! [rejected]git-annex - git-annex (non-fast-forward) You probably need to run git annex merge. If you pull and then immediatly push, git-annex does not get a chance to run, so its behavior of normally transparently handling merging of its branch doesn't happen. So the sequence needs to be pull, git annex merge, push I'd like to get a post-pull hook into git so git annex merge can run automatically after pulls, but it's not there yet. (Git's post-merge hook could be used but does not always get run after a pull.) -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: Is there any interest in a patchset for mr to manage which repos are being handled?
Adam Spiers wrote:
Firstly, I built a library of skip functions:
https://github.com/aspiers/mr-config/blob/master/lib/skippers
which lets me write things like:
[$HOME/.GIT/adamspiers.org/gnupg.sec]
skip = default_skipper || missing_exe gpg
I'm with you so far; this is how I use mr, so in a way it's how mr is
designed to be used.
However, in the upstream mr, this is not fully implemented yet because
it does not prevent checkouts of lazy repositories:
http://thread.gmane.org/gmane.comp.version-control.home-dir/396/focus=398
To solve this, I knew mr would need a mechanism for referring to a
single repository, which in turn would require a new namespace for
repositories.
This still seems a roundabout way to solve that problem.
Why not just:
lazy() {
if [ $MR_ACTION = checkout ]; then
if [ $MR_FORCE ]; then
return 1
else
echo skipping checkout of lazy repo (set
MR_FORCE=1 to enable)
return 0
fi
elif [ -d $MR_REPO ]; then
return 1
else
return 0
fi
}
Then maybe make --force set MR_FORCE, and to enable one you just:
mr --directory somerepo --force checkout
except that it's more direct, since if you enable 'foo', surely you
would checkout 'foo' immediately after. Then the only missing piece
is 'disable'. Personally I don't need this (yet, at least). But if
you really needed it, the lazy() skipper could easily be extended (or
a new skipper written) to perform an extra check:
test -d .mrdisabled
rm -rf seems a good way to disable a lazy repo.
--
see shy jo
signature.asc
Description: Digital signature
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home
Re: Cabal won’t install git-annex, complains about Control.Monad.IO.Control
Sean Whitton wrote: Currently reviewing my setup; switching from madduck’s vcsh to RichiH’s, and from unison to git-annex for my ~/var/ directory of massive media files and backups and the like. Liking it so far. Have installed ghs using my distribution’s package (CRUX, it’s source-based) and have installed cabal, but I get this error when running the installation command suggested on the git-annex wiki: , | Configuring git-annex-3.20111203... | Preprocessing executables for git-annex-3.20111203... | Building git-annex-3.20111203... | | Annex.hs:25:8: | Could not find module `Control.Monad.IO.Control': | Use -v to see a list of the files searched for. | cabal: Error: some packages failed to install: | git-annex-3.20111203 failed during the building phase. The exception was: | ExitFailure 1 ` Anyone know why this might be? This is my first experience with Haskell. I have a new-monad-control branch that fixes this; you can get it from git for now. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: New integration branch
Adam Spiers wrote: 9c87f2352214175de307efedb8fd93889a26afbc Can you give an example of when this is needed? I can't remember but I definitely saw it happen at least once :-/ My worry is that, since that really shouldn't happen AFIACS, you were actually seeing a bug. Either that or it's a corner case I have not identified. 602f26714254f3c65389b7665d15d1d5d0e227a9 mr is quite typically (I know, not by you) run inside the repository. Which would leave the user in an apparently empty directory after mr update if an mr update deleted and remade the whole repository. I don't like that; I don't think things in mr should be deleting repositories in general; mr doesn't even delete a repo that has deleted = true, it only warns the user about it. Hmm, that's a fair point, although the only alternative is to change the contents of the directory rather than the directory itself - similarly to how `git checkout' does, for instance. I'll see if I can get around to doing that. Perhaps some of the effort could be reused for implementing download_diff (diff against the archive file). I think you could just use rsync :) cf3388f443b9d7afe6dc7d8a2159b45fb01ab4e4 This is a slow way to make machine-parsable info available -- the similar mr list takes 8 seconds here, since it has to run 169 shells. That's ok when you're just running mr, but I would not like to use a command that depended on that information. Sure, that's why I used a simple on-disk cache: https://github.com/aspiers/kitenet-mr/commit/b60acb2e767b91ca6d406198d7eea1b3f73ad2bf It works fine. I could get more sophisticated and allow per-user configuration of the cache invalidation strategy, e.g. so that it would automatically rebuild the cache when ~/.mrconfig et al. are changed, but manual rebuilds aren't a great hardship. In fact I could even rebuild the cache every time mr runs! If a machine-parseable list of repositories is needed, I think it'd be better to have a perl function that emits it in one go. I don't see how that's possible without ignoring the `skip', `deleted', and `include' parameters. The include parameter is not a big problem, it's unlikely to require more than one shell process, which will be relatively fast. It's not clear to me what should be done about skip and deleted. skip in particular can behave in weird ways, when something like hours_since is used. To handle that all the skips would need to be tested, which would be less work than mr list but still verging on expensive. Depending on the application, it might be better to just dump all the defined repositories including skipped and deleted ones; if the consumer than runs mr in a skipped/deleted repo, mr will do something sane after all. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: alternative mechanisms for including config
Adam Spiers wrote: This may be a good time to discuss the design of the `include' parameter. When you were deciding what its value should be, I guess there were at least three possibilities: (1) a chunk of shell-code which returns the actual shell-code to include (2) a chunk of shell-code which returns a list of names of files to include (3) a delimited list of files to include You went with (1). One advantage of this is the ability to dynamically generate code to include. But this could also be achieved with (2), by generating the files to include and then returning the names of the generated files. Also, with (1), if the shell-code has an issue it's harder to debug because there's no containing file (and line number and surrounding lines) to refer to. The main advantage of (3) is that you don't have to execute any shell code at all. This would facilitate implementation of your suggestion of writing a Perl function to emit the repo list, although there's still the problem of the `skip' parameter, and I suspect too many users are already relying on the dynamic nature of `include' for (3) to be feasible. But might it be worth implementing (2) alongside the existing (1), via a new `includefiles' special parameter? I've made mr show the included line content in error messages now. The speed hit of running that one shell command is minor. It doesn't seem worth bothering users with deprecating the current include, and needless complication to have a separate way with a list of files. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: New integration branch
Joey Hess wrote: It could well not be. mr list -j 10 runs in the same time as mr list -j 1, suggesting the overhead is in something else than actually running the shell. Whoops, bad benchmark, -j comes before action. Anyway, yes, without any calls to system(), mr list takes just 0.35 seconds. Those calls are: 169 mr list: running set -e; # actual list command 118 mr skip: running vcs test # 55 mr list: running skip test set -e; 50 mr deleted: running vcs test (Note that the vcs test is split between skip and deleted, but if those features are removed, the actual list command would trigger the same test, so those don't add overhead.) Moving the git_test etc into perl code would be one way to speed it up for the common case. Adding a special case optimisation to avoid the shell for true and false brings mr list down from 8.50 to 1.81 seconds. The remaining time is here spent running skip tests, I have a lot. Probably looking at sub-1-second times for most people. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: New integration branch
Joey Hess wrote: Moving the git_test etc into perl code would be one way to speed it up for the common case. Adding a special case optimisation to avoid the shell for true and false brings mr list down from 8.50 to 1.81 seconds. The remaining time is here spent running skip tests, I have a lot. Probably looking at sub-1-second times for most people. These optimisations are now in place. joey@gnu:~/src/d-itime mr -q list 1.14user 2.17system 0:05.12elapsed 64%CPU (0avgtext+0avgdata 26368maxresident)k 0inputs+0outputs (0major+269034minor)pagefaults 0swaps joey@gnu:~/src/d-itime ~/src/mr/mr -q list 0.38user 0.02system 0:00.44elapsed 91%CPU (0avgtext+0avgdata 26640maxresident)k 0inputs+0outputs (0major+6429minor)pagefaults 0swaps joey@gnu:~time mr -q list 1.67user 3.86system 0:08.75elapsed 63%CPU (0avgtext+0avgdata 26720maxresident)k 0inputs+0outputs (0major+464487minor)pagefaults 0swaps joey@gnu:~time ~/src/mr/mr -q list 0.56user 0.60system 0:01.78elapsed 65%CPU (0avgtext+0avgdata 26800maxresident)k 0inputs+0outputs (0major+84959minor)pagefaults 0swaps -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: git annex --help doesn't show manual page
Adam Spiers wrote: $ git annex --help No manual entry for git-annex Is this issue related to the fact that I installed git-annex with cabal install? Yes, as far as I know, cabal does not have a way to handle man pages. make install does install one, that git brings up when you run this (git-annex never runs actually). -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: git annex map does not spot graph loops
Adam Spiers wrote: I set up two git annex repos on the local machine which point to each other and then run git annex map, it chews up a load of CPU, presumably trying to traverse the cyclic repository graph without ever noticing there's a loop: Fixed, it only happened when the repos referred to each other with relative paths. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: Removing git-annex Repo
Klaus Ethgen wrote: is there a other way than git checkout git-annex; sed -i -e '/hash/d' **/*(.); git commit -a; git checkout master to remove a repository completely from annex knowledge? Well, that doesn't actually work; if you do that and then pull a independantly changed git-annex branch from a remote, the auto-union-merge will then add back all the lines you removed. Marking the repository untrusted is the usual way. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: new mr patches on their way (not quite ready yet though)
Adam Spiers wrote: I've made a lot of progress since this last mail, and I'm conscious that my branch is now approximately 50 commits ahead of the official master branch, so I think it's time to work on convergence if possible. It would be helpful if you: * separated the stow stuff into its own branch When adding a feature, I need something I can diff against; a chain of commits is ok, or a squashed commit is ok; a chain of commits mixed in with other unrelated changes is not * wrote commit messages that were longer than one line I need to understand a commit before I can apply it, and the choice is either that you type a little but more, or that I spend significant time guessing and likely miss something. A commit such as a2515e7e89f35c8d3291da9a5908b42a8d0bb277 is simple on its face, but is entirely lacking in an explanation of why the change was necessary; in what situation is there a dangling symlink and how did mr fail? Why is adding an error message of BUG: this shouldn't happen justified? Similarly, commit 655f0002ae80e21329ace97447a3a16c577949ec says it fixes a small bug, but neglects to say what the bug is. A commit such as 49163f09b8ff2c70c64076040be772b8d37c84aa or 1dd662640b946d681683c260f1b693cd0522b09f needs significant justification -- how does hardcoded VT100 color codes or a lot of different debug levels make mr better? Only a patch such as 96f2c8875bba4f7225decb60ee905815e2aeaa4a doesn't need more than one line of explanation. * avoid entangling different lines of development I was about to cherry-pick c4a8af985f525c2a1061576e72d526aa515151be until I noticed the last line ties it into the logging level stuff. Here's a summary of the main features of my branch: - A plug-in module for integration with the GNU Stow symlink manager, now well-tested and stable. I have recently become co-maintainer of GNU Stow, and its current git master branch and next official release contain enhancements which make it compatible with mr out-of-the-box. This makes tracking your $HOME dotfiles as easy as: [shell-config] checkout = git clone ... stowable = true Is there really no way to detect that a given repository is stowable by looking at the content of the repository or some other thing? Why not? - A plug-in module which allows tar balls / zip files etc. to be downloaded, unpacked, and used as repositories with minimal effort, e.g. [foo] checkout = mr_download_checkout http://example.com/foo.tar.bz2 A weird feature IMHO, but I use lib/* is sort of a contrib area and I'm willing to accept most any type of module into it. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: new mr patches on their way (not quite ready yet though)
Adam Spiers wrote: Thanks for the reply. Sure - I can syphon those commits off into a stow branch. It bothered me too that they were non-contiguous. In fact, I appreciate that just dumping a whole load of uncategorised commits on your doorstep isn't particularly helpful, so maybe it's better if I create a branch for each change, and then point you at each branch. Would that work for you? There's no need for a separate branch for self-contained changes. Regarding color: well, I'm not sure on what level the question is asked, so forgive me if I'm treating a more technical question as a philosophical one ;-) If it's whether the use of color makes the program any more useful, then I guess it's a matter of individual taste, and I'm aware that some people don't like it. Each to their own, but personally I find that rather mystifying - after all, how many people do you know who use a web browser which renders everything in black and white? Or even within the terminal / CLI environment, how many people dislike the colors generated by ls(1) or git enough to go to the trouble of disabling them? I have not looked at the colors in use here, but IME adding color to terminal programs is often badly done, resulting in an angry fruit salad effect, and often needing additional configuration to disable it, or to tweak the colors to ones visible for colorblind users or users who cannot read yellow text on a white background. Which I can put up with in a mail reader, git diff, or a text editor, but given the very small amount of output done by mr, seems likely excessive. I suppose the idea is to pick out errors from amoung the rather larger amount of output displayed by the version control system, but mr already tries to structure its output to make it easy to do that. Regarding debug levels: without a more fine-grained approach to logging, it would have been too hard for me to understand mr's internals and achieve all the development I was able to. A large part of the challenge for me was understanding the order in which mr parsed / executed things etc. and the original -v was way too verbose to be able to trace this - it would have required me to keep too much stuff in my head at once. In fact, there were a few minor bugs which I spotted precisely because I entered this debugging exercise. Having said that, I freely admit that a debugging system based on numeric levels isn't particularly elegant or sophisticated, and I'd be happy to see something better. But it's a very standard technique industry-wide (c.f. syslog and hundreds of other logging software projects which use the concept of priorities or severities), and it was also The Simplest Thing That Could Possibly Work. I have never seen such a debugging system in which I did not use exactly two modes: none, or turn the dial to 11 and grep for the thing I actually wanted. Because it's up to the end user whether he wants to stow the repository or not, and that decision is entirely dependent on the context in which the repository is being used. But is there no way to tell if a repository is stowed? Why couldn't the checkout just handle registering it with stow (or whatever), and then the other actions could check to see if it was so registered. Wanting to use mr to manage the download and installation of a piece of software is weird? Or the implementation is weird? Wanting to use mr to manage software not in version control is weird. :) -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: new mr patches on their way (not quite ready yet though)
Adam Spiers wrote: OK. I'm not entirely sure I understand what you want though. How would you define self-contained in this context? Any patch that does not depend on any other patch. IMHO it's an impossible task without color. Scanning for a new paragraph (\n\n) is an impossible task? -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: Doc on creating a remo repo without cloning
Thomas Koch wrote: Hi, I'm starting to learn git-annex and tried by creating one local git-annex enabled repo on my laptop. Then I wanted to create another non-bare repo on my server to push to it. I can not access my laptop from the server, since I'm sitting behind a NAT. However there are at least two hurdles when I want to create another repo without cloning: - I can not push to the remote repo without setting receive.denyCurrentBranch ignore Push to a bare repository, and clone the bare repository. - When I do a simple git push, git tries to push the git-annex branch. However I understand that the git-annex branch is local to each repo and should not be shared? I get non-fast-forward complains on a push attempt. No, the git-annex branch contains its state which must be shared between all repositories, so it knows which repositories have which files. You may need to run `git annex merge` before pushing this branch. Most of the time, git-annex automatically does this merge. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: mr: chaining to absolute paths
Adam Spiers wrote: Thanks for the info, but I'm confused because that doesn't seem to correspond exactly with the layout you gave earlier. For example, you said that you have a ~/doc/.mrconfig, but you didn't say that there was a repository tracking ~/doc itself - only that ~/doc had various document repositories inside it. Every directory I showed is a separate repository. So is your ~/doc/.mrconfig tracked by a repository? If so it sounds like you have nested repositories, which git would presumably detect as submodules unless you are using a symlink manager? git does not detect nested git repos as submodules, that would have to be explicitly set up. -- see shy jo ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: mr: chaining to absolute paths
Adam Spiers wrote: On Wed, Nov 02, 2011 at 05:02:13PM -0400, Joey Hess wrote: Adam Spiers wrote: I notice that chaining to absolute paths does not work, e.g.: Is this a feature or a bug? I would have thought it would be useful to chain to absolute paths. Probably because nobody noticed since when you're in ~/foo/bar, ~/foo/bar/.mrconfig will be read anyway without chaining. I probably meant to say ~/foo/.mrconfig fwiw. I'm beginning to suspect that the way I imagine using mr is fundamentally different to everyone else's way. Your previous point about mr working best with locality of reference (i.e. each .mrconfig being in a parent or near ancestor of the directories containing the repos it manages) also contributed to this suspicion. I can understand how that makes for clean .mrconfig files with short relative paths in the section headers, but I can't understand how you could then version control all your .mrconfig files and share them across computers. And if you can't, then doesn't that discard a very large part of the advantage of using mr in the first place? I guess it would really help me if one or two people would be kind enough to briefly describe the way they use mr, e.g. - How is your home directory structured, i.e. where do your mrconfig files and repos live within it, and which mrconfig files point to which repos? Sure: ~ .mrconfig doc .mrconfig (various document repositories) src .mrconfig (many package sources) d-i .mrconfig lib/backup .mrconfig (only exists on a few machines, various repositories) - How many mrconfig files and mr-managed repos do you have? 190 repos, mostly in src - Do you track your mrconfig files with version control? yes - Do you frequently use the -d or -c options? never - Do you usually cd to a particular directory before running mr, and if so, why? I always run mr in the directory I want to affect. Sometimes this directory contains many repositories, sometimes only one. The point of mr is I don't need to care how many underlying repositories there are. If I run it in ~/src/d-i, I want to act on d-i; in ~/src/d-i/package/main-menu I'm only dealing with one package; in ~/src I want to act on all my source repos. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: mr: chaining to absolute paths
Adam Spiers wrote: - Do you track your mrconfig files with version control? yes How do you do that? Are they all in one repo? How do you get each one into the right subdirectory of ~ ? They're checked out by mr as part of the repositories that provide the subdirectories they're in. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: mr: Lazy processing of repositories
Svend Sorensen wrote:
I'm using mr to manage the repositories for the various software that I
track. However, I don't want to check out all the repos by default. (The
list is getting long). I also don't want to make a modification to the
.mrconfig each time I want to check out a repo. (E.g. have 'skip = true'
on all repos, and remove the skip if I decide to check out a repo.)
I'm looking for a way to do lazy checkouts and updates. A lazy repo
would not be checked out unless I run a command to tell mr to check out
the repo. (Something like mr checkout foo.) After I force mr to check
out a lazy repo, mr would act on the repo for future mr runs.
One way to do this is to have skip check for the existence of a
different file for each repo. Creating the file would activate the repo.
Any ideas for a better approach?
Good idea! In mr git, you can now use skip = lazy to get this behavior.
The lazy shell function is built into mr, but this shows how it works:
lazy() {
if [ $MR_ACTION = checkout ] || [ -d $MR_REPO ]; then
return 1
else
return 0
fi
}
--
see shy jo
signature.asc
Description: Digital signature
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home
Re: mr: Lazy processing of repositories
Svend Sorensen wrote: How do I force mr to checkout a lazy repo? 'mr checkout' seems to ignore any arguments, so 'mr checkout repo' skips repos that have 'skip = lazy'. If I manually create the repo directory, mr thinks the repo is already checked out. Yes, this is a use case for mr checkout somehow taking a parameter, or for the stuff that Adam has been talking about to let mr be told which repos to act on. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: mr: Lazy processing of repositories
Joey Hess wrote: Svend Sorensen wrote: How do I force mr to checkout a lazy repo? 'mr checkout' seems to ignore any arguments, so 'mr checkout repo' skips repos that have 'skip = lazy'. If I manually create the repo directory, mr thinks the repo is already checked out. Yes, this is a use case for mr checkout somehow taking a parameter, or for the stuff that Adam has been talking about to let mr be told which repos to act on. Actually, this already works: mr -d foo checkout -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: mr: chaining to absolute paths
Adam Spiers wrote:
I notice that chaining to absolute paths does not work, e.g.:
[$HOME/foo/bar]
checkout = ...
chain = true
This is due to the way the chaining code checks for an .mrconfig in
the chained repository:
if ($parameter eq 'chain'
length $dir $section ne DEFAULT
-e $dir.$section./.mrconfig) {
Is this a feature or a bug? I would have thought it would be useful
to chain to absolute paths.
Probably because nobody noticed since when you're in ~/foo/bar,
~/foo/bar/.mrconfig will be read anyway without chaining. And there's
rarely a reason to use an absolute path.
Fixed in git.
--
see shy jo
signature.asc
Description: Digital signature
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home
Re: various suggestions for mr
Adam Spiers wrote:
I already did this; in fact I *had* to, in order to support GNU stow,
which requires the stow package namespace to be the list of
directories under a single stow directory. If you look for
$STOW_PKG_PATH in the code I originally posted, you'll see:
STOW_DIR=$HOME/.cfg
...
MR_NAME=`basename $MR_REPO`
...
STOW_PKG_PATH=$STOW_DIR/$MR_NAME
and then post_{checkout,update} call the install() function which
does:
ensure_symlink_exists $STOW_PKG_PATH ${MR_REPO%/}
However, the basename operation does not preserve the uniqueness
property which $MR_REPO had, and that's why I say that we need an
additional namespace.
So pick an operation that does? tr / _ would do, for example.
--
see shy jo
signature.asc
Description: Digital signature
___
vcs-home mailing list
vcs-home@lists.madduck.net
http://lists.madduck.net/listinfo/vcs-home
Re: Corrupt git-annex repository
Richard Hartmann wrote: Is there any technical reason that would make git annex init test --uuid=foo impossible? That way, I could re-use the UUID when I _know_ it's OK to reuse them. There is no technical reason that could not be done, but copying the .git/config has the same effect today. As an aside you are using v1 UUIDs, I hugely prefer v4. Is there any way to change which are being generated? It's up to the uuid program in PATH. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: Corrupt git-annex repository
Richard Hartmann wrote: On Sat, Oct 29, 2011 at 18:58, Joey Hess j...@kitenet.net wrote: There is no technical reason that could not be done, but copying the .git/config has the same effect today. OK, so git annex init, edit the UUID manually and then start to add data? That would still leave me with Yet One More repo in the repo list, defeating the initial purpose of re-using UUIDs. copy .git/config before running git annex init. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: Corrupt git-annex repository
Richard Hartmann wrote: fatal: ambiguous argument 'git-annex..refs/remotes/origin/git-annex': unknown revision or path not in the working tree. It seems your repository has lost the git-annex branch. You might try running git fsck to get a better view of the damage, but it's unlikely to fix anything. Your best bet is to re-clone the repository from origin, and preserve the git-annex file contents by moving .git/annex/objects from the broken repository to the new clone. You can also move .git/config over, to preserve the annex UUID setting (and any other configs of course). Then run git annex fsck to make sure its location tracking for that repository is accurate. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: how to refactor path duplication in .mrconfig section headers?
Adam Spiers wrote: If I have multiple repository paths all similar but spread across different .mrconfig files, e.g. in ~/.mrconfig [.config/mr] checkout = ... in ~/.config/mr/config.d/CLI: [$HOME/.git-repos/zsh] ... [$HOME/.git-repos/mutt] ... and in ~/.config/mr/config.d/GUI: [$HOME/.git-repos/urxvt] ... [$HOME/.git-repos/fonts] ... mr can look much nicer if you take advantage of locality and chaining. By locality, I mean putting a mrconfig close to the directories it checks out, rather than in some standards-body controlled directory like ~/.config. For example, you could have: ~/.mrconfig: [.git-repos/CLI] chain = true [.git-repos/GUI] skip = test_non_gui_machine chain = true ~/.git-repos/CLI/.mrconfig: [zsh] ... [mutt] ... ~/.git-repos/GUI/.mrconfig: [urxvt] ... [fonts] ... -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: various suggestions for mr
Adam Spiers wrote: So far mr is clearly winning :-) However, cfgctl does have one or two tricks up its sleeve: - Config modules / packages / repositories / whatever you want to call them are indexed by name within a unique namespace, rather than by directory path, and packages are grouped together into sections. This allows you to easily run any of the actions on: - all the packages (just like running mr from $HOME) - a single package, just by specifying its name without needing to know where it lives, e.g. cfgctl --update zsh would update just the zsh repository - a section (i.e. group of packages) just by specifying its name (e.g. CLI or mail or Xorg) without needing to know where anything lives, e.g. cfgctl --pull Xorg would update all repos containing config relating to my Xorg (previously X11) environment - any packages matching a regular expression e.g. cfgctl --update /emacs/ Having two namespaces for the same thing does not strike me as necessarily a good idea. But if you wanted to do that with mr, you could maybe take advantage of a little-known thing it does with determining the absolute path: joey@gnu:~mkdir namespace joey@gnu:~cd namespace joey@gnu:~/namespaceln -s ~/lib/sound joey@gnu:~/namespaceln -s ~/src/git-annex joey@gnu:~/namespacecd git-annex joey@gnu:~/namespace/git-annexmr update mr update: /home/joey/src/git-annex The only problem with this approach is that it only work when inside the symlinked directory, so mr update in ~/namespace won't update the directories symlinked to there. All in all, I feel that mr has a better design than cfgctl, and has greater longevity. So last night I spent an hour or two doing a quick proof of concept, to see whether I could extend mr to implement the functionality I require, in particular the integration with GNU stow. I'm pleased to say that so far it's looking very promising :-) This is pretty much all that's needed: This seems close to something I could put in mr as an includable library. Could use some documentation though. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: Corrupt git-annex repository
Richard Hartmann wrote: No, the branch was still there. If you want the contents, I can send them off-list. Hmm, either the main git-annex branch or origin/git-annex seems to be missing based on the error message, and I don't think it's the latter. Sounds like a good idea. One question about the UUID, though. I created a repo as replacement for a borked one and after running `git annex init foo`, it set a UUID for that repo. I copied over the old UUID, hoping to reduce clutter from unused repos, but git annex status still told me the local repo had the initial UUID. The above seems to imply that this should work. The only place the uuid for a repository is stored is .git/config. Finally, I think there's still no way to declare a repo permanently gone, i.e. so that it will never show up in any status reports or similar ever again. Or did I just not find it? There is not. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: Corrupt git-annex repository
Richard Hartmann wrote: On Fri, Oct 28, 2011 at 21:23, Joey Hess j...@kitenet.net wrote: Hmm, either the main git-annex branch or origin/git-annex seems to be missing based on the error message, and I don't think it's the latter. Both are there. git log git-annex..origin/git-annex fails, so both are *not* there. (git branch may show them, which means nothing when your repository is corrupted) The only place the uuid for a repository is stored is .git/config. ef90acc8-019e-11e1-9354-9f042d197907 -- test What I meant to say is that the only place that git-annex looks to determine the UUID of the current repository is .git/config. Of course it *stores* information about a UUID in various places -- in your example you told it to store a description test for that UUID, so status displays that even if it doesn't know of any repository that has that UUID. Notice it did not say test (here) There is not. Why? If a UUID is marked as 'gone forever', the other repos could pick that information up over time and purge it from their own stores. Because I don't know how to design a system that has auto-merging and supports outright deletion of UUIDs. Consider this location log file: 1319837256.637657s 1 82e27eac-00d2-11e1-98af-a7c8649fdab4 1319835945.041525s 0 d819e6c8-01a7-11e1-af2b-9f1a8049ae44 If d819e6c8-01a7-11e1-af2b-9f1a8049ae44 is gone forever and is removed, the log file will then contain: 1319837256.637657s 1 82e27eac-00d2-11e1-98af-a7c8649fdab4 But other repositories where that operation has not been performed will still contain the first file. When one of them is union merged back into it, the removed line will necessarily come back. The best that could be done is to add a fourth trust state that is like untrusted UUIDs but causes them to be hidden from view. But this would mean additional work, to handle this edge case -- and simply changing the description of a dead repository seems to work nearly as well. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: git-annex setup with bare remote and optionally bup?
Richard Hartmann wrote: * git annex status does not know about the global annex keys size I think this could be fixed fairly easily using the existing code to list the keys in a non-checked out git branch. One thing I have been pondering is to create a local clone of the bare repo and soft-link its .git/annex/objects to the bare's one. Is this sane or totally crazy? It will defeat git-annex's location tracking so could lead to data loss. Another thing is that I would like this central bare repo to be a common backup point. I.e. it should incorporate bup seamlessly. From how I understand the docs, this is impossible with a bare git-annex repo; hopefully I am wrong. I don't know why bup couldn't be used with a bare repository, but I am unsure if trying to use bup in the same git repository as git-annex is worth the potential complication. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: git-annex setup with bare remote and optionally bup?
Richard Hartmann wrote: On Thu, Oct 27, 2011 at 18:31, Joey Hess j...@kitenet.net wrote: I think this could be fixed fairly easily using the existing code to list the keys in a non-checked out git branch. Sounds good. Would that cover the other noted limitations, as well? Unsure what you mean. It will defeat git-annex's location tracking so could lead to data loss. Obviously, the non-bare repo would need to be untrusted. Assuming it's untrusted, is this save? Unless I can be _sure_ nothing will break, I am not sure if I want to try this just to see that I lost data. It's still not safe. Consider A and B are symlinked and B is untrusted. Now you run git annex drop in B. It checks that A has a copy of a file.. good, it does. So it deletes a file... from both. It would avoid having to have the data twice. Oh, I thought you meant only storing the bup data in a separate branch in the same repository. You can have a bup special remote and store your data there but it then has to be accessed as a special remote, not as a regular bare remote. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: git-annex setup with bare remote and optionally bup?
Richard Hartmann wrote: The problem is that, afaik, I can't have it as a bare special remote. It would be very weird to have a bup repository that is *not* bare. The use case is that I built hosted a server for backups and backups only. As origin, it's used to sync git state between all other repos and always keeps a copy of all files, forever. Obviously, I'd like to keep the bup repo in there, as well. As I said, it's probably possible to use a branch of the same repository for bup as for git-annex, but I'm not sure why it would be worth the setup bother, compared with having a separate repository for bup. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: git-annex setup with bare remote and optionally bup?
Richard Hartmann wrote: On Thu, Oct 27, 2011 at 21:21, Joey Hess j...@kitenet.net wrote: It would be very weird to have a bup repository that is *not* bare. True; what I meant was the merged bup annex, indeed. As I said, it's probably possible to use a branch of the same repository for bup as for git-annex, but I'm not sure why it would be worth the setup bother, compared with having a separate repository for bup. It doubles the amount of disk space used. No, there's no reason to have git-annex send any files to the origin repository if you're storing them on the bup special remote. I could have a remote bare repo as origin, but never copy any files to it. Another special remote for bup to store data in. But how to fsck this beast? A third, host-local, non-bare annex repo to run fsck from(we are talking hundreds of GB)? Or would an annex fsck from a different host run fsck on the bup host? Or can't I fsck bup remotes at all? git-annex does not support fscking special remotes at all, content has to be copied from them before it can be checked. However, bup repositories should be able to be checked with regular git fsck. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: git_status/clustergit ... mr
Michael Nagel wrote: I heard this is kind of a mailing list for mr. I hope this is correct. Anyways, for a long time I have been looking for tools to operate on a collection of (git) repositories and ideally return aggregate results. Until now I used search engines to find such tools, and only found googles repo and Mike Pearce's show_status. I have dubbed the latter clustergit and have been using it ever since. Today a friend told me about Joey Hess' mr, that seems to be able to do a lot of the things I need, but IMHO is comparatively difficult to set up and is not covered by many online tutorials -- which might in consequence lead to the low discoverability using search engines. The name probably doesn't help. I don't see how mr is particularly hard to set up; all it comes down to to add a repository is: git clone git://foo/bar cd bar mr register What do you think of adding an --all switch to mr so I can invoke it like this: mr --all status and it would operate on all directories in the current or specified directory? For simple setups (like mine) this is all that's ever needed and I can always switch to a .mrconfig later. It would make the learning curve less steep and you could create some nice examples in a tutorial to demonstrate (some of the) capabilities of mr. I think it would be more in tune with mr to have a way to go find checked out repositories and register them all. That way you would avoid the overhead of needing to search through possibly many subdirectories each time, and you would have the start of a mrconfig file, which is where most of the power of mr lies. (As you can use it to make mr check out the same things on another computer, or configure special commands to run in some repositories for things like mr update, etc.) -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: git annex fsck in bare repository
Richard Hartmann wrote: Hi all (i.e. Joey), git annex fsck is a no-op in a bare repository. While I can understand that there is no (easy) way to verify the symlinks, the annex objects are there regardless. Wouldn't it make sense to allow me to check repo integrity in bare repos, as well? Yes, fsck could check the size and checksum (if available). It could not check the location log correctness or number of available copies. As an aside, should those smaller issues go into the wiki or onto this list? Both is fine by me. Personally, I would lean towards Keep small stuff on the list, save in wiki if there is a need. I prefer to track such stuff on the wiki. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: managing remotes collaboratively with mr
micah anderson wrote: [DEFAULT] post_checkout = mr addremotes pre_update = mr addremotes Of course these run on all repos, like any mr command. So, what's really needed is: [DEFAULT] post_checkout = mr -d $MR_REPO addremotes pre_update = mr -d $MR_REPO addremotes -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: [mr] Setting command line parameters in a config file?
Richard Hartmann wrote: If the attached qualifies, I can add support for verbose, quiet, stats, and interactive and update the docs. Lot of code bloat. This would probably be a good oportunity to move the global config variables into a hash, which could then be queried and also filled in by loadconfig. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: Running mr commands on one type of repo, only?
Richard Hartmann wrote: I would like to use mr to run `git gc` on all my git repos. While I could add a gc, or repack, option to my configs, I am wondering if there is a way for mr to only act on a certain type of repo. The manpage seems to imply there no way to do this, but I figured asking wouldn't hurt :) [DEFAULT] git_gc = git gc $@ Now when you run mr gc in a svn repo, it will complain, but skip it. joey@gnu:~/src/nslu2-utilsmr gc mr gc: no defined action for svn repository /home/joey/src/nslu2-utils, skipping -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: [mr] Setting command line parameters in a config file?
Richard Hartmann wrote: is there any way to access command line parameters via the config? Setting quiet = 1 jobs = 5 in my main .mrconfig does not yield the desired effect. This is not currently supported, but I'd probably accept a reasonable clean patch. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: managing remotes collaboratively with mr
micah anderson wrote: But what if someone adds a new remote? Because I put things in the .mrconfig as a 'post_checkout' the new remote will not be added to the git repository. I could add the remotes twice, in a post_checkout (for the new person who wants to get them all) and then also as a pre_update, but that seems a bit ugly. [DEFAULT] post_checkout = mr addremotes pre_update = mr addremotes [apache] checkout = git clone gito...@labs.riseup.net:shared-apache apache addremotes = git remote add foo git://foo.ch/apache.git git remote add bar git://bar.net/modules/apache git remote add baz git://github.com/baz/apache.git -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: [announce] Sharebox, a FUSE filesystem relying on git-annex
Dieter Plaetinck wrote: I think having support for this in git-annex would be very useful, even if it's not that efficient: if this can be dealt with in git-annex, individual higherlevel projects like sharebox and dvcs-autosync have less headaches. Not to mention sharebox/dvcs-autosync would need to do really inefficient things to deal with it anyway. (because they can't involve themselves into the actual git/dvcs tricks, they work on a higher level of abstraction), and it might also benefit some users who work with git-annex manually. How do you see this? How hard/cumbersome is it to implement this in git-annex? Why is it inefficient? It's not really clear to me after reading the smudge information on http://www.kernel.org/pub/software/scm/git/docs/gitattributes.html http://git-annex.branchable.com/todo/smudge/ if toobig then git_annex_add file else git_add file git_commit file unfortunately I don't think so: - with dvcs-autosync we often commit early, as in, the file could still be in the process of being written to, or it could be modified again after we added it. From what I understand, we would need to forbid our users from changing the file after it is added to git-annex, and worse: if git-annex does its move file, replace file with symlink trick, while the user is writing to it, this might break things. You're right. However, you would also not want to commit many partial versions of a large file as it was being written. - when a remote A pulls in the changes from remote B, for dropbox-like behavior it should also automatically: * run `git annex get` * git commit .git-annex/*/*.log Does this seem about right? Yes. - deletes will also need to propagate automatically (see next paragraph), still need to figure out how to do that best. Note that dropbox-like behavior is different from the behavior you usually expect from git-annex users. * usual git-annex behavior: every remote stands on it's own, there is no forced being in sync, so that deletes must happen as initiated by the user, and this way you can prevent them from removing files if you expect it could be the last instance of the file. * dropbox-like : remote A remove a file - *all other remotes* should remove the file, so that their working copy looks the same. BUT the file should still be available *somewhere* so that a restore can be initiated (preferably from any of these nodes) I see two solutions here: - centralized: have 1 (or more) remotes that always keep a copy of the files which are being removed on all other remotes, these would be backup-nodes, they don't follow the strict always in sync rule that applies to the regular nodes. (they follow the original git-annex idea more strictly) - decentralized: allow users to remove files by removing the symlink, but still keep the blob in .git-annex on at least one of the nodes, so that it can be restored from that. Yes, that's the default behavior if the symlink is removed. There is then a git annex unused pass that can be used to find and remove unused content when space is needed. Given the size of modern drives, that could be run nightly or something. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: [announce] Sharebox, a FUSE filesystem relying on git-annex
Richard Hartmann wrote: I know Joey pondered this as well, you will find some references on git-annex' ikiwiki. This is needed for S3 in the medium term, anyway. Basically, the plan is to encrypt the files with a symmetric key and then allow access to that key via other keys. That way, you can share some files between machines/people and still make sure no one gets at stuff they shouldn't. The way to encrypt object files' names is still somewhat open to discussion, afaik. Classical dilemma: Where should this be discussed? On this list or within the ikiwiki? Maybe everyone interested should read through the ikiwiki and after some discussion here, we can dump use cases, design decisions etc back into ikiwiki as a TODO once Joey is happy with it? I've put together my current thoughts at http://git-annex.branchable.com/design/encryption/ Comments appreciated in any medium (except watercolors). -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: mr feature request: support variables in .mrconfig
Richard Hartmann wrote: I would love to have variables supported in addition to absolute and relative paths in .mrconfig. At the least, ~ should work and $XDG_CONFIG_HOME (or other generic ones) would be even better. It's not documented, so perhaps it's a bug, but you can already do this. Ie: [$HOME/foo] checkout = git clone ... [$XDG_CONFIG_HOME/bar] checkout = git clone ... -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: [announce] Sharebox, a FUSE filesystem relying on git-annex
Dieter Plaetinck wrote: @Joey: you mentioned you think inotify might be a better backend/paradigm for this than fuse, so do you think implementing git-annex in something like dvcs-autosync is feasible? and/or preferable? Feasable? Certianly. Preferable? I'm in the let a thousand flowers bloom phase. It's spring. :) As Christophe-Marie has pointed out, git-annex makes annexed files semi-immutable, and FUSE can hide that quirk, while inotify watching cannot. That could be confusing for certian users or use cases, if they are not aware of what is going on. Or it could be something quickly learned about how these special replicated directories work, that files have to be copied to be changed. This is also an area I hope to improve in git-annex, by using git smudge filters. So it might get a mode where files can be modified and git commit just annexes the new content. Last time I looked at this, git was not *quite* there to let it be done efficiently. I quite like dvcs-autosync (partially because inotify is more simple than fuse, partially because it currently works already quite well) and I'm interested in making it support space efficient storage of big files; from what I've read it should be possible to do this with git-annex (which should not even change how we currently deal with small files, they would still be in git) but I'm still doing my first baby steps with git-annex so I wouldn't know. Advice very welcome.. All it probably needs at is simplest is something like this (excuse the haskell): toobig - checkFileSize file if toobig then git_annex_add file else git_add file git_commit file Another note : files being tracked with git-annex through sharebox or dvcs-autosync or whatever should always have at least 1 backup copy, so that if the file gets deleted everywhere, it still can be retrieved from somewhere (which raises the interesting question: where will you store this backup copy? introducing a node/repository which will hold backup copies can be considered going to a centralized model; which is something you (Christophe-Marie) try to explicitly avoid, but I think this is not necessarily a problem) This is something git annex goes to large lengths to deal with. It will enforce N backup copies; it tracks which other repositories have which files; it can transfer wanted file contents from other repositories in either a decentralized or a centralized manner; the other repositories can be on other drives of the same computer, or accessible by ssh, or even, now, Amazon S3. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: A git story
This is a problem I'm increasingly stuggling with as I have more and more git repositories, that were converted from old subversion repositories etc. How do I make sure I don't accidentially delete a repository that has historical data I will want later? How do I know which historical repository to look in to find an old dead project? Checking them all into git is appealing, but seems to have a recursion problem. ;) -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
Re: Keeping your dotfiles in git
[EMAIL PROTECTED] wrote: Second, to create all the symlinks you only need a simple command not a script: `ln -s ~/dotfiles/* ~/`. That won't deal with dotfiles that are renamed or deleted. I'm not entirely clear on why, in the examples I've seen, the -s option is used to create symbolic links instead of just using hard links. I've had better luck with sylinks (various things can accidentially break a hard link, in a non-obvious way), but do have to use hardlinks for a few things. IIRC fetchmail refuses to use a symlink for example. Also, I fail to see the point in having a dotfiles directory. It might make it easy to break things up into generic dotfiles that apply to all machines that you use and go in the dotfiles directory, and dotfiles that are only used on particular machines and go in various dotfiles-machinename directories. But aside from that, why not just create the git repository directly in your homedir? Give it a .gitignore file that ignores everything (contains one line with just a *), then add to the repo only the files you want to version with commands like `git add -f ..muttrc`. I have several dotfiles that I don't want to share to every machine with a checkout of my home directory, so need multiple repositories. -- see shy jo signature.asc Description: Digital signature ___ vcs-home mailing list vcs-home@lists.madduck.net http://lists.madduck.net/listinfo/vcs-home
