[wishlist] submodule.update config

[Yaroslav Halchenko]

Relates (but orthogonal) to my other thread

  [wishlist] git submodule update --reset-hard

ATM, it possible to specify per submodule update strategy via
configuration variable submodule.SUBMODULE.update where SUBMODULE is the name
of the corresponding submodule.  But I see no way to specify default update
strategy for all submodules.

>From our conversation in that other thread  I have discovered to myself about
existence of  submodule.recurse  configuration, and there seems to be a few
more (.fetchJobs, .active) where e.g. .active seems to complement per-submodule
submodule.*.active:

yoh@debian:~/proj/misc/git$ git grep '[^.]submodule\.[a-z]' -- 
Documentation/
Documentation/RelNotes/2.14.0.txt: * Many commands learned to pay 
attention to submodule.recurse
Documentation/RelNotes/2.15.0.txt: * "git -c submodule.recurse=yes 
pull" did not work as if the
Documentation/config.txt:include::config/submodule.txt[]
Documentation/config/submodule.txt: update'. If neither 
submodule..active or submodule.active are
Documentation/config/submodule.txt: interact with submodules; 
settings like `submodule.active`
Documentation/config/submodule.txt: submodule.active config option. 
See linkgit:gitsubmodules[7] for
Documentation/config/submodule.txt: as computed via 
`submodule.alternateLocation`. Possible values are
Documentation/git-clone.txt:of multiple entries.  The resulting 
clone has `submodule.active` set to
Documentation/git-clone.txt:Defaults to the `submodule.fetchJobs` 
option.
Documentation/git-submodule.txt:If no path is specified and 
submodule.active has been configured, submodules
Documentation/git-submodule.txt:Defaults to the 
`submodule.fetchJobs` option.
Documentation/gitsubmodules.txt:`submodule.foo.path = path/to/bar`.
Documentation/gitsubmodules.txt:The section `submodule.foo.*` in the 
`.gitmodules` file gives additional
Documentation/gitsubmodules.txt:hints to Git's porcelain layer. For 
example, the `submodule.foo.url`
Documentation/gitsubmodules.txt:  b. if the submodule's path matches 
the pathspec in `submodule.active`
Documentation/gitsubmodules.txt:submodule's path is excluded in the 
pathspec in `submodule.active`, the
Documentation/gitsubmodules.txt:  git config --global submodule.recurse 
true
Documentation/gitsubmodules.txt:your working tree. Alternatively you 
can set 'submodule.recurse' to have
Documentation/technical/api-config.txt:if 
(!git_configset_get_bool(gm_config, "submodule.frotz.ignore", )) {
Documentation/technical/http-protocol.txt:  $GIT_URL: 
http://example.com/git/repo.git/path/submodule.git
Documentation/technical/http-protocol.txt:  URL request:  
http://example.com/git/repo.git/path/submodule.git/info/refs

I wondered, if you think it would be sensible to also add of
submodule.update which would be considered before submodule.SUBMODULE.update
variable possibly defined per submodule.  That would be more inline with desire
to use any of the --merge, --rebase (and hopefully soon --reset-hard)
strategies specified as an option for submodule update, where no per-submodule
handling  is happening.

Thanks in advance for the consideration!
-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

Re: [wishlist] git submodule update --reset-hard

[Yaroslav Halchenko]

On Fri, 07 Dec 2018, Yaroslav Halchenko wrote:


> On Fri, 07 Dec 2018, Stefan Beller wrote:
> > > the initial "git submodule update --reset-hard" is pretty much a
> > > crude workaround for some of those cases, so I would just go earlier in
> > > the history, and redo some things, whenever I could just drop or revert
> > > some selected set of commits.

> > That makes sense.
> > Do you want to give the implementation a try for the --reset-hard switch?

> ok, will do, thanks for the blessing ;-)

The patch is attached (please advise if should be done
differently) and also submitted as PR
https://github.com/git/git/pull/563

I guess it would need more tests.  Took me some time to figure out
why I was getting

fatal: bad value for update parameter

after all my changes to the git-submodule.sh script after looking at an
example commit 42b491786260eb17d97ea9fb1c4b70075bca9523 which introduced
--merge to the update ;-)

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik
From 170296dc661b4bc3d942917ce27288df52ff650d Mon Sep 17 00:00:00 2001
From: Yaroslav Halchenko 
Date: Fri, 7 Dec 2018 21:28:29 -0500
Subject: [PATCH] submodule: Add --reset-hard option for git submodule update

This patch adds a --reset-hard option for the update command to hard
reset submodule(s) to the gitlink for the corresponding submodule in
the superproject.  This feature is desired e.g. to be able to discard
recent changes in the entire hierarchy of the submodules after running

   git reset --hard PREVIOUS_STATE

in the superproject which leaves submodules in their original state,
and

   git reset --hard --recurse-submodules PREVIOUS_STATE

would result in the submodules being checked into detached HEADs.

As in the original  git reset --hard  no checks or any kind of
safe-guards against jumping into some state which was never on the
current branch is done.

must_die_on_failure is not set to  yes to mimic behavior of a update
--checkout strategy, which would leave user with a non-clean state
immediately apparent via  git status  so an informed decision/actions
could be done manually.

Signed-off-by: Yaroslav Halchenko 
---
 Documentation/git-submodule.txt | 12 +++-
 Documentation/gitmodules.txt|  4 ++--
 builtin/submodule--helper.c |  3 ++-
 git-submodule.sh| 10 +-
 submodule.c |  4 
 submodule.h |  1 +
 t/t7406-submodule-update.sh | 17 -
 7 files changed, 45 insertions(+), 6 deletions(-)

diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt
index ba3c4df550..f90a42d265 100644
--- a/Documentation/git-submodule.txt
+++ b/Documentation/git-submodule.txt
@@ -124,7 +124,7 @@ If you really want to remove a submodule from the repository and commit
 that use linkgit:git-rm[1] instead. See linkgit:gitsubmodules[7] for removal
 options.
 
-update [--init] [--remote] [-N|--no-fetch] [--[no-]recommend-shallow] [-f|--force] [--checkout|--rebase|--merge] [--reference ] [--depth ] [--recursive] [--jobs ] [--] [...]::
+update [--init] [--remote] [-N|--no-fetch] [--[no-]recommend-shallow] [-f|--force] [--checkout|--rebase|--merge|--reset-hard] [--reference ] [--depth ] [--recursive] [--jobs ] [--] [...]::
 +
 --
 Update the registered submodules to match what the superproject
@@ -358,6 +358,16 @@ the submodule itself.
 	If the key `submodule.$name.update` is set to `rebase`, this option is
 	implicit.
 
+--reset-hard::
+	This option is only valid for the update command.
+	Hard reset current state to the commit recorded in the	superproject.
+If this option is given, the submodule's HEAD will not get detached
+if it was not detached before. Note that, like with a regular
+git reset --hard  no safe-guards are in place to prevent jumping
+to a commit which was never part of the current branch.
+	If the key `submodule.$name.update` is set to `reset-hard`, this
+	option is implicit.
+
 --init::
 	This option is only valid for the update command.
 	Initialize all submodules for which "git submodule init" has not been
diff --git a/Documentation/gitmodules.txt b/Documentation/gitmodules.txt
index 312b6f9259..e085dbc01f 100644
--- a/Documentation/gitmodules.txt
+++ b/Documentation/gitmodules.txt
@@ -43,8 +43,8 @@ submodule..update::
 	command in the superproject. This is only used by `git
 	submodule init` to initialize the configuration variable of
 	the same name. Allowed values here are 'checkout', 'rebase',
-	'merge' or 'none'. See description of 'update' command in
-	linkgit:git-submodule[1] for their meaning. Note that the
+	'merge', 'reset-hard' or 'none'. See description of 'update' command
+	in l

Re: [wishlist] git submodule update --reset-hard

[Yaroslav Halchenko]

On Fri, 07 Dec 2018, Stefan Beller wrote:
> > the initial "git submodule update --reset-hard" is pretty much a
> > crude workaround for some of those cases, so I would just go earlier in
> > the history, and redo some things, whenever I could just drop or revert
> > some selected set of commits.

> That makes sense.
> Do you want to give the implementation a try for the --reset-hard switch?

ok, will do, thanks for the blessing ;-)

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

Re: [wishlist] git submodule update --reset-hard

[Yaroslav Halchenko]

Hi Stefan,

Thanks for the dialogue!  Replies are embedded below.

On Thu, 06 Dec 2018, Stefan Beller wrote:
> ...
> > > What if the branch differs from the sha1 recorded in the superproject?

> > git reset --hard  itself is an operation which should be done with some
> > level of competence in doing "the right thing" by calling it.  You
> > can hop branches even in current (without any submodules in question)
> > repository with it and cause as much chaos as you desire.

> Right.

> git reset --hard would the branch (as well as the working tree) to the
> given sha1, which is confusing as submodules get involved.

> The Right Thing as of now is the sha1 as found in the
> superprojects gitlink. But as that can be different from any branch
> in the submodule, we'd rather detach the HEAD to make it
> deterministic.

yeap, makes total sense to be the thing do that by default ;-)

> There was a proposal to "re-attach HEAD" in the submodule, i.e.
> if the branch branch points at the same commit, we don't need
> a detached HEAD, but could go with the branch instead.

if I got the idea right, if we are talking about any branch, it
would also non-deterministic since who knows what left over branch(es)
point to that commit.  Not sure if I would have used that ;)

> > If desired though, a number of prevention mechanisms could be in place (but
> > would require option(s) to overcome) to allow submodule to be reset 
> > --hard'ed
> > only when some conditions met (e.g. only to the commit which is among parent
> > commits path of the current branch).  This way wild hops would be prevented,
> > although you might still end up in some feature branch.  But since "reset
> > --hard" itself doesn't have any safe-guards, I do not really think they 
> > should
> > be implemented here either.

> So are you looking for
> a) "stay on submodule branch (i.e. HEAD still points at $branch), and
> reset --hard" such that the submodule has a clean index and at that $branch 
> or
> b) "stay on submodule branch (i.e. HEAD still points at $branch), but $branch 
> is
>set to the gitlink from the superproject, and then a reset --hard
>will have the worktree set to it as well.

yes!

NB "gitlink" -- just now discovered the thing for me.  Thought it would be
called a  subproject  echoing what git diff/log -p shows for submodule commits.

> (a) is what the referenced submodule.repoLike option implements.

sounds like it indeed, thanks for spelling out

> I'd understand the desire for (b) as well, as it is a "real" hard reset on
> the superproject level, without detaching branches.

yeap

> > >   git reset --hard --recurse-submodules HEAD

> > it does indeed some trick(s) but not all seems to be the ones I desire:

> > 1. Seems to migrate submodule's .git directories into the top level
> > .git/modules

> Ah yes, that happens too. This will help once you want to git-rm
> a submodule and checkout states before and after.

yeap ;-) 

> > > undesirable in the sense of still having local changes (that is what
> > > the above reset with `--recurse` would fix) or changed the branch
> > > state? (i.e. is detached but was on a branch before?)

> > right -- I meant the local changes and indeed reset --recurse-submodules
> > indeed seems to recurse nicely.  Then the undesired effect remaining only
> > the detached HEAD

> For that we may want to revive discussions in
> https://public-inbox.org/git/20170501180058.8063-5-sbel...@google.com/

well, isn't that one requires a branch to be specified in .gitmodules?

> > > >   git submodule update --recursive

> > > > I would end up in the detached HEADs within submodules.

> > > > What I want is to retain current branch they are at (or may be possible
> > > > "were in"? reflog records might have that information)

> > > So something like

> > >   git submodule foreach --recursive git reset --hard

> > > ?

> > not quite  -- this would just kill all local changes within each submodule, 
> > not
> > to reset it to the desired state, which wouldn't be specified in such
> > invocation, and is only known to the repo containing it

> With this answer it sounds like you'd want (b) from above.

yeap

> > > You may be interested in
> > > https://public-inbox.org/git/20180927221603.148025-1-sbel...@google.com/
> > > which introduces a switch `submodule.repoLike [ = true]`, which
> > > when set would not detach HEAD in submodules.

> > Thanks! looks interesting -- was there more discussion/activity beyond 
> > those 5
> > posts in the thread?

> Unfortunately there was not.

pity

> > This feature might indeed come handy but if I got it right, it is somewhat
> > complimentary to just having submodule update --reset-hard .  E.g.  
> > submodules
> > might be in different branches (if I am not tracking based on branch 
> > names), so
> > I would not want a recursive checkout with -b|-B.  But we would indeed 
> > benefit
> > from such functionality, since this difficulty of managing branches of
> > submodules I 

Re: [wishlist] git submodule update --reset-hard

[Yaroslav Halchenko]

On Thu, 06 Dec 2018, Stefan Beller wrote:

> On Thu, Dec 6, 2018 at 10:02 AM Yaroslav Halchenko  
> wrote:

> > Dear Git Gurus,

> > I wondered what would be your take on my wishlist request to add
> > --reset-hard option, which would be very similar to regular "update" which
> > checks out necessary commit, but I want it to remain in the branch.

> What if the branch differs from the sha1 recorded in the superproject?

git reset --hard  itself is an operation which should be done with some
level of competence in doing "the right thing" by calling it.  You
can hop branches even in current (without any submodules in question)
repository with it and cause as much chaos as you desire.

If desired though, a number of prevention mechanisms could be in place (but
would require option(s) to overcome) to allow submodule to be reset --hard'ed
only when some conditions met (e.g. only to the commit which is among parent
commits path of the current branch).  This way wild hops would be prevented,
although you might still end up in some feature branch.  But since "reset
--hard" itself doesn't have any safe-guards, I do not really think they should
be implemented here either.

> > Rationale: In DataLad we heavily rely on submodules, and we have established
> > easy ways to do some manipulations across full hierarchies of them. E.g. a
> > single command could introduce a good number of commits across deep 
> > hierarchy
> > of submodules, e.g. while committing changes within deep submodule, while 
> > also
> > doing all necessary commits in the repositories leading to that submodule so
> > the entire tree of them stays in a "clean" state. The difficulty comes when
> > there is a need to just "forget" some changes.  The obvious way is to e.g.

> >git reset --hard PREVIOUS_STATE

>   git reset --hard --recurse-submodules HEAD

> would do the trick

it does indeed some trick(s) but not all seems to be the ones I desire:

1. Seems to migrate submodule's .git directories into the top level
.git/modules

$>  git reset --hard --recurse-submodules HEAD^^^
Migrating git directory of 'famface' from
'/tmp/gobbini/famface/.git' to
'/tmp/gobbini/.git/modules/famface'
Migrating git directory of 'famface/data' from
'/tmp/gobbini/famface/data/.git' to
'/tmp/gobbini/.git/modules/famface/modules/data'
Migrating git directory of 'famface/data/scripts/mridefacer' from
'/tmp/gobbini/famface/data/scripts/mridefacer/.git' to

'/tmp/gobbini/.git/modules/famface/modules/data/modules/scripts/mridefacer'
HEAD is now at 9b4296d [DATALAD] aggregated meta data

we might eventually adopt this default already for years model (git annex seems
to be ok, in that it then replaces .git symlink file with the actual
symlink .git -> ../../.git/modules/...  So things seems to keep working
for annex)

2. It still does the detached HEAD for me

$> git submodule status --recursive  
 2569ab436501a832d35afbbe9cc20ffeb6077eb1 famface (2569ab4)
 f1e8c9b8b025c311424283b9711efc6bc906ba2b famface/data (BIDS-v1.0.1)
 49b0fe42696724c2a8492f999736056e51b77358 
famface/data/scripts/mridefacer (49b0fe4)


> > in the top level repository.  But that leaves all the submodules now in
> > the undesired state.  If I do

> undesirable in the sense of still having local changes (that is what
> the above reset with `--recurse` would fix) or changed the branch
> state? (i.e. is detached but was on a branch before?)

right -- I meant the local changes and indeed reset --recurse-submodules
indeed seems to recurse nicely.  Then the undesired effect remaining only
the detached HEAD

> >   git submodule update --recursive

> > I would end up in the detached HEADs within submodules.

> > What I want is to retain current branch they are at (or may be possible
> > "were in"? reflog records might have that information)

> So something like

>   git submodule foreach --recursive git reset --hard

> ?

not quite  -- this would just kill all local changes within each submodule, not
to reset it to the desired state, which wouldn't be specified in such
invocation, and is only known to the repo containing it

> You may be interested in
> https://public-inbox.org/git/20180927221603.148025-1-sbel...@google.com/
> which introduces a switch `submodule.repoLike [ = true]`, which
> when set would not detach HEAD in submodules.

Thanks! looks interesting -- was there more discussion/activity beyond those 5
posts in the thread?
https://public-inbox.org/git/87h8i9ift4@evledraar.gmail.com/#r 

This feature might indeed come handy but if I got it right, it is somewhat
complimentary to just having submodule update --reset-hard .  

[wishlist] git submodule update --reset-hard

[Yaroslav Halchenko]

Dear Git Gurus,

I wondered what would be your take on my wishlist request to add
--reset-hard option, which would be very similar to regular "update" which
checks out necessary commit, but I want it to remain in the branch.

Rationale: In DataLad we heavily rely on submodules, and we have established
easy ways to do some manipulations across full hierarchies of them. E.g. a
single command could introduce a good number of commits across deep hierarchy
of submodules, e.g. while committing changes within deep submodule, while also
doing all necessary commits in the repositories leading to that submodule so
the entire tree of them stays in a "clean" state. The difficulty comes when
there is a need to just "forget" some changes.  The obvious way is to e.g. 

   git reset --hard PREVIOUS_STATE

in the top level repository.  But that leaves all the submodules now in
the undesired state.  If I do

  git submodule update --recursive

I would end up in the detached HEADs within submodules.  

What I want is to retain current branch they are at (or may be possible
"were in"? reflog records might have that information)

Example:

# Have to use datalad install  since  git clone --recurse-submodules
# seems to not consider alternative locations for submodules' .git/
# with url being just a relative path, and where submodules aren't 
# all residing up under toplevel URL .git/

$> datalad install -r http://datasets.datalad.org/labs/gobbini/.git
[INFO   ] Cloning http://datasets.datalad.org/labs/gobbini/.git into 
'/tmp/gobbini' 
install(ok): /tmp/gobbini (dataset) 

[INFO   ] Installing  recursively 
[INFO   ] Cloning http://datasets.datalad.org/labs/gobbini/famface/.git 
into '/tmp/gobbini/famface' 
[INFO   ] Cloning 
http://datasets.datalad.org/labs/gobbini/famface/data/.git into 
'/tmp/gobbini/famface/data'   
[INFO   ] access to dataset sibling "datasets.datalad.org" not 
auto-enabled, enable with:   
|   datalad siblings -d "/tmp/gobbini/famface/data" enable 
-s datasets.datalad.org 
[INFO   ] Cloning 
http://datasets.datalad.org/labs/gobbini/famface/data/scripts/mridefacer/.git 
[2 other candidates] into '/tmp/gobbini/famface/data/scripts/mridefacer' 
action summary: 

  install (ok: 4)

so I have a hierarchy in a good state and all checked out in master
branch

$> cd gobbini

$> git submodule status --recursive   
 b9071a6bc9f7665f7c75549c63d29f16d40e8af7 famface (heads/master)
 e59ba76b42f219bdf14b6b547dd6d9cc0ed5227f famface/data 
(BIDS-v1.0.1-3-ge59ba76b)
 5d8036c0aaeebb448a00df6296ddc9f799efdd1f 
famface/data/scripts/mridefacer (heads/master)

$> git submodule foreach --recursive cat .git/HEAD 
Entering 'famface'
ref: refs/heads/master
Entering 'famface/data'
ref: refs/heads/master
Entering 'famface/data/scripts/mridefacer'
ref: refs/heads/master


and if I do roll back

$> git reset --hard HEAD^^^
HEAD is now at 9b4296d [DATALAD] aggregated meta data
changes on filesystem:  

 famface | 2 +-

and default update --recursive

$> git submodule update --recursive
Submodule path 'famface': checked out 
'2569ab436501a832d35afbbe9cc20ffeb6077eb1'
Submodule path 'famface/data': checked out 
'f1e8c9b8b025c311424283b9711efc6bc906ba2b'
Submodule path 'famface/data/scripts/mridefacer': checked out 
'49b0fe42696724c2a8492f999736056e51b77358'

I end up in detached HEADs

$> git submodule status --recursive 
 2569ab436501a832d35afbbe9cc20ffeb6077eb1 famface (2569ab4)
 f1e8c9b8b025c311424283b9711efc6bc906ba2b famface/data (BIDS-v1.0.1)
 49b0fe42696724c2a8492f999736056e51b77358 
famface/data/scripts/mridefacer (49b0fe4)


I do see that there is a "custom command" way to do it via
"submodule..update" config setting, but that is not easy to use for my
case since all the `` would be different to specify !git reset --hard for
all of them via config option and I could not find any way to "glob" config
(like submodule.*.update).  But in effect that is probably what I need:

# restarting from a clean state here
$> git -c submodule.famface.update='!git reset --hard' submodule update 
--recursive
HEAD is now at 2569ab4 [DATALAD] aggregated meta data
Submodule path 'famface': 'git reset --hard 
2569ab436501a832d35afbbe9cc20ffeb6077eb1'
Submodule path 'famface/data': checked out 
'f1e8c9b8b025c311424283b9711efc6bc906ba2b'
Submodule path 'famface/data/scripts/mridefacer': checked out 

Re: wishlist: "--cached|--staged" to "git commit file(s)"

[Yaroslav Halchenko]

On Mon, 13 Aug 2018, Junio C Hamano wrote:
> > command.  ATM there is no non-interactive (via --patch/--interactive I
> > think it is possible) way to commit selected subset of staged files not
> > from the worktree (as it is done with "git commit file(s)") but from the
> > index.

> Hmph, so

>   edit A B C
>   git add A B
>   edit A B C
>   git commit --cached B

> would create a commit that is different from the original HEAD by
> the edit to file B you did in the first step, but not the edit to A
> or C, or edit to B after you did "git add"?

yes

> That sounds like a useful feature.  If you do not need that "exclude
> the later edit since the last 'git add'", you can do "git commit B",
> but if you added crufts to B in the working tree since you added a
> good copy of B to the index, that would not do what you want to do.

exactly

> Sounds like a good starter project for somebody who wants to try to
> get their hands dirty...

asking "around" ;-)


-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

wishlist: "--cached|--staged" to "git commit file(s)"

[Yaroslav Halchenko]

Dear Git developers,

Following up on my quick question/discussion on IRC a few days
back:  Please consider adding --cached or --staged option for 

git commit file(s)

command.  ATM there is no non-interactive (via --patch/--interactive I
think it is possible) way to commit selected subset of staged files not
from the worktree (as it is done with "git commit file(s)") but from the
index.

Original usecase (I would be happy to elaborate):  committing specified
by user files in "direct" mode of git-annex, where it (via git annex
add) first converts files to the symlinks within index (they remain
original files within worktree), so we cannot use "git commit file(s)"
since then they would get committed disregarding their state in index.

But thinking about it I realized that it was quite a common use case to
me, that I typically stage all modified files whenever I reach "it is no
longer broken" stage, and then decide to commit that staged state in a
number of separate commits, and only then commit worktree state.

Current workaround as suggested on IRC is to 

- cp .git/index .git/index2
- GIT_INDEX_FILE=.git/index2 git reset files_not_to_be_committed
- GIT_INDEX_FILE=.git/index2 git commit
- rm .git/index2

or may be there is another (better) way?

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

please resolve a mystery for me: what is j-c-diff exactly? ;)

[Yaroslav Halchenko]

Hi Junio et al,

j-c-diff is "used" in the docs within git, git-annex, and other places
discussing git.  But I failed to find it to seek for an ultimate prototypical
example of the diff command used by git ppl ;)

$> git log -Sj-c-diff -p Documentation/gitattributes.txt 
commit 2cc3167c688d1c91bc4cb9b1caa737b9d4971056
Author: Junio C Hamano 
Date:   Mon Apr 23 00:21:02 2007 -0700

Document "diff=driver" attribute

Signed-off-by: Junio C Hamano 

diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt
index 126871756d..d2edb9b14a 100644
--- a/Documentation/gitattributes.txt
+++ b/Documentation/gitattributes.txt
@@ -151,8 +151,34 @@ Unspecified::
text, it is treated as text.  Otherwise it would
generate `Binary files differ`.
 
-Any other value set to `diff` attribute is ignored and git acts
-as if the attribute is left unspecified.
+String::
+
+ Diff is shown using the specified custom diff driver.
+ The driver program is given its input using the same
+ calling convention as used for GIT_EXTERNAL_DIFF
+ program.
+
+
+Defining a custom diff driver
+^
+
+The definition of a diff driver is done in `gitconfig`, not
+`gitattributes` file, so strictly speaking this manual page is a
+wrong place to talk about it.  However...
+
+To define a custom diff driver `jcdiff`, add a section to your
+`$GIT_DIR/config` file (or `$HOME/.gitconfig` file) like this:
+
+
+[diff "jcdiff"]
+ command = j-c-diff
+
+
+When git needs to show you a diff for the path with `diff`
+attribute set to `jcdiff`, it calls the command you specified
+with the above configuration, i.e. `j-c-diff`, with 7
+parameters, just like `GIT_EXTERNAL_DIFF` program is called.
+See gitlink:git[7] for details.

Cheers!
-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

external diff driver is not used for diff --stat?

[Yaroslav Halchenko]

Dear Git Peoples,

I am torturing git and git-annex here trying to compare some logs from a
run of a software recorded in two different branches.  As many other
tools, software often logs its version, elapsed times etc, so diff becomes not 
of interest to me:

$> PATH=~/proj/misc/git/INSTALL-2.16.1/bin:$PATH git diff test-18.0.09 
test-18.0.05+git24-gb25b21054_dfsg.1-1_nd90+1 -- 
AFNI_data6/FT_analysis/FT.results/out.allcostX.txt-git
diff --git a/AFNI_data6/FT_analysis/FT.results/out.allcostX.txt-git 
b/AFNI_data6/FT_analysis/FT.results/out.allcostX.txt-git
index db85c9be..5f4a704d 100644
--- a/AFNI_data6/FT_analysis/FT.results/out.allcostX.txt-git
+++ b/AFNI_data6/FT_analysis/FT.results/out.allcostX.txt-git
@@ -1,4 +1,4 @@
-++ 3dAllineate: AFNI version=AFNI_18.0.09 (Jan 19 2018) [64-bit]
+++ 3dAllineate: AFNI 
version=Debian-18.0.05+git24-gb25b21054~dfsg.1-1~nd90+1 (Jan 23 2018) [64-bit]
 ++ Authored by: Zhark the Registrator
 ++ Source dataset: ./anat_final.FT+tlrc.HEAD
 ++ Base dataset:   ./final_epi_vr_base_min_outlier+tlrc.HEAD
@@ -28,5 +28,5 @@ volume 0
lpa  = 0.921773
lpc+ = 0.310739
ncd  = 0.967007
-++ 3dAllineate: total CPU time = 0.0 sec  Elapsed = 1.5
+++ 3dAllineate: total CPU time = 0.0 sec  Elapsed = 1.3


so I came up with a simple differ to exclude those:

$> cat ~/bin/git-annex-diff-wrapper
#!/usr/bin/env bash 

LANG=C diff --color=always --ignore-matching-lines="\(AFNI 
version=\|time.*Elapsed\)" -u "$2" "$5"  

which works as it should (sorry for long lines, just wanted to not cut out
anything which might be of relevance)

$> PATH=~/proj/misc/git/INSTALL-2.16.1/bin:$PATH 
GIT_EXTERNAL_DIFF='git-annex diffdriver  -- ~/bin/git-annex-diff-wrapper --' 
git diff --ext-diff test-18.0.09 test-18.0.05+git24-gb25b21054_dfsg.1-1_nd90+1 
-- AFNI_data6/FT_analysis/FT.results/out.allcostX.txt-git 
# no output received

(and even on annexed files -- whoohoo).

The problem comes that --stat seems to be not using the external diff (it is
line the same as above just with --stat):

$> PATH=~/proj/misc/git/INSTALL-2.16.1/bin:$PATH 
GIT_EXTERNAL_DIFF='git-annex diffdriver  -- ~/bin/git-annex-diff-wrapper --' 
git diff --ext-diff test-18.0.09 test-18.0.05+git24-gb25b21054_dfsg.1-1_nd90+1 
--stat -- AFNI_data6/FT_analysis/FT.results/out.allcostX.txt-git
 AFNI_data6/FT_analysis/FT.results/out.allcostX.txt-git | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)


A shortcoming or somehow "by design"? ;)

PS Please CC me in replies

Cheers!

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

Re: [PATCH] merge: teach -Xours/-Xtheirs to symbolic link merge

[Yaroslav Halchenko]

On Thu, 28 Dec 2017, Elijah Newren wrote:
> > Teach a similar trick to the codepath that deals with merging two
> > conflicting changes to symbolic links.

> Saw this change referenced in the "what's cooking" emails and decided
> to review this.  The code changes look obviously correct to me, and
> the testcase looks good too.

> Reviewed-by: Elijah Newren <new...@gmail.com>

> (and perhaps we should also add in "Tested-by: Yaroslav Halchenko
> <y...@onerussian.com>" ?  At least, that was my thought based on
> https://public-inbox.org/git/20170926133703.7gtk5ztkhqvfx...@hopa.kiewit.dartmouth.edu/
> )

I would be honored to wear a badge of the git-tested-by-er!  FWIW
I can reconfirm, that the patch did work out nicely for me back then

Thanks!
-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

Re: Q: rational for $XDG_CONFIG_HOME/git/config to be "non global" or just a bug?

[Yaroslav Halchenko]

On Mon, 18 Dec 2017, Jeff King wrote:

> To complete that abstraction it seems like reading via "--global" should
> read from both (in the same precedence order that normal config lookup
> uses).

FWIW +1 from me on that ;)

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

Re: [PATCH] doc: clarify usage of XDG_CONFIG_HOME config file

[Yaroslav Halchenko]

On Tue, 12 Dec 2017, Jacob Keller wrote:

> > And then "and the other files will not be read" can be dropped from
> > the first sentence of this paragraph?

> > Yaroslav on the original thread mentioned that reading codepath
> > without --file or --global does not limit to one of the three, and
> > this section is about "If not set explicitly with `--file`", so we'd
> > need to make sure if the above is what happens in reality (or update
> > the proposed clarification to match the reality).

> I'm pretty sure it does not read XDG_CONFIG_HOME unless ~/.gitconfig
> is missing. I tried a few things, but it was 2am for me, so I may be
> mis-remembering.

It always read it for non--global

$> ( HOME=/tmp/HOME; rm -rf $HOME; mkdir -p $HOME/.config/git; echo -e 
"[user]\n name=home" > $HOME/.gitconfig; echo -e "[user]\n name=xdg\n 
name2=xdg2" > $HOME/.config/git/config; git config user.name; git config 
user.name2; ) 
home
xdg2

and it doesn't read it for --global

$> ( HOME=/tmp/HOME; rm -rf $HOME; mkdir -p $HOME/.config/git; echo -e 
"[user]\n name=home" > $HOME/.gitconfig; echo -e "[user]\n name=xdg\n 
name2=xdg2" > $HOME/.config/git/config; git config --global user.name; git 
config --global user.name2; )   
 
home

unless ~/.gitconfig is missing

$> ( HOME=/tmp/HOME; rm -rf $HOME; mkdir -p $HOME/.config/git; echo -e 
"[user]\n name=xdg\n name2=xdg2" > $HOME/.config/git/config; git config 
--global user.name; git config --global user.name2; ) 
xdg
xdg2


-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

Re: Q: rational for $XDG_CONFIG_HOME/git/config to be "non global" or just a bug?

[Yaroslav Halchenko]

On Mon, 11 Dec 2017, Junio C Hamano wrote:

> Jonathan Nieder  writes:

> > I think the documentation

> > ~/.gitconfig
> > User-specific configuration file. Also called "global"
> > configuration file.

> > should be clarified --- e.g. it could say

> > $XDG_CONFIG_HOME/git/config
> > ~/.gitconfig
> > User-specific configuration files. Because options in
> > these files are not specific to any repository, thes
> > are sometimes called global configuration files.

> Yeah, I think that makes sense.

> > As for "git config --global", I think the best thing would be to split
> > it into two options: something like "git config --user" and "git
> > config --xdg-user".  That way, it is unambiguous which configuration
> > file the user intends to inspect or modify.  When a user calls "git
> > config --global" and both files exist, it could warn that the command
> > is ambiguous.

> > Thoughts?

> I actually thought that the plan was "you either have this, or the
> other one, never both at the same time" (and I think those who
> pushed the XDG thing in to the system made us favor it over the
> traditional one).  So as long as --global updates the one that
> exists, and updates XDG one when both or neither do, I think we
> should be OK.  And from that viewpoint, we definitely do not want
> two kinds of --global to pretend as if we support use of both at the
> same time.

note that atm $XDG_CONFIG_HOME/git/config is read as --global iff
~/.gitconfig is absent and read always without --global.  So it is
flipping between "global" and "some kind of non-global but user-specific
configuration file" (so sounds like  a global to me ;) )

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

Re: Q: rational for $XDG_CONFIG_HOME/git/config to be "non global" or just a bug?

[Yaroslav Halchenko]

On Mon, 11 Dec 2017, Jonathan Nieder wrote:
> > Example to show that TFM outlines precedence and --global correctly:

> > $> grep xdg .gitconfig .config/git/config
> > .gitconfig:xdg-and-user = user
> > .config/git/config: xdg = xdg
> > .config/git/config: xdg-and-user = xdg
> > $> git config user.xdg ; git config user.xdg-and-user
> > xdg
> > user

> I agree, this is confusing.

> Reverse engineering from source, I find that git reads the following
> files in sequence:

>   system:
>   /etc/gitconfig
>   global:
>   $XDG_CONFIG_HOME/git/config
>   $HOME/.gitconfig
>   repo:
>   $GIT_DIR/config
>   commandline:
>   options passed with -c or GIT_CONFIG_PARAMETERS

> These terms (system, global, repo, etc) are accessible in code as
> current_config_scope().  I don't think there's any user-visible effect
> to $XDG_CONFIG_HOME/git/config and $HOME/.gitconfig both being global
> --- it would probably be a good cleanup to rename the scope for one of
> them.

Well, we have got at least one user/contributor now who uses
$XDG_CONFIG_HOME/git/config in favor of ~/.gitconfig since it makes it
easier for modular user configuration.

> I think the documentation

>   ~/.gitconfig
>   User-specific configuration file. Also called "global"
>   configuration file.

> should be clarified --- e.g. it could say

>   $XDG_CONFIG_HOME/git/config
>   ~/.gitconfig
>   User-specific configuration files. Because options in
>   these files are not specific to any repository, thes
>   are sometimes called global configuration files.

> As for "git config --global", I think the best thing would be to split
> it into two options: something like "git config --user" and "git
> config --xdg-user".  That way, it is unambiguous which configuration
> file the user intends to inspect or modify.  When a user calls "git
> config --global" and both files exist, it could warn that the command
> is ambiguous.

why ambiguous?  as long as both are consistently called global, and the
overloading rules are clear for reading -- nothing ambigous.  The only
ambigous logic would be for writing.

> Thoughts?

Well -- my main functionality concern that ATM
$XDG_CONFIG_HOME/git/config is (as of 2.15.0) only --global for writing
but not for regular reading (as I demonstrated in the original email)

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

Q: rational for $XDG_CONFIG_HOME/git/config to be "non global" or just a bug?

[Yaroslav Halchenko]

Dear Git Gurus,

We [1] have got confused a bit about this recent addition of handling
$XDG_CONFIG_HOME/git/config -- is it --global or not? ;)

According to the man git-config (v 2.15.0 in debian)

   --global
   For writing options: write to global ~/.gitconfig file rather than
   the repository .git/config, write to $XDG_CONFIG_HOME/git/config
   file if this file exists and the ~/.gitconfig file doesn’t.

   For reading options: read only from global ~/.gitconfig and from
   $XDG_CONFIG_HOME/git/config rather than from all available files.

   See also the section called “FILES”.

suggesting that $XDG_CONFIG_HOME/git/config is a part of the "--global" config
space, which it is not, which is also later described in FILES:

   $(prefix)/etc/gitconfig
   System-wide configuration file.

   $XDG_CONFIG_HOME/git/config
   Second user-specific configuration file. If $XDG_CONFIG_HOME is not 
set or empty, $HOME/.config/git/config will be used. Any
   single-valued variable set in this file will be overwritten by 
whatever is in ~/.gitconfig. It is a good idea not to create this file if
   you sometimes use older versions of Git, as support for this file 
was added fairly recently.

   ~/.gitconfig
   User-specific configuration file. Also called "global" configuration 
file.

which

1. says that $XDG_CONFIG_HOME/git/config is the "Second user-specific ..."
   suggesting that it should be the one read AFTER the first user-specific...
   I guess that the first one is the ~/.gitconfig , but then why the first one
   overrides the settings of the second one ? ;)  (as described above in TFM and
   see below for an example)

2. why $XDG_CONFIG_HOME/git/config is not a part of the "global" configuration?

   I always assumed that "global" is ALL settings defined for a user,
   which are not specific to a repository.

   It is double-confusing since, as --global doc describes (and example
   below shows), git config --global --add modifies the
   $XDG_CONFIG_HOME/git/config if there is no ~/.gitconfig

   Actually the doc for --global for "reading" seems to be not correct,
   that the file is not consulted for --global (see below)

Example to show that TFM outlines precedence and --global correctly:

$> grep xdg .gitconfig .config/git/config  
.gitconfig:xdg-and-user = user
.config/git/config: xdg = xdg
.config/git/config: xdg-and-user = xdg
$> git config user.xdg ; git config user.xdg-and-user
xdg  
user
$> git config --global user.xdg# so outputs nothing
$> git config --global user.xdg-and-user
user

$> mv .gitconfig{,.aside}
$> git config --global --add user.new value 
$> cat .config/git/config 
[user]
 xdg = xdg
 xdg-and-user = xdg
 new = value


So, is that simply a bug and $XDG_CONFIG_HOME/git/config should be
consulted for --global reading and doc should be adjusted to
state that it is a part of "global" config in FILES description?
Or it shouldn't be --global (then writing should be fixed, and
documentation adjusted to exclude it from --global)
Or am I just confused? ;)

thanks in advance for the clarification!

[1]  https://github.com/datalad/datalad/pull/2019#issuecomment-350757960
-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

Re: -s theirs use-case(s) Was: BUG: merge -s theirs is not in effect

[Yaroslav Halchenko]

On Wed, 27 Sep 2017, Yaroslav Halchenko wrote:
> > And at that point, use of "-s ours" is no longer a workaround for
> > lack of "-s theirs".  It is a proper part of the desired semantics,
> > i.e. from the point of view of the surviving canonical history line,
> > you want to preserve what it did, nullifying what the other line of
> > history did.

> > So I still do not think the above scenario justifies "-s theirs".

> ok, when you describe it like this (in my case I rarely cared about the
> side of the merge), then indeed I might better do the entire dance with
> git reset --hard theirstate; git merge -s ours HEAD@{1}
> and live happily with the left side being the one always correct and
> hide "my" mistakes ;)  will keep it in mind

ha -- was about to use it, which reminded me about this use case!

NB pardon me for not so wonderful ascii-diagrams as yours but hopefully
still helps

  x-o-o-o-x-o-odebian
 //
 x---x-releases -- should just linearly sweep through 
releases I care about
/   /
   R1  R2  various release branches/tags
   A   B   which have differing commits
  /   /
  ---o---ooomasteg

For packaging some packages for Debian, with my git-rotten-soul, I am trying to
keep my debian packaging  (in a branch "debian") on top of upstream
releases.  But the problem comes whenever upstream releases from "release
branches", which are never merged into master and might have different and
conflicting changes.  

So, it becomes impossible to maintain a linearly progressing "debian"
branch without adding a middle-man between upstream releases (in release
branches), and debian branch (should progress linearly forward) -- "releases"
branch.  See e.g.  http://github.com/neurodebian/pandas and its releases
branch.

so I use -s theirs for "linearizing" the branched up development history

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

Re: -s theirs use-case(s) Was: BUG: merge -s theirs is not in effect

[Yaroslav Halchenko]

On Wed, 27 Sep 2017, Junio C Hamano wrote:

> > and that is where the gotcha comes -- what if "my" changes were already
> > published?  then I would like to avoid the rebase, and would -s theirs
> > to choose "their" solution in favor of mine and be able to push so
> > others could still "fast-forward" to the new state.

> > So -- as to me it remains 'symmetric' ;)

> I do not necessarily agree.  Once you decide that their history is
> the mainline, you'd rather want to treat your line of development as
> a side branch and make a merge in that direction, i.e. the first
> parent of the resulting merge is a commit on their history and the
> second parent is the last bad one of your history.  So you would end
> up using "checkout their-history && merge -s ours your-history" to
> keep the first-parenthood sensible.

> And at that point, use of "-s ours" is no longer a workaround for
> lack of "-s theirs".  It is a proper part of the desired semantics,
> i.e. from the point of view of the surviving canonical history line,
> you want to preserve what it did, nullifying what the other line of
> history did.

> So I still do not think the above scenario justifies "-s theirs".

ok, when you describe it like this (in my case I rarely cared about the
side of the merge), then indeed I might better do the entire dance with
git reset --hard theirstate; git merge -s ours HEAD@{1}
and live happily with the left side being the one always correct and
hide "my" mistakes ;)  will keep it in mind

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

Re: -X theirs does not resolve symlink conflict Was: BUG: merge -s theirs is not in effect

[Yaroslav Halchenko]

On Tue, 26 Sep 2017, Junio C Hamano wrote:
> >> I do not recall people talking about symbolic links but the case of
> >> binary files has been on the wishlist for a long time, and I do not
> >> know of anybody who is working on (or is planning to work on) it.

> > Ah, I misremembered.

> > We've addressed the "binary files" case back in 2012 with a944af1d
> > ("merge: teach -Xours/-Xtheirs to binary ll-merge driver",
> > 2012-09-08).  I do not know offhand if it is just as easy to plumb
> > the MERGE_FAVOR_{OURS,THEIRS} bits thru the symbolic link codepath,
> > like that patch did to the binary file codepath.

> Perhaps the attached (totally untested) patch might be a good
> starting point.  I do not know if you are interested in hacking on
> Git, and I do not feel offended if you are not, but perhaps somebody

I would have felt honored to "hack on Git" but neither my C-foo is up to
par, neither there would be more time I could adequately allocate for
such endeavor.   So meanwhile I am trying to contribute in hopefully
constructive "whining" while exploiting git.

> else might get interested in seeing if this #leftoverbits is a good
> direction to go in, and finishing it with docs and tests if it is
> ;-)


>  merge-recursive.c | 17 +
>  1 file changed, 13 insertions(+), 4 deletions(-)

> >...<

This patch worked beautifully in my usecase!:

$> rm -rf /tmp/repo1; mkdir /tmp/repo1; cd /tmp/repo1; git init .; ln -s sym1 
link; echo 1 > file; git add file link; git commit -m 'common'; git co -b b1 ; 
ln -sf b1link link; echo "b1 file" >| file; git commit -m 'b2 changes' -a; git 
co master; ln -sf masterlink link; echo "master file" >| file; git commit -m 
'also modified in master' -a; git merge -X theirs --no-edit b1; ls -l link; cat 
file   
warning: templates not found /home/yoh/share/git-core/templates 
   
Initialized empty Git repository in /tmp/repo1/.git/
[master (root-commit) d2e9010] common
 2 files changed, 2 insertions(+)
 create mode 100644 file
 create mode 12 link
Switched to a new branch 'b1'
[b1 a2b1321] b2 changes
 2 files changed, 2 insertions(+), 2 deletions(-)
Switched to branch 'master'
[master fbb4ba7] also modified in master
 2 files changed, 2 insertions(+), 2 deletions(-)
Auto-merging link
Auto-merging file
Merge made by the 'recursive' strategy.
 file | 2 +-
 link | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
lrwxrwxrwx 1 yoh yoh 6 Sep 26 09:32 link -> b1link
b1 file

I also tried -s ours and no explicit -s, and they did as prescribed as
well

PS thanks for the CCs
-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

Re: -s theirs use-case(s) Was: BUG: merge -s theirs is not in effect

[Yaroslav Halchenko]

On Tue, 26 Sep 2017, Junio C Hamano wrote:

> Yaroslav Halchenko <y...@onerussian.com> writes:

> > 1. As a workaround for absence of -m theirs I using mtheirs git alias:
> > (I believe provided to me awhile back here on the list):

> > mtheirs = !sh -c 'git merge -s ours --no-commit $1 && git read-tree -m 
> > -u $1' -

> > and it worked fine for my usecases

> > 2. I think that if there is a reason for -s ours to exist, so there for -s 
> > theirs
> > since it is just the directionality of merges which changes between the two

> Just on this point.  They are not exactly symmetric.

> Imagine there are some undesirable changes you want to vanquish from
> the world, but they have already built on useful changes on top of
> the undesirable changes.  A hypothetical history might look like
> this:

>  B---C
> /
>X---X---A
>   /
>   ---o---o your mainline

> where 'X' denotes those unwanted changes.

> >...<

> The symmetiric case where _you_ have wrong changes do not need "-s
> theirs".  These mistakes X are yours, so are the changes depend on
> them:

>  B---C
> /
>X---X---A
>   /
>   ---o---o their mainline

> and you can just rebase A, B and C on top of their mainline while
> getting rid of Xs yourself before publishing.

and that is where the gotcha comes -- what if "my" changes were already
published?  then I would like to avoid the rebase, and would -s theirs
to choose "their" solution in favor of mine and be able to push so
others could still "fast-forward" to the new state.

So -- as to me it remains 'symmetric' ;)

> There may be valid workflows that benefit from "-s theirs", and I
> would not be surprised at all if we found more of them in the past 9
> years since we had the "why -s theirs does not exist" discussion in
> 2008.  But "because -s ours can be used in reverse to emulate" is
> not a valid excuse to add "-s theirs".  It can be used a rationale
> against adding it (e.g. "-s theirs generally is discouraged because
> it forsters a bad workflow, but in a very rare case where it might
> be useful, you can always check out their branch and merge yours
> using '-s ours' to emulate it, so we do not lose any functionality
> even if we did not add it"), though.

sure, git is flexible, so workarounds could always be found, but often
many options are just a matter of convenience.  And here -s theirs would
be one of them besides my other use case where it is a somewhat "by
design" workflow, and -s theirs is use to take their exact state I would
improve upon (before committing the "merge")

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

-s theirs use-case(s) Was: BUG: merge -s theirs is not in effect

[Yaroslav Halchenko]

On Mon, 25 Sep 2017, Junio C Hamano wrote:
>It is a different matter to resurrect the age old discussion that
>happend in the summer of 2008 if '-s theirs' should or should not
>exist.  In short, the previous discussion can be summarised to
>"we don't want '-s theirs' as it encourages the wrong workflow".

>
> https://public-inbox.org/git/alpine.DEB.1.00.0807290123300.2725@eeepc-johanness/
>https://public-inbox.org/git/7vtzen7bul@gitster.siamese.dyndns.org/
>https://public-inbox.org/git/20080720192130.6...@nanako3.lavabit.com/

>It is OK for people to come with new perspective and bring new
>ideas to the table.  We learned from experience while using Git
>for longer and are wiser than what we were back then, and might
>be able to make a better decision ;-)

FWIW

1. As a workaround for absence of -m theirs I using mtheirs git alias:
(I believe provided to me awhile back here on the list):

mtheirs = !sh -c 'git merge -s ours --no-commit $1 && git read-tree -m -u 
$1' -

and it worked fine for my usecases

2. I think that if there is a reason for -s ours to exist, so there for -s 
theirs
since it is just the directionality of merges which changes between the two

3. My most frequently used use-case for -m theirs strategy is repositories such 
as 

http://datasets.datalad.org/openfmri/ds01/.git

where we construct "datalad dataset" by crawling the web resource(s), and
workflow consists of 3 branches:

incoming -- content from the web "as is"
incoming-processed   -- content from the web "processed" (fully automatically),
e.g. tarballs extracted etc
master   -- the "final" result, delivered to public

incoming-processed   is formed  by  -s theirs --no-commit  incoming, then all
content needed to be extracted/processed (since last such merge point) is
processed and commit is done.  Such "merge" allows us to establish a point of
previous "processing state" so we could react appropriately whenever anything
in "incoming" branch changes (so that there is a new commit).

  And then incoming-processed is merged (regular recursive) into the
master branch, which might have further "manual" tune ups.

PS thanks for CCing replies

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

-X theirs does not resolve symlink conflict Was: BUG: merge -s theirs is not in effect

[Yaroslav Halchenko]

On Mon, 25 Sep 2017, Junio C Hamano wrote:

> Yaroslav Halchenko <y...@onerussian.com> writes:

> > d'oh, indeed there is no git-merge-theirs  neither in debian pkg or a 
> > freshly
> > built git  and I found a rogue script in the PATH (which did nothing
> > apparently, sorry!). BUT I was originally mislead by the --help/manpage:

> Ahh, you're right.  The text does make readers expect "-s theirs" to
> exist.
> ...
>  * I hope this should help things a bit.

yes it does. Thanks.  And that is where I realized that I should have used -X
theirs (not -s theirs), as the instruction on the option for the
(recursive) merge.  And now problem is more specific:

- conflict within file content editing was resolved as instructed
  (taking "theirs" version)

- BUT symlink was not taken from "theirs" and left as unresolved conflict:

$> rm -rf /tmp/repo1; mkdir /tmp/repo1; cd /tmp/repo1; git init .; ln -s sym1 
link; echo 1 > file; git add file link; git commit -m 'common'; git co -b b1 ; 
ln -sf b1link link; echo "b1 file" >| file; git commit -m 'b2 changes' -a; git 
co master; ln -sf masterlink link; echo "master file" >| file; git commit -m 
'also modified in master' -a; git merge -X theirs --no-edit b1; ls -l link; cat 
file
warning: templates not found /home/yoh/share/git-core/templates
Initialized empty Git repository in /tmp/repo1/.git/
[master (root-commit) f0b75bc] common
 2 files changed, 2 insertions(+)
 create mode 100644 file
 create mode 12 link
Switched to a new branch 'b1'
[b1 45c93ca] b2 changes
 2 files changed, 2 insertions(+), 2 deletions(-)
Switched to branch 'master'
[master 0ee6db2] also modified in master
 2 files changed, 2 insertions(+), 2 deletions(-)
Auto-merging link
CONFLICT (content): Merge conflict in link
Auto-merging file
Automatic merge failed; fix conflicts and then commit the result.
lrwxrwxrwx 1 yoh yoh 10 Sep 25 10:21 link -> masterlink
b1 file
changes on filesystem:  

 link | Unmerged
cached/staged changes:
 file | 2 +-
 link | Unmerged


PS I will followup on -s theirs in a split thread
PSS Thanks for CCing me your replies
-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

Re: BUG: merge -s theirs is not in effect (does the same as -s ours)

[Yaroslav Halchenko]

On Mon, 25 Sep 2017, Junio C Hamano wrote:

> Yaroslav Halchenko <y...@onerussian.com> writes:

> > My interest was to get remote branch "merge" the changes in the
> > branch taking the branch's version (primarily alternative symlinks
> > for git-annex'ed content) over the version in master (previous
> > merge of a similar branch).  Unfortunately -s theirs seems to do
> > actually -s ours

> What does

> ls $(git --exec-path) | grep git-merge

NB when running git just built, --exec-path reports some non existing dir
in ~:

$> git --exec-path 
/home/yoh/libexec/git-core
$> ls -l /home/yoh/libexec/git-core
ls: cannot access '/home/yoh/libexec/git-core': No such file or directory
$> which git
/home/yoh/proj/misc/git/git

> say?  

> The official Git never shipped "git-merge-theirs" as far as I know,
> and it should not exist (neither should "git merge -s theirs"; you
> can use "git reset --hard theirs" instead).

d'oh, indeed there is no git-merge-theirs  neither in debian pkg or a freshly
built git  and I found a rogue script in the PATH (which did nothing
apparently, sorry!). BUT I was originally mislead by the --help/manpage:


MERGE STRATEGIES
   The merge mechanism (git merge and git pull commands) allows the backend 
merge strategies to be chosen with -s option. Some strategies can also take 
their own options, which can be passed by giving -X
   arguments to git merge and/or git pull.
   ...
   recursive
   This can only resolve two heads using a 3-way merge algorithm. When 
there is more than one common ancestor that can be used for 3-way merge, it 
creates a merged tree of the common ancestors and uses that as
   the reference tree for the 3-way merge. This has been reported to 
result in fewer merge conflicts without causing mismerges by tests done on 
actual merge commits taken from Linux 2.6 kernel development
   history. Additionally this can detect and handle merges involving 
renames. This is the default merge strategy when pulling or merging one branch.

   The recursive strategy can take the following options:

   ours
   This option forces conflicting hunks to be auto-resolved cleanly 
by favoring our version. ...
   theirs
   This is the opposite of ours.


(Documentation/merge-strategies.txt in the sources I guess)

PS thanks for CCing me in replies!
-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

BUG: merge -s theirs is not in effect (does the same as -s ours)

[Yaroslav Halchenko]

My interest was to get remote branch "merge" the changes in the branch taking
the branch's version (primarily alternative symlinks for git-annex'ed content)
over the version in master (previous merge of a similar branch).  Unfortunately
-s theirs seems to do actually -s ours -- symlinks and content is taken from
the 'ours' branch instead of theirs.  workaround -- perform -s ours of
master within the branch, and then ff of master to that state:

$> git --version
git version 2.14.1.729.g59c0ea183a

$> rm -rf /tmp/repo1; mkdir /tmp/repo1; cd /tmp/repo1; git init .; ln -s sym1 
link; echo 1 > file; git add file link; git commit -m 'common'; git co -b b1 ; 
ln -sf b1link link; echo "b1 file" >| file; git commit -m 'b2 changes' -a; git 
co master; ln -sf masterlink link; echo "master file" >| file; git commit -m 
'also modified in master' -a; git merge -s theirs --no-edit b1; ls -l link; cat 
file
E: could not determine git repository root
warning: templates not found /home/yoh/share/git-core/templates
Initialized empty Git repository in /tmp/repo1/.git/
[master (root-commit) b6a69d0] common
 2 files changed, 2 insertions(+)
 create mode 100644 file
 create mode 12 link
Switched to a new branch 'b1'
[b1 739eb85] b2 changes
 2 files changed, 2 insertions(+), 2 deletions(-)
Switched to branch 'master'
[master 18a2da4] also modified in master
 2 files changed, 2 insertions(+), 2 deletions(-)
args: b6a69d0c0c2500530cba8bc2987a1f79998b5e74 -- HEAD 
739eb853c480b729ec07da533610243e3a6d69ee
Merge made by the 'theirs' strategy.
lrwxrwxrwx 1 yoh yoh 10 Sep 24 19:58 link -> masterlink
master file


-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

Re: [PATCH] strbuf: clear errno before calling getdelim(3)

[Yaroslav Halchenko]

On Thu, 10 Aug 2017, Jeff King wrote:

> On Thu, Aug 10, 2017 at 10:56:40PM +0200, René Scharfe wrote:

> > getdelim(3) returns -1 at the end of the file and if it encounters an
> > error, but sets errno only in the latter case.  Set errno to zero before
> > calling it to avoid misdiagnosing an out-of-memory condition due to a
> > left-over value from some other function call.

> Looks good to me.

> > Do we need to save and restore the original value of errno?  I doubt it,
> > but didn't think deeply about it, yet.

> I'd say no. Anybody depending on strbuf_getwholeline() is clearly
> already wrong in the error case. And in general I think we assume that
> syscalls can clear errno on success if they choose to (this isn't a
> syscall, but obviously it is calling some).

Shouldn't ideally errno being reset to 0 upon check of the syscall
successful run right after that syscall?  (I also see some spots within
git code where it sets errno to ENOMEM)


-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

Re: fatal: Out of memory, getdelim failed under NFS mounts

[Yaroslav Halchenko]

On Thu, 10 Aug 2017, Jeff King wrote:

> On Thu, Aug 10, 2017 at 09:58:37PM +0200, René Scharfe wrote:

> > > So the function is returning -1 and leaving ENOMEM in errno on
> > > Yaroslav's system.

> > > I wonder if we are truly hitting out of memory, though.  The same
> > > symptom could bee seen if getdelim() does not touch errno when it
> > > returns -1, but some other system call earlier set it to ENOMEM,
> > > for example.

> > That can happen when the end of a file is reached; getdelim() returns
> > -1 and leaves errno unchanged.  So we need to set errno to 0 just
> > before that call.

> Good catch. That's a bug in my original conversoin of
> strbuf_getwholeline().

I think this did it!  at least on this simple test... yet to test a bit
more in those scenarios we ran into it before while testing git-annex.

commit 36ef5e3ad2c187d3be664c33dbc8c06e59bceaf4 (HEAD -> bf-seterrno0)
Author: Yaroslav O. Halchenko 
Date:   Thu Aug 10 20:26:47 2017 +

BF: set errno to 0 before getdelim call to get unbiased assesment later

diff --git a/strbuf.c b/strbuf.c
index 89d22e3b0..323c49ceb 100644
--- a/strbuf.c
+++ b/strbuf.c
@@ -476,6 +476,7 @@ int strbuf_getwholeline(struct strbuf *sb, FILE *fp, int 
term)
/* Translate slopbuf to NULL, as we cannot call realloc on it */
if (!sb->alloc)
sb->buf = NULL;
+   errno = 0;
r = getdelim(>buf, >alloc, term, fp);
 
if (r > 0) {

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

Re: fatal: Out of memory, getdelim failed under NFS mounts

[Yaroslav Halchenko]

Thank you Junio

On Thu, 10 Aug 2017, Junio C Hamano wrote:
> There is only one getdelim() call, which was introduced in v2.5.0
> timeframe, and it is used like this:

>   r = getdelim(>buf, >alloc, term, fp);

>   if (r > 0) {
>   sb->len = r;
>   return 0;
>   }
>   assert(r == -1);

>   /*
>* Normally we would have called xrealloc, which will try to free
>* memory and recover. But we have no way to tell getdelim() to do so.
>* Worse, we cannot try to recover ENOMEM ourselves, because we have
>* no idea how many bytes were read by getdelim.
>*
>* Dying here is reasonable. It mirrors what xrealloc would do on
>* catastrophic memory failure. We skip the opportunity to free pack
>* memory and retry, but that's unlikely to help for a malloc small
>* enough to hold a single line of input, anyway.
>*/
>   if (errno == ENOMEM)
>   die("Out of memory, getdelim failed");

> So the function is returning -1 and leaving ENOMEM in errno on
> Yaroslav's system.  

> I wonder if we are truly hitting out of memory, though.  The same
> symptom could bee seen if getdelim() does not touch errno when it
> returns -1, but some other system call earlier set it to ENOMEM,
> for example.

> If the same version of Git is recompiled there without HAVE_GETDELIM
> defined, would it still die with out of memory (presumably inside
> the call to strbuf_grow() in the strbuf_getwholeline() function)?

will check now...  for my own education (rotten by Python) -- how
do you know which syscall set errno to be analyzed at this specific
point?  may be it was already set to ENOMEM before entry to this
function?

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

Re: fatal: Out of memory, getdelim failed under NFS mounts

[Yaroslav Halchenko]

Thank you René!  comments/answers embedded below

On Thu, 10 Aug 2017, René Scharfe wrote:

> Am 09.08.2017 um 19:39 schrieb Yaroslav Halchenko:
> > More context (may be different issue(s)) could be found at
> > http://git-annex.branchable.com/forum/git-annex_add_out_of_memory_error/
> > but currently I am consistently reproducing it while running
> > git (1:2.11.0-3 debian stretch build) within debian stretch singularity
> > environment [1].

> > External system is Centos 6.9, and git 1.7.1 (and installed in modules
> > 2.0.4) do not show similar buggy behavior.

> > NFS mounted partitions are bind mounted inside the sinularity space and
> > when I try to do some git operations, I get that error inconsistently , e.g.

> > yhalchen@discovery:/mnt/scratch/yoh/datalad$ git pull --ff-only origin 
> > master
> > fatal: Out of memory, getdelim failed
> > error: git://github.com/datalad/datalad did not send all necessary 
> > objects

> > yhalchen@discovery:/mnt/scratch/yoh/datalad$ git pull --ff-only origin 
> > master
> > fatal: Out of memory, getdelim failed
> > error: git://github.com/datalad/datalad did not send all necessary 
> > objects

> > yhalchen@discovery:/mnt/scratch/yoh/datalad$ git pull --ff-only origin 
> > master
> > From git://github.com/datalad/datalad
> >  * branch  master -> FETCH_HEAD
> > fatal: Out of memory, getdelim failed

> > and some times it succeeds.  So it smells that some race condition
> > somewhere...?

> I doubt the type of file system matters.  

So far it has been a very consistent indicator.  I did not manage to get
this error while performing the same operation under /tmp (bind to local
mounted drive), where it also feels going faster (again suggesting that
original issue is some kind of a race)

> The questions are: How much
> main memory do you have, what is git trying to cram into it, is there
> a way to reduce the memory footprint or do you need to add more RAM?
> ... reordered ...
> "free" and "ulimit -a" can help you find out how much memory you can
> use.

I think those aren't the reason:

yhalchen@discovery:/mnt/scratch/yoh/datalad$ free -h
  totalusedfree  shared  buff/cache   available
Mem:   126G2.5G 90G652K 33G123G
Swap:  127G1.7M127G
yhalchen@discovery:/mnt/scratch/yoh/datalad$ ulimit
unlimited

> > any recommendations on how to pin point the "offender"? ;)
> Running "GIT_TRACE=1 git pull --ff-only origin master" would be a
> good start, I think, to find out which of the different activities
> that pull is doing causes the out-of-memory error.

samples of bad, and then good runs (from eyeballing -- the same until
error message):

yhalchen@discovery:/mnt/scratch/yoh$ cat git_trace_bad.log
14:05:25.782270 git.c:371   trace: built-in: git 'pull' '--ff-only' 
'origin' 'master'
14:05:25.795036 run-command.c:350   trace: run_command: 'fetch' 
'--update-head-ok' 'origin' 'master'
14:05:25.795332 exec_cmd.c:116  trace: exec: 'git' 'fetch' 
'--update-head-ok' 'origin' 'master'
14:05:25.797212 git.c:371   trace: built-in: git 'fetch' 
'--update-head-ok' 'origin' 'master'
14:05:25.904088 run-command.c:350   trace: run_command: 'rev-list' 
'--objects' '--stdin' '--not' '--all' '--quiet'
14:05:26.085954 run-command.c:350   trace: run_command: 'index-pack' 
'--stdin' '--fix-thin' '--keep=fetch-pack 11652 on 
discovery.hpcc.dartmouth.edu' '--pack_header=2,103'
14:05:26.086333 exec_cmd.c:116  trace: exec: 'git' 'index-pack' 
'--stdin' '--fix-thin' '--keep=fetch-pack 11652 on 
discovery.hpcc.dartmouth.edu' '--pack_header=2,103'
14:05:26.088382 git.c:371   trace: built-in: git 'index-pack' 
'--stdin' '--fix-thin' '--keep=fetch-pack 11652 on 
discovery.hpcc.dartmouth.edu' '--pack_header=2,103'
14:05:26.133326 run-command.c:350   trace: run_command: 'rev-list' 
'--objects' '--stdin' '--not' '--all' '--quiet'
14:05:26.133688 exec_cmd.c:116  trace: exec: 'git' 'rev-list' 
'--objects' '--stdin' '--not' '--all' '--quiet'
14:05:26.135493 git.c:371   trace: built-in: git 'rev-list' 
'--objects' '--stdin' '--not' '--all' '--quiet'
fatal: Out of memory, getdelim failed
error: git://github.com/datalad/datalad did not send all necessary objects

14:05:26.138838 run-command.c:350   trace: run_command: 'gc' '--auto'
14:05:26.139131 exec_cmd.c:116  trace: exec: 'git' 'gc' '--auto'
14:05:26.141108 git.c:371   trace: built-in: git 'gc' '--auto'


yhalchen@discovery:/mnt/scratch/yoh$ cat git_trace_good.log
14:05:37.851862 git.c:371   trace: built-in: git 'pull' '--ff-only' 
'origi

fatal: Out of memory, getdelim failed under NFS mounts

[Yaroslav Halchenko]

Dear Git gurus,

More context (may be different issue(s)) could be found at 
http://git-annex.branchable.com/forum/git-annex_add_out_of_memory_error/
but currently I am consistently reproducing it while running 
git (1:2.11.0-3 debian stretch build) within debian stretch singularity
environment [1].  

External system is Centos 6.9, and git 1.7.1 (and installed in modules
2.0.4) do not show similar buggy behavior.

NFS mounted partitions are bind mounted inside the sinularity space and
when I try to do some git operations, I get that error inconsistently , e.g.

yhalchen@discovery:/mnt/scratch/yoh/datalad$ git pull --ff-only origin 
master
fatal: Out of memory, getdelim failed
error: git://github.com/datalad/datalad did not send all necessary 
objects

yhalchen@discovery:/mnt/scratch/yoh/datalad$ git pull --ff-only origin 
master
fatal: Out of memory, getdelim failed
error: git://github.com/datalad/datalad did not send all necessary 
objects

yhalchen@discovery:/mnt/scratch/yoh/datalad$ git pull --ff-only origin 
master
From git://github.com/datalad/datalad
 * branch  master -> FETCH_HEAD
fatal: Out of memory, getdelim failed

and some times it succeeds.  So it smells that some race condition
somewhere...?

any recommendations on how to pin point the "offender"? ;)  Here is the
trailer of one of the straced calls:

...
[pid 19713] getcwd("/ihome/yhalchen/datalad", 129) = 24
[pid 19713] stat(".git", {st_mode=S_IFDIR|0755, st_size=322, ...}) = 0
[pid 19713] lstat(".git/HEAD", {st_mode=S_IFREG|0644, st_size=41, ...}) = 0
[pid 19713] open(".git/HEAD", O_RDONLY) = 3
[pid 19713] read(3, "39f80454d31cfb691b006302b1f29dee"..., 255) = 41
[pid 19713] read(3, "", 214)= 0
[pid 19713] close(3)= 0
[pid 19713] lstat(".git/commondir", 0x7ffc1a571190) = -1 ENOENT (No such file 
or directory)
[pid 19713] access(".git/objects", X_OK) = 0
[pid 19713] access(".git/refs", X_OK)   = 0
[pid 19713] lstat(".git/commondir", 0x7ffc1a571120) = -1 ENOENT (No such file 
or directory)
[pid 19713] open(".git/config", O_RDONLY) = 3
[pid 19713] fstat(3, {st_mode=S_IFREG|0644, st_size=257, ...}) = 0
[pid 19713] mmap(NULL, 528384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, 
-1, 0) = 0x2ba7894e7000
[pid 19713] read(3, "[core]\n\trepositoryformatversion "..., 524288) = 257
[pid 19713] read(3, "", 524288) = 0
[pid 19713] close(3)= 0
[pid 19713] munmap(0x2ba7894e7000, 528384) = 0
[pid 19713] stat(".", {st_mode=S_IFDIR|0755, st_size=907, ...}) = 0
[pid 19713] getcwd("/ihome/yhalchen/datalad", 129) = 24
[pid 19713] chdir(".")  = 0
[pid 19713] getcwd("/ihome/yhalchen/datalad", 130) = 24
[pid 19713] lstat("/ihome/yhalchen/datalad", {st_mode=S_IFDIR|0755, 
st_size=907, ...}) = 0
[pid 19713] chdir("/ihome/yhalchen/datalad") = 0
[pid 19713] stat(".git", {st_mode=S_IFDIR|0755, st_size=322, ...}) = 0
[pid 19713] lstat(".git/commondir", 0x7ffc1a571140) = -1 ENOENT (No such file 
or directory)
[pid 19713] access("/etc/gitconfig", R_OK) = -1 ENOENT (No such file or 
directory)
[pid 19713] access("/ihome/yhalchen/.config/git/config", R_OK) = -1 ENOENT (No 
such file or directory)
[pid 19713] access("/ihome/yhalchen/.gitconfig", R_OK) = -1 ENOENT (No such 
file or directory)
[pid 19713] access(".git/config", R_OK) = 0
[pid 19713] open(".git/config", O_RDONLY) = 3
[pid 19713] fstat(3, {st_mode=S_IFREG|0644, st_size=257, ...}) = 0
[pid 19713] brk(0x2ba78afea000) = 0x2ba78afea000
[pid 19713] read(3, "[core]\n\trepositoryformatversion "..., 524288) = 257
[pid 19713] read(3, "", 524288) = 0
[pid 19713] close(3)= 0
[pid 19713] access("/etc/gitconfig", R_OK) = -1 ENOENT (No such file or 
directory)
[pid 19713] access("/ihome/yhalchen/.config/git/config", R_OK) = -1 ENOENT (No 
such file or directory)
[pid 19713] access("/ihome/yhalchen/.gitconfig", R_OK) = -1 ENOENT (No such 
file or directory)
[pid 19713] access(".git/config", R_OK) = 0
[pid 19713] open(".git/config", O_RDONLY) = 3
[pid 19713] fstat(3, {st_mode=S_IFREG|0644, st_size=257, ...}) = 0
[pid 19713] read(3, "[core]\n\trepositoryformatversion "..., 524288) = 257
[pid 19713] read(3, "", 524288) = 0
[pid 19713] close(3)= 0
[pid 19713] open(".git/objects/pack", 
O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
[pid 19713] fstat(3, {st_mode=S_IFDIR|0755, st_size=270, ...}) = 0
[pid 19713] getdents(3, /* 6 entries */, 524288) = 336
[pid 19713] 
access(".git/objects/pack/pack-fd9e70075570d8ec41f12605852f54f1cb9771a8.keep", 
F_OK) = -1 ENOENT (No such file or directory)
[pid 19713] 
stat(".git/objects/pack/pack-fd9e70075570d8ec41f12605852f54f1cb9771a8.pack", 
{st_mode=S_IFREG|0444, st_size=10019975, ...}) = 0
[pid 19713] 
access(".git/objects/pack/pack-f1fc124e3aa1619d65a6ba56219f84871a762775.keep", 
F_OK) = -1 ENOENT (No such file or directory)
[pid 19713] 

Re: in case you want a use-case with lots of submodules

[Yaroslav Halchenko]

On Mon, 19 Jun 2017, Stefan Beller wrote:

> On Mon, Jun 19, 2017 at 8:59 AM, Yaroslav Halchenko <y...@onerussian.com> 
> wrote:
> > Hi All,

> > On a recent trip I've listened to the git minutes podcast episode and
> > got excited to hear  Stefan Beller (CCed just in case) describing
> > ongoing work on submodules mechanism.  I got excited, since e.g.
> > performance improvements would be of great benefit to us too.

> If you're mostly interested in performance improvements of the status
> quo (i.e. "make git-submodule fast"), then the work of Prathamesh
> Chavan (cc'd) might be more interesting to you than what I do.
> He is porting git-submodule (which is mostly a shell script nowadays)
> to C, such that we can save a lot of process invocations and can do
> processing within one process.

ah -- cool.  I would be eager to test it out, thanks!  would be
interesting to see if it positively affects our overall performance.
Pointers to that development would be welcome!

> > http://datasets.datalad.org ATM provides quite a sizeable (ATM 370
> > repositories, up to 4 levels deep) hierarchy of git/git-annex
> > repositories all tied together via git submodules mechanism.  And as the
> > collection grows, interactions with it become slower, so additional
> > options (such as --ignore-submodules=dirty  to status) become our
> > friends.

> I am not as much concerned about the 370 number than about the
> 4 layers of nesting. In my experience the nested submodule case
> is a little bit error prone and the bug reports are not as frequent as
> there are not as many users of nesting, yet(?)

well -- part of the story here is that we are forced to use/have full
blown .git/ directories (for git-annex symlinks to content files to
work) within submodules instead of .git file with a reference under
parent's .git/modules.   So we can 'slice' at any level and I
guess that is why may be avoiding some possibly issues due to nesting
and the "parent has all .git/modules" approach.

> In a neighboring thread on the mailing list we have a discussion
> on the usefulness of being on branches than in detached HEAD
> in the submodules.
> https://public-inbox.org/git/0092cdd27c5f9d418b0f3e9b5d05be0801028...@sbs2011.opfingen.plc2.de/

> This would not break non-ambiguously, rather it would add
> ease of use.

that is indeed a common caveat... I am not sure if any heuristic
approach would provide a 'bullet proof' solution.  I might even prefer a
hardcoded 'branch-name' to be listed/associated with each submodule
within .gitmodules.  In the datalad case, detached HEAD is common
whenever someone installs "outdated" (branch of which progressed
forward) submodule.  In this case we just check if the branch after "git
clone"  (but before git submodule update) includes the pointed by
Subproject commit, and if so -- we announce that it must be the branch
(so far it is always "master" branch anyways ;) )

> > So I thought to share this as a use-case happen you need more
> > motivation or just a real-case test-bed for your work.  And thank
> > you again for making Git even Greater.

> Thanks for the motivation. :)

the least I could do ;)

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

in case you want a use-case with lots of submodules

[Yaroslav Halchenko]

Hi All,

On a recent trip I've listened to the git minutes podcast episode and
got excited to hear  Stefan Beller (CCed just in case) describing
ongoing work on submodules mechanism.  I got excited, since e.g.
performance improvements would be of great benefit to us too.

In our project, http://datalad.org, git submodules is the basic
mechanism to bring multiple "datasets" (mix of git and git-annex'ed
repositories)  under the same roof so we could non-ambiguously
version them all at any level.

http://datasets.datalad.org ATM provides quite a sizeable (ATM 370
repositories, up to 4 levels deep) hierarchy of git/git-annex
repositories all tied together via git submodules mechanism.  And as the
collection grows, interactions with it become slower, so additional
options (such as --ignore-submodules=dirty  to status) become our
friends.

So I thought to share this as a use-case happen you need more
motivation or just a real-case test-bed for your work.  And thank
you again for making Git even Greater.

P.S. Please CCme in your replies (if any), I am not on the list

With best regards,
-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

inconsistent output from git add about ignored files

[Yaroslav Halchenko]

Dear Git gurus,

It seems that there is some inconsistency in output of git while it is
ignoring files:  

if a single path within ignored directory is provided  -- git outputs
the file path.  If multiple files are provided (some of which aren't
ignored) -- git outputs only the ignored directory name:

% git --version
git version 2.10.2
% git status
On branch master
Untracked files:
  (use "git add ..." to include in what will be committed)

bu

nothing added to commit but untracked files present (use "git add" to track)
% cat .gitignore
*.me
% git add blah.me/bu
The following paths are ignored by one of your .gitignore files:
blah.me/bu
Use -f if you really want to add them.
% git add blah.me/bu bu
The following paths are ignored by one of your .gitignore files:
blah.me
Use -f if you really want to add them.
% git status
On branch master
Changes to be committed:
  (use "git reset HEAD ..." to unstage)

new file:   bu



So note that in the first case it reports "blah.me/bu" whenever in the
second -- only "blah.me".

P.S. Please CC us in your replies.

With best regards,
-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

Re: [wishlist?] make submodule commands robust to having non-submodule Subprojects

[Yaroslav Halchenko]

On Thu, 15 Sep 2016, Junio C Hamano wrote:

> >> which then stops without even looking at other submodules.

> >> I think it would be more logical to make it a 'warning:' not a 'fatal:' and
> >> proceed.

> Making "git submodule" listing to continue from that point may be
> one thing, but do we have a sensible way in "git submodule add" to
> allow the user to recover from this condition?  That is, "git add"
> is a right way to tell the core level that there is a gitlink, but
> as Yaroslav correctly observed in the early part of his message,
> having that gitlink alone is not good enough for the world view of 
> "git submodule" that sits at higher-layer.  And the usual way to
> tell the submodule layer that there is a submodule is with "git
> submodule add", but

>   $ git init top
> $ cd top
> $ git commit --allow-empty -m 'initial in top'
> $ git init sub
> $ git -C sub commit --allow-empty -m 'initial in sub'

> $ git add sub
>   $ git submodule
> fatal: no submodule mapping found in .gitmodules for path 'sub'

>   $ git submodule add ./sub sub
> 'sub' already exists in the index
>   $ git submodule add -f ./sub sub
> 'sub' already exists in the index

FWIW
I could have sworn that I have tried to 'submodule add' it and it
worked... but pragmatically I just did edit .gitmodules, added the
record for it, committed it, and then iirc git submodule update --init
which seemed to make  git happy... FTR:

$> git submodule add ./sub ./sub
'sub' already exists in the index

$> git submodule add ./sub/ ./sub/
'sub' already exists in the index

$> vim .gitmodules

$> git add .gitmodules
cached/staged changes:  


  
 .gitmodules | 4 
 sub | 1 +

$> git submodule update --init
Submodule 'sub' (/tmp/111/top/sub) registered for path 'sub'
cached/staged changes:  


  
 .gitmodules | 4 
 sub | 1 +

$> git commit -m 'added finally'
[master aa6d912] added finally
 2 files changed, 5 insertions(+)
 create mode 100644 .gitmodules
 create mode 16 sub

$> git submodule
 6f574b298ef51aebd36daafad450a3e38802ca03 sub (heads/master)


> I highly suspect that the user will then get stuck at this point,
> after trying to "submodule add" and then even attempting to force
> it.

> I think that is a more pressing thing to address.  Once we make it
> easier for the user to bring a half-initialized submodule properly
> into the world view of the submodule subsystem, we would have to
> worry about the reported failure case even less and you do not need 
> to pile on workaround options to let things continue in a state that
> is half-broken (that is, in a state that is perfectly sane to the
> core layer, but is not liked by the submodule layer).

do you foresee any unpleasant side-effects from above manual editing
.gitmodules/submodule update --init ?

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

Re: [wishlist?] make submodule commands robust to having non-submodule Subprojects

[Yaroslav Halchenko]

On Thu, 15 Sep 2016, Stefan Beller wrote:

> > I think it would be more logical to make it a 'warning:' not a 'fatal:' and
> > proceed.

> So maybe we would want to introduce a switch
>   `--existing-but-unconfigure-gitlinks=(warn|ignore)`
> as well as
> `git config submodule.existing-but-unconfigured (warn|ignore)`
> for a more permanent solution?

possibly ignorant question:  is  gitlink === Subprojector a
Subproject is a kinda of a gitlink and there are other gitlinks which
aren't Subprojects? ;)

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

[wishlist?] make submodule commands robust to having non-submodule Subprojects

[Yaroslav Halchenko]

NB echos some questions of mine a few days back on IRC about Subprojects
and submodules 

If e.g. you just 'git add' a subdirectory which is a git repository, git
adds it as a subproject but doesn't initiate any entry in .gitmodules
since it is the job done by submodule and git core itself is
agnostic of those beasts.

But having then this "Subproject"s which aren't registered as submodules
(and I haven't found any other use for them besides being a submodule)
brakes "git submodule" commands, e.g.

$> git submodule
 cc6a09ac06c13cf06b4f4c8b54cda9a535e4e385 ds01 (2.0.0+4)
 0a9f3b66e06a2137311a537b7377c336f1fb30ad ds02 (1.0.0-3-g0a9f3b6)
 9da7e4f4221699915645ac2003298c6aba2db109 ds03 (1.1.0+4)
 fe16cacb5cb9b4d53c50e498298fab182500e147 ds05 (2.0.0+3)
 6898d99ff3ba26880183ed3672a458a7fcde1737 ds06 (2.0.0+2)
 bbd10f634fe87e9d5853df3a891edbdb18cda7f9 ds07 (2.0.0+3)
 138e6730193c0585a69b8baf5b9d7a4439e83ecc ds08 (2.0.0+2)
 ddf3a4cf7ce51a01a664e6faff4b8334b8414b1f ds09 (2.0.1+1)
 7fa73b4df8166dba950c7dc07c3f8cdd50fca313 ds11 (1.0.0-5-g7fa73b4)
fatal: no submodule mapping found in .gitmodules for path 'ds17

which then stops without even looking at other submodules.

I think it would be more logical to make it a 'warning:' not a 'fatal:' and
proceed.

Thank you for consideration
-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

Re: git submodule add spits unrelated to actual problem error msg about .gitignore

[Yaroslav Halchenko]

On September 14, 2016 3:32:11 PM EDT, Stefan Beller  wrote:
!
>I think we could chop off "2>&1" as that would have exposed the
>underlying error.
>
>Another way to go would be to use verbose git-add and grep for
>the string "add '$sm_path'".
>
> if test -z "$force" && ! git add --verbose --dry-run
>--ignore-missing "$sm_path" |grep "add $sm_path"
>
>git-add already gives the correct (the same error message) for  the
>ignored files, so maybe we'd just do:
>
># no need for a if, but this single line will do:
>test -z "$force" && git add --dry-run git.o >/dev/null || exit 1

FWIW Imho exposing error is good but not sufficient alone, since custom 
gitignore message would still be confusing.
-- 
Sent from a phone which beats iPhone.

git submodule add spits unrelated to actual problem error msg about .gitignore

[Yaroslav Halchenko]

I have spent some time chasing the wild goose (well - the .gitignore
file) after getting:

$> git-submodule add --name fcx-1 ./fcx-1/ ./fcx-1/
The following path is ignored by one of your .gitignore files:
fcx-1
Use -f if you really want to add it.

long story short -- the culprit is this piece of code in git-submodule:

if test -z "$force" && ! git add --dry-run --ignore-missing "$sm_path" > 
/dev/null 2>&1
then
eval_gettextln "The following path is ignored by one of your .gitignore 
files:
\$sm_path
Use -f if you really want to add it." >&2
exit 1
fi


so if anything goes wrong in git add, it just reports  this error
message.

FTR -- actual problem in my case was:

$> git add --dry-run --ignore-missing fcx-1
fatal: Unable to create 
'/mnt/datasets/datalad/crawl/crcns/.git/index.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.

;-)

Cheers!
P.S. Please CC me in replies
-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

Re: [RFC 0/3] http: avoid repeatedly adding curl easy to curlm

[Yaroslav Halchenko]

On Tue, 13 Sep 2016, Eric Wong wrote:
> What is unclear to me is how only Yaroslav's repository seems to
> trigger this bug after all these years...

Thank you Eric very much for tracking down this issue!  Since issue is
intermittent, I guess people just didn't bother going through reporting
if they got an error once in a while, and/or could have attributed
to somehow misran update-server-info.

> However, I am fairly sure this fixes the bug Yaroslav
> encountered.  This patch series is also needed for 2.9.3 and
> perhaps older maintenance tracks for distros.

FWIW I have tested your branch locally - and do not observe that bug any
longer.  Thanks again!

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

git clone http:// fails some times with "Request for d53.. aborted"

[Yaroslav Halchenko]

even when (v 2.7.0) ran on the box where the server is, so
unlikely to be network issue

or from my laptop (v 2.9.3) with ok but wifi with a weakish signal to the
access point:

$> ( set -e; for s in {1..100}; do rm -rf fbirn_phaseIII ; git clone 
http://datasets.datalad.org/nidm/fbirn_phaseIII/.git; done; )
Cloning into 'fbirn_phaseIII'...
Checking connectivity... done.
Cloning into 'fbirn_phaseIII'...
error: Request for d53302dfc7ad13b786923927021039d21a10d5bd aborted
error: Unable to find d53302dfc7ad13b786923927021039d21a10d5bd under 
http://datasets.datalad.org/nidm/fbirn_phaseIII/.git
Cannot obtain needed tree d53302dfc7ad13b786923927021039d21a10d5bd
while processing commit 22dd4c49417cad6f4082ac2aebef45da8b6e473d.
error: fetch failed.

even if I build fresh 2.10.0, discovered that we can trace now CURL calls

$> export PATH=$HOME/proj/misc/git:$PATH
$> git version
git version 2.10.0
$> ( set -e; for s in {1..100}; do rm -rf fbirn_phaseIII ; 
GIT_TRACE_CURL=/tmp/git-trace-curl-failed.log git clone 
http://datasets.datalad.org/nidm/fbirn_phaseIII/.git &&  mv 
/tmp/git-trace-curl-failed.log /tmp/git-trace-curl-good.log || exit 1; done; ) 
Cloning into 'fbirn_phaseIII'...
warning: templates not found /home/yoh/share/git-core/templates
... a few more ...
Cloning into 'fbirn_phaseIII'...
warning: templates not found /home/yoh/share/git-core/templates
error: Request for d53302dfc7ad13b786923927021039d21a10d5bd aborted
error: Unable to find d53302dfc7ad13b786923927021039d21a10d5bd under 
http://datasets.datalad.org/nidm/fbirn_phaseIII/.git
Cannot obtain needed tree d53302dfc7ad13b786923927021039d21a10d5bd
while processing commit 22dd4c49417cad6f4082ac2aebef45da8b6e473d.
error: fetch failed.
( set -e; for s in {1..100}; do; rm -rf fbirn_phaseIII; GIT_TRACE_CURL= git  )  
18.70s user 8.01s system 34% cpu 1:16.84 total

NB 
- note that it is d53302dfc7ad13b786923927021039d21a10d5bd all the time!
- that one corresponds to a tree
$> git show d53302dfc7ad13b786923927021039d21a10d5bd
tree d53302dfc7ad13b786923927021039d21a10d5bd

1ec/
da9/
- I do not see that object being reported as failed to be found on the server:

$> grep d5/3302dfc7ad13b786923927021039d21a10d5bd 
/var/log/apache2/datasets.datalad.org-error.log
$> grep d5/3302dfc7ad13b786923927021039d21a10d5bd 
/var/log/apache2/datasets.datalad.org-access.log
129.170.31.34 - - [09/Sep/2016:14:14:25 -0400] "GET 
/nidm/fbirn_phaseIII/.git/objects/d5/3302dfc7ad13b786923927021039d21a10d5bd 
HTTP/1.1" 200 72
...
- and the logs say that only "good" run had it requested:

$> zgrep 3302dfc7ad13b786923927021039d21a10d5bd git-trace-curl-*.log.gz 
git-trace-curl-good.log.gz:18:09:13.749268 http.c:517  => Send 
header: GET 
/nidm/fbirn_phaseIII/.git/objects/d5/3302dfc7ad13b786923927021039d21a10d5bd 
HTTP/1.1


and here are those two GIT_TRACE_CURL traces from good and bad runs -- may be
someone with a better clue could figure it out or any other information I
should provide to help pin point why that "request for d5... aborted" ;)

http://www.onerussian.com/tmp/git-trace-curl-good.log.gz
http://www.onerussian.com/tmp/git-trace-curl-failed.log.gz

Thanks in advance and please CC me in replies
-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

Re: wishlist; unify behavior while cloning non-bare repos over http to be in line with ssh/local

[Yaroslav Halchenko]

On Tue, 10 May 2016, Junio C Hamano wrote:
> >> The necessary update to the client might be as simple as using
> >> $GIVEN_URL/.git/ and attempting the request again after seeing the
> >> probe for $GIVEN_URL/info/refs fails.

> > Sure -- workarounds are possible,...

> Just so that there is no misunderstanding, the above was not a
> workaround but is an outline of an implementation of updated http
> client shipped with Git.

ah, sorry, I have indeed might have misread it.  So we are on the same
page -- that is I saw also as the tentative implementation ;)

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: wishlist; unify behavior while cloning non-bare repos over http to be in line with ssh/local

[Yaroslav Halchenko]

On Tue, 10 May 2016, Jacob Keller wrote:
> > The necessary update to the client might as simple as using
> > $GIVEN_URL/.git/ and attempting the request again after seeing the
> > probe for $GIVEN_URL/info/refs fails.
> I know at least Jenkin's Git plugin has a workaround to solve this
> issue that is quite similar.

On Tue, 10 May 2016, Junio C Hamano wrote:

> >> traverse website since could lead to dangerous places.  But .git is under
> >> originating url directory, as well as info/ or HEAD or any other object
> >> accessed by git, so IMHO this concern is not a concern.

> I am afraid that the reason why you saw no response is primarily
> because nobody is interested in extending dumb commit-walker HTTP
> transport after the world has largely moved on and abandoned it.

> The necessary update to the client might as simple as using
> $GIVEN_URL/.git/ and attempting the request again after seeing the
> probe for $GIVEN_URL/info/refs fails.

Sure -- workarounds are possible, and we are at the state that many dependent
projects seems are doing that already (as above noted Jenkin's Git plugin, smth
along the lines probably done by github for https as well).  In my case I even
have managed to erect a lovely apache rewrite rule which seems to work, so I
can just 'git clone --recursive' a collection of 30 submodules without a hiccup
(we are also interested in .git/annex part here ;) ).  Citing here if it
comes handy for anyone

# To overcome http://thread.gmane.org/gmane.comp.version-control.git/293777
# we need to rewrite urls so that there is no need for explicit .git/
RewriteEngine On
RewriteCond "!.*/\.git/.*"
RewriteRule 
"(.*?/)((?http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: wishlist; unify behavior while cloning non-bare repos over http to be in line with ssh/local

[Yaroslav Halchenko]

On Fri, 06 May 2016, Yaroslav Halchenko wrote:

> Dear Git Folks,

> Originally this issue was mentioned in previous thread [1], and I have decided
> to bring it into a separate thread.  ATM there is a dichotomy in git behavior
> between cloning non-bare repos:  if I clone over ssh or just locally by
> providing url without trailing /.git, git senses for /.git and works just fine
> with ssh or local repositories, but fails for "dummy" http ones, the demo
> script is here [2] which produces output listed below.  From which you can see
> that  cloning using http URL to the repository without /.git fails (git 
> version
> 2.8.1, Debian).  As it was noted in [1], concern could have been to not
> traverse website since could lead to dangerous places.  But .git is under
> originating url directory, as well as info/ or HEAD or any other object
> accessed by git, so IMHO this concern is not a concern.
> ...

If there is a better venue (bug tracker?) to spark the interest
and discussion, please let me know ;)

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

wishlist; unify behavior while cloning non-bare repos over http to be in line with ssh/local

[Yaroslav Halchenko]

Dear Git Folks,

Originally this issue was mentioned in previous thread [1], and I have decided
to bring it into a separate thread.  ATM there is a dichotomy in git behavior
between cloning non-bare repos:  if I clone over ssh or just locally by
providing url without trailing /.git, git senses for /.git and works just fine
with ssh or local repositories, but fails for "dummy" http ones, the demo
script is here [2] which produces output listed below.  From which you can see
that  cloning using http URL to the repository without /.git fails (git version
2.8.1, Debian).  As it was noted in [1], concern could have been to not
traverse website since could lead to dangerous places.  But .git is under
originating url directory, as well as info/ or HEAD or any other object
accessed by git, so IMHO this concern is not a concern.

So do you think that cloning from  http urls could be adjusted so git senses
for presence of .git/ subdirectory if originating url is missing necessary
/info/refs? ?  That would make behavior uniform and help us and users in
many use-cases IMHO (in particular with relative paths to submodules as in
[1])

Thank you in advance, and please maintain CC

$> /tmp/democlone
Initiating repo
cloning via ssh: localhost:/tmp/repo1
Cloning into 'repo1-ssh'...
remote: Counting objects: 3, done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (3/3), done.
Checking connectivity... done.
SUCCESS
cloning locally: /tmp/repo1
Cloning into 'repo1-local'...
done.
SUCCESS
starting local http server
Serving HTTP on 0.0.0.0 port 8080 ...
cloning via http: http://localhost:8080/repo1
Cloning into 'repo1-http-failed'...
127.0.0.1 - - [06/May/2016 09:13:41] code 404, message File not found
127.0.0.1 - - [06/May/2016 09:13:41] "GET 
/repo1/info/refs?service=git-upload-pack HTTP/1.1" 404 -
fatal: repository 'http://localhost:8080/repo1/' not found
FAILED
doing with /.git: http://localhost:8080/repo1/.git
Cloning into 'repo1-http'...
127.0.0.1 - - [06/May/2016 09:13:41] "GET 
/repo1/.git/info/refs?service=git-upload-pack HTTP/1.1" 200 -
127.0.0.1 - - [06/May/2016 09:13:41] "GET /repo1/.git/HEAD HTTP/1.1" 200 -
127.0.0.1 - - [06/May/2016 09:13:41] "GET 
/repo1/.git/objects/36/31fa81eb6422349035c915a5a11b177688f491 HTTP/1.1" 200 -
127.0.0.1 - - [06/May/2016 09:13:41] "GET 
/repo1/.git/objects/12/ddeb87cc045cd67063e95125fbeb014bd1d9b1 HTTP/1.1" 200 -
127.0.0.1 - - [06/May/2016 09:13:41] "GET 
/repo1/.git/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391 HTTP/1.1" 200 -
Checking connectivity... done.
SUCCESS



[1] problems serving non-bare repos with submodules over http
http://thread.gmane.org/gmane.comp.version-control.git/292032
[2] http://www.onerussian.com/tmp/democlone
-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 'next'ed --allow-unrelated-histories could cause lots of grief

[Yaroslav Halchenko]

On Thu, 21 Apr 2016, Junio C Hamano wrote:
> >> It is not very productive to make such an emotional statement
> >> without substantiating _why_ a merge that adds a new root, which was
> >> declared in the thread above as "crap" (in the context of the kernel
> >> project),

> > Sorry if my statement sounded too emotional ;)  I will outline some of
> > the use-cases below.

> Thanks.  Emotional is fine, as long as you _also_ have useful
> information.

Gotcha:  I will follow "emotional + useful == fine" advice closer
next time ;)

Thank you a lot for the suggested patch with the env variable
workaround!
-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 'next'ed --allow-unrelated-histories could cause lots of grief

[Yaroslav Halchenko]

On Thu, 21 Apr 2016, Junio C Hamano wrote:

> Yaroslav Halchenko <y...@onerussian.com> writes:

> > I have decided to try 2.8.1.369.geae769a available from debian
> > experimental and through our (datalad) tests failing I became
> > aware of the 

> > 
> > https://github.com/git/git/pull/158/commits/e379fdf34fee96cd205be83ff4e71699bdc32b18
> > merge: refuse to create too cool a merge by default

> > which is planned for the next release.

> See http://thread.gmane.org/gmane.linux.kernel.gpio/15365/focus=15401
> for the backstory.

THANKS for the link!

> As this is to allow maintainers at higher levels of hierarchy not to
> have to worry about stupid mistakes happen at maintainers at lower
> levels, I'm afraid that turning this into an opt-in safety would
> defeat the point of the change in a major way.

> > ... BUT not sure if it is so
> > important as to cause a change in behavior on which some projects using
> > git through the cmdline interface might have been relying upon for
> > years!

> It is not very productive to make such an emotional statement
> without substantiating _why_ a merge that adds a new root, which was
> declared in the thread above as "crap" (in the context of the kernel
> project),

Sorry if my statement sounded too emotional ;)  I will outline some of
the use-cases below.

Meanwhile, I could argue that 'crap' in the above thread refers to the
entirety of the branch and thus more to the originating detached
root commit.  Action of creating of such a branch was also described  as
someone 'has done something TOTALLY INSANE'. And as "maintainers at
higher levels" expressed: "You actually have to *work* at making shit
like this".

I do see now the reason for introduced change of behavior clearer BUT I
would still argue that it should better be supported somehow by at least
a configuration option to not make poor mortals like yours truly
to sweat to stay compatible with multiple versions of git.

> is necessary and is a good idea in "some projects".  Maybe
> there is a valid use case that those from the kernel land didn't
> think about.

FWIW

- for git-annex (Joey was CCed from the beginning, not sure if annex
  would be affected though), it would be merging of git-annex
  branches while joining multiple annexes for the sync (e.g. by git
  annex assistant). 

- In our (datalad) case, repository is initialized with stock 'master'
  branch which carries configuration, which then used by the 'crawler'
  to initiate 'incoming' branch to contain (only) incoming data, which
  is later branched into 'incoming-processed' and later merged into
  'master', which is where such problematic merge would happen.  With
  such a workflow we can easily maintain custom changes within master,
  while automate changes to the crawled and processed data within
  those other branches being merged into master for final consumption.

  As a somewhat ugly workaround, we could probably artificially
  create an empty initial commit (forgot even how to do this magic) and
  branch-off it I guess, but I see other unit-tests failing as well, so
  I guess the situation is more chronic.

- In Debian packaging world, people often use 'debian' overlay branch
  which has no common ancestor with 'upstream' sources, but which need
  to be merged for 'in-tree' work.  I don't remember though if any of
  the tools rely on this feature (gbp iirc overlays not through the
  merge), but I know that I have used it quite a few times
  (interactively though, so yes -- could add a flag).

> > Given that git is quite 'self-healing', i.e. if someone has managed to
> > make a merge he didn't intend to, there is always a way back (e.g., as
> > simple as git reset --hard HEAD^),

> That is only true if people notice.  A mere warning would not be an
> effective prevention measure for a user who has to perform dozens of
> merges a day.

Could be argued...  Generally git's warnings is not something to be
ignored IMHO.  Do you ignore a balk of them on a daily basis?  if so --
they might better be downgraded to some kind of 'info' level msg then.

> I am personally on the fence, but right now I am on the side of
> keeping the behaviour as implemented and documented, simply because
> I haven't heard anything concrete to convince me why some people
> need to regularly do a "crap" merge (in other words, in what context
> these are not "crap" merges but ability to create them is a valuable
> part of everyday workflow).

once again -- IMHO it wasn't a 'merge' which was a crap, but the state
of the branch to be merged, and getting to that stage was non-trivial
endeavor as well ;)

Since the referenced situation happened only in 2016, I think that such
merges leading to undesired outcomes were quite a rare incident.  On the
ot

'next'ed --allow-unrelated-histories could cause lots of grief

[Yaroslav Halchenko]

Dear Git Gurus,

I guess whenever it rains it indeed pours, so it is me whining again.

I have decided to try 2.8.1.369.geae769a available from debian
experimental and through our (datalad) tests failing I became
aware of the 


https://github.com/git/git/pull/158/commits/e379fdf34fee96cd205be83ff4e71699bdc32b18
merge: refuse to create too cool a merge by default

which is planned for the next release.  I guess it is indeed a
worthwhile accident-prevention measure BUT not sure if it is so
important as to cause a change in behavior on which some projects using
git through the cmdline interface might have been relying upon for
years!

Given that git is quite 'self-healing', i.e. if someone has managed to
make a merge he didn't intend to, there is always a way back (e.g., as
simple as git reset --hard HEAD^), I am really not sure how valuable
such change of default behavior would be?  Could it may be made into a
warning instead? or reversed option "--dont-allow-unrelated-histories"?

Moreover, it was explicitly stated that "no configuration variable to
enable this by default exists and will not be added", which would cause
3rd party scripts/code/projects relying on previous behavior  to provide
version specific handling (either to add that
--allow-unrelated-histories or not)... very cumbersome!  If nothing else
remains, could there at least be a config option which we could
then use regardless of the version of git we are using for such merges?

P.S. Please maintain CC list

Thank you in advance for your consideration,
-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: problems serving non-bare repos with submodules over http

[Yaroslav Halchenko]

NB Thank you for the lively discussion!

On Wed, 20 Apr 2016, Stefan Beller wrote:

> >> So currently the protocol doesn't allow to even specify the submodules
> >> directories.

> > Depends on what you exactly mean by "the protocol", but the
> > networking protocol is about accessing a single repository.  It is
> > up to you to decide where to go next after learning what you can
> > learn from the result, typically by following what appears in
> > the .gitmodules file.

> Right. But the .gitmodules file is not sufficient.

why?

> >...<

> I think on a hosting site they could even coexist when having the
> layout as above.

>  top.git/
>  top.git/refs/{heads,tags,...}/...
>  top.git/objects/...
>  sub.git/
>  sub.git/refs/{heads,tags,...}/...
>  sub.git/objects/...

>  # the following only exist in non bare:
>  top.git/modules/sub.git/
>  top.git/modules/sub.git/refs/{heads,tags,...}/...
>  top.git/modules/sub.git/objects/...

> The later files would be more reflective of what you *really*
> want if you clone from top.git.

may be there is no need for assumptions and .gitmodules should be
sufficient?

- absolute url in .gitmodules provides absolute URL/path to the
  submodule of interest, regardless either submodule is present in
  originating repository as updated submodule.  Either cloning it
  instead of original repository would be more efficient is already a
  heuristic which might fail miserably (may be I have a faster
  connection to the original repository pointed by the absolute
  url than to this particular repository)

- relative url in .gitmodules provides relative location to the location
  of the "top" repository, and that is only when that submodule "absolute"
  url should be resolved relative to the one of the "top" repository 

NB I will consider it a separate issue either relative paths
without '../' prefix are having any sense in bare repositories.

or have I missed the point?
-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: problems serving non-bare repos with submodules over http

[Yaroslav Halchenko]

On Wed, 20 Apr 2016, Stefan Beller wrote:
> > I do realize that the situation is quite uncommon, partially I guess due
> > to git submodules mechanism flexibility and power on one hand and
> > under-use (imho) on the other, which leads to discovery of regressions
> > [e.g. 1] and corner cases as mine.

> Thanks for fixing the under-use and reporting bugs. :)

I am thrilled to help ;)

> > [1] http://thread.gmane.org/gmane.comp.version-control.git/288064
> > [2] http://www.onerussian.com/tmp/git-web-submodules.sh

> > My use case:  We are trying to serve a git repository with submodules
> > specified with relative paths over http from a simple web server.  With a 
> > demo
> > case and submodule specification [complete script to reproduce including the
> > webserver using python is at 2] such as

> > (git)hopa:/tmp/gitxxmsxYFO[master]git
> > $> tree
> > .
> > ├── f1
> > └── sub1
> > └── f2

> > $> cat .gitmodules
> > [submodule "sub1"]
> > path = sub1
> > url = ./sub1


> > 1. After cloning

> > git clone http://localhost:8080/.git

> >I cannot 'submodule update' the sub1 in the clone since its url after
> >'submodule init' would be  http://localhost:8080/.git/sub1 .  If I 
> > manually fix
> >it up -- it seems to proceed normally since in original repository I have
> >sub1/.git/ directory and not the "gitlink" for that submodule.

> So the expected URL would be  http://localhost:8080/sub1/.git ?

ATM, yes

> I thought you could leave out the .git prefix, i.e. you can type

>  git clone http://localhost:8080

> and Git will recognize the missing .git and try that as well. The relative URL
> would then be constructed as http://localhost:8080/sub1, which will use the
> same mechanism to find the missing .git ending.

[note1] Unfortunately it is not the case ATM (git version
2.8.1.369.geae769a, output is interspersed with log from the python's simple
http server):

$> git clone http://localhost:8080 xxx   
Cloning into 'xxx'... 
127.0.0.1 - - [20/Apr/2016 15:01:25] code 404, message File not found
127.0.0.1 - - [20/Apr/2016 15:01:25] "GET /info/refs?service=git-upload-pack 
HTTP/1.1" 404 -
fatal: repository 'http://localhost:8080/' not found


> > 2. If I serve the clone [2 demos that too] itself, there is no easy remedy 
> > at
> >all since sub1/.git is not a directory but a gitlink.

> Not sure I understand the second question.

If I serve via http a repository where sub1/.git is a "gitlink":

(git)hopa:/tmp/gitxxmsxYFO_[master]
$> cat sub1/.git 
gitdir: ../.git/modules/sub1

Such repository cannot be cloned:

(git)hopa:/tmp/gitxxmsxYFO_[master]git
$> git clone http://localhost:8080/sub1 /tmp/xxx
Cloning into '/tmp/xxx'...  
127.0.0.1 - - [20/Apr/2016 15:04:01] code 404, message File not found
127.0.0.1 - - [20/Apr/2016 15:04:01] "GET 
/sub1/info/refs?service=git-upload-pack HTTP/1.1" 404 -
fatal: repository 'http://localhost:8080/sub1/' not found

$> git clone http://localhost:8080/sub1/.git /tmp/xxx 
Cloning into '/tmp/xxx'...
127.0.0.1 - - [20/Apr/2016 15:04:06] code 404, message File not found
127.0.0.1 - - [20/Apr/2016 15:04:06] "GET 
/sub1/.git/info/refs?service=git-upload-pack HTTP/1.1" 404 -
fatal: repository 'http://localhost:8080/sub1/.git/' not found


> > N.B. I haven't approached nested submodules case yet in [2]

> > I wondered

> > a. could 'git clone' (probably actually some relevant helper used by fetch
> >etc) acquire ability to sense for URL/.git if URL itself doesn't point 
> > to a
> >usable git repository?

> So you mean in case of relative submodules, we need to take the parent
> url, and remove the ".git" at the end and try again if we cannot find
> the submodule?

that would be the a.2 which I have forgotten to outline ;)

in a.  I was suggesting what you have assumed [note 1 above] would be
happening (but doesn't) ATM: that /.git would be automagically sensed.

> > I think this could provide complete remedy for 1 since then relative 
> > urls
> > would be properly assembled, with similar 'sensing' for /.git for the 
> > final urls

> > I guess we could do it with rewrites/forwards on the "server side",
> > but it wouldn't be generally acceptable solution.

> > b. is there a better or already existing way to remedy my situation?

> > c. shouldn't "git clone" (or the relevant helper) be aware of remote
> >/.git possibly being a gitlink file within submodule?

> Oh. I think that non-bare repositories including submodules are not designed
> to be cloned, because they are for use in the file system.

Well -- that is the beauty of git being a distributed VCS, that non-bare repos
seems to be as nicely cloneable as bare ones. And in general it seems to work
with submodules as well, since they should be the "consistent"
philosophically... 

>  Even a local clone fails:

> # gerrit is a project I know which also has 

problems serving non-bare repos with submodules over http

[Yaroslav Halchenko]

Dear Git Folks,

I do realize that the situation is quite uncommon, partially I guess due
to git submodules mechanism flexibility and power on one hand and
under-use (imho) on the other, which leads to discovery of regressions
[e.g. 1] and corner cases as mine.

[1] http://thread.gmane.org/gmane.comp.version-control.git/288064
[2] http://www.onerussian.com/tmp/git-web-submodules.sh

My use case:  We are trying to serve a git repository with submodules
specified with relative paths over http from a simple web server.  With a demo
case and submodule specification [complete script to reproduce including the
webserver using python is at 2] such as

(git)hopa:/tmp/gitxxmsxYFO[master]git
$> tree
.
├── f1
└── sub1
└── f2

$> cat .gitmodules
[submodule "sub1"]
path = sub1
url = ./sub1


1. After cloning 

git clone http://localhost:8080/.git

   I cannot 'submodule update' the sub1 in the clone since its url after
   'submodule init' would be  http://localhost:8080/.git/sub1 .  If I manually 
fix
   it up -- it seems to proceed normally since in original repository I have
   sub1/.git/ directory and not the "gitlink" for that submodule.

2. If I serve the clone [2 demos that too] itself, there is no easy remedy at
   all since sub1/.git is not a directory but a gitlink.

N.B. I haven't approached nested submodules case yet in [2]

I wondered

a. could 'git clone' (probably actually some relevant helper used by fetch
   etc) acquire ability to sense for URL/.git if URL itself doesn't point to a
   usable git repository?

I think this could provide complete remedy for 1 since then relative urls
would be properly assembled, with similar 'sensing' for /.git for the final 
urls

I guess we could do it with rewrites/forwards on the "server side",
but it wouldn't be generally acceptable solution.

b. is there a better or already existing way to remedy my situation?

c. shouldn't "git clone" (or the relevant helper) be aware of remote
   /.git possibly being a gitlink file within submodule?


Thank you in advance for your thoughts and feedback on this case.

P.S. Please maintain the CC list in replies.
-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: git leaves behind .git/COMMIT_EDITMSG non-shared in --shared non-bare repo

[Yaroslav Halchenko]

fwiw that file is created not only by interactive tasks by with a regular 
commit -m msg as well.

On December 19, 2015 5:40:43 AM EST, Johannes Schindelin 
<johannes.schinde...@gmx.de> wrote:
>Hi Yaroslav,
>
>On Fri, 18 Dec 2015, Yaroslav Halchenko wrote:
>
>> Not sure for what batch operations that file is actually useful,
>
>None. This file is written when you commit interactively. It is deleted
>afterwards, unless aborted in a fatal manner.
>
>So I would try to find out who the heck is working interactively
>without
>finishing their commit message in that shared directory.
>
>Ciao,
>Johannes

-- 
Sent from a phone which beats iPhone.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

git leaves behind .git/COMMIT_EDITMSG non-shared in --shared non-bare repo

[Yaroslav Halchenko]

Not sure for what batch operations that file is actually useful, but the
story is that if we have a shared git repo (I know -- might not be as
common of a situation but possible and allowed to happen), then if one
from the shared group commits within that repository, it becomes
impossible for another person to commit.  

git does take care about chmod'ing all the files under .git/objects etc
for --shared operation, but leaves .git/COMMIT_EDITMSG at the
mercy of user's umask.  IMHO correct resolution, if leaving that file
behind is necessary, is to chmod it in the same fashion as any other
internal .git file in --shared mode -- with group write permission.

I have reproduced that behavior with today's version of git as of
2.7.0.rc1.5.gf3adf45.

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

gitk: highlighting commits touching path with globs doesn't work for files list

[Yaroslav Halchenko]

Hi GIT Gurus,

Highlighting files (in patch view) and commits which modified those
files is a really nice feature to have.   Often it is needed to
highlight based on a glob expression for files, e.g. '*Makefile*'
-- that seems to highlight the commits but files in the patch view
are no longer highlighted, which significantly reduces usefulness of
such a feature.

Also, those choices (Exact/IgnCase/Regexp) seems to have no relation
with the edit box when looking for touching path files, so shouldn't
it be somehow disabled/emptied to avoid users trying to create the
ultimate filepattern regexp for no good reason?  (the same for next
field on where to search through -- All Fields/... ) ;)

-- 
Yaroslav O. Halchenko
Postdoctoral Fellow,   Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html