Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-19 Thread Jens Lehmann
Am 18.04.2013 01:17, schrieb Philip Oakley:
 Would it be possible to summarise the key points and proposals of where the 
 subject is now?

Here you go, time to post our third iteration of the comparison
list, containing two updates:

- easier coding was removed from the advantages

- git submodule foreach was retired from the disadvantages

As in the two first versions, the issues in parentheses had been
brought up but were dismissed and are only kept for reference
together with the reason why they aren't relevant anymore. Only
those preceded by a '*' are still considered valid.


Advantages:

* Information is stored in one place, no need to lookup stuff in
  another file/blob.

* No need to cd-to-toplevel to change configuration in the
  .gitmodules file, the special tools to edit link information
  will work in any subdirectory.

(It is all but clear that this approach will lead to easier
coding, some parts of the code - like rm and mv - will profit
from that while others won't, e.g. we have to implement the link
object manipulation tools that are not needed for .gitmodules
and we get another indirection retrieving the submodule commit
from the link object. And then there is the fact that the new
code would have to catch up with functionality already coded
using .gitmodules, like the status/diff ignore and the fetch
flags).

(We currently need a checked out work tree to access the
.gitmodules file, but there is ongoing work to read the
configuration directly from the database)

(While it is easier to merge the link object, a .gitmodules
aware merge driver would work just as well)


Disadvantages:

* Changes in user visible behavior, compatibility problems when
  Git versions are mixed.

* Special tools are needed to edit submodule information where
  currently a plain editor is sufficient and a standard format
  is used.

* merge conflicts are harder to resolve and require special git
  commands, solving them in .gitmodules is way more intuitive
  as users are already used to conflict markers.

* With .gitmodules we lose a central spot where configuration
  concerning many submodules can be stored

(git submodule foreach becomes harder to implement is not the
case, as that command currently also walks all tree objects and
does not read the list of submodules from the .gitmodules file)

(When we also put the submodule name in the link object we could
also retain the ability to repopulated moved submodules from
their old repo, which is found by that name)

(That a link object can have no unstaged counterpart that a file
easily has can be fixed by special casing this, e.g. in using a
file in .git/link-specs/)


As no new arguments have been brought up, it all boils down to a
change that'll hurt users badly and won't fix any issue relevant
to them. It'll bring them a flag day after which the .gitmodules
is gone and they'll have to learn new tools to update and merge
the submodule metadata (and not only the users, GUIs have to
follow and implement support for something which currently is a
perfectly normal merge conflict in a file). You'd have to smoke
really weird stuff to even consider such a change under these
circumstances (or you don't care one bit about your users).

 The submodules does need 'fixing', as does agreeing the problem and abuse 
 cases.

Sure, but almost all problems I know about are work tree related,
so changing the internal representation buys us nothing here. It
will not magically do a bisect over submodules or will recursively
update submodule work trees, and all that stuff won't be easier to
code either just because we have to get the information from a new
object instead of a gitlink/.gitmodules combo.

Let's just close this case and get back to working on things that
users will actually profit from.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-18 Thread Ramkumar Ramachandra
Philip Oakley wrote:
 Would it be possible to summarise the key points and proposals of where the
 subject is now?

Sure.

If you want an update from the current approach, wait for a v2; I'm
cooking it for some time, and getting some resulting ideas merged into
upstream early (look for clone.submoduleGitDir on the list, for
instance).  When upstream is in better shape to ease in a better
fundamental design, I'll post my v2 to the list.  I'll refrain from
posting any updates now, because I don't think the resulting
discussion will generate any value.

If you want to know what this thread was about, I think [1] and [2]
summarize my arguments quite well.

[1]: http://thread.gmane.org/gmane.comp.version-control.git/220047/focus=220436
[2]: http://thread.gmane.org/gmane.comp.version-control.git/220047/focus=220495
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-17 Thread Duy Nguyen
On Mon, Apr 8, 2013 at 7:23 AM, Jonathan Nieder jrnie...@gmail.com wrote:
 Ramkumar Ramachandra wrote:

 It's about the core object code of git parsing links, as
 opposed to a fringe submodule.c/ submodule.sh parsing .gitmodules.

 What's stopping the core object code of git parsing .gitmodules?  What
 is the core object code?  How does this compare to other metadata
 files like .gitattributes and .gitignore?

Somewhat related to the topic. Why can't .gitattributes be used for
storing what's currently in .gitmodules?
--
Duy
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-17 Thread Ramkumar Ramachandra
Duy Nguyen wrote:
 Somewhat related to the topic. Why can't .gitattributes be used for
 storing what's currently in .gitmodules?

It can.  It's just a small syntax change from key = value attributes
inside a toplevel [submodule name] section separated by newlines, to
a path marked with multiple key=value attributes separated by
whitespace.  However, we don't want to make this change because these
submodule attributes are somewhat different from .gitattributes
attributes.

Roughly speaking, the current .gitmodules design treats submodule
directories as directories with special attributes, with two
differences: these directories have a special mode in the index, and a
commit object is created in the database to represent the partial
state of this submodule.  If you think about it, the information
stored in the commit object is no less/ no more important than the
path-attribute mapping in .gitmodules.  I was arguing for using a
special OBJ_LINK to represent the full state of the submodule, and
doing away with the attributes altogether, but not everyone agrees.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-17 Thread Duy Nguyen
On Wed, Apr 17, 2013 at 9:06 PM, Ramkumar Ramachandra
artag...@gmail.com wrote:
 Duy Nguyen wrote:
 Somewhat related to the topic. Why can't .gitattributes be used for
 storing what's currently in .gitmodules?

 It can.  It's just a small syntax change from key = value attributes
 inside a toplevel [submodule name] section separated by newlines, to
 a path marked with multiple key=value attributes separated by
 whitespace.  However, we don't want to make this change because these
 submodule attributes are somewhat different from .gitattributes
 attributes.

 Roughly speaking, the current .gitmodules design treats submodule
 directories as directories with special attributes, with two
 differences: these directories have a special mode in the index, and a
 commit object is created in the database to represent the partial
 state of this submodule.

That was my thinking. .gitmodules would break if a user moves the
submodule manually (or even if .gitattributes is used)

 If you think about it, the information
 stored in the commit object is no less/ no more important than the
 path-attribute mapping in .gitmodules.  I was arguing for using a
 special OBJ_LINK to represent the full state of the submodule, and
 doing away with the attributes altogether, but not everyone agrees.

Include me to those everyone. url feels like a local thing that should
not stay in object database (another way of looking at it is like an
email address: the primary one fixed in stone in commits with .mailmap
for future substitution). Other attributes like .update,
.fetchRecursiveSubmodules... definitely should not be stored in object
database. I think if they are stored in the submodule's config file,
then the manual move problem above will go away.

And if you're dead set on storing some submodule state in object
database, why not reuse tag object with some nea header lines?
--
Duy
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-17 Thread Ramkumar Ramachandra
Duy Nguyen wrote:
 Include me to those everyone. url feels like a local thing that should
 not stay in object database (another way of looking at it is like an
 email address: the primary one fixed in stone in commits with .mailmap
 for future substitution).

We've been over this several times in earlier emails.  That's like
saying that a blob should not be stored in the object database,
because it is not fixed in stone (my OBJ_LINK is just a special kind
of blob, as I've repeated many times already).  I don't rely on what I
feel, which is why I started out by posting an implementation: the
implementation seems to indicate that getting an OBJ_LINK will
simplify a lot of things.  And that is my primary criterion for
deciding: if the implementation is simple and elegant, it must clearly
be doing something right.

Again, I'm not saying that my approach is Correct and Final.  What I'm
saying is: Here's what I've done.  Something interesting is going on.
 It's probably worth a look?

 Other attributes like .update,
 .fetchRecursiveSubmodules... definitely should not be stored in object
 database.

Coffee and other beverages definitely should be served cold.
All very nice to say, but I don't see any rationale.

 I think if they are stored in the submodule's config file,
 then the manual move problem above will go away.

What?  The submodule's .git/config?  Why should a submodule repository
know that it is being used as a submodule?  What inherent properties
of a git repository change if it is being used as a submodule?

 And if you're dead set on storing some submodule state in object
 database,

I'm not.  I'm just saying that it seems to be an interesting
alternative approach.  Considering that nobody else brought up a real
alternative approach, and chose to just keep defending .gitmodules to
the death, it's the only other approach we have.

 why not reuse tag object with some nea header lines?

Or a unified blob, which is currently what we have.  The point is to
have structured parseable information that the object-parsing code of
git code and easily slurp and give to the rest of git-core.

Please clear your reading backlog to avoid bringing up the same points
over and over again.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-17 Thread Duy Nguyen
On Wed, Apr 17, 2013 at 9:56 PM, Ramkumar Ramachandra
artag...@gmail.com wrote:
 why not reuse tag object with some nea header lines?

 Or a unified blob, which is currently what we have.  The point is to
 have structured parseable information that the object-parsing code of
 git code and easily slurp and give to the rest of git-core.

I think you misunderstood. I meant instead of introducing new object
type OBJ_LINK, you can reuse tag object and add new header lines for
your purposes.

 Please clear your reading backlog to avoid bringing up the same points
 over and over again.

Yep. I'll shut up until it's cleared.
--
Duy
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-17 Thread Ramkumar Ramachandra
Duy Nguyen wrote:
 On Wed, Apr 17, 2013 at 9:56 PM, Ramkumar Ramachandra
 artag...@gmail.com wrote:
 why not reuse tag object with some nea header lines?

 Or a unified blob, which is currently what we have.  The point is to
 have structured parseable information that the object-parsing code of
 git code and easily slurp and give to the rest of git-core.

 I think you misunderstood. I meant instead of introducing new object
 type OBJ_LINK, you can reuse tag object and add new header lines for
 your purposes.

Oh, I interpreted your typo nea as neat, when you meant new.
Yeah, it's worth exploring: I don't know what backward compatibility
benefits it will yield yet.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-17 Thread Junio C Hamano
Duy Nguyen pclo...@gmail.com writes:

 Somewhat related to the topic. Why can't .gitattributes be used for
 storing what's currently in .gitmodules?

You _could_ use gitattributes to encode, but it goes against what a
gitattributes file does or is for.  It is a mechanism to associate
groups of paths (that may not even exist) to a set of attributes.
You could list a single pattern that happens to match a single path
and at the implementation level you may be able to make it work, but
at the design/philosophical level, it is wrong.

We need info on each submodule and we need to key it with the name
of the submodule, not with its path.  At any given time, a single
submodule lives at (at most) one path, so you could still use path
as a key in the .gitattributes, but when you need to move the
submodule path, you would need to update the entry for the submodule
in .gitattributes file by finding a pattern that match the old path
and making it a pattern that match the new path.

We have a much more suitable file format that we use to associate
various values to keys: the config format.  Also having a file that
is only about submodules and nothing else means we could write a
content-aware smart ll-merge driver that can take advantage of the
knowledge that it is written in the config format and it talks about
submodules.

The answer to why can't question is no.  No, there is no reason
why you can't use it. We don't do it, because it just does not make
sense.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-17 Thread Philip Oakley

From: Ramkumar Ramachandra artag...@gmail.com
Sent: Wednesday, April 17, 2013 12:56 PM


We've been over this several times in earlier emails.  [...]



Again, I'm not saying that my approach is Correct and Final.  What I'm
saying is: Here's what I've done.  Something interesting is going on.
It's probably worth a look?


[...]

The point is to
have structured parseable information that the object-parsing code of
git code and easily slurp and give to the rest of git-core.

Please clear your reading backlog to avoid bringing up the same points
over and over again.
--


Ram,
The email thread is pretty long with a lot of too and fro, that would be 
difficult to catch up on (too much $dayjob+$family vs $sparetime).


Would it be possible to summarise the key points and proposals of where 
the subject is now?


The submodules does need 'fixing', as does agreeing the problem and 
abuse cases.


Philip 


--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-10 Thread Philip Oakley

From: Ramkumar Ramachandra artag...@gmail.com
Sent: Monday, April 08, 2013 10:03 PM

This is going nowhere.  You're stuck at making the current submodule
system work, not answering my questions, diverting conversation,
repeatedly asking the same stupid questions, labelling everything that
I say subjective, and refusing to look at the objective counterpart
(aka, the code).  It's clear to me that no matter how many more emails
I write, you're not going to concede.

I'm not interested in wasting any more of my time with this nonsense.

I give up.
--

Please don't give up. It is a bit of a 'wicked' problem [1].

Yes to taking a rest, stepping back and trying to summarise/review what 
was discussed.


I couldn't keep up with all the discussion, and I doubt many others kept 
up, especially those who have been frustrated in their (mis-) use of 
submodules. Do remember that Junio has multiple roles which belie the 
softness of the word 'maintainer'. It includes Defender of the 
Heritage in the same way that keepers of ancient monuments will want 
visitors to enjoy the site, but rail against a garish new stainless 
steel and glass entrance to the Colosseum (choose you local heritage 
site) (see [1] again).


I get confused (about sub-modules) with msysgit where git.git is a 
sub-module, and is the fastest moving (an inversion of control issue), 
and when hacking at (just) the msys level when the git sub-module isn't 
in sync.


In many ways sub-module tracking is like file renames and empty 
directories (both of which come up a lot). The submodule meta 
information issue has great similarity to the empty directories issue. 
It's about meta information, not about content (which is certified 
verified by sha1), and about how users know what is going on and get a 
(natural) feeling of control (without upsetting other users/controllers) 
.


regards

Philip
[now to schedule some time to do the catch up reading. $dayjob beckons]

[1] www.poppendieck.com/wicked.htm 


--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-09 Thread Jakub Narębski
Junio C Hamano wrote:
 Ramkumar Ramachandra wrote:

 2. If we want to make git-submodule a part of git-core (which I think
everyone agrees with), we will need to make the information in
.gitmodules available more easily to the rest of git-core.

 Care to define more easily which is another subjective word?  The
 .gitmodules file uses the bog-standard configuration format that can
 be easily read with the config.c infrastructure.  It is a separate
 matter that git_config() API is cumbersome to use, but improving it
 would help not just .gitmodules but also the regular non-submodule
 users of Git.  There is a topic in the works to read data in that
 ^^
 format from core Heiko is working on.
  ^

BTW. this is something that I was missing to implement better
submodule support in gitweb (and thus git-instaweb) than just
marking it as submodule in 'tree' view.

-- 
Jakub Narębski



--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-08 Thread Jens Lehmann
Am 07.04.2013 23:30, schrieb Ramkumar Ramachandra:
 Jonathan Nieder wrote:
 What's stopping the core object code of git parsing .gitmodules?

Just to clarify that: git core already does that. A git grep
gitmodules_config shows it is parsed by some git core commands:
checkout, commit, the diff family and fetch. Others will follow
in the recursive update series. And git mv support will teach
that command to manipulate the .gitmodules file (and I hope that
a patch teaching git rm to remove the section from .gitmodules
will be accepted in the near future).

 Nothing, except that it's perversely unnatural for object parsing code
 to parse something outside the object store.

Hmm, at least the unstaged .gitmodules file has to be parsed from
the file system. And Heiko's current work on parsing .gitmodules
directly from the object store will help here too, right?

 How does this compare to other metadata
 files like .gitattributes and .gitignore?
 
 .gitignore and .gitattributes are parsed in dir.c, where git treats
 worktree paths.  It's quite nicely integrated.

And .gitmodules is parsed in submodule.c where Git treats
.gitmodules entries. So I don't see a problem here, what am I
missing?
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-08 Thread Ramkumar Ramachandra
Jens Lehmann wrote:
 Hmm, at least the unstaged .gitmodules file has to be parsed from
 the file system.

You seem to be touting it as a distinct advantage.  In my opinion,
.gitmodules is a wart that needs to be done away with: it should _not_
be on the filesystem, just like a commit object isn't on the
filesystem.  Getting links to unstage is two hours of work, tops.  And
I'm the one writing the whole thing, so I don't see what everyone else
is complaining about.

 And Heiko's current work on parsing .gitmodules
 directly from the object store will help here too, right?

Ofcourse, you _can_ parse a blob into a struct.  It's just extremely
gross to treat a blob located in a certain tree path differently from
other blobs.  It's a perverse violation of git's fundamental design,
and I'm strongly against such a change.

What I still fail to understand is why you keep mentioning
work-in-progress.  You've had five years in which you haven't been
able to do things that I did in two days.  Yes, you _can_ keep
.gitmodules and hack around everything, but why do you _want_ to do
that?  Preserving backward compatibility is not *that* important, in
my opinion.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-08 Thread Jonathan Nieder
Hi Ram,

Ramkumar Ramachandra wrote:

 In my opinion,
 .gitmodules is a wart that needs to be done away with: it should _not_
 be on the filesystem, just like a commit object isn't on the
 filesystem.

What do you think of .gitignore and .gitattributes?  Should they be
somewhere other than the filesystem as well?

[...]
 What I still fail to understand is why you keep mentioning
 work-in-progress.  You've had five years in which you haven't been
 able to do things that I did in two days.

I don't think Jens had any obligation to work on submodules and
nothing else for the last five years. ;-)

If you end up convincing others that your tools are worth working
on and those tools pleasantly take care of the same workflows that
submodules do, then I imagine people will be happy to migrate.

Speaking only for myself, I actually prefer the submodule UI, despite
not being thrilled with the
single-.gitmodules-file-at-the-root-of-the-worktree feature.  So I
will not be working on your proposed redesign, unless it evolves
enough to be as pleasant a UI as (the long proposed UI of) submodules.

Hope that helps,
Jonathan
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-08 Thread Jonathan Nieder
Ramkumar Ramachandra wrote:
 Jens Lehmann wrote:

 Hmm, at least the unstaged .gitmodules file has to be parsed from
 the file system.

 You seem to be touting it as a distinct advantage.

To clarify what I said in a side thread: yes, as long as the submodule
metadata includes the hostname I am downloading a library from, having
it in an ordinary file is an advantage.

The problem with URLs (and especially hostnames) is that they change.
When my project's previous domain name is lost because the hosting
company lost interest, I want to be able to grep for all instances of
that domain name in my project's documentation and metadata and change
them all at once with a simple command like the following:

git grep -l -F -e oldhost.example.com |
xargs sed -i -e s/oldhost.example.com/newhost.example.com/g

When I clone a project with --no-recurse-submodules, I want to be able
to see what other servers will be contacted when I run git checkout
--recurse-submodules.  The current .gitmodules file lets me find that
out with a simple, intuitive command:

cat .gitmodules

I might change some URLs locally, because I know that some project's
upstream has moved.

git submodule init
git config --edit

On the other hand, the single .gitmodules file will be a pain to merge
if multiple branches modify it.  So I do look forward to a merge
strategy that deals more intelligently with its content, and wouldn't
have minded a design that split this information into multiple files
if we were starting over.

Jonathan
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-08 Thread Ramkumar Ramachandra
Jonathan Nieder wrote:
 What do you think of .gitignore and .gitattributes?  Should they be
 somewhere other than the filesystem as well?

I would argue that .gitignore and .gitattributes are done right.  They
are integrated into a very mature part of git-core very well, and
their nature is fundamentally different from that of .gitmodules.

.gitignore and .gitattributes specify extended globs (see: wildmatch)
rules to apply on the worktree, and can be in multiple places in the
worktree.  They apply strictly on the current worktree; they have
nothing to do with the index, and have no interaction with other
objects in the repository.  Now, you might argue that they should be
part of the tree object, but I will disagree because they don't
operate on concrete entries in the tree but rather extended globs that
match worktree paths.  .gitmodules, on the other hand, specifies
fundamental repository composition: it should be a special object in
the tree precisely because it changes the fundamental meaning of one
concrete tree entry.  It has nothing to do with path treatment in the
worktree, and hence has nothing to do with .gitattributes
or.gitignore.

 I don't think Jens had any obligation to work on submodules and
 nothing else for the last five years. ;-)

I know.  What I'm saying is that his current approach is just filled
with tons of unnecessary complexity, inelegance, and pain.  This is
evidenced by the fact that the current submodule system is pathetic
after five years of work (and I don't think the developers working on
it were particularly incompetent or lazy).

 If you end up convincing others that your tools are worth working
 on and those tools pleasantly take care of the same workflows that
 submodules do, then I imagine people will be happy to migrate.

Yes, I'm planning a strict superset of the current submodule system
features.  After some thought, I've decided not to have any feature
regressions in my first version for merge (although that means a lot
of work for me).

 Speaking only for myself, I actually prefer the submodule UI, despite
 not being thrilled with the
 single-.gitmodules-file-at-the-root-of-the-worktree feature.  So I
 will not be working on your proposed redesign, unless it evolves
 enough to be as pleasant a UI as (the long proposed UI of) submodules.

I'm very interested in building a pleasant UI.  I've always been a
person who cares deeply about UI: this is evidenced by my recent
remote.pushdefault patch, and my pull.autostash WIP.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-08 Thread Ramkumar Ramachandra
Jonathan Nieder wrote:
 git grep -l -F -e oldhost.example.com |
 xargs sed -i -e s/oldhost.example.com/newhost.example.com/g

Yes, I've had to do this too: in a proxied environment I had to
s/git:\/\//https:\/\//.  So yes, we will have features to operate on
multiple links at the same time.  I'm thinking something fine-grained
that allows you to pick which links to operate on.  It's currently a
vague thought, and I'm not sure what the implementation will look
like.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-08 Thread Duy Nguyen
On Fri, Apr 5, 2013 at 5:55 AM, Jonathan Nieder jrnie...@gmail.com wrote:
 Ramkumar Ramachandra wrote:

 1. 'git add' should not go past submodule boundaries.  I should not be
able to 'git add clayoven/' or 'git add clayoven/LICENSE'.  In
addition, the shell completion also needs to be fixed.

 Yep.  This is a bug.

I notice that this case is handled by git-add, but there is probably a
bug somewhere. Ram, can you make a test case for this?
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-08 Thread Duy Nguyen
On Mon, Apr 8, 2013 at 7:08 PM, Ramkumar Ramachandra artag...@gmail.com wrote:
 Jonathan Nieder wrote:
 What do you think of .gitignore and .gitattributes?  Should they be
 somewhere other than the filesystem as well?

 I would argue that .gitignore and .gitattributes are done right.  They
 are integrated into a very mature part of git-core very well, and
 their nature is fundamentally different from that of .gitmodules.

Probably off-topic, but I'm starting to find .gitignore can be found
in every directory a burden to day-to-day git operations. So imo it's
not done right entirely ;-)

 .gitignore and .gitattributes specify extended globs (see: wildmatch)
 rules to apply on the worktree, and can be in multiple places in the
 worktree.  They apply strictly on the current worktree; they have
 nothing to do with the index, and have no interaction with other
 objects in the repository.

Index operations sometimes read these .git{ignore,attributes}. I
believe git-archive reads worktree's .gitattributes, so it's not
really just about worktree.

 I don't think Jens had any obligation to work on submodules and
 nothing else for the last five years. ;-)

 I know.  What I'm saying is that his current approach is just filled
 with tons of unnecessary complexity, inelegance, and pain.  This is
 evidenced by the fact that the current submodule system is pathetic
 after five years of work (and I don't think the developers working on
 it were particularly incompetent or lazy).

I don't follow this thread closely, but I think there's a common
ground where improvements can benefit both approaches. There are a lot
of problems for deep integration and erasing submodule's boundaries
from UI perspective. I think maybe you can work on that first, gain
experience along the way, and maintain the link-object changes
separately. Maybe someday you will manage to switch .gitmodules with
it. Or maybe I'm wrong (partly because I did not read the whole
thread)
-- 
Duy
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-08 Thread Ramkumar Ramachandra
Duy Nguyen wrote:
 Probably off-topic, but I'm starting to find .gitignore can be found
 in every directory a burden to day-to-day git operations. So imo it's
 not done right entirely ;-)

Why is it a burden?  I would argue that the tooling support is not yet
there, but git check-ignore is a step in the right direction.  What
alternate design would you propose, just out of curiosity?

 Index operations sometimes read these .git{ignore,attributes}. I
 believe git-archive reads worktree's .gitattributes, so it's not
 really just about worktree.

I should've said largely, only affects the current worktree.

 I don't follow this thread closely, but I think there's a common
 ground where improvements can benefit both approaches. There are a lot
 of problems for deep integration and erasing submodule's boundaries
 from UI perspective. I think maybe you can work on that first, gain
 experience along the way, and maintain the link-object changes
 separately. Maybe someday you will manage to switch .gitmodules with
 it. Or maybe I'm wrong (partly because I did not read the whole
 thread)

Yes, there is some common ground.  But:

1. The inspiration for fixing fundamental design problems comes from
my redesign.  For instance, I would've never discovered the git add
bug if I'd not attempted to git add (as opposed to the unnatural
abstraction that git submodule add presents).

2. I think it is absolutely imperative that we do the redesign now,
before we've descended too far into the madness that the current
design is.  I think I'm capable of doing the redesign now, with some
help and support from the list.  My attitude doesn't align with the
I'm feeling lazy; why don't we postpone it? argument.  Let's finish
what I started now: I'm more than willing to dedicate the next few
months full-time towards finishing this and getting it merged.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-08 Thread Ramkumar Ramachandra
Duy Nguyen wrote:
 Probably off-topic, but I'm starting to find .gitignore can be found
 in every directory a burden to day-to-day git operations. So imo it's
 not done right entirely ;-)

Or are you saying it's hard to implement elegantly and efficiently in
git-core?  If so, I agree wholeheartedly.  I'm not yet sure what to do
about the situation.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-08 Thread Duy Nguyen
On Mon, Apr 8, 2013 at 9:06 PM, Ramkumar Ramachandra artag...@gmail.com wrote:
 Duy Nguyen wrote:
 Probably off-topic, but I'm starting to find .gitignore can be found
 in every directory a burden to day-to-day git operations. So imo it's
 not done right entirely ;-)

 Why is it a burden?  I would argue that the tooling support is not yet
 there, but git check-ignore is a step in the right direction.  What
 alternate design would you propose, just out of curiosity?

You don't know if .gitignore is there, so you need to check for it in
every directory. If we fixed its location (e.g. worktree's top) we
would not have to look in every directory. Then again it may be a bit
inconvenient that way. If you remove a directory, you also remove
.gitignore rules inside when you distribute .gitignore files.
Otherwise you need to clean up top .gitignore once in a while.

 1. The inspiration for fixing fundamental design problems comes from
 my redesign.  For instance, I would've never discovered the git add
 bug if I'd not attempted to git add (as opposed to the unnatural
 abstraction that git submodule add presents).

I actually spotted a similar use of git-add in the test suite [1]. You
see, it's a bug that should be fixed but in that particular case, it's
valid to add something inside a submodule. I wanted to fix that with
my read_directory rewrite (part of the pathspec stuff) but never got
around to finish it and eventually gave up, which leads to your next
point..

 2. I think it is absolutely imperative that we do the redesign now,
 before we've descended too far into the madness that the current
 design is.  I think I'm capable of doing the redesign now, with some
 help and support from the list.  My attitude doesn't align with the
 I'm feeling lazy; why don't we postpone it? argument.  Let's finish
 what I started now: I'm more than willing to dedicate the next few
 months full-time towards finishing this and getting it merged.

Good luck. Bug such a big work usually requires more than one
volunteer. If you haven't convinced (*) the community it's right,
maybe you should take a few days thinking about it again before
implementing.

[1] http://thread.gmane.org/gmane.comp.version-control.git/177454
(*) just a feeling after a quick glance, I may be terribly wrong again
--
Duy
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-08 Thread Ramkumar Ramachandra
Duy Nguyen wrote:
 Good luck. Bug such a big work usually requires more than one
 volunteer. If you haven't convinced (*) the community it's right,
 maybe you should take a few days thinking about it again before
 implementing.

Yes, I'm thinking about it before rushing in to implement it.

There will always be resistance to change, especially when it involves
a change that breaks a working implementation.  If anything, the
resistance is only going to get worse with time, as people pile more
and more hacks on top of the current submodule implementation.  I say:
do it now, before we lose steam.

As far as I can tell, I'm completely unbiased: I have no vested
interests in either implementation, and I just want to see the best
implementation win.  My conviction in the new approach has only
strengthened after discussions on this thread: there must be some
reason for that, no?

Frankly, I was hoping that atleast one or two people on the thread
would take my side of the argument (or atleast tell me that I'm not
deranged), but that hasn't happened.  Nevertheless, I hope to convince
more people by doing more work and posting a beautifully working
implementation.

I'm already prepared for the worst case: I'll be forced to dump all my
work and be disappointed with the git community.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-08 Thread Junio C Hamano
Jonathan Nieder jrnie...@gmail.com writes:

[snipped everything I agree with...]

 On the other hand, the single .gitmodules file will be a pain to merge
 if multiple branches modify it.  So I do look forward to a merge
 strategy that deals more intelligently with its content, and wouldn't
 have minded a design that split this information into multiple files
 if we were starting over.

I find it a sensible suggestion to have a content-aware merge
driver.  Such a custom merge driver to help merging a structured
datafile in the config format will have other uses when we need to
do more than the current system (outside submodules there will be
other things frotz that need information about frotz in the
future, and a .gitfrotz file would be one possible way to do so).

I do not think it needs to be split per-submodule.

When a submodule in the common ancestor was at path dirA/, and you
are merging with another branch that moved it to path dirB/, the
contents of .gitmodules file for that module (that is identified by
its name) will need a three-way merge of its .path element:

common ancestor:submodule.name.path = dirA/
ours:   submodule.name.path = dirA/
theirs: submodule.name.path = dirB/

And your content-aware merge driver should be able to do the
resolving by following the usual three-way merge rules.  We started
from the same dirA/ and only they changed, so the result is dirB/.

By the way, that's a merge driver (which deals with per-path
content merge), not a strategy (which deals with the entire tree
level merge).
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-08 Thread Junio C Hamano
Ramkumar Ramachandra artag...@gmail.com writes:

 As far as I can tell, I'm completely unbiased: I have no vested
 interests in either implementation,...
 ...
 Frankly, I was hoping that atleast one or two people on the thread
 would take my side of the argument (or atleast tell me that I'm not
 deranged), but that hasn't happened.

Aren't these two quite contradicting?

After listening to what others tell one with an unbiased mind and
finding that nobody agrees with what one initially proposed, an
unbiased person would step back, take a deep breath and think again,
before insulting the intelligence of others with a dissapointed,
like this:

 I'm already prepared for the worst case: I'll be forced to dump all my
 work and be disappointed with the git community.

Would it be possible that (at least some part of, or possibly all
of) your ideas had some merit, but with all your hostility against
the current system and the work that went behind it, you did not
communicate well enough to make others understand you?

What I found very hard to read in this thread was that your messages
all went like this:

 1. In the current system, I have to be at the top level of a
submodule to work in it (or some other problems).

 2. I will fix it in a more elegant way.

 3. I have to have a new object at the submodule path, not the
current submodule is a commit bound at the submodule path, and
information about the submodule is in .gitmodules.

There was very little concrete explanation on how #3 leads to #2,
i.e. the overall design of your new system and how it will work,
other than you would read what we currently write in .gitmodules
from a new kind of object.

When an alternative solution was suggested, all your responses were
full of subjective inelegant and ugly, and at least I couldn't
read much substance in it (here, the usual me might add maybe
others differ, but after reading this thread, I strongly suspect
that others share this problem).
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-08 Thread Ramkumar Ramachandra
Junio C Hamano wrote:
 Would it be possible that (at least some part of, or possibly all
 of) your ideas had some merit, but with all your hostility against
 the current system and the work that went behind it, you did not
 communicate well enough to make others understand you?

Agreed.  My annoyance with the current system did go a little
overboard, and I've been having a splitting headache for the last few
days.

 What I found very hard to read in this thread was that your messages
 all went like this:

  1. In the current system, I have to be at the top level of a
 submodule to work in it (or some other problems).

  2. I will fix it in a more elegant way.

  3. I have to have a new object at the submodule path, not the
 current submodule is a commit bound at the submodule path, and
 information about the submodule is in .gitmodules.

 There was very little concrete explanation on how #3 leads to #2,
 i.e. the overall design of your new system and how it will work,
 other than you would read what we currently write in .gitmodules
 from a new kind of object.

I had no way of expressing what I wanted to do except by writing code
when I started off this thread, but am in much better shape now.  Let
me try to explain my fundamental assumptions and code in a concise way
now.

1. Having a toplevel .gitmodules means that any git-core command like
add/ rm/ mv will be burdened with looking for the .gitmodules at the
toplevel of the worktree and editing it appropriately along with
whatever it was built to do (ie. writing to the index and committing
it).  This is highly unnatural.  Putting the information in link
objects means that we get a more natural UI + warts like
cd-to-toplevel disappear with no extra code.

2. If we want to make git-submodule a part of git-core (which I think
everyone agrees with), we will need to make the information in
.gitmodules available more easily to the rest of git-core.  One way to
do it without breaking anything is to unpack the root tree, look for
an entry with the path .gitmodules and handle it different from other
blobs: ie. parse it into structured data that the rest of git-core can
consume.  However, I think it is very gross as the blob is not
inherently special in any way: it's just incidentally stored at a
specific tree path.  The alternative is to have an inherently special
kind of blob (ie. link object).  In the git-core code, I can simply
match for a link object and operate on it accordingly.  As opposed to
matching a blob object, and its tree path.  Moreover, this means that
the user can simply git edit-link link from anywhere in the worktree
instead of having to refer to the appropriate section in the toplevel
.gitmodules.

3. Currently diffing/ merging one huge .gitmodules file is a mess, as
it doesn't have to conform to a strict format.  This means that I can
get conflicts between these two:

url = gh:artagnon/clayoven
url =gh:artagnon/clayoven

Moreover, since the fields are not ordered, a simple reordering of the
fields will cause a merge conflict.  The correct way to fix this is to
split up .gitmodules into many logical files, have a git
edit-gitmodules which reduces user input to a strict format, and then
write custom diff/merge drivers.  My proposal involves having a git
edit-link, and teaching git-core to diff/merge appropriately.  The
information is already in logical bits.

4. The only seeming disadvantage of not having a file accessible via
the filesystem is that it doesn't behave like a full blob.  But it
does; the code to unstage a link object (emulation) is actually very
simple: I'm currently writing it.

5. Having a first-class link object comes with functional advantages.
It means that I can have a ref pointing to link objects and easily
initialize a nested repository without having to initialize the
containing repository (ie. essentially replacing repo). We can have
true floating submodules, which is really nice in my opinion: you can
fix a library at v3.1 and switch it to v3.2 at some point in the
future without using ugly SHA-1 hexes anywhere.

6. While it is possible to work top-down from the current system, that
approach is clearly taking too long and is too painful.  This explains
why submodules haven't come a long way in the last five years.  With
my approach, I'm trying to make life simpler for everyone: it will
suddenly become much easier to hack on submodules, and it can improve
more rapidly over the next five years.  I'm not thinking about
short-term fixes precisely for this reason: the long-term goal is
worth a little bit of short-term inconvenience.

7. I estimate that replacing the current submodule system without
feature regressions will not take a lot of effort and can be done with
minimal breakages.  It's not a lot of code or anything very complex.
We just have to follow along the lines of how git-core handles blobs,
and write a little bit of code to make links behave like blobs (I'm
halfway done with this already).

Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-08 Thread Ramkumar Ramachandra
Christian Couder wrote:
 What if instead of a git submodule I want to have an hg, or, God/Linus/deity
 forbid, an SVN submodule, inside my git worktree?
 What if I just want a very big movie or .tgz downloaded from somewhere else?

Since the link object is rooted to the tree, it's impossible to have
anything but a working copy in the link directory.  How can I have a
non-git-worktree link directory without breaking checkout?  I think
that making it too generic will make the entire submodule experience
suffer, because the implementation must be coded according to the
lowest-common-denominator.  This is the mistake that the tool mr
makes: since it's so generic, it can't provide very powerful
functionality specifically for git repositories.

I'll try to think of something else.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-08 Thread Junio C Hamano
Junio C Hamano gits...@pobox.com writes:

 Jonathan Nieder jrnie...@gmail.com writes:

 [snipped everything I agree with...]

 On the other hand, the single .gitmodules file will be a pain to merge
 if multiple branches modify it.  So I do look forward to a merge
 strategy that deals more intelligently with its content, and wouldn't
 have minded a design that split this information into multiple files
 if we were starting over.

 I find it a sensible suggestion to have a content-aware merge
 driver.  Such a custom merge driver to help merging a structured
 datafile in the config format will have other uses when we need to
 do more than the current system (outside submodules there will be
 other things frotz that need information about frotz in the
 future, and a .gitfrotz file would be one possible way to do so).

 I do not think it needs to be split per-submodule.

Another thing to think about is what to do when/if we want to
express this is the default that applies to all submodules.  For
example, a superproject that binds multiple submodules may want to
say When on this branch, make all submodules also on 'next'.

With a unified single place that holds information about all
submodules, it is trivial to add a default section, perhaps like
this:

[default]
branch = next
[submodule framework]
url = ...
path = framework
[submodule common]
url = ...
path = common
branch = master ;# regardless of other modules...

on top of the submodule.name.branch mechanism for floating
checkout (the default is of course not limited to branch but
applies in general).

It is not obvious where such a default piece should go once you
start splitting these into per-submodule files, be it a separate but
still in-tree file that is different from the submodule it desribes,
or a blob-like object that sits at the path for the submodule in the
tree and in the index as Ram wants to do (as I kept saying, the
storage mechanism is not fundamental).

This is similar to why .gitattributes is easy to work with, I think.
You can describe the information about paths in that file (which
lives at a place different from the paths that are described), and
you can have a catch-all rule in it.

This is a tangent, but you could build a system that attaches
attributes to individual paths and hide the attributes from the
working tree filesystem (think: svn:blah) and have a set of special
commands (think: svn propset, proplist, etc.) to work with them, and
that is an equally valid way to implement attributes (it does not
make .gitattributes less valid way to do so, though).





--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-08 Thread Junio C Hamano
Ramkumar Ramachandra artag...@gmail.com writes:

 1. Having a toplevel .gitmodules means that any git-core command like
 add/ rm/ mv will be burdened with looking for the .gitmodules at the
 toplevel of the worktree and editing it appropriately along with
 whatever it was built to do (ie. writing to the index and committing
 it).

Burdened is a subjective word.  What's bad about having a single place
you know you can read and find out information about things?  You
have to learn about them to do anything specific to them anyway.

 This is highly unnatural.

Unnatural is a subjective word, and there is no justification I see
here in your message.

 Putting the information in link
 objects means that we get a more natural UI + warts like
 cd-to-toplevel disappear with no extra code.

I do not see how link objects _means_ natural UI, yet, without
an explanation how one leads to the other.

What does cd-to-toplevel have anything to do with it?  In case you
did not notice, all the core commands internally cd-to-toplevel and
carry the prefix information while doing so, and prepend the
prefix to user-supplied paths to find which path the user is talking
about.  So cd to toplevel before starting to carry the operation out
is a natural pattern inside Git.  As many people already told you,
the user has to run 'git submodule' from the top-level of the
submodule working tree is a simple oversight of the implementation.

 2. If we want to make git-submodule a part of git-core (which I think
 everyone agrees with), we will need to make the information in
 .gitmodules available more easily to the rest of git-core.

Care to define more easily which is another subjective word?  The
.gitmodules file uses the bog-standard configuration format that can
be easily read with the config.c infrastructure.  It is a separate
matter that git_config() API is cumbersome to use, but improving it
would help not just .gitmodules but also the regular non-submodule
users of Git.  There is a topic in the works to read data in that
format from core Heiko is working on.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-08 Thread Ramkumar Ramachandra
Junio C Hamano wrote:
 Ramkumar Ramachandra artag...@gmail.com writes:

 1. Having a toplevel .gitmodules means that any git-core command like
 add/ rm/ mv will be burdened with looking for the .gitmodules at the
 toplevel of the worktree and editing it appropriately along with
 whatever it was built to do (ie. writing to the index and committing
 it).

 Burdened is a subjective word.  What's bad about having a single place
 you know you can read and find out information about things?  You
 have to learn about them to do anything specific to them anyway.

Burden refers to the extra work of looking for a file in the
worktree, when this is completely unnecessary if you use a link
object.

 This is highly unnatural.

 Unnatural is a subjective word, and there is no justification I see
 here in your message.

git's design follows along the lines of the UNIX philosophy: do one
thing, do it well.  git add/ rm/ mv have a very sharply defined task:
they first lock the index file, read the_index using read_cache(),
build a cache_entry struct using user-supplied data (this might
involve worktree code from dir.c to recurse subdirectories, for
example), write that cache_entry to the_index (removing existing
entries with cache_tree_invalidate_path() if necessary), and finally
write the_index to the index file, releasing the lock.  Would you
agree that any operation that doesn't follow along this line is
unnatural?  What then, does writing a special file in the worktree
(aka .gitmodules) have to do with this entire process?

Does git diff/ commit/ add/ rm or any other command you can think of
rely on a special file in the worktree (aka .gitmodules) to be checked
out?  Then why does git submodule require it?  Isn't this a
requirement that is inconsistent with the rest of git-core?

 Putting the information in link
 objects means that we get a more natural UI + warts like
 cd-to-toplevel disappear with no extra code.

 I do not see how link objects _means_ natural UI, yet, without
 an explanation how one leads to the other.

I should've said means an easy route to get the existing UI to work
with little or no additional code.  Making the submodule information
available to git-core is precisely what leads to this.  In
index_path(), you can inject a case for S_IFDIR to write a link object
to the database, writing the sha1 to the supplied argument.  This is
not unnatural in any way, because we're just following along the lines
of the S_IFLNK codepath, which writes a blob object to the database.
Now index_path() is called by add_to_index(), which is the master
function for adding anything to the index.  Therefore, git add just
works.  git rm is much simpler: it calls remove_file_from_index()
which in turn calls cache_tree_invalidate() and
remove_index_entry_at().  Once the entry is removed from the cache,
our job is done.  The link object will be cleaned up at gc-time.  git
mv is just a combination of git rm and git add: it invalidates an
existing entry and adds a new one with a different name.

There is no special .gitmodules to take care of.

 What does cd-to-toplevel have anything to do with it?  In case you
 did not notice, all the core commands internally cd-to-toplevel and
 carry the prefix information while doing so, and prepend the
 prefix to user-supplied paths to find which path the user is talking
 about.  So cd to toplevel before starting to carry the operation out
 is a natural pattern inside Git.  As many people already told you,
 the user has to run 'git submodule' from the top-level of the
 submodule working tree is a simple oversight of the implementation.

Yes, I am aware.  I'm piggy-banking on the mature parts of git-core to
get functionality that I would otherwise have to write by hand.  The
current implementation needs to hand-code this, and hasn't done it yet
(presumably because it's non-trivial).

 2. If we want to make git-submodule a part of git-core (which I think
 everyone agrees with), we will need to make the information in
 .gitmodules available more easily to the rest of git-core.

 Care to define more easily which is another subjective word?  The
 .gitmodules file uses the bog-standard configuration format that can
 be easily read with the config.c infrastructure.  It is a separate
 matter that git_config() API is cumbersome to use, but improving it
 would help not just .gitmodules but also the regular non-submodule
 users of Git.  There is a topic in the works to read data in that
 format from core Heiko is working on.

This goes both ways: the information is both easier to read and write.
 I can easily create a link object from anywhere: index_path() or
cmd_edit_link().  To do this, I just have to call write_sha1_file()
with the buffer filled out and with the parameter link_type (which is
already defined).  To access the data in a link, I have to fill out a
tree_desc with HEAD, an unpack_tree_opts with a custom callback, and
pass it to unpack_trees().  An example of a custom callback:

Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-08 Thread Junio C Hamano
Ramkumar Ramachandra artag...@gmail.com writes:

 Does git diff/ commit/ add/ rm or any other command you can think of
 rely on a special file in the worktree (aka .gitmodules) to be checked
 out?

Try git add foo~ with usual suspect in .gitignore ;-)
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-08 Thread Ramkumar Ramachandra
Junio C Hamano wrote:
 Ramkumar Ramachandra artag...@gmail.com writes:

 Does git diff/ commit/ add/ rm or any other command you can think of
 rely on a special file in the worktree (aka .gitmodules) to be checked
 out?

 Try git add foo~ with usual suspect in .gitignore ;-)

First, it's not a hard requirement: in the worst case, git add will
add the file even without a -f.  Second, I've already argued about how
I think this is the right design: What part of that do you disagree
with?  What alternate design do you propose?
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-08 Thread Junio C Hamano
Ramkumar Ramachandra artag...@gmail.com writes:

 Junio C Hamano wrote:
 Ramkumar Ramachandra artag...@gmail.com writes:

 Does git diff/ commit/ add/ rm or any other command you can think of
 rely on a special file in the worktree (aka .gitmodules) to be checked
 out?

 Try git add foo~ with usual suspect in .gitignore ;-)

 First, it's not a hard requirement: in the worst case, git add will
 add the file even without a -f.

In the same sense .gitmodules is not a hard requirement, either.  I
use a submodule without .gitmodules in one of my repositories (the
top-level houses the source to generete my dotfiles and is cloned to
my environment at work, but the submodule houses my private files
that live only at home).  The gitlink entry in the index and the
tree and presence of the .git repository in the submodule checkout
(where it exists) is sufficient to make the layout work.

If your complaints were I cannot make X work with the current
system, even with changes to git-submodule and some core part of the
system, and I think the reason is because the way module information
is stored is in a separate file .gitmodules, with a concrete X,
people who are more versed with the submodule subsystem may be able
to help you come up with a cleaner solution without throwing the
baby with the bathwater, but I do not think we saw any concrete X
mentioned.

The same sentence followed by ... and with an object of a new type
stored at the path of the submodule, I can make it work by doing A,
B and C, with concrete A, B and C, some people may be interested in
pursuing that avenue with you, but I do not think we saw such
combinations of X, A, B, C either.

If all of your argument starts from I think .gitmodules is ugly
because it is not an object of a separate type stored at the path of
the submodule, and here are the reasons why I think it is ugly, I
have nothing more to say to you.  That ugly is at best skewed
aesthetics, and each and every example that comes up in this
discussion, like this 'git add' works with .gitignore, and the one
I sent on .gitattributes vs .gitmodules on the default in the
nearby subthread to Jonathan, makes me realize that .gitmodules is
_more_ in line with the rest of the system, not less.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-08 Thread Jens Lehmann
Ok, here comes an updated version of our comparison list which
I updated with what I read in recent discussions. As I said
earlier, please speak up if I missed anything (or forgot to add
anyone to the CC).

I picked up one advantage (no need to cd-to-toplevel to edit
.gitmodules) two new disadvantages (foreach and default
submodule config) and retired one Ram showed a solution for
(the unstaged gitlink).


Advantages:

* Information is stored in one place, no need to lookup stuff in
  another file/blob.

* Easier coding, as we find all information in a single object.

* No need to cd-to-toplevel to change configuration in the
  .gitmodules file, the special tools to edit link information
  will work in any subdirectory.

(We currently need a checked out work tree to access the
.gitmodules file, but there is ongoing work to read the
configuration directly from the database)

(While it is easier to merge the link object, a .gitmodules
aware merge driver would work just as well)


Disadvantages:

* Changes in user visible behavior, possible compatibility
  problems when Git versions are mixed.

* Special tools are needed to edit submodule information where
  currently a plain editor is sufficient and a standard format
  is used.

* merge conflicts are harder to resolve and require special git
  commands, solving them in .gitmodules is way more intuitive
  as users are already used to conflict markers.

* git submodule foreach becomes harder to implement

* With .gitmodules we lose a central spot where configuration
  concerning many submodules can be stored

(I think when we also put the submodule name in the object we
could also retain the ability to repopulated moved submodules
from their old repo, which is found by that name)

(That a link object can have no unstaged counterpart that a file
easily has can be fixed by special casing this, e.g. in using a
file in .git/link-specs/)


Hmm, while it is still too early to close the polls, it looks
to me as most advantages are about easier coding while most
disadvantages hit the user. That makes it more understandable
for me why Ram is so convinced of his approach and why on the
other hand submodule users like myself are rather sceptical. I
think we need some more advantages that users will directly
profit from, the cd-to-toplevel for .gitmodules is definitely
not enough to support the change Ram is proposing. What other
advantages are missing here?
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-08 Thread Ramkumar Ramachandra
This is going nowhere.  You're stuck at making the current submodule
system work, not answering my questions, diverting conversation,
repeatedly asking the same stupid questions, labelling everything that
I say subjective, and refusing to look at the objective counterpart
(aka, the code).  It's clear to me that no matter how many more emails
I write, you're not going to concede.

I'm not interested in wasting any more of my time with this nonsense.

I give up.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-08 Thread Jeff King
On Mon, Apr 08, 2013 at 10:41:57PM +0200, Jens Lehmann wrote:

 (While it is easier to merge the link object, a .gitmodules
 aware merge driver would work just as well)

I have not been following this thread that closely, so apologies if I
missed it, but one thing I have not seen mention of is how the extra
information inside the gitlink object will require extra merge effort.

Imagine I have two branches; one updates the submodule's commit pointer,
and the other updates some meta-information about the submodule (e.g.,
it points the URL to a new host). In the current system, one change goes
into .gitmodules, and the other goes into the gitlink path. In a new
combined object, there is a conflict and we must do content-level
merging on it (which presumably would be done with a specialized merge
driver).

So I think that in some cases .gitmodules creates more conflicts
(submodule A and submodule B are touched and have a textual conflict),
and sometimes the combined object would create more objects (you touch
two parts of the of the combined object). The solution in both cases is
a smarter merge driver that understands which parts semantically
conflict and which parts do not.

-Peff
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-07 Thread Ramkumar Ramachandra
Junio C Hamano wrote:
 I think it is too premature to discuss _your_ code.  The patches do
 not even tell us anything about how much more work is needed to
 merely make Git with your patches work properly again.  For one
 thing, I suspect that you won't even be able to repack a repository
 that has OBJ_LINK only with the patches you posted.

Let me try to rephrase my original request: I'm an inexperienced
contributor trying to do something very ambitious.  Having authored a
huge part of it, Linus and you understand git-core much better than I
can ever hope to understand.  These are things that you need to tell
me after reading the patches.  I only have a rough idea to make Git
work properly with my patches again: I can't know for sure until I
write all the code.  What's more?  Your guess will probably be better
than mine after you read the code.

You're asking me to submit a perfect 40 or 50 part series that's a
potential candidate for merging.  I'm sorry to say this, but I'm
incapable of doing that without posting intermediate work and getting
help.  Frankly, it's a very unreasonable expectation; I don't think
anyone except you or Linus can even get close after making such a
fundamental change.

I might end up writing all the code (which I'm perfectly okay with),
and all I'm asking from you is to constantly keep picking my brain (by
reviewing my code and posting good critiques).  Am I being
unreasonable?

 At this point the only thing that we can gain from reading your
 patch is that you can write C to do _something_, but that something
 is so fuzzily explained that we do not know what to make of that
 knowledge that you write good (or bad, we don't know) C.

C is irrelevant here: I'm not asking you for style/ structuring tips;
I'm asking you for a critique on the implementation of this idea.
Linus raised some good points after reading [0/7] that I countered,
and there isn't anything else either anyone can raise without reading
more.  Code speaks more clearly than the English in [0/7]: you should
be able to deduce a lot more intent and direction.

 It would be much more productive to learn what these specific issues
 X, Y and Z are, and if the problems you are having with existing
 solutions are really fundamental that need changes to object layer
 to solve.

To reiterate: link does not make possible something that is not
fundamentally _possible_ with a .githack and a 100k-line Perl script.
At its core, every variant of submodules does this.  What I'm
essentially proposing: break up the information in .githack into
smaller bits and create a new object type so it can be parsed by
git-core easily.  If you agree that my proposal doesn't make
impossible what was possible earlier, and that it makes life easier
for everyone, we should be good to go.

When the series matures, we can investigate the other implementations
in greater detail so we can pick out more optional fields to add to
the link object before getting it merged.  This is not the right time
to do that: we're currently trying to get git-core working with the
mandatory fields.

 I do not think we have heard anything concrete and usable about what
 you are trying to achieve yet.

I'll try to rephrase your concerns here:

0. We don't know if this approach will yield a mergeable series at
all, because it breaks so many things and is so difficult to complete.

1. We don't know how much work is needed to bring the series to a
point where it is in a mergeable state.  There is no timeline
specified.

2. We can't build an exhaustive list of the problems that this new
approach will solve (ie. we haven't finalized the optional fields).

3. We don't have anything useable yet.

And your non-concerns should be:

1. We know that this approach won't fundamentally limit us in not
being able to solve some specific problems that the .githack approach
solves.

2. We know that this approach makes life easier for everyone, and
there are significant concrete benefits to teaching git-core about
links.

I agree with all this fully.  I don't have a concrete roadmap; we'll
just have to dive in based on what we've seen so far, and hope that
we're able to finish what we started.  So, my final question is: are
you still not convinced that this approach shows a lot of potential,
and is worth exploring now?
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-07 Thread Junio C Hamano
Ramkumar Ramachandra artag...@gmail.com writes:

 Junio C Hamano wrote:
 I think it is too premature to discuss _your_ code.  The patches do
 not even tell us anything about how much more work is needed to
 merely make Git with your patches work properly again.  For one
 thing, I suspect that you won't even be able to repack a repository
 that has OBJ_LINK only with the patches you posted.

 ...
 You're asking me to submit a perfect 40 or 50 part series...

Not at all.  And please do not start _coding_.

When the design is not clear enough that a 7-patch series is not
ready to be reviewed, certainly 50-patch series will not be.  Not
until you can explain what you are trying to solve and convince
others why other less disruptive approaches are fundamentally
unworkable, and why we need to change the object layer.

 To reiterate: link does not make possible something that is not
 fundamentally _possible_ with a .githack and a 100k-line Perl script.

 At its core, every variant of submodules does this.  What I'm
 essentially proposing: break up the information in .githack into
 smaller bits and create a new object type so it can be parsed by
 git-core easily.

The .gitmodules file is designed to be easily parsable by the config
infrastructure and implemented as such already, thank-you-very-much.

Why do you keep calling an already working solution with derogatory
misspelling?  That only gives others an impression that you do not
understand how the current system works, and pursuade them not to
waste time responding to you.  Stop it.

 When the series matures, we can investigate the other implementations
 in greater detail so we can pick out more optional fields to add to
 the link object before getting it merged.

Sorry, but that is not how open source works in general, and
certainly not how this project works.

We do not add disruptive change just for the sake of changing it to
break a working system, make an extra work to clean up the fallout
for ourselves (i.e. your 40 to 50 patch series, but honestly
speaking I expect it would be more like a 4 months work for a full
time engineer or two), for unproven design (that has not yet to be
illustrated) to solve problems (that has not yet to be explained),
without knowing that

 (1) the problems are worth solving;

 (2) the design will solve the problems; and

 (3) solving the same problems without such a disruptive change is
 impossible, or so cumbersome that it will be far larger than
 the work needed to clean-up the fallout of the disruptive
 change.

So what are your X, Y, Z?  You still haven't answered that question.

For that matter, you didn't answer the same question that was more
tersely phrased by Linus in the very first response in the thread:

 Linus Torvalds wrote:
 I don't dispute that a new link object might be a good idea, but
 there's no explanation of the actual format of this thing anywhere,
 and what the real advantages would be. A clearer this is the design,
 this is the format of the link object, and this is what it buys us
 would be a good idea.

 Yeah, I need help with that.  I've just stuffed in whatever fields
 popped into my mind first.  The current ones are:


And what you listed were your back-then-current thinking on actual
format.

What are the real advantages?  How are they used?  What do they
allow us to do what we cannot do with .gitmodules (or repo or
gitslave for that matter)?  What do they buy us?

What problem are you trying to solve?

I have this suspicion that you do not have to change anything in the
object layer to make Git behave very differently from the current
submodule implementation.  For example, if your gripe were (I am
just speculating without any input from you in this thread) that
each submodule working tree has .git at their top and there is no
unified view from the top-level [*1*], we certainly can solve it
without any change to the object layer.

We currently add a cache entry that has the commit object name to
the index from the tree object when we check out the superproject,
and create a separate repository with a working tree when we
instantiate a submodule.

This arrangement does not have to be fundamental. It is a design
choice of one particular working tree layout, which is totally local
to individual superproject working tree.

You could arrange a single index (the one in the superproject) to
hold the tree contents from the commit in the submodule, while
noting the original commit object name in a new mandatory extension
section in the index. The index will have a unified view of the
whole tree, and we do not have to have a .git at the root of each
submodule working tree (be it a directory or a gitfile).

I think the message where I talked about the bind idea in the list
archive URLs I gave you earlier would give you such a layout, and
you should go read it again to understand how the flow from object
database to index to working tree back to index back to object
database was envisioned to 

Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-07 Thread Ramkumar Ramachandra
Your overall hostility is unappreciated.  The burden of proof is on
me, while you calmly sit back and criticize anything that breaks the
current working state, and refuse to look at the implementation.
Anyway, here we go again.

Junio C Hamano wrote:
 Not at all.  And please do not start _coding_.

You've successfully killed all my enthusiasm.  Congratulations.

 When the design is not clear enough that a 7-patch series is not
 ready to be reviewed, certainly 50-patch series will not be.  Not
 until you can explain what you are trying to solve and convince
 others why other less disruptive approaches are fundamentally
 unworkable, and why we need to change the object layer.

So, my final question is: are you still not convinced that this
approach shows a lot of potential, and is worth exploring now?
No.

I don't know how many times to repeat this: No, Junio.  A less
disruptive approach is _not_ fundamentally unworkable.  You can spend
the next five years fixing submodule.c/ git-submodule.sh, or can take
a step back and think about why it's in such pathetic shape right now.

 To reiterate: link does not make possible something that is not
 fundamentally _possible_ with a .githack and a 100k-line Perl script.

 At its core, every variant of submodules does this.  What I'm
 essentially proposing: break up the information in .githack into
 smaller bits and create a new object type so it can be parsed by
 git-core easily.

 The .gitmodules file is designed to be easily parsable by the config
 infrastructure and implemented as such already, thank-you-very-much.

You're missing the point.  Who parses .gitmodules?  submodule.c and
git-submodule.sh, as opposed to a link being parsed by git-core.  How
is it any different?  That's what my series is trying to answer.

 Why do you keep calling an already working solution with derogatory
 misspelling?  That only gives others an impression that you do not
 understand how the current system works, and pursuade them not to
 waste time responding to you.  Stop it.

I don't see why you have to get offended by my deliberate misspelling:
we're not emotionally attached to software, and I'm merely criticizing
what I think is a bad hack.  I'm not pointing out the concrete
limitations of git-submodule precisely because they can be fixed
without any changes to the object layer: this thread will become a
discussion about how to fix submodule.c/ git-submodule.sh.  You want
floating submodules?  Fine, we'll write a helper script that
auto-commits to superproject everytime the SHA-1 changes.  Everything
_can_ be done.

What exactly don't I understand about the current system, apart from
the fact that everybody is super-rigid and defensive about what
already works?

Let us take a moment to look at the current state of git-submodule
(note that this is after many years of hard work).  This is just off
the top of my head:

1. To add a submodule, you can't git add.  You need to git submodule
add.  And only from the toplevel directory.  You can't first clone and
then add either: a git submodule add clones, adds lines to
.gitmodules, AND stages everything.

2. There is currently no way to remove a submodule.  You have to git
rm it, remove the lines in .gitmodules, and remove the GITDIR from
.git/modules.

3. It is currently impossible to git mv a submodule, because of the
amount of gymnastics required to relocate the object store, rewrite
the .gitmodules and stage the correct changes.

4. It is currently impossible to do true floating submodules, because
we're using a commit object in-tree.

5. You have to execute all submodule commands from the toplevel of the worktree.

6. It is currently impossible to initialize a nested submodule without
initializing the container submodule.  If I really want this, I have
to trade-off composability and use repo.

What is going on?  Either the people working on git-submodule are
horribly incompetent, or there's some fundamental problem.  I believe
the problem is the latter and have tried to show that the above quirks
can be fixed in a much simpler way with two days of work.  What part
of this didn't you understand?

 Sorry, but that is not how open source works in general, and
 certainly not how this project works.

 We do not add disruptive change just for the sake of changing it to
 break a working system, make an extra work to clean up the fallout
 for ourselves (i.e. your 40 to 50 patch series, but honestly
 speaking I expect it would be more like a 4 months work for a full
 time engineer or two), for unproven design (that has not yet to be
 illustrated) to solve problems (that has not yet to be explained),
 without knowing that

  (1) the problems are worth solving;

  (2) the design will solve the problems; and

  (3) solving the same problems without such a disruptive change is
  impossible, or so cumbersome that it will be far larger than
  the work needed to clean-up the fallout of the disruptive
  change.

 So what are your X, Y, Z?  

Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-07 Thread Ramkumar Ramachandra
I suspect you're overtly worried about the fallout of such a
disruptive change.  If so, you could've just said: Ram, I like the
idea.  But what breakages do you estimate we'll have to deal with?
instead of attacking the idea and repeatedly questioning its purpose.
So, I'll make a rough guess based on the first iteration I intend to
get merged:

- Not all the git submodule subcommands will work. add/ status/ init/
deinit are easy to rewrite, but stuff like --recursive and foreach
might be slightly problematic as I already pointed out earlier.  We'll
have to code depending on how far you think the first iteration should
go.  After a few iterations, we can make 'git submodule' just print
This command is deprecated.  Please read `man gitsubmodules`.

- All existing repositories with submodules will not be supported.  My
plan to deal with this: Have git-core code detect commit objects
in-tree and disable things like diff.  As soon as the user executes
the first 'git submodule' command, remove all existing submodules,
along with .gitmodules and re-add them as link objects.  Then print a
message saying: We've just migrated your submodules to the new
format.  Please commit this.

That's really it.  It's certainly not earth-shattering breakage; and I
think the inconvenience it causes is more than compensated by its
beautiful design and UI/UX.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-07 Thread Ramkumar Ramachandra
John Keeping wrote:
 Meaning that every repository using submodules need to have a flag day
 when all of the people using it switch to the new Git version at once?

No, I would be totally against a migration that involves a flag-day.
What I meant is that having old-style submodule side-by-side with
new-style submodules is confusing (think about people using an older
version and getting confused), and that we should disallow it.  Users
will still be able to use existing repositories with new versions of
git with a few caveats:
1. They won't be able to add new new submodules without migrating all
existing submodules.
2. git ls-tree will show the in-tree object incorrectly as a link (ie.
not commit).

That's about it, I think.  Obviously, everyone working on the
repository has to upgrade to a new version of git before they can use
new-style submodules.

 I think you need a much better argument than it makes the
 implementation more beautiful to convince users that a flag day is
 necessary.

There is no flag day necessary, and that is not my argument at all:
new-style submodules brings lots of new functionality to the table.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-07 Thread John Keeping
On Sun, Apr 07, 2013 at 10:12:28PM +0530, Ramkumar Ramachandra wrote:
 John Keeping wrote:
  Meaning that every repository using submodules need to have a flag day
  when all of the people using it switch to the new Git version at once?
 
 No, I would be totally against a migration that involves a flag-day.
 What I meant is that having old-style submodule side-by-side with
 new-style submodules is confusing (think about people using an older
 version and getting confused), and that we should disallow it.  Users
 will still be able to use existing repositories with new versions of
 git with a few caveats:
 1. They won't be able to add new new submodules without migrating all
 existing submodules.
 2. git ls-tree will show the in-tree object incorrectly as a link (ie.
 not commit).
 
 That's about it, I think.  Obviously, everyone working on the
 repository has to upgrade to a new version of git before they can use
 new-style submodules.

So not a flag day, but still some point at which the repository
transitions to will not work with Git older than version X.  And if
you need to add a new submodule then you cannot delay that transition
any longer.

  I think you need a much better argument than it makes the
  implementation more beautiful to convince users that a flag day is
  necessary.
 
 There is no flag day necessary, and that is not my argument at all:
 new-style submodules brings lots of new functionality to the table.

I haven't seen anywhere a concise list of what functionality this is.
Do you have a simple bulleted list of what new features this would
allow?
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-07 Thread Ramkumar Ramachandra
John Keeping wrote:
 So not a flag day, but still some point at which the repository
 transitions to will not work with Git older than version X.  And if
 you need to add a new submodule then you cannot delay that transition
 any longer.

Yes, that is true.  I don't see any way out of this.

 I haven't seen anywhere a concise list of what functionality this is.
 Do you have a simple bulleted list of what new features this would
 allow?

Sure, I'll write it out for you from an end-user perspective:

0. Great UI/UX.  No more cd-to-toplevel, and a beautiful set of native
commands that are consistent with the overall design of git-core.
Which means: clone (to put something in an unstaged place), add (to
stage), and commit (to commit the change).  There's now exactly one
place in your worktree (which is represented as one file in git; think
of it a sort of symlink)  to look in for all the information.  git
cat-link link to figure out its parameters, git edit-link to edit
its parameters: no more find the matching pwd in .gitmodules in
toplevel.  To remove a submodule, just git rm.  And git mv works!

1. True floating submodules.  You can have a submodule checked out at
`master` or `v3.1`: no more detached HEADs in submodules unless you
want fixed submodules.  No additional cruft required to do the
floating: the information is native, in a link object.

2. Initializing a nested submodule without having to initialize the
outer one: no more repo XML nonsense.  And it's composable: you don't
need to put the information about all submodules in one central place.

3. Ability to have very many large submodule repositories without the
performance hit.  It makes sense to block stat() from going through
when you have floating submodules.  This means that many levels of
nesting are very easily possible.

4. It's suddenly much easier to add new features to this
implementation.  You don't need to do the kind of gymnastics you'd
have to do if you were hacking on submodule.c/ git-submodule.sh.

This is basically how great design plays out.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-07 Thread John Keeping
On Sun, Apr 07, 2013 at 10:52:50PM +0530, Ramkumar Ramachandra wrote:
 Sure, I'll write it out for you from an end-user perspective:

To play Devil's Advocate for a bit...

 0. Great UI/UX.  No more cd-to-toplevel, and a beautiful set of native
 commands that are consistent with the overall design of git-core.
 Which means: clone (to put something in an unstaged place), add (to
 stage), and commit (to commit the change).  There's now exactly one
 place in your worktree (which is represented as one file in git; think
 of it a sort of symlink)  to look in for all the information.  git
 cat-link link to figure out its parameters, git edit-link to edit
 its parameters: no more find the matching pwd in .gitmodules in
 toplevel.  To remove a submodule, just git rm.  And git mv works!

Presumably now without .git/config support, so I can't override the
checked-in settings without my own custom branch.  Even carrying a dirty
working tree seems problematic here since a checked-out link object is a
directory, which can't have information like the remote URL in it.

 1. True floating submodules.  You can have a submodule checked out at
 `master` or `v3.1`: no more detached HEADs in submodules unless you
 want fixed submodules.  No additional cruft required to do the
 floating: the information is native, in a link object.

Can't I do that now with submodule.name.branch and git submodule
update --remote --rebase and friends?

 2. Initializing a nested submodule without having to initialize the
 outer one: no more repo XML nonsense.  And it's composable: you don't
 need to put the information about all submodules in one central place.

How does this interact when there is the following structure:

super
`-- sub
`-- subsub   (specified by sub)

and subsub is specified as a submodule in *both* super and sub but with
different settings.  Do I get different behaviour depending on $PWD?

 3. Ability to have very many large submodule repositories without the
 performance hit.  It makes sense to block stat() from going through
 when you have floating submodules.  This means that many levels of
 nesting are very easily possible.

Can't I already control this to some degree?  Certainly the following
commands take different amounts of time to run:

git status
git -c status.submodulesummary=true status

 4. It's suddenly much easier to add new features to this
 implementation.  You don't need to do the kind of gymnastics you'd
 have to do if you were hacking on submodule.c/ git-submodule.sh.
 
 This is basically how great design plays out.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-07 Thread Ramkumar Ramachandra
John Keeping wrote:
 With the clarifications Ram's provided in this thread, I think there are
 also some important regressions in functionality in his proposal (at
 least as it currently stands), particularly losing the .gitconfig
 overrides.

If we want the entire feature list in the very first iteration, it's
going to be huge.

 The only proposed change that seems to me to be impossible with the
 current .gitmodules approach is the submodule in a non-initialized
 submodule feature, but I've never seen anyone ask for that and it seems
 likely to open a whole can of worms where the behaviour is likely to
 vary with $PWD.  The current hierarchical approach provides sensible
 encapsulation of repositories and is simple to understand: once you're
 in a repository nothing above its root directory affects you.

That can be implemented in the current submodule system too, fwiw.

 It doesn't seem to me that it's harder than I'd like to add a feature I
 want is a good reason to subject all users of submodules to a lot of
 pain migrating to some new implementation that doesn't work the way
 they're used to and which will mean they have to deal with complaints
 when people using an older version of Git can't clone their repository
 (and I doubt we want this mailing list to be flooded with such
 complaints either).

Like I've said before: there is nothing that _cannot_ be done with the
current submodule system.  To see the real advantages of this new
submodule system, you have to think like a developer, not an end-user.
 Focusing just on end-user happiness is a very myopic way to develop
software, and I think the git community is better than that.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-07 Thread Ramkumar Ramachandra
This reminds me of the commit generation numbers thread.

But how can we determine ancestry?
Use the commit timestamp.
But what if there are clock skews?
Put in a slop.

It breaks existing stuff, and it's hard to show any end-user benefit.
I fear this proposal will meet with the same fate.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-07 Thread Jens Lehmann
Am 07.04.2013 20:44, schrieb Ramkumar Ramachandra:
 Jens Lehmann wrote:
 The whole feature list is full of red herrings like this which
 have nothing to do with the advantages of a new object, but talk
 about UI issues which are easy to solve in both worlds.
 
 Really?  git-submodule.sh was written in 2007, and does not have git
 mv or cd-to-toplevel restriction removed to date.  What does that say
 about git-submodule?

That there is still some work to do, which I never denied and am
actively working on (see git mv support in pu, which tackles one
of the UI issues you mentioned).

 I specifically said end-user's perspective.  Why exactly would I be
 talking about the advantages of the link object?

Because they are all that matters when it comes to decide if a link
object should be introduced to replace the current model. We should
discuss the differences in the UI that result from introducing such
an object, not the stuff that is still missing from our current
implementation (as that has to be coded either way and can not be
taken in favor of either solution). And we can additionally also
talk about the differences in hacking on git, where I concede that
putting everything into a single object could lead to shorter code
than having to consult a .gitmodules file for that (even though I
believe these arguments are much less important than UI changes).

Just to be sure: I think we agree that both approaches are capable
of allowing all relevant use cases, because they store the same
information?

Disclaimer: I am not opposed to the link object per se, but after
all we are talking about severely changing user visible behavior.
So I want to see striking evidence that we gain something from it,
discussed separately from UI deficiencies of the current code (no
cd-to-toplevel please ;-).

So I started putting together a list of advantages and one of
disadvantages of the new link object compared to the current model.
We can extend and refine that to see what your proposal would mean
for us. After all we are talking about severely changing user
visible behavior, so we need convincing reasons to do that.


Advantages:

* Information is stored in one place, no need to lookup stuff in
  another file/blob.

* Easier coding, as we find all information in a single object.

(I did not forget to add the point that you currently need a
checked out work tree to access the .gitmodules file, as there is
ongoing work to read the configuration directly from the database)

(Another advantage would be that it is easier to merge the link
object, but a - still to be coded - .gitmodules aware merge driver
would work just as well)


Disadvantages:

* Changes in user visible behavior, possible compatibility
  problems when Git versions are mixed.

* Special tools are needed to edit submodule information where
  currently a plain editor is sufficient.

* merge conflicts are harder to resolve and require special git
  commands, solving them in .gitmodules is way more intuitive
  as users are already used to conflict markers.

* A link object has no unstaged counterpart that a file easily
  has. What would that mean for adding a submodule and then
  unstaging it (or how could we add a submodule unstaged, like
  you proposed in another email)?

(I think when we also put the submodule name in the object we
could also retain the ability to repopulated moved submodules
from their old repo, which is found by that name)


I'm not saying that this list is complete, I just wrote down
what came to mind. When we e.g. find workable solutions to the
Disadvantages we can remove them from the list and append them
in parentheses for later reference like I did here. Does that
sound like a plan?
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-07 Thread Ramkumar Ramachandra
Jens Lehmann wrote:
 Just to be sure: I think we agree that both approaches are capable
 of allowing all relevant use cases, because they store the same
 information?

Yes.

 Disclaimer: I am not opposed to the link object per se, but after
 all we are talking about severely changing user visible behavior.
 So I want to see striking evidence that we gain something from it,
 discussed separately from UI deficiencies of the current code (no
 cd-to-toplevel please ;-).

The only mandatory user-visible behavior change is the absence of
.gitmodules.  The git submodule subcommand will be have to be present
and made to work, whether we like it or not.

 (I did not forget to add the point that you currently need a
 checked out work tree to access the .gitmodules file, as there is
 ongoing work to read the configuration directly from the database)

Read the configuration from the database?  How?
Also, I want refs quite badly: I really can't stand repo.

 (Another advantage would be that it is easier to merge the link
 object, but a - still to be coded - .gitmodules aware merge driver
 would work just as well)

It's very simple to implement: if you turn it into a blob, you can
diff and merge as usual.

 Disadvantages:

 * Changes in user visible behavior, possible compatibility
   problems when Git versions are mixed.

Agreed.

 * Special tools are needed to edit submodule information where
   currently a plain editor is sufficient.

Um, I actually really like this.  I don't want to cd-to-toplevel, open
up my .gitmodules and look for the relevant section.  And it's a very
simple tool: see the git cat-file that I posted earlier.

 * merge conflicts are harder to resolve and require special git
   commands, solving them in .gitmodules is way more intuitive
   as users are already used to conflict markers.

There shouldn't be that many merge conflicts to begin with!  It
happens because you've stuffed all the information into one gigantic
.gitmodules.  With links, life is *much* easier: you already have a
tight buffer format and a predefined order in which the key/value
pairs will appear.  But yes, we will require to grow git-core to merge
links seamlessly.

 * A link object has no unstaged counterpart that a file easily
   has. What would that mean for adding a submodule and then
   unstaging it (or how could we add a submodule unstaged, like
   you proposed in another email)?

Adding a submodule untracked (not unstaged) is possible, and is
default: git clone gets the submodules, and you have to use git add to
stage it.  I agree that you can't edit-link and have an unstaged
change, but I really don't care about that.

 (I think when we also put the submodule name in the object we
 could also retain the ability to repopulated moved submodules
 from their old repo, which is found by that name)

Hm, considering that the information is not present anywhere
(certainly not in the tree), this is probably a good idea.  We'd have
the history of the submodule's name too.

 I'm not saying that this list is complete, I just wrote down
 what came to mind. When we e.g. find workable solutions to the
 Disadvantages we can remove them from the list and append them
 in parentheses for later reference like I did here. Does that
 sound like a plan?

Yes, good plan.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-07 Thread Ramkumar Ramachandra
Jens Lehmann wrote:
 * Easier coding, as we find all information in a single object.

It's not just the difference between a single location versus multiple
locations.  It's about the core object code of git parsing links, as
opposed to a fringe submodule.c/ submodule.sh parsing .gitmodules.
When you push git-submodule.sh into core, you'll have to constantly
call functions to parse .gitmodules and get the information.  With
links, all that information is free, provided you've parsed the
object.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-07 Thread John Keeping
On Mon, Apr 08, 2013 at 02:19:10AM +0530, Ramkumar Ramachandra wrote:
 Jens Lehmann wrote:
  * A link object has no unstaged counterpart that a file easily
has. What would that mean for adding a submodule and then
unstaging it (or how could we add a submodule unstaged, like
you proposed in another email)?
 
 Adding a submodule untracked (not unstaged) is possible, and is
 default: git clone gets the submodules, and you have to use git add to
 stage it.  I agree that you can't edit-link and have an unstaged
 change, but I really don't care about that.

I do.  I quite often use git add -p to sort things out and submodules
currently fit into that seamlessly: I can add the submodule and then
wait until later to commit it, without needing to either clone and
remember to submodule add later or commit and play with rebase.

Losing the ability to do that is a major usability regression as far as
I'm concerned.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-07 Thread Jonathan Nieder
Ramkumar Ramachandra wrote:

 It's about the core object code of git parsing links, as
 opposed to a fringe submodule.c/ submodule.sh parsing .gitmodules.

What's stopping the core object code of git parsing .gitmodules?  What
is the core object code?  How does this compare to other metadata
files like .gitattributes and .gitignore?
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-06 Thread Ramkumar Ramachandra
Hi again,

So we've thought about it for some time, and I really need you to
start reviewing the code now.
I'll just summarize what we've discussed so far:

1. The malleability argument doesn't hold, because we're proposing a
link object with optional fields.

2. The local-fork argument doesn't hold, because users will be
rebasing changes to the link object in exactly the same way as they
currently do with the blob object .gitmodules.

3. The worktree argument doesn't hold, because we're proposing to
treat the link object as nothing more than a blob object that can be
parsed by git-core.  It will stage and unstage just like a blob.
Sure, it's not accessible directly by the filesystem: so what?  What
is the difference does `emacsclient .gitmodules` versus `git edit-link
clayoven` make to the end-user?

4. The diff-confusion argument is just another by-the-way, but it
doesn't really hold either.  Currently, we see:

- Subproject commit b83492
+ Subproject commit 39ab2f

(with diff.submodule set to log, we can actually see the log of the
submodule between these two commits.  With links, we will see:

- checkout_rev = b83492
+ checkout_rev = 39ab2f

There's nothing that prevents us from respecting diff.submodule (some
minor glue code will have to be written; that's all).

*. There is actually one thing that .gitmodules does better than
links.  foreach.  It's trivial to implement with .gitmodules and hard
to implement with links: with .gitmodules, the paths of all the
submodules are in one place.  But with links, we'll have to
unpack_trees() every tree in the entire repository, and dig through it
to find all the link objects to initialize.  Basically, inefficient
and inelegant.  However, I don't think this is a big problem in
practice, since this is not exactly a common operation: I'd probably
want to recurse-submodules once at clone time.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-06 Thread Junio C Hamano
Ramkumar Ramachandra artag...@gmail.com writes:

 So we've thought about it for some time, and I really need you to
 start reviewing the code now.

 I'll just summarize what we've discussed so far:
 ...

I do not think we have heard anything concrete and usable about what
you are trying to achieve yet.

You may be proposing to discard baby with bathwater.  We haven't
seen an evidence that the change is really worth having.  We do not
even know what you are trying to change, other than I want to add a
new object type to largely replicate what is recorded in .gitmodules
file.  What are you trying to solve?

I want to have a project for an appliance, that binds two
projects, the kernel and the appliance's userspace.  The
usual suspects to use to implement such a project would be
Git submodule, repo, or Gitslave.

I want to be able to do X and Y and Z in managing such a
project.

If I try to use submodule, I cannot see how I could make it
do X for _this thing_, and it is not a bug in the
implementation but is fundamental because of _this and
that_.  If I try to use repo, .. the same, and the same
for Gitslave. ..

I propose to add a new gitlink object recorded in the tree
and in the index, and the said cases X, Y and Z can be
solved in _such and such way_.  We cannot solve it without
having a new gitlink object recorded in the tree object
because of _this and that reason_.

I think it is too premature to discuss _your_ code.  The patches do
not even tell us anything about how much more work is needed to
merely make Git with your patches work properly again.  For one
thing, I suspect that you won't even be able to repack a repository
that has OBJ_LINK only with the patches you posted.

At this point the only thing that we can gain from reading your
patch is that you can write C to do _something_, but that something
is so fuzzily explained that we do not know what to make of that
knowledge that you write good (or bad, we don't know) C.

It would be much more productive to learn what these specific issues
X, Y and Z are, and if the problems you are having with existing
solutions are really fundamental that need changes to object layer
to solve.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-05 Thread Ramkumar Ramachandra
Ramkumar Ramachandra wrote:
 After some discussion, I hope to be able to finalize a list of fields
 that will suffice for (nearly) everything.

The task is actually much easier than this.  All we have to do is
finalize the list of fields that will mandatorily be written to the
link object.  As I might have indicated in my series, this is:
upstream_url, checkout_rev, and ref_name.  Really, the user only needs
to supply a valid upstream_url: after a clone, everything else can be
inferred (with the exception of a ref_name conflict; I don't like
auto-mangling).

Other fields are like .git/config fields.  We can add new key/value
pairs in the future, without worrying about migration.  A problem
arises only if we want to add a new mandatory field, change the
default value of a key, or deprecate an existing key/value pair.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-05 Thread Linus Torvalds
On Thu, Apr 4, 2013 at 1:04 PM, Ramkumar Ramachandra artag...@gmail.com wrote:
 Linus Torvalds wrote:
 Or you could also just edit and carry a dirty .gitmodules around for
 your personal use-case.

 I'm sorry, but a dirty worktree is unnecessarily painful to work with.

Bzzt. Wrong.

A dirty worktree is not only easy to work with (I do it all the time,
having random test-patches in my tree that I never even intend to
commit), it's a *requirement*.

One thing that git does really really well is merging. And one of the
reasons why git does merging well (apart from the obvious meta-issue:
it's what I care about) is that it not only has the stable information
in the object database, it also has the staging information in the
index, *and* it has dirty data in the working tree.

You absolutely need all three. Having an edit command to edit stable
data (or staging data) is broken. Trust me, I've been there, done
that, got the T-shirt and know it is wrong. The whole stable objects
+ index + dirty worktree is FUNDAMENTALLY the right way to work,
and it *has* to work that way for merges to work well.

The only things that we don't have dirty data for in the worktree is
creating commits and tags, but those aren't relevant for the merging
process anyway, in the sense that you never change them for merging,
you create them *after* merging (and this is fundamental, and not just
a git implementation issue).

So you absolutely need a dirty worktree. You need it for testing, and
you need it for merging. Having a model where you don't have a
in-progress entity that works as a temporary is absolutely and
entirely wrong.

   Linus
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-05 Thread Ramkumar Ramachandra
Linus Torvalds wrote:
 So you absolutely need a dirty worktree. You need it for testing, and
 you need it for merging. Having a model where you don't have a
 in-progress entity that works as a temporary is absolutely and
 entirely wrong.

I agree entirely.  My comment was just a by the way, and specific to
how people work with .gitmodules: I didn't imply any strong notions of
Right or Wrong with respect to dirty worktrees in general.  So, yes:
links stage and unstage, just like blobs do.

Oh, and I'm currently writing infrastructure to work with links like
blobs.  Here's a WIP: git cat-link link is exactly the same as cat
file, to the end user.

-- 8 --
From d8a1de6f9075771dde6f1fde9ffa193dce386a17 Mon Sep 17 00:00:00 2001
From: Ramkumar Ramachandra artag...@gmail.com
Date: Fri, 5 Apr 2013 19:42:56 +0530
Subject: [PATCH] builtin/cat-link: implement new builtin

This is a simple program that calls unpack_trees() with a custom
callback that just prints the contents of whatever objects were
matched using revs.prune_data.  Blobs can be cat'ed directly from the
filesystem, so this program is primarily useful for links; git
cat-link link shows it up like a blob.

We will use this program to build edit-link.

Signed-off-by: Ramkumar Ramachandra artag...@gmail.com
---
 Makefile   |  3 +-
 builtin.h  |  1 +
 builtin/cat-link.c | 83 ++
 diff-lib.c | 10 +++
 diff.h |  6 
 git.c  |  1 +
 6 files changed, 98 insertions(+), 6 deletions(-)
 create mode 100644 builtin/cat-link.c

diff --git a/Makefile b/Makefile
index cd4b6f9..28194d7 100644
--- a/Makefile
+++ b/Makefile
@@ -349,7 +349,7 @@ GIT-VERSION-FILE: FORCE
 
 # CFLAGS and LDFLAGS are for the users to override from the command line.
 
-CFLAGS = -g -O2 -Wall
+CFLAGS = -g -O0 -Wall
 LDFLAGS =
 ALL_CFLAGS = $(CPPFLAGS) $(CFLAGS)
 ALL_LDFLAGS = $(LDFLAGS)
@@ -893,6 +893,7 @@ BUILTIN_OBJS += builtin/blame.o
 BUILTIN_OBJS += builtin/branch.o
 BUILTIN_OBJS += builtin/bundle.o
 BUILTIN_OBJS += builtin/cat-file.o
+BUILTIN_OBJS += builtin/cat-link.o
 BUILTIN_OBJS += builtin/check-attr.o
 BUILTIN_OBJS += builtin/check-ignore.o
 BUILTIN_OBJS += builtin/check-ref-format.o
diff --git a/builtin.h b/builtin.h
index faef559..be0160d 100644
--- a/builtin.h
+++ b/builtin.h
@@ -49,6 +49,7 @@ extern int cmd_blame(int argc, const char **argv, const char 
*prefix);
 extern int cmd_branch(int argc, const char **argv, const char *prefix);
 extern int cmd_bundle(int argc, const char **argv, const char *prefix);
 extern int cmd_cat_file(int argc, const char **argv, const char *prefix);
+extern int cmd_cat_link(int argc, const char **argv, const char *prefix);
 extern int cmd_checkout(int argc, const char **argv, const char *prefix);
 extern int cmd_checkout_index(int argc, const char **argv, const char *prefix);
 extern int cmd_check_attr(int argc, const char **argv, const char *prefix);
diff --git a/builtin/cat-link.c b/builtin/cat-link.c
new file mode 100644
index 000..14dd92b
--- /dev/null
+++ b/builtin/cat-link.c
@@ -0,0 +1,83 @@
+/*
+ * Copyright (c) 2013 Ramkumar Ramachandra
+ */
+#include cache.h
+#include tree.h
+#include cache-tree.h
+#include unpack-trees.h
+#include commit.h
+#include diff.h
+#include revision.h
+
+static int cat_file(struct cache_entry **src, struct unpack_trees_options *o) {
+   int cached, match_missing = 1;
+   unsigned dirty_submodule = 0;
+   unsigned int mode;
+   const unsigned char *sha1;
+   struct cache_entry *idx = src[0];
+   struct cache_entry *tree = src[1];
+   struct rev_info *revs = o-unpack_data;
+   enum object_type type;
+   unsigned long size;
+   char *buf;
+
+   cached = o-index_only;
+   if (ce_path_match(idx ? idx : tree, revs-prune_data)) {
+   if (get_stat_data(idx, sha1, mode, cached, match_missing,
+   dirty_submodule, NULL)  0)
+   die(Something went wrong!);
+   buf = read_sha1_file(sha1, type, size);
+   printf(%s, buf);
+   }
+   return 0;
+}
+
+int cmd_cat_link(int argc, const char **argv, const char *prefix)
+{
+   struct unpack_trees_options opts;
+   int cached = 1;
+   struct rev_info revs;
+   struct tree *tree;
+   struct tree_desc tree_desc;
+   struct object_array_entry *ent;
+
+   if (argc  2)
+   die(Usage: git cat-link link);
+
+   init_revisions(revs, prefix);
+   setup_revisions(argc, argv, revs, NULL); /* For revs.prune_data */
+   add_head_to_pending(revs);
+
+   /* Hack to diff against index; we create a dummy tree for the
+  index information */
+   if (!revs.pending.nr) {
+   struct tree *tree;
+   tree = lookup_tree(EMPTY_TREE_SHA1_BIN);
+   add_pending_object(revs, tree-object, HEAD);
+   }
+
+   if (read_cache()  0)
+   

Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-05 Thread Junio C Hamano
Ramkumar Ramachandra artag...@gmail.com writes:

 Junio C Hamano wrote:
 git log -p .gitmodules would be a way to review what changed in
 the information about submodules.  Don't you need git log-link for
 exactly the same reason why you need git diff-link in the first
 place?

 So you may not have suggested it, but I suspect that was only
 because you haven't had enough time to think things through.

 What is this git log -p .gitmodules doing?  It's walking down the
 commit history, and picking out the commits in which that blob
 changed.  Then it's diffing the blobs in those commits with each
 other.  Why is git log -p link any different?  We already know how
 to diff blobs, and we just need a way to diff links.

You already forget what you invented git diff-link as a solution
for, perhaps?

By recording the submodules themselve and information _about_ the
submodules separately (the latter is in .gitmodules), git diff A
can show the difference in submodule A, while git diff .gitmodules
can show a change, which is a possibly in-working-tree-only proposed
change, in information about submoudules.

Once you start recording the latter also at path A, it becomes
unclear what git diff A should show.

That is what I said in the message, to which you invented diff-link
as a solution to the unclear-ness.

Am I misremembering the flow of discussion in this thread?
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-05 Thread Ramkumar Ramachandra
Junio C Hamano wrote:
 Once you start recording the latter also at path A, it becomes
 unclear what git diff A should show.

 That is what I said in the message, to which you invented diff-link
 as a solution to the unclear-ness.

I just thought it would be a stopgap until we get diff to support
links natively.  Obviously, when we get native diff support, 'log -p'
will be able to show differences as well.  As it turns out from my
little experiment with cat-link, it's really easy to get native diff
support, and I'm targeting that directly instead of a scripted
solution.

As for the unclearness issue, it's a little more complicated than
that: a non-floating submodule could've previously been a floating
one, or vice-versa.  As of this moment, I'm only planning to show
differences between link buffers.  In the case when two consecutive
commits change a link checkout_rev (where floating is set to 0), we
can come up with something like the current diff.submodule = log.  I
see no cause to worry about the interface of that now.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-04 Thread Linus Torvalds
On Thu, Apr 4, 2013 at 11:30 AM, Ramkumar Ramachandra
artag...@gmail.com wrote:

 The purpose of this series is to convince you that we've made a lot of
 fundamental mistakes while designing submodules, and that we should
 fix them now.  [1/7] argues for a new object type, and this is the
 core of the idea.

I don't dispute that a new link object might be a good idea, but
there's no explanation of the actual format of this thing anywhere,
and what the real advantages would be. A clearer this is the design,
this is the format of the link object, and this is what it buys us
would be a good idea. Also, one of the arguments against using link
objects originally was that the format wasn't stable, and in
particular the address of the actual submodule repository might differ
for different people. So when adding a new object type, explaining
*why* the format of such an object is globally stable (since it will
be part of the SHA1 of the object) is a big deal.

Linus
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-04 Thread Jonathan Nieder
Ramkumar Ramachandra wrote:

 The purpose of this series is to convince you that we've made a lot of
 fundamental mistakes while designing submodules, and that we should
 fix them now.  [1/7] argues for a new object type, and this is the
 core of the idea.

Oh, dear.

Shouldn't it be possible to explain the same thing using a test
script illustrating intended UI?

[...]
 $ git clone gh:artagnon/varlog
 $ cd varlog
 $ git clone gh:artagnon/clayoven
 # Notice how it puts clayoven.git in ~/bare

I really would like to be able to continue doing something like

git clone --recurse-submodules git://repo.or.cz/cgit.git
# never mind!
rm -fr cgit

without leaving any clutter behind.  I have used systems that kept
state in my home directory before and found them a pain in the neck to
debug.  Others may disagree, though.

[...]
 # Again, just works!  No cd-to-toplevel nonsense

Didn't Jens mention that git-submodule requiring that one work
at the toplevel is just a (presumably easily fixable) bug?

[...]
 If you think this is all a big waste of time, and that we should focus
 on improving git-submodule.sh, you're probably deranged.  Because it's

I don't think that *you* should focus on improving git-submodule, as
long as you are not using it and dislike its design.  But I do think
it's strange to at the same time

 1) tell me I'm deranged for liking submodules
 2) dismiss other experiments that have been created as alternatives

I like experimentation, which means sometimes having tools whose
purposes overlap, and I like when it's possible to help something
evolve to be better, even far enough to interoperate with or replace
uses of another tool.

I also believe in live and let live.  That means that even if
someone is a little crazy, if they are not actively harmful, I do not
destroy their tools.

That probably marks me as deranged.

Hope that helps,
Jonathan
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-04 Thread Ramkumar Ramachandra
Linus Torvalds wrote:
 I don't dispute that a new link object might be a good idea, but
 there's no explanation of the actual format of this thing anywhere,
 and what the real advantages would be. A clearer this is the design,
 this is the format of the link object, and this is what it buys us
 would be a good idea.

Yeah, I need help with that.  I've just stuffed in whatever fields
popped into my mind first.  The current ones are:

1. upstream_url: this records the upstream URL.  No need to keep a .gitmodules.

2. checkout_rev: this records the ref to check out the submodule to.
As opposed to a concrete SHA-1, this allows for more flexibility; you
can put refs/heads/master and have truly floating submodules.

3. ref_name: this specifies what name the ref under
refs/modules/branch/ should use.

4. floating: this bit specifies whether to record a concrete SHA-1 in
checkout_rev.

5. statthrough: this bit specifies whether git should stat() through
the worktree.  We can turn it off on big repositories for performance
reasons.

 Also, one of the arguments against using link
 objects originally was that the format wasn't stable, and in
 particular the address of the actual submodule repository might differ
 for different people. So when adding a new object type, explaining
 *why* the format of such an object is globally stable (since it will
 be part of the SHA1 of the object) is a big deal.

After some discussion, I hope to be able to finalize a list of fields
that will suffice for (nearly) everything.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-04 Thread Jonathan Nieder
Ramkumar Ramachandra wrote:

 1. 'git add' should not go past submodule boundaries.  I should not be
able to 'git add clayoven/' or 'git add clayoven/LICENSE'.  In
addition, the shell completion also needs to be fixed.

Yep.  This is a bug.

 2. An empty directory containing a .git file is a perfectly valid
worktree, but does not show up in the superproject's 'git status'
output.  How can it be treated like an empty directory?

Stated like that, it doesn't sound like a bug.  Git since very early
has deliberately not tracked files or directories named .git.

Do you need this as a way of importing from a foreign VCS when someone
has accidentally checked in a .git directory along with everything
else?

Thanks,
Jonathan
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-04 Thread Jonathan Nieder
Jonathan Nieder wrote:
 Ramkumar Ramachandra wrote:

 The purpose of this series is to convince you that we've made a lot of
 fundamental mistakes while designing submodules, and
[...]
 Shouldn't it be possible to explain the same thing using a test
 script illustrating intended UI?

Sorry, I sent this reply too quickly.  Your explanation to Linus
clarified the idea.

Regards,
Jonathan
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-04 Thread Linus Torvalds
On Thu, Apr 4, 2013 at 11:52 AM, Ramkumar Ramachandra
artag...@gmail.com wrote:

 1. upstream_url: this records the upstream URL.  No need to keep a 
 .gitmodules.

 2. checkout_rev: this records the ref to check out the submodule to.
 As opposed to a concrete SHA-1, this allows for more flexibility; you
 can put refs/heads/master and have truly floating submodules.

 3. ref_name: this specifies what name the ref under
 refs/modules/branch/ should use.

 4. floating: this bit specifies whether to record a concrete SHA-1 in
 checkout_rev.

 5. statthrough: this bit specifies whether git should stat() through
 the worktree.  We can turn it off on big repositories for performance
 reasons.

So the thing is (and this was pretty much the original basis for
.gitmodules) that pretty  much *all* of the above fields are quite
possibly site-specific, rather than globally stable.

So I actually conceptually like (and liked) the notion of a link
object, but I just don't think it is necessarily practically useful,
exactly because different installations of the *same* supermodule
might well want to have different setups wrt these submodule fields.

My gut feel is that yes, .gitmodules was always a bit of a hack, but
it's a *working* hack, and it does have advantages exactly because
it's more fluid than an actual git object (which by definition has to
be set 100% in stone). If there are things you feel it does wrong
(like the git add bug that is being discussed elsewhere), I wonder
if it's not best to at least try to fix/extend them in the current
model. The features you seem to be after (ie that whole
floating/refname thing) don't seem fundamentally antithetical to the
current model (a commit SHA1 of all zeroes for floating, with a new
refname field in .submodules? I dunno)..

  Linus
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-04 Thread Junio C Hamano
Linus Torvalds torva...@linux-foundation.org writes:

 ... The features you seem to be after (ie that whole
 floating/refname thing) don't seem fundamentally antithetical to the
 current model (a commit SHA1 of all zeroes for floating, with a new
 refname field in .submodules? I dunno)..

Just on this part.

I think Heiko and Jens's (by the way, why aren't they on the Cc:
list when this topic is clearly discussing submodules?  Don't we
want to learn how the current submodule subsystem is used to solve
what real-world problems?) .gitmodules updates is exactly going in
that direction.

 - A submodule can be marked as floating in .gitmodules and be
   specified how (typially, use the tip of this branch in the
   submodule);

 - Running submodule update a floating submodule does not detach
   the submodule working tree to commit in the index of the
   superproject; instead it will use the specified branch tip;

 - A floating submodule records a concrete commit object name in the
   index of the superproject (no need to stuff an unusual SHA-1
   there to signal that the submodule is floating---it is recorded
   in the .gitmodules).  Thanks to this, a release out of the
   top-level can still describe the state of the entire tree;

 - It would be normal for the commit recorded in the index of the
   superproject not to match what is checked out in the submodule
   working tree (i.e. the tip of the branch in the submodule may
   have advanced).  A traditional non-floating submodule has many
   mechanisms to be noisy about this situation to prevent users from
   making an incomplete commits, but they may have to be toned down
   or squelched for floating submodules.

Anything I missed, Jens, Heiko?


--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-04 Thread Ramkumar Ramachandra
Linus Torvalds wrote:
 So the thing is (and this was pretty much the original basis for
 .gitmodules) that pretty  much *all* of the above fields are quite
 possibly site-specific, rather than globally stable.

 So I actually conceptually like (and liked) the notion of a link
 object, but I just don't think it is necessarily practically useful,
 exactly because different installations of the *same* supermodule
 might well want to have different setups wrt these submodule fields.

 My gut feel is that yes, .gitmodules was always a bit of a hack, but
 it's a *working* hack, and it does have advantages exactly because
 it's more fluid than an actual git object (which by definition has to
 be set 100% in stone). If there are things you feel it does wrong
 (like the git add bug that is being discussed elsewhere), I wonder
 if it's not best to at least try to fix/extend them in the current
 model. The features you seem to be after (ie that whole
 floating/refname thing) don't seem fundamentally antithetical to the
 current model (a commit SHA1 of all zeroes for floating, with a new
 refname field in .submodules? I dunno)..

Let's compare the two alternatives: .gitmodules versus link object.
If I want my fork of .gitmodules, I create a commit on top.  If I want
my fork of the link object, I create a link object, plus tree object,
plus commit object on top of that.  But the commit still rebases fine.

On malleability, have you looked at [5/7], where I create edit-link
(dead code; half done)?  The buffer looks just like a .gitmodules
buffer.  Fundamentally, what is the difference between this and a
blob?  git-core can parse it into structured data that it can slurp
easily.

I don't want full float or nothing.  I want in-betweens too, and refs are great.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-04 Thread Ramkumar Ramachandra
Linus Torvalds wrote:
  don't seem fundamentally antithetical to the
 current model

I don't think it's fundamentally antithetical either.  This basically
makes the life of git-submodule much simpler, and eventually obsolete
it away completely.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-04 Thread Linus Torvalds
On Thu, Apr 4, 2013 at 12:36 PM, Ramkumar Ramachandra
artag...@gmail.com wrote:

 Let's compare the two alternatives: .gitmodules versus link object.
 If I want my fork of .gitmodules, I create a commit on top.

Or you could also just edit and carry a dirty .gitmodules around for
your personal use-case.

I don't know if anybody does that, but it should work fine.

And I don't see what you can do with the link objects that you cannot
do with .gitmodules. That's what it really boils down to. .gitmodules
do actually work. Your extensions would work with them too.

   Linus
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-04 Thread Ramkumar Ramachandra
Linus Torvalds wrote:
 On Thu, Apr 4, 2013 at 12:36 PM, Ramkumar Ramachandra
 artag...@gmail.com wrote:

 Let's compare the two alternatives: .gitmodules versus link object.
 If I want my fork of .gitmodules, I create a commit on top.

 Or you could also just edit and carry a dirty .gitmodules around for
 your personal use-case.

Just take the link's buffer with you everywhere.  All you have to do
is git edit-link name and paste the file's contents there, instead
of opening .gitmodules directly in your editor.

 And I don't see what you can do with the link objects that you cannot
 do with .gitmodules. That's what it really boils down to. .gitmodules
 do actually work. Your extensions would work with them too.

If it came to that, you could write a huge Perl script to solve
everything with a .githack.  It breaks the internal symmetry of the
repository, which is why git-submodule is having such a field day.
I'm trying to prove, in my series, that making fundamental changes
lets us get rid of a huge amount of complexity.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-04 Thread Ramkumar Ramachandra
Junio C Hamano wrote:
 I think Heiko and Jens's (by the way, why aren't they on the Cc:
 list when this topic is clearly discussing submodules?  Don't we
 want to learn how the current submodule subsystem is used to solve
 what real-world problems?) .gitmodules updates is exactly going in
 that direction.

Because it's pointless.  We're not discussing a git-submodule
alternative.  We're discussing how to fix git-core so that
git-submodule becomes much simpler; to the extent that it will be
unnecessary soon.

git-submodule is years of hard work and it can do a limited version of
floating with great difficulty.  Mine is two days of work, and can
already do true floating submodules.  What is going on?
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-04 Thread Ramkumar Ramachandra
Linus Torvalds wrote:
 Or you could also just edit and carry a dirty .gitmodules around for
 your personal use-case.

I'm sorry, but a dirty worktree is unnecessarily painful to work with.
 I don't think anyone objects to committing, if they can understand
basic rebase.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-04 Thread Ramkumar Ramachandra
Ramkumar Ramachandra wrote:
 Just take the link's buffer with you everywhere.  All you have to do
 is git edit-link name and paste the file's contents there, instead
 of opening .gitmodules directly in your editor.

On this.  The buffer doesn't have to conform to a tight spec: we can
just expose a .gitconfig-like buffer and reduce it to a tight spec
before writing out.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-04 Thread Jens Lehmann
Am 04.04.2013 21:17, schrieb Junio C Hamano:
 Linus Torvalds torva...@linux-foundation.org writes:
 
 ... The features you seem to be after (ie that whole
 floating/refname thing) don't seem fundamentally antithetical to the
 current model (a commit SHA1 of all zeroes for floating, with a new
 refname field in .submodules? I dunno)..
 
 Just on this part.
 
 I think Heiko and Jens's (by the way, why aren't they on the Cc:
 list when this topic is clearly discussing submodules?  Don't we
 want to learn how the current submodule subsystem is used to solve
 what real-world problems?) .gitmodules updates is exactly going in
 that direction.
 
  - A submodule can be marked as floating in .gitmodules and be
specified how (typially, use the tip of this branch in the
submodule);
 
  - Running submodule update a floating submodule does not detach
the submodule working tree to commit in the index of the
superproject; instead it will use the specified branch tip;
 
  - A floating submodule records a concrete commit object name in the
index of the superproject (no need to stuff an unusual SHA-1
there to signal that the submodule is floating---it is recorded
in the .gitmodules).  Thanks to this, a release out of the
top-level can still describe the state of the entire tree;
 
  - It would be normal for the commit recorded in the index of the
superproject not to match what is checked out in the submodule
working tree (i.e. the tip of the branch in the submodule may
have advanced).  A traditional non-floating submodule has many
mechanisms to be noisy about this situation to prevent users from
making an incomplete commits, but they may have to be toned down
or squelched for floating submodules.
 
 Anything I missed, Jens, Heiko?

Nope, that perfectly sums it up.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-04 Thread Jens Lehmann
Am 04.04.2013 21:04, schrieb Linus Torvalds:
 My gut feel is that yes, .gitmodules was always a bit of a hack, but
 it's a *working* hack, and it does have advantages exactly because
 it's more fluid than an actual git object (which by definition has to
 be set 100% in stone).

Exactly. The flexibility of the .gitmodules file will really help us
when it comes to the next feature that submodules are going to learn
after recursive update: automatically initialize and then populate
certain submodules during the clone of the superproject. You have to
be able to configure that per submodule, which needs a new config
option in .gitmodules. Others may follow for different use cases.

While starting to grok submodules I was wondering myself if the data
stored in .gitmodules would better be stored in an extended gitlink
object, but I learned soon that the scope of the data that has to be
stored there was not clear at that time (and still isn't). So I'm
not opposed per se to adding a special object containing all that
information, but I strongly believe we are not even close to
considering such a step (and won't be for quite some time and maybe
never will).
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-04 Thread Ramkumar Ramachandra
Jens Lehmann wrote:
 Exactly. The flexibility of the .gitmodules file will really help us
 when it comes to the next feature that submodules are going to learn
 after recursive update:

That's like saying that the flexibility of a blob is invaluable: let's
throw out all the other objects, and make do with blobs.  Ofcourse we
make mistakes: we didn't put a generation number in the commit object,
for instance (I'm not arguing about whether it's right or wrong: just
that some people think it's a mistake).

 While starting to grok submodules I was wondering myself if the data
 stored in .gitmodules would better be stored in an extended gitlink
 object, but I learned soon that the scope of the data that has to be
 stored there was not clear at that time (and still isn't). So I'm
 not opposed per se to adding a special object containing all that
 information, but I strongly believe we are not even close to
 considering such a step (and won't be for quite some time and maybe
 never will).

Nonsense.  We will think through it before freezing the format, like
we did with the other objects.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-04 Thread Junio C Hamano
Jens Lehmann jens.lehm...@web.de writes:

 While starting to grok submodules I was wondering myself if the data
 stored in .gitmodules would better be stored in an extended gitlink
 object, but I learned soon that the scope of the data that has to be
 stored there was not clear at that time (and still isn't). So I'm
 not opposed per se to adding a special object containing all that
 information, but I strongly believe we are not even close to
 considering such a step (and won't be for quite some time and maybe
 never will).

I actually think the storage is more or less an orthogonal issue.

The format must be defined to be extensible (nobody is perfect and
if you wait for an exhaustive list of attributes that cover all use
cases including the ones that haven't even been invented yet, you
will get nowhere), and designed carefully to reduce the chance of
allowing the extended/optional bit to express the same thing in two
different ways to make sure the object name will not become
unnecessarily unstable, but you can start small, keep adding
optional fields, and be prepared to design an upgrade path when you
need to add new mandatory fields---that cannot be helped whether you
record the information about submodules in .gitmodules or a new
blob-ish object at the location where the submodule tree should
reside in the index and the tree.

However, the current .gitmodules design, even though it originally
was invented as a way to carry information other than what a single
commit object name from an otherwise unrelated project can express
without having to change anything in-core, has a few practical
merits.  The information _about_ submodules is stored separately
(i.e. in the .gitmodules file) from submodules themselves, and it
may be a good thing.

When you are changing information _about_ submodules (e.g. you may
be updating the recommended URL to fetch it from), you can use the
usual tools like git diff to see how it changed, just like changes
to any other file.  If the information _about_ a submodule A is
stored at path A, and at the same time you have a working tree that
corresponds to the root of the submodule A at that path, it gets
unclear what git diff A should report.  Should it report the
change in the submodule itself, or should it report the change in
the information _about_ the submodule?  By separating these two
concepts to two different places, .gitmodules design solves the
issue nicely.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-04 Thread Ramkumar Ramachandra
Junio C Hamano wrote:
 When you are changing information _about_ submodules (e.g. you may
 be updating the recommended URL to fetch it from), you can use the
 usual tools like git diff to see how it changed, just like changes
 to any other file.  If the information _about_ a submodule A is
 stored at path A, and at the same time you have a working tree that
 corresponds to the root of the submodule A at that path, it gets
 unclear what git diff A should report.  Should it report the
 change in the submodule itself, or should it report the change in
 the information _about_ the submodule?  By separating these two
 concepts to two different places, .gitmodules design solves the
 issue nicely.

git diff-link.  Just turn it into a buffer and diff as usual.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-04 Thread Junio C Hamano
Ramkumar Ramachandra artag...@gmail.com writes:

 Junio C Hamano wrote:
 When you are changing information _about_ submodules (e.g. you may
 be updating the recommended URL to fetch it from), you can use the
 usual tools like git diff to see how it changed, just like changes
 to any other file.  If the information _about_ a submodule A is
 stored at path A, and at the same time you have a working tree that
 corresponds to the root of the submodule A at that path, it gets
 unclear what git diff A should report.  Should it report the
 change in the submodule itself, or should it report the change in
 the information _about_ the submodule?  By separating these two
 concepts to two different places, .gitmodules design solves the
 issue nicely.

 git diff-link.  Just turn it into a buffer and diff as usual.

Sounds like you are saying that you can pile a new command on top of
new command to solve what the existing tools people are familar with
can already solve in a consistent way without adding anything new.
Are you going to dupliate various options to git diff and git
log in git diff-link?  Will you then next need git log-link?
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-04 Thread Ramkumar Ramachandra
Junio C Hamano wrote:
 Sounds like you are saying that you can pile a new command on top of
 new command to solve what the existing tools people are familar with
 can already solve in a consistent way without adding anything new.
 Are you going to dupliate various options to git diff and git
 log in git diff-link?  Will you then next need git log-link?

What I'm saying is: As always, we start with plumbing and work our way
up to porcelain.  We do have git diff-files, diff-index, diff-tree, so
I don't see what the problem with diff-link is.  The point is that we
can get an initial scripted version out quickly.

And no, I never suggested a git log-link.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-04 Thread Junio C Hamano
Ramkumar Ramachandra artag...@gmail.com writes:

 Junio C Hamano wrote:
 Sounds like you are saying that you can pile a new command on top of
 new command to solve what the existing tools people are familar with
 can already solve in a consistent way without adding anything new.
 Are you going to dupliate various options to git diff and git
 log in git diff-link?  Will you then next need git log-link?

 What I'm saying is: As always, we start with plumbing and work our way
 up to porcelain.  We do have git diff-files, diff-index, diff-tree, so
 I don't see what the problem with diff-link is.  The point is that we
 can get an initial scripted version out quickly.

 And no, I never suggested a git log-link.

git log -p .gitmodules would be a way to review what changed in
the information about submodules.  Don't you need git log-link for
exactly the same reason why you need git diff-link in the first
place?

So you may not have suggested it, but I suspect that was only
because you haven't had enough time to think things through.


--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [RFC/PATCH 0/7] Rework git core for native submodules

2013-04-04 Thread Ramkumar Ramachandra
Junio C Hamano wrote:
 git log -p .gitmodules would be a way to review what changed in
 the information about submodules.  Don't you need git log-link for
 exactly the same reason why you need git diff-link in the first
 place?

 So you may not have suggested it, but I suspect that was only
 because you haven't had enough time to think things through.

What is this git log -p .gitmodules doing?  It's walking down the
commit history, and picking out the commits in which that blob
changed.  Then it's diffing the blobs in those commits with each
other.  Why is git log -p link any different?  We already know how
to diff blobs, and we just need a way to diff links.
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html