Re: Composing git repositories
Am 05.04.2013 07:27, schrieb Duy Nguyen: > On Fri, Apr 5, 2013 at 3:53 PM, Junio C Hamano wrote: >>> A brief summary or outcome from these links in the comment would >>> be nice. >> >> A summary of what to consider in Documentation/technical/ somewhere >> may be a very welcome addition. Thanks for volunteering ;-). > > No thanks :-) I did not really follow this thread to make such > contribution. Still have some work to do with other topics. I'll do that (unless someone else steps up). I had these links in my todo list with the intent of adding them to my github page, but I agree they make more sense in Documentation/technical. Will see when I find a time slot to read through the whole threads to give them meaningful captions. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
On Fri, Apr 5, 2013 at 3:53 PM, Junio C Hamano wrote: >> A brief summary or outcome from these links in the comment would >> be nice. > > A summary of what to consider in Documentation/technical/ somewhere > may be a very welcome addition. Thanks for volunteering ;-). No thanks :-) I did not really follow this thread to make such contribution. Still have some work to do with other topics. -- Duy -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Duy Nguyen writes: > Should someone add these links to the source code (maybe as a comment > in submodule.c, or above the definition of S_IFGITLINK in cache.h)? They were given in response to a request for reading material to learn background. Most of the straw-man outlines raised in these old discussoin threads are very different from what we ended up doing, and their value is mostly to learn what kind of use cases should one consider if one were designing subproject support from scratch, what possible approaches were considered and what were found already lacking in them before we even had a working prototype (in other words, to learn where one should _not_ go). Link themselves are not very useful and definitely not appropriate as a comment next to S_IFGITLINK, as S_IFGITLINK is not the only thing that implements the subproject support, and not all the ideals dreamed in these threads are realized with the implementation (yet). > A brief summary or outcome from these links in the comment would > be nice. A summary of what to consider in Documentation/technical/ somewhere may be a very welcome addition. Thanks for volunteering ;-). -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
On Thu, Apr 4, 2013 at 5:40 PM, Junio C Hamano wrote: >> Not on the current design but the discussion before that round that >> influenced the outcome greatly was this: >> >> http://thread.gmane.org/gmane.comp.version-control.git/14486/focus=14492 >> >> where we discussed a separate "gitlink" type of object. >> >> And obviously this discussion is also a must read: >> >> http://thread.gmane.org/gmane.comp.version-control.git/44106 >> > > Found a bit more relevant and probably more important (at the design > level) discussion history for people interested in understanding why > the things are as they are (without which we cannot make progress > while avoiding mistakes): > > http://thread.gmane.org/gmane.comp.version-control.git/15072 > http://thread.gmane.org/gmane.comp.version-control.git/31941/focus=32302 > http://thread.gmane.org/gmane.comp.version-control.git/47466/focus=47621 Should someone add these links to the source code (maybe as a comment in submodule.c, or above the definition of S_IFGITLINK in cache.h)? A brief summary or outcome from these links in the comment would be nice. -- Duy -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Junio C Hamano writes: > Jonathan Nieder writes: > >> If you are curious, at a quieter time it might be useful to ask for >> pointers to the discussions that led to the current design, and folks >> on the list might be glad to help. > > Not on the current design but the discussion before that round that > influenced the outcome greatly was this: > > http://thread.gmane.org/gmane.comp.version-control.git/14486/focus=14492 > > where we discussed a separate "gitlink" type of object. > > And obviously this discussion is also a must read: > > http://thread.gmane.org/gmane.comp.version-control.git/44106 > > I vaguely recall asking (or seeing somebody ask) why Linus ended up > with using "commit in index" without introducing a separate gitlink > type, but I didn't find it. IIRC, the answer was "it turned out > that we didn't need it" or something like that, which I tend to > agree. Found a bit more relevant and probably more important (at the design level) discussion history for people interested in understanding why the things are as they are (without which we cannot make progress while avoiding mistakes): http://thread.gmane.org/gmane.comp.version-control.git/15072 http://thread.gmane.org/gmane.comp.version-control.git/31941/focus=32302 http://thread.gmane.org/gmane.comp.version-control.git/47466/focus=47621 -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Am 02.04.2013 20:35, schrieb Ramkumar Ramachandra: > Jens Lehmann wrote: >> But I think we recently learned to support that use case with >> submodules. I think there are two floating models: >> >> - Tracked: >> [...] >> >> - Untracked: >> Some people just want "the newest" tip of a branch checked out in >> the submodule and update that from time to time (I suspect this >> is because they are used to SVN externals, which I believe work >> that way). You throw away reproducibility, which I think is not >> good and not the way I expect Git to work. >> [...] > > Nope, it has nothing to do with SVN externals; I've never used them. > And no, all repositories aren't created equal. I should be able to > add in magit.git into my dotfiles repository without worrying about > which commits the other repositories were at a particular commit. If > my project depends on the bleeding edge of poppler and girarra, I > should always be able to tell what commits in each subproject the > build was passing in. In other words, I should be able to freely > mixed floating and fixed submodules. There's no reason for one to be > Right, and the other to be a shunned second-class citizen. But you can currently mix floating and fixed submodules, as each submodule can be configured differently. Or am I missing something here? -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Am 02.04.2013 19:44, schrieb Ramkumar Ramachandra: > Jonathan Nieder wrote: >> Elated is probably not the right word. More "annoyed at being told >> their work is ugly without an accompanying concrete and actionable bug >> report". :) > > If I had an actionable report, I'd have started hammering patches > instead of wasting everyone's time here. I'm was presenting fragments > of my thoughts, hoping that it turn into concrete actionable work > after exchanging a few emails. I'm also annoyed that it didn't > happen. But didn't we already note three worthwhile patches (get rid of the top-level requirement and relocate a .git directory at "add" time and anytime later)? -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Ramkumar Ramachandra wrote: > What will I be merging and rebasing? One configuration file stuffed > with miscellaneous repositories. Don't you think this is highly > unpleasant? I spoke too fast. Isn't that exactly what we do with .gitmodules today (I'm not saying it's ideal, but I can't think of an alternative)? Yes, you're right: a simple script with a configuration file can manage only floating submodules quite well. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Jeff King wrote: > I'm happy to make my dump available to anyone who wants it, but it's > kind of big (about 1.4G uncompressed). Thanks. Can you put it up publicly somewhere (Dropbox comes to mind), and send me a link? -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Jonathan Nieder wrote: > Ramkumar Ramachandra wrote: >> Jonathan Nieder wrote: > >>> $ git clone git://repo.or.cz/git.git > [...] >>> Don't forget to "git clone -b todo git://repo.or.cz/git.git >>> git/Meta" >>> for maintenance scripts. >>> $ >> >> Nope, it's not mandatory for everyone to use dotfiles.git in exactly >> the same way either. In other words: I'm not sitting in an office and >> working with my colleagues on exactly the same things, in exactly the >> same way; wasn't that the Subversion age? Some might decide to >> initialize a few submodules, change the URLs of some, and remove some. > > Can't a script pointed to in README handle all these things? If it came to that, it's quite possible to write one giant Perl script to do whatever I want (see: git-slave). What will I be merging and rebasing? One configuration file stuffed with miscellaneous repositories. Don't you think this is highly unpleasant? Why would I want to write slightly different scripts for each of my repositories? I'd want to write down several different adjustable knobs to use something that's well-integrated into core git, like we do in .git/config (except we can't share a .gitconfig yet). > I don't think this hostile way of explaining things is warranted. :/ Sorry about that. No hostility intended. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Ramkumar Ramachandra wrote: > Jonathan Nieder wrote: >> $ git clone git://repo.or.cz/git.git [...] >> Don't forget to "git clone -b todo git://repo.or.cz/git.git git/Meta" >> for maintenance scripts. >> $ > > Nope, it's not mandatory for everyone to use dotfiles.git in exactly > the same way either. In other words: I'm not sitting in an office and > working with my colleagues on exactly the same things, in exactly the > same way; wasn't that the Subversion age? Some might decide to > initialize a few submodules, change the URLs of some, and remove some. Can't a script pointed to in README handle all these things? > I'd want my private fork to have commits changing "initialize > submodule quux" to "don't initialize submodule quux", and be able to > rebase that on top of upstream. These would be patches to comment or uncomment repositories in the list used by your "setup" script. > Why are you leaning towards solutions > for very narrow usecases? I don't think this hostile way of explaining things is warranted. :/ Yours, Jonathan -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Seth Robertson wrote: > > In message > , > Ramkumar Ramachandra writes: > > As a user inexperienced with recursive submodules (I've only used them > in this repository), I found it highly confusing. Thanks for clearing > them up. > > You may want to investigate third party alternatives like gitslave > http://gitslave.sf.net Depending on what problem you are trying to > solve, it can be better (or worse) to use compared to submodules. I'm not looking for yet another specialized tool for my special combined-repository workflow. I'm looking to build one generalized tool that will be able to do various kinds of compositions equally well. > [...] Thanks. I learnt one more workflow that people want to use. It's one large Perl script. Quite an involved hack. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Jonathan Nieder wrote: > That sounds similar to what Junio does with the Meta subdirectory in > his git development worktree. I don't think submodules are a good > fit, but it might make sense to start respecting a .motd file to allow > the following in a hypothetical world where everyone who clones git > uses the same scripts Junio does: > > $ git clone git://repo.or.cz/git.git > Cloning into 'git'... > remote: Counting objects: 151283, done. > remote: Compressing objects: 100% (38546/38546), done. > remote: Total 151283 (delta 111004), reused 151073 (delta 110797) > Receiving objects: 100% (151283/151283), 36.39 MiB | 7.66 MiB/s, done. > Resolving deltas: 100% (111004/111004), done. > > Don't forget to "git clone -b todo git://repo.or.cz/git.git git/Meta" > for maintenance scripts. > $ Nope, it's not mandatory for everyone to use dotfiles.git in exactly the same way either. In other words: I'm not sitting in an office and working with my colleagues on exactly the same things, in exactly the same way; wasn't that the Subversion age? Some might decide to initialize a few submodules, change the URLs of some, and remove some. I'd want my private fork to have commits changing "initialize submodule quux" to "don't initialize submodule quux", and be able to rebase that on top of upstream. Why are you leaning towards solutions for very narrow usecases? -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Jonathan Nieder writes: > ..., but it might make sense to start respecting a .motd file to allow > the following in a hypothetical world where everyone who clones git > uses the same scripts Junio does: > > $ git clone git://repo.or.cz/git.git > Cloning into 'git'... > remote: Counting objects: 151283, done. > remote: Compressing objects: 100% (38546/38546), done. > remote: Total 151283 (delta 111004), reused 151073 (delta 110797) > Receiving objects: 100% (151283/151283), 36.39 MiB | 7.66 MiB/s, done. > Resolving deltas: 100% (111004/111004), done. > > Don't forget to "git clone -b todo git://repo.or.cz/git.git git/Meta" > for maintenance scripts. > $ > > That would allow you to include an arbitrary setup script (including > cloning dependencies as well as running "autoreconf" or whatever) and > give people cloning a quick reminder to inspect it if paranoid and > then run it. I do not think motd is a good fit for the above; the message will disappear any time once you are done cloning. Depending on how common the use of Meta/ should be, it is more appropriate to have such an instruction in README or some other places as tracked contents. I do not know if the above is related to the problem Ram is trying to solve or a totally orthogonal issue, though. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Ramkumar Ramachandra wrote: > I should be able to > add in magit.git into my dotfiles repository For this, ideally what we'd want is a file that lists repositories that should be cloned at the same time as cloning the dotfiles repository, to reconstitute your dotfiles on a new machine. From then on, when acting on the dotfiles repository (with "git status", "git commit", etc) git should not pay attention to the magit subdirectory at all, because there is no coupling between the two. Did I get that right? That sounds similar to what Junio does with the Meta subdirectory in his git development worktree. I don't think submodules are a good fit, but it might make sense to start respecting a .motd file to allow the following in a hypothetical world where everyone who clones git uses the same scripts Junio does: $ git clone git://repo.or.cz/git.git Cloning into 'git'... remote: Counting objects: 151283, done. remote: Compressing objects: 100% (38546/38546), done. remote: Total 151283 (delta 111004), reused 151073 (delta 110797) Receiving objects: 100% (151283/151283), 36.39 MiB | 7.66 MiB/s, done. Resolving deltas: 100% (111004/111004), done. Don't forget to "git clone -b todo git://repo.or.cz/git.git git/Meta" for maintenance scripts. $ That would allow you to include an arbitrary setup script (including cloning dependencies as well as running "autoreconf" or whatever) and give people cloning a quick reminder to inspect it if paranoid and then run it. Jonathan -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Jens Lehmann wrote: > But I think we recently learned to support that use case with > submodules. I think there are two floating models: > > - Tracked: > [...] > > - Untracked: > Some people just want "the newest" tip of a branch checked out in > the submodule and update that from time to time (I suspect this > is because they are used to SVN externals, which I believe work > that way). You throw away reproducibility, which I think is not > good and not the way I expect Git to work. > [...] Nope, it has nothing to do with SVN externals; I've never used them. And no, all repositories aren't created equal. I should be able to add in magit.git into my dotfiles repository without worrying about which commits the other repositories were at a particular commit. If my project depends on the bleeding edge of poppler and girarra, I should always be able to tell what commits in each subproject the build was passing in. In other words, I should be able to freely mixed floating and fixed submodules. There's no reason for one to be Right, and the other to be a shunned second-class citizen. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Jonathan Nieder writes: > If you are curious, at a quieter time it might be useful to ask for > pointers to the discussions that led to the current design, and folks > on the list might be glad to help. Not on the current design but the discussion before that round that influenced the outcome greatly was this: http://thread.gmane.org/gmane.comp.version-control.git/14486/focus=14492 where we discussed a separate "gitlink" type of object. And obviously this discussion is also a must read: http://thread.gmane.org/gmane.comp.version-control.git/44106 I vaguely recall asking (or seeing somebody ask) why Linus ended up with using "commit in index" without introducing a separate gitlink type, but I didn't find it. IIRC, the answer was "it turned out that we didn't need it" or something like that, which I tend to agree. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
On Tue, Apr 02, 2013 at 11:14:49PM +0530, Ramkumar Ramachandra wrote: > > If you are curious, at a quieter time it might be useful to ask for > > pointers to the discussions that led to the current design, and folks > > on the list might be glad to help. > > Will do. The search on GMane is no good, and taking a local dump to > search using real tools is just too painful; does someone already have > a local dump? Yes, I have a maildir with the complete list archive, which I index using mairix (or sometimes grep if I'm doing something particularly tricky). I find the search on gmane to be abysmal. I'm happy to make my dump available to anyone who wants it, but it's kind of big (about 1.4G uncompressed). -Peff -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Jonathan Nieder wrote: > Elated is probably not the right word. More "annoyed at being told > their work is ugly without an accompanying concrete and actionable bug > report". :) If I had an actionable report, I'd have started hammering patches instead of wasting everyone's time here. I'm was presenting fragments of my thoughts, hoping that it turn into concrete actionable work after exchanging a few emails. I'm also annoyed that it didn't happen. > If you are curious, at a quieter time it might be useful to ask for > pointers to the discussions that led to the current design, and folks > on the list might be glad to help. Will do. The search on GMane is no good, and taking a local dump to search using real tools is just too painful; does someone already have a local dump? -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
On Mon, Apr 1, 2013 at 8:14 AM, Jens Lehmann wrote:
> Am 01.04.2013 01:50, schrieb Phil Hord:
>> On Sun, Mar 31, 2013 at 4:34 PM, Ramkumar Ramachandra
>> wrote:
>>> Jens Lehmann wrote:
Guess what: submodules are the solution for a certain set of use
cases, and tools like subtree are a solution for another set of
use cases. There is no silver bullet.
>>>
>>> That's the core of your argument: submodules already solve what it
>>> was meant to, and we can't get it to solve a larger class of problems.
>>> In other words, you're implying that it's impossible to build a tool
>>> that will be able to compose git repositories in a way that solves a
>>> very large class of problems. I don't see conclusive proof for this,
>>> so I have to disagree.
>>
>> I think it is possible to solve larger classes of problems with
>> submodules, but it is a harder problem than you seem to think. In any
>> case I do not think you need to re-engineer submodules to improve
>> them.
>>
>> Sumodules are good for preserving history. When properly managed,
>> they answer the question git always answers, "Where was my code in the
>> past?" I would like proper management to be easier, but I understand
>> why it is difficult; and I see it getting easier.
>
> Exactly.
>
>> Some users also want submodules to handle other tasks, like "Import
>> branch-tracked upstream changes (i.e. git pull origin master)." This
>> too is useful, but it is a different problem than submodules'
>> primarily try to solve. But they do already solve _part_ of that
>> problem ("Show me how these modules are related"), so it seems a
>> trivial thing to ask them also to handle the "floating branch" task.
>> The trick is to handle this task in a way that does not break the task
>> they are designed and used for already.
>
> But I think we recently learned to support that use case with
> submodules. I think there are two floating models:
>
> - Tracked:
> Follow a branch in the submodule and let git help you to advance
> the submodule to the tip of that branch at certain times, but
> still record a certain SHA-1 in the superproject to maintain
> reproducibility. We support this since 1.8.2 (see 06b1abb5 by
> Trevor).
Thanks. I followed that thread closely, but I thought the patch had
stalled again. I'm glad to see it in master
> - Untracked:
> Some people just want "the newest" tip of a branch checked out in
> the submodule and update that from time to time (I suspect this
> is because they are used to SVN externals, which I believe work
> that way). You throw away reproducibility, which I think is not
> good and not the way I expect Git to work. But that use case is
> achieved with a simple script and some config settings telling
> Git to don't even look for changes in the submodule anymore, and
> submodule infrastructure will set up everything for you after
> cloning the superproject. You run your custom script from time
> to time and have a truly floating submodule.
>
> So to me it looks we support both floating models with current Git's
> submodule infrastructure.
Well, for that matter, the "tracked" floating tip was also a simple
script once a upon a time. To say we support both is nothing new.
You could say we supported both in 1.8.1, and now we support "Tracked"
a little bit better in 1.8.2.
I think the difference is that everyone's expectation for "Untracked"
is a little bit different; or maybe it is just dangerous enough that
it should not be a core feature.
But I ain't complainin'.
Phil
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Am 01.04.2013 01:50, schrieb Phil Hord:
> On Sun, Mar 31, 2013 at 4:34 PM, Ramkumar Ramachandra
> wrote:
>> Jens Lehmann wrote:
>>> Guess what: submodules are the solution for a certain set of use
>>> cases, and tools like subtree are a solution for another set of
>>> use cases. There is no silver bullet.
>>
>> That's the core of your argument: submodules already solve what it
>> was meant to, and we can't get it to solve a larger class of problems.
>> In other words, you're implying that it's impossible to build a tool
>> that will be able to compose git repositories in a way that solves a
>> very large class of problems. I don't see conclusive proof for this,
>> so I have to disagree.
>
> I think it is possible to solve larger classes of problems with
> submodules, but it is a harder problem than you seem to think. In any
> case I do not think you need to re-engineer submodules to improve
> them.
>
> Sumodules are good for preserving history. When properly managed,
> they answer the question git always answers, "Where was my code in the
> past?" I would like proper management to be easier, but I understand
> why it is difficult; and I see it getting easier.
Exactly.
> Some users also want submodules to handle other tasks, like "Import
> branch-tracked upstream changes (i.e. git pull origin master)." This
> too is useful, but it is a different problem than submodules'
> primarily try to solve. But they do already solve _part_ of that
> problem ("Show me how these modules are related"), so it seems a
> trivial thing to ask them also to handle the "floating branch" task.
> The trick is to handle this task in a way that does not break the task
> they are designed and used for already.
But I think we recently learned to support that use case with
submodules. I think there are two floating models:
- Tracked:
Follow a branch in the submodule and let git help you to advance
the submodule to the tip of that branch at certain times, but
still record a certain SHA-1 in the superproject to maintain
reproducibility. We support this since 1.8.2 (see 06b1abb5 by
Trevor).
- Untracked:
Some people just want "the newest" tip of a branch checked out in
the submodule and update that from time to time (I suspect this
is because they are used to SVN externals, which I believe work
that way). You throw away reproducibility, which I think is not
good and not the way I expect Git to work. But that use case is
achieved with a simple script and some config settings telling
Git to don't even look for changes in the submodule anymore, and
submodule infrastructure will set up everything for you after
cloning the superproject. You run your custom script from time
to time and have a truly floating submodule.
So to me it looks we support both floating models with current Git's
submodule infrastructure.
> Some other users want submodules to solve the problem of composition,
> like "Show me the combined log of all these submodules." (Replace
> "show log" with "diff", "merge", "bisect" or even "rebase" if you
> like.) I think this is where Jens is leaning when working to improve
> the user experience. But this direction does not require
> re-architecting the fundamentals of submodules.
Correct. The only major change needed for that was to move the .git
directories into the .git directory of the superproject to prepare
for recursive update. But that is done under the hood and didn't
touch the fundamentals of using gitlinks and .gitmodules, it is just
a change in the layout of the local clone.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Am 31.03.2013 22:34, schrieb Ramkumar Ramachandra: >> Are you aware that current Git code already stats all files across >> all submodules recursive by default? So (again) no problem here, we >> do that already (unless configured otherwise). > > I didn't know that. Why does it do this? To show the user work tree changes inside the submodules too. It was really easy to forget to commit accompanying submodule changes when preparing a superproject commit before that (and that broke quite some builds at my $dayjob until we fixed that). >> Guess what: submodules are the solution for a certain set of use >> cases, and tools like subtree are a solution for another set of >> use cases. There is no silver bullet. > > That's the core of your argument: submodules already solve what it > was meant to, and we can't get it to solve a larger class of problems. > In other words, you're implying that it's impossible to build a tool > that will be able to compose git repositories in a way that solves a > very large class of problems. I don't see conclusive proof for this, > so I have to disagree. > > To summarize, everyone seems to be elated with the current state of > submodules and is vehemently defending it. I'm a little unhappy, but > am unable to express my discontent in better prose. I just think it is too early for the "let's do things differently and redesign stuff" phase. Before doing that we have to clear up the things that currently don't work for you or are confusing you about submodules and see if they are already solved (which could lead to a documentation update) or what it would take to fix them using current submodule infrastructure. I have the strong feeling that after we did that, no design changes are necessary anymore. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
In message , Ramkumar Ramachandra writes: As a user inexperienced with recursive submodules (I've only used them in this repository), I found it highly confusing. Thanks for clearing them up. You may want to investigate third party alternatives like gitslave http://gitslave.sf.net Depending on what problem you are trying to solve, it can be better (or worse) to use compared to submodules. It provides a usually conceptually easier method to group subprojects together into a superproject. You can replace practically any git command you want with "gits" and it will usually work as you might more or less expect. Conceptually it has a list of git repositories and will execute the listed git command on each in turn. It seems to resolve most of the issues that you raise, but of course it has some warts of its own. Some could be resolved with sufficient effort, others are fundamental. (An example of the latter, you cannot trivially tell what commit in other repositories a particular user was at when he made a commit in a specific repository (absent a gits tag being created). An example of the former, if you have git output paging turned on and many subprojects to check out, `gits clone` pages the output and more to the point, blocks the clones until you page through the output which you must typically do many times). -Seth Robertson -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
On Sun, Mar 31, 2013 at 4:34 PM, Ramkumar Ramachandra
wrote:
> Thanks for taking the time and effort to review my thoughts.
>
> Jens Lehmann wrote:
>> A
>> commit is the thing to record here because it *is* the perfect fit
>
> Might be, but saying that doesn't help one bit. I want to know why.
>
>> I want to improve the user experience
>> of submodules and don't care much in what language we achieve that.
>
> You missed the point entirely. If git didn't have a commit object,
> would you use a special kind of blob and code around everything to
> avoid fixing a more fundamental issue?
>
>> What happens when you rename "magit" to "foo" in that branch and want
>> to check out an older commit of the same branch? That is one of the
>> reasons why that belongs in to a checked in .gitmodules and not
>> someplace untracked.
>
> Good point. I learnt something new.
>
>> Are you aware that current Git code already stats all files across
>> all submodules recursive by default? So (again) no problem here, we
>> do that already (unless configured otherwise).
>
> I didn't know that. Why does it do this?
>
>> Guess what: submodules are the solution for a certain set of use
>> cases, and tools like subtree are a solution for another set of
>> use cases. There is no silver bullet.
>
> That's the core of your argument: submodules already solve what it
> was meant to, and we can't get it to solve a larger class of problems.
> In other words, you're implying that it's impossible to build a tool
> that will be able to compose git repositories in a way that solves a
> very large class of problems. I don't see conclusive proof for this,
> so I have to disagree.
I think it is possible to solve larger classes of problems with
submodules, but it is a harder problem than you seem to think. In any
case I do not think you need to re-engineer submodules to improve
them.
Sumodules are good for preserving history. When properly managed,
they answer the question git always answers, "Where was my code in the
past?" I would like proper management to be easier, but I understand
why it is difficult; and I see it getting easier.
Some users also want submodules to handle other tasks, like "Import
branch-tracked upstream changes (i.e. git pull origin master)." This
too is useful, but it is a different problem than submodules'
primarily try to solve. But they do already solve _part_ of that
problem ("Show me how these modules are related"), so it seems a
trivial thing to ask them also to handle the "floating branch" task.
The trick is to handle this task in a way that does not break the task
they are designed and used for already.
Some other users want submodules to solve the problem of composition,
like "Show me the combined log of all these submodules." (Replace
"show log" with "diff", "merge", "bisect" or even "rebase" if you
like.) I think this is where Jens is leaning when working to improve
the user experience. But this direction does not require
re-architecting the fundamentals of submodules.
> To summarize, everyone seems to be elated with the current state of
> submodules and is vehemently defending it.
That's a gross overstatement. Everyone understands that is
complicated and dangerous tweak a machine in operation. Such tweaks
should be safe, prudent and justifiable, in roughly that order.
Phil
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Ramkumar Ramachandra wrote: > Jens Lehmann wrote: >> A >> commit is the thing to record here because it *is* the perfect fit > > Might be, but saying that doesn't help one bit. I want to know why. [...] > To summarize, everyone seems to be elated with the current state of > submodules and is vehemently defending it. I'm a little unhappy, but > am unable to express my discontent in better prose. Let's just go > back to writing patches, and come back to this if and when I have a > full design. Elated is probably not the right word. More "annoyed at being told their work is ugly without an accompanying concrete and actionable bug report". :) If you are curious, at a quieter time it might be useful to ask for pointers to the discussions that led to the current design, and folks on the list might be glad to help. Hope that helps, Jonathan -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Thanks for taking the time and effort to review my thoughts. Jens Lehmann wrote: > A > commit is the thing to record here because it *is* the perfect fit Might be, but saying that doesn't help one bit. I want to know why. > I want to improve the user experience > of submodules and don't care much in what language we achieve that. You missed the point entirely. If git didn't have a commit object, would you use a special kind of blob and code around everything to avoid fixing a more fundamental issue? > What happens when you rename "magit" to "foo" in that branch and want > to check out an older commit of the same branch? That is one of the > reasons why that belongs in to a checked in .gitmodules and not > someplace untracked. Good point. I learnt something new. > Are you aware that current Git code already stats all files across > all submodules recursive by default? So (again) no problem here, we > do that already (unless configured otherwise). I didn't know that. Why does it do this? > Guess what: submodules are the solution for a certain set of use > cases, and tools like subtree are a solution for another set of > use cases. There is no silver bullet. That's the core of your argument: submodules already solve what it was meant to, and we can't get it to solve a larger class of problems. In other words, you're implying that it's impossible to build a tool that will be able to compose git repositories in a way that solves a very large class of problems. I don't see conclusive proof for this, so I have to disagree. To summarize, everyone seems to be elated with the current state of submodules and is vehemently defending it. I'm a little unhappy, but am unable to express my discontent in better prose. Let's just go back to writing patches, and come back to this if and when I have a full design. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Am 28.03.2013 10:16, schrieb Ramkumar Ramachandra: > Jens Lehmann wrote: >> Unless you acknowledge that submodules are a different repo, you'll >> always run into problems. I believe future enhancements will make >> this less tedious, but in the end they will stay separate repos >> (which is the whole point, you'd want to use a different approach >> - e.g. subtree - if you want to put stuff from different upstreams >> into a single repo without keeping the distinction where all that >> came from). > > I acknowledge that it's a different repository. It's just that I > think that our current design has too many seams: why do you think > it's impossible to make it seamless? > > git-subtree is not an answer to anything. Dumping all the history > into one repository has its limited usecases, but it is no solution. Guess what: submodules are the solution for a certain set of use cases, and tools like subtree are a solution for another set of use cases. There is no silver bullet. >> What else than a commit object should that be??? Submodules are >> there to have a different upstream for a part of your work tree, >> and that means a commit in that repo is the only sane thing to >> record in the superproject. A lot of thought has been put into >> this, and it is definitely a good choice [1]. > > Linus argues that it shouldn't be a tree object, and I agree with > that. I don't see an argument that says that the commit object is a > perfect fit (probably because it's not). There was discussion about what to record in the index/commit of the superproject in early submodule days (some time before I became involved in Git, seems I currently cannot find a link to that). A commit is the thing to record here because it *is* the perfect fit, as some years of submodule experience show. >> How? The "submodules suck, we should try a completely different >> approach" thingy comes up from time to time, but so far nobody >> could provide a viable alternative to what we currently do. > > My argument is not "submodules suck; we should throw them out of the > window, and start from scratch" at all. I'm merely questioning the > fundamental assumptions that submodules make, instead of proposing > that we work around everything in shell. We don't have to be married > to the existing implementation of submodules and try to fix all the > problems in shell. You cannot simply change the fundamental assumptions of submodules and expect them to be the same thing afterwards. And it doesn't matter at all if we "fix all the problems in shell" or in C-code, we'll fix the remaining problems that are fixable in whatever part of Git it makes sense. And I don't have the impression you have an idea about what submodules are good at, where they can be improved and what problems they'll probably never solve. >> And apart from that, let's not forget we identified some valuable >> improvements to submodules in this thread: >> [...] >> All of those are topics I like to see materialize, and you are >> welcome to tackle them. > > Allow me a few days to think about changing the fundamental building > blocks to make our shell hackery easier. Please go ahead, but if your goal is "to make our shell hackery easier" I'm not interested. I want to improve the user experience of submodules and don't care much in what language we achieve that. And I can't see anything fundamental being wrong with submodules but strongly believe they are a perfect match for some very important use cases (some of which I see happening at my $dayjob for some years now), so I still don't see what you are trying to "fix" here. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Am 28.03.2013 12:48, schrieb Ramkumar Ramachandra: > Okay, here's a first draft of the new design. The new mediator object > should look like: > > name = git > ref = v1.7.8 > > The name is looked up in refs/modules/, which in turn looks like: > > [submodule "git"] > origin = gh:artagnon/git > path = git > [submodule "magit"] > origin = gh:magit/magit > path = git/extensions/magit What happens when you rename "magit" to "foo" in that branch and want to check out an older commit of the same branch? That is one of the reasons why that belongs in to a checked in .gitmodules and not someplace untracked. > This solves the two problems that I brought up earlier: > - Floating submodules (which are _necessary_ if you don't want to > propagate commits upwards to the root). If you don't want that either don't use submodules or set the ignore config so you won't be bothered with any changes to the submodules. Floating up to the submodule's tip can be easily achieved with a script (possibly checked in in the superproject). You loose the reproducibility by doing that, but that's what you asked for. No problem here. > - Initializing a nested submodule without having to initialize all the > submodules in the path leading up to it. You cannot access a nested sub-submodule without its parent telling you what submodules it has. Otherwise the first level submodule would not be self-contained, so you'll need to check it out too to access the sub-submodules. Nothing to fix here either. > However, I suspect that we can put more information the mediator > object to make life easier for the parent repository and make seams > disappear. I'm currently thinking about what information git core > needs to behave smoothly with submodules. To me your proposal is trying to fix non-issues and breaking stuff that works, so I see no improvement. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Am 28.03.2013 11:01, schrieb Ramkumar Ramachandra: > Jonathan Nieder wrote: >> Do you mean that you wish you could ignore subrepository boundaries >> and use commands like >> >> git clone --recurse-submodules http://git.zx2c4.com/cgit >> cd cgit >> vi git/cache.h >> ... edit edit edit ... >> git add --recurse-submodules git/cache.h >> git commit --recurse-submodules >> git push --recurse-submodules >> >> , possibly with configuration to allow the --recurse-submodules to be >> implied, and have everything work out well? > > Do you realize how difficult this is to implement? We'll need to > patch all the git commands to essentially do what we'd get for free if > the submodule were a tree object instead of a commit object (although > I'm not saying that's the Right thing to do). Some caveats: > > - If we maintain one global index, and try to emulate git-subtree > using submodules, we've lost. It's going to take freakin' ages to > stat billions of files across hundreds of nested sumodules. One major > advantage of having repository boundaries is separate object stores, > indexes, worktrees: little ones that git is designed to work with. Are you aware that current Git code already stats all files across all submodules recursive by default? So (again) no problem here, we do that already (unless configured otherwise). > - Auto-splitting commits that touch multiple submodules/ superproject > at once. Although git-subtree does this, I think it's horribly ugly. You don't like it, but what technical argument is hidden here I'm missing? > - Auto-propagating commits upwards to the superproject is another big > challenge. I think the current design of anchoring to a specific > commit SHA-1 has its usecases, but is unwieldy when things become big. > We have to fix that first. What??? Again there is nothing to "fix" here, "anchoring to a specific commit SHA-1" is *the* most prominent use case (think reproducibility of the whole work tree), floating submodules are the oddball here. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Ramkumar Ramachandra wrote: > Do you realize how difficult this is to implement? We'll need to > patch all the git commands to essentially do what we'd get for free if > the submodule were a tree object instead of a commit object (although > I'm not saying that's the Right thing to do). What are you talking about? Yes, of course I realize that recursing over subprojects that are managed as separate git repositories requires writing new code. That's why people have started to write such code. They seem to think it's worth it. Meanwhile others with different designs in mind have written other tools. Use cases even overlap a little, so they can compete. That is exactly as it should be. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Junio C Hamano wrote: > As I said in another thread, your top-level may be only a part in > somebody else's project, and what you consider just a part of your > project may be the whole project to somebody else. If you pick one > location to store both for the above clone, e.g. cgit/.git (it could > be cgit/.ram-git or any other name), embedding it in a yet larger > project (perhaps having both cgit and gitolite to give a one-stop > solution for hosting services) later would face the same issue as > Ram seemed to be complaining. It needs to address what happens when > that cgit/.git (or whatever name) gets in the way in the scope of > the larger project. That is why I said Ram's rant, using subjective > words like "elegant", without sound technical justification, did not > make much sense to me. I was having a lot of difficulty writing down my thoughts. Thank you for providing an illustrative example. It is terribly hard to do with our current implementation: we'd have to rewrite the "gitdir: " lines in all the .git files in the submodule trees and rebuild all the .git/modules paths. I'm thinking that we need to separate the object stores from the worktrees for good. For a project with no submodules, the object store can be present in .git/ of the toplevel directory, like it is now. The moment submodules are added, all the object stores should be relocated to a place outside the worktree. So my ~/src might look like: dotfiles.git/, auto-complete.git/, magit.git/, git-commit-mode.git/, yasnippet.git/ and dotfiles/. dotfiles/ contains lots of worktrees stitched together nicely, pointing to these object stores in ~/src. This would certainly get rid of the asymmetry for good. Now, we can focus our attention on composing git worktrees. What is a worktree? A tree object pointed to by the commit object referred to by HEAD. What we need to do is embed one tree inside another using a mediating object to establish repository boundaries, while not introducing an ugly seam. If you think about it, the mediator we've picked conveys little/ no information to the parent; it says: "there's a commit with this SHA-1 present in this submodule, but I can't tell you the commit message, tree object, branch, remote, or anything else" (obviously because the commit isn't present in the parent's object store). So, the mediator might as well have been a SHA-1 string. And we have an ugly .gitmodules conveying the remote and the branch. Why can't we stuff more information into the mediating object and get rid of .gitmodules altogether? Okay, here's a first draft of the new design. The new mediator object should look like: name = git ref = v1.7.8 The name is looked up in refs/modules/, which in turn looks like: [submodule "git"] origin = gh:artagnon/git path = git [submodule "magit"] origin = gh:magit/magit path = git/extensions/magit The ref could be 'master', 'HEAD~1', or even a commit SHA-1 (to do the current anchored-submodules). Finally, there's a .git file in the worktree, which contains a "gitdir: " line pointing to the object store, as before. This solves the two problems that I brought up earlier: - Floating submodules (which are _necessary_ if you don't want to propagate commits upwards to the root). - Initializing a nested submodule without having to initialize all the submodules in the path leading up to it. However, I suspect that we can put more information the mediator object to make life easier for the parent repository and make seams disappear. I'm currently thinking about what information git core needs to behave smoothly with submodules. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Jonathan Nieder wrote: > Do you mean that you wish you could ignore subrepository boundaries > and use commands like > > git clone --recurse-submodules http://git.zx2c4.com/cgit > cd cgit > vi git/cache.h > ... edit edit edit ... > git add --recurse-submodules git/cache.h > git commit --recurse-submodules > git push --recurse-submodules > > , possibly with configuration to allow the --recurse-submodules to be > implied, and have everything work out well? Do you realize how difficult this is to implement? We'll need to patch all the git commands to essentially do what we'd get for free if the submodule were a tree object instead of a commit object (although I'm not saying that's the Right thing to do). Some caveats: - If we maintain one global index, and try to emulate git-subtree using submodules, we've lost. It's going to take freakin' ages to stat billions of files across hundreds of nested sumodules. One major advantage of having repository boundaries is separate object stores, indexes, worktrees: little ones that git is designed to work with. - Auto-splitting commits that touch multiple submodules/ superproject at once. Although git-subtree does this, I think it's horribly ugly. - Auto-propagating commits upwards to the superproject is another big challenge. I think the current design of anchoring to a specific commit SHA-1 has its usecases, but is unwieldy when things become big. We have to fix that first. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Jens Lehmann wrote: > Unless you acknowledge that submodules are a different repo, you'll > always run into problems. I believe future enhancements will make > this less tedious, but in the end they will stay separate repos > (which is the whole point, you'd want to use a different approach > - e.g. subtree - if you want to put stuff from different upstreams > into a single repo without keeping the distinction where all that > came from). I acknowledge that it's a different repository. It's just that I think that our current design has too many seams: why do you think it's impossible to make it seamless? git-subtree is not an answer to anything. Dumping all the history into one repository has its limited usecases, but it is no solution. > What else than a commit object should that be??? Submodules are > there to have a different upstream for a part of your work tree, > and that means a commit in that repo is the only sane thing to > record in the superproject. A lot of thought has been put into > this, and it is definitely a good choice [1]. Linus argues that it shouldn't be a tree object, and I agree with that. I don't see an argument that says that the commit object is a perfect fit (probably because it's not). > How? The "submodules suck, we should try a completely different > approach" thingy comes up from time to time, but so far nobody > could provide a viable alternative to what we currently do. My argument is not "submodules suck; we should throw them out of the window, and start from scratch" at all. I'm merely questioning the fundamental assumptions that submodules make, instead of proposing that we work around everything in shell. We don't have to be married to the existing implementation of submodules and try to fix all the problems in shell. > And apart from that, let's not forget we identified some valuable > improvements to submodules in this thread: > [...] > All of those are topics I like to see materialize, and you are > welcome to tackle them. Allow me a few days to think about changing the fundamental building blocks to make our shell hackery easier. Thanks. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Am 27.03.2013 18:02, schrieb Ramkumar Ramachandra: > Junio C Hamano wrote: >> Ramkumar Ramachandra writes: >>> Junio C Hamano wrote: So you have to stash it somewhere. We could have made it to move them to $HOME/.safeplace or somewhere totally unrelated to the superproject. So in that sense, the repositories are *not* owned by the superproject in any way. However, you are working within the context of the superproject on the submodule after all, and somewhere under $GIT_DIR/ of the superorject is not too wrong a place to use as such a safe place. >>> >>> Thanks for the explanation. >> >> What do you _exactly_ mean by that? You understood why things are >> arranged in that way, and no longer think that it is unnecessary, >> ugly and unwieldy to stash the real copy of $GIT_DIR of submodules >> away from their working trees and store them inside $GIT_DIR/modules >> of the superproject? > > In essence, git commands are built to act on pure worktrees. It's > trivially Correct to pretend that an object store present in the > toplevel directory (as .git/) of the worktree doesn't exist, but it's > quite non-trivial to handle a .git directory anywhere else in the > worktree. Since we built git ground-up to act on a single > repository's worktree, embedding one repository inside another is a > hack: as a "workaround", we simply relocate the object store of the > submodule repository. Submodules work pretty well, no matter if you call them a "hack". And what you call a "workaround" allows us to move, remove and recreate submodules, which is one of *the* major inconveniences submodules currently have. > Even then, working with one worktree embedded > inside another is something git never designed for: it explains why I > have to literally fight with git when using submodules (no offense > Jens; it's a very hard problem). Unless you acknowledge that submodules are a different repo, you'll always run into problems. I believe future enhancements will make this less tedious, but in the end they will stay separate repos (which is the whole point, you'd want to use a different approach - e.g. subtree - if you want to put stuff from different upstreams into a single repo without keeping the distinction where all that came from). > Representing submodules as commit objects in the tree is also a hack. > I'm sorry, but a submodule is not a commit object. We need a fifth > object type if we want them to be first-class citizens. What else than a commit object should that be??? Submodules are there to have a different upstream for a part of your work tree, and that means a commit in that repo is the only sane thing to record in the superproject. A lot of thought has been put into this, and it is definitely a good choice [1]. > Sorry, I'm deviating. I learnt why you think the hack is necessary > and not "too wrong". As I explained above, the entire design is > asymmetric and inelegant; I think we can do much better than this. How? The "submodules suck, we should try a completely different approach" thingy comes up from time to time, but so far nobody could provide a viable alternative to what we currently do. And apart from that, let's not forget we identified some valuable improvements to submodules in this thread: *) Get rid of the "toplevel" requirement *) Add functionality to relocate the object store out of the work tree (either "git submodule to-gitfile" or something similar, maybe even as a separate script in contrib) *) Add an option to "git submodule add" (and/or maybe a config option) to relocate the object store immediately on adding an already present submodule All of those are topics I like to see materialize, and you are welcome to tackle them. [1]: http://article.gmane.org/gmane.comp.version-control.git/151857/ -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Junio C Hamano wrote: > I however do not see the implementation detail of having (or not > having) separate $GIT_DIR for component projects having anything to > do with the goal of that ideal. Yeah, I think the current gitlink-instead-of-full-git-dir-for-submodules implementation that can allow "git rm" to remove a submodule and "git checkout --recurse-submodules" to later revive it is good. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Jonathan Nieder writes: > Ramkumar Ramachandra wrote: > >>Even then, working with one worktree embedded >> inside another is something git never designed for: it explains why I >> have to literally fight with git when using submodules > > Do you mean that you wish you could ignore subrepository boundaries > and use commands like > > git clone --recurse-submodules http://git.zx2c4.com/cgit > cd cgit > vi git/cache.h > ... edit edit edit ... > git add --recurse-submodules git/cache.h > git commit --recurse-submodules > git push --recurse-submodules > > , possibly with configuration to allow the --recurse-submodules to be > implied, and have everything work out well? > ... > I think something like that is a goal for submodules in the long > term,... As you hinted with "complications" below, I have to wonder what should happen when the above "git add" touches anything outside "git" subdirectory. But such an administrative details (the project boundary is primarily not an implementation detail but is a social issue) aside, I agree that overall it would be a good user experience. I however do not see the implementation detail of having (or not having) separate $GIT_DIR for component projects having anything to do with the goal of that ideal. Where and how do you envision the metainformation about the component projects are stored in such a clone? It does not have to be cgit/.git, but you would need to somehow store things we store in $GIT_DIR for cgit itself and git in the current system. If you pick one location to store both, I would imagine that it would still be somewhere under the cgit directory. As I said in another thread, your top-level may be only a part in somebody else's project, and what you consider just a part of your project may be the whole project to somebody else. If you pick one location to store both for the above clone, e.g. cgit/.git (it could be cgit/.ram-git or any other name), embedding it in a yet larger project (perhaps having both cgit and gitolite to give a one-stop solution for hosting services) later would face the same issue as Ram seemed to be complaining. It needs to address what happens when that cgit/.git (or whatever name) gets in the way in the scope of the larger project. That is why I said Ram's rant, using subjective words like "elegant", without sound technical justification, did not make much sense to me. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Ramkumar Ramachandra wrote: >Even then, working with one worktree embedded > inside another is something git never designed for: it explains why I > have to literally fight with git when using submodules Do you mean that you wish you could ignore subrepository boundaries and use commands like git clone --recurse-submodules http://git.zx2c4.com/cgit cd cgit vi git/cache.h ... edit edit edit ... git add --recurse-submodules git/cache.h git commit --recurse-submodules git push --recurse-submodules , possibly with configuration to allow the --recurse-submodules to be implied, and have everything work out well? I think something like that is a goal for submodules in the long term, with a caveat that there are complications in that different projects (the parent project and subproject) can have different contribution guidelines, review and release schedules, and so on. If submodules are not working for you today, you may find some of Jens's submodule improvement patches interesting, or you may want to look into alternatives that make different assumptions, such as entirely independent repositories and tools like "mr" that iterate over them. Hope that helps, Jonathan -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Ramkumar Ramachandra writes: > Sorry, I'm deviating. I learnt why you think the hack is necessary > and not "too wrong". OK. > As I explained above, the entire design is > asymmetric and inelegant; I think we can do much better than this. I personally find the "explained above" part making not much sense. Maybe others can comment on it. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Junio C Hamano wrote: > Ramkumar Ramachandra writes: >> Junio C Hamano wrote: >>> So you have to stash it somewhere. We could have made it to move >>> them to $HOME/.safeplace or somewhere totally unrelated to the >>> superproject. So in that sense, the repositories are *not* owned by >>> the superproject in any way. However, you are working within the >>> context of the superproject on the submodule after all, and >>> somewhere under $GIT_DIR/ of the superorject is not too wrong a >>> place to use as such a safe place. >> >> Thanks for the explanation. > > What do you _exactly_ mean by that? You understood why things are > arranged in that way, and no longer think that it is unnecessary, > ugly and unwieldy to stash the real copy of $GIT_DIR of submodules > away from their working trees and store them inside $GIT_DIR/modules > of the superproject? In essence, git commands are built to act on pure worktrees. It's trivially Correct to pretend that an object store present in the toplevel directory (as .git/) of the worktree doesn't exist, but it's quite non-trivial to handle a .git directory anywhere else in the worktree. Since we built git ground-up to act on a single repository's worktree, embedding one repository inside another is a hack: as a "workaround", we simply relocate the object store of the submodule repository. Even then, working with one worktree embedded inside another is something git never designed for: it explains why I have to literally fight with git when using submodules (no offense Jens; it's a very hard problem). Representing submodules as commit objects in the tree is also a hack. I'm sorry, but a submodule is not a commit object. We need a fifth object type if we want them to be first-class citizens. Sorry, I'm deviating. I learnt why you think the hack is necessary and not "too wrong". As I explained above, the entire design is asymmetric and inelegant; I think we can do much better than this. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Ramkumar Ramachandra writes: > Junio C Hamano wrote: >> So you have to stash it somewhere. We could have made it to move >> them to $HOME/.safeplace or somewhere totally unrelated to the >> superproject. So in that sense, the repositories are *not* owned by >> the superproject in any way. However, you are working within the >> context of the superproject on the submodule after all, and >> somewhere under $GIT_DIR/ of the superorject is not too wrong a >> place to use as such a safe place. > > Thanks for the explanation. What do you _exactly_ mean by that? You understood why things are arranged in that way, and no longer think that it is unnecessary, ugly and unwieldy to stash the real copy of $GIT_DIR of submodules away from their working trees and store them inside $GIT_DIR/modules of the superproject? > The paths in .git/modules are unnecessary ugly and unwieldy, > especially in the case of multiple levels of nesting... -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Junio C Hamano wrote: > So you have to stash it somewhere. We could have made it to move > them to $HOME/.safeplace or somewhere totally unrelated to the > superproject. So in that sense, the repositories are *not* owned by > the superproject in any way. However, you are working within the > context of the superproject on the submodule after all, and > somewhere under $GIT_DIR/ of the superorject is not too wrong a > place to use as such a safe place. Thanks for the explanation. The paths in .git/modules are unnecessary ugly and unwieldy, especially in the case of multiple levels of nesting: I'll look into converting it to a flat structure using .git, while handling name conflicts. I'll also look into adding a feature to relocate this/ using an object store from an existing clone. > Look for floating submodules in the list archive. The most relevant message thread I could find was [1], back from 2011. You argue that floating submodules are Wrong, and that there is no real usecase for it. Like I explained earlier, I'm looking at one tool that solves a superset of the problems mr, repo, submodules, subtrees, and other tools solve. I really like the way repo allows me to work on a meta project like Android or ChromiumOS, but hate that it allows for zero composition. To move forward, I have the following design thoughts (elaborating on my previous email): 1. If .gitmodules is tracked like a normal file, it is absolutely impossible to tell the possible dependencies of the superproject without cloning it entirely, and looking at the .gitmodules file in each of the branches. Can't we have it as a special ref instead, so I can `git fetch` just that ref to figure out the dependencies? 2. True composition requires that I be able to specify the entire manifest (for nested submodules) in the toplevel .gitmodules, or break it up as I see fit. This is currently impossible, and brings us back to #1: the manifest for b/, b/c are in the toplevel repo's special ref, and I need to fetch c's special ref to figure out what d is (or error out if no such thing exists). 3. True floating submodules are impossible, because a change in the submodule means a change in the commit object referenced by the superproject's tree object; diff-tree will see that some content has changed in the repository. We can represent that diff however we want (using diff.submodule), but we can't change the fact that the change has to be committed. Fixing this will require nothing short of introducing a new kind of object (say "submodule" object which can be a concrete SHA-1 or point to a ref). Do you think thinking about these things is worthwhile? I see more complaints like [2] as git adoption in the industry increases, but we have no solution: we can't make git scale to super-large repositories, and we have no real way to compose smaller repositories. Hacks like repo sadden me. [1]: http://thread.gmane.org/gmane.comp.version-control.git/185164 [2]: http://thread.gmane.org/gmane.comp.version-control.git/189776 -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Composing git repositories
Ramkumar Ramachandra writes: > Apart from the implementation glitches, I don't like the design; > submodules don't compose well: > > 1. There's an inherent asymmetry between the superproject and each of > the subprojects, because the superproject owns all the object stores. > Why is it absolutely necessary to relocate the object stores? Imagine doing "git checkout oldbranch" in the superproject when your current branch has a submodule A bound to it, but the oldbranch that is an old version of the superproject did not (yet) have it. You obviously want the directory A disappear, but you would be unhappy if you have to lose A/.git and everything in it while doing so. If you are lucky, you can re-clone, but you may have your own changes. So you have to stash it somewhere. We could have made it to move them to $HOME/.safeplace or somewhere totally unrelated to the superproject. So in that sense, the repositories are *not* owned by the superproject in any way. However, you are working within the context of the superproject on the submodule after all, and somewhere under $GIT_DIR/ of the superorject is not too wrong a place to use as such a safe place. > 3. The current implementation only allows me to compose with commit > objects, but what if I want to compose with refs? ie. What if I want > to track the tip of the 'master' of a submodule in a superproject? Look for floating submodules in the list archive. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
