Re: [PATCH 1/2] store submodule in common dir
On Wed, Aug 15, 2018 at 1:20 AM Junio C Hamano wrote: > Theoretically we should be able to make modules/kernel%2fv2.[24] > additional "worktree"s of modules/kernel%2fv2.6, but given that > these are all "bare" repositories without an attached working tree, > I am not sure how that would supposed to work. Thinking about > having multiple worktrees on a single bare repository makes me head > spin and ache X-<;-) I could only read the mail subject this morning and thought a bit about it on the train, and came to the same conclusion that "git worktree" is the way to go. The bareness should not affect this at all. If you meant core.bare, it can only affect the main repo. But what I'm thinking is repos with only linked worktrees and no main one. So when you "submodule update" a repo, you "clone -n", then "git worktree add" to put a worktree in place. The repo is strictly speaking not bare, it just does not have a main worktree. The main HEAD should be detached so that it does not block any worktree from checking out branches. If we could get away without the main HEAD, that would be great, but HEAD is part of the repo signature so it has to stay (and we may end up keeping more objects for this HEAD even after every other heads don't need the same objects anymore). This also solves the problem with mupltiple worktrees on a repo with submodules, where we have the same problems (if I add a new worktree of the superproject, where do the new submodules go?). In this case ideally all worktrees should have separate ref namespaces to maintain the "separate repo" view that we currently have, but I guess we can live with sharing refs for a while. And simply sharing $GIT_DIR/modules won't work. For starter, it breaks existing setups. What I've wanted to do is adding a shared "common" directory. Then you could have "common/modules" (among other future common stuff), which is shared across all worktrees, but you don't have to add extra share rules since it's already covered by the "common" rule. -- Duy
Re: [PATCH 1/2] store submodule in common dir
On Wed, 2018-08-15 at 10:28 +0200, Joakim Tjernlund wrote: > On Tue, 2018-08-14 at 16:20 -0700, Junio C Hamano wrote: > > CAUTION: This email originated from outside of the organization. Do not > > click links or open attachments unless you recognize the sender and know > > the content is safe. > > > > > > Junio C Hamano writes: > > > > > My understanding of what Joakim wants to do is to have a top-level > > > project that has three subdirectories, e.g. kernel/v2.2, kernel/v2.4 > > > and kernel/v2.6, each of which is a submodule that houses these > > > versions of Linux kernel source, but only clone Linus's repository > > > (as the up-to-late tree has all the necessary history to check out > > > these past development tracks). And that should be doable with > > > just the main checkout, without any additional worktree (it's just > > > the matter of having .git/modules/kernel%2fv2.6/ directory pointed > > > by two symlinks from .git/modules/kernel%2fv2.[24], or something > > > like that). > > > > Actually I take the last part of that back. When thought naively > > about, it may appear that it should be doable, but because each of > > the modules/* directory in the top-level project has to serve as the > > $GIT_DIR for each submodule checkout, and the desire is to have > > these three directories to have checkout of three different > > branches, a single directory under modules/. that is shared among > > three submodules would *not* work---they must have separate index, > > HEAD, etc. > > > > Theoretically we should be able to make modules/kernel%2fv2.[24] > > additional "worktree"s of modules/kernel%2fv2.6, but given that > > these are all "bare" repositories without an attached working tree, > > I am not sure how that would supposed to work. Thinking about > > having multiple worktrees on a single bare repository makes me head > > spin and ache X-<;-) > > You nailed it ! :) > My head spins just reading this so I think I got my answer. I can be done > but will be tricky to impl. > I will keep an eye on how submodules develops, sure would be a welcome > feature. > > Jocke On a related note, it would be great if git could support sparse checkout/clone for submodules, now one have to manually add .git/info/sparse-checkout. If sparse clone and sparse checkout could be saved in the submodule, then there would be no problem with 3 copies of the same repo in my submodules. Jocke
Re: [PATCH 1/2] store submodule in common dir
On Tue, 2018-08-14 at 16:20 -0700, Junio C Hamano wrote: > CAUTION: This email originated from outside of the organization. Do not click > links or open attachments unless you recognize the sender and know the > content is safe. > > > Junio C Hamano writes: > > > My understanding of what Joakim wants to do is to have a top-level > > project that has three subdirectories, e.g. kernel/v2.2, kernel/v2.4 > > and kernel/v2.6, each of which is a submodule that houses these > > versions of Linux kernel source, but only clone Linus's repository > > (as the up-to-late tree has all the necessary history to check out > > these past development tracks). And that should be doable with > > just the main checkout, without any additional worktree (it's just > > the matter of having .git/modules/kernel%2fv2.6/ directory pointed > > by two symlinks from .git/modules/kernel%2fv2.[24], or something > > like that). > > Actually I take the last part of that back. When thought naively > about, it may appear that it should be doable, but because each of > the modules/* directory in the top-level project has to serve as the > $GIT_DIR for each submodule checkout, and the desire is to have > these three directories to have checkout of three different > branches, a single directory under modules/. that is shared among > three submodules would *not* work---they must have separate index, > HEAD, etc. > > Theoretically we should be able to make modules/kernel%2fv2.[24] > additional "worktree"s of modules/kernel%2fv2.6, but given that > these are all "bare" repositories without an attached working tree, > I am not sure how that would supposed to work. Thinking about > having multiple worktrees on a single bare repository makes me head > spin and ache X-<;-) You nailed it ! :) My head spins just reading this so I think I got my answer. I can be done but will be tricky to impl. I will keep an eye on how submodules develops, sure would be a welcome feature. Jocke
Re: [PATCH 1/2] store submodule in common dir
Junio C Hamano wrote: > Theoretically we should be able to make modules/kernel%2fv2.[24] > additional "worktree"s of modules/kernel%2fv2.6, but given that > these are all "bare" repositories without an attached working tree, > I am not sure how that would supposed to work. Thinking about > having multiple worktrees on a single bare repository makes me head > spin and ache X-<;-) Agreed about spinning head. This is why I suggested at [1] that anyone intereseted in this start with description of their proposed design, which would have three benefits: - after implementation, it would document the intent behind whatever we choose to do, hopefully saving people debugging or improving this code some head spinning - it would allow subject matter experts on-list to suggest refinements and simplifications - it would avoid the interested contributor going too far down a blind alley in case their initial proposal has a fatal flaw I also agree with the "theoretically we should be able to make it work". As described in [1], I think most of this is work we're going to have to do eventually, as part of properly supporting multiple worktrees for a superproject. But I don't want to set wrong expectations: this will be hard. Thanks, Jonathan [1] https://public-inbox.org/git/20180814231049.gh142...@aiede.svl.corp.google.com/
Re: [PATCH 1/2] store submodule in common dir
On Tue, Aug 14, 2018 at 4:20 PM Junio C Hamano wrote: > > Junio C Hamano writes: > > > My understanding of what Joakim wants to do is to have a top-level > > project that has three subdirectories, e.g. kernel/v2.2, kernel/v2.4 > > and kernel/v2.6, each of which is a submodule that houses these > > versions of Linux kernel source, but only clone Linus's repository > > (as the up-to-late tree has all the necessary history to check out > > these past development tracks). And that should be doable with > > just the main checkout, without any additional worktree (it's just > > the matter of having .git/modules/kernel%2fv2.6/ directory pointed > > by two symlinks from .git/modules/kernel%2fv2.[24], or something > > like that). > > Actually I take the last part of that back. When thought naively > about, it may appear that it should be doable, but because each of > the modules/* directory in the top-level project has to serve as the > $GIT_DIR for each submodule checkout, and the desire is to have > these three directories to have checkout of three different > branches, a single directory under modules/. that is shared among > three submodules would *not* work---they must have separate index, > HEAD, etc. > > Theoretically we should be able to make modules/kernel%2fv2.[24] > additional "worktree"s of modules/kernel%2fv2.6, but given that > these are all "bare" repositories without an attached working tree, > I am not sure how that would supposed to work. Thinking about > having multiple worktrees on a single bare repository makes me head > spin and ache X-<;-) Well the worktree feature would do all this in a safe manner, but the existing feature of just cloning the submodules with a reference pointer to another repository at least dedupes some of the object store, although warnings need to be read carefully. Regarding the worktree, I would think we'd want to have git worktree --recurse-submodules [list, add, etc] that would do a sensible thing for each action on the worktrees, but you seem to propose to have one of the three submodules the main worktree and the other two would be just worktrees of the first. I guess you could just * init/update one of the submodules * add their worktrees manually pointed to where the 2nd and 3rd submodule would go. * make a symbolic link from ln -s .git/modules/<1st>/worktrees/<2nd> .git/modules/<2nd> ln -s .git/modules/<1st>/worktrees/<3rd> .git/modules/<3rd> as then submodule operations should "just work" and by having the worktrees in-place where the submodules are, we'd also have all the protection against overzealous garbage collection, too. Stefan
Re: [PATCH 1/2] store submodule in common dir
On Tue, Aug 14, 2018 at 4:04 PM Junio C Hamano wrote: > > Stefan Beller writes: > > > Signed-off-by: Stefan Beller > > --- > > path.c | 1 + > > 1 file changed, 1 insertion(+) > > > > On Tue, Aug 14, 2018 at 3:27 PM Joakim Tjernlund > > wrote: > >> > >> I am trying to create 3 submodules from the same git repo, each pointing > >> to a different branch. > >> Since the repo is somewhat large, I don't want the 3 submodules to clone > >> the same repo 3 > >> times, I want one clone and then have the 3 submodules to point to > >> different commits. > >> > >> Is this possible? If not, could it be added? > > > > yup. > > > > According to recent discussions, it would be just this patch. > > (plus some unspecified amount of work, TBD). > > > > I thought about proposing something proper later, but here is the WIP patch. > > > > Thanks, > > Stefan > > My understanding of what Joakim wants to do is to have a top-level > project that has three subdirectories, e.g. kernel/v2.2, kernel/v2.4 > and kernel/v2.6, each of which is a submodule that houses these > versions of Linux kernel source, but only clone Linus's repository > (as the up-to-late tree has all the necessary history to check out > these past development tracks). And that should be doable with > just the main checkout, without any additional worktree (it's just > the matter of having .git/modules/kernel%2fv2.6/ directory pointed > by two symlinks from .git/modules/kernel%2fv2.[24], or something > like that). Ah! I misunderstood due to fast reading. For that I think you are interested in the feature added in d92a39590d1 (Add --reference option to git submodule., 2009-05-04), i.e. both the update and add command take the --reference flag that can be pointed at another repository such as an outside clone of these three submodules, so some deduplication will be performed. > Isn't "common_dir" stuff used to decide if each of separate > "worktree" instance (of the superproject) shares ".git/$stuff" > with each other? > > Unless I am grossly misinterpreting the original question, I fail to > see how changing .git/modules to be shared across worktrees possibly > affects anything. I am puzzled... I did misunderstand grossly. Stefan
Re: [PATCH 1/2] store submodule in common dir
Junio C Hamano writes: > My understanding of what Joakim wants to do is to have a top-level > project that has three subdirectories, e.g. kernel/v2.2, kernel/v2.4 > and kernel/v2.6, each of which is a submodule that houses these > versions of Linux kernel source, but only clone Linus's repository > (as the up-to-late tree has all the necessary history to check out > these past development tracks). And that should be doable with > just the main checkout, without any additional worktree (it's just > the matter of having .git/modules/kernel%2fv2.6/ directory pointed > by two symlinks from .git/modules/kernel%2fv2.[24], or something > like that). Actually I take the last part of that back. When thought naively about, it may appear that it should be doable, but because each of the modules/* directory in the top-level project has to serve as the $GIT_DIR for each submodule checkout, and the desire is to have these three directories to have checkout of three different branches, a single directory under modules/. that is shared among three submodules would *not* work---they must have separate index, HEAD, etc. Theoretically we should be able to make modules/kernel%2fv2.[24] additional "worktree"s of modules/kernel%2fv2.6, but given that these are all "bare" repositories without an attached working tree, I am not sure how that would supposed to work. Thinking about having multiple worktrees on a single bare repository makes me head spin and ache X-<;-)
Re: [PATCH 1/2] store submodule in common dir
Stefan Beller writes:
> Signed-off-by: Stefan Beller
> ---
> path.c | 1 +
> 1 file changed, 1 insertion(+)
>
> On Tue, Aug 14, 2018 at 3:27 PM Joakim Tjernlund
> wrote:
>>
>> I am trying to create 3 submodules from the same git repo, each pointing to
>> a different branch.
>> Since the repo is somewhat large, I don't want the 3 submodules to clone the
>> same repo 3
>> times, I want one clone and then have the 3 submodules to point to different
>> commits.
>>
>> Is this possible? If not, could it be added?
>
> yup.
>
> According to recent discussions, it would be just this patch.
> (plus some unspecified amount of work, TBD).
>
> I thought about proposing something proper later, but here is the WIP patch.
>
> Thanks,
> Stefan
My understanding of what Joakim wants to do is to have a top-level
project that has three subdirectories, e.g. kernel/v2.2, kernel/v2.4
and kernel/v2.6, each of which is a submodule that houses these
versions of Linux kernel source, but only clone Linus's repository
(as the up-to-late tree has all the necessary history to check out
these past development tracks). And that should be doable with
just the main checkout, without any additional worktree (it's just
the matter of having .git/modules/kernel%2fv2.6/ directory pointed
by two symlinks from .git/modules/kernel%2fv2.[24], or something
like that).
Isn't "common_dir" stuff used to decide if each of separate
"worktree" instance (of the superproject) shares ".git/$stuff"
with each other?
Unless I am grossly misinterpreting the original question, I fail to
see how changing .git/modules to be shared across worktrees possibly
affects anything. I am puzzled...
>
> diff --git a/path.c b/path.c
> index 34f0f98349a..64c9821b834 100644
> --- a/path.c
> +++ b/path.c
> @@ -115,6 +115,7 @@ static struct common_dir common_list[] = {
> { 1, 1, 1, "logs/HEAD" },
> { 0, 1, 1, "logs/refs/bisect" },
> { 0, 1, 0, "lost-found" },
> + { 0, 1, 0, "modules" },
> { 0, 1, 0, "objects" },
> { 0, 1, 0, "refs" },
> { 0, 1, 1, "refs/bisect" },
[PATCH 1/2] store submodule in common dir
Signed-off-by: Stefan Beller
---
path.c | 1 +
1 file changed, 1 insertion(+)
On Tue, Aug 14, 2018 at 3:27 PM Joakim Tjernlund
wrote:
>
> I am trying to create 3 submodules from the same git repo, each pointing to a
> different branch.
> Since the repo is somewhat large, I don't want the 3 submodules to clone the
> same repo 3
> times, I want one clone and then have the 3 submodules to point to different
> commits.
>
> Is this possible? If not, could it be added?
yup.
According to recent discussions, it would be just this patch.
(plus some unspecified amount of work, TBD).
I thought about proposing something proper later, but here is the WIP patch.
Thanks,
Stefan
diff --git a/path.c b/path.c
index 34f0f98349a..64c9821b834 100644
--- a/path.c
+++ b/path.c
@@ -115,6 +115,7 @@ static struct common_dir common_list[] = {
{ 1, 1, 1, "logs/HEAD" },
{ 0, 1, 1, "logs/refs/bisect" },
{ 0, 1, 0, "lost-found" },
+ { 0, 1, 0, "modules" },
{ 0, 1, 0, "objects" },
{ 0, 1, 0, "refs" },
{ 0, 1, 1, "refs/bisect" },
--
2.18.0.865.gffc8e1a3cd6-goog
