Re: [RFC PATCH 0/7] Implement ref namespaces as a ref storage backend

[Richard Maw]

On Thu, Aug 24, 2017 at 06:17:07PM +0200, Michael Haggerty wrote:
> On Sun, Aug 13, 2017 at 9:36 PM, Richard Maw  wrote:
> > [...]
> > Fortunately the pluggable ref backends work provided an easier starting 
> > point.
> 
> :-) I'm glad my years-long obsession is finally yielding fruit.
> 
> First a general comment about the approach...
> 
> I've always thought that a workable "Git with namespaces" would
> probably look more like git worktrees:
> 
> * One main repository holding all of the objects and all of the
> non-pseudo references.
> 
> * One lightweight directory per namespace-view, holding the
> "core.namespace" config and the pseudorefs. HEAD should probably be
> stored in the main repository (?).

I had pondered something like this as a later extension,
since it's an iterable way of finding out which namespaces exist in a repository
and allows you to explicitly say which namespaces are meant to exist
since looking for all (refs/namespaces/[^/]+/).*HEAD
will also return HEADs that were created by requesting a symbolic ref creation.

Given how many of my problems have been because
namespaced head isn't located at ${GITDIR}/HEAD
I'm not sure how this helps.

> Both the main repository and the namespace-view directories would probably be
> bare, though perhaps somebody can think of an application for allowing
> non-bare repositories.

The only application I've come up with so far is:

1.  Your git server stores configuration in git repositories (gitano does).
2.  The configuration is stored in a git namespace (gitano wants to do this).
3.  Instead of making a clone of the namespace, making changes and pushing that
it would be nicer to create a workspace to make the changes in instead.

It's not a particularly strong application,
since normally you'd be administering by doing a clone remotely,
and even if that doesn't work you can do a clone locally and push that.

Similarly you should be able to create a workspace
and check out the namespace's head yourself.

> Even though this scheme implies the need for extra directories, I
> think that it would make it easier to fix a lot of your problems:
> 
> * Each namespace-view could define its own namespace,
> quasi-permanently. You wouldn't have to pass it via the environment.
> (You might even want to *forbid* changing the namespace via the
> environment or command line!) So fetches and pushes from one namespace
> to another would work correctly.
> 
> * There would be one place for each namespace-view's pseudorefs. You
> wouldn't have to squeeze them into a single reference tree.

I'm not sure that un-namespaced pseudorefs are a problem.
At least for the git server use-case you wouldn't typically have them.
It would be nice for it to just work of course.

> * The main repository would know all of the references of all of the
> namespace views, so maintenance tools like `git gc` would work without
> changes.

The same is/will be true for worktrees from walking the gitdirs
stored in the commondir.

> * The namespace-view directories wouldn't be mistakable for full
> repositories, so tools like `git gc` would refuse to run in them. I
> think this would also make it a little bit harder for reference values
> to "leak" from one namespace-view to another.
> 
> * Remote access could use different paths for different
> namespace-views. The externally-visible path need not, of course, be
> the same as the path of the namespace-view's directory.

This talk of worktrees got me thinking about an alternative implementation
where instead of mangling refs to add and remove prefixes
a separate git directory is maintained, like worktrees,
and the namespace handling in the refs backend would be
to create a files (or reftable) backend for the namespace's git directory.

Having a separate namespave-view would be like having a bare worktree,
which would presumably be just the gitfile.

This would require extra tooling to create namespaces
since you can't just create a sub-namespace by copying some refs
and adding a symbolic ref for HEAD.

However since you can't (or couldn't last time I tried to find a way to)
push a symbolic ref, so the git server needs extra tooling anyway.
You would lose the ability to trivially see sub-namespaces by fetching refs,
but the only use I can think of for that is mirroring for backup.

> By the way, there are certainly still places in the code that don't go
> through the refs API (e.g., the one that Junio found). That's because
> the refs API has still not been used for anything very interesting, so
> the bugs haven't been flushed out. I see you've found some more.
> That's because you're doing something interesting :-)

> > [...]
> > Bugs
> > 
> >
> > Most boil down to how special refs like HEAD are handled.
> >
> > 1.  Logged messages display the namespaced path,
> > which a human may deal with but confuses the test suite.
> 
> I think it's clear that the logged messages should reflect the shorter
> reference 

Re: [RFC PATCH 0/7] Implement ref namespaces as a ref storage backend

[Michael Haggerty]

On Sun, Aug 13, 2017 at 9:36 PM, Richard Maw  wrote:
> Forewarning: I don't consider this work complete
> and am unlikely to find time to finish it any time soon.
> I've mostly sent this because it may include valuable feedback
> on how well the ref storage backends works
> from trying to use it to change how git namespaces work.
>
> Introduction
> 
>
> I work on a git server called Gitano,
> and I'd like to add support for git namespaces to: [...]

Thanks so much for your efforts and your description of the problems
that you faced. That will be really valuable for whomever might follow
up on your work (even if it is you :-) ).

> Unfortunately namespace handling was never implemented for any other part of 
> git
> and at least gitolite makes use of namespaces,
> and will have to work around it not being implemented fully,
> but implementing it more fully will break work-arounds.

I agree that the current namespace feature is not a great foundation
for future work.

> [...]
> Fortunately the pluggable ref backends work provided an easier starting point.

:-) I'm glad my years-long obsession is finally yielding fruit.

First a general comment about the approach...

I've always thought that a workable "Git with namespaces" would
probably look more like git worktrees:

* One main repository holding all of the objects and all of the
non-pseudo references.

* One lightweight directory per namespace-view, holding the
"core.namespace" config and the pseudorefs. HEAD should probably be
stored in the main repository (?). Both the main repository and the
namespace-view directories would probably be bare, though perhaps
somebody can think of an application for allowing non-bare
repositories.

Even though this scheme implies the need for extra directories, I
think that it would make it easier to fix a lot of your problems:

* Each namespace-view could define its own namespace,
quasi-permanently. You wouldn't have to pass it via the environment.
(You might even want to *forbid* changing the namespace via the
environment or command line!) So fetches and pushes from one namespace
to another would work correctly.

* There would be one place for each namespace-view's pseudorefs. You
wouldn't have to squeeze them into a single reference tree.

* The main repository would know all of the references of all of the
namespace views, so maintenance tools like `git gc` would work without
changes.

* The namespace-view directories wouldn't be mistakable for full
repositories, so tools like `git gc` would refuse to run in them. I
think this would also make it a little bit harder for reference values
to "leak" from one namespace-view to another.

* Remote access could use different paths for different
namespace-views. The externally-visible path need not, of course, be
the same as the path of the namespace-view's directory.

By the way, there are certainly still places in the code that don't go
through the refs API (e.g., the one that Junio found). That's because
the refs API has still not been used for anything very interesting, so
the bugs haven't been flushed out. I see you've found some more.
That's because you're doing something interesting :-)

> [...]
> Bugs
> 
>
> Most boil down to how special refs like HEAD are handled.
>
> 1.  Logged messages display the namespaced path,
> which a human may deal with but confuses the test suite.

I think it's clear that the logged messages should reflect the shorter
reference names, and it is the test suite that needs to be fixed.

> 2.  Reflogs for namespaced HEAD are not updated.
>
> This is because resolving HEAD to split the transaction's updates
> to add a log only update to HEAD works by transaction_prepare resolving 
> HEAD
> using its own ref store rather than the main one,
> so the namespace translation isn't performed.
> See split_head_update.
>
> The fix for this may be to move the transaction mangling out of the 
> backend,
> unless it should be implied that every backend implementation
> must be responsible for symbolic ref reflog updates implicitly.

It probably makes sense for the namespace layer to do this step.

I think there is a similar problem with `split_symref_update()`. Here
the problem is trickier, because you don't know how to split the
update until you have locked the symref, but the locking necessarily
has to happen in the main-repo backend. So I think there will be
places where the main-repo backend needs to call back to the namespace
layer for some things, like deciding what reference names to use in
error messages and things.

You'd also want to prevent actions in a namespace-view from affecting
references outside of that namespace. For example, you shouldn't be
able to follow a symref from a namespace-view ref to another reference
in a different namespace. This also implies some cooperation between
the file-level backend and the namespace layer.

I guess it is also clear that symrefs 

Re: [RFC PATCH 0/7] Implement ref namespaces as a ref storage backend

[Richard Maw]

On Tue, Aug 15, 2017 at 10:13:22AM -0700, Junio C Hamano wrote:
> Richard Maw  writes:
> 
> > This is not my first attempt to improve the git namespace handling in git.
> > I tried last year, but it took me so long that all the ref handling code 
> > changed
> > and I would have had to start from scratch.
> >
> > Fortunately the pluggable ref backends work provided an easier starting 
> > point.
> 
> Yeah, I also made an ultra-brief foray into ref backends a few weeks
> ago, and found that Michael did an excellent job identifying the
> building blocks backends may want to implement differently and
> abstracting out major parts of the ref processing.  I also hit some
> of the same issues you mention, e.g. "HEAD" and other funny refs.
> 
> I do suspect that the current GIT_NAMESPACE thing may have outlived
> its usefulness and with the pluggable ref backend thing in place, we
> may want to redesign how support for multiple views into the same
> repository is done.  I do not have a need for such a thing myself,
> but I am glad somebody is looking into it ;-)

It was great to be able to get something mostly working this time around,
which would not have been possible without the pluggable ref backends.

I've no intention of giving up just yet,
though it'll be a while before I can devote significant time to it.

I'll be keeping an eye on the refdb backend.
If it in the process fixes the issues I'd been having that'd be so much better
since I've not got the time or community standing to champion big changes.

Re: [RFC PATCH 0/7] Implement ref namespaces as a ref storage backend

[Junio C Hamano]

Richard Maw  writes:

> This is not my first attempt to improve the git namespace handling in git.
> I tried last year, but it took me so long that all the ref handling code 
> changed
> and I would have had to start from scratch.
>
> Fortunately the pluggable ref backends work provided an easier starting point.

Yeah, I also made an ultra-brief foray into ref backends a few weeks
ago, and found that Michael did an excellent job identifying the
building blocks backends may want to implement differently and
abstracting out major parts of the ref processing.  I also hit some
of the same issues you mention, e.g. "HEAD" and other funny refs.

I do suspect that the current GIT_NAMESPACE thing may have outlived
its usefulness and with the pluggable ref backend thing in place, we
may want to redesign how support for multiple views into the same
repository is done.  I do not have a need for such a thing myself,
but I am glad somebody is looking into it ;-)

Re: [RFC PATCH 0/7] Implement ref namespaces as a ref storage backend

[Stefan Beller]

> Technical description
> =
>
> This patch series adds a new refs backend, stacking on top of the files 
> backend,
> based on whether `core.namespace` is set in git config.

Currently there is another Big Thing getting started in in the refs backend.
https://public-inbox.org/git/CAJo=hJtg0PAVHT1phbArdra8+4LfnEEuaj3fBid==bxkzgh...@mail.gmail.com/
Maybe it is worth looking into that as well? Reftables solve
the problem of scaling (i.e. a repo containing a million refs works
just fine, and fast(!) for reading/writing refs, which is not the case
for packed refs)