Thanks for this feedback, Bert. Several important points are emerging. The first thing to get straight is what future we want for file externals. File externals should behave like directory externals. The directory externals system is far from a perfect solution to the high-level challenges that users try to solve, but it has a lot of precedent behind it now and is at least fairly logical. Despite us having released file externals in 1.6 with several differences from directory externals, I think absolutely all those differences should be regarded as bugs and we should be bringing them back into line.
(If we do know of any way in which file externals behave "better" than directory externals, that should be considered as a potential future improvement for externals in general.) In order to know what we're dealing with, we need to check and write down the current behavioural differences, but in re-implementing file externals handling, it's unquestionably the "directory externals" behaviour that we need to be heading for. On Fri, 2010-08-20, Bert Huijben wrote: > > -----Original Message----- > > From: Julian Foad [mailto:julian.f...@wandisco.com] > > * An 'external' or 'switched' WC node is an immediate child of a > > versioned WC directory. [3] Changed to "parent or grandparent etc.". > > * An 'external' or 'switched' WC node is attached to a repository > > location other than the one implied by its WC parent directory. > > An external directory is not connected to it's parent. (You don't get > it in the list of children using entries and/or nodes). To the parent > it is just an unversioned directory, but if you look inside you find a > working copy root that might (accidentally) be from the same > repository. And it can be a few levels deeper than just an immediate > child. OK: an external directory is a WC root. I hadn't contemplated the importance of that until now, but I think we can almost *define* the required behaviour of externals in terms of that statement. "An external directory is a nested WC that libsvn_client knows something about: namely it knows how and when to create it and to use it." I'm suggesting "libsvn_client" here as an arbitrary but important level of abstraction where a "complex WC" is implemented, and below which there is a concept of "single WC" (single wc_id) that has no notion of "externals". We might choose to put this "complex WC" concept into libsvn_wc instead, but it's useful for the conceptual distinction to exist somewhere in the APIs, and this is probably a very good candidate for a chunk of complexity that we would do well to keep out of libsvn_wc. Let's see where that approach will take us. [...] > > * Client operations treat the node as part of its parent WC in some > > ways, and treat it as special in other ways, depending on the > > command and on user-specified options. > > Yes, > > Well, there are no real operations on external directories as they are > just unversioned to the parent working copy. At least two client operations, "svn update" and "svn status", recognize externals and descend into them. > External files... specialized code... specialized code... specialized > code. > > External files are part of the parent working copy, so things like > (recursive) locks and things apply on them. But these locks don't > apply on directory externals as they are in a different working copy. [...] > > How should we represent external files and external dirs in the WC in a > > unified manner? Do we need to explicitly mark them as 'external'? What > > differences in behaviour should 'external' nodes and 'switched' nodes > > exhibit within the WC? (Heh, those were just meant to be examples of questions that this whole topic might help to answer.) > One option would be to pull file externals out of their parent working > copy and make them their own working copy root. We can handle this in > WC-DB. (Create a different wcroot id, defining the file to be its own > root. Fix the update handling and we are done). Yes! If we are committed to making file externals like directory externals, we need to re-specify a file external as "a separate WC consisting of a single file". And when we try to do that, we have to re-think our notion of a "WC root". A WC root will no longer always be a directory and will not always be able to hold its own .svn admin area inside it. Recall that the ability to check out and work on a single file from a repository has been requested quite a few times in the life of Subversion. That concept of a single-file checkout is alien to WC-1 and to some assumptions built on our old implementation of WCs, but is not in itself at all alien to the version control concepts of Subversion. So a single-file WC is not specifically an "externals" concept, but rather a general concept that will be useful in implementing externals. > > Differences in WC State > > ======================= > > > > This table documents the differences in WC state at a high level. > > > > +----------------------+----------------------+----------------------+ > > | | switched | external | > > +----------------------+----------------------+----------------------+ > > | | | | > > | WC path affected | Existing child | Non-existent child | > > File external: existing child. > > | | | | > > | Target URL | Same repo | Same or other repo | > File external: same repo > > | | | | > > | Pinned to a revision | No [2] | Optional; recorded | > > | | | in 'svn:externals' | > File external: recorded on the node itself. Are you saying the pin revision is recorded on the node itself? As well as in the 'svn:externals' property value of course? > (And it's currently impossible to find the svn:externals definition > for a specific external without looking at all it's ancestor > directories) Heh, that's a nuisance but not a logical difficulty. > > | | | | > > | Initial depth | Set by "switch" cmd | Infinity | > > | | | | > > +----------------------+----------------------+----------------------+ > > > > ### Anything else? > > > > > > Differences in Client Commands (Current behaviour) > > ================================================== > > > > The following differences occur when a command encounters an 'external' > > or 'switched' node during its operation. This table is to document the > > current (1.6.x) behaviour. > > > > ### We'll need some test scripts to help gather this data. > > > > +----------------------+----------------------+----------------------+ > > | | switched | external | > > +----------------------+----------------------+----------------------+ > > | | | | > > | recursive descent, | yes | optional | > > | in general | | | > > | | | | > > | status | report & descend | report [& descend] | > > | | | | > > | propget/list/etc. | ? | ? | > yes | no > > | | | | > > | diff | ? | ? | > yes | no > > | | | | > > | commit | descend | [descend?] | > Yes | no (except for some buggy behavior relying on passing a limiting depth) > > | | | | > > | update | ? | ? | > Yes | optional > > | | | | > > | switch | Can switch a WC root | No UI exists for an | > > | | but it's not then | external outside | > > | | recorded as such. | of a WC. | > (externals definitions are normally updated after a switch by handling > the svn:externals property changes) > > | | | | > > | ... | | | > > File externals are in all these cases handled as switch. > > | | | | > > +----------------------+----------------------+----------------------+ [...] Updated in r988207. Thanks. > > Footnotes > > ========= > > > > [2] The ability for "svn update -rX" to "pin" that part of the WC at > > rX has been requested by users and seems like a reasonable (clean, > > moderately useful) feature. > > I wouldn't want just using -r to pin the revision, but I certainly > like the idea of a --pin option. And if we generalize this we can also > use it to deny commits on pinned nodes. (Which is currently a feature > request for file externals with a fixed revision) Right. We *could* implement features such as denying commits on pinned (file) externals right now, without generalizing the pinning in that way, but it would be cleaner that way. However, if we agree (as I think we do) that all externals should be separate WCs, then that particular problem becomes less of an issue because commit won't descend into them, at least not automatically. > > [4] A possible exception is that "update" should obey the "pinned > > revision number" if present. But that need not necessarily be an > > exception: we might choose to define that functionality on all WC > > nodes but only use it on 'external' nodes. > > See my answer on [2]. I like this generalized pinned idea as option to > replace our checks on file externals. > > (First thing that comes to mind is how it would interact with 'svn > switch on some parent') That makes me think: the concept of pinning a WC subtree isn't just "this needs to stay at revision R" but rather "this needs to stay pointing to $REPOS/RELPATH at revision R". And, moving mentally sideways from the idea of "pinning to a revision" towards "pinning to a tag", there is the important variant the "floating tag": "this needs to stay pointing to $REPOS/RELPATH at HEAD" which is pretty much exactly what a revisionless svn:external property says. But we don't have a strong requirement for this general pinning, and the client can already tell from the svn:externals property what it should do in the case of an external. I can't think of how it would be *better* to implement it as a general WC feature at this time. The focus of this thread is swinging towards how to unify file externals with directory externals. And we're starting to find some powerful, independent concepts which we can combine into a design, rather than treating every requirement as a separate local problem. So far, we've considered: * The paired concepts of - a "single WC" having a single wc_id, and - "externals" as a way for a higher layer to link together a set of "single WCs". * The single-file checkout. - useful in itself - could be used to enable file externals * The concept of "pinning" to a specific revision or node-rev. - could be useful in itself - could be used as a building block for pinned externals - no need to implement this generalization at the moment; it's just a thought to keep in mind. - Julian
