Re: [PATCH] Hackathon project: Dumping viewspec

[Philip Martin]

Julian Foad  writes:

> I haven't looked further into this, but what I think is the core of
> the problem is there's no way to generate a WC directory with its
> depth set to 'infinity' but sparse inside, without checking out the
> whole tree.

It might be difficult to do automatically, but there is a trick you can
use if a suitable file is available.  Suppose you want all of

  ^/subversion

but with

  ^/subversion/trunk
  ^/subversion/branches
  ^/subversion/tags

empty, then:

svn co --depth empty http://svn.apache.org/repos/asf/subversion wc
svn up --depth empty wc/trunk wc/branches wc/tags
svn sw --ignore-ancestry ^/subversion/README wc/trunk
svn sw --ignore-ancestry ^/subversion/README wc/branches
svn sw --ignore-ancestry ^/subversion/README wc/tags
svn up --set-depth infinity wc
svn sw --ignore-ancestry --set-depth empty ^/subversion/trunk wc/trunk
svn sw --ignore-ancestry --set-depth empty ^/subversion/branches wc/branches
svn sw --ignore-ancestry --set-depth empty ^/subversion/tags wc/tags

The file that allows this trick doesn't need to be in ^/subversion,
using ^/subversion/trunk/subversion/include/svn_version.h would work
just as well.

This isn't a new idea -- it is based on a trick that was used in the
days before the clients supported sparse working copies, see:

https://gcc.gnu.org/wiki/SvnSetup#Optimize_disk_usage

-- 
Philip

Re: [PATCH] Hackathon project: Dumping viewspec

[Julian Foad]

Branko Čibej wrote:

On 18.03.2018 13:43, Julian Foad wrote:

Branko Čibej wrote:

On 16.03.2018 18:33, Julian Foad wrote:

Suggested exercises for the reader (you, plural):

    * implement a UI to choose the output format (currently 'svn info
--viewspec' is hard-coded to produce that 'format 2')


Why?


To experiment with the two existing implementations of "recreate WC
from a viewspec" -- which are direct command-line cmds and
svn-viewspec.py.


Again, why? Pick one and use it. [...]


Gosh, it was only a suggestion! You are permitted to disregard it.

[...]

We need to design and implement the APIs that we will need to make an
efficient implementation.


I disagree. I've seen no overwhelming argument in favour of requiring
new APIs for this. Your argument is "efficiency" but in terms of what?
[...]


Maybe I'm wrong but we discussed this at the hackathon and didn't see a 
way to do the operation efficiently (data throughput, time taken) with 
the current APIs.


I haven't looked further into this, but what I think is the core of the 
problem is there's no way to generate a WC directory with its depth set 
to 'infinity' but sparse inside, without checking out the whole tree.


When you (or someone) can demonstrate otherwise, then all your other 
suggestions will make sense.


- Julian

Re: [PATCH] Hackathon project: Dumping viewspec

[Branko Čibej]

On 18.03.2018 13:43, Julian Foad wrote:
> Branko Čibej wrote:
>> On 16.03.2018 18:33, Julian Foad wrote:
>>> I implemented an output format compatible with 'svn-viewspec.py' in
>>> r1826990. Then I updated that output format to also support 'switched'
>>> and revisions, in r1826993. This version outputs a header declaring
>>> 'Format: 2', and svn-viewspec.py currently barfs on reading that.
>>>
>>> Suggested exercises for the reader (you, plural):
>>>
>>>    * implement a UI to choose the output format (currently 'svn info
>>> --viewspec' is hard-coded to produce that 'format 2')
>>
>> Why?
>
> To experiment with the two existing implementations of "recreate WC
> from a viewspec" -- which are direct command-line cmds and
> svn-viewspec.py.

Again, why? Pick one and use it. It's not as if there was some subtle
tradeoff here that needs lots of experimenting with two options in order
to reach a decision. Also: svn-viewspec.py uses command-line cmds so
there's no difference in practice (which leads to my other discussion,
see below).

>> 'svn-viewspec.py' is not a supported tool; 'svn info --viewspec'
>> does not exist in 1.10. We don't have any compatibility guarantees to
>> worry about.
>
> Yes.
>
>>>    * update 'svn-viewspec.py' to implement those 'format 2' extensions
>>>
>>>    * start implementing the API functions that 'svn-viewspec.py' needs
>>
>> Again, why? 'svn-viewspec.py' uses the standard command-line tools. It
>> follows that all API functions it needs already exist.
>
> An efficient implementation is not yet possible. New APIs are needed.
> The existing implementation checks out the root, then modifies each
> subdirectory in depth-first order. If the root is depth-infinity, then
> it first checks out everything, then proceeds to delete the unwanted
> bits (and the same applies at any level where depth is reduced).

That is an inefficiency in the implementation of svn-viewspec.py, not a
lack of APIs.

> We need to design and implement the APIs that we will need to make an
> efficient implementation.

I disagree. I've seen no overwhelming argument in favour of requiring
new APIs for this. Your argument is "efficiency" but in terms of what?
The number of client operations? Network data transfer? Or perhaps
efficiency of long-term support for new APIs?

If the issue is data transfer (and local I/O), then the onus is on
svn-viewspec.py to properly analyse the viewspec and construct a minimal
set of client operations to reproduce the working copy structure. If you
invent new APIs to do that instead, you'll have to support them
indefinitely in the client library, and that is, IMO, a less efficient
use of available resources.

> Working with bindings to APIs seems like a good way to do that.

Sure, but not at this point in time. We should only add new APIs as the
existing client(s) need them. On the other hand, svn-viewspec.py is a
good vehicle for developing the necessary methods for efficiently
reproducing working copy topology.

I suggest things should be done in this order:

 1. Fix the most glaring problems in svn-viewspec.py (such as use of
system() instead of popen)
 2. Improve the algorithm for constructing a working copy from the
viewspec, using only existing command-line client operations
 3. Consider what kind of client operations (implying new or improved
APIs) are required to optimise the algorithm further, and implement
those (note: exposing these operations in the command-line client
will give users access to new functionality without having to resort
to using the APIs directly)
 4. Once the above is done, we can then decide to either convert
svn-viewspec.py to use the bindings, or to implement 'svn checkout
--viewspec' instead


"Adding new APIs" seems like an obvious way to proceed, but we have
long-standing counter-examples in our code:

  * the MTCC API, which was experimental and is now private and used by
exactly one tool, and which, as far as I can remember, no-one has
really requested was made public;
  * editor v2 ... unfortunately (by my error in judgement) exposed in
JavaHL but, again, not as a public C API.


-- Brane

Re: [PATCH] Hackathon project: Dumping viewspec

[Julian Foad]

Branko Čibej wrote:

On 16.03.2018 18:33, Julian Foad wrote:

I implemented an output format compatible with 'svn-viewspec.py' in
r1826990. Then I updated that output format to also support 'switched'
and revisions, in r1826993. This version outputs a header declaring
'Format: 2', and svn-viewspec.py currently barfs on reading that.

Suggested exercises for the reader (you, plural):

   * implement a UI to choose the output format (currently 'svn info
--viewspec' is hard-coded to produce that 'format 2')


Why?


To experiment with the two existing implementations of "recreate WC from 
a viewspec" -- which are direct command-line cmds and svn-viewspec.py.



'svn-viewspec.py' is not a supported tool; 'svn info --viewspec'
does not exist in 1.10. We don't have any compatibility guarantees to
worry about.


Yes.


   * update 'svn-viewspec.py' to implement those 'format 2' extensions

   * start implementing the API functions that 'svn-viewspec.py' needs


Again, why? 'svn-viewspec.py' uses the standard command-line tools. It
follows that all API functions it needs already exist.


An efficient implementation is not yet possible. New APIs are needed. 
The existing implementation checks out the root, then modifies each 
subdirectory in depth-first order. If the root is depth-infinity, then 
it first checks out everything, then proceeds to delete the unwanted 
bits (and the same applies at any level where depth is reduced).


We need to design and implement the APIs that we will need to make an 
efficient implementation.


Working with bindings to APIs seems like a good way to do that.

- Julian

Re: [PATCH] Hackathon project: Dumping viewspec

[Branko Čibej]

On 16.03.2018 18:33, Julian Foad wrote:
> Johan Corveleyn wrote:
>> Julian Foad wrote:
>>> We need to look at using the viewspec as input next. Semantically
>>> speaking,
>>> these sorts of things:
>>>
>>> * a way to check out a new WC to match the spec;
>>> * a way to modify an existing WC to match the spec;
>>> * a way to modify/checkout a WC of a *different* branch, to match
>>> the spec
>>> except for its URLs (maybe switch URLs pointing inside this 'branch'
>>> or WC
>>> get adjusted as if they are relative, and other switch URLs stay
>>> absolute?);
>>
>> Ack.
>>
>> FWIW, the viewspec format of svn-viewspec.py [1] (which can serve as
>> inspiration I guess) requires the viewspec to have a "Url", and then a
>> series of path rules. The Url serves as a base url, the path rules are
>> relative to that base url.
>
> I implemented an output format compatible with 'svn-viewspec.py' in
> r1826990. Then I updated that output format to also support 'switched'
> and revisions, in r1826993. This version outputs a header declaring
> 'Format: 2', and svn-viewspec.py currently barfs on reading that.
>
> Suggested exercises for the reader (you, plural):
>
>   * implement a UI to choose the output format (currently 'svn info
> --viewspec' is hard-coded to produce that 'format 2')

Why? 'svn-viewspec.py' is not a supported tool; 'svn info --viewspec'
does not exist in 1.10. We don't have any compatibility guarantees to
worry about.

>   * update 'svn-viewspec.py' to implement those 'format 2' extensions
>
>   * start implementing the API functions that 'svn-viewspec.py' needs

Again, why? 'svn-viewspec.py' uses the standard command-line tools. It
follows that all API functions it needs already exist.

>   * expose those API functions in the Python bindings and convert
> 'svn-viewspec.py' to use them

Frankly, if all necessary functionality exists in the command-line
client, what's the benefit of requiring python bindings for
svn-viewspec.py? What _should_ be done is to stop using the system()
call to invoke the command-line and use the popen module instead (as we
do in our test suite).

-- Brane

Re: [PATCH] Hackathon project: Dumping viewspec

[Julian Foad]

Johan Corveleyn wrote:

Julian Foad wrote:

We need to look at using the viewspec as input next. Semantically speaking,
these sorts of things:

* a way to check out a new WC to match the spec;
* a way to modify an existing WC to match the spec;
* a way to modify/checkout a WC of a *different* branch, to match the spec
except for its URLs (maybe switch URLs pointing inside this 'branch' or WC
get adjusted as if they are relative, and other switch URLs stay absolute?);


Ack.

FWIW, the viewspec format of svn-viewspec.py [1] (which can serve as
inspiration I guess) requires the viewspec to have a "Url", and then a
series of path rules. The Url serves as a base url, the path rules are
relative to that base url.


I implemented an output format compatible with 'svn-viewspec.py' in 
r1826990. Then I updated that output format to also support 'switched' 
and revisions, in r1826993. This version outputs a header declaring 
'Format: 2', and svn-viewspec.py currently barfs on reading that.


Suggested exercises for the reader (you, plural):

  * implement a UI to choose the output format (currently 'svn info 
--viewspec' is hard-coded to produce that 'format 2')


  * update 'svn-viewspec.py' to implement those 'format 2' extensions

  * start implementing the API functions that 'svn-viewspec.py' needs

  * expose those API functions in the Python bindings and convert 
'svn-viewspec.py' to use them



So here too, I suppose a viewspec can contain a "base url", but the
user should be able to override that base url when applying the
viewspec (with checkout, update, switch, export, ...).


Yes, this is true with either format.

- Julian



[1] 
http://svn.apache.org/repos/asf/subversion/trunk/tools/client-side/svn-viewspec.py

Re: [PATCH] Hackathon project: Dumping viewspec

[Johan Corveleyn]

On Thu, Mar 15, 2018 at 11:19 PM, Julian Foad  wrote:
> Johan Corveleyn wrote:
>>
>> On Sun, Nov 26, 2017, Stefan  wrote:
>>>
>>> On 25/11/2017, Stefan Sperling wrote:

 On Fri, Nov 24, 2017, Bert Huijben wrote:
>
> At the Aachen hackathon I promised to write some code to spit out the
> sparse
> definition of a working copy, or in other words some initial dumb
> viewspec
> output.
> $ svn switch --list \SharpSvn\trunk


 Has a new 'svn viewspec' been subcommand considered?
 'switch --list' reminds me of our 'switch --relocate' mistake
 from the past ;)
>>>
>>>
>>> Indeed it was. FWIW I agree there are good arguments for a new viewspec
>>> subcommand.
>>>
>>> The alternative would be to use "svn list --generate-viewspec" and "svn
>>> switch/checkout --use-viewspec < viewspecfile" or something like this.
>>> The obvious downside would be that one subcommand would be used to
>>> generate the viewspec while another one would be used to apply it. I
>>> think Bert brought up other arguments against adding it to "svn list".
>>
>>
>> I prefer adding the "export the viewspec info from this WC" to "svn
>> info", because that's what we already use to obtain info from a
>> working copy (including depth and working revision). Perhaps "svn info
>> -R --viewspec". Let's say this would generate some structured
>> information in a well defined syntax.
>
>
> I have committed Bert's patch, with the command modified to be "svn info
> --viewspec", in http://svn.apache.org/r1826864
>
> Let's see where it leads.

Awesome! Thanks Julian (and Bert of course).

> First I'd like to 'clean up' the formatting a bit to make it easier to read,
> and add some tests.
>
>> The exported info can then be used as optional input for several commands:
>>
>>  svn checkout $URL --apply-viewspec vspecfile.txt
>>  svn update . --apply-viewspec vspecfile.txt
>>  svn switch . --apply-viewspec vspecfile.txt   (perhaps the
>> viewspec contains switched subtrees, which necessitates the 'switch'
>> command to execute)
>
>
> We need to look at using the viewspec as input next. Semantically speaking,
> these sorts of things:
>
> * a way to check out a new WC to match the spec;
> * a way to modify an existing WC to match the spec;
> * a way to modify/checkout a WC of a *different* branch, to match the spec
> except for its URLs (maybe switch URLs pointing inside this 'branch' or WC
> get adjusted as if they are relative, and other switch URLs stay absolute?);

Ack.

FWIW, the viewspec format of svn-viewspec.py [1] (which can serve as
inspiration I guess) requires the viewspec to have a "Url", and then a
series of path rules. The Url serves as a base url, the path rules are
relative to that base url.

So here too, I suppose a viewspec can contain a "base url", but the
user should be able to override that base url when applying the
viewspec (with checkout, update, switch, export, ...).

[1] 
http://svn.apache.org/repos/asf/subversion/trunk/tools/client-side/svn-viewspec.py

-- 
Johan

Re: [PATCH] Hackathon project: Dumping viewspec

[Julian Foad]

Johan Corveleyn wrote:

On Sun, Nov 26, 2017, Stefan  wrote:

On 25/11/2017, Stefan Sperling wrote:

On Fri, Nov 24, 2017, Bert Huijben wrote:

At the Aachen hackathon I promised to write some code to spit out the sparse
definition of a working copy, or in other words some initial dumb viewspec
output.
$ svn switch --list \SharpSvn\trunk


Has a new 'svn viewspec' been subcommand considered?
'switch --list' reminds me of our 'switch --relocate' mistake
from the past ;)


Indeed it was. FWIW I agree there are good arguments for a new viewspec
subcommand.

The alternative would be to use "svn list --generate-viewspec" and "svn
switch/checkout --use-viewspec < viewspecfile" or something like this.
The obvious downside would be that one subcommand would be used to
generate the viewspec while another one would be used to apply it. I
think Bert brought up other arguments against adding it to "svn list".


I prefer adding the "export the viewspec info from this WC" to "svn
info", because that's what we already use to obtain info from a
working copy (including depth and working revision). Perhaps "svn info
-R --viewspec". Let's say this would generate some structured
information in a well defined syntax.


I have committed Bert's patch, with the command modified to be "svn info 
--viewspec", in http://svn.apache.org/r1826864


Let's see where it leads.

First I'd like to 'clean up' the formatting a bit to make it easier to 
read, and add some tests.



The exported info can then be used as optional input for several commands:

 svn checkout $URL --apply-viewspec vspecfile.txt
 svn update . --apply-viewspec vspecfile.txt
 svn switch . --apply-viewspec vspecfile.txt   (perhaps the
viewspec contains switched subtrees, which necessitates the 'switch'
command to execute)


We need to look at using the viewspec as input next. Semantically 
speaking, these sorts of things:


* a way to check out a new WC to match the spec;
* a way to modify an existing WC to match the spec;
* a way to modify/checkout a WC of a *different* branch, to match the 
spec except for its URLs (maybe switch URLs pointing inside this 
'branch' or WC get adjusted as if they are relative, and other switch 
URLs stay absolute?);


- Julian

Re: [PATCH] Hackathon project: Dumping viewspec

[Johan Corveleyn]

On Sun, Nov 26, 2017 at 11:50 PM, Stefan  wrote:
> On 25/11/2017 16:12, Stefan Sperling wrote:
>> On Fri, Nov 24, 2017 at 02:42:05PM +0100, Bert Huijben wrote:
>>>  Hi,
>>>
>>> At the Aachen hackathon I promised to write some code to spit out the sparse
>>> definition of a working copy, or in other words some initial dumb viewspec
>>> output.
>>>
>>> Testing this on a test working copy with
>>> [[
>>> $ svn switch --list \SharpSvn\trunk
>> Has a new 'svn viewspec' been subcommand considered?
>> 'switch --list' reminds me of our 'switch --relocate' mistake
>> from the past ;)
>
> Indeed it was. FWIW I agree there are good arguments for a new viewspec
> subcommand.
>
> The alternative would be to use "svn list --generate-viewspec" and "svn
> switch/checkout --use-viewspec < viewspecfile" or something like this.
> The obvious downside would be that one subcommand would be used to
> generate the viewspec while another one would be used to apply it. I
> think Bert brought up other arguments against adding it to "svn list".

I prefer adding the "export the viewspec info from this WC" to "svn
info", because that's what we already use to obtain info from a
working copy (including depth and working revision). Perhaps "svn info
-R --viewspec". Let's say this would generate some structured
information in a well defined syntax.

The exported info can then be used as optional input for several commands:

svn checkout $URL --apply-viewspec vspecfile.txt
svn update . --apply-viewspec vspecfile.txt
svn switch . --apply-viewspec vspecfile.txt   (perhaps the
viewspec contains switched subtrees, which necessitates the 'switch'
command to execute)

Maybe the viewspec could also be read from standard input (by using '-'):

svn info -R --viewspec WC1 | svn update WC2 --viewspec -

?

A separate 'viewspec' subcommand might also be interesting, but I'm
not sure what we'll be using it for exactly (only for exporting
viewspecs from WCs?).
I think we need to think about two different things:

(1) Export / describe the viewspec of a specific working copy.
(2) Apply a viewspec to a working copy (or when creating a working copy)

IMHO, the "applying" needs to be part of several commands that create
/ modify working copies (checkout, update, switch, perhaps export).

-- 
Johan

Re: [PATCH] Hackathon project: Dumping viewspec

[Stefan]

On 25/11/2017 16:12, Stefan Sperling wrote:
> On Fri, Nov 24, 2017 at 02:42:05PM +0100, Bert Huijben wrote:
>>  Hi,
>>
>> At the Aachen hackathon I promised to write some code to spit out the sparse
>> definition of a working copy, or in other words some initial dumb viewspec
>> output.
>>
>> Testing this on a test working copy with
>> [[
>> $ svn switch --list \SharpSvn\trunk
> Has a new 'svn viewspec' been subcommand considered?
> 'switch --list' reminds me of our 'switch --relocate' mistake
> from the past ;)

Indeed it was. FWIW I agree there are good arguments for a new viewspec
subcommand.

The alternative would be to use "svn list --generate-viewspec" and "svn
switch/checkout --use-viewspec < viewspecfile" or something like this.
The obvious downside would be that one subcommand would be used to
generate the viewspec while another one would be used to apply it. I
think Bert brought up other arguments against adding it to "svn list".

Regards,
Stefan

Re: [PATCH] Hackathon project: Dumping viewspec

[Stefan Sperling]

On Fri, Nov 24, 2017 at 02:42:05PM +0100, Bert Huijben wrote:
>   Hi,
> 
> At the Aachen hackathon I promised to write some code to spit out the sparse
> definition of a working copy, or in other words some initial dumb viewspec
> output.
> 
> Testing this on a test working copy with
> [[
> $ svn switch --list \SharpSvn\trunk

Has a new 'svn viewspec' been subcommand considered?
'switch --list' reminds me of our 'switch --relocate' mistake
from the past ;)

[PATCH] Hackathon project: Dumping viewspec

[Bert Huijben]

Hi,

At the Aachen hackathon I promised to write some code to spit out the sparse
definition of a working copy, or in other words some initial dumb viewspec
output.

Testing this on a test working copy with
[[
$ svn switch --list \SharpSvn\trunk
]]
I can now see output like
[[
svn checkout https://ctf.open.collab.net/svn/repos/sharpsvn/trunk@4003
/SharpSvn/trunk
svn update -r 4004
/SharpSvn/trunk/imports/patches/openssl/201603-capi-sha-openssl-1.0.2.patch
svn switch ^/branches/1.800x/contrib@4004 /SharpSvn/trunk/contrib
svn update -r 0 /SharpSvn/trunk/LICENSE.txt
svn update -r 4004 /SharpSvn/trunk/src/SharpPlink --set-depth files
svn update -r 4004 /SharpSvn/trunk/src/SharpSvn --set-depth files
svn update --set-depth files /SharpSvn/trunk/src/SharpSvn/Commands
svn update --set-depth exclude /SharpSvn/trunk/build.info
]]

So this new command spits out the shell commands necessary to create a new
working copy with exactly the same layout and revisions as the existing
working copy.

This could be the starting point for an improved format, or for creating
scripts that transform this in something more useful, e.g. by filtering out
unnecessary details.


I put the current code in a patch file attached to this message to allow
discussing this feature before putting it on trunk.

Bert




viewspec-dump.patch
Description: Binary data