Re: Svn server images / appliances / packages

[Daniel Shahaf]

Nathan Hartman wrote on Fri, 10 Jan 2020 15:07 +00:00:
> But: The reference system would be one particular configuration known 
> to work, with all the kinks ironed out.

Isn't that just

cd subversion/tests/cmdline && ./davautocheck.sh --no-tests

?

Re: "svn list -v" column alignment issue

[Johan Corveleyn]

On Fri, Jan 10, 2020 at 3:28 PM Vincent Lefevre  wrote:
> On 2020-01-08 13:12:21 +, Daniel Shahaf wrote:
> > Vincent Lefevre wrote on Wed, Jan 08, 2020 at 10:28:01 +0100:
> > > On 2020-01-07 16:10:52 +, Daniel Shahaf wrote:
> > > > Vincent Lefevre wrote on Tue, Jan 07, 2020 at 16:17:10 +0100:
> > > > > On 2019-12-23 06:35:08 +, Daniel Shahaf wrote:
> > > > > > Two things are not immediately clear to me:
> > > > > >
> > > > > > - This info is only needed by the cmdline client, not by other
> > > > > >   library users. This suggests the API should be generic. A
> > > > > >   per-client cache, maybe? There's already a
> > > > > >   svn_client_ctx_t::client_name so it's not unprecedented.
> > > > >
> > > > > Well, you don't know what the library users will do. Perhaps the cache
> > > > > will benefit them too. The value can be used by them or not.
> > > >
> > > > I think you've missed my point.  I'm saying we should try to design the 
> > > > cache
> > > > API in a way that will allow it to be used not only for 'svn ls -v' but 
> > > > also
> > > > for other things.  The API does not exist to serve the cmdline client; 
> > > > it
> > > > exists to serve _all_ svn clients.
> > >
> > > I entirely agree. But I'm still confused by your first sentence above
> > > "This info is only needed by the cmdline client, not by other library
> > > users." or perhaps you misplaced the "not"?
> >
> > The length of the longest author name is needed by 'svn ls -v'.
> >
> > The length of the longest author name is unlikely to be needed by
> > other consumers of the svn_client_* API.
>
> A reimplementation of "svn ls -v" or "svn blame" may find this useful
> information. Graphical clients may be interested in a cached full list
> of authors to compute the size of the corresponding column.

Since both 'svn ls -v' and 'svn blame' can be called on a repository
URL, I don't really see the point in caching the max(length(author))
of the working copy.

The output from a URL would still be mis-aligned. It would make the
presentation inconsistent between calling it on a working copy and on
a URL.

Just my 2 cents ...
-- 
Johan

Re: 20th anniversary press?

[Doug Robinson]

Nathan:

Let me see what we can do.

Cheers.

Doug

On Fri, Jan 10, 2020 at 10:20 AM Nathan Hartman 
wrote:

> On Fri, Jan 10, 2020 at 10:07 AM Doug Robinson 
> wrote:
>
>> Folks:
>>
>> On Mon, Jan 6, 2020 at 9:21 AM Nathan Hartman 
>> wrote:
>>
>>> It's super cool that Subversion's birthday is Leap Day 2000. [1]
>>>
>>
>> That's amazing!  First commit on a leap day?!  Cool.
>>
>
> Not just any leap day, one that happens only every 400 years.
>
> Let me know if WANdisco can help the celebration in any way.
>>
>
> We need testimonials, preferably from highly visible corporate users;
> anything that answers the question "Who uses Subversion?" for the press
> release.
>
> If you can help us secure some testimonials, that will be very helpful and
> much appreciated!
>
> Cheers,
> Nathan
>
>
>

-- 
*DOUGLAS B ROBINSON* SENIOR PRODUCT MANAGER

T +1 925 396 1125
*E* doug.robin...@wandisco.com

-- 


* *

**The *LiveData* Company
*Find out more 
*wandisco.com *



 



THIS MESSAGE AND ANY ATTACHMENTS 
ARE CONFIDENTIAL, PROPRIETARY AND MAY BE PRIVILEGED
*


If this message was 
misdirected, WANdisco, Inc. and its subsidiaries, ("WANdisco") does not 
waive any confidentiality or privilege. If you are not the intended 
recipient, please notify us immediately and destroy the message without 
disclosing its contents to anyone. Any distribution, use or copying of this 
email or the information it contains by other than an intended recipient is 
unauthorized. The views and opinions expressed in this email message are 
the author's own and may not reflect the views and opinions of WANdisco, 
unless the author is authorized by WANdisco to express such views or 
opinions on its behalf. All email sent to or from this address is subject 
to electronic storage and review by WANdisco. Although WANdisco operates 
anti-virus programs, it does not accept responsibility for any damage 
whatsoever caused by viruses being passed.

Re: 20th anniversary press?

[Nathan Hartman]

On Fri, Jan 10, 2020 at 10:07 AM Doug Robinson 
wrote:

> Folks:
>
> On Mon, Jan 6, 2020 at 9:21 AM Nathan Hartman 
> wrote:
>
>> It's super cool that Subversion's birthday is Leap Day 2000. [1]
>>
>
> That's amazing!  First commit on a leap day?!  Cool.
>

Not just any leap day, one that happens only every 400 years.

Let me know if WANdisco can help the celebration in any way.
>

We need testimonials, preferably from highly visible corporate users;
anything that answers the question "Who uses Subversion?" for the press
release.

If you can help us secure some testimonials, that will be very helpful and
much appreciated!

Cheers,
Nathan

Re: Svn server images / appliances / packages

[Nathan Hartman]

On Fri, Jan 10, 2020 at 9:40 AM Julian Foad  wrote:

> Branko Čibej wrote:
> > Nathan Hartman wrote:
> >> If there is a defined "reference system," that makes it possible to
> >> focus on making that configuration as turn-key as possible, and also
> >> to document it as well as possible.
>
> If I understand correctly, the point is to concentrate on one system in
> our efforts to improve the experience of running Subversion, so that we
> avoid diluting our efforts with discussing and accommodating the
> differences between systems.


I think you understand my point but just to be extra clear, I'd like to
emphasize that I'm *not* suggesting to let other systems fall by the
wayside. We would continue having buildbots on different platforms, support
building on different platforms, etc. You would still have infinitely many
possible ways to configure and run a Subversion server, including locally,
svnserve, and httpd.

But: The reference system would be one particular configuration known to
work, with all the kinks ironed out. It would be the system that
documentation writers of a "Subversion Administration" manual could focus
on without diluting their exposition. If someone has trouble with their
system, we could ask "what are you doing differently from the reference
system?" and they'd have a reference point for comparison.

Cheers,
Nathan

Re: 20th anniversary press?

[Doug Robinson]

Folks:

On Mon, Jan 6, 2020 at 9:21 AM Nathan Hartman 
wrote:

> Daniel Shahaf  wrote:
> >
> > We're coming up on the 20th anniversary of the first commit to
> > Subversion's CVS repository:
> >
> > [[[
> > % svn log ^/subversion -r 0:HEAD -l 3 | vipe
> > 
> > r836421 | kfogel | 2000-02-29 20:32:07 -0600 (Tue, 29 Feb 2000) | 2 lines
> > Changed paths:
> > A /subversion/trunk/src
> > A /subversion/trunk/src/LICENSE
> >
> > initial checkin
> >
> > 
> > % ]]]
> >
> > Sounds like an excuse to do a press release?  Or something to mention in
> > the announcement of 1.14.0-LTS, at least?
>
> It's super cool that Subversion's birthday is Leap Day 2000. [1]
>

That's amazing!  First commit on a leap day?!  Cool.


> I think it's a great idea to celebrate with a press release.
>

Let me know if WANdisco can help the celebration in any way.

Cheers.

Doug


>
> Let me know how I can help.
>
>
> [1] That day is cool for a variety of reasons. It's the year 2000. It
> has a leap day, contrary to most years divisible by 100 (because it's
> also divisible by 400 [2]). And it's the birthday of the best version
> control system ever!
>
> [2] https://www.timeanddate.com/date/leapyear.html
>
> Nathan
>


-- 
*DOUGLAS B ROBINSON* SENIOR PRODUCT MANAGER

T +1 925 396 1125
*E* doug.robin...@wandisco.com

-- 


* *

**The *LiveData* Company
*Find out more 
*wandisco.com *



 



THIS MESSAGE AND ANY ATTACHMENTS 
ARE CONFIDENTIAL, PROPRIETARY AND MAY BE PRIVILEGED
*


If this message was 
misdirected, WANdisco, Inc. and its subsidiaries, ("WANdisco") does not 
waive any confidentiality or privilege. If you are not the intended 
recipient, please notify us immediately and destroy the message without 
disclosing its contents to anyone. Any distribution, use or copying of this 
email or the information it contains by other than an intended recipient is 
unauthorized. The views and opinions expressed in this email message are 
the author's own and may not reflect the views and opinions of WANdisco, 
unless the author is authorized by WANdisco to express such views or 
opinions on its behalf. All email sent to or from this address is subject 
to electronic storage and review by WANdisco. Although WANdisco operates 
anti-virus programs, it does not accept responsibility for any damage 
whatsoever caused by viruses being passed.

Re: Svn server images / appliances / packages

[Julian Foad]

Branko Čibej wrote:

Nathan Hartman wrote:
If there is a defined "reference system," that makes it possible to 
focus on making that configuration as turn-key as possible, and also 
to document it as well as possible.


If I understand correctly, the point is to concentrate on one system in 
our efforts to improve the experience of running Subversion, so that we 
avoid diluting our efforts with discussing and accommodating the 
differences between systems.


Discussing and accommodating other systems should and can still happen, 
but it can happen separately (different discussion threads, different 
times, different people) so as not to distract (so much) from that 
primary effort.


That sounds like a good plan.

A reference system [...] is a [...] configuration with the 
best probability of success, the best documentation, and the best 
understanding of the potential pitfalls and their solutions or 
workarounds. [...] It could be docker based, or not.


On most "sane" OSes, administrators are used to the way their upstream 
packager sets up things. [...] 
The most we could do would be for such cases would be to provide 
example configurations for, e.g., LDAP/AD integration etc. That's really 
a part of end-user documentation and should go into the wiki.


That seems to miss the point.  A native installation on any "sane" OS 
would make a perfectly adequate reference platform, as would a Docker 
installation.


The OS that would be best served by a reference installation (or 
convenience binaries, if you prefer) would be, not so surprisingly, 
Windows. [...]


I'm not quite sure how to interpret that.  The primary goal is not to 
benefit the chosen platform (although that would of course happen) but 
rather to make it easy for contributors to concentrate on something. 
The system to concentrate on therefore needs to be one that a large 
subset of contributors can work with.  I personally am here because I 
support FOSS in general.  Although I am willing to help anyone who wants 
to make Subversion also work well on Windows, I would not support 
favouritism toward any proprietary software.


From my limited experience, it seems the most suitable "reference 
system" options, ones that are open and that a large sector people in 
this space are likely able to work with, would be:


  * Ubuntu (server, latest LTS)
  * Docker (probably using an Alpine image, or maybe Ubuntu)

- Julian

Re: 20th anniversary press?

[Nathan Hartman]

On Thu, Jan 9, 2020 at 2:28 PM Nathan Hartman 
wrote:

> On Thu, Jan 9, 2020 at 9:00 AM Daniel Shahaf 
> wrote:
>
>> Nathan Hartman wrote on Mon, Jan 06, 2020 at 09:21:33 -0500:
>> > Daniel Shahaf  wrote:
>> > >
>> > > We're coming up on the 20th anniversary of the first commit to
>> > > Subversion's CVS repository:
>> > >
>> > > [[[
>> > > % svn log ^/subversion -r 0:HEAD -l 3 | vipe
>> > >
>> 
>> > > r836421 | kfogel | 2000-02-29 20:32:07 -0600 (Tue, 29 Feb 2000) | 2
>> lines
>> > > Changed paths:
>> > > A /subversion/trunk/src
>> > > A /subversion/trunk/src/LICENSE
>> > >
>> > > initial checkin
>> > >
>> > >
>> 
>> > > % ]]]
>> > >
>> > > Sounds like an excuse to do a press release?  Or something to mention
>> in
>> > > the announcement of 1.14.0-LTS, at least?
>> >
>> >
>> > It's super cool that Subversion's birthday is Leap Day 2000. [1]
>> >
>> > I think it's a great idea to celebrate with a press release.
>> >
>> > Let me know how I can help.
>>
>> Would you like to liaise with Sally / press@ about this?
>
>
> Sure. I'll be happy to.
>

According to Sally, for the press release to be meaningful, we should
secure some testimonials from highly-visible users, preferably corporate.
Apparently the first question the media asks is "who uses you."

It's up to us to secure those testimonials. This is pretty important. The
biggest thing we can do to gain more help around here is to attract more
interest and awareness.

We have corporate (and even government) customers who have posted questions
to users@. I propose to ask there. I'm asking here first: Please ask (very
nicely) any contacts you may have.

Thanks,
Nathan

Re: "svn list -v" column alignment issue

[Vincent Lefevre]

On 2020-01-08 13:12:21 +, Daniel Shahaf wrote:
> Vincent Lefevre wrote on Wed, Jan 08, 2020 at 10:28:01 +0100:
> > On 2020-01-07 16:10:52 +, Daniel Shahaf wrote:
> > > Vincent Lefevre wrote on Tue, Jan 07, 2020 at 16:17:10 +0100:
> > > > On 2019-12-23 06:35:08 +, Daniel Shahaf wrote:
> > > > > Two things are not immediately clear to me:
> > > > > 
> > > > > - This info is only needed by the cmdline client, not by other
> > > > >   library users. This suggests the API should be generic. A
> > > > >   per-client cache, maybe? There's already a
> > > > >   svn_client_ctx_t::client_name so it's not unprecedented.
> > > > 
> > > > Well, you don't know what the library users will do. Perhaps the cache
> > > > will benefit them too. The value can be used by them or not.
> > > 
> > > I think you've missed my point.  I'm saying we should try to design the 
> > > cache
> > > API in a way that will allow it to be used not only for 'svn ls -v' but 
> > > also
> > > for other things.  The API does not exist to serve the cmdline client; it
> > > exists to serve _all_ svn clients.
> > 
> > I entirely agree. But I'm still confused by your first sentence above
> > "This info is only needed by the cmdline client, not by other library
> > users." or perhaps you misplaced the "not"?
> 
> The length of the longest author name is needed by 'svn ls -v'.
> 
> The length of the longest author name is unlikely to be needed by 
> other consumers of the svn_client_* API.

A reimplementation of "svn ls -v" or "svn blame" may find this useful
information. Graphical clients may be interested in a cached full list
of authors to compute the size of the corresponding column.

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: [#4843] svn wc verify -- pristine files consistency check, and possibly repair

[Johan Corveleyn]

On Fri, Jan 10, 2020 at 11:27 AM Julian Foad  wrote:
>
> Filed as http://subversion.apache.org/issue/4843 .
>
> We could start by defining more precisely what it needs to do.
>
> Some aims, in order from highest priority:
>* check if any pristine file's content is corrupted (according to its
> filename hash)
>  - report and rename/delete corrupted pristines
>* check if any pristines are missing (according to wc.db)
>* fetch missing (or corrupted) pristines from the repository
>* verify wc.db 'pristine' table entries against other tables
>
> Checking for content corruption by recalculating the checksums is going
> to be slow -- there is no getting away from that -- so this most
> important check is probably going to be the last one we run, and we may
> choose to make it optional.  That's fine.
>
> We could check quickly:
>* for each pristine file listed in the DB:
>  - file is present
>  - file size matches the DB
>  - file mod-time matches the DB
>
> The existing 'cleanup' implementation contains a function
> 'pristine_cleanup_wcroot' which has in its doc string:
>
> [[[
>TODO: Ideas for possible extra clean-up operations:
>
>* Check and correct all the refcounts.  Identify any rows missing
>  from the 'pristine' table.  [...]
>
>* Check the checksums.  (Very expensive to check them all, so find
>  a way to not check them all.)
>
>* Check for pristine files missing from disk but referenced in the
>  'pristine' table.
>
>* Repair any pristine files missing from disk and/or rows missing
>  from the 'pristine' table and/or bad checksums.  Generally
>  requires contacting the server, so requires support at a higher
>  level than this function.
>
>* Identify any pristine text files on disk that are not referenced
>  in the DB, and delete them.
> ]]]
>
> The refcounts are references within the DB from nodes to the 'pristines'
> table.  They are enforced by SQLite with 'REFERENCES' clauses in the
> schema, though I saw one comment somewhere saying this was "in debug
> builds" so we might want to double-check.
>
> I am not aware of problems in the consistency of the DB tables, so I
> don't think checking that is a priority.  Though I don't have hard
> evidence, from problems reported over the years I think corrupted and
> missing pristine files on disk is the main concern.

Ah, how time flies ... I first started reporting / asking about
missing and corrupted pristines shortly after WC-NG was released in
1.7, because at that point you couldn't "fix" a broken WC anymore by
deleting the affected subdir. It's happened rarely in recent years,
but back then I've had to fix such corruptions regularly for
colleagues that ran into them. I just wanted to say: big +1 for taking
this on.

FWIW, some threads for historians:

https://svn.haxx.se/dev/archive-2011-07/0001.shtml ("wc-ng and
recoverability of corrupt wc's")
Before the final release of wc-ng I started asking about this problem
-- Philip and Daniel then already suggested two potential ways for
manual fixing: using "svn up -r0 $corrupted" and using "svn up
--set-depth exclude $corrupted".

https://svn.haxx.se/users/archive-2012-03/0458.shtml ("svn 1.7: how to
recover from a lost pristine file")
Some real-world experience. The wc was also stuck with a non-empty
work_queue ("E155037: Previous operation has not finished; run
'cleanup' if it was interrupted" -- where cleanup reports the same
error). Was fixed by first "manually emptying the work_queue in
wc.db", after which "svn up --set-depth exclude" fixed it.

https://svn.haxx.se/dev/archive-2012-06/0185.shtml: some further
feedback to devs after we discussed this issue a bit during the 2012
Hackathon. It also mentions another "fix" strategy, because the other
two didn't work: "in wc.db, set the presence to "not-present" in
NODES, and run 'svn update'".

https://svn.haxx.se/dev/archive-2012-09/0304.shtml ("SmartSVN (by
Wandisco) - repair working copy feature")
The (commercial) product SmartSVN had a feature listed as "Guided
fixing of rare working copy problems" in their professional version
(still listed on their https://www.smartsvn.com/compare-editions/
page). It has saved me and my colleagues a couple of times back then.
It did things like correcting checksums, refcounts, recovering missing
/ corrupt pristines, ...

https://svn.haxx.se/dev/archive-2013-04/0426.shtml ("Pristine text
missing - cleanup doesn't work")
Some more real-world feedback by Julian :-).


Obviously it would be much better if svn could detect and fix such
corruptions itself, instead of having to jump through (potentially
dangerous) hoops :-).

-- 
Johan

Re: Svn server images / appliances / packages

[Branko Čibej]

On 10.01.2020 06:14, Nathan Hartman wrote:
> On Thu, Jan 9, 2020 at 11:05 AM Julian Foad  > wrote:
>
> Branko Čibej wrote:
> > [... Docker...] as much a PITA as maintaining any other server. The
> > installation step is the *least* of your worries.
>
> Well, that's a good point.  Maybe the goal of making svn more easily
> deployable is better served by doing something towards making it more
> set-and-forget, removing pitfalls and unnecessary complexities...
>
> Any more concrete thoughts in that direction, anyone?
>
>
> Anything we can do to make Subversion more deployable is potentially
> more people attracted to the project as contributors.
>
> One thought is a "reference system." That would comprise a specific
> hardware platform, specific OS, dependencies, and configuration.
>
> If there is a defined "reference system," that makes it possible to
> focus on making that configuration as turn-key as possible, and also
> to document it as well as possible.
>
> A reference system doesn't mean that everyone is forced to run their
> Subversion server that way, but it is a way of saying that if you're
> between hobbyist and enterprise, here's the configuration with the
> best probability of success, the best documentation, and the best
> understanding of the potential pitfalls and their solutions or
> workarounds.
>
> Now, I recognize that everyone has their favorite OS etc, so that
> might make it difficult to reach consensus on what that "reference
> system" should be.
>
> It could be docker based, or not.

On most "sane" OSes, administrators are used to the way their upstream
packager sets up things. For example, on anything derived from Debian,
you'll have a structured /etc/apache2 and installing libapache2-mod-svn
will put config files with the necessary incantations in the expected
places. The most we could do would be for such cases would be to provide
example configurations for, e.g., LDAP/AD integration etc. That's really
a part of end-user documentation and should go into the wiki.

The OS that would be best served by a reference installation (or
convenience binaries, if you prefer) would be, not so surprisingly,
Windows. However I'd hesitate to provide those without co-ordination
with the httpd and APR projects.

-- Brane

Re: [#4843] svn wc verify -- pristine files consistency check, and possibly repair

[Julian Foad]

Filed as http://subversion.apache.org/issue/4843 .

We could start by defining more precisely what it needs to do.

Some aims, in order from highest priority:
  * check if any pristine file's content is corrupted (according to its 
filename hash)

- report and rename/delete corrupted pristines
  * check if any pristines are missing (according to wc.db)
  * fetch missing (or corrupted) pristines from the repository
  * verify wc.db 'pristine' table entries against other tables

Checking for content corruption by recalculating the checksums is going 
to be slow -- there is no getting away from that -- so this most 
important check is probably going to be the last one we run, and we may 
choose to make it optional.  That's fine.


We could check quickly:
  * for each pristine file listed in the DB:
- file is present
- file size matches the DB
- file mod-time matches the DB

The existing 'cleanup' implementation contains a function 
'pristine_cleanup_wcroot' which has in its doc string:


[[[
  TODO: Ideas for possible extra clean-up operations:

  * Check and correct all the refcounts.  Identify any rows missing
from the 'pristine' table.  [...]

  * Check the checksums.  (Very expensive to check them all, so find
a way to not check them all.)

  * Check for pristine files missing from disk but referenced in the
'pristine' table.

  * Repair any pristine files missing from disk and/or rows missing
from the 'pristine' table and/or bad checksums.  Generally
requires contacting the server, so requires support at a higher
level than this function.

  * Identify any pristine text files on disk that are not referenced
in the DB, and delete them.
]]]

The refcounts are references within the DB from nodes to the 'pristines' 
table.  They are enforced by SQLite with 'REFERENCES' clauses in the 
schema, though I saw one comment somewhere saying this was "in debug 
builds" so we might want to double-check.


I am not aware of problems in the consistency of the DB tables, so I 
don't think checking that is a priority.  Though I don't have hard 
evidence, from problems reported over the years I think corrupted and 
missing pristine files on disk is the main concern.


- Julian

Re: [RFC] svn wc verify -- pristine files consistency check, and possibly repair

[Branko Čibej]

On 10.01.2020 06:30, Nathan Hartman wrote:
> On Thu, Jan 9, 2020 at 11:38 AM Julian Foad  > wrote:
>
> Proposal: a feature to check whether a Subversion WC's pristine texts
> (and potentially other metadata) are all present and uncorrupted.
> Possible second stage: ability to repair some problems.
>
> Why?
>
> Different factors can cause corruption, including:
>
>    * the user accidentally changing .svn files by running a
> search-and-replace etc.;
>    * bugs in Subversion;
>    * "random" corruption caused by hardware or other software.
>
> One customer I know of recently found corruption of the "pristine
> checksum mismatch" kind in some WCs when trying to commit from
> them, and
> was looking for a way to check whether other WCs were valid ahead of
> finding a problem at commit time.  That is not the first time
> users have
> experienced WC corruption.  The usual suggestion, "check out a fresh
> WC", is a blunt tool and may leave a user with residual fear,
> uncertainty and doubt.
>
> Right now, there is no good and easy way to check if a WC's pristines
> are present and correct.
>
> Does it make sense as a feature proposal?  Thoughts?
>
>
> I think both items makes sense as a feature proposal.
>
> It would make sense (at least in my mind) that 'svn cleanup' should
> perform whatever checks, and either fix (if it can) or re-fetch from
> the server (particularly since we already ask the user to run cleanup
> after interrupted operations).

I was thinking the same thing, only not by default because this could be
a fairly expensive operation. Still, it we have --vacuum-pristines, we
could add --verify-pristines.

Having a new 'svn wc ' subcommand would of course be nice, but
for consistency with the current status quo, adding an option to 'svn
cleanup' (perhaps 'svn wc cleanup' at some future date ...) would be good.

-- Brane