Re: Test failures with Python 3 (Re: PMCs: any Hackathon requests? (deadline 11 October))

[Daniel Shahaf]

Yasuhito FUTATSUKI wrote on Sun, 13 Oct 2019 04:01 +00:00:
> On 2019/10/13 7:24, Daniel Shahaf wrote:
> > Yasuhito FUTATSUKI wrote on Sat, 12 Oct 2019 03:01 +00:00:
> >> If textual comparison is sufficient here, it is right to open file
> >> text mode (with suitable, unified set of `encoding', `errors', and 
> >> `newline'
> >> parameter). Otherwise, if strict comparison is needed, we must avoid 
> >> unwanted,
> >> not one-on-one corresponding conversion from bytes to str applied by 
> >> Python.
> >> In the latter case, it may be rather incorrect to use
> >> compare_and_display_lines().
> > 
> > Good question.  I suspect textual comparison would suffice here, because
> > this is a tree conflicts test, not a keywords semantics test, and the
> > test case seems to revolve around the tree changes, not around the
> > newline characters.
> 
> I see. Now I agree it would suffice here.
>   
> > So, how about:
> > 
> > 1. Make the test use non-binary mode for changing and reading the
> > file 'lambda'.
> > 
> > 2. Locally revert the C part of r1841731 and make sure the modified test
> > still (correctly) fails.  (That revision both added the test and
> > fixed the bug the test checks for.)
> 
> So it looks sufficient to me.

Cool.  Will you perchance have time to do this?  No worries if not.

Also, what about the svnadmin_tests.py patch you posted upthread?  Is
there a reason not to go ahead and commit it to trunk?  (and even nominate
it for backport in 1.13.x/STATUS)

Cheers,

Daniel

Re: Subversion semantics: no no-op changes

[Branko Čibej]

On 14.10.2019 10:07, Johan Corveleyn wrote:
> On Fri, Oct 11, 2019 at 4:56 PM Julian Foad  wrote:
> ...
>> Some of the existing svn protocols and APIs explicitly preserve certain
>> no-op changes.  For example, one user reported [2] that in their svn
>> history (converted from CVS) they would "hate to lose" the historical
>> record that "svn log -v" reports "file text changed" for a certain no-op
>> file change.  When I eliminated this no-op change from "dump", without
>> due care to backward compatibility, it was considered a regression and
>> reverted [#4598].  There are valid arguments for preserving backward
>> compatibility in some places.  However, I propose such behaviour should
>> be considered obsolete and broken, and a migration path should be
>> planned to get away from it.
> Hi Julian,
>
> As the user in question, who reported this loss of history at $company
> while dumping/loading our repo: at the time, I was mainly concerned
> with backwards compatibility, wanting to preserve our existing history
> as close to 100% as possible. I was also very much surprised by this
> change in behaviour. Another couple of years have passed, and now I
> think: meh, it's probably not such a big deal.

Well actually it is. And here's why:

If you have a repository with such no-op changes; and you want to do a
dump/load cycle in order to, I don't know, optimize the repo or make new
features available. Then, if the dump or the load step drops no-op
changes, all existing working copies suddenly are no longer compatible
with the new repository, even if you preserver the UUID. You can't just
'svn relocate' any more (or replace the repo on the server), you have to
do a fresh checkout, or expect possibly corrupting side effects.

-- Brane

Issue tracker housecleaning: SVN-1722

[Nathan Hartman]

A short time ago, in a galaxy nearby... or, in the thread
"Fwd: PMCs: any Hackathon requests? (deadline 11 October)"...

Julian Foad wrote:
>> Nathan Hartman wrote:
>> https://issues.apache.org/jira/browse/SVN-1722 "svn
>> diff may missreport a revision as the working copy."
>
> Errm... last reviewed 9 years ago.  Is it really what it says it is?
>
> Sadly, we have hundreds of unreviewed (or not recently reviewed) open
> issues.  Some are misleadingly labeled "bite-sized" but are really
> symptoms of deeper problems.  Can we take this opportunity to do
> something about that?  Like, maybe start bringing one old issue to the
> dev list for review each week?

I later said I would bring one issue to the list for review each week.
So here's the first one... SVN-1722 is as good a candidate as any.

Summary: I think this issue should be closed.

It looks as though this issue was fixed somewhere between 1.8.19 and
1.12.2.

I think the issue tracker was not updated because the fix might have
occurred as part of other refactorings or new features in the code.

In 2004, when the issue was opened, it was reported that the problem
manifest when svn diff was run with the following arguments in the
reproduction script:

+ svn diff -r BASE:HEAD
+ svn diff -r HEAD:BASE
+ svn diff -r HEAD
+ svn diff -r 0:HEAD
+ svn diff -r 1

Later, in 2010, C. Michael Pilato commented in the issue tracker and
showed one form that was fixed and another that that was still broken.
The correct form was output by:

+ svn diff -r BASE:HEAD

The broken form was still being output by:

+ svn diff -r HEAD:BASE

I ran the reproduction script against 1.8.19 and against 1.12.2
(respectively the oldest and newest installed versions I happen to
have at the moment):

When run against 1.8.19, the following forms produce the broken output:

+ svn diff -r HEAD:BASE
+ svn diff -r HEAD
+ svn diff -r 1

When run against 1.12.2, all of the forms appear correct.

I did not setup and run a bisect to find the exact commit where the
issue gets fixed, but some manual investigation turned up this very
promising-looking candidate:

r1570053

and these slightly less-promising-looking ones:

r1619452
r1570266

The reproduction script (copied from issue tracker):

$ cat runtest
rm -rf repo test test2
svnadmin create repo
svn checkout file:///`pwd`/repo test
svn checkout file:///`pwd`/repo test2
cd test
echo "test" >file
svn add file
svn -m test commit
cd ../test2
svn diff -r BASE:HEAD
svn diff -r HEAD:BASE
svn diff -r BASE
svn diff -r HEAD
svn diff -r 0:BASE
svn diff -r 0:HEAD
svn diff -r 0:1
svn diff -r 1:0
svn diff -r 0
svn diff -r 1

Following is the output when run against 1.8.19; note the incorrect
"working copy" labels when -r is HEAD:BASE, HEAD, and 1:

$ sh -x runtest
+ rm -rf repo test test2
+ svnadmin create repo
++ pwd
+ svn checkout file:Users/nate/mount/ramdrive/test/repo test
Checked out revision 0.
++ pwd
+ svn checkout file:Users/nate/mount/ramdrive/test/repo test2
Checked out revision 0.
+ cd test
+ echo test
+ svn add file
A file
+ svn -m test commit
Adding file
Transmitting file data .
Committed revision 1.
+ cd ../test2
+ svn diff -r BASE:HEAD
Index: file
===
--- file (revision 0)
+++ file (revision 1)
@@ -0,0 +1 @@
+test
+ svn diff -r HEAD:BASE
Index: file
===
--- file (revision 1)
+++ file (working copy)
@@ -1 +0,0 @@
-test
+ svn diff -r BASE
+ svn diff -r HEAD
Index: file
===
--- file (revision 1)
+++ file (working copy)
@@ -1 +0,0 @@
-test
+ svn diff -r 0:BASE
+ svn diff -r 0:HEAD
Index: file
===
--- file (revision 0)
+++ file (revision 1)
@@ -0,0 +1 @@
+test
+ svn diff -r 0:1
Index: file
===
--- file (revision 0)
+++ file (revision 1)
@@ -0,0 +1 @@
+test
+ svn diff -r 1:0
Index: file
===
--- file (revision 1)
+++ file (revision 0)
@@ -1 +0,0 @@
-test
+ svn diff -r 0
+ svn diff -r 1
Index: file
===
--- file (revision 1)
+++ file (working copy)
@@ -1 +0,0 @@
-test
$

Following is the output when run against 1.12.2:

$ sh -x runtest
+ rm -rf repo test test2
+ svnadmin create repo
+ pwd
+ svn checkout file:home/nate/ramdrive/test/repo test
Checked out revision 0.
+ pwd
+ svn checkout file:home/nate/ramdrive/test/repo test2
Checked out revision 0.
+ cd test
+ echo test
+ svn add file
A file
+ svn -m test commit
Adding file
Transmitting file data .done
Committing transaction...
Committed revision 1.
+ cd ../test2
+ svn diff -r BASE:HEAD
Index: file
===
--- file(nonexistent)
+++ file

Re: Subversion semantics: no no-op changes

[Julian Foad]

Eric S. Raymond wrote:
> Then I no longer undersrand what we are talking about.
 
OK, no worries. I was just hoping you'd twig what I'm clumsily trying to get 
at, but I'll have to write it up properly some time.
 
- Julian

Re: Subversion semantics: no no-op changes

[Eric S. Raymond]

Julian Foad :
> Eric S. Raymond wrote:
> > Julian Foad :
> > > [...]  I have come across a number of kinds of what could
> > > be called a "no-op change" or perhaps better described as "I touched this
> > > but did not change its value".
> [...]
> > No-op commits are specifically awkward for
> [...]
> 
> Eric, thank you for your thoughts on no-op commits.
> 
> Unfortunately I failed to make clear that I didn't mean to talk about an
> entire commit that is in total a no-op.  That is of course one manifestation
> of a no-op change, and I would certainly be interested in discussing it.
> 
> However, in this thread I specifically meant to talk about all the possible
> variants of micro-change within a commit (or within a diff or update or
> merge or ...).  For the sake of simplicity, let's say we're talking about a
> commit that also contains at least one "operative" change.
> 
> - Julian

Then I no longer undersrand what we are talking about.
-- 
http://www.catb.org/~esr/;>Eric S. Raymond

Re: Make check with different client and server versions

[Julian Foad]

+1. I'd like to support us doing this.

Nathan Hartman wrote:

How would we do this?


As a starting point, for each client version to be tested, the new 
multi-combination test driver should:


  * Run the regression test suite that is supplied with that client 
version.  (This will be easiest because most variation is associated 
with client side changes.)


  * Tell the test suite ("make check") which server version to expect.

This is partly done: the Python and C tests take an argument

  --server-minor-version
Set the minor version for the server ('3'..'14')

  or, for the C tests, docs apparently not updated recently:
set the minor version for the server ('3', '4', '5', or '6')

I'm not sure exactly how one sets up the server appropriately, before 
running with that option, for various kinds of server.  Maybe 'make 
check' and/or 'make svnserveautocheck' and/or 'make davautocheck' have 
ways to specify how to find and run the desired server version.  We'll 
probably need to check and update that.



Ideally, later, the tests should also be divided or tagged so we can 
select sets of tests:

  - client-server tests
  - client-only tests
  - server-only tests
because we could then eliminate running redundant sets.  Initially, that 
isn't critical.




(1) Which versions are we interested in cross-testing in this manner?


Start with a simple fixed set, such as

  - (client: trunk, server: 1.10)
  - (client: 1.10, server: trunk)

Review later, once that's working.

(I suggested 1.10 there because it's the most recent LTS version, but 
the important thing is just to start with something.)



(2) How do we handle differences between versions?

[...]
Is the test driver program supposed to contain knowledge of these 
differences and prevent some of the tests from running under certain 
combinations of client and server versions?


Annotate the tests according to what server versions they require.

This is at least partly done.  The test suite already uses conditionals like

  if svntest.main.options.server_minor_version < 9:

I'm not sure if this is already done everywhere it needs to be.  I would 
expect to see some of the Python "decorators" such as


  @SkipUnless(server_authz_has_aliases)

using "server_minor_version" but I don't see any.



(3) How do we handle dependencies?


Initially: whatever works.  Probably neatest to install the dependencies 
for each built version of Subversion into a location dedicated to that 
version of Subversion, so they don't affect each other and don't affect 
the rest of the system.


- Julian

Re: Subversion semantics: no no-op changes

[Julian Foad]

Eric S. Raymond wrote:

Julian Foad :

[...]  I have come across a number of kinds of what could
be called a "no-op change" or perhaps better described as "I touched this
but did not change its value".

[...]
No-op commits are specifically awkward for 

[...]

Eric, thank you for your thoughts on no-op commits.

Unfortunately I failed to make clear that I didn't mean to talk about an 
entire commit that is in total a no-op.  That is of course one 
manifestation of a no-op change, and I would certainly be interested in 
discussing it.


However, in this thread I specifically meant to talk about all the 
possible variants of micro-change within a commit (or within a diff or 
update or merge or ...).  For the sake of simplicity, let's say we're 
talking about a commit that also contains at least one "operative" change.


- Julian

Re: Python3 work [was: The run up to Subversion 1.13.0]

[Johan Corveleyn]

On Fri, Oct 11, 2019 at 2:38 PM Yasuhito FUTATSUKI  wrote:
>
> On 2019-09-30 06:32, Johan Corveleyn wrote:
> ...
> > At this point, I think the only thing blocking reintegration of this
> > branch into trunk seems to be the C4115 warning / error. As I said, I
> > have no informed opinion either way (and Brane was okay with turning
> > it into a warning). So if Bert doesn't reply within a couple of days,
> > I'd say we are good to go.
>
> ... and more than a week has past. So I've commited the patch address
> to it on swig-py3 branch, as r1868290.

Great. I've retested it all on Windows (once with Python 2.7.16, and
once with Python 3.7.4) and it looks good to me.

+1 to reintegrate the branch into trunk.

-- 
Johan

Re: Subversion semantics: no no-op changes

[Johan Corveleyn]

On Fri, Oct 11, 2019 at 4:56 PM Julian Foad  wrote:
...
> Some of the existing svn protocols and APIs explicitly preserve certain
> no-op changes.  For example, one user reported [2] that in their svn
> history (converted from CVS) they would "hate to lose" the historical
> record that "svn log -v" reports "file text changed" for a certain no-op
> file change.  When I eliminated this no-op change from "dump", without
> due care to backward compatibility, it was considered a regression and
> reverted [#4598].  There are valid arguments for preserving backward
> compatibility in some places.  However, I propose such behaviour should
> be considered obsolete and broken, and a migration path should be
> planned to get away from it.

Hi Julian,

As the user in question, who reported this loss of history at $company
while dumping/loading our repo: at the time, I was mainly concerned
with backwards compatibility, wanting to preserve our existing history
as close to 100% as possible. I was also very much surprised by this
change in behaviour. Another couple of years have passed, and now I
think: meh, it's probably not such a big deal.

Do note however that at the time of that mail in 2015 (and issue
#4598), it was also determined that the change in behaviour in 1.9 (no
longer dumping those no-op changes) was unintentional (side effect of
another change by Stefan Fuhrmann). That's one of the reasons why it
was considered a regression.

Anyway, that's all "history" now. I just wanted to say: if y'all
decide to get rid of no-op changes, I won't oppose it at all cost for
the sake of backwards compatibility any more :-). So FWIW, consider my
opinion "neutral" on the matter.

-- 
Johan

> [2] Email: "No-op changes no longer dumped by 'svnadmin dump' in 1.9",
>  from Johan Corveleyn to dev@, 2015-09-21,
>  https://svn.haxx.se/dev/archive-2015-09/0269.shtml
>
> http://mail-archives.apache.org/mod_mbox/subversion-dev/201509.mbox/%3ccab84ubve8qnepbpvab__yqjiddoyjfn2+m9mpcdbxzcwmcp...@mail.gmail.com%3E
>
> [#4598] "No-op changes no longer dumped by 'svnadmin dump' in 1.9",
>  https://subversion.apache.org/issue/4598
>  https://issues.apache.org/jira/browse/SVN-4598
>
> [#4623] "no-op prop change not preserved across dump/load"
>  https://subversion.apache.org/issue/4623
>  https://issues.apache.org/jira/browse/SVN-4623

Re: Better choice for Linux semaphore than spinlock?

[eponymous alias]

 Perhaps these links might be of help in some way:

https://webkit.org/blog/6161/locking-in-webkit/
https://blog.mozilla.org/nfroyd/2017/03/29/on-mutex-performance-part-1/
https://preshing.com/2018/locks-arent-slow-lock-contention-is/

On Monday, October 7, 2019, 1:56:14 PM PDT, Doug Robinson 
 wrote:

Rüdiger:

On Mon, Oct 7, 2019 at 3:51 PM Ruediger Pluem  wrote:

 On 10/07/2019 08:40 PM, Branko Čibej wrote:
 > On Mon, 7 Oct 2019, 19:47 Doug Robinson,  > wrote:
 >
 > Folks:
 >
 > I spoke with this user late last week. They stated that they can only get 
 > approximately 400 parallel SVN operations
 > before the "system time" consumes all available CPU for an 8-core machine. 
 > Adding more cores won't help because of
 > the nature of spin locks (it makes things worse). Turns out that even with 
 > ~100 parallel SVN operations the "system
 > time" starts becoming significant/measurable (~10%). Both HTTP (mod_dav_svn) 
 > and "svnserve" protocols participate
 > in the lock contention.
 >
 > Your help would be greatly appreciated.
 >
 > Whew. So. Reducing this issue to "use a more efficient lock" is not going to 
 > work, and you provided far too little
 > information to even attempt a diagnosis. For starters, I recommend gathering 
 > as much info as possible (anonymised of
 > course) about the server configuration, everything from httpd an svnserve to 
 > the repository config and underlying
 > filesystem, if possible. Getting stack traces of the "stuck" threads would 
 > be necessary, too. Without knowing exactly
 > what is happening, these kinds of problems are extremely hard to understand, 
 > let alone fix.

 Plus depending on which part of the code requires this lock a different 
locking mechanism that might suit better for
 this use case can possibly be chosen via configuration changes (e.g. httpd 
allows this for most of its locking).

That would be awesome! I'll definitely try to get those stack tracebacks.

Cheers.

Doug