Re: Test failures with Python 3 (Re: PMCs: any Hackathon requests? (deadline 11 October))

[Yasuhito FUTATSUKI]

On 2019-10-12 07:47, Daniel Shahaf wrote:

Yasuhito FUTATSUKI wrote on Sat, Oct 12, 2019 at 05:31:53 +0900:

Yes, it will fix local_missing_dir_endless_loop() itself correctly.
But the stack trace before fix indicate there is at least one problem
in svntest.verify.compare_and_display_lines().

Assume the file contents is broken here. This situation can be simulate
by patch like:

Index: subversion/tests/cmdline/tree_conflict_tests.py
===
--- subversion/tests/cmdline/tree_conflict_tests.py (revision 1868264)
+++ subversion/tests/cmdline/tree_conflict_tests.py (working copy)
@@ -1544,7 +1544,7 @@
contents = open(sbox.ospath('A1/B/lambda'), 'rb').readlines()
svntest.verify.compare_and_display_lines(
  "A1/B/lambda has unexpectected contents", sbox.ospath("A1/B/lambda"),
-[ "This is the file 'lambda'.\n", "This is more content.\n"], contents)
+[ b"This is the file 'lambda'.\n", b"This is not more content.\n"], 
contents)
  ###


then we will got fails.log, contains stack trace for unexpected exception
within the code to construct log message.

[[[
W: A1/B/lambda has unexpectected contents
W: EXPECTED svn-test-work/working_copies/tree_conflict_tests-26/A1/B/lambda 
(match_all=True):
W: CWD: /home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline
Traceback (most recent call last):
   File 
"/home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline/svntest/main.py",
 line 1931, in run
 rc = self.pred.run(sandbox)
   File 
"/home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline/svntest/testcase.py",
 line 178, in run
 result = self.func(sandbox)
   File 
"/home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline/tree_conflict_tests.py",
 line 1547, in local_missing_dir_endless_loop
 [ b"This is the file 'lambda'.\n", b"This is not more content.\n"], 
contents)
   File 
"/home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline/svntest/verify.py",
 line 503, in compare_and_display_lines
 expected.display_differences(message, label, actual)
   File 
"/home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline/svntest/verify.py",
 line 154, in display_differences
 display_lines(message, self.expected, actual, e_label, label)
   File 
"/home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline/svntest/verify.py",
 line 474, in display_lines
 logger.warn('| ' + x.rstrip())
TypeError: can only concatenate str (not "bytes") to str
FAIL:  tree_conflict_tests.py 26: endless loop when resolving local-missing dir
]]]

This is caused by mixing bytes object drived from file contents and str
object to construct log message.


I agree: this FAIL indicates str and bytes are mixed.  My question is:
Why do you think svntest.verify.compare_and_display_lines() needs to be
changed?  That function's names implies it deals with text files, so,
why should compare_and_display_lines() support the case that its third
and fourth parameters (EXPECTED and ACTUAL) are both lists of bytes objects?

In other words, why would changing «'rb'» to «'r'» on line 1544 —
without changing the str literals to bytes literals on line 1547 — not
be a correct solution?


Ah, I thought strict comparison is needed here because of raw mode file I/O.

Index: subversion/tests/cmdline/tree_conflict_tests.py
===
--- subversion/tests/cmdline/tree_conflict_tests.py (revision 1868264)
+++ subversion/tests/cmdline/tree_conflict_tests.py (working copy)
@@ -1518,7 +1518,7 @@
   sbox.simple_move('A/B', 'A/B2')
   sbox.simple_commit()
   sbox.simple_update()
-  main.file_append_binary(sbox.ospath("A/B2/lambda"), "This is more 
content.\n")
+  main.file_append_binary(sbox.ospath("A/B2/lambda"), "This is more 
content.\r\n")
   sbox.simple_commit()
   sbox.simple_update()
 
@@ -1544,7 +1544,7 @@

   contents = open(sbox.ospath('A1/B/lambda'), 'rb').readlines()
   svntest.verify.compare_and_display_lines(
 "A1/B/lambda has unexpectected contents", sbox.ospath("A1/B/lambda"),
-[ "This is the file 'lambda'.\n", "This is more content.\n"], contents)
+[ b"This is the file 'lambda'.\n", b"This is more content.\n"], contents)
 
 
 ###


Above patch will make local_missing_dir_endless_loop test fail because of
strictness of comparison. However,

Index: subversion/tests/cmdline/tree_conflict_tests.py
===
--- subversion/tests/cmdline/tree_conflict_tests.py (revision 1868264)
+++ subversion/tests/cmdline/tree_conflict_tests.py (working copy)
@@ -1518,7 +1518,7 @@
   sbox.simple_move('A/B', 'A/B2')
   sbox.simple_commit()
   sbox.simple_update()
-  main.file_append_binary(sbox.ospath("A/B2/lambda"), "This is more 
content.\n")
+ 

Re: PMCs: any Hackathon requests? (deadline 11 October)

[Daniel Shahaf]

Nathan Hartman wrote on Fri, Oct 11, 2019 at 12:53:23 -0400:
> (2) Once this is more-or-less under control, I'd like to start actively
> looking for volunteers, hackathon events, and any other opportunities
> to get more involvement.

This might be easier to do on an ASF-wide scale.  comdev¹ would be a good place
to start.

Cheers,

Daniel


¹ d...@community.apache.org / https://community.apache.org/

Re: Test failures with Python 3 (Re: PMCs: any Hackathon requests? (deadline 11 October))

[Daniel Shahaf]

Nathan Hartman wrote on Fri, Oct 11, 2019 at 16:50:39 -0400:
> On Fri, Oct 11, 2019 at 4:32 PM Yasuhito FUTATSUKI  
> wrote:
> > This is caused by mixing bytes object drived from file contents and str
> > object to construct log message.
> 
> Does something like this answer help:
> 
> https://stackoverflow.com/questions/31058055/how-do-i-convert-a-python-3-byte-string-variable-into-a-regular-string/31060836
> 
> Something like:
> str(bytes_string, 'utf-8')

Nathan, I appreciate the intent, but Yasuhito and I are both familiar with the
semantics of str and bytes objects in Python 3.  We're not asking what the
difference between bytes and str is, or how to work with them; we are simply
trying to resolve two particular test failures, in svnadmin_ and
tree_conflict_tests.py.  Specifically, we're trying to determine whether file
contents should be handled as str or as bytes, both in the test function and in
the test framework.

Cheers,

Daniel

Re: Test failures with Python 3 (Re: PMCs: any Hackathon requests? (deadline 11 October))

[Daniel Shahaf]

Yasuhito FUTATSUKI wrote on Sat, Oct 12, 2019 at 05:31:53 +0900:
> Yes, it will fix local_missing_dir_endless_loop() itself correctly.
> But the stack trace before fix indicate there is at least one problem
> in svntest.verify.compare_and_display_lines().
> 
> Assume the file contents is broken here. This situation can be simulate
> by patch like:
> 
> Index: subversion/tests/cmdline/tree_conflict_tests.py
> ===
> --- subversion/tests/cmdline/tree_conflict_tests.py (revision 1868264)
> +++ subversion/tests/cmdline/tree_conflict_tests.py (working copy)
> @@ -1544,7 +1544,7 @@
>contents = open(sbox.ospath('A1/B/lambda'), 'rb').readlines()
>svntest.verify.compare_and_display_lines(
>  "A1/B/lambda has unexpectected contents", sbox.ospath("A1/B/lambda"),
> -[ "This is the file 'lambda'.\n", "This is more content.\n"], contents)
> +[ b"This is the file 'lambda'.\n", b"This is not more content.\n"], 
> contents)
>  ###
> 
> 
> then we will got fails.log, contains stack trace for unexpected exception
> within the code to construct log message.
> 
> [[[
> W: A1/B/lambda has unexpectected contents
> W: EXPECTED svn-test-work/working_copies/tree_conflict_tests-26/A1/B/lambda 
> (match_all=True):
> W: CWD: /home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline
> Traceback (most recent call last):
>   File 
> "/home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline/svntest/main.py",
>  line 1931, in run
> rc = self.pred.run(sandbox)
>   File 
> "/home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline/svntest/testcase.py",
>  line 178, in run
> result = self.func(sandbox)
>   File 
> "/home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline/tree_conflict_tests.py",
>  line 1547, in local_missing_dir_endless_loop
> [ b"This is the file 'lambda'.\n", b"This is not more content.\n"], 
> contents)
>   File 
> "/home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline/svntest/verify.py",
>  line 503, in compare_and_display_lines
> expected.display_differences(message, label, actual)
>   File 
> "/home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline/svntest/verify.py",
>  line 154, in display_differences
> display_lines(message, self.expected, actual, e_label, label)
>   File 
> "/home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline/svntest/verify.py",
>  line 474, in display_lines
> logger.warn('| ' + x.rstrip())
> TypeError: can only concatenate str (not "bytes") to str
> FAIL:  tree_conflict_tests.py 26: endless loop when resolving local-missing 
> dir
> ]]]
> 
> This is caused by mixing bytes object drived from file contents and str
> object to construct log message.

I agree: this FAIL indicates str and bytes are mixed.  My question is:
Why do you think svntest.verify.compare_and_display_lines() needs to be
changed?  That function's names implies it deals with text files, so,
why should compare_and_display_lines() support the case that its third
and fourth parameters (EXPECTED and ACTUAL) are both lists of bytes objects?

In other words, why would changing «'rb'» to «'r'» on line 1544 —
without changing the str literals to bytes literals on line 1547 — not
be a correct solution?  

Hope I'm clearer this time.  If not, I'd be happy to clarify.

Cheers,

Daniel

Re: Test failures with Python 3 (Re: PMCs: any Hackathon requests? (deadline 11 October))

[Nathan Hartman]

On Fri, Oct 11, 2019 at 4:32 PM Yasuhito FUTATSUKI  wrote:
> This is caused by mixing bytes object drived from file contents and str
> object to construct log message.

Does something like this answer help:

https://stackoverflow.com/questions/31058055/how-do-i-convert-a-python-3-byte-string-variable-into-a-regular-string/31060836

Something like:
str(bytes_string, 'utf-8')

Re: Test failures with Python 3 (Re: PMCs: any Hackathon requests? (deadline 11 October))

[Yasuhito FUTATSUKI]

On 2019-10-12 02:56, Daniel Shahaf wrote:

Yasuhito FUTATSUKI wrote on Fri, Oct 11, 2019 at 16:35:19 +0900:



The latter also can be fixed by fix_tree_conflict_tests_patch.txt
at least so that the test can be passed on Python 3. However, from above
stack trace, I think it is incomplete because there still exists some sort of
confusion between bytes and str in subversion/tests/cmdline/svntest/*.py

Index: subversion/tests/cmdline/tree_conflict_tests.py
===
--- subversion/tests/cmdline/tree_conflict_tests.py (revision 1868264)
+++ subversion/tests/cmdline/tree_conflict_tests.py (working copy)
@@ -1544,7 +1544,7 @@
contents = open(sbox.ospath('A1/B/lambda'), 'rb').readlines()
svntest.verify.compare_and_display_lines(
  "A1/B/lambda has unexpectected contents", sbox.ospath("A1/B/lambda"),
-[ "This is the file 'lambda'.\n", "This is more content.\n"], contents)
+[ b"This is the file 'lambda'.\n", b"This is more content.\n"], contents)


Why do you think this is incomplete?  The open() call uses mode='rb', so
«contents» will be set to an array of bytes objects, so it'll need to be
compared to an array of bytes objects.  Which is to say, this patch, too, looks
correct to me.


Yes, it will fix local_missing_dir_endless_loop() itself correctly.
But the stack trace before fix indicate there is at least one problem
in svntest.verify.compare_and_display_lines().

Assume the file contents is broken here. This situation can be simulate
by patch like:

Index: subversion/tests/cmdline/tree_conflict_tests.py
===
--- subversion/tests/cmdline/tree_conflict_tests.py (revision 1868264)
+++ subversion/tests/cmdline/tree_conflict_tests.py (working copy)
@@ -1544,7 +1544,7 @@
   contents = open(sbox.ospath('A1/B/lambda'), 'rb').readlines()
   svntest.verify.compare_and_display_lines(
 "A1/B/lambda has unexpectected contents", sbox.ospath("A1/B/lambda"),
-[ "This is the file 'lambda'.\n", "This is more content.\n"], contents)
+[ b"This is the file 'lambda'.\n", b"This is not more content.\n"], 
contents)
 
 
 ###



then we will got fails.log, contains stack trace for unexpected exception
within the code to construct log message.

[[[
W: A1/B/lambda has unexpectected contents
W: EXPECTED svn-test-work/working_copies/tree_conflict_tests-26/A1/B/lambda 
(match_all=True):
W: CWD: /home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline
Traceback (most recent call last):
  File 
"/home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline/svntest/main.py",
 line 1931, in run
rc = self.pred.run(sandbox)
  File 
"/home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline/svntest/testcase.py",
 line 178, in run
result = self.func(sandbox)
  File 
"/home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline/tree_conflict_tests.py",
 line 1547, in local_missing_dir_endless_loop
[ b"This is the file 'lambda'.\n", b"This is not more content.\n"], 
contents)
  File 
"/home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline/svntest/verify.py",
 line 503, in compare_and_display_lines
expected.display_differences(message, label, actual)
  File 
"/home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline/svntest/verify.py",
 line 154, in display_differences
display_lines(message, self.expected, actual, e_label, label)
  File 
"/home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline/svntest/verify.py",
 line 474, in display_lines
logger.warn('| ' + x.rstrip())
TypeError: can only concatenate str (not "bytes") to str
FAIL:  tree_conflict_tests.py 26: endless loop when resolving local-missing dir
]]]

This is caused by mixing bytes object drived from file contents and str
object to construct log message.

Cheers,
--
Yasuhito FUTATSUKI 

Re: Subversion semantics: no no-op changes

[Daniel Shahaf]

Julian Foad wrote on Fri, Oct 11, 2019 at 15:56:31 +0100:
> Hello Eric.

Not to preëmpt Eric, but may I share my thoughts too?

> In conclusion, I consider svn would be a better system -- more predictable,
> testable, composable, etc.; more generally dependable -- and would lose no
> significant value at all -- if we were to explicitly remove no-op changes.
> 
> Does this all ring true and obvious to you, or can you explain better what I
> am getting at and what I'm missing?

What layer do you want to forbid no-op changes at?

At the FS level, a filesystem is an array of trees.  The contents of rN
place some restrictions on what the contents of rN+1 may be: for example,
if foo@N+1 has copyfrom=bar@N, then foo and bar have the same node kind.
At this level, I don't see a reason to forbid two successive trees from being
equal.¹  The question isn't whether that's a _useful_ thing to do, but
whether it has a consistent, well-defined semantics — to which, IMO, the
answer is in the affirmative.  That, incidentally, is also the reason
why «x + 0» is a well-formed arithmetic expression.

In C, it's perfectly valid to write «if (0) abort();», or even «foo; ;».
It is not a syntax error.  It _will_ generate a compiler warning, but
you will be able to compile and run it.

Similarly, I suspect "no no-op changes" is a semantic that belongs at
the same level as "no mixed-revision or switched subtrees": it is
something a user _usually_ don't want to do, but ultimately, it's their
data, not ours.   That's what we do in 'svn merge', which does that
sanity check but makes it opt-outable.  It's actually what we already do
with empty revisions: 'svn commit' won't create an empty revision, but
it _is_ possible to do so if one really wants to.

Use-case: Imagine a company that is regulatorily required to review its
foobar policy once a year.  For that company, it would make perfect
sense to create a commit that _doesn't_ change the contents of the
policy, but does mark the file as changed (in «svn log -qv», etc), with
the log message "Policy reviewed and reaffirmed without changes.".

Yes, it's a metadata-only change.

Makes sense?

Cheers,

Daniel

¹ [Hair splitting: when I say "equal" I mean, "equal as far as a consumer
of the replay or dump API can detect.  That is, node-rev-id's are not
available to distinguish wish.]

P.S. Relevant APIs: svn_fs_contents_different() v. svn_fs_contents_changed().

Re: Test failures with Python 3 (Re: PMCs: any Hackathon requests? (deadline 11 October))

[Daniel Shahaf]

Yasuhito FUTATSUKI wrote on Fri, Oct 11, 2019 at 16:35:19 +0900:
> On 2019-10-11 06:45, Daniel Shahaf wrote:
> ...
> >  From autogen.sh to 'make check', all steps pass with 
> > PYTHON=/usr/bin/python3.
> 
> There were two failures in 1.11 on tests.
> 
> https://lists.apache.org/thread.html/366e77c7443a0c575b47de150a2c394e1e685fde8887c805a656a33d@%3Cusers.subversion.apache.org%3E
> 
> and it seems they still exist in trunk.

Sorry, my bad.  I did see them in my test run too, but incorrectly determined
they were an artifact of my build environment.

> The former failure can be fixed by attached patch, 
> fix_svnadmin_tests_patch.txt

> Index: subversion/tests/cmdline/svnadmin_tests.py
> ===
> --- subversion/tests/cmdline/svnadmin_tests.py(revision 1868264)
> +++ subversion/tests/cmdline/svnadmin_tests.py(working copy)
> @@ -3859,7 +3859,7 @@
>   sbox.repo_url)
>  
>dump_lines = svntest.actions.run_and_verify_dump(sbox.repo_dir)
> -  assert propval + '\n' in dump_lines
> +  assert propval.encode() + b'\n' in dump_lines

+1 to commit.

> The latter also can be fixed by fix_tree_conflict_tests_patch.txt
> at least so that the test can be passed on Python 3. However, from above
> stack trace, I think it is incomplete because there still exists some sort of
> confusion between bytes and str in subversion/tests/cmdline/svntest/*.py
> 
> Index: subversion/tests/cmdline/tree_conflict_tests.py
> ===
> --- subversion/tests/cmdline/tree_conflict_tests.py   (revision 1868264)
> +++ subversion/tests/cmdline/tree_conflict_tests.py   (working copy)
> @@ -1544,7 +1544,7 @@
>contents = open(sbox.ospath('A1/B/lambda'), 'rb').readlines()
>svntest.verify.compare_and_display_lines(
>  "A1/B/lambda has unexpectected contents", sbox.ospath("A1/B/lambda"),
> -[ "This is the file 'lambda'.\n", "This is more content.\n"], contents)
> +[ b"This is the file 'lambda'.\n", b"This is more content.\n"], contents)

Why do you think this is incomplete?  The open() call uses mode='rb', so
«contents» will be set to an array of bytes objects, so it'll need to be
compared to an array of bytes objects.  Which is to say, this patch, too, looks
correct to me.

Cheers,

Daniel

Re: PMCs: any Hackathon requests? (deadline 11 October)

[Nathan Hartman]

On Fri, Oct 11, 2019 at 11:45 AM Julian Foad 
wrote:
> Daniel Shahaf wrote:
> > Julian, do you have any release.py tasks to suggest?
>
> There are certainly some things need doing in release.py.

Unfortunately I didn't hear back in time to put that on the list for
this hackathon. If the event organizer communicates with us, we
might be able to add that if you can describe exactly what you'd
like worked on.

> One thing I've taken from this discussion is there are sometimes
> potential contributors coming along who might take a well defined and
> contained task, so it's useful if we can have tasks well defined and
> discoverable, something we need to improve.

Agreed.

(1) Housecleaning on the issues database and properly marking items as
"bite size," "hackathon ready," or some other appropriate label, will
make it much easier to respond when we get such a request.

(2) Once this is more-or-less under control, I'd like to start actively
looking for volunteers, hackathon events, and any other opportunities
to get more involvement.

Re: PMCs: any Hackathon requests? (deadline 11 October)

[Julian Foad]

Daniel Shahaf wrote:

Julian, do you have any release.py tasks to suggest?


There are certainly some things need doing in release.py.

One thing I've taken from this discussion is there are sometimes 
potential contributors coming along who might take a well defined and 
contained task, so it's useful if we can have tasks well defined and 
discoverable, something we need to improve.


- Julian

Re: Remove From Mailing List

[Julian Foad]

Gillead, Gennar wrote:

Please remove my email from all mailing list.


I sent a request; I think you will have to reply to a confirmation.

Anyone can unsubscribe themself from this list by sending any (e.g. 
blank) email to:



And the equivalent (adding -unsubscribe before the @) for any other 
apache.org list.


- Julian

Subversion semantics: no no-op changes

[Julian Foad]

Hello Eric.

TL;DR:  I explain why I am convinced no-op changes don't belong in the 
Subversion versioning semantics.  With your work on Subversion 
repository and dump stream semantics, is this something you can offer a 
view on?  I have previously failed to convince the developer community [1].



In examining the Subversion versioned data semantics and how the 
protocols and APIs represent them, I have come across a number of kinds 
of what could be called a "no-op change" or perhaps better described as 
"I touched this but did not change its value".


Example:
  - I changed the text of file F from T1 to T1;
  - now "svn log -v" tells me the text of F was "modified";
  - some variants of "svn diff" show no output;
  - some variants of "svn diff" show a diff header with no body.

That is the best known user-visible example.  Other kinds are possible 
too, and a number of examples exist on the server side, e.g. [#4623].


A Subversion client generally does not send no-op changes to a 
repository, but in certain cases it does.  A Subversion repository 
generally does not record and play back any no-op changes that may be 
sent to it, but in certain cases it does.


I am convinced "no-op changes" should be considered meaningless and 
removed from the data model presented to the user.  In protocols and 
APIs, a no-op change should be considered a non-canonical form and a 
transient implementation detail of that particular interface, and 
implementations should not attempt to preserve it.


In the rest of this note I try to explain some angles to the issue.

The Subversion system is built on a main design principle of tree 
snapshots and differencing and merging of trees.  A no-op change is 
out-of-tree metadata about certain pairs of trees.  Carrying such 
metadata around the system in general is fundamentally incompatible with 
that principle.


One practical reason the existing system does not preserve that metadata 
is because, with very few exceptions, the existing interfaces convey 
no-op changes only implicitly, as a side effect of how their explicit 
operations are formulated, and so one differs from another.  For 
example, an interface that represents a file change through multiple 
optional operations, one of which is "on file F, property P changes to 
value V" can convey "property P1's value no-op-changed from V1 to V1, 
while property P2's value was not touched" if we invoke the "change 
property" operation for property P1 but do not invoke it for P2.  On the 
other hand, an interface that represents a file change as a single 
operation, "new file := {text, {properties}}" cannot; the only no-op 
change it can convey is at a coarser granularity, "file F no-op-changed 
its value from {T1,PROPS1} to {T1,PROPS1}".  The kinds of no-op change 
an interface can convey locally is an implementation detail of that 
particular interface, and so cannot be expected to match any other 
interface unless explicitly required and tested, which they mostly are not.


Because the existing interfaces convey no-op-change information only 
incidentally, the system cannot be expected to preserve any particular 
no-op change when data flows through multiple interfaces, through 
commit, checkout, branch, diff, merge, and so on.  Subversion only 
preserves some within very limited scopes (such as the "file changed" 
flag in the "changed paths" list in "svn log -v").


Some of the existing svn protocols and APIs explicitly preserve certain 
no-op changes.  For example, one user reported [2] that in their svn 
history (converted from CVS) they would "hate to lose" the historical 
record that "svn log -v" reports "file text changed" for a certain no-op 
file change.  When I eliminated this no-op change from "dump", without 
due care to backward compatibility, it was considered a regression and 
reverted [#4598].  There are valid arguments for preserving backward 
compatibility in some places.  However, I propose such behaviour should 
be considered obsolete and broken, and a migration path should be 
planned to get away from it.


The snapshots argument is diluted because we already have at least one 
other kind of metadata outside a pure tree-snapshots system: the 
"copy-from" links.  I am not immediately planning to ditch copy-from 
links, though I think there are good reasons, analogous to the reasoning 
about no-op changes, to replace them in a possible future system.  I 
have given some thought to it.  That would be a more visible change to 
the system, of course, though not so much as it might first appear.


The example of a no-op file text change is a simple one.  An example 
with deeper implications is a directory copy combined with replacing one 
of its implicitly copied child with an explicit copy of that child from 
the same source as it was implicitly copied.  Addressing a case like 
this may be as simple as declaring one version as the canonical form, or 
may require further travel down the road of copy-from 

Re: Python3 work [was: The run up to Subversion 1.13.0]

[Yasuhito FUTATSUKI]

On 2019-09-30 06:32, Johan Corveleyn wrote:
...

At this point, I think the only thing blocking reintegration of this
branch into trunk seems to be the C4115 warning / error. As I said, I
have no informed opinion either way (and Brane was okay with turning
it into a warning). So if Bert doesn't reply within a couple of days,
I'd say we are good to go.


... and more than a week has past. So I've commited the patch address
to it on swig-py3 branch, as r1868290.

Thanks,
--
Yasuhito FUTATSUKI 

Remove From Mailing List

[Gillead, Gennar]

Please remove my email from all mailing list.
gennar.gill...@teradata.com
gg123...@teradata.com


Thanks and Best Regards,

Gennar Gillead
Americas Platform Engineering Practice
Teradata Consulting
678-485-8757

Upcoming OOO Dates:
August 16th

[page1image1774944]

17095 Via Del Campo, San Diego, CA 92127
teradata.com

Re: Creating directory copy operations in a dump stream

[Eric S. Raymond]

Johan Corveleyn :
> - Just guessing here, but perhaps the problem with your test is that
> your working copy is "mixed-revision" after you added the three files.
> I.e. the 'trunk' directory is a revision behind, after you added the
> three files. See if it makes a difference if you do "svn up $wc_root"
> before performing the "svn copy trunk branches/stable".

That did the trick, thanks.

For your entertainment, here's how the test koad generator now looks:


## General test load for ancestry-chasing logic

dump=no
verbose=null
while getopts dv opt
do
case $opt in
d) dump=yes;;
v) verbose=stdout;;
esac
done

trap 'rm -fr test-repo test-checkout' 0 1 2 15 

svnaction () {
filename=$1
content=$2
comment=$3
if [ ! -f $filename ]
then
if [ ! -d `dirname $filename` ]
then
mkdir `dirname $filename`
svn add `dirname $filename`
fi
echo "$content" >$filename
svn add $filename
else
echo "$content" >$filename
fi
svn commit -m "$comment" $filename
}

{
set -e
make svn-branchy
cd test-checkout
# Content operations start here
svnaction "trunk/foo.txt" "Now is the time." "More example content" 
svnaction "trunk/bar.txt" "For all good men." "Example content in different 
file" 
svnaction "trunk/baz.txt" "to come to the aid of their country." "And in yet 
another file"
svn up  # Without this, the next copy does file copies.  With it, a directory 
copy. 
svn copy trunk branches/stable
svn commit -m "First directory copy"
svnaction "trunk/foo.txt" "Whether tis nobler in the mind." "Hamlet the Dane 
said this"
svnaction "trunk/bar.txt" "or to take arms against a sea of troubles" "He 
continued"
svnaction "trunk/baz.txt" "and by opposing end them" "The build-up"
svnaction "trunk/foo.txt" "to be,"  "Famous soliloquy begins"
svnaction "branches/foo.txt" "or not to be." "And continues"
svn up
svn copy trunk tags/1.0
svn commit -m "First tag copy"
# We're done
cd ..
} >/dev/$verbose 2>&1
if [ "$dump" = yes ]
then
svnadmin dump -q test-repo
fi


This will get longer and include the most perverse combinations of
deletes and copies I can dream up. The point, of course, is to
torture-test my Subversion dump analyzer.

I have a fairly nice gallmaufry of static stream dumps I've gathered
or made by hand over the years; being able to generate them easily
will be helpful.
-- 
http://www.catb.org/~esr/;>Eric S. Raymond

Re: Adding Apache Subversion to the hackCBS 2.0 task list

[Sally Khudairi]

Rohan, 

We have Apache OFBiz, Ignite, and now SVN providing a list of tasks for the 
hackathon. 

Each Apache Project has detailed what their requirements are, which has been 
forwarded to you. 

I don't know how the event organizers are grouping the development 
themes/activities: we are providing numerous opportunities for participating 
students here. 

If you can kindly follow up with those who have taken the time to pull these 
lists together --Pranay with Apache OFBiz, Ilya and Denis with Apache Ignite, 
and Nathan with Apache SVN-- this will help the specific projects know what 
will be worked on (or however the event will be handled). I have copied Pranay, 
Ilya and Denis here so everyone is aware. 

You wrote:

> ...how we can work together to get Apache out in front of some of the top 
> developers in the world. You can even give some specific Hackathon problem 
> statements to the participants, and let innovation drive through it.

I am not involved with technical development: I am connecting hackCBS with 
Apache Projects and their communities. There is now a comprehensive list of 
Apache-related activities that can be done during the hackathon. 

You also asked us to sponsor and send t-shirts for the event. As mentioned 
earlier, we do not have the budget for either, but a rush order for Apache 
stickers is in production and will be sent to Sajal Bansal.

I trust that you will be able to share next steps and address any concerns with 
Pranay, Ilya, Denis, and Nathan. Email is the primary form of communication at 
The Apache Software Foundation. We look forward to hearing from you. 

Kind regards,
Sally

- - -
Vice President Marketing & Publicity
Vice President Sponsor Relations
The Apache Software Foundation

Tel +1 617 921 8656 | s...@apache.org


On Fri, Oct 11, 2019, at 05:08, Rohan Vij wrote:
> Hi Sally
> 
> Is it possible that we can get over call sometime to discuss this? So, that I 
> am clear of what all needs to be done.
> 
> Regards
> Rohan Vij
> 
> 
> On Fri, 11 Oct, 2019, 2:19 PM Sally Khudairi,  wrote:
>> Thank you, Nathan.
>> 
>> I'm copying Rohan Vij from hackCBS 2.0 here so he can follow up with you 
>> directly.
>> 
>> @Rohan --kindly advise next steps.
>> 
>> Best regards, 
>> Sally
>> 
>> - - - 
>> Vice President Marketing & Publicity
>> Vice President Sponsor Relations
>> The Apache Software Foundation
>> 
>> Tel +1 617 921 8656 | s...@apache.org
>> 
>> On Thu, Oct 10, 2019, at 20:32, Nathan Hartman wrote:
>> >
>> > The Apache Subversion project has several opportunities for the 2-day
>> > hackCBS 2.0 event across these skill areas: web design, swig bindings,
>> > and C programming. Specific items are listed below.
>> > 
>> > Hackathon participants who are interested in working on Subversion
>> > should join our 'dev' mailing list and introduce themselves. To join,
>> > email dev-subscr...@subversion.apache.org. For details, see:
>> > https://subversion.apache.org/mailing-lists.html.
>> > 
>> > We'll be happy to provide whatever guidance we can, as well as review
>> > and (hopefully) apply some quality patches!
>> > 
>> > 
>> > Items:
>> > 
>> > * Web design on "next generation" https://subversion.apache.org:
>> > 
>> > - Incorporate normalize.css and main.css from html5boilerplate.com.
>> > 
>> > - Improve the navigation bar to make the static pages mobile
>> > friendly, while keeping the site in a form that can be
>> > hand-edited.
>> > 
>> > * Swig bindings:
>> > 
>> > - SVN-4781 - expose svn_fs_change_rev_prop2() to swig.
>> > https://issues.apache.org/jira/browse/SVN-4781?issueNumber=4781
>> > 
>> > - SVN-4568 - expose svn_fs_set_warning_func() to swig.
>> > https://issues.apache.org/jira/browse/SVN-4568?issueNumber=4568
>> > 
>> > See:
>> > 
>> > https://svn.apache.org/repos/asf/subversion/trunk/subversion/bindings/swig/INSTALL
>> > 
>> > * C programming:
>> > 
>> > - SVN-4343 - FSFS backend work: "svnadmin verify" should verify
>> > changed-paths list.
>> > https://issues.apache.org/jira/browse/SVN-4343?issueNumber=4343
>> > 
>> > See:
>> > 
>> > https://svn.apache.org/repos/asf/subversion/trunk/subversion/libsvn_fs_fs/structure
>> > 
>> > - SVN-4605 - "svnadmin verify" doesn't verify locks.
>> > https://issues.apache.org/jira/browse/SVN-4605?issueNumber=4605
>> > 
>> > 
>> > Additional useful links:
>> > 
>> > https://ci.apache.org/projects/subversion/nightlies/index.html
>> > https://subversion.apache.org/patches
>> > https://subversion.apache.org/docs/community-guide/
>> > https://subversion.apache.org/docs/api/latest/
>> > https://subversion.apache.org/ideas.html
>> > 
>> > 
>> > If you have any questions, please let us know by emailing
>> > dev@subversion.apache.org!
>> > 
>> > Kind regards,
>> > Nathan Hartman,
>> > Apache Subversion
>> >

Re: Creating directory copy operations in a dump stream

[Johan Corveleyn]

On Fri, Oct 11, 2019 at 11:41 AM Eric S. Raymond  wrote:
>
> After years of eyeballing Subversion dump streams as part of making
> reposurgeon work, I have belatedly realized there is a dump stream
> operation I don't know how to generate with the CLI.  And I need to
> to get good test load coverage.
>
> If, in a normally-set up Subversion repository, I create and commit
> three files under trunk and then do "svn copy trunk branches/stable",
> what I see being generated into the dump is three file copies to
> stable.
>
> That's fine, but I also want to be able to generate a *directory* copy.
> I see these in the dump files I get in bug reports.  Here's an
> example:
>
> Node-path: tags/1.0rc1
> Node-kind: dir
> Node-action: add
> Node-copyfrom-rev: 5
> Node-copyfrom-path: tags/1.0
>
> How do I generate this kind of operation from the CLI - that is, a copy of the
> whole directory rather than a wildcarded copy of all of its files?

I'd try two things:

- Just guessing here, but perhaps the problem with your test is that
your working copy is "mixed-revision" after you added the three files.
I.e. the 'trunk' directory is a revision behind, after you added the
three files. See if it makes a difference if you do "svn up $wc_root"
before performing the "svn copy trunk branches/stable".

- Maybe you can get what you want by performing a server-side copy,
instead of in your working copy and then committing. I.e. "svn copy
$url/trunk $url/branches/stable".

-- 
Johan

Creating directory copy operations in a dump stream

[Eric S. Raymond]

After years of eyeballing Subversion dump streams as part of making
reposurgeon work, I have belatedly realized there is a dump stream
operation I don't know how to generate with the CLI.  And I need to 
to get good test load coverage.

If, in a normally-set up Subversion repository, I create and commit
three files under trunk and then do "svn copy trunk branches/stable",
what I see being generated into the dump is three file copies to
stable.

That's fine, but I also want to be able to generate a *directory* copy.
I see these in the dump files I get in bug reports.  Here's an
example:

Node-path: tags/1.0rc1
Node-kind: dir
Node-action: add
Node-copyfrom-rev: 5
Node-copyfrom-path: tags/1.0

How do I generate this kind of operation from the CLI - that is, a copy of the
whole directory rather than a wildcarded copy of all of its files?
-- 
http://www.catb.org/~esr/;>Eric S. Raymond

All governments are more or less combinations against the
people. . .and as rulers have no more virtue than the ruled. . .
the power of government can only be kept within its constituted
bounds by the display of a power equal to itself, the collected
sentiment of the people.
-- Benjamin Franklin Bache, in a Phildelphia Aurora editorial 1794

Adding Apache Subversion to the hackCBS 2.0 task list

[Sally Khudairi]

Thank you, Nathan.

I'm copying Rohan Vij from hackCBS 2.0 here so he can follow up with you 
directly.

@Rohan --kindly advise next steps.

Best regards, 
Sally

- - - 
Vice President Marketing & Publicity
Vice President Sponsor Relations
The Apache Software Foundation

Tel +1 617 921 8656 | s...@apache.org

On Thu, Oct 10, 2019, at 20:32, Nathan Hartman wrote:
>
> The Apache Subversion project has several opportunities for the 2-day
> hackCBS 2.0 event across these skill areas: web design, swig bindings,
> and C programming. Specific items are listed below.
> 
> Hackathon participants who are interested in working on Subversion
> should join our 'dev' mailing list and introduce themselves. To join,
> email dev-subscr...@subversion.apache.org. For details, see:
> https://subversion.apache.org/mailing-lists.html.
> 
> We'll be happy to provide whatever guidance we can, as well as review
> and (hopefully) apply some quality patches!
> 
> 
> Items:
> 
> * Web design on "next generation" https://subversion.apache.org:
> 
>   - Incorporate normalize.css and main.css from html5boilerplate.com.
> 
>   - Improve the navigation bar to make the static pages mobile
> friendly, while keeping the site in a form that can be
> hand-edited.
> 
> * Swig bindings:
> 
>   - SVN-4781 - expose svn_fs_change_rev_prop2() to swig.
> https://issues.apache.org/jira/browse/SVN-4781?issueNumber=4781
> 
>   - SVN-4568 - expose svn_fs_set_warning_func() to swig.
> https://issues.apache.org/jira/browse/SVN-4568?issueNumber=4568
> 
> See:
> 
> https://svn.apache.org/repos/asf/subversion/trunk/subversion/bindings/swig/INSTALL
> 
> * C programming:
> 
>   - SVN-4343 - FSFS backend work: "svnadmin verify" should verify
> changed-paths list.
> https://issues.apache.org/jira/browse/SVN-4343?issueNumber=4343
> 
> See:
> 
> https://svn.apache.org/repos/asf/subversion/trunk/subversion/libsvn_fs_fs/structure
> 
>   - SVN-4605 - "svnadmin verify" doesn't verify locks.
> https://issues.apache.org/jira/browse/SVN-4605?issueNumber=4605
> 
> 
> Additional useful links:
> 
> https://ci.apache.org/projects/subversion/nightlies/index.html
> https://subversion.apache.org/patches
> https://subversion.apache.org/docs/community-guide/
> https://subversion.apache.org/docs/api/latest/
> https://subversion.apache.org/ideas.html
> 
> 
> If you have any questions, please let us know by emailing
> dev@subversion.apache.org!
> 
> Kind regards,
> Nathan Hartman,
> Apache Subversion
>

Test failures with Python 3 (Re: PMCs: any Hackathon requests? (deadline 11 October))

[Yasuhito FUTATSUKI]

On 2019-10-11 06:45, Daniel Shahaf wrote:
...

 From autogen.sh to 'make check', all steps pass with PYTHON=/usr/bin/python3.


There were two failures in 1.11 on tests.
 
https://lists.apache.org/thread.html/366e77c7443a0c575b47de150a2c394e1e685fde8887c805a656a33d@%3Cusers.subversion.apache.org%3E


and it seems they still exist in trunk.

on FreeBSD 11.1, Python 3.7.0, Subversion trunk@1868235:
[[[
W: CWD: /home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline
Traceback (most recent call last):
  File 
"/home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline/svntest/main.py",
 line 1931, in run
rc = self.pred.run(sandbox)
  File 
"/home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline/svntest/testcase.py",
 line 178, in run
result = self.func(sandbox)
  File 
"/home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline/svnadmin_tests.py",
 line 3862, in dump_no_canonicalize_svndate
assert propval + '\n' in dump_lines
AssertionError
FAIL:  svnadmin_tests.py 69: svnadmin dump shouldn't canonicalize svn:date
]]]

[[[
W: A1/B/lambda has unexpectected contents
W: EXPECTED svn-test-work/working_copies/tree_conflict_tests-26/A1/B/lambda 
(match_all=True):
W: | This is the file 'lambda'.
W: | This is more content.
W: ACTUAL svn-test-work/working_copies/tree_conflict_tests-26/A1/B/lambda:
W: CWD: /home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline
Traceback (most recent call last):
  File 
"/home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline/svntest/main.py",
 line 1931, in run
rc = self.pred.run(sandbox)
  File 
"/home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline/svntest/testcase.py",
 line 178, in run
result = self.func(sandbox)
  File 
"/home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline/tree_conflict_tests.py",
 line 1547, in local_missing_dir_endless_loop
[ "This is the file 'lambda'.\n", "This is more content.\n"], contents)
  File 
"/home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline/svntest/verify.py",
 line 503, in compare_and_display_lines
expected.display_differences(message, label, actual)
  File 
"/home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline/svntest/verify.py",
 line 154, in display_differences
display_lines(message, self.expected, actual, e_label, label)
  File 
"/home/futatuki/work/subversion/vwc/trunk/subversion/tests/cmdline/svntest/verify.py",
 line 478, in display_lines
logger.warn('| ' + x.rstrip())
TypeError: can only concatenate str (not "bytes") to str
FAIL:  tree_conflict_tests.py 26: endless loop when resolving local-missing dir
]]]

The former failure can be fixed by attached patch, fix_svnadmin_tests_patch.txt

The latter also can be fixed by fix_tree_conflict_tests_patch.txt
at least so that the test can be passed on Python 3. However, from above
stack trace, I think it is incomplete because there still exists some sort of
confusion between bytes and str in subversion/tests/cmdline/svntest/*.py

Cheers,
--
Yasuhito FUTATSUKI 
Index: subversion/tests/cmdline/svnadmin_tests.py
===
--- subversion/tests/cmdline/svnadmin_tests.py  (revision 1868264)
+++ subversion/tests/cmdline/svnadmin_tests.py  (working copy)
@@ -3859,7 +3859,7 @@
  sbox.repo_url)
 
   dump_lines = svntest.actions.run_and_verify_dump(sbox.repo_dir)
-  assert propval + '\n' in dump_lines
+  assert propval.encode() + b'\n' in dump_lines
 
 def check_recover_prunes_rep_cache(sbox, enable_rep_sharing):
   """Check 'recover' prunes the rep-cache while enable-rep-sharing is
Index: subversion/tests/cmdline/tree_conflict_tests.py
===
--- subversion/tests/cmdline/tree_conflict_tests.py (revision 1868264)
+++ subversion/tests/cmdline/tree_conflict_tests.py (working copy)
@@ -1544,7 +1544,7 @@
   contents = open(sbox.ospath('A1/B/lambda'), 'rb').readlines()
   svntest.verify.compare_and_display_lines(
 "A1/B/lambda has unexpectected contents", sbox.ospath("A1/B/lambda"),
-[ "This is the file 'lambda'.\n", "This is more content.\n"], contents)
+[ b"This is the file 'lambda'.\n", b"This is more content.\n"], contents)
 
 
 ###