Re: Problems running testsuite on Windows with Python 3
On 2020/05/09 6:43, Johan Corveleyn wrote: > On Fri, May 8, 2020 at 2:49 PM Jun Omae wrote: >> >> Hi, >> >> On 2020/05/07 16:17, Johan Corveleyn wrote: >>> There is still the issue of PYTHONLEGACYWINDOWSSTDIO that is very much >>> blocking for anyone running the testsuite on Windows with Python 3. >>> Worst case, we might have to simply document it, and perhaps emit a >>> warning if it's not set? Or can we simply set that envvar ourselves, >>> automatically, from within some central place in the testsuite (as >>> soon as redirection to a logfile has been requested or something)? >>> Unless you still have some more magic up your sleeve to handle this, >>> Yasuhito :-). >> >> I created patch to resolve the issue of PYTHONLEGACYWINDOWSSTDIO and remove >> uses of os.dup2() in run_tests.py. >> After attached patch, PYTHONLEGACYWINDOWSSTDIO is not needed. >> >> [[[ >> * build/run_tests.py >>(open_logfile): New function returning file-like object which is >> reassignable >>sys.stdout and sys.stderr. >>(TestHarness.run): Use open_logfile() instead of codecs.open(). >>(TestHarness._open_log): Ditto. >>(TestHarness._run_py_test): Reassign sys.stdout and sys.stderr instead of >>uses of os.dup2(). >> >> * subversion/tests/cmdline/svntest/main.py >>(LoggingStdoutHandler): New function to use the value of sys.stdout at >> call >>time. >>(parse_options): Use LoggingStdoutHandler() instead of >>StreamHandler(sys.stdout). >> ]]] >> >> Tested with the following environments: >> >> * Python 3.8.2 and Python 2.7.18 on Windows 10 >> * Python 3.8.2 on Linux >> >> -- >> Jun Omae (大前 潤) > > Nice! I can confirm that this does indeed work perfectly under both > Python 3.8.2 and Python 2.7.17 on Windows 10, without the envvar set. > > Great work :-). +1 to commit. Cheers, -- Yasuhito FUTATSUKI /
Re: Problems running testsuite on Windows with Python 3
On Fri, May 8, 2020 at 2:49 PM Jun Omae wrote: > > Hi, > > On 2020/05/07 16:17, Johan Corveleyn wrote: > > There is still the issue of PYTHONLEGACYWINDOWSSTDIO that is very much > > blocking for anyone running the testsuite on Windows with Python 3. > > Worst case, we might have to simply document it, and perhaps emit a > > warning if it's not set? Or can we simply set that envvar ourselves, > > automatically, from within some central place in the testsuite (as > > soon as redirection to a logfile has been requested or something)? > > Unless you still have some more magic up your sleeve to handle this, > > Yasuhito :-). > > I created patch to resolve the issue of PYTHONLEGACYWINDOWSSTDIO and remove > uses of os.dup2() in run_tests.py. > After attached patch, PYTHONLEGACYWINDOWSSTDIO is not needed. > > [[[ > * build/run_tests.py >(open_logfile): New function returning file-like object which is > reassignable >sys.stdout and sys.stderr. >(TestHarness.run): Use open_logfile() instead of codecs.open(). >(TestHarness._open_log): Ditto. >(TestHarness._run_py_test): Reassign sys.stdout and sys.stderr instead of >uses of os.dup2(). > > * subversion/tests/cmdline/svntest/main.py >(LoggingStdoutHandler): New function to use the value of sys.stdout at call >time. >(parse_options): Use LoggingStdoutHandler() instead of >StreamHandler(sys.stdout). > ]]] > > Tested with the following environments: > > * Python 3.8.2 and Python 2.7.18 on Windows 10 > * Python 3.8.2 on Linux > > -- > Jun Omae (大前 潤) Nice! I can confirm that this does indeed work perfectly under both Python 3.8.2 and Python 2.7.17 on Windows 10, without the envvar set. Great work :-). -- Johan
Re: Problems running testsuite on Windows with Python 3
Hi,
On 2020/05/07 16:17, Johan Corveleyn wrote:
There is still the issue of PYTHONLEGACYWINDOWSSTDIO that is very much
blocking for anyone running the testsuite on Windows with Python 3.
Worst case, we might have to simply document it, and perhaps emit a
warning if it's not set? Or can we simply set that envvar ourselves,
automatically, from within some central place in the testsuite (as
soon as redirection to a logfile has been requested or something)?
Unless you still have some more magic up your sleeve to handle this,
Yasuhito :-).
I created patch to resolve the issue of PYTHONLEGACYWINDOWSSTDIO and remove
uses of os.dup2() in run_tests.py.
After attached patch, PYTHONLEGACYWINDOWSSTDIO is not needed.
[[[
* build/run_tests.py
(open_logfile): New function returning file-like object which is reassignable
sys.stdout and sys.stderr.
(TestHarness.run): Use open_logfile() instead of codecs.open().
(TestHarness._open_log): Ditto.
(TestHarness._run_py_test): Reassign sys.stdout and sys.stderr instead of
uses of os.dup2().
* subversion/tests/cmdline/svntest/main.py
(LoggingStdoutHandler): New function to use the value of sys.stdout at call
time.
(parse_options): Use LoggingStdoutHandler() instead of
StreamHandler(sys.stdout).
]]]
Tested with the following environments:
* Python 3.8.2 and Python 2.7.18 on Windows 10
* Python 3.8.2 on Linux
--
Jun Omae (大前 潤)
* build/run_tests.py
(open_logfile): New function returning file-like object which is reassignable
sys.stdout and sys.stderr.
(TestHarness.run): Use open_logfile() instead of codecs.open().
(TestHarness._open_log): Ditto.
(TestHarness._run_py_test): Reassign sys.stdout and sys.stderr instead of
uses of os.dup2().
* subversion/tests/cmdline/svntest/main.py
(LoggingStdoutHandler): New function to use the value of sys.stdout at call
time.
(parse_options): Use LoggingStdoutHandler() instead of
StreamHandler(sys.stdout).
Index: build/run_tests.py
===
--- build/run_tests.py (revision 1877480)
+++ build/run_tests.py (working copy)
@@ -47,7 +47,7 @@
separated list of test numbers; the default is to run all the tests in it.
'''
-import os, sys, shutil, codecs
+import os, sys, shutil
import re
import logging
import optparse, subprocess, threading, traceback
@@ -141,6 +141,18 @@
else:
return s.decode("latin-1")
+def open_logfile(filename, mode, encoding='utf-8'):
+ if sys.version_info[0] != 2:
+return open(filename, mode, encoding=encoding, errors='surrogateescape')
+ else:
+class Wrapper(object):
+ def __init__(self, stream, encoding):
+self._stream = stream
+self.encoding = encoding
+ def __getattr__(self, name):
+return getattr(self._stream, name)
+return Wrapper(open(filename, mode), encoding)
+
class TestHarness:
'''Test harness for Subversion tests.
'''
@@ -700,7 +712,7 @@
# Copy the truly interesting verbose logs to a separate file, for easier
# viewing.
if xpassed or failed_list:
- faillog = codecs.open(self.faillogfile, 'w', encoding="latin-1")
+ faillog = open_logfile(self.faillogfile, 'w')
last_start_lineno = None
last_start_re = re.compile('^(FAIL|SKIP|XFAIL|PASS|START|CLEANUP|END):')
for lineno, line in enumerate(log_lines):
@@ -733,7 +745,7 @@
'Open the log file with the required MODE.'
if self.logfile:
self._close_log()
- self.log = codecs.open(self.logfile, mode, encoding="latin-1")
+ self.log = open_logfile(self.logfile, mode)
def _close_log(self):
'Close the log file.'
@@ -843,14 +855,13 @@
sys.exit(1)
# setup the output pipes
+old_stdout = sys.stdout.fileno()
if self.log:
sys.stdout.flush()
sys.stderr.flush()
self.log.flush()
- old_stdout = os.dup(sys.stdout.fileno())
- old_stderr = os.dup(sys.stderr.fileno())
- os.dup2(self.log.fileno(), sys.stdout.fileno())
- os.dup2(self.log.fileno(), sys.stderr.fileno())
+ saved_stds = sys.stdout, sys.stderr
+ sys.stdout = sys.stderr = self.log
# These have to be class-scoped for use in the progress_func()
self.dots_written = 0
@@ -891,12 +902,8 @@
# restore some values
if self.log:
- sys.stdout.flush()
- sys.stderr.flush()
- os.dup2(old_stdout, sys.stdout.fileno())
- os.dup2(old_stderr, sys.stderr.fileno())
- os.close(old_stdout)
- os.close(old_stderr)
+ self.log.flush()
+ sys.stdout, sys.stderr = saved_stds
return failed
Index: subversion/tests/cmdline/svntest/main.py
===
--- subversion/tests/cmdline/svntest/main.py(revision 1877480)
+++ subversion/tests/cmdline/svntest/main.py(working copy)
@@ -2084,6 +2084,23 @@
record.levelshort = self._level_short[record.levelno]
return logging.Formatter.format(self, record)
Re: Problems running testsuite on Windows with Python 3
Johan Corveleyn wrote on Thu, 07 May 2020 09:17 +0200: > We should probably also propose all of your latest fixes for backport > in /branches/1.14.x/STATUS, so they can be included in 1.14.1 (and > interested people can already see that the "known testsuite issues on > Windows" are being addressed). > Should we add something to either of these two sections? — https://subversion.apache.org/docs/release-notes/1.14.html#python3 https://subversion.apache.org/docs/release-notes/1.14.html#python3-work-in-progress (despite the anchors' names, the second is not a subsection of the first) Cheers, Daniel
Re: Problems running testsuite on Windows with Python 3
On 2020/05/07 16:17, Johan Corveleyn wrote: > On Wed, May 6, 2020 at 10:36 AM Yasuhito FUTATSUKI > wrote: >> >> On 2020/05/04 4:49, Johan Corveleyn wrote: >>> On Sun, May 3, 2020 at 6:23 PM Yasuhito FUTATSUKI >>> wrote: On 2020/05/03 10:02, Johan Corveleyn wrote: > On Fri, May 1, 2020 at 7:46 PM Yasuhito FUTATSUKI > wrote: >> >> On 2020/04/23 2:05, Yasuhito FUTATSUKI wrote: >> >> It seems following tests are EOL style sensitive, but they didn't >> check without strict EOL style on Windows: >> >> merge_tests.merge_conflict_markers_matching_eol >> merge_tests.merge_eolstyle_handling >> patch_tests.patch_no_svn_eol_style >> patch_tests.patch_with_svn_eol_style >> patch_tests.patch_with_svn_eol_style_uncommitted >> update_tests.conflict_markers_matching_eol >> update_tests.update_eol_style_handling >> >> I've not check that each tests depend that native EOL is '\n' or not. >> However, I think if some tests depend on it, other test cases for >> those aims are needed on Windows, or at least explicitly skip the tests >> on Windows to clarify that those tests check nothing for the aims. >> >> The updated patch attached only make these tests EOL style sensitive >> (and relax check of EOL on Python 2 if keep_eol_style optional >> parameter is not specified). > > Thank you for continuing to work on this, Yasuhito. > > This patch indeed fixes the above 7 tests when running with Python 3 on > Windows. > However, when running with Python 2 there are now 6 tests that fail > (without this patch, all tests are successful with Py2): Thank you for testing. As far as this test result, those test cases don't depend on specific native EOL. > FAIL: merge_tests.py 34: conflict markers should match the file's eol > style > FAIL: patch_tests.py 13: patch target with no svn:eol-style > FAIL: patch_tests.py 14: patch target with svn:eol-style > FAIL: patch_tests.py 15: patch target with uncommitted svn:eol-style > FAIL: patch_tests.py 57: patch a binary file > FAIL: update_tests.py 26: conflict markers should match the file's eol > style > > See fails.log in attachment. I overlooked that result of io.TextIO.read() is unicode on Python 2. I hope updated patch may resolve this issue. >>> >>> Okay, that latest version fixes this test for Python 2: >>> >>> patch_tests.py 57: patch a binary file >>> >>> But the five other failures still remain for Py2: >>> >>> FAIL: merge_tests.py 34: conflict markers should match the file's eol style >>> FAIL: patch_tests.py 13: patch target with no svn:eol-style >>> FAIL: patch_tests.py 14: patch target with svn:eol-style >>> FAIL: patch_tests.py 15: patch target with uncommitted svn:eol-style >>> FAIL: update_tests.py 26: conflict markers should match the file's eol >>> style >>> >>> See fails_py2.log in attachment. >> >> I read those test code again, and I found that they don't distinct >> text I/O and binary I/O when they write to files in some case, So I >> updated the patch address them. (sain_keep_eol_style_win_patch_20200506.txt) > > Yes, it does :-). Perfect! With this version of the patch, all tests > pass again with Python 2. Thanks. I think they do what we want them to do now. >>> For Python 3 we're getting close. After this patch and the one for >>> svnrdump_tests, we only have 2 failures left with Python 3: >> >> Please don't forget our goal is not to make the tests be passed but >> to make the tests check what we want to check, on Windows both with >> py2 and py3. > > Agreed. And thank you for being so thorough. It's great that you took > the time to analyse the tests so deeply. > >> I've fixed the test codes to make them what the author intended I think. >> If the results of py2 and py3 are still different, the test code are >> still broken. With a broken test, we can't warrant ether the test target >> code is correct or the test case is correct, even if the tests can be >> passed. So we should also check the test cases (scenarios and expected >> results) themselves are correct on Windows(, and I didn't it... at >> least, yet). Now I've confirmed those cases themselves. It seems they don't depend on what is EOL character(s), except value of variable "native_nl". >>> FAIL: svnadmin_tests.py 35: detect denormalized names and name collisions >> >> This is caused by output message of svnadmin, containing non UTF-8 >> character. Attached patch fix_svnadmin_tests_patch.txt address it, >> however I don't have confidence because I don't know what charset/encoding >> svnadmin on Windows use. With Python 2, output message is treated >> as bytes as is, this is not affected. > > Well, I can confirm that the test passes on my Windows 10 system with > this patch. Both with Python 3 and Python 2. I can't comment on the > general validity (this is a
Re: Problems running testsuite on Windows with Python 3
On Wed, May 6, 2020 at 10:36 AM Yasuhito FUTATSUKI wrote: > > On 2020/05/04 4:49, Johan Corveleyn wrote: > > On Sun, May 3, 2020 at 6:23 PM Yasuhito FUTATSUKI > > wrote: > >> > >> On 2020/05/03 10:02, Johan Corveleyn wrote: > >>> On Fri, May 1, 2020 at 7:46 PM Yasuhito FUTATSUKI > >>> wrote: > > On 2020/04/23 2:05, Yasuhito FUTATSUKI wrote: > > It seems following tests are EOL style sensitive, but they didn't > check without strict EOL style on Windows: > > merge_tests.merge_conflict_markers_matching_eol > merge_tests.merge_eolstyle_handling > patch_tests.patch_no_svn_eol_style > patch_tests.patch_with_svn_eol_style > patch_tests.patch_with_svn_eol_style_uncommitted > update_tests.conflict_markers_matching_eol > update_tests.update_eol_style_handling > > I've not check that each tests depend that native EOL is '\n' or not. > However, I think if some tests depend on it, other test cases for > those aims are needed on Windows, or at least explicitly skip the tests > on Windows to clarify that those tests check nothing for the aims. > > The updated patch attached only make these tests EOL style sensitive > (and relax check of EOL on Python 2 if keep_eol_style optional > parameter is not specified). > >>> > >>> Thank you for continuing to work on this, Yasuhito. > >>> > >>> This patch indeed fixes the above 7 tests when running with Python 3 on > >>> Windows. > >>> However, when running with Python 2 there are now 6 tests that fail > >>> (without this patch, all tests are successful with Py2): > >> > >> Thank you for testing. > >> > >> As far as this test result, those test cases don't depend on specific > >> native EOL. > >> > >>> FAIL: merge_tests.py 34: conflict markers should match the file's eol > >>> style > >>> FAIL: patch_tests.py 13: patch target with no svn:eol-style > >>> FAIL: patch_tests.py 14: patch target with svn:eol-style > >>> FAIL: patch_tests.py 15: patch target with uncommitted svn:eol-style > >>> FAIL: patch_tests.py 57: patch a binary file > >>> FAIL: update_tests.py 26: conflict markers should match the file's eol > >>> style > >>> > >>> See fails.log in attachment. > >> > >> I overlooked that result of io.TextIO.read() is unicode on Python 2. > >> I hope updated patch may resolve this issue. > > > > Okay, that latest version fixes this test for Python 2: > > > > patch_tests.py 57: patch a binary file > > > > But the five other failures still remain for Py2: > > > > FAIL: merge_tests.py 34: conflict markers should match the file's eol style > > FAIL: patch_tests.py 13: patch target with no svn:eol-style > > FAIL: patch_tests.py 14: patch target with svn:eol-style > > FAIL: patch_tests.py 15: patch target with uncommitted svn:eol-style > > FAIL: update_tests.py 26: conflict markers should match the file's eol > > style > > > > See fails_py2.log in attachment. > > I read those test code again, and I found that they don't distinct > text I/O and binary I/O when they write to files in some case, So I > updated the patch address them. (sain_keep_eol_style_win_patch_20200506.txt) Yes, it does :-). Perfect! With this version of the patch, all tests pass again with Python 2. > > For Python 3 we're getting close. After this patch and the one for > > svnrdump_tests, we only have 2 failures left with Python 3: > > Please don't forget our goal is not to make the tests be passed but > to make the tests check what we want to check, on Windows both with > py2 and py3. Agreed. And thank you for being so thorough. It's great that you took the time to analyse the tests so deeply. > I've fixed the test codes to make them what the author intended I think. > If the results of py2 and py3 are still different, the test code are > still broken. With a broken test, we can't warrant ether the test target > code is correct or the test case is correct, even if the tests can be > passed. So we should also check the test cases (scenarios and expected > results) themselves are correct on Windows(, and I didn't it... at > least, yet). > > > FAIL: svnadmin_tests.py 35: detect denormalized names and name collisions > > This is caused by output message of svnadmin, containing non UTF-8 > character. Attached patch fix_svnadmin_tests_patch.txt address it, > however I don't have confidence because I don't know what charset/encoding > svnadmin on Windows use. With Python 2, output message is treated > as bytes as is, this is not affected. Well, I can confirm that the test passes on my Windows 10 system with this patch. Both with Python 3 and Python 2. I can't comment on the general validity (this is a bit above my head), but it does work :-). > > FAIL: svndumpfilter_tests.py 7: svndumpfilter with an empty prefix > > It seems this is just same reason as svnrdump_tests.py was broken. > (fix_svndumpfilter_tests_patch.txt) Yes, perfect! It works. Awesome work, Yasuhito! With these
Re: Problems running testsuite on Windows with Python 3
On 2020/05/04 4:49, Johan Corveleyn wrote: > On Sun, May 3, 2020 at 6:23 PM Yasuhito FUTATSUKI wrote: >> >> On 2020/05/03 10:02, Johan Corveleyn wrote: >>> On Fri, May 1, 2020 at 7:46 PM Yasuhito FUTATSUKI >>> wrote: On 2020/04/23 2:05, Yasuhito FUTATSUKI wrote: It seems following tests are EOL style sensitive, but they didn't check without strict EOL style on Windows: merge_tests.merge_conflict_markers_matching_eol merge_tests.merge_eolstyle_handling patch_tests.patch_no_svn_eol_style patch_tests.patch_with_svn_eol_style patch_tests.patch_with_svn_eol_style_uncommitted update_tests.conflict_markers_matching_eol update_tests.update_eol_style_handling I've not check that each tests depend that native EOL is '\n' or not. However, I think if some tests depend on it, other test cases for those aims are needed on Windows, or at least explicitly skip the tests on Windows to clarify that those tests check nothing for the aims. The updated patch attached only make these tests EOL style sensitive (and relax check of EOL on Python 2 if keep_eol_style optional parameter is not specified). >>> >>> Thank you for continuing to work on this, Yasuhito. >>> >>> This patch indeed fixes the above 7 tests when running with Python 3 on >>> Windows. >>> However, when running with Python 2 there are now 6 tests that fail >>> (without this patch, all tests are successful with Py2): >> >> Thank you for testing. >> >> As far as this test result, those test cases don't depend on specific >> native EOL. >> >>> FAIL: merge_tests.py 34: conflict markers should match the file's eol style >>> FAIL: patch_tests.py 13: patch target with no svn:eol-style >>> FAIL: patch_tests.py 14: patch target with svn:eol-style >>> FAIL: patch_tests.py 15: patch target with uncommitted svn:eol-style >>> FAIL: patch_tests.py 57: patch a binary file >>> FAIL: update_tests.py 26: conflict markers should match the file's eol >>> style >>> >>> See fails.log in attachment. >> >> I overlooked that result of io.TextIO.read() is unicode on Python 2. >> I hope updated patch may resolve this issue. > > Okay, that latest version fixes this test for Python 2: > > patch_tests.py 57: patch a binary file > > But the five other failures still remain for Py2: > > FAIL: merge_tests.py 34: conflict markers should match the file's eol style > FAIL: patch_tests.py 13: patch target with no svn:eol-style > FAIL: patch_tests.py 14: patch target with svn:eol-style > FAIL: patch_tests.py 15: patch target with uncommitted svn:eol-style > FAIL: update_tests.py 26: conflict markers should match the file's eol style > > See fails_py2.log in attachment. I read those test code again, and I found that they don't distinct text I/O and binary I/O when they write to files in some case, So I updated the patch address them. (sain_keep_eol_style_win_patch_20200506.txt) > For Python 3 we're getting close. After this patch and the one for > svnrdump_tests, we only have 2 failures left with Python 3: Please don't forget our goal is not to make the tests be passed but to make the tests check what we want to check, on Windows both with py2 and py3. I've fixed the test codes to make them what the author intended I think. If the results of py2 and py3 are still different, the test code are still broken. With a broken test, we can't warrant ether the test target code is correct or the test case is correct, even if the tests can be passed. So we should also check the test cases (scenarios and expected results) themselves are correct on Windows(, and I didn't it... at least, yet). > FAIL: svnadmin_tests.py 35: detect denormalized names and name collisions This is caused by output message of svnadmin, containing non UTF-8 character. Attached patch fix_svnadmin_tests_patch.txt address it, however I don't have confidence because I don't know what charset/encoding svnadmin on Windows use. With Python 2, output message is treated as bytes as is, this is not affected. > FAIL: svndumpfilter_tests.py 7: svndumpfilter with an empty prefix It seems this is just same reason as svnrdump_tests.py was broken. (fix_svndumpfilter_tests_patch.txt) Cheers, -- Yasuhito FUTATSUKI / Fix eol style treatment in command tests on Windows. [in subversion/tests/cmdline] * merge_tests.py (merge_conflict_markers_matching_eol, merge_eolstyle_handling), * patch_tests.py (patch_no_svn_eol_style, patch_with_svn_eol_style, patch_with_svn_eol_style_uncommitted), * update_tests.py (conflict_markers_matching_eol, update_eol_style_handling): Specify keep_eol_style = True evne if platform is Windows. * merge_tests.py (merge_conflict_markers_matching_eol), * patch_tests.py (patch_no_svn_eol_style, patch_with_svn_eol_style, patch_with_svn_eol_style_uncommitted), * update_tests.py
Re: Problems running testsuite on Windows with Python 3
On Sun, May 3, 2020 at 6:23 PM Yasuhito FUTATSUKI wrote: > > On 2020/05/03 10:02, Johan Corveleyn wrote: > > On Fri, May 1, 2020 at 7:46 PM Yasuhito FUTATSUKI > > wrote: > >> > >> On 2020/04/23 2:05, Yasuhito FUTATSUKI wrote: > >>> On 2020/04/22 23:04, Johan Corveleyn wrote: > On Wed, Apr 22, 2020 at 3:50 PM Yasuhito FUTATSUKI > wrote: > >>> > > ... and what is worse, at least > > > >> FAIL: merge_tests.py 34: conflict markers should match the file's eol > >> style > > > > this one seems to be broken even with Python 2.7, on Windows. > > (I'll post about it later). > > That's strange. It does succeed on my system when running with Python > 2.7.17. I had "All successful" test runs for [fsfs] x [ra_local, > ra_serf, ra_svn]. > >>> > >>> Yes, I don't doubt this test is passed on Windows with Python 2.7, > >>> but I doubt this test does not check just what we want to check. > >>> > >>> As far as I read the description of the test and about KEEP_EOL_STYLE > >>> option introduced in r1743445. > >>> > >>> merge_tests.py (merge_conflict_markers_matching_eol): > >>> [[[ > >>> # eol-style handling during merge with conflicts, scenario 1: > >>> # when a merge creates a conflict on a file, make sure the file and files > >>> # r, r and .mine are in the eol-style defined for that file. > >>> ]]] > >>> > >>> So I think comparion of expected and actual should be done without > >>> eol-style translation. However it try to check with translation on > >>> Windows environment. > >> > >> It seems following tests are EOL style sensitive, but they didn't > >> check without strict EOL style on Windows: > >> > >> merge_tests.merge_conflict_markers_matching_eol > >> merge_tests.merge_eolstyle_handling > >> patch_tests.patch_no_svn_eol_style > >> patch_tests.patch_with_svn_eol_style > >> patch_tests.patch_with_svn_eol_style_uncommitted > >> update_tests.conflict_markers_matching_eol > >> update_tests.update_eol_style_handling > >> > >> I've not check that each tests depend that native EOL is '\n' or not. > >> However, I think if some tests depend on it, other test cases for > >> those aims are needed on Windows, or at least explicitly skip the tests > >> on Windows to clarify that those tests check nothing for the aims. > >> > >> The updated patch attached only make these tests EOL style sensitive > >> (and relax check of EOL on Python 2 if keep_eol_style optional > >> parameter is not specified). > > > > Thank you for continuing to work on this, Yasuhito. > > > > This patch indeed fixes the above 7 tests when running with Python 3 on > > Windows. > > However, when running with Python 2 there are now 6 tests that fail > > (without this patch, all tests are successful with Py2): > > Thank you for testing. > > As far as this test result, those test cases don't depend on specific > native EOL. > > > FAIL: merge_tests.py 34: conflict markers should match the file's eol style > > FAIL: patch_tests.py 13: patch target with no svn:eol-style > > FAIL: patch_tests.py 14: patch target with svn:eol-style > > FAIL: patch_tests.py 15: patch target with uncommitted svn:eol-style > > FAIL: patch_tests.py 57: patch a binary file > > FAIL: update_tests.py 26: conflict markers should match the file's eol > > style > > > > See fails.log in attachment. > > I overlooked that result of io.TextIO.read() is unicode on Python 2. > I hope updated patch may resolve this issue. Okay, that latest version fixes this test for Python 2: patch_tests.py 57: patch a binary file But the five other failures still remain for Py2: FAIL: merge_tests.py 34: conflict markers should match the file's eol style FAIL: patch_tests.py 13: patch target with no svn:eol-style FAIL: patch_tests.py 14: patch target with svn:eol-style FAIL: patch_tests.py 15: patch target with uncommitted svn:eol-style FAIL: update_tests.py 26: conflict markers should match the file's eol style See fails_py2.log in attachment. For Python 3 we're getting close. After this patch and the one for svnrdump_tests, we only have 2 failures left with Python 3: FAIL: svnadmin_tests.py 35: detect denormalized names and name collisions FAIL: svndumpfilter_tests.py 7: svndumpfilter with an empty prefix See fails_py3.log in attachment. -- Johan fails_py2.log Description: Binary data fails_py3.log Description: Binary data
Re: Problems running testsuite on Windows with Python 3
On 2020/05/03 10:02, Johan Corveleyn wrote:
> On Fri, May 1, 2020 at 7:46 PM Yasuhito FUTATSUKI wrote:
>>
>> On 2020/04/23 2:05, Yasuhito FUTATSUKI wrote:
>>> On 2020/04/22 23:04, Johan Corveleyn wrote:
On Wed, Apr 22, 2020 at 3:50 PM Yasuhito FUTATSUKI
wrote:
>>>
> ... and what is worse, at least
>
>> FAIL: merge_tests.py 34: conflict markers should match the file's eol
>> style
>
> this one seems to be broken even with Python 2.7, on Windows.
> (I'll post about it later).
That's strange. It does succeed on my system when running with Python
2.7.17. I had "All successful" test runs for [fsfs] x [ra_local,
ra_serf, ra_svn].
>>>
>>> Yes, I don't doubt this test is passed on Windows with Python 2.7,
>>> but I doubt this test does not check just what we want to check.
>>>
>>> As far as I read the description of the test and about KEEP_EOL_STYLE
>>> option introduced in r1743445.
>>>
>>> merge_tests.py (merge_conflict_markers_matching_eol):
>>> [[[
>>> # eol-style handling during merge with conflicts, scenario 1:
>>> # when a merge creates a conflict on a file, make sure the file and files
>>> # r, r and .mine are in the eol-style defined for that file.
>>> ]]]
>>>
>>> So I think comparion of expected and actual should be done without
>>> eol-style translation. However it try to check with translation on
>>> Windows environment.
>>
>> It seems following tests are EOL style sensitive, but they didn't
>> check without strict EOL style on Windows:
>>
>> merge_tests.merge_conflict_markers_matching_eol
>> merge_tests.merge_eolstyle_handling
>> patch_tests.patch_no_svn_eol_style
>> patch_tests.patch_with_svn_eol_style
>> patch_tests.patch_with_svn_eol_style_uncommitted
>> update_tests.conflict_markers_matching_eol
>> update_tests.update_eol_style_handling
>>
>> I've not check that each tests depend that native EOL is '\n' or not.
>> However, I think if some tests depend on it, other test cases for
>> those aims are needed on Windows, or at least explicitly skip the tests
>> on Windows to clarify that those tests check nothing for the aims.
>>
>> The updated patch attached only make these tests EOL style sensitive
>> (and relax check of EOL on Python 2 if keep_eol_style optional
>> parameter is not specified).
>
> Thank you for continuing to work on this, Yasuhito.
>
> This patch indeed fixes the above 7 tests when running with Python 3 on
> Windows.
> However, when running with Python 2 there are now 6 tests that fail
> (without this patch, all tests are successful with Py2):
Thank you for testing.
As far as this test result, those test cases don't depend on specific
native EOL.
> FAIL: merge_tests.py 34: conflict markers should match the file's eol style
> FAIL: patch_tests.py 13: patch target with no svn:eol-style
> FAIL: patch_tests.py 14: patch target with svn:eol-style
> FAIL: patch_tests.py 15: patch target with uncommitted svn:eol-style
> FAIL: patch_tests.py 57: patch a binary file
> FAIL: update_tests.py 26: conflict markers should match the file's eol style
>
> See fails.log in attachment.
I overlooked that result of io.TextIO.read() is unicode on Python 2.
I hope updated patch may resolve this issue.
Thanks,
--
Yasuhito FUTATSUKI
Fix eol style treatment in command tests on Windows.
[in subversion/tests/cmdline]
* merge_tests.py (merge_conflict_markers_matching_eol,
merge_eolstyle_handling),
* patch_tests.py (patch_no_svn_eol_style,
patch_with_svn_eol_style,
patch_with_svn_eol_style_uncommitted),
* update_tests.py (conflict_markers_matching_eol,
update_eol_style_handling):
Specify keep_eol_style = True evne if platform is Windows.
* merge_tests.py (merge_conflict_markers_matching_eol),
* patch_tests.py (patch_with_svn_eol_style,
patch_with_svn_eol_style_uncommitted),
* conflict_markers_matching_eol):
Switch per platform eol value for 'native' svn:eol-style
* svntest/wc.py (State.from_wc):
Use io.open() explicitly to specify 'newline' parameter for universal newline,
even on Python 2. With this change, '\r' end of line chracters in files
are also translated to '\n' if keep_eol_style=False (or unspicified) on
Python 2.
Index: subversion/tests/cmdline/merge_tests.py
===
--- subversion/tests/cmdline/merge_tests.py (revision 1877314)
+++ subversion/tests/cmdline/merge_tests.py (working copy)
@@ -3323,16 +3323,12 @@
mu_path = sbox.ospath('A/mu')
- # CRLF is a string that will match a CRLF sequence read from a text file.
- # ### On Windows, we assume CRLF will be read as LF, so it's a poor test.
if os.name == 'nt':
-crlf = '\n'
+native_nl = '\r\n'
else:
-crlf = '\r\n'
+native_nl = '\n'
+ crlf = '\r\n'
- # Strict EOL style matching breaks Windows tests at least with Python 2
- keep_eol_style = not
Re: [PATCH] fix svnrdump_tests.py on Windows with Python 3 (Re: Problems running testsuite on Windows with Python 3)
On 2020/05/03 10:04, Johan Corveleyn wrote:
> On Sat, May 2, 2020 at 7:37 PM Yasuhito FUTATSUKI wrote:
>>
>> On 2020/04/19 3:26, Johan Corveleyn wrote:
>>
>>> This is the list of tests that still fail (66):
>>
>>> FAIL: svnrdump_tests.py 1: dump: standard sbox repos
>> ...
>>> FAIL: svnrdump_tests.py 57: load: invalid svn:date revprop in r1
>>
>> These failures in svnrdump_tests.py seems to be caused in EOL character
>> conversion only for Windows, and by change of map() function return
>> value type between Python 2 and 3.
>>
>> The patch below address to it. This will fix the exceptions in
>> fails2.log, however other problems can be found.
>> [[[
>> Index: subversion/tests/cmdline/svntest/actions.py
>> ===
>> --- subversion/tests/cmdline/svntest/actions.py (revision 1877286)
>> +++ subversion/tests/cmdline/svntest/actions.py (working copy)
>> @@ -418,7 +418,7 @@
>># Since main.run_svnrdump() uses binary mode, normalize the stderr
>># line endings on Windows ourselves.
>>if sys.platform == 'win32':
>> -err = map(lambda x : x.replace('\r\n', '\n'), err)
>> +err = [x.replace('\r\n', '\n') for x in err]
>>
>># Ignore "consider upgrade" warnings to allow regression tests to pass
>># when run against a 1.6 mod_dav_svn.
>> ]]]
>
> Perfect! This fixes all the svnrdump tests for Python 3 on Windows
> (and to be sure I confirmed they also still work with Python 2).
Thanks! commited in 1877318.
> Thanks,
>
Thanks,
--
Yasuhito FUTATSUKI /
Re: [PATCH] fix svnrdump_tests.py on Windows with Python 3 (Re: Problems running testsuite on Windows with Python 3)
On Sat, May 2, 2020 at 7:37 PM Yasuhito FUTATSUKI wrote:
>
> On 2020/04/19 3:26, Johan Corveleyn wrote:
>
> > This is the list of tests that still fail (66):
>
> > FAIL: svnrdump_tests.py 1: dump: standard sbox repos
> ...
> > FAIL: svnrdump_tests.py 57: load: invalid svn:date revprop in r1
>
> These failures in svnrdump_tests.py seems to be caused in EOL character
> conversion only for Windows, and by change of map() function return
> value type between Python 2 and 3.
>
> The patch below address to it. This will fix the exceptions in
> fails2.log, however other problems can be found.
> [[[
> Index: subversion/tests/cmdline/svntest/actions.py
> ===
> --- subversion/tests/cmdline/svntest/actions.py (revision 1877286)
> +++ subversion/tests/cmdline/svntest/actions.py (working copy)
> @@ -418,7 +418,7 @@
># Since main.run_svnrdump() uses binary mode, normalize the stderr
># line endings on Windows ourselves.
>if sys.platform == 'win32':
> -err = map(lambda x : x.replace('\r\n', '\n'), err)
> +err = [x.replace('\r\n', '\n') for x in err]
>
># Ignore "consider upgrade" warnings to allow regression tests to pass
># when run against a 1.6 mod_dav_svn.
> ]]]
Perfect! This fixes all the svnrdump tests for Python 3 on Windows
(and to be sure I confirmed they also still work with Python 2).
Thanks,
--
Johan
Re: Problems running testsuite on Windows with Python 3
On Fri, May 1, 2020 at 7:46 PM Yasuhito FUTATSUKI wrote: > > On 2020/04/23 2:05, Yasuhito FUTATSUKI wrote: > > On 2020/04/22 23:04, Johan Corveleyn wrote: > >> On Wed, Apr 22, 2020 at 3:50 PM Yasuhito FUTATSUKI > >> wrote: > > > >>> ... and what is worse, at least > >>> > FAIL: merge_tests.py 34: conflict markers should match the file's eol > style > >>> > >>> this one seems to be broken even with Python 2.7, on Windows. > >>> (I'll post about it later). > >> > >> That's strange. It does succeed on my system when running with Python > >> 2.7.17. I had "All successful" test runs for [fsfs] x [ra_local, > >> ra_serf, ra_svn]. > > > > Yes, I don't doubt this test is passed on Windows with Python 2.7, > > but I doubt this test does not check just what we want to check. > > > > As far as I read the description of the test and about KEEP_EOL_STYLE > > option introduced in r1743445. > > > > merge_tests.py (merge_conflict_markers_matching_eol): > > [[[ > > # eol-style handling during merge with conflicts, scenario 1: > > # when a merge creates a conflict on a file, make sure the file and files > > # r, r and .mine are in the eol-style defined for that file. > > ]]] > > > > So I think comparion of expected and actual should be done without > > eol-style translation. However it try to check with translation on > > Windows environment. > > It seems following tests are EOL style sensitive, but they didn't > check without strict EOL style on Windows: > > merge_tests.merge_conflict_markers_matching_eol > merge_tests.merge_eolstyle_handling > patch_tests.patch_no_svn_eol_style > patch_tests.patch_with_svn_eol_style > patch_tests.patch_with_svn_eol_style_uncommitted > update_tests.conflict_markers_matching_eol > update_tests.update_eol_style_handling > > I've not check that each tests depend that native EOL is '\n' or not. > However, I think if some tests depend on it, other test cases for > those aims are needed on Windows, or at least explicitly skip the tests > on Windows to clarify that those tests check nothing for the aims. > > The updated patch attached only make these tests EOL style sensitive > (and relax check of EOL on Python 2 if keep_eol_style optional > parameter is not specified). Thank you for continuing to work on this, Yasuhito. This patch indeed fixes the above 7 tests when running with Python 3 on Windows. However, when running with Python 2 there are now 6 tests that fail (without this patch, all tests are successful with Py2): FAIL: merge_tests.py 34: conflict markers should match the file's eol style FAIL: patch_tests.py 13: patch target with no svn:eol-style FAIL: patch_tests.py 14: patch target with svn:eol-style FAIL: patch_tests.py 15: patch target with uncommitted svn:eol-style FAIL: patch_tests.py 57: patch a binary file FAIL: update_tests.py 26: conflict markers should match the file's eol style See fails.log in attachment. -- Johan fails.log Description: Binary data
[PATCH] fix svnrdump_tests.py on Windows with Python 3 (Re: Problems running testsuite on Windows with Python 3)
On 2020/04/19 3:26, Johan Corveleyn wrote:
> This is the list of tests that still fail (66):
> FAIL: svnrdump_tests.py 1: dump: standard sbox repos
...
> FAIL: svnrdump_tests.py 57: load: invalid svn:date revprop in r1
These failures in svnrdump_tests.py seems to be caused in EOL character
conversion only for Windows, and by change of map() function return
value type between Python 2 and 3.
The patch below address to it. This will fix the exceptions in
fails2.log, however other problems can be found.
[[[
Index: subversion/tests/cmdline/svntest/actions.py
===
--- subversion/tests/cmdline/svntest/actions.py (revision 1877286)
+++ subversion/tests/cmdline/svntest/actions.py (working copy)
@@ -418,7 +418,7 @@
# Since main.run_svnrdump() uses binary mode, normalize the stderr
# line endings on Windows ourselves.
if sys.platform == 'win32':
-err = map(lambda x : x.replace('\r\n', '\n'), err)
+err = [x.replace('\r\n', '\n') for x in err]
# Ignore "consider upgrade" warnings to allow regression tests to pass
# when run against a 1.6 mod_dav_svn.
]]]
Cheers,
--
Yasuhito FUTATSUKI /
Re: Problems running testsuite on Windows with Python 3
On 2020/04/23 2:05, Yasuhito FUTATSUKI wrote:
> On 2020/04/22 23:04, Johan Corveleyn wrote:
>> On Wed, Apr 22, 2020 at 3:50 PM Yasuhito FUTATSUKI
>> wrote:
>
>>> ... and what is worse, at least
>>>
FAIL: merge_tests.py 34: conflict markers should match the file's eol
style
>>>
>>> this one seems to be broken even with Python 2.7, on Windows.
>>> (I'll post about it later).
>>
>> That's strange. It does succeed on my system when running with Python
>> 2.7.17. I had "All successful" test runs for [fsfs] x [ra_local,
>> ra_serf, ra_svn].
>
> Yes, I don't doubt this test is passed on Windows with Python 2.7,
> but I doubt this test does not check just what we want to check.
>
> As far as I read the description of the test and about KEEP_EOL_STYLE
> option introduced in r1743445.
>
> merge_tests.py (merge_conflict_markers_matching_eol):
> [[[
> # eol-style handling during merge with conflicts, scenario 1:
> # when a merge creates a conflict on a file, make sure the file and files
> # r, r and .mine are in the eol-style defined for that file.
> ]]]
>
> So I think comparion of expected and actual should be done without
> eol-style translation. However it try to check with translation on
> Windows environment.
It seems following tests are EOL style sensitive, but they didn't
check without strict EOL style on Windows:
merge_tests.merge_conflict_markers_matching_eol
merge_tests.merge_eolstyle_handling
patch_tests.patch_no_svn_eol_style
patch_tests.patch_with_svn_eol_style
patch_tests.patch_with_svn_eol_style_uncommitted
update_tests.conflict_markers_matching_eol
update_tests.update_eol_style_handling
I've not check that each tests depend that native EOL is '\n' or not.
However, I think if some tests depend on it, other test cases for
those aims are needed on Windows, or at least explicitly skip the tests
on Windows to clarify that those tests check nothing for the aims.
The updated patch attached only make these tests EOL style sensitive
(and relax check of EOL on Python 2 if keep_eol_style optional
parameter is not specified).
Cheers,
--
Yasuhito FUTATSUKI /
Fix eol style treatment in command tests on Windows.
[in subversion/tests/cmdline]
* merge_tests.py (merge_conflict_markers_matching_eol,
merge_eolstyle_handling),
* patch_tests.py (patch_no_svn_eol_style,
patch_with_svn_eol_style,
patch_with_svn_eol_style_uncommitted),
* update_tests.py (conflict_markers_matching_eol,
update_eol_style_handling):
Specify keep_eol_style = True evne if platform is Windows.
* merge_tests.py (merge_conflict_markers_matching_eol),
* patch_tests.py (patch_with_svn_eol_style,
patch_with_svn_eol_style_uncommitted),
* conflict_markers_matching_eol):
Switch per platform eol value for 'native' svn:eol-style
* svntest/wc.py (State.from_wc):
Use io.open() explicitly to specify 'newline' parameter for universal newline,
even on Python 2. With this change, '\r' end of line chracters in files
are also translated to '\n' if keep_eol_style=False (or unspicified) on
Python 2.
Index: subversion/tests/cmdline/merge_tests.py
===
--- subversion/tests/cmdline/merge_tests.py (revision 1876908)
+++ subversion/tests/cmdline/merge_tests.py (working copy)
@@ -3323,16 +3323,12 @@
mu_path = sbox.ospath('A/mu')
- # CRLF is a string that will match a CRLF sequence read from a text file.
- # ### On Windows, we assume CRLF will be read as LF, so it's a poor test.
if os.name == 'nt':
-crlf = '\n'
+native_nl = '\r\n'
else:
-crlf = '\r\n'
+native_nl = '\n'
+ crlf = '\r\n'
- # Strict EOL style matching breaks Windows tests at least with Python 2
- keep_eol_style = not svntest.main.is_os_windows()
-
# Checkout a second working copy
wc_backup = sbox.add_wc_path('backup')
svntest.actions.run_and_verify_svn(None, [], 'checkout',
@@ -3349,8 +3345,8 @@
path_backup = os.path.join(wc_backup, 'A', 'mu')
# do the test for each eol-style
- for eol, eolchar in zip(['CRLF', 'CR', 'native', 'LF'],
- [crlf, '\015', '\n', '\012']):
+ for eol, eolchar in zip(['CRLF', 'CR','native', 'LF'],
+ [crlf, '\015', native_nl, '\012']):
# rewrite file mu and set the eol-style property.
svntest.main.file_write(mu_path, "This is the file 'mu'."+ eolchar, 'wb')
svntest.main.run_svn(None, 'propset', 'svn:eol-style', eol, mu_path)
@@ -3445,7 +3441,7 @@
expected_backup_disk,
expected_backup_status,
expected_backup_skip,
- keep_eol_style=keep_eol_style)
+ keep_eol_style=True)
# cleanup for next run
svntest.main.run_svn(None, 'revert', '-R', wc_backup)
Re: Problems running testsuite on Windows with Python 3
On 2020/04/23 5:53, Johan Corveleyn wrote: > On Tue, Apr 21, 2020 at 5:14 PM Yasuhito FUTATSUKI > wrote: > >> On 2020/04/21 4:03, Johan Corveleyn wrote: >>> On Mon, Apr 20, 2020 at 2:16 PM Yasuhito FUTATSUKI >> >> >> So I wrote a patch for testing (not for commit. it is need to brush up even if it can work). At least it should work (no regression) on Python 2.7, Python 3 with and without PYTHONLEGACYWINDOWSSTDIO on Windows. (I confirmed on Python 2.7 and Python 3.7 on FreeBSD). Thanks, -- Yasuhito FUTATSUKI >>> >>> Perfect! Works well on Windows, with Py 2.7, 3.7 and 3.8.2, with or >> without >>> the envvar defined. >>> Thanks! >> >> Thank you! >> >> Then I update the patch to prepare to commit. However I don't have >> confidence function name, variable names, and comment, etc. >> >> Could anyone please refine it? >> >> Thanks, >> -- >> Yasuhito FUTATSUKI / >> > > Hi Yasuhito, > > Whoops, I'm not sure anymore if your console fix is entirely correct (or > perhaps it's missing some extra changes in other parts of the testsuite). > > I was experimenting with your latest patches (for > merge_conflict_markers_matching_eol), which failed under Python 3.7.7 (I > only applied your first patch of the latest post, not the second, just to > see what would happen). Anyway, the point is that, with a failing test, > there's something going wrong with the output. > Where's that "I" output on the console coming from? Log statements should > have been redirected to logfiles. It seems my second patch for Windows console I/O still not enough. Perhaps those extra output to console came from internal buffer in some layer, which didn't flushed via flush() method just before redirection, or there exist other reference of old sys.stdout, sys.stderr I'm overlooking. Traceback in log file indicates those problem occurs around spawn_process() in run_tests.py > Any idea? I'll read code again, however I'm not sure I can solve this problem Cheers, -- Yasuhito FUTATSUKI /
Re: Problems running testsuite on Windows with Python 3
On Tue, Apr 21, 2020 at 5:14 PM Yasuhito FUTATSUKI wrote: > On 2020/04/21 4:03, Johan Corveleyn wrote: > > On Mon, Apr 20, 2020 at 2:16 PM Yasuhito FUTATSUKI > > > > >> So I wrote a patch for testing (not for commit. it is need to brush up > >> even if it can work). > >> > >> At least it should work (no regression) on Python 2.7, Python 3 with > >> and without PYTHONLEGACYWINDOWSSTDIO on Windows. > >> > >> (I confirmed on Python 2.7 and Python 3.7 on FreeBSD). > >> > >> Thanks, > >> -- > >> Yasuhito FUTATSUKI > >> > > > > Perfect! Works well on Windows, with Py 2.7, 3.7 and 3.8.2, with or > without > > the envvar defined. > > Thanks! > > Thank you! > > Then I update the patch to prepare to commit. However I don't have > confidence function name, variable names, and comment, etc. > > Could anyone please refine it? > > Thanks, > -- > Yasuhito FUTATSUKI / > Hi Yasuhito, Whoops, I'm not sure anymore if your console fix is entirely correct (or perhaps it's missing some extra changes in other parts of the testsuite). I was experimenting with your latest patches (for merge_conflict_markers_matching_eol), which failed under Python 3.7.7 (I only applied your first patch of the latest post, not the second, just to see what would happen). Anyway, the point is that, with a failing test, there's something going wrong with the output. On the console I got: [[[ C:\research\svn\dev\subversion-1.14.0-rc2>runtest.bat -t merge Testing Release configuration on local repository. [1/1] merge_tests.pyFAILURE At least one test was SKIPPED, checking R:\test_release-tmerge\tests.log SKIP: merge_tests.py 81: merge with broken symlinks in target SKIP: merge_tests.py 116: executable flag is maintained during binary merge At least one test XFAILED, checking R:\test_release-tmerge\tests.log XFAIL: merge_tests.py 49: avoid repeated merges for cyclic merging XFAIL: merge_tests.py 64: merge target with non inheritable mergeinfo XFAIL: merge_tests.py 114: don't inherit bogus mergeinfo XFAIL: merge_tests.py 115: don't inherit bogus working mergeinfo At least one test FAILED, checking R:\test_release-tmerge\tests.log FAIL: merge_tests.py 34: conflict markers should match the file's eol style FAIL: merge_tests.py 35: handle eol-style propchange during merge Summary of test results: 133 tests PASSED 2 tests SKIPPED 4 tests XFAILED 2 tests FAILED Python version: 3.7.7. SUMMARY: Some tests failed I: CMD: svnadmin.exe create svn-test-work\local_tmp\repos --compatible-version=1.14 --fs-type=fsfs I: I: CMD: svn.exe import -m "Log message for revision 1." svn-test-work\local_tmp\greekfiles file:///R:/test_release-tmerge/subversion/tests/cmdline/svn-test-work/local_tmp/repos --config-dir R:\test_release-tmerge\subversion\tests\cmdline\svn-test-work\local_tmp\config --password rayjandom --no-auth-cache --username jrandom I: I: Adding svn-test-work\local_tmp\greekfiles\A I: Adding svn-test-work\local_tmp\greekfiles\A\B I: Adding svn-test-work\local_tmp\greekfiles\A\B\E I: Adding svn-test-work\local_tmp\greekfiles\A\B\E\alpha I: Adding svn-test-work\local_tmp\greekfiles\A\B\E\beta I: Adding svn-test-work\local_tmp\greekfiles\A\B\F I: Adding svn-test-work\local_tmp\greekfiles\A\B\lambda I: Adding svn-test-work\local_tmp\greekfiles\A\C I: Adding svn-test-work\local_tmp\greekfiles\A\D I: Adding svn-test-work\local_tmp\greekfiles\A\D\G I: Adding svn-test-work\local_tmp\greekfiles\A\D\G\pi I: Adding svn-test-work\local_tmp\greekfiles\A\D\G\rho I: Adding svn-test-work\local_tmp\greekfiles\A\D\G\tau I: Adding svn-test-work\local_tmp\greekfiles\A\D\H I: Adding svn-test-work\local_tmp\greekfiles\A\D\H\chi I: Adding svn-test-work\local_tmp\greekfiles\A\D\H\omega I: Adding svn-test-work\local_tmp\greekfiles\A\D\H\psi I: Adding svn-test-work\local_tmp\greekfiles\A\D\gamma I: Adding svn-test-work\local_tmp\greekfiles\A\mu I: Adding svn-test-work\local_tmp\greekfiles\iota I: Committing transaction... I: Committed revision 1. I: CMD: svnadmin.exe create svn-test-work\local_tmp\trojan --compatible-version=1.14 --fs-type=fsfs I: I: CMD: svn.exe import -m "Log message for revision 1." svn-test-work\local_tmp\trojanfiles file:///R:/test_release-tmerge/subversion/tests/cmdline/svn-test-work/local_tmp/trojan --config-dir R:\test_release-tmerge\subversion\tests\cmdline\svn-test-work\local_tmp\config --password rayjandom --no-auth-cache --username jrandom I: I: Adding svn-test-work\local_tmp\trojanfiles\.@ I: Adding svn-test-work\local_tmp\trojanfiles\.@kappa I: Adding svn-test-work\local_tmp\trojanfiles\@ I: Adding svn-test-work\local_tmp\trojanfiles\@@ I: Adding svn-test-work\local_tmp\trojanfiles\@omicron@ I: Adding svn-test-work\local_tmp\trojanfiles\@zeta I: Adding
Re: Problems running testsuite on Windows with Python 3
On 2020/04/22 23:04, Johan Corveleyn wrote:
> On Wed, Apr 22, 2020 at 3:50 PM Yasuhito FUTATSUKI
> wrote:
>> ... and what is worse, at least
>>
>>> FAIL: merge_tests.py 34: conflict markers should match the file's eol style
>>
>> this one seems to be broken even with Python 2.7, on Windows.
>> (I'll post about it later).
>
> That's strange. It does succeed on my system when running with Python
> 2.7.17. I had "All successful" test runs for [fsfs] x [ra_local,
> ra_serf, ra_svn].
Yes, I don't doubt this test is passed on Windows with Python 2.7,
but I doubt this test does not check just what we want to check.
As far as I read the description of the test and about KEEP_EOL_STYLE
option introduced in r1743445.
merge_tests.py (merge_conflict_markers_matching_eol):
[[[
# eol-style handling during merge with conflicts, scenario 1:
# when a merge creates a conflict on a file, make sure the file and files
# r, r and .mine are in the eol-style defined for that file.
]]]
So I think comparion of expected and actual should be done without
eol-style translation. However it try to check with translation on
Windows environment.
[[[
# CRLF is a string that will match a CRLF sequence read from a text file.
# ### On Windows, we assume CRLF will be read as LF, so it's a poor test.
if os.name == 'nt':
crlf = '\n'
else:
crlf = '\r\n'
# Strict EOL style matching breaks Windows tests at least with Python 2
keep_eol_style = not svntest.main.is_os_windows()
...
# do the test for each eol-style
for eol, eolchar in zip(['CRLF', 'CR', 'native', 'LF'],
[crlf, '\015', '\n', '\012']):
...
svntest.actions.run_and_verify_merge2(wc_backup, cur_rev - 1, cur_rev,
sbox.repo_url, None,
expected_backup_output,
expected_mergeinfo_output,
expected_elision_output,
expected_backup_disk,
expected_backup_status,
expected_backup_skip,
keep_eol_style=keep_eol_style)
]]]
So, svntest.actions.run_and_verify_merge2 is called with
keep_eol_style=False.
The docstring of svntest.actons.run_andverify_merge2 said:
[[[
"""
...
If KEEP_EOL_STYLE is set, don't let Python normalize the EOL when
reading working copy contents as text files. It has no effect on
binary files.
...
"""
]]]
It seems that this works only for Python 3. Even KEEP_EOL_STYLE=False,
Python 2.7 on Windows still read '\r' as it is, as I didn't expect
when I read it first time.
To make it work as I expected (and just work as Python 3 do), apply
this patch:
[[[
Index: subversion/tests/cmdline/svntest/wc.py
===
--- subversion/tests/cmdline/svntest/wc.py (revision 1876827)
+++ subversion/tests/cmdline/svntest/wc.py (working copy)
@@ -28,6 +28,7 @@
import re
import logging
import pprint
+import io
if sys.version_info[0] >= 3:
# Python >=3.0
@@ -686,9 +687,9 @@
if os.path.isfile(node):
try:
if keep_eol_style:
- contents = open(node, 'r', newline='').read()
+ contents = io.open(node, 'r', newline='').read()
else:
- contents = open(node, 'r').read()
+ contents = io.open(node, 'r').read()
except:
contents = open(node, 'rb').read()
else:
]]]
and this will make merge_tests.merge_conflict_markers_matching_eol()
and perhaps some other tests fail on Windows with Python 2.7.
a fix for merge_tests.merge_conflict_markers_matching_eol and
merge_tests.merge_eolstyle_handling will be:
[[[
Index: subversion/tests/cmdline/merge_tests.py
===
--- subversion/tests/cmdline/merge_tests.py (revision 1876827)
+++ subversion/tests/cmdline/merge_tests.py (working copy)
@@ -3323,16 +3323,12 @@
mu_path = sbox.ospath('A/mu')
- # CRLF is a string that will match a CRLF sequence read from a text file.
- # ### On Windows, we assume CRLF will be read as LF, so it's a poor test.
if os.name == 'nt':
-crlf = '\n'
+native_nl = '\r\n'
else:
-crlf = '\r\n'
+native_nl = '\n'
+ crlf = '\r\n'
- # Strict EOL style matching breaks Windows tests at least with Python 2
- keep_eol_style = not svntest.main.is_os_windows()
-
# Checkout a second working copy
wc_backup = sbox.add_wc_path('backup')
svntest.actions.run_and_verify_svn(None, [], 'checkout',
@@ -3349,8 +3345,8 @@
path_backup = os.path.join(wc_backup, 'A', 'mu')
# do the test for each eol-style
- for eol, eolchar in zip(['CRLF', 'CR', 'native', 'LF'],
- [crlf, '\015', '\n', '\012']):
+ for eol,
Re: Problems running testsuite on Windows with Python 3
On Wed, Apr 22, 2020 at 3:50 PM Yasuhito FUTATSUKI wrote: > > On 2020/04/22 22:18, Nathan Hartman wrote: > > > I'd like to add something along these lines to the Release Notes, > > under "Known Issues in the Release." > > > > Please check for accuracy: > > > > (I haven't tested this myself -- haven't reached this point yet in my > > Windows setup!) > > > > [[[ > > > > When using Python 3.6 or higher on Windows, the test suite fails to > > run because of changes in Python's implementation of console I/O. > > > > Several workarounds are available: > > > > Either: > > > > * Use Python 2.7, or Python 3.5 or below, or > > > > * Run the test suite with the --log-to-stdout command line argument, > > or > > > > * Set the PYTHONLEGACYWINDOWSSTDIO environment variable before running > > the test suite, or > > > > * Apply r1876707 and r[insert number here] and run the test suite. > > r1876707 isn't fix for console I/O issue, it is fix for another > problem in the testsuite to run it on Windows with Python 3. > > Also, there are still 65 of failure in test on Windows with > Python 3. Indeed. Actually 68 test failures if none of your post-1.14.0-rc2 patches are applied. There is also a problem running gen-make.py with Python 3, when configuring for a Windows build in Debug configuration (I didn't notice that earlier on because I always build a Release configuration). I'll post about that separately. > ... and what is worse, at least > > > FAIL: merge_tests.py 34: conflict markers should match the file's eol style > > this one seems to be broken even with Python 2.7, on Windows. > (I'll post about it later). That's strange. It does succeed on my system when running with Python 2.7.17. I had "All successful" test runs for [fsfs] x [ra_local, ra_serf, ra_svn]. -- Johan
Re: Problems running testsuite on Windows with Python 3
On 2020/04/22 22:18, Nathan Hartman wrote: > I'd like to add something along these lines to the Release Notes, > under "Known Issues in the Release." > > Please check for accuracy: > > (I haven't tested this myself -- haven't reached this point yet in my > Windows setup!) > > [[[ > > When using Python 3.6 or higher on Windows, the test suite fails to > run because of changes in Python's implementation of console I/O. > > Several workarounds are available: > > Either: > > * Use Python 2.7, or Python 3.5 or below, or > > * Run the test suite with the --log-to-stdout command line argument, > or > > * Set the PYTHONLEGACYWINDOWSSTDIO environment variable before running > the test suite, or > > * Apply r1876707 and r[insert number here] and run the test suite. r1876707 isn't fix for console I/O issue, it is fix for another problem in the testsuite to run it on Windows with Python 3. Also, there are still 65 of failure in test on Windows with Python 3. ... and what is worse, at least > FAIL: merge_tests.py 34: conflict markers should match the file's eol style this one seems to be broken even with Python 2.7, on Windows. (I'll post about it later). > > This problem is (expected to be) fixed in 1.14.1. > > ]]] > > Where "insert number here" would be Yasuhito's latest patch. > > Nathan > Cheers, -- Yasuhito FUTATSUKI /
Re: Problems running testsuite on Windows with Python 3
Nathan Hartman wrote on Wed, 22 Apr 2020 13:18 +00:00: > Please check for accuracy: > > (I haven't tested this myself -- haven't reached this point yet in my > Windows setup!) > I don't have a Windows build either, but a few suggested edits: > [[[ > > When using Python 3.6 or higher on Windows, the test suite fails to > run because of changes in Python's implementation of console I/O. > Quote the error message here for greppability? > Several workarounds are available: > > Either: > > * Use Python 2.7, or Python 3.5 or below, or > Suggest to s/3.5/3.5.x/ to make it clear that 3.5.x with x>0 are fine. > * Run the test suite with the --log-to-stdout command line argument, > or > > * Set the PYTHONLEGACYWINDOWSSTDIO environment variable before running > the test suite, or s/variable/variable to a non-empty string/ (reference: https://docs.python.org/3/using/cmdline.html#envvar-PYTHONLEGACYWINDOWSSTDIO) > * Apply r1876707 and r[insert number here] and run the test suite. > > This problem is (expected to be) fixed in 1.14.1. > > ]]]
Re: Problems running testsuite on Windows with Python 3
Earlier, in the thread "schedule for 1.14.0 release"... On Wed, Apr 22, 2020 at 8:15 AM Johan Corveleyn wrote: > In fact, there is very little pressure on getting this in 1.14.0 GA, > because (1) it only affects testsuite-runners on Windows, of which > there are extremely few :-), and (2) the workaround is simple (just > use Py 2.7 for running the testsuite on Windows). Not worth taking > risks for IMHO. > > The only actual "damage" is perhaps marketing. Now we can't claim > "Full Py3 support" in big letters, we need to add a footnote pointing > to a known issues section in the release notes or something ... I'd like to add something along these lines to the Release Notes, under "Known Issues in the Release." Please check for accuracy: (I haven't tested this myself -- haven't reached this point yet in my Windows setup!) [[[ When using Python 3.6 or higher on Windows, the test suite fails to run because of changes in Python's implementation of console I/O. Several workarounds are available: Either: * Use Python 2.7, or Python 3.5 or below, or * Run the test suite with the --log-to-stdout command line argument, or * Set the PYTHONLEGACYWINDOWSSTDIO environment variable before running the test suite, or * Apply r1876707 and r[insert number here] and run the test suite. This problem is (expected to be) fixed in 1.14.1. ]]] Where "insert number here" would be Yasuhito's latest patch. Nathan
Re: Problems running testsuite on Windows with Python 3
On Tue, Apr 21, 2020 at 11:14 AM Yasuhito FUTATSUKI wrote:
> > Perfect! Works well on Windows, with Py 2.7, 3.7 and 3.8.2, with or without
> > the envvar defined.
> > Thanks!
>
> Thank you!
>
> Then I update the patch to prepare to commit. However I don't have
> confidence function name, variable names, and comment, etc.
>
> Could anyone please refine it?
Hello Yasuhito,
Thanks for finding a solution for the problem!
The function name, variable names, etc., look good to me.
I am only proposing a change for the log message, to provide more
context and also link to this email thread for future reference.
Hope this helps...
Cheers,
Nathan
Fix an issue when running tests on Windows with Python 3.6 and later.
Since Python 3.6 on Windows, io.IOBase object overlaid tty is incompatible
with other file like objects. This makes it impossible to redirect input or
output between file and tty without changing file descriptor number by using
os.dup2() without changing the overlaid object [1].
This prevents our test suite from running correctly because it attempts to
redirect stdout to a file using os.dup2() [2].
This is a workaround for it.
[1] _io._WindowsConsoleIO breaks in the face of fd redirection
https://bugs.python.org/issue30555
[2]
https://lists.apache.org/thread.html/r87a33f0046949fe59188e844832297d8a00e675c4e59a1ab01005e0e%40%3Cdev.subversion.apache.org%3E
* build/run_tests.py
(rebuild_textio): New.
(TestHarness._run_py_test): Rebuild IO object after os.dup2() if it is
needed.
Found by: jcorvel
Index: build/run_tests.py
===
--- build/run_tests.py (revision 1876712)
+++ build/run_tests.py (working copy)
@@ -48,6 +48,7 @@
'''
import os, sys, shutil, codecs
+import io
import re
import logging
import optparse, subprocess, threading, traceback
@@ -141,6 +142,10 @@
else:
return s.decode("latin-1")
+def rebuild_textio(fd, mode):
+ '''Rebuild an io.BaseIO object from file descriptor for sys.std*'''
+ return open(fd, mode, encoding='utf-8', errors='replace', closefd=False)
+
class TestHarness:
'''Test harness for Subversion tests.
'''
@@ -844,6 +849,16 @@
# setup the output pipes
if self.log:
+ # Since Python 3.6 on Windows, redirecting tty to file or file to tty
+ # by using os.dup2 breaks overlaid IO object.
+ # (See https://bugs.python.org/issue30555).
+ # To avoid this, we rebuild IO object after os.dup2 if it is needed.
+ need_rebuild_stdout = (isinstance(sys.stdout, io.IOBase)
+ and not isinstance(sys.stdout.buffer.raw,
+io.FileIO))
+ need_rebuild_stderr = (isinstance(sys.stderr, io.IOBase)
+ and not isinstance(sys.stderr.buffer.raw,
+io.FileIO))
sys.stdout.flush()
sys.stderr.flush()
self.log.flush()
@@ -851,6 +866,10 @@
old_stderr = os.dup(sys.stderr.fileno())
os.dup2(self.log.fileno(), sys.stdout.fileno())
os.dup2(self.log.fileno(), sys.stderr.fileno())
+ if need_rebuild_stdout:
+sys.stdout = rebuild_textio(sys.stdout.fileno(), 'w')
+ if need_rebuild_stderr:
+sys.stderr = rebuild_textio(sys.stderr.fileno(), 'w')
# These have to be class-scoped for use in the progress_func()
self.dots_written = 0
@@ -897,6 +916,10 @@
os.dup2(old_stderr, sys.stderr.fileno())
os.close(old_stdout)
os.close(old_stderr)
+ if need_rebuild_stdout:
+sys.stdout = rebuild_textio(sys.stdout.fileno(), 'w')
+ if need_rebuild_stderr:
+sys.stderr = rebuild_textio(sys.stderr.fileno(), 'w')
return failed
Re: Problems running testsuite on Windows with Python 3
On 2020/04/21 4:03, Johan Corveleyn wrote:
> On Mon, Apr 20, 2020 at 2:16 PM Yasuhito FUTATSUKI
>> So I wrote a patch for testing (not for commit. it is need to brush up
>> even if it can work).
>>
>> At least it should work (no regression) on Python 2.7, Python 3 with
>> and without PYTHONLEGACYWINDOWSSTDIO on Windows.
>>
>> (I confirmed on Python 2.7 and Python 3.7 on FreeBSD).
>>
>> Thanks,
>> --
>> Yasuhito FUTATSUKI
>>
>
> Perfect! Works well on Windows, with Py 2.7, 3.7 and 3.8.2, with or without
> the envvar defined.
> Thanks!
Thank you!
Then I update the patch to prepare to commit. However I don't have
confidence function name, variable names, and comment, etc.
Could anyone please refine it?
Thanks,
--
Yasuhito FUTATSUKI /
Fix an issue on run tests on Windows with Python 3.6 and later.
Since Python 3.6 on Windows, io.IOBase object overlaid tty is incompatible
with which of other file like objects. It makes impossible to redirect
input or output between file and tty without changing file descriptor number
by using os.dup2() without changing overlaid object[1]. This is a work
around for it.
[1] _io._WindowsConsoleIO breaks in the face of fd redirection
https://bugs.python.org/issue30555
* build/run_tests.py
(rebuild_textio): New.
(TestHarness._run_py_test): Rebuild IO object after os.dup2() if it is needed.
Reported by: jcorvel
Index: build/run_tests.py
===
--- build/run_tests.py (revision 1876712)
+++ build/run_tests.py (working copy)
@@ -48,6 +48,7 @@
'''
import os, sys, shutil, codecs
+import io
import re
import logging
import optparse, subprocess, threading, traceback
@@ -141,6 +142,10 @@
else:
return s.decode("latin-1")
+def rebuild_textio(fd, mode):
+ '''Rebuild an io.BaseIO object from file descriptor for sys.std*'''
+ return open(fd, mode, encoding='utf-8', errors='replace', closefd=False)
+
class TestHarness:
'''Test harness for Subversion tests.
'''
@@ -844,6 +849,16 @@
# setup the output pipes
if self.log:
+ # Since Python 3.6 on Windows, redirecting tty to file or file to tty
+ # by using os.dup2 breaks overlaid IO object.
+ # (See https://bugs.python.org/issue30555).
+ # To avoid this, we rebuild IO object after os.dup2 if it is needed.
+ need_rebuild_stdout = (isinstance(sys.stdout, io.IOBase)
+ and not isinstance(sys.stdout.buffer.raw,
+io.FileIO))
+ need_rebuild_stderr = (isinstance(sys.stderr, io.IOBase)
+ and not isinstance(sys.stderr.buffer.raw,
+io.FileIO))
sys.stdout.flush()
sys.stderr.flush()
self.log.flush()
@@ -851,6 +866,10 @@
old_stderr = os.dup(sys.stderr.fileno())
os.dup2(self.log.fileno(), sys.stdout.fileno())
os.dup2(self.log.fileno(), sys.stderr.fileno())
+ if need_rebuild_stdout:
+sys.stdout = rebuild_textio(sys.stdout.fileno(), 'w')
+ if need_rebuild_stderr:
+sys.stderr = rebuild_textio(sys.stderr.fileno(), 'w')
# These have to be class-scoped for use in the progress_func()
self.dots_written = 0
@@ -897,6 +916,10 @@
os.dup2(old_stderr, sys.stderr.fileno())
os.close(old_stdout)
os.close(old_stderr)
+ if need_rebuild_stdout:
+sys.stdout = rebuild_textio(sys.stdout.fileno(), 'w')
+ if need_rebuild_stderr:
+sys.stderr = rebuild_textio(sys.stderr.fileno(), 'w')
return failed
Re: Problems running testsuite on Windows with Python 3
On Mon, Apr 20, 2020 at 2:16 PM Yasuhito FUTATSUKI wrote: > On 2020/04/20 9:13, Branko Čibej wrote: > > On 19.04.2020 23:58, Yasuhito FUTATSUKI wrote: > > > > >> On 2020/04/19 22:53, Branko Čibej wrote: > >>> On 19.04.2020 14:47, Johan Corveleyn wrote: > Would you also have an idea on how to fix the > PYTHONLEGACYWINDOWSSTDIO issue? > I mean: would it ever be possible to run the testsuite without > defining that envvar? Or will that remain necessary? > > > > >> If reconstruction of _io._WindowsConsoleIO from stored old descripor is > >> allowed (i.e. documented in the specification), it seems to be possible > >> to write work around. > > > > > > Given that these are private symbols, at least according to their names, > > they could change in any minor Python version. :( > > Yes, indeed. > > However, at least on Python 3.6.6, if given file descptor 1 is pointing > to tty, the result of > > open(1, 'w', encoding='utf-8', closefd=False).buffer.raw > > is _io._WindowsConsoleIO object, and it seems to be able to used for > console output. > > So I wrote a patch for testing (not for commit. it is need to brush up > even if it can work). > > At least it should work (no regression) on Python 2.7, Python 3 with > and without PYTHONLEGACYWINDOWSSTDIO on Windows. > > (I confirmed on Python 2.7 and Python 3.7 on FreeBSD). > > Thanks, > -- > Yasuhito FUTATSUKI > Perfect! Works well on Windows, with Py 2.7, 3.7 and 3.8.2, with or without the envvar defined. Thanks! -- Johan
Re: Problems running testsuite on Windows with Python 3
On 20.04.2020 14:13, Yasuhito FUTATSUKI wrote: > On 2020/04/20 9:13, Branko Čibej wrote: >> On 19.04.2020 23:58, Yasuhito FUTATSUKI wrote: > > >>> On 2020/04/19 22:53, Branko Čibej wrote: On 19.04.2020 14:47, Johan Corveleyn wrote: > Would you also have an idea on how to fix the PYTHONLEGACYWINDOWSSTDIO > issue? > I mean: would it ever be possible to run the testsuite without > defining that envvar? Or will that remain necessary? > > >>> If reconstruction of _io._WindowsConsoleIO from stored old descripor is >>> allowed (i.e. documented in the specification), it seems to be possible >>> to write work around. >> >> Given that these are private symbols, at least according to their names, >> they could change in any minor Python version. :( > Yes, indeed. > > However, at least on Python 3.6.6, if given file descptor 1 is pointing > to tty, the result of > > open(1, 'w', encoding='utf-8', closefd=False).buffer.raw > > is _io._WindowsConsoleIO object, and it seems to be able to used for > console output. Ah, very nice, I didn't know you could open() an existing file descriptor in Python3. -- Brane
Re: Problems running testsuite on Windows with Python 3
On 2020/04/20 9:13, Branko Čibej wrote: > On 19.04.2020 23:58, Yasuhito FUTATSUKI wrote: >> On 2020/04/19 22:53, Branko Čibej wrote: >>> On 19.04.2020 14:47, Johan Corveleyn wrote: Would you also have an idea on how to fix the PYTHONLEGACYWINDOWSSTDIO issue? I mean: would it ever be possible to run the testsuite without defining that envvar? Or will that remain necessary? >> If reconstruction of _io._WindowsConsoleIO from stored old descripor is >> allowed (i.e. documented in the specification), it seems to be possible >> to write work around. > > > Given that these are private symbols, at least according to their names, > they could change in any minor Python version. :( Yes, indeed. However, at least on Python 3.6.6, if given file descptor 1 is pointing to tty, the result of open(1, 'w', encoding='utf-8', closefd=False).buffer.raw is _io._WindowsConsoleIO object, and it seems to be able to used for console output. So I wrote a patch for testing (not for commit. it is need to brush up even if it can work). At least it should work (no regression) on Python 2.7, Python 3 with and without PYTHONLEGACYWINDOWSSTDIO on Windows. (I confirmed on Python 2.7 and Python 3.7 on FreeBSD). Thanks, -- Yasuhito FUTATSUKI Index: build/run_tests.py === --- build/run_tests.py (revision 1876712) +++ build/run_tests.py (working copy) @@ -48,6 +48,7 @@ ''' import os, sys, shutil, codecs +import io import re import logging import optparse, subprocess, threading, traceback @@ -844,6 +845,12 @@ # setup the output pipes if self.log: + need_stdout_reconstruct = (isinstance(sys.stdout, io.IOBase) + and not isinstance(sys.stdout.buffer.raw, +io.FileIO)) + need_stderr_reconstruct = (isinstance(sys.stderr, io.IOBase) + and not isinstance(sys.stderr.buffer.raw, +io.FileIO)) sys.stdout.flush() sys.stderr.flush() self.log.flush() @@ -851,6 +858,12 @@ old_stderr = os.dup(sys.stderr.fileno()) os.dup2(self.log.fileno(), sys.stdout.fileno()) os.dup2(self.log.fileno(), sys.stderr.fileno()) + if need_stdout_reconstruct: +sys.stdout = open(sys.stdout.fileno(), 'w', encoding='utf-8', + errors='replace', closefd=False) + if need_stderr_reconstruct: +sys.stderr = open(sys.stderr.fileno(), 'w', encoding='utf-8', + errors='replace', closefd=False) # These have to be class-scoped for use in the progress_func() self.dots_written = 0 @@ -897,6 +910,12 @@ os.dup2(old_stderr, sys.stderr.fileno()) os.close(old_stdout) os.close(old_stderr) + if need_stdout_reconstruct: +sys.stdout = open(sys.stdout.fileno(), 'w', encoding='utf-8', + errors='replace', closefd=False) + if need_stderr_reconstruct: +sys.stderr = open(sys.stderr.fileno(), 'w', encoding='utf-8', + errors='replace', closefd=False) return failed
Re: Problems running testsuite on Windows with Python 3
On 19.04.2020 23:58, Yasuhito FUTATSUKI wrote: > I've commited the patch for blame_tests.py in 1876734. Thank you. > > On 2020/04/19 22:53, Branko Čibej wrote: >> On 19.04.2020 14:47, Johan Corveleyn wrote: >>> Would you also have an idea on how to fix the PYTHONLEGACYWINDOWSSTDIO >>> issue? >>> I mean: would it ever be possible to run the testsuite without >>> defining that envvar? Or will that remain necessary? >> >> Depending on how the testsuite is implemented, which I forget in detail, >> it might be. The os.dup2() method allows child processes to inherit the >> redirected standard handles. Reassigning os.stdout will not do that. >> There might be a way to maintain the inheritance behaviour by creating >> child processes differently. > Actually I tried the patch below (in FreeBSD), but it is not enogh. > Duplication of file descriptor is still needed. Output during test is > redirected to the log file, however the results of the tests written > in Python are not redirected. Yes, that is exactly what I suspected would happen. > [[[ > Index: build/run_tests.py ... > ]]] > > If reconstruction of _io._WindowsConsoleIO from stored old descripor is > allowed (i.e. documented in the specification), it seems to be possible > to write work around. Given that these are private symbols, at least according to their names, they could change in any minor Python version. :( -- Brane
Re: Problems running testsuite on Windows with Python 3
I've commited the patch for blame_tests.py in 1876734. Thank you.
On 2020/04/19 22:53, Branko Čibej wrote:
> On 19.04.2020 14:47, Johan Corveleyn wrote:
>> Would you also have an idea on how to fix the PYTHONLEGACYWINDOWSSTDIO issue?
>> I mean: would it ever be possible to run the testsuite without
>> defining that envvar? Or will that remain necessary?
>
>
> Depending on how the testsuite is implemented, which I forget in detail,
> it might be. The os.dup2() method allows child processes to inherit the
> redirected standard handles. Reassigning os.stdout will not do that.
> There might be a way to maintain the inheritance behaviour by creating
> child processes differently.
Actually I tried the patch below (in FreeBSD), but it is not enogh.
Duplication of file descriptor is still needed. Output during test is
redirected to the log file, however the results of the tests written
in Python are not redirected.
[[[
Index: build/run_tests.py
===
--- build/run_tests.py (revision 1876712)
+++ build/run_tests.py (working copy)
@@ -847,10 +847,8 @@
sys.stdout.flush()
sys.stderr.flush()
self.log.flush()
- old_stdout = os.dup(sys.stdout.fileno())
- old_stderr = os.dup(sys.stderr.fileno())
- os.dup2(self.log.fileno(), sys.stdout.fileno())
- os.dup2(self.log.fileno(), sys.stderr.fileno())
+ old_stdout = sys.stdout
+ old_stderr = sys.stderr
# These have to be class-scoped for use in the progress_func()
self.dots_written = 0
@@ -867,7 +865,8 @@
if self.dots_written < dot_count:
dots_to_write = dots - self.dots_written
self.dots_written = dots
-os.write(old_stdout, b'.' * dots_to_write)
+old_stdout.write('.' * dots_to_write)
+old_stdout.flush()
self.progress_lock.release()
serial_only = hasattr(prog_mod, 'serial_only') and prog_mod.serial_only
@@ -886,7 +885,8 @@
test_selection=test_nums)
except svntest.Failure:
if self.log:
-os.write(old_stdout, b'.' * dot_count)
+old_stdout.write('.' * dot_count)
+old_stdout.flush()
failed = True
# restore some values
@@ -893,10 +893,8 @@
if self.log:
sys.stdout.flush()
sys.stderr.flush()
- os.dup2(old_stdout, sys.stdout.fileno())
- os.dup2(old_stderr, sys.stderr.fileno())
- os.close(old_stdout)
- os.close(old_stderr)
+ sys.stdout = old_stdout
+ sys.stderr = old_stderr
return failed
]]]
If reconstruction of _io._WindowsConsoleIO from stored old descripor is
allowed (i.e. documented in the specification), it seems to be possible
to write work around.
Cheers,
--
Yasuhito FUTATSUKI /
Re: Problems running testsuite on Windows with Python 3
On 19.04.2020 14:47, Johan Corveleyn wrote: > On Sun, Apr 19, 2020 at 10:49 AM Yasuhito FUTATSUKI > wrote: >> On 2020/04/19 3:26, Johan Corveleyn wrote: >>> On Fri, Apr 17, 2020 at 11:21 PM Yasuhito FUTATSUKI >>> wrote: On 2020/04/17 5:53, Johan Corveleyn wrote: > Running the testsuite with Py 3.8.2 with PYTHONLEGACYWINDOWSSTDIO > defined in the environment allows it to complete. It results in 68 > FAILS (for ra_local x fsfs). See attached fails.log. I hope a part of the problems may fixed by attached patch, however, I can't see why these test was passed in Python 2.7. >>> Thanks! That fixes two of the tests. These two now pass after applying >>> your patch (so feel free to commit -- I also checked again with Python >>> 2.7 and still all tests successful): >> Thank you for testing. I commited it in r1876707. >> >> One more fix. >>> FAIL: blame_tests.py 18: blame_youngest_to_oldest >> The patch attached address to this one. > Confirmed! Thanks :-). > > Would you also have an idea on how to fix the PYTHONLEGACYWINDOWSSTDIO issue? > I mean: would it ever be possible to run the testsuite without > defining that envvar? Or will that remain necessary? Depending on how the testsuite is implemented, which I forget in detail, it might be. The os.dup2() method allows child processes to inherit the redirected standard handles. Reassigning os.stdout will not do that. There might be a way to maintain the inheritance behaviour by creating child processes differently. So yeah ... this is a bit of serious nonsense from the Python devs. Breaking decades' worth of legacy code is never a good idea. -- Brane
Re: Problems running testsuite on Windows with Python 3
On Sun, Apr 19, 2020 at 10:49 AM Yasuhito FUTATSUKI wrote: > > On 2020/04/19 3:26, Johan Corveleyn wrote: > > On Fri, Apr 17, 2020 at 11:21 PM Yasuhito FUTATSUKI > > wrote: > >> > >> On 2020/04/17 5:53, Johan Corveleyn wrote: > >> > >>> Running the testsuite with Py 3.8.2 with PYTHONLEGACYWINDOWSSTDIO > >>> defined in the environment allows it to complete. It results in 68 > >>> FAILS (for ra_local x fsfs). See attached fails.log. > >> > >> I hope a part of the problems may fixed by attached patch, however, > >> I can't see why these test was passed in Python 2.7. > > > > Thanks! That fixes two of the tests. These two now pass after applying > > your patch (so feel free to commit -- I also checked again with Python > > 2.7 and still all tests successful): > > Thank you for testing. I commited it in r1876707. > > One more fix. > > FAIL: blame_tests.py 18: blame_youngest_to_oldest > > The patch attached address to this one. Confirmed! Thanks :-). Would you also have an idea on how to fix the PYTHONLEGACYWINDOWSSTDIO issue? I mean: would it ever be possible to run the testsuite without defining that envvar? Or will that remain necessary? (see https://stackoverflow.coma/questions/52373180/python-on-windows-handle-invalid-when-redirecting-stdout-writing-to-file) -- Johan
Re: Problems running testsuite on Windows with Python 3
On 2020/04/19 3:26, Johan Corveleyn wrote: > On Fri, Apr 17, 2020 at 11:21 PM Yasuhito FUTATSUKI > wrote: >> >> On 2020/04/17 5:53, Johan Corveleyn wrote: >> >>> Running the testsuite with Py 3.8.2 with PYTHONLEGACYWINDOWSSTDIO >>> defined in the environment allows it to complete. It results in 68 >>> FAILS (for ra_local x fsfs). See attached fails.log. >> >> I hope a part of the problems may fixed by attached patch, however, >> I can't see why these test was passed in Python 2.7. > > Thanks! That fixes two of the tests. These two now pass after applying > your patch (so feel free to commit -- I also checked again with Python > 2.7 and still all tests successful): Thank you for testing. I commited it in r1876707. One more fix. > FAIL: blame_tests.py 18: blame_youngest_to_oldest The patch attached address to this one. Thanks, -- Yasuhito FUTATSUKI / Use svntest.main.file_write to avoid end of line conversion. * subversion/tests/cmdline/blame_tests.py (blame_youngest_to_oldest) Reported by: jcorvel Index: subversion/tests/cmdline/blame_tests.py === --- subversion/tests/cmdline/blame_tests.py (revision 1876706) +++ subversion/tests/cmdline/blame_tests.py (working copy) @@ -958,8 +958,7 @@ sbox.simple_commit() #r3 # Delete a line. - with open(iota_moved, 'w') as f: -f.write(line) + svntest.main.file_write(iota_moved, line) sbox.simple_commit() #r4 expected_output = [
Re: Problems running testsuite on Windows with Python 3
On 2020/04/17 5:53, Johan Corveleyn wrote: > Running the testsuite with Py 3.8.2 with PYTHONLEGACYWINDOWSSTDIO > defined in the environment allows it to complete. It results in 68 > FAILS (for ra_local x fsfs). See attached fails.log. I hope a part of the problems may fixed by attached patch, however, I can't see why these test was passed in Python 2.7. Cheers, -- Yasuhito FUTATSUKI / Escape special characters in path when construct expected output regex. * subversion/tests/cmdline/svntest/actions.py (_run_and_verify_resolve) Reported by: jcorvel Index: subversion/tests/cmdline/svntest/actions.py === --- subversion/tests/cmdline/svntest/actions.py (revision 1876517) +++ subversion/tests/cmdline/svntest/actions.py (working copy) @@ -1923,7 +1923,7 @@ "Merge conflicts in '" + path + "' marked as resolved.\n" for path in expected_paths]), verify.UnorderedRegexListOutput([ -"Conflict in property.*at '" + path + "' marked as resolved.\n" \ +"Conflict in property.*at '" + re.escape(path) + "' marked as resolved.\n" \ for path in expected_paths]), verify.UnorderedOutput([ "Tree conflict at '" + path + "' marked as resolved.\n" for path in
