Re: Building SVN (dependencies) on Windows

[Adam Humpherys]

For what it's worth, I was able to get my build environment working based
on the script here: https://github.com/Jan-E/svn-windows which builds
(most) of the dependencies, but uses a pre-built version of Apache 2.4 from
apachelounge.com.

--
Adam Humpherys




On Wed, May 6, 2020 at 4:32 AM  wrote:

> Quite a few of the Windows dependencies can easily be built by the VCPKG
> package manager.
>
>
>
> For SharpSvn I maintained quite a few custom NAnt scripts that do about
> the same work (
> https://ctf.open.collab.net/svn/repos/sharpsvn/trunk/imports
>
> (or https://github.com/AmpScm/SharpSvn/tree/master/imports, but I think
> this mirror lags a bit )
>
>
>
> Somewhere in the CollabNet open source repositories there should be a
> slightly simpler -not as many development options- MSBuild version of the
> scripts to build the dependencies, but I’m not sure at which url.
>
>
>
>   Bert
>
> *Van:* Graham Bloice 
> *Verzonden:* donderdag 23 april 2020 18:17
> *Aan:* dev@subversion.apache.org
> *Onderwerp:* Re: Building SVN (dependencies) on Windows
>
>
>
> Apologies for butting in, but as a long time maintainer of the (mainly
> Windows) build process for Wireshark, another cross platform project, I'd
> like to offer my thoughts on such matters to the svn project.
>
>1. We moved to CMake quite some time ago, getting rid of automake and
>nmake and VS solution files.  Such a relief to have one build tool, usable
>on all platforms.  This is obviously a big step to take.
>2. Wireshark has lots of dependencies on 3rd party libraries and on
>Windows the usual issues of obtaining them and where to put them.  The
>project has "solved" this by maintaining an svn repo (even though the
>project has switched to git for the sources) of "-dev" zip packages
>containing the items required to build; headers, .lib and .dll files, e.g.
>https://anonsvn.wireshark.org/wireshark-win64-libs/trunk/packages/Part
>of the CMake generation step runs a PowerShell script that checks for
>updates, downloads the required zips and expands them to a "known"
>location, configurable by env vars.  The locations of these files are
>passed as hints to the CMake findxxx modules (modified by the project) so
>that CMake is happy.  Some build tools and libraries are exclude from this
>process and have to be manually installed, e.g. Perl, Python, Qt.
>3. Production of the "-dev" packages takes a few forms, cross building
>on OpenSUSE or using vcpkg, the details are listed for each library, e.g.
>https://www.wireshark.org/docs/wsdg_html_chunked/ChLibsGLib.html.
>When a library is updated, a single "tag" file in the main source tree is
>updated causing all builders to check for changes.
>4. All required artefacts are built\copied by CMake directives into
>the run-time directory so tests can be run without requiring installation.
>5. The installers are also built by CMake and pick up the required
>artefacts from the run-time directory.
>
> I see a number of advantages with the Wireshark solution:
>
>1. Much smaller barrier to entry for those attempting to build.  Still
>quite a setup needed, hence the developer's guide step-by-step section, see
>here:https://www.wireshark.org/docs/wsdg_html_chunked/ChSetupWin32.html
>2. No need to build (or even download unless there are changes) the
>3rd party libraries reducing build times
>3. Consistency across builds for all users, no local
>variations causing oddities.
>
> You can see the sausages being made over at the Wireshark buildbot:
> https://buildbot.wireshark.org/wireshark-master/waterfall, in particular
> the "ran CMake" generation step, although if there are no updates to the
> libraries it's a standard CMake experience.
>
>
>
> I'm not subscribed to the list, just observe infrequently via the
> svn.haxx.se archive, so if you have any questions please add me in
> replies.
>
>
>
> P.S. I did build svn on Windows once, many years ago and I found it an
> awkward process.
>
>
>
> --
>
> Graham Bloice
>

Re: SVN 1.14 release: please fix your buildbot

[Yasuhito FUTATSUKI]

On 2020/05/06 21:02, Yasuhito FUTATSUKI wrote:
> On 2020/05/06 20:38, b...@qqmail.nl wrote:
>> This buildbot runs python 2.X (and has for years) and now the tests are
>> broken. I don't remember seeing changes to the actual Conflict handing code
>> in Subversion itself, so...
>> I still think this is related to the changes to the test framework causing
>> regressions on 2.7.
> 
> In #2922, the test is passed with Python 2.7.8, however since #2923 it seems
> the buildbot uses Python 3.7.4, if I'm looking bad place.
^not

> https://ci.apache.org/builders/svn-windows-ra/builds/2922
> https://ci.apache.org/builders/svn-windows-ra/builds/2923

Please see end of some lines:
https://ci.apache.org/builders/svn-windows-ra/builds/2923/steps/Test%20fsfs%2Bsvn/logs/stdio


>>
>> This buildbot does the testing on Windows with python 2.7... But errors on
>> this bot are often ignored by *nix developers. (It reported errors for some
>> weeks because build.conf was broken for Windows... and once that was fixed
>> this new problem existed so 'nobody' noticed)
>>
>> -Oorspronkelijk bericht-
>> Van: Yasuhito FUTATSUKI  
>> Verzonden: woensdag 6 mei 2020 13:19
>> Aan: dev@subversion.apache.org; b...@qqmail.nl
>> Onderwerp: Re: SVN 1.14 release: please fix your buildbot
>>
>> On 2020/05/06 19:48, b...@qqmail.nl wrote:
>>> The Windows buildbots currently produce actual errors, most likely 
>>> caused by recent changes to the python test framework. (It is 
>>> expecting to see \r, but now sees \n in several text conflicts)
>>
>> The reason of failures aren't recent changes of the python test framework,
>> but no one cared of the tests on Windows with Python 3.  Please see the
>> thread "Problems running testsuite on Windows with Python 3" [1].
>>
>> We are now working to try to fix it.
>>
>> [1]
>> https://lists.apache.org/thread.html/r87a33f0046949fe59188e844832297d8a00e67
>> 5c4e59a1ab01005e0e%40%3Cdev.subversion.apache.org%3E
>>
>> Cheers,
>> --
>> Yasuhito FUTATSUKI /
>>
> 

Cheers,
-- 
Yasuhito FUTATSUKI /

Re: SVN 1.14 release: please fix your buildbot

[Yasuhito FUTATSUKI]

On 2020/05/06 20:38, b...@qqmail.nl wrote:
> This buildbot runs python 2.X (and has for years) and now the tests are
> broken. I don't remember seeing changes to the actual Conflict handing code
> in Subversion itself, so...
> I still think this is related to the changes to the test framework causing
> regressions on 2.7.

In #2922, the test is passed with Python 2.7.8, however since #2923 it seems
the buildbot uses Python 3.7.4, if I'm looking bad place.

https://ci.apache.org/builders/svn-windows-ra/builds/2922
https://ci.apache.org/builders/svn-windows-ra/builds/2923

> 
> This buildbot does the testing on Windows with python 2.7... But errors on
> this bot are often ignored by *nix developers. (It reported errors for some
> weeks because build.conf was broken for Windows... and once that was fixed
> this new problem existed so 'nobody' noticed)
> 
> -Oorspronkelijk bericht-
> Van: Yasuhito FUTATSUKI  
> Verzonden: woensdag 6 mei 2020 13:19
> Aan: dev@subversion.apache.org; b...@qqmail.nl
> Onderwerp: Re: SVN 1.14 release: please fix your buildbot
> 
> On 2020/05/06 19:48, b...@qqmail.nl wrote:
>> The Windows buildbots currently produce actual errors, most likely 
>> caused by recent changes to the python test framework. (It is 
>> expecting to see \r, but now sees \n in several text conflicts)
> 
> The reason of failures aren't recent changes of the python test framework,
> but no one cared of the tests on Windows with Python 3.  Please see the
> thread "Problems running testsuite on Windows with Python 3" [1].
> 
> We are now working to try to fix it.
> 
> [1]
> https://lists.apache.org/thread.html/r87a33f0046949fe59188e844832297d8a00e67
> 5c4e59a1ab01005e0e%40%3Cdev.subversion.apache.org%3E
> 
> Cheers,
> --
> Yasuhito FUTATSUKI /
> 


-- 
Yasuhito FUTATSUKI /

RE: SVN 1.14 release: please fix your buildbot

[bert]

This buildbot runs python 2.X (and has for years) and now the tests are
broken. I don't remember seeing changes to the actual Conflict handing code
in Subversion itself, so...
I still think this is related to the changes to the test framework causing
regressions on 2.7.

This buildbot does the testing on Windows with python 2.7... But errors on
this bot are often ignored by *nix developers. (It reported errors for some
weeks because build.conf was broken for Windows... and once that was fixed
this new problem existed so 'nobody' noticed)

-Oorspronkelijk bericht-
Van: Yasuhito FUTATSUKI  
Verzonden: woensdag 6 mei 2020 13:19
Aan: dev@subversion.apache.org; b...@qqmail.nl
Onderwerp: Re: SVN 1.14 release: please fix your buildbot

On 2020/05/06 19:48, b...@qqmail.nl wrote:
> The Windows buildbots currently produce actual errors, most likely 
> caused by recent changes to the python test framework. (It is 
> expecting to see \r, but now sees \n in several text conflicts)

The reason of failures aren't recent changes of the python test framework,
but no one cared of the tests on Windows with Python 3.  Please see the
thread "Problems running testsuite on Windows with Python 3" [1].

We are now working to try to fix it.

[1]
https://lists.apache.org/thread.html/r87a33f0046949fe59188e844832297d8a00e67
5c4e59a1ab01005e0e%40%3Cdev.subversion.apache.org%3E

Cheers,
--
Yasuhito FUTATSUKI /

Re: SVN 1.14 release: please fix your buildbot

[Yasuhito FUTATSUKI]

On 2020/05/06 19:48, b...@qqmail.nl wrote:
> The Windows buildbots currently produce actual errors, most likely caused by
> recent changes to the python test framework. (It is expecting to see \r, but
> now sees \n in several text conflicts)

The reason of failures aren't recent changes of the python test framework,
but no one cared of the tests on Windows with Python 3.  Please see the thread
"Problems running testsuite on Windows with Python 3" [1].

We are now working to try to fix it.

[1] 
https://lists.apache.org/thread.html/r87a33f0046949fe59188e844832297d8a00e675c4e59a1ab01005e0e%40%3Cdev.subversion.apache.org%3E

Cheers,
-- 
Yasuhito FUTATSUKI /

RE: SVN 1.14 release: please fix your buildbot

[bert]

The Windows buildbots currently produce actual errors, most likely caused by
recent changes to the python test framework. (It is expecting to see \r, but
now sees \n in several text conflicts)

Bert

-Oorspronkelijk bericht-
Van: Stefan Sperling  
Verzonden: donderdag 2 april 2020 16:54
Aan: Bert Huijben ; Branko Čibej ;
dev@subversion.apache.org
Onderwerp: Re: SVN 1.14 release: please fix your buildbot

On Mon, Mar 16, 2020 at 09:27:28AM +0100, Stefan Sperling wrote:
> Hi Bert and Brane,
> 
> Your windows-ra and macosx-apr1.3-nothread buildbots are currently
failing.
> 
> If possible please fix them, or retire them if they are no longer needed.
> 
> For release management it would help us to have a set of only green 
> buildbots to start with, so we can easily identify any new problems 
> across all platforms that pop up during the release testing phase as 
> fixes get committed.
> 
> Thanks,
> Stefan

Hi,

Both buildbots are still failing.

Unless someone can help out with getting these failures fixed, I will just
have to ignore these problems and go ahead with rolling a new release
anyway.

RE: Building SVN (dependencies) on Windows

[bert]

Quite a few of the Windows dependencies can easily be built by the VCPKG 
package manager.

 

For SharpSvn I maintained quite a few custom NAnt scripts that do about the 
same work (https://ctf.open.collab.net/svn/repos/sharpsvn/trunk/imports

(or   
https://github.com/AmpScm/SharpSvn/tree/master/imports, but I think this mirror 
lags a bit )

 

Somewhere in the CollabNet open source repositories there should be a slightly 
simpler -not as many development options- MSBuild version of the scripts to 
build the dependencies, but I’m not sure at which url.

 

  Bert

Van: Graham Bloice  
Verzonden: donderdag 23 april 2020 18:17
Aan: dev@subversion.apache.org
Onderwerp: Re: Building SVN (dependencies) on Windows

 

Apologies for butting in, but as a long time maintainer of the (mainly Windows) 
build process for Wireshark, another cross platform project, I'd like to offer 
my thoughts on such matters to the svn project.

1.  We moved to CMake quite some time ago, getting rid of automake and 
nmake and VS solution files.  Such a relief to have one build tool, usable on 
all platforms.  This is obviously a big step to take.
2.  Wireshark has lots of dependencies on 3rd party libraries and on 
Windows the usual issues of obtaining them and where to put them.  The project 
has "solved" this by maintaining an svn repo (even though the project has 
switched to git for the sources) of "-dev" zip packages containing the items 
required to build; headers, .lib and .dll files, e.g.  
https://anonsvn.wireshark.org/wireshark-win64-libs/trunk/packages/Part of 
the CMake generation step runs a PowerShell script that checks for updates, 
downloads the required zips and expands them to a "known" location, 
configurable by env vars.  The locations of these files are passed as hints to 
the CMake findxxx modules (modified by the project) so that CMake is happy.  
Some build tools and libraries are exclude from this process and have to be 
manually installed, e.g. Perl, Python, Qt.
3.  Production of the "-dev" packages takes a few forms, cross building on 
OpenSUSE or using vcpkg, the details are listed for each library, e.g. 
https://www.wireshark.org/docs/wsdg_html_chunked/ChLibsGLib.html.  When a 
library is updated, a single "tag" file in the main source tree is updated 
causing all builders to check for changes.
4.  All required artefacts are built\copied by CMake directives into the 
run-time directory so tests can be run without requiring installation.
5.  The installers are also built by CMake and pick up the required 
artefacts from the run-time directory.

I see a number of advantages with the Wireshark solution:

1.  Much smaller barrier to entry for those attempting to build.  Still 
quite a setup needed, hence the developer's guide step-by-step section, see 
here:https://www.wireshark.org/docs/wsdg_html_chunked/ChSetupWin32.html
2.  No need to build (or even download unless there are changes) the 3rd 
party libraries reducing build times
3.  Consistency across builds for all users, no local variations causing 
oddities.

You can see the sausages being made over at the Wireshark buildbot: 
https://buildbot.wireshark.org/wireshark-master/waterfall, in particular the 
"ran CMake" generation step, although if there are no updates to the libraries 
it's a standard CMake experience.

 

I'm not subscribed to the list, just observe infrequently via the svn.haxx.se 
  archive, so if you have any questions please add me in 
replies.

 

P.S. I did build svn on Windows once, many years ago and I found it an awkward 
process.

 

-- 

Graham Bloice

RE: Svnmucc sensitive to order of mkdir

[bert]

1.8.15 just orders the operations based on path, so you may be lucky that it 
works, but there are a lot of other examples where pre 1.9 fail caused by the 
simplicity of the code there. (Thinks like renaming directories and then 
applying operations). With 1.9 there is a backing framework that should allow 
options like –parent to be added without too much hassle. (I know the path 
exists check that knows the previously scheduled operations is there… It is 
probably the reason your example of adding a directory before its parent 
fails). 

 

I think parsing the commandline with optional flags might be the hardest part 
here, as svnmucc doesn’t have such options on subcommands yet as far as I know.

 

  Bert

 

Van: 钱海远(Nathan)  
Verzonden: dinsdag 28 april 2020 14:45
Aan: dev@subversion.apache.org
Onderwerp: Svnmucc sensitive to order of mkdir

 

Dear all,

 

I   was discuss with Daniel Shahaf about the 
Issue: https://issues.apache.org/jira/browse/SVN-4854

 

SVN 1.10.6 change the svnmucc , it’s sensitive with mkdir operations, if a want 
to mkdir /foot/bar when /foot is not exists , svnmucc only can work with 
svnmucc mkdir ‘/foor’ mkdir ‘/foot/bar’ . Cannot work with svnmucc mkdir 
‘/foot/bar’ mkdir ‘/foot’. But it’s both success with 1.8.15 version.

 

Daniel Shahaf told me the new subversion is much more sensitive to order of 
operation.

 

So, can we set the option --parent with svnmucc mkdir?

 

  _  

Best Regards!

Haiyuan Qian

R & D Management Group

Hangzhou Hikvision Digital Technology Co.,Ltd

No.555 Qianmo Road, Binjiang District, Hangzhou 310052, China 

M (86)18969199712

 

本邮件及其附件含有海康威视公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！
 

This e-mail and its attachments contain confidential information from 
HIKVISION, which is intended only for  the person or entity whose address is 
listed above. Any use of the information contained herein in any way 
(including, but not limited to, total or partial disclosure, reproduction, or 
dissemination) by persons other  than the intended recipient(s) is prohibited. 
If you receive this e-mail in error, please notify the sender by phone or email 
immediately and delete it!

 

 

  _  

CONFIDENTIALITY NOTICE: This electronic message is intended to be viewed only 
by the individual or entity to whom it is addressed. It may contain information 
that is privileged, confidential and exempt from disclosure under applicable 
law. Any dissemination, distribution or copying of this communication is 
strictly prohibited without our prior permission. If the reader of this message 
is not the intended recipient, or the employee or agent responsible for 
delivering the message to the intended recipient, or if you have received this 
communication in error, please notify us immediately by return e-mail and 
delete the original message and any copies of it from your computer system. For 
further information about Hikvision company. please see our website at 
www.hikvision.com  

Re: Problems running testsuite on Windows with Python 3

[Yasuhito FUTATSUKI]

On 2020/05/04 4:49, Johan Corveleyn wrote:
> On Sun, May 3, 2020 at 6:23 PM Yasuhito FUTATSUKI  wrote:
>>
>> On 2020/05/03 10:02, Johan Corveleyn wrote:
>>> On Fri, May 1, 2020 at 7:46 PM Yasuhito FUTATSUKI  
>>> wrote:

 On 2020/04/23 2:05, Yasuhito FUTATSUKI wrote:

 It seems following tests are EOL style sensitive, but they didn't
 check without strict EOL style on Windows:

   merge_tests.merge_conflict_markers_matching_eol
   merge_tests.merge_eolstyle_handling
   patch_tests.patch_no_svn_eol_style
   patch_tests.patch_with_svn_eol_style
   patch_tests.patch_with_svn_eol_style_uncommitted
   update_tests.conflict_markers_matching_eol
   update_tests.update_eol_style_handling

 I've not check that each tests depend that native EOL is '\n' or not.
 However, I think if some tests depend on it, other test cases for
 those aims are needed on Windows, or at least explicitly skip the tests
 on Windows to clarify that those tests check nothing for the aims.

 The updated patch attached only make these tests EOL style sensitive
 (and relax check of EOL on Python 2 if keep_eol_style optional
 parameter is not specified).
>>>
>>> Thank you for continuing to work on this, Yasuhito.
>>>
>>> This patch indeed fixes the above 7 tests when running with Python 3 on 
>>> Windows.
>>> However, when running with Python 2 there are now 6 tests that fail
>>> (without this patch, all tests are successful with Py2):
>>
>> Thank you for testing.
>>
>> As far as this test result, those test cases don't depend on specific
>> native EOL.
>>
>>> FAIL:  merge_tests.py 34: conflict markers should match the file's eol style
>>> FAIL:  patch_tests.py 13: patch target with no svn:eol-style
>>> FAIL:  patch_tests.py 14: patch target with svn:eol-style
>>> FAIL:  patch_tests.py 15: patch target with uncommitted svn:eol-style
>>> FAIL:  patch_tests.py 57: patch a binary file
>>> FAIL:  update_tests.py 26: conflict markers should match the file's eol 
>>> style
>>>
>>> See fails.log in attachment.
>>
>> I overlooked that result of io.TextIO.read() is unicode on Python 2.
>> I hope updated patch may resolve this issue.
> 
> Okay, that latest version fixes this test for Python 2:
> 
> patch_tests.py 57: patch a binary file
> 
> But the five other failures still remain for Py2:
> 
> FAIL:  merge_tests.py 34: conflict markers should match the file's eol style
> FAIL:  patch_tests.py 13: patch target with no svn:eol-style
> FAIL:  patch_tests.py 14: patch target with svn:eol-style
> FAIL:  patch_tests.py 15: patch target with uncommitted svn:eol-style
> FAIL:  update_tests.py 26: conflict markers should match the file's eol style
> 
> See fails_py2.log in attachment.

I read those test code again, and I found that they don't distinct
text I/O and binary I/O when they write to files in some case, So I
updated the patch address them. (sain_keep_eol_style_win_patch_20200506.txt)

> For Python 3 we're getting close. After this patch and the one for
> svnrdump_tests, we only have 2 failures left with Python 3:

Please don't forget our goal is not to make the tests be passed but
to make the tests check what we want to check, on Windows both with
py2 and py3.

I've fixed the test codes to make them what the author intended I think.
If the results of py2 and py3 are still different, the test code are
still broken. With a broken test, we can't warrant ether the test target
code is correct or the test case is correct, even if the tests can be
passed. So we should also check the test cases (scenarios and expected
results) themselves are correct on Windows(, and I didn't it... at
least, yet).
 
> FAIL:  svnadmin_tests.py 35: detect denormalized names and name collisions

This is caused by output message of svnadmin, containing non UTF-8
character. Attached patch fix_svnadmin_tests_patch.txt address it,
however I don't have confidence because I don't know what charset/encoding
svnadmin on Windows use. With Python 2, output message is treated
as bytes as is, this is not affected.

> FAIL:  svndumpfilter_tests.py 7: svndumpfilter with an empty prefix

It seems this is just same reason as svnrdump_tests.py was broken.
(fix_svndumpfilter_tests_patch.txt)

Cheers,
-- 
Yasuhito FUTATSUKI /
Fix eol style treatment in command tests on Windows.

[in subversion/tests/cmdline]

* merge_tests.py (merge_conflict_markers_matching_eol,
  merge_eolstyle_handling),
* patch_tests.py (patch_no_svn_eol_style,
  patch_with_svn_eol_style,
  patch_with_svn_eol_style_uncommitted),
* update_tests.py (conflict_markers_matching_eol,
   update_eol_style_handling):
 Specify keep_eol_style = True evne if platform is Windows.

* merge_tests.py (merge_conflict_markers_matching_eol),
* patch_tests.py (patch_no_svn_eol_style,
  patch_with_svn_eol_style,
  patch_with_svn_eol_style_uncommitted),
* update_tests.py