Re: mailer.py commit says TypeError: must be unicode, not str

[Nico Kadel-Garcia]

On Wed, Jan 31, 2018 at 10:40 PM, Kenneth Porter  wrote:
> --On Wednesday, January 31, 2018 7:23 PM -0800 Kenneth Porter
>  wrote:
>
>> fp = builtins.open(file, 'w+') # avoid namespace clash with
>># trimmed-down svn_fs_open()
>
>
> I'm now thinking the problem is in the open call, and that I'm somehow
> getting a Python 3 open function even though I've got Python 2.7 installed.
> Should the mode be 'wb' instead of 'w+'? That would insure that the raw data
> from the Subversion object is getting dumped into the temporary fle without
> interpretation. I don't understand why update (denoted by the plus) is
> wanted. The temp file isn't being read from.

If you are on RHEL based operating systems, some of them install both,
and sometimes with some fascinating "PATH" settings. It can be very
handy to edit customized .py scripts to explicitly call the version of
Python you reall want to use.

Re: mailer.py commit says TypeError: must be unicode, not str

[Kenneth Porter]

--On Wednesday, January 31, 2018 7:23 PM -0800 Kenneth Porter 
 wrote:



fp = builtins.open(file, 'w+') # avoid namespace clash with
   # trimmed-down svn_fs_open()


I'm now thinking the problem is in the open call, and that I'm somehow 
getting a Python 3 open function even though I've got Python 2.7 installed. 
Should the mode be 'wb' instead of 'w+'? That would insure that the raw 
data from the Subversion object is getting dumped into the temporary fle 
without interpretation. I don't understand why update (denoted by the plus) 
is wanted. The temp file isn't being read from.

Re: mailer.py commit says TypeError: must be unicode, not str

[Kenneth Porter]

--On Tuesday, January 23, 2018 7:14 AM -0800 Kenneth Porter 
 wrote:



   File "/usr/lib64/python2.7/site-packages/svn/fs.py", line 87, in
_dump_contents
 fp.write(chunk)
TypeError: must be unicode, not str


Here's the code where this is going wrong. I think svn_stream_read is 
returning a byte stream and the file object here is expecting a unicode 
string. Is there a missing decode('utf-8') call? (I'm a very new Python 
coder but have lots of experience in C++ and some understanding of unicode.)


Package is subversion-python-1.7.14-11.el7_4.x86_64 in CentOS 7.4.


From /usr/lib64/python2.7/site-packages/svn/fs.py


 def _dump_contents(self, file, root, path, pool=None):
   fp = builtins.open(file, 'w+') # avoid namespace clash with
  # trimmed-down svn_fs_open()
   if path is not None:
 stream = file_contents(root, path, pool)
 try:
   while True:
 chunk = _svncore.svn_stream_read(stream, 
_svncore.SVN_STREAM_CHUNK_SIZE)

 if not chunk:
   break
 fp.write(chunk)
 finally:
   _svncore.svn_stream_close(stream)
   fp.close()

BTW, I found this nice treatment of unicode in Python 2 and 3:

Re: Building 1.8.19 from source tarball

[Daniel Shahaf]

Stefan Sperling wrote on Wed, 31 Jan 2018 16:58 +0100:
> On Wed, Jan 31, 2018 at 10:45:07AM -0500, Nathan Hartman wrote:
> > The get-deps.sh script that comes with the source tarball is fetching
> > apr-1.4.6 and apr-util-1.5.1. There are newer versions available,
> > namely apr-1.6.3 and apr-util-1.6.1.
> 
> You should be able to use current APR releases without issues.

We don't update the get-deps.sh recommended version whenever APR makes
a release; we only update that in the run-up to a 1.x.0 release and when
upstream APR has a high-severity bug.  Consequently, it's common that
even on the day a Subversion release is made, its get-deps.sh wouldn't point to
the latest APR version, but to whichever was latest at the time of the 
preceding 1.x.0.

Re: Building 1.8.19 from source tarball

[Stefan Sperling]

On Wed, Jan 31, 2018 at 10:45:07AM -0500, Nathan Hartman wrote:
> Hi all,
> 
> I'm planning to update a Subversion server from 1.8.11 to 1.8.19.
> 
> I'm building from the source tarball in order to run the test suites
> of svn and its dependencies.
> 
> The get-deps.sh script that comes with the source tarball is fetching
> apr-1.4.6 and apr-util-1.5.1. There are newer versions available,
> namely apr-1.6.3 and apr-util-1.6.1.
> 
> Is it better to use the newer versions or the ones called for in the
> get-deps.sh script?
> 
> Thanks,
> Nathan Hartman

You should be able to use current APR releases without issues.

Building 1.8.19 from source tarball

[Nathan Hartman]

Hi all,

I'm planning to update a Subversion server from 1.8.11 to 1.8.19.

I'm building from the source tarball in order to run the test suites
of svn and its dependencies.

The get-deps.sh script that comes with the source tarball is fetching
apr-1.4.6 and apr-util-1.5.1. There are newer versions available,
namely apr-1.6.3 and apr-util-1.6.1.

Is it better to use the newer versions or the ones called for in the
get-deps.sh script?

Thanks,
Nathan Hartman

RE: Issue while loading the SVN Dump SVN version 1.9.7

[Santosh Kondapuram]

Hi Johan,

As you suggested I have removed the leading spaces from line 39 
(enable-rep-sharing=false) and this time it worked and was able to successfully 
load the problematic revision.
So does this conclude I have the sha-1 colliding files in my repo ? Also what 
are the next steps to catchup the latest revisions from the master node ?
Appreciate all your help and great working with you on this issue !!!

FYI,

By the way I have tried running the following command " svnadmin load -M 0 
/u01/svn/repos < incdump724413.txt "with rep-sharing enabled and still see the 
same issue. I have tried doing this before the above work around which worked.

Thanks,
Santosh.

-Original Message-
From: Johan Corveleyn [mailto:jcor...@gmail.com] 
Sent: Wednesday, January 31, 2018 8:59 AM
To: Santosh Kondapuram 
Cc: users@subversion.apache.org
Subject: Re: Issue while loading the SVN Dump SVN version 1.9.7

On Wed, Jan 31, 2018 at 2:44 PM, Johan Corveleyn  wrote:
> On Wed, Jan 31, 2018 at 9:28 AM, Santosh Kondapuram 
>  wrote:
>> Hi Johan,
>>
>> Sorry for the delayed response as I was in long vacation.
>> Yes, I don’t think we are hitting the real sha-1 collision in our repository 
>> and as you said it might be another bug in the sha-1 collision detection 
>> code.
>> I am sure that no one committed the sha-1-colliding files from  
>> https://shattered.io Our repository is very huge with 1 Million + revisions 
>> and I am seeing the issue only while loading the specific revision 724413. I 
>> am able to load the revisions before and after it.
>> As you suggested I have now loaded the dump till 724412 which is the 
>> revision before the problematic one with rep sharing enabled.
>> It's not allowing me to load the problematic revision with rep-sharing 
>> disabled and getting below message.
>>
>> FYI,
>>
>> [root@vitech-svn-slave-test-01 svndump]# svnadmin load /u01/svn/repos 
>> < incdump724413.txt
>> svnadmin: E22: Error while parsing config file: 
>> /u01/svn/repos/db/fsfs.conf:
>> svnadmin: E22: line 39: Option expected
>
> Hm, that indicates that there was a syntax error on line 39 in the 
> db/fsfs.conf file you edited (when you were trying to disable 
> rep-sharing). Can you take another look and try again with rep-sharing 
> disabled? That line in fsfs.conf should be (without leading spaces):
>
> enable-rep-sharing = false
>
> Now, I'm not sure what we'll be able to do next to figure this out.
> First step is probably find out which two files are colliding, and 
> what their contents is exactly. It's extremely unlikely that there is 
> a real sha1 collision, but we have to rule it out I suppose.

Santosh,

Can you double-check that using -M 0 for the 'svnadmin load' operation doesn't 
solve the problem (while keeping rep-sharing enabled)?

So:
svnadmin load -M 0 /u01/svn/repos < incdump724413.txt

It's just that this works around the only currently known bug in the 
sha1-collision-detection code, so I want to be really sure that this workaround 
doesn't help in your case and we need to look further.

--
Johan

This e-mail message and any files transmitted with it may contain confidential 
and proprietary information and are intended solely for the use of the 
individual or entity to which they are addressed. Any unauthorized review, use, 
disclosure or distribution is strictly prohibited. If you have received this 
e-mail in error please notify the sender by reply email and destroy all copies 
of the original message. Thank you for your cooperation.

Re: Issue while loading the SVN Dump SVN version 1.9.7

[Johan Corveleyn]

On Wed, Jan 31, 2018 at 2:44 PM, Johan Corveleyn  wrote:
> On Wed, Jan 31, 2018 at 9:28 AM, Santosh Kondapuram
>  wrote:
>> Hi Johan,
>>
>> Sorry for the delayed response as I was in long vacation.
>> Yes, I don’t think we are hitting the real sha-1 collision in our repository 
>> and as you said it might be another bug in the sha-1 collision detection 
>> code.
>> I am sure that no one committed the sha-1-colliding files from  
>> https://shattered.io
>> Our repository is very huge with 1 Million + revisions and I am seeing the 
>> issue only while loading the specific revision 724413. I am able to load the 
>> revisions before and after it.
>> As you suggested I have now loaded the dump till 724412 which is the 
>> revision before the problematic one with rep sharing enabled.
>> It's not allowing me to load the problematic revision with rep-sharing 
>> disabled and getting below message.
>>
>> FYI,
>>
>> [root@vitech-svn-slave-test-01 svndump]# svnadmin load /u01/svn/repos < 
>> incdump724413.txt
>> svnadmin: E22: Error while parsing config file: 
>> /u01/svn/repos/db/fsfs.conf:
>> svnadmin: E22: line 39: Option expected
>
> Hm, that indicates that there was a syntax error on line 39 in the
> db/fsfs.conf file you edited (when you were trying to disable
> rep-sharing). Can you take another look and try again with rep-sharing
> disabled? That line in fsfs.conf should be (without leading spaces):
>
> enable-rep-sharing = false
>
> Now, I'm not sure what we'll be able to do next to figure this out.
> First step is probably find out which two files are colliding, and
> what their contents is exactly. It's extremely unlikely that there is
> a real sha1 collision, but we have to rule it out I suppose.

Santosh,

Can you double-check that using -M 0 for the 'svnadmin load' operation
doesn't solve the problem (while keeping rep-sharing enabled)?

So:
svnadmin load -M 0 /u01/svn/repos < incdump724413.txt

It's just that this works around the only currently known bug in the
sha1-collision-detection code, so I want to be really sure that this
workaround doesn't help in your case and we need to look further.

-- 
Johan

Re: Issue while loading the SVN Dump SVN version 1.9.7

[Johan Corveleyn]

On Wed, Jan 31, 2018 at 9:28 AM, Santosh Kondapuram
 wrote:
> Hi Johan,
>
> Sorry for the delayed response as I was in long vacation.
> Yes, I don’t think we are hitting the real sha-1 collision in our repository 
> and as you said it might be another bug in the sha-1 collision detection code.
> I am sure that no one committed the sha-1-colliding files from  
> https://shattered.io
> Our repository is very huge with 1 Million + revisions and I am seeing the 
> issue only while loading the specific revision 724413. I am able to load the 
> revisions before and after it.
> As you suggested I have now loaded the dump till 724412 which is the revision 
> before the problematic one with rep sharing enabled.
> It's not allowing me to load the problematic revision with rep-sharing 
> disabled and getting below message.
>
> FYI,
>
> [root@vitech-svn-slave-test-01 svndump]# svnadmin load /u01/svn/repos < 
> incdump724413.txt
> svnadmin: E22: Error while parsing config file: 
> /u01/svn/repos/db/fsfs.conf:
> svnadmin: E22: line 39: Option expected

Hm, that indicates that there was a syntax error on line 39 in the
db/fsfs.conf file you edited (when you were trying to disable
rep-sharing). Can you take another look and try again with rep-sharing
disabled? That line in fsfs.conf should be (without leading spaces):

enable-rep-sharing = false

Now, I'm not sure what we'll be able to do next to figure this out.
First step is probably find out which two files are colliding, and
what their contents is exactly. It's extremely unlikely that there is
a real sha1 collision, but we have to rule it out I suppose.

-- 
Johan

RE: Searching for a C++ API

[Bert Huijben]

> -Original Message-
> From: Branko Čibej [mailto:br...@apache.org]
> Sent: maandag 29 januari 2018 13:54
> To: users@subversion.apache.org
> Subject: Re: Searching for a C++ API
> 
> On 29.01.2018 11:45, R developer wrote:
> > Hello all,
> >
> > I have been involved in writing a closed source application that among
> > other things maintains a checkout from an SVN repository, the
> > application used to be written in C# so at the moment we are used to
> > SharpSvn. Recently the decision was made to incorporate the
> > functionality into a mobile c++ app (yes, I do wish we could use a
> > different language, but for now that's out of our scope).
> >
> > Is there a C++ library available somewhere?
> > One that can be used in a closed source app and compiles both on
> > Windows and Linux (and preferably other platforms like OSx, iOS, Android).
> > Usage of c++11, boost and/or Qt frameworks is fine as our app already
> > uses them.

It is not a real C++ library, but there might be some code that you can copy 
from the SharpSvn library code for your usage. SharpSvn is C++/CLI, to allow 
operation with the .Net environment, but there is enough plain C++ inside that 
might be able to get you started with the C library. The code is Apache 
licensed so you can mostly use it however you like.

Bert

RE: Issue while loading the SVN Dump SVN version 1.9.7

[Santosh Kondapuram]

Hi Johan,

Sorry for the delayed response as I was in long vacation.
Yes, I don’t think we are hitting the real sha-1 collision in our repository 
and as you said it might be another bug in the sha-1 collision detection code.
I am sure that no one committed the sha-1-colliding files from  
https://shattered.io
Our repository is very huge with 1 Million + revisions and I am seeing the 
issue only while loading the specific revision 724413. I am able to load the 
revisions before and after it.
As you suggested I have now loaded the dump till 724412 which is the revision 
before the problematic one with rep sharing enabled.
It's not allowing me to load the problematic revision with rep-sharing disabled 
and getting below message.

FYI,

[root@vitech-svn-slave-test-01 svndump]# svnadmin load /u01/svn/repos < 
incdump724413.txt
svnadmin: E22: Error while parsing config file: /u01/svn/repos/db/fsfs.conf:
svnadmin: E22: line 39: Option expected
[root@vitech-svn-slave-test-01 svndump]#


If I re-enable the rep-sharing and trying to load the problematic revision, I 
get the error while loading the specific file"SecurityServiceImpl.java" of that 
revision number. Please let me know if you need any further details.
Just to reiterate my source SVN is 1.9.5 and target is 1.9.7 version

FYI,

[root@vitech-svn-slave-test-01 svndump]# svnadmin load /u01/svn/repos < 
incdump724413.txt
<<< Started new transaction, based on original revision 724413
 * editing path : 
v3-web-product/branches/8.3.0.279_IPERS/benefits/Scripts/DML_Exception/web_dml_admin_businessEntityRelationsHistory.sql
 ... done.
 * editing path : 
v3-web-product/branches/8.3.0.279_IPERS/benefits/Scripts/DML_Exception/web_dml_admin_securityApplicationUserHistory.sql
 ... done.
 * editing path : 
v3-web-product/branches/8.3.0.279_IPERS/benefits/Scripts/DML_Exception/web_dml_admin_securityRoleUserHistory.sql
 ... done.
 * editing path : 
v3-web-product/branches/8.3.0.279_IPERS/benefits/Scripts/DML_Exception/web_dml_admin_userMaintenance_audit_Labels.sql
 ... done.
 * editing path : 
v3-web-product/branches/8.3.0.279_IPERS/benefits/src/com/vitechinc/v3/admin/model/BusinessEntityRelationHist.hbm
 ... done.
 * editing path : 
v3-web-product/branches/8.3.0.279_IPERS/benefits/src/com/vitechinc/v3/admin/model/BusinessEntityRelationHist.java
 ... done.
 * editing path : 
v3-web-product/branches/8.3.0.279_IPERS/benefits/src/com/vitechinc/v3/admin/model/SecurityApplicationUsrHist.hbm
 ... done.
 * editing path : 
v3-web-product/branches/8.3.0.279_IPERS/benefits/src/com/vitechinc/v3/admin/model/SecurityApplicationUsrHist.java
 ... done.
 * editing path : 
v3-web-product/branches/8.3.0.279_IPERS/benefits/src/com/vitechinc/v3/admin/model/SecurityRoleUserHist.hbm
 ... done.
 * editing path : 
v3-web-product/branches/8.3.0.279_IPERS/benefits/src/com/vitechinc/v3/admin/model/SecurityRoleUserHist.java
 ... done.
 * editing path : 
v3-web-product/branches/8.3.0.279_IPERS/benefits/src/com/vitechinc/v3/admin/model/base/BaseBusinessEntityRelationHist.java
 ... done.
 * editing path : 
v3-web-product/branches/8.3.0.279_IPERS/benefits/src/com/vitechinc/v3/admin/model/base/BaseSecurityApplicationUsrHist.java
 ... done.
 * editing path : 
v3-web-product/branches/8.3.0.279_IPERS/benefits/src/com/vitechinc/v3/admin/model/base/BaseSecurityRoleUserHist.java
 ... done.
 * editing path : 
v3-web-product/branches/8.3.0.279_IPERS/benefits/src/com/vitechinc/v3/admin/services/SecurityService.java
 ... done.
 * editing path : 
v3-web-product/branches/8.3.0.279_IPERS/benefits/src/com/vitechinc/v3/admin/services/SecurityServiceImpl.java
 ...svnadmin: E16: SHA1 of reps '669684 7 1143 195972 
efbdb058ce857b2860cfa245f014f0b9 9db457be74545c184242e57208bf1d56db1f15b2 
724412-fiyk/_16' and '-1 0 929 195972 efbdb058ce857b2860cfa245f014f0b9 
9db457be74545c184242e57208bf1d56db1f15b2 724412-fiyk/_16' matches 
(9db457be74545c184242e57208bf1d56db1f15b2) but contents differ
svnadmin: E160004: Filesystem is corrupt
svnadmin: E200014: Checksum mismatch while reading representation:
   expected:  efbdb058ce857b2860cfa245f014f0b9
 actual:  04a53277f405bcbab8a3b1fff9f8c6e0

Thanks,
Santosh.
-Original Message-
From: Johan Corveleyn [mailto:jcor...@gmail.com] 
Sent: Thursday, January 25, 2018 4:27 PM
To: Santosh Kondapuram 
Cc: users@subversion.apache.org
Subject: Re: Issue while loading the SVN Dump SVN version 1.9.7

On Thu, Jan 25, 2018 at 10:40 AM, Santosh Kondapuram 
 wrote:
> Hi Johan,
>
> I have tried reloading the dump as you suggested with  -M 0 option but still 
> I am running in to the same issue.
> Seems like the svn admin could not load the dump due to sha1 collision files. 
> So the question is how do I identify the sha1 colliding files ? does the 
> error stack trace say something about it ?
>  As per the sha1-advisory note if we create a Subversion permission 
>