Bug#809209: libsvn_subr-1.so.1: svn: E235000: dirent_uri.c:2335: assertion

[Philipp Marek]

> > I thought that the API is 
> > defined for UTF8-strings, so I can understand about using urlencode for
> > [%\/?&] - but for "normal", printable UTF8 characters?
> 
> Well, the description of svn_path_canonicalize[0] leaves vague the full
> specification of what transformations will be applied:
...
> > > I'm inclined to close this as the API being used incorrectly.
> > So, I should just make FSVS pass eg. "ö" as "%C3%B6" instead?
> 
> I'd suggest using svn_path_canonicalize (or one the more granular
> replacements, depending on the minimal version of SVN you want to
> impose).  Then you know you're using a properly formed path.
Done, see
http://fsvs.tigris.org/source/browse/fsvs?view=rev=2487


Thanks for your help!!

Bug#809209: libsvn_subr-1.so.1: svn: E235000: dirent_uri.c:2335: assertion

[Philipp Marek]

Hi James,

thank you very much for your patience!


> > > Granted, the assert is new but it's just asserting the contract of the
> > > APIs.  The URIs were always supposed to be canonicalized.
> > Yeah, and I even have code that deals with a slash at the end and so on.
> > 
> > The other idea is that the requirements have changed - perhaps an UTF8 URL 
> > like 
> > file:///tmp/fsvs-test-1044/repos/trunk/ä-30305/ö-13157/§-2186
> > isn't allowed anymore, but the non-ASCII characters have to be encoded?
> 
> From what I can tell, that was always supposed to be the case.
What was? That non-ASCII needs to be encoded? I thought that the API is 
defined for UTF8-strings, so I can understand about using urlencode for
[%\/?&] - but for "normal", printable UTF8 characters?

> I'm inclined to close this as the API being used incorrectly.
So, I should just make FSVS pass eg. "ö" as "%C3%B6" instead?

Bug#809209: libsvn_subr-1.so.1: svn: E235000: dirent_uri.c:2335: assertion

[James McCoy]

On Sun, Jan 10, 2016 at 12:07:15PM +0100, Philipp Marek wrote:
> Hi James,
> 
> thank you very much for your patience!
> 
> 
> > > > Granted, the assert is new but it's just asserting the contract of the
> > > > APIs.  The URIs were always supposed to be canonicalized.
> > > Yeah, and I even have code that deals with a slash at the end and so on.
> > > 
> > > The other idea is that the requirements have changed - perhaps an UTF8 
> > > URL 
> > > like 
> > > file:///tmp/fsvs-test-1044/repos/trunk/ä-30305/ö-13157/§-2186
> > > isn't allowed anymore, but the non-ASCII characters have to be encoded?
> > 
> > From what I can tell, that was always supposed to be the case.
> What was? That non-ASCII needs to be encoded?

Yes.

> I thought that the API is 
> defined for UTF8-strings, so I can understand about using urlencode for
> [%\/?&] - but for "normal", printable UTF8 characters?

Well, the description of svn_path_canonicalize[0] leaves vague the full
specification of what transformations will be applied:

  This involves collapsing redundant "/./" elements, removing multiple
  adjacent separator characters, removing trailing separator characters,
  and possibly other semantically inoperative transformations.

[0]: 
https://subversion.apache.org/docs/api/latest/svn__path_8h.html#a71d5716a18e04df175633a65ddac7809

Testing shows that the canonical version of the path does indeed encode
the UTF-8 characters:

  $ python
  Python 2.7.11 (default, Dec  9 2015, 00:29:25)
  [GCC 5.3.1 20151205] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  >>> from svn import core
  >>> 
core.svn_path_canonicalize('file:///tmp/fsvs-test-1044/repos/trunk/ä-30305/ö-13157/§-2186')
  'file:///tmp/fsvs-test-1044/repos/trunk/%C3%A4-30305/%C3%B6-13157/%C2%A7-2186'

> > I'm inclined to close this as the API being used incorrectly.
> So, I should just make FSVS pass eg. "ö" as "%C3%B6" instead?

I'd suggest using svn_path_canonicalize (or one the more granular
replacements, depending on the minimal version of SVN you want to
impose).  Then you know you're using a properly formed path.

Cheers,
-- 
James
GPG Key: 4096R/331BA3DB 2011-12-05 James McCoy 

Bug#809209: libsvn_subr-1.so.1: svn: E235000: dirent_uri.c:2335: assertion

[James McCoy]

On Fri, Jan 08, 2016 at 08:25:57AM +0100, Philipp Marek wrote:
> > Granted, the abort needs to be fixed,

Or not.  The library is asserting the expected semantics of its API.
The semantics didn't change, just how strictly they were being checked
did.

> > Ok, during SVN 1.7 development (r957893) moved a bunch of code from
> > svn_ra_local__split_URL to a new function,
> > svn_uri_get_dirent_from_file_url.  When that happened, the assert you're
> > hitting was added:
> > 
> >   assert(svn_uri_is_canonical(url, pool));
> > 
> > You're not canonicalizing the URL before passing it into svn_ra_open().
> > You probably want to use svn_path_canonicalize() (or if you can accept a
> > minimum SVN version of 1.6, one of svn_dirent_canonicalize,
> > svn_uri_canonicalize, svn_relpath_canonicalize) in url__parse instead of
> > manually trying to make it canonical.
> > 
> > Granted, the assert is new but it's just asserting the contract of the
> > APIs.  The URIs were always supposed to be canonicalized.
> Yeah, and I even have code that deals with a slash at the end and so on.
> 
> 
> The other idea is that the requirements have changed - perhaps an UTF8 URL 
> like 
> file:///tmp/fsvs-test-1044/repos/trunk/ä-30305/ö-13157/§-2186
> isn't allowed anymore, but the non-ASCII characters have to be encoded?

>From what I can tell, that was always supposed to be the case.

I'm inclined to close this as the API being used incorrectly.

Cheers,
-- 
James
GPG Key: 4096R/331BA3DB 2011-12-05 James McCoy 

Bug#809209: libsvn_subr-1.so.1: svn: E235000: dirent_uri.c:2335: assertion

[James McCoy]

On Mon, Dec 28, 2015 at 11:27:28AM +0100, Ph. Marek wrote:
> During a run with my automated test suite for FSVS I noticed that some 
> things that previously worked do not any more.
> I can't really give a hard range of SVN versions, but I guess 2 or 3 years 
> ago that wasn't a problem yet ;/

Hmmm.  I tested all the way back to SVN 1.5 and I either get the abort
you're reporting or a failure trying to setup the working copy in order
to run the test.

  $ make BINARY=../src/fsvs TEST_LIST=037\* -C tests
  make: Entering directory '/home/jamessan/src/debian.org/fsvs/fsvs-1.2.6/tests'
  test -d /tmp/fsvs-test-1000 || mkdir -p /tmp/fsvs-test-1000
  test -d /tmp/fsvs-test-1000/waa || mkdir -p /tmp/fsvs-test-1000/waa
  test -d /tmp/fsvs-test-1000/conf || mkdir -p /tmp/fsvs-test-1000/conf
  Preparing default repository.


  An error occurred: Couldn't open a repository (180001)
in url__open_session: 
svn_ra_open("file:///tmp/fsvs-test-1000/repos/trunk"): Unable to connect to a 
repository at URL 'file:///tmp/fsvs-test-1000/repos/trunk'
  make[3]: *** [prepare_wc] Error 2


  An error occurred: Couldn't open a repository (180001)
in url__open_session: 
svn_ra_open("file:///tmp/fsvs-test-1000/repos/trunk"): Unable to connect to a 
repository at URL 'file:///tmp/fsvs-test-1000/repos/trunk'
  make[3]: *** [prepare_wc] Error 2
  make[2]: *** [prepare_wcs] Error 2
  make[1]: *** [prepare_clean] Error 2
  Makefile:166: recipe for target '/tmp/fsvs-test-1000/default-repos.dump' 
failed
  make: *** [/tmp/fsvs-test-1000/default-repos.dump] Error 2
  make: Leaving directory '/home/jamessan/src/debian.org/fsvs/fsvs-1.2.6/tests'

I didn't get a single successful run of the test.

Granted, the abort needs to be fixed, but not being able to run the test
makes it hard to bisect where things broke.

Cheers,
-- 
James
GPG Key: 4096R/331BA3DB 2011-12-05 James McCoy 

Bug#809209: libsvn_subr-1.so.1: svn: E235000: dirent_uri.c:2335: assertion

[James McCoy]

On Thu, Jan 07, 2016 at 07:33:38PM -0500, James McCoy wrote:
> On Mon, Dec 28, 2015 at 11:27:28AM +0100, Ph. Marek wrote:
> > During a run with my automated test suite for FSVS I noticed that some 
> > things that previously worked do not any more.
> > I can't really give a hard range of SVN versions, but I guess 2 or 3 years 
> > ago that wasn't a problem yet ;/
> 
> Hmmm.  I tested all the way back to SVN 1.5 and I either get the abort
> you're reporting or a failure trying to setup the working copy in order
> to run the test.

Ok, during SVN 1.7 development (r957893) moved a bunch of code from
svn_ra_local__split_URL to a new function,
svn_uri_get_dirent_from_file_url.  When that happened, the assert you're
hitting was added:

  assert(svn_uri_is_canonical(url, pool));

You're not canonicalizing the URL before passing it into svn_ra_open().
You probably want to use svn_path_canonicalize() (or if you can accept a
minimum SVN version of 1.6, one of svn_dirent_canonicalize,
svn_uri_canonicalize, svn_relpath_canonicalize) in url__parse instead of
manually trying to make it canonical.

Granted, the assert is new but it's just asserting the contract of the
APIs.  The URIs were always supposed to be canonicalized.

Cheers,
-- 
James
GPG Key: 4096R/331BA3DB 2011-12-05 James McCoy 

Bug#809209: libsvn_subr-1.so.1: svn: E235000: dirent_uri.c:2335: assertion

[Philipp Marek]

Hi James,


>   $ make BINARY=../src/fsvs TEST_LIST=037\* -C tests
...
>   An error occurred: Couldn't open a repository (180001)
> in url__open_session: 
> svn_ra_open("file:///tmp/fsvs-test-1000/repos/trunk"): Unable to connect to a 
> repository at URL 'file:///tmp/fsvs-test-1000/repos/trunk'
>   make[3]: *** [prepare_wc] Error 2
...
> I didn't get a single successful run of the test.
Just sitting in src/ and running "make run-tests TEST_LIST=037*" doesn't 
work for you? Is the repository in "/tmp/fsvs-test-1000/repos/" created 
for you?

Hmmm, what could be different?
I've got /tmp on a tmpfs.


> Granted, the abort needs to be fixed, but not being able to run the test
> makes it hard to bisect where things broke.
Yeah, right.


> Ok, during SVN 1.7 development (r957893) moved a bunch of code from
> svn_ra_local__split_URL to a new function,
> svn_uri_get_dirent_from_file_url.  When that happened, the assert you're
> hitting was added:
> 
>   assert(svn_uri_is_canonical(url, pool));
> 
> You're not canonicalizing the URL before passing it into svn_ra_open().
> You probably want to use svn_path_canonicalize() (or if you can accept a
> minimum SVN version of 1.6, one of svn_dirent_canonicalize,
> svn_uri_canonicalize, svn_relpath_canonicalize) in url__parse instead of
> manually trying to make it canonical.
> 
> Granted, the assert is new but it's just asserting the contract of the
> APIs.  The URIs were always supposed to be canonicalized.
Yeah, and I even have code that deals with a slash at the end and so on.


The other idea is that the requirements have changed - perhaps an UTF8 URL 
like 
file:///tmp/fsvs-test-1044/repos/trunk/ä-30305/ö-13157/§-2186
isn't allowed anymore, but the non-ASCII characters have to be encoded?


Thank you for your help!

Bug#809209: libsvn_subr-1.so.1: svn: E235000: dirent_uri.c:2335: assertion

[Ph. Marek]

Package: libsvn1
Version: 1.9.2-3+b1
Severity: normal
File: /usr/lib/x86_64-linux-gnu/libsvn_subr-1.so.1


Original subject before shortening:
/usr/lib/x86_64-linux-gnu/libsvn_subr-1.so.1: svn: E235000: In file 
'.../dirent_uri.c' line 2335: assertion failed (svn_uri_is_canonical(url, pool))

During a run with my automated test suite for FSVS I noticed that some 
things that previously worked do not any more.
I can't really give a hard range of SVN versions, but I guess 2 or 3 years 
ago that wasn't a problem yet ;/


Basically, my code does an "svn_ra_open()" with an URL of 
"file:///tmp/fsvs-test-1044/repos/trunk/ä-30305/ö-13157/§-2186"; the 
repository base is (easily guessed ;) at 
"file:///tmp/fsvs-test-1044/repos/".

I guess that the non-ASCII characters in there are not parsed correctly 
(any more).


The stack trace shows this, with the last three being in FSVS:

Program received signal SIGABRT, Aborted.
0x7684a657 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x7684a657 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x7684ba2a in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x77b0558e in svn_error_abort_on_malfunction () from 
/usr/lib/x86_64-linux-gnu/libsvn_subr-1.so.1
#3  0x77b03a45 in svn_uri_get_dirent_from_file_url () from 
/usr/lib/x86_64-linux-gnu/libsvn_subr-1.so.1
#4  0x7587b74d in svn_ra_local__split_URL () from 
/usr/lib/x86_64-linux-gnu/libsvn_ra_local-1.so.1
#5  0x7587af3d in ?? () from 
/usr/lib/x86_64-linux-gnu/libsvn_ra_local-1.so.1
#6  0x77698d43 in svn_ra_open4 () from 
/usr/lib/x86_64-linux-gnu/libsvn_ra-1.so.1
#7  0x77695c3d in svn_ra_open3 () from 
/usr/lib/x86_64-linux-gnu/libsvn_ra-1.so.1
#8  0x77695c66 in svn_ra_open2 () from 
/usr/lib/x86_64-linux-gnu/libsvn_ra-1.so.1
#9  0x77695d26 in svn_ra_open () from 
/usr/lib/x86_64-linux-gnu/libsvn_ra-1.so.1
#10 0x0040b88e in url__open_session (session=session@entry=0x0, 
missing_dirs=missing_dirs@entry=0x7fffdf98) at url.c:1103
#11 0x0040f5c3 in ci__work (root=0x7fffe080, argc=0x0, 
argv=0x7fffe278) at commit.c:1138
#12 0x00404aa8 in main (argc=, args=0x7fffe268, 
env=) at fsvs.c:1408


The repository exists, but has no content yet.
FSVS tries to open this RA path; previously it would simply get ENOENT 
back, so it could determine which intermediate directories it had to create 
- but now that an assertion is triggered, it won't get to that point any 
more, but just "crashes" before.


In case that's of interest, this is the "037_mkdir_base" test in FSVS.


Thank you!


-- System Information:
Debian Release: stretch/sid
  APT prefers testing
  APT policy: (990, 'testing'), (500, 'unstable'), (500, 'stable'), (1, 
'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 4.2.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=de_AT.UTF-8, LC_CTYPE=de_AT.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages libsvn1:amd64 depends on:
ii  libapr11.5.2-3
ii  libaprutil11.5.4-1+b1
ii  libc6  2.21-4
ii  libdb5.3   5.3.28-11
ii  libexpat1  2.1.0-7
ii  libldap-2.4-2  2.4.42+dfsg-2
ii  libsasl2-2 2.1.26.dfsg1-14+b1
ii  libserf-1-11.3.8-1+b1
ii  libsqlite3-0   3.9.2-1
ii  zlib1g 1:1.2.8.dfsg-2+b1

libsvn1:amd64 recommends no packages.

libsvn1:amd64 suggests no packages.

-- no debconf information

-- 
: Ing. Philipp Marek
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com :

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.