Re: Slow ISVNClient.getChangelists on Linux/NFS share

2017-10-19 Thread Thomas Singer 

On 2017-10-19 10:44, Johan Corveleyn wrote:

I have no idea what ISVNClient.getChangelists corresponds to compared
with the native 'svn' commandline. But one thing that might be
relevant: in the client-side configuration file 'config' there are two
properties that can have a big influence here:

(see %APPDATA%/Subversion/config on Windows, or ~/.subversion/config on Unix)
[[[
### Section for configuring working copies.
[working-copy]
### Set to a list of the names of specific clients that should use
### exclusive SQLite locking of working copies.  This increases the
### performance of the client but prevents concurrent access by
### other clients.  Third-party clients may also support this
### option.
### Possible values:
###   svn(the command line client)
# exclusive-locking-clients =
### Set to true to enable exclusive SQLite locking of working
### copies by all clients using the 1.8 APIs.  Enabling this may
### cause some clients to fail to work properly. This does not have
### to be set for exclusive-locking-clients to work.
# exclusive-locking = false
]]]

So with the above options you can make all clients (that use that
config file) respect "exclusive-locking" (with the exclusive-locking
flag), or just a subset of specific client applications (with the
exclusive-locking-clients property).

Perhaps your commandline 'svn' invocation uses the standard
~/.subversion/config file, which has the exclusive-locking flag set,
and your SmartSVN application might use another one (it can choose
another "config dir").


Thank you for this suggestion. I've tried with

exclusive-locking = true

but it makes no difference. I don't know what to enter for

exclusive-locking-clients =

How does SVN detect the client name when using JavaHL?

--
Best regards,
Thomas Singer
=
syntevo GmbH
http://www.syntevo.com
http://www.syntevo.com/blog

Re: Slow ISVNClient.getChangelists on Linux/NFS share

2017-10-19 Thread Johan Corveleyn 
On Thu, Oct 19, 2017 at 8:30 AM, Thomas Singer
 wrote:
> On 18.10.2017 19:56, Branko Čibej wrote:
>>
>> On 18.10.2017 13:31, Thomas Singer wrote:
>>>
>>> Hi,
>>>
>>> When performing following steps on my old Linux test machine (with
>>> slow hard disk):
>>>
>>> - have a SVN working copy at /home/user/test
>>> - sudo apt install nfs-kernel-server
>>> - add following line to /etc/exports:
>>>/home/user/test *(rw,sync,no_root_squash)
>>> - start the NFS server:
>>>sudo systemctl start nfs-kernel-server.service
>>> - mount the NFS share:
>>>sudo mount localhost:/home/user/test /home/user/test.nfs
>>>
>>> and then open /home/user/test.nfs in SmartSVN 9.2 (using SVN 1.9
>>> JavaHL binaries), adding/removing a file is very slow. It boils down
>>> to the call ISVNClient.getChangelists which takes ~8s on the NFS share
>>> (/home/user/test.nfs). First, I thought, it would be caused by the
>>> native-Java overhead calling the call-back ~11,000 times for my
>>> working copy, but when using the working copy directly
>>> (/home/user/test), the method just takes <1s though the ~11,000 times
>>> call-back invocations are still there.
>>>
>>> My working copy has no local modifications, no untracked or ignored
>>> files, no changelists.
>>>
>>> Is it expected that this method (ISVNClient.getChangelists) is so slow
>>> on a NFS share even if there are no changelists?
>>
>>
>> I don't know if it's "expected" but I bet that NFS is killing SQLite
>> performance.
>>
>>
>> https://www.mail-archive.com/sqlite-users@mailinglists.sqlite.org/msg88989.html
>>
>> I'm not sure about the reason but the most likely answer, apart from
>> slow data rate and latency when compared to a local filesystem (which,
>> in your case on loopback, should not be an issue), is that the OS can't
>> really use a cache for files on NFS since it has no way to know whether
>> or not it's valid. With a lot of random-access reads and writes, that
>> can be a HUGE slowdown, as you found.
>>
>>
>> Also this:
>> https://sqlite.org/faq.html#q5
>>
>> In other words, Subversion working copies on NFS are, and have always
>> been, a bad idea; not only because of SQLite but also because
>> Subversion's code itself relies on atomic renames, which NFS does not
>> provide.
>>
>> -- Brane
>
>
> What SVN command (on command line) I should test to get a similar result as
> from ISVNClient.getChangelists? I've tried "git status" and it just needs
> <1s on the NFS working copy.

I assume you meant "svn status", not "git status".

I have no idea what ISVNClient.getChangelists corresponds to compared
with the native 'svn' commandline. But one thing that might be
relevant: in the client-side configuration file 'config' there are two
properties that can have a big influence here:

(see %APPDATA%/Subversion/config on Windows, or ~/.subversion/config on Unix)
[[[
### Section for configuring working copies.
[working-copy]
### Set to a list of the names of specific clients that should use
### exclusive SQLite locking of working copies.  This increases the
### performance of the client but prevents concurrent access by
### other clients.  Third-party clients may also support this
### option.
### Possible values:
###   svn(the command line client)
# exclusive-locking-clients =
### Set to true to enable exclusive SQLite locking of working
### copies by all clients using the 1.8 APIs.  Enabling this may
### cause some clients to fail to work properly. This does not have
### to be set for exclusive-locking-clients to work.
# exclusive-locking = false
]]]

So with the above options you can make all clients (that use that
config file) respect "exclusive-locking" (with the exclusive-locking
flag), or just a subset of specific client applications (with the
exclusive-locking-clients property).

Perhaps your commandline 'svn' invocation uses the standard
~/.subversion/config file, which has the exclusive-locking flag set,
and your SmartSVN application might use another one (it can choose
another "config dir").

-- 
Johan

Re: Slow ISVNClient.getChangelists on Linux/NFS share

2017-10-19 Thread Branko Čibej 
On 19.10.2017 10:07, Thomas Singer wrote:
> On 2017-10-19 9:05, Branko Čibej wrote:
>> On 19.10.2017 08:30, Thomas Singer wrote:
>>> On 18.10.2017 19:56, Branko Čibej wrote:
 On 18.10.2017 13:31, Thomas Singer wrote:
> Hi,
>
> When performing following steps on my old Linux test machine (with
> slow hard disk):
>
> - have a SVN working copy at /home/user/test
> - sudo apt install nfs-kernel-server
> - add following line to /etc/exports:
>     /home/user/test *(rw,sync,no_root_squash)
> - start the NFS server:
>     sudo systemctl start nfs-kernel-server.service
> - mount the NFS share:
>     sudo mount localhost:/home/user/test /home/user/test.nfs
>
> and then open /home/user/test.nfs in SmartSVN 9.2 (using SVN 1.9
> JavaHL binaries), adding/removing a file is very slow. It boils down
> to the call ISVNClient.getChangelists which takes ~8s on the NFS
> share
> (/home/user/test.nfs). First, I thought, it would be caused by the
> native-Java overhead calling the call-back ~11,000 times for my
> working copy, but when using the working copy directly
> (/home/user/test), the method just takes <1s though the ~11,000 times
> call-back invocations are still there.
>
> My working copy has no local modifications, no untracked or ignored
> files, no changelists.
>
> Is it expected that this method (ISVNClient.getChangelists) is so
> slow
> on a NFS share even if there are no changelists?

 I don't know if it's "expected" but I bet that NFS is killing SQLite
 performance.

 https://www.mail-archive.com/sqlite-users@mailinglists.sqlite.org/msg88989.html



 I'm not sure about the reason but the most likely answer, apart from
 slow data rate and latency when compared to a local filesystem (which,
 in your case on loopback, should not be an issue), is that the OS
 can't
 really use a cache for files on NFS since it has no way to know
 whether
 or not it's valid. With a lot of random-access reads and writes, that
 can be a HUGE slowdown, as you found.


 Also this:
 https://sqlite.org/faq.html#q5

 In other words, Subversion working copies on NFS are, and have always
 been, a bad idea; not only because of SQLite but also because
 Subversion's code itself relies on atomic renames, which NFS does not
 provide.

 -- Brane
>>>
>>> What SVN command (on command line) I should test to get a similar
>>> result as from ISVNClient.getChangelists?
>>
>> If "adding/removing a file" is any indication, then "svn add" or "svn
>> remove" should be comparable.
>
> Sorry, this was misleading. Adding/removing a file cause a refresh of
> which ISVNClient.getChangelists takes the longest time. With SmartSVN
> Foundation ISVNClient.getChangelists is not invoked and hence much
> faster when refreshing after a command, e.g. add or remove.
>
> Which is the SVN command line equivalent of ISVNClient.getChangelists?

The only one I can find that actually uses the
svn_client_get_changelists() function is 'svn update --changelist', but
of course it also does an update, so the comparison is not entirely valid.

-- Brane

Re: Slow ISVNClient.getChangelists on Linux/NFS share

2017-10-19 Thread Branko Čibej 
On 19.10.2017 08:30, Thomas Singer wrote:
> On 18.10.2017 19:56, Branko Čibej wrote:
>> On 18.10.2017 13:31, Thomas Singer wrote:
>>> Hi,
>>>
>>> When performing following steps on my old Linux test machine (with
>>> slow hard disk):
>>>
>>> - have a SVN working copy at /home/user/test
>>> - sudo apt install nfs-kernel-server
>>> - add following line to /etc/exports:
>>>    /home/user/test *(rw,sync,no_root_squash)
>>> - start the NFS server:
>>>    sudo systemctl start nfs-kernel-server.service
>>> - mount the NFS share:
>>>    sudo mount localhost:/home/user/test /home/user/test.nfs
>>>
>>> and then open /home/user/test.nfs in SmartSVN 9.2 (using SVN 1.9
>>> JavaHL binaries), adding/removing a file is very slow. It boils down
>>> to the call ISVNClient.getChangelists which takes ~8s on the NFS share
>>> (/home/user/test.nfs). First, I thought, it would be caused by the
>>> native-Java overhead calling the call-back ~11,000 times for my
>>> working copy, but when using the working copy directly
>>> (/home/user/test), the method just takes <1s though the ~11,000 times
>>> call-back invocations are still there.
>>>
>>> My working copy has no local modifications, no untracked or ignored
>>> files, no changelists.
>>>
>>> Is it expected that this method (ISVNClient.getChangelists) is so slow
>>> on a NFS share even if there are no changelists?
>>
>> I don't know if it's "expected" but I bet that NFS is killing SQLite
>> performance.
>>
>> https://www.mail-archive.com/sqlite-users@mailinglists.sqlite.org/msg88989.html
>>
>>
>> I'm not sure about the reason but the most likely answer, apart from
>> slow data rate and latency when compared to a local filesystem (which,
>> in your case on loopback, should not be an issue), is that the OS can't
>> really use a cache for files on NFS since it has no way to know whether
>> or not it's valid. With a lot of random-access reads and writes, that
>> can be a HUGE slowdown, as you found.
>>
>>
>> Also this:
>> https://sqlite.org/faq.html#q5
>>
>> In other words, Subversion working copies on NFS are, and have always
>> been, a bad idea; not only because of SQLite but also because
>> Subversion's code itself relies on atomic renames, which NFS does not
>> provide.
>>
>> -- Brane
>
> What SVN command (on command line) I should test to get a similar
> result as from ISVNClient.getChangelists?

If "adding/removing a file" is any indication, then "svn add" or "svn
remove" should be comparable.

-- Brane

Re: Slow ISVNClient.getChangelists on Linux/NFS share

2017-10-19 Thread Thomas Singer 

On 18.10.2017 19:56, Branko Čibej wrote:

On 18.10.2017 13:31, Thomas Singer wrote:

Hi,

When performing following steps on my old Linux test machine (with
slow hard disk):

- have a SVN working copy at /home/user/test
- sudo apt install nfs-kernel-server
- add following line to /etc/exports:
   /home/user/test *(rw,sync,no_root_squash)
- start the NFS server:
   sudo systemctl start nfs-kernel-server.service
- mount the NFS share:
   sudo mount localhost:/home/user/test /home/user/test.nfs

and then open /home/user/test.nfs in SmartSVN 9.2 (using SVN 1.9
JavaHL binaries), adding/removing a file is very slow. It boils down
to the call ISVNClient.getChangelists which takes ~8s on the NFS share
(/home/user/test.nfs). First, I thought, it would be caused by the
native-Java overhead calling the call-back ~11,000 times for my
working copy, but when using the working copy directly
(/home/user/test), the method just takes <1s though the ~11,000 times
call-back invocations are still there.

My working copy has no local modifications, no untracked or ignored
files, no changelists.

Is it expected that this method (ISVNClient.getChangelists) is so slow
on a NFS share even if there are no changelists?


I don't know if it's "expected" but I bet that NFS is killing SQLite
performance.

https://www.mail-archive.com/sqlite-users@mailinglists.sqlite.org/msg88989.html

I'm not sure about the reason but the most likely answer, apart from
slow data rate and latency when compared to a local filesystem (which,
in your case on loopback, should not be an issue), is that the OS can't
really use a cache for files on NFS since it has no way to know whether
or not it's valid. With a lot of random-access reads and writes, that
can be a HUGE slowdown, as you found.


Also this:
https://sqlite.org/faq.html#q5

In other words, Subversion working copies on NFS are, and have always
been, a bad idea; not only because of SQLite but also because
Subversion's code itself relies on atomic renames, which NFS does not
provide.

-- Brane


What SVN command (on command line) I should test to get a similar result 
as from ISVNClient.getChangelists? I've tried "git status" and it just 
needs <1s on the NFS working copy.


--
Best regards,
Thomas Singer
=
syntevo GmbH
http://www.syntevo.com
http://www.syntevo.com/blog

Re: Slow ISVNClient.getChangelists on Linux/NFS share

2017-10-18 Thread Branko Čibej 
On 18.10.2017 13:31, Thomas Singer wrote:
> Hi,
>
> When performing following steps on my old Linux test machine (with
> slow hard disk):
>
> - have a SVN working copy at /home/user/test
> - sudo apt install nfs-kernel-server
> - add following line to /etc/exports:
>   /home/user/test *(rw,sync,no_root_squash)
> - start the NFS server:
>   sudo systemctl start nfs-kernel-server.service
> - mount the NFS share:
>   sudo mount localhost:/home/user/test /home/user/test.nfs
>
> and then open /home/user/test.nfs in SmartSVN 9.2 (using SVN 1.9
> JavaHL binaries), adding/removing a file is very slow. It boils down
> to the call ISVNClient.getChangelists which takes ~8s on the NFS share
> (/home/user/test.nfs). First, I thought, it would be caused by the
> native-Java overhead calling the call-back ~11,000 times for my
> working copy, but when using the working copy directly
> (/home/user/test), the method just takes <1s though the ~11,000 times
> call-back invocations are still there.
>
> My working copy has no local modifications, no untracked or ignored
> files, no changelists.
>
> Is it expected that this method (ISVNClient.getChangelists) is so slow
> on a NFS share even if there are no changelists?

I don't know if it's "expected" but I bet that NFS is killing SQLite
performance.

https://www.mail-archive.com/sqlite-users@mailinglists.sqlite.org/msg88989.html

I'm not sure about the reason but the most likely answer, apart from
slow data rate and latency when compared to a local filesystem (which,
in your case on loopback, should not be an issue), is that the OS can't
really use a cache for files on NFS since it has no way to know whether
or not it's valid. With a lot of random-access reads and writes, that
can be a HUGE slowdown, as you found.


Also this:
https://sqlite.org/faq.html#q5

In other words, Subversion working copies on NFS are, and have always
been, a bad idea; not only because of SQLite but also because
Subversion's code itself relies on atomic renames, which NFS does not
provide.

-- Brane

Slow ISVNClient.getChangelists on Linux/NFS share

2017-10-18 Thread Thomas Singer 

Hi,

When performing following steps on my old Linux test machine (with slow 
hard disk):


- have a SVN working copy at /home/user/test
- sudo apt install nfs-kernel-server
- add following line to /etc/exports:
  /home/user/test *(rw,sync,no_root_squash)
- start the NFS server:
  sudo systemctl start nfs-kernel-server.service
- mount the NFS share:
  sudo mount localhost:/home/user/test /home/user/test.nfs

and then open /home/user/test.nfs in SmartSVN 9.2 (using SVN 1.9 JavaHL 
binaries), adding/removing a file is very slow. It boils down to the 
call ISVNClient.getChangelists which takes ~8s on the NFS share 
(/home/user/test.nfs). First, I thought, it would be caused by the 
native-Java overhead calling the call-back ~11,000 times for my working 
copy, but when using the working copy directly (/home/user/test), the 
method just takes <1s though the ~11,000 times call-back invocations are 
still there.


My working copy has no local modifications, no untracked or ignored 
files, no changelists.


Is it expected that this method (ISVNClient.getChangelists) is so slow 
on a NFS share even if there are no changelists?


--
Best regards,
Thomas Singer
=
syntevo GmbH
http://www.syntevo.com
http://www.syntevo.com/blog