Re: EXTERNAL: Re: svn version 1.10 lack of robustness in presence of flaky network

[Johan Corveleyn]

On Wed, Apr 24, 2019 at 12:41 PM Marlow, Andrew
 wrote:
> -Original Message-
> From: Johan Corveleyn 
...
> > Where are you checking out your software? Is it on a local disk?
> Yes.

Okay, then there must be something strange with this local disk; or a
second process is interfering with 'svn' trying to lock its database.
In any case, no network is involved then, so "flaky network" can't be
the cause of your problem. The error you received specifically points
to a problem while locking the working copy database:

[[[
svn: E200033: Another process is blocking the working copy database,
or the underlying filesystem does not support file locking; if the
working copy is on a network filesystem, make sure file locking has
been enabled on the file server
svn: E200033: sqlite[S5]: database is locked
svn: E200042: Additional errors:
svn: E200033: sqlite[S5]: database is locked
svn: E200030: sqlite[S1]: cannot start a transaction within a transaction
svn: E200030: sqlite[S1]: cannot start a transaction within a transaction
]]]

To be clear: the "working copy database" is an sqlite database in the
"wc.db" file inside the .svn directory at the root of your working
copy.

I'm not sure what else to suggest, but "something" is preventing 'svn'
from locking its database; and since you're using a local disk, it
seems to me that "network" is not involved in this locking process.

BTW, what version of SVN are you using?

-- 
Johan

RE: EXTERNAL: Re: svn version 1.10 lack of robustness in presence of flaky network

[Marlow, Andrew]

Reply below:

-Original Message-
From: Johan Corveleyn 
Sent: 24 April 2019 11:30
To: Marlow, Andrew 
Cc: users@subversion.apache.org
Subject: Re: EXTERNAL: Re: svn version 1.10 lack of robustness in presence of 
flaky network

> >Regarding the comment that was made, I don't know exactly how the svn repo 
> >is hosted in the corporate environment I am in. It is accessed via an http 
> >URL (not https, I know). The underlying filesystem is Windows because every 
> >now and then we get aggro due to the case preserving behaviour of the 
> >Windows filesystem. But is it on a network share? Not sure, but I don't 
> >think so. IMO that would be an extremely bad setup.

>>The network share Stefan and I are referring to is not on the server side 
>>(which is accessed via https, that's fine; and
>> we're not concerned with how that server is set up). We're talking about the 
>> client-side location where you're putting your working copy.
[snip]
Yes, thanks for clearing that point up.

> Where are you checking out your software? Is it on a local disk?
Yes.

>Or on a CIFS mounted filesystem (Windows)
No.

> or on an NFS-mounted filesystem, or SMB, ... if you're on Solaris?
No.

> If in step 2 you're indeed using some network share as the location for your 
> working copy, that could be problematic.
Indeed, especially with the network I am using.

> As Stefan said, this is discouraged, and you should try to put working copies 
> on local disks.
Good job I'm not, then. 😊


--
Johan
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. FIS is a trading name of the following 
companies: Fidelity Information Services Ltd (registered in England 
No.02225203), FIS Payments (UK) Ltd (No.04215488), FIS Asiapacrim Holdings Ltd 
(No.06707320), Certegy Card Services Ltd (No.03517639) and Efunds International 
Ltd (No.01930117), all with their registered office at Floor 1, 51/53 Hagley 
Road, Birmingham B16 8TU, United Kingdom; and FIS Payments (Ireland) Ltd 
(registered in Ireland No.126879), with its registered office at 25/28 North 
Wall Quay, Dublin 1, D01 H104, Ireland. FIS Payments (UK) Ltd is authorised and 
regulated by the Financial Conduct Authority; some services are covered by the 
Financial Ombudsman Service (in the UK). Calls to and from the companies may be 
recorded for quality purposes. All of the above companies are part of FIS 
(Fidelity National Information Services Inc.).

Re: EXTERNAL: Re: svn version 1.10 lack of robustness in presence of flaky network

[Johan Corveleyn]

On Wed, Apr 24, 2019 at 11:51 AM Marlow, Andrew
 wrote:
>
> Sorry everyone, I mis-spoke below. While looking at this issue I was also 
> filing a bug report on the maven jar signing plugin, which also has a problem 
> when the network is flaky. That is the thing that was calling out to the 
> timestamp server, not svn. I got mixed up, sorry.
>
> Regarding the comment that was made, I don't know exactly how the svn repo is 
> hosted in the corporate environment I am in. It is accessed via an http URL 
> (not https, I know). The underlying filesystem is Windows because every now 
> and then we get aggro due to the case preserving behaviour of the Windows 
> filesystem. But is it on a network share? Not sure, but I don't think so. IMO 
> that would be an extremely bad setup.
>

The network share Stefan and I are referring to is not on the server
side (which is accessed via https, that's fine; and we're not
concerned with how that server is set up). We're talking about the
client-side location where you're putting your working copy. I'm not
sure you fully understand the difference between "repository"
(server-side, single source of truth of the entire history) and
"working copy" (client-side checkout, where you perform local work to
perform commits from, and "update" to get other's changes). See [1]
for some more explanation.

Where are you checking out your software? Is it on a local disk? Or on
a CIFS mounted filesystem (Windows) or on an NFS-mounted filesystem,
or SMB, ... if you're on Solaris?

For example, are you doing the following?

1) Log into your Windows pc.

2) cd \\some\network\share (or cd M:\subdir; where M: is a mapped
network drive)  <- this is where you're putting your working copy

3) svn checkout https://svnserver.example.com/path/to/repo  <- this is
the URL to the repository, which can be hosted on Windows, Solaris,
Linux, ... whatever


If in step 2 you're indeed using some network share as the location
for your working copy, that could be problematic. As Stefan said, this
is discouraged, and you should try to put working copies on local
disks. And if you really must work with networked filesystems for your
working copy, you should look at the configuration options in the
[working-copy] section of the client-side configuration file "config"
(see [2] for some general information about the client-side config
files). Maybe some tweaks there can help.

[1] http://svnbook.red-bean.com/nightly/en/svn.basic.version-control-basics.html
[2] http://svnbook.red-bean.com/nightly/en/svn.advanced.confarea.html

-- 
Johan

RE: EXTERNAL: Re: svn version 1.10 lack of robustness in presence of flaky network

[Marlow, Andrew]

Sorry everyone, I mis-spoke below. While looking at this issue I was also 
filing a bug report on the maven jar signing plugin, which also has a problem 
when the network is flaky. That is the thing that was calling out to the 
timestamp server, not svn. I got mixed up, sorry.

Regarding the comment that was made, I don't know exactly how the svn repo is 
hosted in the corporate environment I am in. It is accessed via an http URL 
(not https, I know). The underlying filesystem is Windows because every now and 
then we get aggro due to the case preserving behaviour of the Windows 
filesystem. But is it on a network share? Not sure, but I don't think so. IMO 
that would be an extremely bad setup.

-Original Message-
From: Marlow, Andrew 
Sent: 24 April 2019 09:52
To: users@subversion.apache.org
Subject: RE: EXTERNAL: Re: svn version 1.10 lack of robustness in presence of 
flaky network

Reply below:

-Original Message-
From: Stefan Sperling 
Sent: 24 April 2019 07:54
To: Marlow, Andrew 
Cc: Johan Corveleyn ; users@subversion.apache.org
Subject: EXTERNAL: Re: svn version 1.10 lack of robustness in presence of flaky 
network

On Wed, Apr 24, 2019 at 12:55:47AM +0200, Johan Corveleyn wrote:
> On Mon, Apr 22, 2019 at 9:22 AM Marlow, Andrew
>  wrote:
> > Hello everyone,
> > I got this error below during an svn co command. It left my workspace in a 
> > bad state from which I had to do svn cleanup before trying again (the retry 
> > worked):
[snip]
> > I think this happens when the network is flaky. This error happened on 
> > windows but I have also seen it happen on solaris 10.  Has anyone else seen 
> > this? If it is due to network flakiness then perhaps svn should retry to 
> > work around this transparently, and thus be more robust? Perhaps it could 
> > retry up to 3 times with a sleep a 1 second between retries?
>
> Is your working copy on a network filesystem (CIFS, NFS, ...)?
[snip]
While working copies on networks filesystems should generally work, such use is 
strongly discouraged.
[snip]
No. There is no network filesystem (CIFS) here. What about when I am using 
solaris? That's proof that it cannot be due to CIFS. The internet is being used 
because that it is where the timestamp server is. When the code tries to call 
out to that timeserver it has to travel via the corporate network, including 
navigating via the internet proxy. This network call fails randomly due a flaky 
network.

The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. FIS is a trading name of the following 
companies: Fidelity Information Services Ltd (registered in England 
No.02225203), FIS Payments (UK) Ltd (No.04215488), FIS Asiapacrim Holdings Ltd 
(No.06707320), Certegy Card Services Ltd (No.03517639) and Efunds International 
Ltd (No.01930117), all with their registered office at Floor 1, 51/53 Hagley 
Road, Birmingham B16 8TU, United Kingdom; and FIS Payments (Ireland) Ltd 
(registered in Ireland No.126879), with its registered office at 25/28 North 
Wall Quay, Dublin 1, D01 H104, Ireland. FIS Payments (UK) Ltd is authorised and 
regulated by the Financial Conduct Authority; some services are covered by the 
Financial Ombudsman Service (in the UK). Calls to and from the companies may be 
recorded for quality purposes. All of the above companies are part of FIS 
(Fidelity National Information Services Inc.).
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. FIS is a trading name of the following 
companies: Fidelity Information Services Ltd (registered in England 
No.02225203), FIS Payments (UK) Ltd (No.04215488), FIS Asiapacrim Holdings Ltd 
(No.06707320), Certegy Card Services Ltd (No.03517639) and Efunds International 
Ltd (No.01930117), all with their registered office at Floor 1, 51/53 Hagley 
Road, Birmingham B16 8TU, United Kingdom; and FIS Payments (Ireland) Ltd 
(registered in Ireland No.126879), with its registered office at 25/28 North 
Wall Quay, Dublin 1, D01 H104, Ireland. FIS Payments (UK) Ltd is authorised and 
regulated by the Financial Conduct Authority; some services are covered by the 
Financial Ombudsman Service (in the UK). Calls to and from the companies may be 
rec

RE: EXTERNAL: Re: svn version 1.10 lack of robustness in presence of flaky network

[Marlow, Andrew]

Reply below:

-Original Message-
From: Stefan Sperling 
Sent: 24 April 2019 07:54
To: Marlow, Andrew 
Cc: Johan Corveleyn ; users@subversion.apache.org
Subject: EXTERNAL: Re: svn version 1.10 lack of robustness in presence of flaky 
network

On Wed, Apr 24, 2019 at 12:55:47AM +0200, Johan Corveleyn wrote:
> On Mon, Apr 22, 2019 at 9:22 AM Marlow, Andrew
>  wrote:
> > Hello everyone,
> > I got this error below during an svn co command. It left my workspace in a 
> > bad state from which I had to do svn cleanup before trying again (the retry 
> > worked):
[snip]
> > I think this happens when the network is flaky. This error happened on 
> > windows but I have also seen it happen on solaris 10.  Has anyone else seen 
> > this? If it is due to network flakiness then perhaps svn should retry to 
> > work around this transparently, and thus be more robust? Perhaps it could 
> > retry up to 3 times with a sleep a 1 second between retries?
>
> Is your working copy on a network filesystem (CIFS, NFS, ...)?
[snip]
While working copies on networks filesystems should generally work, such use is 
strongly discouraged.
[snip]
No. There is no network filesystem (CIFS) here. What about when I am using 
solaris? That's proof that it cannot be due to CIFS. The internet is being used 
because that it is where the timestamp server is. When the code tries to call 
out to that timeserver it has to travel via the corporate network, including 
navigating via the internet proxy. This network call fails randomly due a flaky 
network.

The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. FIS is a trading name of the following 
companies: Fidelity Information Services Ltd (registered in England 
No.02225203), FIS Payments (UK) Ltd (No.04215488), FIS Asiapacrim Holdings Ltd 
(No.06707320), Certegy Card Services Ltd (No.03517639) and Efunds International 
Ltd (No.01930117), all with their registered office at Floor 1, 51/53 Hagley 
Road, Birmingham B16 8TU, United Kingdom; and FIS Payments (Ireland) Ltd 
(registered in Ireland No.126879), with its registered office at 25/28 North 
Wall Quay, Dublin 1, D01 H104, Ireland. FIS Payments (UK) Ltd is authorised and 
regulated by the Financial Conduct Authority; some services are covered by the 
Financial Ombudsman Service (in the UK). Calls to and from the companies may be 
recorded for quality purposes. All of the above companies are part of FIS 
(Fidelity National Information Services Inc.).

Re: svn version 1.10 lack of robustness in presence of flaky network

[Stefan Sperling]

On Wed, Apr 24, 2019 at 12:55:47AM +0200, Johan Corveleyn wrote:
> On Mon, Apr 22, 2019 at 9:22 AM Marlow, Andrew
>  wrote:
> > Hello everyone,
> >
> > I got this error below during an svn co command. It left my workspace in a 
> > bad state from which I had to do svn cleanup before trying again (the retry 
> > worked):
> >
> > svn: E200033: Another process is blocking the working copy database, or the 
> > underlying filesystem does not support file locking; if the working copy is 
> > on a network filesystem, make sure file locking has been enabled on the 
> > file server
> > svn: E200033: sqlite[S5]: database is locked
> > svn: E200042: Additional errors:
> > svn: E200033: sqlite[S5]: database is locked
> > svn: E200030: sqlite[S1]: cannot start a transaction within a transaction
> > svn: E200030: sqlite[S1]: cannot start a transaction within a transaction
> >
> > I think this happens when the network is flaky. This error happened on 
> > windows but I have also seen it happen on solaris 10.  Has anyone else seen 
> > this? If it is due to network flakiness then perhaps svn should retry to 
> > work around this transparently, and thus be more robust? Perhaps it could 
> > retry up to 3 times with a sleep a 1 second between retries?
> >
> 
> Is your working copy on a network filesystem (CIFS, NFS, ...)? And are
> you talking about a flaky network between your machine and its
> networked filesystem? If so, I think there is not much we can do about
> it ... the filesystem you're checking out to should reliable. There is
> already a retry loop in some places for putting checked out files into
> place, to work around locks from antivirus software etc, ... (but the
> sqlite database itself should be reliably available).

While working copies on networks filesystems should generally work,
such use is strongly discouraged.

So far, all reasons I've heard for putting working copies on network
drives turned out to be backed by bad or misinformed decisions about
the development process or allocation of hardware resources.
Moving to local disks improved not just the SVN user experience but
also repaired a broken process in such cases.

So put working copies on a local disk, preferrably SSDs. Working copies
should be considered temporary and disposable copies of your data.
The repository on the server is the important and permanent database
which must be protected and backed up, not the working copy.

If your working copy is too large for a modern SSD (really?),
consider sparse working copies
(http://svnbook.red-bean.com/nightly/en/svn.advanced.sparsedirs.html)
and/or reorganize your project such that parts of it can be checked
out and built in isolation.

That said, if you really must use a network filesystem, you should
look into tweaking the following options in Subversion's 'config' file:

### Section for configuring working copies.
[working-copy]
### Set to a list of the names of specific clients that should use
### exclusive SQLite locking of working copies.  This increases the
### performance of the client but prevents concurrent access by
### other clients.  Third-party clients may also support this
### option.
### Possible values:
###   svn(the command line client)
# exclusive-locking-clients =
### Set to true to enable exclusive SQLite locking of working
### copies by all clients using the 1.8 APIs.  Enabling this may
### cause some clients to fail to work properly. This does not have
### to be set for exclusive-locking-clients to work.
# exclusive-locking = false
### Set the SQLite busy timeout in milliseconds: the maximum time
### the client waits to get access to the SQLite database before
### returning an error.  The default is 1, i.e. 10 seconds.
### Longer values may be useful when exclusive locking is enabled.
# busy-timeout = 1

Re: svn version 1.10 lack of robustness in presence of flaky network

[Johan Corveleyn]

On Mon, Apr 22, 2019 at 9:22 AM Marlow, Andrew
 wrote:
> Hello everyone,
>
> I got this error below during an svn co command. It left my workspace in a 
> bad state from which I had to do svn cleanup before trying again (the retry 
> worked):
>
> svn: E200033: Another process is blocking the working copy database, or the 
> underlying filesystem does not support file locking; if the working copy is 
> on a network filesystem, make sure file locking has been enabled on the file 
> server
> svn: E200033: sqlite[S5]: database is locked
> svn: E200042: Additional errors:
> svn: E200033: sqlite[S5]: database is locked
> svn: E200030: sqlite[S1]: cannot start a transaction within a transaction
> svn: E200030: sqlite[S1]: cannot start a transaction within a transaction
>
> I think this happens when the network is flaky. This error happened on 
> windows but I have also seen it happen on solaris 10.  Has anyone else seen 
> this? If it is due to network flakiness then perhaps svn should retry to work 
> around this transparently, and thus be more robust? Perhaps it could retry up 
> to 3 times with a sleep a 1 second between retries?
>

Is your working copy on a network filesystem (CIFS, NFS, ...)? And are
you talking about a flaky network between your machine and its
networked filesystem? If so, I think there is not much we can do about
it ... the filesystem you're checking out to should reliable. There is
already a retry loop in some places for putting checked out files into
place, to work around locks from antivirus software etc, ... (but the
sqlite database itself should be reliably available).

Or are you talking about flakiness of the network between your svn
client and the svn server? In that case, I would not expect this kind
of error.

-- 
Johan

svn version 1.10 lack of robustness in presence of flaky network

[Marlow, Andrew]

Hello everyone,

I got this error below during an svn co command. It left my workspace in a bad 
state from which I had to do svn cleanup before trying again (the retry worked):

svn: E200033: Another process is blocking the working copy database, or the 
underlying filesystem does not support file locking; if the working copy is on 
a network filesystem, make sure file locking has been enabled on the file server
svn: E200033: sqlite[S5]: database is locked
svn: E200042: Additional errors:
svn: E200033: sqlite[S5]: database is locked
svn: E200030: sqlite[S1]: cannot start a transaction within a transaction
svn: E200030: sqlite[S1]: cannot start a transaction within a transaction

I think this happens when the network is flaky. This error happened on windows 
but I have also seen it happen on solaris 10.  Has anyone else seen this? If it 
is due to network flakiness then perhaps svn should retry to work around this 
transparently, and thus be more robust? Perhaps it could retry up to 3 times 
with a sleep a 1 second between retries?

Andrew Marlow
Consultant developer, Apex
38th Floor, 25 Canada Square,
Canary Wharf, London E14 5LQ
T:  020-8081-2367 / 07966-451-521
E: andrew.mar...@fisglobal.com
FIS | Empowering the Financial World [cid:image001.png@01D112FA.C4A77D90] 
 [cid:image002.png@01D112FA.C4A77D90] 
 [cid:image003.png@01D112FA.C4A77D90] 

FIS Apex (UK) Limited * Registered in England and Wales No. 3573008 * 
Registered Office: 38th floor, 25 Canada Square, London, E14 5LQ, United Kingdom
[50_3]

The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. FIS is a trading name of the following 
companies: Fidelity Information Services Ltd (registered in England 
No.02225203), FIS Payments (UK) Ltd (No.04215488), FIS Asiapacrim Holdings Ltd 
(No.06707320), Certegy Card Services Ltd (No.03517639) and Efunds International 
Ltd (No.01930117), all with their registered office at Floor 1, 51/53 Hagley 
Road, Birmingham B16 8TU, United Kingdom; and FIS Payments (Ireland) Ltd 
(registered in Ireland No.126879), with its registered office at 25/28 North 
Wall Quay, Dublin 1, D01 H104, Ireland. FIS Payments (UK) Ltd is authorised and 
regulated by the Financial Conduct Authority; some services are covered by the 
Financial Ombudsman Service (in the UK). Calls to and from the companies may be 
recorded for quality purposes. All of the above companies are part of FIS 
(Fidelity National Information Services Inc.).