Hello,
I'm looking for guidance from Subversion administrators regarding a
performance issue in our production SVN environment.
We have been experiencing significant performance degradation across
multiple Unreal Engine projects. SVN updates, commits, checkouts and even
opening the TortoiseSVN Repository Browser have become slow.
I'm trying to identify whether the issue relates to Subversion, Apache
HTTPD, repository storage, CIFS/SMB or the underlying infrastructure.
1. Our environment
-
Operating System: Ubuntu 20.04.5 LTS
-
Kernel: Linux 5.15
-
Subversion access: Apache HTTPD over HTTPS
-
SVN port: 18080
-
Repository format: FSFS
-
Repository storage: CIFS/SMB network share
-
CIFS mount: //moon/repo mounted at /mnt/repo
-
SMB version: 3.1.1
The repositories are stored under:
/mnt/repo/data/repositories/
2. Performance symptoms
We are experiencing the following:
-
SVN updates and commits have become significantly slower across multiple
projects.
-
TortoiseSVN Repository Browser took approximately 9 minutes to
initialize.
-
A 14.7 GB checkout took approximately 13 minutes.
-
A 2.7 GB commit took approximately 4 minutes.
-
A previous 15 GB commit completed in approximately 5–7 minutes.
-
Unreal Engine cache reimports from network storage have also become
slower.
-
Unreal rendering performance on the farm has reportedly degraded as well.
Artists use local Unreal working copies, while central SVN repositories are
stored on network storage.
3. Diagnostics performed
During a slow 14.7 GB checkout, we observed:
-
Linux I/O pressure reaching approximately 23%.
-
Multiple Apache httpd processes in uninterruptible D state.
-
Kernel wait channels included:
-
cifs_call_async
-
compound_send_recv
-
cifs_readdata
-
wait_on_page_bit_common
The SVN repository storage is mounted through CIFS/SMB.
TCP connectivity to the SVN server on port 18080 is successful.
4. Questions
1.
Can FSFS repositories hosted on CIFS/SMB network storage experience
significant performance degradation due to storage latency or locking
behaviour?
2.
Are the observed Apache D-state processes and CIFS-related wait channels
consistent with delays accessing repository storage?
3.
What is the recommended method to distinguish between Subversion
processing delays, Apache HTTPD delays, CIFS/SMB latency and physical
storage bottlenecks?
4.
Are there any known performance or reliability concerns with hosting
production FSFS repositories on CIFS/SMB shares?
5.
What read-only diagnostic commands or measurements would you recommend
before making any configuration changes?
6.
Are there any particular Apache HTTPD, mod_dav_svn or Subversion
settings that we should investigate?
We are not looking to make random configuration changes or rebuild
repositories without understanding the root cause.
Any guidance on systematically identifying the bottleneck would be greatly
appreciated.
*We have also involved our internal IT and production teams in parallel to
investigate the infrastructure and application-side components. *
Thank you.
--
____________________________________________________________________________
Disclaimer: - This email and any files transmitted with it are strictly
confidential and intended solely for the use of the individual or entity to
whom they are addressed. If you have received this email in error, please
notify the system manager. This message contains confidential information
and is intended only for the individual named. If you are not the named
addressee, you should not disseminate, distribute or copy this email.
Please notify the sender immediately by email if you have received this
email by mistake and delete this email from your system. If you are not the
intended recipient, you are notified that disclosing, copying, distributing
or taking any action in reliance on the contents of this information is
strictly prohibited.