Hi all,
Something that has been plaguing our 64 player server for a while is
seemingly random lag spikes, but after some detailed investigation into the
problem a co-admin and I have managed to find the reason.
We tried everything to find out why this was happening, including buying new
hardware and changing bandwidth providers. After all that, we are 100%
convinced it was srcds and I finally have the proof.
In brief, more detailed report follows:
- The way srcds calculates the file name for storing uploaded sprays causes
lag.
In order to get past the spam in the console[1], we reduced the server to 32
players. We then were able to observe lag spikes in the game, and by
coincidence, monitoring the server console we saw this error:
CreateFragmentsFromFile: 'downloads/ee9becfc.dat' doesn't exist.
[message is repeated several times]
In the time this message is printed, a lag spike occurs in the game. In the
past, it has been advised to "ignore" this message - while this is safe to
do so, when you have a popular server (as we do) it happens far too
regularly.
And the reason why this is bad? We looked into more detail by using strace
on srcds to see what happens, here is the result just as the
CreateFragmentsFromFile message appears (note: we run Debian Linux):
open("/cs/css/cstrike/downloads",
O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 13
fstat64(13, {st_mode=S_IFDIR|0755, st_size=48, ...}) = 0
fcntl64(13, F_SETFD, FD_CLOEXEC) = 0
getdents64(13, /* 2 entries */, 131072) = 48
getdents64(13, /* 0 entries */, 131072) = 0
close(13) = 0
The downloads directory is where srcds stores uploaded sprays from clients.
To put this into perspective, we have, on average, 3000 clients connecting
to our server a day, and if the majority of clients have a custom spray,
this will ultimately get uploaded to the srcds server.
For example:
banana:/cs/css/cstrike# ls -l downloads | wc -l
52235
52000 files over a three month period!
Linux programmers on the list may notice that using getdents(2) on that many
files, not matter how fast the bus, disk or whatever, *will* take a short
moment - i.e. the exact length of one, random lag spike.
So to conclude, my questions are:
- Why doesn't the server clean up the downloads folder?
- How does the server generate the filenames used to store uploaded sprays?
- Can this be fixed?
Our solution at the moment is to simply delete all content in the downloads
folder and/or set sv_allowupload to zero, but you get even more
CreateFragmentsFromFile errors in the console (but no lag!).
We used a lot of time and effort to get to the bottom of this. I'd
appreciate if Valve would at least acknowledge this.
[1] - Remove the console message that says "Free sound list is full!" - it
is absolutely meaningless.
--
Bart King -- http://www.bart666.com
+44 781 219 5654 -- PGP: 0xC9C3EB8B
_______________________________________________
To unsubscribe, edit your list preferences, or view the list archives, please
visit:
http://list.valvesoftware.com/mailman/listinfo/hlds_linux
