Well we at least confirmed it's definitely not a hardware issue. Two
separate systems in separate locations have the exact same problem. The
only difference is one is the 1270 and one is the 1230. Same OS.
On Sat, May 11, 2013 at 1:27 PM, Abdulrahman Abdulkawi <
[email protected]
wrote:
@Fletcher Dunn,
On linux the CPU affinity is auto-detected/decided by the srcds process,
which is normal (same as hlds). I monitored the threads created by the
process: they do not spike and only 3 are consecutively running. The
actual
srcds process is what spikes (the srcds process cpu% does not include the
threads % usage when monitoring with htop, which is how I know only the
srcds spikes, not it's threads).
Without -replay does reduce CPU usage which makes it not go 'overboard'
when the process spikes.
I have tried using strace although there's way too many lines/events to
figure out at which point it spiked. I will however soon try and see if I
can have it log only certain events with timestamp.
I've taken the following excerpts from the strace log (in random order):
--- Lines Beginaccept(12, 0xffe5aeec, [16]) = -1 EAGAIN
(Resource temporarily unavailable)recv(16, 0xffe5bf8f, 1, MSG_PEEK)
=
-1 EAGAIN (Resource temporarily unavailable)
send(42, "\2\1\0\0\0\262\2\0\0", 9, MSG_NOSIGNAL) = 9
futex(0xf73c8598, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xf73c8594, {FUTEX_OP_SET,
0, FUTEX_OP_CMP_GT, 1}) = 1futex(0xf73c857c, FUTEX_WAKE_PRIVATE, 1) =
1futex(0xf73c8558, FUTEX_WAKE_PRIVATE, 1) = 1futex(0xdc16fce4,
FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily
unavailable)futex(0xdc16fce4, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x128048d4, FUTEX_WAIT_PRIVATE, 2, NULL) = 0futex(0x128048d4,
FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x9f76a24, FUTEX_WAIT_PRIVATE, 7, {29, 999961633}) =
0futex(0x9f76a08, FUTEX_WAKE_PRIVATE, 1) = 0--- Lines End
Upon server startup there are a lot of lines for each
model/material/scripts...etc:
open("/home/tf2/badwater/hl2/materials/models/weapons/v_machete/models/weapons/v_machete",
O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No
such
file or directory)
I doubt that explains the spikes but probably has an impact to server
restart/startup?
Does any of that help?
Date: Sat, 11 May 2013 10:54:36 -0400
From: [email protected]
To: [email protected]; [email protected]
Subject: Re: [hlds_linux] Huge sv drops and var spikes in net_graph 4.
Just tested without SourceTV, no difference at all.
On Sat, May 11, 2013 at 10:07 AM, Essay Tew Phaun <[email protected]>
wrote:
So I've got two demos here. The first one, go to about tick 19000 and
wait
to around the point where Fred 2.0 joins. The server drops to 5
frames. I
removed Sourcemod and recorded on another server. I wasn't able to
get
those huge drops but there are big var spikes ~10 and it *seems* to
happen
as clients join.
http://dropcanvas.com/8itl3/1
http://dropcanvas.com/8itl3/2
I can't think of what else it could be. We haven't changed anything
about
the setup from before Steampipe to after.
On Mon, May 6, 2013 at 1:00 PM, Bjorn Wielens <[email protected]>
wrote:
Thanks for investigating, Fletcher.
I can tell you that in our case, we have an 8 core system with no
CPU
affinity assigned (with 12 TF2 servers running comfortably with CPU
to
spare, srcds generally consumes 30% but can get mid 40s when full of
players), Watching top, the load is fairly evenly distributed across
the
individual cores, and monitoring resource usage showed only one
process
spiking in usage at a time.
For what it's worth, I haven't had the issue recur after disabling
replay
on the servers in question. For me, that would mean the next step is
to
re-enable it and see if the problem returns - I'll post back here
when I
get those results (though it's entirely possible that the additional
replay
overhead is the metaphorical straw on the camel's back) and any
additional
information I uncover.
________________________________
From: Fletcher Dunn <[email protected]>
To: Half-Life dedicated Linux server mailing list <
[email protected]>
Cc: "Half-Life dedicated Win32 server mailing list (
[email protected])" <[email protected]>
Sent: Monday, May 6, 2013 1:41:18 PM
Subject: Re: [hlds_linux] Huge sv drops and var spikes in net_graph
4.
I was able to get a trace from a Windows server operator that was
experiencing a similar problem. The problem in that case was that
it
was
spending all of its time compressing snapshots. The problem was
exacerbated by some process affinity settings which were
counterproductive
and putting all the threads on the same core, while other cores were
idle.
Tracking the overall CPU usage can be quite deceiving when
investigating
"lag". Lower CPU usage is not necessarily better! For example, in
this
case if the process affinity were not set and more cores were
allowed
to be
used, then the CPU usage on this 4-core system would have gone above
25%
for a shorter burst, which would have been preferable to it being
pegged at
25% (a single core saturated for a longer period of time).
I think it is a reasonable guess that his problem is similar to the
one
you guys are seeing. (Of course they could be totally unrelated, so
take
all this with a grain of salt.) But assuming that they are similar
problems, there are two things to investigate. First, did something
change
that is doing something subtly different with process affinity that
it
wasn't doing before, or is the process affinity thing totally a
config
issue with one server? (Or if anybody else is tweaking processor
affinities, do you fully understand the implications of it?) When
you are
looking at the CPU usage, is the process pegging a single CPU, or is
it
pegging at some round fraction like 1/4 or 1/8? The second thing to
investigate is, what is happening at the high level that is causing
the
snapshot data to be so big? Do network activity spikes correlates
with the
CPU spikes? If that is the case, then we can use -netspike or
other
tools
that you guys are
probably more familiar with than I am to track down what the entity
data
is that is causing all the busywork.
We have very powerful performance analysis tools we can use on
Windows to
track these problems down. I only wish more windows server
operators
users
would get setup with them! Please help us help you! It's not that
difficult to get set up. Email me directly and I can get you a CD
key.
https://support.steampowered.com/kb_article.php?ref=3059-RTXB-2672
-----Original Message-----
From: [email protected] [mailto:
[email protected]] On Behalf Of Essay Tew
Phaun
Sent: Saturday, May 04, 2013 11:54 AM
To: Half-Life dedicated Linux server mailing list
Subject: Re: [hlds_linux] Huge sv drops and var spikes in net_graph
4.
Hmm. We didn't have this problem before Steampipe though. We just
use
the
STV with a couple of slots open on it so that an admin can connect
and
watch for reported players.
On Sat, May 4, 2013 at 2:44 PM, ics <[email protected]> wrote:
STV is known to cause issues and hogs a lot of memory. That's why
rarely no one uses it on public servers.
-ics
Essay Tew Phaun kirjoitti:
Interesting. We're not hosting a replay, but we do have STV.
On Sat, May 4, 2013 at 12:06 PM, Bjorn Wielens <
[email protected]>
wrote:
Sounds like the same problem I'm having (see last few posts in
'CPU
Increase' thread).
______________________________**__
From: Michael Johansen <[email protected]>
To: Half-Life dedicated Linux server mailing list <
[email protected].**com
<[email protected]>>
Sent: Saturday, May 4, 2013 12:43:59 PM
Subject: Re: [hlds_linux] Huge sv drops and var spikes in
net_graph 4.
We've noticed this as well, no complaints though as the players
does
not seem to notice it very much.
Date: Sat, 4 May 2013 11:40:54 -0400
From: [email protected]
To:
[email protected].
**com<[email protected]
om>
Subject: [hlds_linux] Huge sv drops and var spikes in net_graph
4.
We never had this issue prior to Steampipe, but we're dealing
with
these very random and rare spikes of lag where the server
framerate
will
suddenly
drop from 66 to below 20 and var will spike to 20+. We're
hosting 6
24
slot
servers on the following:
E3-1230v2
16GB
CentOS 6.4 64bit
I was able to capture a screenshot of one of the drops but
they're
very hard to catch since they appear very very briefly.
http://i.imgur.com/wxp8P3N.png
server.cfg
sv_minrate 35000
sv_maxrate 200000
sv_maxupdaterate 67
sv_minupdaterate 66
sv_mincmdrate 66
sv_maxcmdrate 67
sv_client_cmdrate_difference 100
Thanks
______________________________**_________________
To unsubscribe, edit your list preferences, or view the list
archives,
please visit:
https://list.valvesoftware.
**com/cgi-bin/mailman/listinfo/**hlds_li
nux<
https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_li
nux>
______________________________**_________________
To unsubscribe, edit your list preferences, or view the list
archives, please visit:
https://list.valvesoftware.
**com/cgi-bin/mailman/listinfo/**hlds_lin
ux<
https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linu
x> ______________________________**_________________
To unsubscribe, edit your list preferences, or view the list
archives, please visit:
https://list.valvesoftware.
**com/cgi-bin/mailman/listinfo/**hlds_lin
ux<
https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linu
x>
______________________________**_________________
To unsubscribe, edit your list preferences, or view the list
archives, please visit:
https://list.valvesoftware.
**com/cgi-bin/mailman/listinfo/**hlds_linu
x<
https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux>
______________________________**_________________
To unsubscribe, edit your list preferences, or view the list
archives,
please visit:
https://list.valvesoftware.
**com/cgi-bin/mailman/listinfo/**hlds_linux
<
https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux
_______________________________________________
To unsubscribe, edit your list preferences, or view the list
archives,
please visit:
https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux
_______________________________________________
To unsubscribe, edit your list preferences, or view the list
archives,
please visit:
https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux
_______________________________________________
To unsubscribe, edit your list preferences, or view the list
archives,
please visit:
https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux
_______________________________________________
To unsubscribe, edit your list preferences, or view the list archives,
please visit:
https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux
_______________________________________________
To unsubscribe, edit your list preferences, or view the list archives,
please visit:
https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux
_______________________________________________
To unsubscribe, edit your list preferences, or view the list archives,
please visit:
https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux
