@Fletcher Dunn,
On linux the CPU affinity is auto-detected/decided by the srcds process, which
is normal (same as hlds). I monitored the threads created by the process: they
do not spike and only 3 are consecutively running. The actual srcds process is
what spikes (the srcds process cpu% does not include the threads % usage when
monitoring with htop, which is how I know only the srcds spikes, not it's
threads).
Without -replay does reduce CPU usage which makes it not go 'overboard' when
the process spikes.
I have tried using strace although there's way too many lines/events to figure
out at which point it spiked. I will however soon try and see if I can have it
log only certain events with timestamp.
I've taken the following excerpts from the strace log (in random order):
--- Lines Beginaccept(12, 0xffe5aeec, [16]) = -1 EAGAIN (Resource
temporarily unavailable)recv(16, 0xffe5bf8f, 1, MSG_PEEK) = -1 EAGAIN
(Resource temporarily unavailable)
send(42, "\2\1\0\0\0\262\2\0\0", 9, MSG_NOSIGNAL) = 9
futex(0xf73c8598, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xf73c8594, {FUTEX_OP_SET, 0,
FUTEX_OP_CMP_GT, 1}) = 1futex(0xf73c857c, FUTEX_WAKE_PRIVATE, 1) =
1futex(0xf73c8558, FUTEX_WAKE_PRIVATE, 1) = 1futex(0xdc16fce4,
FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily
unavailable)futex(0xdc16fce4, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x128048d4, FUTEX_WAIT_PRIVATE, 2, NULL) = 0futex(0x128048d4,
FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x9f76a24, FUTEX_WAIT_PRIVATE, 7, {29, 999961633}) = 0futex(0x9f76a08,
FUTEX_WAKE_PRIVATE, 1) = 0--- Lines End
Upon server startup there are a lot of lines for each
model/material/scripts...etc:
open("/home/tf2/badwater/hl2/materials/models/weapons/v_machete/models/weapons/v_machete",
O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such
file or directory)
I doubt that explains the spikes but probably has an impact to server
restart/startup?
Does any of that help?
> Date: Sat, 11 May 2013 10:54:36 -0400
> From: [email protected]
> To: [email protected]; [email protected]
> Subject: Re: [hlds_linux] Huge sv drops and var spikes in net_graph 4.
>
> Just tested without SourceTV, no difference at all.
>
>
>
>
> On Sat, May 11, 2013 at 10:07 AM, Essay Tew Phaun <[email protected]> wrote:
>
> > So I've got two demos here. The first one, go to about tick 19000 and wait
> > to around the point where Fred 2.0 joins. The server drops to 5 frames. I
> > removed Sourcemod and recorded on another server. I wasn't able to get
> > those huge drops but there are big var spikes ~10 and it *seems* to happen
> > as clients join.
> >
> > http://dropcanvas.com/8itl3/1
> > http://dropcanvas.com/8itl3/2
> >
> > I can't think of what else it could be. We haven't changed anything about
> > the setup from before Steampipe to after.
> >
> >
> > On Mon, May 6, 2013 at 1:00 PM, Bjorn Wielens <[email protected]> wrote:
> >
> >>
> >>
> >> Thanks for investigating, Fletcher.
> >>
> >> I can tell you that in our case, we have an 8 core system with no CPU
> >> affinity assigned (with 12 TF2 servers running comfortably with CPU to
> >> spare, srcds generally consumes 30% but can get mid 40s when full of
> >> players), Watching top, the load is fairly evenly distributed across the
> >> individual cores, and monitoring resource usage showed only one process
> >> spiking in usage at a time.
> >>
> >>
> >> For what it's worth, I haven't had the issue recur after disabling replay
> >> on the servers in question. For me, that would mean the next step is to
> >> re-enable it and see if the problem returns - I'll post back here when I
> >> get those results (though it's entirely possible that the additional replay
> >> overhead is the metaphorical straw on the camel's back) and any additional
> >> information I uncover.
> >>
> >>
> >>
> >>
> >>
> >> ________________________________
> >> From: Fletcher Dunn <[email protected]>
> >> To: Half-Life dedicated Linux server mailing list <
> >> [email protected]>
> >> Cc: "Half-Life dedicated Win32 server mailing list (
> >> [email protected])" <[email protected]>
> >> Sent: Monday, May 6, 2013 1:41:18 PM
> >> Subject: Re: [hlds_linux] Huge sv drops and var spikes in net_graph 4.
> >>
> >>
> >> I was able to get a trace from a Windows server operator that was
> >> experiencing a similar problem. The problem in that case was that it was
> >> spending all of its time compressing snapshots. The problem was
> >> exacerbated by some process affinity settings which were counterproductive
> >> and putting all the threads on the same core, while other cores were idle.
> >>
> >> Tracking the overall CPU usage can be quite deceiving when investigating
> >> "lag". Lower CPU usage is not necessarily better! For example, in this
> >> case if the process affinity were not set and more cores were allowed to be
> >> used, then the CPU usage on this 4-core system would have gone above 25%
> >> for a shorter burst, which would have been preferable to it being pegged at
> >> 25% (a single core saturated for a longer period of time).
> >>
> >> I think it is a reasonable guess that his problem is similar to the one
> >> you guys are seeing. (Of course they could be totally unrelated, so take
> >> all this with a grain of salt.) But assuming that they are similar
> >> problems, there are two things to investigate. First, did something change
> >> that is doing something subtly different with process affinity that it
> >> wasn't doing before, or is the process affinity thing totally a config
> >> issue with one server? (Or if anybody else is tweaking processor
> >> affinities, do you fully understand the implications of it?) When you are
> >> looking at the CPU usage, is the process pegging a single CPU, or is it
> >> pegging at some round fraction like 1/4 or 1/8? The second thing to
> >> investigate is, what is happening at the high level that is causing the
> >> snapshot data to be so big? Do network activity spikes correlates with the
> >> CPU spikes? If that is the case, then we can use -netspike or other tools
> >> that you guys are
> >> probably more familiar with than I am to track down what the entity data
> >> is that is causing all the busywork.
> >>
> >> We have very powerful performance analysis tools we can use on Windows to
> >> track these problems down. I only wish more windows server operators users
> >> would get setup with them! Please help us help you! It's not that
> >> difficult to get set up. Email me directly and I can get you a CD key.
> >> https://support.steampowered.com/kb_article.php?ref=3059-RTXB-2672
> >>
> >> -----Original Message-----
> >> From: [email protected] [mailto:
> >> [email protected]] On Behalf Of Essay Tew Phaun
> >> Sent: Saturday, May 04, 2013 11:54 AM
> >> To: Half-Life dedicated Linux server mailing list
> >> Subject: Re: [hlds_linux] Huge sv drops and var spikes in net_graph 4.
> >>
> >> Hmm. We didn't have this problem before Steampipe though. We just use the
> >> STV with a couple of slots open on it so that an admin can connect and
> >> watch for reported players.
> >>
> >>
> >> On Sat, May 4, 2013 at 2:44 PM, ics <[email protected]> wrote:
> >>
> >> > STV is known to cause issues and hogs a lot of memory. That's why
> >> > rarely no one uses it on public servers.
> >> >
> >> > -ics
> >> >
> >> > Essay Tew Phaun kirjoitti:
> >> >
> >> > Interesting. We're not hosting a replay, but we do have STV.
> >> >>
> >> >>
> >> >> On Sat, May 4, 2013 at 12:06 PM, Bjorn Wielens <[email protected]>
> >> wrote:
> >> >>
> >> >> Sounds like the same problem I'm having (see last few posts in 'CPU
> >> >>> Increase' thread).
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>> ______________________________**__
> >> >>> From: Michael Johansen <[email protected]>
> >> >>> To: Half-Life dedicated Linux server mailing list <
> >> >>> [email protected].**com
> >> >>> <[email protected]>>
> >> >>> Sent: Saturday, May 4, 2013 12:43:59 PM
> >> >>> Subject: Re: [hlds_linux] Huge sv drops and var spikes in net_graph 4.
> >> >>>
> >> >>>
> >> >>> We've noticed this as well, no complaints though as the players does
> >> >>> not seem to notice it very much.
> >> >>>
> >> >>> Date: Sat, 4 May 2013 11:40:54 -0400
> >> >>>> From: [email protected]
> >> >>>> To:
> >> >>>> [email protected].**com<[email protected]
> >> >>>> om>
> >> >>>> Subject: [hlds_linux] Huge sv drops and var spikes in net_graph 4.
> >> >>>>
> >> >>>> We never had this issue prior to Steampipe, but we're dealing with
> >> >>>> these very random and rare spikes of lag where the server framerate
> >> >>>> will
> >> >>>>
> >> >>> suddenly
> >> >>>
> >> >>>> drop from 66 to below 20 and var will spike to 20+. We're hosting 6
> >> >>>> 24
> >> >>>>
> >> >>> slot
> >> >>>
> >> >>>> servers on the following:
> >> >>>>
> >> >>>> E3-1230v2
> >> >>>> 16GB
> >> >>>> CentOS 6.4 64bit
> >> >>>>
> >> >>>> I was able to capture a screenshot of one of the drops but they're
> >> >>>> very hard to catch since they appear very very briefly.
> >> >>>>
> >> >>>> http://i.imgur.com/wxp8P3N.png
> >> >>>>
> >> >>>> server.cfg
> >> >>>>
> >> >>>> sv_minrate 35000
> >> >>>> sv_maxrate 200000
> >> >>>> sv_maxupdaterate 67
> >> >>>> sv_minupdaterate 66
> >> >>>> sv_mincmdrate 66
> >> >>>> sv_maxcmdrate 67
> >> >>>> sv_client_cmdrate_difference 100
> >> >>>>
> >> >>>>
> >> >>>> Thanks
> >> >>>> ______________________________**_________________
> >> >>>> To unsubscribe, edit your list preferences, or view the list
> >> >>>> archives,
> >> >>>>
> >> >>> please visit:
> >> >>>
> >> >>>> https://list.valvesoftware.**com/cgi-bin/mailman/listinfo/**hlds_li
> >> >>>> nux<https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_li
> >> >>>> nux>
> >> >>>>
> >> >>> ______________________________**_________________
> >> >>> To unsubscribe, edit your list preferences, or view the list
> >> >>> archives, please visit:
> >> >>> https://list.valvesoftware.**com/cgi-bin/mailman/listinfo/**hlds_lin
> >> >>> ux<https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linu
> >> >>> x> ______________________________**_________________
> >> >>> To unsubscribe, edit your list preferences, or view the list
> >> >>> archives, please visit:
> >> >>> https://list.valvesoftware.**com/cgi-bin/mailman/listinfo/**hlds_lin
> >> >>> ux<https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linu
> >> >>> x>
> >> >>>
> >> >>> ______________________________**_________________
> >> >> To unsubscribe, edit your list preferences, or view the list
> >> >> archives, please visit:
> >> >> https://list.valvesoftware.**com/cgi-bin/mailman/listinfo/**hlds_linu
> >> >> x<https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux>
> >> >>
> >> >
> >> >
> >> > ______________________________**_________________
> >> > To unsubscribe, edit your list preferences, or view the list archives,
> >> > please visit:
> >> > https://list.valvesoftware.**com/cgi-bin/mailman/listinfo/**hlds_linux
> >> > <https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux>
> >> >
> >> _______________________________________________
> >> To unsubscribe, edit your list preferences, or view the list archives,
> >> please visit:
> >> https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux
> >>
> >> _______________________________________________
> >> To unsubscribe, edit your list preferences, or view the list archives,
> >> please visit:
> >> https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux
> >> _______________________________________________
> >> To unsubscribe, edit your list preferences, or view the list archives,
> >> please visit:
> >> https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux
> >>
> >
> >
> _______________________________________________
> To unsubscribe, edit your list preferences, or view the list archives, please
> visit:
> https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux
_______________________________________________
To unsubscribe, edit your list preferences, or view the list archives, please
visit:
https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux
