Well we at least confirmed it's definitely not a hardware issue. Two separate systems in separate locations have the exact same problem. The only difference is one is the 1270 and one is the 1230. Same OS.
On Sat, May 11, 2013 at 1:27 PM, Abdulrahman Abdulkawi <[email protected] > wrote: > @Fletcher Dunn, > On linux the CPU affinity is auto-detected/decided by the srcds process, > which is normal (same as hlds). I monitored the threads created by the > process: they do not spike and only 3 are consecutively running. The actual > srcds process is what spikes (the srcds process cpu% does not include the > threads % usage when monitoring with htop, which is how I know only the > srcds spikes, not it's threads). > Without -replay does reduce CPU usage which makes it not go 'overboard' > when the process spikes. > I have tried using strace although there's way too many lines/events to > figure out at which point it spiked. I will however soon try and see if I > can have it log only certain events with timestamp. > I've taken the following excerpts from the strace log (in random order): > --- Lines Beginaccept(12, 0xffe5aeec, [16]) = -1 EAGAIN > (Resource temporarily unavailable)recv(16, 0xffe5bf8f, 1, MSG_PEEK) = > -1 EAGAIN (Resource temporarily unavailable) > send(42, "\2\1\0\0\0\262\2\0\0", 9, MSG_NOSIGNAL) = 9 > futex(0xf73c8598, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xf73c8594, {FUTEX_OP_SET, > 0, FUTEX_OP_CMP_GT, 1}) = 1futex(0xf73c857c, FUTEX_WAKE_PRIVATE, 1) = > 1futex(0xf73c8558, FUTEX_WAKE_PRIVATE, 1) = 1futex(0xdc16fce4, > FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily > unavailable)futex(0xdc16fce4, FUTEX_WAKE_PRIVATE, 1) = 0 > futex(0x128048d4, FUTEX_WAIT_PRIVATE, 2, NULL) = 0futex(0x128048d4, > FUTEX_WAKE_PRIVATE, 1) = 0 > futex(0x9f76a24, FUTEX_WAIT_PRIVATE, 7, {29, 999961633}) = > 0futex(0x9f76a08, FUTEX_WAKE_PRIVATE, 1) = 0--- Lines End > Upon server startup there are a lot of lines for each > model/material/scripts...etc: > open("/home/tf2/badwater/hl2/materials/models/weapons/v_machete/models/weapons/v_machete", > O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such > file or directory) > I doubt that explains the spikes but probably has an impact to server > restart/startup? > Does any of that help? > > Date: Sat, 11 May 2013 10:54:36 -0400 > > From: [email protected] > > To: [email protected]; [email protected] > > Subject: Re: [hlds_linux] Huge sv drops and var spikes in net_graph 4. > > > > Just tested without SourceTV, no difference at all. > > > > > > > > > > On Sat, May 11, 2013 at 10:07 AM, Essay Tew Phaun <[email protected]> > wrote: > > > > > So I've got two demos here. The first one, go to about tick 19000 and > wait > > > to around the point where Fred 2.0 joins. The server drops to 5 > frames. I > > > removed Sourcemod and recorded on another server. I wasn't able to get > > > those huge drops but there are big var spikes ~10 and it *seems* to > happen > > > as clients join. > > > > > > http://dropcanvas.com/8itl3/1 > > > http://dropcanvas.com/8itl3/2 > > > > > > I can't think of what else it could be. We haven't changed anything > about > > > the setup from before Steampipe to after. > > > > > > > > > On Mon, May 6, 2013 at 1:00 PM, Bjorn Wielens <[email protected]> > wrote: > > > > > >> > > >> > > >> Thanks for investigating, Fletcher. > > >> > > >> I can tell you that in our case, we have an 8 core system with no CPU > > >> affinity assigned (with 12 TF2 servers running comfortably with CPU to > > >> spare, srcds generally consumes 30% but can get mid 40s when full of > > >> players), Watching top, the load is fairly evenly distributed across > the > > >> individual cores, and monitoring resource usage showed only one > process > > >> spiking in usage at a time. > > >> > > >> > > >> For what it's worth, I haven't had the issue recur after disabling > replay > > >> on the servers in question. For me, that would mean the next step is > to > > >> re-enable it and see if the problem returns - I'll post back here > when I > > >> get those results (though it's entirely possible that the additional > replay > > >> overhead is the metaphorical straw on the camel's back) and any > additional > > >> information I uncover. > > >> > > >> > > >> > > >> > > >> > > >> ________________________________ > > >> From: Fletcher Dunn <[email protected]> > > >> To: Half-Life dedicated Linux server mailing list < > > >> [email protected]> > > >> Cc: "Half-Life dedicated Win32 server mailing list ( > > >> [email protected])" <[email protected]> > > >> Sent: Monday, May 6, 2013 1:41:18 PM > > >> Subject: Re: [hlds_linux] Huge sv drops and var spikes in net_graph 4. > > >> > > >> > > >> I was able to get a trace from a Windows server operator that was > > >> experiencing a similar problem. The problem in that case was that it > was > > >> spending all of its time compressing snapshots. The problem was > > >> exacerbated by some process affinity settings which were > counterproductive > > >> and putting all the threads on the same core, while other cores were > idle. > > >> > > >> Tracking the overall CPU usage can be quite deceiving when > investigating > > >> "lag". Lower CPU usage is not necessarily better! For example, in > this > > >> case if the process affinity were not set and more cores were allowed > to be > > >> used, then the CPU usage on this 4-core system would have gone above > 25% > > >> for a shorter burst, which would have been preferable to it being > pegged at > > >> 25% (a single core saturated for a longer period of time). > > >> > > >> I think it is a reasonable guess that his problem is similar to the > one > > >> you guys are seeing. (Of course they could be totally unrelated, so > take > > >> all this with a grain of salt.) But assuming that they are similar > > >> problems, there are two things to investigate. First, did something > change > > >> that is doing something subtly different with process affinity that it > > >> wasn't doing before, or is the process affinity thing totally a config > > >> issue with one server? (Or if anybody else is tweaking processor > > >> affinities, do you fully understand the implications of it?) When > you are > > >> looking at the CPU usage, is the process pegging a single CPU, or is > it > > >> pegging at some round fraction like 1/4 or 1/8? The second thing to > > >> investigate is, what is happening at the high level that is causing > the > > >> snapshot data to be so big? Do network activity spikes correlates > with the > > >> CPU spikes? If that is the case, then we can use -netspike or other > tools > > >> that you guys are > > >> probably more familiar with than I am to track down what the entity > data > > >> is that is causing all the busywork. > > >> > > >> We have very powerful performance analysis tools we can use on > Windows to > > >> track these problems down. I only wish more windows server operators > users > > >> would get setup with them! Please help us help you! It's not that > > >> difficult to get set up. Email me directly and I can get you a CD > key. > > >> https://support.steampowered.com/kb_article.php?ref=3059-RTXB-2672 > > >> > > >> -----Original Message----- > > >> From: [email protected] [mailto: > > >> [email protected]] On Behalf Of Essay Tew > Phaun > > >> Sent: Saturday, May 04, 2013 11:54 AM > > >> To: Half-Life dedicated Linux server mailing list > > >> Subject: Re: [hlds_linux] Huge sv drops and var spikes in net_graph 4. > > >> > > >> Hmm. We didn't have this problem before Steampipe though. We just use > the > > >> STV with a couple of slots open on it so that an admin can connect and > > >> watch for reported players. > > >> > > >> > > >> On Sat, May 4, 2013 at 2:44 PM, ics <[email protected]> wrote: > > >> > > >> > STV is known to cause issues and hogs a lot of memory. That's why > > >> > rarely no one uses it on public servers. > > >> > > > >> > -ics > > >> > > > >> > Essay Tew Phaun kirjoitti: > > >> > > > >> > Interesting. We're not hosting a replay, but we do have STV. > > >> >> > > >> >> > > >> >> On Sat, May 4, 2013 at 12:06 PM, Bjorn Wielens <[email protected]> > > >> wrote: > > >> >> > > >> >> Sounds like the same problem I'm having (see last few posts in > 'CPU > > >> >>> Increase' thread). > > >> >>> > > >> >>> > > >> >>> > > >> >>> > > >> >>> ______________________________**__ > > >> >>> From: Michael Johansen <[email protected]> > > >> >>> To: Half-Life dedicated Linux server mailing list < > > >> >>> [email protected].**com > > >> >>> <[email protected]>> > > >> >>> Sent: Saturday, May 4, 2013 12:43:59 PM > > >> >>> Subject: Re: [hlds_linux] Huge sv drops and var spikes in > net_graph 4. > > >> >>> > > >> >>> > > >> >>> We've noticed this as well, no complaints though as the players > does > > >> >>> not seem to notice it very much. > > >> >>> > > >> >>> Date: Sat, 4 May 2013 11:40:54 -0400 > > >> >>>> From: [email protected] > > >> >>>> To: > > >> >>>> [email protected]. > **com<[email protected] > > >> >>>> om> > > >> >>>> Subject: [hlds_linux] Huge sv drops and var spikes in net_graph > 4. > > >> >>>> > > >> >>>> We never had this issue prior to Steampipe, but we're dealing > with > > >> >>>> these very random and rare spikes of lag where the server > framerate > > >> >>>> will > > >> >>>> > > >> >>> suddenly > > >> >>> > > >> >>>> drop from 66 to below 20 and var will spike to 20+. We're > hosting 6 > > >> >>>> 24 > > >> >>>> > > >> >>> slot > > >> >>> > > >> >>>> servers on the following: > > >> >>>> > > >> >>>> E3-1230v2 > > >> >>>> 16GB > > >> >>>> CentOS 6.4 64bit > > >> >>>> > > >> >>>> I was able to capture a screenshot of one of the drops but > they're > > >> >>>> very hard to catch since they appear very very briefly. > > >> >>>> > > >> >>>> http://i.imgur.com/wxp8P3N.png > > >> >>>> > > >> >>>> server.cfg > > >> >>>> > > >> >>>> sv_minrate 35000 > > >> >>>> sv_maxrate 200000 > > >> >>>> sv_maxupdaterate 67 > > >> >>>> sv_minupdaterate 66 > > >> >>>> sv_mincmdrate 66 > > >> >>>> sv_maxcmdrate 67 > > >> >>>> sv_client_cmdrate_difference 100 > > >> >>>> > > >> >>>> > > >> >>>> Thanks > > >> >>>> ______________________________**_________________ > > >> >>>> To unsubscribe, edit your list preferences, or view the list > > >> >>>> archives, > > >> >>>> > > >> >>> please visit: > > >> >>> > > >> >>>> https://list.valvesoftware. > **com/cgi-bin/mailman/listinfo/**hlds_li > > >> >>>> nux< > https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_li > > >> >>>> nux> > > >> >>>> > > >> >>> ______________________________**_________________ > > >> >>> To unsubscribe, edit your list preferences, or view the list > > >> >>> archives, please visit: > > >> >>> https://list.valvesoftware. > **com/cgi-bin/mailman/listinfo/**hlds_lin > > >> >>> ux< > https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linu > > >> >>> x> ______________________________**_________________ > > >> >>> To unsubscribe, edit your list preferences, or view the list > > >> >>> archives, please visit: > > >> >>> https://list.valvesoftware. > **com/cgi-bin/mailman/listinfo/**hlds_lin > > >> >>> ux< > https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linu > > >> >>> x> > > >> >>> > > >> >>> ______________________________**_________________ > > >> >> To unsubscribe, edit your list preferences, or view the list > > >> >> archives, please visit: > > >> >> https://list.valvesoftware. > **com/cgi-bin/mailman/listinfo/**hlds_linu > > >> >> x< > https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux> > > >> >> > > >> > > > >> > > > >> > ______________________________**_________________ > > >> > To unsubscribe, edit your list preferences, or view the list > archives, > > >> > please visit: > > >> > https://list.valvesoftware. > **com/cgi-bin/mailman/listinfo/**hlds_linux > > >> > <https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux > > > > >> > > > >> _______________________________________________ > > >> To unsubscribe, edit your list preferences, or view the list archives, > > >> please visit: > > >> https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux > > >> > > >> _______________________________________________ > > >> To unsubscribe, edit your list preferences, or view the list archives, > > >> please visit: > > >> https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux > > >> _______________________________________________ > > >> To unsubscribe, edit your list preferences, or view the list archives, > > >> please visit: > > >> https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux > > >> > > > > > > > > _______________________________________________ > > To unsubscribe, edit your list preferences, or view the list archives, > please visit: > > https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux > > _______________________________________________ > To unsubscribe, edit your list preferences, or view the list archives, > please visit: > https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux > _______________________________________________ To unsubscribe, edit your list preferences, or view the list archives, please visit: https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux
