Re: [hlds_linux] Huge sv drops and var spikes in net_graph 4.
I was able to get a trace from a Windows server operator that was experiencing a similar problem. The problem in that case was that it was spending all of its time compressing snapshots. The problem was exacerbated by some process affinity settings which were counterproductive and putting all the threads on the same core, while other cores were idle. Tracking the overall CPU usage can be quite deceiving when investigating lag. Lower CPU usage is not necessarily better! For example, in this case if the process affinity were not set and more cores were allowed to be used, then the CPU usage on this 4-core system would have gone above 25% for a shorter burst, which would have been preferable to it being pegged at 25% (a single core saturated for a longer period of time). I think it is a reasonable guess that his problem is similar to the one you guys are seeing. (Of course they could be totally unrelated, so take all this with a grain of salt.) But assuming that they are similar problems, there are two things to investigate. First, did something change that is doing something subtly different with process affinity that it wasn't doing before, or is the process affinity thing totally a config issue with one server? (Or if anybody else is tweaking processor affinities, do you fully understand the implications of it?) When you are looking at the CPU usage, is the process pegging a single CPU, or is it pegging at some round fraction like 1/4 or 1/8? The second thing to investigate is, what is happening at the high level that is causing the snapshot data to be so big? Do network activity spikes correlates with the CPU spikes? If that is the case, then we can use -netspike or other tools that you guys are probably more familiar with than I am to track down what the entity data is that is causing all the busywork. We have very powerful performance analysis tools we can use on Windows to track these problems down. I only wish more windows server operators users would get setup with them! Please help us help you! It's not that difficult to get set up. Email me directly and I can get you a CD key. https://support.steampowered.com/kb_article.php?ref=3059-RTXB-2672 -Original Message- From: hlds_linux-boun...@list.valvesoftware.com [mailto:hlds_linux-boun...@list.valvesoftware.com] On Behalf Of Essay Tew Phaun Sent: Saturday, May 04, 2013 11:54 AM To: Half-Life dedicated Linux server mailing list Subject: Re: [hlds_linux] Huge sv drops and var spikes in net_graph 4. Hmm. We didn't have this problem before Steampipe though. We just use the STV with a couple of slots open on it so that an admin can connect and watch for reported players. On Sat, May 4, 2013 at 2:44 PM, ics i...@ics-base.net wrote: STV is known to cause issues and hogs a lot of memory. That's why rarely no one uses it on public servers. -ics Essay Tew Phaun kirjoitti: Interesting. We're not hosting a replay, but we do have STV. On Sat, May 4, 2013 at 12:06 PM, Bjorn Wielens uniac...@yahoo.ca wrote: Sounds like the same problem I'm having (see last few posts in 'CPU Increase' thread). __**__ From: Michael Johansen michs...@live.no To: Half-Life dedicated Linux server mailing list hlds_linux@list.valvesoftware.**com hlds_linux@list.valvesoftware.com Sent: Saturday, May 4, 2013 12:43:59 PM Subject: Re: [hlds_linux] Huge sv drops and var spikes in net_graph 4. We've noticed this as well, no complaints though as the players does not seem to notice it very much. Date: Sat, 4 May 2013 11:40:54 -0400 From: sc2p...@gmail.com To: hlds_linux@list.valvesoftware.**comhlds_linux@list.valvesoftware.c om Subject: [hlds_linux] Huge sv drops and var spikes in net_graph 4. We never had this issue prior to Steampipe, but we're dealing with these very random and rare spikes of lag where the server framerate will suddenly drop from 66 to below 20 and var will spike to 20+. We're hosting 6 24 slot servers on the following: E3-1230v2 16GB CentOS 6.4 64bit I was able to capture a screenshot of one of the drops but they're very hard to catch since they appear very very briefly. http://i.imgur.com/wxp8P3N.png server.cfg sv_minrate 35000 sv_maxrate 20 sv_maxupdaterate 67 sv_minupdaterate 66 sv_mincmdrate 66 sv_maxcmdrate 67 sv_client_cmdrate_difference 100 Thanks __**_ To unsubscribe, edit your list preferences, or view the list archives, please visit: https://list.valvesoftware.**com/cgi-bin/mailman/listinfo/**hlds_li nuxhttps://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_li nux __**_ To unsubscribe, edit your list preferences, or view the list archives, please visit: https://list.valvesoftware.**com/cgi-bin/mailman/listinfo/**hlds_lin
Re: [hlds_linux] Huge sv drops and var spikes in net_graph 4.
Hey, Can I get a CD key? I might fire up a few Windows servers again to send you guys some data Thanks, On Mon, May 6, 2013 at 9:41 AM, Fletcher Dunn fletch...@valvesoftware.comwrote: I was able to get a trace from a Windows server operator that was experiencing a similar problem. The problem in that case was that it was spending all of its time compressing snapshots. The problem was exacerbated by some process affinity settings which were counterproductive and putting all the threads on the same core, while other cores were idle. Tracking the overall CPU usage can be quite deceiving when investigating lag. Lower CPU usage is not necessarily better! For example, in this case if the process affinity were not set and more cores were allowed to be used, then the CPU usage on this 4-core system would have gone above 25% for a shorter burst, which would have been preferable to it being pegged at 25% (a single core saturated for a longer period of time). I think it is a reasonable guess that his problem is similar to the one you guys are seeing. (Of course they could be totally unrelated, so take all this with a grain of salt.) But assuming that they are similar problems, there are two things to investigate. First, did something change that is doing something subtly different with process affinity that it wasn't doing before, or is the process affinity thing totally a config issue with one server? (Or if anybody else is tweaking processor affinities, do you fully understand the implications of it?) When you are looking at the CPU usage, is the process pegging a single CPU, or is it pegging at some round fraction like 1/4 or 1/8? The second thing to investigate is, what is happening at the high level that is causing the snapshot data to be so big? Do network activity spikes correlates with the CPU spikes? If that is the case, then we can use -netspike or other tools that you guys are probably more familiar with than I am to track down what the entity data is that is causing all the busywork. We have very powerful performance analysis tools we can use on Windows to track these problems down. I only wish more windows server operators users would get setup with them! Please help us help you! It's not that difficult to get set up. Email me directly and I can get you a CD key. https://support.steampowered.com/kb_article.php?ref=3059-RTXB-2672 -Original Message- From: hlds_linux-boun...@list.valvesoftware.com [mailto: hlds_linux-boun...@list.valvesoftware.com] On Behalf Of Essay Tew Phaun Sent: Saturday, May 04, 2013 11:54 AM To: Half-Life dedicated Linux server mailing list Subject: Re: [hlds_linux] Huge sv drops and var spikes in net_graph 4. Hmm. We didn't have this problem before Steampipe though. We just use the STV with a couple of slots open on it so that an admin can connect and watch for reported players. On Sat, May 4, 2013 at 2:44 PM, ics i...@ics-base.net wrote: STV is known to cause issues and hogs a lot of memory. That's why rarely no one uses it on public servers. -ics Essay Tew Phaun kirjoitti: Interesting. We're not hosting a replay, but we do have STV. On Sat, May 4, 2013 at 12:06 PM, Bjorn Wielens uniac...@yahoo.ca wrote: Sounds like the same problem I'm having (see last few posts in 'CPU Increase' thread). __**__ From: Michael Johansen michs...@live.no To: Half-Life dedicated Linux server mailing list hlds_linux@list.valvesoftware.**com hlds_linux@list.valvesoftware.com Sent: Saturday, May 4, 2013 12:43:59 PM Subject: Re: [hlds_linux] Huge sv drops and var spikes in net_graph 4. We've noticed this as well, no complaints though as the players does not seem to notice it very much. Date: Sat, 4 May 2013 11:40:54 -0400 From: sc2p...@gmail.com To: hlds_linux@list.valvesoftware.**comhlds_linux@list.valvesoftware.c om Subject: [hlds_linux] Huge sv drops and var spikes in net_graph 4. We never had this issue prior to Steampipe, but we're dealing with these very random and rare spikes of lag where the server framerate will suddenly drop from 66 to below 20 and var will spike to 20+. We're hosting 6 24 slot servers on the following: E3-1230v2 16GB CentOS 6.4 64bit I was able to capture a screenshot of one of the drops but they're very hard to catch since they appear very very briefly. http://i.imgur.com/wxp8P3N.png server.cfg sv_minrate 35000 sv_maxrate 20 sv_maxupdaterate 67 sv_minupdaterate 66 sv_mincmdrate 66 sv_maxcmdrate 67 sv_client_cmdrate_difference 100 Thanks __**_ To unsubscribe, edit your list preferences, or view the list archives, please visit: https://list.valvesoftware.**com/cgi-bin/mailman/listinfo/**hlds_li nuxhttps://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_li
Re: [hlds_linux] Huge sv drops and var spikes in net_graph 4.
Thanks for investigating, Fletcher. I can tell you that in our case, we have an 8 core system with no CPU affinity assigned (with 12 TF2 servers running comfortably with CPU to spare, srcds generally consumes 30% but can get mid 40s when full of players), Watching top, the load is fairly evenly distributed across the individual cores, and monitoring resource usage showed only one process spiking in usage at a time. For what it's worth, I haven't had the issue recur after disabling replay on the servers in question. For me, that would mean the next step is to re-enable it and see if the problem returns - I'll post back here when I get those results (though it's entirely possible that the additional replay overhead is the metaphorical straw on the camel's back) and any additional information I uncover. From: Fletcher Dunn fletch...@valvesoftware.com To: Half-Life dedicated Linux server mailing list hlds_linux@list.valvesoftware.com Cc: Half-Life dedicated Win32 server mailing list (h...@list.valvesoftware.com) h...@list.valvesoftware.com Sent: Monday, May 6, 2013 1:41:18 PM Subject: Re: [hlds_linux] Huge sv drops and var spikes in net_graph 4. I was able to get a trace from a Windows server operator that was experiencing a similar problem. The problem in that case was that it was spending all of its time compressing snapshots. The problem was exacerbated by some process affinity settings which were counterproductive and putting all the threads on the same core, while other cores were idle. Tracking the overall CPU usage can be quite deceiving when investigating lag. Lower CPU usage is not necessarily better! For example, in this case if the process affinity were not set and more cores were allowed to be used, then the CPU usage on this 4-core system would have gone above 25% for a shorter burst, which would have been preferable to it being pegged at 25% (a single core saturated for a longer period of time). I think it is a reasonable guess that his problem is similar to the one you guys are seeing. (Of course they could be totally unrelated, so take all this with a grain of salt.) But assuming that they are similar problems, there are two things to investigate. First, did something change that is doing something subtly different with process affinity that it wasn't doing before, or is the process affinity thing totally a config issue with one server? (Or if anybody else is tweaking processor affinities, do you fully understand the implications of it?) When you are looking at the CPU usage, is the process pegging a single CPU, or is it pegging at some round fraction like 1/4 or 1/8? The second thing to investigate is, what is happening at the high level that is causing the snapshot data to be so big? Do network activity spikes correlates with the CPU spikes? If that is the case, then we can use -netspike or other tools that you guys are probably more familiar with than I am to track down what the entity data is that is causing all the busywork. We have very powerful performance analysis tools we can use on Windows to track these problems down. I only wish more windows server operators users would get setup with them! Please help us help you! It's not that difficult to get set up. Email me directly and I can get you a CD key. https://support.steampowered.com/kb_article.php?ref=3059-RTXB-2672 -Original Message- From: hlds_linux-boun...@list.valvesoftware.com [mailto:hlds_linux-boun...@list.valvesoftware.com] On Behalf Of Essay Tew Phaun Sent: Saturday, May 04, 2013 11:54 AM To: Half-Life dedicated Linux server mailing list Subject: Re: [hlds_linux] Huge sv drops and var spikes in net_graph 4. Hmm. We didn't have this problem before Steampipe though. We just use the STV with a couple of slots open on it so that an admin can connect and watch for reported players. On Sat, May 4, 2013 at 2:44 PM, ics i...@ics-base.net wrote: STV is known to cause issues and hogs a lot of memory. That's why rarely no one uses it on public servers. -ics Essay Tew Phaun kirjoitti: Interesting. We're not hosting a replay, but we do have STV. On Sat, May 4, 2013 at 12:06 PM, Bjorn Wielens uniac...@yahoo.ca wrote: Sounds like the same problem I'm having (see last few posts in 'CPU Increase' thread). __**__ From: Michael Johansen michs...@live.no To: Half-Life dedicated Linux server mailing list hlds_linux@list.valvesoftware.**com hlds_linux@list.valvesoftware.com Sent: Saturday, May 4, 2013 12:43:59 PM Subject: Re: [hlds_linux] Huge sv drops and var spikes in net_graph 4. We've noticed this as well, no complaints though as the players does not seem to notice it very much. Date: Sat, 4 May 2013 11:40:54 -0400 From: sc2p...@gmail.com To: hlds_linux@list.valvesoftware.**comhlds_linux@list.valvesoftware.c om Subject: [hlds_linux] Huge sv drops
[hlds_linux] crash problems in Freak Fortress 2 and Vanilla Server
hi, im new to the HLDS mailing list, i've got a problem that nobody has seemed to be able/willing to diagnosis, i only know a medium tech knowledge of servers, because im mostly focusing on creating the actual content (maps, models, sounds) instead. My Freak Fortress 2 and Vanilla Server has been crashing a lot, on map change, or (final) round end. so here's my SM logs, FF2: http://pastebin.com/u9qCyPqk TF2: http://pastebin.com/wtn4W3z6 ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux
Re: [hlds_linux] crash problems in Freak Fortress 2 and Vanilla Server
On Mon, 2013-05-06 at 19:36 +0200, Albert Kim wrote: hi, im new to the HLDS mailing list, i've got a problem that nobody has seemed to be able/willing to diagnosis, i only know a medium tech knowledge of servers, because im mostly focusing on creating the actual content (maps, models, sounds) instead. My Freak Fortress 2 and Vanilla Server has been crashing a lot, on map change, or (final) round end. That's a load of SourceMod error messages. Have you tried asking in the SourceMod forums? Do you have the newest versions of all plugins? If you have installed the SteamPipe beta gamedata, have you deleted it again now that it has gone official and the automatic updater took over? Cheers, ~~ Ondra ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux
Re: [hlds_linux] crash problems in Freak Fortress 2 and Vanilla Server
Source mod is far from vanilla... Sent from my iPhone 5 On May 6, 2013, at 10:51 AM, Ondřej Hošek ondra.ho...@gmail.com wrote: On Mon, 2013-05-06 at 19:36 +0200, Albert Kim wrote: hi, im new to the HLDS mailing list, i've got a problem that nobody has seemed to be able/willing to diagnosis, i only know a medium tech knowledge of servers, because im mostly focusing on creating the actual content (maps, models, sounds) instead. My Freak Fortress 2 and Vanilla Server has been crashing a lot, on map change, or (final) round end. That's a load of SourceMod error messages. Have you tried asking in the SourceMod forums? Do you have the newest versions of all plugins? If you have installed the SteamPipe beta gamedata, have you deleted it again now that it has gone official and the automatic updater took over? Cheers, ~~ Ondra ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux
Re: [hlds_linux] crash problems in Freak Fortress 2 and Vanilla Server
This is.. pretty frightful. You really should look into putting some try/catches or something into your plugins (or bug the devs to) if you're getting this many. I could keep poking through but at this point you really need to strip EVERYTHING OUT and start putting it back together piece by piece. We are not SourceMod/FreakFortress support, and while we (hlds_linux) generally have the brainpower to figure out what might be causing a problem with SourceMod, this is a thread you probably should take up over at SourceMod forums. Also you say your Vanilla server is crashing a lot. What mods is that running? On Mon, May 6, 2013 at 12:29 PM, DontWannaName! ad...@topnotchclan.comwrote: Source mod is far from vanilla... Sent from my iPhone 5 On May 6, 2013, at 10:51 AM, Ondřej Hošek ondra.ho...@gmail.com wrote: On Mon, 2013-05-06 at 19:36 +0200, Albert Kim wrote: hi, im new to the HLDS mailing list, i've got a problem that nobody has seemed to be able/willing to diagnosis, i only know a medium tech knowledge of servers, because im mostly focusing on creating the actual content (maps, models, sounds) instead. My Freak Fortress 2 and Vanilla Server has been crashing a lot, on map change, or (final) round end. That's a load of SourceMod error messages. Have you tried asking in the SourceMod forums? Do you have the newest versions of all plugins? If you have installed the SteamPipe beta gamedata, have you deleted it again now that it has gone official and the automatic updater took over? Cheers, ~~ Ondra ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux -- - Gordon Reynolds ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux
Re: [hlds_linux] crash problems in Freak Fortress 2 and Vanilla Server
Looks to me from the vanilla log ( which isn't actually vanilla) that something has paused source mod. First place I would look is metamod. All the errors I see point to the same issue. On May 6, 2013 5:27 PM, Gordon Reynolds thisisgordonsem...@gmail.com wrote: This is.. pretty frightful. You really should look into putting some try/catches or something into your plugins (or bug the devs to) if you're getting this many. I could keep poking through but at this point you really need to strip EVERYTHING OUT and start putting it back together piece by piece. We are not SourceMod/FreakFortress support, and while we (hlds_linux) generally have the brainpower to figure out what might be causing a problem with SourceMod, this is a thread you probably should take up over at SourceMod forums. Also you say your Vanilla server is crashing a lot. What mods is that running? On Mon, May 6, 2013 at 12:29 PM, DontWannaName! ad...@topnotchclan.com wrote: Source mod is far from vanilla... Sent from my iPhone 5 On May 6, 2013, at 10:51 AM, Ondřej Hošek ondra.ho...@gmail.com wrote: On Mon, 2013-05-06 at 19:36 +0200, Albert Kim wrote: hi, im new to the HLDS mailing list, i've got a problem that nobody has seemed to be able/willing to diagnosis, i only know a medium tech knowledge of servers, because im mostly focusing on creating the actual content (maps, models, sounds) instead. My Freak Fortress 2 and Vanilla Server has been crashing a lot, on map change, or (final) round end. That's a load of SourceMod error messages. Have you tried asking in the SourceMod forums? Do you have the newest versions of all plugins? If you have installed the SteamPipe beta gamedata, have you deleted it again now that it has gone official and the automatic updater took over? Cheers, ~~ Ondra ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux -- - Gordon Reynolds ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux
