Re: [hlds_linux] netspike.txt
Mine is too big to open in Windows unfortunately. On Wed, Jul 11, 2012 at 11:38 AM, ics i...@ics-base.net wrote: I think i'm seeing a pattern which causes the overflows and there's nothing *I* can do about these things. This is just 3 clients from 2 different servers but i suspect the rest will be showing the same stuff. The rest of the data is just plain small compared to these on those snapshots. StringTable soundprecache : 8257 bits (1032.125 bytes) StringTable instancebaseline : 160672 bits (20084.000 bytes) StringTable userinfo :11919 bits (1489.875 bytes) StringTable DynamicModels : 7128 bits ( 891.000 bytes) Total Delta : 130135 bits (16266.875 bytes) StringTable soundprecache :11173 bits (1396.625 bytes) StringTable instancebaseline : 192452 bits (24056.500 bytes) StringTable userinfo : 7610 bits ( 951.250 bytes) StringTable DynamicModels : 8030 bits (1003.750 bytes) Total Delta : 100317 bits (12539.625 bytes) StringTable soundprecache :10887 bits (1360.875 bytes) StringTable instancebaseline : 176476 bits (22059.500 bytes) StringTable userinfo : 8675 bits (1084.375 bytes) StringTable DynamicModels :11910 bits (1488.750 bytes) Total Delta : 110850 bits (13856.250 bytes) -ics 11.7.2012 20:40, Fletcher Dunn kirjoitti: Ah, if you are seeing it mostly on the saxton servers, then that could be a clue. There's could be some big array or object or something in the saxton entities being networked inefficiently. The netspike file should make it obvious where the bandwidth is going. But if you need some help analyzing it, email it to me directly. -Original Message- From: hlds_linux-bounces@list.**valvesoftware.comhlds_linux-boun...@list.valvesoftware.com[mailto: hlds_linux-bounces@**list.valvesoftware.comhlds_linux-boun...@list.valvesoftware.com] On Behalf Of Erik-jan Riemers Sent: Wednesday, July 11, 2012 10:30 AM To: Half-Life dedicated Linux server mailing list Subject: Re: [hlds_linux] netspike.txt When you update, put in the notes the cvars or whatever you can setup for those option. Since I only see it on a couple of servers (saxton servers mostly, but hey I can understand) its actually a nice feedback to know that clients are having issues and it could be related too a plugin too. -Original Message- From: hlds_linux-bounces@list.**valvesoftware.comhlds_linux-boun...@list.valvesoftware.com [mailto:hlds_linux-bounces@**list.valvesoftware.comhlds_linux-boun...@list.valvesoftware.com] On Behalf Of Fletcher Dunn Sent: woensdag 11 juli 2012 18:58 To: Half-Life dedicated Linux server mailing list; ics Subject: Re: [hlds_linux] netspike.txt The netspike file has long existed in the Source engine as a way to track which entities were causing large spikes of network traffic. There's a convar you can use to configure a threshold when it will get dumped. (I think by default it's off.) Recently there was a change where, if a client was dropped to the infamous snapshot overflow error, it would re-run the network packing code for that client, forcing the netspike file to be dumped, to help figure out what entities are the cause of the problem. That could definitely cause CPU spikes. I didn't realize we had shipped with that debugging mode on. We'll make it configurable by convar for the next release. In the meantime, each time it dumps a network trace, it means you dropped a client. It needed to send them a full snapshot of the game state, but the snapshot was too big, so it had to drop the client. You might want to see which entities are responsible for so much traffic, because you are dropping players. If there's something particular to your server, this trace will help you tune it. If you find a problem you think is global to the game, please report it here. Your humble servant, Fletch -Original Message- From: hlds_linux-bounces@list.**valvesoftware.comhlds_linux-boun...@list.valvesoftware.com [mailto:hlds_linux-bounces@**list.valvesoftware.comhlds_linux-boun...@list.valvesoftware.com] On Behalf Of Thorsten Knoll Sent: Wednesday, July 11, 2012 8:39 AM To: ics; Half-Life dedicated Linux server mailing list Subject: Re: [hlds_linux] netspike.txt i only have it on my registered servers with quickplay. on my not registered servers there´s no netspike.txt. -moss Am 11.07.2012 12:47, schrieb ics: I have that file on 4 servers from 0.5 to 2.17 gb in filesize. I guess opening and writing to that file could cause the cpu spikes some are having. What is this file for? Can someone from valve enlighten the matter? -ics - Alkuperäinen viesti - yes it looks like debuggin info of some sort. My two servers have those files to and both are 2.0Gb each On 2012-07-11 11:20, Jesse Molina wrote: Interesting. This file has indeed appeared on some servers on a host which I manage. In one case, the file is now 1.3GB
Re: [hlds_linux] netspike.txt
I have on 5 gig netspike file and one 3 gig. :/ On Thu, Jul 12, 2012 at 10:36 AM, DontWannaName! ad...@topnotchclan.comwrote: Mine is too big to open in Windows unfortunately. On Wed, Jul 11, 2012 at 11:38 AM, ics i...@ics-base.net wrote: I think i'm seeing a pattern which causes the overflows and there's nothing *I* can do about these things. This is just 3 clients from 2 different servers but i suspect the rest will be showing the same stuff. The rest of the data is just plain small compared to these on those snapshots. StringTable soundprecache : 8257 bits (1032.125 bytes) StringTable instancebaseline : 160672 bits (20084.000 bytes) StringTable userinfo :11919 bits (1489.875 bytes) StringTable DynamicModels : 7128 bits ( 891.000 bytes) Total Delta : 130135 bits (16266.875 bytes) StringTable soundprecache :11173 bits (1396.625 bytes) StringTable instancebaseline : 192452 bits (24056.500 bytes) StringTable userinfo : 7610 bits ( 951.250 bytes) StringTable DynamicModels : 8030 bits (1003.750 bytes) Total Delta : 100317 bits (12539.625 bytes) StringTable soundprecache :10887 bits (1360.875 bytes) StringTable instancebaseline : 176476 bits (22059.500 bytes) StringTable userinfo : 8675 bits (1084.375 bytes) StringTable DynamicModels :11910 bits (1488.750 bytes) Total Delta : 110850 bits (13856.250 bytes) -ics 11.7.2012 20:40, Fletcher Dunn kirjoitti: Ah, if you are seeing it mostly on the saxton servers, then that could be a clue. There's could be some big array or object or something in the saxton entities being networked inefficiently. The netspike file should make it obvious where the bandwidth is going. But if you need some help analyzing it, email it to me directly. -Original Message- From: hlds_linux-bounces@list.**valvesoftware.comhlds_linux-boun...@list.valvesoftware.com[mailto: hlds_linux-bounces@**list.valvesoftware.comhlds_linux-boun...@list.valvesoftware.com] On Behalf Of Erik-jan Riemers Sent: Wednesday, July 11, 2012 10:30 AM To: Half-Life dedicated Linux server mailing list Subject: Re: [hlds_linux] netspike.txt When you update, put in the notes the cvars or whatever you can setup for those option. Since I only see it on a couple of servers (saxton servers mostly, but hey I can understand) its actually a nice feedback to know that clients are having issues and it could be related too a plugin too. -Original Message- From: hlds_linux-bounces@list.**valvesoftware.comhlds_linux-boun...@list.valvesoftware.com [mailto:hlds_linux-bounces@**list.valvesoftware.comhlds_linux-boun...@list.valvesoftware.com] On Behalf Of Fletcher Dunn Sent: woensdag 11 juli 2012 18:58 To: Half-Life dedicated Linux server mailing list; ics Subject: Re: [hlds_linux] netspike.txt The netspike file has long existed in the Source engine as a way to track which entities were causing large spikes of network traffic. There's a convar you can use to configure a threshold when it will get dumped. (I think by default it's off.) Recently there was a change where, if a client was dropped to the infamous snapshot overflow error, it would re-run the network packing code for that client, forcing the netspike file to be dumped, to help figure out what entities are the cause of the problem. That could definitely cause CPU spikes. I didn't realize we had shipped with that debugging mode on. We'll make it configurable by convar for the next release. In the meantime, each time it dumps a network trace, it means you dropped a client. It needed to send them a full snapshot of the game state, but the snapshot was too big, so it had to drop the client. You might want to see which entities are responsible for so much traffic, because you are dropping players. If there's something particular to your server, this trace will help you tune it. If you find a problem you think is global to the game, please report it here. Your humble servant, Fletch -Original Message- From: hlds_linux-bounces@list.**valvesoftware.comhlds_linux-boun...@list.valvesoftware.com [mailto:hlds_linux-bounces@**list.valvesoftware.comhlds_linux-boun...@list.valvesoftware.com] On Behalf Of Thorsten Knoll Sent: Wednesday, July 11, 2012 8:39 AM To: ics; Half-Life dedicated Linux server mailing list Subject: Re: [hlds_linux] netspike.txt i only have it on my registered servers with quickplay. on my not registered servers there´s no netspike.txt. -moss Am 11.07.2012 12:47, schrieb ics: I have that file on 4 servers from 0.5 to 2.17 gb in filesize. I guess opening and writing to that file could cause the cpu spikes some are having. What is this file for? Can someone from valve enlighten the matter? -ics - Alkuperäinen viesti - yes it looks like debuggin info of some sort. My two servers have those files to and both are 2.0Gb each On 2012-07-11 11:20, Jesse Molina wrote
[hlds_linux] netspike.txt
What is this file? I haven't noticed it before, but it's showing up in my tf folder. Looks like a log of packet information being sent to clients. Why is this showing up now, and how can this logging be disabled? 40602.961200/2703847 Player [Player.X][7][adr:XX.XXX.XXX.XX:46611] was sent a datagram 32 bits (4.000 bytes) NET_Tick : 70 bits ( 8.750 bytes) [soundprecache] 3745:)weapons/scatter_gun_double_shoot.wav : 323 bits ( 40.375 bytes) [soundprecache] 3746:)weapons/scatter_gun_double_shoot_crit.wa : 112 bits ( 14.000 bytes) [soundprecache] 3747:)weapons/pickaxe_swing1.wav : 168 bits ( 21.000 bytes) [soundprecache] 3748:)weapons/pickaxe_swing2.wav : 64 bits ( 8.000 bytes) [soundprecache] 3749:)weapons/pickaxe_swing3.wav : 64 bits ( 8.000 bytes) [soundprecache] 3750:weapons/blade_slice_2.wav : 214 bits ( 26.750 bytes) [soundprecache] 3751:weapons/blade_slice_3.wav : 64 bits ( 8.000 bytes) [soundprecache] 3752:weapons/blade_slice_4.wav : 64 bits ( 8.000 bytes) [soundprecache] 3753:)weapons\sniper_railgun_dry_fire.wav : 248 bits ( 31.000 bytes) [soundprecache] 3754:)weapons\sniper_railgun_charged_shot_01.w : 176 bits ( 22.000 bytes) [soundprecache] 3755:)weapons\sniper_railgun_charged_shot_02.w : 120 bits ( 15.000 bytes) [soundprecache] 3756:)weapons\sniper_railgun_charged_shot_crit : 160 bits ( 20.000 bytes) [soundprecache] 3757:)weapons\sniper_railgun_charged_shot_crit : 160 bits ( 20.000 bytes) [soundprecache] 3758:)weapons\sniper_railgun_world_reload.wav : 152 bits ( 19.000 bytes) [soundprecache] 3759:)weapons\sniper_railgun_no_fire.wav : 112 bits ( 14.000 bytes) [soundprecache] 3760:)weapons\sniper_railgun_single_01.wav : 128 bits ( 16.000 bytes) [soundprecache] 3761:)weapons\sniper_railgun_single_02.wav : 72 bits ( 9.000 bytes) [soundprecache] 3762:)weapons/flame_thrower_bb_start.wav : 232 bits ( 29.000 bytes) [soundprecache] 3763:)weapons/flame_thrower_bb_loop_crit.wav : 128 bits ( 16.000 bytes) [soundprecache] 3764:)weapons/flame_thrower_bb_loop.wav : 56 bits ( 7.000 bytes) [soundprecache] 3765:)weapons/ambassador_shoot.wav : 184 bits ( 23.000 bytes) [soundprecache] 3766:)weapons/ambassador_shoot_crit.wav : 96 bits ( 12.000 bytes) [soundprecache] 3767:)weapons/demo_sword_hit_world1.wav : 224 bits ( 28.000 bytes) [soundprecache] 3768:)weapons/demo_sword_hit_world2.wav : 64 bits ( 8.000 bytes) [soundprecache] 3769:vo/sword_hit01.wav : 158 bits ( 19.750 bytes) [soundprecache] 3770:vo/sword_hit02.wav : 64 bits ( 8.000 bytes) [soundprecache] 3771:vo/sword_hit03.wav : 64 bits ( 8.000 bytes) [soundprecache] 3772:vo/sword_hit04.wav : 64 bits ( 8.000 bytes) [soundprecache] 3773:vo/sword_hit05.wav : 64 bits ( 8.000 bytes) [soundprecache] 3774:vo/sword_hit06.wav : 64 bits ( 8.000 bytes) [soundprecache] 3775:vo/sword_hit07.wav : 64 bits ( 8.000 bytes) [soundprecache] 3776:vo/sword_hit08.wav : 64 bits ( 8.000 bytes) [soundprecache] 3777:vo/sword_hit09.wav : 64 bits ( 8.000 bytes) [soundprecache] 3778:vo/sword_hit10.wav : 72 bits ( 9.000 bytes) [soundprecache] 3779:vo/sword_idle01.wav : 104 bits ( 13.000 bytes) [soundprecache] 3780:vo/sword_idle02.wav : 64 bits ( 8.000 bytes) [soundprecache] 3781:vo/sword_idle03.wav : 64 bits ( 8.000 bytes) [soundprecache] 3782:vo/sword_idle04.wav : 64 bits ( 8.000 bytes) [soundprecache] 3783:vo/sword_idle05.wav : 64 bits ( 8.000 bytes) [soundprecache] 3784:vo/sword_idle06.wav : 64 bits ( 8.000 bytes) [soundprecache] 3785:vo/sword_idle07.wav : 64 bits ( 8.000 bytes) [soundprecache] 3786:vo/sword_idle08.wav : 64 bits ( 8.000 bytes) [soundprecache] 3787:vo/sword_idle09.wav : 64 bits ( 8.000 bytes) [soundprecache] 3788:vo/sword_idle10.wav : 72 bits ( 9.000 bytes) [soundprecache] 3789:vo/sword_idle11.wav : 64 bits ( 8.000 bytes) [soundprecache] 3790:vo/sword_idle12.wav : 64 bits ( 8.000 bytes) [soundprecache]
Re: [hlds_linux] netspike.txt
On 11.07.2012 07:58, Russell Smith wrote: What is this file? I haven't noticed it before, but it's showing up in my tf folder. Looks like a log of packet information being sent to clients. I've noticed that file as well, and its gotten pretty large on some of my TF2s. (Deleting it seems not to harm the server, though) /Peter ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux
Re: [hlds_linux] netspike.txt
Interesting. This file has indeed appeared on some servers on a host which I manage. In one case, the file is now 1.3GB in size. I suspect Valve has noticed the same thing I noticed. When the Pyromania update came out, I noticed that my host was using much more bandwidth than usual. There seemed to be a spike happening, intermittently, for limited periods of time. I observed a 30% traffic in/out increase for 30-60 seconds at a time. All clients on all servers on that particular host, and all servers on that host, would suddenly start sending much more data, for no apparent reason. I only have a 10base-T uplink on this host, so I am limited to 10Mbps. All four of the servers were full that day. However, due to the traffic spikes, we had to shut one of them down to prevent exceeding the circuit limitation. This is how the issue first came to my attention. I did a tcpdump of the traffic but whatever the issue was, it was embedded in the game traffic itself and was being represented by all clients on all four servers at the exact same time. If I was to guess, something like an authentication request or item-server related activity was suddenly causing all clients and the server to suddenly transmit much more data. Whatever it was, it would have been something that affected all clients across those four servers on the same host. Valve would have noticed huge spikes like this since ALL of their servers and clients would suddenly start transmitting 20-30% more traffic. I am guessing they are trying to debug the problem by collecting evidence. That's just a guess though. Russell Smith wrote: What is this file? I haven't noticed it before, but it's showing up in my tf folder. Looks like a log of packet information being sent to clients. Why is this showing up now, and how can this logging be disabled? 40602.961200/2703847 Player [Player.X][7][adr:XX.XXX.XXX.XX:46611] was sent a datagram 32 bits (4.000 bytes) NET_Tick : 70 bits ( 8.750 bytes) [soundprecache] 3745:)weapons/scatter_gun_double_shoot.wav : 323 bits ( 40.375 bytes) [soundprecache] 3746:)weapons/scatter_gun_double_shoot_crit.wa : 112 bits ( 14.000 bytes) [soundprecache] 3747:)weapons/pickaxe_swing1.wav : 168 bits ( 21.000 bytes) [soundprecache] 3748:)weapons/pickaxe_swing2.wav : 64 bits ( 8.000 bytes) [soundprecache] 3749:)weapons/pickaxe_swing3.wav : 64 bits ( 8.000 bytes) [soundprecache] 3750:weapons/blade_slice_2.wav : 214 bits ( 26.750 bytes) [soundprecache] 3751:weapons/blade_slice_3.wav : 64 bits ( 8.000 bytes) [soundprecache] 3752:weapons/blade_slice_4.wav : 64 bits ( 8.000 bytes) [soundprecache] 3753:)weapons\sniper_railgun_dry_fire.wav : 248 bits ( 31.000 bytes) [soundprecache] 3754:)weapons\sniper_railgun_charged_shot_01.w : 176 bits ( 22.000 bytes) [soundprecache] 3755:)weapons\sniper_railgun_charged_shot_02.w : 120 bits ( 15.000 bytes) [soundprecache] 3756:)weapons\sniper_railgun_charged_shot_crit : 160 bits ( 20.000 bytes) [soundprecache] 3757:)weapons\sniper_railgun_charged_shot_crit : 160 bits ( 20.000 bytes) [soundprecache] 3758:)weapons\sniper_railgun_world_reload.wav : 152 bits ( 19.000 bytes) [soundprecache] 3759:)weapons\sniper_railgun_no_fire.wav : 112 bits ( 14.000 bytes) [soundprecache] 3760:)weapons\sniper_railgun_single_01.wav : 128 bits ( 16.000 bytes) [soundprecache] 3761:)weapons\sniper_railgun_single_02.wav : 72 bits ( 9.000 bytes) [soundprecache] 3762:)weapons/flame_thrower_bb_start.wav : 232 bits ( 29.000 bytes) [soundprecache] 3763:)weapons/flame_thrower_bb_loop_crit.wav : 128 bits ( 16.000 bytes) [soundprecache] 3764:)weapons/flame_thrower_bb_loop.wav : 56 bits ( 7.000 bytes) [soundprecache] 3765:)weapons/ambassador_shoot.wav : 184 bits ( 23.000 bytes) [soundprecache] 3766:)weapons/ambassador_shoot_crit.wav : 96 bits ( 12.000 bytes) [soundprecache] 3767:)weapons/demo_sword_hit_world1.wav : 224 bits ( 28.000 bytes) [soundprecache] 3768:)weapons/demo_sword_hit_world2.wav : 64 bits ( 8.000 bytes) [soundprecache] 3769:vo/sword_hit01.wav : 158 bits ( 19.750 bytes) [soundprecache] 3770:vo/sword_hit02.wav : 64 bits ( 8.000 bytes) [soundprecache] 3771:vo/sword_hit03.wav : 64 bits ( 8.000 bytes) [soundprecache] 3772:vo/sword_hit04.wav : 64 bits ( 8.000 bytes) [soundprecache] 3773:vo/sword_hit05.wav : 64 bits ( 8.000 bytes) [soundprecache] 3774:vo/sword_hit06.wav : 64 bits ( 8.000 bytes) [soundprecache] 3775:vo/sword_hit07.wav : 64 bits ( 8.000 bytes)
Re: [hlds_linux] netspike.txt
yes it looks like debuggin info of some sort. My two servers have those files to and both are 2.0Gb each On 2012-07-11 11:20, Jesse Molina wrote: Interesting. This file has indeed appeared on some servers on a host which I manage. In one case, the file is now 1.3GB in size. I suspect Valve has noticed the same thing I noticed. When the Pyromania update came out, I noticed that my host was using much more bandwidth than usual. There seemed to be a spike happening, intermittently, for limited periods of time. I observed a 30% traffic in/out increase for 30-60 seconds at a time. All clients on all servers on that particular host, and all servers on that host, would suddenly start sending much more data, for no apparent reason. I only have a 10base-T uplink on this host, so I am limited to 10Mbps. All four of the servers were full that day. However, due to the traffic spikes, we had to shut one of them down to prevent exceeding the circuit limitation. This is how the issue first came to my attention. I did a tcpdump of the traffic but whatever the issue was, it was embedded in the game traffic itself and was being represented by all clients on all four servers at the exact same time. If I was to guess, something like an authentication request or item-server related activity was suddenly causing all clients and the server to suddenly transmit much more data. Whatever it was, it would have been something that affected all clients across those four servers on the same host. Valve would have noticed huge spikes like this since ALL of their servers and clients would suddenly start transmitting 20-30% more traffic. I am guessing they are trying to debug the problem by collecting evidence. That's just a guess though. Russell Smith wrote: What is this file? I haven't noticed it before, but it's showing up in my tf folder. Looks like a log of packet information being sent to clients. Why is this showing up now, and how can this logging be disabled? 40602.961200/2703847 Player [Player.X][7][adr:XX.XXX.XXX.XX:46611] was sent a datagram 32 bits (4.000 bytes) NET_Tick : 70 bits ( 8.750 bytes) [soundprecache] 3745:)weapons/scatter_gun_double_shoot.wav : 323 bits ( 40.375 bytes) [soundprecache] 3746:)weapons/scatter_gun_double_shoot_crit.wa : 112 bits ( 14.000 bytes) [soundprecache] 3747:)weapons/pickaxe_swing1.wav : 168 bits ( 21.000 bytes) [soundprecache] 3748:)weapons/pickaxe_swing2.wav : 64 bits ( 8.000 bytes) [soundprecache] 3749:)weapons/pickaxe_swing3.wav : 64 bits ( 8.000 bytes) [soundprecache] 3750:weapons/blade_slice_2.wav : 214 bits ( 26.750 bytes) [soundprecache] 3751:weapons/blade_slice_3.wav : 64 bits ( 8.000 bytes) [soundprecache] 3752:weapons/blade_slice_4.wav : 64 bits ( 8.000 bytes) [soundprecache] 3753:)weapons\sniper_railgun_dry_fire.wav : 248 bits ( 31.000 bytes) [soundprecache] 3754:)weapons\sniper_railgun_charged_shot_01.w : 176 bits ( 22.000 bytes) [soundprecache] 3755:)weapons\sniper_railgun_charged_shot_02.w : 120 bits ( 15.000 bytes) [soundprecache] 3756:)weapons\sniper_railgun_charged_shot_crit : 160 bits ( 20.000 bytes) [soundprecache] 3757:)weapons\sniper_railgun_charged_shot_crit : 160 bits ( 20.000 bytes) [soundprecache] 3758:)weapons\sniper_railgun_world_reload.wav : 152 bits ( 19.000 bytes) [soundprecache] 3759:)weapons\sniper_railgun_no_fire.wav : 112 bits ( 14.000 bytes) [soundprecache] 3760:)weapons\sniper_railgun_single_01.wav : 128 bits ( 16.000 bytes) [soundprecache] 3761:)weapons\sniper_railgun_single_02.wav : 72 bits ( 9.000 bytes) [soundprecache] 3762:)weapons/flame_thrower_bb_start.wav : 232 bits ( 29.000 bytes) [soundprecache] 3763:)weapons/flame_thrower_bb_loop_crit.wav : 128 bits ( 16.000 bytes) [soundprecache] 3764:)weapons/flame_thrower_bb_loop.wav : 56 bits ( 7.000 bytes) [soundprecache] 3765:)weapons/ambassador_shoot.wav : 184 bits ( 23.000 bytes) [soundprecache] 3766:)weapons/ambassador_shoot_crit.wav : 96 bits ( 12.000 bytes) [soundprecache] 3767:)weapons/demo_sword_hit_world1.wav : 224 bits ( 28.000 bytes) [soundprecache] 3768:)weapons/demo_sword_hit_world2.wav : 64 bits ( 8.000 bytes) [soundprecache] 3769:vo/sword_hit01.wav : 158 bits ( 19.750 bytes) [soundprecache] 3770:vo/sword_hit02.wav : 64 bits ( 8.000 bytes) [soundprecache] 3771:vo/sword_hit03.wav : 64 bits ( 8.000 bytes) [soundprecache] 3772:vo/sword_hit04.wav : 64 bits ( 8.000 bytes) [soundprecache] 3773:vo/sword_hit05.wav : 64 bits ( 8.000 bytes) [soundprecache] 3774:vo/sword_hit06.wav
Re: [hlds_linux] netspike.txt
I also have these files, but strangely enough its not on every server. Only a few though. 2012/7/11 daniel jokiaho daniel.joki...@gmail.com yes it looks like debuggin info of some sort. My two servers have those files to and both are 2.0Gb each On 2012-07-11 11:20, Jesse Molina wrote: Interesting. This file has indeed appeared on some servers on a host which I manage. In one case, the file is now 1.3GB in size. I suspect Valve has noticed the same thing I noticed. When the Pyromania update came out, I noticed that my host was using much more bandwidth than usual. There seemed to be a spike happening, intermittently, for limited periods of time. I observed a 30% traffic in/out increase for 30-60 seconds at a time. All clients on all servers on that particular host, and all servers on that host, would suddenly start sending much more data, for no apparent reason. I only have a 10base-T uplink on this host, so I am limited to 10Mbps. All four of the servers were full that day. However, due to the traffic spikes, we had to shut one of them down to prevent exceeding the circuit limitation. This is how the issue first came to my attention. I did a tcpdump of the traffic but whatever the issue was, it was embedded in the game traffic itself and was being represented by all clients on all four servers at the exact same time. If I was to guess, something like an authentication request or item-server related activity was suddenly causing all clients and the server to suddenly transmit much more data. Whatever it was, it would have been something that affected all clients across those four servers on the same host. Valve would have noticed huge spikes like this since ALL of their servers and clients would suddenly start transmitting 20-30% more traffic. I am guessing they are trying to debug the problem by collecting evidence. That's just a guess though. Russell Smith wrote: What is this file? I haven't noticed it before, but it's showing up in my tf folder. Looks like a log of packet information being sent to clients. Why is this showing up now, and how can this logging be disabled? 40602.961200/2703847 Player [Player.X][7][adr:XX.XXX.XXX.**XX:46611] was sent a datagram 32 bits (4.000 bytes) NET_Tick : 70 bits ( 8.750 bytes) [soundprecache] 3745:)weapons/scatter_gun_**double_shoot.wav : 323 bits ( 40.375 bytes) [soundprecache] 3746:)weapons/scatter_gun_**double_shoot_crit.wa : 112 bits ( 14.000 bytes) [soundprecache] 3747:)weapons/pickaxe_swing1.**wav : 168 bits ( 21.000 bytes) [soundprecache] 3748:)weapons/pickaxe_swing2.**wav : 64 bits ( 8.000 bytes) [soundprecache] 3749:)weapons/pickaxe_swing3.**wav : 64 bits ( 8.000 bytes) [soundprecache] 3750:weapons/blade_slice_2.wav : 214 bits ( 26.750 bytes) [soundprecache] 3751:weapons/blade_slice_3.wav : 64 bits ( 8.000 bytes) [soundprecache] 3752:weapons/blade_slice_4.wav : 64 bits ( 8.000 bytes) [soundprecache] 3753:)weapons\sniper_railgun_**dry_fire.wav : 248 bits ( 31.000 bytes) [soundprecache] 3754:)weapons\sniper_railgun_**charged_shot_01.w : 176 bits ( 22.000 bytes) [soundprecache] 3755:)weapons\sniper_railgun_**charged_shot_02.w : 120 bits ( 15.000 bytes) [soundprecache] 3756:)weapons\sniper_railgun_**charged_shot_crit : 160 bits ( 20.000 bytes) [soundprecache] 3757:)weapons\sniper_railgun_**charged_shot_crit : 160 bits ( 20.000 bytes) [soundprecache] 3758:)weapons\sniper_railgun_**world_reload.wav : 152 bits ( 19.000 bytes) [soundprecache] 3759:)weapons\sniper_railgun_**no_fire.wav : 112 bits ( 14.000 bytes) [soundprecache] 3760:)weapons\sniper_railgun_**single_01.wav : 128 bits ( 16.000 bytes) [soundprecache] 3761:)weapons\sniper_railgun_**single_02.wav : 72 bits ( 9.000 bytes) [soundprecache] 3762:)weapons/flame_thrower_**bb_start.wav : 232 bits ( 29.000 bytes) [soundprecache] 3763:)weapons/flame_thrower_**bb_loop_crit.wav : 128 bits ( 16.000 bytes) [soundprecache] 3764:)weapons/flame_thrower_**bb_loop.wav : 56 bits ( 7.000 bytes) [soundprecache] 3765:)weapons/ambassador_**shoot.wav : 184 bits ( 23.000 bytes) [soundprecache] 3766:)weapons/ambassador_**shoot_crit.wav : 96 bits ( 12.000 bytes) [soundprecache] 3767:)weapons/demo_sword_hit_**world1.wav : 224 bits ( 28.000 bytes) [soundprecache] 3768:)weapons/demo_sword_hit_**world2.wav : 64 bits ( 8.000 bytes) [soundprecache] 3769:vo/sword_hit01.wav : 158 bits ( 19.750 bytes) [soundprecache] 3770:vo/sword_hit02.wav : 64 bits ( 8.000 bytes) [soundprecache] 3771:vo/sword_hit03.wav : 64 bits (
Re: [hlds_linux] netspike.txt
I have that file on 4 servers from 0.5 to 2.17 gb in filesize. I guess opening and writing to that file could cause the cpu spikes some are having. What is this file for? Can someone from valve enlighten the matter? -ics - Alkuperäinen viesti - yes it looks like debuggin info of some sort. My two servers have those files to and both are 2.0Gb each On 2012-07-11 11:20, Jesse Molina wrote: Interesting. This file has indeed appeared on some servers on a host which I manage. In one case, the file is now 1.3GB in size. I suspect Valve has noticed the same thing I noticed. When the Pyromania update came out, I noticed that my host was using much more bandwidth than usual. There seemed to be a spike happening, intermittently, for limited periods of time. I observed a 30% traffic in/out increase for 30-60 seconds at a time. All clients on all servers on that particular host, and all servers on that host, would suddenly start sending much more data, for no apparent reason. I only have a 10base-T uplink on this host, so I am limited to 10Mbps. All four of the servers were full that day. However, due to the traffic spikes, we had to shut one of them down to prevent exceeding the circuit limitation. This is how the issue first came to my attention. I did a tcpdump of the traffic but whatever the issue was, it was embedded in the game traffic itself and was being represented by all clients on all four servers at the exact same time. If I was to guess, something like an authentication request or item-server related activity was suddenly causing all clients and the server to suddenly transmit much more data. Whatever it was, it would have been something that affected all clients across those four servers on the same host. Valve would have noticed huge spikes like this since ALL of their servers and clients would suddenly start transmitting 20-30% more traffic. I am guessing they are trying to debug the problem by collecting evidence. That's just a guess though. Russell Smith wrote: What is this file? I haven't noticed it before, but it's showing up in my tf folder. Looks like a log of packet information being sent to clients. Why is this showing up now, and how can this logging be disabled? 40602.961200/2703847 Player [Player.X][7][adr:XX.XXX.XXX.XX:46611] was sent a datagram 32 bits (4.000 bytes) NET_Tick : 70 bits ( 8.750 bytes) [soundprecache] 3745:)weapons/scatter_gun_double_shoot.wav : 323 bits ( 40.375 bytes) [soundprecache] 3746:)weapons/scatter_gun_double_shoot_crit.wa : 112 bits ( 14.000 bytes) [soundprecache] 3747:)weapons/pickaxe_swing1.wav : 168 bits ( 21.000 bytes) [soundprecache] 3748:)weapons/pickaxe_swing2.wav : 64 bits ( 8.000 bytes) [soundprecache] 3749:)weapons/pickaxe_swing3.wav : 64 bits ( 8.000 bytes) [soundprecache] 3750:weapons/blade_slice_2.wav : 214 bits ( 26.750 bytes) [soundprecache] 3751:weapons/blade_slice_3.wav : 64 bits ( 8.000 bytes) [soundprecache] 3752:weapons/blade_slice_4.wav : 64 bits ( 8.000 bytes) [soundprecache] 3753:)weapons\sniper_railgun_dry_fire.wav : 248 bits ( 31.000 bytes) [soundprecache] 3754:)weapons\sniper_railgun_charged_shot_01.w : 176 bits ( 22.000 bytes) [soundprecache] 3755:)weapons\sniper_railgun_charged_shot_02.w : 120 bits ( 15.000 bytes) [soundprecache] 3756:)weapons\sniper_railgun_charged_shot_crit : 160 bits ( 20.000 bytes) [soundprecache] 3757:)weapons\sniper_railgun_charged_shot_crit : 160 bits ( 20.000 bytes) [soundprecache] 3758:)weapons\sniper_railgun_world_reload.wav : 152 bits ( 19.000 bytes) [soundprecache] 3759:)weapons\sniper_railgun_no_fire.wav : 112 bits ( 14.000 bytes) [soundprecache] 3760:)weapons\sniper_railgun_single_01.wav : 128 bits ( 16.000 bytes) [soundprecache] 3761:)weapons\sniper_railgun_single_02.wav : 72 bits ( 9.000 bytes) [soundprecache] 3762:)weapons/flame_thrower_bb_start.wav : 232 bits ( 29.000 bytes) [soundprecache] 3763:)weapons/flame_thrower_bb_loop_crit.wav : 128 bits ( 16.000 bytes) [soundprecache] 3764:)weapons/flame_thrower_bb_loop.wav : 56 bits ( 7.000 bytes) [soundprecache] 3765:)weapons/ambassador_shoot.wav : 184 bits ( 23.000 bytes) [soundprecache] 3766:)weapons/ambassador_shoot_crit.wav : 96 bits ( 12.000 bytes) [soundprecache] 3767:)weapons/demo_sword_hit_world1.wav : 224 bits ( 28.000 bytes) [soundprecache] 3768:)weapons/demo_sword_hit_world2.wav : 64 bits ( 8.000 bytes) [soundprecache] 3769:vo/sword_hit01.wav : 158 bits ( 19.750 bytes) [soundprecache] 3770:vo/sword_hit02.wav : 64 bits ( 8.000 bytes) [soundprecache] 3771:vo/sword_hit03.wav : 64 bits ( 8.000 bytes)
Re: [hlds_linux] netspike.txt
i only have it on my registered servers with quickplay. on my not registered servers there´s no netspike.txt. -moss Am 11.07.2012 12:47, schrieb ics: I have that file on 4 servers from 0.5 to 2.17 gb in filesize. I guess opening and writing to that file could cause the cpu spikes some are having. What is this file for? Can someone from valve enlighten the matter? -ics - Alkuperäinen viesti - yes it looks like debuggin info of some sort. My two servers have those files to and both are 2.0Gb each On 2012-07-11 11:20, Jesse Molina wrote: Interesting. This file has indeed appeared on some servers on a host which I manage. In one case, the file is now 1.3GB in size. I suspect Valve has noticed the same thing I noticed. When the Pyromania update came out, I noticed that my host was using much more bandwidth than usual. There seemed to be a spike happening, intermittently, for limited periods of time. I observed a 30% traffic in/out increase for 30-60 seconds at a time. All clients on all servers on that particular host, and all servers on that host, would suddenly start sending much more data, for no apparent reason. I only have a 10base-T uplink on this host, so I am limited to 10Mbps. All four of the servers were full that day. However, due to the traffic spikes, we had to shut one of them down to prevent exceeding the circuit limitation. This is how the issue first came to my attention. I did a tcpdump of the traffic but whatever the issue was, it was embedded in the game traffic itself and was being represented by all clients on all four servers at the exact same time. If I was to guess, something like an authentication request or item-server related activity was suddenly causing all clients and the server to suddenly transmit much more data. Whatever it was, it would have been something that affected all clients across those four servers on the same host. Valve would have noticed huge spikes like this since ALL of their servers and clients would suddenly start transmitting 20-30% more traffic. I am guessing they are trying to debug the problem by collecting evidence. That's just a guess though. Russell Smith wrote: What is this file? I haven't noticed it before, but it's showing up in my tf folder. Looks like a log of packet information being sent to clients. Why is this showing up now, and how can this logging be disabled? 40602.961200/2703847 Player [Player.X][7][adr:XX.XXX.XXX.XX:46611] was sent a datagram 32 bits (4.000 bytes) NET_Tick : 70 bits ( 8.750 bytes) [soundprecache] 3745:)weapons/scatter_gun_double_shoot.wav : 323 bits ( 40.375 bytes) [soundprecache] 3746:)weapons/scatter_gun_double_shoot_crit.wa : 112 bits ( 14.000 bytes) [soundprecache] 3747:)weapons/pickaxe_swing1.wav : 168 bits ( 21.000 bytes) [soundprecache] 3748:)weapons/pickaxe_swing2.wav : 64 bits ( 8.000 bytes) [soundprecache] 3749:)weapons/pickaxe_swing3.wav : 64 bits ( 8.000 bytes) [soundprecache] 3750:weapons/blade_slice_2.wav : 214 bits ( 26.750 bytes) [soundprecache] 3751:weapons/blade_slice_3.wav : 64 bits ( 8.000 bytes) [soundprecache] 3752:weapons/blade_slice_4.wav : 64 bits ( 8.000 bytes) [soundprecache] 3753:)weapons\sniper_railgun_dry_fire.wav : 248 bits ( 31.000 bytes) [soundprecache] 3754:)weapons\sniper_railgun_charged_shot_01.w : 176 bits ( 22.000 bytes) [soundprecache] 3755:)weapons\sniper_railgun_charged_shot_02.w : 120 bits ( 15.000 bytes) [soundprecache] 3756:)weapons\sniper_railgun_charged_shot_crit : 160 bits ( 20.000 bytes) [soundprecache] 3757:)weapons\sniper_railgun_charged_shot_crit : 160 bits ( 20.000 bytes) [soundprecache] 3758:)weapons\sniper_railgun_world_reload.wav : 152 bits ( 19.000 bytes) [soundprecache] 3759:)weapons\sniper_railgun_no_fire.wav : 112 bits ( 14.000 bytes) [soundprecache] 3760:)weapons\sniper_railgun_single_01.wav : 128 bits ( 16.000 bytes) [soundprecache] 3761:)weapons\sniper_railgun_single_02.wav : 72 bits ( 9.000 bytes) [soundprecache] 3762:)weapons/flame_thrower_bb_start.wav : 232 bits ( 29.000 bytes) [soundprecache] 3763:)weapons/flame_thrower_bb_loop_crit.wav : 128 bits ( 16.000 bytes) [soundprecache] 3764:)weapons/flame_thrower_bb_loop.wav : 56 bits ( 7.000 bytes) [soundprecache] 3765:)weapons/ambassador_shoot.wav : 184 bits ( 23.000 bytes) [soundprecache] 3766:)weapons/ambassador_shoot_crit.wav : 96 bits ( 12.000 bytes) [soundprecache] 3767:)weapons/demo_sword_hit_world1.wav : 224 bits ( 28.000 bytes) [soundprecache] 3768:)weapons/demo_sword_hit_world2.wav : 64 bits ( 8.000 bytes) [soundprecache] 3769:vo/sword_hit01.wav : 158 bits ( 19.750 bytes) [soundprecache] 3770:vo/sword_hit02.wav : 64 bits ( 8.000 bytes) [soundprecache] 3771:vo/sword_hit03.wav : 64 bits ( 8.000 bytes) [soundprecache] 3772:vo/sword_hit04.wav : 64 bits ( 8.000 bytes) [soundprecache] 3773:vo/sword_hit05.wav : 64 bits ( 8.000 bytes) [soundprecache]
Re: [hlds_linux] netspike.txt
The netspike file has long existed in the Source engine as a way to track which entities were causing large spikes of network traffic. There's a convar you can use to configure a threshold when it will get dumped. (I think by default it's off.) Recently there was a change where, if a client was dropped to the infamous snapshot overflow error, it would re-run the network packing code for that client, forcing the netspike file to be dumped, to help figure out what entities are the cause of the problem. That could definitely cause CPU spikes. I didn't realize we had shipped with that debugging mode on. We'll make it configurable by convar for the next release. In the meantime, each time it dumps a network trace, it means you dropped a client. It needed to send them a full snapshot of the game state, but the snapshot was too big, so it had to drop the client. You might want to see which entities are responsible for so much traffic, because you are dropping players. If there's something particular to your server, this trace will help you tune it. If you find a problem you think is global to the game, please report it here. Your humble servant, Fletch -Original Message- From: hlds_linux-boun...@list.valvesoftware.com [mailto:hlds_linux-boun...@list.valvesoftware.com] On Behalf Of Thorsten Knoll Sent: Wednesday, July 11, 2012 8:39 AM To: ics; Half-Life dedicated Linux server mailing list Subject: Re: [hlds_linux] netspike.txt i only have it on my registered servers with quickplay. on my not registered servers there´s no netspike.txt. -moss Am 11.07.2012 12:47, schrieb ics: I have that file on 4 servers from 0.5 to 2.17 gb in filesize. I guess opening and writing to that file could cause the cpu spikes some are having. What is this file for? Can someone from valve enlighten the matter? -ics - Alkuperäinen viesti - yes it looks like debuggin info of some sort. My two servers have those files to and both are 2.0Gb each On 2012-07-11 11:20, Jesse Molina wrote: Interesting. This file has indeed appeared on some servers on a host which I manage. In one case, the file is now 1.3GB in size. I suspect Valve has noticed the same thing I noticed. When the Pyromania update came out, I noticed that my host was using much more bandwidth than usual. There seemed to be a spike happening, intermittently, for limited periods of time. I observed a 30% traffic in/out increase for 30-60 seconds at a time. All clients on all servers on that particular host, and all servers on that host, would suddenly start sending much more data, for no apparent reason. I only have a 10base-T uplink on this host, so I am limited to 10Mbps. All four of the servers were full that day. However, due to the traffic spikes, we had to shut one of them down to prevent exceeding the circuit limitation. This is how the issue first came to my attention. I did a tcpdump of the traffic but whatever the issue was, it was embedded in the game traffic itself and was being represented by all clients on all four servers at the exact same time. If I was to guess, something like an authentication request or item-server related activity was suddenly causing all clients and the server to suddenly transmit much more data. Whatever it was, it would have been something that affected all clients across those four servers on the same host. Valve would have noticed huge spikes like this since ALL of their servers and clients would suddenly start transmitting 20-30% more traffic. I am guessing they are trying to debug the problem by collecting evidence. That's just a guess though. Russell Smith wrote: What is this file? I haven't noticed it before, but it's showing up in my tf folder. Looks like a log of packet information being sent to clients. Why is this showing up now, and how can this logging be disabled? 40602.961200/2703847 Player [Player.X][7][adr:XX.XXX.XXX.XX:46611] was sent a datagram 32 bits (4.000 bytes) NET_Tick : 70 bits ( 8.750 bytes) [soundprecache] 3745:)weapons/scatter_gun_double_shoot.wav : 323 bits ( 40.375 bytes) [soundprecache] 3746:)weapons/scatter_gun_double_shoot_crit.wa : 112 bits ( 14.000 bytes) [soundprecache] 3747:)weapons/pickaxe_swing1.wav : 168 bits ( 21.000 bytes) [soundprecache] 3748:)weapons/pickaxe_swing2.wav : 64 bits ( 8.000 bytes) [soundprecache] 3749:)weapons/pickaxe_swing3.wav : 64 bits ( 8.000 bytes) [soundprecache] 3750:weapons/blade_slice_2.wav : 214 bits ( 26.750 bytes) [soundprecache] 3751:weapons/blade_slice_3.wav : 64 bits ( 8.000 bytes) [soundprecache] 3752:weapons/blade_slice_4.wav : 64 bits ( 8.000 bytes) [soundprecache] 3753:)weapons\sniper_railgun_dry_fire.wav : 248 bits ( 31.000 bytes) [soundprecache] 3754:)weapons\sniper_railgun_charged_shot_01.w : 176 bits ( 22.000 bytes) [soundprecache] 3755:)weapons
Re: [hlds_linux] netspike.txt
When you update, put in the notes the cvars or whatever you can setup for those option. Since I only see it on a couple of servers (saxton servers mostly, but hey I can understand) its actually a nice feedback to know that clients are having issues and it could be related too a plugin too. -Original Message- From: hlds_linux-boun...@list.valvesoftware.com [mailto:hlds_linux-boun...@list.valvesoftware.com] On Behalf Of Fletcher Dunn Sent: woensdag 11 juli 2012 18:58 To: Half-Life dedicated Linux server mailing list; ics Subject: Re: [hlds_linux] netspike.txt The netspike file has long existed in the Source engine as a way to track which entities were causing large spikes of network traffic. There's a convar you can use to configure a threshold when it will get dumped. (I think by default it's off.) Recently there was a change where, if a client was dropped to the infamous snapshot overflow error, it would re-run the network packing code for that client, forcing the netspike file to be dumped, to help figure out what entities are the cause of the problem. That could definitely cause CPU spikes. I didn't realize we had shipped with that debugging mode on. We'll make it configurable by convar for the next release. In the meantime, each time it dumps a network trace, it means you dropped a client. It needed to send them a full snapshot of the game state, but the snapshot was too big, so it had to drop the client. You might want to see which entities are responsible for so much traffic, because you are dropping players. If there's something particular to your server, this trace will help you tune it. If you find a problem you think is global to the game, please report it here. Your humble servant, Fletch -Original Message- From: hlds_linux-boun...@list.valvesoftware.com [mailto:hlds_linux-boun...@list.valvesoftware.com] On Behalf Of Thorsten Knoll Sent: Wednesday, July 11, 2012 8:39 AM To: ics; Half-Life dedicated Linux server mailing list Subject: Re: [hlds_linux] netspike.txt i only have it on my registered servers with quickplay. on my not registered servers there´s no netspike.txt. -moss Am 11.07.2012 12:47, schrieb ics: I have that file on 4 servers from 0.5 to 2.17 gb in filesize. I guess opening and writing to that file could cause the cpu spikes some are having. What is this file for? Can someone from valve enlighten the matter? -ics - Alkuperäinen viesti - yes it looks like debuggin info of some sort. My two servers have those files to and both are 2.0Gb each On 2012-07-11 11:20, Jesse Molina wrote: Interesting. This file has indeed appeared on some servers on a host which I manage. In one case, the file is now 1.3GB in size. I suspect Valve has noticed the same thing I noticed. When the Pyromania update came out, I noticed that my host was using much more bandwidth than usual. There seemed to be a spike happening, intermittently, for limited periods of time. I observed a 30% traffic in/out increase for 30-60 seconds at a time. All clients on all servers on that particular host, and all servers on that host, would suddenly start sending much more data, for no apparent reason. I only have a 10base-T uplink on this host, so I am limited to 10Mbps. All four of the servers were full that day. However, due to the traffic spikes, we had to shut one of them down to prevent exceeding the circuit limitation. This is how the issue first came to my attention. I did a tcpdump of the traffic but whatever the issue was, it was embedded in the game traffic itself and was being represented by all clients on all four servers at the exact same time. If I was to guess, something like an authentication request or item-server related activity was suddenly causing all clients and the server to suddenly transmit much more data. Whatever it was, it would have been something that affected all clients across those four servers on the same host. Valve would have noticed huge spikes like this since ALL of their servers and clients would suddenly start transmitting 20-30% more traffic. I am guessing they are trying to debug the problem by collecting evidence. That's just a guess though. Russell Smith wrote: What is this file? I haven't noticed it before, but it's showing up in my tf folder. Looks like a log of packet information being sent to clients. Why is this showing up now, and how can this logging be disabled? 40602.961200/2703847 Player [Player.X][7][adr:XX.XXX.XXX.XX:46611] was sent a datagram 32 bits (4.000 bytes) NET_Tick : 70 bits ( 8.750 bytes) [soundprecache] 3745:)weapons/scatter_gun_double_shoot.wav : 323 bits ( 40.375 bytes) [soundprecache] 3746:)weapons/scatter_gun_double_shoot_crit.wa : 112 bits ( 14.000 bytes) [soundprecache] 3747:)weapons/pickaxe_swing1.wav : 168 bits ( 21.000 bytes) [soundprecache] 3748:)weapons/pickaxe_swing2.wav : 64 bits
Re: [hlds_linux] netspike.txt
Ah, if you are seeing it mostly on the saxton servers, then that could be a clue. There's could be some big array or object or something in the saxton entities being networked inefficiently. The netspike file should make it obvious where the bandwidth is going. But if you need some help analyzing it, email it to me directly. -Original Message- From: hlds_linux-boun...@list.valvesoftware.com [mailto:hlds_linux-boun...@list.valvesoftware.com] On Behalf Of Erik-jan Riemers Sent: Wednesday, July 11, 2012 10:30 AM To: Half-Life dedicated Linux server mailing list Subject: Re: [hlds_linux] netspike.txt When you update, put in the notes the cvars or whatever you can setup for those option. Since I only see it on a couple of servers (saxton servers mostly, but hey I can understand) its actually a nice feedback to know that clients are having issues and it could be related too a plugin too. -Original Message- From: hlds_linux-boun...@list.valvesoftware.com [mailto:hlds_linux-boun...@list.valvesoftware.com] On Behalf Of Fletcher Dunn Sent: woensdag 11 juli 2012 18:58 To: Half-Life dedicated Linux server mailing list; ics Subject: Re: [hlds_linux] netspike.txt The netspike file has long existed in the Source engine as a way to track which entities were causing large spikes of network traffic. There's a convar you can use to configure a threshold when it will get dumped. (I think by default it's off.) Recently there was a change where, if a client was dropped to the infamous snapshot overflow error, it would re-run the network packing code for that client, forcing the netspike file to be dumped, to help figure out what entities are the cause of the problem. That could definitely cause CPU spikes. I didn't realize we had shipped with that debugging mode on. We'll make it configurable by convar for the next release. In the meantime, each time it dumps a network trace, it means you dropped a client. It needed to send them a full snapshot of the game state, but the snapshot was too big, so it had to drop the client. You might want to see which entities are responsible for so much traffic, because you are dropping players. If there's something particular to your server, this trace will help you tune it. If you find a problem you think is global to the game, please report it here. Your humble servant, Fletch -Original Message- From: hlds_linux-boun...@list.valvesoftware.com [mailto:hlds_linux-boun...@list.valvesoftware.com] On Behalf Of Thorsten Knoll Sent: Wednesday, July 11, 2012 8:39 AM To: ics; Half-Life dedicated Linux server mailing list Subject: Re: [hlds_linux] netspike.txt i only have it on my registered servers with quickplay. on my not registered servers there´s no netspike.txt. -moss Am 11.07.2012 12:47, schrieb ics: I have that file on 4 servers from 0.5 to 2.17 gb in filesize. I guess opening and writing to that file could cause the cpu spikes some are having. What is this file for? Can someone from valve enlighten the matter? -ics - Alkuperäinen viesti - yes it looks like debuggin info of some sort. My two servers have those files to and both are 2.0Gb each On 2012-07-11 11:20, Jesse Molina wrote: Interesting. This file has indeed appeared on some servers on a host which I manage. In one case, the file is now 1.3GB in size. I suspect Valve has noticed the same thing I noticed. When the Pyromania update came out, I noticed that my host was using much more bandwidth than usual. There seemed to be a spike happening, intermittently, for limited periods of time. I observed a 30% traffic in/out increase for 30-60 seconds at a time. All clients on all servers on that particular host, and all servers on that host, would suddenly start sending much more data, for no apparent reason. I only have a 10base-T uplink on this host, so I am limited to 10Mbps. All four of the servers were full that day. However, due to the traffic spikes, we had to shut one of them down to prevent exceeding the circuit limitation. This is how the issue first came to my attention. I did a tcpdump of the traffic but whatever the issue was, it was embedded in the game traffic itself and was being represented by all clients on all four servers at the exact same time. If I was to guess, something like an authentication request or item-server related activity was suddenly causing all clients and the server to suddenly transmit much more data. Whatever it was, it would have been something that affected all clients across those four servers on the same host. Valve would have noticed huge spikes like this since ALL of their servers and clients would suddenly start transmitting 20-30% more traffic. I am guessing they are trying to debug the problem by collecting evidence. That's just a guess though. Russell Smith wrote: What is this file? I haven't noticed it before, but it's showing up
Re: [hlds_linux] netspike.txt
I don't know about others but i do have issue where a player downloads a custom map and gets dropped off from a full server with a snapshot overflow. It doesn't have to be larger than 24 slot server. I myself get that with each download of a new map for the most of the time. My PC has done downloading the map (any map will do) and it will load and load (i hear the hdd scream in misery) and at the end, i just get snapshot overlow. Other than that, i have no issues for dropped players that i know of, unless they get dropped by something causing them to drop within the game itself. I know some maps, especially badly made custom ones can cause issues like this but most cases that map is bloated with too many entities already or too many entities being active on the same area while rockets and other bits are spawning out. However i run 2 completely vanilla servers for TF2 and one of those, the most popular server i had, had the largest file to date (2,56GB) and i just had to let those files go so i cannot dug in to those but i'm sure i'll get more data. -ics 11.7.2012 19:58, Fletcher Dunn kirjoitti: The netspike file has long existed in the Source engine as a way to track which entities were causing large spikes of network traffic. There's a convar you can use to configure a threshold when it will get dumped. (I think by default it's off.) Recently there was a change where, if a client was dropped to the infamous snapshot overflow error, it would re-run the network packing code for that client, forcing the netspike file to be dumped, to help figure out what entities are the cause of the problem. That could definitely cause CPU spikes. I didn't realize we had shipped with that debugging mode on. We'll make it configurable by convar for the next release. In the meantime, each time it dumps a network trace, it means you dropped a client. It needed to send them a full snapshot of the game state, but the snapshot was too big, so it had to drop the client. You might want to see which entities are responsible for so much traffic, because you are dropping players. If there's something particular to your server, this trace will help you tune it. If you find a problem you think is global to the game, please report it here. Your humble servant, Fletch -Original Message- From: hlds_linux-boun...@list.valvesoftware.com [mailto:hlds_linux-boun...@list.valvesoftware.com] On Behalf Of Thorsten Knoll Sent: Wednesday, July 11, 2012 8:39 AM To: ics; Half-Life dedicated Linux server mailing list Subject: Re: [hlds_linux] netspike.txt i only have it on my registered servers with quickplay. on my not registered servers there´s no netspike.txt. -moss Am 11.07.2012 12:47, schrieb ics: I have that file on 4 servers from 0.5 to 2.17 gb in filesize. I guess opening and writing to that file could cause the cpu spikes some are having. What is this file for? Can someone from valve enlighten the matter? -ics - Alkuperäinen viesti - yes it looks like debuggin info of some sort. My two servers have those files to and both are 2.0Gb each On 2012-07-11 11:20, Jesse Molina wrote: Interesting. This file has indeed appeared on some servers on a host which I manage. In one case, the file is now 1.3GB in size. I suspect Valve has noticed the same thing I noticed. When the Pyromania update came out, I noticed that my host was using much more bandwidth than usual. There seemed to be a spike happening, intermittently, for limited periods of time. I observed a 30% traffic in/out increase for 30-60 seconds at a time. All clients on all servers on that particular host, and all servers on that host, would suddenly start sending much more data, for no apparent reason. I only have a 10base-T uplink on this host, so I am limited to 10Mbps. All four of the servers were full that day. However, due to the traffic spikes, we had to shut one of them down to prevent exceeding the circuit limitation. This is how the issue first came to my attention. I did a tcpdump of the traffic but whatever the issue was, it was embedded in the game traffic itself and was being represented by all clients on all four servers at the exact same time. If I was to guess, something like an authentication request or item-server related activity was suddenly causing all clients and the server to suddenly transmit much more data. Whatever it was, it would have been something that affected all clients across those four servers on the same host. Valve would have noticed huge spikes like this since ALL of their servers and clients would suddenly start transmitting 20-30% more traffic. I am guessing they are trying to debug the problem by collecting evidence. That's just a guess though. Russell Smith wrote: What is this file? I haven't noticed it before, but it's showing up in my tf folder. Looks like a log of packet information being sent to clients. Why is this showing up now, and how
Re: [hlds_linux] netspike.txt
Fletcher, I have been seeing CPU spikes since the last few updates. I had some spikes last night that didn't coincide with anything being dumped to netspike.txt. Is it possible this is still related to debug information being collected, but not dumped to the file? My netspike.txt files aren't particularly large right now as I just did a clean install Monday night on both my servers. Regards, Russell On 11.07.2012 09:58, Fletcher Dunn wrote: The netspike file has long existed in the Source engine as a way to track which entities were causing large spikes of network traffic. There's a convar you can use to configure a threshold when it will get dumped. (I think by default it's off.) Recently there was a change where, if a client was dropped to the infamous snapshot overflow error, it would re-run the network packing code for that client, forcing the netspike file to be dumped, to help figure out what entities are the cause of the problem. That could definitely cause CPU spikes. I didn't realize we had shipped with that debugging mode on. We'll make it configurable by convar for the next release. In the meantime, each time it dumps a network trace, it means you dropped a client. It needed to send them a full snapshot of the game state, but the snapshot was too big, so it had to drop the client. You might want to see which entities are responsible for so much traffic, because you are dropping players. If there's something particular to your server, this trace will help you tune it. If you find a problem you think is global to the game, please report it here. Your humble servant, Fletch ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux
Re: [hlds_linux] netspike.txt
From what I see even 4 bytes are not enough these days :( The debug code is broken because for some players it starts to log out every packet, not only when an overflow occurs. I have two servers with a 2GB netspike.txt file, both being filled with multiple messages about one player. Some logged datagrams are small, even less than 100 bytes. I assume that when this happens the server is almost killed. Both files have exactly 2147483647 bytes and last logged datagram is truncated. Something like: 114346.040083/7609262 Player [name][16][adr:1.2.3.4:6] was sent a datagram 319697 bits (39962.125 bytes) 114351.395064/7609619 Player [name][16][adr:1.2.3.4:6] was sent a datagram 21340 bits (2667.500 bytes) 114351.590024/7609632 Player [name][16][adr:1.2.3.4:6] was sent a datagram 20041 bits (2505.125 bytes) 114351.770071/7609644 Player [name][16][adr:1.2.3.4:6] was sent a datagram 18259 bits (2282.375 bytes) 114351.875036/7609651 Player [name][16][adr:1.2.3.4:6] was sent a datagram 17725 bits (2215.625 bytes) 114351.965024/7609657 Player [name][16][adr:1.2.3.4:6] was sent a datagram 17213 bits (2151.625 bytes) 114352.055024/7609663 Player [name][16][adr:1.2.3.4:6] was sent a datagram 1601 bits ( 200.125 bytes) 114352.100024/7609666 Player [name][16][adr:1.2.3.4:6] was sent a datagram 2274 bits ( 284.250 bytes) etc... -Original Message- From: hlds_linux-boun...@list.valvesoftware.com [mailto:hlds_linux-boun...@list.valvesoftware.com] On Behalf Of Fletcher Dunn Sent: Wednesday, July 11, 2012 7:58 PM To: Half-Life dedicated Linux server mailing list; ics Subject: Re: [hlds_linux] netspike.txt The netspike file has long existed in the Source engine as a way to track which entities were causing large spikes of network traffic. There's a convar you can use to configure a threshold when it will get dumped. (I think by default it's off.) Recently there was a change where, if a client was dropped to the infamous snapshot overflow error, it would re-run the network packing code for that client, forcing the netspike file to be dumped, to help figure out what entities are the cause of the problem. That could definitely cause CPU spikes. I didn't realize we had shipped with that debugging mode on. We'll make it configurable by convar for the next release. In the meantime, each time it dumps a network trace, it means you dropped a client. It needed to send them a full snapshot of the game state, but the snapshot was too big, so it had to drop the client. You might want to see which entities are responsible for so much traffic, because you are dropping players. If there's something particular to your server, this trace will help you tune it. If you find a problem you think is global to the game, please report it here. Your humble servant, Fletch ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux
Re: [hlds_linux] netspike.txt
I think i'm seeing a pattern which causes the overflows and there's nothing *I* can do about these things. This is just 3 clients from 2 different servers but i suspect the rest will be showing the same stuff. The rest of the data is just plain small compared to these on those snapshots. StringTable soundprecache : 8257 bits (1032.125 bytes) StringTable instancebaseline : 160672 bits (20084.000 bytes) StringTable userinfo :11919 bits (1489.875 bytes) StringTable DynamicModels : 7128 bits ( 891.000 bytes) Total Delta : 130135 bits (16266.875 bytes) StringTable soundprecache :11173 bits (1396.625 bytes) StringTable instancebaseline : 192452 bits (24056.500 bytes) StringTable userinfo : 7610 bits ( 951.250 bytes) StringTable DynamicModels : 8030 bits (1003.750 bytes) Total Delta : 100317 bits (12539.625 bytes) StringTable soundprecache :10887 bits (1360.875 bytes) StringTable instancebaseline : 176476 bits (22059.500 bytes) StringTable userinfo : 8675 bits (1084.375 bytes) StringTable DynamicModels :11910 bits (1488.750 bytes) Total Delta : 110850 bits (13856.250 bytes) -ics 11.7.2012 20:40, Fletcher Dunn kirjoitti: Ah, if you are seeing it mostly on the saxton servers, then that could be a clue. There's could be some big array or object or something in the saxton entities being networked inefficiently. The netspike file should make it obvious where the bandwidth is going. But if you need some help analyzing it, email it to me directly. -Original Message- From: hlds_linux-boun...@list.valvesoftware.com [mailto:hlds_linux-boun...@list.valvesoftware.com] On Behalf Of Erik-jan Riemers Sent: Wednesday, July 11, 2012 10:30 AM To: Half-Life dedicated Linux server mailing list Subject: Re: [hlds_linux] netspike.txt When you update, put in the notes the cvars or whatever you can setup for those option. Since I only see it on a couple of servers (saxton servers mostly, but hey I can understand) its actually a nice feedback to know that clients are having issues and it could be related too a plugin too. -Original Message- From: hlds_linux-boun...@list.valvesoftware.com [mailto:hlds_linux-boun...@list.valvesoftware.com] On Behalf Of Fletcher Dunn Sent: woensdag 11 juli 2012 18:58 To: Half-Life dedicated Linux server mailing list; ics Subject: Re: [hlds_linux] netspike.txt The netspike file has long existed in the Source engine as a way to track which entities were causing large spikes of network traffic. There's a convar you can use to configure a threshold when it will get dumped. (I think by default it's off.) Recently there was a change where, if a client was dropped to the infamous snapshot overflow error, it would re-run the network packing code for that client, forcing the netspike file to be dumped, to help figure out what entities are the cause of the problem. That could definitely cause CPU spikes. I didn't realize we had shipped with that debugging mode on. We'll make it configurable by convar for the next release. In the meantime, each time it dumps a network trace, it means you dropped a client. It needed to send them a full snapshot of the game state, but the snapshot was too big, so it had to drop the client. You might want to see which entities are responsible for so much traffic, because you are dropping players. If there's something particular to your server, this trace will help you tune it. If you find a problem you think is global to the game, please report it here. Your humble servant, Fletch -Original Message- From: hlds_linux-boun...@list.valvesoftware.com [mailto:hlds_linux-boun...@list.valvesoftware.com] On Behalf Of Thorsten Knoll Sent: Wednesday, July 11, 2012 8:39 AM To: ics; Half-Life dedicated Linux server mailing list Subject: Re: [hlds_linux] netspike.txt i only have it on my registered servers with quickplay. on my not registered servers there´s no netspike.txt. -moss Am 11.07.2012 12:47, schrieb ics: I have that file on 4 servers from 0.5 to 2.17 gb in filesize. I guess opening and writing to that file could cause the cpu spikes some are having. What is this file for? Can someone from valve enlighten the matter? -ics - Alkuperäinen viesti - yes it looks like debuggin info of some sort. My two servers have those files to and both are 2.0Gb each On 2012-07-11 11:20, Jesse Molina wrote: Interesting. This file has indeed appeared on some servers on a host which I manage. In one case, the file is now 1.3GB in size. I suspect Valve has noticed the same thing I noticed. When the Pyromania update came out, I noticed that my host was using much more bandwidth than usual. There seemed to be a spike happening, intermittently, for limited periods of time. I observed a 30% traffic in/out increase for 30-60 seconds at a time. All clients on all servers on that particular host, and all servers on that host, would suddenly start sending much more data
