Shot in the dark here... SELinux maybe? Another shot - if this is a virtualized instance, it may be a hypervisor issue.
Dan -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Christian Rößner Sent: Friday, May 16, 2014 7:05 AM To: [email protected] Cc: [email protected] Subject: Re: [Ntop] Very high load I have recompiled the kernel with automatic grsec configuration->Server->Performance I will give feedback later... Am 16.05.2014 um 11:27 schrieb Christian Rößner <[email protected]>: >> What is high load for you? > > Well, it is not the value 2.x or 3.x in the load avergae. It is that I can > not understand, why CPU load ist near zero, disk IO is low and the system > behaves like being under heaviest load. For example, if I am ssh-ing to the > machine that can take up to a minute until I get a shell prompt. Even > powering the system down with acpi did take endless until I decided to power > off the server hard (which you can see in the RAID controller output). > > And we just speak about 40-50 MBit/s. I thought, ntopng is a userland > application. So if the load comes from decoding packets, why can I not see > this in the CPUs (looking at top). > > The ssh-daemon is running on a separate NIC. > > I played around with IO scheduler CFQ/deadlin. I tuned CONFIG_HZ 250->1000. I > tried out full preempt. No changes. Always the same result. And I can not > understand that this new server with lots of power feels exactly the same as > the old G4. So I think, something is going wrong here. > > I mean there are 8 cores and 8 HT. The disks are SAS and the NICs are > gigabit. And 40-50 MBit bring the server to near unusable state. How can this > be? > > So I come here to the list to report feedback :) > > Maybe it has to do with some GRSec options in the kernel? What could > cause trouble? I have seen that ntopng does anonymous mamory mapping. > Not knowing what this is, but I had to grant access for this with > paxctl. So maybe there are some other options that might conflict and > so I hope you developers might be possible to help me :) > > Thanks a lot in advance > > Christian > >> Remember ntopng will decode every packet at L7 to try to identify the >> protocol, that means It will read the full packet, every packet. >> >> I have ntopng in a very normal server (Intel(R) Xeon(R) CPU X3430 @ 2.40GHz >> and 8Gb RAM), with 1 switch port mirrored to 1 interface on the server with >> arround 400Mbps+ and 53k+ pps and this is my load average. >> >> ntop@ntop:~$ top >> top - 19:35:51 up 101 days, 6:26, 2 users, load average: 3.90, 4.73, 4.49 >> Tasks: 92 total, 3 running, 89 sleeping, 0 stopped, 0 zombie >> Cpu(s): 3.5%us, 1.6%sy, 0.0%ni, 80.1%id, 13.5%wa, 0.1%hi, >> 1.1%si, 0.0%st with those TOP Proccess >> 14564 ntop 20 0 1965m 1.1g 5400 S 42 14.2 61:40.36 ntopng >> 16400 redis 20 0 163m 153m 280 R 26 1.9 0:00.13 redis-server >> 14312 root 20 0 18748 4940 1720 D 6 0.1 4:02.32 dibbler-server >> 1156 redis 20 0 163m 153m 612 R 2 1.9 5888:00 redis-server >> >> This is the normal behaviour and cpu load, at least as far i know. >> >> Regards, >> >> >> El 15/05/2014 18:12, Christian Rößner escribió: >>> Hi, >>> >>> I had installed ntopng from SVN and compiled it on an HP ProLiant G4 server >>> with 6GB RAM and 2 CPUs. Okay, load got high and therefor I decided to by a >>> new machine for that task. Now it is a HP ProLiant SEM316M1 (same as >>> DL160G6) with 24GB RAM. Tripple channel. And I still have too high load on >>> the system. I am totally stuck and frustrated. >>> >>> I have mirrored 3 witch ports and they are connected to a separate port on >>> the server. The totally traffic that arrives on this NIC is no more than >>> 50MBit/s. So this is really less. >>> >>> I have installed irqbalance and numad. The RAID-controller is battery >>> backed up. >>> >>> So currently I have no clue, what is going wrong. As this ntopng is from >>> SVN, I have updated several times. Always doing a "make clean" and >>> ./configure. >>> >>> The system is a Gentoo hardened 3.13.10 kernel. >>> >>> I have done paxctl -cm on the ntopng binary. >>> >>> This is my current ntopng configuration: >>> >>> # ntopng.conf >>> # >>> >>> -n=3 >>> -i=enp2s0f1 >>> -d=/opt/ntopng/data >>> -G=/opt/ntopng/var/run/ntopng.pid >>> -H=0 >>> -w=8080 >>> -W=8443 >>> -m=192.168.2.0/24,193.239.104.0/22,217.199.204.10/32 >>> -p=tcp:216@SSH >>> -q=1 >>> -A=0 >>> -U=ntopng >>> -e >>> >>> I have attached a log.txt, gzip compressed, which should give a good >>> overview of this hardware and current configuration. I really, really was >>> happy, if somewhat could point me to a problem with this server/ntopng. >>> >>> svn info >>> Path: . >>> Working Copy Root Path: /usr/local/src/ntopng >>> URL: >>> https://svn.ntop.org/svn/ntop/trunk/ntopng >>> >>> Repository Root: >>> https://svn.ntop.org/svn/ntop >>> >>> Repository UUID: a02cd0c1-8e76-42e1-a119-56f9641475e2 >>> Revision: 7598 >>> Node Kind: directory >>> Schedule: normal >>> Last Changed Author: deri >>> Last Changed Rev: 7597 >>> Last Changed Date: 2014-05-06 17:12:17 +0200 (Tue, 06 May 2014) >>> >>> Thanks a lot in advance >>> >>> -Christian Rößner >>> >>> >>> >>> >>> >>> -- >>> [*] sys4 AG >>> >>> >>> http://sys4.de >>> , +49 (89) 30 90 46 64 >>> Franziskanerstraße 15, 81669 München >>> >>> Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 >>> Vorstand: Patrick Ben Koetter, Marc Schiffbauer >>> Aufsichtsratsvorsitzender: Florian Kirstein >>> >>> >>> >>> >>> _______________________________________________ >>> Ntop mailing list >>> >>> [email protected] >>> http://listgateway.unipi.it/mailman/listinfo/ntop >> >> -- >> Daniel Baeza >> Centro de Observación de Red >> Dpto. Internet y Telefonía >> Television Costa Blanca S.L. >> Telf. 966190565 >> WEB: >> http://www.tvt.es >> >> Correo: >> [email protected] >> >> >> --AVISO LEGAL-- >> >> En cumplimiento de la Ley Orgánica 15/1999, de 13 de diciembre de protección >> de datos de carácter personal, se pone en conocimiento del destinatario del >> presente correo electrónico, que los datos incluidos en este mensaje, están >> dirigidos exclusivamente al citado destinatario cuyo nombre aparece en el >> encabezamiento, por lo que si usted no es la persona interesada rogamos nos >> comunique el error de envío y se abstenga de realizar copias del mensaje o >> de los datos contenidos en el mismo o remitirlo o entregarlo a otra persona, >> procediendo a borrarlo de inmediato. >> Asimismo le informamos que sus datos de correo han quedado incluidos en >> nuestra base de datos a fin de dirigirle, por este medio, comunicaciones >> comerciales, profesionales e informativas y que usted dispone de los >> derechos de acceso, rectificación, cancelación y especificación de los >> mismos, derechos que podrá hacer efectivos dirigiéndose a Televisión Costa >> Blanca, S.L., C/ San Policarpo 41 Bajo. C.P: 03181 Torrevieja (Alicante). >> >> _______________________________________________ >> Ntop mailing list >> [email protected] >> http://listgateway.unipi.it/mailman/listinfo/ntop > > -Christian Rößner > > -- > [*] sys4 AG > > http://sys4.de, +49 (89) 30 90 46 64 > Franziskanerstraße 15, 81669 München > > Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 > Vorstand: Patrick Ben Koetter, Marc Schiffbauer > Aufsichtsratsvorsitzender: Florian Kirstein > > _______________________________________________ > Ntop mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop -Christian Rößner -- [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein _______________________________________________ Ntop mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop Confidential: This electronic message and all contents contained may be privileged, confidential or otherwise protected from disclosure. The information is intended to be for the addressee only. If you are not the addressee, any disclosure, copy, distribution or use of the contents of this message is prohibited. If you have received this electronic message in error, please notify me immediately by return email and destroy the original message and all copies. _______________________________________________ Ntop mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop
