Thanks Lennart I am using AWS instances: server1 - c3.2xlarge server2 - i2.xlarge
it seems like graylog2-server is working very hard to consume messages from kafka cluster, can I configure rate limit for graylog2-server so it won't take all CPU resources? On Wednesday, May 7, 2014 10:23:59 PM UTC+3, lennart wrote: > > Looks like your systems are just overloaded and you need faster > hardware or a scale out on more machines. > > On Tue, May 6, 2014 at 4:13 PM, Yossi Nachum <[email protected]<javascript:>> > wrote: > > Hi, > > > > I am trying to run the following graylog2 system: > > > > server1: graylog2-server-v0.21 + graylog2-radio-v0.20 + kafka + > graylog2-web > > > > server2: elasticsearch > > > > when I am sending a lot of log messages (~20K per second) the lag in the > > kafka server start to increase and then I get the following messages in > > zookeeper log: > > > > [2014-05-05 17:27:13,144] INFO Accepted socket connection from > > /127.0.0.1:38581(org.apache.zookeeper.server.NIOServerCnxn) > > > > [2014-05-05 17:27:13,144] INFO Accepted socket connection from > > /127.0.0.1:38582(org.apache.zookeeper.server.NIOServerCnxn) > > > > [2014-05-05 17:27:13,144] INFO Client attempting to renew session > > 0x145ccf8c9a00174 at > > /127.0.0.1:38582(org.apache.zookeeper.server.NIOServerCnxn) > > > > [2014-05-05 17:27:13,145] INFO Invalid session 0x145ccf8c9a00174 for > client > > /127.0.0.1:38582, probably expired > > (org.apache.zookeeper.server.NIOServerCnxn) > > > > [2014-05-05 17:27:13,145] INFO Closed socket connection for client > > /127.0.0.1:38582 which had sessionid 0x145ccf8c9a00174 > > (org.apache.zookeeper.server.NIOServerCnxn) > > > > [2014-05-05 17:27:13,148] INFO Client attempting to renew session > > 0x145ccf8c9a00175 at > > /127.0.0.1:38581(org.apache.zookeeper.server.NIOServerCnxn) > > > > [2014-05-05 17:27:13,148] INFO Invalid session 0x145ccf8c9a00175 for > client > > /127.0.0.1:38581, probably expired > > (org.apache.zookeeper.server.NIOServerCnxn) > > > > [2014-05-05 17:27:13,148] INFO Closed socket connection for client > > /127.0.0.1:38581 which had sessionid 0x145ccf8c9a00175 > > (org.apache.zookeeper.server.NIOServerCnxn) > > > > [2014-05-05 17:27:13,161] INFO Accepted socket connection from > > /127.0.0.1:38586(org.apache.zookeeper.server.NIOServerCnxn) > > > > [2014-05-05 17:27:13,172] INFO Client attempting to establish new > session at > > /127.0.0.1:38586(org.apache.zookeeper.server.NIOServerCnxn) > > > > [2014-05-05 17:27:13,173] INFO Accepted socket connection from > > /127.0.0.1:38588(org.apache.zookeeper.server.NIOServerCnxn) > > > > [2014-05-05 17:27:13,174] INFO Client attempting to establish new > session at > > /127.0.0.1:38588(org.apache.zookeeper.server.NIOServerCnxn) > > > > [2014-05-05 17:27:13,183] INFO Established session 0x145ccf8c9a00176 > with > > negotiated timeout 6000 for client /127.0.0.1:38586 > > (org.apache.zookeeper.server.NIOServerCnxn) > > > > [2014-05-05 17:27:13,184] INFO Established session 0x145ccf8c9a00177 > with > > negotiated timeout 6000 for client /127.0.0.1:38588 > > (org.apache.zookeeper.server.NIOServerCnxn) > > > > [2014-05-05 17:27:21,000] INFO Expiring session 0x145ccf8c9a00176, > timeout > > of 6000ms exceeded (org.apache.zookeeper.server.ZooKeeperServer) > > > > [2014-05-05 17:27:21,000] INFO Expiring session 0x145ccf8c9a00177, > timeout > > of 6000ms exceeded (org.apache.zookeeper.server.ZooKeeperServer) > > > > [2014-05-05 17:27:21,001] INFO Processed session termination for > sessionid: > > 0x145ccf8c9a00176 (org.apache.zookeeper.server.PrepRequestProcessor) > > > > [2014-05-05 17:27:21,001] INFO Processed session termination for > sessionid: > > 0x145ccf8c9a00177 (org.apache.zookeeper.server.PrepRequestProcessor) > > > > [2014-05-05 17:27:21,002] INFO Closed socket connection for client > > /127.0.0.1:38586 which had sessionid 0x145ccf8c9a00176 > > (org.apache.zookeeper.server.NIOServerCnxn) > > > > [2014-05-05 17:27:21,004] INFO Closed socket connection for client > > /127.0.0.1:38588 which had sessionid 0x145ccf8c9a00177 > > (org.apache.zookeeper.server.NIOServerCnxn) > > > > [2014-05-05 17:27:56,146] INFO Accepted socket connection from > > /127.0.0.1:38760(org.apache.zookeeper.server.NIOServerCnxn) > > > > [2014-05-05 17:27:56,146] INFO Accepted socket connection from > > /127.0.0.1:38761(org.apache.zookeeper.server.NIOServerCnxn) > > > > [2014-05-05 17:27:56,146] INFO Client attempting to renew session > > 0x145ccf8c9a00176 at > > /127.0.0.1:38760(org.apache.zookeeper.server.NIOServerCnxn) > > > > [2014-05-05 17:27:56,147] INFO Invalid session 0x145ccf8c9a00176 for > client > > /127.0.0.1:38760, probably expired > > (org.apache.zookeeper.server.NIOServerCnxn) > > > > [2014-05-05 17:27:56,147] INFO Client attempting to renew session > > 0x145ccf8c9a00177 at > > /127.0.0.1:38761(org.apache.zookeeper.server.NIOServerCnxn) > > > > [2014-05-05 17:27:56,147] INFO Invalid session 0x145ccf8c9a00177 for > client > > /127.0.0.1:38761, probably expired > > (org.apache.zookeeper.server.NIOServerCnxn) > > > > [2014-05-05 17:27:56,148] INFO Closed socket connection for client > > /127.0.0.1:38760 which had sessionid 0x145ccf8c9a00176 > > (org.apache.zookeeper.server.NIOServerCnxn) > > > > [2014-05-05 17:27:56,148] INFO Closed socket connection for client > > /127.0.0.1:38761 which had sessionid 0x145ccf8c9a00177 > > (org.apache.zookeeper.server.NIOServerCnxn) > > > > [2014-05-05 17:27:56,151] INFO Accepted socket connection from > > /127.0.0.1:38762(org.apache.zookeeper.server.NIOServerCnxn) > > > > [2014-05-05 17:27:56,173] INFO Client attempting to establish new > session at > > /127.0.0.1:38762(org.apache.zookeeper.server.NIOServerCnxn) > > > > [2014-05-05 17:27:56,176] INFO Accepted socket connection from > > /127.0.0.1:38767(org.apache.zookeeper.server.NIOServerCnxn) > > > > [2014-05-05 17:27:56,176] INFO Established session 0x145ccf8c9a00178 > with > > negotiated timeout 6000 for client /127.0.0.1:38762 > > (org.apache.zookeeper.server.NIOServerCnxn) > > > > [2014-05-05 17:27:56,177] INFO Client attempting to establish new > session at > > /127.0.0.1:38767(org.apache.zookeeper.server.NIOServerCnxn) > > > > [2014-05-05 17:27:56,179] INFO Established session 0x145ccf8c9a00179 > with > > negotiated timeout 6000 for client /127.0.0.1:38767 > > (org.apache.zookeeper.server.NIOServerCnxn) > > > > A lot of timeouts and graylog2-server try to reconnect. > > > > On server1 I see that graylog2-server consume all cpu resources > > > > what can I do? where I can change zookeeper client timeout in > > graylog2-server? > > > > Thanks > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "graylog2" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to [email protected] <javascript:>. > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
