Looks like your systems are just overloaded and you need faster hardware or a scale out on more machines.
On Tue, May 6, 2014 at 4:13 PM, Yossi Nachum <[email protected]> wrote: > Hi, > > I am trying to run the following graylog2 system: > > server1: graylog2-server-v0.21 + graylog2-radio-v0.20 + kafka + graylog2-web > > server2: elasticsearch > > when I am sending a lot of log messages (~20K per second) the lag in the > kafka server start to increase and then I get the following messages in > zookeeper log: > > [2014-05-05 17:27:13,144] INFO Accepted socket connection from > /127.0.0.1:38581(org.apache.zookeeper.server.NIOServerCnxn) > > [2014-05-05 17:27:13,144] INFO Accepted socket connection from > /127.0.0.1:38582(org.apache.zookeeper.server.NIOServerCnxn) > > [2014-05-05 17:27:13,144] INFO Client attempting to renew session > 0x145ccf8c9a00174 at > /127.0.0.1:38582(org.apache.zookeeper.server.NIOServerCnxn) > > [2014-05-05 17:27:13,145] INFO Invalid session 0x145ccf8c9a00174 for client > /127.0.0.1:38582, probably expired > (org.apache.zookeeper.server.NIOServerCnxn) > > [2014-05-05 17:27:13,145] INFO Closed socket connection for client > /127.0.0.1:38582 which had sessionid 0x145ccf8c9a00174 > (org.apache.zookeeper.server.NIOServerCnxn) > > [2014-05-05 17:27:13,148] INFO Client attempting to renew session > 0x145ccf8c9a00175 at > /127.0.0.1:38581(org.apache.zookeeper.server.NIOServerCnxn) > > [2014-05-05 17:27:13,148] INFO Invalid session 0x145ccf8c9a00175 for client > /127.0.0.1:38581, probably expired > (org.apache.zookeeper.server.NIOServerCnxn) > > [2014-05-05 17:27:13,148] INFO Closed socket connection for client > /127.0.0.1:38581 which had sessionid 0x145ccf8c9a00175 > (org.apache.zookeeper.server.NIOServerCnxn) > > [2014-05-05 17:27:13,161] INFO Accepted socket connection from > /127.0.0.1:38586(org.apache.zookeeper.server.NIOServerCnxn) > > [2014-05-05 17:27:13,172] INFO Client attempting to establish new session at > /127.0.0.1:38586(org.apache.zookeeper.server.NIOServerCnxn) > > [2014-05-05 17:27:13,173] INFO Accepted socket connection from > /127.0.0.1:38588(org.apache.zookeeper.server.NIOServerCnxn) > > [2014-05-05 17:27:13,174] INFO Client attempting to establish new session at > /127.0.0.1:38588(org.apache.zookeeper.server.NIOServerCnxn) > > [2014-05-05 17:27:13,183] INFO Established session 0x145ccf8c9a00176 with > negotiated timeout 6000 for client /127.0.0.1:38586 > (org.apache.zookeeper.server.NIOServerCnxn) > > [2014-05-05 17:27:13,184] INFO Established session 0x145ccf8c9a00177 with > negotiated timeout 6000 for client /127.0.0.1:38588 > (org.apache.zookeeper.server.NIOServerCnxn) > > [2014-05-05 17:27:21,000] INFO Expiring session 0x145ccf8c9a00176, timeout > of 6000ms exceeded (org.apache.zookeeper.server.ZooKeeperServer) > > [2014-05-05 17:27:21,000] INFO Expiring session 0x145ccf8c9a00177, timeout > of 6000ms exceeded (org.apache.zookeeper.server.ZooKeeperServer) > > [2014-05-05 17:27:21,001] INFO Processed session termination for sessionid: > 0x145ccf8c9a00176 (org.apache.zookeeper.server.PrepRequestProcessor) > > [2014-05-05 17:27:21,001] INFO Processed session termination for sessionid: > 0x145ccf8c9a00177 (org.apache.zookeeper.server.PrepRequestProcessor) > > [2014-05-05 17:27:21,002] INFO Closed socket connection for client > /127.0.0.1:38586 which had sessionid 0x145ccf8c9a00176 > (org.apache.zookeeper.server.NIOServerCnxn) > > [2014-05-05 17:27:21,004] INFO Closed socket connection for client > /127.0.0.1:38588 which had sessionid 0x145ccf8c9a00177 > (org.apache.zookeeper.server.NIOServerCnxn) > > [2014-05-05 17:27:56,146] INFO Accepted socket connection from > /127.0.0.1:38760(org.apache.zookeeper.server.NIOServerCnxn) > > [2014-05-05 17:27:56,146] INFO Accepted socket connection from > /127.0.0.1:38761(org.apache.zookeeper.server.NIOServerCnxn) > > [2014-05-05 17:27:56,146] INFO Client attempting to renew session > 0x145ccf8c9a00176 at > /127.0.0.1:38760(org.apache.zookeeper.server.NIOServerCnxn) > > [2014-05-05 17:27:56,147] INFO Invalid session 0x145ccf8c9a00176 for client > /127.0.0.1:38760, probably expired > (org.apache.zookeeper.server.NIOServerCnxn) > > [2014-05-05 17:27:56,147] INFO Client attempting to renew session > 0x145ccf8c9a00177 at > /127.0.0.1:38761(org.apache.zookeeper.server.NIOServerCnxn) > > [2014-05-05 17:27:56,147] INFO Invalid session 0x145ccf8c9a00177 for client > /127.0.0.1:38761, probably expired > (org.apache.zookeeper.server.NIOServerCnxn) > > [2014-05-05 17:27:56,148] INFO Closed socket connection for client > /127.0.0.1:38760 which had sessionid 0x145ccf8c9a00176 > (org.apache.zookeeper.server.NIOServerCnxn) > > [2014-05-05 17:27:56,148] INFO Closed socket connection for client > /127.0.0.1:38761 which had sessionid 0x145ccf8c9a00177 > (org.apache.zookeeper.server.NIOServerCnxn) > > [2014-05-05 17:27:56,151] INFO Accepted socket connection from > /127.0.0.1:38762(org.apache.zookeeper.server.NIOServerCnxn) > > [2014-05-05 17:27:56,173] INFO Client attempting to establish new session at > /127.0.0.1:38762(org.apache.zookeeper.server.NIOServerCnxn) > > [2014-05-05 17:27:56,176] INFO Accepted socket connection from > /127.0.0.1:38767(org.apache.zookeeper.server.NIOServerCnxn) > > [2014-05-05 17:27:56,176] INFO Established session 0x145ccf8c9a00178 with > negotiated timeout 6000 for client /127.0.0.1:38762 > (org.apache.zookeeper.server.NIOServerCnxn) > > [2014-05-05 17:27:56,177] INFO Client attempting to establish new session at > /127.0.0.1:38767(org.apache.zookeeper.server.NIOServerCnxn) > > [2014-05-05 17:27:56,179] INFO Established session 0x145ccf8c9a00179 with > negotiated timeout 6000 for client /127.0.0.1:38767 > (org.apache.zookeeper.server.NIOServerCnxn) > > A lot of timeouts and graylog2-server try to reconnect. > > On server1 I see that graylog2-server consume all cpu resources > > what can I do? where I can change zookeeper client timeout in > graylog2-server? > > Thanks > > -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
