What time does this normally happen? Is it after logrotate has run? Does collector know it needs to re-open the logs after they're rotated? It may be trying to read from the old inodes and doesn't realize it needs to switch.
It is possible to configure logrotate to deal with this on its own without changing the inode. On Fri, Apr 8, 2016 at 11:20 AM, Mirza Dedic <[email protected]> wrote: > Hello, > > We are using graylog-collector v0.4.2 on AIX 7.1 and we can start the > logger and it works for awhile (couple of hours) and then it suddenly stops > sending logs with no error messages in the logs.. > > I also changed the COLLECTOR_JAVA_DEFAULTS to -Xms512m -Xmx768m but no > luck. > > On the graylog-server the collector is visible while it is working, when > it stops sending logs the collector is no longer visible on the server side > but the graylog-collector process is still running in AIX with no errors in > logs. I have to kill the process (kill -9 since kill -15 does not stop it) > and then restart the process. > > Any idea how to troubleshoot the cause of this? > > *The collector.conf consists of this..* > server-url = "http://172.16.x.x:12900"; > enable-registration = true > collector-id = "file:config/collector-id" > > inputs { > 1 { > type = "file" > path = "/usr1/dbs/1.lg" > outputs = "gelf-tcp" > } > 2 { > type = "file" > path = "/usr1/dbs/2.lg" > outputs = "gelf-tcp" > } > 3 { > type = "file" > path = "/usr1/dbs/3.lg" > outputs = "gelf-tcp" > } > 4 { > type = "file" > path = "/usr1/dbs/4.lg" > outputs = "gelf-tcp" > } > 5 { > type = "file" > path = "/usr1/dbs/5.lg" > outputs = "gelf-tcp" > } > 6 { > type = "file" > path = "/usr1/dbs/6.lg" > outputs = "gelf-tcp" > } > 7 { > type = "file" > path = "/usr1/dbs/7.lg" > outputs = "gelf-tcp" > } > 8 { > type = "file" > path = "/usr1/dbs/8.lg" > outputs = "gelf-tcp" > } > } > > outputs { > gelf-tcp { > type = "gelf" > host = "172.16.x.x" > port = 12201 > client-tls = false > client-queue-size = 5120 > client-connect-timeout = 10000 > client-reconnect-delay = 2000 > client-tcp-no-delay = true > client-send-buffer-size = 32768 > } > } > > *In the log all we see is...* > [main] cli.commands.Run - Starting Collector v0.4.2 (commit 2609a38) > [main] cli.commands.Run - Running on AIX AIX 7.1 (ppc64) > [main] collector.utils.CollectorId - Collector ID: > 19c8fecb-0f04-4503-9943-15cfbac98458 > [main] outputs.gelf.GelfOutput - Starting GELF transport: > org.graylog2.gelfclient.GelfConfiguration@3c8510bd > [main] cli.commands.Run - Service RUNNING: BufferProcessor [RUNNING] > [main] cli.commands.Run - Service RUNNING: FileObserver [RUNNING] > [main] cli.commands.Run - Service RUNNING: MetricService [RUNNING] > [main] cli.commands.Run - Service RUNNING: MemoryReporterService [RUNNING] > [main] cli.commands.Run - Service RUNNING: HeartbeatService [RUNNING] > [main] cli.commands.Run - Service RUNNING: FileInput{id='1', > path-set='SinglePathSet{path=/usr1/dbs/1.lg}', reader-buffer-size='102400', > message-fields='MessageFields{}', charset='UTF-8', outputs='gelf-tcp', > content-splitter='NEWLINE', reader-interval='100'} > [main] cli.commands.Run - Service RUNNING: FileInput{id='2', > path-set='SinglePathSet{path=/usr1/dbs/2.lg}', reader-buffer-size='102400', > message-fields='MessageFields{}', charset='UTF-8', outputs='gelf-tcp', > content-splitter='NEWLINE', reader-interval='100'} > [main] cli.commands.Run - Service RUNNING: FileInput{id='3', > path-set='SinglePathSet{path=/usr1/dbs/3.lg}', reader-buffer-size='102400', > message-fields='MessageFields{}', charset='UTF-8', outputs='gelf-tcp', > content-splitter='NEWLINE', reader-interval='100'} > [main] cli.commands.Run - Service RUNNING: FileInput{id='4', > path-set='SinglePathSet{path=/usr1/dbs/4.lg}', reader-buffer-size='102400', > message-fields='MessageFields{}', charset='UTF-8', outputs='gelf-tcp', > content-splitter='NEWLINE', reader-interval='100'} > [main] cli.commands.Run - Service RUNNING: FileInput{id='5', > path-set='SinglePathSet{path=/usr1/dbs/5.lg}', reader-buffer-size='102400', > message-fields='MessageFields{}', charset='UTF-8', outputs='gelf-tcp', > content-splitter='NEWLINE', reader-interval='100'} > [main] cli.commands.Run - Service RUNNING: FileInput{id='6', > path-set='SinglePathSet{path=/usr1/dbs/6.lg}', reader-buffer-size='102400', > message-fields='MessageFields{}', charset='UTF-8', outputs='gelf-tcp', > content-splitter='NEWLINE', reader-interval='100'} > [main] cli.commands.Run - Service RUNNING: GelfOutput{port='12201', > id='gelf-tcp', client-send-buffer-size='32768', host='172.16.x.x', > inputs='', client-reconnect-delay='2000', client-connect-timeout='10000', > client-tcp-no-delay='true', client-queue-size='5120'} > [main] cli.commands.Run - Service RUNNING: FileInput{id='7', > path-set='SinglePathSet{path=/usr1/dbs/7.lg}', reader-buffer-size='102400', > message-fields='MessageFields{}', charset='UTF-8', outputs='gelf-tcp', > content-splitter='NEWLINE', reader-interval='100'} > [main] cli.commands.Run - Service RUNNING: FileInput{id='8', > path-set='SinglePathSet{path=/usr1/dbs/8.lg}', reader-buffer-size='102400', > message-fields='MessageFields{}', charset='UTF-8', outputs='gelf-tcp', > content-splitter='NEWLINE', reader-interval='100'}', reader-interval='100'} > > > -- > You received this message because you are subscribed to the Google Groups > "Graylog Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/graylog2/3806ada3-9eb1-4126-a415-d7c0bf997ae1%40googlegroups.com > <https://groups.google.com/d/msgid/graylog2/3806ada3-9eb1-4126-a415-d7c0bf997ae1%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- No matter what we think of Linux versus FreeBSD, etc., the one thing I really like about Linux is that it has Microsoft worried. Anything that kicks a monopoly in the pants has got to be good for something. - Chris Johnson -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAL5rfGVtRG4WMck8oKzeEfcbEH_7XLA0-e%2BEoFdZZ9oQJBLj9A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
