What size are the logs files? Do they stop running when they reach a certain size, or when the collector has transferred a certain amount of data? I'm wondering about bugs in the java from using INTs or something too small to handle the size of the files, 2G or 4G numbers come to mind. Do small files fail as well, or just large ones? Does the service creating those files ever truncate them?
Does your aix host have an lsof command? Can it tell you if java still has the log file open, and the pid? If so, can you strace (or equivalent) on that pid and see what it's doing? Can you tell if the collect is stuck reading from the log, or is it stuck trying to write it to the graylob service? On Mon, Apr 11, 2016 at 9:26 AM, Mirza Dedic <mirza.de...@outlook.com> wrote: > Hi Joi, > > This happens randomly after the graylog-collector has started (usually > within a couple of hours of running). The logs being monitored do not have > any rotation on them (they grow forever) so I don't think it is a rotation > issue. > > Any other ideas to try? I am stuck and I really need the graylog-collector > to run on our AIX box. > > On Friday, April 8, 2016 at 9:35:23 AM UTC-7, Joi Owen wrote: >> >> What time does this normally happen? Is it after logrotate has run? >> Does collector know it needs to re-open the logs after they're rotated? It >> may be trying to read from the old inodes and doesn't realize it needs to >> switch. >> >> It is possible to configure logrotate to deal with this on its own >> without changing the inode. >> >> >> On Fri, Apr 8, 2016 at 11:20 AM, Mirza Dedic <mirza...@outlook.com> >> wrote: >> >>> Hello, >>> >>> We are using graylog-collector v0.4.2 on AIX 7.1 and we can start the >>> logger and it works for awhile (couple of hours) and then it suddenly stops >>> sending logs with no error messages in the logs.. >>> >>> I also changed the COLLECTOR_JAVA_DEFAULTS to -Xms512m -Xmx768m but no >>> luck. >>> >>> On the graylog-server the collector is visible while it is working, when >>> it stops sending logs the collector is no longer visible on the server side >>> but the graylog-collector process is still running in AIX with no errors in >>> logs. I have to kill the process (kill -9 since kill -15 does not stop it) >>> and then restart the process. >>> >>> Any idea how to troubleshoot the cause of this? >>> >>> *The collector.conf consists of this..* >>> server-url = "http://172.16.x.x:12900"; >>> enable-registration = true >>> collector-id = "file:config/collector-id" >>> >>> inputs { >>> 1 { >>> type = "file" >>> path = "/usr1/dbs/1.lg" >>> outputs = "gelf-tcp" >>> } >>> 2 { >>> type = "file" >>> path = "/usr1/dbs/2.lg" >>> outputs = "gelf-tcp" >>> } >>> 3 { >>> type = "file" >>> path = "/usr1/dbs/3.lg" >>> outputs = "gelf-tcp" >>> } >>> 4 { >>> type = "file" >>> path = "/usr1/dbs/4.lg" >>> outputs = "gelf-tcp" >>> } >>> 5 { >>> type = "file" >>> path = "/usr1/dbs/5.lg" >>> outputs = "gelf-tcp" >>> } >>> 6 { >>> type = "file" >>> path = "/usr1/dbs/6.lg" >>> outputs = "gelf-tcp" >>> } >>> 7 { >>> type = "file" >>> path = "/usr1/dbs/7.lg" >>> outputs = "gelf-tcp" >>> } >>> 8 { >>> type = "file" >>> path = "/usr1/dbs/8.lg" >>> outputs = "gelf-tcp" >>> } >>> } >>> >>> outputs { >>> gelf-tcp { >>> type = "gelf" >>> host = "172.16.x.x" >>> port = 12201 >>> client-tls = false >>> client-queue-size = 5120 >>> client-connect-timeout = 10000 >>> client-reconnect-delay = 2000 >>> client-tcp-no-delay = true >>> client-send-buffer-size = 32768 >>> } >>> } >>> >>> *In the log all we see is...* >>> [main] cli.commands.Run - Starting Collector v0.4.2 (commit 2609a38) >>> [main] cli.commands.Run - Running on AIX AIX 7.1 (ppc64) >>> [main] collector.utils.CollectorId - Collector ID: >>> 19c8fecb-0f04-4503-9943-15cfbac98458 >>> [main] outputs.gelf.GelfOutput - Starting GELF transport: >>> org.graylog2.gelfclient.GelfConfiguration@3c8510bd >>> [main] cli.commands.Run - Service RUNNING: BufferProcessor [RUNNING] >>> [main] cli.commands.Run - Service RUNNING: FileObserver [RUNNING] >>> [main] cli.commands.Run - Service RUNNING: MetricService [RUNNING] >>> [main] cli.commands.Run - Service RUNNING: MemoryReporterService >>> [RUNNING] >>> [main] cli.commands.Run - Service RUNNING: HeartbeatService [RUNNING] >>> [main] cli.commands.Run - Service RUNNING: FileInput{id='1', >>> path-set='SinglePathSet{path=/usr1/dbs/1.lg}', reader-buffer-size='102400', >>> message-fields='MessageFields{}', charset='UTF-8', outputs='gelf-tcp', >>> content-splitter='NEWLINE', reader-interval='100'} >>> [main] cli.commands.Run - Service RUNNING: FileInput{id='2', >>> path-set='SinglePathSet{path=/usr1/dbs/2.lg}', reader-buffer-size='102400', >>> message-fields='MessageFields{}', charset='UTF-8', outputs='gelf-tcp', >>> content-splitter='NEWLINE', reader-interval='100'} >>> [main] cli.commands.Run - Service RUNNING: FileInput{id='3', >>> path-set='SinglePathSet{path=/usr1/dbs/3.lg}', reader-buffer-size='102400', >>> message-fields='MessageFields{}', charset='UTF-8', outputs='gelf-tcp', >>> content-splitter='NEWLINE', reader-interval='100'} >>> [main] cli.commands.Run - Service RUNNING: FileInput{id='4', >>> path-set='SinglePathSet{path=/usr1/dbs/4.lg}', reader-buffer-size='102400', >>> message-fields='MessageFields{}', charset='UTF-8', outputs='gelf-tcp', >>> content-splitter='NEWLINE', reader-interval='100'} >>> [main] cli.commands.Run - Service RUNNING: FileInput{id='5', >>> path-set='SinglePathSet{path=/usr1/dbs/5.lg}', reader-buffer-size='102400', >>> message-fields='MessageFields{}', charset='UTF-8', outputs='gelf-tcp', >>> content-splitter='NEWLINE', reader-interval='100'} >>> [main] cli.commands.Run - Service RUNNING: FileInput{id='6', >>> path-set='SinglePathSet{path=/usr1/dbs/6.lg}', reader-buffer-size='102400', >>> message-fields='MessageFields{}', charset='UTF-8', outputs='gelf-tcp', >>> content-splitter='NEWLINE', reader-interval='100'} >>> [main] cli.commands.Run - Service RUNNING: GelfOutput{port='12201', >>> id='gelf-tcp', client-send-buffer-size='32768', host='172.16.x.x', >>> inputs='', client-reconnect-delay='2000', client-connect-timeout='10000', >>> client-tcp-no-delay='true', client-queue-size='5120'} >>> [main] cli.commands.Run - Service RUNNING: FileInput{id='7', >>> path-set='SinglePathSet{path=/usr1/dbs/7.lg}', reader-buffer-size='102400', >>> message-fields='MessageFields{}', charset='UTF-8', outputs='gelf-tcp', >>> content-splitter='NEWLINE', reader-interval='100'} >>> [main] cli.commands.Run - Service RUNNING: FileInput{id='8', >>> path-set='SinglePathSet{path=/usr1/dbs/8.lg}', reader-buffer-size='102400', >>> message-fields='MessageFields{}', charset='UTF-8', outputs='gelf-tcp', >>> content-splitter='NEWLINE', reader-interval='100'}', reader-interval='100'} >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Graylog Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to graylog2+u...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/graylog2/3806ada3-9eb1-4126-a415-d7c0bf997ae1%40googlegroups.com >>> <https://groups.google.com/d/msgid/graylog2/3806ada3-9eb1-4126-a415-d7c0bf997ae1%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> >> No matter what we think of Linux versus FreeBSD, etc., the one thing I >> really like about Linux is that it has Microsoft worried. Anything >> that kicks a monopoly in the pants has got to be good for something. >> - Chris Johnson >> >> -- > You received this message because you are subscribed to the Google Groups > "Graylog Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to graylog2+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/graylog2/ad5bd1a9-bb37-40fa-9f46-2aa42cc874d7%40googlegroups.com > <https://groups.google.com/d/msgid/graylog2/ad5bd1a9-bb37-40fa-9f46-2aa42cc874d7%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- No matter what we think of Linux versus FreeBSD, etc., the one thing I really like about Linux is that it has Microsoft worried. Anything that kicks a monopoly in the pants has got to be good for something. - Chris Johnson -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAL5rfGXErR_o2sUh-Mizme_19j6eDzgbHy89sGmzLxyiU7mO2Q%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
