Most of the messages are flowing to splunk fine. A few are failing to get
through - possibly we are flooding splunk - hence those errors.
But what I dont understand is why the process then stops processing any more
messages.
On Sun, May 8, 2011 at 6:07 AM, Filippo Balicchia <fbalicc...@gmail.com>wrote:
> Does your splunk accept you connection ?
>
> From stacktrace seems that refused your connection
>
>
> 2011-05-07 12:41:47.042077500 Caused by: java.net.ConnectException:
> Connection refused2011-05-07 12:41:47.042078500 at
>
>
> --Filippo
>
>
> 2011/5/7 Tom Howe <tomh...@artcore.com>:
> > I have a simple java app that collect files of disk, converts the [json]
> > contents and then forwards it to a splunk instance over tcp using mina.
> >
> >
> > <endpoint id="input" uri="${input.endpoint}"/>
> > <endpoint id="output" uri="${output.endpoint}"/>
> >
> > <route autoStartup="true" xmlns="
> > http://camel.apache.org/schema/spring";>
> > <from ref="input"/>
> > <convertBodyTo type="java.lang.String"/>
> > <bean ref="jsonToSplunk"/>
> > <to ref="splunk" />
> > </route>
> >
> > input.endpoint =
> > file:/dir/to/files?delete=true&recursive=true&maxMessagesPerPoll=500
> > output.endpoint =
> > mina:tcp://splunkhost/?textline=true&textlineDelimiter=UNIX&sync=false
> >
> > It also emits a heartbeat message every minute over JMS using camel
> quartz
> > endpoint.
> >
> > The problem I am having is that after about a day of running, the process
> > stops picking up files and sending them to splunk and stops sending the
> > heartbeat JMS messages. However, the log is still showing that it is
> > emitting messages.
> >
> > It's like the camel routes have just stopped working properly but I have
> no
> > idea why.
> >
> >
> > I see a bunch of these exceptions in the log like the following.. which
> is
> > suggesting it is failing to connect to splunk sometimes.
> >
> > 2011-05-07 12:41:50.457659500 2011-05-07 13:41:50,456 [Camel (main)
> thread
> > #0 - file:///home/wfe/var/ispy/events] ERROR
> > org.apache.camel.processor.DeadLetterChannel - Failed delivery for
> > exchangeId: ID-vp-prod-wfe-a-61766-1304769141692-0-95088. Exhausted after
> > delivery attempt: 11 caught: org.apache.mina.common.RuntimeIOException:
> > Failed to get the session.. Processed by failure processor:
> > sendTo(Endpoint[activemq://splunk.error] InOnly)
> > 2011-05-07 12:41:50.457664500 org.apache.mina.common.RuntimeIOException:
> > Failed to get the session.
> > 2011-05-07 12:41:50.457952500 at
> >
> org.apache.mina.common.support.DefaultConnectFuture.getSession(DefaultConnectFuture.java:63)
> > 2011-05-07 12:41:50.457953500 at
> >
> org.apache.camel.component.mina.MinaProducer.openConnection(MinaProducer.java:211)
> > 2011-05-07 12:41:50.457954500 at
> >
> org.apache.camel.component.mina.MinaProducer.process(MinaProducer.java:80)
> > 2011-05-07 12:41:50.457955500 at
> >
> org.apache.camel.impl.converter.AsyncProcessorTypeConverter$ProcessorToAsyncProcessorBridge.process(AsyncProcessorTypeConverter.java:50)
> > 2011-05-07 12:41:50.457963500 at
> >
> org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:77)
> > 2011-05-07 12:41:50.457964500 at
> >
> org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:104)
> > ..snip..
> > 2011-05-07 12:41:47.042077500 Caused by: java.net.ConnectException:
> > Connection refused2011-05-07 12:41:47.042078500 at
> > sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > 2011-05-07 12:41:47.042082500 at
> > sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
> > 2011-05-07 12:41:47.042083500 at
> >
> org.apache.mina.transport.socket.nio.SocketConnector.processSessions(SocketConnector.java:293)
> > 2011-05-07 12:41:47.042084500 at
> >
> org.apache.mina.transport.socket.nio.SocketConnector.access$800(SocketConnector.java:53)
> > 2011-05-07 12:41:47.042085500 at
> >
> org.apache.mina.transport.socket.nio.SocketConnector$Worker.run(SocketConnector.java:383)
> > 2011-05-07 12:41:47.042089500 at
> >
> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
> > 2011-05-07 12:41:47.042090500 ... 3 more
> >
> > Any idea if this could be causing the routes to stop?
> >
> > Let me know if I should provide extra information.
> >
> > Thanks, Tom
> >
>
