Yes, I was referring to the Flume Persistent Appender. I will open an enhancement request in JIRA.In the meanwhile, we will consider using local disk instead of a remote filesystem to decrease the likelihood of the issue occurring.
Thanks, - Arkin On Fri, Apr 4, 2014 at 2:46 PM, Ralph Goers <[email protected]>wrote: > I'm assuming you are using the Flume Persistent Appender which uses > Berkeley DB based on the logs below. From the log it appears that Berkeley > DB's file handle is stale and meets to be closed and reopened. I haven't > checked but Berkeley DB might have a setting for this, otherwise the Flume > Persistent Manager would need to deal with this condition. That would be a > big change as the Database object is currently immutable. > > Ralph > > On Apr 4, 2014, at 12:04 PM, Arkin Yetis <[email protected]> wrote: > > > We use the Flume Appender. Our logging stopped after a certain point in > > time and we noticed the exception at the end of this message in our > > application logs. It looks like there was an issue with the filesystem. > But > > although the filesystem has recovered, the appender (or probably the > > persistence mechanism it uses) was stuck in this state and it took an > > application restart for it to continue logging. It does not look like > there > > is a recovery mechanism or if there is one it failed. > > Would you like me to open a log4j JIRA ticket for this? Or is this > > something that can be prevented by something simple you can share over > > e-mail such as a certain configuration setting? > > > > Thanks, > > - Arkin > > > > Exception stack is: > > 1. Stale NFS file handle (java.io.IOException) > > java.io.RandomAccessFile:-2 (null) > > 2. Environment invalid because of previous exception: (JE 5.0.73) > > /app/logs/abs-workflow/flumeDir java.io.IOException: Stale NFS file > handle > > LOG_READ: IOException on read, log is likely invalid. Environment is > > invalid and must be closed. fetchTarget of 0x542/0x4af13c parent IN=5 IN > > class=com.sleepycat.je.tree.BIN lastFullVersion=0x543/0x62d6c5 > > lastLoggedVersion=0x543/0x62d6c5 parent.getDirty()=true state=0 > > (com.sleepycat.je.EnvironmentFailureException) > > com.sleepycat.je.log.FileManager:1883 (null) > > > > > ******************************************************************************** > > Root Exception stack trace:java.io.IOException: Stale NFS file handle > > at java.io.RandomAccessFile.readBytes(Native Method) > > at java.io.RandomAccessFile.read(RandomAccessFile.java:338) > > at > > > com.sleepycat.je.log.FileManager.readFromFileInternal(FileManager.java:1918) > > at > com.sleepycat.je.log.FileManager.readFromFile(FileManager.java:1869) > > at > com.sleepycat.je.log.FileManager.readFromFile(FileManager.java:1807) > > at com.sleepycat.je.log.FileSource.getBytes(FileSource.java:56) > > at > > > com.sleepycat.je.log.LogManager.getLogEntryFromLogSource(LogManager.java:919) > > at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:848) > > at > > > com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:809) > > at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1412) > > at com.sleepycat.je.tree.BIN.fetchTarget(BIN.java:1251) > > at com.sleepycat.je.dbi.CursorImpl.fetchCurrent(CursorImpl.java:2261) > > at > > > com.sleepycat.je.dbi.CursorImpl.getCurrentAlreadyLatched(CursorImpl.java:1466) > > at com.sleepycat.je.dbi.CursorImpl.getNext(CursorImpl.java:1593) > > at > > > com.sleepycat.je.cleaner.UtilizationProfile.getObsoleteDetail(UtilizationProfile.java:632) > > at > > > com.sleepycat.je.cleaner.FileProcessor.processFile(FileProcessor.java:439) > > at > > com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:289) > > at > > com.sleepycat.je.cleaner.FileProcessor.onWakeup(FileProcessor.java:148) > > at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:163) > > at java.lang.Thread.run(Thread.java:662) > > > ******************************************************************************** > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
