RE: File system full causes log4cxx to crash
>Probably a limit you can configure with your shell. In bash it's ulimit -c unlimited to get full core dumps and ulimit -a >to see the settings (0 bytes by default on my shell.) It and gdb might not be helpful anyway if your log4cxx library is >stripped of debugging symbols. If you do pursue getting a core, please don't send it to me :). Install gdb and get the >stack trace. We're a month from beta and I still have other bugs to fix, so I'm putting this one to rest for now...
Re: File system full causes log4cxx to crash
On 2008-09-23 11:21, Peter Steele wrote:
/mnt: write failed, filesystem is full
terminate called after throwing an instance of
'log4cxx::helpers::IOException'
what(): IO Exception : status code = 28
Abort trap: 6 (core dumped)
I'm stumped here. The only places I see an IOException thrown in 0.9.7 code are:
* datagramsocket.cpp
* properties.cpp
There are some child classes, InterruptedIOException and SocketTimeoutException,
and then children of those all defined in socketimpl.h. Since my sample app was
using a FileAppender, I don't get how an IOException is thrown.
Unfortunately, despite it saying a core was dumped, there doesn't seem
to be a core file. Not sure how to proceed from here.
Probably a limit you can configure with your shell. In bash it's ulimit -c
unlimited to get full core dumps and ulimit -a to see the settings (0 bytes by
default on my shell.) It and gdb might not be helpful anyway if your log4cxx
library is stripped of debugging symbols. If you do pursue getting a core,
please don't send it to me :). Install gdb and get the stack trace.
I'm hoping someone else chimes in with ideas. I guess as a second test, you
could try my non-log4cxx program against that full filesystem. Maybe modify a
copy of it to try with an fstream. Both should fail gracefully.
--
Jacob Anawalt
Gecko Software, Inc.
[EMAIL PROTECTED]
435-752-8026
// 2008-Sep-22 18:00
// Simple test application to see how stdc libraries
// react to a full filesystem.
// gcc -ofullfs-c fullfs.c
#include /* open */
#include /* open */
#include /* printf */
#include /* write */
#include /* errno */
int
main(void)
{
int i;
int estat;
int status;
int fd;
ssize_t bytes;
const size_t bufSz = 8096;
char buf[bufSz];
estat = 0;
i = 0;
fd = open("/mnt/test",O_WRONLY|O_TRUNC|O_CREAT);
if(fd != -1) {
printf("Opened file fd %d\n",fd);
while(1)
{
bytes = write(fd,buf,bufSz);
if(bytes == bufSz) {
++i;
printf("Write block, filesize now %d\n",i*bufSz);
}
else if( bytes == -1 ) {
status = errno;
perror("write()");
printf("errno: %d\n");
estat = 1;
break;
}
else {
printf("Partial write of %d bytes\n",bytes);
}
};
}
else {
perror("open()");
estat = 1; /* failure */
}
return estat;
}
RE: File system full causes log4cxx to crash
Peter - To add to your comments: In our systems (real-time call processing engines for carrier networks) a s/w fault is many orders of magnitude worse than no logs being available. One thing that happens in our customer environments is that when something goes wrong... the IT/sysadmin folks will look at syslog entries -- and if the appender or logger framework were to log something they'd see it. The more 'process' oriented companies will have their NOC's review the logs and trap major issues as soon as they happen. Renny Koshy "Peter Steele" <[EMAIL PROTECTED]> 09/23/2008 01:26 PM Please respond to "Log4CXX User" To "Log4CXX User" cc Subject RE: File system full causes log4cxx to crash I’ll add my own agreement to this. It’s not acceptable for a logger to crash an application when the volume becomes full. My solution to use a try/catch works for us for now since we only have a single appender, but as was mentioned below if we decided to add a second appender the try/catch is a very big hammer, potentially preventing the other appender from running. Still, it’s better than the application crashing on us. From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: Tuesday, September 23, 2008 10:20 AM To: Log4CXX User Subject: Re: File system full causes log4cxx to crash I agree wholeheartedly If it gets into something like the scenario mentioned below, maybe the logger can put a syslog entry, or console/stderr output (as configured by the user in some global setting)? Renny Koshy [EMAIL PROTECTED] (J.T. Conklin) 09/23/2008 01:12 PM Please respond to "Log4CXX User" To "Log4CXX User" cc Subject Re: File system full causes log4cxx to crash "Jacob L. Anawalt" <[EMAIL PROTECTED]> writes: > I am glad it worked out for you. At the same time I am concerned that > you had to do this and I didn't. I would like to reproduce the problem > so that I can handle it, preferably without putting try/catch around > every logging statement... Some time back, I sent a message to the list asking what the behavior of an appender that throws an exception should be. As it is, a stray exception can unwind all the way through the library into the calling code. To my mind, wrapping log invocations with try/catch blocks isn't a completely satisfactory solution, as if you have multiple appenders associated with a logger, the exception will prevent subsequent appenders from being invoked. Personally, I think log4cxx should provide a guarantee that a log invocation will not throw an exception under any circumstances. One of the common use cases for emitting a log event is when you're in a error handling path, the last thing you want to worry about is the logging framework making hash of things. Two alternatives that come to mind are that every appender's append() method needs to have a try/catch block to squelch any exceptions, or that the appendLoopOnAppenders should have a try/catch block around each call to doAppend() invocation. Or since all it takes is one bad appender (and even if all appenders that are distributed with log4cxx are fixed, there's no guarantee that user's appenders will also be), a hybrid of the two approaches where the try/catch block around doAppend() logs an internal message about the unhandled exception. Basically, a framework shouldn't allow a callback to subvert or compromise the function of the framework itself. --jtc -- J.T. Conklin
RE: File system full causes log4cxx to crash
I'll add my own agreement to this. It's not acceptable for a logger to
crash an application when the volume becomes full. My solution to use a
try/catch works for us for now since we only have a single appender, but
as was mentioned below if we decided to add a second appender the
try/catch is a very big hammer, potentially preventing the other
appender from running. Still, it's better than the application crashing
on us.
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED]
Sent: Tuesday, September 23, 2008 10:20 AM
To: Log4CXX User
Subject: Re: File system full causes log4cxx to crash
I agree wholeheartedly
If it gets into something like the scenario mentioned below, maybe the
logger can put a syslog entry, or console/stderr output (as configured
by the user in some global setting)?
Renny Koshy
[EMAIL PROTECTED] (J.T. Conklin)
09/23/2008 01:12 PM
Please respond to
"Log4CXX User"
To
"Log4CXX User"
cc
Subject
Re: File system full causes log4cxx to crash
"Jacob L. Anawalt" <[EMAIL PROTECTED]> writes:
> I am glad it worked out for you. At the same time I am concerned that
> you had to do this and I didn't. I would like to reproduce the problem
> so that I can handle it, preferably without putting try/catch around
> every logging statement...
Some time back, I sent a message to the list asking what the behavior
of an appender that throws an exception should be. As it is, a stray
exception can unwind all the way through the library into the calling
code.
To my mind, wrapping log invocations with try/catch blocks isn't a
completely satisfactory solution, as if you have multiple appenders
associated with a logger, the exception will prevent subsequent
appenders from being invoked.
Personally, I think log4cxx should provide a guarantee that a log
invocation will not throw an exception under any circumstances. One
of the common use cases for emitting a log event is when you're in a
error handling path, the last thing you want to worry about is the
logging framework making hash of things.
Two alternatives that come to mind are that every appender's append()
method needs to have a try/catch block to squelch any exceptions, or
that the appendLoopOnAppenders should have a try/catch block around
each call to doAppend() invocation. Or since all it takes is one bad
appender (and even if all appenders that are distributed with log4cxx
are fixed, there's no guarantee that user's appenders will also be), a
hybrid of the two approaches where the try/catch block around
doAppend() logs an internal message about the unhandled exception.
Basically, a framework shouldn't allow a callback to subvert or
compromise the function of the framework itself.
--jtc
--
J.T. Conklin
RE: File system full causes log4cxx to crash
>I wonder if the difference might be in our iostreams libraries. >Would you mind debugging the exception and sharing who's throwing it and why? My C++ Unix debugging skills aren't great (I'm a Windows guy normally). I don't gdb or any of the other Unix debuggers. The error I get when I don't have the try/catch is: # ./a.out /mnt: write failed, filesystem is full terminate called after throwing an instance of 'log4cxx::helpers::IOException' what(): IO Exception : status code = 28 Abort trap: 6 (core dumped) Unfortunately, despite it saying a core was dumped, there doesn't seem to be a core file. Not sure how to proceed from here.
Re: File system full causes log4cxx to crash
I agree wholeheartedly If it gets into something like the scenario mentioned below, maybe the logger can put a syslog entry, or console/stderr output (as configured by the user in some global setting)? Renny Koshy [EMAIL PROTECTED] (J.T. Conklin) 09/23/2008 01:12 PM Please respond to "Log4CXX User" To "Log4CXX User" cc Subject Re: File system full causes log4cxx to crash "Jacob L. Anawalt" <[EMAIL PROTECTED]> writes: > I am glad it worked out for you. At the same time I am concerned that > you had to do this and I didn't. I would like to reproduce the problem > so that I can handle it, preferably without putting try/catch around > every logging statement... Some time back, I sent a message to the list asking what the behavior of an appender that throws an exception should be. As it is, a stray exception can unwind all the way through the library into the calling code. To my mind, wrapping log invocations with try/catch blocks isn't a completely satisfactory solution, as if you have multiple appenders associated with a logger, the exception will prevent subsequent appenders from being invoked. Personally, I think log4cxx should provide a guarantee that a log invocation will not throw an exception under any circumstances. One of the common use cases for emitting a log event is when you're in a error handling path, the last thing you want to worry about is the logging framework making hash of things. Two alternatives that come to mind are that every appender's append() method needs to have a try/catch block to squelch any exceptions, or that the appendLoopOnAppenders should have a try/catch block around each call to doAppend() invocation. Or since all it takes is one bad appender (and even if all appenders that are distributed with log4cxx are fixed, there's no guarantee that user's appenders will also be), a hybrid of the two approaches where the try/catch block around doAppend() logs an internal message about the unhandled exception. Basically, a framework shouldn't allow a callback to subvert or compromise the function of the framework itself. --jtc -- J.T. Conklin
Re: File system full causes log4cxx to crash
"Jacob L. Anawalt" <[EMAIL PROTECTED]> writes: > I am glad it worked out for you. At the same time I am concerned that > you had to do this and I didn't. I would like to reproduce the problem > so that I can handle it, preferably without putting try/catch around > every logging statement... Some time back, I sent a message to the list asking what the behavior of an appender that throws an exception should be. As it is, a stray exception can unwind all the way through the library into the calling code. To my mind, wrapping log invocations with try/catch blocks isn't a completely satisfactory solution, as if you have multiple appenders associated with a logger, the exception will prevent subsequent appenders from being invoked. Personally, I think log4cxx should provide a guarantee that a log invocation will not throw an exception under any circumstances. One of the common use cases for emitting a log event is when you're in a error handling path, the last thing you want to worry about is the logging framework making hash of things. Two alternatives that come to mind are that every appender's append() method needs to have a try/catch block to squelch any exceptions, or that the appendLoopOnAppenders should have a try/catch block around each call to doAppend() invocation. Or since all it takes is one bad appender (and even if all appenders that are distributed with log4cxx are fixed, there's no guarantee that user's appenders will also be), a hybrid of the two approaches where the try/catch block around doAppend() logs an internal message about the unhandled exception. Basically, a framework shouldn't allow a callback to subvert or compromise the function of the framework itself. --jtc -- J.T. Conklin
Re: File system full causes log4cxx to crash
On 2008-09-23 06:16, Peter Steele wrote: With 0.9.7 your app threw and exception and aborted when the log volume filled up. Putting a try/catch around the log call prevented the crash. I put the same try/catch in my wrapper functions and that worked as well, so that's all there was to it. I like simple solutions. I am glad it worked out for you. At the same time I am concerned that you had to do this and I didn't. I would like to reproduce the problem so that I can handle it, preferably without putting try/catch around every logging statement... The only throw I see in Logger is in getEffectiveLevel. The forcedLog method called by the macros gets to callAppenders where in the sample app it should hit a console and a file appender. That does appendLoopOnAppenders to call doAppend, which calls Writer::append, checkEntryConditions and subAppend. On to PatternLayout::format and PatternConverter::format. No throw. Did I follow the wrong chain? I wonder if the difference might be in our iostreams libraries. Would you mind debugging the exception and sharing who's throwing it and why? Thank you, -- Jacob Anawalt Gecko Software, Inc. [EMAIL PROTECTED] 435-752-8026
RE: File system full causes log4cxx to crash
With 0.9.7 your app threw and exception and aborted when the log volume filled up. Putting a try/catch around the log call prevented the crash. I put the same try/catch in my wrapper functions and that worked as well, so that's all there was to it. I like simple solutions. Thanks for pointing me in the right direction. -Original Message- From: Peter Steele [mailto:[EMAIL PROTECTED] Sent: Monday, September 22, 2008 5:46 PM To: Log4CXX User Subject: RE: File system full causes log4cxx to crash Okay, thanks, I'll give it a try. -Original Message- From: Jacob L. Anawalt [mailto:[EMAIL PROTECTED] Sent: Monday, September 22, 2008 4:24 PM To: Log4CXX User Subject: Re: File system full causes log4cxx to crash On 2008-09-22 17:03, Peter Steele wrote: > We're using 0.9.7 in fact, under FreeBSD. Our case is bit more > Maybe we need to catch this IOException in our wrapper functions? The IOException was in the 0.10.x code. I didn't see any exception around the writing to files in 0.9.7. As near as I could tell the iostream library was handing the full filesytem error by not writing any more but the rest of the program and logging kept working. Perhaps it is something with the C wrappers. You could try my test program with a small file based filesystem loop mounted at /mnt -- Jacob Anawalt Gecko Software, Inc. [EMAIL PROTECTED] 435-752-8026
RE: File system full causes log4cxx to crash
Okay, thanks, I'll give it a try. -Original Message- From: Jacob L. Anawalt [mailto:[EMAIL PROTECTED] Sent: Monday, September 22, 2008 4:24 PM To: Log4CXX User Subject: Re: File system full causes log4cxx to crash On 2008-09-22 17:03, Peter Steele wrote: > We're using 0.9.7 in fact, under FreeBSD. Our case is bit more > Maybe we need to catch this IOException in our wrapper functions? The IOException was in the 0.10.x code. I didn't see any exception around the writing to files in 0.9.7. As near as I could tell the iostream library was handing the full filesytem error by not writing any more but the rest of the program and logging kept working. Perhaps it is something with the C wrappers. You could try my test program with a small file based filesystem loop mounted at /mnt -- Jacob Anawalt Gecko Software, Inc. [EMAIL PROTECTED] 435-752-8026
Re: File system full causes log4cxx to crash
On 2008-09-22 17:03, Peter Steele wrote:
We're using 0.9.7 in fact, under FreeBSD. Our case is bit more
Maybe we need to catch this IOException in our wrapper functions?
The IOException was in the 0.10.x code. I didn't see any exception around the
writing to files in 0.9.7. As near as I could tell the iostream library was
handing the full filesytem error by not writing any more but the rest of the
program and logging kept working.
Perhaps it is something with the C wrappers. You could try my test program with
a small file based filesystem loop mounted at /mnt
--
Jacob Anawalt
Gecko Software, Inc.
[EMAIL PROTECTED]
435-752-8026
// 2008-Sep-22 19:30
// Simple test application to test how log4cxx handles a full
// filesystem. /mnt was a small 1MiB ext2 filesystem (loop mounted
// file). When it filled up, the console logging, and thus the
// program, continued to work and did not crash. New logging
// events stopped writing to the file.
//
// This test was done with log4cxx 0.9.7.
// g++ -ofullfs-log4cxx fullfs-log4cxx.cpp -llog4cxx
#include
#include
#include
#include
int
main(void)
{
log4cxx::BasicConfigurator::configure();
log4cxx::LayoutPtr l(new log4cxx::PatternLayout(
"%d{%Y-%m-%d %H:%M:%S} %-5p %c %x - %m%n"));
log4cxx::AppenderPtr a(new log4cxx::FileAppender(l,"/mnt/test"));
a->setName("FILE");
log4cxx::LoggerPtr logger(log4cxx::Logger::getLogger("MyApp"));
logger->addAppender(a);
log4cxx::LoggerPtr root(log4cxx::Logger::getRootLogger());
while(1) {
LOG4CXX_DEBUG(logger,"bla bla bla bla bla bla bla");
}
return EXIT_SUCCESS;
}
RE: File system full causes log4cxx to crash
>Looking at FileOutputStream::write in svn head I see that if apr_file_write >doesn't return APR_SUCCESS, it throws an IOException with the write call's >status. I am not familar enough with the code to see who/what should catch that, >but to work like 0.9.7 I'd expect it to be caught at some appropriate level to >disable that particular appender. (Maybe in WriterAppender::append?) If the >exception is not caught perhaps it is the crash you see. We're using 0.9.7 in fact, under FreeBSD. Our case is bit more complicated in that we are using log4cxx in a C application, with wrapper functions to let us call the log4cxx functions from our C layer. Maybe we need to catch this IOException in our wrapper functions? I'll have to investigate this. Thanks for the idea. Peter
Re: File system full causes log4cxx to crash
On 2008-09-20 11:37, Peter Steele wrote: We've experienced several cases of our logging volume becoming full during the course of execution of our application. What we would like to happen in this case is simply to lose any log further messages that are written after the file system becomes full I wanted to know what would happen to my programs using log4cxx (0.9.7) in this case so I wrote a couple of test apps and created a 1MiB block file and mounted it as a loop device in /mnt. First I tried a simple C app using open and blocking write to see what happens. By picking a buffer size of 8096 I was able to see full writes, a partial write and then an error with perror reporting 'No space left on device'. My code exits the loop at this point. Next I tried a simple C++ log4cxx app that writes to the disk and console via a "MyApp" logger. After it fill the disk the app continues to work and continues to write to the console. Since the writer is ofstream based, I imagine the bad bit was set and further writes just get skipped. Looking at FileOutputStream::write in svn head I see that if apr_file_write doesn't return APR_SUCCESS, it throws an IOException with the write call's status. I am not familar enough with the code to see who/what should catch that, but to work like 0.9.7 I'd expect it to be caught at some appropriate level to disable that particular appender. (Maybe in WriterAppender::append?) If the exception is not caught perhaps it is the crash you see. -- Jacob Anawalt Gecko Software, Inc. [EMAIL PROTECTED] 435-752-8026
RE: File system full causes log4cxx to crash
Peter - Sorry, been a bit busy lately, and missed this one... I don't believe it's a controllable behavior, from what I recall it is an OS level behavior on what happens when it is out of resources. I've seen apps not using Log4CXX do the same thing. Or worse, I've seen apps get "stuck". The only solution we've seen -- as a general case -- is to monitor the volume via SNMP or using a simple script, and notify the admins BEFORE it fails. Regards, Renny Koshy "Peter Steele" <[EMAIL PROTECTED]> 09/20/2008 01:37 PM Please respond to "Log4CXX User" To "Log4CXX User" cc Subject RE: File system full causes log4cxx to crash Does not one have any comments on this? Surely someone must have experienced this scenario, specially the volume where logs are being recorded becoming filled up. From: Peter Steele [mailto:[EMAIL PROTECTED] Sent: Friday, September 19, 2008 11:46 AM To: Log4CXX User Subject: File system full causes log4cxx to crash We’ve experienced several cases of our logging volume becoming full during the course of execution of our application. What we would like to happen in this case is simply to lose any log further messages that are written after the file system becomes full, but what happens instead is log4cxx crashes and brings down our application. We’re using the RollingFileAppender on a Unix box. Is there some way to configure log4cxx to behave a little less drastically in a case like this? Ideally, the best behavior in a case like this would be for log4cxx to delete the oldest backup log file and continue logging, but even simply suspending logging would be better than a crash. Any suggestions?
RE: File system full causes log4cxx to crash
I have a couple of comments: 1. The RollingFileAppender allows you to know a priori exactly how much disk space is required. Is there an amount of space you can be sure will exist? Worst case, if you own the environment, you could create a partition and log there. 2. I think a default behavior of deleting the oldest logs in this situation would be a bad idea - stop logging would be better. A disk full situation may be the result of an application error or some kind of DoS attack, the details of which might be logged and subsequently lost if the oldest logs are deleted. This behavior seems like it could be configurable but, by default, I would not delete existing logs. These aren't the comments you're looking for, I'm sure, but I have never run into this problem before so I cannot provide a real solution. You didn't mention how it is that you know that log4cxx is the cause of the crash. Is the crash from any log write when the volume is full, or does it happen when the log attempts to roll? You didn't mention what Unix and version you are using. This would be helpful to attempt to reproduce. Sorry I can't be more help to your particular problem, but I wanted to weigh in on the possible solution implementation. On Sat, 20 Sep 2008, Peter Steele wrote: Does not one have any comments on this? Surely someone must have experienced this scenario, specially the volume where logs are being recorded becoming filled up. From: Peter Steele [mailto:[EMAIL PROTECTED] Sent: Friday, September 19, 2008 11:46 AM To: Log4CXX User Subject: File system full causes log4cxx to crash We've experienced several cases of our logging volume becoming full during the course of execution of our application. What we would like to happen in this case is simply to lose any log further messages that are written after the file system becomes full, but what happens instead is log4cxx crashes and brings down our application. We're using the RollingFileAppender on a Unix box. Is there some way to configure log4cxx to behave a little less drastically in a case like this? Ideally, the best behavior in a case like this would be for log4cxx to delete the oldest backup log file and continue logging, but even simply suspending logging would be better than a crash. Any suggestions?
RE: File system full causes log4cxx to crash
Does not one have any comments on this? Surely someone must have experienced this scenario, specially the volume where logs are being recorded becoming filled up. From: Peter Steele [mailto:[EMAIL PROTECTED] Sent: Friday, September 19, 2008 11:46 AM To: Log4CXX User Subject: File system full causes log4cxx to crash We've experienced several cases of our logging volume becoming full during the course of execution of our application. What we would like to happen in this case is simply to lose any log further messages that are written after the file system becomes full, but what happens instead is log4cxx crashes and brings down our application. We're using the RollingFileAppender on a Unix box. Is there some way to configure log4cxx to behave a little less drastically in a case like this? Ideally, the best behavior in a case like this would be for log4cxx to delete the oldest backup log file and continue logging, but even simply suspending logging would be better than a crash. Any suggestions?
