On Fri, Oct 4, 2013 at 1:21 PM, David Lang <[email protected]> wrote: > It's not just about memory. Keep in mind that files are currently only > >> closed when they are evicted from cache. So this can lead to tons of open >> files. Also, a high number is very dangerous when multiple dynafile >> actions >> are being used (which is common). Let's say we have 20 such actions, that >> means we potentially have a total of 20,000 files open (worst case). The >> whole omfile file cache system can need a refactoring, and I think I >> prefer >> to keep the numbers very conservative rather than risk blowing the system. >> > > well, there's conservative and then there's what rsyslog currently has. :-) >
Of course, it's always debatable ;) > > but your comment brings up a question, you make it sound like there is a > separate dynafile cach for each action. Is this the case or is it one > filehandle cache? No, that's actually the case, one cache per action. This is why I am so hesistant to just increase it's size. > Given that you could have multiple actions write to the same file, I would > have expected a single cache to serialize the writes to the file. > No, that never happened in traditional syslog and rsyslog carried that concept over. That's also why I would like to refactor that thing, but there are big performance implications (just think about the then-necessary locking operations and the like). it seems that what's really needed is some way of detecting that a file > handle has been inactive for a while and closing it. I'll have to think > about hwo to cheaply do this. > It's fairly easy nowadays. We just need a backround thread which periodically wakes up and checks cache entry age. The "time" used can be wakeup ticks, so this can be done efficiently as well. Previously, this was no good solution as it caused wakeups, which were criticial on power-critical systems. Now that we move more and more into the datacenter, that's no longer a real concern. But even though it is easy, it must be done ;) Given the fact that I don't have sponsoring for any of what we discuss here, I need to focus. My decision was to put the focus on imfile, as I know that many more folks are unhappy (and rightfully so!) with the current state than there are with the dynafile cache. You can solve all problems with dynafiles with proper configuration, but there are many use cases you really can't do currently with imfile (like monitoring a directory). So I prefer to put my effort into making something possible that currently is impossible vs. improving the usability of something that you can already do... Rainer _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
