Re: [Simple-evcorr-users] intstates and HUP
2015-04-03 3:57 GMT+03:00 David Lang da...@lang.hm: On Thu, 2 Apr 2015, Risto Vaarandi wrote: 2015-04-02 1:22 GMT+03:00 David Lang da...@lang.hm: On Wed, 1 Apr 2015, Risto Vaarandi wrote: hi David, is my understanding correct that you would like to have pre_restart event, in order to execute the following steps: 1) when HUP is received, match pre_restart synthetic event and save event correlation state to disk, 2) let the HUP handling procedure to drop all event correlation state from memory, 3) match sec_restart synthetic event and load event correlation state from disk back to memory If my understanding is correct and you want to use the above sequence for making HUP a log rotation signal, there would be a number of non-trivial issues to handle. Firstly, while some state like contexts can be stored and restored to/from disk in a straightforward way, things get a bit more complex with storing/restoring event correlation operations. Secondly, receiving HUP also means that all inputs will be closed and reopened, and at reopen sec always jumps to the very end of each input file. This means that any lines appended to input files between close and reopen will be missed. In short, I would never recommend using the HUP signal just for rotating log files, since HUP's true purpose is doing full restart (in fact, it is almost identical to taking sec process down and starting a new instance). In my use case, the input is a pipe, so unless sec is falling behind, little, if anything would be lost. I agree it's not the best thing to do, for rotation, but here's another case. I have an instance of SEC that's keeping running totals of things in logs, and I have it writing the totals out every minute. I also have it write out the totals when it shuts down. But if it gets a HUP, it looses all it's current data and has no chance to write it out or do anything else with it. Different treatment of HUP is actually one of the differences between stateful and stateless unix daemons. It is a common practice to use HUP for reinitializing a daemon which normally means reopening inputs/outputs, reloading configuration and starting with a clean state. Daemons like rsyslog are largely stateless -- they don't keep any state about processed events in memory, since their main purpose is fast forwarding of events from inputs to outputs. For this reason, sending a HUP to rsyslog largely means reopening input/outputs and reloading your configuration. Thus, if you haven't modified your configuration, you can easily use HUP for log rotation, since you have no state to lose in memory. not actually true with rsyslog. If it did a re-read of it's config, lots of things can change and there's a lot of state that can be maintained. But because HUP is so commonly used for log rotation, and a full startup takes so long, HUP was redefined to just be a log rotation event (close inputs and outputs) rather than a full reparse of the config. Too many people were loosing too much data with the old way of doing things. We could have gone down the route of defining a different signal for the rotation and kept HUP as the full restart, but the result would have been more lost logs for people over the years. Did we do the wrong thing? arguable. From a purist point of view we did, from a pragmatic point of view probably not. With stateful daemons like sec and snort the HUP signal is not just about reopening input/output channels and reloading rules/signatures, but you always lose state which has been stored in memory. As a result, with stateful daemons the HUP signal can not have a dual role of reinitialization and log rotation. That is why sec uses a different signal (USR2) designed solely for log rotation. That's all well and good, but why do you loose more state when you do a HUP than when you do a full shutdown? I'm not even talking about restoring state after the HUP, I'm talking about the ability to output running counters. If I have a counter that's accumulating data that I output every hour (via a calendar command) and it's 1 sec before the end of the hour. If I do a full kill/restart I have the chance to output the data before I shutdown But if I do a HUP instead, that data is irrevokably lost. why should a full kill be more data friendly than a HUP? David Lang hi David, when we leave data storing on SEC_SHUTDOWN completely aside, and just compare HUP and full restart as such, there are data items which are not lost at HUP. Since the user can set up his custom Perl data structures from perlfunc patterns, but also with 'lcall', 'eval' and other such actions, these data structures are not cleared. Since custom perl hashes, arrays and variables are not standard sec items and might have special semantics, resetting them automatically is not a good idea, and futhermore, the user can clear them easily on the reception of sec_restart event. So
Re: [Simple-evcorr-users] intstates and HUP
On Fri, 3 Apr 2015, Risto Vaarandi wrote: 2015-04-03 3:57 GMT+03:00 David Lang da...@lang.hm: On Thu, 2 Apr 2015, Risto Vaarandi wrote: 2015-04-02 1:22 GMT+03:00 David Lang da...@lang.hm: On Wed, 1 Apr 2015, Risto Vaarandi wrote: hi David, is my understanding correct that you would like to have pre_restart event, in order to execute the following steps: 1) when HUP is received, match pre_restart synthetic event and save event correlation state to disk, 2) let the HUP handling procedure to drop all event correlation state from memory, 3) match sec_restart synthetic event and load event correlation state from disk back to memory If my understanding is correct and you want to use the above sequence for making HUP a log rotation signal, there would be a number of non-trivial issues to handle. Firstly, while some state like contexts can be stored and restored to/from disk in a straightforward way, things get a bit more complex with storing/restoring event correlation operations. Secondly, receiving HUP also means that all inputs will be closed and reopened, and at reopen sec always jumps to the very end of each input file. This means that any lines appended to input files between close and reopen will be missed. In short, I would never recommend using the HUP signal just for rotating log files, since HUP's true purpose is doing full restart (in fact, it is almost identical to taking sec process down and starting a new instance). In my use case, the input is a pipe, so unless sec is falling behind, little, if anything would be lost. I agree it's not the best thing to do, for rotation, but here's another case. I have an instance of SEC that's keeping running totals of things in logs, and I have it writing the totals out every minute. I also have it write out the totals when it shuts down. But if it gets a HUP, it looses all it's current data and has no chance to write it out or do anything else with it. Different treatment of HUP is actually one of the differences between stateful and stateless unix daemons. It is a common practice to use HUP for reinitializing a daemon which normally means reopening inputs/outputs, reloading configuration and starting with a clean state. Daemons like rsyslog are largely stateless -- they don't keep any state about processed events in memory, since their main purpose is fast forwarding of events from inputs to outputs. For this reason, sending a HUP to rsyslog largely means reopening input/outputs and reloading your configuration. Thus, if you haven't modified your configuration, you can easily use HUP for log rotation, since you have no state to lose in memory. not actually true with rsyslog. If it did a re-read of it's config, lots of things can change and there's a lot of state that can be maintained. But because HUP is so commonly used for log rotation, and a full startup takes so long, HUP was redefined to just be a log rotation event (close inputs and outputs) rather than a full reparse of the config. Too many people were loosing too much data with the old way of doing things. We could have gone down the route of defining a different signal for the rotation and kept HUP as the full restart, but the result would have been more lost logs for people over the years. Did we do the wrong thing? arguable. From a purist point of view we did, from a pragmatic point of view probably not. With stateful daemons like sec and snort the HUP signal is not just about reopening input/output channels and reloading rules/signatures, but you always lose state which has been stored in memory. As a result, with stateful daemons the HUP signal can not have a dual role of reinitialization and log rotation. That is why sec uses a different signal (USR2) designed solely for log rotation. That's all well and good, but why do you loose more state when you do a HUP than when you do a full shutdown? I'm not even talking about restoring state after the HUP, I'm talking about the ability to output running counters. If I have a counter that's accumulating data that I output every hour (via a calendar command) and it's 1 sec before the end of the hour. If I do a full kill/restart I have the chance to output the data before I shutdown But if I do a HUP instead, that data is irrevokably lost. why should a full kill be more data friendly than a HUP? David Lang hi David, when we leave data storing on SEC_SHUTDOWN completely aside, and just compare HUP and full restart as such, there are data items which are not lost at HUP. Since the user can set up his custom Perl data structures from perlfunc patterns, but also with 'lcall', 'eval' and other such actions, these data structures are not cleared. Since custom perl hashes, arrays and variables are not standard sec items and might have special semantics, resetting them automatically is not a good idea, and futhermore, the user can clear
Re: [Simple-evcorr-users] intstates and HUP
2015-04-03 11:56 GMT+03:00 David Lang da...@lang.hm That's all well and good, but why do you loose more state when you do a HUP than when you do a full shutdown? I'm not even talking about restoring state after the HUP, I'm talking about the ability to output running counters. If I have a counter that's accumulating data that I output every hour (via a calendar command) and it's 1 sec before the end of the hour. If I do a full kill/restart I have the chance to output the data before I shutdown But if I do a HUP instead, that data is irrevokably lost. why should a full kill be more data friendly than a HUP? David Lang hi David, when we leave data storing on SEC_SHUTDOWN completely aside, and just compare HUP and full restart as such, there are data items which are not lost at HUP. Since the user can set up his custom Perl data structures from perlfunc patterns, but also with 'lcall', 'eval' and other such actions, these data structures are not cleared. Since custom perl hashes, arrays and variables are not standard sec items and might have special semantics, resetting them automatically is not a good idea, and futhermore, the user can clear them easily on the reception of sec_restart event. So if you keep the counters in a perl hash/array, they will survive HUP. ahh, I didn't realize this. I assumed that in clearing everything it would clear the entire namespace. I also had a look at the official documentation and realized that this aspect should be explicitly mentioned. That's another thing to fix in the next version, apart from few other documentation improvements. But if someone was using the report functionality, that would be cleared, and giving them a chance to do something with the data first is a good idea. However, I do see your point that SEC_SHUTDOWN and SEC_STARTUP events allow to save and restore state when shutting down a process and starting a new one, while SEC_RESTART alone does not allow for the same functionality with HUP. I suppose we could add another synthetic event like SEC_PENDING_RESTART which is the last one seen before state gets dropped on HUP -- would that help to address your problem? That's the type of thing I was looking for when I started. As John suggested, we shoudl probably have pending actions for the other signals as well while we are messing in the area David Lang I will look into it and check the code how best to implement them. kind regards, risto -- Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/___ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
