2015-04-03 3:57 GMT+03:00 David Lang <da...@lang.hm>:
> On Thu, 2 Apr 2015, Risto Vaarandi wrote:
>
> 2015-04-02 1:22 GMT+03:00 David Lang <da...@lang.hm>:
>>
>> On Wed, 1 Apr 2015, Risto Vaarandi wrote:
>>>
>>> hi David,
>>>
>>>>
>>>> is my understanding correct that you would like to have pre_restart
>>>> event,
>>>> in order to execute the following steps:
>>>> 1) when HUP is received, match pre_restart synthetic event and save
>>>> event
>>>> correlation state to disk,
>>>> 2) let the HUP handling procedure to drop all event correlation state
>>>> from
>>>> memory,
>>>> 3) match sec_restart synthetic event and load event correlation state
>>>> from
>>>> disk back to memory
>>>>
>>>> If my understanding is correct and you want to use the above sequence
>>>> for
>>>> making HUP a log rotation signal, there would be a number of non-trivial
>>>> issues to handle. Firstly, while some state like contexts can be stored
>>>> and
>>>> restored to/from disk in a straightforward way, things get a bit more
>>>> complex with storing/restoring event correlation operations. Secondly,
>>>> receiving HUP also means that all inputs will be closed and reopened,
>>>> and
>>>> at reopen sec always jumps to the very end of each input file. This
>>>> means
>>>> that any lines appended to input files between close and reopen will be
>>>> missed. In short, I would never recommend using the HUP signal just for
>>>> rotating log files, since HUP's true purpose is doing full restart (in
>>>> fact, it is almost identical to taking sec process down and starting a
>>>> new
>>>> instance).
>>>>
>>>>
>>> In my use case, the input is a pipe, so unless sec is falling behind,
>>> little, if anything would be lost.
>>>
>>> I agree it's not the best thing to do, for rotation, but here's another
>>> case.
>>>
>>> I have an instance of SEC that's keeping running totals of things in
>>> logs,
>>> and I have it writing the totals out every minute. I also have it write
>>> out
>>> the totals when it shuts down.
>>>
>>> But if it gets a HUP, it looses all it's current data and has no chance
>>> to
>>> write it out or do anything else with it.
>>>
>>>
>> Different treatment of HUP is actually one of the differences between
>> stateful and stateless unix daemons. It is a common practice to use HUP
>> for
>> reinitializing a daemon which normally means reopening inputs/outputs,
>> reloading configuration and starting with a clean state. Daemons like
>> rsyslog are largely stateless -- they don't keep any state about processed
>> events in memory, since their main purpose is fast forwarding of events
>> from inputs to outputs. For this reason, sending a HUP to rsyslog largely
>> means reopening input/outputs and reloading your configuration. Thus, if
>> you haven't modified your configuration, you can easily use HUP for log
>> rotation, since you have no state to lose in memory.
>>
>
> not actually true with rsyslog. If it did a re-read of it's config, lots
> of things can change and there's a lot of state that can be maintained. But
> because HUP is so commonly used for log rotation, and a full startup takes
> so long, HUP was redefined to just be a log rotation event (close inputs
> and outputs) rather than a full reparse of the config. Too many people were
> loosing too much data with the old way of doing things.
>
> We could have gone down the route of defining a different signal for the
> rotation and kept HUP as the full restart, but the result would have been
> more lost logs for people over the years. Did we do the wrong thing?
> arguable. From a purist point of view we did, from a pragmatic point of
> view probably not.
>
> With stateful daemons like sec and snort the HUP signal is not just about
>> reopening input/output channels and reloading rules/signatures, but you
>> always lose state which has been stored in memory. As a result, with
>> stateful daemons the HUP signal can not have a dual role of
>> reinitialization and log rotation. That is why sec uses a different signal
>> (USR2) designed solely for log rotation.
>>
>
> That's all well and good, but why do you loose more state when you do a
> HUP than when you do a full shutdown?
>
> I'm not even talking about restoring state after the HUP, I'm talking
> about the ability to output running counters.
>
> If I have a counter that's accumulating data that I output every hour (via
> a calendar command) and it's 1 sec before the end of the hour.
>
> If I do a full kill/restart I have the chance to output the data before I
> shutdown
>
> But if I do a HUP instead, that data is irrevokably lost.
>
> why should a full kill be more data friendly than a HUP?
>
> David Lang
>
hi David,
when we leave data storing on SEC_SHUTDOWN completely aside, and just
compare HUP and full restart as such, there are data items which are not
lost at HUP. Since the user can set up his custom Perl data structures from
perlfunc patterns, but also with 'lcall', 'eval' and other such actions,
these data structures are not cleared. Since custom perl hashes, arrays and
variables are not standard sec items and might have special semantics,
resetting them automatically is not a good idea, and futhermore, the user
can clear them easily on the reception of sec_restart event. So if you keep
the counters in a perl hash/array, they will survive HUP.
However, I do see your point that SEC_SHUTDOWN and SEC_STARTUP events allow
to save and restore state when shutting down a process and starting a new
one, while SEC_RESTART alone does not allow for the same functionality with
HUP. I suppose we could add another synthetic event like
SEC_PENDING_RESTART which is the last one seen before state gets dropped on
HUP -- would that help to address your problem?
kind regards,
risto
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users