2015-04-02 1:22 GMT+03:00 David Lang <da...@lang.hm>:
> On Wed, 1 Apr 2015, Risto Vaarandi wrote:
>
> hi David,
>>
>> is my understanding correct that you would like to have pre_restart event,
>> in order to execute the following steps:
>> 1) when HUP is received, match pre_restart synthetic event and save event
>> correlation state to disk,
>> 2) let the HUP handling procedure to drop all event correlation state from
>> memory,
>> 3) match sec_restart synthetic event and load event correlation state from
>> disk back to memory
>>
>> If my understanding is correct and you want to use the above sequence for
>> making HUP a log rotation signal, there would be a number of non-trivial
>> issues to handle. Firstly, while some state like contexts can be stored
>> and
>> restored to/from disk in a straightforward way, things get a bit more
>> complex with storing/restoring event correlation operations. Secondly,
>> receiving HUP also means that all inputs will be closed and reopened, and
>> at reopen sec always jumps to the very end of each input file. This means
>> that any lines appended to input files between close and reopen will be
>> missed. In short, I would never recommend using the HUP signal just for
>> rotating log files, since HUP's true purpose is doing full restart (in
>> fact, it is almost identical to taking sec process down and starting a new
>> instance).
>>
>
> In my use case, the input is a pipe, so unless sec is falling behind,
> little, if anything would be lost.
>
> I agree it's not the best thing to do, for rotation, but here's another
> case.
>
> I have an instance of SEC that's keeping running totals of things in logs,
> and I have it writing the totals out every minute. I also have it write out
> the totals when it shuts down.
>
> But if it gets a HUP, it looses all it's current data and has no chance to
> write it out or do anything else with it.
>
Different treatment of HUP is actually one of the differences between
stateful and stateless unix daemons. It is a common practice to use HUP for
reinitializing a daemon which normally means reopening inputs/outputs,
reloading configuration and starting with a clean state. Daemons like
rsyslog are largely stateless -- they don't keep any state about processed
events in memory, since their main purpose is fast forwarding of events
from inputs to outputs. For this reason, sending a HUP to rsyslog largely
means reopening input/outputs and reloading your configuration. Thus, if
you haven't modified your configuration, you can easily use HUP for log
rotation, since you have no state to lose in memory.
With stateful daemons like sec and snort the HUP signal is not just about
reopening input/output channels and reloading rules/signatures, but you
always lose state which has been stored in memory. As a result, with
stateful daemons the HUP signal can not have a dual role of
reinitialization and log rotation. That is why sec uses a different signal
(USR2) designed solely for log rotation.
If one wants to reload modified rules without doing full reinit and losing
all state, the ABRT signal should be used. If sec has been started with the
--keepopen command line option, sec will even keep input files open on the
reception of ABRT, so there is no danger of losing data. (Actually, the
current default command line setting is --nokeepopen, but maybe we should
change it to --keepopen in the next version.)
>
> Since you mentioned a discussion in rsyslog mailing list about forwarding
>> HUP signal from rsyslog to its child processes, and possibly replacing HUP
>> with another signal, there was a somewhat similar discussion in the sec
>> mailing list few years ago. The main idea of the discussion was the
>> limitation of child process runtime, and a relevant recipe was posted to
>> the SEC FAQ: http://simple-evcorr.sourceforge.net/FAQ.html#20
>>
>
> I was starting from #15
>
>
> The above recipe can be easily reused for handling your problem by rolling
>> out a similar Perl wrapper:
>>
>> #!/usr/bin/perl -w
>> #
>> # wrapper.pl -- this wrapper starts sec and if HUP is sent to
>> # the wrapper, it sends USR2 signal to the sec process
>> #
>> if (scalar(@ARGV) < 1) { exit(1); }
>> $pid = fork();
>> if ($pid == -1) {
>> exit(1);
>> } elsif ($pid == 0) {
>> exec("@ARGV");
>> } else {
>> $SIG{HUP} = sub { kill USR2, $pid; };
>> $SIG{TERM} = sub { kill TERM, $pid; exit(0); };
>> waitpid($pid, 0);
>> exit($? >> 8);
>> }
>>
>>
>> If you start the following command line from rsyslog:
>> wrapper.pl /usr/bin/sec --conf=/etc/sec/sec.conf --notail --input=-
>> every HUP signal that is forwarded to wrapper.pl will be replaced with
>> USR2
>> and sent to sec. Also, since the sec process inherits its standard input
>> from the wrapper, it can receive input from the pipe that rsyslog has
>> created. While using a wrapper might initially seem a bit complex, rsyslog
>> v5 has to use wrapper script anyway for programs with command line options
>> (one example can be found at http://simple-evcorr.
>> sourceforge.net/FAQ.html#2).
>> With v5 you can easily modify the above wrapper for handling both tasks,
>> so
>> no additional complexity will be added to your setup.
>>
>
> I'm using v8.8, and 7.x is in current distros, so v5 stuff is very quickly
> becoming distant legacy.
>
I see. Nevertheless, the wrapper is probably the simplest way to remap
signals without changing the current rsyslog code.
> It's good to have options at both ends to deal with signals like this. But
> I'm not sure that this completely avoids the uses of being able to do
> something before data is lost when receiving something other than a full
> shutdown.
>
If you are a bit worried about USR2 semantics, then it only closes all
outputs -- network and unix file system sockets, output file handles opened
by 'write*' actions, connection to syslog and sec's own log file.
Connection to syslog and sec's log file will be immediately reopened, while
sockets and output files are reopened on demand -- whenever you have a next
invocation of 'tcpsock', 'write' and other such action, the action opens
the relevant output channel again. Therefore, you can't possibly lose state
from memory or miss input events, since input channels and event
correlation state are not touched.
kind regards,
risto
>
> Hope this helps,
>>
>
> It does, thanks.
>
> David Lang
>
>
> risto
>>
>>
>> 2015-04-01 9:54 GMT+03:00 David Lang <da...@lang.hm>:
>>
>> On Tue, 31 Mar 2015, John P. Rouillard wrote:
>>>
>>> Hi David:
>>>>
>>>> In message <alpine.deb.2.02.1503311452250.26...@nftneq.ynat.uz>,
>>>> David Lang writes:
>>>>
>>>>> before I realized that I really only needed USR2, I was sending sec
>>>>> a HUP to get it to close all it's output files (for log rotation),
>>>>> but I was running into the problem that the HUP restart looses all
>>>>>
>>>> context.
>>>
>>>>
>>>> Not just context but all correlation operations and variables (however
>>>> variables set explicitly via perl functions/expressions IIRC aren't
>>>> flushed).
>>>>
>>>> There is the SEC_RESTART event that shows up after the HUP finishes and
>>>>> all state is cleared, but it doesn't look like SEC_SHUTDOWN takes place
>>>>> prior to the HUP.
>>>>>
>>>>
>>>> Yeah I can see the agrument for that, but I would really expect a
>>>> SEC_SHUTDOWN only when it's really being shut down.
>>>>
>>>> Am I missing something? or is there no way to save state when a HUP
>>>>> is received?
>>>>>
>>>>
>>>> I don't think you are missing anything. I am not sure if it makes sense
>>>>
>>> to
>>>
>>>> create events for:
>>>>
>>>> PRE_SEC_RESTART - this event will be the last event that SEC sees before
>>>> reloading its configuration (HUP)
>>>>
>>>> PRE_SEC_SOFTRESTART - this event will be the last event that SEC sees
>>>> before reloading its configuration (ABRT)
>>>>
>>>> PRE_SEC_LOGROTATE - this event will be the last event that SEC sees
>>>> before reopening its log file and closing its outputs (USR2)
>>>>
>>>> just for completeness.
>>>>
>>>> I can't think of an occasion where I would send a HUP to a SEC and
>>>> want to retain the state. (Then again I don't find keeping state and
>>>> reloading it after a shutdown/restart (SIGTERM/ service restart) to be
>>>> very useful given all the caveats associated with it.)
>>>>
>>>> I am interested in what your use case would be for these theoretical
>>>> events.
>>>>
>>>
>>> This started with me running sec from rsyslog and hitting the file
>>> rotation
>>> problem because when rsyslog is sent a HUP to have it close it's outputs,
>>> sec
>>> doesn't receive anything. So I slapped a quick HUP for sec in my rotate
>>> script
>>> and reported this on the rsyslog mailing list. A day later, there's a
>>> feature in
>>> rsyslog git so that when a HUP is sent to rsyslog, it will send a HUP on
>>> to
>>> programs that it's started.
>>>
>>> Thinks seemed to be working, until I setup another ruleset in sec to
>>> alert
>>> for
>>> logs disappearing on my (if the last log is >3 min ago, send one e-mail),
>>> but
>>> this wasn't firing reliably because the logs were rotating every 5 min
>>> and
>>> the
>>> HUP erased the sec history.
>>>
>>> so I changed my rotate script to send USR2 and reported this to the
>>> rsyslog list
>>> to see about making it so that when rsyslog receives a HUP it can send a
>>> different signal to it's child processes.
>>>
>>> That's pending, but in the meantime, as I'm working on various sec
>>> rulesets, I
>>> may have a bunch of different copies of sec running, each receiving a
>>> different
>>> feed (different subsets of traffic, and different contents in what is
>>> being
>>> sent). It's FAR easier to do killall -HUP sec than to figure out which of
>>> the
>>> several copies of sec I need to send the signal to.
>>>
>>> So, eventually, I will be able to work around the issues that triggered
>>> my
>>> desire for a pre_restart event. But as the system gets more complex and
>>> grows
>>> more instances of sec running, working around the lack is going to keep
>>> getting
>>> more complex.
>>>
>>> David Lang
>>>
>>>
>>> ------------------------------------------------------------
>>> ------------------
>>> Dive into the World of Parallel Programming The Go Parallel Website,
>>> sponsored
>>> by Intel and developed in partnership with Slashdot Media, is your hub
>>> for
>>> all
>>> things parallel software development, from weekly thought leadership
>>> blogs
>>> to
>>> news, videos, case studies, tutorials and more. Take a look and join the
>>> conversation now. http://goparallel.sourceforge.net/
>>> _______________________________________________
>>> Simple-evcorr-users mailing list
>>> Simple-evcorr-users@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>>>
>>>
>>
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
