Risto, I cannot change the timestamps, as they are coded in the files I'm working with (old data). Going forward, I also cannot guarantee that I will have control over the source data, and it is likely that it will have a similar granularity. Finally, there's no need for that level of granularity downstream; it is only necessary that two events with the same timestamp are sequenced so that it is clear which happened first.
It does seem a bit kludgy, but if I try to reduce the kludge in my SEC file, I am almost certain to have to introduce kludge somewhere else, which is worse (this way I only have SEC to maintain, rather than additional technology). This way, I can feed relevant log files to SEC from any source, and trust that I'll be able to handle them, so long as they follow standard Asterisk log format. Thanks again for your generous advice. Jim -- Jim Van Meggelen ClearlyCore Inc. +1-416-639-6001 (DID) +1-877-253-2716 (Canada) +1-866-644-7729 (USA) +1-416-425-6111 x6001 jim.vanmegge...@clearlycore.com [ http://www.clearlycore.com/ | http://www.clearlycore.com ] Asterisk: The Definitive Guide FIFTH EDITION NOW AVAILABLE TO DOWNLOAD: [ https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf | https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf ] > From: "Risto Vaarandi" <risto.vaara...@gmail.com> > To: "Jim Van Meggelen" <jim.vanmegge...@clearlycore.com> > Cc: "simple-evcorr-users" <simple-evcorr-users@lists.sourceforge.net> > Sent: Saturday, 23 September, 2023 03:51:02 > Subject: Re: [Simple-evcorr-users] Storing a sequence counter in a context > hi Jim, > the solutions from my previous post were provided without knowing the root > cause > of the issue. If the main reason for the problem is the nature of the > timestamps in the logs (that is, they are provided with an accuracy of a > second), I would recommend using a different logging scheme with > high-resolution timestamps. For example, as David has mentioned in his post, > using syslog servers with RFC5424-based syslog protocol ( [ > https://datatracker.ietf.org/doc/html/rfc5424 | > https://datatracker.ietf.org/doc/html/rfc5424 ] ) provides such timestamps > out-of-the-box. I am not sure whether you can change the logging scheme for > these particular application messages, but if you can, these timestamps are > highly useful for many purposes like the investigation of past system faults > and security incidents. > kind regards, > risto >> hi Jim, >> let me provide some suggestions how to accomplish this task. First, you could >> utilize the context based approach that you have described in your post. In >> that case, the numeral that has been retrieved from the context with the >> 'pop' >> action needs to be incremented, and you can do it with the 'lcall' action >> that >> invokes the relevant Perl code. Here is one example (I have added several of >> extra actions into the rule for initializing the context): >> type=single >> ptype=regexp >> pattern=call_id (\S+) >> desc=call id $1 >> action=exists %e call_uniqueid_seq_$1; \ >> if %e ( none ) else ( add call_uniqueid_seq_$1 0); \ >> pop call_uniqueid_seq_$1 %sequence_num; \ >> lcall %sequence_num %sequence_num -> ( sub { $_[0] + 1 } ); \ >> add call_uniqueid_seq_$1 %sequence_num >> The first two actions check if the context call_uniqueid_seq_* for the given >> call exists, and if not, 0 is written into that context to initialize it: >> exists %e call_uniqueid_seq_$1; \ >> if %e ( none ) else ( add call_uniqueid_seq_$1 0); >> The 'pop' and 'add' actions (the 3rd and 5th action in the above rule) are >> identical to the actions from your email. >> The 'lcall' action (4th action in the above rule) increments the counter >> value >> retrieved from the context by executing the following Perl one-line function: >> sub { $_[0] + 1 } >> That function takes the value of its parameter (that is, %sequence_num) and >> returns an incremented value which is stored back into %sequence_num. Note >> that >> %sequence_num appears twice in the 'lcall' action, since it acts both as an >> input parameter for the Perl function and as the variable where the return >> value from the function is stored. >> There is also an alternative way for addressing the same assignment. This >> approach involves keeping the counter for each call ID not in a SEC context, >> but rather in a Perl hash table. That approach is more efficient, since the >> counter value is not copied back and forth between the Perl function and the >> SEC context each time it needs to be incremented. Instead, the counter value >> stays in a Perl data structure during these increments. Here is a relevant >> rule >> example: >> type=single >> ptype=regexp >> pattern=call_id (\S+) >> desc=call id $1 >> action=lcall %sequence_num $1 -> ( sub { ++$calls{$_[0]} } ); \ >> write - the new value for counter for call $1 is %sequence_num >> The above example involves the use of a Perl hash (associative array) >> 'calls', >> where the call ID serves as a key into the hash. For each key (call ID), the >> hash stores the counter for that call ID. Each time the function >> sub { ++$calls{$_[0]} } >> gets called, the counter for the relevant call ID (the value of $1) is >> incremented in the hash 'calls', and the result is stored to %sequence_num. >> If you just want to retrieve the current value of the counter for some call >> ID >> (the value of $1) without incrementing it, you can simply invoke the >> following >> action: >> lcall %sequence_num $1 -> ( sub { $calls{$_[0]} } ) >> With both the context-based and Perl-hash-based approach, one needs to think >> about dropping stale contexts (or stale hash keys) for calls that are no >> longer >> relevant. With contexts, you can remove them with the 'delete' action. In the >> case you use the Perl hash for storing the counters, you can use the >> following >> 'lcall' action for deleting a key from the hash 'calls': >> lcall %sequence_num $1 -> ( sub { delete $calls{$_[0]} } ) >> I hope the above examples are useful. >> kind regards, >> risto >> Kontakt Jim Van Meggelen (< [ mailto:jim.vanmegge...@clearlycore.com | >> jim.vanmegge...@clearlycore.com ] >) kirjutas kuupäeval R, 22. september 2023 >> kell 19:50: >>> I am using SEC for an atypical use case, where rather than using log events >>> to >>> correlate events for things like security events, I am using it to parse >>> through Asterisk logs in order to capture a sequence of phone call events >>> (call >>> comes in, auto attendant answers, user selects a digit, call is transferred, >>> etc). I have SEC write each to a CSV file, which can then be processed >>> downstream for reporting (for example, I produced a Sankey chart using this >>> data). >>> SEC has proven fantastic for this, however one minor issue has been that the >>> timestamps in the logs are not granular to smaller than a second, so it's >>> possible for two or more events to occur within the same second. Generally >>> this >>> doesn't cause a problem, but when sorting the CSV file elsewhere, this can >>> result in out-of-sequence lines, if they both contain the exact same >>> timestamp. >>> So, what I've been trying to figure out is how to store a counter that is >>> tied >>> to the uniqueid of the call, and increment that with each event. I figured >>> I'd >>> be able to do this by storing an integer in the event store of a context >>> (tied >>> to the uniqueid for that call). I can then increment it as the various log >>> lines of the call are processed in turn.
_______________________________________________ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
