The message you are looking for wasn't from me (I just started the thread), look
in the mailing list archives on Nov 5 for the subject line:
Re: [Simple-evcorr-users] dealing with JSON based logs
David Lang
On Thu, 29 Nov 2012, Boyles, Gary P wrote:
Date: Thu, 29 Nov 2012 16:17:18
easier to read, easier to parse, win-win.
David Lang
On Wed, 5 Dec 2012, Boyles, Gary P wrote:
> John,
> I would say that the 2nd format is most definitely easier to read.
> Gary Boyles
>
> -Original Message-
> From: John P. Rouillard [mailto:rou...@cs.umb.edu]
> Se
generate and to parse, but it's not that much harder. I
think the human readability of the second is worth the cost.
David Lang
--
LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
Remotely acces
Much of this formatting stuff is useful even if just writing to files, so it
may be that the right thing is to define the transport at open time and then
have a function call to format the message at log time.
thoughts?
David Lang
--
sec --intevents IIRC.
Even if the values do change occasionally, you are probably better off setting
them at startup, then re-checking them (either based of calendar rules or when
a
'reload values' pattern shows up in a log message) rat
On Thu, 7 Feb 2013, Risto Vaarandi wrote:
> On 02/06/2013 11:59 PM, David Lang wrote:
>> I think this sort of thing would be useful, a lot of 'action scripts' end up
>> being trivial wrappers to do these sorts of things, and opening/closing files
>> and starting p
sending you the message. If you only have one
Juniper, you can get away with desc being a fixed string.
David Lang
On Fri, 29 Mar 2013, Vernon Nelson wrote:
Date: Fri, 29 Mar 2013 16:12:18 -0400
From: Vernon Nelson
To: simple-evcorr-users@lists.sourceforge.net
Subject: [Simple-evcorr-users] trap
e 'internal' messages and
that instance generates all the alerts, and deals with correlation rules that
need to take into account the different feeds.
David Lang
--
Introducing AppDynamics Lite, a free troubles
TraditionalFileFormat
& ~
#:programname, isequal, "ossec" /var/log/ossec;RSYSLOG_TraditionalFileFormat
& ~
:rawmsg, contains, " netips-"
|/var/fifo/tippingpoint;RSYSLOG_TraditionalFileFormat
& ~
# filter out some high volume log entries that we don't have any r
x27;|',$_);
$known_admin_ips{$junk[0]}{$junk[1]} = 1;}; close(FILE); )
You can setup a trigger to reload the table by just sending a log line that SEC
matches with a rule like:
type=Single
desc=reLoad known admin IPs
ptype=SubStr
continue=TakeNext
pattern=reload k
ve a different way of
addressing the problems, the drawbacks from having the pipe always return EOF
seems pretty significant.
David Lang
--
How ServiceNow helps IT people transform IT departments:
1. A cloud service to a
s the logs into a report, and then a singlewiththreshold that
exports the data when it fires.
The problem is expiring the old data from the report. I can think of ways to do
this, but not clean ones.
David Lang
#more than 15 failed logins
type=singlewiththreshold
desc=Possible brute force at
try changing sec-comware to redirect stdout and stderr from sec to some file so
that you can see what it's complaining about.
David Lang
On Wed, 26 Jun 2013, Orangepeel Beef wrote:
> Date: Wed, 26 Jun 2013 16:56:04 -0700
> From: Orangepeel Beef
> To: simple-evcorr-users@lists.s
ission to write.
David Lang
>
> On Wed, Jun 26, 2013 at 4:01 PM, David Lang wrote:
>
>> try changing sec-comware to redirect stdout and stderr from sec to some
>> file so that you can see what it's complaining about.
>>
>> David Lang
>>
>> On We
the latest development version (as of yesterday, git version) of rsyslog allows
a newer syntax to be used that allows for command-line arguments to be used on
the command.
David Lang
On Thu, 27 Jun 2013, Risto Vaarandi wrote:
> Date: Thu, 27 Jun 2013 11:14:15 +0300
> From: Risto Vaarandi
> To:
ry well want to have multiple rules processing the same log line
(takenext=continue) so that you can do different things with the log messages,
but more info is needed to create the specific rules.
David Lang--
This
egy that they used can be used with SEC.
The advantage of this compared to just combining your config files is that this
can scale out to multiple instances of SEC, potentially running on multiple
machines as needed.
David Lang
---
buffer and just prepend all 10 contexts to the report. These will
> likely be out of order but at least the info is all there.
>
> But these are both lousy ideas. Does anybody have a better idea?
I would be looking to limit based on time instead of number of events.
On Thu, 22 Aug 2013, John P. Rouillard wrote:
> In message ,
> David Lang writes:
>
>> On Thu, 22 Aug 2013, John P. Rouillard wrote:
>>> In trying to troubleshoot a problem with an application, I want to
>>> include the prior 10 minutes of iostat info along with
be no logs arriving for the first 40 seconds of
a
minute.
David Lang
--
Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
Discover the easy way to master current and previous Microsoft technologies
a
Thanks, that's what I thought I was reading, but I wasn't sure.
David Lang
On Mon, 9 Sep 2013, Risto Vaarandi wrote:
> hi David,
> unfortunately, eventgroup rule does not have this particular functionality,
> and if you would like to ensure reporting precisely after 1 minu
reading the man page, it looks like they both have a window parameter, so I'm
not understanding what the difference between them are.
David Lang
--
LIMITED TIME SALE - Full Year of Microsoft Training For Just $
set
to be written to /var/tmp
I don't know if the problem is with rsyslog, SEC, or RHEL 6.x
so I figured I'd crosspost to both lists to see if anyone had an idea :-)
David Lang
--
LIMITED TIME SALE - Full Year o
On Thu, 19 Sep 2013, Risto Vaarandi wrote:
> On 09/19/2013 06:16 AM, David Lang wrote:
>> I've started running SEC from rsyslog via omprog and it's running, but when
>> it
>> tries to write the dumpfile, nothing happens.
>>
>> I did a cut-n-paste of t
On Thu, 19 Sep 2013, David Lang wrote:
> On Thu, 19 Sep 2013, Risto Vaarandi wrote:
>
>> On 09/19/2013 06:16 AM, David Lang wrote:
>>> I've started running SEC from rsyslog via omprog and it's running, but
>>> when it
>>> tries to write the dumpfi
On Mon, 23 Sep 2013, John P. Rouillard wrote:
> In message ,
> David Lang writes:
>>>> On 09/19/2013 06:16 AM, David Lang wrote:
>>>>> I've started running SEC from rsyslog via omprog and it's
>>>>> running, but when it tries to write the
ls and can be taken down only
> with SIGKILL. Apart from SIGTERM, I also tried SIGSEGV and SIGFPE which
> should all terminate 'cat', but this does not happen.
> kind regards,
> risto
>
>
> 2013/9/24 David Lang
>
>> On Mon, 23 Sep 2013, John P. Rouillard wrote:
; second action
just add a ; at the end of your existing action and put in your definition for
the second action.
David Lang
--
October Webinars: Code for Performance
Free Intel webinars can help you accelerate applicat
te with one rule
can be accessed by your perl code in any other rule.
you don't _have_ to use varmap.
If your flatten routine sets a variable %hash and then returns a reference to
it, you can have other commands just access $hash{key}
David Lang
-
On Fri, 27 Sep 2013, Mark D. Nagel wrote:
> On 9/27/2013 4:14 PM, David Lang wrote:
>>
>> remember that Perl variables (including hashes) that you create with one
>> rule can be
>> accessed by your perl code in any other rule.
>>
>> you don't _have_
onable to say that SEC requires
that your times be consistent, and that it will make reasonable attempts at
doing the right thing if they aren't, but re-processing an arbitrary amount of
old log data (most of which SEC will no longer have access to) if an old
timestamp arrives is not reason
> output them to a text file)..
Yes, SEC can read from a text file and apply it's rules to what it finds. Just
list that file as an input for SEC/
David Lang
--
October Webinars: Code for Performance
Free Intel w
_r/"]
[unique_id "Uo9nBX8AAQEAAASiAh8A"]
What i try to is, to match the words in the logs, Cross-site Scripting
AND CRITCAL.
I have check this regex with online checker, it can mathced, but
unfortunately not able to match in SEC.
pattern=(Cross-site Scripting \
nd is that time limits in SEC are based on wall-clock
time, not the timestamps in the log messages. So if a UDP flood attack causes
SEC to bog down and get way behind in processing messages, it's possible that
some of your other rules may not act the way you expect them to.
David Lang
On T
ts of snmptt_cisco.conf, starting
from about line 110 where the errors are being reported.
If the contents are what's listed in () of the error message, the answer is that
whatever is in that file isn't a valid SEC configuration fil
pattern2= Loss Of Signal
the first thing that jumps out at me is that there is a very strong probability
that the same log line will match pattern and pattern2, I think this can cause a
lot of confusion.
David Lang
desc2=_lossOfSignal:Flapping
action2=write - Loss Of Signal Flapping
can you include a sample of the relevant log messages?
David Lang
On Wed, 29 Jan 2014, George Lakovski wrote:
Date: Wed, 29 Jan 2014 16:28:19 +0200
From: George Lakovski
To: simple-evcorr-users@lists.sourceforge.net
Subject: [Simple-evcorr-users] RANCID SEC Cisco intergration
Hi,
I need
you would need to create it with the mkfifo command.
David Lang
On Thu, 6 Feb 2014, andrewarnier wrote:
Date: Thu, 6 Feb 2014 16:19:07 +0800
From: andrewarnier
To: simple-evcorr-users@lists.sourceforge.net
Subject: [Simple-evcorr-users] missing SEC_fifo
Hi all,
I want to using SEC
nd explain what you are trying to do
you want to see a particular message from snort, then do what?
David Lang--
Managing the Performance of Cloud-Based Applications
Take advantage of what the Cloud has to offer - Av
what do you have writing logs to the fifo?
remember, that's not a file, it's a connection point. Think of it as the
equivalent of | in a shell statement, just showing up on the filesystem.
David Lang
On Tue, 11 Feb 2014, andrewarnier wrote:
> Hi restro,
> When I running /bin/
short answer: Yes, everything in SEC can be used for correlating (or if it
can't, it should be removed)
now, that wasn't the answer to what you really wanted to know, so could you try
reframing your question?
David Lang
On Thu, 6 Mar 2014, Rolf Nufable wrote:
could the jump
people want to do the exact same
thing in response to the exact same conditions.
What is it that you want to monitor for?
David Lang
On Sun, 6 Apr 2014, Tim Peiffer wrote:
> I have a co-worker that is trying to introduce me to Raspberry Pi and
> his facilities for monitoring an HD TV h
27;s still running and what logs it
generates.
I know this seems unhelpful, but even when you spend 7 figures on a log
alerting
system, default rules are still worthless, and you spend a lot of time creating
custom rules (or a lot of money paying the vendor professional services to
write
the r
> shellcmd or another way.
rather than sending this to an external script, have you looked at using a perl
snippit inside SEC?
you avoid the need to do a regex and then send it out, and you avoid the
startup
overhead of the external scri
Keep in mind that there are several things that can cause SEC to see the logs in
a different order than they were generated in, so be careful about ordering
requirements.
David Lang
On Thu, 26 Jun 2014, Risto Vaarandi wrote:
For detecting sequences of events, you could use the following
Even on from the same device there are numerous things that can reorder logs,
the network can reorder logs, rsyslog can end up reordering logs, etc.
It doesn't happen a lot, but if you depend on the order, you will miss
correlations.
David Lang
On Mon, 30 Jun 2014, Risto Vaarandi
include the exact string '%deploymentId' and not its value in
> order to match (e.g. "status of %deploymentId to dead").
right, variables defined in one rule only last for that rule
> There are at least two ways to make this work:
>
> 1) use a context to tie the tw
yslog-ng.
take a look at this paper
http://static.usenix.org/events/lisa10/tech/full_papers/Krizak.pdf it's not
about SEC, but the ideas presented for splitting the work across multiple
machines, but then combining the results is applicable.
David Lang
-arm and fire every 2 hours, but
until I can get the simple version working, trying to extend it beyond that
isn't happening.
David Lang
--
Dive into the World of Parallel Programming The Go Parallel Website, sponso
context lifetime (and taking other measures for
> preventing context deletion) will have no effect when invoked from
> action-on-expire action list.
I was afraid it was something like this.
I think that what I'll try is to replace the create action with a log action,
and then have an
state is cleared, but it doesn't look like SEC_SHUTDOWN takes place prior to
the
HUP.
Am I missing something? or is there no way to save state when a HUP is received?
David Lang
--
Dive into the World of Parallel P
On Tue, 31 Mar 2015, John P. Rouillard wrote:
> Hi David:
>
> In message ,
> David Lang writes:
>> before I realized that I really only needed USR2, I was sending sec
>> a HUP to get it to close all it's output files (for log rotation),
>> but I was running in
ions at both ends to deal with signals like this. But I'm
not sure that this completely avoids the uses of being able to do something
before data is lost when receiving something other than a full shutdown.
> Hope this helps,
It does, thanks.
David Lang
> risto
>
>
> 2015-04-01 9
me that my ;login article has gone past the paywall stage and is
now available for free.
https://www.usenix.org/system/files/login/articles/09_lang-online.pdf
David Lang--
Dive into the World of Parallel Programming The G
y more time parsing or matching data than it needs
to.
David Lang--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub f
On Thu, 2 Apr 2015, Risto Vaarandi wrote:
> 2015-04-02 1:30 GMT+03:00 David Lang :
>
>> On Wed, 18 Mar 2015, Risto Vaarandi wrote:
>>
>> hi all,
>>>
>>> last week, I had a conference presentation "Simple Event Correlator - Best
>>> P
On Thu, 2 Apr 2015, Risto Vaarandi wrote:
> 2015-04-02 1:22 GMT+03:00 David Lang :
>
>> On Wed, 1 Apr 2015, Risto Vaarandi wrote:
>>
>> hi David,
>>>
>>> is my understanding correct that you would like to have pre_restart event,
>>> in order
On Fri, 3 Apr 2015, Risto Vaarandi wrote:
> 2015-04-03 3:57 GMT+03:00 David Lang :
>
>> On Thu, 2 Apr 2015, Risto Vaarandi wrote:
>>
>> 2015-04-02 1:22 GMT+03:00 David Lang :
>>>
>>> On Wed, 1 Apr 2015, Risto Vaarandi wrote:
>>>>
>>>
nt rules that match the different log entries
and then have a rule that looks for both contexts to be raised and generate an
alert at that time (possibly including data from each of the two contexts)
David Lang--
:16:02 2015: Deleting SEC internal context 'SEC_INTERNAL_EVENT'
what can I do to try and get more info from sec about what it's seeing happen?
This was happening every few weeks, but today it's happening much more
frequently (twice in a minute in the sample logs above)
David La
ith log rotation, but while I see
some of this that is happening at the log rotation time, I'm seeing other times
well clear of the minute boundry when log rotation takes place.
David Lang
> Kind regards, risto
> On Jul 15, 2015 10:29 PM, "David Lang" wrote:
>
>>
On Thu, 16 Jul 2015, Risto Vaarandi wrote:
> 2015-07-15 23:07 GMT+03:00 David Lang :
>
>> On Wed, 15 Jul 2015, Risto Vaarandi wrote:
>>
>> Hi David,
>>> I noticed that sec is running without --notail option, but this causes sec
>>> to stay around even
esulting file to see the most recent logs it's processed. That will tell you if
it's way behind (although sec using 100% cpu for any significant amountof time
will tell you is is not
processing logs.
David Lang
On Tue, 25 Aug 2015, Ganji, Shashirekha Yadav wrote:
> David,
>
> We are forwarding all devices logs to syslog server and using different
> facilities based on the technologies.
>
> I see actual device logs coming around 8:00pm on our syslog local files
were
> perfectly fine without any delays.
only if all events are in the same file. If the different alerts are in
different files, it could be behind in one compared to the others.
David Lang
> Thanks,
> shashi
>
> -Original Message-
> From: David Lang [mailto:da...@
to
match and run multiple instances of sec.
and keep an eye on the cpu utilization to catch if it's maxing out.
David Lang
> Thanks,
> shashi
>
> -Original Message-
> From: Ganji, Shashirekha Yadav
> Sent: Monday, August 24, 2015 5:11 PM
> To: 'David Lang
done
> processing?
the variables are local to the rule, but you can use varmap to save them from
one rule and access them from others (this is designed to avoid needing to do
the parsing multiple times)
David Lang
-
to doing this
manually (detect the first event, set a context that will expire in X tiem and
do nothing when it expires)
pair of events where it notices the first event and if the second
has anyone put together the code that would be needed to detect if sec or log
delivery is falling behind? something along the order of 'if the timestamp in
the logs is > X min behind current, alert'
t; (sub {perl code here});pipe 'SEC is behind %o minutes. Log
time: $1' /bin/mailx -s "SEC: %t > SEC is behind 1+ minutes." u...@somewhere.edu
David Lang
On Mon, 4 Jan 2016, Todd M. Hall wrote:
> Date: Mon, 4 Jan 2016 13:25:26 -0600 (CST)
> From: Todd M. Hall
&
et matched
with the time= line) put into the $1-$5 variables, it would make this more
reliable, and shouldn't hurt any existing configs, because they can't be using
$
does this sound like a reasonable thing to do? and does it sound like an easy
thin
lookup the info anyway (to decide if the
rule matches), so setting it in a way that can be retrieved should be
cheap/free.
David Lang
On Wed, 10 Feb 2016, Risto Vaarandi wrote:
> when thinking quickly about it, it might be a better idea to provide time
> related data through action list vari
arsing is horribly slow)
then in the action, I call sec with a bunch of parameters so that it logs to a
file (but not too much), has a dumpfile defined, creates events and contexts for
startup/shutdown/restart, and when rsyslog is sent a HUP to roll it's logs, sec
will get USR2 instead
we may as well add them in. I know I've run across a few oddities and
it
seems better to add them all than to play wack-a-mole on them over time.
As for the other variables, they seem to cover things, but I wish there was a
good way to not have three versions of everything. But I guess that w
haracters if we
think we need them (although most should be able to be entered with normal perl
escaping rules)
do we need variables for characters that are considered special by SEC?
(quotes,
semicolens, percent, etc?) does it make sense to just define the entire ascii
set?
David Lang
&g
et a context with a timer for 10 min, when the canelation comes in delete
the context. If the timer expires, it will take the action specified to alert.
David Lang
--
___
S
I think 4K works well, larger may end up getting broken up by the OS in any
case. Any benchmarks showing improvement above 4K?
David Lang
--
Check out the vibrant tech community on one of the world's most
en
now how to extend this timeout with systemd (it's always possible that
they decided that nobody needed to do that, but I'd bet that there is a way
somewhere)
David Lang
--
Check out the vibrant tech community on one
try writing the log to /dev/log rather than sending it over the network
if you are needing to send it over the network, check your buffers, are they
getting full (and is your receiving syslog daemon keeping up)
On Tue, 27 Aug 2019, Santhosh Kumar wrote:
Date: Tue, 27 Aug 2019 10:55:44 +0900
restart more
readily.
David Lang
___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
messages to say that their part of the correlation has tested
true and then have another instance that processes these partial correlation
messages and decide if the combined correlation is matched.
David Lang
___
Simple-evcorr-users mailing list
Simple
in avg).
Od: David Lang
Odoslané: štvrtok 26. marca 2020 1:24
Komu: Richard Ostrochovský
Kópia: simple-evcorr-users@lists.sourceforge.net
Predmet: Re: [Simple-evcorr-users] SEC CPU utilization
Much of the time you can trivially split your rules and then run
infrastructure, ideally on a dedicated system (or set of systems) so having
multiple instances, each eating a core, is a feature not a bug ;-)
David Lang
On Thu, 2 Apr 2020, Richard Ostrochovský wrote:
Date: Thu, 2 Apr 2020 00:15:08 +0200
From: Richard Ostrochovský
To: simple-evcorr-users
perl operations on a hash are surprisingly efficient. If you store your context
in a hash, it can be very efficient to add/remove/check specific items. What is
not efficient is aging things out based on time.
David Lang
On Thu, 2 Apr 2020, Richard Ostrochovský wrote:
Date: Thu, 2 Apr 2020
most syslog servers have the ability to log with subsecond accuracy, and
RFC-5424 requires it in the spec.
David Lang
___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple
85 matches
Mail list logo
