Re: [graylog2] debugging pipelines is... difficult

2016-07-18 Thread Edmundo Alvarez 
I added this Github issue so you can track the issue I mentioned in point 
number 2: 
https://github.com/Graylog2/graylog-plugin-pipeline-processor/issues/46

Cheers,
Edmundo

> On 18 Jul 2016, at 10:51, Edmundo Alvarez  wrote:
> 
> I spent some time debugging the issue, and I found two of them:
> 
> 1. The when expression should be wrapped in a "to_bool" function, otherwise 
> the parser gets confused about it and replaces it with "false":
>
> to_bool(regex("[^0-9a-zA-Z]([0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+)[^0-9a-zA-Z].*[^0-9a-zA-Z]([0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+)[^0-9a-zA-Z]",to_string($message.message)).matches)
> 
> 2. There seems to be some problems when handling strings containing 
> backslashes. You need to escape them so they get parsed, but then the escape 
> character is still being used in the regular expression. I will investigate 
> further and keep you posted on that.
> 
> Cheers,
> Edmundo
> 
>> On 13 Jul 2016, at 12:31, Jason Haar  wrote:
>> 
>> 
>> On Mon, Jul 11, 2016 at 11:28 AM, Jason Haar  wrote:
>> If I take the regex I wrote in this rule (as per first email), replace '\\' 
>> with '\', then the regex works fine via egrep. It's a simple "when, do this" 
>> type statement: I can't see what's gone wrong in it
>> 
>> Oh - and thanks to your comment about the regex needing to match the entire 
>> line, I put ".*" at the beginning and end - but it made no difference. Still 
>> no Cisco syslog messages (as above) match
>> 
>> 
>> -- 
>> Cheers
>> 
>> Jason Haar
>> Information Security Manager, Trimble Navigation Ltd.
>> Phone: +1 408 481 8171
>> PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Graylog Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to graylog2+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/graylog2/CAFChrgJZng%2Bzc-iZ%2Bv73%2Bd8Q6YatVATaDtj2R%3Dd7sR9iXZfbHQ%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/BA27A691-42D6-46BD-80B5-988211F400B3%40graylog.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] debugging pipelines is... difficult

2016-07-18 Thread Edmundo Alvarez 
I spent some time debugging the issue, and I found two of them:

1. The when expression should be wrapped in a "to_bool" function, otherwise the 
parser gets confused about it and replaces it with "false":

to_bool(regex("[^0-9a-zA-Z]([0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+)[^0-9a-zA-Z].*[^0-9a-zA-Z]([0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+)[^0-9a-zA-Z]",to_string($message.message)).matches)

2. There seems to be some problems when handling strings containing 
backslashes. You need to escape them so they get parsed, but then the escape 
character is still being used in the regular expression. I will investigate 
further and keep you posted on that.

Cheers,
Edmundo

> On 13 Jul 2016, at 12:31, Jason Haar  wrote:
> 
> 
> On Mon, Jul 11, 2016 at 11:28 AM, Jason Haar  wrote:
> If I take the regex I wrote in this rule (as per first email), replace '\\' 
> with '\', then the regex works fine via egrep. It's a simple "when, do this" 
> type statement: I can't see what's gone wrong in it
> 
> Oh - and thanks to your comment about the regex needing to match the entire 
> line, I put ".*" at the beginning and end - but it made no difference. Still 
> no Cisco syslog messages (as above) match
> 
> 
> -- 
> Cheers
> 
> Jason Haar
> Information Security Manager, Trimble Navigation Ltd.
> Phone: +1 408 481 8171
> PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Graylog Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/graylog2/CAFChrgJZng%2Bzc-iZ%2Bv73%2Bd8Q6YatVATaDtj2R%3Dd7sR9iXZfbHQ%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/4A90E6BA-9C9C-4D9C-ADE8-787ADEFB1D54%40graylog.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] debugging pipelines is... difficult

2016-07-13 Thread Jason Haar 
On Mon, Jul 11, 2016 at 11:28 AM, Jason Haar  wrote:

> If I take the regex I wrote in this rule (as per first email), replace
> '\\' with '\', then the regex works fine via egrep. It's a simple "when, do
> this" type statement: I can't see what's gone wrong in it
>

Oh - and thanks to your comment about the regex needing to match the entire
line, I put ".*" at the beginning and end - but it made no difference.
Still no Cisco syslog messages (as above) match


-- 
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +1 408 481 8171
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/CAFChrgJZng%2Bzc-iZ%2Bv73%2Bd8Q6YatVATaDtj2R%3Dd7sR9iXZfbHQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] debugging pipelines is... difficult

2016-07-10 Thread Jason Haar 
On Fri, Jul 8, 2016 at 10:32 PM, Edmundo Alvarez 
wrote:

>
> It's hard to tell what is wrong from here, since we can't exactly see how
> your messages look like. Could you share a couple of messages with us?
>
> Please be aware that at the moment, the "regex" function needs to match
> the whole string:
> https://github.com/Graylog2/graylog-plugin-pipeline-processor/issues/35
>

Sure thing

So that rule is attempting to extract any TWO ipv4 addresses detected in
any form of syslog message. So to give you some examples of when that
occurs, we'll look to Cisco firewalls/routers


%ASA-4-106023: Deny tcp src inside:192.168.3.79/57577 dst outside:
54.171.242.51/843 by access-group "acl_inside" [0x2923dc37, 0x0]
%ASA-7-710006: EIGRP request discarded from 192.168.23.1 to
inside:224.0.0.10
%ASA-4-106023: Deny tcp src inside:192.168.4.52/62508 dst outside:
21.125.185.18/5287 by access-group

If I take the regex I wrote in this rule (as per first email), replace '\\'
with '\', then the regex works fine via egrep. It's a simple "when, do
this" type statement: I can't see what's gone wrong in it

I have another pipeline with two rules and it's working just fine - it
seems to be the regex in this that is at fault, but I can't see how

-- 
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +1 408 481 8171
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/CAFChrgK8T6t_728ynFmH2ePHMx9dhsFYq4stfk1DVcyrdtCRPw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] debugging pipelines is... difficult

2016-07-08 Thread Edmundo Alvarez 
Hi Jason,

It's hard to tell what is wrong from here, since we can't exactly see how your 
messages look like. Could you share a couple of messages with us?

Please be aware that at the moment, the "regex" function needs to match the 
whole string: 
https://github.com/Graylog2/graylog-plugin-pipeline-processor/issues/35

We will release a Graylog 2.1.0 alpha really soon, and it would be really 
helpful to know what you think about the new pipeline simulator. It's still a 
work in progress, just as the pipeline processor, but hopefully will help you 
debug your pipelines and rules.

Regards,
Edmundo

> On 07 Jul 2016, at 23:31, Jason Haar  wrote:
> 
> 
> On Wed, Jul 6, 2016 at 9:50 PM, Jochen Schalanda  wrote:
> there's something coming up in Graylog 2.1.0 which will vastly simplify 
> testing pipeline rules.
> 
> That's great to hear. Any suggestions as to what's wrong with my rule?
> 
> Thanks
> 
> 
> -- 
> Cheers
> 
> Jason Haar
> Information Security Manager, Trimble Navigation Ltd.
> Phone: +1 408 481 8171
> PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Graylog Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/graylog2/CAFChrgL7rcbe_rFpciwxs%3D5%3Dh%3D%3DXC7E3mdXWyO-skSP4ZjidCg%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/CE4236F5-A849-43F0-A8B2-BA9BD79A359C%40graylog.com.
For more options, visit https://groups.google.com/d/optout.