Re: [graylog2] Multiple source IP addresses to one Stream group - HOW? POSSIBLE? A BETTER WAY?
Exactly :) Also keep in mind that messages can exist in multiple streams since it’s just a form of tagging and filtering. If you really, really need to use the approach you outlined there a 2 options - one would be to vote for the implementation of issues #381 and #382 another would be to use the excellent REST API in Graylog to manipulate the streams any way you would like. Btw Aidan - did you ever find the ring you lost at the Zabbix Conference a few years back? — HenrikJ On 31 May 2015, at 22:29, Brandon bk...@alias454studios.com wrote: I can see the benefit of having a stream template feature but to repeat what Henrik asked, why does each source need its own separate stream? Not asking to deter your question, I'm trying to understand the use case. A stream rule can be setup to find ALL messages that hit on the disconnect keyword and route to a single stream. The disconnect stream would contain messages from ALL sources that send in that message (source 1.1.1.1, 1.1.1.2, 1.1.1.3 etc.). From within the stream, you can then use the quick filters feature to find a list of all unique sources and add that to a dashboard gadget. This can be repeated for the 10 different keywords and then one only has to manage 10 streams instead of 10,000. Regards, Brandon On 05/31/2015 11:00 AM, Aidan Venn wrote: Hi, its like having a template applied to sources. change the template changes all the related sources. Like in Zabbix. Kind Regards Aidan On Thursday, May 28, 2015 at 9:40:20 AM UTC+1, Aidan Venn wrote: https://lh3.googleusercontent.com/-VXS0tYSBx3Y/VWYbA0x3z0I/Dg8/7ZikVzm-U_U/s1600/Untitled.pngHi, Garylog Newbie Please see picture attached. I have three streams matching a single source IP and warning keywords from logs: source IP: 192.168.0.1 stream 1-keyword:disconnect steram 2-keyword:loss stream 3-keyword:fail I want to group these streams and apply to multiple (1000 +) source IP addresses to benefit future scalability and large scale administration. Basically for each source IP they will be three or more streams but I only have to configure/edit the group once. I don`t want to have 1000 devices then have to copy each stream and then change the source IP address match. 10 keyword stream x 1000 devices would then equal 1 streams in total to configure and edit. This would be very time consuming. Especially if I had to make a change. One change to the group would apply to all. A one to many relationship. How can I do this? Perhaps my approach/idaea is incorrect so any recommendations would be great. Kind Regards Aidan Venn -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com mailto:graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com mailto:graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: exporting data from searches not working properly
Hello
Found the issue as well, only the message is by default exported so I had
to create an extractor to override the default message with the full
message
I used the split and index, using { as splitting characters
Thanks!
Mark
On Sunday, May 31, 2015 at 1:49:07 AM UTC+10, graylog...@gmail.com wrote:
Hello
I'm using the production OVA (not the beta) of Graylog
I noticed that when I try to export the results of a search, the message
field is trunked, see example below:
The full message is full_message
*{1331892651000, 4776, Success, Security,
Microsoft-Windows-Security-Auditing, The computer attempted to validate
the credentials for an account.Authentication Package:
MICROSOFT_AUTHENTICATION_PACKAGE_V1_0Logon Account: mr636cSource
Workstation: INHYIMR636CError Code: 0x0 }*
http://192.168.1.123/search?rangetype=relativefields=source%2Cmessagewidth=1920relative=3600from=to=q=mr636c#
In the exported CSV log I have only this:
*{1331892651000, 4634, Success, Security, Microsoft-Windows*
Is there anyway to fix this?
Thanks a lot
Mark
--
You received this message because you are subscribed to the Google Groups
graylog2 group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Graylog 1.1.0-beta.2 collector issue in webinterface
Bernd, looks like it is solved in 1.10.4-rc1. Thank you. On Thursday, May 28, 2015 at 5:54:33 PM UTC+2, Bernd Ahlers wrote: Arie, thanks for he report. There is an issue and a pull request to fix the issue on GitHub. https://github.com/Graylog2/graylog2-web-interface/issues/1334 https://github.com/Graylog2/graylog2-server/pull/1190 This will be fixed in the next beta or rc. Regards, Bernd Arie [Thu, May 28, 2015 at 07:12:30AM -0700] wrote: Hi Bernd, Just installed and tried it, the error is still there. Tested it with a windows and linux collector, and in both cases, no results. Arie. On Thursday, May 28, 2015 at 3:58:56 PM UTC+2, Bernd Ahlers wrote: Arie, thanks for the report. Do you still have that problem with beta.3? Bernd Arie [Thu, May 28, 2015 at 06:22:49AM -0700] wrote: Hi All, When we look @ System Collectors and select show messages, no messages are show in the UI. Messages are visible with a normal search. Running on centos-6.6 / elastic 1.5.2 / JRE 1.8 hth,, Arie -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+u...@googlegroups.com javascript:. For more options, visit https://groups.google.com/d/optout. -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH - A Graylog company Steckelhörn 11 20457 Hamburg Germany Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+u...@googlegroups.com javascript:. For more options, visit https://groups.google.com/d/optout. -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH - A Graylog company Steckelhörn 11 20457 Hamburg Germany Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Multiple source IP addresses to one Stream group - HOW? POSSIBLE? A BETTER WAY?
Hi Brandon, I see now what you and Henrik mean. That’s a far more suitable solution and one I will start to have a look at. My original approach / mind-set wasn`t correct. Thanks for yours and Henriks help. I`ve been playing with graylog for about 10 days on and off now and loving it! Kind Regards Aidan Venn On Thursday, May 28, 2015 at 9:40:20 AM UTC+1, Aidan Venn wrote: https://lh3.googleusercontent.com/-VXS0tYSBx3Y/VWYbA0x3z0I/Dg8/7ZikVzm-U_U/s1600/Untitled.png Hi, Garylog Newbie Please see picture attached. I have three streams matching a single source IP and warning keywords from logs: source IP: 192.168.0.1 stream 1-keyword:disconnect steram 2-keyword:loss stream 3-keyword:fail I want to group these streams and apply to multiple (1000 +) source IP addresses to benefit future scalability and large scale administration. Basically for each source IP they will be three or more streams but I only have to configure/edit the group once. I don`t want to have 1000 devices then have to copy each stream and then change the source IP address match. 10 keyword stream x 1000 devices would then equal 1 streams in total to configure and edit. This would be very time consuming. Especially if I had to make a change. One change to the group would apply to all. A one to many relationship. How can I do this? Perhaps my approach/idaea is incorrect so any recommendations would be great. Kind Regards Aidan Venn -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: grok extractors not working
Jesse, We've just tried to reproduce this issue on 1.1.0-RC.1 but it works as expected. Could you give that a try in a test environment, please? Thanks, Kay On Saturday, 30 May 2015 13:42:52 UTC+2, Bernd Ahlers wrote: Jesse, thank you for the update. I created an issue in GitHub for this with a link to this mailing list thread. https://github.com/Graylog2/graylog2-server/issues/1192 I also started to test with the detailed data you submitted but did not see any problems. I was testing on 1.1.0-rc.1 though. Next step is to test all of this with 1.0.2 (which you are running). I will let you know once I have any updates. Thank you! Bernd Jesse Skrivseth [Fri, May 29, 2015 at 11:41:52AM -0700] wrote: I'm not sure why, but suddenly the extractors are working today without any further action on my part. There seems to be a very long delay between when an extractor is configured and when it is in effect, at least in this environment. Another thing to note is that the data on this input is TLS encrypted GELF via TCP, and the data is coming in from NXLog using GELF_TCP. On Thursday, May 28, 2015 at 3:25:05 PM UTC-6, Kay Röpke wrote: I'm not an expert on the OVAs so I would recommend simply setting up a test instance to check this. Or you can wait until I get to it in the (my) morning ;) -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH - A Graylog company Steckelhörn 11 20457 Hamburg Germany Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Multiple source IP addresses to one Stream group - HOW? POSSIBLE? A BETTER WAY?
Thanks Henrik. Lol - I remember a crazy party with lots of crazy zabbix guys and lots of champagne! I never found the ring and I am now on my forth in five years! regards On Monday, June 1, 2015 at 7:25:21 AM UTC+1, Henrik Johansen wrote: Exactly :) Also keep in mind that messages can exist in multiple streams since it’s just a form of tagging and filtering. If you really, really need to use the approach you outlined there a 2 options - one would be to vote for the implementation of issues #381 and #382 another would be to use the excellent REST API in Graylog to manipulate the streams any way you would like. Btw Aidan - did you ever find the ring you lost at the Zabbix Conference a few years back? — HenrikJ On 31 May 2015, at 22:29, Brandon bk...@alias454studios.com javascript: wrote: I can see the benefit of having a stream template feature but to repeat what Henrik asked, why does each source need its own separate stream? Not asking to deter your question, I'm trying to understand the use case. A stream rule can be setup to find ALL messages that hit on the disconnect keyword and route to a single stream. The disconnect stream would contain messages from ALL sources that send in that message (source 1.1.1.1, 1.1.1.2, 1.1.1.3 etc.). From within the stream, you can then use the quick filters feature to find a list of all unique sources and add that to a dashboard gadget. This can be repeated for the 10 different keywords and then one only has to manage 10 streams instead of 10,000. Regards, Brandon On 05/31/2015 11:00 AM, Aidan Venn wrote: Hi, its like having a template applied to sources. change the template changes all the related sources. Like in Zabbix. Kind Regards Aidan On Thursday, May 28, 2015 at 9:40:20 AM UTC+1, Aidan Venn wrote: https://lh3.googleusercontent.com/-VXS0tYSBx3Y/VWYbA0x3z0I/Dg8/7ZikVzm-U_U/s1600/Untitled.png Hi, Garylog Newbie Please see picture attached. I have three streams matching a single source IP and warning keywords from logs: source IP: 192.168.0.1 stream 1-keyword:disconnect steram 2-keyword:loss stream 3-keyword:fail I want to group these streams and apply to multiple (1000 +) source IP addresses to benefit future scalability and large scale administration. Basically for each source IP they will be three or more streams but I only have to configure/edit the group once. I don`t want to have 1000 devices then have to copy each stream and then change the source IP address match. 10 keyword stream x 1000 devices would then equal 1 streams in total to configure and edit. This would be very time consuming. Especially if I had to make a change. One change to the group would apply to all. A one to many relationship. How can I do this? Perhaps my approach/idaea is incorrect so any recommendations would be great. Kind Regards Aidan Venn -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+u...@googlegroups.com javascript:. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+u...@googlegroups.com javascript:. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: grok extractors not working
Thanks to everyone for continuing to pursue this odd issue. Arie - We are using nxlog-ce version 2.9.1347 Kay - I can't seem to recreate the problem (yet) in a test environment, whether 1.0.2 or 1.1.0. There are some (possibly irrelevant) differences between test and production, but I'll mention them anyway. Production is built on Amazon Web Services using the provided 1.0.2 AMIs. Test is running locally from the 1.0.2 OVA images in ESXi. Test and production have different volumes of data coming in and different devices are sending logs. The only place I seem to be having trouble is in production with messages coming from the one network appliance I am focused on at the moment. I haven't had a need to add/update extractors for other devices yet, so I'm not sure if the problem is limited to that one device or is universal for this production instance. I will test more generally, with more devices, and see if I can find a pattern. On Monday, June 1, 2015 at 4:33:08 AM UTC-6, Kay Roepke wrote: Jesse, We've just tried to reproduce this issue on 1.1.0-RC.1 but it works as expected. Could you give that a try in a test environment, please? Thanks, Kay -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: how to keep the log message in one field?
That is one way to do it, this works up to server 2003, server 2008 and so
on is a little different,
this way there is better handling of the logs.
define ROOT C:\Program Files\nxlog
#define ROOT C:\Program Files (x86)\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
Extension gelf
Module xm_gelf
/Extension
Input in
Moduleim_mseventlog
Sources Application,System
/Input
Output out
Module om_udp
Host10.64.91.18
Port8000
OutputType GELF
/Output
Route 1
Pathin = out
/Route
Op maandag 1 juni 2015 09:04:28 UTC+2 schreef graylog...@gmail.com:
Hello
Found the issue, it was the configuration of NXLOG, I had to tell NXLOG
that the input was multiline and the headline/endline were {}, I changed
the nxlog.conf as below:
Extension gelf
Module xm_gelf
/Extension
Extension multiline
Module xm_multiline
HeaderLine /^{/
EndLine /^}/
/Extension
Input in
Module im_file
File/media/winlogs/*
SavePos TRUE
Recursive TRUE
InputType multiline
/Input
Output out
Module om_udp
Host127.0.0.1
Port12201
OutputType GELF
/Output
#Output out
#Module om_file
#File /tmp/output
#/Output
On Sunday, May 31, 2015 at 1:49:06 AM UTC+10, graylog...@gmail.com wrote:
Hello
I'm having a problem with graylog and nxlog feed
I have a huge archive of windows event logs, I have been trying to import
these logs into graylog using nxlog and gelf
It all works well, nxlog pickup the logs and imports them but the
messages are being split in several records rather tha a single one,
Example if the event log contains the follow
*{1331892664000, 4624, Success, Security,
Microsoft-Windows-Security-Auditing, An account was successfully logged
on.*
*Subject:*
* Security ID: S-1-0-0*
* Account Name: -*
* Account Domain: -*
* Logon ID: 0x0*
*Logon Type: 3*
*This event is generated when a logon session is created. It is generated
on the computer that was accessed.*
*Key length indicates the length of the generated session key. This will
be 0 if no session key was requested. } *
It gets loaded into graylog as:
Record 1: *{1331892664000, 4624, Success, Security,
Microsoft-Windows-Security-Auditing, An account was successfully logged
on.*
Record 2: *Subject*
Record 3*: **Security ID: S-1-0-0*
etc.
etc
I just would like to have all the message stored in one record
Do you have any idea how this could be achieved?
Thanks!
Mark
--
You received this message because you are subscribed to the Google Groups
graylog2 group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: how to keep the log message in one field?
Can you explain or show what the difference is for 2008/2012 or did I
misunderstand you and the config you describe below applies also to
2008/2012?
Thanks,
- Marco
On 06/01/2015 12:41 PM, Arie wrote:
That is one way to do it, this works up to server 2003, server 2008
and so on is a little different,
this way there is better handling of the logs.
define ROOT C:\Program Files\nxlog
#define ROOT C:\Program Files (x86)\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
Extension gelf
Module xm_gelf
/Extension
Input in
Moduleim_mseventlog
Sources Application,System
/Input
Output out
Module om_udp
Host10.64.91.18
Port8000
OutputType GELF
/Output
Route 1
Pathin = out
/Route
Op maandag 1 juni 2015 09:04:28 UTC+2 schreef graylog...@gmail.com:
Hello
Found the issue, it was the configuration of NXLOG, I had to tell
NXLOG that the input was multiline and the headline/endline were
{}, I changed the nxlog.conf as below:
Extension gelf
Module xm_gelf
/Extension
Extension multiline
Module xm_multiline
HeaderLine /^{/
EndLine /^}/
/Extension
Input in
Module im_file
File/media/winlogs/*
SavePos TRUE
Recursive TRUE
InputType multiline
/Input
Output out
Module om_udp
Host127.0.0.1
Port12201
OutputType GELF
/Output
#Output out
#Module om_file
#File /tmp/output
#/Output
On Sunday, May 31, 2015 at 1:49:06 AM UTC+10, graylog...@gmail.com
wrote:
Hello
I'm having a problem with graylog and nxlog feed
I have a huge archive of windows event logs, I have been
trying to import these logs into graylog using nxlog and gelf
It all works well, nxlog pickup the logs and imports them but
the messages are being split in several records rather tha a
single one,
Example if the event log contains the follow
/{1331892664000, 4624, Success, Security,
Microsoft-Windows-Security-Auditing, An account was
successfully logged on./
/
/
/Subject:/
/Security ID:S-1-0-0/
/Account Name:-/
/Account Domain:-/
/Logon ID:0x0/
/
/
/Logon Type:3/
/
/
/
/
/This event is generated when a logon session is created. It
is generated on the computer that was accessed./
/
/
/Key length indicates the length of the generated session key.
This will be 0 if no session key was requested. } /
/
/
It gets loaded into graylog as:
Record 1: /{1331892664000, 4624, Success, Security,
Microsoft-Windows-Security-Auditing, An account was
successfully logged on./
Record 2: /Subject/
Record 3/: //Security ID:S-1-0-0/
/
/
etc.
etc
/
/
/
/
I just would like to have all the message stored in one record
Do you have any idea how this could be achieved?
Thanks!
Mark
/
/
/
/
--
You received this message because you are subscribed to the Google
Groups graylog2 group.
To unsubscribe from this group and stop receiving emails from it, send
an email to graylog2+unsubscr...@googlegroups.com
mailto:graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
graylog2 group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: how to keep the log message in one field?
Hello
Thanks for info but my case is different (I think!)
If I'm not wrong your configuration for NXLOG is to fetch live eventlogs,
in my case I have a huge archive (5TB) of windows logs that have been
already exported as text file, so I'm not accessing the live eventlogs on a
windows system.
Best regards
Mark
On Sunday, May 31, 2015 at 1:49:06 AM UTC+10, graylog...@gmail.com wrote:
Hello
I'm having a problem with graylog and nxlog feed
I have a huge archive of windows event logs, I have been trying to import
these logs into graylog using nxlog and gelf
It all works well, nxlog pickup the logs and imports them but the messages
are being split in several records rather tha a single one,
Example if the event log contains the follow
*{1331892664000, 4624, Success, Security,
Microsoft-Windows-Security-Auditing, An account was successfully logged
on.*
*Subject:*
* Security ID: S-1-0-0*
* Account Name: -*
* Account Domain: -*
* Logon ID: 0x0*
*Logon Type: 3*
*This event is generated when a logon session is created. It is generated
on the computer that was accessed.*
*Key length indicates the length of the generated session key. This will
be 0 if no session key was requested. } *
It gets loaded into graylog as:
Record 1: *{1331892664000, 4624, Success, Security,
Microsoft-Windows-Security-Auditing, An account was successfully logged
on.*
Record 2: *Subject*
Record 3*: **Security ID: S-1-0-0*
etc.
etc
I just would like to have all the message stored in one record
Do you have any idea how this could be achieved?
Thanks!
Mark
--
You received this message because you are subscribed to the Google Groups
graylog2 group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
