[graylog2] Re: Graylog Best Practices

[Sean McGurk]

Thanks, Jochen,

I perhaps didn't make myself clear in my question - I have a number of 
Graylog collectors running on different instances and my question was more 
whether I should create a separate input on a distinct port for each of 
these collectors or just create one input and have all the collectors send 
to that one input.

In the end, I went with the second approach, so I have one GELF TCP input 
started on port 12201, which aggregates the data from all the collectors.

I then have created a number of streams to route the incoming data, where 
they can be separated by log (and application) type.

I preferred this approach as it meant from a security point of view, I only 
had to open one more port and the 'Streams' concept allowed me to segment 
my log messages.

Seán

On Tuesday, 1 December 2015 16:41:40 UTC, Sean McGurk wrote:
>
> Hi there,
>
> I have set up a Graylog server with a number (7) of input sources.
>
> My question is, when configuring Graylog, is it better to open a number of 
> ports on the Graylog server and have each port receive messages from a 
> particular source or is it better to only open 1 port and receive all 
> inputs via this 1 port?
>
> Thanks,
>
> Seán
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/7c662b78-65ee-4eb0-8186-551b63147d5f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] importing old logs from syslog server

[Darin Perusich]

Hello All,

I'm new to graylog so bare with me if I'm asking a stupid question, but the 
little bit a research I've done on this subject hasn't been as fruitful as 
I've hoped. I have a syslog server with years of archived logs where the 
logs are arranged under a directory structure like 
/logs/$hostip/$year/$month/$day/$facility/$facility.log, and any long older 
than 3 days is compressed to conserve space. I want to import these logs 
into graylog and it looks like the most common tools used for importing log 
files is graylog-collector or logstash, but from looking at configuration 
examples it looks like these tools only monitor individual log files and 
cannot recursively traverse a directory structure and send the logs.

Are the collector or logstash able to traverse a directory structure and 
send any files it finds to graylog? If this is possible can someone provide 
or point me to some configuration examples? Also would the compressed logs 
need to be unpacked before sending them to the server?

If the tools are not able to traverse a directory structure can someone 
provide some guidance on an alternate method to import the logs? I came 
across someone using netcat, but it wasn't overly apparent how they 
accomplished the task. Scripting something to rip down the directory 
structure and pipe the files through whatever filters are need to send them 
to the server is simple enough, I just want to ensure I've setup the 
correct inputs on the server.

Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/b1770456-c27c-4fb3-834c-2035bee80c48%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Re: Graylog Best Practices

[Joi Owen]

One benefit of having separated inputs is that you can isolate unique
extractors to only the input that provides the fields of interest, reducing
the load of having to parse for those fields on log data arriving from
unrelated sources.


On Wed, Dec 2, 2015 at 10:40 AM, Sean McGurk 
wrote:

> Thanks, Jochen,
>
> I perhaps didn't make myself clear in my question - I have a number of
> Graylog collectors running on different instances and my question was more
> whether I should create a separate input on a distinct port for each of
> these collectors or just create one input and have all the collectors send
> to that one input.
>
> In the end, I went with the second approach, so I have one GELF TCP input
> started on port 12201, which aggregates the data from all the collectors.
>
> I then have created a number of streams to route the incoming data, where
> they can be separated by log (and application) type.
>
> I preferred this approach as it meant from a security point of view, I
> only had to open one more port and the 'Streams' concept allowed me to
> segment my log messages.
>
> Seán
>
> On Tuesday, 1 December 2015 16:41:40 UTC, Sean McGurk wrote:
>>
>> Hi there,
>>
>> I have set up a Graylog server with a number (7) of input sources.
>>
>> My question is, when configuring Graylog, is it better to open a number
>> of ports on the Graylog server and have each port receive messages from a
>> particular source or is it better to only open 1 port and receive all
>> inputs via this 1 port?
>>
>> Thanks,
>>
>> Seán
>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Graylog Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/graylog2/7c662b78-65ee-4eb0-8186-551b63147d5f%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>



-- 

No matter what we think of Linux versus FreeBSD, etc., the one thing I
really like about Linux is that it has Microsoft worried. Anything
that kicks a monopoly in the pants has got to be good for something.
- Chris Johnson

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/CAL5rfGViaTCAQw4iCOgYH96Ghpq_sDoo7uVBzCb49LaSxmU9xA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Hostnames not working

[Chris]

Not sure whats going on  but the devices are showing up as source 
ipaddresses instead of as hostnames i set force rdns in the conf, and i 
tested my reverse dns and i fixed everything so its matched up rdns vs 
fwddns and still not working ... 

i have set force_syslog_rdns = true but that didn't help

Graylog server 1.2.1 

[root@GrayLog2 ~]# host 172.16.10.11
11.10.16.172.in-addr.arpa domain name pointer TEST-1.

[root@GrayLog2 ~]# nslookup TEST-1
Server: 172.16.0.252
Address:172.16.0.252#53

Name:   TEST-1
Address: 172.16.10.11



[root@GrayLog2 ~]# host TEST-1
UTS450-1 has address 172.16.10.11

[root@GrayLog2 ~]# nslookup  TEST-1
Server: 172.16.0.252
Address:172.16.0.252#53

Name:   TEST-1
Address: 172.16.10.11




-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/fbd07426-c48b-43c7-bba3-f7530f2ffbdb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Re: Graylog Best Practices

[Jason Haar]

On 03/12/15 07:17, Joi Owen wrote:
> One benefit of having separated inputs is that you can isolate unique
> extractors to only the input that provides the fields of interest,
> reducing the load of having to parse for those fields on log data
> arriving from unrelated sources.
>
The way I look at it is that you have two ways of massaging the data.
One is to do it on the "client" end (eg via graylog-collector, pygelf,
etc) and one is on the "server" end via extractors.

So what I have ended up with is all our syslog data goes in via the
syslog connector and relies on extractors to create the fields, and all
other forms of data input go via pygelf scripts - and I code into it how
I want the fields to be defined - and they go in via a GELF/TLS connector

The extractor method has the advantage that you can centralize all your
massaging, but the client-based massaging has the advantage that you
remove workload from the graylog-servers (and is way more powerful of
course)

Jason

 
>
> On Wed, Dec 2, 2015 at 10:40 AM, Sean McGurk  > wrote:
>
> Thanks, Jochen,
>
> I perhaps didn't make myself clear in my question - I have a
> number of Graylog collectors running on different instances and my
> question was more whether I should create a separate input on a
> distinct port for each of these collectors or just create one
> input and have all the collectors send to that one input.
>
> In the end, I went with the second approach, so I have one GELF
> TCP input started on port 12201, which aggregates the data from
> all the collectors.
>
> I then have created a number of streams to route the incoming
> data, where they can be separated by log (and application) type.
>
> I preferred this approach as it meant from a security point of
> view, I only had to open one more port and the 'Streams' concept
> allowed me to segment my log messages.
>
> Seán
>
> On Tuesday, 1 December 2015 16:41:40 UTC, Sean McGurk wrote:
>
> Hi there,
>
> I have set up a Graylog server with a number (7) of input sources.
>
> My question is, when configuring Graylog, is it better to open
> a number of ports on the Graylog server and have each port
> receive messages from a particular source or is it better to
> only open 1 port and receive all inputs via this 1 port?
>
> Thanks,
>
> Seán
>
>
> -- 
> You received this message because you are subscribed to the Google
> Groups "Graylog Users" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to graylog2+unsubscr...@googlegroups.com
> .
> To view this discussion on the web visit
> 
> https://groups.google.com/d/msgid/graylog2/7c662b78-65ee-4eb0-8186-551b63147d5f%40googlegroups.com
> 
> .
>
>
> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> -- 
>
> No matter what we think of Linux versus FreeBSD, etc., the one thing I
> really like about Linux is that it has Microsoft worried. Anything
> that kicks a monopoly in the pants has got to be good for something.
> - Chris Johnson
>
> -- 
> You received this message because you are subscribed to the Google
> Groups "Graylog Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to graylog2+unsubscr...@googlegroups.com
> .
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/graylog2/CAL5rfGViaTCAQw4iCOgYH96Ghpq_sDoo7uVBzCb49LaSxmU9xA%40mail.gmail.com
> .
> For more options, visit https://groups.google.com/d/optout.


-- 
Cheers

Jason Haar
Corporate Information Security Manager, Trimble Navigation Ltd.
Phone: +1 408 481 8171
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/565F3F92.2010707%40trimble.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog collector and timestamp

[Jochen Schalanda]

Hi Alex,

the Graylog Collector file input basically just reads in text files and 
sends them line-by-line to Graylog. It doesn't parse those lines in any 
way. If you want to use the original timestamp of the log messages in your 
file(s) as message timestamp in Graylog, you'll have to add one or more 
extractors to the GELF TCP input which parses the message and extracts the 
correct timestamp.


Cheers,
Jochen


On Tuesday, 1 December 2015 17:20:02 UTC+1, Alex B. wrote:
>
> Hello, using graylog 1.2.2 and collector 0.4.1, there is a big difference 
> between graylog timestamp and log file timestamp.
>
> A line in a logfile with a 17:11:34,887 timestamp can have a 17:11:53.328 
> timestamp in graylog, which is a 20 seconds difference !
>
> I'm currently testing collector to replace nxlog, don't have this problem 
> with nxlog as you can apply parsedate on date field and send it as event 
> time.
>
> Ty
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/69108a19-ecf8-4be5-8f64-693add61b35e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog Best Practices

[Jochen Schalanda]

Hi Seán,

you cannot bind multiple inputs to the same port (or more precisely the 
same IP address and port), so I guess you don't have a choice but to open 
multiple ports for multiple inputs.


Cheers,
Jochen

On Tuesday, 1 December 2015 17:41:40 UTC+1, Sean McGurk wrote:
>
> Hi there,
>
> I have set up a Graylog server with a number (7) of input sources.
>
> My question is, when configuring Graylog, is it better to open a number of 
> ports on the Graylog server and have each port receive messages from a 
> particular source or is it better to only open 1 port and receive all 
> inputs via this 1 port?
>
> Thanks,
>
> Seán
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/93a01466-3f79-4f86-956a-d270ae355b44%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog collector and timestamp

[Alex B.]

Working like a charm, thank you :)

Le mercredi 2 décembre 2015 10:33:51 UTC+1, Jochen Schalanda a écrit :
>
> Hi Alex,
>
> the Graylog Collector file input basically just reads in text files and 
> sends them line-by-line to Graylog. It doesn't parse those lines in any 
> way. If you want to use the original timestamp of the log messages in your 
> file(s) as message timestamp in Graylog, you'll have to add one or more 
> extractors to the GELF TCP input which parses the message and extracts the 
> correct timestamp.
>
>
> Cheers,
> Jochen
>
>
> On Tuesday, 1 December 2015 17:20:02 UTC+1, Alex B. wrote:
>>
>> Hello, using graylog 1.2.2 and collector 0.4.1, there is a big difference 
>> between graylog timestamp and log file timestamp.
>>
>> A line in a logfile with a 17:11:34,887 timestamp can have a 17:11:53.328 
>> timestamp in graylog, which is a 20 seconds difference !
>>
>> I'm currently testing collector to replace nxlog, don't have this problem 
>> with nxlog as you can apply parsedate on date field and send it as event 
>> time.
>>
>> Ty
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/d8d0280c-d3e1-48dc-ab13-36e1b285da41%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Hostnames not working

[Drew Miranda]

I don't think graylog does any reverse DNS. How are you sending logs to gray 
log? 

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/c2475d8c-e61a-4ca9-be13-b4971ef3db8b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] importing old logs from syslog server

[Drew Miranda]

I did something similar as a proof of concept but it was far from elegant.

In short:

1. Use nxlog to listen to a file and configure a rule that uses the date of the 
log message and not the current date (which it would do if we don't create this 
rule)
2. Use something that reads your log file(s) one line at a time and appends 
each line to the file being monitored by nxlog

*in some cases the date time format is not directly parseable by nxlog and a 
script is required to parse on the correct format.

https://nxlog.org/node/295#idp9098336

Sorry this is so convoluted. This is a feature that has been requested so it is 
possible we may see a native way to do this in the future.

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/e956ec6d-3f94-40be-a3c8-147ea7502ed2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Rewrite log with extractor

[Drew Miranda]

I believe something like this should be possible with drools rules

http://docs.graylog.org/en/1.2/pages/drools.html

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/84003687-af21-4277-8a4b-15dd4e6cb316%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.