[graylog2] Re: Missing old data after upgrade to 1.3.4

[Idan Lerer]

Hi,

The config look OK,
I see the following indexes and look like old data (before the April 14th 
is not part of the index
Why Graylog didn't take / index the old data ?
How I can enforce it ?

{
  "total" : 6,
  "ranges" : [ {
"index_name" : "graylog_5",
"begin" : "1970-01-01T00:00:00.000Z",
"end" : "1970-01-01T00:00:00.000Z",
"calculated_at" : "2016-04-18T11:09:10.089Z",
"took_ms" : 267
  }, {
"index_name" : "graylog_0",
"begin" : "2016-04-14T19:46:32.000Z",
"end" : "2016-04-15T17:09:09.833Z",
"calculated_at" : "2016-04-15T20:20:56.963Z",
"took_ms" : 556
  }, {
"index_name" : "graylog_1",
"begin" : "2016-04-15T17:00:57.000Z",
"end" : "2016-04-15T20:21:27.780Z",
"calculated_at" : "2016-04-15T20:21:28.314Z",
"took_ms" : 62
  }, {
"index_name" : "graylog_2",
"begin" : "2016-04-15T20:20:21.000Z",
"end" : "2016-04-16T15:53:49.737Z",
"calculated_at" : "2016-04-16T15:53:50.181Z",
"took_ms" : 414
  }, {
"index_name" : "graylog_3",
"begin" : "2016-04-16T15:49:46.000Z",
"end" : "2016-04-17T14:37:28.688Z",
"calculated_at" : "2016-04-17T14:37:30.067Z",
"took_ms" : 278
  }, {
"index_name" : "graylog_4",
"begin" : "2016-04-17T14:32:09.000Z",
"end" : "2016-04-18T11:09:09.743Z",
"calculated_at" : "2016-04-18T11:09:10.087Z",
"took_ms" : 291
  } ]
}


On Monday, April 18, 2016 at 8:16:38 AM UTC-7, Jochen Schalanda wrote:
>
> Hi Idan,
>
> please make sure that the elasticsearch_index_prefix 
> 
>  
> and the elasticsearch_cluster_name 
> 
>  
> settings are correct and correspond with the actual cluster name of your 
> Elasticsearch node(s).
>
> Additionally please check the logs of your Elasticsearch node(s) and the 
> output of the following curl command (replace localhost with the host name 
> or IP address of one of your Elasticsearch nodes):
>
> $ curl http://localhost:9200/_cat/indices?v
>
>
>
> Cheers,
> Jochen
>
> On Friday, 15 April 2016 22:43:46 UTC+2, Idan Lerer wrote:
>>
>> Hi Jochen,
>> Thanks for your quick reply.
>>
>> I ran the following
>>
>> wget 
>> https://packages.graylog2.org/releases/graylog-omnibus/ubuntu/graylog_latest.deb
>> sudo graylog-ctl stop
>> sudo dpkg -G -i graylog_latest.deb
>> sudo graylog-ctl reconfigure
>>
>> I still see all my dashboard I configured but I can see only the data 
>> since the upgrade
>>
>>
>> On Friday, April 15, 2016 at 10:27:55 AM UTC-7, Jochen Schalanda wrote:
>>>
>>> Hi Idan,
>>>
>>> how exactly did you upgrade Graylog? What's the configuration of your 
>>> Graylog server and your Elasticsearch node(s)?
>>>
>>>
>>> Cheers,
>>> Jochen
>>>
>>> On Friday, 15 April 2016 18:52:08 UTC+2, Idan Lerer wrote:

 Hello,
 I just upgraded from Graylog v1.3.4 from1.1.3.
 All my settings saved and I can see all my collection / dashboards ETC.
 But I can't see all the data collected by the Graylong before the 
 upgrade.
 When I try to search old data before the upgrade I'm not getting any 
 results.
 On the disk, I do see the Graylog disk 
 /dev/mapper/graylog--indices-graylog--indices have the same used space as 
 it was before the upgrade.
 Where is all my old data ? what I'm missing ?

>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/c2146fc3-bb07-4cc5-9438-737ce0b6868d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Stream or other? Forward everything to SIEM

[Fred Blaise]

Hi Jochen,

OK sounds good to me then :-)

Cheers,
fred

On Monday, April 18, 2016 at 5:36:28 PM UTC+2, Jochen Schalanda wrote:
>
> Hi Fred,
>
> - How does a stream scale? Do we have some benchmarks available?
>>
>
> Streams are relatively lightweight, depending on the stream rules. In your 
> case a simple check for the existence of the message or timestamp fields 
> would be completely sufficient to catch all messages and that's a very 
> cheap check. So if you're not operating your Graylog cluster at 100% 
> utilization, creating that catch-all stream should be fine.
>  
>
>> - Since we need to catch everything, would that even be the right option?
>
>
> That depends what you want to do with those messages additionally. It's 
> clearly one viable option for what you've described in your first email.
>
>
> Cheers,
> Jochen
>
> On Sunday, 17 April 2016 20:33:42 UTC+2, Fred Blaise wrote:
>>
>> Hello all,
>>
>> Currently running on latest 1.3.x, I have to somehow forward all logs 
>> events to a proprietary SIEM, preferably unaltered, so that the receiving 
>> end can apply its own filters and patterns.
>> My current architecture is much like the one shown in the graylog's doc 
>> (prod), including a pair of HAproxy, going down to the graylog-servers.
>>
>> I am guessing I have 2 options:
>> 1. Put a pair of Logstash (or similar) between the HAproxy and the 
>> graylog-server. The LS would split the traffic before it reaches the 
>> graylog-server: 1 flow would go straight to the proprietary SIEM, the other 
>> flow would continue on to the graylog-server
>>
>> 2. Let the message come down to a stream, catch all of them, and output 
>> them to the SIEM using a (probably) custom output.
>>
>> I am actually wondering about option 2.
>> - How does a stream scale? Do we have some benchmarks available?
>> - Since we need to catch everything, would that even be the right option?
>>
>> Any other ideas?
>>
>> Thank you!
>> fred
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/b66ecdbb-3be8-4c14-8dae-3299ecd0d7dc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Stream or other? Forward everything to SIEM

[Jochen Schalanda]

Hi Fred,

- How does a stream scale? Do we have some benchmarks available?
>

Streams are relatively lightweight, depending on the stream rules. In your 
case a simple check for the existence of the message or timestamp fields 
would be completely sufficient to catch all messages and that's a very 
cheap check. So if you're not operating your Graylog cluster at 100% 
utilization, creating that catch-all stream should be fine.
 

> - Since we need to catch everything, would that even be the right option?


That depends what you want to do with those messages additionally. It's 
clearly one viable option for what you've described in your first email.


Cheers,
Jochen

On Sunday, 17 April 2016 20:33:42 UTC+2, Fred Blaise wrote:
>
> Hello all,
>
> Currently running on latest 1.3.x, I have to somehow forward all logs 
> events to a proprietary SIEM, preferably unaltered, so that the receiving 
> end can apply its own filters and patterns.
> My current architecture is much like the one shown in the graylog's doc 
> (prod), including a pair of HAproxy, going down to the graylog-servers.
>
> I am guessing I have 2 options:
> 1. Put a pair of Logstash (or similar) between the HAproxy and the 
> graylog-server. The LS would split the traffic before it reaches the 
> graylog-server: 1 flow would go straight to the proprietary SIEM, the other 
> flow would continue on to the graylog-server
>
> 2. Let the message come down to a stream, catch all of them, and output 
> them to the SIEM using a (probably) custom output.
>
> I am actually wondering about option 2.
> - How does a stream scale? Do we have some benchmarks available?
> - Since we need to catch everything, would that even be the right option?
>
> Any other ideas?
>
> Thank you!
> fred
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/5f8e119f-96ee-473d-a9c7-91d50f0c8244%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Nodes- Connection to machines

[Jochen Schalanda]

Hi Sikender,

I'm not sure what you want to know. Could you please try to rephrase your 
questions or add more details?


Cheers,
Jochen

On Thursday, 14 April 2016 07:29:52 UTC+2, sikender...@acesred.com wrote:
>
> Hi JOchen,
>
>
>
> I dont know what might be the reason But I cant able to access REST API. 
>  And one more thing is like, If i provide more nodes to server ad access 
> graylog? What are the defects I will be facing ? 
> For example; 
>
> I have 10 environments, Can I access 5 environments in 1 node and other 5 
> in another node ? IS it fine If i do that ? 
>
> Thank you Jochen. 
>
> Regards
> Sikender
>
> On Monday, April 11, 2016 at 3:36:22 AM UTC-7, Jochen Schalanda wrote:
>>
>> Hi Sikender,
>>
>> if you're using Graylog 1.3.x, the Graylog REST API must be accessible by 
>> the Graylog web interface. If you're using Graylog 2.x, the Graylog REST 
>> API must additionally be accessible by your web browser (since the web 
>> interface is now a single-page application directly communicating with the 
>> Graylog REST API from your browser).
>>
>> Cheers,
>> Jochen
>>
>> On Thursday, 7 April 2016 10:36:53 UTC+2, sikender...@acesred.com wrote:
>>>
>>> HI Jochen,
>>>
>>>
>>> My graylog server is different server compare to client server.  my 
>>> graylog server is x.x.x.x:9000 and client server is on x.x.x.x:port.   DO 
>>> we need this API access to perform well? I have mu graylog server up and 
>>> able to receive logs though?
>>>
>>> When I run lsof -l :12900;
>>> It gives me output like:
>>>
>>> COMMAND   PIDUSER   FD   TYPE DEVICE SIZE/OFF NODE NAME
>>> java12450 graylog  137u  IPv6 877480  0t0  TCP localhost:12900 
>>> (LISTEN)
>>>
>>>
>>>
>>>
>>> On Thursday, April 7, 2016 at 1:01:07 AM UTC-7, Jochen Schalanda wrote:

 Hi Sikender,

 the loopback interface  is 
 always only accessible from the very same machine, so your client or 
 browser also needs to run on that machine in order to access the Graylog 
 REST API at http://127.0.0.1:12900/.

 If that's already the case, then you need to check the logs of your 
 Graylog server for error messages. Checking that the actual process is 
 running (e. g. sudo lsof -i :12900 or sudo netstat -tplen | grep 
 :12900) would also help.

 Cheers,
 Jochen

 On Thursday, 7 April 2016 09:37:36 UTC+2, sikender...@acesred.com 
 wrote:
>
> Hi Jochen,
>
>
> The graylog-server is running. 
>
> Status is "graylog-server (pid  24233) is running..."  But I can't 
> able to access http://127.0.0.1:12900/ . Even graylog server and 
> server web are on the same server. ?? CAn you please tell me where am I 
> going wrong. 
>
>
> Thank you 
> Sikender
>
>
> On Thursday, March 31, 2016 at 2:54:27 AM UTC-7, Jochen Schalanda 
> wrote:
>>
>> Hi Sikender,
>>
>> make sure that your Graylog server is running and that the web 
>> interface can reach it at http://127.0.0.1:12900/ (which will only 
>> work if it's running on the same machine).
>>
>> Cheers,
>> Jochen
>>
>> On Thursday, 31 March 2016 01:45:07 UTC+2, sikender...@acesred.com 
>> wrote:
>>>
>>> H Jochen, 
>>>
>>>
>>> Awesome. That works for me. Now I am able to send logs so easily :) 
>>>
>>> Seems everything is fine , but when I see the logs of graylog-web 
>>> server after restart, I see something like ; 
>>>
>>>
>>> 2016-03-30T02:14:25.471-04:00 - [INFO] - from play in main
>>> Application started (Prod)
>>>
>>> 2016-03-30T02:14:25.614-04:00 - [INFO] - from play in main
>>> Listening for HTTP on /0:0:0:0:0:0:0:0:9000
>>>
>>> 2016-03-30T02:15:55.038-04:00 - [INFO] - from play in New I/O worker 
>>> #13
>>> Starting application default Akka system.
>>>
>>> 2016-03-30T17:12:42.253-04:00 - [ERROR] - from 
>>> org.graylog2.restclient.lib.ApiClient in pool-22-thread-1
>>> Connection refused: /127.0.0.1:12900 to 
>>> http://127.0.0.1:12900/system/metrics/multiple
>>>
>>> 2016-03-30T17:12:46.427-04:00 - [ERROR] - from 
>>> org.graylog2.restclient.lib.ApiClient in servernodes-refresh-0
>>> Connection refused: /127.0.0.1:12900 to 
>>> http://127.0.0.1:12900/system/cluster/node
>>> "application.log" 145L, 7809C
>>>
>>>
>>> Can you please tell me where exactly it is pointing to ! 
>>>
>>>
>>> Thank you 
>>>
>>>
>>> On Wednesday, March 30, 2016 at 2:10:03 AM UTC-7, Jochen Schalanda 
>>> wrote:

 Hi Sikender,

 you cannot bind two inputs to the same network interface (in this 
 case 0.0.0.0:12201). One of those GELF TCP inputs has to use 
 another port (e. g. 12201 or anything above 1024).

 Cheers,
 Jochen

 On Wednesday, 30 March 2016 

[graylog2] Re: Multiple retention times still not possible?

[Jochen Schalanda]

Hi tokred,

support for multiple (concurrent and possibly incompatible) index schemes 
and retention times is not included in Graylog 2.0.0 and currently isn't on 
the roadmap in the mid-term.


Cheers,
Jochen

On Friday, 15 April 2016 14:49:59 UTC+2, tok...@gmx.net wrote:
>
> Hi all,
>
> I really appreciate the recent developments of Graylog towards 2.0 and big 
> applause for the efforts of the developers!
>
> However, I am still desperately hoping/waiting for a features which has 
> been number 1 on the idea portal for almost a year (!) - multiple indices 
> with different retention times. From my understanding, the archiving 
> feature does not cover this, right?
>
> Kindly ask for any plans or alternatives.
>
>
> Best regards,
> tokred
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/2ef14a79-a99f-4851-8010-bfca5297aca2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Can't delete an input in Graylog 2.0 Beta.3

[Dennis Oelkers]

Hey Jayica,

it looks like the input you are trying to stop does not exist anymore. Are you 
sre that it is still existing? Do you still see it in the inputs page after 
refreshing?

Kind regards,
D.

--
Tel.: +49 (0)40 609 452 077
Fax.: +49 (0)40 609 452 078

TORCH GmbH - A Graylog company
Steckelhörn 11
20457 Hamburg
Germany

Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175
Geschäftsführer: Lennart Koopmann (CEO)

> On 18.04.2016, at 01:04, Jayica  wrote:
> 
> I just installed Graylog 2.0.0-beta.3 and created an input (which was working 
> fine until I added an elastic IP to my Graylog AWS instance - while it was up 
> and running).  Now I can't stop or delete the input.
> 
> Error upon attempting to stop the input:
> Input 'Graylog Lab Server - Linux' could not be stopped
> Request to stop input 'Graylog Lab Server - Linux' failed. Check your Graylog 
> logs for more information.
> 
> Per Graylog logs:
> 2016-04-17T22:54:33.525Z WARN  [ProxiedResource] Unable to call 
> http://50.27.235.63:12900/system/inputstates/571402e2ff18d6124cd36b30 on node 
> , result: Not Found
> 
> Error upon attempting to delete the input:
> Could not delete input
> Deleting input 'Graylog Lab Server - Linux' failed with status: Error: cannot 
> DELETE http://50.27.235.63:12900/system/inputs/571402e2ff18d6124cd36b30 (404)
> 
> 
> Is there a file on the Graylog server (Ubuntu Linux 14.04) somewhere i can 
> delete/cleanup to remove this input?
> 
> I can successfully create new inputs, however, they are not appearing in the 
> Inputs blotter.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Graylog Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/graylog2/1b990cc3-498a-43d5-b181-d8b732f4c981%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/C147C1D6-28F9-4B4B-B592-F5EEBEE4CCF3%40graylog.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: graylog collector and cpu load

[Jochen Schalanda]

Hi Hasan,

I'm not quite sure what you want to know. Could you please try to rephrase 
your question or add more details?

Cheers,
Jochen

On Friday, 15 April 2016 20:23:19 UTC+2, hasan akgöz wrote:
>
> Hello community,
>
> I just wonder , if the service stops graylog-server or elasticsearch . Does 
> it make load in the log source?
>
> have a nice day.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/2c39627a-0ccc-4f38-8bdd-6a76f2e3a44b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Missing old data after upgrade to 1.3.4

[Jochen Schalanda]

Hi Idan,

please make sure that the elasticsearch_index_prefix 

 
and the elasticsearch_cluster_name 

 
settings are correct and correspond with the actual cluster name of your 
Elasticsearch node(s).

Additionally please check the logs of your Elasticsearch node(s) and the 
output of the following curl command (replace localhost with the host name 
or IP address of one of your Elasticsearch nodes):

$ curl http://localhost:9200/_cat/indices?v



Cheers,
Jochen

On Friday, 15 April 2016 22:43:46 UTC+2, Idan Lerer wrote:
>
> Hi Jochen,
> Thanks for your quick reply.
>
> I ran the following
>
> wget 
> https://packages.graylog2.org/releases/graylog-omnibus/ubuntu/graylog_latest.deb
> sudo graylog-ctl stop
> sudo dpkg -G -i graylog_latest.deb
> sudo graylog-ctl reconfigure
>
> I still see all my dashboard I configured but I can see only the data 
> since the upgrade
>
>
> On Friday, April 15, 2016 at 10:27:55 AM UTC-7, Jochen Schalanda wrote:
>>
>> Hi Idan,
>>
>> how exactly did you upgrade Graylog? What's the configuration of your 
>> Graylog server and your Elasticsearch node(s)?
>>
>>
>> Cheers,
>> Jochen
>>
>> On Friday, 15 April 2016 18:52:08 UTC+2, Idan Lerer wrote:
>>>
>>> Hello,
>>> I just upgraded from Graylog v1.3.4 from1.1.3.
>>> All my settings saved and I can see all my collection / dashboards ETC.
>>> But I can't see all the data collected by the Graylong before the 
>>> upgrade.
>>> When I try to search old data before the upgrade I'm not getting any 
>>> results.
>>> On the disk, I do see the Graylog disk 
>>> /dev/mapper/graylog--indices-graylog--indices have the same used space as 
>>> it was before the upgrade.
>>> Where is all my old data ? what I'm missing ?
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/5ad7a5fc-cea7-4788-aae5-b03f75166688%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Can't delete an input in Graylog 2.0 Beta.3

[Jochen Schalanda]

Hi Jayica,

it looks like the given URL used by the web interface isn't correct.

Please make sure that the correct URL is given in the rest_transport_uri 
setting in the Graylog configuration file (see 
https://github.com/Graylog2/graylog2-server/blob/2.0.0-beta.3/misc/graylog.conf#L39-L45
).


Cheers,
Jochen

On Monday, 18 April 2016 01:04:36 UTC+2, Jayica wrote:
>
> I just installed Graylog 2.0.0-beta.3 and created an input (which was 
> working fine until I added an elastic IP to my Graylog AWS instance - while 
> it was up and running).  Now I can't stop or delete the input.
>
> Error upon attempting to stop the input:
> Input 'Graylog Lab Server - Linux' could not be stopped
> Request to stop input 'Graylog Lab Server - Linux' failed. Check your 
> Graylog logs for more information.
>
> Per Graylog logs:
> 2016-04-17T22:54:33.525Z WARN  [ProxiedResource] Unable to call 
> http://50.27.235.63:12900/system/inputstates/571402e2ff18d6124cd36b30 on 
> node , result: Not Found
>
> Error upon attempting to delete the input:
> Could not delete input
> Deleting input 'Graylog Lab Server - Linux' failed with status: Error: 
> cannot DELETE 
> http://50.27.235.63:12900/system/inputs/571402e2ff18d6124cd36b30 (404)
>
>
> Is there a file on the Graylog server (Ubuntu Linux 14.04) somewhere i can 
> delete/cleanup to remove this input?
>
> I can successfully create new inputs, however, they are not appearing in 
> the Inputs blotter.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/1f86e802-8627-42f9-be8c-46629eba6d0f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog /var parition always increases

[Jochen Schalanda]

Hi Roberto,

what's the configuration of your Graylog servers? What's the directory 
inside /var with the largest size (check with du or similar tools 
)?

What's the output of the following curl command (replace localhost with the 
host name or IP address of one of your Elasticsearch nodes):

$ curl http://localhost:9200/_cat/indices?v



Cheers,
Jochen

On Monday, 18 April 2016 14:12:01 UTC+2, roberto...@gmail.com wrote:
>
> Dear, I have GRaylog 1.3 and I setup a /var partition of 1.5 TB.
>
> I define 10 indexes of 3 days each, and every index is deleted after that.
>
> In despite of this strategy, the /var partition of Graylog server always 
> increases and when it reaches 95% aprox, the Graylog stop logging.
>
> What can I do in order to maintain the /var partition size in a value that 
> never reaches the 90%???
>
> Because in the firts days of my Graylog server, I had 3 indexes of 3 days 
> each, after that 20 indexes of 3 days, now 10 indexes of 3 days...and 
> always my /var partition gets nearly 100%.
>
> Thanks a lot,
>
> Roberto
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/9d668fed-1ee5-4389-a6fb-2233ae6da703%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: varnish and varnishncsa UDP log pipe

[Gregory Boddin]

Hey,

Sorry for the late reply but :

I created a small utility to forward JSON objects (from stdin or a file) to 
a GELF UDP server..

It's written in C and is fast enough to live as an agent next on your 
varnish server without adding too much load and without the need of a full 
scripting environment (python/ruby/js).

https://github.com/gboddin/json2gelf

There's an exemple on how to create a JSON packet from varnishncsa here 
: https://github.com/gboddin/json2gelf/blob/master/scripts/varnish2json

Just pipe varnish2json in json2gelf to start sending gelf packets to your 
graylog/logstash/.

We're using it in production, feel free to use it for any application able 
to output json logging.

It avoids UDP fragmentation by using the chunk specification of the GELF 
protocol. Shall your MTU be exotic, you can always change the #define in 
json2gelf.c

Hope it helps !

On Sunday, November 1, 2015 at 3:11:37 AM UTC+1, Dhyaneshwaran Muralidharan 
wrote:
>
> Hi Razvan,
>
>  Can you please explain how exactly you converted the varnish logs to Json 
> ? I was trying to do the same but was worried if the log itself could have 
> strings which could make the Json invalid .
>
> Currently we have the format like below :
> -F %h %u %t "%r" %s %B "%{Referer}i" "%{User-agent}i"
>
> --Dhyan
>
> On Wednesday, December 3, 2014 at 6:23:57 PM UTC-5, Razvan Grigore wrote:
>>
>> Hello and thank you for this great piece of software!
>>
>> I joined one of the author's live presentations in Hamburg, and since 
>> then I am using graylog2 as logging solution for the company I work for.
>>
>> Most of the basic stuff works great, it's just the varnish logging I 
>> can't seem to make it work. I found very little about this couple online - 
>> very strange, no extractors, no group topic, nothing. Varnish is great!
>>
>> My idea was to pipe the logs via simple UDP from varnishncsa (log daemon) 
>> instead of storing them in a file and have rsyslog parse them and send the 
>> to graylog - avoid unnecessary IO.
>>
>> Since varnishncsa has the -F option, I specified a JSON format according 
>> to GELF specification, and this works great, I can see all entries in stdin 
>> like a tail -f of the log file, all valid JSON.
>>
>> Now the part that is not working, is that I try to pipe this output to nc 
>> (netcat) via UDP. The strange this is, SOME logs appear, after long time, 
>> but most of them are lost.
>>
>> I tried the nc command with both -w 1 and without, seems the same, my 
>> guess is that netcat keeps the connection open and sends multiple messages 
>> until the max packet size is reached, making graylog receive invalid 
>> formatted JSON.
>>
>> I should mention that the input is working, I can receive messages in 
>> GELF format from PHP via UDP on that port without any problem. If needed, I 
>> can provide the exact command I am running.
>>
>> Thank you for your help!
>> Razvan, HH
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/1a6175b6-cd81-4832-9460-6b7b07dd7d51%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Graylog /var parition always increases

[robertocarna36]

Dear, I have GRaylog 1.3 and I setup a /var partition of 1.5 TB.

I define 10 indexes of 3 days each, and every index is deleted after that.

In despite of this strategy, the /var partition of Graylog server always 
increases and when it reaches 95% aprox, the Graylog stop logging.

What can I do in order to maintain the /var partition size in a value that 
never reaches the 90%???

Because in the firts days of my Graylog server, I had 3 indexes of 3 days 
each, after that 20 indexes of 3 days, now 10 indexes of 3 days...and 
always my /var partition gets nearly 100%.

Thanks a lot,

Roberto

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/dbdecac3-2271-4b9b-8ae2-8e2fcb8aeceb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.