Re: [graylog2] TCP Syslog "input" channel restarting all the time?
Jason, thank you for the report. We have a similar issue open: https://github.com/Graylog2/graylog2-server/issues/1105 We haven't been able to reproduce this. I will try again with your setup. Can you send us the syslog-ng configuration snippet for Graylog? That would be helpful. Thanks, Bernd Jason Haar [Tue, Jul 28, 2015 at 07:37:54PM -0700] wrote: >Hi there > >I'm using syslog-ng to feed in data via a syslog/TCP channel and it's >continually (every 10 seconds) dropping the TCP channel - forcing syslog-ng >to restart it > >2015-07-29T02:26:31+00:00 syslog.server syslog notice syslog-ng[30512]: >Syslog connection broken; fd='408', server='AF_INET(192.168.6.3:1514)', >time_reopen='10' >2015-07-29T02:26:41+00:00 syslog.server syslog notice syslog-ng[30512]: >Syslog connection established; fd='465', >server='AF_INET(192.168.6.3:1514)', local='AF_INET(0.0.0.0:0)' >2015-07-29T02:26:41+00:00 syslog.server syslog notice syslog-ng[30512]: >Syslog connection broken; fd='465', server='AF_INET(192.168.6.3:1514)', >time_reopen='10' >2015-07-29T02:26:51+00:00 syslog.server syslog notice syslog-ng[30512]: >Syslog connection established; fd='379', >server='AF_INET(192.168.6.3:1514)', local='AF_INET(0.0.0.0:0)' >2015-07-29T02:26:51+00:00 syslog.server syslog notice syslog-ng[30512]: >Syslog connection broken; fd='379', server='AF_INET(192.168.6.3:1514)', >time_reopen='10' >2015-07-29T02:27:01+00:00 syslog.server syslog notice syslog-ng[30512]: >Syslog connection established; fd='476', >server='AF_INET(192.168.6.3:1514)', local='AF_INET(0.0.0.0:0)' >2015-07-29T02:27:02+00:00 syslog.server syslog notice syslog-ng[30512]: >Syslog connection broken; fd='476', server='AF_INET(192.168.6.3:1514)', >time_reopen='10' > > >tcpdump shows normal data flow followed by two TCP resets coming back from >the graylog-1.1.5 server - so it's definitely graylog that's borking. > >BTW, this system *is working*: I'm seeing these syslogs flowing in - can do >searches/etc - but I assume I'm losing some records due to this issue. I >even created a xinetd.d based tcp service on the graylog server that just >logged what it received to a file, configured the syslog server to send to >both tcp channels - and it's running fine with no restarts (ie tcpdump of >both ports only shows TCP resets on the graylog port not the xinetd port). >So I think that implies it isn't the OS (CentOS-7) > >Whatever the root cause is should be logged somewhere - can someone point >out to me how the debug this? > >Thanks > >Jason > >-- >You received this message because you are subscribed to the Google Groups >"graylog2" group. >To unsubscribe from this group and stop receiving emails from it, send an >email to graylog2+unsubscr...@googlegroups.com. >For more options, visit https://groups.google.com/d/optout. -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH - A Graylog company Steckelhörn 11 20457 Hamburg Germany Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: how to use sessionid with rest api
Hi Jochen,
Many thanks for the prompt reply. It works exactly as you described it.
Best Regards,
Emde
On Tuesday, July 28, 2015 at 5:48:47 PM UTC+3, Jochen Schalanda wrote:
>
> Hi,
>
> the session handling of Graylog is kind of special (i. e. using a very
> custom mechanism).
>
> You'll have to use the session ID as user name and the special password
> "session" as credentials and send those as standard Authorization header
> (Basic Auth) to the Graylog server.
>
> In the end your requests will look like http://${
> SESSION_ID}:sess...@graylog.example.net:12900/foo/bar.
>
>
> Cheers,
> Jochen
>
> On Tuesday, 28 July 2015 16:21:56 UTC+2, Emde wrote:
>>
>> Hello,
>>
>> I am accesing the REST API of Graylog from java.
>>
>> I want to use a sessionId to make the calls to the REST api without
>> username and password. I haven't found any documentation reading how this
>> should be set in order for graylog to accept it. I have tried the following
>> and all of them return 401 Unauthorized:
>>
>> URLConnection graylogConnection = graylogUrl.openConnection();
>> graylogConnection.setRequestProperty("sessionid", sessionId);
>> graylogConnection.setRequestProperty("session-id", sessionId);
>> graylogConnection.setRequestProperty("session_id", sessionId);
>> graylogConnection.setRequestProperty("id", sessionId);
>> graylogConnection.setRequestProperty("authorization", "Bearer " +
>> sessionId);
>> graylogConnection.setRequestProperty("JSESSIONID", sessionId);
>> graylogConnection.setRequestProperty("Cookie", "JSESSIONID=" + sessionId);
>>
>> I got the sessionId value from the system/sessions resource and has a
>> valid expiry date(it is not expired).
>> I also tried all of the above with sessionId Base64 encoded.
>>
>> Could anyone suggest how I should set the sessionId in order for graylog
>> to accept it?
>>
>
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] TCP Syslog "input" channel restarting all the time?
Hi there I'm using syslog-ng to feed in data via a syslog/TCP channel and it's continually (every 10 seconds) dropping the TCP channel - forcing syslog-ng to restart it 2015-07-29T02:26:31+00:00 syslog.server syslog notice syslog-ng[30512]: Syslog connection broken; fd='408', server='AF_INET(192.168.6.3:1514)', time_reopen='10' 2015-07-29T02:26:41+00:00 syslog.server syslog notice syslog-ng[30512]: Syslog connection established; fd='465', server='AF_INET(192.168.6.3:1514)', local='AF_INET(0.0.0.0:0)' 2015-07-29T02:26:41+00:00 syslog.server syslog notice syslog-ng[30512]: Syslog connection broken; fd='465', server='AF_INET(192.168.6.3:1514)', time_reopen='10' 2015-07-29T02:26:51+00:00 syslog.server syslog notice syslog-ng[30512]: Syslog connection established; fd='379', server='AF_INET(192.168.6.3:1514)', local='AF_INET(0.0.0.0:0)' 2015-07-29T02:26:51+00:00 syslog.server syslog notice syslog-ng[30512]: Syslog connection broken; fd='379', server='AF_INET(192.168.6.3:1514)', time_reopen='10' 2015-07-29T02:27:01+00:00 syslog.server syslog notice syslog-ng[30512]: Syslog connection established; fd='476', server='AF_INET(192.168.6.3:1514)', local='AF_INET(0.0.0.0:0)' 2015-07-29T02:27:02+00:00 syslog.server syslog notice syslog-ng[30512]: Syslog connection broken; fd='476', server='AF_INET(192.168.6.3:1514)', time_reopen='10' tcpdump shows normal data flow followed by two TCP resets coming back from the graylog-1.1.5 server - so it's definitely graylog that's borking. BTW, this system *is working*: I'm seeing these syslogs flowing in - can do searches/etc - but I assume I'm losing some records due to this issue. I even created a xinetd.d based tcp service on the graylog server that just logged what it received to a file, configured the syslog server to send to both tcp channels - and it's running fine with no restarts (ie tcpdump of both ports only shows TCP resets on the graylog port not the xinetd port). So I think that implies it isn't the OS (CentOS-7) Whatever the root cause is should be logged somewhere - can someone point out to me how the debug this? Thanks Jason -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Node not taking messages
I ended up figuring it out. Total oversight on my side.
On Monday, July 27, 2015 at 1:42:45 PM UTC-5, Mike Daoust wrote:
>
> Im completely drawing a blank here.
>
> Have a 3 GL server nodes. Only 2 of them are taking messages. The node
> not getting any messages is starting just fine and is listed in the
> web.conf of all web nodes. Any suggestions?
>
>
>
>
>
> 2015-07-27T13:14:14.160-05:00 INFO [node] [Internal-Logging03-GL] started
>> 2015-07-27T13:14:14.295-05:00 INFO [service] [Internal-Logging03-GL]
>> detected_master
>> [Internal-Logging01-ES][BzV1fQD9Q6m10QrTFZB44g][Internal-Logging01][inet[/10.242.228.161:9300]],
>>
>> added
>> {[Internal-Logging01-ES][BzV1fQD9Q6m10QrTFZB44g][Internal-Logging01][inet[/10.242.228.161:9300]],[Internal-Logging01-GL][zDEBwMlESBuvo43FiPEzlg][Internal-Logging01][inet[/10.242.228.161:9350]]{client=true,
>>
>> data=false,
>> master=false},[Internal-Logging03-ES][yPdS241YQ7WIuGaoKCSYww][Internal-Logging03][inet[/10.242.230.162:9300]],[Internal-Logging02-GL][22EmM-TARRCwr2-G8CaPSw][Internal-Logging02][inet[/10.242.228.166:9350]]{client=true,
>>
>> data=false,
>> master=false},[Internal-Logging02-ES][PUD74j4qRjCmQvfr6U3zKw][Internal-Logging02][inet[/10.242.228.166:9300]],},
>>
>> reason: zen-disco-receive(from master
>> [[Internal-Logging01-ES][BzV1fQD9Q6m10QrTFZB44g][Internal-Logging01][inet[/10.242.228.161:9300]]])
>> 2015-07-27T13:14:18.654-05:00 INFO [RestApiService] Adding security
>> context factory:
>>
>> 2015-07-27T13:14:18.674-05:00 INFO [RestApiService] Started REST API at <
>> http://internal-logging03.apps.appriver.corp:12900/>
>> 2015-07-27T13:14:18.676-05:00 INFO [ServiceManagerListener] Services are
>> healthy
>> 2015-07-27T13:14:18.678-05:00 INFO [ServerBootstrap] Services started,
>> startup times in ms: {InputSetupService [RUNNING]=5,
>> GroovyShellSetupService [RUNNING]=11, MetricsReporterService [RUNNING]=11,
>> OutputSetupService [RUNNING]=15, BufferSynchronizerService [RUNNING]=15,
>> DashboardRegistryService [RUNNING]=16, KafkaJournal [RUNNING]=19,
>> PeriodicalsService [RUNNING]=28, JournalReader [RUNNING]=35,
>> IndexerSetupService [RUNNING]=3374, RestApiService [RUNNING]=7716}
>> 2015-07-27T13:14:18.680-05:00 INFO [InputSetupService] Triggering
>> launching persisted inputs, node transitioned from Uninitialized?[LB:DEAD]
>> to Running?[LB:ALIVE]
>> 2015-07-27T13:14:18.687-05:00 INFO [ServerBootstrap] Graylog server up
>> and running.
>> 2015-07-27T13:14:18.725-05:00 INFO [InputStateListener] Input [GELF
>> UDP/55b643c00cf2b25b612bda04] is now STARTING
>> 2015-07-27T13:14:18.959-05:00 INFO [InputStateListener] Input [GELF
>> UDP/55b643c00cf2b25b612bda04] is now RUNNING
>> 2015-07-27T13:14:20.650-05:00 INFO [AbstractValidatingSessionManager]
>> Enabling session validation scheduler...
>> 2015-07-27T13:14:20.753-05:00 INFO [Version] HV01: Hibernate
>> Validator 5.1.2.Final
>> ^C
>
>
>
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] Graylog web interface routine disconnects
Happy Tuesday, Graylog community >From time-to-time, I find that the web interface cannot contact a graylog server. This occurs in both clustered and non-clustered environments. To simplify things, I'm focusing only on the all-in-one instances for now. This is a Graylog 1.1.4 instance running in AWS under the provided AMI. Workload is steady, but load average looks fine, node stats don't show exhausted buffers, machine is spec'd well enough (4 proc, 32gb RAM). Refreshing the page once or twice is usually enough to bring it back to life. When this occurs, I don't see errors in /var/log/graylog/elasticsearch/current /var/log/graylog/server/current /var/log/graylog/mongodb/current I do see some connection errors in /var/log/graylog/nginx/error.log (IPs and DNS masked) 2015/07/26 21:06:44 [error] 32592#0: *3 connect() failed (111: Connection refused) while connecting to upstream, client: x.x.x.x, server: , request: "GET /a/metrics/info?t=1437944810321 HTTP/1.1", upstream: "http://127.0.0.1:9000/a/metrics/info?t=1437944810321";, host: "example.com", referrer: "https://example.com/search?rangetype=relative&relative=604800&q=&interval=hour&page=1&fields=message%2Csource&sortField=timestamp&sortOrder=desc"; 2015/07/26 21:06:44 [error] 32592#0: *3 connect() failed (111: Connection refused) while connecting to upstream, client: x.x.x.x, server: , request: "GET /a/connection/available HTTP/1.1", upstream: "http://127.0.0.1:9000/a/connection/available";, host: "example.com", referrer: "https://example.com/disconnected"; 2015/07/26 21:07:55 [error] 32592#0: *14 upstream prematurely closed connection while reading upstream, client: x.x.x.x, server: , request: "POST /a/metrics/383/biyvtlrv/xhr?t=1437944870413 HTTP/1.1", upstream: "http://127.0.0.1:9000/a/metrics/383/biyvtlrv/xhr?t=1437944870413";, host: "example.com", referrer: "https://example.com/search"; 2015/07/26 21:07:55 [error] 32592#0: *9 connect() failed (111: Connection refused) while connecting to upstream, client: x.x.x.x, server: , request: "POST /a/metrics/433/qfc23hgj/xhr?t=1437944881323 HTTP/1.1", upstream: "http://127.0.0.1:9000/a/metrics/433/qfc23hgj/xhr?t=1437944881323";, host: "example.com", referrer: "https://example.com/search?rangetype=relative&relative=604800&q=&interval=hour&page=1&fields=message%2Csource&sortField=timestamp&sortOrder=desc"; 2015/07/26 21:07:56 [error] 32592#0: *9 connect() failed (111: Connection refused) while connecting to upstream, client: x.x.x.x, server: , request: "GET /a/connection/available HTTP/1.1", upstream: "http://127.0.0.1:9000/a/connection/available";, host: "example.com", referrer: "https://example.com/disconnected"; 2015/07/26 21:08:41 [error] 508#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: x.x.x.x, server: , request: "GET /a/connection/available HTTP/1.1", upstream: "http://127.0.0.1:9000/a/connection/available";, host: "example.com", referrer: "https://example.com/disconnected"; 2015/07/26 21:08:42 [error] 508#0: *4 connect() failed (111: Connection refused) while connecting to upstream, client: x.x.x.x, server: , request: "GET /a/metrics/info?t=1437944917665 HTTP/1.1", upstream: "http://127.0.0.1:9000/a/metrics/info?t=1437944917665";, host: "example.com", referrer: "https://example.com/search"; 2015/07/26 21:08:42 [error] 508#0: *4 connect() failed (111: Connection refused) while connecting to upstream, client: x.x.x.x, server: , request: "GET /a/system/notifications HTTP/1.1", upstream: "http://127.0.0.1:9000/a/system/notifications";, host: "example.com", referrer: "https://example.com/search"; 2015/07/26 21:08:43 [error] 508#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: x.x.x.x, server: , request: "GET /a/connection/available HTTP/1.1", upstream: "http://127.0.0.1:9000/a/connection/available";, host: "example.com", referrer: "https://example.com/disconnected"; Since everything is self-contained, I'm not sure why a call to 127.0.0.1:9000 would be refused. ufw is disabled. Any ideas where else to look? -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: how to use sessionid with rest api
Hi,
the session handling of Graylog is kind of special (i. e. using a very
custom mechanism).
You'll have to use the session ID as user name and the special password
"session" as credentials and send those as standard Authorization header
(Basic Auth) to the Graylog server.
In the end your requests will look like
http://${SESSION_ID}:sess...@graylog.example.net:12900/foo/bar.
Cheers,
Jochen
On Tuesday, 28 July 2015 16:21:56 UTC+2, Emde wrote:
>
> Hello,
>
> I am accesing the REST API of Graylog from java.
>
> I want to use a sessionId to make the calls to the REST api without
> username and password. I haven't found any documentation reading how this
> should be set in order for graylog to accept it. I have tried the following
> and all of them return 401 Unauthorized:
>
> URLConnection graylogConnection = graylogUrl.openConnection();
> graylogConnection.setRequestProperty("sessionid", sessionId);
> graylogConnection.setRequestProperty("session-id", sessionId);
> graylogConnection.setRequestProperty("session_id", sessionId);
> graylogConnection.setRequestProperty("id", sessionId);
> graylogConnection.setRequestProperty("authorization", "Bearer " +
> sessionId);
> graylogConnection.setRequestProperty("JSESSIONID", sessionId);
> graylogConnection.setRequestProperty("Cookie", "JSESSIONID=" + sessionId);
>
> I got the sessionId value from the system/sessions resource and has a
> valid expiry date(it is not expired).
> I also tried all of the above with sessionId Base64 encoded.
>
> Could anyone suggest how I should set the sessionId in order for graylog
> to accept it?
>
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Setting default TTL for new indices
Hi, Also, I tried using the elasticsearch_config_file setting in Graylog > config to make it point to /etc/elasticsearch/elasticsearch.yml but then > I ran into the problem described here, so I had to roll back: > https://groups.google.com/forum/#!topic/graylog2/PvP2BYNe0xg Sounds like you tried to use the configuration file for your normal Elasticsearch node(s) with Graylog. That won't work. You have to use a configuration file which only contains the bare minimum of what you want to change for the Elasticsearch client node used inside Graylog. Many options are also overridden by Graylog directly (see configuration settings prefixed with elasticsearch_ in the Graylog configuration file). Cheers, Jochen On Tuesday, 28 July 2015 15:21:32 UTC+2, ZeroUno wrote: > > Il 28/07/15 14:13, Jochen Schalanda ha scritto: > > > you can add your index template to the Elasticsearch configuration file > > ( > https://www.elastic.co/guide/en/elasticsearch/reference/0.90/indices-templates.html#config) > > > > referenced in the elasticsearch_config_file setting in the Graylog > > configuration file > > Thanks, but the link you posted just talks about putting the template > file into the config/templates directory, which is what I already did, > > Also, I tried using the elasticsearch_config_file setting in Graylog > config to make it point to /etc/elasticsearch/elasticsearch.yml but then > I ran into the problem described here, so I had to roll back: > https://groups.google.com/forum/#!topic/graylog2/PvP2BYNe0xg > > Anyway, I found the problem: my template file was missing a comma before > "message", sorry :-P. I should have looked better at the logs. > > So now as far as I can see my template file is working fine, I can even > only use the "message" mapping without the need to add the "_default_" > one. > Apart from this, I added the indices.ttl.interval setting in > elasticsearch.yml, this should tell it how often expired documents must > be deleted, e.g. "7d" for 7 days. > > -- > 01 > > -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] how to use sessionid with rest api
Hello,
I am accesing the REST API of Graylog from java.
I want to use a sessionId to make the calls to the REST api without
username and password. I haven't found any documentation reading how this
should be set in order for graylog to accept it. I have tried the following
and all of them return 401 Unauthorized:
URLConnection graylogConnection = graylogUrl.openConnection();
graylogConnection.setRequestProperty("sessionid", sessionId);
graylogConnection.setRequestProperty("session-id", sessionId);
graylogConnection.setRequestProperty("session_id", sessionId);
graylogConnection.setRequestProperty("id", sessionId);
graylogConnection.setRequestProperty("authorization", "Bearer " +
sessionId);
graylogConnection.setRequestProperty("JSESSIONID", sessionId);
graylogConnection.setRequestProperty("Cookie", "JSESSIONID=" + sessionId);
I got the sessionId value from the system/sessions resource and has a valid
expiry date(it is not expired).
I also tried all of the above with sessionId Base64 encoded.
Could anyone suggest how I should set the sessionId in order for graylog to
accept it?
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Setting default TTL for new indices
Il 28/07/15 14:13, Jochen Schalanda ha scritto: you can add your index template to the Elasticsearch configuration file (https://www.elastic.co/guide/en/elasticsearch/reference/0.90/indices-templates.html#config) referenced in the elasticsearch_config_file setting in the Graylog configuration file Thanks, but the link you posted just talks about putting the template file into the config/templates directory, which is what I already did, Also, I tried using the elasticsearch_config_file setting in Graylog config to make it point to /etc/elasticsearch/elasticsearch.yml but then I ran into the problem described here, so I had to roll back: https://groups.google.com/forum/#!topic/graylog2/PvP2BYNe0xg Anyway, I found the problem: my template file was missing a comma before "message", sorry :-P. I should have looked better at the logs. So now as far as I can see my template file is working fine, I can even only use the "message" mapping without the need to add the "_default_" one. Apart from this, I added the indices.ttl.interval setting in elasticsearch.yml, this should tell it how often expired documents must be deleted, e.g. "7d" for 7 days. -- 01 -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Setting default TTL for new indices
Hi, you can add your index template to the Elasticsearch configuration file ( https://www.elastic.co/guide/en/elasticsearch/reference/0.90/indices-templates.html#config) referenced in the elasticsearch_config_file setting in the Graylog configuration file ( https://github.com/Graylog2/graylog2-server/blob/0.20.1/misc/graylog2.conf#L41-43). This file is being used for additional configuration of the Elasticsearch node client being used inside Graylog itself. This being said, you should really upgrade to a more recent of Graylog as there've been substantial improvements regarding performance and stability since Graylog2 0.20.x. Cheers, Jochen On Tuesday, 28 July 2015 13:42:47 UTC+2, ZeroUno wrote: > > Il 28/07/15 12:06, ZeroUno ha scritto: > > > I tried with templates: I created this file as > > templates/graylog2_ttl_template.json inside /etc/elasticsearch, which is > > where elasticsearch configuration can be found: > > A small update: I can see this can work if I create the template by curl > -XPUT localhost:9200/_template/graylog2_ttl_template -d '...' > > But for my purpose using a file would be much better. > Anyone knows why this is not working? > > Thanks. > > -- > 01 > > -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Setting default TTL for new indices
Il 28/07/15 12:06, ZeroUno ha scritto: I tried with templates: I created this file as templates/graylog2_ttl_template.json inside /etc/elasticsearch, which is where elasticsearch configuration can be found: A small update: I can see this can work if I create the template by curl -XPUT localhost:9200/_template/graylog2_ttl_template -d '...' But for my purpose using a file would be much better. Anyone knows why this is not working? Thanks. -- 01 -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Setting default TTL for new indices
Hi,
I'm using the following software versions which I'm not allowed to
upgrade, on RedHat:
graylog2 0.20.1
elasticsearch 0.90.10
Graylog gets messages from logstash.
I need to set a default TTL for all incoming messages, but I'm having
trouble with this.
Most online resources simply suggest this:
curl -XPUT "http://localhost:9200/graylog2/message/_mapping";
-d'{"message": {"_ttl" : { "enabled" : true, "default" : "30d" }}}'
But, as far as I understand, this only adds the default _ttl for an
index called "graylog2".
In my configuration instead (which I guess is a default) graylog cycles
through many indices, calling them "graylog2_1", graylog2_2", and so on.
So I cannot just set _ttl for existing indices: I also need to define a
default value for all new indices that graylog will create.
I tried with templates: I created this file as
templates/graylog2_ttl_template.json inside /etc/elasticsearch, which is
where elasticsearch configuration can be found:
{
"graylog2_ttl_template" : {
"template" : "graylog2*",
"mappings" : {
"_default_" : {
"_ttl" : {
"enabled" : true,
"default" : "180d"
}
}
"message" : {
"_ttl" : {
"enabled" : true,
"default" : "180d"
}
}
}
}
}
(I used both "message" and "_default_" just to be sure)
Then I restarted elasticsearch, and in graylog GUI I manually cycled the
deflector, letting it create a new index. But the mapping for the
message type in the new index is still the old one, which has _ttl
enabled but with no default duration.
What am I missing? Why isn't the template used?
What is the right way to accomplish this?
Thank you for any info.
--
01
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Add Role to Graylog2
Hi Tim, user roles will be supported in Graylog 1.2.0. You can follow https://github.com/Graylog2/graylog2-server/pull/1322 if you want to stay up-to-date on the issue. Cheers, Jochen On Tuesday, 28 July 2015 10:07:30 UTC+2, tim lewis wrote: > > Is there a way to add a new role to Graylog? We would like to control > access to different groups of users with potentially different sets of > permissions/responsibilities and not have to manually add permissions to > each account. Does Graylog support this concept? Thank you all for your > help. > -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Help with install script
Hi Joe,
if you want to automate the installation of Graylog and its dependencies,
you should use one of our existing recipes/modules/playbooks for Chef,
Puppet, or Ansible instead of writing your own bash script; see
http://docs.graylog.org/en/1.1/pages/installation/config_management_tools.html
for details.
Cheers,
Jochen
On Tuesday, 28 July 2015 10:07:30 UTC+2, Joe wrote:
>
> I am trying to install mongodb, elasticsearch, graylog2 and
> graylog2-web-interface on one server using Ubuntu 14.04, but I am getting
> errors such as java does not have enough memory. Ill post my script and
> hopefully someone can tell me what I am doing wrong and will help someone
> else down the line.
>
> #!/bin/bash
>
> #
> # Should use NTP on all machines for time sync
> #
>
> #Install Java 7
> cd
> sudo apt-add-repository ppa:webupd8team/java
> sudo apt-get update
> sudo apt-get install oracle-java7-installer
> read -rep $'\n+++Java Done.+++\n\nPress [Enter] key to continue...'
>
> #Install MongoDB -
> http://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/
> cd
> sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10
> printf "deb http://repo.mongodb.org/apt/ubuntu "$(lsb_release
> -sc)"/mongodb-org/3.0 multiverse" | sudo tee
> /etc/apt/sources.list.d/mongodb-org-3.0.list
> sudo apt-get update
> sudo apt-get install -y mongodb-org
> sudo service mongod start
> printf "\nVerify that the mongod process has started successfully by
> checking the ..."
> printf "contents of the log file at /var/log/mongodb/mongod.log for a line
> reading ..."
> printf "[initandlisten] waiting for connections on port "
> gnome-terminal -e 'nano /var/log/mongodb/mongod.log'
> read -rep $'\n\n+++MongoDB Done.+++\n\nPress [Enter] key to continue...'
>
> #Install elasticsearch - https://www.elastic.co/downloads/elasticsearch
> cd
> wget
> https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.0.deb
> sudo dpkg -i elasticsearch-1.7.0.deb
> gnome-terminal -e 'sudo nano /etc/elasticsearch/elasticsearch.yml'
> read -rep $'\nEdit Lines:\n\tcluster.name: graylog2\n\tnetwork.bind_host:
> 127.0.0.1\n\tAdd: script.disable_dynamic: true\n\nPress enter when done
> editing and saved.'
> sudo service elasticsearch restart
> sudo apt-get install curl
> curl -XGET 'http://localhost:9200/_cluster/health?pretty=true' #might
> fail if not enough time, but good
> read -rep $'\n+++ElasticSearch Done.+++\n\nPress [Enter] key to
> continue...'
>
>
> #Install graylog-server
> cd
> wget
> https://packages.graylog2.org/releases/graylog2-server/graylog-1.1.4.tgz
> tar xvfz graylog-1.1.4.tgz
> cd graylog-1.1.4
> sudo mkdir /etc/graylog The website directions assume this is a
> dir?
> sudo mkdir /etc/graylog/server
> sudo cp graylog.conf.example /etc/graylog/server/server.conf
> sudo apt-get install pwgen #Check if not already installed
> SECRET=$(pwgen -s 96 1)
> sudo -E sed -i -e 's/password_secret =.*/password_secret = '$SECRET'/'
> /etc/graylog/server/server.conf
> PASSWORD=$(echo -n password | shasum -a 256 | awk '{print $1}')
> sudo -E sed -i -e 's/root_password_sha2 =.*/root_password_sha2 =
> '$PASSWORD'/' /etc/graylog/server/server.conf
> gnome-terminal -e 'sudo nano /etc/graylog/server/server.conf'
> read -rep $'\n Just change elasticsearch_shards to 0 or \n\nEdit file
> with directions here:
> http://docs.graylog.org/en/1.1/pages/installation/manual_setup.html#downloading-and-extracting-the-server\n\nPress
>
> enter when done editing and saved.'
> #sudo nano /etc/graylog/server/server.conf
> sudo apt-get install openjdk-7-jre
> cd bin/
> ./graylogctl start
> read -rep $'\n+++GrayLog-Server Done.+++\n\nPress [Enter] key to
> continue...'
>
> #Install graylog-web interface
> cd
> wget
> https://packages.graylog2.org/releases/graylog2-web-interface/graylog-web-interface-1.1.4.tgz
> tar xvfz graylog-web-interface-1.1.4.tgz
> cd graylog-web-interface-1.1.4
> SECRET=$(pwgen -s 96 1)
> sudo -E sed -i -e
> 's/application\.secret=""/application\.secret="'$SECRET'"/'
> conf/graylog-web-interface.conf
> gnome-terminal -e 'sudo nano conf/graylog-web-interface.conf'
> read -rep $'\n Just change
> graylog2-server.uris=\"http://127.0.0.1:12900/\";
> or \n\nEdit file with directions here:
> http://docs.graylog.org/en/1.1/pages/installation/manual_setup.html#downloading-and-extracting-the-server
> '
> #sudo nano conf/graylog-web-interface.conf
> bin/graylog-web-interface
> read -rep $'\n+++GrayLog-Web-Interface Done.+++\n\nPress [Enter] key to
> continue...'
>
>
>
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Logging in GELF over TLS
Hi Russ, Graylog supports Structured Syslog as defined per RFC 5424 ( https://tools.ietf.org/html/rfc5424) which means that it will automatically extract data from syslog messages containing structured data elements. Cheers, Jochen On Monday, 27 July 2015 19:00:30 UTC+2, Russ wrote: > > Hi Jochen, > > Thanks for the reply! > > When you mention using structured syslog data through syslog, how does > that work? Does graylog automatically detect structured data inside a > standard syslog message? I tried including JSON blobs and key value pairs > in my syslog message but it didn't seem to pick it up. Maybe I need to > setup an extractor? Or would this be like CEE for rsyslog? > > Thanks for your help! > > -Russ > > On Monday, July 27, 2015 at 1:32:00 AM UTC-7, Jochen Schalanda wrote: >> >> Hi Russ, >> >> most third-party libraries only support sending GELF over UDP, some also >> support TCP, and very few support GELF over TCP+TLS. For example our own >> Java-based gelfclient (https://github.com/Graylog2/gelfclient) supports >> all three modes. If you're missing a specific transport mode in some >> library, you should contact the author of the respective project and see >> what's required to get support for TLS into the library. >> >> The support for GELF over TCP+TLS is relatively new (introduced in >> Graylog 1.0.0), so some libraries just lag behind or nobody really asked >> for encrypted transport yet. >> >> FWIW you might work around this limitation by spanning a VPN underneath >> the logging clients and the Graylog servers in order to move the encryption >> to a "lower" layer in the stack or use structured syslog to include more >> information in syslog messages sent to Graylog. >> >> >> Cheers, >> Jochen >> >> On Monday, 27 July 2015 10:15:50 UTC+2, Russ wrote: >>> >>> Hi Everyone, >>> >>> I'm trying to run graylog with a huge amount of incoming messages per >>> second. To speed up indexing and make the data more easily queryable I'm >>> interested in using GELF. When I do it over UDP it works perfectly; it's >>> super efficient. >>> >>> However, I can't find much information (libraries/modules/etc for Java >>> and Python) with folks logging in GELF over TCP with TLS. Unfortunately I >>> have some requirements that won't allow me ship the messages over an >>> unencrypted connection and am forced to used to TLS. I can solve for this >>> now by using rsyslog but it's not in GELF format. >>> >>> I'm curious if I'm not finding anything about GELF over TLS because: >>> >>> (A) there is a reason this sort of thing shouldn't be done, >>> (B) I'm not looking in the right places >>> (C) Just hasn't been a library developed that does this yet. >>> (D) Something else. >>> >>> I started prototyping something with Python (socket/ssl) to do it but >>> didn't want to get too far down the path if there is something fundamental >>> I'm missing. They GELF documentation mentions TCP introduces headaches >>> which I can understand from a high level but I'm not so much of an expert >>> to know if I'll get myself in over my head if I write something to do this. >>> >>> Thanks for your help! >>> >>> -Russ >>> >> -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Node not taking messages
Hi Mike,
how are you sending messages to those Graylog nodes? Do you see any
throughput for the inputs on that node in the web interface?
Cheers,
Jochen
On Monday, 27 July 2015 20:42:45 UTC+2, Mike Daoust wrote:
>
> Im completely drawing a blank here.
>
> Have a 3 GL server nodes. Only 2 of them are taking messages. The node
> not getting any messages is starting just fine and is listed in the
> web.conf of all web nodes. Any suggestions?
>
>
>
>
>
> 2015-07-27T13:14:14.160-05:00 INFO [node] [Internal-Logging03-GL] started
>> 2015-07-27T13:14:14.295-05:00 INFO [service] [Internal-Logging03-GL]
>> detected_master
>> [Internal-Logging01-ES][BzV1fQD9Q6m10QrTFZB44g][Internal-Logging01][inet[/10.242.228.161:9300]],
>>
>> added
>> {[Internal-Logging01-ES][BzV1fQD9Q6m10QrTFZB44g][Internal-Logging01][inet[/10.242.228.161:9300]],[Internal-Logging01-GL][zDEBwMlESBuvo43FiPEzlg][Internal-Logging01][inet[/10.242.228.161:9350]]{client=true,
>>
>> data=false,
>> master=false},[Internal-Logging03-ES][yPdS241YQ7WIuGaoKCSYww][Internal-Logging03][inet[/10.242.230.162:9300]],[Internal-Logging02-GL][22EmM-TARRCwr2-G8CaPSw][Internal-Logging02][inet[/10.242.228.166:9350]]{client=true,
>>
>> data=false,
>> master=false},[Internal-Logging02-ES][PUD74j4qRjCmQvfr6U3zKw][Internal-Logging02][inet[/10.242.228.166:9300]],},
>>
>> reason: zen-disco-receive(from master
>> [[Internal-Logging01-ES][BzV1fQD9Q6m10QrTFZB44g][Internal-Logging01][inet[/10.242.228.161:9300]]])
>> 2015-07-27T13:14:18.654-05:00 INFO [RestApiService] Adding security
>> context factory:
>>
>> 2015-07-27T13:14:18.674-05:00 INFO [RestApiService] Started REST API at <
>> http://internal-logging03.apps.appriver.corp:12900/>
>> 2015-07-27T13:14:18.676-05:00 INFO [ServiceManagerListener] Services are
>> healthy
>> 2015-07-27T13:14:18.678-05:00 INFO [ServerBootstrap] Services started,
>> startup times in ms: {InputSetupService [RUNNING]=5,
>> GroovyShellSetupService [RUNNING]=11, MetricsReporterService [RUNNING]=11,
>> OutputSetupService [RUNNING]=15, BufferSynchronizerService [RUNNING]=15,
>> DashboardRegistryService [RUNNING]=16, KafkaJournal [RUNNING]=19,
>> PeriodicalsService [RUNNING]=28, JournalReader [RUNNING]=35,
>> IndexerSetupService [RUNNING]=3374, RestApiService [RUNNING]=7716}
>> 2015-07-27T13:14:18.680-05:00 INFO [InputSetupService] Triggering
>> launching persisted inputs, node transitioned from Uninitialized?[LB:DEAD]
>> to Running?[LB:ALIVE]
>> 2015-07-27T13:14:18.687-05:00 INFO [ServerBootstrap] Graylog server up
>> and running.
>> 2015-07-27T13:14:18.725-05:00 INFO [InputStateListener] Input [GELF
>> UDP/55b643c00cf2b25b612bda04] is now STARTING
>> 2015-07-27T13:14:18.959-05:00 INFO [InputStateListener] Input [GELF
>> UDP/55b643c00cf2b25b612bda04] is now RUNNING
>> 2015-07-27T13:14:20.650-05:00 INFO [AbstractValidatingSessionManager]
>> Enabling session validation scheduler...
>> 2015-07-27T13:14:20.753-05:00 INFO [Version] HV01: Hibernate
>> Validator 5.1.2.Final
>> ^C
>
>
>
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] Help with install script
I am trying to install mongodb, elasticsearch, graylog2 and
graylog2-web-interface on one server using Ubuntu 14.04, but I am getting
errors such as java does not have enough memory. Ill post my script and
hopefully someone can tell me what I am doing wrong and will help someone
else down the line.
#!/bin/bash
#
# Should use NTP on all machines for time sync
#
#Install Java 7
cd
sudo apt-add-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java7-installer
read -rep $'\n+++Java Done.+++\n\nPress [Enter] key to continue...'
#Install MongoDB -
http://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/
cd
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10
printf "deb http://repo.mongodb.org/apt/ubuntu "$(lsb_release
-sc)"/mongodb-org/3.0 multiverse" | sudo tee
/etc/apt/sources.list.d/mongodb-org-3.0.list
sudo apt-get update
sudo apt-get install -y mongodb-org
sudo service mongod start
printf "\nVerify that the mongod process has started successfully by
checking the ..."
printf "contents of the log file at /var/log/mongodb/mongod.log for a line
reading ..."
printf "[initandlisten] waiting for connections on port "
gnome-terminal -e 'nano /var/log/mongodb/mongod.log'
read -rep $'\n\n+++MongoDB Done.+++\n\nPress [Enter] key to continue...'
#Install elasticsearch - https://www.elastic.co/downloads/elasticsearch
cd
wget
https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.0.deb
sudo dpkg -i elasticsearch-1.7.0.deb
gnome-terminal -e 'sudo nano /etc/elasticsearch/elasticsearch.yml'
read -rep $'\nEdit Lines:\n\tcluster.name: graylog2\n\tnetwork.bind_host:
127.0.0.1\n\tAdd: script.disable_dynamic: true\n\nPress enter when done
editing and saved.'
sudo service elasticsearch restart
sudo apt-get install curl
curl -XGET 'http://localhost:9200/_cluster/health?pretty=true' #might fail
if not enough time, but good
read -rep $'\n+++ElasticSearch Done.+++\n\nPress [Enter] key to continue...'
#Install graylog-server
cd
wget
https://packages.graylog2.org/releases/graylog2-server/graylog-1.1.4.tgz
tar xvfz graylog-1.1.4.tgz
cd graylog-1.1.4
sudo mkdir /etc/graylog The website directions assume this is a dir?
sudo mkdir /etc/graylog/server
sudo cp graylog.conf.example /etc/graylog/server/server.conf
sudo apt-get install pwgen #Check if not already installed
SECRET=$(pwgen -s 96 1)
sudo -E sed -i -e 's/password_secret =.*/password_secret = '$SECRET'/'
/etc/graylog/server/server.conf
PASSWORD=$(echo -n password | shasum -a 256 | awk '{print $1}')
sudo -E sed -i -e 's/root_password_sha2 =.*/root_password_sha2 =
'$PASSWORD'/' /etc/graylog/server/server.conf
gnome-terminal -e 'sudo nano /etc/graylog/server/server.conf'
read -rep $'\n Just change elasticsearch_shards to 0 or \n\nEdit file
with directions here:
http://docs.graylog.org/en/1.1/pages/installation/manual_setup.html#downloading-and-extracting-the-server\n\nPress
enter when done editing and saved.'
#sudo nano /etc/graylog/server/server.conf
sudo apt-get install openjdk-7-jre
cd bin/
./graylogctl start
read -rep $'\n+++GrayLog-Server Done.+++\n\nPress [Enter] key to
continue...'
#Install graylog-web interface
cd
wget
https://packages.graylog2.org/releases/graylog2-web-interface/graylog-web-interface-1.1.4.tgz
tar xvfz graylog-web-interface-1.1.4.tgz
cd graylog-web-interface-1.1.4
SECRET=$(pwgen -s 96 1)
sudo -E sed -i -e
's/application\.secret=""/application\.secret="'$SECRET'"/'
conf/graylog-web-interface.conf
gnome-terminal -e 'sudo nano conf/graylog-web-interface.conf'
read -rep $'\n Just change
graylog2-server.uris=\"http://127.0.0.1:12900/\"; or \n\nEdit file with
directions here:
http://docs.graylog.org/en/1.1/pages/installation/manual_setup.html#downloading-and-extracting-the-server'
#sudo nano conf/graylog-web-interface.conf
bin/graylog-web-interface
read -rep $'\n+++GrayLog-Web-Interface Done.+++\n\nPress [Enter] key to
continue...'
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] Add Role to Graylog2
Is there a way to add a new role to Graylog? We would like to control access to different groups of users with potentially different sets of permissions/responsibilities and not have to manually add permissions to each account. Does Graylog support this concept? Thank you all for your help. -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
