[graylog2] Re: One of the nodes in the cluster not processing messages

[Brett Child]

Just had a similar issue today.  Everything was up but search wasn't 
working and messages were not processing.  A restart of ES seemed to fix 
it.  Hope this helps.  

On Thursday, December 1, 2016 at 11:57:01 AM UTC-7, iskuruts wrote:
>
> One of the nodes in the cluster is not processing messages.  It has 
> 475,745 unprocessed messages.  How do I force it to process messages?  The 
> elasticsearch is up as well as graylog service. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/ba8d767e-3499-4b58-8144-834151c39694%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Re: Graylog V2.1.2 - getting GELF HTTP working with simple HTTP POST/JSON body

[Mike Norris]

Jochen

Thanks for this suggestion, but it still isn't working

I ran ipconfig /all on my windows box via the command line
and got IP address 10.0.75.1 for my Docker container  (see below)

I tried using 10.0.75.1 as the bind address in Graylog inputs, this failed
to start up, so I reset it to 0.0.0.0
- this 0.0.0.0 setting allows the GELF HTTP input listener to start
successfully, 10.0.75.1 will not

Then I entered 10.0.75.1:9000 in my browser and the Graylog login page
appeared

Then I tried to send a HTTP Post with JSON body to address 10.0.75.1:12201,
as still get the same old error
e.g. Invoke-WebRequest : Unable to connect to the remote server

If I enter http://10.0.75.1:12201/gelf in my browser I also get the same
kind of error

IS it possible that the /gelf part of the URL is wrong? at a basic level
the browser will display a login page for http://10.0.75.1:900

I believe 10.0.75.1 is reachable but :12202 or /gelf is not ??

I remain confused by GELF over HTTP, its not https is it ???

Mike

--- output from ipconfig /all)
*Ethernet adapter vEthernet (DockerNAT):*

*   Connection-specific DNS Suffix  . :*
*   Description . . . . . . . . . . . : Hyper-V Virtual Ethernet Adapter*
*   Physical Address. . . . . . . . . : 00-15-5D-15-4C-00*
*   DHCP Enabled. . . . . . . . . . . : No*
*   Autoconfiguration Enabled . . . . : Yes*
*   IPv4 Address. . . . . . . . . . . : 10.0.75.1(Preferred)*
*   Subnet Mask . . . . . . . . . . . : 255.255.255.0*
On Tue, Dec 6, 2016 at 3:46 PM Jochen Schalanda  wrote:

> Hi Mike,
>
> if you bind the GELF HTTP input to 127.0.0.1, only processes on the same
> machine (or running in the Docker container) can access it.
>
> Try binding the GELF HTTP input to the public IP address of the Docker
> container or 0.0.0.0.
>
> Cheers,
> Jochen
>
>
> On Monday, 5 December 2016 20:57:54 UTC+1, Mike Norris wrote:
>
> Hi
>
> I am having some troubles technically with a GELF HTTP input
>
> I cannot POST a simple sample JSON message to my GELF endpoint, *I keep
> getting "Unable to connect to the remote server"*
>
> Graylog has been installed as a Docker runtime container(s) on my Windows
> 10 Surface laptop
>
> Starting Graylog up is simple, the Docker process worked first time, so I
> have the following
>
> *a. Graylog V2.1.2 running*
>
> *b. I can login to the console via htttp:127.0.0.1:9000
>  as admin/admin*
>
> *c. I've created a basic GELF HTTP input in System ... Inputs*
>
> Type = GELF HTTP
> Bind address 127.0.0.1
> Port 12201
> removed any suggestion of TLS authentication, which is [optional] anyway
>
> This endpoint is running ok
>
> So in theory I have a listening process on port 12201 waiting and ready
> for GELF messages
>
> *d. I've tried POSTing simple requests as per the examples
> in http://docs.graylog.org/en/2.1/pages/gelf.html
> *
>
> *e. My tool of choice for the client is Windows .NET and I've tried both
> PowerShell and C# programs*
>
> All I think I have to do is deliver a correct json payload in a sim ple
> HHTP request to the right endpoint for it to work
>
> *Example - PowerShell code*
>
> Set-ExecutionPolicy Unrestricted
> Get-ExecutionPolicy
>
> $resource = "http://localhost:12201/gelf;
>
> $body = '"{version": "1.1","host": "example.org","short_message": "A
> short message that helps you identify what is going on","full_message":
> "Backtrace here\n\nmore stuff","timestamp": 1385053862.3072,"level":
> 1,"_user_id": 9001,"_some_info":"foo","_some_env_var":"bar"}'
>
> Invoke-WebRequest -Uri "http://127.0.0.1:12201/gelf; -Method Post
> -TransferEncoding "gzip" -ContentType "application/json; charset=UTF-8"
> -Body $body
>
> I initailly wrote the PowerShell code without invoking gzip compression,
> I've recently added that to no avail
>
> *f. I don't understand why the error I get is Unable to connect, as
> Graylog is quite happy to serve pages on the same IP address on port 9000*
>
> The GELF INPUT is in a running state?
>
>
> Is this one of those examples where my input is wrong and the error
> message is misleading?
>
>
>
>
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Graylog Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/graylog2/J7Ja2RGzE_4/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/graylog2/b7452a2a-4bee-404c-b371-92494c4987d2%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send 

[graylog2] Re: Pipeline with multiple streams and data accumulation

[Evgeny Shepelyuk]

Well, I'm actually interested in data accumulation.
Let me rephrase, is it possible for pipeline to analyze two input streams, 
compare then and in some condition send a message to this stream ?

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/f5243bb5-e631-4544-8e7f-09b1e436e09c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Single or double quotes in json messages for configuration via api

[Marius Sturm]

Hi,
the right syntax is to use single quotes, but this is problematic with
curl because the json string itself is usually put in single quotes.
So your call looks like this:

curl -X POST -d '{some json}'

now you can't simply put another single quote in the json string like
-d '{some' json}' that would terminate the string at that point.
One way is to use unicode for that. You can do it like:

curl -X POST -d '{some \u0027 json}'

A complete call to create e.g. an collector input looks then like this:

curl -v -u admin:admin -H "Content-Type: application/json" -X POST -d
'{"backend": "filebeat","type": "file","name":
"file-input","properties": {"paths":
"[\u0027/var/log/*.log\u0027]","scan_frequency": "10s","encoding":
"plain","ignore_older": "0","document_type": "log","exclude_lines":
"[]","include_lines": "[]","tail_files": true},"forward_to":
"5846e531adca4c0315bc6e8b"}'
http://192.168.1.109:9000/api/plugins/org.graylog.plugins.collector/configurations/5846e51eadca4c0315bc6e77/inputs

Cheers,
Marius


On 6 December 2016 at 00:06, Evgueni Gordienko  wrote:
> Hi,
>
> When I curl with GET method on collector config info I get
> "paths" : "[\"/var/log/*.log\",\"/var/log/messages\",\"/var/log/secure\"]"
>
> So which form is correct in POST for configuration:
>
> "properties": {
> "paths": "['/var/log/*.log','/var/log/messages']",
>
> or
>
> "properties": {
> "paths":
> "[\"/var/log/*.log\",\"/var/log/messages\",\"/var/log/secure\"]",
>
> Thanks,
> Evgueni
>
> --
> You received this message because you are subscribed to the Google Groups
> "Graylog Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/graylog2/752950d1-c531-49c4-9aa9-d6853e64d8fc%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Developer

Tel.: +49 (0)40 609 452 077
Fax.: +49 (0)40 609 452 078

TORCH GmbH - A Graylog Company
Poolstraße 21
20335 Hamburg
Germany

https://www.graylog.com

Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175
Geschäftsführer: Lennart Koopmann (CEO)

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/CAMqbBbK3UPBUtHRzrKbekUMXQ-%3D1ZV9WKi%3DMh1K7f8GengnatQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: One of the nodes in the cluster not processing messages

[Jochen Schalanda]

Hi,

provided that the Elasticsearch cluster is up and healthy, a restart of the 
Graylog node might help.

Cheers,
Jochen

On Thursday, 1 December 2016 19:57:01 UTC+1, iskuruts wrote:
>
> One of the nodes in the cluster is not processing messages.  It has 
> 475,745 unprocessed messages.  How do I force it to process messages?  The 
> elasticsearch is up as well as graylog service. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/d64a7726-c2c5-492a-bdb2-453ace501d17%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Saved search with specific fields (as favourites)

[Jochen Schalanda]

Hi Daniel,

please see https://github.com/Graylog2/graylog2-server/issues/3071 for a 
related issue on GitHub.

Cheers,
Jochen

On Wednesday, 30 November 2016 15:58:40 UTC+1, danie...@inner-active.com 
wrote:
>
> Hi all,
> I'd like to execute a search with a specific fields, and to save that 
> field any time (without configuring it any time...).
>
> Is there an option to do so within GrayLog?
>
> The field I'd like to search is (and only!) "full_message" field.
> How to do so?
>
> Thanks a lot!
> Daniel
> daniel@inner-active.com
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/74daea98-b79c-4652-841f-bfb967ff1263%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Pipeline with multiple streams and data accumulation

[Jochen Schalanda]

Hi Evgeny,

the pipelines don't trigger any alarms, they simply filter and mutate 
messages.

This being said, message correlation in alerts (or in general) is currently 
not possible.

Cheers,
Jochen

On Tuesday, 29 November 2016 14:20:58 UTC+1, Evgeny Shepelyuk wrote:
>
> Hello, 
>
> Is it possible to write a pipeline that will trigger alerts comparing data 
> from 2 streams ?
> For instance, one stream - is user registration, another stream is 
> clicking confirmation emails
> So, is it possible to create a pipeline that will trigger an alert when 
> count from both streams ain't equal during the day ?
>
> Thanks
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/6d1797c0-472e-4942-bbbc-6a9ebb597a1b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: extractor not working properly

[Jochen Schalanda]

For reference: https://github.com/Graylog2/graylog2-server/issues/3131

On Monday, 28 November 2016 14:18:38 UTC+1, vivek reddy wrote:
>
> Hi,
>
> I am reading raw message from kafka with message as '2016-11-24 
> 15:10:59.529' 8 'PUSH_VODAF1_MSG' 'PUSH_VODAF1_MSG [ esmeid and sessions 
> unbind/stopped ]'
>
> i applied copy extractor on this message with csv convertor, however 
> graylog is failing to create those csv columns.
>
> csv seperator/delim = SYN ("\u0016")
> csv quote char = '
> csv escape char = \
>
> when the csv separator is "," ,graylog is properly parsing, but it fails 
> whenever we use SYN as a seperator.
>
> ny help would be welcome.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/b3021ea7-b409-456b-8b4f-2aa056ce1203%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Source field not alway correctly filled.

[Jochen Schalanda]

Hi,

this looks like some problem on the emitting client. Make sure that it can 
resolve its own hostname and that it's added to the log message.

Cheers,
Jochen

On Friday, 25 November 2016 14:37:03 UTC+1, leck...@gmail.com wrote:
>
> Hello ,
> I am running version 2.0 of graylog in a configuration: Load Balancder in 
> front of cluster of 3 graylog servers.
> 95 percent of messages are shown also with the sopurce field correctly 
> filled with the name of the originating machine.
> But there are 5% of messages, where in the source field there is only the 
> internal IP of Load balancer
>
> Here is example of such "bad" message with details. The difference I see 
> is that 
> source should have origin hostname or IP, but it only has LB IP.
>
> the facility , level and application fields are also not filled correctly
> They are filled in case of "OK" messages.
>
>
> Received by*: tcp syslog ngray4* on  ngray1 
> 
> Stored in index: graylog_235
> facility: Unknown
> level: -1message: 2016-11-25 13:26:20,308  authentication valid: false 
> for username: kralsource: 10.139.123.1gl2_remote_ip: 
> 10.139.123.1gl2:source_node: 
> 220e6aee-550e-4b7d-ba7b-414d40e9abc6gl2_remote_ip: 10.139.123.1
>
>
> BTW, this original message was sent form Windows using nxlog. Maybe there 
> is some way to remap message in nxlog to show source 
> also in graylog ?
>
> Thanks!
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/ad838a14-39d8-4579-9781-bbaadb868d6f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Possible Cause Of Memory Leak Graylog 2(v2.1.2)

[Jochen Schalanda]

Hi Yogesh,

please post your complete Graylog configuration file and describe your 
setup in more detail.

Cheers,
Jochen

On Friday, 25 November 2016 02:34:53 UTC+1, Yogesh Rao wrote:
>
> Hi,
>
> We have just started using graylog2 (2.1.2) on centos 7. I keep getting 
> below exception after few hours
>
> 2016-11-24T17:25:13.737+05:30 WARN [ProxiedResource] Unable to call 
> http://10.101.160.95:9000/api/system on node 
>
> java.net.SocketTimeoutException: timeout
>
> at okio.Okio$3.newTimeoutException(Okio.java:210) ~[graylog.jar:?]
>
> at okio.AsyncTimeout.exit(AsyncTimeout.java:288) ~[graylog.jar:?]
>
> at okio.AsyncTimeout$2.read(AsyncTimeout.java:242) ~[graylog.jar:?]
>
> at okio.RealBufferedSource.indexOf(RealBufferedSource.java:325) 
> ~[graylog.jar:?]
>
> at okio.RealBufferedSource.indexOf(RealBufferedSource.java:314) 
> ~[graylog.jar:?]
>
> at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:210) 
> ~[graylog.jar:?]
>
> at okhttp3.internal.http.Http1xStream.readResponse(Http1xStream.java:186) 
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.http.Http1xStream.readResponseHeaders(Http1xStream.java:127) 
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:53)
>  
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>  
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
>  
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>  
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
>  
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:109) 
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>  
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
>  
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) 
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>  
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:124)
>  
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>  
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
>  
> ~[graylog.jar:?]
>
> at 
> org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:59)
>  
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>  
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
>  
> ~[graylog.jar:?]
>
> at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:170) 
> ~[graylog.jar:?]
>
> at okhttp3.RealCall.execute(RealCall.java:60) ~[graylog.jar:?]
>
> at retrofit2.OkHttpCall.execute(OkHttpCall.java:174) ~[graylog.jar:?]
>
> at 
> org.graylog2.shared.rest.resources.ProxiedResource.lambda$null$0(ProxiedResource.java:76)
>  
> ~[graylog.jar:?]
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_102]
>
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  
> [?:1.8.0_102]
>
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  
> [?:1.8.0_102]
>
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102]
>
> Caused by: java.net.SocketTimeoutException: Read timed out
>
> at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_102]
>
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) 
> ~[?:1.8.0_102]
>
> at java.net.SocketInputStream.read(SocketInputStream.java:170) 
> ~[?:1.8.0_102]
>
> at java.net.SocketInputStream.read(SocketInputStream.java:141) 
> ~[?:1.8.0_102]
>
> at okio.Okio$2.read(Okio.java:138) ~[graylog.jar:?]
>
> at okio.AsyncTimeout$2.read(AsyncTimeout.java:238) ~[graylog.jar:?]
>
> ... 29 more
>
> 2016-11-24T17:25:13.751+05:30 WARN [jvm] 
> [graylog-b8d88e71-dbbb-44ee-9951-2bcdb45b047a] [gc][old][19743][529] 
> duration [12.4s], collections [1]/[12.6s], total [12.4s]/[10.4m], memory 
> [3.7gb]->[3.7gb]/[3.8gb], all_pools {[young] 
> [1.6gb]->[1.6gb]/[1.6gb]}{[survivor] [188.7mb]->[191.2mb]/[204.7mb]}{[old] 
> [1.9gb]->[1.9gb]/[2gb]}
> 2016-11-24T17:25:13.821+05:30 INFO [BeatsCodec] inside 
> decodeMessagesRawMessage{id=52182873-b20e-11e6-9b98-14feb5ea6076, 
> journalOffset=960249, codec=beats, payloadSize=4169, 
> timestamp=2016-11-24T06:22:05.798Z, remoteAddress=/10.101.160.6:49324}
>
>
> Heap Size set for Graylog2 is 4G, I noticed the heap size jumps up from 
> 1.5G to 4G as soon as the exceptions are 

[graylog2] Re: [IndexerSetupService] Could not connect to Elasticsearch. [IndexerSetupService] If you're using multicast, check that it is working in your network and that Elasticsearch is accessible.

[Jochen Schalanda]

Hi Dylan,

it seems like you're overthinking this.

If you run Elasticsearch and Graylog on the same node, you don't need to 
configure anything except for the cluster name really.

Keep the defaults for everything except cluster.name in the Elasticsearch 
configuration and 
read 
http://docs.graylog.org/en/2.1/pages/configuration/elasticsearch.html#configuration
 
for more details.

Cheers,
Jochen

On Thursday, 24 November 2016 16:29:57 UTC+1, Dylan Humphreys wrote:
>
> Yes, its this one again. Sadly none of the other cases of this seem to fix 
> my problem, although I have an idea of where things are going wrong.
>
> bddvgl1:~ # dpkg -l elasticsearch graylog-server | grep ii | awk '{print 
> $1,$2,$3}' 
> ii elasticsearch 2.3.5 
> ii graylog-server 2.1.2-1 
> bddvgl1:~ # 
>
> My Elasticsearch cluster (currently one node to make things simple) is 
> hale and hearty:
>
> bddvgl1:~ # curl -s bddvgl1:9200/_cluster/health?pretty 
>  
> { 
>  "cluster_name" : "graylog", 
>  "status" : "green", 
>  "timed_out" : false, 
>  "number_of_nodes" : 1, 
>  "number_of_data_nodes" : 1, 
>  "active_primary_shards" : 0, 
>  "active_shards" : 0, 
>  "relocating_shards" : 0, 
>  "initializing_shards" : 0, 
>  "unassigned_shards" : 0, 
>  "delayed_unassigned_shards" : 0, 
>  "number_of_pending_tasks" : 0, 
>  "number_of_in_flight_fetch" : 0, 
>  "task_max_waiting_in_queue_millis" : 0, 
>  "active_shards_percent_as_number" : 100.0 
> } 
> bddvgl1:~ # 
>
>
> I think the "problem" is with (or with my understanding of) the 
> elasticsearch node graylog uses to communicate with the cluster. I 
> *believe* I have that binding to 127.0.0.1
> (from my server.conf)
>
> elasticsearch_cluster_name = graylog 
> elasticsearch_node_name_prefix = graylog-
> elasticsearch_transport_tcp_port = 9300
> elasticsearch_network_host = 127.0.0.1 
> elasticsearch_network_bind_host = 127.0.0.1
> elasticsearch_http_enabled = true
>
> Sure enough, when I start graylog, I get this:
>
> bddvgl1:~ # netstat -tulpn | grep -E "(9300|9200)" 
> tcp6   0  0 127.0.0.1:9200  :::* 
>LISTEN  15627/java   
> tcp6   0  0 10.0.34.128:9200:::* 
>LISTEN  15281/java   
> tcp6   0  0 127.0.0.1:9300  :::* 
>LISTEN  15627/java   
> tcp6   0  0 10.0.34.128:9300:::* 
>LISTEN  15281/java   
> bddvgl1:~ # 
>
>
> The 9200 & 9300 on the RFC1918 addresses are my elasticsearch cluster that 
> shows healthy above. 
> I enabled http on the graylog elasticsearch node to help diagnose my 
> problem. Sure enough:
>
> bddvgl1:~ # curl -s localhost:9200/_cluster/health?pretty 
> { 
>  "error" : { 
>"root_cause" : [ { 
>  "type" : "master_not_discovered_exception", 
>  "reason" : null 
>} ], 
>"type" : "master_not_discovered_exception", 
>"reason" : null 
>  }, 
>  "status" : 503 
> }
>
>
> Which is essentially what Im getting in the logs.
>
> I have tried this:
> elasticsearch_discovery_zen_ping_unicast_hosts = bddvgl1:9300
> and this 
> elasticsearch_discovery_zen_ping_unicast_hosts = 10.0.34.128:9300 
> and this 
> elasticsearch_discovery_zen_ping_unicast_hosts = 127.0.0.1:9300
>
> But it never seems to connect. In fact If I comment that out entirely, I 
> still get the same error mentioned in the subject.
>
> Primarily, where am I going wrong? also, does elasticsearch_blah refer to 
> the graylog elasticsearch instance, or (in the case 
> of elasticsearch_discovery_zen_ping_unicast_hosts) does that ask for the 
> ACTUAL data storing nodes of my elasticsearch cluster?
>
> Thanks in advance!
>
> Dylan
>
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/40f2b825-5759-45f0-a718-7817c3092bbe%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: HELP-ME Duplications mensages

[Jochen Schalanda]

Hi Anderson,

do all "duplicated" messages have the same timestamp and the same message 
ID or are they different?

Cheers,
Jochen

On Wednesday, 23 November 2016 16:29:56 UTC+1, Anderson Gabriel wrote:
>
> Hello, I have a graylog with elasticsearch configured on only 1 server. I 
> am capturing jboss logs. But my messages are replicating for more than 
> 20x each.
>
> Configuration logstash:
>
> input {
> filet {
> Type => "aeq-pro"
> Add_field => { "ugcs_server" => "172.29.1.114"}
> Add_field => { "ugcs_application" => "aeq-pro"}
> Add_field => { "ugcs_environment" => "production"}
> Add_field => { "ugcs_type" => "jboss-pro"}
> Path => "/nfs/prod/logs/jboss6/aeq/server.log"
> Codec => multiline {
> Pattern => "^% {TIMESTAMP_ISO8601}"
> Negate => true
> What => previous
>  }
> }
> }
>
> filter {
> mutate {
> Type => "aeq-pro"
> Add_tag => [ "aeq-pro"]
> Remove_tag => [ "multiline"]
> }
> }
>
> # Output
>
> output {
> gelf {
> Host => "172.29.1.181"
> Port = "" 12212 "
> Full_message => ""
> }
> }
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/951963c4-2af4-463f-abbe-816ca133a6f8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Splitting indices for different input

[Jochen Schalanda]

Hi Jonas,

this will be possible in Graylog 2.2.0.

Cheers,
Jochen

On Wednesday, 23 November 2016 10:40:44 UTC+1, Jonas K wrote:
>
>
> ##german version below ## 
>
> Hey guys, 
>
>   
> ##Sorry for my bad english## 
>
> it is possible write incoming messages in different indices? 
>
>   
>
> The problem is, that we have some Windows AD-Server, who write very much 
> traffic in the indices and we have some database-Server, who write not so 
> much message in the indices. 
>
> We want to store the db message longer in Graylog, but the indices close 
> to early because the AD-Server write too much in the indices. 
>
> It is possible that the AD-Server write in his own indices and the 
> Database-Server write in his own indices? 
>
> Or is this impossible? 
>
>   
>
> Many thanks and best regards 
>
>   
>
> Jonas 
>
> Hey, 
>
>   
>
> Wir haben hier ein kleines Problem. 
>
> Zurzeit nutzen wir Graylog für  3 AD-Server und ein paar Datenbankserver. 
>
> Wir möchten die Logs der Datenbandserver länger in Graylog behalten. Dies 
> geht aber nicht, da die AD-Server sehr viele Logs produzieren und die 
> Indizes laufen dadurch auch zu schnell voll und werden dann gelöscht. (mehr 
> Speicher ist zur Zeit leider nicht möglich) 
>
> Ist es möglich, dass die AD-Server in ihre eigenen Indizes schreiben und 
> die Datenbankserver  auch in ihre eigenen Indizes schreiben? 
>
> Als Beispiel: 
>
> Die beiden bekommen jeweils 5 Indices in der Standartgröße( also 2000 
> Nachrichten) und wenn von den AD-Servern die Indices voll sind, wird nur 
> der älteste Index gelöscht und die Indizes von den D-Servern bleiben 
> unangetastet. 
>
> Oder ist das nicht möglich? 
>
>   
>
> Besten Dank schon mal im Vorhinein 
>
>   
>
>   
>
> Jonas 
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/08c9db10-8016-4c75-9fe7-23502b0ba2dd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: TimeStamp

[Jochen Schalanda]

Hi,

you have to use some filter (e. g. a Grok extractor with the pattern 
%{COMMONAPACHELOG}) to extract the date from your web server access logs 
and substitute the timestamp field of the message.

Cheers,
Jochen

On Wednesday, 23 November 2016 07:15:30 UTC+1, suj...@bw.ae wrote:
>
> Hello,
>
> I have few servers. Graylog is fetching logs from those server. Graylog is 
> showing correct timestamp of the logs which is generated those server. But 
> there is difference of 9 OR 6 OR 4 seconds.
>
> In below example you can see the first timestamp is showing 6 seconds more 
> than the log
>
> 2016-11-23 09:53:09.000 server 
> server1 AccesLogs xx.xx.xx.xx. 5 - - [23/Nov/*2016:09:53:03* +0400] "GET 
> /sdfdsfdsf/sdfsdfdsf/sdfdsfsdfdsfds.js HTTP/1.1" 200 4343
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/5fac8e67-b9ca-4464-a57d-fbb847981e18%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog Web Interface is disconnected

[Jochen Schalanda]

Hi Rodion,

make sure that your Elasticsearch cluster is up and healthy. Check the logs 
of your ES nodes for further details.

Cheers,
Jochen

On Tuesday, 29 November 2016 09:41:27 UTC+1, Rodion Ovodnev wrote:
>
> Help me please! What's the problem, look at the log that correct?
>
> 1.> 
> 
> >
>
> 2.> 
> 
> >
>
> 3.> 
> 
> ?
>
>
>
> graylog/server.log: 
> 2016-11-29 15:48:11,503 ERROR: 
> org.graylog2.shared.rest.exceptionmappers.AnyExceptionClassMapper - 
> Unhandled exception in REST resource
> org.elasticsearch.ElasticsearchTimeoutException: Timeout waiting for task.
> at 
> org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:73)
> at 
> org.graylog2.indexer.elasticsearch.GlobalTimeoutClient$GlobalTimeoutActionFuture.actionGet(GlobalTimeoutClient.java:163)
> at org.graylog2.indexer.searches.Searches.count(Searches.java:190)
> at 
> org.graylog2.dashboards.widgets.SearchResultCountWidget.computeInternal(SearchResultCountWidget.java:73)
> at 
> org.graylog2.dashboards.widgets.SearchResultCountWidget.compute(SearchResultCountWidget.java:68)
> at 
> org.graylog2.dashboards.widgets.DashboardWidget$ComputationResultSupplier.get(DashboardWidget.java:155)
> at 
> org.graylog2.dashboards.widgets.DashboardWidget$ComputationResultSupplier.get(DashboardWidget.java:151)
> at 
> com.google.common.base.Suppliers$ExpiringMemoizingSupplier.get(Suppliers.java:192)
> at 
> org.graylog2.dashboards.widgets.DashboardWidget.getComputationResult(DashboardWidget.java:123)
> at 
> org.graylog2.rest.resources.dashboards.DashboardsResource.widgetValue(DashboardsResource.java:355)
> at sun.reflect.GeneratedMethodAccessor106.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at 
> org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
> at 
> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
> at 
> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
> at 
> org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205)
> at 
> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
> at 
> org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
> at 
> org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
> at 
> org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
> at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
> at 
> org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
> at 
> org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
> at 
> org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
> at 
> org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
> at 
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> at 
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at 
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
> at 
> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
> at 
> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
> at 
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at 
> org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:606)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

[graylog2] Re: Graylog V2.1.2 - getting GELF HTTP working with simple HTTP POST/JSON body

[Jochen Schalanda]

Hi Mike,

if you bind the GELF HTTP input to 127.0.0.1, only processes on the same 
machine (or running in the Docker container) can access it.

Try binding the GELF HTTP input to the public IP address of the Docker 
container or 0.0.0.0.

Cheers,
Jochen

On Monday, 5 December 2016 20:57:54 UTC+1, Mike Norris wrote:
>
> Hi
>
> I am having some troubles technically with a GELF HTTP input
>
> I cannot POST a simple sample JSON message to my GELF endpoint, *I keep 
> getting "Unable to connect to the remote server"*
>
> Graylog has been installed as a Docker runtime container(s) on my Windows 
> 10 Surface laptop
>
> Starting Graylog up is simple, the Docker process worked first time, so I 
> have the following
>
> *a. Graylog V2.1.2 running*
>
> *b. I can login to the console via htttp:127.0.0.1:9000 
>  as admin/admin*
>
> *c. I've created a basic GELF HTTP input in System ... Inputs*
>
> Type = GELF HTTP
> Bind address 127.0.0.1
> Port 12201
> removed any suggestion of TLS authentication, which is [optional] anyway
>
> This endpoint is running ok 
>
> So in theory I have a listening process on port 12201 waiting and ready 
> for GELF messages
>
> *d. I've tried POSTing simple requests as per the examples 
> in http://docs.graylog.org/en/2.1/pages/gelf.html 
> *
>
> *e. My tool of choice for the client is Windows .NET and I've tried both 
> PowerShell and C# programs*
>
> All I think I have to do is deliver a correct json payload in a sim ple 
> HHTP request to the right endpoint for it to work
>
> *Example - PowerShell code*
>
> Set-ExecutionPolicy Unrestricted
> Get-ExecutionPolicy
>
> $resource = "http://localhost:12201/gelf;
>
> $body = '"{version": "1.1","host": "example.org","short_message": "A 
> short message that helps you identify what is going on","full_message": 
> "Backtrace here\n\nmore stuff","timestamp": 1385053862.3072,"level": 
> 1,"_user_id": 9001,"_some_info":"foo","_some_env_var":"bar"}'
>
> Invoke-WebRequest -Uri "http://127.0.0.1:12201/gelf; -Method Post 
> -TransferEncoding "gzip" -ContentType "application/json; charset=UTF-8" 
> -Body $body
>
> I initailly wrote the PowerShell code without invoking gzip compression, 
> I've recently added that to no avail
>
> *f. I don't understand why the error I get is Unable to connect, as 
> Graylog is quite happy to serve pages on the same IP address on port 9000*
>
> The GELF INPUT is in a running state?
>
>
> Is this one of those examples where my input is wrong and the error 
> message is misleading?
>
>
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/b7452a2a-4bee-404c-b371-92494c4987d2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Sorting messages by more than one field in Graylog API

[Jochen Schalanda]

Hi Łukasz,

sorting by more than 1 field is currently not possible. Please create a 
feature request at https://github.com/Graylog2/graylog2-server/issues in 
which you describe your use cases if you think this is missing in Graylog.

Cheers,
Jochen

On Tuesday, 6 December 2016 15:25:16 UTC+1, Łukasz Tarasiewicz wrote:
>
> Hello. 
> Using Graylog extensively in a project, we found that it seems impossible 
> to sort messages returned by Graylog API by more than one field. The 
> default value of the "sort" key is to sort by "timestamp" in a descending 
> order. This can be overriden, and another key can be used for sorting 
> (fileld:asc / field:desc). However, we'd like to be able to sort by two 
> keys, e.g. "subject_name" and "log_created_at". Such sorting behaviour is 
> readily offered by Elastic, but Graylog API seems to impose this limitation 
> - that whenever we try to sort by more than one key, it defaults to its 
> standard sorting (by timestamp).
>
> To visualize better: http://prntscr.com/dg1mho
>
> The question I'd like to ask - is it possible to sort messages by more 
> than one field when interacting with Graylog through the API ?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/d0072d62-ad7d-4493-ab64-64ef3e54a565%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] My experience with collector-sidecar in Windows

[Michael Taylor]

I got collector-sidecar to work on my windows systems and here are some 
tips. I'm using the latest graylog in Docker.


   - Update your graylog configuration to use GRAYLOG_WEB_ENDPOINT_URI if 
   you haven't already, since this URI can now be under the same port as the 
   web interface.
   - Beats by default uses TCP, make sure you're forwarding the right 
   protocol for port 5044.
   - Install the right architecture of the collector-sidecar. If you get it 
   wrong it will be looking for config files in the wrong places.
   - Collector-sidecar needs permission to write into the program files 
   directories or you'll get errors about the content ID. Running it manually 
   will usually result in UAC blocking the changes. Either run it as an 
   administrator or install it as a service and give its service user write 
   permission on its program files directory.
   - No changes should be necessary to the configuration if you entered the 
   right information during installation. If collector-sidecar doesn't 
   connect, troubleshoot permissions and connections to the graylog URI.
   - Even if the configuration fails, the collector will show up in the 
   list of collectors in graylog if the connection is made.
   - Both winlogbeats and filebeats need some kind of configuration or the 
   collector will show failed. You need two inputs and two outputs, one each 
   for filebeats and one each for winlogbeats.
   - In the output configuration, make sure you get the syntax right for 
   the host to connect to. You're making a JSON array of strings, so you need 
   the brackets and single quotes around the IP/hostname:port.
   - When winlogbeats first connects, it will dump the entire event log to 
   graylog, which could take a long time and take a lot of bandwidth.


If there's a way to only get the tail of the logs I haven't found it yet. I 
also haven't yet figured out how configuration precedence works if two tags 
have different configurations.


-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/c0b5053c-274c-40ad-9aad-79e93e5097bf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Sorting messages by more than one field in Graylog API

[Łukasz Tarasiewicz]

Hello. 
Using Graylog extensively in a project, we found that it seems impossible 
to sort messages returned by Graylog API by more than one field. The 
default value of the "sort" key is to sort by "timestamp" in a descending 
order. This can be overriden, and another key can be used for sorting 
(fileld:asc / field:desc). However, we'd like to be able to sort by two 
keys, e.g. "subject_name" and "log_created_at". Such sorting behaviour is 
readily offered by Elastic, but Graylog API seems to impose this limitation 
- that whenever we try to sort by more than one key, it defaults to its 
standard sorting (by timestamp).

To visualize better: http://prntscr.com/dg1mho

The question I'd like to ask - is it possible to sort messages by more than 
one field when interacting with Graylog through the API ?

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/af93cadd-9b95-4b50-8c68-21165a6af40a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Disk Journal / Kafka Input / Throttling

[cazy]

Hi Jochen,

I think what Eli is driving at is that throttling starts too late. I think 
there should be a (configurable) threshold that starts throttling when the 
internal journal exceeds a certail level.

Cheers,
Carlos

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/c21073fd-e9be-40a7-8bfb-51bf6de47533%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.