[graylog2] Did not find meta info of this node. Re-registering.

2016-10-19 Thread Jimmy Chen
We have recently started seeing this in our graylog collector server. I've 
searched through threads of others reporting this issue but none of which 
seems to apply. Namely, NTP service is running for all nodes and synced to 
the same local server. I also bumped up the resources for the collector 
node to 12 cores and 4 GB of ram but it is still happening. This is 
severely impacting our logging as when this message appears, it seems like 
the collector node drops out and stops processing any messages momentarily.

2016-10-19 12:52:21,315 WARN : org.graylog2.periodical.NodePingThread - Did 
not find meta info of this node. Re-registering.
2016-10-19 12:52:25,109 WARN : org.graylog2.periodical.NodePingThread - Did 
not find meta info of this node. Re-registering.
2016-10-19 12:52:28,647 WARN : org.graylog2.periodical.NodePingThread - Did 
not find meta info of this node. Re-registering.
2016-10-19 12:52:32,009 WARN : org.graylog2.periodical.NodePingThread - Did 
not find meta info of this node. Re-registering.
2016-10-19 12:52:35,562 WARN : org.graylog2.periodical.NodePingThread - Did 
not find meta info of this node. Re-registering.

Any suggestions on how to fix this?

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/cb9e77d1-098b-477e-affd-d89259f2fe20%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


[graylog2] Re: Shortened Xml in message field

2016-10-19 Thread 4BRobby
Hello Jochen,
I have nospecial extractors running. Maybe I can send you the real (not 
anonymized) XML on a private way? To me it seems like it is a special 
sequence in that XML that is causing this. I will try to reproduce it first 
by myself.
The message that shows the shortened XML is still there.
I found this out because I am now tracking sizes of all messages sent to 
Graylog to see big sized messages that should be shortened before sending 
them. I investigated further and this one was a very large message but in 
my framework internally shortened to 64kb (the full_message field) and ".." 
appended. So the XML got cut. Nevertheless the message in graylog is 
showing closing tags that were never sent and a full_message field of 
~200byte. Very weired because I do no XML processing in the logging 
framework as well just plain strings.
Thanks
Robby

Am Mittwoch, 19. Oktober 2016 12:41:16 UTC+2 schrieb Jochen Schalanda:
>
> Hi Robby,
>
> I cannot reproduce your problem.
>
> I've created a GELF UDP input on an installation of Graylog 2.1.1 and sent 
> a GELF message to it with the XML example from your first email in the 
> full_text message field.
>
> Everything has been ingested and is being shown as intended (i. e. 
> unparsed and the verbatim XML). Maybe some extractor is interfering with 
> your payload or even the client already sends the modified GELF message.
>
> Graylog and Elasticsearch itself do not parse XML from message fields by 
> default.
>
> For reproduction, here's what I did:
>
> Create GELF UDP input with the following configuration and no extractors:
>
> bind_address: 127.0.0.1
> decompress_size_limit: 8388608
> override_source: 
> port: 12201
> recv_buffer_size: 16384
>
>
> Send message to Graylog:
>
>
> $ echo '{"version":"1.1", "message":"XML TEST", "full_message":" version=\"1.0\" encoding=\"utf-8\"?>\n   \n 
>  Success\n  DoSomething\n 
>  8\n \n 0\n \n 
> 
>
> Message in Graylog:
>
> $ curl -i -u admin:admin 
> http://127.0.0.1:9000/api/messages/graylog2_27/43c3aca0-95e7-11e6-8fc7-6c4008b8fc28?pretty=true
> HTTP/1.1 200 OK
> X-Graylog-Node-ID: cd03ee44-b2a7-4824-be16-bb7456149dbd
> X-Runtime-Microseconds: 1887
> Content-Type: application/json
> Date: Wed, 19 Oct 2016 10:38:40 GMT
> Content-Length: 1975
>
> {
>   "message" : {
> "fields" : {
>   "full_message" : "\n   
> \n  Success\n  DoSomething\n   
>8\n \n 0\n \n   
>      "streams" : [ "552b92b2e4b0c055e41ffb8e", "0001" 
> ],
>   "gl2_source_node" : "cd03ee44-b2a7-4824-be16-bb7456149dbd",
>   "_id" : "43c3aca0-95e7-11e6-8fc7-6c4008b8fc28",
>   "source" : "unknown",
>   "message" : "XML TEST",
>   "gl2_source_input" : "5592913ec026f261b8d5ce9a",
>   "timestamp" : "2016-10-19T10:31:58.824Z"
> },
> "streams" : [ ],
> "source_input_id" : null,
> "filter_out" : false,
> "journal_offset" : -9223372036854775808,
> "message" : "XML TEST",
> "id" : "43c3aca0-95e7-11e6-8fc7-6c4008b8fc28",
> "source" : "unknown",
> "field_names" : [ "full_message", "streams", "gl2_source_node", "_id", 
> "source", "message", "gl2_source_input", "timestamp" ],
> "complete" : true,
> "timestamp" : "2016-10-19T10:31:58.824Z",
> "validation_errors" : "",
> "fields_entries" : [ {
>   "full_message" : "\n   
> \n  Success\n  DoSomething\n   
>8\n \n 0\n \n   
>    }, {
>   "streams" : [ "552b92b2e4b0c055e41ffb8e", "0001" 
> ]
> }, {
>   "gl2_source_node" : "cd03ee44-b2a7-4824-be16-bb7456149dbd"
> }, {
>   "_id" : "43c3aca0-95e7-11e6-8fc7-6c4008b8fc28"
> }, {
>   "source" : "unknown"
> }, {
>   "message" : "XML TEST"
> }, {
>   "gl2_source_input" : "5592913ec026f261b8d5ce9a"
> }, {
>   "timestamp" : "2016-10-19T10:31:58.824Z"
> } ],
> "field_count" : 8,
> "stream_ids" : [ "552b92b2e4b0c055e41ffb8e", 
> "0001" ],
> "is_source_inet_address" : false,
> "inet_address" : null
>   },
>   "index" : "graylog2_27"
> }
>
>
> Screenshot:
>
>
> 
>
>
> Cheers,
> Jochen
>
> On Tuesday, 18 October 2016 14:10:11 UTC+2, 4BRobby wrote:
>>
>> Hell Jochen,
>>
>> Enclosed my configuration if this helps. It is the OVA appliance, latest 
>> graylog version.
>>
>> A new log worked b.t.w. so it may be the problem that the .. at one time 
>> breaks the xml tag itself. 
>> Nevertheless it is an unexpected behaviour that the xml even gets parsed 
>> out of the message. XML is of course base64 coded like all other log 
>> message bodies (full_message) that are send to Graylog.
>>
>> I have found no information about this behaviour so far.
>>
>> I had to anonymize 

[graylog2] Does Graylog server save a copy of the original log messages before indexing the message

2016-10-19 Thread Wayne
Hi All,

Let's say we send a query and search a couple of records, now we would like 
to retrieve the original text message. Does Graylog keep the original copy 
of the log message?

In addition, the disk based journal seems to keep some data, but not 
completely visible. Are those the copy of the messages?

Thanks,

Wayne



-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/f43cf707-2941-4f10-bf31-ca077f7da68b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


[graylog2] Re: Some fields generated from Extractor are not searchable

2016-10-19 Thread Wayne
Hi Jochen,

What is strange about it is that the "Stream" rules apparently work with 
the field "log_message", but a search query does not work. 

I send a raw Elastic query and still not much information about why it is 
not working.

The custom mapping is useful if the data type is not the default string 
type. However, the log_message field is still string type. So it may not 
make much difference if I set up custom mapping for this field?

Thanks,

Wayne


On Wednesday, October 19, 2016 at 12:22:32 PM UTC-4, Jochen Schalanda wrote:
>
> Hi Wayne,
>
> On Wednesday, 19 October 2016 17:36:10 UTC+2, Wayne wrote:
>>
>> Is there additional configuration that is required to ensure all the 
>> extracted fields to be searchable?
>>
>
> See 
> http://docs.graylog.org/en/2.1/pages/configuration/elasticsearch.html#custom-index-mappings
>  
> for details.
>
> Cheers,
> Jochen 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/8876b4c5-a05a-4f58-8810-164570aec67d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


[graylog2] Re: Some fields generated from Extractor are not searchable

2016-10-19 Thread Jochen Schalanda
Hi Wayne,

On Wednesday, 19 October 2016 17:36:10 UTC+2, Wayne wrote:
>
> Is there additional configuration that is required to ensure all the 
> extracted fields to be searchable?
>

See 
http://docs.graylog.org/en/2.1/pages/configuration/elasticsearch.html#custom-index-mappings
 
for details.

Cheers,
Jochen 

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/e272e3fd-1aeb-440d-b7ae-3a49f886d88c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


[graylog2] Some fields generated from Extractor are not searchable

2016-10-19 Thread Wayne
Hi All,

I configured a couple of Extractors to extract fields from the log message. 
Some fields can be searched, but others can not be searched.


Example:

I have a field called "level" (log level) and it can be searched. I can 
also see this field listed as a property in search index 
logstash-.MM.dd. I have another field called "log_message" and it is 
not searchable. When I checked the mapping, it is not listed as a property 
in logstash-.MM.dd. 

When I check the mapping in search index graylog-x, both are listed.

The failed search example:

If I use message field, I can search a record with a string Exception in 
the message within 2 hours of time frame, but if I use log_message field 
(remove the timestamp part and contains the string Exception), I can not 
search the record although the string is in the log_message field.

Is there additional configuration that is required to ensure all the 
extracted fields to be searchable?

Thanks,

Wayne


Note:

I access url to check the fields and mapping in each search index:

http://localhost:9200/_mappings

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/7ffe7dcd-9a0d-4ee5-a099-9d7d40f20f7b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


[graylog2] Re: Redirect logs from /varlog/messages to a Graylog server

2016-10-19 Thread Jochen Schalanda
Hi Benbrahim,

see https://github.com/Graylog2/graylog-guide-syslog-linux#readme for 
instructions how to configure rsyslog or syslog-ng to forward logs to 
Graylog.

Cheers,
Jochen

On Wednesday, 19 October 2016 15:30:06 UTC+2, Benbrahim Anass wrote:
>
> Hello
> i'm wondering if it is possible to redirect all /var/log/message of a 
> syslog server to a distant graylog server
> Thanks alot
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/b3c57ce6-2139-4566-8de4-c573c9a401ba%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


[graylog2] Redirect logs from /varlog/messages to a Graylog server

2016-10-19 Thread Benbrahim Anass
Hello
i'm wondering if it is possible to redirect all /var/log/message of a 
syslog server to a distant graylog server
Thanks alot

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/6ff6a9f5-752c-4f99-9ca9-5762c85ff315%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


[graylog2] Re: Is it possible to delete logs in Graylog 2.0 based on source?

2016-10-19 Thread Jochen Schalanda
Hi Anant,

On Wednesday, 19 October 2016 14:43:39 UTC+2, Anant Sawant wrote:
>
> I would like delete logs of system A for a particular date.
>
> So is it possible to know in which indices the logs from System A are 
> stored and remove those particular indices/logs on a particular date?
>

That's not possible with Graylog directly but you can remove these messages 
from Elasticsearch using the delete-by-query 
API: 
https://www.elastic.co/guide/en/elasticsearch/plugins/2.4/plugins-delete-by-query.html

Cheers,
Jochen

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/e5866af7-ae7b-473f-87e3-b4a80e252193%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


[graylog2] Re: Internal message queue for graylog2?

2016-10-19 Thread Jochen Schalanda
Hi Wayne,

On Wednesday, 19 October 2016 15:07:07 UTC+2, Wayne wrote:
>
> It is stated in 2.1 document that Kafka and RabbitMQ can be configured as 
> transport queue.
>
> What are the use cases/scenarios which we need to do the above 
> configuration considering Graylog already has its own way to persist the 
> messages?
>

It can be useful for connecting offsite locations with bad network 
connection or if log messages aren't exclusively consumed by Graylog.

If you can't come up with a use case for using a message broker like 
RabbitMQ or Apache Kafka, it's probably not necessary for you…

Cheers,
Jochen

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/7621ac60-5354-4edd-8d84-7d422f8be61d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


[graylog2] Re: Graylog is restarting...

2016-10-19 Thread Jochen Schalanda
Hi Shane,

On Wednesday, 19 October 2016 14:04:32 UTC+2, Shane wrote:
>
> We have seen this server have issues in the past - typically due to the 
> Graylog not handling having more than one eth interface well at all.
>
>
So, what did you do in the past to solve this issue?

By the way, Graylog doesn't have a problem with multiple network 
interfaces. The configuration script for the OVA (graylog-ctl) might have, 
but I'm not sure about that…

Cheers,
Jochen

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/9420ba0e-9ab5-459a-9bba-7e8e5c84dc02%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


[graylog2] Re: Graylog is restarting...

2016-10-19 Thread Jochen Schalanda
Hi Shane,

On Wednesday, 19 October 2016 14:04:32 UTC+2, Shane wrote:
>
> The server was updated using: 
>
> $ wget 
> https://packages.graylog2.org/releases/graylog-omnibus/ubuntu/graylog_latest.deb
> $ sudo graylog-ctl stop
> $ sudo dpkg -G -i graylog_latest.deb
> $ sudo graylog-ctl reconfigure
>
>
All of these commands produce some output which might contain warning or 
error messages.

Cheers,
Jochen

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/f07e63a7-ac69-4073-b259-f1e40e49c11f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


[graylog2] Re: Internal message queue for graylog2?

2016-10-19 Thread Wayne
Hi Jochen,

It is stated in 2.1 document that Kafka and RabbitMQ can be configured as 
transport queue.

What are the use cases/scenarios which we need to do the above 
configuration considering Graylog already has its own way to persist the 
messages? 

Thanks,

Wayne


On Wednesday, October 19, 2016 at 6:48:05 AM UTC-4, Jochen Schalanda wrote:
>
> Hi Wayne,
>
> Graylog writes messages into a disk journal once they have been received 
> and will only remove them from the journal again, if they've been 
> successfully been indexed into Elasticsearch.
>
> Cheers,
> Jochen
>
> On Tuesday, 18 October 2016 18:41:50 UTC+2, Wayne wrote:
>>
>> Hi All,
>>
>> I would like to understand how Graylog is trying to send message without 
>> additional configuration with kafka or RabbitMQ. 
>>
>> I am currently using Graylog collector sidebar to configure filebeat to 
>> send the tail of application log messages to Graylog server, and I am not 
>> sure if there is any internal message queue to hold messages in case of 
>> high load.
>>
>>
>> Thanks,
>>
>> Wayne
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/4c84b271-98da-4e86-ba16-63c16f196a26%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


[graylog2] Is it possible to delete logs in Graylog 2.0 based on source?

2016-10-19 Thread Anant Sawant
Hi Graylog Team


We have a single instance of graylog running with single node.
We have pointed two sensors (two systems) via syslog udp to graylog. When i 
do a search in all messages it shows 42 million events searched in three 
indices namely

graylog2_0
graylog2_1
graylog2_2

Now my query is that as mentioned we have two system from where logs are 
being pushed to graylog. System A and System B.

I would like delete logs of system A for a particular date.

So is it possible to know in which indices the logs from System A are 
stored and remove those particular indices/logs on a particular date?

Thanks in advance


-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/40ca6243-5107-4add-857f-fa82e2e0e035%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


[graylog2] Re: Graylog is restarting...

2016-10-19 Thread Shane
It's difficult to provide more details when there are no errors at all in 
any of the logs, with the only indication of the issue being the text 

"Graylog is restarting...

There is no Graylog web application running at the moment, please reload 
this page in a minute. It can take up to 1-2 minutes until all services are 
running properly. In case this is a permanent error, check the following:"

when attempting to load the UI.

The server was updated using: 

$ wget 
https://packages.graylog2.org/releases/graylog-omnibus/ubuntu/graylog_latest.deb
$ sudo graylog-ctl stop
$ sudo dpkg -G -i graylog_latest.deb
$ sudo graylog-ctl reconfigure

We have seen this server have issues in the past - typically due to the Graylog 
not handling having more than one eth interface well at all.




On Wednesday, 19 October 2016 12:46:45 UTC+1, Jochen Schalanda wrote:
>
> Hi Shane,
>
> On Wednesday, 19 October 2016 13:43:23 UTC+2, Shane wrote:
>>
>> Same thing for me on the 2.1.1 after attempting an upgrade from 2.1.0. No 
>> logs of any value and just a message "Graylog is restarting..." - 
>> struggling to see the value in this product. `gralog-ctl tail` also of no 
>> use.
>>
>
> Without any more details, such as the exact steps you did to upgrade your 
> Graylog installation, we can't help you. Maybe the logs do have some value 
> if you're looking for the right things.
>
> Restarting the VM might also help.
>
> Cheers,
> Jochen 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/9810f47b-7de1-41b7-a0c1-69d0c4f1949f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


[graylog2] Re: Graylog is restarting...

2016-10-19 Thread Jochen Schalanda
Hi Shane,

On Wednesday, 19 October 2016 13:43:23 UTC+2, Shane wrote:
>
> Same thing for me on the 2.1.1 after attempting an upgrade from 2.1.0. No 
> logs of any value and just a message "Graylog is restarting..." - 
> struggling to see the value in this product. `gralog-ctl tail` also of no 
> use.
>

Without any more details, such as the exact steps you did to upgrade your 
Graylog installation, we can't help you. Maybe the logs do have some value 
if you're looking for the right things.

Restarting the VM might also help.

Cheers,
Jochen 

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/ad52292b-da3d-4beb-9d71-514827c22e21%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


[graylog2] Re: Graylog is restarting...

2016-10-19 Thread Shane
Same thing for me on the 2.1.1 after attempting an upgrade from 2.1.0. No 
logs of any value and just a message "Graylog is restarting..." - 
struggling to see the value in this product. `gralog-ctl tail` also of no 
use.


On Thursday, 28 July 2016 09:15:17 UTC+1, Jochen Schalanda wrote:
>
> Hi Lino,
>
> please check the logs of the Graylog process in /var/log/graylog/* for 
> error messages.
>
> Cheers,
> Jochen
>
> On Wednesday, 27 July 2016 23:37:13 UTC+2, Lino Edgar wrote:
>>
>> Hi Community
>>
>>
>> Greetings
>>
>>
>> Excuse me, after install Graylog2 is not able to display the webpage, I 
>> have the next messaje "Graylog is restarting..."
>>
>>
>> I have checked the status services and it seems that is correct
>>
>>
>> ubuntu@graylog:~$ sudo graylog-ctl status
>> run: elasticsearch: (pid 906) 1327s; run: log: (pid 861) 1328s
>> run: etcd: (pid 907) 1327s; run: log: (pid 904) 1327s
>> run: graylog-server: (pid 1773) 294s; run: log: (pid 853) 1328s
>> run: mongodb: (pid 905) 1327s; run: log: (pid 862) 1328s
>> run: nginx: (pid 903) 1328s; run: log: (pid 854) 1328s
>>
>> Could you indicate me if is necessary to check any else?
>>
>>
>>
>> I have installed the OVA appliance
>>
>>
>> Any idea?
>>
>>
>> Thanks in advance
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/b602bc53-b992-43db-baeb-d72d28f08369%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


[graylog2] Re: Internal message queue for graylog2?

2016-10-19 Thread Jochen Schalanda
Hi Wayne,

Graylog writes messages into a disk journal once they have been received 
and will only remove them from the journal again, if they've been 
successfully been indexed into Elasticsearch.

Cheers,
Jochen

On Tuesday, 18 October 2016 18:41:50 UTC+2, Wayne wrote:
>
> Hi All,
>
> I would like to understand how Graylog is trying to send message without 
> additional configuration with kafka or RabbitMQ. 
>
> I am currently using Graylog collector sidebar to configure filebeat to 
> send the tail of application log messages to Graylog server, and I am not 
> sure if there is any internal message queue to hold messages in case of 
> high load.
>
>
> Thanks,
>
> Wayne
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/f8f432f2-f61e-40c8-88be-0775faf0f665%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


[graylog2] Re: Shortened Xml in message field

2016-10-19 Thread Jochen Schalanda
Hi Robby,

I cannot reproduce your problem.

I've created a GELF UDP input on an installation of Graylog 2.1.1 and sent 
a GELF message to it with the XML example from your first email in the 
full_text message field.

Everything has been ingested and is being shown as intended (i. e. unparsed 
and the verbatim XML). Maybe some extractor is interfering with your 
payload or even the client already sends the modified GELF message.

Graylog and Elasticsearch itself do not parse XML from message fields by 
default.

For reproduction, here's what I did:

Create GELF UDP input with the following configuration and no extractors:

bind_address: 127.0.0.1
decompress_size_limit: 8388608
override_source: 
port: 12201
recv_buffer_size: 16384


Send message to Graylog:


$ echo '{"version":"1.1", "message":"XML TEST", "full_message":"\n   \n 
 Success\n  DoSomething\n 
 8\n \n 0\n \n 



Cheers,
Jochen

On Tuesday, 18 October 2016 14:10:11 UTC+2, 4BRobby wrote:
>
> Hell Jochen,
>
> Enclosed my configuration if this helps. It is the OVA appliance, latest 
> graylog version.
>
> A new log worked b.t.w. so it may be the problem that the .. at one time 
> breaks the xml tag itself. 
> Nevertheless it is an unexpected behaviour that the xml even gets parsed 
> out of the message. XML is of course base64 coded like all other log 
> message bodies (full_message) that are send to Graylog.
>
> I have found no information about this behaviour so far.
>
> I had to anonymize the message (That is why it reads nested). One tag is 
> called result - but only numbers and usual text in the tag values, Nothing 
> special really.
>
> Thanks
> Robby
>
>
> Am Dienstag, 18. Oktober 2016 12:45:50 UTC+2 schrieb Jochen Schalanda:
>>
>> Hi,
>>
>> On Tuesday, 18 October 2016 12:27:45 UTC+2, 4BRobby wrote:
>>>
>>> Configuraiton: There is no transformation for the message processing, 
>>> all to default.
>>>
>>
>> That's not what I've asked for. Please provide the requested information, 
>> otherwise we might not be able to help you.
>>
>> Cheers,
>> Jochen 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/cda4020d-0595-4c34-bd1e-d597366ba6c6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


[graylog2] Re: geolocation

2016-10-19 Thread Jochen Schalanda
Hi,

On Wednesday, 19 October 2016 10:17:59 UTC+2, mani...@qrsolutions.in wrote:
>
> And now only I notice that Pfsense log doesn't contain any (log/lat) geo 
> values. It is possible to create map without geo values??
>

No, that's not possible and that's exactly what the GeoIP Processor is for.

Please read http://docs.graylog.org/en/2.1/pages/geolocation.html for 
details and setup instructions.

Cheers,
Jochen

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/0a6c6df5-0285-4644-a272-83ae17332223%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


[graylog2] Re: error on sudo graylog-ctl reconfigure after upgrade

2016-10-19 Thread Nathaniel Brassington
Just an update. I restored the server from backup and reapplied the update 
without doing a sudo apt-get update and upgrade. The process failed at the 
same point. Service still down.

On Tuesday, 18 October 2016 11:47:43 UTC+1, Jochen Schalanda wrote:
>
> Hi Nathaniel,
>
> what exactly (step-by-step) did you do to upgrade Graylog?
>
> FWIW, port 4001 is etcd and it seems like it didn't start properly.
>
> Cheers,
> Jochen
>
> On Tuesday, 18 October 2016 12:33:02 UTC+2, Nathaniel Brassington wrote:
>>
>> sudo graylog-ctl reconfigure
>> Starting Chef Client, version 12.6.0
>> Compiling Cookbooks...
>> Recipe: graylog::default
>>   * directory[/etc/graylog] action create (up to date)
>>   Converging 103 resources
>> Recipe: apt::default
>>   * file[/var/lib/apt/periodic/update-success-stamp] action nothing 
>> (skipped 
>> due  
>> 
>> to action :nothing)
>>   * directory[/etc/apt/apt.conf.d] action create (up to date)
>>   * cookbook_file[/etc/apt/apt.conf.d/15update-stamp] action create (up 
>> to date)
>>   * execute[apt-get update] action nothing (skipped due to action 
>> :nothing)
>>   * execute[apt-get autoremove] action nothing (skipped due to action 
>> :nothing)
>>   * execute[apt-get autoclean] action nothing (skipped due to action 
>> :nothing)
>>   * execute[apt-get-update-periodic] action run (skipped due to not_if)
>>   * directory[/var/cache/local] action create (up to date)
>>   * directory[/var/cache/local/preseeding] action create (up to date)
>>   * template[/etc/apt/apt.conf.d/10recommends] action create (up to date)
>>   * apt_package[apt-transport-https] action install (up to date)
>> Recipe: graylog::default
>>   * directory[/etc/graylog] action nothing (skipped due to action 
>> :nothing)
>> Recipe: graylog::users
>>   * group[graylog] action create (up to date)
>>   * user[graylog] action create (up to date)
>>   * directory[/var/opt/graylog] action create (up to date)
>>   * template[/var/opt/graylog/.gitconfig] action create (up to date)
>>   * directory[/var/opt/graylog/data] action create (up to date)
>> Recipe: graylog::authbind
>>   * directory[/etc/authbind/byport] action create (up to date)
>>   * file[/etc/authbind/byport/!514] action create (up to date)
>> Recipe: runit::upstart
>>   * cookbook_file[/etc/init/graylog-runsvdir.conf] action create (up to 
>> date)
>>   * execute[initctl status graylog-runsvdir] action run
>> - execute initctl status graylog-runsvdir
>>   * execute[initctl start graylog-runsvdir] action run (skipped due to 
>> only_if)
>> Recipe: runit::svloggelfd
>>   * 
>> cookbook_file[/opt/graylog/embedded/cookbooks/cache/svloggelfd-0.2.1.tar.gz] 
>>  
>> 
>> action create (up to date)
>>   * execute[extract svloggelfd] action run (skipped due to not_if)
>> Recipe: timezone-ii::default
>>   * apt_package[tzdata] action install (up to date)
>> Recipe: timezone-ii::debian
>>   * template[/etc/timezone] action create
>> - update content in file /etc/timezone from 188731 to a60f4b
>> --- /etc/timezone   2016-10-18 11:29:35.619822451 +0100
>> +++ /etc/.timezone20161018-3852-1fe92to 2016-10-18 
>> 11:30:34.988793520 
>> +0   
>>   
>> 100
>> @@ -1,2 +1,2 @@
>> -Europe/London
>> +GB
>>   * bash[dpkg-reconfigure tzdata] action nothing (skipped due to action 
>> :nothing 
>> 
>> )
>> Recipe: graylog::etcd
>>   * directory[/var/log/graylog/etcd] action create (up to date)
>>   * directory[/var/opt/graylog/data/etcd] action create (up to date)
>>   * directory[/opt/graylog/sv/etcd] action create (up to date)
>>   * directory[/opt/graylog/sv/etcd/log] action create (up to date)
>>   * directory[/opt/graylog/sv/etcd/log/main] action create (up to date)
>>   * template[/opt/graylog/sv/etcd/run] action create (up to date)
>>   * template[/opt/graylog/sv/etcd/log/run] action create (up to date)
>>   * template[/var/log/graylog/etcd/config] action create (up to date)
>>   * ruby_block[reload etcd svlogd configuration] action nothing (skipped 
>> due 
>> to   
>>
>> action :nothing)
>>   * file[/opt/graylog/sv/etcd/down] action delete (up to date)
>>   * link[/opt/graylog/init/etcd] action create (up to date)
>>   * link[/opt/graylog/service/etcd] action create (up to date)
>>   * ruby_block[supervise_etcd_sleep] action run (skipped due to not_if)
>>   * 

[graylog2] Re: Graylog Training Courses

2016-10-19 Thread 'Stefan Krüger' via Graylog Users
any news on this?
 

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/a89a9381-8783-4e47-89ac-7c1df71d34ab%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


[graylog2] Re: geolocation

2016-10-19 Thread manimaran

Hi,

Please find the details below.

And now only I notice that Pfsense log doesn't contain any (log/lat) geo 
values. It is possible to create map without geo values??




 


On Wednesday, October 19, 2016 at 9:44:26 AM UTC+5:30, Aykisn wrote:
>
> Did you do every steps of the documentation about this ?
> Is there a pfsense_filter_sourceip_geolocation field in your list of 
> fields ?
>

-- 


**Disclaimer**

"This email and any attachments are confidential and are for the intended 
addressee[s] only. Unauthorised use of this communication is prohibited. If 
you have received this communication in error, please notify the sender and 
remove them from your system. Confidentiality is not waived or lost by 
reason of the mistaken delivery to you. Please scan this email and any 
attachment(s) for viruses. It is your responsibility to check them before 
opening"

End of Disclaimer*

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/3b191c7b-d488-42bf-a0fc-f1115fc93b7d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


[graylog2] Re: Forward an Existing Log file to Graylog ( Syslog Server ==> graylog )

2016-10-19 Thread Benbrahim Anass
Thanks Jochen For the Reply,
what i meant is that i already have a syslog server gathering logs from 
differents equipements in my newtrok ( routers, switches, other servers..) 
the idea is to forward those logs to the graylog without the need of 
reconfiguring everything to work with graylog

Le mercredi 19 octobre 2016 08:58:02 UTC+2, Jochen Schalanda a écrit :
>
> Hi Benbrahim,
>
> most syslog daemons support sending logs to remote locations, so take a 
> look at https://github.com/Graylog2/graylog-guide-syslog-linux#readme for 
> information how to configure rsyslog and syslog-ng to work with Graylog.
>
> If you have only some log files and want to send them to Graylog, you can 
> use log shippers such as nxlog or Filebeat to accomplish this: 
> http://docs.graylog.org/en/2.1/pages/collector_sidecar.html
>
> Cheers,
> Jochen
>
> On Wednesday, 19 October 2016 08:54:36 UTC+2, Benbrahim Anass wrote:
>>
>> Hi everyone,
>> i have a question, well, i have a systlog server already configured, i'm 
>> wondering if it is possible to forward the existing log file on the server 
>> toward the Graylog server .
>> thanks
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/18449deb-9ad0-455c-922f-057408a759f6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


[graylog2] Re: Forward an Existing Log file to Graylog ( Syslog Server ==> graylog )

2016-10-19 Thread Benbrahim Anass
Thanks Jochen For the Reply,
what i meant is that i already have a syslog server gathering trafic from 
differents equipements in my newtrok ( routers, switches, other servers..) 
the idea is to forward that trafic to the graylog without the need of 
reconfiguring everything to work with graylog

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/ede97a5b-5f9c-4b62-9446-df9acb095d1d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


[graylog2] Re: Forward an Existing Log file to Graylog ( Syslog Server ==> graylog )

2016-10-19 Thread Benbrahim Anass


Le mercredi 19 octobre 2016 08:54:36 UTC+2, Benbrahim Anass a écrit :
>
> Hi everyone,
> i have a question, well, i have a systlog server already configured, i'm 
> wondering if it is possible to forward the existing log file on the server 
> toward the Graylog server .
> thanks
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/935f2e18-7a7f-4838-940d-0c8cb3502299%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


[graylog2] Re: graylog2 timestamp not from application log message

2016-10-19 Thread Jochen Schalanda
Hi Wayne,

On Tuesday, 18 October 2016 20:01:11 UTC+2, Wayne wrote:
>
> The problem is that when an alert email is sent, the Date is showing UTC 
> time.
>

Yes, that's intentional. The alert emails aren't linked to any Graylog 
user, so it's not possible to use the configured timezone of any Graylog 
user to transform the timestamp of messages in these emails.

 

> Is it something that will be fixed later?
>

That's rather unlikely.

Cheers,
Jochen

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/7c315df8-3c0f-42bb-9c45-468d432b5788%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


[graylog2] Re: Forward an Existing Log file to Graylog ( Syslog Server ==> graylog )

2016-10-19 Thread Jochen Schalanda
Hi Benbrahim,

most syslog daemons support sending logs to remote locations, so take a 
look at https://github.com/Graylog2/graylog-guide-syslog-linux#readme for 
information how to configure rsyslog and syslog-ng to work with Graylog.

If you have only some log files and want to send them to Graylog, you can 
use log shippers such as nxlog or Filebeat to accomplish 
this: http://docs.graylog.org/en/2.1/pages/collector_sidecar.html

Cheers,
Jochen

On Wednesday, 19 October 2016 08:54:36 UTC+2, Benbrahim Anass wrote:
>
> Hi everyone,
> i have a question, well, i have a systlog server already configured, i'm 
> wondering if it is possible to forward the existing log file on the server 
> toward the Graylog server .
> thanks
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/4d060ecf-5e01-4522-94f3-b2a2d13b880e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


[graylog2] Forward an Existing Log file to Graylog ( Syslog Server ==> graylog )

2016-10-19 Thread Benbrahim Anass
Hi everyone,
i have a question, well, i have a systlog server already configured, i'm 
wondering if it is possible to forward the existing log file on the server 
toward the Graylog server .
thanks

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/e4357a8b-a0d5-4fab-9f9e-b90a27ef5033%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.