[graylog2] Need help in scaling up my Graylog-elasticsearch-logstash setup

2016-04-14 Thread Drew Miranda
You are going to want to have a good bit more ram for large amount of log processing. Ideally you will want to setup multiple vms to each handle a role as well (e.g. VM or 2 for elasticsearch, Vm or two for graylog nodes) I have 2 elasticsearch search nodes with 12gb ran each and I still like

[graylog2] Graylog System Sizing Problem?

2016-04-14 Thread Drew Miranda
I don't believe there is any relation between hard disk space and available cpu for graylog. To understand what your requirement are for hardware we need to answer two questions: 1. How many messages per second do you want to process. Depending on the message size and complexity of steam and

[graylog2] Graylog 1.3.4 elasticsearch cluster red / unassigned shards

2016-04-14 Thread Drew Miranda
What version of elastic search and how many data nodes? Do you have replicas enabled/ how many? The error says "not allowed, reason: [NO(shard cannot be allocated on same node [mB5gKQroSu6XQNzNkeHzxQ] it already exists on)]" Are there any errors in the elastic search logs? I found this which

[graylog2] Graylog collector

2016-04-14 Thread Drew Miranda
Check out the collector documentation page here http://docs.graylog.org/en/1.3/pages/collector.html You can define what log files to ship to graylog in the collector configuration file. Configuring steam rules is done via the gray log web interface. Hope that helps. -- You received this

[graylog2] [ANNOUNCE] Graylog v2.0.0-beta.3 has been released

2016-04-14 Thread Drew Miranda
I just want to say, thank you for making such an incredible and valuable product. -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to

[graylog2] Re: Web Interface Certificate differences from v1 and v2

2016-04-14 Thread Drew Miranda
Okay, quick update, I did some quick searching and found this, https://community.oracle.com/thread/1534464?start=0 which sounds exactly like the issue. My cert chain file does have extra characters in it. I'll test this tomorrow. -- You received this message because you are subscribed to the

[graylog2] Re: Web Interface Certificate differences from v1 and v2

2016-04-14 Thread Drew Miranda
Thanks for that command. So I'm able to extract my private key from the original Java keysyore (because this is where the original private key was created) and convert it to p12 and then pkcs8. I can verify the key is ASCII readable and is encoded and passes checked when viewing via openssl.

[graylog2] [ANNOUNCE] Graylog v2.0.0-beta.3 has been released

2016-04-14 Thread Lennart Koopmann
Hi everyone, we just released Graylog v2.0.0-beta.3. Read more in the announcement: * https://www.graylog.org/blog/53-graylog-v2-0-beta-3-released Thanks, Lennart -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this

[graylog2] Re: web interface with v2.0 appears to require direct REST access?

2016-04-14 Thread Michael Taylor
Yes it's required to connect to TCP 12900. That's also where the sidecar and collectors connect. I think the reasoning was that you ought to have the ability to open the firewall to the server since you're already using a bunch of ports to send it the log data it needs. The web interface isn't

[graylog2] Re: date being recognized as a string

2016-04-14 Thread mantica . msys
...An Update: Graylog created an error. Oh no, something went wrong! (You caused a org.graylog2.restclient.lib.APIException. API call failed POST http://@127.0.0.1:12900/system/deflector/cycle returned 500 Internal Server Error body: {"type":"ApiError","message":"mapping [properties]"}) With

[graylog2] Re: date being recognized as a string

2016-04-14 Thread mantica . msys
Hi Jochen, actually we already did this step. We tried this: curl -XPUT http://localhost:9200/_template/timestamp -d' { "template": "*", "mappings": { "_default_": { "@timestamp": { "type": "date", "format": "dateOptionalTime" } } } } ' Any clues how to

[graylog2] Re: extracting date field from json

2016-04-14 Thread mantica . msys
Hi there, did you find a solution for this meanwhile? I have the same problem here. Any help is very welcomed? Thanks - cheers -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it,

Re: [graylog2] Re: Can I convert a field from string to integer?

2016-04-14 Thread Jochen Schalanda
Hi Jason, we might add an interface for managing Elasticsearch mappings/index templates in a future version of Graylog, but right now it's not on the immediate roadmap. Cheers, Jochen On Wednesday, 13 April 2016 22:07:53 UTC+2, Jason Haar wrote: > > It would be great if graylog had an

[graylog2] Re: Data type of @timestamp is sometimes string instead of date

2016-04-14 Thread mantica . msys
Hi, I have the exact problem. Did you find a soution to this? cheers -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com.

[graylog2] Re: Copy field containing date as a new date field does not work

2016-04-14 Thread Jochen Schalanda
Hi, for reference: this question has been answered in https://groups.google.com/forum/#!topic/graylog2/LkBvIgDlcoo (post by the same author). Cheers, Jochen On Wednesday, 13 April 2016 18:25:55 UTC+2, grayl...@gmx.de wrote: > > Unfortunately Graylog saves a field in ElasticSearch instead of

[graylog2] Re: Data type of @timestamp is sometimes string instead of date

2016-04-14 Thread Jochen Schalanda
Hi, if you want to ensure that a document field has always the same type, you'll have to create a custom index mapping ( https://www.elastic.co/guide/en/elasticsearch/reference/1.7/mapping.html) and make sure it will be applied to new indices by creating an index template with your custom

[graylog2] Re: date being recognized as a string

2016-04-14 Thread Jochen Schalanda
Hi, if you want to ensure that a document field has always the same type, you'll have to create a custom index mapping ( https://www.elastic.co/guide/en/elasticsearch/reference/1.7/mapping.html) and make sure it will be applied to new indices by creating an index template with your custom

[graylog2] Re: date being recognized as a string

2016-04-14 Thread mantica . msys
Hi all, I would like to give some details. We have a field @timestamp. But it is in the false format. It is recognized as string instead as beging recognized as date. We defined the input as being JSON. The extractor defnitions seemed to be clean. With this extractor we should have had as a

[graylog2] date being recognized as a string

2016-04-14 Thread mantica . msys
Hi there, we have incoming events where a date field is being recognized as a string instead of being recogniized as a date. With the extractor we tried a conversion. In Kibana we still see the incoming events the date being recognized as a string. Does someone has a hint how to fix this?