[graylog2] Need help in scaling up my Graylog-elasticsearch-logstash setup
You are going to want to have a good bit more ram for large amount of log processing. Ideally you will want to setup multiple vms to each handle a role as well (e.g. VM or 2 for elasticsearch, Vm or two for graylog nodes) I have 2 elasticsearch search nodes with 12gb ran each and I still like it isn't awkward memory wise. My two graylog nodes also have 4gb each with r CPUs. Generally you will want to throw as much RAM as you can at elasticsearch search because there is a steep performance hit with its disk operations. Unless you have all flash disks maybe. As far as CPU, you will want to throw as much cpu as you can at the graylog server nodes. It eats up A LOT of cpu to process stream and extractor rules. Disk space is only something to consider based on how many messages you want to store at any given time and for how long. The more disk space the better. Hope that helps! -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/4d66e8a2-ec0e-4cfc-a7ac-3539ad5b62a0%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Graylog System Sizing Problem?
I don't believe there is any relation between hard disk space and available cpu for graylog. To understand what your requirement are for hardware we need to answer two questions: 1. How many messages per second do you want to process. Depending on the message size and complexity of steam and extractor rules, a few cpus (e.g. 4) can process a thousand or so messages a second. This does vary on CPU type sure but I'm basing this off of a virtual machine on a shared host. 2. How long do we want to reattain processed messages. This also somewhat depends on your message per second count. I don't think 20gb is much space considering how cheap storage is. I recommend a few hundred GB if you can spare it. I find it helpful to be able to search back easily through historical logs. Hope that helps! If you have any specific questions I'll be happy to try and answer them. -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/92542704-eaf1-4888-adfa-39e7de7f915f%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Graylog 1.3.4 elasticsearch cluster red / unassigned shards
What version of elastic search and how many data nodes? Do you have replicas enabled/ how many? The error says "not allowed, reason: [NO(shard cannot be allocated on same node [mB5gKQroSu6XQNzNkeHzxQ] it already exists on)]" Are there any errors in the elastic search logs? I found this which may be relevant https://groups.google.com/forum/m/#!msg/elasticsearch/sztApzHmTjY/Nbr0N3EtfhMJ And maybe http://stackoverflow.com/questions/34619265/elasticsearch-unassigned-shards-with-two-nodes-different-machines-1-master-bo -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/c57275f8-9ffa-4696-8056-6ed350e9a1f8%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Graylog collector
Check out the collector documentation page here http://docs.graylog.org/en/1.3/pages/collector.html You can define what log files to ship to graylog in the collector configuration file. Configuring steam rules is done via the gray log web interface. Hope that helps. -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/6cb218be-26ad-4354-ae22-c1400aa5261b%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] [ANNOUNCE] Graylog v2.0.0-beta.3 has been released
I just want to say, thank you for making such an incredible and valuable product. -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/0c2a827b-9b94-4964-9100-936df9276ae3%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Web Interface Certificate differences from v1 and v2
Okay, quick update, I did some quick searching and found this, https://community.oracle.com/thread/1534464?start=0 which sounds exactly like the issue. My cert chain file does have extra characters in it. I'll test this tomorrow. -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/6ac83424-cea8-43b9-a0d9-5ed507cbb10f%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Web Interface Certificate differences from v1 and v2
Thanks for that command. So I'm able to extract my private key from the original Java keysyore (because this is where the original private key was created) and convert it to p12 and then pkcs8. I can verify the key is ASCII readable and is encoded and passes checked when viewing via openssl. However, I'm still confused about the public cert. I think the documentation is saying I need a x.509 pkcs8 cert but there seems to be a contradiction. Anything that I can find and all commands that generate a valid pkcs8 cert only seem to contain the private key. If I just out an ASCII readable pem of the base 64 certificate chain, to me, that appears to be the closest I can get. So anyways, this is all to say I have what I think are the correct cert and key yet I'm still getting that same error. I've probably spent about 4 hours testing and reading up on x.509. Any help? -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/379f4269-8050-4064-b83c-8e73eadaa939%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] [ANNOUNCE] Graylog v2.0.0-beta.3 has been released
Hi everyone, we just released Graylog v2.0.0-beta.3. Read more in the announcement: * https://www.graylog.org/blog/53-graylog-v2-0-beta-3-released Thanks, Lennart -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CADRA1nmRdZvqhEs4RcydhY03shd-rU%3DMWtWCRgW%3D6pu_ozp5jQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: web interface with v2.0 appears to require direct REST access?
Yes it's required to connect to TCP 12900. That's also where the sidecar and collectors connect. I think the reasoning was that you ought to have the ability to open the firewall to the server since you're already using a bunch of ports to send it the log data it needs. The web interface isn't separate any more, so one more port for the REST interface wasn't supposed to be a problem. But proxy firewalls and remote access may turn out to be a larger issue than the developers expected. I'm currently having problems getting my URIs correct in Docker on a bunch of hosts that don't tell me the container's IP until after the container is created. On Wednesday, April 13, 2016 at 8:14:16 PM UTC-4, Jason Haar wrote: > > Hi there > > Under graylog-1.3.4 I had published graylog-web behind a WAF - which > nicely mapped https://graylog.internet.domain to > http://graylog.intranet.domain (notice the different domain names too) > > With v2.0 I can't get this to work. Now it appears graylog returns content > with hardwired URLs that are defined by rest_listen_uri? That means we end > up with browser errors as they are talking to the WAF over HTTPS and the > content contains HTTP links - to port 12900. Bad. > > Am I correct that graylog-v2 requires browsers to talk to non-web ports > (ie 12900)? That's quite a change. The comments say "Must be reachable by > other Graylog server nodes if you run a cluster" - no mention of this being > required by web browsers. > > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/8c3040cb-61f0-4519-a5d2-e9bf55178d15%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: date being recognized as a string
...An Update: Graylog created an error.
Oh no, something went wrong!
(You caused a org.graylog2.restclient.lib.APIException. API call failed POST
http://@127.0.0.1:12900/system/deflector/cycle returned 500 Internal Server
Error body: {"type":"ApiError","message":"mapping [properties]"})
With this input:
curl -XPUT
http://peng-elastic0-dev.gb-lon1.elk.metroscales.io:9200/_template/ts_json -d
'{"template": "json*","mappings": {"properties": {"@timestamp": {"type":
"date","format": "dateOptionalTime"
greetings
Stacktrace:
org.graylog2.restclient.lib.ApiClientImpl$ApiRequestBuilder#handleResponse
(ApiClientImpl.java:511)
org.graylog2.restclient.lib.ApiClientImpl$ApiRequestBuilder#execute
(ApiClientImpl.java:441)
org.graylog2.restclient.models.IndexService#cycleDeflector
(IndexService.java:114)
controllers.IndicesController#cycleDeflector (IndicesController.java:183)
Routes$$anonfun$routes$1$$anonfun$applyOrElse$63$$anonfun$apply$637#apply
(routes_routing.scala:2142)
Routes$$anonfun$routes$1$$anonfun$applyOrElse$63$$anonfun$apply$637#apply
(routes_routing.scala:2142)
play.core.Router$HandlerInvokerFactory$$anon$4#resultCall (Router.scala:264)
play.core.Router$HandlerInvokerFactory$JavaActionInvokerFactory$$anon$15$$anon$1#invocation
(Router.scala:255)
play.core.j.JavaAction$$anon$1#call (JavaAction.scala:55)
play.GlobalSettings$1#call (GlobalSettings.java:67)
play.mvc.Security$AuthenticatedAction#call (Security.java:44)
play.core.j.JavaAction$$anonfun$11#apply (JavaAction.scala:82)
play.core.j.JavaAction$$anonfun$11#apply (JavaAction.scala:82)
scala.concurrent.impl.Future$PromiseCompletingRunnable#liftedTree1$1
(Future.scala:24)
scala.concurrent.impl.Future$PromiseCompletingRunnable#run (Future.scala:24)
play.core.j.HttpExecutionContext$$anon$2#run (HttpExecutionContext.scala:40)
play.api.libs.iteratee.Execution$trampoline$#execute (Execution.scala:46)
play.core.j.HttpExecutionContext#execute (HttpExecutionContext.scala:32)
scala.concurrent.impl.Future$#apply (Future.scala:31)
scala.concurrent.Future$#apply (Future.scala:485)
play.core.j.JavaAction$class#apply (JavaAction.scala:82)
play.core.Router$HandlerInvokerFactory$JavaActionInvokerFactory$$anon$15$$anon$1#apply
(Router.scala:252)
play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5#apply
(Action.scala:130)
play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5#apply
(Action.scala:130)
play.utils.Threads$#withContextClassLoader (Threads.scala:21)
play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4#apply
(Action.scala:129)
play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4#apply
(Action.scala:128)
scala.Option#map (Option.scala:145)
play.api.mvc.Action$$anonfun$apply$1#apply (Action.scala:128)
play.api.mvc.Action$$anonfun$apply$1#apply (Action.scala:121)
play.api.libs.iteratee.Iteratee$$anonfun$mapM$1#apply (Iteratee.scala:483)
play.api.libs.iteratee.Iteratee$$anonfun$mapM$1#apply (Iteratee.scala:483)
play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1#apply
(Iteratee.scala:519)
play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1#apply
(Iteratee.scala:519)
play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$14#apply
(Iteratee.scala:496)
play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$14#apply
(Iteratee.scala:496)
scala.concurrent.impl.Future$PromiseCompletingRunnable#liftedTree1$1
(Future.scala:24)
scala.concurrent.impl.Future$PromiseCompletingRunnable#run (Future.scala:24)
akka.dispatch.TaskInvocation#run (AbstractDispatcher.scala:41)
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask#exec
(AbstractDispatcher.scala:393)
scala.concurrent.forkjoin.ForkJoinTask#doExec (ForkJoinTask.java:260)
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue#runTask
(ForkJoinPool.java:1339)
scala.concurrent.forkjoin.ForkJoinPool#runWorker (ForkJoinPool.java:1979)
scala.concurrent.forkjoin.ForkJoinWorkerThread#run
(ForkJoinWorkerThread.java:107)
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/0de3e0ac-53ce-450d-9148-b22c6b93f7f2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: date being recognized as a string
Hi Jochen,
actually we already did this step. We tried this:
curl -XPUT http://localhost:9200/_template/timestamp -d'
{
"template": "*",
"mappings": {
"_default_": {
"@timestamp": {
"type": "date",
"format": "dateOptionalTime"
}
}
}
}
'
Any clues how to proceed?
Thanks & best regards
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/fab5d770-dcf0-4616-b8f2-bccda62e0f3d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: extracting date field from json
Hi there, did you find a solution for this meanwhile? I have the same problem here. Any help is very welcomed? Thanks - cheers -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/2b75cb33-4c7a-406a-a76b-935b644d4b40%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: Can I convert a field from string to integer?
Hi Jason, we might add an interface for managing Elasticsearch mappings/index templates in a future version of Graylog, but right now it's not on the immediate roadmap. Cheers, Jochen On Wednesday, 13 April 2016 22:07:53 UTC+2, Jason Haar wrote: > > It would be great if graylog had an interface for doing this. I don't know > if this is the intent, but graylog sort of "hides" ES almost entirely, so > being able to use graylog to redefine field names would be useful and in > keeping with that concept (it would obviously just be a wrapper around the > actual ES template) > > Good timing on this topic. I've just adding a new GELF feed of data > containing a "lastSeen" field and was wondering how to make it a field like > "timestamp". Now I know ;-) > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/5d39bb2d-1f1b-4a3c-93a7-3e29b0a6ffb7%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Data type of @timestamp is sometimes string instead of date
Hi, I have the exact problem. Did you find a soution to this? cheers -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/054d3351-0eb3-40b8-b345-9f01ef6546da%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Copy field containing date as a new date field does not work
Hi, for reference: this question has been answered in https://groups.google.com/forum/#!topic/graylog2/LkBvIgDlcoo (post by the same author). Cheers, Jochen On Wednesday, 13 April 2016 18:25:55 UTC+2, grayl...@gmx.de wrote: > > Unfortunately Graylog saves a field in ElasticSearch instead of date. > > Therefore I tried to copy the field in a new field as type date. > > I tried it that way but without success. Can somebody help? > > Message: 2016-04-13T16:18:24.739Z > > Converter: Convert to date type > > Format string: -MM-dd'HH:mm:ss.SSSZ > > Time zone: UTC > > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/7639cca8-d188-4258-aca6-63fa82ddf941%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Data type of @timestamp is sometimes string instead of date
Hi, if you want to ensure that a document field has always the same type, you'll have to create a custom index mapping ( https://www.elastic.co/guide/en/elasticsearch/reference/1.7/mapping.html) and make sure it will be applied to new indices by creating an index template with your custom mapping ( https://www.elastic.co/guide/en/elasticsearch/reference/1.7/indices-templates.html ). Cheers, Jochen On Friday, 8 April 2016 15:53:44 UTC+2, grayl...@gmx.de wrote: > > Hello, > > We have a problem with the @timestamp field. Sometimes it is wrongly > created as string instead of date. > > We use the following pipeline: > > Docker -- mixed logs in GELF format --> graylog2-server1 -- only JSON logs > in GELF format --> graylog2-server2. > > Both graylog2 servers are connected to the same ElasticSearch cluster. > Each graylog2-server uses a separate index-pattern. > > Graylog2 version is 1.3.4. > > The indexes of graylog2-server2 should always create the @timestamp field > as date. Is there a way to force graylog2 to use always date? > > Thanx in advancd! > > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/8bd72ac8-dc79-4c77-8040-f6b61bd0ced0%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: date being recognized as a string
Hi, if you want to ensure that a document field has always the same type, you'll have to create a custom index mapping ( https://www.elastic.co/guide/en/elasticsearch/reference/1.7/mapping.html) and make sure it will be applied to new indices by creating an index template with your custom mapping ( https://www.elastic.co/guide/en/elasticsearch/reference/1.7/indices-templates.html ). Cheers, Jochen On Thursday, 14 April 2016 11:35:23 UTC+2, mantic...@gmail.com wrote: > > Hi all, > > I would like to give some details. > We have a field @timestamp. > But it is in the false format. > It is recognized as string instead as beging recognized as date. > We defined the input as being JSON. > The extractor defnitions seemed to be clean. With this extractor we should > have had as a result the @timestamp as date. > But how can we test it? > Also: Taking a look to Elasticsearch via Kibana we see the @timestamp > field being in a string format. > So, either: The format conversion did not function we the extractor, what > I did not think. > Or some template in elasticsearch transforms the now-date-field back to > string. > Does someone have a clue? > Either: How can we check the extractor conversion went good. > Or: If elasticsearch is the source for this erratic behaviour. > > Thanks! > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/38f77029-18f8-4449-b2e7-057dd68b1eaf%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: date being recognized as a string
Hi all, I would like to give some details. We have a field @timestamp. But it is in the false format. It is recognized as string instead as beging recognized as date. We defined the input as being JSON. The extractor defnitions seemed to be clean. With this extractor we should have had as a result the @timestamp as date. But how can we test it? Also: Taking a look to Elasticsearch via Kibana we see the @timestamp field being in a string format. So, either: The format conversion did not function we the extractor, what I did not think. Or some template in elasticsearch transforms the now-date-field back to string. Does someone have a clue? Either: How can we check the extractor conversion went good. Or: If elasticsearch is the source for this erratic behaviour. Thanks! -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/9b78743a-a838-40df-944b-3c42bf47111d%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] date being recognized as a string
Hi there, we have incoming events where a date field is being recognized as a string instead of being recogniized as a date. With the extractor we tried a conversion. In Kibana we still see the incoming events the date being recognized as a string. Does someone has a hint how to fix this? Thanks! -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/b47bbe5e-1eb5-42a9-a623-ffab9d7a8933%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
