[graylog2] Re: email alert receiver issue
Yes, they are. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: 1.0.1 Spontaneous restart followed by memory shortage
Thanks. For the benefit of anyone reading this who uses the AWS images,
which are based on Ubuntu and use Chef to (re)configure everything, it ends
up the relevant line is in
/opt/graylog/embedded/cookbooks/graylog/recipes/elasticsearch.rb:
:max_memory = #{(node.memory.total.to_i * 0.6 ).floor / 1024}m
I changed 0.6 to 0.3 and ran a graylog-ctl reconfigure, which results in
1185m instead of 2371m being thrown at it. I'll monitor and report back if
any problems. For now this is just a test installation and loads are very
low.
Still, it seems like the default AWS configuration is cutting it too close
to the 4GB the docs on its distribution page recommend. And I'm beginning
to see that using this image instead of a manual install onto a basic
Centos Amazon linux VM was a false savings of time...
On Monday, April 13, 2015 at 10:57:00 PM UTC+3, Arie wrote:
4 GB isn't a lot if you have it all on one machine.
- You could start to give ES 1 GB of memory at max. (ES_HEAP_SIZE=1g) on
centos in /etc/sysconfig/elasticsearch.
- Second is to lower the field cache in elasticsearch.yml with:
indices.fielddata.cache.size: 40% (or even lower)
- Check is there is some swapping, this suf can't stand it.
- Is there a lot of data stored and kept over time? ES tries to keep as
much of the field data
in memory as possible. If you do not need a lot of history, consider to
keep data not
that long, and configure graylog the correct way for this.
Install a plugin to check how your instance of es is running.
...
--
You received this message because you are subscribed to the Google Groups
graylog2 group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] Journal utilization is too high and uncommited messages
Dear, I've installed the current versions of Graylog and Elasticsearch: graylog-server 1.0.1-1 / graylog-web 1.0.1-1 / graylog2-stream-dashboard 0.90.0-1 /elasticsearch 1.5.1 My server is Debian Wheezy, with 2 processors and 20 GB RAM (now I have 15 GB free). Everything works OK, but because of the high volume of received logs, I get these two error messages: Journal utilization is too high 9 minutes ago Journal utilization is too high and may go over the limit soon. Please verify that your Elasticsearch cluster is healthy and fast enough. You may also want to review your Graylog journal settings and set a higher limit. (Node: *b7b62947-250e-473b-b8df-7083d6df9886*, journal utilization: 101.0%) Uncommited messages deleted from journal 9 minutes ago Some messages were deleted from the Graylog journal before they could be written to Elasticsearch. Please verify that your Elasticsearch cluster is healthy and fast enough. You may also want to review your Graylog journal settings and set a higher limit. (Node: *b7b62947-250e-473b-b8df-7083d6df9886*) Also the JVM in the Node tab is using 750MB of 972MB heap space, and there are 1 million messages in the journal. Please, how can I tune the system in order to avoid these messages and expand the heap space??? I'm using the default settings for elasticsearch and graylog. Special thanks, Roberto -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Search advise
I don't need to use streams, I just do whatever search I what, then in the 'Search Results' on the right, click on the + to the left of 'source' and press the 'quick values' button. You can export that to a dashboard, too. My dashboard has a number of these. (I just wish the dashboard version had the clickable magnifying glass to make searching for #1 easier.) On Friday, April 10, 2015 at 4:41:33 PM UTC-5, Mark Moorcroft wrote: This is probably a dumb newb question, but at this moment it's not obvious to me. If I have a saved search like: dropping event AND queue is full Is it possible to see the list of Sources with the number of logged events per source ONLY, instead of 10 pages of results? I guess you could call that a summary. It's often the case I don't care about the details. I'm just in a hurry to go look at the clients in question so I just want to see which ones are spitting out the messages. Part of the problem is probably that I have yet to embrace Streams because it's really not clear to me how they work. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Improving Drools support (discussion / PR scoping)
Hi all, I'd like to start addressing issue #499 https://github.com/Graylog2/graylog2-server/issues/499. I've done some improvements on my own, but I'd like to work them into something that can be merged back upstream. There are a few issues I'm looking at addressing: 1. Lack of UI exposure / CRUD operations for Drools file(s) 2. No auto-reload of rules if Drools file(s) change 3. Support for only one Drools file From my understanding so far, it looks like most of this functionality should involve building up the DroolsEngine class. So far, I've taken care of #2 by storing rules file strings in a cache and doing a FS read / compare / update at some interval. If a rules file has changed, the rules are redeployed and the cache is updated. I'm looking to move onto UI integration today and eventually to multiple file support. With the hope of eventually creating a PR for this stuff, I thought I'd ask to see if you guys see this as being on the right track. I'm not sure if you already have a long-term plan scoped out for Drools or the DroolsEngine class. Please let me know if you have any extra details or gotchas that might be helpful along the way. Best, Dylan Hingey -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Upgrading Graylog 1.0.0 to 1.0.1 on AWS has broken it
I've tried to upgrade a Graylog instance (running everything) on AWS using the instructions from here: https://github.com/Graylog2/graylog2-images/tree/master/ova Which are to run the commands: wget https://packages.graylog2.org/releases/graylog2-omnibus/ubuntu/graylog_latest.deb sudo rm /var/lib/dpkg/info/graylog.postrm sudo graylog-ctl stop sudo dpkg -G -i graylog_latest.deb sudo graylog-ctl reconfigure However this breaks at the reconfigure stage. * execute[/opt/graylog/embedded/bin/graylog-ctl start graylog-server] action run - execute /opt/graylog/embedded/bin/graylog-ctl start graylog-server * ruby_block[add node to server list] action run Error executing action `run` on resource 'ruby_block[add node to server list]' Errno::ECONNREFUSED --- Connection refused - connect(2) for 127.0.0.1 port 4001 Which is I think because the etcd server isn't started: ubuntu@graylog:~$ sudo graylog-ctl status run: elasticsearch: (pid 1019) 629s; run: log: (pid 1012) 629s down: etcd: 1s, normally up, want up; run: log: (pid 1011) 629s run: graylog-server: (pid 1786) 588s; run: log: (pid 1008) 629s run: graylog-web: (pid 1021) 629s; run: log: (pid 1018) 629s run: mongodb: (pid 1014) 629s; run: log: (pid 1009) 629s run: nginx: (pid 4562) 1s; run: log: (pid 1010) 629s It tries to start it in the Chef recipe: * execute[/opt/graylog/embedded/bin/graylog-ctl start etcd] action run - execute /opt/graylog/embedded/bin/graylog-ctl start etcd Recipe: graylog::elasticsearch * directory[/var/log/graylog/elasticsearch] action create (up to date) which doesn't moan, but looking in the etcd log, it's reporting this every second: 2015-04-13_13:13:37.96364 2015/04/13 13:13:37 etcd: listening for peers on http://localhost:2380 2015-04-13_13:13:37.96367 2015/04/13 13:13:37 etcd: listening for peers on http://localhost:7001 2015-04-13_13:13:37.96368 2015/04/13 13:13:37 etcd: listening for client requests on http://0.0.0.0:2379 2015-04-13_13:13:37.96370 2015/04/13 13:13:37 etcd: listening for client requests on http://0.0.0.0:4001 2015-04-13_13:13:37.96511 2015/04/13 13:13:37 etcdserver: recovered store from snapshot at index 1310131 2015-04-13_13:13:37.96522 2015/04/13 13:13:37 etcdserver: name = default 2015-04-13_13:13:37.96522 2015/04/13 13:13:37 etcdserver: data dir = /var/opt/graylog/data/etcd 2015-04-13_13:13:37.96523 2015/04/13 13:13:37 etcdserver: heartbeat = 100ms 2015-04-13_13:13:37.96523 2015/04/13 13:13:37 etcdserver: election = 1000ms 2015-04-13_13:13:37.96523 2015/04/13 13:13:37 etcdserver: snapshot count = 1 2015-04-13_13:13:37.96524 2015/04/13 13:13:37 etcdserver: advertise client URLs = http://localhost:2379,http://localhost:4001 2015-04-13_13:13:37.96524 2015/04/13 13:13:37 etcdserver: loaded cluster information from store: default=http://localhost:2380,default=http://localhost:7001 2015-04-13_13:13:37.98912 2015/04/13 13:13:37 etcdserver: read wal error: unexpected EOF Not sure what the error means though, nor can I work out how to fix it. Any ideas? -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Extracter regexp refuses to match a log message with : in it
I'm trying to extract a port name from a log message such as this one (copied from my rsyslog permanent archive before it was transfered on into graylog 1.0.1) *2015-04-13T22:42:19-05:00 10.146.156.20 INFO: Port 1:37 link up, 100Mbps FULL duplex* I want to extract the port name, which in this line is *1:37* but nothing, absolutely nothing I've tried has worked. I have no problem extracting that field from lines like: *2015-04-13T11:06:16-05:00 10.144.24.91 INFO: Port 7 link up, 100Mbps FULL duplex* I've tried Port (\d+), Port (\S+), Port ([\d\:]+), Port (\d+:\d+), Port (\d*:?\d+) and even Port (.+) link, all with and without ^.+ and .+$ endings, and nothing works. I can always get the port out when it's just digits, but as soon as the input contains a colon, it refuses to match. I've spent two hours trying trick after trick and nothing has worked. I've been writing regexp in perl for decades so I'm pretty confident of my basic understanding of regexps. I've studied the Java documentation as well and don't see any reason why this continues to fail. What really, really is bugging me is that *ALL of those patterns worked fine in the extractor editor test page*, but once I save the extractor and go try to use it, it fails. I'm selecting actual messages out of the input and loading the messages up to test against. The only thing I can think of is that something about the underlying java is puking on the : in the content being matched, and it's causing the test to fail. Just for grins, I looked at the indexer page, and I see bunches of this: *MapperParsingException[failed to parse [port]]; nested: NumberFormatException[For input string: 1:3];* But I have specifically told this extractor to NOT convert the thing to a number. I even tried forcing in a 'lowercase' converter, but that didn't help, either. It appears that the extractor is insisting on converting the field to a number before creating it, despite what I told it to do with the converter settings. I've searched through the group posts here and found the ones where variable white space was an issue; I've checked against the original content (see above) and that isn't the issue. (I tried using \s+, a space, etc, and those made no difference, either.) Can anyone show me a pattern that will properly return a match for *1:37*? And have it properly set the new field? Here's a copy/paste of the extractor as it exists right now, it's giving me port fields with values only when the values are one or more digits. None of them with : are getting set. Trying to extract data from *message* into *port*, leaving the original intact. Configuration: - regex_value: ^.+INFO:\s+Port\s+(\S+)\s.+$ Converters - uppercase Any suggestions would be most welcome. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Artifical Ignorance in Graylog2?
Hi, I recently listen to an older packet pushers podcast episode, where syslog logging and graylog2 was featured. In the podcast, this old post on Artificial Ignorance was mentioned: http://www.ranum.com/security/computer_security/papers/ai/ I wonder if there is a good solution/workflow in implementing this in Graylog2 ? I can create a stream and add patterns I want to ignore one by one, but doing it from Stream view is not very efficient... From the Search view, can create an extractor if I mark a log entry, but it would be nice if Graylog also would allow creating rules into a Stream too. (Currently it's only allowed to match a log entry against a Stream) /P -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] kafka as datasource for Graylog
I was trying to figure out how to use Kafka as a datasource for the graylog. It seems like there is no standard connector. I'm doing the transfer using Storm and Json log message format, but I wonder if there is a simpler way. Thanks. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Customize dashboard widget size and allow anonymous dashboard access
Is it possible to grant anonymous access to certain dashboards (and have a direct URL to that dashboard)? Em segunda-feira, 30 de março de 2015 02:57:25 UTC+2, Emad Khurshid escreveu: Hey Lennart, I have the same question if you can specifically reply to the point 1 made by Brantley Hobbs for accessing the dashboards without authentication which we wish to see on TV's On Wednesday, April 16, 2014 at 8:57:31 AM UTC+10, lennart wrote: Hey Brantley, The different widget sizes are definitely coming and are already on the roadmap. Mind opening a feature request in the issue tracker for the login problem? That would be great. Thank you very much, Lennart On Fri, Apr 11, 2014 at 2:52 PM, Brantley Hobbs brantle...@gmail.com wrote: Hi all! I did some brief searching through the group and didn't see the answers to these questions, so I thought I'd start a new topic. We want to set up some Graylog2 dashboards on TV's to hang in common areas. These are essentially appliances that boot to a full-screen browser already pointed at the dashboard. I have a couple of issues that are giving me trouble however. 1. We have to do a login on the page before we can see the dashboard. Since these devices are meant to be without keyboard or mouse, this is problematic (each restart requires you to plug in a keyboard just to hit enter). 2. The televisions we have chosen are only 720p; a side-by-side search_result_chart and search_result_count widget won't fit on a standard 1280x720 720p display. Is it possible to grant anonymous access to certain dashboards (and have a direct URL to that dashboard)? Is it possible to tweak the widgets to change their display size? Thanks! Brantley Hobbs University of Georgia -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+u...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Sending GELF messages from a Java application with a specified TTL
What i'm trying to accomplish is to send GELF log messages from within a Java application with a specific TTL for each level type. For example a ERROR, INFO, WARN level I would like to have a TTL of 3 Months, while a DEBUG level I would like to have a TTL of 1 Month. The question is, can you specify the TTL information with the GELF format ? As TTL is not a standard GELF field and would go to additional fields, and if you put it as an additional field, when Graylog send the data to Elasticsearch it is not interpreted as TTL information, just as an additional field with a string. I have also tried to do the TTL from the Graylog side with DROOLS processing but was not able to properly introduce the TTL field with DROOLS so can't say yet if that is a feasible alternative. Did anyone attempt to do this and have any feedback ? I am running on Graylog 0.20 but right now i'm upgrading the to Graylog 1.x and tying to see what's new there :) -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Multitenancy
Hello world. I have just started working with graylog2. I have it running in Docker and I'm capturing Windows Event Logs as Syslog UDP. It works very well so far! I have a few questions about visibility and scoping. Imagine you want to capture log data from numerous tenants and you don't want the tenants to see each other's data. Does graylog2 support this? I suppose it could work by directing data to different receiver nodes and having them both push up to a shared parent. This hierarchy would mean each tenant would have to search their local node, but the parent server would have all the data. I assume that graylog2 allows for such a hierarchy, but I'm not sure just yet. Even so, are there features or plugins that support this kind of isolation at the user level? Any advice is much appreciated. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Increase JVM heap space
People, in my graylog server I have a lot of incoming logs and in the Node tab of Graylog web (version 1.0.1) I can see too many processing messages and the processing status bar is always near the maximum. How can I increase the JVM heap space in order to avoid journaling??? At the moment the JVM heap space is 972 MB. Thanks a lot, Alejandro -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Multitenancy
Jochen, Thanks! This looks like an ideal solution. On Tuesday, April 14, 2015 at 12:09:22 PM UTC-6, Jochen Schalanda wrote: Hi Jesse, you could achieve something like you've described with streams (see http://docs.graylog.org/en/1.0/pages/streams.html) where every stream contains the messages of one tenant. For this to work, you'd have to create each stream for each tenant explicitly, there's no automatism for it yet. Users/tenants can be granted access to one or more streams so they can only search within their own log messages. Cheers, Jochen On Tuesday, 14 April 2015 18:04:56 UTC+2, Jesse Skrivseth wrote: Hello world. I have just started working with graylog2. I have it running in Docker and I'm capturing Windows Event Logs as Syslog UDP. It works very well so far! I have a few questions about visibility and scoping. Imagine you want to capture log data from numerous tenants and you don't want the tenants to see each other's data. Does graylog2 support this? I suppose it could work by directing data to different receiver nodes and having them both push up to a shared parent. This hierarchy would mean each tenant would have to search their local node, but the parent server would have all the data. I assume that graylog2 allows for such a hierarchy, but I'm not sure just yet. Even so, are there features or plugins that support this kind of isolation at the user level? Any advice is much appreciated. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] 50k + messages per second
Hey folks I have a new project that Im looking for some insight on. we are testing out logging some high volume data that is between 65 and 100k per second. What would you all think would be an optimal config? With higher loads do you find that having everything separate vs full stack offers better performance? Thanks Mike -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Multitenancy
Hi Jesse, you could achieve something like you've described with streams (see http://docs.graylog.org/en/1.0/pages/streams.html) where every stream contains the messages of one tenant. For this to work, you'd have to create each stream for each tenant explicitly, there's no automatism for it yet. Users/tenants can be granted access to one or more streams so they can only search within their own log messages. Cheers, Jochen On Tuesday, 14 April 2015 18:04:56 UTC+2, Jesse Skrivseth wrote: Hello world. I have just started working with graylog2. I have it running in Docker and I'm capturing Windows Event Logs as Syslog UDP. It works very well so far! I have a few questions about visibility and scoping. Imagine you want to capture log data from numerous tenants and you don't want the tenants to see each other's data. Does graylog2 support this? I suppose it could work by directing data to different receiver nodes and having them both push up to a shared parent. This hierarchy would mean each tenant would have to search their local node, but the parent server would have all the data. I assume that graylog2 allows for such a hierarchy, but I'm not sure just yet. Even so, are there features or plugins that support this kind of isolation at the user level? Any advice is much appreciated. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
