Hi Joshua, I use Puppet, so not managing config files myself. To my knowledge though, server.conf for Graylog and mongo[sd].conf is the only non ES config files.
> On 9/12/2016, at 08:29, Joshua Waclawski <[email protected]> wrote: > > I can definitely do that. While I'm doing that, would you be able to assist > me otherwise by helping to locate the configuration files for each of the > services? > >> On Thursday, December 8, 2016 at 2:16:37 PM UTC-5, Werner van der Merwe >> wrote: >> Hi Joshua, >> >> I am going to have to make a few assumptions here. The fact that your ES >> cluster is red will directly result in your journal filling up. Graylog will >> journal all the messages until ES appears green before it sends the >> messages. >> >> I'd focus my time on looking through the ES logs for indications of what is >> causing Elasticsearch to be red. Until that goes green, nothing you change >> in mongo or Graylog will resolve the problem. >> >> Let us know if the ES log reports anything. >> >> Good luck! >> >>> On 9/12/2016, at 08:01, Joshua Waclawski <[email protected]> wrote: >>> >>> Hello Werner, >>> >>> Thanks for the reply and I apologize for the long delay, my project was >>> shelved briefly to handle some new work. With that out of the way I can >>> get back into this project... >>> >>> Well to start out, my graylog server crashes almost weekly due to the >>> application running out of some sort of resource, be it memory or journal >>> space; I'd like to figure out what configurations to change to stop this. >>> I'm running this on a dual-core server with 4 gb of memory and plenty of >>> disk space. >>> >>> Here are the errors I'm actually looking at on my server now: >>> Elasticsearch Cluster Unhealthy(RED) >>> Uncommited Messages Deleted from Journal >>> Journal Utilization is Too High >>> This is running on a default installation using the AWS-EC2 image. I have >>> 11 servers reporting to this server with a 1 minute average of 10 msg/s, so >>> the load on this server is incredibly low for what Graylog should be able >>> to handle. On top of that we have several new vulnerabilities on the >>> system that need addressing. >>> >>> My biggest problem right now is trying to find all of the configuration >>> files that I need to review and change. The documentation available does >>> not match how everything is setup in the EC2 image available on Amazon Web >>> Services. The mongod.cnf file required to configure mongoDB doesn't exist >>> on this server and the location stated in the documentation for the JVM >>> settings file is incorrect, as it's not in the /etc/default folder. I even >>> tried doing a search for the graylog-server file for JVM configuration and >>> it showed me three, all of which are in the /opt/graylog directory, so I >>> have no clue which one to edit. >>> >>> All I really need right now is a mapping of where the configuration files >>> are on the EC2 image and I should be able to go from there. >>> >>>> On Monday, November 21, 2016 at 5:43:52 PM UTC-5, Werner van der Merwe >>>> wrote: >>>> Hi Joshua, >>>> >>>> Hardware requirements: >>>> It is obviously very difficult to give you exact numbers. The requirements >>>> for 300 syslog messages vs 300 multi-line logs where one extracts 50 key >>>> value pairs per entry will have different requirements. That said, 300 >>>> messages is trivial and you can get away with very low resources, at a >>>> guess I'd say 1 or 2 cores and 4-8Gb ram will be more than adequate, >>>> again, depending on what you are sending it. >>>> >>>> Configs are actually fairly straight-forward. Most GrayLog config has >>>> moved to Mongo, so you only _really_ need to worry about server.conf file >>>> in /etc/graylog/server. >>>> Keep in mind that Mongo and Elasticsearch is used by Graylog, but not >>>> included, so their configs are managed separately. >>>> >>>> If you have more specific questions around the config, that might help as >>>> well. >>>> >>>> Apart from that, if you are using puppet, the manifest they provide is >>>> really good and does everything for you! >>>> >>>> You mention you've hit a roadblock - are you struggling to get the system >>>> running, is it running slow? Might help to let us know a bit more detail >>>> on what is keeping you from moving forward? >>>> >>>> Cheers >>>> >>>>> On Friday, November 18, 2016 at 5:50:17 AM UTC+13, Joshua Waclawski wrote: >>>>> As the title states, I'm pretty new to Graylog and Elasticsearch. I've >>>>> read the documentation thoroughly and I've watched a few educational >>>>> videos describing how elasticsearch works from the ground up; everything >>>>> is very, very cool and I'm excited to start using it! Using the AMI >>>>> provided on the Github, I've setup an EC2 instance and have started work >>>>> on learning how to configure and use this tool, but I've hit a bit of a >>>>> road block and need some answers... >>>>> Hardware requirements - what exactly are they? I'm attempting to deploy >>>>> graylog to an environment that receives no more than maybe 200-300 >>>>> messages per second, if that. I can't imagine that managing a few >>>>> thousand logs per minute requires 12gb of RAM to do, but I'm new to >>>>> Elasticsearch so I'm asking for clarification. Every white paper, forum >>>>> post, blog post, or guide that I've read so far assumes 5000+ messages >>>>> per second. >>>>> Configurations - This is what annoys me the most. The configuration >>>>> files are very, very scattered (at least they seem so) and the official >>>>> documentation does a very poor job of explaining what's required to >>>>> configure for basic functionality on a single server. Using the AWS AMI, >>>>> what configuration files need editing to inflict changes upon the system? >>>>> I'm seeing .conf, .yml, and .cfg files all over my operating system with >>>>> seemingly redundant settings that I can't find explanations for. Again, >>>>> this could be my ignorance of the architecture, but the file hierarchy is >>>>> explained no where. >>>>> That's really it for now.. if I can hammer out these answers with some >>>>> level of certainty then I'll have what I need to move forward in >>>>> configuring and testing this software. As thing stand now, I have no >>>>> idea if I'm editing the proper config file half the time as all I receive >>>>> are errors in logs and alerts in the webUI. >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "Graylog Users" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/graylog2/KlZkOlCdffE/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/graylog2/904086fc-f92f-415c-b78b-4ca327b04894%40googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to a topic in the Google > Groups "Graylog Users" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/graylog2/KlZkOlCdffE/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/graylog2/2c36bd59-86db-4ad7-92d1-421603f07958%40googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/7AD98830-4B4B-4DEE-8D5F-B9D338546233%40gmail.com. For more options, visit https://groups.google.com/d/optout.
