Hi Rob, rginga - Excellent information, exactly what I was hoping to see.
For the RAM requirements, would it be safe to estimate the required RAM on the master by looking at the combined utilization of the build servers? For instance, I see that the Jenkins and Java processes combined are using about 8GB of ram on one of our build servers. Would this this translate directly to the master utilization? So just add up the utilization for the build servers and that is the baseline for the master? If so then this machine is going to need TONS of ram. Or I am going to have to start pruning how much Jenkins 'remembers' about builds. That may not go over very well with the teams. Thanks, ~ Dave On Wednesday, November 12, 2014 9:06:00 AM UTC-5, Rob Mandeville wrote: > > If you move most/all of the actual jobs off to slave nodes on other > machines, then the Jenkins server host needs to: > > > > · Interact with the user (minimal hardware requirements if your > users don’t use auto-refresh; auto-refresh could greatly increase CPU needs) > > · Retrieve data, artifacts, and logs from the slave machines > (needs good network connectivity and large, fast storage capacity) > > · Hold data for every build (but not artifacts and logs) in core > (requires great gobs of memory) > > > > You certainly don’t need one core per slave executor on the master, unless > you expect all your jobs to be “chatty” all the time. Generally, capturing > the data, logs, and artifacts from a build requires one or two orders of > magnitude less CPU time than actually running the build. You’re likely to > get bigger speed gains by increasing the speed of your disk than that of > your CPU. > > > > Every build that your Jenkins server “remembers” will have its data stored > on disk and in memory, and its artifacts and logs will remain on disk until > and unless requested by a user or another job. If this was the whole > story, then memory and disk needs would increase without bounds. However, > you can set jobs to only remember a certain number of builds and/or a > certain number of days of builds. You can separately limit how many logs > and artifacts it will keep. Use this feature judiciously, preferably on > every job you define. If you have a need to store results “since the > beginning of time”, you have a big problem. At a previous job, I had that > and had to parse my build logs and write the data required to a database, > only using Jenkins to run builds and not to analyze the results. > > > > I currently run Jenkins servers and clients on VMs, and don’t have any > problems with them. This might make it easier to do whole-host backups of > your Jenkins system (if you have the space for that) for disaster recovery. > > > > Any combined master/slave machine can be converted to a slave-only machine > without more hardware; you might even be able to increase the number of > executors. In terms of server CPU and network use, adding executors to an > existing node is slightly less expensive than adding nodes. > > > > Finally, you have said that you are running a small number of Jenkins > standalone servers. I would suspect that a master-only machine needs no > more CPU or disk than a standalone host, and may just need more disk > capacity. You should be able to sacrifice one standalone system to serve > as the master, and the rest of them can be converted to slave-only hosts. > > > > *From:* [email protected] <javascript:> [mailto: > [email protected] <javascript:>] *On Behalf Of *David Brooks > *Sent:* Wednesday, November 12, 2014 8:44 AM > *To:* [email protected] <javascript:> > *Subject:* Specs for new Master server > > > > I have searched the archives and don't see a direct answer to this > question. I apologize if this has already been asked and answered. > > My team currently runs a few Jenkins stand-alone build servers, lots of > jobs each, some CI jobs, not a lot of concurrency in builds, but some. > > I have been asked to migrate our environment to a master/slave setup so > the job management can be run from one server. My questions is this: what > kind of hardware would a master-only machine need to be successful? The > current stand-alone servers will become the slaves. > > So here are the questions I am starting with: > > - It seems from what I have read in this and other forums that it is a > good idea to have one CPU core per executor. Is this true for the master > as well as the server doing the builds? Or can we scale back? > - I have also read that the master can become I/O bound while moving > the logs and build results around. (I think I have that right but some > threads suggest that Jenkins can become CPU bound as a result of being I/O > bound, an interaction that I don't fully understand.) The network > backbone > isn't an issue but I can spec out a variety of different storage > solutions. I would obviously like to avoid spending money on storage we > don't need but I need to build something that will function. > - How much RAM should this master-only machine have? > - All of our servers are virtual. That helps in that they can be > rescaled if needed, but are there any special considerations that a > virtual > master introduces that I should know about? > - Is there anything that I should be thinking about when converting > the stand-alone servers to slaves? I have already modeled the process of > getting a Jenkins server to be both a stand-alone server and a slave so > that I can do the migration without bringing the servers down. That works > fine. But is there anything about being a slave server that would change > the system requirements? > > I know I haven't provided enough details to come up with a spec, and that > is sort of intentional. I hope to learn instead how to do the analysis, > what the tradeoffs are, what the design considerations are, etc. so I can > build the spec. This won't be the last time I have to visit this > environment and I would like to build the knowledge and skills necessary to > be able to manage it. > > Thanks for any suggestions! > > ~ Dave > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > For more options, visit https://groups.google.com/d/optout. > > Click here > <https://www.mailcontrol.com/sr/p!fUEiNMhaHGX2PQPOmvUkWM85sEKD4+D6Lwpry0Oxo!FtZi7dOxJosDGAD0hjxQDZBLv8O0BTRzr!jqbv!p6A==> > > to report this email as spam. > > ------------------------------ > This e-mail and the information, including any attachments it contains, > are intended to be a confidential communication only to the person or > entity to whom it is addressed and may contain information that is > privileged. If the reader of this message is not the intended recipient, > you are hereby notified that any dissemination, distribution or copying of > this communication is strictly prohibited. If you have received this > communication in error, please immediately notify the sender and destroy > the original message. > > Thank you. > > Please consider the environment before printing this email. > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
