On Tue, 7 Nov 2017 at 14:56, Eric D Helms <ericdhe...@gmail.com> wrote:
> On Thu, Nov 2, 2017 at 5:05 PM, Ivan Necas <ine...@redhat.com> wrote: > >> I lean towards the push model here. The main reason is >> the simpler way to publish the instrumentation data from whatever >> process we want to track. Also, my understanding is, that we don't care >> only if the service is up or down (readiness and liveness) but also >> about trends during the processing. >> > > In reading about push vs. pull, my biggest issue with it is that the > application has to have knowledge of where it's pushing. Whereas the pull > model allows an application to say I have metrics here and anything that > knows how to scrape and interpret those metrics can grab them at their > leisure. This provides nicer de-coupling and potentially more choice if > there is a standard-ish data format used to expose the metrics. > > >> >> Eric: could you more describe the 5 web applications requiring 5 >> monitoring containers? >> I might be missing where this implication came from? >> > > This part comes from if you do the sidecar method of running something > like a statsd process. A sidecar is essentially just running two containers > in a pod where one container is considered the main application and the > other an addon that provides some additional value or performs some actions > without being baked into your main application container. If you picture > this idea, and then think about pod scaling (in Kube), for every scale up > you'd also be adding another statsd process container. This might be fine > in practice, but could in theory be overkill. > Let me ask again: why new stats process for every container? It seems to me like other infra service, similar to db: you don't run a separate db process for every container either. I might be missing something. -- Ivan > Another method with statsd is simply to run it on the host itself and have > the containers send data to it but this provides some security concerns > from what I understand. > > The biggest limiting factor appears to be how forking webservers are > handled and probably constraints us the most. Lukas, have you seen anything > related to being to define what the metrics are and how they get published > being able to be separated from the publishing mechanism? My thinking being > if we started with statsd and wrote code within the application generating > statsd metrics, if at a later point one could simply say now publish this > via HTTP endpoint in Prometheus data style for scraping? > > Eric > > >> >> -- Ivan >> >> On Wed, Nov 1, 2017 at 4:54 PM, Lukas Zapletal <l...@redhat.com> wrote: >> >> Does Prometheus only not work in a multi-process Rails web server? >> Does it >> >> work for a single process multi-threaded web server? This is an >> interesting >> >> roadblock given you'd expect this to affect lots of webserver across >> >> multiple languages out there. >> > >> > Any Rails app that has multiple processes needs currently to figure >> > out how to deliver data to the HTTP endpoint. E.g. store it in a >> > database or something, which is not the best approach. >> > >> > Absolutely, it lacks quite important feature right there. It stems >> > from the design which is pull-based. >> > >> >> Yes, standard practice is to think about one container per pod (in a >> >> Kubernetes environment). However, there are patterns for things like >> log >> >> aggregation and monitoring such as doing a sidecar container that >> ensures >> >> co-location. The part I don't entirely get with sidecars is if I scale >> the >> >> pod to say 5, I get 5 web applications and 5 monitoring containers and >> that >> >> seems odd. Which I why I think the tendency is towards models where >> your >> >> single process/application is the end point for your metrics to be >> scrapped >> >> by an outside agent or services. >> >> >> >> I agree you want the collector to be separate, but if your web >> application >> >> is down what value would a monitoring endpoint being alive provide? The >> >> application would be down, thus no metrics to serve up. The other >> exporters >> >> such as the one exporting metrics about the underlying system would be >> >> responsible for giving system metrics. In the Kube world, this is >> handled by >> >> readiness and liveness probes for Kubenernetes to re-spin the >> container if >> >> it stops responding. >> > >> > In container world, monitoring agents are running on hosts, not in >> > containers themselves. And collector agents can be 1:1 or 1:N (e.g. >> > for each container host). I am not sure I follow you here. Why you >> > don't see added value again? Monitoring agent without any apps >> > connected is as useful as ssh deamon waiting for connections. >> > >> > Let me put it this way - push approach seems to be more appropriate >> > for multi process Ruby application than pull approach. That's what we >> > are discussing here, unless there are better protocols/agents I am not >> > aware of. >> > >> > Honestly, pull approach via simple HTTP REST API seems cleaner but it >> > is just not good fit and also it creates other unnecessary >> > responsibility on the app itself. You are working on containerizing >> > Foreman, so it is also actually against this effort. >> > >> > Anyway, let me throw another integration. Collectd has an agent (or >> > plugin) that opens a local socket which can be used to receive data >> > from other applications. I wrote Ruby client library the other day >> > (https://github.com/lzap/collectd-uxsock) but I believe this make no >> > difference than statsd - you still need a local process to gather the >> > data. >> > >> > -- >> > Later, >> > Lukas @lzap Zapletal >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups "foreman-dev" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an email to foreman-dev+unsubscr...@googlegroups.com. >> > For more options, visit https://groups.google.com/d/optout. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "foreman-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to foreman-dev+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > Eric D. Helms > Red Hat Engineering > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
