Thanks Ivan, I'll definitely check out Kamon and inquire further about Typesafe Console.
One thing you mentioned that I didn't understand: "As mentioned on the "Why no mailbox size in Akka 2?" blog, the collections backing some mailbox implementations have O(n) time implementations of .size(), we are experimenting on attaching counters to message queues to overcome this problem and have a constant time implementation of .numberOfMessages, *but that shouldn't be a limitation to monitoring mailboxes.*" Are you're saying that the O(n) time .size() implementation *shouldn't*stop folks from monitoring mailboxes (contrary to the blog post and you guys working on queue counters instead)? If so, can you explain, or if not, what did you mean? thx, Chris On Wed, Mar 26, 2014 at 3:00 PM, Ivan Topolnjak <[email protected]> wrote: > Chris, > > 1) Generic JVM metrics >> > > There are many approaches on how to get JVM metrics out there, and > Edward's suggestion seems pretty cool.. I'll just skip this item. > > 2) Generic Akka metrics (mailbox sizes, message rates, etc.) >> > > Here comes the tricky part. I totally agree that having access to the > mailbox size inside an actor is a bad idea and have no complaints about its > removal from the actor API, but I would never say that it is discouraged to > monitor mailbox sizes for some of your actors! In my experience with Akka, > the most common cause of OutOfMemoryErrors are unbounded mailboxes pilling > up messages because some part of the application got slower than we > thought, in some areas of our apps where we didn't think that back pressure > control was necessary... sometimes a little increase in processing times > for an actor might seem harmless, if you only look at processing times, but > it can end up pilling thousands of messages in a mailbox over time and a > simple processing time increase might turn into a app crash.. you > definitely want to keep an eye on some mailboxes on your application. Also, > monitoring min and max mailbox sizes you get the correct answers to > questions like ¿do I need to make this actor a router?, ¿do I need to tweak > this dispatcher?, never underestimate the amount of info you can infer from > looking at some mailbox metrics! > > You mentioned in your first post that the Typesafe Console is not > recommended for production usage, but as far as I known there is no such > limitation, I suggest you to ask for more information on the > typesafe-console mailing list [1] to make sure you get the most up to date > info. The applications we currently have in production are monitored by > Kamon [2] and we are pushing our metrics data to NewRelic, but Kamon can > offer you the metrics you want and you can then push those metrics to > whatever metrics backend you prefer. We heard a lot about statsd lately so > I just created an issue to track integration with statsd [3], which should > be really easy to achieve. As mentioned on the "Why no mailbox size in Akka > 2?" blog, the collections backing some mailbox implementations have O(n) > time implementations of .size(), we are experimenting on attaching counters > to message queues to overcome this problem and have a constant time > implementation of .numberOfMessages, but that shouldn't be a limitation to > monitoring mailboxes. I'll be writing more docs about how we do metrics > collection and update you guys as soon as it is available in our site. Oh! > we also have processing-time and time-in-mailbox metrics! > > 3) Application-specific metrics >> > > Kamon also provides you with tracing facilities [4], the current > documentation available on the site just talks about passing a TraceContext > around, but that's not the only thing we do with traces, once a trace is > finished we generate elapsed time metrics for the entire trace and Kamon > can also automatically recognize when a request starts/ends for Spray > applications (Play! support is almost there!). Kamon also has custom > metrics support but it's API needs some more love before we spread the > word. I can't show our NewRelic custom metrics dashboard since they contain > some sensitive information, but here is a overview page for one of our > applications [5] that is generated with the data we get out of Kamon. > > Currently Kamon can do a lot more than what you can see on our site, just > give us some time and the info will be there :), feel free to ask anything > you need on the mailing list [6], we are here to help. I hope you find this > information useful, best regards! > > > [1] https://groups.google.com/forum/#!forum/typesafe-console > [2] http://kamon.io > [3] https://github.com/kamon-io/Kamon/issues/24 > [4] http://kamon.io/core/tracing/ > [5] http://i.imgur.com/hq3TeeW.png > [6] https://groups.google.com/forum/#!forum/kamon-user > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to a topic in the > Google Groups "Akka User List" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/akka-user/lZ3UjPoy1go/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
