Roland, I certainly feel like moving towards bounded queues as the rule rather than the exception is the way to go. Setting reasonable bounds is a task requiring deep knowledge of the application domain, actor/dispatcher dynamics and running platforms (just to name a few), and it is difficult to propose a checklist that will guide people through the optimal bounds for their case, but I guess that with time and more information taken out of live systems the community will develop a better understanding on this topics and it will be a lot easier and common to set bounds. Maybe introducing bounds in terms of low-latency and performance might rise more interest in the topic.
Your proposal about firing monitoring events upon reaching some mailbox size watermarks seems pretty neat, we will certainly consider adding some of that goodness into Kamon :), is there any ticket tracking that feature in assembla/github?, maybe we can give a hand on adding this feature to akka as well. Patrick, thanks for mentioning LoggingMailbox, it certainly looks pretty nice and people can use it right away, will keep it on my toolbox. have a nice day guys, best regards! On Thu, Mar 27, 2014 at 5:43 AM, Patrik Nordwall <[email protected]>wrote: > The LoggingMailbox <https://gist.github.com/patriknw/5946678> has helped > me debugging bottlenecks many times. > /Patrik > > > > On Thu, Mar 27, 2014 at 8:22 AM, Roland Kuhn <[email protected]> wrote: > >> Thanks for sharing, this will be very useful for people searching this >> forum! >> >> Regards, >> >> Roland >> >> 26 mar 2014 kl. 20:55 skrev Edward Steel <[email protected]>: >> >> We're using a combination of things at work. Sensu when alerts are needed >> and graphite/statsd for collecting, displaying, monitoring metrics. >> Statsd's pretty powerful, and graphite does the job as far as powering >> dashboards (there are some alternatives though). >> >> More specifically: >> >> 1) java.lang.management._ + statsd -- an actor schedules itself at the >> rate we need to report heap size and gc time. We also monitor sub-jvm >> system stats in statsd using diamond. >> >> 2) Statsd, ad-hoc as needed. Mailbox size isn't really accessible, and >> trying to monitor it is discouraged ( >> http://letitcrash.com/post/17707262394/why-no-mailboxsize-in-akka-2). >> Instead you can monitor time between the message being sent from one actor >> and worked on by another (by injecting timestamps in the message), or you >> can monitor all requests and see if there's a time lag between reducing the >> load on the sender and a corresponding reduction appearing on the receiver >> (while it chews through its mailbox). However if you monitor message count >> and various timings, especially if you measure timing for specific >> components and compare them, you can generally see bottlenecks pretty >> clearly. >> >> 3) Statsd again. We use it a ton. And since the client is nio/udp and >> supports sampling, we never see a problem with overhead. The worst you'll >> see is strange graphs if you overload graphite itself. >> >> You're very welcome to try our statsd client. It's on github at >> github.com/hootsuite/statsd-client . It's pretty much a wrapper around >> etsy's client with some scala niceties and typesafe config. It's missing >> some examples but is pretty straightforward (see >> https://github.com/hootsuite/statsd-client/blob/master/src/main/scala/com/hootsuite/statsd/StatsdReporting.scala >> ) >> >> Would always love to hear of better solutions! I've heard vague rumours >> typesafe are working on something for production. >> >> Cheers, >> Edd >> >> >> On 26 March 2014 11:48, Chris Toomey <[email protected]> wrote: >> >>> Are any of you folks in this group running production Akka systems, and >>> if so, which of the following classes of metrics are you collecting and >>> monitoring? >>> >>> 1) Generic JVM metrics >>> >>> 2) Generic Akka metrics (mailbox sizes, message rates, etc.) >>> >>> 3) Application-specific metrics >>> >>> If you're collecting/monitoring generic Akka metrics, how are you doing >>> it? >>> >>> If you're not, is it because 1) and 3) are sufficient, or because you've >>> not found a good way to do 2)? >>> >>> Thanks for any guidance. >>> >>> Chris >>> >>> >>> On Tuesday, March 25, 2014 7:25:38 AM UTC-7, Chris Toomey wrote: >>>> >>>> We're developing our first Akka application and are thus new to the >>>> larger ecosystem. I've spent the last hour or so searching for Akka >>>> monitoring tools but haven't come across much of anything as far as >>>> production monitoring tools. I see that Typesafe Console is aimed at >>>> development time and not recommended for production usage, and I've seen a >>>> few things on github, but that's about it. >>>> >>>> What tools are folks using or would you recommend using to monitor >>>> production Akka applications for Akka-specific metrics like message rates, >>>> mailbox sizes, actor counts, etc.? >>>> >>>> thx, >>>> Chris >>>> >>>> >>> -- >>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>> >>>>>>>>>> Check the FAQ: >>> http://doc.akka.io/docs/akka/current/additional/faq.html >>> >>>>>>>>>> Search the archives: >>> https://groups.google.com/group/akka-user >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Akka User List" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/akka-user. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> >> >> >> >> *Dr. Roland Kuhn* >> *Akka Tech Lead* >> Typesafe <http://typesafe.com/> - Reactive apps on the JVM. >> twitter: @rolandkuhn >> <http://twitter.com/#!/rolandkuhn> >> >> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > > Patrik Nordwall > Typesafe <http://typesafe.com/> - Reactive apps on the JVM > Twitter: @patriknw > JOIN US. REGISTER TODAY! <http://www.scaladays.org/> > Scala <http://www.scaladays.org/> > Days <http://www.scaladays.org/> > June 16th-18th, <http://www.scaladays.org/> > Berlin <http://www.scaladays.org/> > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
