Hi Jeff,
On Thursday, 16 January 2014 at 04:34, Jeff Crilly wrote: > Thx for the feedback! > > Re "ouliving"... Afaict, the app just stops getting connections, so its hard > to tell there is a problem from the "inside". (Fwiw, this app also connects > to other systems which might be down; hence the app needs to deal with > failure and recover those connections -- simply exiting the app is not an > option in those scenarios.) > So that the app stops getting connections shouldn’t be related to this bug. This bug is about the Netty thread pools not being properly shut down when the Akka actor system is shut down. Have you, or your ops guys, taken any kind of diagnostics, i.e. thread/heap dump, of the system when it fails? > As for "periodically"... good question.. one thing i'm asking for is the > "rate of leak". It is not on the order of "minutes", more like "days", and > may actually be "weeks" (I know this sounds odd, but i dont have direct > access to the production systems and need to go through a couple layers of IT > folks in different geos and timezones.) > That doesn’t sound odd, just cumbersome and way to common. ;) B/ > > Anyhow, I just wanted to check on the options/alternatives that might not be > obvious. I'm a fan of getting on the latest stable (but not bleeding edge). > > -jeff > > > > On Wednesday, January 15, 2014 2:25:09 AM UTC-8, Björn Antonsson wrote: > > Hi Jeff, > > > > > > On Tuesday, 14 January 2014 at 03:54, Jeff Crilly wrote: > > > > > We've had some recent "production issues" and it looks like this is the > > > exact issue biting us; (this results in a need to restart our appservers > > > rather periodically.) > > > The app is on akka 2.1.4. > > > > > > > I’m just curious as to what you mean by periodically. Also, why does you > > app server outlive the actor systems created inside it? You shouldn’t need > > to start/stop actor systems frequently. > > > > > I scanned the akka 2.1.x to 2.2.x migration guide a while ago (and also > > > just now), and it didn't seem like a "drop in library" change. (Am I > > > missing something?) > > > > > > > You are right that there are a number of changes between 2.1 and 2.2 that > > might make upgrading (depending on you application) include some rewriting > > of you application. The easiest thing is probably to try to build against > > 2.2.x. > > > > > Any other thoughts on how we might mitigate the issue alternatively w/o > > > 2.2 upgrade? (Thought i'd ask anyhow.) > > > > > > (Or do we.. ahem.. need to bite the bullet and do the upgrade? This is > > > an application that is undergoing minor bug fixes, but not major changes, > > > hence rolling a full regression etc is not quite in the plan; we were > > > looking for a narrow fix/solution rather than (afaict) refactoring the > > > akka usage.) > > > > > > > I think that upgrading to 2.2 is the only available solution for the thread > > leakage right now, unless you keep the same actor system around in the app > > server. > > > > B/ > > > > > thx! > > > > > > On Sunday, June 23, 2013 9:55:35 AM UTC-7, √ wrote: > > > > I just want to clarify that there is no ship date for a 2.1.5 release > > > > at this date. > > > > > > > > Cheers, > > > > √ > > > > > > > > > > > > > > > > On Sun, Jun 23, 2013 at 12:44 PM, Matei Zaharia <[email protected]> > > > > wrote: > > > > > Awesome, thanks! > > > > > > > > > > Matei > > > > > > > > > > On Jun 23, 2013, at 12:39 PM, √iktor Ҡlang <[email protected]> > > > > > wrote: > > > > > > https://github.com/akka/akka/pull/1559/files > > > > > > > > > > > > > > > > > > On Sat, Jun 22, 2013 at 7:16 PM, Matei Zaharia > > > > > > <[email protected]> wrote: > > > > > > > Yes, it was in 2.1.4, and it looks like it's been fixed in > > > > > > > https://www.assembla.com/spaces/akka/tickets/3436. I'm curious, > > > > > > > will the fix be merged into 2.1.5, or should I upgrade to 2.2? > > > > > > > > > > > > > > Matei > > > > > > > > > > > > > > > > > > > > > On Wednesday, June 12, 2013 8:07:53 AM UTC-4, Patrik Nordwall > > > > > > > wrote: > > > > > > > > Thanks for reporting. We will look into it. > > > > > > > > Is it akka 2.1.4? Otherwise, please try with that version. > > > > > > > > /Patrik > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Jun 9, 2013 at 1:07 AM, Matei Zaharia > > > > > > > > <[email protected]> wrote: > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > I've noticed that in Akka 2.1.x, setting the "daemonic" flag > > > > > > > > > to true no longer ensures that all of Akka's threads are > > > > > > > > > daemons. Instead, if you register any remote actors, there > > > > > > > > > are some leftover threads like this: > > > > > > > > > > > > > > > > > > "New I/O server boss #18" prio=10 tid=0x00007f83949eb000 > > > > > > > > > nid=0x5fb runnable [0x00007f836d0a8000] > > > > > > > > > java.lang.Thread.State: RUNNABLE > > > > > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > > > > > > > > > at > > > > > > > > > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228) > > > > > > > > > at > > > > > > > > > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81) > > > > > > > > > at > > > > > > > > > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > > > > > > > > > - locked <0x00000000feae8170> (a sun.nio.ch.Util$2) > > > > > > > > > - locked <0x00000000feae8180> (a > > > > > > > > > java.util.Collections$UnmodifiableSet) > > > > > > > > > - locked <0x00000000feae8128> (a > > > > > > > > > sun.nio.ch.EPollSelectorImpl) > > > > > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > > > > > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) > > > > > > > > > at > > > > > > > > > org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163) > > > > > > > > > at > > > > > > > > > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206) > > > > > > > > > at > > > > > > > > > org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42) > > > > > > > > > at > > > > > > > > > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > > > > > > > > > at > > > > > > > > > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > > > > > > > > > at > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > > > > > > > > > at > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > > > > > > > > > at java.lang.Thread.run(Thread.java:722) > > > > > > > > > > > > > > > > > > "New I/O worker #17" prio=10 tid=0x00007f83949d9000 nid=0x5fa > > > > > > > > > runnable [0x00007f836d1a9000] > > > > > > > > > java.lang.Thread.State: RUNNABLE > > > > > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > > > > > > > > > at > > > > > > > > > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228) > > > > > > > > > at > > > > > > > > > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81) > > > > > > > > > at > > > > > > > > > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > > > > > > > > > - locked <0x00000000feb200e8> (a sun.nio.ch.Util$2) > > > > > > > > > - locked <0x00000000feb200f8> (a > > > > > > > > > java.util.Collections$UnmodifiableSet) > > > > > > > > > - locked <0x00000000feb200a0> (a > > > > > > > > > sun.nio.ch.EPollSelectorImpl) > > > > > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > > > > > > > > > at > > > > > > > > > org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64) > > > > > > > > > at > > > > > > > > > org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409) > > > > > > > > > at > > > > > > > > > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206) > > > > > > > > > at > > > > > > > > > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90) > > > > > > > > > at > > > > > > > > > org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) > > > > > > > > > at > > > > > > > > > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > > > > > > > > > at > > > > > > > > > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > > > > > > > > > at > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > > > > > > > > > at > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > > > > > > > > > at java.lang.Thread.run(Thread.java:722) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Calling ActorSystem.shutdown() does shut these down but, I'd > > > > > > > > > like them to be daemons even if you don't call shutdown(). > > > > > > > > > This seems to have changed from Akka 2.0.x. Is this a known > > > > > > > > > problem or an intended behavior? I've seen one thread related > > > > > > > > > to it here: > > > > > > > > > https://www.assembla.com/spaces/akka/tickets/2733#/activity/ticket: > > > > > > > > > but that seemed to be due to shutdown() not shutting things > > > > > > > > > down, whereas I'd like these threads to be daemons in the > > > > > > > > > first place. Is there any known workaround for it? > > > > > > > > > > > > > > > > > > -- > > > > > > > > > >>>>>>>>>> Read the docs: http://akka.io/docs/ > > > > > > > > > >>>>>>>>>> Check the FAQ: http://akka.io/faq/ > > > > > > > > > >>>>>>>>>> Search the archives: > > > > > > > > > >>>>>>>>>> https://groups.google.com/group/akka-user > > > > > > > > > --- > > > > > > > > > You received this message because you are subscribed to the > > > > > > > > > Google Groups "Akka User List" group. > > > > > > > > > To unsubscribe from this group and stop receiving emails from > > > > > > > > > it, send an email to [email protected]. > > > > > > > > > To post to this group, send email to [email protected]. > > > > > > > > > > > > > > > > > > Visit this group at > > > > > > > > > http://groups.google.com/group/akka-user?hl=en. > > > > > > > > > For more options, visit > > > > > > > > > https://groups.google.com/groups/opt_out. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > Patrik Nordwall > > > > > > > > Typesafe (http://typesafe.com/) - The software stack for > > > > > > > > applications that scale > > > > > > > > Twitter: @patriknw > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > >>>>>>>>>> Read the docs: http://akka.io/docs/ > > > > > > > >>>>>>>>>> Check the FAQ: http://akka.io/faq/ > > > > > > > >>>>>>>>>> Search the archives: > > > > > > > >>>>>>>>>> https://groups.google.com/group/akka-user > > > > > > > --- > > > > > > > You received this message because you are subscribed to the > > > > > > > Google Groups "Akka User List" group. > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > send an email to [email protected]. > > > > > > > To post to this group, send email to [email protected]. > > > > > > > Visit this group at http://groups.google.com/group/akka-user. > > > > > > > > > > > > > > For more options, visit https://groups.google.com/groups/opt_out. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Viktor Klang > > > > > > Director of Engineering > > > > > > Typesafe (http://www.typesafe.com/) > > > > > > > > > > > > Twitter: @viktorklang > > > > > > > > > > > > -- > > > > > > >>>>>>>>>> Read the docs: http://akka.io/docs/ > > > > > > >>>>>>>>>> Check the FAQ: http://akka.io/faq/ > > > > > > >>>>>>>>>> Search the archives: > > > > > > >>>>>>>>>> https://groups.google.com/group/akka-user > > > > > > --- > > > > > > You received this message because you are subscribed to a topic in > > > > > > the Google Groups "Akka User List" group. > > > > > > To unsubscribe from this topic, visit > > > > > > https://groups.google.com/d/topic/akka-user/tLBZu5Sr0vU/unsubscribe. > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > [email protected]. > > > > > > > > > > > > To post to this group, send email to [email protected]. > > > > > > Visit this group at http://groups.google.com/group/akka-user. > > > > > > For more options, visit https://groups.google.com/groups/opt_out. > > > > > > > > > > > > > > > > > > > > > > -- > > > > > >>>>>>>>>> Read the docs: http://akka.io/docs/ > > > > > >>>>>>>>>> Check the FAQ: http://akka.io/faq/ > > > > > >>>>>>>>>> Search the archives: > > > > > >>>>>>>>>> https://groups.google.com/group/akka-user > > > > > --- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "Akka User List" group. > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > send an email to [email protected]. > > > > > To post to this group, send email to [email protected]. > > > > > Visit this group at http://groups.google.com/group/akka-user. > > > > > For more options, visit https://groups.google.com/groups/opt_out. > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Viktor Klang > > > > Director of Engineering > > > > Typesafe (http://www.typesafe.com/) > > > > > > > > Twitter: @viktorklang > > > > > > -- > > > >>>>>>>>>> Read the docs: http://akka.io/docs/ > > > >>>>>>>>>> Check the FAQ: http://akka.io/faq/ > > > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > > > --- > > > You received this message because you are subscribed to the Google Groups > > > "Akka User List" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [email protected] (javascript:). > > > To post to this group, send email to [email protected] > > > (javascript:). > > > Visit this group at http://groups.google.com/group/akka-user. > > > For more options, visit https://groups.google.com/groups/opt_out. > > > > -- > > Björn Antonsson > > Typesafe (http://typesafe.com/) – Reactive Apps on the JVM > > > > twitter: @bantonsson (http://twitter.com/#!/bantonsson) > > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: http://akka.io/faq/ > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > (mailto:[email protected]). > To post to this group, send email to [email protected] > (mailto:[email protected]). > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/groups/opt_out. -- Björn Antonsson Typesafe (http://typesafe.com/) – Reactive Apps on the JVM twitter: @bantonsson (http://twitter.com/#!/bantonsson) -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: http://akka.io/faq/ >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/groups/opt_out.
