Do you say that you have 600 live threads? Have you verified that with
jstack or a profiler?
The count doesn't mean that the threads are alive. Threads are stopped when
idle for some time. E.g. the fork-join pool has an idle timeout of around 2
seconds. For fun you can try to schedule something every second instead of
1.5.minutes.
The thread-pool-executor has a allow-core-timeout setting that you can try
to set to off.
PinnedDispatcher with fixed-pool-size = 2 does not make sense.
On Sat, Mar 19, 2016 at 2:47 AM, Jesse C <crimson.core...@gmail.com> wrote:
> I've got an application that is using akka and I'm seeing some really
> strange behavior. I'm assuming this is something dumb that I'm doing in my
> configuration.
>
> The setup is
> CentOS 6.7
> Java 1.8.0_71-b15
> Scala 2.11.7
> Akka 2.4.2
>
> Here is what I'm seeing. First noticed the issue because the application
> hit the system thread limit today and crashed (CentOS defaults to 1024
> processes/threads per user). That seemed very odd to me since it is pretty
> lightweight and there is only one actor in the application. So I started
> monitoring it. From the Java side, it appears that nothing is amiss. Using
> jstack I see the expected number of threads and nothing grows over time.
> However, watching the process on the OS side, I see a fairly rapidly
> growing number of threads being created. After about 8 hours, I'm up to
> around 600 native threads. It seems like threads are being created and not
> being destroyed, but Java is losing track of them somehow.
>
> I've got some other akka applications with the same basic stack that are
> not exhibiting this behavior. That leads me to conclude that either my
> configuration on this particular service is just wrong or my configuration
> is tickling a bug somewhere.
>
> Here is what seems to be the relevant config pieces:
>
> I'm creating a config for my actor system so that it will use a really
> basic custom mailbox I wrote to make sure messages get ordered:
> private[this]
> val actorConfig = ConfigFactory.parseString("""
> prio-dispatcher {
> mailbox-type = "com.company.server.PriorityMailbox"
> }
> """).withFallback(conf)
>
> Then I make my actor system with this config
>
> private[this]
> val system = ActorSystem("mySystem", actorConfig)
>
>
>
> Mailbox is here:
> class PriorityMailbox(settings: ActorSystem.Settings, config: Config)
> extends UnboundedStablePriorityMailbox(PriorityMailbox.priority)
>
> I create my actor with:
>
> private[this]
> val priorityActor = system.actorOf(PriorityActor.props(db).withDispatcher(
> "prio-dispatcher"))
>
> The only other thing I do that in anyway really interacts with the actor
> system at a thread/scheduling level is in the preStart for the
> priorityActor, where I set up a schedule to send a message:
>
> override
> def preStart()
> {
> import context.dispatcher
> context.system.scheduler.schedule(1.5 minutes, 1.5 minutes, self,
> NEWCALLBACK)
> }
>
> And that's basically it. And yet, I get what appears to be an unbounded
> number of OS threads over time.
>
> I tried tweaking my actorConfig to a fixed-sized thread pool
>
> private[this]
> val actorConfig = ConfigFactory.parseString("""
> prio-dispatcher {
> mailbox-type = "com.company.server.PriorityMailbox"
> type = PinnedDispatcher
> executor = "thread-pool-executor"
> thread-pool-executor {
> fixed-pool-size = 2
> }
> throughput = 1
> }
> """).withFallback(conf)
>
> And that did not change anything.
>
> So here is what else I've been noticing. When I use jstack to print out
> the current threads I usually see something like this at the top
> "mySystem-prio-dispatcher-395" #3305 prio=5 os_prio=0
> tid=0x00007f64e1642000 nid=0x6c49 waiting on condition [0x00007f6668fd3000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0000000080177350> (a akka.dispatch.
> ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:
> 2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.
> java:1979)
> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(
> ForkJoinWorkerThread.java:107)
>
>
> This isn't unexpected, since you expect a ForkJoinPool to clean-up old
> threads and spawn new ones over time. But it does appear to be the only
> thing in the system that is actually spawning new threads. Which would seem
> to indicate to me that this the most likely source of the thread leak. And
> that number after the thread, which I'm pretty sure is basically a running
> count of the total number of threads that have been made in the process,
> keeps going up.
>
>
> Anyone have any thoughts on what I might be doing wrong or places I can
> look for more information.
>
> cheers,
> Jesse C
>
>
> --
> >>>>>>>>>> Read the docs: http://akka.io/docs/
> >>>>>>>>>> Check the FAQ:
> http://doc.akka.io/docs/akka/current/additional/faq.html
> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to akka-user+unsubscr...@googlegroups.com.
> To post to this group, send email to akka-user@googlegroups.com.
> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>
--
Patrik Nordwall
Akka Tech Lead
Lightbend <http://www.lightbend.com/> - Reactive apps on the JVM
Twitter: @patriknw
[image: Lightbend] <http://www.lightbend.com/>
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
