I've got an application that is using akka and I'm seeing some really
strange behavior. I'm assuming this is something dumb that I'm doing in my
configuration.
The setup is
CentOS 6.7
Java 1.8.0_71-b15
Scala 2.11.7
Akka 2.4.2
Here is what I'm seeing. First noticed the issue because the application
hit the system thread limit today and crashed (CentOS defaults to 1024
processes/threads per user). That seemed very odd to me since it is pretty
lightweight and there is only one actor in the application. So I started
monitoring it. From the Java side, it appears that nothing is amiss. Using
jstack I see the expected number of threads and nothing grows over time.
However, watching the process on the OS side, I see a fairly rapidly
growing number of threads being created. After about 8 hours, I'm up to
around 600 native threads. It seems like threads are being created and not
being destroyed, but Java is losing track of them somehow.
I've got some other akka applications with the same basic stack that are
not exhibiting this behavior. That leads me to conclude that either my
configuration on this particular service is just wrong or my configuration
is tickling a bug somewhere.
Here is what seems to be the relevant config pieces:
I'm creating a config for my actor system so that it will use a really
basic custom mailbox I wrote to make sure messages get ordered:
private[this]
val actorConfig = ConfigFactory.parseString("""
prio-dispatcher {
mailbox-type = "com.company.server.PriorityMailbox"
}
""").withFallback(conf)
Then I make my actor system with this config
private[this]
val system = ActorSystem("mySystem", actorConfig)
Mailbox is here:
class PriorityMailbox(settings: ActorSystem.Settings, config: Config)
extends UnboundedStablePriorityMailbox(PriorityMailbox.priority)
I create my actor with:
private[this]
val priorityActor = system.actorOf(PriorityActor.props(db).withDispatcher(
"prio-dispatcher"))
The only other thing I do that in anyway really interacts with the actor
system at a thread/scheduling level is in the preStart for the
priorityActor, where I set up a schedule to send a message:
override
def preStart()
{
import context.dispatcher
context.system.scheduler.schedule(1.5 minutes, 1.5 minutes, self,
NEWCALLBACK)
}
And that's basically it. And yet, I get what appears to be an unbounded
number of OS threads over time.
I tried tweaking my actorConfig to a fixed-sized thread pool
private[this]
val actorConfig = ConfigFactory.parseString("""
prio-dispatcher {
mailbox-type = "com.company.server.PriorityMailbox"
type = PinnedDispatcher
executor = "thread-pool-executor"
thread-pool-executor {
fixed-pool-size = 2
}
throughput = 1
}
""").withFallback(conf)
And that did not change anything.
So here is what else I've been noticing. When I use jstack to print out the
current threads I usually see something like this at the top
"mySystem-prio-dispatcher-395" #3305 prio=5 os_prio=0
tid=0x00007f64e1642000 nid=0x6c49 waiting on condition [0x00007f6668fd3000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000080177350> (a akka.dispatch.
ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:
2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.
java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(
ForkJoinWorkerThread.java:107)
This isn't unexpected, since you expect a ForkJoinPool to clean-up old
threads and spawn new ones over time. But it does appear to be the only
thing in the system that is actually spawning new threads. Which would seem
to indicate to me that this the most likely source of the thread leak. And
that number after the thread, which I'm pretty sure is basically a running
count of the total number of threads that have been made in the process,
keeps going up.
Anyone have any thoughts on what I might be doing wrong or places I can
look for more information.
cheers,
Jesse C
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
