Hi,
I have a problem in my current project and I cannot pinpoint the exact
cause. I implemented an ETL and what I have is a controller actor
dispatching files to process to a router (SmallestMailboxPool with one file
being one message), which dispatch them to file processors. In the end, the
processors notifies back the controller of the outcome. My datasource
(HikariCP) is held within an actor attached to the guardian so it restarts
when necessary.
My controller and processors actors are attached to a *pinned dispatcher*
because they use JDBC to insert files data into a DB. What I'm trying to do
is handle failures of the database for the file processors. I used a
Circuit Breaker like so :
File processors (simplified and without logging) :
override def preStart(): Unit = {
breaker = new CircuitBreaker(context.system.scheduler,
maxFailures = 1,
callTimeout = 30 seconds,
resetTimeout = 10 seconds)(context.dispatcher)
}
[...]
case Forwarded(file) =>
try{
breaker.withSyncCircuitBreaker {
val file = doParsingAndValidation()
file match {
case Success(parsedFile) =>
using((connectionFactory ? GetConnection).mapTo[Connection
].waitForResult) { conn =>
conn.setAutoCommit(false)
doJDBCStuff()
conn.commit()
controller ! Processed(parsedFile)
}
case Failure(err) =>
controller ! Malformed(file, err)
}
}
} catch {
case ex: Exception => context.system.scheduler.scheduleOnce(5
seconds, self, Forwarded(file))(context.dispatcher)
}
So basically when the database is down the breaker is gonna switch to Open
at first exception (therefore resending message to self). After 10 seconds
it's gonna be in HalfOpen state and allow only one file through (while all
others will trigger the CircuitBreakerOpenException and get resent). If the
allowed one works it's all good, otherwise the breaker is restored to Open
state again.
Anyway, the logic works in itself : When I stop the database it loops
through messages every 5 seconds to fail each time and resends them. *My
real problem is that some files aren't processed !*
I looked at my logs and ran YourKit to find out that processors process
files normally until a crash occurs, at which point only some of them
(seems random) switch into Open state. Eventually these ones switch to
Closed and resume their processing but the ones that didn't go into Open
state just keep waiting ! What I get from this is that some actors aren't
crashing (because they never go into Open state and I don't have any logs
for them) and are keeping files in their mailbox without doing anything.
Here's a sample logs <http://pastebin.com/HGciueNq> file (here only one
processors switched state)
So I guess my logic is flawed and I missed something ? Thanks for any help.
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
