Hi Mario, Thanks for the reproducer. Since you don't use PullMode (it was introduced in 2.3.x I think, but it is not used by default even there) I assume that an OP_READ should be registered on the selector, so this might be a bug.
Can you verify that this exists on 2.3.5? -Endre On Thu, Aug 28, 2014 at 4:24 PM, Mario Camou <[email protected]> wrote: > Hi Roland, Endre and the rest :), > > Here is an excerpted/simplified version of what we're doing (hopefully I > didn't mess things up too badly during sanitization): > > class ConnectionListener(endpoint: InetSocketAddress) extends Actor with > ActorLogging { > import akka.io.{ IO, Tcp } > > implicit val system = context.system > IO(Tcp) ! Tcp.Bind(self, endpoint) > > log.info("listening on {}", endpoint) > > override def receive = { > case Tcp.Connected(remote, _) ⇒ handleConnected(remote, sender()) > case Terminated(handler) ⇒ log info s"Handler terminated: > $handler" > } > > // The pipeline is built using Google Protobuf > private def pipelineStages = > new ProtoBufStage >> > new LengthFieldFrame(maxSize = maxFrameSize) >> > new TcpReadWriteAdapter > > private def handleConnected(remote: InetSocketAddress, connection: > ActorRef): Unit = { > log info s"Client connected from IP $remote" > > val init = TcpPipelineHandler withLogger (log, pipelineStages) > > val handler = context.actorOf(Props(classOf[ConnectionHandler], init, > connection)), > s"gateway-connection-handler-${UUID.randomUUID().toString}" > ) > > val pipeline = context.actorOf(TcpPipelineHandler.props(init, > connection, handler).withDeploy(Deploy.local)) > connection ! Tcp.Register(pipeline) > context watch handler > } > } > > class ConnectionHandler( > init: Init[WithinActorContext, ServerMessage, ClientMessage], > connection: ActorRef) extends Actor with ActorLogging { > > var pipeline: Option[ActorRef] = None > > def receive = LoggingReceive { > case init.Event(c: ClientMessage) ⇒ > pipeline = Some(sender) > processMessage(c) > > case s: ServerMessage ⇒ pipeline foreach { _ ! init.Command(s) > } > case _: Tcp.ConnectionClosed ⇒ handleClosedConnection > case Terminated(`connection`) ⇒ handleClosedConnection > } > > private def processMessage(c: ClientMessage): Unit = { > log info s"Processing message $c" > } > > private def handleClosedConnection: Unit = { > log info s"Client has disconnected" > } > } > > What we see is, the ConnectionHandler is waiting for messages from the > client (via the pipeline - this was started in 2.2 when pipelines were a > thing) or to send to the client. The thing is, if the server isn't sending > anything and the client dies in an unexpected fashion (kill -9 or power > loss), we never get the Tcp.ConnectionClosed message. > > Endre mentioned SO_KEEPALIVE. I've already tried adding options = > List(KeepAlive(true))to the Tcp.Bind() message, however, according to the > TCP specification: > > *Keep-alive packets MUST only be sent when no data or acknowledgement > packets have been received for the connection within an interval. This > interval MUST be configurable and MUST default to no less than two hours.* > > > 2 hours is much too long for our application, we actually need it within a > few minutes at the latest. The problem is that lowering this at the OS > level affects all processes running on the server, which I've found that > can break other stuff. > > As I said, what I find most weird about this, is that the socket IS > actually being closed (checking by netstat -an) but Akka is not getting a > notification. > > One more thing. If you do it over localhost, it will actually work fine, > since localhost DOES correctly detect the other process dying. It's only > when you are running on 2 separate hosts that the problem arises. > > Hoping we can sort this out, > -Mario. > > -- > I want to change the world but they won't give me the source code. > > > On Thu, Aug 28, 2014 at 11:42 AM, Akka Team <[email protected]> > wrote: > >> >> >> >> On Wed, Aug 27, 2014 at 9:25 PM, Roland Kuhn <[email protected]> wrote: >> >>> Hi Mario, >>> >>> re-reading the docs >>> <http://docs.oracle.com/javase/8/docs/api/java/nio/channels/SelectionKey.html> >>> it >>> does sound like the channel should be woken up if it has either OP_READ or >>> OP_WRITE registered. The latter only is registered when you try to write >>> something without success, but the former should always be registered >>> unless you suspended reading (or did not resume it for pollMode). Are you >>> certain that the socket is ready for reading when you make this test? >>> >> >> Fair point, Roland. >> >> -Endre >> >> >>> >>> Regards, >>> >>> Roland >>> >>> 27 aug 2014 kl. 21:12 skrev Mario Camou <[email protected]>: >>> >>> Hi Roland, >>> >>> It's not happening... >>> >>> If you do it through localhost it works fine, but in the case where the >>> client is on a different machine from the Akka server, it doesn't happen. >>> I've tried with kill -9 and also pulling the power plug on the client box. >>> >>> As I said before, when doing a netstat -an on the server side the socket >>> eventually disappears, but Akka doesn't find out about it until you >>> actually try to write data to the socket. I've been Googling around and >>> diving into the akka.io source code, and all I've found is people >>> saying that the Java ServerChannel doesn't tell you when the connection is >>> closed until you try to send something (which sounds like a really bad lack >>> in Java if that is true). >>> >>> For the moment I'm starting to do the same thing I did in the other >>> project back when this thread started: defining a new "Heartbeat" message >>> type and having a Scheduler periodically (I'm thinking every 30 seconds) >>> send it, but everybody else on my team to whome I've talked about this >>> finds it really strange. >>> >>> Any ideas? Of course, I'm always open to it being a case of PEBKAC.... >>> >>> Cheers, >>> -Mario. >>> >>> >>> -Mario. >>> >>> -- >>> I want to change the world but they won't give me the source code. >>> >>> >>> On Wed, Aug 27, 2014 at 5:23 PM, Roland Kuhn <[email protected]> wrote: >>> >>>> Hi Mario, >>>> >>>> just to get this straight: you have a connection from process A to >>>> process B, possibly on different machines. You kill -9 process A and >>>> process B is an Akka application using IO actors. If that is right then >>>> there should definitely be a ConnectionClosed event if at the time of the >>>> kill a TCP connection was indeed established. >>>> >>>> Regards, >>>> >>>> Roland >>>> >>>> 26 aug 2014 kl. 21:17 skrev Mario Camou <[email protected]>: >>>> >>>> Hi again, >>>> >>>> Sorry to reopen this thread after so long, but I'm in another project >>>> with a similar problem. >>>> >>>> One thing I'm noticing is that, even after the OS closes the connection >>>> (i.e., it doesn't appear in netstat -an) my handler isn't getting any of >>>> the Tcp.ConnectionClosed messages. Is that normal? It looks bizarre to me. >>>> >>>> What I'm doing is, starting a client that makes a connection, then >>>> doing a kill -9 to simulate the client crashing (or the client machine >>>> turning off). The socket stays there for a while (again checking via >>>> netstat -an) but eventually the OS closes it. Akka is not sending any >>>> message to indicate that the socket has been closed. >>>> >>>> Any ideas short of doing what Roiland suggested earlier? (i.e., >>>> modifying the protocol to periodically send some data). >>>> >>>> Thanks, >>>> -Mario. >>>> >>>> On Friday, December 27, 2013 9:26:46 PM UTC+1, Mario Camou wrote: >>>>> >>>>> Hi, >>>>> >>>>> I’m using Akka I/O and am trying to set the SO_TIMEOUT socket value. >>>>> >>>>> Looking at the docs, I see that the Tcp.Bind message receives a >>>>> Traversable[SocketOption]. However, looking at the docs for >>>>> akka.io.Inet.SocketOption, the only subclasses are Broadcast, KeepAlive, >>>>> OOBInline, ReceiveBufferSize, ReuseAddress, SendBufferSize, TcpNoDelay and >>>>> TrafficClass. The superclass for the only SoTimeout class I see >>>>> (akka.actor.IO.SoTimeout) is akka.actor.IO.SocketOption which I assume is >>>>> part of the old (Iteratee-based) Akka I/O. >>>>> >>>>> So, how to set the SO_TIMEOUT using the new Akka I/O? Also, how to >>>>> reset the SO_TIMEOUT once the socket has been opened? >>>>> >>>>> In my use case I need to set a relatively short SO_TIMEOUT value when >>>>> the socket is first connected, and bump it up a bit after receiving the >>>>> first actual message. >>>>> >>>>> Thanks, >>>>> -Mario >>>>> — >>>>> I want to change the world, but they won’t give me the source code >>>>> >>>>> >>>> -- >>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>> >>>>>>>>>> Check the FAQ: >>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>> >>>>>>>>>> Search the archives: >>>> https://groups.google.com/group/akka-user >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "Akka User List" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at http://groups.google.com/group/akka-user. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>>> >>>> >>>> *Dr. Roland Kuhn* >>>> *Akka Tech Lead* >>>> Typesafe <http://typesafe.com/> – Reactive apps on the JVM. >>>> twitter: @rolandkuhn >>>> <http://twitter.com/#!/rolandkuhn> >>>> >>>> >>>> -- >>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>> >>>>>>>>>> Check the FAQ: >>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>> >>>>>>>>>> Search the archives: >>>> https://groups.google.com/group/akka-user >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "Akka User List" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at http://groups.google.com/group/akka-user. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> -- >>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>> >>>>>>>>>> Check the FAQ: >>> http://doc.akka.io/docs/akka/current/additional/faq.html >>> >>>>>>>>>> Search the archives: >>> https://groups.google.com/group/akka-user >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Akka User List" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/akka-user. >>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> >>> >>> *Dr. Roland Kuhn* >>> *Akka Tech Lead* >>> Typesafe <http://typesafe.com/> – Reactive apps on the JVM. >>> twitter: @rolandkuhn >>> <http://twitter.com/#!/rolandkuhn> >>> >>> -- >>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>> >>>>>>>>>> Check the FAQ: >>> http://doc.akka.io/docs/akka/current/additional/faq.html >>> >>>>>>>>>> Search the archives: >>> https://groups.google.com/group/akka-user >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Akka User List" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/akka-user. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Akka Team >> Typesafe - The software stack for applications that scale >> Blog: letitcrash.com >> Twitter: @akkateam >> >> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Akka Team Typesafe - The software stack for applications that scale Blog: letitcrash.com Twitter: @akkateam -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
