On Thu, Mar 19, 2015 at 12:42 PM, Sam Halliday <[email protected]> wrote:
> Hi all, > > We are using Spray IO (with the wandoulabs WebSockets layer) on really old > RHEL5 boxes in our QA environments. > > Bizarrely, our server beta release was working fine on one box, but > failing to write messages on another, despite the kernels and software > versions being identical. > > Clients were able to connect to the server, but as soon as the server > started to write to the socket, we got this sort of thing: > > 19 Mar 15 10:56:55.542 > HttpServerConnectionakka://MDES/user/IO-UHTTP/listener-0/0 [ > MDES-akka.actor.default-dispatcher-4] WARN - CommandFailed for Tcp.Write > text frame: <garbled bytes> ... > 19 Mar 15 10:56:55.543 > HttpServerConnectionakka://MDES/user/IO-UHTTP/listener-0/0 [ > MDES-akka.actor.default-dispatcher-4] WARN - event pipeline: dropped > CommandFailed(Write(ByteString(<lots of numbers>),NoAck(null))) > > > The boxes are running "Red Hat Enterprise Linux Server release 5.8 > (Tikanga)" with 2.6.18-308.el5 on x86_64 cores. We're using scala 2.11.5, > Java 1.6.0_40 and Akka 2.3.8 / Spray-IO 1.3.2. > > We spotted that the kernel parameters were different on the boxes, this > being the diff: > > net.core.rmem_default = 262144 > net.core.rmem_max = 16777216 > net.core.wmem_default = 262144 > net.core.wmem_max = 16777216 > net.ipv4.tcp_rmem = 4096 4194304 16777216 > net.ipv4.tcp_sack = 1 > net.ipv4.tcp_timestamps = 1 > net.ipv4.tcp_window_scaling = 1 > net.ipv4.tcp_wmem = 4096 4194304 16777216 > > and by using those parameters on all boxes the problems went away. > > However, we have no control over setting these parameters on PROD boxes so > we need a workaround that works without them. > > But this is extremely concerning, why was the failure happening because of > kernel parameters? Is this a bug in NIO, Spray IO, or Spray-WebSockets? > Wandoulabs aren't doing anything unusual as you can see > https://github.com/wandoulabs/spray-websocket/blob/master/spray-websocket/src/main/scala/spray/can/server/UpgradableHttpListener.scala > > Most importantly, we need a workaround... does anybody have any > suggestions? > > The current theory is that the default kernel buffer size is too low to > accept the outbound WebSocket frames. > That sounds like an explanation. I did some testing with sending large ByteString with akka.io.Tcp. It splits it up in chunks of the size of the configured direct-buffer-size, but no CommandFailed. I'm afraid I don't know why it fails with your kernel settings. You might get a better answer at the Spray mailing list. Regards, Patrik > On the failing boxes (which we can't change), this is > > $ cat /proc/sys/net/ipv4/tcp_wmem > 4096 16384 4194304 > > and our messages are a few kb each of JSON. > > Best regards, > Sam > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Patrik Nordwall Typesafe <http://typesafe.com/> - Reactive apps on the JVM Twitter: @patriknw -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
