Hi all, We are using Spray IO (with the wandoulabs WebSockets layer) on really old RHEL5 boxes in our QA environments.
Bizarrely, our server beta release was working fine on one box, but failing to write messages on another, despite the kernels and software versions being identical. Clients were able to connect to the server, but as soon as the server started to write to the socket, we got this sort of thing: 19 Mar 15 10:56:55.542 HttpServerConnectionakka://MDES/user/IO-UHTTP/listener-0/0 [ MDES-akka.actor.default-dispatcher-4] WARN - CommandFailed for Tcp.Write text frame: <garbled bytes> ... 19 Mar 15 10:56:55.543 HttpServerConnectionakka://MDES/user/IO-UHTTP/listener-0/0 [ MDES-akka.actor.default-dispatcher-4] WARN - event pipeline: dropped CommandFailed(Write(ByteString(<lots of numbers>),NoAck(null))) The boxes are running "Red Hat Enterprise Linux Server release 5.8 (Tikanga)" with 2.6.18-308.el5 on x86_64 cores. We're using scala 2.11.5, Java 1.6.0_40 and Akka 2.3.8 / Spray-IO 1.3.2. We spotted that the kernel parameters were different on the boxes, this being the diff: net.core.rmem_default = 262144 net.core.rmem_max = 16777216 net.core.wmem_default = 262144 net.core.wmem_max = 16777216 net.ipv4.tcp_rmem = 4096 4194304 16777216 net.ipv4.tcp_sack = 1 net.ipv4.tcp_timestamps = 1 net.ipv4.tcp_window_scaling = 1 net.ipv4.tcp_wmem = 4096 4194304 16777216 and by using those parameters on all boxes the problems went away. However, we have no control over setting these parameters on PROD boxes so we need a workaround that works without them. But this is extremely concerning, why was the failure happening because of kernel parameters? Is this a bug in NIO, Spray IO, or Spray-WebSockets? Wandoulabs aren't doing anything unusual as you can see https://github.com/wandoulabs/spray-websocket/blob/master/spray-websocket/src/main/scala/spray/can/server/UpgradableHttpListener.scala Most importantly, we need a workaround... does anybody have any suggestions? The current theory is that the default kernel buffer size is too low to accept the outbound WebSocket frames. On the failing boxes (which we can't change), this is $ cat /proc/sys/net/ipv4/tcp_wmem 4096 16384 4194304 and our messages are a few kb each of JSON. Best regards, Sam -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
