> I have some skeletial client/server code in C. It just needs to morphed to your test case. I can’t see me getting that done today unless I get blocked on what I need to get done.
Hello Kirk, I'm still banging my head trying to understand this latency issue. Did you have time to use your C code to try to reproduce this problem? I'm not a C programmer, but if you are busy I can try to adapt your skeletal client/server C code to the use-case in question. I'm currently clueless and unable to make progress. It happens on MacOS, Linux and Windows so it does not look like a OS-related issue. Looks more like a JVM or CPU issue. Thanks! -JC On Thursday, April 13, 2017 at 1:59:48 AM UTC-5, Kirk Pepperdine wrote: > > > Normally when I run into “can’t scale down” problems in Java you have to > be concerned about methods on the critical path not being hot enough to be > compiled. However I’d give this one a low probability because the knock on > latency is typically 2-3x what you’d see under load. So, this seems some > how connected to a buffer with a timer. Under load you get fill and fire > and of course the scale down is fire on timeout ‘cos you rarely is ever > fill. > > Have you looked at this problem using Wireshark or a packet sniffer in > your network? Another trick is to directly instrument the Socket read, > write methods. You can do that with BCI or simply just hand modify the code > and preload it on the bootstrap class path. > > I have some skeletial client/server code in C. It just needs to morphed to > your test case. I can’t see me getting that done today unless I get blocked > on what I need to get done. > > Kind regards, > Kirk > > On Apr 13, 2017, at 6:45 AM, J Crawford <latency...@mail.com <javascript:>> > wrote: > > Very good idea, Mike. If I only knew C :) I'll try to hire a C coder on > UpWork.com or Elance.com to do that. It shouldn't be hard for someone who > knows C network programming. I hope... > > Thanks! > > -JC > > On Wednesday, April 12, 2017 at 11:37:28 PM UTC-5, mikeb01 wrote: >> >> Rewrite the test in C to eliminate the JVM as the cause of the slowdown? >> >> On 13 April 2017 at 16:31, J Crawford <latency...@mail.com> wrote: >> >>> Ok, this is a total mystery. Tried a bunch of strategies with no luck: >>> >>> 1. Checked the cpu frequency with i7z_64bit. No variance in the >>> frequency. >>> >>> 2. Disabled all power management. No luck. >>> >>> 3. Changed TCP Congestion Control Algorithm. No luck. >>> >>> 4. Set net.ipv4.tcp_slow_start_after_idle to false. No luck. >>> >>> 5. Tested with UDP implementation. No luck. >>> >>> 6. Placed the all sockets in blocking mode just for the heck of it. No >>> luck, same problem. >>> >>> I'm out of pointers now and don't know where to run. This is an >>> important latency problem that I must understand as it affects my trading >>> system. >>> >>> Anyone who has any clue of what might be going on, please throw some >>> light. Also, if you run the provided Server and Client code in your own >>> environment/machine (over localhost/loopback) you will see that it does >>> happen. >>> >>> Thanks! >>> >>> -JC >>> >>> On Wednesday, April 12, 2017 at 10:23:17 PM UTC-5, Todd L. Montgomery >>> wrote: >>>> >>>> The short answer is that no congestion control algorithm is suited for >>>> low latency trading and in all cases, using raw UDP will be better for >>>> latency. Congestion control is about fairness. Latency in trading has >>>> nothing to do with fairness. >>>> >>>> The long answer is that to varying degrees, all congestion control must >>>> operate at high or complete utilization to probe. Those based on loss (all >>>> variants of CUBIC, Reno, etc.) must be operating in congestion avoidance >>>> or >>>> be in slow start. Those based on RTT (Vegas) or RTT/Bottleneck Bandwidth >>>> (BBR) must be probing for more bandwidth to determine change in RTT (as a >>>> "replacement" for loss). >>>> >>>> So, the case of sending only periodically is somewhat antithetical to >>>> the operating point that all congestion control must operate at while >>>> probing. And the reason all appropriate congestion control algorithms I >>>> know of reset upon not operating at high utilization. >>>> >>>> You can think of it this way.... the network can only sustain X >>>> msgs/sec, but X is a (seemingly random) nonlinear function of time. How do >>>> you determine X at any given time without operating at that point? You can >>>> not, that I know of, predict X without operating at X. >>>> >>>> On Wed, Apr 12, 2017 at 6:54 PM, J Crawford <latency...@mail.com> >>>> wrote: >>>> >>>>> Hi Todd, >>>>> >>>>> I'm trying several TCP Congestion algorithms here: westwood, >>>>> highspeed, veno, etc. >>>>> >>>>> No luck so far, but there are many more I haven't tried. I'm using >>>>> this answer to change the TCP congestion algo: >>>>> http://unix.stackexchange.com/a/278217 >>>>> >>>>> Does anyone know what TCP congestion algorithm is the best for >>>>> low-latency? Or the best for the single message scenario I've described? >>>>> *This >>>>> looks like an important configuration for trading, when a single order >>>>> needs to go out after some time and you don't want it to go out at a >>>>> slower >>>>> speed.* >>>>> >>>>> Thanks! >>>>> >>>>> -JC >>>>> >>>>> On Wednesday, April 12, 2017 at 5:38:40 PM UTC-5, Todd L. Montgomery >>>>> wrote: >>>>>> >>>>>> Mike has the best point, I think. 30 seconds between sends will cause >>>>>> the congestion window to close. Depending on what is in use (CUBIC vs. >>>>>> Reno), this will change behavior. >>>>>> >>>>>> -- Todd >>>>>> >>>>>> On Wed, Apr 12, 2017 at 3:27 PM, Greg Young <gregor...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> You are likely measuring wrong and just have not figured out how yet. >>>>>>> >>>>>>> On Wed, Apr 12, 2017 at 8:56 PM, J Crawford <latency...@mail.com> >>>>>>> wrote: >>>>>>> > The SO question has the source codes of a simple server and client >>>>>>> that >>>>>>> > demonstrate and isolate the problem. Basically I'm timing the >>>>>>> latency of a >>>>>>> > ping-pong (client-server-client) message. I start by sending one >>>>>>> message >>>>>>> > every 1 millisecond. I wait for 200k messages to be sent so that >>>>>>> the HotSpot >>>>>>> > has a chance to optimize the code. Then I change my pause time >>>>>>> from 1 >>>>>>> > millisecond to 30 seconds. For my surprise my write and read >>>>>>> operation >>>>>>> > become considerably slower. >>>>>>> > >>>>>>> > I don't think it is a JIT/HotSpot problem. I was able to pinpoint >>>>>>> the slower >>>>>>> > method to the native JNI calls to write (write0) and read. Even if >>>>>>> I change >>>>>>> > the pause from 1 millisecond to 1 second, problem persists. >>>>>>> > >>>>>>> > I was able to observe that on MacOS and Linux. >>>>>>> > >>>>>>> > Does anyone here have a clue of what can be happening? >>>>>>> > >>>>>>> > Note that I'm disabling Nagle's Algorithm with setTcpNoDelay(true). >>>>>>> > >>>>>>> > SO question with code and output: >>>>>>> > >>>>>>> http://stackoverflow.com/questions/43377600/socketchannel-why-if-i-write-msgs-quickly-the-latency-of-each-message-is-low-b >>>>>>> > >>>>>>> > Thanks! >>>>>>> > >>>>>>> > -JC >>>>>>> > >>>>>>> > -- >>>>>>> > You received this message because you are subscribed to the Google >>>>>>> Groups >>>>>>> > "mechanical-sympathy" group. >>>>>>> > To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an >>>>>>> > email to mechanical-sympathy+unsubscr...@googlegroups.com. >>>>>>> > For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Studying for the Turing test >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "mechanical-sympathy" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to mechanical-sympathy+unsubscr...@googlegroups.com. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "mechanical-sympathy" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to mechanical-sympathy+unsubscr...@googlegroups.com. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "mechanical-sympathy" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to mechanical-sympathy+unsubscr...@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> > -- > You received this message because you are subscribed to the Google Groups > "mechanical-sympathy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to mechanical-sympathy+unsubscr...@googlegroups.com <javascript:>. > For more options, visit https://groups.google.com/d/optout. > > > -- You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group. To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
