Hi, Could it be caused by speculative execution/the tight wait loop? You can probably test in C with a pause instruction in the loop...
Best, Nikolay On Tue, Apr 18, 2017 at 9:24 AM, Kirk Pepperdine <[email protected]> wrote: > Some code written, I’ll take this offline > > > On Apr 17, 2017, at 5:28 PM, J Crawford <[email protected]> wrote: > > > > I have some skeletial client/server code in C. It just needs to morphed > to your test case. I can’t see me getting that done today unless I get > blocked on what I need to get done. > > Hello Kirk, I'm still banging my head trying to understand this latency > issue. Did you have time to use your C code to try to reproduce this > problem? I'm not a C programmer, but if you are busy I can try to adapt > your skeletal client/server C code to the use-case in question. > > I'm currently clueless and unable to make progress. It happens on MacOS, > Linux and Windows so it does not look like a OS-related issue. Looks more > like a JVM or CPU issue. > > Thanks! > > -JC > > > On Thursday, April 13, 2017 at 1:59:48 AM UTC-5, Kirk Pepperdine wrote: >> >> >> Normally when I run into “can’t scale down” problems in Java you have to >> be concerned about methods on the critical path not being hot enough to be >> compiled. However I’d give this one a low probability because the knock on >> latency is typically 2-3x what you’d see under load. So, this seems some >> how connected to a buffer with a timer. Under load you get fill and fire >> and of course the scale down is fire on timeout ‘cos you rarely is ever >> fill. >> >> Have you looked at this problem using Wireshark or a packet sniffer in >> your network? Another trick is to directly instrument the Socket read, >> write methods. You can do that with BCI or simply just hand modify the code >> and preload it on the bootstrap class path. >> >> I have some skeletial client/server code in C. It just needs to morphed >> to your test case. I can’t see me getting that done today unless I get >> blocked on what I need to get done. >> >> Kind regards, >> Kirk >> >> On Apr 13, 2017, at 6:45 AM, J Crawford <[email protected]> wrote: >> >> Very good idea, Mike. If I only knew C :) I'll try to hire a C coder on >> UpWork.com <http://upwork.com/> or Elance.com <http://elance.com/> to do >> that. It shouldn't be hard for someone who knows C network programming. I >> hope... >> >> Thanks! >> >> -JC >> >> On Wednesday, April 12, 2017 at 11:37:28 PM UTC-5, mikeb01 wrote: >>> >>> Rewrite the test in C to eliminate the JVM as the cause of the slowdown? >>> >>> On 13 April 2017 at 16:31, J Crawford <[email protected]> wrote: >>> >>>> Ok, this is a total mystery. Tried a bunch of strategies with no luck: >>>> >>>> 1. Checked the cpu frequency with i7z_64bit. No variance in the >>>> frequency. >>>> >>>> 2. Disabled all power management. No luck. >>>> >>>> 3. Changed TCP Congestion Control Algorithm. No luck. >>>> >>>> 4. Set net.ipv4.tcp_slow_start_after_idle to false. No luck. >>>> >>>> 5. Tested with UDP implementation. No luck. >>>> >>>> 6. Placed the all sockets in blocking mode just for the heck of it. No >>>> luck, same problem. >>>> >>>> I'm out of pointers now and don't know where to run. This is an >>>> important latency problem that I must understand as it affects my trading >>>> system. >>>> >>>> Anyone who has any clue of what might be going on, please throw some >>>> light. Also, if you run the provided Server and Client code in your own >>>> environment/machine (over localhost/loopback) you will see that it does >>>> happen. >>>> >>>> Thanks! >>>> >>>> -JC >>>> >>>> On Wednesday, April 12, 2017 at 10:23:17 PM UTC-5, Todd L. Montgomery >>>> wrote: >>>>> >>>>> The short answer is that no congestion control algorithm is suited for >>>>> low latency trading and in all cases, using raw UDP will be better for >>>>> latency. Congestion control is about fairness. Latency in trading has >>>>> nothing to do with fairness. >>>>> >>>>> The long answer is that to varying degrees, all congestion control >>>>> must operate at high or complete utilization to probe. Those based on loss >>>>> (all variants of CUBIC, Reno, etc.) must be operating in congestion >>>>> avoidance or be in slow start. Those based on RTT (Vegas) or >>>>> RTT/Bottleneck >>>>> Bandwidth (BBR) must be probing for more bandwidth to determine change in >>>>> RTT (as a "replacement" for loss). >>>>> >>>>> So, the case of sending only periodically is somewhat antithetical to >>>>> the operating point that all congestion control must operate at while >>>>> probing. And the reason all appropriate congestion control algorithms I >>>>> know of reset upon not operating at high utilization. >>>>> >>>>> You can think of it this way.... the network can only sustain X >>>>> msgs/sec, but X is a (seemingly random) nonlinear function of time. How do >>>>> you determine X at any given time without operating at that point? You can >>>>> not, that I know of, predict X without operating at X. >>>>> >>>>> On Wed, Apr 12, 2017 at 6:54 PM, J Crawford <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Todd, >>>>>> >>>>>> I'm trying several TCP Congestion algorithms here: westwood, >>>>>> highspeed, veno, etc. >>>>>> >>>>>> No luck so far, but there are many more I haven't tried. I'm using >>>>>> this answer to change the TCP congestion algo: >>>>>> http://unix.stackexchange.com/a/278217 >>>>>> >>>>>> Does anyone know what TCP congestion algorithm is the best for >>>>>> low-latency? Or the best for the single message scenario I've described? >>>>>> *This >>>>>> looks like an important configuration for trading, when a single order >>>>>> needs to go out after some time and you don't want it to go out at a >>>>>> slower >>>>>> speed.* >>>>>> >>>>>> Thanks! >>>>>> >>>>>> -JC >>>>>> >>>>>> On Wednesday, April 12, 2017 at 5:38:40 PM UTC-5, Todd L. Montgomery >>>>>> wrote: >>>>>>> >>>>>>> Mike has the best point, I think. 30 seconds between sends will >>>>>>> cause the congestion window to close. Depending on what is in use (CUBIC >>>>>>> vs. Reno), this will change behavior. >>>>>>> >>>>>>> -- Todd >>>>>>> >>>>>>> On Wed, Apr 12, 2017 at 3:27 PM, Greg Young <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> You are likely measuring wrong and just have not figured out how >>>>>>>> yet. >>>>>>>> >>>>>>>> On Wed, Apr 12, 2017 at 8:56 PM, J Crawford <[email protected]> >>>>>>>> wrote: >>>>>>>> > The SO question has the source codes of a simple server and >>>>>>>> client that >>>>>>>> > demonstrate and isolate the problem. Basically I'm timing the >>>>>>>> latency of a >>>>>>>> > ping-pong (client-server-client) message. I start by sending one >>>>>>>> message >>>>>>>> > every 1 millisecond. I wait for 200k messages to be sent so that >>>>>>>> the HotSpot >>>>>>>> > has a chance to optimize the code. Then I change my pause time >>>>>>>> from 1 >>>>>>>> > millisecond to 30 seconds. For my surprise my write and read >>>>>>>> operation >>>>>>>> > become considerably slower. >>>>>>>> > >>>>>>>> > I don't think it is a JIT/HotSpot problem. I was able to pinpoint >>>>>>>> the slower >>>>>>>> > method to the native JNI calls to write (write0) and read. Even >>>>>>>> if I change >>>>>>>> > the pause from 1 millisecond to 1 second, problem persists. >>>>>>>> > >>>>>>>> > I was able to observe that on MacOS and Linux. >>>>>>>> > >>>>>>>> > Does anyone here have a clue of what can be happening? >>>>>>>> > >>>>>>>> > Note that I'm disabling Nagle's Algorithm with >>>>>>>> setTcpNoDelay(true). >>>>>>>> > >>>>>>>> > SO question with code and output: >>>>>>>> > http://stackoverflow.com/questions/43377600/socketchannel- >>>>>>>> why-if-i-write-msgs-quickly-the-latency-of-each-message-is-low-b >>>>>>>> > >>>>>>>> > Thanks! >>>>>>>> > >>>>>>>> > -JC >>>>>>>> > >>>>>>>> > -- >>>>>>>> > You received this message because you are subscribed to the >>>>>>>> Google Groups >>>>>>>> > "mechanical-sympathy" group. >>>>>>>> > To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an >>>>>>>> > email to [email protected]. >>>>>>>> > For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Studying for the Turing test >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "mechanical-sympathy" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "mechanical-sympathy" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "mechanical-sympathy" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "mechanical-sympathy" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> >> >> > -- > You received this message because you are subscribed to the Google Groups > "mechanical-sympathy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to the Google Groups > "mechanical-sympathy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
