Hi, I was talking about your server spin-waiting. Not sure it applies in your case, but from http://x86.renejeschke.de/html/file_module_x86_id_232.html
When executing a "spin-wait loop," a Pentium 4 or Intel Xeon processor > suffers a severe performance penalty when exiting the loop because it > detects a possible memory order violation. The PAUSE instruction provides a > hint to the processor that the code sequence is a spin-wait loop. The > processor uses this hint to avoid the memory order violation in most > situations, which greatly improves processor performance. For this reason, > it is recommended that a PAUSE instruction be placed in all spin-wait loops. > On second thought, this probably is far less impact-full than the latency spike you observe On Thu, Apr 20, 2017 at 8:22 AM, J Crawford <[email protected]> wrote: > Hi Nikolay, > > Thanks for trying to help. Can you elaborate on "speculative execution" > and how do you think it could be affecting the socket latency? > > My tight loop for pausing is indeed working (the program actually "pauses" > as expected) so not sure what you mean. > > Thanks again! > > -JC > > On Wednesday, April 19, 2017 at 1:15:24 AM UTC-5, Nikolay Tsankov wrote: >> >> Hi, >> >> Could it be caused by speculative execution/the tight wait loop? You can >> probably test in C with a pause instruction in the loop... >> >> Best, >> Nikolay >> >> On Tue, Apr 18, 2017 at 9:24 AM, Kirk Pepperdine <[email protected]> >> wrote: >> >>> Some code written, I’ll take this offline >>> >>> >>> On Apr 17, 2017, at 5:28 PM, J Crawford <[email protected]> wrote: >>> >>> >>> > I have some skeletial client/server code in C. It just needs to >>> morphed to your test case. I can’t see me getting that done today unless I >>> get blocked on what I need to get done. >>> >>> Hello Kirk, I'm still banging my head trying to understand this latency >>> issue. Did you have time to use your C code to try to reproduce this >>> problem? I'm not a C programmer, but if you are busy I can try to adapt >>> your skeletal client/server C code to the use-case in question. >>> >>> I'm currently clueless and unable to make progress. It happens on MacOS, >>> Linux and Windows so it does not look like a OS-related issue. Looks more >>> like a JVM or CPU issue. >>> >>> Thanks! >>> >>> -JC >>> >>> >>> On Thursday, April 13, 2017 at 1:59:48 AM UTC-5, Kirk Pepperdine wrote: >>>> >>>> >>>> Normally when I run into “can’t scale down” problems in Java you have >>>> to be concerned about methods on the critical path not being hot enough to >>>> be compiled. However I’d give this one a low probability because the knock >>>> on latency is typically 2-3x what you’d see under load. So, this seems some >>>> how connected to a buffer with a timer. Under load you get fill and fire >>>> and of course the scale down is fire on timeout ‘cos you rarely is ever >>>> fill. >>>> >>>> Have you looked at this problem using Wireshark or a packet sniffer in >>>> your network? Another trick is to directly instrument the Socket read, >>>> write methods. You can do that with BCI or simply just hand modify the code >>>> and preload it on the bootstrap class path. >>>> >>>> I have some skeletial client/server code in C. It just needs to morphed >>>> to your test case. I can’t see me getting that done today unless I get >>>> blocked on what I need to get done. >>>> >>>> Kind regards, >>>> Kirk >>>> >>>> On Apr 13, 2017, at 6:45 AM, J Crawford <[email protected]> wrote: >>>> >>>> Very good idea, Mike. If I only knew C :) I'll try to hire a C coder on >>>> UpWork.com <http://upwork.com/> or Elance.com <http://elance.com/> to >>>> do that. It shouldn't be hard for someone who knows C network programming. >>>> I hope... >>>> >>>> Thanks! >>>> >>>> -JC >>>> >>>> On Wednesday, April 12, 2017 at 11:37:28 PM UTC-5, mikeb01 wrote: >>>>> >>>>> Rewrite the test in C to eliminate the JVM as the cause of the >>>>> slowdown? >>>>> >>>>> On 13 April 2017 at 16:31, J Crawford <[email protected]> wrote: >>>>> >>>>>> Ok, this is a total mystery. Tried a bunch of strategies with no luck: >>>>>> >>>>>> 1. Checked the cpu frequency with i7z_64bit. No variance in the >>>>>> frequency. >>>>>> >>>>>> 2. Disabled all power management. No luck. >>>>>> >>>>>> 3. Changed TCP Congestion Control Algorithm. No luck. >>>>>> >>>>>> 4. Set net.ipv4.tcp_slow_start_after_idle to false. No luck. >>>>>> >>>>>> 5. Tested with UDP implementation. No luck. >>>>>> >>>>>> 6. Placed the all sockets in blocking mode just for the heck of it. >>>>>> No luck, same problem. >>>>>> >>>>>> I'm out of pointers now and don't know where to run. This is an >>>>>> important latency problem that I must understand as it affects my trading >>>>>> system. >>>>>> >>>>>> Anyone who has any clue of what might be going on, please throw some >>>>>> light. Also, if you run the provided Server and Client code in your own >>>>>> environment/machine (over localhost/loopback) you will see that it does >>>>>> happen. >>>>>> >>>>>> Thanks! >>>>>> >>>>>> -JC >>>>>> >>>>>> On Wednesday, April 12, 2017 at 10:23:17 PM UTC-5, Todd L. Montgomery >>>>>> wrote: >>>>>>> >>>>>>> The short answer is that no congestion control algorithm is suited >>>>>>> for low latency trading and in all cases, using raw UDP will be better >>>>>>> for >>>>>>> latency. Congestion control is about fairness. Latency in trading has >>>>>>> nothing to do with fairness. >>>>>>> >>>>>>> The long answer is that to varying degrees, all congestion control >>>>>>> must operate at high or complete utilization to probe. Those based on >>>>>>> loss >>>>>>> (all variants of CUBIC, Reno, etc.) must be operating in congestion >>>>>>> avoidance or be in slow start. Those based on RTT (Vegas) or >>>>>>> RTT/Bottleneck >>>>>>> Bandwidth (BBR) must be probing for more bandwidth to determine change >>>>>>> in >>>>>>> RTT (as a "replacement" for loss). >>>>>>> >>>>>>> So, the case of sending only periodically is somewhat antithetical >>>>>>> to the operating point that all congestion control must operate at while >>>>>>> probing. And the reason all appropriate congestion control algorithms I >>>>>>> know of reset upon not operating at high utilization. >>>>>>> >>>>>>> You can think of it this way.... the network can only sustain X >>>>>>> msgs/sec, but X is a (seemingly random) nonlinear function of time. How >>>>>>> do >>>>>>> you determine X at any given time without operating at that point? You >>>>>>> can >>>>>>> not, that I know of, predict X without operating at X. >>>>>>> >>>>>>> On Wed, Apr 12, 2017 at 6:54 PM, J Crawford <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Todd, >>>>>>>> >>>>>>>> I'm trying several TCP Congestion algorithms here: westwood, >>>>>>>> highspeed, veno, etc. >>>>>>>> >>>>>>>> No luck so far, but there are many more I haven't tried. I'm using >>>>>>>> this answer to change the TCP congestion algo: >>>>>>>> http://unix.stackexchange.com/a/278217 >>>>>>>> >>>>>>>> Does anyone know what TCP congestion algorithm is the best for >>>>>>>> low-latency? Or the best for the single message scenario I've >>>>>>>> described? *This >>>>>>>> looks like an important configuration for trading, when a single order >>>>>>>> needs to go out after some time and you don't want it to go out at a >>>>>>>> slower >>>>>>>> speed.* >>>>>>>> >>>>>>>> Thanks! >>>>>>>> >>>>>>>> -JC >>>>>>>> >>>>>>>> On Wednesday, April 12, 2017 at 5:38:40 PM UTC-5, Todd L. >>>>>>>> Montgomery wrote: >>>>>>>>> >>>>>>>>> Mike has the best point, I think. 30 seconds between sends will >>>>>>>>> cause the congestion window to close. Depending on what is in use >>>>>>>>> (CUBIC >>>>>>>>> vs. Reno), this will change behavior. >>>>>>>>> >>>>>>>>> -- Todd >>>>>>>>> >>>>>>>>> On Wed, Apr 12, 2017 at 3:27 PM, Greg Young <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> You are likely measuring wrong and just have not figured out how >>>>>>>>>> yet. >>>>>>>>>> >>>>>>>>>> On Wed, Apr 12, 2017 at 8:56 PM, J Crawford <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> > The SO question has the source codes of a simple server and >>>>>>>>>> client that >>>>>>>>>> > demonstrate and isolate the problem. Basically I'm timing the >>>>>>>>>> latency of a >>>>>>>>>> > ping-pong (client-server-client) message. I start by sending >>>>>>>>>> one message >>>>>>>>>> > every 1 millisecond. I wait for 200k messages to be sent so >>>>>>>>>> that the HotSpot >>>>>>>>>> > has a chance to optimize the code. Then I change my pause time >>>>>>>>>> from 1 >>>>>>>>>> > millisecond to 30 seconds. For my surprise my write and read >>>>>>>>>> operation >>>>>>>>>> > become considerably slower. >>>>>>>>>> > >>>>>>>>>> > I don't think it is a JIT/HotSpot problem. I was able to >>>>>>>>>> pinpoint the slower >>>>>>>>>> > method to the native JNI calls to write (write0) and read. Even >>>>>>>>>> if I change >>>>>>>>>> > the pause from 1 millisecond to 1 second, problem persists. >>>>>>>>>> > >>>>>>>>>> > I was able to observe that on MacOS and Linux. >>>>>>>>>> > >>>>>>>>>> > Does anyone here have a clue of what can be happening? >>>>>>>>>> > >>>>>>>>>> > Note that I'm disabling Nagle's Algorithm with >>>>>>>>>> setTcpNoDelay(true). >>>>>>>>>> > >>>>>>>>>> > SO question with code and output: >>>>>>>>>> > http://stackoverflow.com/questions/43377600/socketchannel- >>>>>>>>>> why-if-i-write-msgs-quickly-the-latency-of-each-message-is-low-b >>>>>>>>>> > >>>>>>>>>> > Thanks! >>>>>>>>>> > >>>>>>>>>> > -JC >>>>>>>>>> > >>>>>>>>>> > -- >>>>>>>>>> > You received this message because you are subscribed to the >>>>>>>>>> Google Groups >>>>>>>>>> > "mechanical-sympathy" group. >>>>>>>>>> > To unsubscribe from this group and stop receiving emails from >>>>>>>>>> it, send an >>>>>>>>>> > email to [email protected]. >>>>>>>>>> > For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Studying for the Turing test >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "mechanical-sympathy" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to [email protected] >>>>>>>>>> . >>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "mechanical-sympathy" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "mechanical-sympathy" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "mechanical-sympathy" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "mechanical-sympathy" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "mechanical-sympathy" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "mechanical-sympathy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
