So the only way I was able to successfully identify the suspicious code was
to route a percentage of my production traffic to a stubbed route that I
incrementally added back pieces of our implementation into. What I found
was that we started getting spikes when the entity(as[CaseClassFromJson])
stubbed
was added back in. To figure out if it was the json parsing or 'POST'
entity consumption itself, I replaced that class with a string - turns out
we experience the latency spikes with that as well (on low traffic as noted
earlier in this thread).
I by no means have a deep understanding of streams, but it makes me wonder
if the way I have our code consuming the entity is not correct.
On Monday, October 30, 2017 at 4:27:13 PM UTC-4, Gary Malouf wrote:
>
> Hi Roland - thank you for the tip. We shrunk the thread pool size down to
> 1, but were disheartened to still see the latency spikes. Using Kamon's
> tracing library (which we validated with various tests to ensure it's own
> numbers are most likely correct), we could not find anything in our code
> within the route that was causing the latency (it all appeared to be
> classified to be that route but no code segments within it).
>
> As mentioned earlier, running loads of 100-1000 requests/second completely
> hides the issue (save for the max latency) as everything through 99th
> percentiles is under a few milliseconds.
>
> On Tuesday, October 24, 2017 at 2:23:07 AM UTC-4, rkuhn wrote:
>>
>> You could try to decrease your thread pool size to 1 to exclude wakeup
>> latencies when things (like CPU cores) have gone to sleep.
>>
>> Regards, Roland
>>
>> Sent from my iPhone
>>
>> On 23. Oct 2017, at 22:49, Gary Malouf <malou...@gmail.com> wrote:
>>
>> Yes, it gets parsed using entity(as[]) with spray-json support. Under a
>> load test of say 1000 requests/second these latencies are not visible in
>> the percentiles - they are easy to see because this web server is getting
>> 10-20 requests/second currently. Trying to brainstorm if a dispatcher
>> needed to be tuned or something of that sort but have yet to see evidence
>> supporting that.
>>
>> path("foos") {
>> traceName("FooSelection") {
>> entity(as[ExternalPageRequest]) { pr =>
>> val spr = toSelectionPageRequest(pr)
>> shouldTracePageId(spr.pageId).fold(
>> Tracer.currentContext.withNewSegment(s"Page-${pr.pageId}", "PageTrace", "
>> kamon") {
>> processPageRequestAndComplete(pr, spr)
>> },
>> processPageRequestAndComplete(pr, spr)
>> )
>> }
>> }
>>
>> }
>>
>> On Mon, Oct 23, 2017 at 4:42 PM, Viktor Klang <viktor...@gmail.com>
>> wrote:
>>
>>> And you consume the entityBytes I presume?
>>>
>>> On Mon, Oct 23, 2017 at 10:35 PM, Gary Malouf <malou...@gmail.com>
>>> wrote:
>>>
>>>> It is from when I start the Kamon trace (just inside of my
>>>> path("myawesomepath") declaration until (theoretically) a 'complete' call
>>>> is made.
>>>>
>>>> path("myawesomepath") {
>>>> traceName("CoolStory") {
>>>> ///do some stuff
>>>> complete("This is great")
>>>> } }
>>>>
>>>> For what it's worth, this route is a 'POST' call.
>>>>
>>>> On Mon, Oct 23, 2017 at 4:30 PM, Viktor Klang <viktor...@gmail.com>
>>>> wrote:
>>>>
>>>>> No, I mean, is it from first-byte-received to last-byte-sent or what?
>>>>>
>>>>> On Mon, Oct 23, 2017 at 10:22 PM, Gary Malouf <malou...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> We are using percentiles computed via Kamon 0.6.8. In a very low
>>>>>> request rate environment like this, it takes roughly 1 super slow
>>>>>> request/second to throw off the percentiles (which is what I think is
>>>>>> happening).
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Oct 23, 2017 at 4:20 PM, Viktor Klang <viktor...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> What definition of latency are you using? (i.e. how is it derived)
>>>>>>>
>>>>>>> On Mon, Oct 23, 2017 at 10:11 PM, Gary Malouf <malou...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Konrad,
>>>>>>>>
>>>>>>>> Our real issue is that we can not reproduce the results. The web
>>>>>>>> server we are having latency issues with is under peak load of 10-15
>>>>>>>> requests/second - obviously not much to deal with.
>>>>>>>>
>>>>>>>> When we use load tests (https://github.com/apigee/apib), it's easy
>>>>>>>> for us to throw a few thousand requests/second at it and get latencies
>>>>>>>> in
>>>>>>>> the ~ 3 ms range. We use kamon to track internal metrics - what we
>>>>>>>> see is
>>>>>>>> that our 95th and 99th percentiles only look bad under the production
>>>>>>>> traffic but not under load tests.
>>>>>>>>
>>>>>>>> I've since used kamon to print out the actual requests trying to
>>>>>>>> find any pattern in them to hint at what's wrong in my own code, but
>>>>>>>> they
>>>>>>>> seem to be completely random. What we do know is that downgrading to
>>>>>>>> spray
>>>>>>>> gets us 99.9th percentile latencies under 2ms, so something related to
>>>>>>>> the
>>>>>>>> upgrade is allowing this.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Gary
>>>>>>>>
>>>>>>>> On Tuesday, October 17, 2017 at 12:07:51 PM UTC-4, Konrad Malawski
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Step 1 – don’t panic ;-)
>>>>>>>>> Step 2 – as I already asked for, please share actual details of
>>>>>>>>> the benchmarks. It is not good to discuss benchmarks without any
>>>>>>>>> insight
>>>>>>>>> into what / how exactly you’re measuring.
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Cheers,
>>>>>>>>> Konrad 'ktoso <http://kto.so>' Malawski
>>>>>>>>> Akka <http://akka.io/> @ Lightbend <http://lightbend.com/>
>>>>>>>>>
>>>>>>>>> On October 12, 2017 at 15:31:19, Gary Malouf (malou...@gmail.com)
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> We have a web service that we just finished migrating from spray
>>>>>>>>> 1.3 to Akka-Http 10.0.9. While in most cases it is performing well,
>>>>>>>>> we are
>>>>>>>>> seeing terrible 99th percentile latencies 300-450ms range) starting
>>>>>>>>> from a
>>>>>>>>> very low request rate (10/second) on an ec2 m3.large.
>>>>>>>>>
>>>>>>>>> Our service does not do anything complicated - it does a few Map
>>>>>>>>> lookups and returns a response to a request. In spray, even 99th
>>>>>>>>> percentile latencies were on the order of 1-3 ms, so we are
>>>>>>>>> definitely
>>>>>>>>> concerned. Connections as with many pixel-type servers are
>>>>>>>>> short-lived ->
>>>>>>>>> we actually pass the Connection: Close header intentionally in our
>>>>>>>>> responses.
>>>>>>>>>
>>>>>>>>> Is there any obvious tuning that should be done on the server
>>>>>>>>> configuration that others have found?
>>>>>>>>> --
>>>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>> >>>>>>>>>> Check the FAQ:
>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>> >>>>>>>>>> Search the archives:
>>>>>>>>> https://groups.google.com/group/akka-user
>>>>>>>>> ---
>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>>> Groups "Akka User List" group.
>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>> send an email to akka-user+...@googlegroups.com.
>>>>>>>>> To post to this group, send email to akka...@googlegroups.com.
>>>>>>>>> Visit this group at https://groups.google.com/group/akka-user.
>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>
>>>>>>>>> --
>>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>> >>>>>>>>>> Check the FAQ:
>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>> >>>>>>>>>> Search the archives:
>>>>>>>> https://groups.google.com/group/akka-user
>>>>>>>> ---
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "Akka User List" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>> send an email to akka-user+...@googlegroups.com.
>>>>>>>> To post to this group, send email to akka...@googlegroups.com.
>>>>>>>> Visit this group at https://groups.google.com/group/akka-user.
>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Cheers,
>>>>>>> √
>>>>>>>
>>>>>>> --
>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>> >>>>>>>>>> Check the FAQ:
>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>> >>>>>>>>>> Search the archives:
>>>>>>> https://groups.google.com/group/akka-user
>>>>>>> ---
>>>>>>> You received this message because you are subscribed to a topic in
>>>>>>> the Google Groups "Akka User List" group.
>>>>>>> To unsubscribe from this topic, visit
>>>>>>> https://groups.google.com/d/topic/akka-user/-_C9jCPDwts/unsubscribe.
>>>>>>> To unsubscribe from this group and all its topics, send an email to
>>>>>>> akka-user+...@googlegroups.com.
>>>>>>> To post to this group, send email to akka...@googlegroups.com.
>>>>>>> Visit this group at https://groups.google.com/group/akka-user.
>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>> >>>>>>>>>> Check the FAQ:
>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>> >>>>>>>>>> Search the archives:
>>>>>> https://groups.google.com/group/akka-user
>>>>>> ---
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Akka User List" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to akka-user+...@googlegroups.com.
>>>>>> To post to this group, send email to akka...@googlegroups.com.
>>>>>> Visit this group at https://groups.google.com/group/akka-user.
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Cheers,
>>>>> √
>>>>>
>>>>> --
>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>> >>>>>>>>>> Check the FAQ:
>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>> >>>>>>>>>> Search the archives:
>>>>> https://groups.google.com/group/akka-user
>>>>> ---
>>>>> You received this message because you are subscribed to a topic in the
>>>>> Google Groups "Akka User List" group.
>>>>> To unsubscribe from this topic, visit
>>>>> https://groups.google.com/d/topic/akka-user/-_C9jCPDwts/unsubscribe.
>>>>> To unsubscribe from this group and all its topics, send an email to
>>>>> akka-user+...@googlegroups.com.
>>>>> To post to this group, send email to akka...@googlegroups.com.
>>>>> Visit this group at https://groups.google.com/group/akka-user.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
>>>> --
>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>>> >>>>>>>>>> Check the FAQ:
>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>> >>>>>>>>>> Search the archives:
>>>> https://groups.google.com/group/akka-user
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Akka User List" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to akka-user+...@googlegroups.com.
>>>> To post to this group, send email to akka...@googlegroups.com.
>>>> Visit this group at https://groups.google.com/group/akka-user.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>>
>>>
>>> --
>>> Cheers,
>>> √
>>>
>>> --
>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>> >>>>>>>>>> Check the FAQ:
>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>> >>>>>>>>>> Search the archives:
>>> https://groups.google.com/group/akka-user
>>> ---
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "Akka User List" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/akka-user/-_C9jCPDwts/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> akka-user+...@googlegroups.com.
>>> To post to this group, send email to akka...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/akka-user.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>> >>>>>>>>>> Check the FAQ:
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Akka User List" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to akka-user+...@googlegroups.com.
>> To post to this group, send email to akka...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/akka-user.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
