Hi Vanger,
thanks for the report. Have you changed the value of
akka.http.host-connection-pool.min-connections
(https://github.com/akka/akka-http/blob/master/akka-http-core/src/main/resources/reference.conf#L233)?
That would explain the behavior where the pool tries to keep connections
alive to hosts it connected to previously.
In any case, it shouldn't loop infinitely in such a case. Can you create an
issue with that problem?
Thanks,
Johannes
On Monday, July 10, 2017 at 5:52:34 PM UTC+2, Vanger B wrote:
>
> We have simple proxy based on Akka Http low-level API. Time to time we can
> observe very strange behavior of the proxy. We are still gathering info,
> but I think we already have enough for a post in this google group.
> At any time, our service can start consuming CPU and never stops to do so
> by itself, but can be "cured" with restart. We found no correlations with
> traffic volumes nor with particular HTTP endpoints. But it more likely to
> be related with DNS manipulations, we are hosted in AWS and proxy routes
> traffic between resources that appears and disappears dynamically (for
> instance: DNS records and LoadBalancers) and it seems that connection pool
> could have bad times when that happens. However, that assumption may be
> wrong.
>
> It looks like it only consumes available CPU time but don't steal time
> from other threads, proxy works normal (if you can call it "normal") with
> 100% CPU consumption. When this happens, only "average response time" is
> affected - some constant time (tens of ms.) added for this useless
> activity. However, we aren't agree to live with that :)
>
> What do we have:
>
> 1. Akka-http 10.0.8
> 2. We setup our proxy as shown below:
>
> val src = Http(system).bind(interface = host, port = port)
>
> src.to(Sink.foreach { connection =>
> connection.handleWithAsyncHandler { request =>
> //*rerouting original request here*
> }
> }).run()
>
> 3. We completely isolated machine form outside world and we are pretty
> sure that no external requests get to the proxy. However, different
> monitoring utilities show there are incoming and outgoing "traffic" on the
> instance.
> 4. There are no application logs at all without external requests. If
> we turn root log level to TRACE (without service restart): we only see
> Akka
> logging with repeating pattern:
> 5. [07.07.2017 18:42:21.796] DEBUG
> [Proxy-akka.actor.default-dispatcher-13915] [akka.stream.impl.io.TLSActor]
> closing output
> [07.07.2017 18:42:21.797] DEBUG
> [Proxy-akka.actor.default-dispatcher-13915]
> [a.h.impl.engine.client.PoolGateway] [1] <unconnected> connection was
> closed by peer while no requests were in flight
> [07.07.2017 18:42:21.797] DEBUG
> [Proxy-akka.actor.default-dispatcher-13915]
> [a.h.impl.engine.client.PoolGateway] [1] Idle -> Unconnected
> [07.07.2017 18:42:21.797] DEBUG
> [Proxy-akka.actor.default-dispatcher-13915]
> [a.h.impl.engine.client.PoolGateway] [1] Unconnected -> Idle
> [07.07.2017 18:42:21.797] DEBUG
> [Proxy-akka.actor.default-dispatcher-13915]
> [a.h.impl.engine.client.PoolGateway] [1] <unconnected> Establishing
> connection...
> [07.07.2017 18:42:21.797] DEBUG
> [Proxy-akka.actor.default-dispatcher-13915] [akka.stream.impl.io.TLSActor]
> closing output
> [07.07.2017 18:42:21.814] DEBUG
> [Proxy-akka.actor.default-dispatcher-16344]
> [a.h.impl.engine.client.PoolGateway] [2] <unconnected> connection was
> closed by peer while no requests were in flight
> [07.07.2017 18:42:21.814] DEBUG
> [Proxy-akka.actor.default-dispatcher-16344] [akka.stream.impl.io.TLSActor]
> closing output
> [07.07.2017 18:42:21.814] DEBUG
> [Proxy-akka.actor.default-dispatcher-16344]
> [a.h.impl.engine.client.PoolGateway] [2] Idle -> Unconnected
> [07.07.2017 18:42:21.814] DEBUG
> [Proxy-akka.actor.default-dispatcher-16344]
> [a.h.impl.engine.client.PoolGateway] [2] Unconnected -> Idle
> [07.07.2017 18:42:21.814] DEBUG
> [Proxy-akka.actor.default-dispatcher-16344]
> [a.h.impl.engine.client.PoolGateway] [2] <unconnected> Establishing
> connection...
> [07.07.2017 18:42:21.814] DEBUG
> [Proxy-akka.actor.default-dispatcher-16344] [akka.stream.impl.io.TLSActor]
> closing output
> [07.07.2017 18:42:21.814] DEBUG
> [Proxy-akka.actor.default-dispatcher-16344]
> [akka.io.TcpOutgoingConnection]
> Resolving loadbala-1x4ty7dih7yp4-545989797.us-east-1.elb.amazonaws.com
> before connecting
> [07.07.2017 18:42:21.814] DEBUG
> [Proxy-akka.actor.default-dispatcher-16344]
> [akka.io.TcpOutgoingConnection]
> Could not establish connection to [
> loadbala-1x4ty7dih7yp4-545989797.us-east-1.elb.amazonaws.com:443] due
> to java.net.UnknownHostException:
> loadbala-1x4ty7dih7yp4-545989797.us-east-1.elb.amazonaws.com
> [07.07.2017 18:42:21.815] DEBUG
> [Proxy-akka.actor.default-dispatcher-16344]
> [a.h.impl.engine.client.PoolGateway] [0] <unconnected> connection was
> closed by peer while no requests were in flight
> 6. Looks like that loop tries to execute as fast as it possibly can
> and consumes 100% CPU and pollutes GBs of logs in minutes. Again, there
> are
> no other sources of logs except shown above.
> 7. We have some profiling info from the problematic instances (and we
> still have couple of them isolated and consuming CPU, for fun &
> research...).
> I've attached screenshot from visualvm profiling: Live threads (note
> that there are no application threads visible and they aren't filtered
> out), CPU consumption by threads with different sorting (self time %, self
> time CPU, total time CPU).
>
> We did quite a few experiments, but not enough to reproduce the issue, so
> we are not sure which information is useful and which is not. We can
> provide additional data from the "bad" instances if it's needed.
>
> We would appreciate any help with investigation :\
>
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
