[
https://issues.apache.org/jira/browse/CALCITE-1315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15378490#comment-15378490
]
Josh Elser commented on CALCITE-1315:
-------------------------------------
bq. I don't know the semantics of NoHttpResponseException. Is it possible for a
server to repeatedly return NoHttpResponseException? If so, should you limit
the number of retries, to avoid a loop?
Not sure if it's not re-used elsewhere (don't have enough experience with HC to
be confident that they wouldnt' do something screwy), but, under normal
circumstances, we shouldn't ever get into a situation where we can connect to a
server, send a request, and get no bytes back from that server (if it's
dead/dying, it should stop accepting the connection before we send our request).
That said, I'm not a huge fan of our existing logic inside of the
AvaticaHttpClient impls for the reason you point out. We have very little
insight into retries happening too (often because logging is minimized because
we're in the context of sqlline or similar). Consolidating retry logic and
making sure we have appropriate options to control those client-side retries as
well as options exposed via the JDBC url would be a very good thing, but I need
to put some more thought into the knobs that are actually necessary (to avoid
death-by-configuratin).
Let me file another issue to look at this for 1.9.0.
> Automatically retry request when a server fails to deliver an HTTP response
> ---------------------------------------------------------------------------
>
> Key: CALCITE-1315
> URL: https://issues.apache.org/jira/browse/CALCITE-1315
> Project: Calcite
> Issue Type: Bug
> Components: avatica
> Reporter: Josh Elser
> Assignee: Josh Elser
> Fix For: avatica-1.9.0
>
>
> I've been running my load-balancer test using HAProxy between an Avatica
> client and multiple avatica servers. With the help of [~speleato], we've
> noticed that we're still seeing some intermittent failures in the client when
> we take down a backend server.
> The client reports a NoHttpResponseException:
> {noformat}
> java.lang.RuntimeException: org.apache.http.NoHttpResponseException:
> localhost:8888 failed to respond
> at
> org.apache.calcite.avatica.remote.AvaticaCommonsHttpClientImpl.send(AvaticaCommonsHttpClientImpl.java:149)
> at
> org.apache.calcite.avatica.remote.RemoteProtobufService._apply(RemoteProtobufService.java:44)
> at
> org.apache.calcite.avatica.remote.ProtobufService.apply(ProtobufService.java:69)
> at
> org.apache.calcite.avatica.remote.RemoteMeta$14.call(RemoteMeta.java:307)
> at
> org.apache.calcite.avatica.remote.RemoteMeta$14.call(RemoteMeta.java:305)
> at
> org.apache.calcite.avatica.AvaticaConnection.invokeWithRetries(AvaticaConnection.java:715)
> at
> org.apache.calcite.avatica.remote.RemoteMeta.fetch(RemoteMeta.java:304)
> at
> org.apache.calcite.avatica.MetaImpl$FetchIterator.moveNext(MetaImpl.java:908)
> at
> org.apache.calcite.avatica.MetaImpl$FetchIterator.next(MetaImpl.java:893)
> at
> org.apache.calcite.avatica.util.IteratorCursor.next(IteratorCursor.java:46)
> at
> org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:218)
> at sqlline.BufferedRows.<init>(BufferedRows.java:37)
> at sqlline.SqlLine.print(SqlLine.java:1649)
> at sqlline.Commands.execute(Commands.java:833)
> at sqlline.Commands.sql(Commands.java:732)
> at sqlline.SqlLine.dispatch(SqlLine.java:807)
> at sqlline.SqlLine.runCommands(SqlLine.java:1710)
> at sqlline.Commands.run(Commands.java:1285)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
> at sqlline.SqlLine.dispatch(SqlLine.java:803)
> at sqlline.SqlLine.initArgs(SqlLine.java:613)
> at sqlline.SqlLine.begin(SqlLine.java:656)
> at sqlline.SqlLine.start(SqlLine.java:398)
> at sqlline.SqlLine.main(SqlLine.java:292)
> at
> org.apache.phoenix.queryserver.client.SqllineWrapper.main(SqllineWrapper.java:83)
> Caused by: org.apache.http.NoHttpResponseException: localhost:8888 failed to
> respond
> at
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
> at
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
> at
> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
> at
> org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
> at
> org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
> at
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
> at
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
> at
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
> at
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
> at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
> at
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
> at
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at
> org.apache.calcite.avatica.remote.AvaticaCommonsHttpClientImpl.execute(AvaticaCommonsHttpClientImpl.java:157)
> at
> org.apache.calcite.avatica.remote.AvaticaCommonsHttpClientImpl.send(AvaticaCommonsHttpClientImpl.java:134)
> ... 28 more
> {noformat}
> I'm pretty sure what's happening (and what I can't see as something which we
> can change in the loadbalancer itself), is that a client submits a request to
> the LB which is then sent to the backend server, the backend server starts
> processing that request, but fails to ever respond. Instead of the LB sending
> back an HTTP/503 (as we've seen it do before), the client just gets a
> terminated connection while waiting for the response. I'm not surprised that
> HAProxy does not automatically retry this for us.
> So, the Avatica client can just retry this instead like we do on HTTP/503's
> already.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
