Josh Elser created CALCITE-1315:
-----------------------------------
Summary: Automatically retry request when a server fails to
deliver an HTTP response
Key: CALCITE-1315
URL: https://issues.apache.org/jira/browse/CALCITE-1315
Project: Calcite
Issue Type: Bug
Components: avatica
Reporter: Josh Elser
Assignee: Josh Elser
Fix For: avatica-1.9.0
I've been running my load-balancer test using HAProxy between an Avatica client
and multiple avatica servers. With the help of [~speleato], we've noticed that
we're still seeing some intermittent failures in the client when we take down a
backend server.
The client reports a NoHttpResponseException:
{noformat}
java.lang.RuntimeException: org.apache.http.NoHttpResponseException:
localhost:8888 failed to respond
at
org.apache.calcite.avatica.remote.AvaticaCommonsHttpClientImpl.send(AvaticaCommonsHttpClientImpl.java:149)
at
org.apache.calcite.avatica.remote.RemoteProtobufService._apply(RemoteProtobufService.java:44)
at
org.apache.calcite.avatica.remote.ProtobufService.apply(ProtobufService.java:69)
at
org.apache.calcite.avatica.remote.RemoteMeta$14.call(RemoteMeta.java:307)
at
org.apache.calcite.avatica.remote.RemoteMeta$14.call(RemoteMeta.java:305)
at
org.apache.calcite.avatica.AvaticaConnection.invokeWithRetries(AvaticaConnection.java:715)
at
org.apache.calcite.avatica.remote.RemoteMeta.fetch(RemoteMeta.java:304)
at
org.apache.calcite.avatica.MetaImpl$FetchIterator.moveNext(MetaImpl.java:908)
at
org.apache.calcite.avatica.MetaImpl$FetchIterator.next(MetaImpl.java:893)
at
org.apache.calcite.avatica.util.IteratorCursor.next(IteratorCursor.java:46)
at
org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:218)
at sqlline.BufferedRows.<init>(BufferedRows.java:37)
at sqlline.SqlLine.print(SqlLine.java:1649)
at sqlline.Commands.execute(Commands.java:833)
at sqlline.Commands.sql(Commands.java:732)
at sqlline.SqlLine.dispatch(SqlLine.java:807)
at sqlline.SqlLine.runCommands(SqlLine.java:1710)
at sqlline.Commands.run(Commands.java:1285)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
at sqlline.SqlLine.dispatch(SqlLine.java:803)
at sqlline.SqlLine.initArgs(SqlLine.java:613)
at sqlline.SqlLine.begin(SqlLine.java:656)
at sqlline.SqlLine.start(SqlLine.java:398)
at sqlline.SqlLine.main(SqlLine.java:292)
at
org.apache.phoenix.queryserver.client.SqllineWrapper.main(SqllineWrapper.java:83)
Caused by: org.apache.http.NoHttpResponseException: localhost:8888 failed to
respond
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
at
org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at
org.apache.calcite.avatica.remote.AvaticaCommonsHttpClientImpl.execute(AvaticaCommonsHttpClientImpl.java:157)
at
org.apache.calcite.avatica.remote.AvaticaCommonsHttpClientImpl.send(AvaticaCommonsHttpClientImpl.java:134)
... 28 more
{noformat}
I'm pretty sure what's happening (and what I can't see as something which we
can change in the loadbalancer itself), is that a client submits a request to
the LB which is then sent to the backend server, the backend server starts
processing that request, but fails to ever respond. Instead of the LB sending
back an HTTP/503 (as we've seen it do before), the client just gets a
terminated connection while waiting for the response. I'm not surprised that
HAProxy does not automatically retry this for us.
So, the Avatica client can just retry this instead like we do on HTTP/503's
already.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
