[
https://issues.apache.org/jira/browse/CALCITE-1315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser resolved CALCITE-1315.
---------------------------------
Resolution: Fixed
Fixed in
https://git1-us-west.apache.org/repos/asf?p=calcite.git;a=commit;h=799e1b083b57d2295681d3e58c4d88563f7ccc91
> Automatically retry request when a server fails to deliver an HTTP response
> ---------------------------------------------------------------------------
>
> Key: CALCITE-1315
> URL: https://issues.apache.org/jira/browse/CALCITE-1315
> Project: Calcite
> Issue Type: Bug
> Components: avatica
> Reporter: Josh Elser
> Assignee: Josh Elser
> Fix For: avatica-1.9.0
>
>
> I've been running my load-balancer test using HAProxy between an Avatica
> client and multiple avatica servers. With the help of [~speleato], we've
> noticed that we're still seeing some intermittent failures in the client when
> we take down a backend server.
> The client reports a NoHttpResponseException:
> {noformat}
> java.lang.RuntimeException: org.apache.http.NoHttpResponseException:
> localhost:8888 failed to respond
> at
> org.apache.calcite.avatica.remote.AvaticaCommonsHttpClientImpl.send(AvaticaCommonsHttpClientImpl.java:149)
> at
> org.apache.calcite.avatica.remote.RemoteProtobufService._apply(RemoteProtobufService.java:44)
> at
> org.apache.calcite.avatica.remote.ProtobufService.apply(ProtobufService.java:69)
> at
> org.apache.calcite.avatica.remote.RemoteMeta$14.call(RemoteMeta.java:307)
> at
> org.apache.calcite.avatica.remote.RemoteMeta$14.call(RemoteMeta.java:305)
> at
> org.apache.calcite.avatica.AvaticaConnection.invokeWithRetries(AvaticaConnection.java:715)
> at
> org.apache.calcite.avatica.remote.RemoteMeta.fetch(RemoteMeta.java:304)
> at
> org.apache.calcite.avatica.MetaImpl$FetchIterator.moveNext(MetaImpl.java:908)
> at
> org.apache.calcite.avatica.MetaImpl$FetchIterator.next(MetaImpl.java:893)
> at
> org.apache.calcite.avatica.util.IteratorCursor.next(IteratorCursor.java:46)
> at
> org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:218)
> at sqlline.BufferedRows.<init>(BufferedRows.java:37)
> at sqlline.SqlLine.print(SqlLine.java:1649)
> at sqlline.Commands.execute(Commands.java:833)
> at sqlline.Commands.sql(Commands.java:732)
> at sqlline.SqlLine.dispatch(SqlLine.java:807)
> at sqlline.SqlLine.runCommands(SqlLine.java:1710)
> at sqlline.Commands.run(Commands.java:1285)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
> at sqlline.SqlLine.dispatch(SqlLine.java:803)
> at sqlline.SqlLine.initArgs(SqlLine.java:613)
> at sqlline.SqlLine.begin(SqlLine.java:656)
> at sqlline.SqlLine.start(SqlLine.java:398)
> at sqlline.SqlLine.main(SqlLine.java:292)
> at
> org.apache.phoenix.queryserver.client.SqllineWrapper.main(SqllineWrapper.java:83)
> Caused by: org.apache.http.NoHttpResponseException: localhost:8888 failed to
> respond
> at
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
> at
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
> at
> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
> at
> org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
> at
> org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
> at
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
> at
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
> at
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
> at
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
> at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
> at
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
> at
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at
> org.apache.calcite.avatica.remote.AvaticaCommonsHttpClientImpl.execute(AvaticaCommonsHttpClientImpl.java:157)
> at
> org.apache.calcite.avatica.remote.AvaticaCommonsHttpClientImpl.send(AvaticaCommonsHttpClientImpl.java:134)
> ... 28 more
> {noformat}
> I'm pretty sure what's happening (and what I can't see as something which we
> can change in the loadbalancer itself), is that a client submits a request to
> the LB which is then sent to the backend server, the backend server starts
> processing that request, but fails to ever respond. Instead of the LB sending
> back an HTTP/503 (as we've seen it do before), the client just gets a
> terminated connection while waiting for the response. I'm not surprised that
> HAProxy does not automatically retry this for us.
> So, the Avatica client can just retry this instead like we do on HTTP/503's
> already.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
