Sergey Shelukhin created KNOX-755:
-------------------------------------
Summary: retry logic for replayBuffer limit errors is incorrect.
Key: KNOX-755
URL: https://issues.apache.org/jira/browse/KNOX-755
Project: Apache Knox
Issue Type: Bug
Reporter: Sergey Shelukhin
Hive receives corrupted thrift requests when using Knox with Hive with a large
query and insufficient replayBuffer:
{noformat}
org.apache.thrift.transport.TTransportException
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:354)
at
org.apache.thrift.protocol.TBinaryProtocol.readString(TBinaryProtocol.java:347)
at
org.apache.hive.service.cli.thrift.TExecuteStatementReq$TExecuteStatementReqStandardScheme.read(TExecuteStatementReq.java:618)
...
{noformat}
It seems that the retry logic for this error is incorrect, as follows (names
changed to generic):
{noformat}
2016-10-05 15:25:51,104 DEBUG http.wire (Wire.java:wire(63)) - >>
"[0x80][0x1][0x0][0x1][0x0][0x0][0x0][0x10]ExecuteStatement[0x0][0x0][0x0]...![0x88]SELECT
1 AS `number_of_records`,[\n]"
...
2016-10-05 15:25:51,117 DEBUG http.wire (Wire.java:wire(77)) - >> "
`tablename`.`columnn"
2016-10-05 15:25:51,118 DEBUG http.wire (Wire.java:wire(63)) - >> "[\r][\n]"
...
2016-10-05 15:25:51,119 INFO client.DefaultHttpClient
(DefaultRequestDirector.java:tryExecute(726)) - I/O exception
(java.io.IOException) caught when processing request: Hit replay buffer max
limit
2016-10-05 15:25:51,120 DEBUG client.DefaultHttpClient
(DefaultRequestDirector.java:tryExecute(731)) - Hit replay buffer max limit
java.io.IOException: Hit replay buffer max limit
at
org.apache.hadoop.gateway.dispatch.CappedBufferHttpEntity$ReplayStream.read(CappedBufferHttpEntity.java:143)
at java.io.InputStream.read(InputStream.java:101)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1792)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744)
at
org.apache.hadoop.gateway.dispatch.CappedBufferHttpEntity.writeTo(CappedBufferHttpEntity.java:93)
at
org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:98)
{noformat}
However, then it retries:
{noformat}
2016-10-05 15:25:51,121 INFO client.DefaultHttpClient
(DefaultRequestDirector.java:tryExecute(733)) - Retrying request
2016-10-05 15:25:51,121 DEBUG client.DefaultHttpClient
(DefaultRequestDirector.java:tryExecute(703)) - Reopening the direct connection.
{noformat}
After auth (for which the same incorrect request as below is sent, but not
parsed due to 401), it sends the thing again with correct auth header, as
follows:
{noformat}
2016-10-05 15:25:51,166 DEBUG client.DefaultHttpClient
(DefaultRequestDirector.java:tryExecute(713)) - Attempt 3 to execute request
2016-10-05 15:25:51,166 DEBUG conn.DefaultClientConnection
(DefaultClientConnection.java:sendRequestHeader(269)) - Sending request: POST
/cliservice?doAs=... HTTP/1.1
2016-10-05 15:25:51,167 DEBUG http.wire (Wire.java:wire(63)) - >> "POST
/cliservice?doAs=... HTTP/1.1[\r][\n]"
...
2016-10-05 15:25:51,169 DEBUG http.wire (Wire.java:wire(63)) - >>
"Authorization: Negotiate ...
2016-10-05 15:25:51,170 DEBUG http.wire (Wire.java:wire(63)) - >> "[\r][\n]"
...
2016-10-05 15:25:51,172 DEBUG http.wire (Wire.java:wire(63)) - >> "1000[\r][\n]"
2016-10-05 15:25:51,173 DEBUG http.wire (Wire.java:wire(63)) - >>
"[0x80][0x1][0x0][0x1][0x0][0x0][0x0][0x10]ExecuteStatement[0x0] ...
![0x88]SELECT 1 AS `number_of_records`,[\n]"
...
2016-10-05 15:25:51,186 DEBUG http.wire (Wire.java:wire(77)) - >> "
`tablename`.`columnn"
2016-10-05 15:25:51,187 DEBUG http.wire (Wire.java:wire(63)) - >> "[\r][\n]"
2016-10-05 15:25:51,187 DEBUG http.wire (Wire.java:wire(63)) - >> "1f3[\r][\n]"
2016-10-05 15:25:51,187 DEBUG http.wire (Wire.java:wire(63)) - >> "ther` AS
`anothercolumnnameother`,[\n]"
... rest of the query
{noformat}
Note that there's a gap at "columnn", where "columnname" should be.
This results in the above error when reading the request, and error 500 on
gateway side.
I think the retry logic should be fixed to send the correct buffer, or removed
for this type of error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
