jenrryyou opened a new pull request #1516:
URL: https://github.com/apache/incubator-brpc/pull/1516
### 复现条件
1. channel的连接模式设置为single
2. 客户端并发发起创建stream的请求
2. 服务端出现限流或者overcrowded等状态（为了快速复现，可以把max_concurrency设置很小）
3. 客户端解析服务端返回数据，报解析错误（PARSE_ERROR_ABSOLUTELY_WRONG)
### 原因
#### 服务端限流或者拥堵，StreamAccept没有执行
当客户端使用baidu_rpc协议发起创建新的stream请求（创建Stream）时，服务端判断限流或者拥堵，直接报错，不再处理请求。(baidu_rpc_protocol.cpp)。这样的后果是服务端的`brpc::StreamAccept`没有执行，服务端stream
socket没有创建，controller的repsonse_stream就没有设置。
```cpp
void ProcessRpcRequest(InputMessageBase* msg_base) {
......
if (!server->IsRunning()) {
cntl->SetFailed(ELOGOFF, "Server is stopping");
break;
}
if (socket->is_overcrowded()) {
cntl->SetFailed(EOVERCROWDED, "Connection to %s is overcrowded",
butil::endpoint2str(socket->remote_side()).c_str());
break;
}
if (!server_accessor.AddConcurrency(cntl.get())) {
cntl->SetFailed(
ELIMIT, "Reached server's max_concurrency=%d",
server->options().max_concurrency);
break;
}
if (FLAGS_usercode_in_pthread && TooManyUserCode()) {
cntl->SetFailed(ELIMIT, "Too many user code to run when"
" -usercode_in_pthread is on");
break;
}
......
}
```
#### 服务端响应数据没有用Stream格式封装
服务端`ProcessRpcResponse`返回数据时如果存在服务端的stream
socket就通过stream协议封装数据，否则通过原始协议baidu_rpc返回数据。由于服务端的stream创建失败，stream socket
没有创建出来，于是返回数据是用baidu_rpc协议封装的数据。
#### 客户端解析响应出错
因为客户端多个stream共享host socket，host
socket的preferred协议已经为设置为PROTOCOL_STREAMING_RPC（有概率在发送新建的stream请求的时候preferred协议短暂设置为PROTOCOL_BAIDU_STD，这里也是存在race
issue的），这时收到一个PROTOCOL_BAIDU_STD的数据用就会抛PARSE_ERROR_ABSOLUTELY_WRONG的错误，导致客户端看不到真正的报错原因（封装在rpc格式里的报错信息）
```cpp
ParseResult InputMessenger::CutInputMessage(
Socket* m, size_t* index, bool read_eof) {
// Try preferred handler first. The preferred_index is set on last
// selection or by client.
if (preferred >= 0 && preferred <= max_index
&& _handlers[preferred].parse != NULL) {
ParseResult result =
_handlers[preferred].parse(&m->_read_buf, m, read_eof,
_handlers[preferred].arg);
if (result.is_ok() ||
result.error() == PARSE_ERROR_NOT_ENOUGH_DATA) {
*index = preferred;
return result;
} else if (result.error() != PARSE_ERROR_TRY_OTHERS) {
// Critical error, return directly.
LOG_IF(ERROR, result.error() == PARSE_ERROR_TOO_BIG_DATA)
<< "A message from " << m->remote_side()
<< "(protocol=" << _handlers[preferred].name
<< ") is bigger than " << FLAGS_max_body_size
<< " bytes, the connection will be closed."
" Set max_body_size to allow bigger messages";
return result;
}
if (m->CreatedByConnect() &&
// baidu_std may fall to streaming_rpc
(ProtocolType)preferred != PROTOCOL_BAIDU_STD) {
// The protocol is fixed at client-side, no need to try others.
LOG(ERROR) << "Fail to parse response from " << m->remote_side()
<< " by " << _handlers[preferred].name
<< " at client-side";
return MakeParseError(PARSE_ERROR_ABSOLUTELY_WRONG);
}
// Clear context before trying next protocol which probably has
// an incompatible context with the current one.
if (m->parsing_context()) {
m->reset_parsing_context(NULL);
}
m->set_preferred_index(-1);
}
```
### 问题修复
原来的逻辑是要服务端的stream创建成功，才用stream rpc协议封装数据。
为了修复数据解析问题，我们先检查has_remote_stream，如果有就把原来baidu_rpc协议格式的response封装到stream
rpc数据帧里发送给客户端。另外由于server可能返回INVALID_STREAM_ID，客户端ProcessRpcResponse 需要做对应处理。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
