caiconghui edited a comment on pull request #3703:
URL: https://github.com/apache/incubator-doris/pull/3703#issuecomment-635199557
As a conclusion, the http server is based on libevent , which use
reactor model to process http request, some http requests are processed by one
thread. it assumes that every process function in libevent thread is non
blocked, but there are some blocked operations in our current implementaion,
such as rpc calls to fe and wait for write data operation finish. once a stream
load is blocked, it will affect the following stream load, which cannot be
processed in time.
So This PR manily add some mertics to trace the Potential bottlenecks
when stream load get bad performance. and change the default value of http
server thread num to reduce the num of http requeset for per libevent server
thread for one time.
For now. the main time cost for stream load seems to be commit and
publis txn, write data operation if we load many data into doris. If the stream
load operation really cost much time, what we can do is to think a way to
escape the heavy stream load blocking other http request
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
