caiconghui edited a comment on pull request #3703:
URL: https://github.com/apache/incubator-doris/pull/3703#issuecomment-635199557
As a conclusion, the http server is based on libevent , which use reactor
model to process http request, some http requests are processed by one thread.
it assumes that every process function in libevent thread is non blocked, but
there are some blocked operations in our current implementaion, such as rpc
calls to fe and wait for write data operation finish. once a stream load is
blocked, it will affect the following stream load, which cannot be processed in
time.
So This PR manily add some mertics to trace the Potential bottlenecks when
stream load get bad performance. and change the default value of http server
thread num to reduce the num of http requeset for per libevent server thread
for one time.
For now. the main time cost for stream load seems to be commit and
publis txn, write data operation if we load many data into doris. If the stream
load operation really cost much time, what we can do is to think a way to
escape the heavy stream load blocking other http request
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
