caiconghui opened a new issue #3743: URL: https://github.com/apache/incubator-doris/issues/3743
Now, Doris still get bad performance for High concurrency stream load， and I add some metric to trace the some bottleneck for issue #3696, and modify some be config to make it a litte better. However, the locked range in db level is too big, when one slow task thread get db lock, it can easily block other tasks. I do some test for stream load, sadly, for db level lock, the total time cost for 5000 stream load in one java client is not stable, sometimes about 15s, sometimes need 30s. By analyzing the return result info, I find some commit task performance are poor, because commit task can only executed one by one when get db write lock. And I try to use table lock instead of db lock to do some testing, now, the whole time cost is much more stable , about 11s for whole 5000 stream load. Although some publish task still need 1s to complete, but table lock can significantly reduce the lock competition. So I make this proposal to make some improvement to stream load performance. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
