morningman opened a new issue #5280: URL: https://github.com/apache/incubator-doris/issues/5280
Doris' load performance is very bad in some cases. Especially when using `broker load` to load large quantities of data, the system resources may be idle, but the load is very slow. Currently I found the following problems: 1. The execution plan of Broker Load has only one concurrency on a single node, so even if the system resources are idle, the load speed cannot be improved by increasing the concurrency. 2. Memtable is written to disk serially. By default, a disk has two flush threads, and the current flush thread pool strategy is serial execution. In fact, each memtable can be written to disk in parallel. 3. A loadChannel may correspond to data sent by multiple OlapTableSinks, and because of the problem of LoadChannel lock granularity, the data receiving is a serial operation. Therefore, in some cases, even if the BE node is added, the load speed cannot be improved. I am trying to solve these problem. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
