caiconghui opened a new issue #3368:
URL: https://github.com/apache/incubator-doris/issues/3368
The main purpose of this proposal is to make doris stream load more rubust
with higher concurrent peformance.
Now, in our production environment, we mainly use stream load to load
business data into doris.
And we found the following problem:
1. the transaction processing in GlobalTransactionMgr performance is poor,
because the transaction lock is global without any isolation, for some
operation we need to iterate the whole map.
2. Fe thrift server use Executors.newFixedThreadPool and
Executors.newCachedThreadPool to construct thread pool, which may cause oom or
too many thread to be created and then crash
3. the rpc cache client num is not limited in be and when rpc client
timeout, the request task is still blocked in fe, the client encounter
rpc_timeout and retry, it may cause disaster for fe sometimes.
4. the tablet writer open cost too much time, because when user doesn't
specify the partition, the all tablet writer would be opened.
Implementation
So the following work we need to implement:
1. Support db level isolation for transaction processing and use
ArrayDeque to stored the finished transaction
2. Remove all code that used newFixedThreadPool or newCachedThreadPool,
construct the thread pool explicitly, support thread pool metric monitor
3. use client pool to manage rpc client, set the core and max cache client
num, use try lock instead of lock to support server task be can be canceled in
time due to timeout
4. open the tablet writer when needed not open all when partitions of
table are not specified.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
