DarvenDuan opened a new pull request, #39103:
URL: https://github.com/apache/doris/pull/39103
## Proposed changes
The current implementation of Doris only allows to commit a single stream
2PC transaction per TLoadTxn2PCRequest. This can lead to performance issues in
high concurrency scenarios, such as with the flink-doris-connector:
Each 2PC commit request acquires a write lock on the table and the database.
With multiple concurrent write requests using stream load 2PC to a single
table, Doris sequentially locks and unlocks the table and database multiple
times during the commit phase. If these transactions were batched into a single
request, Doris could process them collectively, requiring only one lock on the
table and database. Additionally, Doris can write the edit log in a batch mode,
which could further enhance performance.
In my test case, I tested 1000 commit requests: 100 tables and 10 concurrent
writes per table.
commit cost:
- **1.533s** in original commit model
- **0.038s** in batch commit model
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
