[
https://issues.apache.org/jira/browse/RATIS-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17867904#comment-17867904
]
Duong commented on RATIS-2129:
------------------------------
One of the reasons for the long time to acquire readLock is that RaftLog
performs some inefficient things when holding writeLock.
Look at RaftLogbase.appendEntry implementation, some pieces can be moved out of
the writeLock, such as "preAppendTransaction" or the validation based on
"getSerializedSize" (which can be expensive for 1st call in a protobuf object).
{code:java}
try(AutoCloseableLock writeLock = writeLock()) {
final long nextIndex = getNextIndex();
// This is called here to guarantee strict serialization of callback
executions in case
// the SM wants to attach a logic depending on ordered execution in the log
commit order.
try {
operation = operation.preAppendTransaction();
} catch (StateMachineException e) {
throw e;
} catch (IOException e) {
throw new StateMachineException(memberId, e);
}
// build the log entry after calling the StateMachine
final LogEntryProto e = operation.initLogEntry(term, nextIndex);
int entrySize = e.getSerializedSize();
if (entrySize > maxBufferSize) {
throw new StateMachineException(memberId, new RaftLogIOException(
"Log entry size " + entrySize + " exceeds the max buffer limit of " +
maxBufferSize));
}
.... {code}
Also, in the profile of RaftLogbase.appendEntry, within the writeLock (and in
the critical path of the single thread to process incoming transaction), a
significant cost is spent on TermIndex.valueOf, which can be easily passed from
outside. This call is quite expensive because it involves putting TermIndex
into a global cache. !Screenshot 2024-07-22 at 4.40.07
PM.png|width=863,height=318!
> Low replication performance low because GrpcLogAppender is often blocked by
> RaftLog's readLock
> ----------------------------------------------------------------------------------------------
>
> Key: RATIS-2129
> URL: https://issues.apache.org/jira/browse/RATIS-2129
> Project: Ratis
> Issue Type: Bug
> Components: server
> Affects Versions: 3.1.0
> Reporter: Duong
> Priority: Major
> Labels: Performance, performance
> Attachments: Screenshot 2024-07-22 at 4.40.07 PM.png,
> dn_echo_leader_profile.html, image-2024-07-22-15-25-46-155.png
>
>
> Today, the GrpcLogAppender thread makes a lot of calls that need RaftLog's
> readLock. In an active environment, RaftLog is always busy appending
> transactions from clients, thus writeLock is frequently busy. This makes the
> replication performance slow.
> See the [^dn_echo_leader_profile.html], or in the picture below, the purple
> is the time taken to acquire readLock from RaftLog.
> # !image-2024-07-22-15-25-46-155.png|width=854,height=425!
> So far, I'm not sure if this is a regression from a recent change in
> 3.1.0/3.0.0, or if it's been always the case.
> A few early considerations:
> # The rate of calling RaftLog per GrpcLogAppender seems to be too high.
> Instead of calling RaftLog multiple, maybe the log appended can call once to
> obtain all the required information?
> # Can RaftLog expose those data without requiring a read lock?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
