[
https://issues.apache.org/jira/browse/KUDU-3500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17751771#comment-17751771
]
ASF subversion and git services commented on KUDU-3500:
-------------------------------------------------------
Commit 6c049687f60e90cbdac6f6ec039a528d13664a6b in kudu's branch
refs/heads/master from Alexey Serbin
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=6c049687f ]
KUDU-3500 don't start operations timed out in prepare queue
While troubleshooting a performance issue where the prepare queue for
a tablet was very long, I noticed that tablet servers start write
operations that correspond to RPCs that have already timed out. Most
likely, the client that sent the RPC had already detected the timeout
and expected that the write would have failed already.
As a simple optimization, this patch updates the logic of the OpDriver
class to respond with TimedOut error status right away when a write
operation that has already timed out while waiting in the prepare queue
is dispatched to the prepare thread. That helps with clearing the queue
and processing not-yet-timed-out requests from the queue faster,
increasing the overall robustness of a tablet server when the load
is high and the node's CPU and disk IO bandwidth are saturated.
A new tablet metric 'ops_timed_out_in_prepare_queue' is introduced to
track the number of WriteRequestPB RPCs timed out in the tablet's prepare
queue and responded with TimedOut error status even before starting
the PREPARE phase for the corresponding operation.
This patch also adds a new test to cover the new functionality.
Change-Id: I202ce6b5e425439b50c0751d7f7406e69b8e751a
Reviewed-on: http://gerrit.cloudera.org:8080/20300
Tested-by: Kudu Jenkins
Reviewed-by: Abhishek Chennaka <[email protected]>
> Don't start write operations timed out in the tablet's prepare queue
> --------------------------------------------------------------------
>
> Key: KUDU-3500
> URL: https://issues.apache.org/jira/browse/KUDU-3500
> Project: Kudu
> Issue Type: Improvement
> Components: tserver
> Reporter: Alexey Serbin
> Assignee: Alexey Serbin
> Priority: Major
>
> While troubleshooting one performance issue where the prepare queue of a
> tablet was very long, I noticed that tablet servers start write operations
> that correspond to RPCs that have already timed out. Most likely, the client
> that sent the RPC had already detected the timeout and expected that the
> write would have failed already, so there isn't much sense to start such
> operations anyway.
> As a simple optimization, tablet servers shouldn't even start the PREPARE
> phase for such operations, but respond with TimedOut error status right away
> when dispatched them to the prepare thread. Doing so would help with
> clearing the prepare queue and processing not-yet-timed-out requests from the
> queue faster, increasing the overall robustness of a tablet server when the
> load is high and the node's CPU and disk IO bandwidth are saturated.
> A new metric should be introduced to track the number of WriteRequestPB RPCs
> timed out in the prepare queue and responded with TimedOut error status
> before starting the PREPARE phase for the corresponding operations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
