[
https://issues.apache.org/jira/browse/KUDU-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16054778#comment-16054778
]
Todd Lipcon commented on KUDU-2046:
-----------------------------------
[~aserbin] and I discussed this a bit on Slack on Friday. I also chatted with
[~danburkert] this morning about it. A few of the ideas we discussed:
*1 (Simple approach)*- even if the user specifies a long timeout like 30
seconds, we can send the Write RPCs with a short timeout to the leader (eg 5
seconds). Then, if it times out, we still have a lot of remaining time to go
look up new locations and hopefully send to the new leader. The downside is
that if the leader has not changed, then we risk just filling up network pipes
with our retries.
*2 (Fancier approach)* - we could introduce a concept such as a "pre-timeout".
Given a 30 second user-specified timeout, we'd send the write RPC to the leader
with that full timeout, but with a 5-second pre-timeout. If the pre-timeout
elapses, then we can fire off the write to the other two replicas (perhaps with
an optional cache refresh in there). These secondary requests would have the
remaining 25 seconds set as their timeout. When any request comes back
successful, we know the write was successful.
> Client should fail over timed out write operations
> --------------------------------------------------
>
> Key: KUDU-2046
> URL: https://issues.apache.org/jira/browse/KUDU-2046
> Project: Kudu
> Issue Type: Bug
> Components: client
> Affects Versions: 1.4.0
> Reporter: Todd Lipcon
>
> This is similar to KUDU-1034, which was partially fixed in
> 4263b037844fca595a35f99479fbb5765ba7a443. That fix was partial in the
> following ways:
> - it works when the client times out during negotiation, since the
> negotiation timeout is typically shorter than the total write operation
> timeout
> - if the client has an already-estabished connection, the Write RPC is sent
> with a timeout equal to the entire user-specified timeout. So, if the server
> side is hung, then the client will fully time out the user operation and
> return an error before it tries any other replica. The fix for KUDU-1034
> ensured that the server was marked down, thus fixing any *future* write
> attempt, but doesn't prevent the TimedOut error from bubbling up to the
> original caller.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
