[ 
https://issues.apache.org/jira/browse/CASSANDRA-19215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17828423#comment-17828423
 ] 

Alex Petrov commented on CASSANDRA-19215:
-----------------------------------------

[~curlylrt] we've determined that this change might be insufficient and have 
worked on a larger change that might be a better fit. I hope to submit a 
reworked version this week.

> "Query start time" in native transport request threads should be the task 
> enqueue time
> --------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-19215
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-19215
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Messaging/Client
>            Reporter: Runtian Liu
>            Assignee: Alex Petrov
>            Priority: Normal
>             Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x
>
>         Attachments: ci_summary.html, result_details.tar.gz
>
>
> Recently, our Cassandra 4.0.6 cluster experienced an outage due to a surge in 
> expensive traffic from the application side. This surge involved a large 
> volume of costly read queries, which took a considerable amount of time to 
> process on the server side. The client had timeout settings; if a request 
> timed out, it might trigger the sending of new requests. Since the server 
> nodes were overloaded, numerous nodes had hundreds of thousands of tasks 
> queued in the Native-Transport-Request pending queue. I expected that once 
> the application ceased sending requests, the server node would quickly return 
> to normal, as most requests in the queue were over half an hour old and 
> should have timed out rapidly, clearing the queue. However, it actually took 
> an hour to clear the native transport's pending queue, even with native 
> transport disabled. Upon examining the code, I noticed that for read/write 
> requests, the 
> [queryStartNanoTime|https://github.com/apache/cassandra/blob/cassandra-4.0/src/java/org/apache/cassandra/transport/Dispatcher.java#L78],
>  which determines if a request has timed out, only begins when the task 
> starts processing. This means that no matter how long a request has been 
> pending, it doesn't contribute to the timeout. I believe this is incorrect. 
> The timer should start when the Cassandra server receives the request or when 
> it enqueues the task, not when the request/task begins processing. This way, 
> an overloaded node with many pending tasks can quickly discard timed-out 
> requests and recover from an outage once new requests stop.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

Reply via email to