Stefania created CASSANDRA-11320:
------------------------------------
Summary: Improve backoff policy for cqlsh COPY FROM
Key: CASSANDRA-11320
URL: https://issues.apache.org/jira/browse/CASSANDRA-11320
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Stefania
Assignee: Stefania
Fix For: 3.x
Currently we have an exponential back-off policy in COPY FROM that kicks in
when timeouts are received. However there are two limitations:
* it does not cover new requests and therefore we may not back-off sufficiently
to give time to an overloaded server to recover
* the pause is performed in the receiving thread and therefore we may not
process server messages quickly enough
There is a static throttling mechanism in rows per second from feeder to worker
processes (the INGESTRATE) but the feeder has no idea of the load of each
worker process. However it's easy to keep track of how many chunks a worker
process has yet to read by introducing a bounded semaphore.
The idea is to move the back-off pauses to the worker processes main thread so
as to include all messages, new and retries, not just the retries that timed
out. The worker process will not read new chunks during the back-off pauses,
and the feeder process can then look at the number of pending chunks before
sending new chunks to a worker process.
[~aholmber], [~aweisberg] what do you think?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
