[jira] [Updated] (CASSANDRA-11320) Improve backoff policy for cqlsh COPY FROM
[ https://issues.apache.org/jira/browse/CASSANDRA-11320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tyler Hobbs updated CASSANDRA-11320: Status: Open (was: Patch Available) > Improve backoff policy for cqlsh COPY FROM > -- > > Key: CASSANDRA-11320 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11320 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Stefania >Assignee: Stefania > Labels: doc-impacting > Fix For: 3.0.x, 3.x > > > Currently we have an exponential back-off policy in COPY FROM that kicks in > when timeouts are received. However there are two limitations: > * it does not cover new requests and therefore we may not back-off > sufficiently to give time to an overloaded server to recover > * the pause is performed in the receiving thread and therefore we may not > process server messages quickly enough > There is a static throttling mechanism in rows per second from feeder to > worker processes (the INGESTRATE) but the feeder has no idea of the load of > each worker process. However it's easy to keep track of how many chunks a > worker process has yet to read by introducing a bounded semaphore. > The idea is to move the back-off pauses to the worker processes main thread > so as to include all messages, new and retries, not just the retries that > timed out. The worker process will not read new chunks during the back-off > pauses, and the feeder process can then look at the number of pending chunks > before sending new chunks to a worker process. > [~aholmber], [~aweisberg] what do you think? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-11320) Improve backoff policy for cqlsh COPY FROM
[ https://issues.apache.org/jira/browse/CASSANDRA-11320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joshua McKenzie updated CASSANDRA-11320: Reviewer: Tyler Hobbs > Improve backoff policy for cqlsh COPY FROM > -- > > Key: CASSANDRA-11320 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11320 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Stefania >Assignee: Stefania > Labels: doc-impacting > Fix For: 3.0.x, 3.x > > > Currently we have an exponential back-off policy in COPY FROM that kicks in > when timeouts are received. However there are two limitations: > * it does not cover new requests and therefore we may not back-off > sufficiently to give time to an overloaded server to recover > * the pause is performed in the receiving thread and therefore we may not > process server messages quickly enough > There is a static throttling mechanism in rows per second from feeder to > worker processes (the INGESTRATE) but the feeder has no idea of the load of > each worker process. However it's easy to keep track of how many chunks a > worker process has yet to read by introducing a bounded semaphore. > The idea is to move the back-off pauses to the worker processes main thread > so as to include all messages, new and retries, not just the retries that > timed out. The worker process will not read new chunks during the back-off > pauses, and the feeder process can then look at the number of pending chunks > before sending new chunks to a worker process. > [~aholmber], [~aweisberg] what do you think? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-11320) Improve backoff policy for cqlsh COPY FROM
[ https://issues.apache.org/jira/browse/CASSANDRA-11320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefania updated CASSANDRA-11320: - Fix Version/s: 3.0.x > Improve backoff policy for cqlsh COPY FROM > -- > > Key: CASSANDRA-11320 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11320 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Stefania >Assignee: Stefania > Labels: doc-impacting > Fix For: 3.0.x, 3.x > > > Currently we have an exponential back-off policy in COPY FROM that kicks in > when timeouts are received. However there are two limitations: > * it does not cover new requests and therefore we may not back-off > sufficiently to give time to an overloaded server to recover > * the pause is performed in the receiving thread and therefore we may not > process server messages quickly enough > There is a static throttling mechanism in rows per second from feeder to > worker processes (the INGESTRATE) but the feeder has no idea of the load of > each worker process. However it's easy to keep track of how many chunks a > worker process has yet to read by introducing a bounded semaphore. > The idea is to move the back-off pauses to the worker processes main thread > so as to include all messages, new and retries, not just the retries that > timed out. The worker process will not read new chunks during the back-off > pauses, and the feeder process can then look at the number of pending chunks > before sending new chunks to a worker process. > [~aholmber], [~aweisberg] what do you think? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-11320) Improve backoff policy for cqlsh COPY FROM
[ https://issues.apache.org/jira/browse/CASSANDRA-11320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefania updated CASSANDRA-11320: - Tester: (was: Ariel Weisberg) Status: Patch Available (was: Awaiting Feedback) > Improve backoff policy for cqlsh COPY FROM > -- > > Key: CASSANDRA-11320 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11320 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Stefania >Assignee: Stefania > Labels: doc-impacting > Fix For: 3.x > > > Currently we have an exponential back-off policy in COPY FROM that kicks in > when timeouts are received. However there are two limitations: > * it does not cover new requests and therefore we may not back-off > sufficiently to give time to an overloaded server to recover > * the pause is performed in the receiving thread and therefore we may not > process server messages quickly enough > There is a static throttling mechanism in rows per second from feeder to > worker processes (the INGESTRATE) but the feeder has no idea of the load of > each worker process. However it's easy to keep track of how many chunks a > worker process has yet to read by introducing a bounded semaphore. > The idea is to move the back-off pauses to the worker processes main thread > so as to include all messages, new and retries, not just the retries that > timed out. The worker process will not read new chunks during the back-off > pauses, and the feeder process can then look at the number of pending chunks > before sending new chunks to a worker process. > [~aholmber], [~aweisberg] what do you think? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-11320) Improve backoff policy for cqlsh COPY FROM
[ https://issues.apache.org/jira/browse/CASSANDRA-11320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefania updated CASSANDRA-11320: - Tester: Ariel Weisberg Status: Awaiting Feedback (was: In Progress) > Improve backoff policy for cqlsh COPY FROM > -- > > Key: CASSANDRA-11320 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11320 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Stefania >Assignee: Stefania > Labels: doc-impacting > Fix For: 3.x > > > Currently we have an exponential back-off policy in COPY FROM that kicks in > when timeouts are received. However there are two limitations: > * it does not cover new requests and therefore we may not back-off > sufficiently to give time to an overloaded server to recover > * the pause is performed in the receiving thread and therefore we may not > process server messages quickly enough > There is a static throttling mechanism in rows per second from feeder to > worker processes (the INGESTRATE) but the feeder has no idea of the load of > each worker process. However it's easy to keep track of how many chunks a > worker process has yet to read by introducing a bounded semaphore. > The idea is to move the back-off pauses to the worker processes main thread > so as to include all messages, new and retries, not just the retries that > timed out. The worker process will not read new chunks during the back-off > pauses, and the feeder process can then look at the number of pending chunks > before sending new chunks to a worker process. > [~aholmber], [~aweisberg] what do you think? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-11320) Improve backoff policy for cqlsh COPY FROM
[ https://issues.apache.org/jira/browse/CASSANDRA-11320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefania updated CASSANDRA-11320: - Labels: doc-impacting (was: ) > Improve backoff policy for cqlsh COPY FROM > -- > > Key: CASSANDRA-11320 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11320 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Stefania >Assignee: Stefania > Labels: doc-impacting > Fix For: 3.x > > > Currently we have an exponential back-off policy in COPY FROM that kicks in > when timeouts are received. However there are two limitations: > * it does not cover new requests and therefore we may not back-off > sufficiently to give time to an overloaded server to recover > * the pause is performed in the receiving thread and therefore we may not > process server messages quickly enough > There is a static throttling mechanism in rows per second from feeder to > worker processes (the INGESTRATE) but the feeder has no idea of the load of > each worker process. However it's easy to keep track of how many chunks a > worker process has yet to read by introducing a bounded semaphore. > The idea is to move the back-off pauses to the worker processes main thread > so as to include all messages, new and retries, not just the retries that > timed out. The worker process will not read new chunks during the back-off > pauses, and the feeder process can then look at the number of pending chunks > before sending new chunks to a worker process. > [~aholmber], [~aweisberg] what do you think? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
