[
https://issues.apache.org/jira/browse/CASSANDRA-21640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Semb Wever updated CASSANDRA-21640:
-------------------------------------------
Description:
Three faults in the Jenkins pipeline, each measured on a Kubernetes clone of
ci-cassandra.apache.org. CI only; no production code changes.
h3. 1. generateTestReports queues behind the whole test fan-out
It runs on {{{}cassandra-medium{}}}, which is also the label of ~460 test
splits, and it is the build's last task, so it begins only once the fan-out has
drained: 109 minutes of a 163 minute build, for ~3 minutes of work. Within it,
each {{xargs -P}} and {{xz -T}} sizes itself from {{{}nproc{}}}, which answers
for the node and not for the container's cgroup quota: measured at 8 against a
2 cpu limit, so every fan-out ran eight ways under a quota of two. The
per-target {{ant junitreport}} loop is also serial.
h3. 2. Agent pods churn instead of queueing
Every podTemplate gives a pod 30 seconds to become ready
({{{}slaveConnectTimeout{}}}), and {{agent.waitForPodSec}} is 180. Only a warm
node meets 30 seconds; a cold one is an instance launch, a boot, a kubelet join
and two image pulls. Too short a deadline does not queue, it churns: the
controller deletes the pod, the build asks again, and the nodes are still not
there. Observed in one 24 minute window: 1,110 pods created and deleted, 0
agents connected, and 483 {{KubernetesProvisioningLimits ... went below zero}}
warnings, the plugin losing count of its own caps. The cloud meanwhile
provisions nodes for pods that no longer exist.
Separately, the caps do not match measured demand. Over a pre-commit tests
stage, medium held all 100 of its agents with 202 tasks queued behind them and
large held all 160 with 467 queued, while small used 19 pods in the whole build
and never held more than 2 at once.
h3. 3. No build reports which cell came near its timeout
A cell that reaches {{timeout(time: command.timeout_hours)}} is retried once
and then fails the build with {{{}Retryable interruption: Timeout has been
exceeded{}}}, naming neither the cell nor how close the others came.
{{_split_tests}} deals an alphabetically sorted class list round-robin with
{{{}split -n r/K/N{}}}, which balances the count of classes and knows nothing
of their duration, so long classes can collide. On builds cassandra-eks-k8s #14
to #19, {{test-burn}} split 3 of 4 was the build's worst cell every time, 47.1
to 58.2 minutes against a 60 minute deadline and 2.0 to 2.6 times its target's
median.
h3. Patch
https://github.com/apache/cassandra/compare/trunk...thelastpickle:cassandra:mck/ci-perf/trunk
* {{generateTestReports}} takes a new {{cassandra-report}} label on a node
pool of its own, added as the {{agent-dind-report}} podTemplate. A site with no
spare nodes can put the label on its medium agents.
* The report merge runs one {{ant junitreport}} per target, three at a time
under an explicit {{{}-Xmx{}}}, with dind's memory limit raised to match.
* A {{cpus()}} shell function reads the cgroup v2 quota from
{{/proc/self/cgroup}} and {{{}cpu.max{}}}, falling back to {{nproc}} where
there is no limit or on cgroup v1. Every {{xargs -P}} and {{xz -T}} uses it.
* {{slaveConnectTimeout}} 600 and {{waitForPodSec}} 900, the two deadlines on
one provisioning attempt, so both had to move. {{idleMinutes}} 5, which returns
a node inside the autoscaler's 10 minute scale-down window.
* {{instanceCap}} to measured demand cut to the account's quotas: small 20,
report 4, medium 150, large 306. {{containerCap}} is their sum, so no pool's
share is decided by which holds its agents longest, and
{{maxRequestsPerHostStr}} scales with it.
* New {{{}.build/ci/cell_balance.py{}}}: per target, how evenly the splits
divided, what the worst cell ran, and how much of a cell was setup rather than
tests. Printed to the console and appended to {{{}ci_summary.html{}}}. Guarded,
so it never fails a build.
* {{test-burn}} takes 5 splits. 5 divides none of the gaps between its three
long classes, where 4 divides one of them; its worst chunk is 24.1 minutes
against 46.7.
* Splits stay round-robin, deliberately: which split holds a class stays
predictable from the sorted list alone, so the lever is the split count.
was:
Three faults in the Jenkins pipeline, each measured on a Kubernetes clone of
ci-cassandra.apache.org. CI only; no production code changes.
h3. 1. generateTestReports queues behind the whole test fan-out
It runs on {{{}cassandra-medium{}}}, which is also the label of ~460 test
splits, and it is the build's last task, so it begins only once the fan-out has
drained: 109 minutes of a 163 minute build, for ~3 minutes of work. Within it,
each {{xargs -P}} and {{xz -T}} sizes itself from {{{}nproc{}}}, which answers
for the node and not for the container's cgroup quota: measured at 8 against a
2 cpu limit, so every fan-out ran eight ways under a quota of two. The
per-target {{ant junitreport}} loop is also serial.
h3. 2. Agent pods churn instead of queueing
Every podTemplate gives a pod 30 seconds to become ready
({{{}slaveConnectTimeout{}}}), and {{agent.waitForPodSec}} is 180. Only a warm
node meets 30 seconds; a cold one is an instance launch, a boot, a kubelet join
and two image pulls. Too short a deadline does not queue, it churns: the
controller deletes the pod, the build asks again, and the nodes are still not
there. Observed in one 24 minute window: 1,110 pods created and deleted, 0
agents connected, and 483 {{KubernetesProvisioningLimits ... went below zero}}
warnings, the plugin losing count of its own caps. The cloud meanwhile
provisions nodes for pods that no longer exist.
Separately, the caps do not match measured demand. Over a pre-commit tests
stage, medium held all 100 of its agents with 202 tasks queued behind them and
large held all 160 with 467 queued, while small used 19 pods in the whole build
and never held more than 2 at once.
h3. 3. No build reports which cell came near its timeout
A cell that reaches {{timeout(time: command.timeout_hours)}} is retried once
and then fails the build with {{{}Retryable interruption: Timeout has been
exceeded{}}}, naming neither the cell nor how close the others came.
{{_split_tests}} deals an alphabetically sorted class list round-robin with
{{{}split -n r/K/N{}}}, which balances the count of classes and knows nothing
of their duration, so long classes can collide. On builds cassandra-eks-k8s #14
to #19, {{test-burn}} split 3 of 4 was the build's worst cell every time, 47.1
to 58.2 minutes against a 60 minute deadline and 2.0 to 2.6 times its target's
median.
h3. Patch
* {{generateTestReports}} takes a new {{cassandra-report}} label on a node
pool of its own, added as the {{agent-dind-report}} podTemplate. A site with no
spare nodes can put the label on its medium agents.
* The report merge runs one {{ant junitreport}} per target, three at a time
under an explicit {{{}-Xmx{}}}, with dind's memory limit raised to match.
* A {{cpus()}} shell function reads the cgroup v2 quota from
{{/proc/self/cgroup}} and {{{}cpu.max{}}}, falling back to {{nproc}} where
there is no limit or on cgroup v1. Every {{xargs -P}} and {{xz -T}} uses it.
* {{slaveConnectTimeout}} 600 and {{waitForPodSec}} 900, the two deadlines on
one provisioning attempt, so both had to move. {{idleMinutes}} 5, which returns
a node inside the autoscaler's 10 minute scale-down window.
* {{instanceCap}} to measured demand cut to the account's quotas: small 20,
report 4, medium 150, large 306. {{containerCap}} is their sum, so no pool's
share is decided by which holds its agents longest, and
{{maxRequestsPerHostStr}} scales with it.
* New {{{}.build/ci/cell_balance.py{}}}: per target, how evenly the splits
divided, what the worst cell ran, and how much of a cell was setup rather than
tests. Printed to the console and appended to {{{}ci_summary.html{}}}. Guarded,
so it never fails a build.
* {{test-burn}} takes 5 splits. 5 divides none of the gaps between its three
long classes, where 4 divides one of them; its worst chunk is 24.1 minutes
against 46.7.
* Splits stay round-robin, deliberately: which split holds a class stays
predictable from the sorted list alone, so the lever is the split count.
> CI: generateTestReports queues behind the test fan-out, agent pods churn
> instead of queueing, and reports on split balance
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-21640
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21640
> Project: Apache Cassandra
> Issue Type: Improvement
> Components: CI
> Reporter: Michael Semb Wever
> Assignee: Michael Semb Wever
> Priority: Normal
> Fix For: 5.0.x, 6.0.x, 7.x
>
>
> Three faults in the Jenkins pipeline, each measured on a Kubernetes clone of
> ci-cassandra.apache.org. CI only; no production code changes.
> h3. 1. generateTestReports queues behind the whole test fan-out
> It runs on {{{}cassandra-medium{}}}, which is also the label of ~460 test
> splits, and it is the build's last task, so it begins only once the fan-out
> has drained: 109 minutes of a 163 minute build, for ~3 minutes of work.
> Within it, each {{xargs -P}} and {{xz -T}} sizes itself from {{{}nproc{}}},
> which answers for the node and not for the container's cgroup quota: measured
> at 8 against a 2 cpu limit, so every fan-out ran eight ways under a quota of
> two. The per-target {{ant junitreport}} loop is also serial.
> h3. 2. Agent pods churn instead of queueing
> Every podTemplate gives a pod 30 seconds to become ready
> ({{{}slaveConnectTimeout{}}}), and {{agent.waitForPodSec}} is 180. Only a
> warm node meets 30 seconds; a cold one is an instance launch, a boot, a
> kubelet join and two image pulls. Too short a deadline does not queue, it
> churns: the controller deletes the pod, the build asks again, and the nodes
> are still not there. Observed in one 24 minute window: 1,110 pods created and
> deleted, 0 agents connected, and 483 {{KubernetesProvisioningLimits ... went
> below zero}} warnings, the plugin losing count of its own caps. The cloud
> meanwhile provisions nodes for pods that no longer exist.
> Separately, the caps do not match measured demand. Over a pre-commit tests
> stage, medium held all 100 of its agents with 202 tasks queued behind them
> and large held all 160 with 467 queued, while small used 19 pods in the whole
> build and never held more than 2 at once.
> h3. 3. No build reports which cell came near its timeout
> A cell that reaches {{timeout(time: command.timeout_hours)}} is retried once
> and then fails the build with {{{}Retryable interruption: Timeout has been
> exceeded{}}}, naming neither the cell nor how close the others came.
> {{_split_tests}} deals an alphabetically sorted class list round-robin with
> {{{}split -n r/K/N{}}}, which balances the count of classes and knows nothing
> of their duration, so long classes can collide. On builds cassandra-eks-k8s
> #14 to #19, {{test-burn}} split 3 of 4 was the build's worst cell every time,
> 47.1 to 58.2 minutes against a 60 minute deadline and 2.0 to 2.6 times its
> target's median.
> h3. Patch
> https://github.com/apache/cassandra/compare/trunk...thelastpickle:cassandra:mck/ci-perf/trunk
> * {{generateTestReports}} takes a new {{cassandra-report}} label on a node
> pool of its own, added as the {{agent-dind-report}} podTemplate. A site with
> no spare nodes can put the label on its medium agents.
> * The report merge runs one {{ant junitreport}} per target, three at a time
> under an explicit {{{}-Xmx{}}}, with dind's memory limit raised to match.
> * A {{cpus()}} shell function reads the cgroup v2 quota from
> {{/proc/self/cgroup}} and {{{}cpu.max{}}}, falling back to {{nproc}} where
> there is no limit or on cgroup v1. Every {{xargs -P}} and {{xz -T}} uses it.
> * {{slaveConnectTimeout}} 600 and {{waitForPodSec}} 900, the two deadlines
> on one provisioning attempt, so both had to move. {{idleMinutes}} 5, which
> returns a node inside the autoscaler's 10 minute scale-down window.
> * {{instanceCap}} to measured demand cut to the account's quotas: small 20,
> report 4, medium 150, large 306. {{containerCap}} is their sum, so no pool's
> share is decided by which holds its agents longest, and
> {{maxRequestsPerHostStr}} scales with it.
> * New {{{}.build/ci/cell_balance.py{}}}: per target, how evenly the splits
> divided, what the worst cell ran, and how much of a cell was setup rather
> than tests. Printed to the console and appended to {{{}ci_summary.html{}}}.
> Guarded, so it never fails a build.
> * {{test-burn}} takes 5 splits. 5 divides none of the gaps between its three
> long classes, where 4 divides one of them; its worst chunk is 24.1 minutes
> against 46.7.
> * Splits stay round-robin, deliberately: which split holds a class stays
> predictable from the sorted list alone, so the lever is the split count.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
