[
https://issues.apache.org/jira/browse/CASSANDRA-20311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17927435#comment-17927435
]
Michael Semb Wever edited comment on CASSANDRA-20311 at 2/15/25 7:05 PM:
-------------------------------------------------------------------------
CI
- 5.0 pre-commit w/ upgrades:
[^ci_summary_thelastpickle_mck-20311-5.0_156.html] and
[^results_details_thelastpickle_mck-20311-5.0_156.tar.xz]
7 failures, unrelated.
was (Author: michaelsembwever):
CI
- 5.0 pre-commit w/ upgrades:
[^ci_summary_thelastpickle_mck-20311-5.0_156.html] and
[^results_details_thelastpickle_mck-20311-5.0_156.tar.xz]
> Adjust 5.0 and trunk Jenkinsfile's splits configuration
> --------------------------------------------------------
>
> Key: CASSANDRA-20311
> URL: https://issues.apache.org/jira/browse/CASSANDRA-20311
> Project: Apache Cassandra
> Issue Type: Task
> Components: CI
> Reporter: Michael Semb Wever
> Assignee: Michael Semb Wever
> Priority: Normal
> Fix For: 5.0.x, 5.x
>
> Attachments: ci_summary_thelastpickle_mck-20311-5.0_156.html,
> results_details_thelastpickle_mck-20311-5.0_156.tar.xz
>
>
> Using CASSANDRA-20157, analyse
> # what splits are problematic
> # what test types are configured with too many splits, or not enough
> *Analyse Process*
> The data could be graphed, for better visualisation, and identification of
> mean, stddev, and trends. But the following poor man's approach is fine for
> first pass.
> Download the jenkins consoleText logs we want to analyse
> {noformat}
> for i in $(seq $first_build $last_build) ; do
> wget -q https://ci-cassandra.apache.org/job/Cassandra-5.0/$i/consoleText
> bash -c "grep '] Time ' consoleText | awk -F']' '{print $2}' | sort -u >
> Cassandra-5.0_${i}_timings.txt"
> rm consoleText
> done
> {noformat}
> Alternative, if using nightlies.a.o webdav mount (the consoleText.xz files
> have been manually downloaded and put into place).
> {noformat}
> for i in $(seq $first_build $last_build) ; do
> bash -c "xzgrep '] Time '
> <nightlies_webdav_mount>/cassandra/Cassandra-5.0/${i}/consoleText.xz | awk
> -F']' '{print $2}' | sort -u > Cassandra-5.0_${i}_timings.txt"
> done
> {noformat}
> (1)
> {code}
> for i in $(seq $first_build $last_build) ; do grep " 01:0"
> Cassandra-5.0_${i}_timings.txt | awk '{print $3}' ; done | sort | uniq -c |
> sort -rn
> {code}
> Target those individual splits that have timed out (duration of 1 hour) the
> most.
> (2a)
> For test types that have too few splits (are timing out at the 1 hour mark
> too often).
> {code}
> for i in $(seq $first_build $last_build) ; do grep " 01:0"
> Cassandra-5.0_${i}_timings.txt | awk '{print $3}' ; done | awk -F_ '{print
> $1}' | sort | uniq -c | sort -rn
> {code}
> (2b)
> For test types that have too many splits (are finishing faster than ten
> minutes).
> {code}
> for i in $(seq $first_build $last_build) ; do grep " 00:0"
> Cassandra-5.0_${i}_timings.txt | awk '{print $3}' ; done | awk -F_ '{print
> $1}' | sort | uniq -c | sort -rn
> {code}
> This output will be weighted by those test types that have more splits (but
> that's ok because that's where we can save time / improve throughput most).
> Replace "Cassandra-5.0_${i}_timings.txt" with
> "Cassandra-trunk_${i}_timings.txt" for trunk analysis, naturally.
> *Results 5.0*
> builds 352 - 385
> (1)
> {noformat}
> 5 test-cdc_jdk11_python_3.8_no_amd64_2_8:
> 4 test-latest_jdk17_python_3.8_no_amd64_2_8:
> 4 dtest-upgrade_jdk11_python_3.8_no_amd64_40_64:
> 4 dtest-upgrade_jdk11_python_3.8_no_amd64_37_64:
> 3 test-system-keyspace-directory_jdk11_python_3.8_no_amd64_7_8:
> 3 test-oa_jdk17_python_3.8_no_amd64_5_8:
> 3 test-oa_jdk11_python_3.8_no_amd64_3_8:
> 3 test-compression_jdk17_python_3.8_no_amd64_8_8:
> 3 test-compression_jdk17_python_3.8_no_amd64_7_8:
> 3 test-compression_jdk17_python_3.8_no_amd64_2_8:
> 3 test-cdc_jdk11_python_3.8_no_amd64_4_8:
> 3 dtest-upgrade_jdk11_python_3.8_no_amd64_54_64:
> 3 dtest-upgrade_jdk11_python_3.8_no_amd64_53_64:
> 3 dtest-upgrade_jdk11_python_3.8_no_amd64_28_64:
> 3 dtest-upgrade-novnode_jdk11_python_3.8_no_amd64_10_64:
> 3 dtest-upgrade-large_jdk11_python_3.8_no_amd64_12_32:
> …
> {noformat}
> (format here is
> `<test_type>\-<jdk>\-<python>\-<cython>\-<arch>\-<split>\-<splits>:`)
> (2a)
> {noformat}
> 54 dtest-upgrade
> 43 dtest-upgrade-novnode
> 18 test-oa
> 18 test-compression
> 15 test-cdc
> 15 dtest-upgrade-large
> 14 test-system-keyspace-directory
> 14 test-latest
> 14 dtest-upgrade-novnode-large
> …
> {noformat}
> (2b)
> {noformat}
> 535 dtest-latest
> 495 dtest
> 341 long-test
> 306 dtest-novnode
> 167 jvm-dtest-upgrade
> 102 cqlsh-test
> 90 dtest-upgrade-large
> 90 dtest-large-latest
> 89 dtest-large
> 77 dtest-upgrade-novnode-large
> 49 stress-test
> 49 fqltool-test
> 34 dtest-large-novnode
> 24 simulator-dtest
> …
> {noformat}
> *Results trunk*
> builds 1988 - 2021
> (1)
> {noformat}
> 11 test-compression_jdk11_python_3.8_no_amd64_2_8:
> 10 test-system-keyspace-directory_jdk17_python_3.8_no_amd64_2_8:
> 9 dtest-upgrade_jdk11_python_3.8_no_amd64_1_64:
> 7 test_jdk17_python_3.8_no_amd64_4_8:
> 6 dtest-upgrade-novnode_jdk11_python_3.8_no_amd64_42_64:
> 4 test-oa_jdk17_python_3.8_no_amd64_7_8:
> 4 dtest-upgrade_jdk11_python_3.8_no_amd64_4_64:
> 3 test_jdk17_python_3.8_no_amd64_6_8:
> 3 test_jdk11_python_3.8_no_amd64_8_8:
> 3 test-oa_jdk11_python_3.8_no_amd64_2_8:
> 3 test-compression_jdk17_python_3.8_no_amd64_3_8:
> 3 dtest-upgrade_jdk11_python_3.8_no_amd64_63_64:
> 3 dtest-upgrade_jdk11_python_3.8_no_amd64_55_64:
> 3 dtest-upgrade_jdk11_python_3.8_no_amd64_36_64:
> 3 dtest-upgrade-novnode_jdk11_python_3.8_no_amd64_3_64:
> 3 dtest-upgrade-novnode_jdk11_python_3.8_no_amd64_37_64:
> 3 dtest-upgrade-novnode_jdk11_python_3.8_no_amd64_30_64:
> 3 dtest-upgrade-novnode_jdk11_python_3.8_no_amd64_2_64:
> 3 dtest-upgrade-novnode_jdk11_python_3.8_no_amd64_14_64:
> 3 dtest-upgrade-novnode-large_jdk11_python_3.8_no_amd64_7_32:
> 3 dtest-upgrade-large_jdk11_python_3.8_no_amd64_19_32:
> 3 dtest-upgrade-large_jdk11_python_3.8_no_amd64_17_32:
> …
> {noformat}
> (2a)
> {noformat}
> 66 dtest-upgrade
> 62 dtest-upgrade-novnode
> 23 test-system-keyspace-directory
> 23 test
> 22 test-compression
> 18 test-oa
> 17 test-latest
> 17 dtest-upgrade-large
> 14 jvm-dtest
> 14 dtest-upgrade-novnode-large
> …
> {noformat}
> (2b)
> {noformat}
> 334 long-test
> 298 dtest-latest
> 291 dtest
> 185 dtest-novnode
> 159 dtest-large-latest
> 151 dtest-large
> 98 cqlsh-test
> 86 dtest-upgrade-large
> 85 dtest-upgrade-novnode-large
> 52 dtest-large-novnode
> 51 stress-test
> 51 fqltool-test
> 45 jvm-dtest-upgrade
> 25 simulator-dtest
> …
> {noformat}
> *Observations*
> - dtest-upgrade (+variants) needs more splits
> - test (+variants) needs more splits
> - dtest (+variants) could reduce splits
> -- taking a closer look, the spread up to the one hour timeout value
> makes this difficult to change
> - long-test could reduce splits
> - dtest-large (+variants) could reduce splits
> - jvm-dtest-upgrade in 5.0 could reduce splits
> - dtest-upgrade-large (+variants) have a lot of variability (both <10 minutes
> and >1 hour)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
