[
https://issues.apache.org/jira/browse/CASSANDRA-19602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18133558#comment-18133558
]
Dmitry Konstantinov commented on CASSANDRA-19602:
-------------------------------------------------
it looks like it is caused by a truncated copy of build/dtest-4.0.22.jar. A
Jenkins unstash read the stash while another split was overwriting it and left
a partial file behind. The fallback build then skipped that jar because it only
checks that the file exists.
{code}
All 23 failures in #691 are the same ClassNotFoundException:
o.a.c.distributed.impl.IsolatedExecutor. They happen in about 1.5 s, in
AbstractCluster$Wrapper.newInstance, right after "testing upgrade from 4.0.22".
Splits 1–2 passed completely. The only tests that passed in splits 3–6 don't
start from 4.0 (for example CompactStoragePagingWithProtocolV41Test).
The ls -l in each split's stage log shows the difference:
┌────────┬─────────────────────────────────────────────────┬────────────────────────┐
│ Splits │ dtest-4.0.22.jar size │ Result
│
├────────┼─────────────────────────────────────────────────┼────────────────────────┤
│ 1, 2 │ 59,077,772 B │ all pass
│
├────────┼─────────────────────────────────────────────────┼────────────────────────┤
│ 3–6 │ 24,952,832 B (exactly 6092 × 4096), mtime 09:33 │ every 4.0→X test
fails │
└────────┴─────────────────────────────────────────────────┴────────────────────────┘
The race sits in buildJVMDTestJars in .jenkins/Jenkinsfile. Each upgrade split
tries unstash jvm_dtests_amd64_11. On any error it builds the jars and stashes
them again under the same name. Timeline from the pipeline step API:
- 09:23:42 – split 1: unstash fails with "No such saved stash". It builds the
jars and stashes them at 09:28:16.
- 09:26:17 – split 2: gets the same "No such saved stash" because split 1
hasn't finished. It builds the jars, then at 09:33:15 logs "Warning:
overwriting stash" while rewriting the tar.gz. That write took until 09:34:31.
- 09:33:23 – split 3: unstashes in the middle of that rewrite. The step fails
after 3.6 s with IOException: Failed to extract jvm_dtests_amd64_11.tar.gz and
leaves a partial build/dtest-4.0.22.jar. The mtime is 09:33, the extraction
time, because the untar aborted before it could set the original mtime.
- Split 3 fallback build: _build_all_dtest_jars only checks [ -f
"${DIST_DIR}/dtest-${version}.jar" ]. So its log says "Skipping dtest jar build
for branch cassandra-4.0 as …/dtest-4.0.22.jar already exists" (09:35:07), and
it builds the other jars.
- 09:36:58 – split 3 overwrites the stash with this bad set. Splits 4, 5 and 6
then unstash it successfully and all get the truncated jar.
{code}
> Test Failure: jvm-dtest-upgrade failing on ClassNotFoundException
> IsolatedExecutor
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-19602
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19602
> Project: Apache Cassandra
> Issue Type: Bug
> Components: CI
> Reporter: Ekaterina Dimitrova
> Assignee: Dmitry Konstantinov
> Priority: Normal
> Labels: green_ci
> Fix For: 5.0.x, 6.0.x, 7.x
>
>
> As seen below plenty of java upgrade tests are failing on trunk:
> https://ci-cassandra.apache.org/job/Cassandra-trunk/1880/#showFailuresLink
> {code:java}
> Tests / jvm-dtest-upgrade jdk11 3/8 /
> org.apache.cassandra.distributed.upgrade.ClusterMetadataUpgradeHarryTest.simpleUpgradeTest-_jdk11_x86_64
> Tests / jvm-dtest-upgrade jdk11 3/8 /
> org.apache.cassandra.distributed.upgrade.CompactStorageHiddenColumnTest.testHiddenColumnWithCompactStorage-_jdk11_x86_64
> Tests / jvm-dtest-upgrade jdk11 3/8 /
> org.apache.cassandra.distributed.upgrade.MixedModeAvailabilityUpgradedCoordinatorTest.testAvailability-_jdk11_x86_64
> Tests / jvm-dtest-upgrade jdk11 3/8 /
> org.apache.cassandra.distributed.upgrade.MixedModeReadRepairDeleteTest.mixedModeReadRepairDeleteRow-_jdk11_x86_64
> Tests / jvm-dtest-upgrade jdk11 3/8 /
> org.apache.cassandra.distributed.upgrade.MixedModeReadRepairDeleteTest.mixedModeReadRepairDeletePartition-_jdk11_x86_64
> Tests / jvm-dtest-upgrade jdk11 3/8 /
> org.apache.cassandra.distributed.upgrade.UpgradeTest.simpleUpgradeWithNetworkAndGossipTest-_jdk11_x86_64
> Tests / jvm-dtest-upgrade jdk11 4/8 /
> org.apache.cassandra.distributed.upgrade.ClusterMetadataUpgradeHintsTest.upgradeWithHintsTest-_jdk11_x86_64
> Tests / jvm-dtest-upgrade jdk11 4/8 /
> org.apache.cassandra.distributed.upgrade.CompactStorageImplicitNullInClusteringTest.testImplicitNullInClusteringWithCompactStorage-_jdk11_x86_64
> Tests / jvm-dtest-upgrade jdk11 4/8 /
> org.apache.cassandra.distributed.upgrade.MixedModeAvailabilityUpgradedReplicaTest.testAvailability-_jdk11_x86_64
> Tests / jvm-dtest-upgrade jdk11 4/8 /
> org.apache.cassandra.distributed.upgrade.MixedModeReadRepairWriteTest.mixedModeReadRepairInsert-_jdk11_x86_64
> Tests / jvm-dtest-upgrade jdk11 4/8 /
> org.apache.cassandra.distributed.upgrade.MixedModeReadRepairWriteTest.mixedModeReadRepairUpdate-_jdk11_x86_64
> Tests / jvm-dtest-upgrade jdk11 5/8 /
> org.apache.cassandra.distributed.upgrade.ClusterMetadataUpgradeHostIdTest.upgradeHostIdUpdateTest-_jdk11_x86_64
> Tests / jvm-dtest-upgrade jdk11 5/8 /
> org.apache.cassandra.distributed.upgrade.CompactStoragePagingTest.testPagingWithCompactStorage-_jdk11_x86_64
> Tests / jvm-dtest-upgrade jdk11 5/8 /
> org.apache.cassandra.distributed.upgrade.MixedModeConsistencyV30Test.testConsistency-_jdk11_x86_64
> Tests / jvm-dtest-upgrade jdk11 5/8 /
> org.apache.cassandra.distributed.upgrade.MixedModeRepairTest.testRepairDuringMajorUpgrade-_jdk11_x86_64
> Tests / jvm-dtest-upgrade jdk11 6/8 /
> org.apache.cassandra.distributed.upgrade.ClusterMetadataUpgradeIgnoreHostsTest.upgradeIgnoreHostsTest-_jdk11_x86_64
> Tests / jvm-dtest-upgrade jdk11 6/8 /
> org.apache.cassandra.distributed.upgrade.CompactStoragePagingWithProtocolV40Test.testPagingWithCompactStorageMultipleClusterings-_jdk11_x86_64
> Tests / jvm-dtest-upgrade jdk11 6/8 /
> org.apache.cassandra.distributed.upgrade.CompactStoragePagingWithProtocolV40Test.testPagingWithCompactStorageSingleClustering-_jdk11_x86_64
> Tests / jvm-dtest-upgrade jdk11 6/8 /
> org.apache.cassandra.distributed.upgrade.CompactStoragePagingWithProtocolV40Test.testPagingWithCompactStorageWithoutClustering-_jdk11_x86_64
> Tests / jvm-dtest-upgrade jdk11 6/8 /
> org.apache.cassandra.distributed.upgrade.MixedModeFrom3LoggedBatchTest.testSimpleStrategy-_jdk11_x86_64
> Tests / jvm-dtest-upgrade jdk11 6/8 /
> org.apache.cassandra.distributed.upgrade.MixedModeTTLOverflowAfterUpgradeTest.testTTLOverflowAfterUpgrade-_jdk11_x86_64
> Tests / jvm-dtest-upgrade jdk11 7/8 /
> org.apache.cassandra.distributed.upgrade.ClusterMetadataUpgradeIgnoreHostTest.upgradeIgnoreHostsTest-_jdk11_x86_64
> Tests / jvm-dtest-upgrade jdk11 7/8 /
> org.apache.cassandra.distributed.upgrade.CompactStoragePagingWithProtocolV41Test.testPagingWithCompactStorageMultipleClusterings-_jdk11_x86_64
> Tests / jvm-dtest-upgrade jdk11 7/8 /
> org.apache.cassandra.distributed.upgrade.CompactStoragePagingWithProtocolV41Test.testPagingWithCompactStorageSingleClustering-_jdk11_x86_64
> Tests / jvm-dtest-upgrade jdk11 7/8 /
> org.apache.cassandra.distributed.upgrade.CompactStoragePagingWithProtocolV41Test.testPagingWithCompactStorageWithoutClustering-_jdk11_x86_64
> Tests / jvm-dtest-upgrade jdk11 7/8 /
> org.apache.cassandra.distributed.upgrade.MixedModeFrom3ReplicationTest.testSimpleStrategy-_jdk11_x86_64
> Tests / jvm-dtest-upgrade jdk11 7/8 /
> org.apache.cassandra.distributed.upgrade.MixedModeTTLOverflowDuringUpgradeTest.testTTLOverflowDuringUpgrade-_jdk11_x86_64
> Tests / jvm-dtest-upgrade jdk11 8/8 /
> org.apache.cassandra.distributed.upgrade.ClusterMetadataUpgradeTest.simpleUpgradeTest-_jdk11_x86_64
> Tests / jvm-dtest-upgrade jdk11 8/8 /
> org.apache.cassandra.distributed.upgrade.DropCompactStorageNullClusteringValuesTest.testNullClusteringValues-_jdk11_x86_64
> Tests / jvm-dtest-upgrade jdk11 8/8 /
> org.apache.cassandra.distributed.upgrade.MixedModeFrom3UnloggedBatchTest.testSimpleStrategy-_jdk11_x86_64
> Tests / jvm-dtest-upgrade jdk11 8/8 /
> org.apache.cassandra.distributed.upgrade.MixedModeWritetimeOrTTLTest.testWritetimeOrTTLDuringUpgrade-_jdk11_x86_64
> {code}
> It was seen in some of the logs:
> {code:java}
> Caused by: java.lang.ClassNotFoundException:
> org.apache.cassandra.distributed.impl.IsolatedExecutor
> {code}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
