[
https://issues.apache.org/jira/browse/CASSANDRA-18949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782378#comment-17782378
]
Ekaterina Dimitrova edited comment on CASSANDRA-18949 at 11/3/23 2:31 AM:
--------------------------------------------------------------------------
There is no more in the
[logs|https://nightlies.apache.org/cassandra/trunk/Cassandra-trunk-test/1814/Cassandra-trunk-test/jdk=jdk_11_latest,label=cassandra,split=1/]
of nightlies than the error reported here.
I tried to reproduce the reported failure on a commit before the fix of
CASSANDRA-18747 unsuccessfully - both locally and in CI -
[https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra?branch=18949-5.0]
One observation is that this test does not create keyspace_04, that one is
created by calling _prepareData,_ which is called by the other two test methods
in the same class but not the one that was reported -
testClearSnapshotWithOlderThanFlag and
testClearSnapshotWithOlderThanTimestampFlag.
So at the end I managed to reproduce by changing the Schema#unload method and
then running the whole test class (if I run the testClearSnapshot_RemoveByName
separately, it passes):
{code:java}
private synchronized void unload(KeyspaceMetadata ksm)
{
distributedKeyspaces = distributedKeyspaces.without(ksm.name);
if(!ksm.name.equals("keyspace_04"))
distributedAndLocalKeyspaces =
distributedAndLocalKeyspaces.without(ksm.name);
this.tableMetadataRefCache = tableMetadataRefCache.withRemovedRefs(ksm);
SchemaDiagnostics.metadataRemoved(this, ksm);
}{code}
I suspect after the fix in CASSANDRA-18747 we can close this one. [~blerer] ,
do you agree with this analysis?
was (Author: e.dimitrova):
There is no more in the logs of nightlies than the error reported here.
I tried to reproduce the reported failure on a commit before the fix of
CASSANDRA-18747 unsuccessfully - both locally and in CI -
[https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra?branch=18949-5.0]
One observation is that this test does not create keyspace_04, that one is
created by calling _prepareData,_ which is called by the other two test methods
in the same class but not the one that was reported -
testClearSnapshotWithOlderThanFlag and
testClearSnapshotWithOlderThanTimestampFlag.
So at the end I managed to reproduce by changing the Schema#unload method and
then running the whole test class (if I run the testClearSnapshot_RemoveByName
separately, it passes):
{code:java}
private synchronized void unload(KeyspaceMetadata ksm)
{
distributedKeyspaces = distributedKeyspaces.without(ksm.name);
if(!ksm.name.equals("keyspace_04"))
distributedAndLocalKeyspaces =
distributedAndLocalKeyspaces.without(ksm.name);
this.tableMetadataRefCache = tableMetadataRefCache.withRemovedRefs(ksm);
SchemaDiagnostics.metadataRemoved(this, ksm);
}{code}
I suspect after the fix in CASSANDRA-18747 we can close this one. [~blerer] ,
do you agree with this analysis?
> Test failure:
> org.apache.cassandra.tools.nodetool.ClearSnapshotTest.testClearSnapshot_RemoveByName-.jdk11.arch=x86_64.python2.7
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-18949
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18949
> Project: Cassandra
> Issue Type: Bug
> Components: CI
> Reporter: Ekaterina Dimitrova
> Priority: Normal
> Fix For: 5.0-rc
>
>
> Seen here:
> [https://ci-cassandra.apache.org/job/Cassandra-trunk/1747/testReport/org.apache.cassandra.tools.nodetool/ClearSnapshotTest/testClearSnapshot_RemoveByName__jdk11_arch_x86_64_python2_7/]
> h3.
> {code:java}
> Error Message
> [bin/nodetool, -p, 36753, -h, 127.0.0.1, snapshot, -t, some-name] exited with
> code 2 stderr: error: Unknown keyspace keyspace_04 -- StackTrace --
> java.lang.AssertionError: Unknown keyspace keyspace_04 at
> org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:324) at
> org.apache.cassandra.db.Keyspace.lambda$open$0(Keyspace.java:162) at
> org.apache.cassandra.utils.concurrent.LoadingMap.blockingLoadIfAbsent(LoadingMap.java:105)
> at
> org.apache.cassandra.schema.Schema.maybeAddKeyspaceInstance(Schema.java:251)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:162) at
> org.apache.cassandra.db.Keyspace.open(Keyspace.java:151) at
> com.google.common.collect.Iterators$6.transform(Iterators.java:828) at
> com.google.common.collect.TransformedIterator.next(TransformedIterator.java:52)
> at
> org.apache.cassandra.service.StorageService.takeSnapshot(StorageService.java:4356)
> at
> org.apache.cassandra.service.StorageService.takeSnapshot(StorageService.java:4221)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method) at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566) at
> sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71) at
> jdk.internal.reflect.GeneratedMethodAccessor1.invoke(Unknown Source) at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566) at
> java.base/sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:260) at
> java.management/com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> at
> java.management/com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> at
> java.management/com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> at
> java.management/com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at
> java.management/com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> at
> java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:809)
> at
> java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
> at
> java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1466)
> at
> java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1307)
> at
> java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1399)
> at
> java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:827)
> at java.base/jdk.internal.reflect.GeneratedMethodAccessor16.invoke(Unknown
> Source) at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566) at
> java.rmi/sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:359)
> at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:200) at
> java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:197) at
> java.base/java.security.AccessController.doPrivileged(Native Method) at
> java.rmi/sun.rmi.transport.Transport.serviceCall(Transport.java:196) at
> java.rmi/sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:562)
> at
> java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:796)
> at
> java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:677)
> at java.base/java.security.AccessController.doPrivileged(Native Method) at
> java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:676)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829) stdout: Requested
> creating snapshot(s) for [all keyspaces] with snapshot name [some-name] and
> options {skipFlush=false}
> Stacktrace
> junit.framework.AssertionFailedError: [bin/nodetool, -p, 36753, -h,
> 127.0.0.1, snapshot, -t, some-name] exited with code 2 stderr: error: Unknown
> keyspace keyspace_04 -- StackTrace -- java.lang.AssertionError: Unknown
> keyspace keyspace_04 at
> org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:324) at
> org.apache.cassandra.db.Keyspace.lambda$open$0(Keyspace.java:162) at
> org.apache.cassandra.utils.concurrent.LoadingMap.blockingLoadIfAbsent(LoadingMap.java:105)
> at
> org.apache.cassandra.schema.Schema.maybeAddKeyspaceInstance(Schema.java:251)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:162) at
> org.apache.cassandra.db.Keyspace.open(Keyspace.java:151) at
> com.google.common.collect.Iterators$6.transform(Iterators.java:828) at
> com.google.common.collect.TransformedIterator.next(TransformedIterator.java:52)
> at
> org.apache.cassandra.service.StorageService.takeSnapshot(StorageService.java:4356)
> at
> org.apache.cassandra.service.StorageService.takeSnapshot(StorageService.java:4221)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method) at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at jdk.internal.reflect.GeneratedMethodAccessor1.invoke(Unknown Source) at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at
> java.management/com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> at
> java.management/com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> at
> java.management/com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> at
> java.management/com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at
> java.management/com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> at
> java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:809)
> at
> java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
> at
> java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1466)
> at
> java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1307)
> at
> java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1399)
> at
> java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:827)
> at java.base/jdk.internal.reflect.GeneratedMethodAccessor16.invoke(Unknown
> Source) at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at
> java.rmi/sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:359)
> at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:200) at
> java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:197) at
> java.base/java.security.AccessController.doPrivileged(Native Method) at
> java.rmi/sun.rmi.transport.Transport.serviceCall(Transport.java:196) at
> java.rmi/sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:562)
> at
> java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:796)
> at
> java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:677)
> at java.base/java.security.AccessController.doPrivileged(Native Method) at
> java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:676)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829) stdout: Requested
> creating snapshot(s) for [all keyspaces] with snapshot name [some-name] and
> options {skipFlush=false} at
> org.apache.cassandra.tools.ToolRunner$ToolResult.assertExitCode(ToolRunner.java:445)
> at
> org.apache.cassandra.tools.ToolRunner$ToolResult.assertOnExitCode(ToolRunner.java:439)
> at
> org.apache.cassandra.tools.ToolRunner$ToolResult.assertOnCleanExit(ToolRunner.java:492)
> at
> org.apache.cassandra.tools.ToolRunner$ToolResult.assertOnCleanExit(ToolRunner.java:487)
> at
> org.apache.cassandra.tools.nodetool.ClearSnapshotTest.testClearSnapshot_RemoveByName(ClearSnapshotTest.java:78)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method) at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
