[
https://issues.apache.org/jira/browse/CASSANDRA-8653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14971286#comment-14971286
]
Sam Tunnicliffe edited comment on CASSANDRA-8653 at 10/23/15 4:23 PM:
----------------------------------------------------------------------
bq. This ticket mentions trunk, but any reason to think 3.0 is immune to that?
It's actually the 3.0 that the test upgrades to, or it would be except it's
currently skipped (waiting on CASSANDRA-9704, so should be re-enabled). When I
run locally I see no auth problems and the test completes as expected. It fails
though because of an unexpected ERROR in the log of node1, which is thrown just
after the last node is upgraded:
{noformat}
ERROR [HintsDispatcher:2] 2015-10-23 17:18:53,942 CassandraDaemon.java:195 -
Exception in thread Thread[HintsDispatcher:2,1,main]
java.lang.RuntimeException: java.nio.file.NoSuchFileException:
/home/sam/.ccm/repository/gitCOLONcassandra-3.0/data/hints/ac459445-1f7f-45f2-b9a8-2b185df34845-1445617063586-1.hints
at
org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:55)
~[main/:na]
at org.apache.cassandra.io.util.ChannelProxy.<init>(ChannelProxy.java:66)
~[main/:na]
at
org.apache.cassandra.hints.ChecksummedDataInput.open(ChecksummedDataInput.java:63)
~[main/:na]
at org.apache.cassandra.hints.HintsReader.open(HintsReader.java:77)
~[main/:na]
at
org.apache.cassandra.hints.HintsDispatcher.create(HintsDispatcher.java:71)
~[main/:na]
at
org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:242)
~[main/:na]
at
org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:219)
~[main/:na]
at
org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:198)
~[main/:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_60]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_60]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
Caused by: java.nio.file.NoSuchFileException:
/home/sam/.ccm/repository/gitCOLONcassandra-3.0/data/hints/ac459445-1f7f-45f2-b9a8-2b185df34845-1445617063586-1.hints
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
~[na:1.8.0_60]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
~[na:1.8.0_60]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
~[na:1.8.0_60]
at
sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
~[na:1.8.0_60]
at java.nio.channels.FileChannel.open(FileChannel.java:287) ~[na:1.8.0_60]
at java.nio.channels.FileChannel.open(FileChannel.java:335) ~[na:1.8.0_60]
at
org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:51)
~[main/:na]
... 12 common frames omitted
{noformat}
was (Author: beobal):
b.q This ticket mentions trunk, but any reason to think 3.0 is immune to that?
It's actually the 3.0 that the test upgrades to, or it would be except it's
currently skipped (waiting on CASSANDRA-9704, so should be re-enabled). When I
run locally I see no auth problems and the test completes as expected. It fails
though because of an unexpected ERROR in the log of node1, which is thrown just
after the last node is upgraded:
{noformat}
RROR [HintsDispatcher:2] 2015-10-23 17:18:53,942 CassandraDaemon.java:195 -
Exception in thread Thread[HintsDispatcher:2,1,main]
java.lang.RuntimeException: java.nio.file.NoSuchFileException:
/home/sam/.ccm/repository/gitCOLONcassandra-3.0/data/hints/ac459445-1f7f-45f2-b9a8-2b185df34845-1445617063586-1.hints
at
org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:55)
~[main/:na]
at org.apache.cassandra.io.util.ChannelProxy.<init>(ChannelProxy.java:66)
~[main/:na]
at
org.apache.cassandra.hints.ChecksummedDataInput.open(ChecksummedDataInput.java:63)
~[main/:na]
at org.apache.cassandra.hints.HintsReader.open(HintsReader.java:77)
~[main/:na]
at
org.apache.cassandra.hints.HintsDispatcher.create(HintsDispatcher.java:71)
~[main/:na]
at
org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:242)
~[main/:na]
at
org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:219)
~[main/:na]
at
org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:198)
~[main/:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_60]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_60]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
Caused by: java.nio.file.NoSuchFileException:
/home/sam/.ccm/repository/gitCOLONcassandra-3.0/data/hints/ac459445-1f7f-45f2-b9a8-2b185df34845-1445617063586-1.hints
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
~[na:1.8.0_60]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
~[na:1.8.0_60]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
~[na:1.8.0_60]
at
sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
~[na:1.8.0_60]
at java.nio.channels.FileChannel.open(FileChannel.java:287) ~[na:1.8.0_60]
at java.nio.channels.FileChannel.open(FileChannel.java:335) ~[na:1.8.0_60]
at
org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:51)
~[main/:na]
... 12 common frames omitted
{noformat}
> Upgrading to trunk with auth throws exception
> ---------------------------------------------
>
> Key: CASSANDRA-8653
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8653
> Project: Cassandra
> Issue Type: Bug
> Reporter: Philip Thompson
> Assignee: Sam Tunnicliffe
> Fix For: 3.0.0
>
> Attachments: node1.log, node2.log, node3.log
>
>
> When running Sam's upgrade_internal_auth_dtest, I am seeing the following
> exception (amongst others) in the log file of the second node to be upgraded
> to trunk from 2.1:
> {code}
> ERROR [GossipStage:1] 2015-01-20 13:46:21,679 CassandraDaemon.java:170 -
> Exception in thread Thread[GossipStage:1,5,main]
> java.lang.NoClassDefFoundError:
> org/apache/cassandra/transport/Event$TopologyChange$Change
> at
> org.apache.cassandra.transport.Server$EventNotifier.onJoinCluster(Server.java:374)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:1668)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.onChange(StorageService.java:1384)
> ~[main/:na]
> at
> org.apache.cassandra.gms.Gossiper.doOnChangeNotifications(Gossiper.java:1094)
> ~[main/:na]
> at
> org.apache.cassandra.gms.Gossiper.applyNewStates(Gossiper.java:1076)
> ~[main/:na]
> at
> org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1034)
> ~[main/:na]
> at
> org.apache.cassandra.gms.GossipDigestAckVerbHandler.doVerb(GossipDigestAckVerbHandler.java:58)
> ~[main/:na]
> 1554 - Node /127.0.0.1 state jump to normal
> ERROR [GossipStage:1] 2015-01-20 13:46:21,679 CassandraDaemon.java
> :170 - Exception in thread Thread[GossipStage:1,5,main]
> java.lang.NoClassDefFoundError: org/apache/cassandra/transport/Eve
> nt$TopologyChange$Change
> at org.apache.cassandra.transport.Server$EventNotifier.onJ
> oinCluster(Server.java:374) ~[main/:na]
> at org.apache.cassandra.service.StorageService.handleState
> Normal(StorageService.java:1668) ~[main/:na]
> at org.apache.cassandra.service.StorageService.onChange(St
> orageService.java:1384) ~[main/:na]
> at org.apache.cassandra.gms.Gossiper.doOnChangeNotificatio
> ns(Gossiper.java:1094) ~[main/:na]
> at org.apache.cassandra.gms.Gossiper.applyNewStates(Gossip
> er.java:1076) ~[main/:na]
> at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gos
> siper.java:1034) ~[main/:na]
> at org.apache.cassandra.gms.GossipDigestAckVerbHandler.doV
> erb(GossipDigestAckVerbHandler.java:58) ~[main/:na]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
> ~[main/:na]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[na:1.7.0_67]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[na:1.7.0_67]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_67]
> Caused by: java.lang.ClassNotFoundException:
> org.apache.cassandra.transport.Event$TopologyChange$Change
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> ~[na:1.7.0_67]
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> ~[na:1.7.0_67]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[na:1.7.0_67]
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> ~[na:1.7.0_67]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> ~[na:1.7.0_67]
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> ~[na:1.7.0_67]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> ~[na:1.7.0_67]
> ... 11 common frames omitted
> WARN [Thread-10] 2015-01-20 13:46:34,500 IncomingTcpConnection.java:91 -
> UnknownColumnFamilyException reading from socket; closing
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=5f2fbdad-91f1-3946-bd25-d5da3a5c35ec
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
> ~[main/:na]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
> ~[main/:na]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
> ~[main/:na]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
> ~[main/:na]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
> ~[main/:na]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> ~[main/:na]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
> ~[main/:na]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
> ~[main/:na]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
> ~[main/:na]
> {code}
> I have included the logs of all three nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
