[ https://issues.apache.org/jira/browse/CASSANDRA-10756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15022232#comment-15022232 ]
Stefan Podkowinski commented on CASSANDRA-10756: ------------------------------------------------ It's a bit unfortunate that the netty promises will not yield an exception or any result at all in case the executor has already been shut down. Adding a simple guard to not execute destroy twice seems to prevent the issue to happen. > Timeout failures in NativeTransportService.testConcurrentDestroys unit test > --------------------------------------------------------------------------- > > Key: CASSANDRA-10756 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10756 > Project: Cassandra > Issue Type: Bug > Reporter: Joel Knighton > > History of test on trunk > [here|http://cassci.datastax.com/job/trunk_testall/lastCompletedBuild/testReport/org.apache.cassandra.service/NativeTransportServiceTest/testConcurrentDestroys/history/]. > I've seen these failures across 3.0/trunk for a while. I ran the test looping > locally for a while and the timeout is fairly easy to reproduce. The timeout > appears to be an indefinite hang and not a timing issue. > When the timeout occurs, the following stack trace is at the end of the logs > for the unit test. > {code} > ERROR [ForkJoinPool.commonPool-worker-1] 2015-11-22 21:30:53,635 Failed to > submit a listener notification task. Event loop shut down? > java.util.concurrent.RejectedExecutionException: event executor terminated > at > io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:745) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:322) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:728) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.util.concurrent.DefaultPromise.execute(DefaultPromise.java:671) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.util.concurrent.DefaultPromise.notifyLateListener(DefaultPromise.java:641) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:138) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:93) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:28) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.group.DefaultChannelGroupFuture.<init>(DefaultChannelGroupFuture.java:116) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.group.DefaultChannelGroup.close(DefaultChannelGroup.java:275) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.group.DefaultChannelGroup.close(DefaultChannelGroup.java:167) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > org.apache.cassandra.transport.Server$ConnectionTracker.closeAll(Server.java:277) > [main/:na] > at org.apache.cassandra.transport.Server.close(Server.java:180) > [main/:na] > at org.apache.cassandra.transport.Server.stop(Server.java:116) > [main/:na] > at java.util.Collections$SingletonSet.forEach(Collections.java:4767) > ~[na:1.8.0_60] > at > org.apache.cassandra.service.NativeTransportService.stop(NativeTransportService.java:136) > ~[main/:na] > at > org.apache.cassandra.service.NativeTransportService.destroy(NativeTransportService.java:144) > ~[main/:na] > at > org.apache.cassandra.service.NativeTransportServiceTest.lambda$withService$102(NativeTransportServiceTest.java:201) > ~[classes/:na] > at java.util.stream.IntPipeline$3$1.accept(IntPipeline.java:233) > ~[na:1.8.0_60] > at > java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:110) > ~[na:1.8.0_60] > at java.util.Spliterator$OfInt.forEachRemaining(Spliterator.java:693) > ~[na:1.8.0_60] > at > java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) > ~[na:1.8.0_60] > at > java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) > ~[na:1.8.0_60] > at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747) > ~[na:1.8.0_60] > at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721) > ~[na:1.8.0_60] > at java.util.stream.AbstractTask.compute(AbstractTask.java:316) > ~[na:1.8.0_60] > at > java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) > ~[na:1.8.0_60] > at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) > ~[na:1.8.0_60] > at > java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) > ~[na:1.8.0_60] > at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) > ~[na:1.8.0_60] > at > java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) > ~[na:1.8.0_60] > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)