[
https://issues.apache.org/jira/browse/CASSANDRA-5689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698565#comment-13698565
]
Alex Zarutin edited comment on CASSANDRA-5689 at 7/3/13 4:37 AM:
-----------------------------------------------------------------
General comments:
I use CCM based installed C* cluster, so I do not use init script to stop and
start it. So, what I did, I creaated subset of ..../debian/init script to be
able to run it from the shell against the cluster started by CCM. Platform is a
Mac laptop.
Number of the tests, including 1 node and 3 nodes cluster did not show any
NullPointer or any other exceptions.
Full repro steps and init1, init2, init3 scripts stripped to the functionality
of stopping cluster are attached.
was (Author: alexzar):
General comments:
I use CCM based installed C* cluster, so I do not use init script to stop and
start it. So, what I did, I creaated subset of ..../debian/init script to be
able to run it from the shell against the cluster started by CCM.
Comment regarding this particular issue. Node1 went down (in the logs) but
still stay as the java process after I run $nodetool drain
I) logs to see after running $nodetool disablegossip
==> /Users/alexzarutin/.ccm/test-1.2.6/node1/logs/system.log <==
WARN [RMI TCP Connection(2)-10.10.10.130] 2013-07-02 20:45:21,896
StorageService.java (line 288) Stopping gossip by operator request
INFO [RMI TCP Connection(2)-10.10.10.130] 2013-07-02 20:45:21,897
Gossiper.java (line 1108) Announcing shutdown
==> /Users/alexzarutin/.ccm/test-1.2.6/node2/logs/system.log <==
INFO [GossipStage:1] 2013-07-02 20:45:23,899 Gossiper.java (line 785)
InetAddress /127.0.0.1 is now DOWN
==> /Users/alexzarutin/.ccm/test-1.2.6/node3/logs/system.log <==
INFO [GossipStage:1] 2013-07-02 20:45:23,899 Gossiper.java (line 785)
InetAddress /127.0.0.1 is now DOWN
INFO [HANDSHAKE-/127.0.0.1] 2013-07-02 20:45:24,078 OutboundTcpConnection.java
(line 399) Handshaking version with /127.0.0.1
==> /Users/alexzarutin/.ccm/test-1.2.6/node2/logs/system.log <==
INFO [HANDSHAKE-/127.0.0.1] 2013-07-02 20:45:24,523 OutboundTcpConnection.java
(line 399) Handshaking version with /127.0.0.1
II) logs after sleeping 5 seconds and running $nodetool disablebinary
==> /Users/alexzarutin/.ccm/test-1.2.6/node1/logs/system.log <==
INFO [RMI TCP Connection(6)-10.10.10.130] 2013-07-02 20:48:04,149 Server.java
(line 148) Stop listening for CQL clients
III) logs after running $nodetool disablethrift
==> /Users/alexzarutin/.ccm/test-1.2.6/node1/logs/system.log <==
INFO [RMI TCP Connection(8)-10.10.10.130] 2013-07-02 20:49:08,008
ThriftServer.java (line 116) Stop listening to thrift clients
IV) logs after running $nodetool drain
==> /Users/alexzarutin/.ccm/test-1.2.6/node1/logs/system.log <==
INFO [RMI TCP Connection(10)-10.10.10.130] 2013-07-02 20:50:40,713
StorageService.java (line 897) DRAINING: starting drain process
INFO [RMI TCP Connection(10)-10.10.10.130] 2013-07-02 20:50:40,714
Gossiper.java (line 1108) Announcing shutdown
INFO [RMI TCP Connection(10)-10.10.10.130] 2013-07-02 20:50:42,715
MessagingService.java (line 685) Waiting for messaging service to quiesce
INFO [ACCEPT-/127.0.0.1] 2013-07-02 20:50:42,716 MessagingService.java (line
895) MessagingService shutting down server thread.
INFO [RMI TCP Connection(10)-10.10.10.130] 2013-07-02 20:50:42,718
ColumnFamilyStore.java (line 630) Enqueuing flush of
Memtable-local@788901055(52/52 serialized/live bytes, 4 ops)
INFO [RMI TCP Connection(10)-10.10.10.130] 2013-07-02 20:50:42,718
ColumnFamilyStore.java (line 630) Enqueuing flush of
Memtable-peers@1810922770(410/410 serialized/live bytes, 32 ops)
INFO [FlushWriter:2] 2013-07-02 20:50:42,719 Memtable.java (line 461) Writing
Memtable-local@788901055(52/52 serialized/live bytes, 4 ops)
INFO [FlushWriter:2] 2013-07-02 20:50:42,728 Memtable.java (line 495)
Completed flushing
/Users/alexzarutin/.ccm/test-1.2.6/node1/data/system/local/system-local-ic-11-Data.db
(84 bytes) for commitlog position ReplayPosition(segmentId=1372823015605,
position=81617)
INFO [FlushWriter:2] 2013-07-02 20:50:42,729 Memtable.java (line 461) Writing
Memtable-peers@1810922770(410/410 serialized/live bytes, 32 ops)
INFO [FlushWriter:2] 2013-07-02 20:50:42,735 Memtable.java (line 495)
Completed flushing
/Users/alexzarutin/.ccm/test-1.2.6/node1/data/system/peers/system-peers-ic-4-Data.db
(323 bytes) for commitlog position ReplayPosition(segmentId=1372823015605,
position=81617)
INFO [CompactionExecutor:22] 2013-07-02 20:50:42,736 CompactionTask.java (line
105) Compacting
[SSTableReader(path='/Users/alexzarutin/.ccm/test-1.2.6/node1/data/system/peers/system-peers-ic-2-Data.db'),
SSTableReader(path='/Users/alexzarutin/.ccm/test-1.2.6/node1/data/system/peers/system-peers-ic-1-Data.db'),
SSTableReader(path='/Users/alexzarutin/.ccm/test-1.2.6/node1/data/system/peers/system-peers-ic-3-Data.db'),
SSTableReader(path='/Users/alexzarutin/.ccm/test-1.2.6/node1/data/system/peers/system-peers-ic-4-Data.db')]
INFO [CompactionExecutor:22] 2013-07-02 20:50:42,743 CompactionTask.java (line
262) Compacted 4 sstables to
[/Users/alexzarutin/.ccm/test-1.2.6/node1/data/system/peers/system-peers-ic-5,].
986 bytes to 394 (~39% of original) in 7ms = 0.053678MB/s. 7 total rows, 2
unique. Row merge counts were {1:0, 2:0, 3:1, 4:1, }
INFO [RMI TCP Connection(10)-10.10.10.130] 2013-07-02 20:50:42,893
StorageService.java (line 897) DRAINED
Pay attention to the following line. On this point node1 is seems is going
down. However, this process is still up and running.
INFO [ACCEPT-/127.0.0.1] 2013-07-02 20:50:42,716 MessagingService.java (line
895) MessagingService shutting down server thread.
So running ./init1 (no indication in the logs), ./init2 and ./init3 will fully
stop the cluster - zero java processes. There are no NullPointer or any other
exceptions.
Full repro steps and init1, init2, init3 scripts stripped to the functionality
of stopping cluster are attached.
> NPE shutting down Cassandra trunk (cassandra-1.2.5-989-g70dfb70)
> ----------------------------------------------------------------
>
> Key: CASSANDRA-5689
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5689
> Project: Cassandra
> Issue Type: Bug
> Components: API
> Affects Versions: 1.2.0
> Environment: Ubuntu Precise with Oracle Java 7u25.
> Reporter: Blair Zajac
> Assignee: Alex Zarutin
> Priority: Trivial
> Attachments: CASSANDRA-5689.txt, init1, init2, init3
>
>
> I built Cassandra from git trunk at cassandra-1.2.5-989-g70dfb70 using the
> debian/ package. I have a shell script to shut down Cassandra:
> {code}
> $nodetool disablegossip
> sleep 5
> $nodetool disablebinary
> $nodetool disablethrift
> $nodetool drain
> /etc/init.d/cassandra stop
> {code}
> Shutting it down I get this exception on all three nodes:
> {code}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.cassandra.transport.Server.close(Server.java:156)
> at org.apache.cassandra.transport.Server.stop(Server.java:107)
> at
> org.apache.cassandra.service.StorageService.stopNativeTransport(StorageService.java:347)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
> at
> com.sun.jmx.remote.security.MBeanServerAccessController.invoke(MBeanServerAccessController.java:468)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
> at
> javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
> at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1427)
> at
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at sun.rmi.transport.Transport$1.run(Transport.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
