[
https://issues.apache.org/jira/browse/CASSANDRA-8306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14212694#comment-14212694
]
Michael Shuler commented on CASSANDRA-8306:
-------------------------------------------
Could you explain exactly your process of how you were adding this node to your
cluster? The provided log is 171480 lines of tombstone processing and only 54
lines of INFO,WARN normal log entries.
> exception in nodetool enablebinary
> ----------------------------------
>
> Key: CASSANDRA-8306
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8306
> Project: Cassandra
> Issue Type: Bug
> Reporter: Rafał Furmański
> Assignee: Russ Hatch
> Attachments: system.log.zip
>
>
> I was trying to add new node (db4) to existing cluster - with no luck. I
> can't see any errors in system.log. nodetool status shows, that node is
> joining into cluster (for many hours). Attaching error and cluster info:
> {code}
> root@db4:~# nodetool enablebinary
> error: Error starting native transport: null
> -- StackTrace --
> java.lang.RuntimeException: Error starting native transport: null
> at
> org.apache.cassandra.service.StorageService.startNativeTransport(StorageService.java:350)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
> at
> javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
> at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)
> at
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at sun.rmi.transport.Transport$1.run(Transport.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> {code}
> root@db4:~# nodetool describecluster
> Cluster Information:
> Name: Production Cluster
> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Schema versions:
> b7e98bb9-717f-3f59-bac4-84bc19544e90: [10.195.15.163,
> 10.195.15.162, 10.195.15.167, 10.195.15.166]
> {code}
> {code}
> root@db4:~# nodetool status
> Datacenter: Ashburn
> ===================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns Host ID
> Rack
> UN 10.195.15.163 12.05 GB 256 ?
> 0a9f478c-80b5-4c15-8b2e-e27df6684c69 RAC1
> UN 10.195.15.162 12.8 GB 256 ?
> c18d2218-ef84-4165-9c3a-05f592f512e9 RAC1
> UJ 10.195.15.167 18.61 GB 256 ?
> 0d3999d9-1e33-4407-bbbd-10cf0a93b3ba RAC1
> UN 10.195.15.166 13.67 GB 256 ?
> df8df3b7-da17-48de-8cf6-1c718dc2fde8 RAC1
> {code}
> I can't even connect to cassandra using cqlsh:
> {code}
> root@db4:~# cqlsh 10.195.15.167
> Connection error: ('Unable to connect to any servers', {'10.195.15.167':
> error(111, "Tried connecting to [('10.195.15.167', 9042)]. Last error:
> Connection refused")})
> {code}
> In OpsCenter I can see that node has status: Active - Joining and Gossip is
> active. Any ideas what could be wrong with this node?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
