Hi,
>From "nodetool status" output, it looks like the cluster is running ok. The
exception itself simply says that data streaming fails during nodetool
rebuild. This could be due to possible network hiccup. It is hard to say.
You need to do further investigation. For example, you can run "nodetool
netstats" and check log file on the target node to get more information
about the failed streaming sessions, such as the source nodes of the failed
streaming sessions and then check the log files on those nodes.
Yabin
On Thu, Sep 22, 2016 at 4:57 PM, laxmikanth sadula <laxmikanth...@gmail.com>
wrote:
> Hi,
>
> We have c* 2.0.17 cluster with 2 DCs - DC1, DC2. We tried to add new data
> center DC3 and ran "nodetool rebuild 'DC1'" and we faced below exception on
> few nodes after some data got streamed to new nodes in new data center DC3.
>
>
> *Exception in thread "main" java.lang.RuntimeException: Error while
> rebuilding node: Stream failed*
> *at
> org.apache.cassandra.service.StorageService.rebuild(StorageService.java:936)*
> *at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)*
> *at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)*
> *at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)*
> *at java.lang.reflect.Method.invoke(Method.java:606)*
> *at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)*
> *at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)*
> *at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)*
> *at java.lang.reflect.Method.invoke(Method.java:606)*
> *at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)*
> *at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)*
> *at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)*
> *at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)*
> *at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)*
> *at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)*
> *at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)*
> *at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)*
> *at
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)*
> *at
> javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)*
> *at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)*
> *at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)*
> *at
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)*
> *at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)*
> *at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)*
> *at java.lang.reflect.Method.invoke(Method.java:606)*
> *at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)*
> *at sun.rmi.transport.Transport$2.run(Transport.java:202)*
> *at sun.rmi.transport.Transport$2.run(Transport.java:199)*
> *at java.security.AccessController.doPrivileged(Native Method)*
> *at sun.rmi.transport.Transport.serviceCall(Transport.java:198)*
> *at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:567)*
> *at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:828)*
> *at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.access$400(TCPTransport.java:619)*
> *at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:684)*
> *at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:681)*
> *at java.security.AccessController.doPrivileged(Native Method)*
> *at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:681)*
> *at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)*
> *at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)*
> *at java.lang.Thread.run(Thread.java:745)*
>
>
> We have 4 user keyspaces , so we altered all keyspaces as below before
> issuing rebuild.
>
> *ALTER KEYSPACE keyspace_name WITH replication = {'class':
> 'NetworkTopologyStrategy', 'DC1': '3' , 'DC2' : '3' , 'DC3' : '3'};*
>
>
> *Output of describecluster*
>
> ./nodetool describecluster
> Cluster Information:
> Name: Ss Cluster
> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Schema versions:
> 3b688e54-47be-39e8-ae45-e71ba98d69e2: [xxx.xxx.198.75, xxx.xxx.198.132,
> xxx.xxx.198.133, xxx.xxx.12.115, xxx.xxx.198.78, xxx.xxx.12.123,
> xxx.xxx.98.205, xxx.xxx.98.219, xxx.xxx.98.220, xxx.xxx.198.167,
> xxx.xxx.98.172, xxx.xxx.98.173, xxx.xxx.98.170, xxx.xxx.198.168,
> xxx.xxx.98.171, xxx.xxx.198.169, xxx.xxx.12.146, xxx.xxx.98.168,
> xxx.xxx.12.145, xxx.xxx.98.169, xxx.xxx.12.144, xxx.xxx.12.143,
> xxx.xxx.12.140, xxx.xxx.12.139, xxx.xxx.198.126, xxx.xxx.12.136,
> xxx.xxx.12.135, xxx.xxx.198.191, xxx.xxx.12.133, xxx.xxx.12.131]
>
>
>
> *nodetool status output:*
>
> ./nodetool status
> Note: Ownership information does not include topology; for complete
> information, specify a keyspace
> Datacenter: DC1
> ===============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns Host ID
> Rack
> UN xxx.xxx.198.75 639.41 GB 256 3.2%
> fdd32c67-3cea-4174-b59b-c1ea14e1a334
> GRP1
> UN xxx.xxx.198.132 581.94 GB 256 3.4%
> 6a465101-29e7-4792-8269-851200a70023
> GRP2
> UN xxx.xxx.198.133 618.22 GB 256 3.6%
> 751ce15a-10f1-44cf-9357-04da7e21b511
> GRP2
> UN xxx.xxx.198.167 566.83 GB 256 3.5%
> 45f6684f-d6a0-4cba-875c-9db459646545
> GRP2
> UN xxx.xxx.198.78 661.98 GB 256 3.3%
> a8332f22-a75f-4d7c-8b71-7284f6fe208f
> GRP3
> UN xxx.xxx.198.126 603.37 GB 256 3.6%
> 71be90d8-97db-4155-b4fc-da59d78331ef
> GRP1
> UN xxx.xxx.198.191 571.8 GB 256 3.2%
> a9023df8-a8b3-484b-a03d-0fdea35007bd
> GRP3
> UN xxx.xxx.198.168 631.98 GB 256 3.5%
> 2302491a-a8b5-4aa6-bda7-f1544064c4e3
> GRP3
> UN xxx.xxx.198.169 631.12 GB 256 3.4%
> d5e5bc3d-38de-4043-abca-08ac09f29a46
> GRP1
> Datacenter: DC2
> ==============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns Host ID
> Rack
> UN xxx.xxx.98.172 630.36 GB 256 3.3%
> 4709fe54-2793-481a-95f0-a780ea55cdf1
> GRP1
> UN xxx.xxx.98.219 606.85 GB 256 3.2%
> 792f3185-d5f2-4f4f-a60e-e0c888871cbb
> GRP3
> UN xxx.xxx.98.173 654.75 GB 256 3.5%
> 345cdedc-d6d4-4b46-9cf3-5dc726af9b99
> GRP3
> UN xxx.xxx.98.170 577.26 GB 256 3.6%
> cfaf5519-4849-49f9-a1c2-6ca13eb19d3c
> GRP3
> UN xxx.xxx.98.205 603.29 GB 256 2.9%
> efafe9fd-65b1-41ec-b285-fa00279410ce
> GRP2
> UN xxx.xxx.98.171 659.41 GB 256 3.4%
> 9585c52a-94ba-4b94-af22-97f80ab8c102
> GRP2
> UN xxx.xxx.98.220 634.14 GB 256 3.1%
> 20a5f0ec-12b3-4db0-9483-f5ce984b362a
> GRP1
> UN xxx.xxx.98.168 617.16 GB 256 3.3%
> 3ee473fc-6415-45c5-9466-4aaad9181e5a
> GRP1
> UN xxx.xxx.98.169 614.25 GB 256 3.4%
> dc1f04d7-0c84-4f8c-95bb-97005baaa328
> GRP2
> Datacenter: DC3
> ===============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns Host ID
> Rack
> UN xxx.xxx.12.143 4.76 GB 256 3.5%
> 626facfb-8346-4f17-aa06-38424d9575ce
> GRP2
> UN xxx.xxx.12.140 4.88 GB 256 3.9%
> 88517bf9-efa6-4214-b503-943c4d5d91ba
> GRP1
> UN xxx.xxx.12.115 4.48 GB 256 3.3%
> 3878532d-3980-40a8-9276-0983eebdc9c6
> GRP1
> UN xxx.xxx.12.139 86.72 GB 256 3.2%
> 2ad980fc-ffec-489e-850d-3b11824bb7c6
> GRP3
> UN xxx.xxx.12.136 4.78 GB 256 3.0%
> 0b25753f-e0c4-448d-8459-6248b3072b79
> GRP3
> UN xxx.xxx.12.135 4.92 GB 256 3.1%
> 29ac1861-da54-421f-b0b7-0ad1230460ac
> GRP2
> UN xxx.xxx.12.133 4.15 GB 256 3.2%
> fd048371-4994-486d-8de5-365e56ec912f
> GRP1
> UN xxx.xxx.12.123 4.78 GB 256 3.3%
> cbbb8973-4a33-4ce9-9610-8a7a1ddf975d
> GRP3
> UN xxx.xxx.12.131 4.44 GB 256 3.1%
> 163961b1-96aa-4ef1-8517-ec0f24f0be39
> GRP2
> UN xxx.xxx.12.146 4.46 GB 256 3.6%
> 23cc92fc-2bd8-453c-ae3c-341cfba172a9
> GRP2
> UN xxx.xxx.12.145 5.18 GB 256 3.2%
> 7a929498-29ab-4921-9676-cce41714d8c3
> GRP1
> UN xxx.xxx.12.144 62.52 GB 256 3.3%
> 1631630d-a6ed-437e-9368-30fccce4f12c
> GRP3
>
> We using GossipingPropertyFileSnitch , so we modified only
> cassandra-rackdc.properties and made NO changes to
> cassandra-topology.properties.
>
> If anybody has faced this problem or know how to fix , please let us know.
>
> Thanks in advance...!!!
>
