Re: Unable to connect to C* nodes except one node (same configuration)

[Junaid Nasir]

figured out the problems, node 1 has configuration error due to a problem
in ansible script, which set rpc_address to localhost.
On node 3, after clearing the logs and restarting I saw this in log file

INFO  [main] 2017-07-26 09:26:36,938 Server.java:160 - Starting
listening for CQL clients on /10.128.1.3:9042 (unencrypted)...
INFO  [main] 2017-07-26 09:26:36,994 ThriftServer.java:119 - Binding
thrift service to /10.128.1.3:9160
INFO  [Thread-3] 2017-07-26 09:26:36,999 ThriftServer.java:136 -
Listening for thrift clients...
ERROR [HintsDispatcher:1] 2017-07-26 09:28:30,275
HintsDispatchExecutor.java:225 - *Failed to dispatch hints file
702e8a31-6441--b569-d2d137d54a5d-1500589737308-1.hints: file is
corrupted ({})*
org.apache.cassandra.io.FSReadError: java.io.EOFException: EOF after
736 bytes out of 1955
at 
org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:284)
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:254)
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:156)
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:137)
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:119)
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:91)
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:259)
[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:242)
[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:220)
[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:199)
[apache-cassandra-3.9.jar:3.9]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[na:1.8.0_131]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_131]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
Caused by: java.io.EOFException: EOF after 736 bytes out of 1955
at 
org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:68)
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.hints.ChecksummedDataInput.readFully(ChecksummedDataInput.java:153)
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:402)
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.hints.HintsReader$BuffersIterator.readBuffer(HintsReader.java:312)
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNextInternal(HintsReader.java:303)
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:280)
~[apache-cassandra-3.9.jar:3.9]
... 15 common frames omitted
ERROR [HintsDispatcher:1] 2017-07-26 09:28:30,276
CassandraDaemon.java:226 - Exception in thread
Thread[HintsDispatcher:1,1,main]
org.apache.cassandra.io.FSReadError: java.io.EOFException: EOF after
736 bytes out of 1955
at 
org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:284)
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:254)
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:156)
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:137)
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:119)
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:91)
~[apache-cassandra-3.9.jar:3.9]
at 

Re: Unable to connect to C* nodes except one node (same configuration)

[Junaid Nasir]

each node has its own ip for listen_address and rpc_address. seed node ip
is fixed to 10.128.1.1 on all nodes. configuration was written using
ansible and I have also verified it.

On Wed, Jul 26, 2017 at 3:52 AM, Erik Forkalsud  wrote:

> On 07/25/2017 05:13 AM, Junaid Nasir wrote:
>
> listen_address: 10.128.1.1
>
> rpc_address: 10.128.1.1
>
> Are these the values on all three nodes?
>
> If so, try with empty values:
>
> listen_address:
>
> rpc_address:
>
> or make sure each node has its own IP address configured.
>
>
>

Re: Unable to connect to C* nodes except one node (same configuration)

[Erik Forkalsud]

On 07/25/2017 05:13 AM, Junaid Nasir wrote:

|listen_address: 10.128.1.1 rpc_address: 10.128.1.1|


Are these the values on all three nodes?

If so, try with empty values:

|listen_address: rpc_address:|

or make sure each node has its own IP address configured.