Re: Streaming Failed during bootstrap of a Replacement node

2019-12-20 Thread Jon Haddad 
Without getting too in the weeds here - ideally yes, you'd be able to
replace the node.  However, the error you're getting looks (at a glance) to
be one of the many range tombstone bugs that were fixed in later versions.

On Fri, Dec 20, 2019 at 11:23 AM Nethi, Manoj 
wrote:

> Hi Jon,
>
> Yes we will upgrade it soon. But before we can upgrade shouldn’t we get
> this lost node in the cluster to be replaced ?
>
>
>
>
>
>
>
> *From:* Jon Haddad 
> *Sent:* Friday, December 20, 2019 2:13 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Streaming Failed during bootstrap of a Replacement node
>
>
>
> *This email is from an external source - **exercise caution regarding
> links and attachments. *
>
>
>
> You should upgrade to Cassandra 3.11.5 before doing anything else.  You're
> running a pretty old and buggy version.  There's been hundreds (maybe
> thousands) of bugs fixed between 3.3 and 3.11.5.
>
>
>
> On Fri, Dec 20, 2019 at 10:46 AM Nethi, Manoj 
> wrote:
>
> Hi,
>
>
>
> We are seeing the following error while bootstrapping a node which is
> replacement of a failed node in a multi DC cluster.
>
>
>
>
>
>
>
> WARN  [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,120
> StreamSession.java:641 - [Stream #f231db30-22da-11ea-b38a-0f6cfab62953]
> Retrying for following error
>
> java.lang.ArrayIndexOutOfBoundsException: -127
>
> at
> org.apache.cassandra.db.RangeTombstone$Bound$Serializer.deserialize(RangeTombstone.java:201)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:355)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:86)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:64)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.StreamReader$StreamDeserializer.hasNext(StreamReader.java:253)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:108)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:120)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.db.ColumnIndex.writeAndBuildIndex(ColumnIndex.java:57)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:153)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:48)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.io.sstable.format.RangeAwareSSTableWriter.append(RangeAwareSSTableWriter.java:99)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.StreamReader.writePartition(StreamReader.java:178)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:106)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50)
> [apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39)
> [apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)
> [apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
> [apache-cassandra-3.3.0.jar:3.3.0]
>
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
>
> ERROR [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,120
> StreamSession.java:520 - [Stream #f231db30-22da-11ea-b38a-0f6cfab62953]
> Streaming error occurred
>
> java.lang.IllegalArgumentException: Unknown type 0
>
> at
> org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:97)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
>
> INFO  [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,124
> StreamResultFuture.java:185 - [Stream
> #f231db30-22da-11ea-b38a-0f6cfab62953] Session with /**.***.**.** is
> complete
>
> W

RE: Streaming Failed during bootstrap of a Replacement node

2019-12-20 Thread Nethi, Manoj 
Hi Jon,
Yes we will upgrade it soon. But before we can upgrade shouldn’t we get this 
lost node in the cluster to be replaced ?



From: Jon Haddad 
Sent: Friday, December 20, 2019 2:13 PM
To: user@cassandra.apache.org
Subject: Re: Streaming Failed during bootstrap of a Replacement node

This email is from an external source - exercise caution regarding links and 
attachments.

You should upgrade to Cassandra 3.11.5 before doing anything else.  You're 
running a pretty old and buggy version.  There's been hundreds (maybe 
thousands) of bugs fixed between 3.3 and 3.11.5.

On Fri, Dec 20, 2019 at 10:46 AM Nethi, Manoj 
mailto:manoj.ne...@fmr.com.invalid>> wrote:
Hi,

We are seeing the following error while bootstrapping a node which is 
replacement of a failed node in a multi DC cluster.



WARN  [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,120 StreamSession.java:641 
- [Stream #f231db30-22da-11ea-b38a-0f6cfab62953] Retrying for following error
java.lang.ArrayIndexOutOfBoundsException: -127
at 
org.apache.cassandra.db.RangeTombstone$Bound$Serializer.deserialize(RangeTombstone.java:201)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:355)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:86)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:64)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.streaming.StreamReader$StreamDeserializer.hasNext(StreamReader.java:253)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:108) 
~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:120) 
~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.db.ColumnIndex.writeAndBuildIndex(ColumnIndex.java:57) 
~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:153)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:48)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.io.sstable.format.RangeAwareSSTableWriter.append(RangeAwareSSTableWriter.java:99)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.streaming.StreamReader.writePartition(StreamReader.java:178)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:106)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50)
 [apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39)
 [apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)
 [apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
 [apache-cassandra-3.3.0.jar:3.3.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
ERROR [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,120 StreamSession.java:520 
- [Stream #f231db30-22da-11ea-b38a-0f6cfab62953] Streaming error occurred
java.lang.IllegalArgumentException: Unknown type 0
at 
org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:97)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
INFO  [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,124 
StreamResultFuture.java:185 - [Stream #f231db30-22da-11ea-b38a-0f6cfab62953] 
Session with /**.***.**.** is complete
WARN  [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,124 
StreamResultFuture.java:212 - [Stream #f231db30-22da-11ea-b38a-0f6cfab62953] 
Stream failed
WARN  [Thread-738] 2019-12-19 23:36:40,126 CompressedInputStream.java:182 - 
Error while reading compressed input stream.
java.nio.channels.ClosedChannelException: null
at 
sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:257) 
~[na:1.8.0_74]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:300) 
~[

Re: Streaming Failed during bootstrap of a Replacement node

2019-12-20 Thread Jon Haddad 
You should upgrade to Cassandra 3.11.5 before doing anything else.  You're
running a pretty old and buggy version.  There's been hundreds (maybe
thousands) of bugs fixed between 3.3 and 3.11.5.

On Fri, Dec 20, 2019 at 10:46 AM Nethi, Manoj 
wrote:

> Hi,
>
>
>
> We are seeing the following error while bootstrapping a node which is
> replacement of a failed node in a multi DC cluster.
>
>
>
>
>
>
>
> WARN  [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,120
> StreamSession.java:641 - [Stream #f231db30-22da-11ea-b38a-0f6cfab62953]
> Retrying for following error
>
> java.lang.ArrayIndexOutOfBoundsException: -127
>
> at
> org.apache.cassandra.db.RangeTombstone$Bound$Serializer.deserialize(RangeTombstone.java:201)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:355)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:86)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:64)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.StreamReader$StreamDeserializer.hasNext(StreamReader.java:253)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:108)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:120)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.db.ColumnIndex.writeAndBuildIndex(ColumnIndex.java:57)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:153)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:48)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.io.sstable.format.RangeAwareSSTableWriter.append(RangeAwareSSTableWriter.java:99)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.StreamReader.writePartition(StreamReader.java:178)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:106)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50)
> [apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39)
> [apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)
> [apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
> [apache-cassandra-3.3.0.jar:3.3.0]
>
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
>
> ERROR [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,120
> StreamSession.java:520 - [Stream #f231db30-22da-11ea-b38a-0f6cfab62953]
> Streaming error occurred
>
> java.lang.IllegalArgumentException: Unknown type 0
>
> at
> org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:97)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
>
> INFO  [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,124
> StreamResultFuture.java:185 - [Stream
> #f231db30-22da-11ea-b38a-0f6cfab62953] Session with /**.***.**.** is
> complete
>
> WARN  [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,124
> StreamResultFuture.java:212 - [Stream
> #f231db30-22da-11ea-b38a-0f6cfab62953] Stream failed
>
> WARN  [Thread-738] 2019-12-19 23:36:40,126 CompressedInputStream.java:182
> - Error while reading compressed input stream.
>
> java.nio.channels.ClosedChannelException: null
>
> at
> sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:257)
> ~[na:1.8.0_74]
>
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:300)
> ~[na:1.8.0_74]
>
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:59)
> ~[na:1.8.0_74]
>
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
> ~[na:1.8.0_74]
>
>

Streaming Failed during bootstrap of a Replacement node

2019-12-20 Thread Nethi, Manoj 
Hi,

We are seeing the following error while bootstrapping a node which is 
replacement of a failed node in a multi DC cluster.



WARN  [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,120 StreamSession.java:641 
- [Stream #f231db30-22da-11ea-b38a-0f6cfab62953] Retrying for following error
java.lang.ArrayIndexOutOfBoundsException: -127
at 
org.apache.cassandra.db.RangeTombstone$Bound$Serializer.deserialize(RangeTombstone.java:201)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:355)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:86)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:64)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.streaming.StreamReader$StreamDeserializer.hasNext(StreamReader.java:253)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:108) 
~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:120) 
~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.db.ColumnIndex.writeAndBuildIndex(ColumnIndex.java:57) 
~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:153)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:48)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.io.sstable.format.RangeAwareSSTableWriter.append(RangeAwareSSTableWriter.java:99)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.streaming.StreamReader.writePartition(StreamReader.java:178)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:106)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50)
 [apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39)
 [apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)
 [apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
 [apache-cassandra-3.3.0.jar:3.3.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
ERROR [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,120 StreamSession.java:520 
- [Stream #f231db30-22da-11ea-b38a-0f6cfab62953] Streaming error occurred
java.lang.IllegalArgumentException: Unknown type 0
at 
org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:97)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
INFO  [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,124 
StreamResultFuture.java:185 - [Stream #f231db30-22da-11ea-b38a-0f6cfab62953] 
Session with /**.***.**.** is complete
WARN  [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,124 
StreamResultFuture.java:212 - [Stream #f231db30-22da-11ea-b38a-0f6cfab62953] 
Stream failed
WARN  [Thread-738] 2019-12-19 23:36:40,126 CompressedInputStream.java:182 - 
Error while reading compressed input stream.
java.nio.channels.ClosedChannelException: null
at 
sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:257) 
~[na:1.8.0_74]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:300) 
~[na:1.8.0_74]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:59) 
~[na:1.8.0_74]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) 
~[na:1.8.0_74]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) 
~[na:1.8.0_74]
at 
org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:172)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) 
[apache-cassandra-3.3.0.jar:3.3.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]



Did any one faced a similar issue ?

RE: Data is not syncing up when we add one more Node(DR) to existing 3 node cluster

2019-12-20 Thread Anil Kumar Ganipineni 
Hi John,

Yes!


Regards,
Anil Ganipineni
Adaequare GSP
GSP Support : 040 660 335 88
Personal : +91 96522 314 80

P Please consider environment before printing this page.

From: John Belliveau 
Sent: 13 December 2019 06:31
To: user@cassandra.apache.org
Subject: Re: Data is not syncing up when we add one more Node(DR) to existing 3 
node cluster

**CAUTION: This email is originated from outside the organization. Be cautious 
before opening the link or attachments. Please verify the email id and source 
of the sender.**
Hi Anil,

In the cassandra.yaml file on your new node in DC2, is the IP address for the 
seeds set to the seed node in DC1?

Best,
John

On Wed, Dec 11, 2019 at 11:09 PM Anil Kumar Ganipineni 
mailto:akganipin...@adaequare.com>> wrote:
Hi All,

We have 3 node cluster on datacentre DC1 and below is our key space 
declaration. The current data size on the cluster is ~10GB. When we add a new 
node on datacentre DC2, the new node is not syncing up with the data, but it is 
showing UN when I run the nodetool status.

CREATE KEYSPACE production WITH REPLICATION = { 'class' : 
'org.apache.cassandra.locator.NetworkTopologyStrategy', 'DC1': '3', 'DC2': '1' 
} AND DURABLE_WRITES = true;


Please provide suggestions to make the new node on DC2 to sync up with existing 
cluster. This is required as the DC2 is our DR in a different region other than 
existing cluster.


Regards,
Anil Ganipineni

P Please consider environment before printing this page.