Re: How to complete bootstrap with exception due to stream failure?

[Jason Kania]

Thanks for the reference to nodetool resetlocalschema as that will come in 
handy in the future. Thanks also for the reference to 
https://issues.apache.org/jira/browse/CASSANDRA-11050 which seems related, but 
I am not sure.

I was doing a bootstrapping on 192.168.10.10 and it had nothing on it to start 
with it. It was in the process of transferring the schema definitions that the 
bootstrap was failing. In the process of trying to get something working, I 
tried adding the dropped columns on the existing node and the new node but had 
no luck with that either.
I finally figured it out so I raised 
https://issues.apache.org/jira/browse/CASSANDRA-11273 with these details and 
the workaround that I found.
  From: Paulo Motta 
 To: "user@cassandra.apache.org" ; Jason Kania 
 
 Sent: Sunday, February 28, 2016 10:01 PM
 Subject: Re: How to complete bootstrap with exception due to stream failure?
   
Were the columns sensor.lastEvaluation and sensordb.lastCheckTime dropped by 
any chance? If so, you might be hitting 
https://issues.apache.org/jira/browse/CASSANDRA-11050, fixed in upcoming 3.4.

If that's the case, you may want to check if nodes other than 192.168.10.10 
have the dropped columns in the system_schema.dropped_columns table, and if so, 
reset the local schema (nodetool resetlocalschema) of 192.168.10.10 to force a 
schema synchronization with other nodes. Another possible workaround is to 
manually include the dropped columns in the system_schema.dropped_columns table 
of 192.168.10.10.

2016-02-27 22:56 GMT-03:00 Jason Kania :

Hi,
I just reran the command and collected following. Any suggestions would be 
appreciated.

Thanks,
Jason

from 192.168.10.8

ERROR [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857 StreamSession.java:635 
- [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Remote peer 192.168.10.10 
failed stream session.
INFO  [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857 
StreamResultFuture.java:182 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Session with /192.168.10.10 is complete
WARN  [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,858 
StreamResultFuture.java:209 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Stream failed

from 192.168.10.8 debug
DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,414 
ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Received Received (79256340--11e5-9f70-7d76a8de8480, #0)
DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,854 
ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Received Retry (f3a137e0-024b-11e5-bb31-0d2316086bf7, #0)
DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,854 
ConnectionHandler.java:334 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Sending File (Header (cfId: f3a137e0-024b-11e5-bb31-0d2316086bf7, #0, version: 
ma, format: BIG, estimated keys: 128, transfer size: 4653, compressed?: true, 
repairedAt: 0, level: 0), file: 
/home/cassandra/data/sensordb/sensor/ma-76-big-Data.db)
DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,854 
CompressedStreamWriter.java:63 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Start streaming file /home/cassandra/data/sensordb/sensor/ma-76-big-Data.db to 
/192.168.10.10, repairedAt = 0, totalSize = 4653
DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,854 
CompressedStreamWriter.java:94 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Finished streaming file /home/cassandra/data/sensordb/sensor/ma-76-big-Data.db 
to /192.168.10.10, bytesTransferred = 4653, totalSize = 4653
DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,855 
ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Received Retry (faa55490-024b-11e5-bb31-0d2316086bf7, #0)
DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,855 
ConnectionHandler.java:334 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Sending File (Header (cfId: faa55490-024b-11e5-bb31-0d2316086bf7, #0, version: 
ma, format: BIG, estimated keys: 128, transfer size: 705, compressed?: true, 
repairedAt: 0, level: 0), file: 
/home/cassandra/data/sensordb/sensorUnit/ma-79-big-Data.db)
DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,856 
CompressedStreamWriter.java:63 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Start streaming file /home/cassandra/data/sensordb/sensorUnit/ma-79-big-Data.db 
to /192.168.10.10, repairedAt = 0, totalSize = 705
DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,856 
CompressedStreamWriter.java:94 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Finished streaming file 
/home/cassandra/data/sensordb/sensorUnit/ma-79-big-Data.db to /192.168.10.10, 
bytesTransferred = 705, totalSize = 705
DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857 
ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Received Session Failed
ERROR [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857 StreamSession.java:635 
- [Stream #c9868f90-ddbb-11e5-80c0-89f591237a

Re: How to complete bootstrap with exception due to stream failure?

[Paulo Motta]

.0.3.jar:3.0.3]
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50)
> [apache-cassandra-3.0.3.jar:3.0.3]
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39)
> [apache-cassandra-3.0.3.jar:3.0.3]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)
> [apache-cassandra-3.0.3.jar:3.0.3]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
> [apache-cassandra-3.0.3.jar:3.0.3]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
> DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,854
> ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
> Received null
>
>
> --
> *From:* Sebastian Estevez 
> *To:* user@cassandra.apache.org; Jason Kania 
> *Sent:* Saturday, February 27, 2016 8:24 PM
> *Subject:* Re: How to complete bootstrap with exception due to stream
> failure?
>
> progress: 361% does not look right (probably a bug).
>
> Can you check the corresponding messages on the other side of the stream?
> I.E. the system log for 192.168.10.8 around 18:02:06?
>
> All the best,
>
> [image: datastax_logo.png] <http://www.datastax.com/>
> Sebastián Estévez
> Solutions Architect | 954 905 8615 | sebastian.este...@datastax.com
> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
> <https://twitter.com/datastax> [image: g+.png]
> <https://plus.google.com/+Datastax/about>
> <http://feeds.feedburner.com/datastax>
> <http://goog_410786983/>
>
> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>
> DataStax is the fastest, most scalable distributed database technology,
> delivering Apache Cassandra to the world’s most innovative enterprises.
> Datastax is built to be agile, always-on, and predictably scalable to any
> size. With more than 500 customers in 45 countries, DataStax is the
> database technology and transactional backbone of choice for the worlds
> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>
> On Sat, Feb 27, 2016 at 6:12 PM, Jason Kania 
> wrote:
>
> Hello,
>
> I am trying to get a node bootstrapped in 3.0.3, but just at the point
> where the bootstrap process is to complete, a broken pipe exception occurs
> so the bootstrap process hangs. Once I kill the bootstrap process, I can
> execute "nodetool bootstrap resume" again and the same problem will occur
> just at the end of the bootstrap exercise. Here is the tail of the log:
>
> [2016-02-27 18:02:05,898] received file
> /home/cassandra/data/sensordb/listedAttributes-7925634011e59f707d76a8de8480/ma-30-big-Data.db
> (progress: 357%)
> [2016-02-27 18:02:06,479] received file
> /home/cassandra/data/sensordb/notification-f7e3eaa0024b11e5bb310d2316086bf7/ma-38-big-Data.db
> (progress: 361%)
> [2016-02-27 18:02:06,884] session with /192.168.10.8 complete (progress:
> 361%)
> [2016-02-27 18:02:06,886] Stream failed
>
> I attempted to run nodetool repair, but get the following which I have
> been told indicates that the replication factor is 1:
>
> root@bull:~# nodetool repair
> [2016-02-27 18:04:55,083] Nothing to repair for keyspace 'sensordb'
>
> Thanks,
>
> Jason
>
>
>
>
>

Re: How to complete bootstrap with exception due to stream failure?

[Jason Kania]

6-02-27 20:37:53,412 
ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Received File (Header (cfId: 79256340--11e5-9f70-7d76a8de8480, #0, version: 
ma, format: BIG, estimated keys: 128, transfer size: 166627, compressed?: true, 
repairedAt: 0, level: 0), file: 
/home/cassandra/data/sensordb/listAttributes-7925634011e59f707d76a8de8480/ma-32-big-Data.db)
DEBUG [STREAM-OUT-/192.168.10.8] 2016-02-27 20:37:53,412 
ConnectionHandler.java:334 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Sending Received (79256340--11e5-9f70-7d76a8de8480, #0)
DEBUG [CompactionExecutor:3] 2016-02-27 20:37:53,833 CompactionTask.java:217 - 
Compacted (e224bef0-ddbb-11e5-80c0-89f591237aca) 4 sstables to 
[/home/cassandra/data/system_distributed/parent_repair_history-deabd734b99d3b9c92e5fd92eb5abf14/ma-5-big,]
 to level=0.  2,743,164 bytes to 685,791 (~25% of original) in 1,096ms = 
0.596735MB/s.  0 total partitions merged to 57.  Partition merge counts were 
{4:57, }
DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,850 
CompressedStreamReader.java:80 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Start receiving file #0 from /192.168.10.8, repairedAt = 0, size = 4653, ks = 
'sensordb', table = 'sensor'.
WARN  [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,851 StreamSession.java:641 
- [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Retrying for following error
java.lang.RuntimeException: Unknown column lastEvaluation during deserialization
    at 
org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
 ~[apache-cassandra-3.0.3.jar:3.0.3]
    at 
org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:87)
 ~[apache-cassandra-3.0.3.jar:3.0.3]
    at 
org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50)
 [apache-cassandra-3.0.3.jar:3.0.3]
    at 
org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39)
 [apache-cassandra-3.0.3.jar:3.0.3]
    at 
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)
 [apache-cassandra-3.0.3.jar:3.0.3]
    at 
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
 [apache-cassandra-3.0.3.jar:3.0.3]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
DEBUG [STREAM-OUT-/192.168.10.8] 2016-02-27 20:37:53,852 
ConnectionHandler.java:334 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Sending Retry (f3a137e0-024b-11e5-bb31-0d2316086bf7, #0)
DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,852 
ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Received null
DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,853 
CompressedStreamReader.java:80 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Start receiving file #0 from /192.168.10.8, repairedAt = 0, size = 705, ks = 
'sensordb', table = 'sensorUnit'.
WARN  [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,854 StreamSession.java:641 
- [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Retrying for following error
java.lang.RuntimeException: Unknown column lastCheckTime during deserialization
    at 
org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
 ~[apache-cassandra-3.0.3.jar:3.0.3]
    at 
org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:87)
 ~[apache-cassandra-3.0.3.jar:3.0.3]
    at 
org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50)
 [apache-cassandra-3.0.3.jar:3.0.3]
    at 
org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39)
 [apache-cassandra-3.0.3.jar:3.0.3]
    at 
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)
 [apache-cassandra-3.0.3.jar:3.0.3]
    at 
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
 [apache-cassandra-3.0.3.jar:3.0.3]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,854 
ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Received null


  From: Sebastian Estevez 
 To: user@cassandra.apache.org; Jason Kania  
 Sent: Saturday, February 27, 2016 8:24 PM
 Subject: Re: How to complete bootstrap with exception due to stream failure?
   
progress: 361% does not look right (probably a bug).

Can you check the corresponding messages on the other side of the stream? I.E. 
the system log for 192.168.10.8 around 18:02:06?
All the best,
Sebastián EstévezSolutions Architect | 954 905 8615 | 
sebastian.este...@datastax.com


DataStax is the fastest, most scalable distributed database technology, 
delivering Apache Cassandra to the world’s most innov

Re: How to complete bootstrap with exception due to stream failure?

[Sebastian Estevez]

progress: 361% does not look right (probably a bug).

Can you check the corresponding messages on the other side of the stream?
I.E. the system log for 192.168.10.8 around 18:02:06?

All the best,


[image: datastax_logo.png] 

Sebastián Estévez

Solutions Architect | 954 905 8615 | sebastian.este...@datastax.com

[image: linkedin.png]  [image:
facebook.png]  [image: twitter.png]
 [image: g+.png]







DataStax is the fastest, most scalable distributed database technology,
delivering Apache Cassandra to the world’s most innovative enterprises.
Datastax is built to be agile, always-on, and predictably scalable to any
size. With more than 500 customers in 45 countries, DataStax is the
database technology and transactional backbone of choice for the worlds
most innovative companies such as Netflix, Adobe, Intuit, and eBay.

On Sat, Feb 27, 2016 at 6:12 PM, Jason Kania  wrote:

> Hello,
>
> I am trying to get a node bootstrapped in 3.0.3, but just at the point
> where the bootstrap process is to complete, a broken pipe exception occurs
> so the bootstrap process hangs. Once I kill the bootstrap process, I can
> execute "nodetool bootstrap resume" again and the same problem will occur
> just at the end of the bootstrap exercise. Here is the tail of the log:
>
> [2016-02-27 18:02:05,898] received file
> /home/cassandra/data/sensordb/listedAttributes-7925634011e59f707d76a8de8480/ma-30-big-Data.db
> (progress: 357%)
> [2016-02-27 18:02:06,479] received file
> /home/cassandra/data/sensordb/notification-f7e3eaa0024b11e5bb310d2316086bf7/ma-38-big-Data.db
> (progress: 361%)
> [2016-02-27 18:02:06,884] session with /192.168.10.8 complete (progress:
> 361%)
> [2016-02-27 18:02:06,886] Stream failed
>
> I attempted to run nodetool repair, but get the following which I have
> been told indicates that the replication factor is 1:
>
> root@bull:~# nodetool repair
> [2016-02-27 18:04:55,083] Nothing to repair for keyspace 'sensordb'
>
> Thanks,
>
> Jason
>

How to complete bootstrap with exception due to stream failure?

[Jason Kania]

Hello,
I am trying to get a node bootstrapped in 3.0.3, but just at the point where 
the bootstrap process is to complete, a broken pipe exception occurs so the 
bootstrap process hangs. Once I kill the bootstrap process, I can execute 
"nodetool bootstrap resume" again and the same problem will occur just at the 
end of the bootstrap exercise. Here is the tail of the log:
[2016-02-27 18:02:05,898] received file 
/home/cassandra/data/sensordb/listedAttributes-7925634011e59f707d76a8de8480/ma-30-big-Data.db
 (progress: 357%)
[2016-02-27 18:02:06,479] received file 
/home/cassandra/data/sensordb/notification-f7e3eaa0024b11e5bb310d2316086bf7/ma-38-big-Data.db
 (progress: 361%)
[2016-02-27 18:02:06,884] session with /192.168.10.8 complete (progress: 361%)
[2016-02-27 18:02:06,886] Stream failed
I attempted to run nodetool repair, but get the following which I have been 
told indicates that the replication factor is 1:
root@bull:~# nodetool repair
[2016-02-27 18:04:55,083] Nothing to repair for keyspace 'sensordb'

Thanks,
Jason