Re: How to complete bootstrap with exception due to stream failure?

2016-02-27 Thread Jason Kania 
Hi,
I just reran the command and collected following. Any suggestions would be 
appreciated.

Thanks,
Jason

from 192.168.10.8

ERROR [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857 StreamSession.java:635 
- [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Remote peer 192.168.10.10 
failed stream session.
INFO  [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857 
StreamResultFuture.java:182 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Session with /192.168.10.10 is complete
WARN  [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,858 
StreamResultFuture.java:209 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Stream failed

from 192.168.10.8 debug
DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,414 
ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Received Received (79256340--11e5-9f70-7d76a8de8480, #0)
DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,854 
ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Received Retry (f3a137e0-024b-11e5-bb31-0d2316086bf7, #0)
DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,854 
ConnectionHandler.java:334 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Sending File (Header (cfId: f3a137e0-024b-11e5-bb31-0d2316086bf7, #0, version: 
ma, format: BIG, estimated keys: 128, transfer size: 4653, compressed?: true, 
repairedAt: 0, level: 0), file: 
/home/cassandra/data/sensordb/sensor/ma-76-big-Data.db)
DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,854 
CompressedStreamWriter.java:63 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Start streaming file /home/cassandra/data/sensordb/sensor/ma-76-big-Data.db to 
/192.168.10.10, repairedAt = 0, totalSize = 4653
DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,854 
CompressedStreamWriter.java:94 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Finished streaming file /home/cassandra/data/sensordb/sensor/ma-76-big-Data.db 
to /192.168.10.10, bytesTransferred = 4653, totalSize = 4653
DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,855 
ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Received Retry (faa55490-024b-11e5-bb31-0d2316086bf7, #0)
DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,855 
ConnectionHandler.java:334 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Sending File (Header (cfId: faa55490-024b-11e5-bb31-0d2316086bf7, #0, version: 
ma, format: BIG, estimated keys: 128, transfer size: 705, compressed?: true, 
repairedAt: 0, level: 0), file: 
/home/cassandra/data/sensordb/sensorUnit/ma-79-big-Data.db)
DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,856 
CompressedStreamWriter.java:63 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Start streaming file /home/cassandra/data/sensordb/sensorUnit/ma-79-big-Data.db 
to /192.168.10.10, repairedAt = 0, totalSize = 705
DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,856 
CompressedStreamWriter.java:94 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Finished streaming file 
/home/cassandra/data/sensordb/sensorUnit/ma-79-big-Data.db to /192.168.10.10, 
bytesTransferred = 705, totalSize = 705
DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857 
ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Received Session Failed
ERROR [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857 StreamSession.java:635 
- [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Remote peer 192.168.10.10 
failed stream session.
DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857 
ConnectionHandler.java:110 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Closing stream connection handler on /192.168.10.10
INFO  [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857 
StreamResultFuture.java:182 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Session with /192.168.10.10 is complete
WARN  [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,858 
StreamResultFuture.java:209 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Stream failed


from 192.168.10.10
[2016-02-27 20:37:53,413] received file 
/home/cassandra/data/sensordb/listedAttributes-7925634011e59f707d76a8de8480/ma-32-big-Data.db
 (progress: 365%)
[2016-02-27 20:37:53,414] received file 
/home/cassandra/data/sensordb/liestedAttributes-7925634011e59f707d76a8de8480/ma-32-big-Data.db
 (progress: 369%)
[2016-02-27 20:37:53,865] session with /192.168.10.8 complete (progress: 369%)
[2016-02-27 20:37:53,866] Stream failed

from 192.168.10.10 debug
DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,201 
CompressedStreamReader.java:80 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] 
Start receiving file #0 from /192.168.10.8, repairedAt = 0, size = 166627, ks = 
'sensordb', table = 'listAttributes'.
DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,412 
CompressedStreamReader.java:110 - [Stream 
#c9868f90-ddbb-11e5-80c0-89f591237aca] Finished receiving file #0 from 
/192.168.10.8 readBytes = 166627, totalSize = 166627
DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,412

Re: How to complete bootstrap with exception due to stream failure?

2016-02-27 Thread Sebastian Estevez 
progress: 361% does not look right (probably a bug).

Can you check the corresponding messages on the other side of the stream?
I.E. the system log for 192.168.10.8 around 18:02:06?

All the best,


[image: datastax_logo.png] 

Sebastián Estévez

Solutions Architect | 954 905 8615 | sebastian.este...@datastax.com

[image: linkedin.png]  [image:
facebook.png]  [image: twitter.png]
 [image: g+.png]







DataStax is the fastest, most scalable distributed database technology,
delivering Apache Cassandra to the world’s most innovative enterprises.
Datastax is built to be agile, always-on, and predictably scalable to any
size. With more than 500 customers in 45 countries, DataStax is the
database technology and transactional backbone of choice for the worlds
most innovative companies such as Netflix, Adobe, Intuit, and eBay.

On Sat, Feb 27, 2016 at 6:12 PM, Jason Kania  wrote:

> Hello,
>
> I am trying to get a node bootstrapped in 3.0.3, but just at the point
> where the bootstrap process is to complete, a broken pipe exception occurs
> so the bootstrap process hangs. Once I kill the bootstrap process, I can
> execute "nodetool bootstrap resume" again and the same problem will occur
> just at the end of the bootstrap exercise. Here is the tail of the log:
>
> [2016-02-27 18:02:05,898] received file
> /home/cassandra/data/sensordb/listedAttributes-7925634011e59f707d76a8de8480/ma-30-big-Data.db
> (progress: 357%)
> [2016-02-27 18:02:06,479] received file
> /home/cassandra/data/sensordb/notification-f7e3eaa0024b11e5bb310d2316086bf7/ma-38-big-Data.db
> (progress: 361%)
> [2016-02-27 18:02:06,884] session with /192.168.10.8 complete (progress:
> 361%)
> [2016-02-27 18:02:06,886] Stream failed
>
> I attempted to run nodetool repair, but get the following which I have
> been told indicates that the replication factor is 1:
>
> root@bull:~# nodetool repair
> [2016-02-27 18:04:55,083] Nothing to repair for keyspace 'sensordb'
>
> Thanks,
>
> Jason
>

How to complete bootstrap with exception due to stream failure?

2016-02-27 Thread Jason Kania 
Hello,
I am trying to get a node bootstrapped in 3.0.3, but just at the point where 
the bootstrap process is to complete, a broken pipe exception occurs so the 
bootstrap process hangs. Once I kill the bootstrap process, I can execute 
"nodetool bootstrap resume" again and the same problem will occur just at the 
end of the bootstrap exercise. Here is the tail of the log:
[2016-02-27 18:02:05,898] received file 
/home/cassandra/data/sensordb/listedAttributes-7925634011e59f707d76a8de8480/ma-30-big-Data.db
 (progress: 357%)
[2016-02-27 18:02:06,479] received file 
/home/cassandra/data/sensordb/notification-f7e3eaa0024b11e5bb310d2316086bf7/ma-38-big-Data.db
 (progress: 361%)
[2016-02-27 18:02:06,884] session with /192.168.10.8 complete (progress: 361%)
[2016-02-27 18:02:06,886] Stream failed
I attempted to run nodetool repair, but get the following which I have been 
told indicates that the replication factor is 1:
root@bull:~# nodetool repair
[2016-02-27 18:04:55,083] Nothing to repair for keyspace 'sensordb'

Thanks,
Jason

Re: Cassandra Multi DC (Active-Active) Setup - Measuring latency & throughput performance

2016-02-27 Thread chandrasekar.krc 
Thanks Bryan for the inputs. One of the tests that I'm trying to do is fire 
write requests in one DC and simultaneously do read requests from other DC 
using cassandra-stress (custom schema option). While reading from the other DC, 
I'm expecting cassandra-stress to throw some error for no data found scenario 
atleast for the first few milli-seconds until the cross data center replication 
is successful.


Likewise, for writes across both DC, does Cassandra have any function similar 
to Oracle SYSDATE/SYSTIMESTAMP which can be used to measure the time difference 
of records across data centers.


Thanks & Regards, Chandra Sekar KR

From: Bryan Cheng 
Sent: Saturday, February 27, 2016 05:01
To: user@cassandra.apache.org
Subject: Re: Cassandra Multi DC (Active-Active) Setup - Measuring latency & 
throughput performance

Hi Chandra,

For write latency, etc. the tools are still largely the same set of tools you'd 
use for single-DC- stuff like tracing, cfhistograms, cassandra-stress come to 
mind. The exact results are going to differ based on your consistency tuning 
(can you get away with LOCAL_QUORUM vs QUORUM?) and read/write patterns.

What other data are you looking to gather?

On Fri, Feb 26, 2016 at 5:53 AM, 
> wrote:

Hi,


Are there any links/resources which describe performance measurement (latency & 
throughput) for a Cassandra Multi DC Active-Active setup across a WAN network 
(20Gbps bandwidth) with 5 nodes in each DC.


Basically, I would like to know how to measure latency of writes when data is 
replicated across DC (local/remote) in active-active cluster setup


Regards, Chandra KR

The information contained in this electronic message and any attachments to 
this message are intended for the exclusive use of the addressee(s) and may 
contain proprietary, confidential or privileged information. If you are not the 
intended recipient, you should not disseminate, distribute or copy this e-mail. 
Please notify the sender immediately and destroy all copies of this message and 
any attachments. WARNING: Computer viruses can be transmitted via email. The 
recipient should check this email and any attachments for the presence of 
viruses. The company accepts no liability for any damage caused by any virus 
transmitted by this email. www.wipro.com

The information contained in this electronic message and any attachments to 
this message are intended for the exclusive use of the addressee(s) and may 
contain proprietary, confidential or privileged information. If you are not the 
intended recipient, you should not disseminate, distribute or copy this e-mail. 
Please notify the sender immediately and destroy all copies of this message and 
any attachments. WARNING: Computer viruses can be transmitted via email. The 
recipient should check this email and any attachments for the presence of 
viruses. The company accepts no liability for any damage caused by any virus 
transmitted by this email. www.wipro.com