[
https://issues.apache.org/jira/browse/CASSANDRA-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yoan Arnaudov updated CASSANDRA-5717:
-------------------------------------
Description:
I've changed the replication factor for one of keyspaces and now I'm running
repairs on column families (manually). I have 3 nodes cluster. Here is the
error in the error log for one of the nodes.
{code:title=Error Log}
ERROR [Streaming to /NODE_IP:1] 2013-07-01 09:31:29,819 CassandraDaemon.java
(line 192) Exception in thread Thread[Streaming to /NODE_IP:1,5,main]
java.lang.RuntimeException: java.io.IOException: Broken pipe
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
at sun.nio.ch.FileChannelImpl.transferToDirectly(Unknown Source)
at sun.nio.ch.FileChannelImpl.transferTo(Unknown Source)
at
org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:93)
at
org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
... 3 more
ERROR [Streaming to /NODE_IP:2] 2013-07-01 09:44:18,372 CassandraDaemon.java
(line 192) Exception in thread Thread[Streaming to /NODE_IP:2,5,main]
java.lang.RuntimeException: java.io.IOException: Broken pipe
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
at sun.nio.ch.FileChannelImpl.transferToDirectly(Unknown Source)
at sun.nio.ch.FileChannelImpl.transferTo(Unknown Source)
at
org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:93)
at
org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
... 3 more
{code}
{code:title=netstats for one of the nodes}
Mode: NORMAL
Not sending any streams.
Streaming from: /IP
KEYSPACE:
/var/lib/cassandra/data/KEYSPACE/ns_history_org/KEYSPACE-ns_history_org-hf-282-Data.db
sections=1 progress=0/1393915753 - 0%
KEYSPACE:
/var/lib/cassandra/data/KEYSPACE/ns_history_org/KEYSPACE-ns_history_org-ic-455-Data.db
sections=1 progress=0/792707 - 0%
Streaming from: /IP
KEYSPACE:
/var/lib/cassandra/data/KEYSPACE/ns_history_org/KEYSPACE-ns_history_org-hf-255-Data.db
sections=1 progress=0/1398197628 - 0%
KEYSPACE:
/var/lib/cassandra/data/KEYSPACE/ns_history_biz/KEYSPACE-ns_history_biz-ic-341-Data.db
sections=1 progress=0/6153542 - 0%
KEYSPACE:
/var/lib/cassandra/data/KEYSPACE/listing/KEYSPACE-listing-hf-539-Data.db
sections=1 progress=0/86968194 - 0%
KEYSPACE:
/var/lib/cassandra/data/KEYSPACE/ns_history_biz/KEYSPACE-ns_history_biz-hf-244-Data.db
sections=1 progress=0/322197762 - 0%
KEYSPACE:
/var/lib/cassandra/data/KEYSPACE/ns_history_biz/KEYSPACE-ns_history_biz-ic-346-Data.db
sections=1 progress=0/6219503 - 0%
KEYSPACE:
/var/lib/cassandra/data/KEYSPACE/listing/KEYSPACE-listing-hf-487-Data.db
sections=1 progress=0/2689291466 - 0%
KEYSPACE:
/var/lib/cassandra/data/KEYSPACE/listing/KEYSPACE-listing-ic-684-Data.db
sections=1 progress=0/3717513 - 0%
KEYSPACE:
/var/lib/cassandra/data/KEYSPACE/ns_history_org/KEYSPACE-ns_history_org-ic-413-Data.db
sections=1 progress=0/22256993 - 0%
KEYSPACE:
/var/lib/cassandra/data/KEYSPACE/listing/KEYSPACE-listing-hf-529-Data.db
sections=1 progress=0/345419053 - 0%
KEYSPACE:
/var/lib/cassandra/data/KEYSPACE/listing/KEYSPACE-listing-hf-534-Data.db
sections=1 progress=0/88759930 - 0%
KEYSPACE:
/var/lib/cassandra/data/KEYSPACE/listing/KEYSPACE-listing-hf-509-Data.db
sections=1 progress=0/365451892 - 0%
Read Repair Statistics:
Attempted: 10696592
Mismatch (Blocking): 48873
Mismatch (Background): 47308
Pool Name Active Pending Completed
Commands n/a 0 339083095
Responses n/a 0 270274923
{code}
Netstats showing zero progress and it's not progressing.
was:
I've changed the replication factor for one of keyspaces and now I'm running
repairs on column families (manually). I have 3 nodes cluster. Here is the
error in the error log for one of the nodes.
{code:title=Error Log}
ERROR [Streaming to /208.94.232.9:1] 2013-07-01 09:31:29,819
CassandraDaemon.java (line 192) Exception in thread Thread[Streaming to
/208.94.232.9:1,5,main]
java.lang.RuntimeException: java.io.IOException: Broken pipe
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
at sun.nio.ch.FileChannelImpl.transferToDirectly(Unknown Source)
at sun.nio.ch.FileChannelImpl.transferTo(Unknown Source)
at
org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:93)
at
org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
... 3 more
ERROR [Streaming to /208.94.232.135:2] 2013-07-01 09:44:18,372
CassandraDaemon.java (line 192) Exception in thread Thread[Streaming to
/208.94.232.135:2,5,main]
java.lang.RuntimeException: java.io.IOException: Broken pipe
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
at sun.nio.ch.FileChannelImpl.transferToDirectly(Unknown Source)
at sun.nio.ch.FileChannelImpl.transferTo(Unknown Source)
at
org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:93)
at
org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
... 3 more
{code}
{code:title=netstats for one of the nodes}
Mode: NORMAL
Not sending any streams.
Streaming from: /IP
KEYSPACE:
/var/lib/cassandra/data/KEYSPACE/ns_history_org/KEYSPACE-ns_history_org-hf-282-Data.db
sections=1 progress=0/1393915753 - 0%
KEYSPACE:
/var/lib/cassandra/data/KEYSPACE/ns_history_org/KEYSPACE-ns_history_org-ic-455-Data.db
sections=1 progress=0/792707 - 0%
Streaming from: /IP
KEYSPACE:
/var/lib/cassandra/data/KEYSPACE/ns_history_org/KEYSPACE-ns_history_org-hf-255-Data.db
sections=1 progress=0/1398197628 - 0%
KEYSPACE:
/var/lib/cassandra/data/KEYSPACE/ns_history_biz/KEYSPACE-ns_history_biz-ic-341-Data.db
sections=1 progress=0/6153542 - 0%
KEYSPACE:
/var/lib/cassandra/data/KEYSPACE/listing/KEYSPACE-listing-hf-539-Data.db
sections=1 progress=0/86968194 - 0%
KEYSPACE:
/var/lib/cassandra/data/KEYSPACE/ns_history_biz/KEYSPACE-ns_history_biz-hf-244-Data.db
sections=1 progress=0/322197762 - 0%
KEYSPACE:
/var/lib/cassandra/data/KEYSPACE/ns_history_biz/KEYSPACE-ns_history_biz-ic-346-Data.db
sections=1 progress=0/6219503 - 0%
KEYSPACE:
/var/lib/cassandra/data/KEYSPACE/listing/KEYSPACE-listing-hf-487-Data.db
sections=1 progress=0/2689291466 - 0%
KEYSPACE:
/var/lib/cassandra/data/KEYSPACE/listing/KEYSPACE-listing-ic-684-Data.db
sections=1 progress=0/3717513 - 0%
KEYSPACE:
/var/lib/cassandra/data/KEYSPACE/ns_history_org/KEYSPACE-ns_history_org-ic-413-Data.db
sections=1 progress=0/22256993 - 0%
KEYSPACE:
/var/lib/cassandra/data/KEYSPACE/listing/KEYSPACE-listing-hf-529-Data.db
sections=1 progress=0/345419053 - 0%
KEYSPACE:
/var/lib/cassandra/data/KEYSPACE/listing/KEYSPACE-listing-hf-534-Data.db
sections=1 progress=0/88759930 - 0%
KEYSPACE:
/var/lib/cassandra/data/KEYSPACE/listing/KEYSPACE-listing-hf-509-Data.db
sections=1 progress=0/365451892 - 0%
Read Repair Statistics:
Attempted: 10696592
Mismatch (Blocking): 48873
Mismatch (Background): 47308
Pool Name Active Pending Completed
Commands n/a 0 339083095
Responses n/a 0 270274923
{code}
Netstats showing zero progress and it's not progressing.
> Repair causes streaming errors
> ------------------------------
>
> Key: CASSANDRA-5717
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5717
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.2.6
> Environment: CentOS release 6.3 (Final)
> Reporter: Yoan Arnaudov
>
> I've changed the replication factor for one of keyspaces and now I'm running
> repairs on column families (manually). I have 3 nodes cluster. Here is the
> error in the error log for one of the nodes.
> {code:title=Error Log}
> ERROR [Streaming to /NODE_IP:1] 2013-07-01 09:31:29,819 CassandraDaemon.java
> (line 192) Exception in thread Thread[Streaming to /NODE_IP:1,5,main]
> java.lang.RuntimeException: java.io.IOException: Broken pipe
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Broken pipe
> at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
> at sun.nio.ch.FileChannelImpl.transferToDirectly(Unknown Source)
> at sun.nio.ch.FileChannelImpl.transferTo(Unknown Source)
> at
> org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:93)
> at
> org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ... 3 more
> ERROR [Streaming to /NODE_IP:2] 2013-07-01 09:44:18,372 CassandraDaemon.java
> (line 192) Exception in thread Thread[Streaming to /NODE_IP:2,5,main]
> java.lang.RuntimeException: java.io.IOException: Broken pipe
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Broken pipe
> at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
> at sun.nio.ch.FileChannelImpl.transferToDirectly(Unknown Source)
> at sun.nio.ch.FileChannelImpl.transferTo(Unknown Source)
> at
> org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:93)
> at
> org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ... 3 more
> {code}
> {code:title=netstats for one of the nodes}
> Mode: NORMAL
> Not sending any streams.
> Streaming from: /IP
> KEYSPACE:
> /var/lib/cassandra/data/KEYSPACE/ns_history_org/KEYSPACE-ns_history_org-hf-282-Data.db
> sections=1 progress=0/1393915753 - 0%
> KEYSPACE:
> /var/lib/cassandra/data/KEYSPACE/ns_history_org/KEYSPACE-ns_history_org-ic-455-Data.db
> sections=1 progress=0/792707 - 0%
> Streaming from: /IP
> KEYSPACE:
> /var/lib/cassandra/data/KEYSPACE/ns_history_org/KEYSPACE-ns_history_org-hf-255-Data.db
> sections=1 progress=0/1398197628 - 0%
> KEYSPACE:
> /var/lib/cassandra/data/KEYSPACE/ns_history_biz/KEYSPACE-ns_history_biz-ic-341-Data.db
> sections=1 progress=0/6153542 - 0%
> KEYSPACE:
> /var/lib/cassandra/data/KEYSPACE/listing/KEYSPACE-listing-hf-539-Data.db
> sections=1 progress=0/86968194 - 0%
> KEYSPACE:
> /var/lib/cassandra/data/KEYSPACE/ns_history_biz/KEYSPACE-ns_history_biz-hf-244-Data.db
> sections=1 progress=0/322197762 - 0%
> KEYSPACE:
> /var/lib/cassandra/data/KEYSPACE/ns_history_biz/KEYSPACE-ns_history_biz-ic-346-Data.db
> sections=1 progress=0/6219503 - 0%
> KEYSPACE:
> /var/lib/cassandra/data/KEYSPACE/listing/KEYSPACE-listing-hf-487-Data.db
> sections=1 progress=0/2689291466 - 0%
> KEYSPACE:
> /var/lib/cassandra/data/KEYSPACE/listing/KEYSPACE-listing-ic-684-Data.db
> sections=1 progress=0/3717513 - 0%
> KEYSPACE:
> /var/lib/cassandra/data/KEYSPACE/ns_history_org/KEYSPACE-ns_history_org-ic-413-Data.db
> sections=1 progress=0/22256993 - 0%
> KEYSPACE:
> /var/lib/cassandra/data/KEYSPACE/listing/KEYSPACE-listing-hf-529-Data.db
> sections=1 progress=0/345419053 - 0%
> KEYSPACE:
> /var/lib/cassandra/data/KEYSPACE/listing/KEYSPACE-listing-hf-534-Data.db
> sections=1 progress=0/88759930 - 0%
> KEYSPACE:
> /var/lib/cassandra/data/KEYSPACE/listing/KEYSPACE-listing-hf-509-Data.db
> sections=1 progress=0/365451892 - 0%
> Read Repair Statistics:
> Attempted: 10696592
> Mismatch (Blocking): 48873
> Mismatch (Background): 47308
> Pool Name Active Pending Completed
> Commands n/a 0 339083095
> Responses n/a 0 270274923
> {code}
> Netstats showing zero progress and it's not progressing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
