[jira] [Commented] (CASSANDRA-5381) java.io.EOFException exception while executing nodetool repair with compression enabled
[ https://issues.apache.org/jira/browse/CASSANDRA-5381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618665#comment-13618665 ] Neil Thomson commented on CASSANDRA-5381: - Hi, yes we also have node encryption enabled. java.io.EOFException exception while executing nodetool repair with compression enabled --- Key: CASSANDRA-5381 URL: https://issues.apache.org/jira/browse/CASSANDRA-5381 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.2.3 Environment: Linux Virtual Machines, Red Hat Enterprise release 6.4, kernel version 2.6.32-358.2.1.el6.x86_64. Each VM has 8GB memory and 4vCPUS. Reporter: Neil Thomson Priority: Minor Very similar to issue reported in CASSANDRA-5105. I have 3 nodes configured in a cluster. The nodes are configured with compression enabled. When attempting a nodetool repair on one node, i get exceptions in the other nodes in the cluster. Disabling compression on the column family allows nodetool repair to run without error. Exception: INFO [Streaming to /3.69.211.179:2] 2013-03-25 12:30:27,874 StreamReplyVerbHandler.java (line 50) Need to re-stream file /var/lib/cassandra/data/rt/values/rt-values-ib-1-Data.db to /3.69.211.179 INFO [Streaming to /3.69.211.179:2] 2013-03-25 12:30:27,991 StreamReplyVerbHandler.java (line 50) Need to re-stream file /var/lib/cassandra/data/rt/values/rt-values-ib-1-Data.db to /3.69.211.179 ERROR [Streaming to /3.69.211.179:2] 2013-03-25 12:30:28,113 CassandraDaemon.java (line 164) Exception in thread Thread[Streaming to /3.69.211.179:2,5,main] java.lang.RuntimeException: java.io.EOFException at com.google.common.base.Throwables.propagate(Throwables.java:160) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.io.EOFException at java.io.DataInputStream.readInt(Unknown Source) at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:193) at org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:114) at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ... 3 more Keyspace configuration is as follows: Keyspace: rt: Replication Strategy: org.apache.cassandra.locator.SimpleStrategy Durable Writes: true Options: [replication_factor:3] Column Families: ColumnFamily: tagname Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 0.1 DC Local Read repair chance: 0.0 Populate IO Cache on flush: false Replicate on write: true Caching: KEYS_ONLY Bloom Filter FP chance: default Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: values Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 0.1 DC Local Read repair chance: 0.0 Populate IO Cache on flush: false Replicate on write: true Caching: KEYS_ONLY Bloom Filter FP chance: default Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5381) java.io.EOFException exception while executing nodetool repair with compression enabled
[ https://issues.apache.org/jira/browse/CASSANDRA-5381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13617038#comment-13617038 ] Yuki Morishita commented on CASSANDRA-5381: --- [~mathijs] you are right and I attached fix to CASSANDRA-5391 which also is reporting the error. thanks for reporting. java.io.EOFException exception while executing nodetool repair with compression enabled --- Key: CASSANDRA-5381 URL: https://issues.apache.org/jira/browse/CASSANDRA-5381 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.2.3 Environment: Linux Virtual Machines, Red Hat Enterprise release 6.4, kernel version 2.6.32-358.2.1.el6.x86_64. Each VM has 8GB memory and 4vCPUS. Reporter: Neil Thomson Priority: Minor Very similar to issue reported in CASSANDRA-5105. I have 3 nodes configured in a cluster. The nodes are configured with compression enabled. When attempting a nodetool repair on one node, i get exceptions in the other nodes in the cluster. Disabling compression on the column family allows nodetool repair to run without error. Exception: INFO [Streaming to /3.69.211.179:2] 2013-03-25 12:30:27,874 StreamReplyVerbHandler.java (line 50) Need to re-stream file /var/lib/cassandra/data/rt/values/rt-values-ib-1-Data.db to /3.69.211.179 INFO [Streaming to /3.69.211.179:2] 2013-03-25 12:30:27,991 StreamReplyVerbHandler.java (line 50) Need to re-stream file /var/lib/cassandra/data/rt/values/rt-values-ib-1-Data.db to /3.69.211.179 ERROR [Streaming to /3.69.211.179:2] 2013-03-25 12:30:28,113 CassandraDaemon.java (line 164) Exception in thread Thread[Streaming to /3.69.211.179:2,5,main] java.lang.RuntimeException: java.io.EOFException at com.google.common.base.Throwables.propagate(Throwables.java:160) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.io.EOFException at java.io.DataInputStream.readInt(Unknown Source) at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:193) at org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:114) at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ... 3 more Keyspace configuration is as follows: Keyspace: rt: Replication Strategy: org.apache.cassandra.locator.SimpleStrategy Durable Writes: true Options: [replication_factor:3] Column Families: ColumnFamily: tagname Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 0.1 DC Local Read repair chance: 0.0 Populate IO Cache on flush: false Replicate on write: true Caching: KEYS_ONLY Bloom Filter FP chance: default Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: values Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 0.1 DC Local Read repair chance: 0.0 Populate IO Cache on flush: false Replicate on write: true Caching: KEYS_ONLY Bloom Filter FP chance: default Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5381) java.io.EOFException exception while executing nodetool repair with compression enabled
[ https://issues.apache.org/jira/browse/CASSANDRA-5381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13617041#comment-13617041 ] Yuki Morishita commented on CASSANDRA-5381: --- [~neil.thomp...@shepway.gov.uk] are you also using internode_encryption? then this is duplicate issue of CASSANDRA-5391. java.io.EOFException exception while executing nodetool repair with compression enabled --- Key: CASSANDRA-5381 URL: https://issues.apache.org/jira/browse/CASSANDRA-5381 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.2.3 Environment: Linux Virtual Machines, Red Hat Enterprise release 6.4, kernel version 2.6.32-358.2.1.el6.x86_64. Each VM has 8GB memory and 4vCPUS. Reporter: Neil Thomson Priority: Minor Very similar to issue reported in CASSANDRA-5105. I have 3 nodes configured in a cluster. The nodes are configured with compression enabled. When attempting a nodetool repair on one node, i get exceptions in the other nodes in the cluster. Disabling compression on the column family allows nodetool repair to run without error. Exception: INFO [Streaming to /3.69.211.179:2] 2013-03-25 12:30:27,874 StreamReplyVerbHandler.java (line 50) Need to re-stream file /var/lib/cassandra/data/rt/values/rt-values-ib-1-Data.db to /3.69.211.179 INFO [Streaming to /3.69.211.179:2] 2013-03-25 12:30:27,991 StreamReplyVerbHandler.java (line 50) Need to re-stream file /var/lib/cassandra/data/rt/values/rt-values-ib-1-Data.db to /3.69.211.179 ERROR [Streaming to /3.69.211.179:2] 2013-03-25 12:30:28,113 CassandraDaemon.java (line 164) Exception in thread Thread[Streaming to /3.69.211.179:2,5,main] java.lang.RuntimeException: java.io.EOFException at com.google.common.base.Throwables.propagate(Throwables.java:160) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.io.EOFException at java.io.DataInputStream.readInt(Unknown Source) at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:193) at org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:114) at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ... 3 more Keyspace configuration is as follows: Keyspace: rt: Replication Strategy: org.apache.cassandra.locator.SimpleStrategy Durable Writes: true Options: [replication_factor:3] Column Families: ColumnFamily: tagname Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 0.1 DC Local Read repair chance: 0.0 Populate IO Cache on flush: false Replicate on write: true Caching: KEYS_ONLY Bloom Filter FP chance: default Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: values Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 0.1 DC Local Read repair chance: 0.0 Populate IO Cache on flush: false Replicate on write: true Caching: KEYS_ONLY Bloom Filter FP chance: default Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5381) java.io.EOFException exception while executing nodetool repair with compression enabled
[ https://issues.apache.org/jira/browse/CASSANDRA-5381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13615171#comment-13615171 ] Mathijs Vogelzang commented on CASSANDRA-5381: -- We have the same issue where all streaming between nodes fails with an EOFException and then too many retries. This started when we upgraded from 1.1.7 to 1.2.2, and didn't go away on subsequent upgrade to 1.2.3. We tried running with/without internode compression and encryption, and found out that when encryption is off, everything works fine (also WITH compression on). With encryption on, it doesn't work, also with internode compression turned off, so for us it definitely has something to do with streaming while internode encryption is enabled. java.io.EOFException exception while executing nodetool repair with compression enabled --- Key: CASSANDRA-5381 URL: https://issues.apache.org/jira/browse/CASSANDRA-5381 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.2.3 Environment: Linux Virtual Machines, Red Hat Enterprise release 6.4, kernel version 2.6.32-358.2.1.el6.x86_64. Each VM has 8GB memory and 4vCPUS. Reporter: Neil Thomson Priority: Minor Very similar to issue reported in CASSANDRA-5105. I have 3 nodes configured in a cluster. The nodes are configured with compression enabled. When attempting a nodetool repair on one node, i get exceptions in the other nodes in the cluster. Disabling compression on the column family allows nodetool repair to run without error. Exception: INFO [Streaming to /3.69.211.179:2] 2013-03-25 12:30:27,874 StreamReplyVerbHandler.java (line 50) Need to re-stream file /var/lib/cassandra/data/rt/values/rt-values-ib-1-Data.db to /3.69.211.179 INFO [Streaming to /3.69.211.179:2] 2013-03-25 12:30:27,991 StreamReplyVerbHandler.java (line 50) Need to re-stream file /var/lib/cassandra/data/rt/values/rt-values-ib-1-Data.db to /3.69.211.179 ERROR [Streaming to /3.69.211.179:2] 2013-03-25 12:30:28,113 CassandraDaemon.java (line 164) Exception in thread Thread[Streaming to /3.69.211.179:2,5,main] java.lang.RuntimeException: java.io.EOFException at com.google.common.base.Throwables.propagate(Throwables.java:160) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.io.EOFException at java.io.DataInputStream.readInt(Unknown Source) at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:193) at org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:114) at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ... 3 more Keyspace configuration is as follows: Keyspace: rt: Replication Strategy: org.apache.cassandra.locator.SimpleStrategy Durable Writes: true Options: [replication_factor:3] Column Families: ColumnFamily: tagname Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 0.1 DC Local Read repair chance: 0.0 Populate IO Cache on flush: false Replicate on write: true Caching: KEYS_ONLY Bloom Filter FP chance: default Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: values Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 0.1 DC Local Read repair chance: 0.0 Populate IO Cache on flush: false Replicate on write: true Caching: KEYS_ONLY Bloom Filter FP chance: default Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA,
[jira] [Commented] (CASSANDRA-5381) java.io.EOFException exception while executing nodetool repair with compression enabled
[ https://issues.apache.org/jira/browse/CASSANDRA-5381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13612665#comment-13612665 ] Yuki Morishita commented on CASSANDRA-5381: --- Do you see the error on the other end(3.69.211.179)? What is it look like? java.io.EOFException exception while executing nodetool repair with compression enabled --- Key: CASSANDRA-5381 URL: https://issues.apache.org/jira/browse/CASSANDRA-5381 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.2.3 Environment: Linux Virtual Machines, Red Hat Enterprise release 6.4, kernel version 2.6.32-358.2.1.el6.x86_64. Each VM has 8GB memory and 4vCPUS. Reporter: Neil Thomson Priority: Minor Very similar to issue reported in CASSANDRA-5105. I have 3 nodes configured in a cluster. The nodes are configured with compression enabled. When attempting a nodetool repair on one node, i get exceptions in the other nodes in the cluster. Disabling compression on the column family allows nodetool repair to run without error. Exception: INFO [Streaming to /3.69.211.179:2] 2013-03-25 12:30:27,874 StreamReplyVerbHandler.java (line 50) Need to re-stream file /var/lib/cassandra/data/rt/values/rt-values-ib-1-Data.db to /3.69.211.179 INFO [Streaming to /3.69.211.179:2] 2013-03-25 12:30:27,991 StreamReplyVerbHandler.java (line 50) Need to re-stream file /var/lib/cassandra/data/rt/values/rt-values-ib-1-Data.db to /3.69.211.179 ERROR [Streaming to /3.69.211.179:2] 2013-03-25 12:30:28,113 CassandraDaemon.java (line 164) Exception in thread Thread[Streaming to /3.69.211.179:2,5,main] java.lang.RuntimeException: java.io.EOFException at com.google.common.base.Throwables.propagate(Throwables.java:160) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.io.EOFException at java.io.DataInputStream.readInt(Unknown Source) at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:193) at org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:114) at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ... 3 more Keyspace configuration is as follows: Keyspace: rt: Replication Strategy: org.apache.cassandra.locator.SimpleStrategy Durable Writes: true Options: [replication_factor:3] Column Families: ColumnFamily: tagname Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 0.1 DC Local Read repair chance: 0.0 Populate IO Cache on flush: false Replicate on write: true Caching: KEYS_ONLY Bloom Filter FP chance: default Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: values Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 0.1 DC Local Read repair chance: 0.0 Populate IO Cache on flush: false Replicate on write: true Caching: KEYS_ONLY Bloom Filter FP chance: default Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5381) java.io.EOFException exception while executing nodetool repair with compression enabled
[ https://issues.apache.org/jira/browse/CASSANDRA-5381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13612687#comment-13612687 ] Neil Thomson commented on CASSANDRA-5381: - Hi. Thanks for getting back so quickly. This is the log on the node being repaired. The nodetool command actually hangs and never returns. [root@livutsds03 ~]# nodetool repair [2013-03-25 14:36:05,177] Nothing to repair for keyspace 'system' [2013-03-25 14:36:05,209] Starting repair command #1, repairing 509 ranges for keyspace rt Complete log : INFO [Thread-10] 2013-03-25 14:36:05,208 StorageService.java (line 2379) Starting repair command #1, repairing 509 ranges for keyspace rt INFO [AntiEntropySessions:1] 2013-03-25 14:36:05,768 AntiEntropyService.java (line 652) [repair #53071fe0-9559-11e2-b719-570d5f987031] new session: will sync /3.69.211.179, /3.69.211.177 on range (376298460076626656,380384391924186156] for rt.[values, tagname] INFO [AntiEntropySessions:1] 2013-03-25 14:36:05,772 AntiEntropyService.java (line 858) [repair #53071fe0-9559-11e2-b719-570d5f987031] requesting merkle trees for values (to [/3.69.211.177, /3.69.211.179]) INFO [AntiEntropyStage:1] 2013-03-25 14:36:06,344 AntiEntropyService.java (line 214) [repair #53071fe0-9559-11e2-b719-570d5f987031] Received merkle tree for values from /3.69.211.177 INFO [AntiEntropyStage:1] 2013-03-25 14:36:06,382 AntiEntropyService.java (line 214) [repair #53071fe0-9559-11e2-b719-570d5f987031] Received merkle tree for values from /3.69.211.179 INFO [AntiEntropyStage:1] 2013-03-25 14:36:06,383 AntiEntropyService.java (line 858) [repair #53071fe0-9559-11e2-b719-570d5f987031] requesting merkle trees for tagname (to [/3.69.211.177, livutsds03/3.69.211.179]) INFO [AntiEntropyStage:1] 2013-03-25 14:36:06,497 AntiEntropyService.java (line 995) [repair #53071fe0-9559-11e2-b719-570d5f987031] Endpoints /3.69.211.177 and /3.69.211.179 have 7 range(s) out of sync for values INFO [AntiEntropyStage:1] 2013-03-25 14:36:06,498 StreamingRepairTask.java (line 128) [streaming task #53cb0c20-9559-11e2-b719-570d5f987031] Performing streaming repair of 7 ranges with /3.69.211.177 INFO [AntiEntropyStage:1] 2013-03-25 14:36:06,502 StreamOut.java (line 159) Stream context metadata [], 0 sstables. INFO [AntiEntropyStage:1] 2013-03-25 14:36:06,503 StreamOutSession.java (line 162) Streaming to /3.69.211.177 INFO [AntiEntropyStage:1] 2013-03-25 14:36:06,531 AntiEntropyService.java (line 214) [repair #53071fe0-9559-11e2-b719-570d5f987031] Received merkle tree for tagname from /3.69.211.177 INFO [Streaming to /3.69.211.177:1] 2013-03-25 14:36:06,620 FileStreamTask.java (line 102) Finished streaming session to /3.69.211.177 INFO [Thread-12] 2013-03-25 14:36:06,672 StreamInSession.java (line 136) Streaming of file /var/lib/cassandra/data/rt/values/rt-values-ib-1-Data.db sections=7 progress=0/16468 - 0% for org.apache.cassandra.streaming.StreamInSession@33469a69 failed: requesting a retry. INFO [AntiEntropyStage:1] 2013-03-25 14:36:06,708 AntiEntropyService.java (line 214) [repair #53071fe0-9559-11e2-b719-570d5f987031] Received merkle tree for tagname from /3.69.211.179 INFO [AntiEntropySessions:2] 2013-03-25 14:36:06,722 AntiEntropyService.java (line 652) [repair #53eb3e50-9559-11e2-b719-570d5f987031] new session: will sync livutsds03/3.69.211.179, /3.69.211.177 on range (1742492608639662568,1810636791599146663] for rt.[values, tagname] INFO [AntiEntropySessions:2] 2013-03-25 14:36:06,723 AntiEntropyService.java (line 858) [repair #53eb3e50-9559-11e2-b719-570d5f987031] requesting merkle trees for values (to [/3.69.211.177, livutsds03/3.69.211.179]) INFO [Thread-14] 2013-03-25 14:36:06,795 StreamInSession.java (line 136) Streaming of file /var/lib/cassandra/data/rt/values/rt-values-ib-1-Data.db sections=7 progress=0/16468 - 0% for org.apache.cassandra.streaming.StreamInSession@33469a69 failed: requesting a retry. INFO [AntiEntropyStage:1] 2013-03-25 14:36:06,820 AntiEntropyService.java (line 989) [repair #53071fe0-9559-11e2-b719-570d5f987031] Endpoints /3.69.211.177 and /3.69.211.179 are consistent for tagname INFO [AntiEntropyStage:1] 2013-03-25 14:36:06,821 AntiEntropyService.java (line 765) [repair #53071fe0-9559-11e2-b719-570d5f987031] tagname is fully synced (1 remaining column family to sync for this session) INFO [AntiEntropyStage:1] 2013-03-25 14:36:06,864 AntiEntropyService.java (line 214) [repair #53eb3e50-9559-11e2-b719-570d5f987031] Received merkle tree for values from /3.69.211.177 INFO [Thread-16] 2013-03-25 14:36:06,918 StreamInSession.java (line 136) Streaming of file /var/lib/cassandra/data/rt/values/rt-values-ib-1-Data.db sections=7 progress=0/16468 - 0% for org.apache.cassandra.streaming.StreamInSession@33469a69 failed: requesting a retry. INFO [AntiEntropyStage:1] 2013-03-25 14:36:06,972 AntiEntropyService.java (line 214) [repair
[jira] [Commented] (CASSANDRA-5381) java.io.EOFException exception while executing nodetool repair with compression enabled
[ https://issues.apache.org/jira/browse/CASSANDRA-5381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13612800#comment-13612800 ] Neil Thomson commented on CASSANDRA-5381: - Please note that this is an issue in 1.2.2 and 1.2.3, i.e. i do not believe that this is an issue that has been introduced in the 1.2.3 release. java.io.EOFException exception while executing nodetool repair with compression enabled --- Key: CASSANDRA-5381 URL: https://issues.apache.org/jira/browse/CASSANDRA-5381 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.2.3 Environment: Linux Virtual Machines, Red Hat Enterprise release 6.4, kernel version 2.6.32-358.2.1.el6.x86_64. Each VM has 8GB memory and 4vCPUS. Reporter: Neil Thomson Priority: Minor Very similar to issue reported in CASSANDRA-5105. I have 3 nodes configured in a cluster. The nodes are configured with compression enabled. When attempting a nodetool repair on one node, i get exceptions in the other nodes in the cluster. Disabling compression on the column family allows nodetool repair to run without error. Exception: INFO [Streaming to /3.69.211.179:2] 2013-03-25 12:30:27,874 StreamReplyVerbHandler.java (line 50) Need to re-stream file /var/lib/cassandra/data/rt/values/rt-values-ib-1-Data.db to /3.69.211.179 INFO [Streaming to /3.69.211.179:2] 2013-03-25 12:30:27,991 StreamReplyVerbHandler.java (line 50) Need to re-stream file /var/lib/cassandra/data/rt/values/rt-values-ib-1-Data.db to /3.69.211.179 ERROR [Streaming to /3.69.211.179:2] 2013-03-25 12:30:28,113 CassandraDaemon.java (line 164) Exception in thread Thread[Streaming to /3.69.211.179:2,5,main] java.lang.RuntimeException: java.io.EOFException at com.google.common.base.Throwables.propagate(Throwables.java:160) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.io.EOFException at java.io.DataInputStream.readInt(Unknown Source) at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:193) at org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:114) at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ... 3 more Keyspace configuration is as follows: Keyspace: rt: Replication Strategy: org.apache.cassandra.locator.SimpleStrategy Durable Writes: true Options: [replication_factor:3] Column Families: ColumnFamily: tagname Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 0.1 DC Local Read repair chance: 0.0 Populate IO Cache on flush: false Replicate on write: true Caching: KEYS_ONLY Bloom Filter FP chance: default Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: values Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 0.1 DC Local Read repair chance: 0.0 Populate IO Cache on flush: false Replicate on write: true Caching: KEYS_ONLY Bloom Filter FP chance: default Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5381) java.io.EOFException exception while executing nodetool repair with compression enabled
[ https://issues.apache.org/jira/browse/CASSANDRA-5381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13613462#comment-13613462 ] Yuki Morishita commented on CASSANDRA-5381: --- Timestamp from the second log does not match the first one, but I assume it's the same retry failed error. Cassandra attempts file streaming retry when error happens, and fails if retry count exceeds max_retry(default 3, but configurable in your cassandra.yaml). The node then sends back streaming session failure message to origin, but if the message cannot be sent somehow, you will encounter EOFException. I cannot tell why your streaming is failing from the log. One thing for sure is the repair would hang if streaming failed. Note that we have to fix in 2.0 because we need to change message format to respond failure for forwarded repair. java.io.EOFException exception while executing nodetool repair with compression enabled --- Key: CASSANDRA-5381 URL: https://issues.apache.org/jira/browse/CASSANDRA-5381 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.2.3 Environment: Linux Virtual Machines, Red Hat Enterprise release 6.4, kernel version 2.6.32-358.2.1.el6.x86_64. Each VM has 8GB memory and 4vCPUS. Reporter: Neil Thomson Priority: Minor Very similar to issue reported in CASSANDRA-5105. I have 3 nodes configured in a cluster. The nodes are configured with compression enabled. When attempting a nodetool repair on one node, i get exceptions in the other nodes in the cluster. Disabling compression on the column family allows nodetool repair to run without error. Exception: INFO [Streaming to /3.69.211.179:2] 2013-03-25 12:30:27,874 StreamReplyVerbHandler.java (line 50) Need to re-stream file /var/lib/cassandra/data/rt/values/rt-values-ib-1-Data.db to /3.69.211.179 INFO [Streaming to /3.69.211.179:2] 2013-03-25 12:30:27,991 StreamReplyVerbHandler.java (line 50) Need to re-stream file /var/lib/cassandra/data/rt/values/rt-values-ib-1-Data.db to /3.69.211.179 ERROR [Streaming to /3.69.211.179:2] 2013-03-25 12:30:28,113 CassandraDaemon.java (line 164) Exception in thread Thread[Streaming to /3.69.211.179:2,5,main] java.lang.RuntimeException: java.io.EOFException at com.google.common.base.Throwables.propagate(Throwables.java:160) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.io.EOFException at java.io.DataInputStream.readInt(Unknown Source) at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:193) at org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:114) at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ... 3 more Keyspace configuration is as follows: Keyspace: rt: Replication Strategy: org.apache.cassandra.locator.SimpleStrategy Durable Writes: true Options: [replication_factor:3] Column Families: ColumnFamily: tagname Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 0.1 DC Local Read repair chance: 0.0 Populate IO Cache on flush: false Replicate on write: true Caching: KEYS_ONLY Bloom Filter FP chance: default Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: values Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 0.1 DC Local Read repair chance: 0.0 Populate IO Cache on flush: false Replicate on write: true Caching: KEYS_ONLY Bloom Filter FP chance: default Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy -- This message is automatically generated by JIRA. If you think it was
