[jira] [Commented] (CASSANDRA-5381) java.io.EOFException exception while executing nodetool repair with compression enabled
[ https://issues.apache.org/jira/browse/CASSANDRA-5381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618665#comment-13618665 ] Neil Thomson commented on CASSANDRA-5381: - Hi, yes we also have node encryption enabled. java.io.EOFException exception while executing nodetool repair with compression enabled --- Key: CASSANDRA-5381 URL: https://issues.apache.org/jira/browse/CASSANDRA-5381 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.2.3 Environment: Linux Virtual Machines, Red Hat Enterprise release 6.4, kernel version 2.6.32-358.2.1.el6.x86_64. Each VM has 8GB memory and 4vCPUS. Reporter: Neil Thomson Priority: Minor Very similar to issue reported in CASSANDRA-5105. I have 3 nodes configured in a cluster. The nodes are configured with compression enabled. When attempting a nodetool repair on one node, i get exceptions in the other nodes in the cluster. Disabling compression on the column family allows nodetool repair to run without error. Exception: INFO [Streaming to /3.69.211.179:2] 2013-03-25 12:30:27,874 StreamReplyVerbHandler.java (line 50) Need to re-stream file /var/lib/cassandra/data/rt/values/rt-values-ib-1-Data.db to /3.69.211.179 INFO [Streaming to /3.69.211.179:2] 2013-03-25 12:30:27,991 StreamReplyVerbHandler.java (line 50) Need to re-stream file /var/lib/cassandra/data/rt/values/rt-values-ib-1-Data.db to /3.69.211.179 ERROR [Streaming to /3.69.211.179:2] 2013-03-25 12:30:28,113 CassandraDaemon.java (line 164) Exception in thread Thread[Streaming to /3.69.211.179:2,5,main] java.lang.RuntimeException: java.io.EOFException at com.google.common.base.Throwables.propagate(Throwables.java:160) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.io.EOFException at java.io.DataInputStream.readInt(Unknown Source) at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:193) at org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:114) at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ... 3 more Keyspace configuration is as follows: Keyspace: rt: Replication Strategy: org.apache.cassandra.locator.SimpleStrategy Durable Writes: true Options: [replication_factor:3] Column Families: ColumnFamily: tagname Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 0.1 DC Local Read repair chance: 0.0 Populate IO Cache on flush: false Replicate on write: true Caching: KEYS_ONLY Bloom Filter FP chance: default Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: values Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 0.1 DC Local Read repair chance: 0.0 Populate IO Cache on flush: false Replicate on write: true Caching: KEYS_ONLY Bloom Filter FP chance: default Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-5381) java.io.EOFException exception while executing nodetool repair with compression enabled
Neil Thomson created CASSANDRA-5381: --- Summary: java.io.EOFException exception while executing nodetool repair with compression enabled Key: CASSANDRA-5381 URL: https://issues.apache.org/jira/browse/CASSANDRA-5381 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.2.3 Environment: Linux Virtual Machines, Red Hat Enterprise release 6.4, kernel version 2.6.32-358.2.1.el6.x86_64. Each VM has 8GB memory and 4vCPUS. Reporter: Neil Thomson Priority: Minor Very similar to issue reported in CASSANDRA-5105. I have 3 nodes configured in a cluster. The nodes are configured with compression enabled. When attempting a nodetool repair on one node, i get exceptions in the other nodes in the cluster. Disabling compression on the column family allows nodetool repair to run without error. Exception: INFO [Streaming to /3.69.211.179:2] 2013-03-25 12:30:27,874 StreamReplyVerbHandler.java (line 50) Need to re-stream file /var/lib/cassandra/data/rt/values/rt-values-ib-1-Data.db to /3.69.211.179 INFO [Streaming to /3.69.211.179:2] 2013-03-25 12:30:27,991 StreamReplyVerbHandler.java (line 50) Need to re-stream file /var/lib/cassandra/data/rt/values/rt-values-ib-1-Data.db to /3.69.211.179 ERROR [Streaming to /3.69.211.179:2] 2013-03-25 12:30:28,113 CassandraDaemon.java (line 164) Exception in thread Thread[Streaming to /3.69.211.179:2,5,main] java.lang.RuntimeException: java.io.EOFException at com.google.common.base.Throwables.propagate(Throwables.java:160) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.io.EOFException at java.io.DataInputStream.readInt(Unknown Source) at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:193) at org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:114) at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ... 3 more Keyspace configuration is as follows: Keyspace: rt: Replication Strategy: org.apache.cassandra.locator.SimpleStrategy Durable Writes: true Options: [replication_factor:3] Column Families: ColumnFamily: tagname Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 0.1 DC Local Read repair chance: 0.0 Populate IO Cache on flush: false Replicate on write: true Caching: KEYS_ONLY Bloom Filter FP chance: default Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: values Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 0.1 DC Local Read repair chance: 0.0 Populate IO Cache on flush: false Replicate on write: true Caching: KEYS_ONLY Bloom Filter FP chance: default Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5381) java.io.EOFException exception while executing nodetool repair with compression enabled
[ https://issues.apache.org/jira/browse/CASSANDRA-5381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13612687#comment-13612687 ] Neil Thomson commented on CASSANDRA-5381: - Hi. Thanks for getting back so quickly. This is the log on the node being repaired. The nodetool command actually hangs and never returns. [root@livutsds03 ~]# nodetool repair [2013-03-25 14:36:05,177] Nothing to repair for keyspace 'system' [2013-03-25 14:36:05,209] Starting repair command #1, repairing 509 ranges for keyspace rt Complete log : INFO [Thread-10] 2013-03-25 14:36:05,208 StorageService.java (line 2379) Starting repair command #1, repairing 509 ranges for keyspace rt INFO [AntiEntropySessions:1] 2013-03-25 14:36:05,768 AntiEntropyService.java (line 652) [repair #53071fe0-9559-11e2-b719-570d5f987031] new session: will sync /3.69.211.179, /3.69.211.177 on range (376298460076626656,380384391924186156] for rt.[values, tagname] INFO [AntiEntropySessions:1] 2013-03-25 14:36:05,772 AntiEntropyService.java (line 858) [repair #53071fe0-9559-11e2-b719-570d5f987031] requesting merkle trees for values (to [/3.69.211.177, /3.69.211.179]) INFO [AntiEntropyStage:1] 2013-03-25 14:36:06,344 AntiEntropyService.java (line 214) [repair #53071fe0-9559-11e2-b719-570d5f987031] Received merkle tree for values from /3.69.211.177 INFO [AntiEntropyStage:1] 2013-03-25 14:36:06,382 AntiEntropyService.java (line 214) [repair #53071fe0-9559-11e2-b719-570d5f987031] Received merkle tree for values from /3.69.211.179 INFO [AntiEntropyStage:1] 2013-03-25 14:36:06,383 AntiEntropyService.java (line 858) [repair #53071fe0-9559-11e2-b719-570d5f987031] requesting merkle trees for tagname (to [/3.69.211.177, livutsds03/3.69.211.179]) INFO [AntiEntropyStage:1] 2013-03-25 14:36:06,497 AntiEntropyService.java (line 995) [repair #53071fe0-9559-11e2-b719-570d5f987031] Endpoints /3.69.211.177 and /3.69.211.179 have 7 range(s) out of sync for values INFO [AntiEntropyStage:1] 2013-03-25 14:36:06,498 StreamingRepairTask.java (line 128) [streaming task #53cb0c20-9559-11e2-b719-570d5f987031] Performing streaming repair of 7 ranges with /3.69.211.177 INFO [AntiEntropyStage:1] 2013-03-25 14:36:06,502 StreamOut.java (line 159) Stream context metadata [], 0 sstables. INFO [AntiEntropyStage:1] 2013-03-25 14:36:06,503 StreamOutSession.java (line 162) Streaming to /3.69.211.177 INFO [AntiEntropyStage:1] 2013-03-25 14:36:06,531 AntiEntropyService.java (line 214) [repair #53071fe0-9559-11e2-b719-570d5f987031] Received merkle tree for tagname from /3.69.211.177 INFO [Streaming to /3.69.211.177:1] 2013-03-25 14:36:06,620 FileStreamTask.java (line 102) Finished streaming session to /3.69.211.177 INFO [Thread-12] 2013-03-25 14:36:06,672 StreamInSession.java (line 136) Streaming of file /var/lib/cassandra/data/rt/values/rt-values-ib-1-Data.db sections=7 progress=0/16468 - 0% for org.apache.cassandra.streaming.StreamInSession@33469a69 failed: requesting a retry. INFO [AntiEntropyStage:1] 2013-03-25 14:36:06,708 AntiEntropyService.java (line 214) [repair #53071fe0-9559-11e2-b719-570d5f987031] Received merkle tree for tagname from /3.69.211.179 INFO [AntiEntropySessions:2] 2013-03-25 14:36:06,722 AntiEntropyService.java (line 652) [repair #53eb3e50-9559-11e2-b719-570d5f987031] new session: will sync livutsds03/3.69.211.179, /3.69.211.177 on range (1742492608639662568,1810636791599146663] for rt.[values, tagname] INFO [AntiEntropySessions:2] 2013-03-25 14:36:06,723 AntiEntropyService.java (line 858) [repair #53eb3e50-9559-11e2-b719-570d5f987031] requesting merkle trees for values (to [/3.69.211.177, livutsds03/3.69.211.179]) INFO [Thread-14] 2013-03-25 14:36:06,795 StreamInSession.java (line 136) Streaming of file /var/lib/cassandra/data/rt/values/rt-values-ib-1-Data.db sections=7 progress=0/16468 - 0% for org.apache.cassandra.streaming.StreamInSession@33469a69 failed: requesting a retry. INFO [AntiEntropyStage:1] 2013-03-25 14:36:06,820 AntiEntropyService.java (line 989) [repair #53071fe0-9559-11e2-b719-570d5f987031] Endpoints /3.69.211.177 and /3.69.211.179 are consistent for tagname INFO [AntiEntropyStage:1] 2013-03-25 14:36:06,821 AntiEntropyService.java (line 765) [repair #53071fe0-9559-11e2-b719-570d5f987031] tagname is fully synced (1 remaining column family to sync for this session) INFO [AntiEntropyStage:1] 2013-03-25 14:36:06,864 AntiEntropyService.java (line 214) [repair #53eb3e50-9559-11e2-b719-570d5f987031] Received merkle tree for values from /3.69.211.177 INFO [Thread-16] 2013-03-25 14:36:06,918 StreamInSession.java (line 136) Streaming of file /var/lib/cassandra/data/rt/values/rt-values-ib-1-Data.db sections=7 progress=0/16468 - 0% for org.apache.cassandra.streaming.StreamInSession@33469a69 failed: requesting a retry. INFO [AntiEntropyStage:1] 2013-03-25 14:36:06,972 AntiEntropyService.java (line 214) [repair
[jira] [Commented] (CASSANDRA-5381) java.io.EOFException exception while executing nodetool repair with compression enabled
[ https://issues.apache.org/jira/browse/CASSANDRA-5381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13612800#comment-13612800 ] Neil Thomson commented on CASSANDRA-5381: - Please note that this is an issue in 1.2.2 and 1.2.3, i.e. i do not believe that this is an issue that has been introduced in the 1.2.3 release. java.io.EOFException exception while executing nodetool repair with compression enabled --- Key: CASSANDRA-5381 URL: https://issues.apache.org/jira/browse/CASSANDRA-5381 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.2.3 Environment: Linux Virtual Machines, Red Hat Enterprise release 6.4, kernel version 2.6.32-358.2.1.el6.x86_64. Each VM has 8GB memory and 4vCPUS. Reporter: Neil Thomson Priority: Minor Very similar to issue reported in CASSANDRA-5105. I have 3 nodes configured in a cluster. The nodes are configured with compression enabled. When attempting a nodetool repair on one node, i get exceptions in the other nodes in the cluster. Disabling compression on the column family allows nodetool repair to run without error. Exception: INFO [Streaming to /3.69.211.179:2] 2013-03-25 12:30:27,874 StreamReplyVerbHandler.java (line 50) Need to re-stream file /var/lib/cassandra/data/rt/values/rt-values-ib-1-Data.db to /3.69.211.179 INFO [Streaming to /3.69.211.179:2] 2013-03-25 12:30:27,991 StreamReplyVerbHandler.java (line 50) Need to re-stream file /var/lib/cassandra/data/rt/values/rt-values-ib-1-Data.db to /3.69.211.179 ERROR [Streaming to /3.69.211.179:2] 2013-03-25 12:30:28,113 CassandraDaemon.java (line 164) Exception in thread Thread[Streaming to /3.69.211.179:2,5,main] java.lang.RuntimeException: java.io.EOFException at com.google.common.base.Throwables.propagate(Throwables.java:160) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.io.EOFException at java.io.DataInputStream.readInt(Unknown Source) at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:193) at org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:114) at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ... 3 more Keyspace configuration is as follows: Keyspace: rt: Replication Strategy: org.apache.cassandra.locator.SimpleStrategy Durable Writes: true Options: [replication_factor:3] Column Families: ColumnFamily: tagname Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 0.1 DC Local Read repair chance: 0.0 Populate IO Cache on flush: false Replicate on write: true Caching: KEYS_ONLY Bloom Filter FP chance: default Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: values Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 0.1 DC Local Read repair chance: 0.0 Populate IO Cache on flush: false Replicate on write: true Caching: KEYS_ONLY Bloom Filter FP chance: default Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
