[jira] [Updated] (CASSANDRA-12386) Exception in thread Thread[MemtableFlushWriter:33,5,main]
[ https://issues.apache.org/jira/browse/CASSANDRA-12386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vin01 updated CASSANDRA-12386: -- Description: In logs i got an exception which looks like :- ERROR [MemtableFlushWriter:33] 2016-08-04 19:40:17,712 CassandraDaemon.java:201 - Exception in thread Thread[MemtableFlushWriter:33,5,main] java.lang.RuntimeException: Last written key DecoratedKey(076710082, 303736373130303832) >= current key DecoratedKey(07671^@^@^@^@^@^@, 3037363731313232333534) writing into /data/cassandra/data//-01ad9750723e11e4bfe0d3887930a87c/./mb-23389-big-Data.db at org.apache.cassandra.io.sstable.format.big.BigTableWriter.beforeAppend(BigTableWriter.java:106) ~[apache-cassandra-3.0.7.jar:3.0.7] at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:145) ~[apache-cassandra-3.0.7.jar:3.0.7] at org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:45) ~[apache-cassandra-3.0.7.jar:3.0.7] at org.apache.cassandra.io.sstable.SSTableTxnWriter.append(SSTableTxnWriter.java:52) ~[apache-cassandra-3.0.7.jar:3.0.7] at org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:379) ~[apache-cassandra-3.0.7.jar:3.0.7] at org.apache.cassandra.db.Memtable.flush(Memtable.java:326) ~[apache-cassandra-3.0.7.jar:3.0.7] at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1071) ~[apache-cassandra-3.0.7.jar:3.0.7] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_91] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_91] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_91] It seems similar to https://issues.apache.org/jira/browse/CASSANDRA-9126 but seems a bit different. Please review. Thanks. was: In logs i got an exception which looks like :- ERROR [MemtableFlushWriter:33] 2016-08-04 19:40:17,712 CassandraDaemon.java:201 - Exception in thread Thread[MemtableFlushWriter:33,5,main] java.lang.RuntimeException: Last written key DecoratedKey(076710082, 303736373130303832) >= current key DecoratedKey(07671^@^@^@^@^@^@, 3037363731313232333534) writing into /data/cassandra/data/eviveportal/memberinfo-01ad9750723e11e4bfe0d3887930a87c/.healthy_living_id_index/mb-23389-big-Data.db at org.apache.cassandra.io.sstable.format.big.BigTableWriter.beforeAppend(BigTableWriter.java:106) ~[apache-cassandra-3.0.7.jar:3.0.7] at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:145) ~[apache-cassandra-3.0.7.jar:3.0.7] at org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:45) ~[apache-cassandra-3.0.7.jar:3.0.7] at org.apache.cassandra.io.sstable.SSTableTxnWriter.append(SSTableTxnWriter.java:52) ~[apache-cassandra-3.0.7.jar:3.0.7] at org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:379) ~[apache-cassandra-3.0.7.jar:3.0.7] at org.apache.cassandra.db.Memtable.flush(Memtable.java:326) ~[apache-cassandra-3.0.7.jar:3.0.7] at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1071) ~[apache-cassandra-3.0.7.jar:3.0.7] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_91] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_91] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_91] It seems similar to https://issues.apache.org/jira/browse/CASSANDRA-9126 but seems a bit different. Please review. Thanks. > Exception in thread Thread[MemtableFlushWriter:33,5,main] > - > > Key: CASSANDRA-12386 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12386 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: CentOS 6.8 x86_64, Cassandra 3.0.7 >Reporter: vin01 >Priority: Minor > > In logs i got an exception which looks like :- > ERROR [MemtableFlushWriter:33] 2016-08-04 19:40:17,712 > CassandraDaemon.java:201 - Exception in thread > Thread[MemtableFlushWriter:33,5,main] > java.lang.RuntimeException: Last written key DecoratedKey(076710082, > 303736373130303832) >= current key DecoratedKey(07671^@^@^@^@^@^@, > 3037363731313232333534) writing into > /data/cassandra/data//-01ad9750723e11e4bfe0d3887930a87c/./mb-23389-big-Data.db > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter.beforeAppend(BigTableWriter.java:106) > ~[apache-cassandra-3.0.7.jar:3.0.7] > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:145) >
[jira] [Created] (CASSANDRA-12386) Exception in thread Thread[MemtableFlushWriter:33,5,main]
vin01 created CASSANDRA-12386: - Summary: Exception in thread Thread[MemtableFlushWriter:33,5,main] Key: CASSANDRA-12386 URL: https://issues.apache.org/jira/browse/CASSANDRA-12386 Project: Cassandra Issue Type: Bug Components: Core Environment: CentOS 6.8 x86_64, Cassandra 3.0.7 Reporter: vin01 Priority: Minor In logs i got an exception which looks like :- ERROR [MemtableFlushWriter:33] 2016-08-04 19:40:17,712 CassandraDaemon.java:201 - Exception in thread Thread[MemtableFlushWriter:33,5,main] java.lang.RuntimeException: Last written key DecoratedKey(076710082, 303736373130303832) >= current key DecoratedKey(07671^@^@^@^@^@^@, 3037363731313232333534) writing into /data/cassandra/data/eviveportal/memberinfo-01ad9750723e11e4bfe0d3887930a87c/.healthy_living_id_index/mb-23389-big-Data.db at org.apache.cassandra.io.sstable.format.big.BigTableWriter.beforeAppend(BigTableWriter.java:106) ~[apache-cassandra-3.0.7.jar:3.0.7] at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:145) ~[apache-cassandra-3.0.7.jar:3.0.7] at org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:45) ~[apache-cassandra-3.0.7.jar:3.0.7] at org.apache.cassandra.io.sstable.SSTableTxnWriter.append(SSTableTxnWriter.java:52) ~[apache-cassandra-3.0.7.jar:3.0.7] at org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:379) ~[apache-cassandra-3.0.7.jar:3.0.7] at org.apache.cassandra.db.Memtable.flush(Memtable.java:326) ~[apache-cassandra-3.0.7.jar:3.0.7] at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1071) ~[apache-cassandra-3.0.7.jar:3.0.7] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_91] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_91] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_91] It seems similar to https://issues.apache.org/jira/browse/CASSANDRA-9126 but seems a bit different. Please review. Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-12349) Adding some new features to cqlsh
vin01 created CASSANDRA-12349:
-
Summary: Adding some new features to cqlsh
Key: CASSANDRA-12349
URL: https://issues.apache.org/jira/browse/CASSANDRA-12349
Project: Cassandra
Issue Type: New Feature
Environment: All
Reporter: vin01
Priority: Minor
I will like to have following features in in cqlsh, I have made a patch to
enable them as well.
1. Aliases.
2. Safe mode (prompt on delete,update,truncate,drop if safe_mode is true).
3. Press q to exit.
Its also shared here ->
https://github.com/vineet01/cassandra/blob/trunk/new_features.txt
Example for aliases :-
cassandra@cqlsh> show
; ALIASES HOST SESSION VERSION
cassandra@cqlsh> show ALIASES ;
Aliases :> {'dk': 'desc keyspaces;', 'sl': 'select * from'}
now if you type dk and press it will auto complete it to "desc keyspace".
Adding an alias from shell :-
cassandra@cqlsh> alias slu=select * from login.user ;
Alias added successfully - sle:select * login.user ;
cassandra@cqlsh> show ALIASES ;
Aliases :> {'slu': 'select * from login.user ;', 'dk': 'desc keyspaces;', 'sl':
'select * from'}
cassandra@cqlsh> sle
Expanded alias to> select * from login.user ;
username | blacklisted | lastlogin | password
Adding an alias directly in file :-
aliases will be kept in same cqlshrc file.
[aliases]
dk = desc keyspaces;
sl = select * from
sle = select * from login.user ;
now if we type just "sle" it will autocomplete rest of it and show next options.
Example of safe mode :-
cassandra@cqlsh> truncate login.user ;
Are you sure you want to do this? (y/n) > n
Not performing any action.
cassandra@cqlsh> updatee login.user set password=null;
Are you sure you want to do this? (y/n) >
Not performing any action.
Initial commit :-
https://github.com/vineet01/cassandra/commit/0bfce2ccfc610021a74a1f82ed24aa63e1b72fec
Current version :- https://github.com/vineet01/cassandra/blob/trunk/bin/cqlsh.py
Please review and suggest any improvements.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12221) Maximum Memory Usage Reached - NoSpamLogger
[ https://issues.apache.org/jira/browse/CASSANDRA-12221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381728#comment-15381728 ] vin01 commented on CASSANDRA-12221: --- Thanks. So shouldn't it be a WARN/ERROR? does it cause any issues ? can it be responsible for High CPU Usage? > Maximum Memory Usage Reached - NoSpamLogger > --- > > Key: CASSANDRA-12221 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12221 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: CentOS 6.8 x86_64, Cassandra-3.0.7 >Reporter: vin01 >Priority: Minor > > I see some logs which look like :- > INFO [SharedPool-Worker-22] 2016-07-17 22:03:02,725 NoSpamLogger.java:91 - > Maximum memory usage reached (536870912 bytes), cannot allocate chunk of > 1048576 bytes > INFO [SharedPool-Worker-33] 2016-07-17 22:18:02,747 NoSpamLogger.java:91 - > Maximum memory usage reached (536870912 bytes), cannot allocate chunk of > 1048576 bytes > INFO [SharedPool-Worker-35] 2016-07-17 22:33:02,829 NoSpamLogger.java:91 - > Maximum memory usage reached (536870912 bytes), cannot allocate chunk of > 1048576 bytes > INFO [SharedPool-Worker-31] 2016-07-17 22:48:02,834 NoSpamLogger.java:91 - > Maximum memory usage reached (536870912 bytes), cannot allocate chunk of > 1048576 bytes > When i get these logs, CPU usage is quite high on the system. > Are they expected? Log message also seems confusing and i am not sure what > memory we are talking about here.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12221) Maximum Memory Usage Reached - NoSpamLogger
[ https://issues.apache.org/jira/browse/CASSANDRA-12221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vin01 updated CASSANDRA-12221: -- Description: I see some logs which look like :- INFO [SharedPool-Worker-22] 2016-07-17 22:03:02,725 NoSpamLogger.java:91 - Maximum memory usage reached (536870912 bytes), cannot allocate chunk of 1048576 bytes INFO [SharedPool-Worker-33] 2016-07-17 22:18:02,747 NoSpamLogger.java:91 - Maximum memory usage reached (536870912 bytes), cannot allocate chunk of 1048576 bytes INFO [SharedPool-Worker-35] 2016-07-17 22:33:02,829 NoSpamLogger.java:91 - Maximum memory usage reached (536870912 bytes), cannot allocate chunk of 1048576 bytes INFO [SharedPool-Worker-31] 2016-07-17 22:48:02,834 NoSpamLogger.java:91 - Maximum memory usage reached (536870912 bytes), cannot allocate chunk of 1048576 bytes When i get these logs, CPU usage is quite high on the system. Are they expected? Log message also seems confusing and i am not sure what memory we are talking about here.. was: I see some logs which look like :- INFO [SharedPool-Worker-22] 2016-07-17 22:03:02,725 NoSpamLogger.java:91 - Maximum memory usage reached (536870912 bytes), cannot allocate chunk of 1048576 bytes INFO [SharedPool-Worker-33] 2016-07-17 22:18:02,747 NoSpamLogger.java:91 - Maximum memory usage reached (536870912 bytes), cannot allocate chunk of 1048576 bytes INFO [SharedPool-Worker-35] 2016-07-17 22:33:02,829 NoSpamLogger.java:91 - Maximum memory usage reached (536870912 bytes), cannot allocate chunk of 1048576 bytes INFO [SharedPool-Worker-31] 2016-07-17 22:48:02,834 NoSpamLogger.java:91 - Maximum memory usage reached (536870912 bytes), cannot allocate chunk of 1048576 bytes When i see these logs, CPU usage is quite high on the system. Are they expected? Log message also seems confusing and i am not sure what memory we are talking about here.. > Maximum Memory Usage Reached - NoSpamLogger > --- > > Key: CASSANDRA-12221 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12221 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: CentOS 6.8 x86_64, Cassandra-3.0.7 >Reporter: vin01 >Priority: Minor > > I see some logs which look like :- > INFO [SharedPool-Worker-22] 2016-07-17 22:03:02,725 NoSpamLogger.java:91 - > Maximum memory usage reached (536870912 bytes), cannot allocate chunk of > 1048576 bytes > INFO [SharedPool-Worker-33] 2016-07-17 22:18:02,747 NoSpamLogger.java:91 - > Maximum memory usage reached (536870912 bytes), cannot allocate chunk of > 1048576 bytes > INFO [SharedPool-Worker-35] 2016-07-17 22:33:02,829 NoSpamLogger.java:91 - > Maximum memory usage reached (536870912 bytes), cannot allocate chunk of > 1048576 bytes > INFO [SharedPool-Worker-31] 2016-07-17 22:48:02,834 NoSpamLogger.java:91 - > Maximum memory usage reached (536870912 bytes), cannot allocate chunk of > 1048576 bytes > When i get these logs, CPU usage is quite high on the system. > Are they expected? Log message also seems confusing and i am not sure what > memory we are talking about here.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-12221) Maximum Memory Usage Reached - NoSpamLogger
vin01 created CASSANDRA-12221: - Summary: Maximum Memory Usage Reached - NoSpamLogger Key: CASSANDRA-12221 URL: https://issues.apache.org/jira/browse/CASSANDRA-12221 Project: Cassandra Issue Type: Bug Components: Core Environment: CentOS 6.8 x86_64, Cassandra-3.0.7 Reporter: vin01 Priority: Minor I see some logs which look like :- INFO [SharedPool-Worker-22] 2016-07-17 22:03:02,725 NoSpamLogger.java:91 - Maximum memory usage reached (536870912 bytes), cannot allocate chunk of 1048576 bytes INFO [SharedPool-Worker-33] 2016-07-17 22:18:02,747 NoSpamLogger.java:91 - Maximum memory usage reached (536870912 bytes), cannot allocate chunk of 1048576 bytes INFO [SharedPool-Worker-35] 2016-07-17 22:33:02,829 NoSpamLogger.java:91 - Maximum memory usage reached (536870912 bytes), cannot allocate chunk of 1048576 bytes INFO [SharedPool-Worker-31] 2016-07-17 22:48:02,834 NoSpamLogger.java:91 - Maximum memory usage reached (536870912 bytes), cannot allocate chunk of 1048576 bytes When i see these logs, CPU usage is quite high on the system. Are they expected? Log message also seems confusing and i am not sure what memory we are talking about here.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-12156) java.lang.ClassCastException During Write Operations
[ https://issues.apache.org/jira/browse/CASSANDRA-12156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369023#comment-15369023 ] vin01 edited comment on CASSANDRA-12156 at 7/9/16 9:14 AM: --- And followin this there were more errors like :- ERROR [SharedPool-Worker-14] 2016-07-09 00:09:23,547 Message.java:611 - Unexpected exception during request; channel = [id: 0x7e101236, /ip.add.re.ss:36421 => $ java.lang.ClassCastException: null no stack-trace for these was there. was (Author: vin01): And followin this there were more errors like :- ERROR [SharedPool-Worker-14] 2016-07-09 00:09:23,547 Message.java:611 - Unexpected exception during request; channel = [id: 0x7e101236, /192.168.100.91:36421 => $ java.lang.ClassCastException: null no stack-trace for these was there. > java.lang.ClassCastException During Write Operations > > > Key: CASSANDRA-12156 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12156 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Centos 6.7, JDK1.8.0_72, Cassandra 3.0.7 >Reporter: vin01 >Priority: Minor > > During a regular ETL process today suddenly i am getting some errors from > cassandra which look like :- > ERROR [SharedPool-Worker-28] 2016-07-09 00:07:04,062 Message.java:611 - > Unexpected exception during request; channel = [id: 0x7e101236, > /ip.add.re.ss:36421 => /ip.add.re.ss:9044] > io.netty.handler.codec.DecoderException: java.lang.ClassCastException: > java.lang.String cannot be cast to [B > at > io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:99) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:971) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:854) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:249) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:149) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:722) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] > Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to [B > at > io.netty.buffer.PooledHeapByteBuf.array(PooledHeapByteBuf.java:280) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > org.apache.cassandra.transport.FrameCompressor$LZ4Compressor.decompress(FrameCompressor.java:191) > ~[apache-cassandra-3.0.7.jar:3.0.7] > at > org.apache.cassandra.transport.Frame$Decompressor.decode(Frame.java:310) >
[jira] [Updated] (CASSANDRA-12156) java.lang.ClassCastException During Write Operations
[ https://issues.apache.org/jira/browse/CASSANDRA-12156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vin01 updated CASSANDRA-12156: -- Priority: Minor (was: Major) > java.lang.ClassCastException During Write Operations > > > Key: CASSANDRA-12156 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12156 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Centos 6.7, JDK1.8.0_72, Cassandra 3.0.7 >Reporter: vin01 >Priority: Minor > > During a regular ETL process today suddenly i am getting some errors from > cassandra which look like :- > ERROR [SharedPool-Worker-28] 2016-07-09 00:07:04,062 Message.java:611 - > Unexpected exception during request; channel = [id: 0x7e101236, > /ip.add.re.ss:36421 => /ip.add.re.ss:9044] > io.netty.handler.codec.DecoderException: java.lang.ClassCastException: > java.lang.String cannot be cast to [B > at > io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:99) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:971) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:854) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:249) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:149) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:722) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] > Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to [B > at > io.netty.buffer.PooledHeapByteBuf.array(PooledHeapByteBuf.java:280) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > org.apache.cassandra.transport.FrameCompressor$LZ4Compressor.decompress(FrameCompressor.java:191) > ~[apache-cassandra-3.0.7.jar:3.0.7] > at > org.apache.cassandra.transport.Frame$Decompressor.decode(Frame.java:310) > ~[apache-cassandra-3.0.7.jar:3.0.7] > at > org.apache.cassandra.transport.Frame$Decompressor.decode(Frame.java:289) > ~[apache-cassandra-3.0.7.jar:3.0.7] > at > io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > ... 18 common frames omitted > it didn't affect the application though but i haven't seen it before and > seems like something is wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12156) java.lang.ClassCastException During Write Operations
[ https://issues.apache.org/jira/browse/CASSANDRA-12156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369023#comment-15369023 ] vin01 commented on CASSANDRA-12156: --- And followin this there were more errors like :- ERROR [SharedPool-Worker-14] 2016-07-09 00:09:23,547 Message.java:611 - Unexpected exception during request; channel = [id: 0x7e101236, /192.168.100.91:36421 => $ java.lang.ClassCastException: null no stack-trace for these was there. > java.lang.ClassCastException During Write Operations > > > Key: CASSANDRA-12156 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12156 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Centos 6.7, JDK1.8.0_72, Cassandra 3.0.7 >Reporter: vin01 > > During a regular ETL process today suddenly i am getting some errors from > cassandra which look like :- > ERROR [SharedPool-Worker-28] 2016-07-09 00:07:04,062 Message.java:611 - > Unexpected exception during request; channel = [id: 0x7e101236, > /ip.add.re.ss:36421 => /ip.add.re.ss:9044] > io.netty.handler.codec.DecoderException: java.lang.ClassCastException: > java.lang.String cannot be cast to [B > at > io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:99) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:971) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:854) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:249) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:149) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:722) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] > Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to [B > at > io.netty.buffer.PooledHeapByteBuf.array(PooledHeapByteBuf.java:280) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > org.apache.cassandra.transport.FrameCompressor$LZ4Compressor.decompress(FrameCompressor.java:191) > ~[apache-cassandra-3.0.7.jar:3.0.7] > at > org.apache.cassandra.transport.Frame$Decompressor.decode(Frame.java:310) > ~[apache-cassandra-3.0.7.jar:3.0.7] > at > org.apache.cassandra.transport.Frame$Decompressor.decode(Frame.java:289) > ~[apache-cassandra-3.0.7.jar:3.0.7] > at > io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > ... 18 common frames omitted > it didn't affect the application
[jira] [Created] (CASSANDRA-12156) java.lang.ClassCastException During Write Operations
vin01 created CASSANDRA-12156: - Summary: java.lang.ClassCastException During Write Operations Key: CASSANDRA-12156 URL: https://issues.apache.org/jira/browse/CASSANDRA-12156 Project: Cassandra Issue Type: Bug Components: Core Environment: Centos 6.7, JDK1.8.0_72, Cassandra 3.0.7 Reporter: vin01 During a regular ETL process today suddenly i am getting some errors from cassandra which look like :- ERROR [SharedPool-Worker-28] 2016-07-09 00:07:04,062 Message.java:611 - Unexpected exception during request; channel = [id: 0x7e101236, /ip.add.re.ss:36421 => /ip.add.re.ss:9044] io.netty.handler.codec.DecoderException: java.lang.ClassCastException: java.lang.String cannot be cast to [B at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:99) ~[netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) ~[netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) ~[netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) ~[netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) ~[netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) ~[netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:971) ~[netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:854) ~[netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:249) ~[netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:149) ~[netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) ~[netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) ~[netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) ~[netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:722) ~[netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326) ~[netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264) ~[netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) ~[netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) ~[netty-all-4.0.23.Final.jar:4.0.23.Final] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to [B at io.netty.buffer.PooledHeapByteBuf.array(PooledHeapByteBuf.java:280) ~[netty-all-4.0.23.Final.jar:4.0.23.Final] at org.apache.cassandra.transport.FrameCompressor$LZ4Compressor.decompress(FrameCompressor.java:191) ~[apache-cassandra-3.0.7.jar:3.0.7] at org.apache.cassandra.transport.Frame$Decompressor.decode(Frame.java:310) ~[apache-cassandra-3.0.7.jar:3.0.7] at org.apache.cassandra.transport.Frame$Decompressor.decode(Frame.java:289) ~[apache-cassandra-3.0.7.jar:3.0.7] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) ~[netty-all-4.0.23.Final.jar:4.0.23.Final] ... 18 common frames omitted it didn't affect the application though but i haven't seen it before and seems like something is wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12031) "LEAK DETECTED" during incremental repairs
[ https://issues.apache.org/jira/browse/CASSANDRA-12031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369016#comment-15369016 ] vin01 commented on CASSANDRA-12031: --- I have moved to 3.0.7 now and haven't seen this yet :) > "LEAK DETECTED" during incremental repairs > -- > > Key: CASSANDRA-12031 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12031 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging > Environment: Centos 6.6, x86_64, Cassandra 2.2.4 >Reporter: vin01 >Priority: Minor > > I encountered some errors during an incremental repair session which look > like :- > ERROR [Reference-Reaper:1] 2016-06-19 03:28:35,884 Ref.java:187 - LEAK > DETECTED: a reference > (org.apache.cassandra.utils.concurrent.Ref$State@2ce0fab3) to class > org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1513857473:Memory@[7f2d462191f0..7f2d46219510) > was not released before the reference was garbage collected > Should i be worried about these? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-11845) Hanging repair in cassandra 2.2.4
[ https://issues.apache.org/jira/browse/CASSANDRA-11845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vin01 updated CASSANDRA-11845: -- Description: So after increasing the streaming_timeout_in_ms value to 3 hours, i was able to avoid the socketTimeout errors i was getting earlier (https://issues.apAache.org/jira/browse/CASSANDRA-11826), but now the issue is repair just stays stuck. current status :- [2016-05-19 05:52:50,835] Repair session a0e590e1-1d99-11e6-9d63-b717b380ffdd for range (-3309358208555432808,-3279958773585646585] finished (progress: 54%) [2016-05-19 05:53:09,446] Repair session a0e590e3-1d99-11e6-9d63-b717b380ffdd for range (8149151263857514385,8181801084802729407] finished (progress: 55%) [2016-05-19 05:53:13,808] Repair session a0e5b7f1-1d99-11e6-9d63-b717b380ffdd for range (3372779397996730299,3381236471688156773] finished (progress: 55%) [2016-05-19 05:53:27,543] Repair session a0e5b7f3-1d99-11e6-9d63-b717b380ffdd for range (-4182952858113330342,-4157904914928848809] finished (progress: 55%) [2016-05-19 05:53:41,128] Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd for range (6499366179019889198,6523760493740195344] finished (progress: 55%) And its 10:46:25 Now, almost 5 hours since it has been stuck right there. Earlier i could see repair session going on in system.log but there are no logs coming in right now, all i get in logs is regular index summary redistribution logs. Last logs for repair i saw in logs :- INFO [RepairJobTask:5] 2016-05-19 05:53:41,125 RepairJob.java:152 - [repair #a0e5df00-1d99-11e6-9d63-b717b380ffdd] TABLE_NAME is fully synced INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairSession.java:279 - [repair #a0e5df00-1d99-11e6-9d63-b717b380ffdd] Session completed successfully INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairRunnable.java:232 - Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd for range (6499366179019889198,6523760493740195344] finished Its an incremental repair, and in "nodetool netstats" output i can see logs like :- Repair e3055fb0-1d9d-11e6-9d63-b717b380ffdd /Node-2 Receiving 8 files, 1093461 bytes total. Already received 8 files, 1093461 bytes total /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80872-big-Data.db 399475/399475 bytes(100%) received from idx:0/Node-2 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80879-big-Data.db 53809/53809 bytes(100%) received from idx:0/Node-2 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80878-big-Data.db 89955/89955 bytes(100%) received from idx:0/Node-2 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80881-big-Data.db 168790/168790 bytes(100%) received from idx:0/Node-2 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80886-big-Data.db 107785/107785 bytes(100%) received from idx:0/Node-2 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80880-big-Data.db 52889/52889 bytes(100%) received from idx:0/Node-2 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80884-big-Data.db 148882/148882 bytes(100%) received from idx:0/Node-2 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80883-big-Data.db 71876/71876 bytes(100%) received from idx:0/Node-2 Sending 5 files, 863321 bytes total. Already sent 5 files, 863321 bytes total /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73168-big-Data.db 161895/161895 bytes(100%) sent to idx:0/Node-2 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-72604-big-Data.db 399865/399865 bytes(100%) sent to idx:0/Node-2 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73147-big-Data.db 149066/149066 bytes(100%) sent to idx:0/Node-2 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-72682-big-Data.db 126000/126000 bytes(100%) sent to idx:0/Node-2 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73173-big-Data.db 26495/26495 bytes(100%) sent to idx:0/Node-2 Repair c0c8af20-1d9c-11e6-9d63-b717b380ffdd /Node-3 Receiving 11 files, 13896288 bytes total. Already received 11 files, 13896288 bytes total /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79186-big-Data.db 1598874/1598874 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79196-big-Data.db 736365/736365 bytes(100%) received
[jira] [Commented] (CASSANDRA-11845) Hanging repair in cassandra 2.2.4
[ https://issues.apache.org/jira/browse/CASSANDRA-11845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344344#comment-15344344 ] vin01 commented on CASSANDRA-11845: --- It worked !! So tuning of following three things made it work :- 1. increasing streaming_timeout_in_ms value to 6 hours. 2. sudo sysctl -w net.ipv4.tcp_keepalive_time=60 net.ipv4.tcp_keepalive_probes=3 net.ipv4.tcp_keepalive_intvl=10 (as per https://docs.datastax.com/en/cassandra/2.2/cassandra/troubleshooting/trblshootIdleFirewall.html) 3. Increasing value of vm.max_map_count (https://docs.datastax.com/en/cassandra/2.2/cassandra/install/installRecommendSettings.html.) Thanks again Paulo! > Hanging repair in cassandra 2.2.4 > - > > Key: CASSANDRA-11845 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11845 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging > Environment: Centos 6 >Reporter: vin01 >Priority: Minor > Attachments: cassandra-2.2.4.error.log, system.log > > > So after increasing the streaming_timeout_in_ms value to 3 hours, i was able > to avoid the socketTimeout errors i was getting earlier > (https://issues.apAache.org/jira/browse/CASSANDRA-11826), but now the issue > is repair just stays stuck. > current status :- > [2016-05-19 05:52:50,835] Repair session a0e590e1-1d99-11e6-9d63-b717b380ffdd > for range (-3309358208555432808,-3279958773585646585] finished (progress: 54%) > [2016-05-19 05:53:09,446] Repair session a0e590e3-1d99-11e6-9d63-b717b380ffdd > for range (8149151263857514385,8181801084802729407] finished (progress: 55%) > [2016-05-19 05:53:13,808] Repair session a0e5b7f1-1d99-11e6-9d63-b717b380ffdd > for range (3372779397996730299,3381236471688156773] finished (progress: 55%) > [2016-05-19 05:53:27,543] Repair session a0e5b7f3-1d99-11e6-9d63-b717b380ffdd > for range (-4182952858113330342,-4157904914928848809] finished (progress: 55%) > [2016-05-19 05:53:41,128] Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd > for range (6499366179019889198,6523760493740195344] finished (progress: 55%) > And its 10:46:25 Now, almost 5 hours since it has been stuck right there. > Earlier i could see repair session going on in system.log but there are no > logs coming in right now, all i get in logs is regular index summary > redistribution logs. > Last logs for repair i saw in logs :- > INFO [RepairJobTask:5] 2016-05-19 05:53:41,125 RepairJob.java:152 - [repair > #a0e5df00-1d99-11e6-9d63-b717b380ffdd] TABLE_NAME is fully synced > INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairSession.java:279 - > [repair #a0e5df00-1d99-11e6-9d63-b717b380ffdd] Session completed successfully > INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairRunnable.java:232 - > Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd for range > (6499366179019889198,6523760493740195344] finished > Its an incremental repair, and in "nodetool netstats" output i can see logs > like :- > Repair e3055fb0-1d9d-11e6-9d63-b717b380ffdd > /Node-2 > Receiving 8 files, 1093461 bytes total. Already received 8 files, > 1093461 bytes total > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80872-big-Data.db > 399475/399475 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80879-big-Data.db > 53809/53809 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80878-big-Data.db > 89955/89955 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80881-big-Data.db > 168790/168790 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80886-big-Data.db > 107785/107785 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80880-big-Data.db > 52889/52889 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80884-big-Data.db > 148882/148882 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80883-big-Data.db > 71876/71876 bytes(100%) received from idx:0/Node-2 > Sending 5 files, 863321 bytes total. Already sent 5 files, 863321 > bytes total > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73168-big-Data.db > 161895/161895 bytes(100%) sent to idx:0/Node-2 > >
[jira] [Commented] (CASSANDRA-11845) Hanging repair in cassandra 2.2.4
[ https://issues.apache.org/jira/browse/CASSANDRA-11845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342169#comment-15342169 ] vin01 commented on CASSANDRA-11845: --- Thanks a lot Paulo! I am going to try it. Present value of vm.max_map_count is "65530" which is default I believe. I am going to increase it to "131072" as recommended by https://docs.datastax.com/en/cassandra/2.2/cassandra/install/installRecommendSettings.html. > Hanging repair in cassandra 2.2.4 > - > > Key: CASSANDRA-11845 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11845 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging > Environment: Centos 6 >Reporter: vin01 >Priority: Minor > Attachments: cassandra-2.2.4.error.log, system.log > > > So after increasing the streaming_timeout_in_ms value to 3 hours, i was able > to avoid the socketTimeout errors i was getting earlier > (https://issues.apAache.org/jira/browse/CASSANDRA-11826), but now the issue > is repair just stays stuck. > current status :- > [2016-05-19 05:52:50,835] Repair session a0e590e1-1d99-11e6-9d63-b717b380ffdd > for range (-3309358208555432808,-3279958773585646585] finished (progress: 54%) > [2016-05-19 05:53:09,446] Repair session a0e590e3-1d99-11e6-9d63-b717b380ffdd > for range (8149151263857514385,8181801084802729407] finished (progress: 55%) > [2016-05-19 05:53:13,808] Repair session a0e5b7f1-1d99-11e6-9d63-b717b380ffdd > for range (3372779397996730299,3381236471688156773] finished (progress: 55%) > [2016-05-19 05:53:27,543] Repair session a0e5b7f3-1d99-11e6-9d63-b717b380ffdd > for range (-4182952858113330342,-4157904914928848809] finished (progress: 55%) > [2016-05-19 05:53:41,128] Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd > for range (6499366179019889198,6523760493740195344] finished (progress: 55%) > And its 10:46:25 Now, almost 5 hours since it has been stuck right there. > Earlier i could see repair session going on in system.log but there are no > logs coming in right now, all i get in logs is regular index summary > redistribution logs. > Last logs for repair i saw in logs :- > INFO [RepairJobTask:5] 2016-05-19 05:53:41,125 RepairJob.java:152 - [repair > #a0e5df00-1d99-11e6-9d63-b717b380ffdd] TABLE_NAME is fully synced > INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairSession.java:279 - > [repair #a0e5df00-1d99-11e6-9d63-b717b380ffdd] Session completed successfully > INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairRunnable.java:232 - > Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd for range > (6499366179019889198,6523760493740195344] finished > Its an incremental repair, and in "nodetool netstats" output i can see logs > like :- > Repair e3055fb0-1d9d-11e6-9d63-b717b380ffdd > /Node-2 > Receiving 8 files, 1093461 bytes total. Already received 8 files, > 1093461 bytes total > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80872-big-Data.db > 399475/399475 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80879-big-Data.db > 53809/53809 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80878-big-Data.db > 89955/89955 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80881-big-Data.db > 168790/168790 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80886-big-Data.db > 107785/107785 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80880-big-Data.db > 52889/52889 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80884-big-Data.db > 148882/148882 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80883-big-Data.db > 71876/71876 bytes(100%) received from idx:0/Node-2 > Sending 5 files, 863321 bytes total. Already sent 5 files, 863321 > bytes total > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73168-big-Data.db > 161895/161895 bytes(100%) sent to idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-72604-big-Data.db > 399865/399865 bytes(100%) sent to idx:0/Node-2 > >
[jira] [Updated] (CASSANDRA-11845) Hanging repair in cassandra 2.2.4
[ https://issues.apache.org/jira/browse/CASSANDRA-11845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vin01 updated CASSANDRA-11845: -- Attachment: system.log > Hanging repair in cassandra 2.2.4 > - > > Key: CASSANDRA-11845 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11845 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging > Environment: Centos 6 >Reporter: vin01 >Priority: Minor > Attachments: cassandra-2.2.4.error.log, system.log > > > So after increasing the streaming_timeout_in_ms value to 3 hours, i was able > to avoid the socketTimeout errors i was getting earlier > (https://issues.apAache.org/jira/browse/CASSANDRA-11826), but now the issue > is repair just stays stuck. > current status :- > [2016-05-19 05:52:50,835] Repair session a0e590e1-1d99-11e6-9d63-b717b380ffdd > for range (-3309358208555432808,-3279958773585646585] finished (progress: 54%) > [2016-05-19 05:53:09,446] Repair session a0e590e3-1d99-11e6-9d63-b717b380ffdd > for range (8149151263857514385,8181801084802729407] finished (progress: 55%) > [2016-05-19 05:53:13,808] Repair session a0e5b7f1-1d99-11e6-9d63-b717b380ffdd > for range (3372779397996730299,3381236471688156773] finished (progress: 55%) > [2016-05-19 05:53:27,543] Repair session a0e5b7f3-1d99-11e6-9d63-b717b380ffdd > for range (-4182952858113330342,-4157904914928848809] finished (progress: 55%) > [2016-05-19 05:53:41,128] Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd > for range (6499366179019889198,6523760493740195344] finished (progress: 55%) > And its 10:46:25 Now, almost 5 hours since it has been stuck right there. > Earlier i could see repair session going on in system.log but there are no > logs coming in right now, all i get in logs is regular index summary > redistribution logs. > Last logs for repair i saw in logs :- > INFO [RepairJobTask:5] 2016-05-19 05:53:41,125 RepairJob.java:152 - [repair > #a0e5df00-1d99-11e6-9d63-b717b380ffdd] TABLE_NAME is fully synced > INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairSession.java:279 - > [repair #a0e5df00-1d99-11e6-9d63-b717b380ffdd] Session completed successfully > INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairRunnable.java:232 - > Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd for range > (6499366179019889198,6523760493740195344] finished > Its an incremental repair, and in "nodetool netstats" output i can see logs > like :- > Repair e3055fb0-1d9d-11e6-9d63-b717b380ffdd > /Node-2 > Receiving 8 files, 1093461 bytes total. Already received 8 files, > 1093461 bytes total > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80872-big-Data.db > 399475/399475 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80879-big-Data.db > 53809/53809 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80878-big-Data.db > 89955/89955 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80881-big-Data.db > 168790/168790 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80886-big-Data.db > 107785/107785 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80880-big-Data.db > 52889/52889 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80884-big-Data.db > 148882/148882 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80883-big-Data.db > 71876/71876 bytes(100%) received from idx:0/Node-2 > Sending 5 files, 863321 bytes total. Already sent 5 files, 863321 > bytes total > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73168-big-Data.db > 161895/161895 bytes(100%) sent to idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-72604-big-Data.db > 399865/399865 bytes(100%) sent to idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73147-big-Data.db > 149066/149066 bytes(100%) sent to idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-72682-big-Data.db > 126000/126000 bytes(100%) sent to idx:0/Node-2 > >
[jira] [Commented] (CASSANDRA-11845) Hanging repair in cassandra 2.2.4
[ https://issues.apache.org/jira/browse/CASSANDRA-11845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341484#comment-15341484 ] vin01 commented on CASSANDRA-11845: --- Thanks Paulo, that solved the network issue but i got another one this time, same setup, incremental repair. [2016-06-21 04:51:30,716] Repair session c7204063-3780-11e6-8610-b717b380ffdd for range (5170031145794801425,5184266996546342699] finished (progress: 29%) Exception occurred during clean-up. java.lang.reflect.UndeclaredThrowableException error: [2016-06-21 04:52:06,659] JMX connection closed. You should check server log for repair status of keyspace KEYSPACE_NAME(Subsequent keyspaces are not going to be repaired). -- StackTrace -- java.io.IOException: [2016-06-21 04:52:06,659] JMX connection closed. You should check server log for repair status of keyspace KEYSPACE_NAME(Subsequent keyspaces are not going to be repaired). at org.apache.cassandra.tools.RepairRunner.handleConnectionFailed(RepairRunner.java:97) at org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:86) at javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:275) at javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:352) at javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:337) at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:248) at javax.management.remote.rmi.RMIConnector.sendNotification(RMIConnector.java:441) at javax.management.remote.rmi.RMIConnector.access$1200(RMIConnector.java:121) at javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1531) at javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1352) at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchOneNotif(ClientNotifForwarder.java:655) at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:607) at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:471) at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452) at com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108) I have attached more logs which have some errors like -> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed And ERROR [StreamReceiveTask:154] 2016-06-21 04:51:58,193 JVMStabilityInspector.java:117 - JVM state determined to be unstable. Exiting forcefully due to: java.lang.OutOfMemoryError: Map failed > Hanging repair in cassandra 2.2.4 > - > > Key: CASSANDRA-11845 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11845 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging > Environment: Centos 6 >Reporter: vin01 >Priority: Minor > Attachments: cassandra-2.2.4.error.log, system.log > > > So after increasing the streaming_timeout_in_ms value to 3 hours, i was able > to avoid the socketTimeout errors i was getting earlier > (https://issues.apAache.org/jira/browse/CASSANDRA-11826), but now the issue > is repair just stays stuck. > current status :- > [2016-05-19 05:52:50,835] Repair session a0e590e1-1d99-11e6-9d63-b717b380ffdd > for range (-3309358208555432808,-3279958773585646585] finished (progress: 54%) > [2016-05-19 05:53:09,446] Repair session a0e590e3-1d99-11e6-9d63-b717b380ffdd > for range (8149151263857514385,8181801084802729407] finished (progress: 55%) > [2016-05-19 05:53:13,808] Repair session a0e5b7f1-1d99-11e6-9d63-b717b380ffdd > for range (3372779397996730299,3381236471688156773] finished (progress: 55%) > [2016-05-19 05:53:27,543] Repair session a0e5b7f3-1d99-11e6-9d63-b717b380ffdd > for range (-4182952858113330342,-4157904914928848809] finished (progress: 55%) > [2016-05-19 05:53:41,128] Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd > for range (6499366179019889198,6523760493740195344] finished (progress: 55%) > And its 10:46:25 Now, almost 5 hours since it has been stuck right there. > Earlier i could see repair session going on in system.log but there are no > logs coming in right now, all i get in logs is regular index summary > redistribution logs. > Last logs for repair i saw in logs :- > INFO [RepairJobTask:5] 2016-05-19 05:53:41,125 RepairJob.java:152 - [repair > #a0e5df00-1d99-11e6-9d63-b717b380ffdd] TABLE_NAME is fully synced > INFO [RepairJobTask:5] 2016-05-19
[jira] [Comment Edited] (CASSANDRA-11845) Hanging repair in cassandra 2.2.4
[ https://issues.apache.org/jira/browse/CASSANDRA-11845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338535#comment-15338535 ] vin01 edited comment on CASSANDRA-11845 at 6/19/16 2:31 PM: After gettint the exception :- ERROR [STREAM-OUT-/NODE_IN_DC_1] 2016-06-19 08:36:10,187 StreamSession.java:524 - [Stream #80b94bf0-3611-11e6-a89a-87602fd2948b] Streaming error occurred java.net.SocketException: Connection reset at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113) ~[na:1.8.0_72] at java.net.SocketOutputStream.write(SocketOutputStream.java:153) ~[na:1.8.0_72] I went on to check network logs around that time. At ASA firewall between DCs i can see lot of deny messages for some packets :- %ASA-6-106015: Deny TCP (no connection) from [NODE_IN_DC_2]/7003 to [NODE_IN_DC_1]/45573 flags ACK on interface inside I think that's the reason for failure. That deny message basically indicates an idle timeout, which lead to an ACK to be sent after connection was already removed from connection pool by firewall. Does cassandra has something to handle such cases? some retry kind of mechanism? was (Author: vin01): At ASA firewall between DCs i can see lot of deny messages for some packets :- %ASA-6-106015: Deny TCP (no connection) from [NODE_IN_DC_2]/7003 to [NODE_IN_DC_1]/45573 flags ACK on interface inside I think that's the reason for failure. That deny message basically indicates an idle timeout, which lead to an ACK to be sent after connection was already removed from connection pool by firewall. Does cassandra has something to handle such cases? some retry kind of mechanism? > Hanging repair in cassandra 2.2.4 > - > > Key: CASSANDRA-11845 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11845 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging > Environment: Centos 6 >Reporter: vin01 >Priority: Minor > Attachments: cassandra-2.2.4.error.log > > > So after increasing the streaming_timeout_in_ms value to 3 hours, i was able > to avoid the socketTimeout errors i was getting earlier > (https://issues.apAache.org/jira/browse/CASSANDRA-11826), but now the issue > is repair just stays stuck. > current status :- > [2016-05-19 05:52:50,835] Repair session a0e590e1-1d99-11e6-9d63-b717b380ffdd > for range (-3309358208555432808,-3279958773585646585] finished (progress: 54%) > [2016-05-19 05:53:09,446] Repair session a0e590e3-1d99-11e6-9d63-b717b380ffdd > for range (8149151263857514385,8181801084802729407] finished (progress: 55%) > [2016-05-19 05:53:13,808] Repair session a0e5b7f1-1d99-11e6-9d63-b717b380ffdd > for range (3372779397996730299,3381236471688156773] finished (progress: 55%) > [2016-05-19 05:53:27,543] Repair session a0e5b7f3-1d99-11e6-9d63-b717b380ffdd > for range (-4182952858113330342,-4157904914928848809] finished (progress: 55%) > [2016-05-19 05:53:41,128] Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd > for range (6499366179019889198,6523760493740195344] finished (progress: 55%) > And its 10:46:25 Now, almost 5 hours since it has been stuck right there. > Earlier i could see repair session going on in system.log but there are no > logs coming in right now, all i get in logs is regular index summary > redistribution logs. > Last logs for repair i saw in logs :- > INFO [RepairJobTask:5] 2016-05-19 05:53:41,125 RepairJob.java:152 - [repair > #a0e5df00-1d99-11e6-9d63-b717b380ffdd] TABLE_NAME is fully synced > INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairSession.java:279 - > [repair #a0e5df00-1d99-11e6-9d63-b717b380ffdd] Session completed successfully > INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairRunnable.java:232 - > Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd for range > (6499366179019889198,6523760493740195344] finished > Its an incremental repair, and in "nodetool netstats" output i can see logs > like :- > Repair e3055fb0-1d9d-11e6-9d63-b717b380ffdd > /Node-2 > Receiving 8 files, 1093461 bytes total. Already received 8 files, > 1093461 bytes total > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80872-big-Data.db > 399475/399475 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80879-big-Data.db > 53809/53809 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80878-big-Data.db > 89955/89955 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80881-big-Data.db > 168790/168790 bytes(100%) received from idx:0/Node-2 >
[jira] [Commented] (CASSANDRA-11845) Hanging repair in cassandra 2.2.4
[ https://issues.apache.org/jira/browse/CASSANDRA-11845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338535#comment-15338535 ] vin01 commented on CASSANDRA-11845: --- At ASA firewall between DCs i can see lot of deny messages for some packets :- %ASA-6-106015: Deny TCP (no connection) from [NODE_IN_DC_2]/7003 to [NODE_IN_DC_1]/45573 flags ACK on interface inside I think that's the reason for failure. That deny message basically indicates an idle timeout, which lead to an ACK to be sent after connection was already removed from connection pool by firewall. Does cassandra has something to handle such cases? some retry kind of mechanism? > Hanging repair in cassandra 2.2.4 > - > > Key: CASSANDRA-11845 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11845 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging > Environment: Centos 6 >Reporter: vin01 >Priority: Minor > Attachments: cassandra-2.2.4.error.log > > > So after increasing the streaming_timeout_in_ms value to 3 hours, i was able > to avoid the socketTimeout errors i was getting earlier > (https://issues.apAache.org/jira/browse/CASSANDRA-11826), but now the issue > is repair just stays stuck. > current status :- > [2016-05-19 05:52:50,835] Repair session a0e590e1-1d99-11e6-9d63-b717b380ffdd > for range (-3309358208555432808,-3279958773585646585] finished (progress: 54%) > [2016-05-19 05:53:09,446] Repair session a0e590e3-1d99-11e6-9d63-b717b380ffdd > for range (8149151263857514385,8181801084802729407] finished (progress: 55%) > [2016-05-19 05:53:13,808] Repair session a0e5b7f1-1d99-11e6-9d63-b717b380ffdd > for range (3372779397996730299,3381236471688156773] finished (progress: 55%) > [2016-05-19 05:53:27,543] Repair session a0e5b7f3-1d99-11e6-9d63-b717b380ffdd > for range (-4182952858113330342,-4157904914928848809] finished (progress: 55%) > [2016-05-19 05:53:41,128] Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd > for range (6499366179019889198,6523760493740195344] finished (progress: 55%) > And its 10:46:25 Now, almost 5 hours since it has been stuck right there. > Earlier i could see repair session going on in system.log but there are no > logs coming in right now, all i get in logs is regular index summary > redistribution logs. > Last logs for repair i saw in logs :- > INFO [RepairJobTask:5] 2016-05-19 05:53:41,125 RepairJob.java:152 - [repair > #a0e5df00-1d99-11e6-9d63-b717b380ffdd] TABLE_NAME is fully synced > INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairSession.java:279 - > [repair #a0e5df00-1d99-11e6-9d63-b717b380ffdd] Session completed successfully > INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairRunnable.java:232 - > Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd for range > (6499366179019889198,6523760493740195344] finished > Its an incremental repair, and in "nodetool netstats" output i can see logs > like :- > Repair e3055fb0-1d9d-11e6-9d63-b717b380ffdd > /Node-2 > Receiving 8 files, 1093461 bytes total. Already received 8 files, > 1093461 bytes total > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80872-big-Data.db > 399475/399475 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80879-big-Data.db > 53809/53809 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80878-big-Data.db > 89955/89955 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80881-big-Data.db > 168790/168790 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80886-big-Data.db > 107785/107785 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80880-big-Data.db > 52889/52889 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80884-big-Data.db > 148882/148882 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80883-big-Data.db > 71876/71876 bytes(100%) received from idx:0/Node-2 > Sending 5 files, 863321 bytes total. Already sent 5 files, 863321 > bytes total > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73168-big-Data.db > 161895/161895 bytes(100%) sent to idx:0/Node-2 > >
[jira] [Updated] (CASSANDRA-12031) "LEAK DETECTED" during incremental repairs
[ https://issues.apache.org/jira/browse/CASSANDRA-12031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vin01 updated CASSANDRA-12031: -- Environment: Centos 6.6, x86_64, Cassandra 2.2.4 (was: Centos 6.6, x86_64) > "LEAK DETECTED" during incremental repairs > -- > > Key: CASSANDRA-12031 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12031 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging > Environment: Centos 6.6, x86_64, Cassandra 2.2.4 >Reporter: vin01 >Priority: Minor > > I encountered some errors during an incremental repair session which look > like :- > ERROR [Reference-Reaper:1] 2016-06-19 03:28:35,884 Ref.java:187 - LEAK > DETECTED: a reference > (org.apache.cassandra.utils.concurrent.Ref$State@2ce0fab3) to class > org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1513857473:Memory@[7f2d462191f0..7f2d46219510) > was not released before the reference was garbage collected > Should i be worried about these? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-12031) "LEAK DETECTED" during incremental repairs
vin01 created CASSANDRA-12031: - Summary: "LEAK DETECTED" during incremental repairs Key: CASSANDRA-12031 URL: https://issues.apache.org/jira/browse/CASSANDRA-12031 Project: Cassandra Issue Type: Bug Components: Streaming and Messaging Environment: Centos 6.6, x86_64 Reporter: vin01 Priority: Minor I encountered some errors during an incremental repair session which look like :- ERROR [Reference-Reaper:1] 2016-06-19 03:28:35,884 Ref.java:187 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@2ce0fab3) to class org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1513857473:Memory@[7f2d462191f0..7f2d46219510) was not released before the reference was garbage collected Should i be worried about these? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11845) Hanging repair in cassandra 2.2.4
[ https://issues.apache.org/jira/browse/CASSANDRA-11845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15334083#comment-15334083 ] vin01 commented on CASSANDRA-11845: --- It never succeeded.. I just keep going with "nodetool repair -full -local" to minimize the inconsistency issues. > Hanging repair in cassandra 2.2.4 > - > > Key: CASSANDRA-11845 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11845 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging > Environment: Centos 6 >Reporter: vin01 >Priority: Minor > Attachments: cassandra-2.2.4.error.log > > > So after increasing the streaming_timeout_in_ms value to 3 hours, i was able > to avoid the socketTimeout errors i was getting earlier > (https://issues.apAache.org/jira/browse/CASSANDRA-11826), but now the issue > is repair just stays stuck. > current status :- > [2016-05-19 05:52:50,835] Repair session a0e590e1-1d99-11e6-9d63-b717b380ffdd > for range (-3309358208555432808,-3279958773585646585] finished (progress: 54%) > [2016-05-19 05:53:09,446] Repair session a0e590e3-1d99-11e6-9d63-b717b380ffdd > for range (8149151263857514385,8181801084802729407] finished (progress: 55%) > [2016-05-19 05:53:13,808] Repair session a0e5b7f1-1d99-11e6-9d63-b717b380ffdd > for range (3372779397996730299,3381236471688156773] finished (progress: 55%) > [2016-05-19 05:53:27,543] Repair session a0e5b7f3-1d99-11e6-9d63-b717b380ffdd > for range (-4182952858113330342,-4157904914928848809] finished (progress: 55%) > [2016-05-19 05:53:41,128] Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd > for range (6499366179019889198,6523760493740195344] finished (progress: 55%) > And its 10:46:25 Now, almost 5 hours since it has been stuck right there. > Earlier i could see repair session going on in system.log but there are no > logs coming in right now, all i get in logs is regular index summary > redistribution logs. > Last logs for repair i saw in logs :- > INFO [RepairJobTask:5] 2016-05-19 05:53:41,125 RepairJob.java:152 - [repair > #a0e5df00-1d99-11e6-9d63-b717b380ffdd] TABLE_NAME is fully synced > INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairSession.java:279 - > [repair #a0e5df00-1d99-11e6-9d63-b717b380ffdd] Session completed successfully > INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairRunnable.java:232 - > Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd for range > (6499366179019889198,6523760493740195344] finished > Its an incremental repair, and in "nodetool netstats" output i can see logs > like :- > Repair e3055fb0-1d9d-11e6-9d63-b717b380ffdd > /Node-2 > Receiving 8 files, 1093461 bytes total. Already received 8 files, > 1093461 bytes total > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80872-big-Data.db > 399475/399475 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80879-big-Data.db > 53809/53809 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80878-big-Data.db > 89955/89955 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80881-big-Data.db > 168790/168790 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80886-big-Data.db > 107785/107785 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80880-big-Data.db > 52889/52889 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80884-big-Data.db > 148882/148882 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80883-big-Data.db > 71876/71876 bytes(100%) received from idx:0/Node-2 > Sending 5 files, 863321 bytes total. Already sent 5 files, 863321 > bytes total > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73168-big-Data.db > 161895/161895 bytes(100%) sent to idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-72604-big-Data.db > 399865/399865 bytes(100%) sent to idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73147-big-Data.db > 149066/149066 bytes(100%) sent to idx:0/Node-2 > >
[jira] [Commented] (CASSANDRA-11919) Failure in nodetool decommission
[
https://issues.apache.org/jira/browse/CASSANDRA-11919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15334064#comment-15334064
]
vin01 commented on CASSANDRA-11919:
---
Replication Factor : 2 for DC1 and 1 for DC2.
CREATE KEYSPACE KEYSPACE_NAME WITH replication = {'class':
'NetworkTopologyStrategy', 'DC1': '2', 'DC2': '1'} AND durable_writes = true;
I was able to remove the node with 'removenode' but it left cluster
inconsistent and i had to perform a full repair on all keyspaces to fix that.
> Failure in nodetool decommission
>
>
> Key: CASSANDRA-11919
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11919
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: Centos 6.6 x86_64, Cassandra 2.2.4
>Reporter: vin01
>Priority: Minor
> Fix For: 2.2.x
>
>
> I keep getting an exception while attempting "nodetool decommission".
> {code}
> ERROR [STREAM-IN-/[NODE_ON_WHICH_DECOMMISSION_RUNNING]] 2016-05-29
> 13:08:39,040 StreamSession.java:524 - [Stream
> #b2039080-25c2-11e6-bd92-d71331aaf180] Streaming error occurred
> java.lang.IllegalArgumentException: Unknown type 0
> at
> org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:96)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:57)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> {code}
> Because of these, decommission process is not succeeding.
> Is interrupting the decommission process safe? Seems like i will have to
> retry to make it work.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11919) Failure in nodetool decommission
[ https://issues.apache.org/jira/browse/CASSANDRA-11919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15306070#comment-15306070 ] vin01 commented on CASSANDRA-11919: --- In "nodetool netstats" , when the decommission task reaches towards end, i get :- Sending 0 files, 32652963850 bytes total. Already sent 0 files, 0 bytes total And it keeps on coming without any change. Will "nodetool removenode" help here? > Failure in nodetool decommission > > > Key: CASSANDRA-11919 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11919 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging > Environment: Centos 6.6 x86_64, Cassandra 2.2.4 >Reporter: vin01 >Priority: Minor > > I keep getting an exception while attempting "nodetool decommission". > ERROR [STREAM-IN-/[NODE_ON_WHICH_DECOMMISSION_RUNNING]] 2016-05-29 > 13:08:39,040 StreamSession.java:524 - [Stream > #b2039080-25c2-11e6-bd92-d71331aaf180] Streaming error occurred > java.lang.IllegalArgumentException: Unknown type 0 > at > org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:96) > ~[apache-cassandra-2.2.4.jar:2.2.4] > at > org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:57) > ~[apache-cassandra-2.2.4.jar:2.2.4] > at > org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) > ~[apache-cassandra-2.2.4.jar:2.2.4] > Because of these, decommission process is not succeeding. > Is interrupting the decommission process safe? Seems like i will have to > retry to make it work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-11919) Failure in nodetool decommission
vin01 created CASSANDRA-11919: - Summary: Failure in nodetool decommission Key: CASSANDRA-11919 URL: https://issues.apache.org/jira/browse/CASSANDRA-11919 Project: Cassandra Issue Type: Bug Components: Streaming and Messaging Environment: Centos 6.6 x86_64, Cassandra 2.2.4 Reporter: vin01 Priority: Minor I keep getting an exception while attempting "nodetool decommission". ERROR [STREAM-IN-/[NODE_ON_WHICH_DECOMMISSION_RUNNING]] 2016-05-29 13:08:39,040 StreamSession.java:524 - [Stream #b2039080-25c2-11e6-bd92-d71331aaf180] Streaming error occurred java.lang.IllegalArgumentException: Unknown type 0 at org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:96) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:57) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) ~[apache-cassandra-2.2.4.jar:2.2.4] Because of these, decommission process is not succeeding. Is interrupting the decommission process safe? Seems like i will have to retry to make it work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-11918) NullPointerException in Cleanup Process
vin01 created CASSANDRA-11918: - Summary: NullPointerException in Cleanup Process Key: CASSANDRA-11918 URL: https://issues.apache.org/jira/browse/CASSANDRA-11918 Project: Cassandra Issue Type: Bug Components: Compaction Environment: Cassandra 2.2.4, Centos 6.6 x86_64 Reporter: vin01 Priority: Minor After adding a new node to a 3 node cluster, i started "nodetool cleanup" as recommended. [data-center1] Node1, Node2, Node3 [data-center2] Node4 It finished successfully on one Node4 in the other data-center and took only one minute to finish. On Node3 in data-center1, it threw many exceptions in between :- ERROR [SharedPool-Worker-2] 2016-05-29 00:00:03,571 ErrorMessage.java:336 - Unexpected exception during request java.lang.NullPointerException: null at com.stratio.cassandra.lucene.IndexSearcher.(IndexSearcher.java:77) ~[cassandra-lucene-index-plugin-2.2.4.0.jar:na] at com.stratio.cassandra.lucene.Index.createSecondaryIndexSearcher(Index.java:249) ~[cassandra-lucene-index-plugin-2.2.4.0.jar:na] at org.apache.cassandra.db.index.SecondaryIndexManager.validateIndexSearchersForQuery(SecondaryIndexManager.java:590) ~[apache-cassandra-2.2.4.j ar:2.2.4] at org.apache.cassandra.cql3.statements.SelectStatement.getValidatedIndexExpressions(SelectStatement.java:608) ~[apache-cassandra-2.2.4.jar:2.2. 4] at org.apache.cassandra.cql3.statements.SelectStatement.getRangeCommand(SelectStatement.java:376) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.cql3.statements.SelectStatement.getPageableCommand(SelectStatement.java:186) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:172) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:76) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:226) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:466) ~[apache-cassandra-2.2.4.jar:2.2.4] at com.stratio.cassandra.lucene.IndexQueryHandler.processPrepared(IndexQueryHandler.java:108) ~[cassandra-lucene-index-plugin-2.2.4.0.jar:na] at org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:142) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401) [apache-cassandra-2.2.4.jar:2.2.4] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_72] at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] I am not sure what is causing it and is it really a problem. Also if cleanup process is interrupted, do i need to restart the node? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-11845) Hanging repair in cassandra 2.2.4
[ https://issues.apache.org/jira/browse/CASSANDRA-11845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292934#comment-15292934 ] vin01 edited comment on CASSANDRA-11845 at 5/20/16 8:27 AM: Thanks Paulo, i restarted all 3 nodes and started repair again and got the errors which i have attached. (cassandra-2.2.4.error.log) Nodetool output for repair session :- [2016-05-20 02:37:59,168] Repair session cffbadd3-1e55-11e6-bd05-b717b380ffdd for range (-8184117312116560831,-8171918810495776305] failed with error Endpoint /Node-3 died (progress: 100%) .. still running (compaction is going on) was (Author: vin01): Thanks Paulo, i restarted all 3 nodes and started repair again and got the errors which i have attached. (cassandra-2.2.4.error.log) Nodetool output for repair session :- [2016-05-20 02:37:59,168] Repair session cffbadd3-1e55-11e6-bd05-b717b380ffdd for range (-8184117312116560831,-8171918810495776305] failed with error Endpoint /Node-3 died (progress: 100%) .. still running > Hanging repair in cassandra 2.2.4 > - > > Key: CASSANDRA-11845 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11845 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging > Environment: Centos 6 >Reporter: vin01 >Priority: Minor > Attachments: cassandra-2.2.4.error.log > > > So after increasing the streaming_timeout_in_ms value to 3 hours, i was able > to avoid the socketTimeout errors i was getting earlier > (https://issues.apAache.org/jira/browse/CASSANDRA-11826), but now the issue > is repair just stays stuck. > current status :- > [2016-05-19 05:52:50,835] Repair session a0e590e1-1d99-11e6-9d63-b717b380ffdd > for range (-3309358208555432808,-3279958773585646585] finished (progress: 54%) > [2016-05-19 05:53:09,446] Repair session a0e590e3-1d99-11e6-9d63-b717b380ffdd > for range (8149151263857514385,8181801084802729407] finished (progress: 55%) > [2016-05-19 05:53:13,808] Repair session a0e5b7f1-1d99-11e6-9d63-b717b380ffdd > for range (3372779397996730299,3381236471688156773] finished (progress: 55%) > [2016-05-19 05:53:27,543] Repair session a0e5b7f3-1d99-11e6-9d63-b717b380ffdd > for range (-4182952858113330342,-4157904914928848809] finished (progress: 55%) > [2016-05-19 05:53:41,128] Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd > for range (6499366179019889198,6523760493740195344] finished (progress: 55%) > And its 10:46:25 Now, almost 5 hours since it has been stuck right there. > Earlier i could see repair session going on in system.log but there are no > logs coming in right now, all i get in logs is regular index summary > redistribution logs. > Last logs for repair i saw in logs :- > INFO [RepairJobTask:5] 2016-05-19 05:53:41,125 RepairJob.java:152 - [repair > #a0e5df00-1d99-11e6-9d63-b717b380ffdd] TABLE_NAME is fully synced > INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairSession.java:279 - > [repair #a0e5df00-1d99-11e6-9d63-b717b380ffdd] Session completed successfully > INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairRunnable.java:232 - > Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd for range > (6499366179019889198,6523760493740195344] finished > Its an incremental repair, and in "nodetool netstats" output i can see logs > like :- > Repair e3055fb0-1d9d-11e6-9d63-b717b380ffdd > /Node-2 > Receiving 8 files, 1093461 bytes total. Already received 8 files, > 1093461 bytes total > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80872-big-Data.db > 399475/399475 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80879-big-Data.db > 53809/53809 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80878-big-Data.db > 89955/89955 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80881-big-Data.db > 168790/168790 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80886-big-Data.db > 107785/107785 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80880-big-Data.db > 52889/52889 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80884-big-Data.db > 148882/148882 bytes(100%) received from idx:0/Node-2 > >
[jira] [Commented] (CASSANDRA-11845) Hanging repair in cassandra 2.2.4
[ https://issues.apache.org/jira/browse/CASSANDRA-11845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292934#comment-15292934 ] vin01 commented on CASSANDRA-11845: --- Thanks Paulo, i restarted all 3 nodes and started repair again and got the errors which i have attached. (cassandra-2.2.4.error.log) Nodetool output for repair session :- [2016-05-20 02:37:59,168] Repair session cffbadd3-1e55-11e6-bd05-b717b380ffdd for range (-8184117312116560831,-8171918810495776305] failed with error Endpoint /Node-3 died (progress: 100%) .. still running > Hanging repair in cassandra 2.2.4 > - > > Key: CASSANDRA-11845 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11845 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging > Environment: Centos 6 >Reporter: vin01 >Priority: Minor > Attachments: cassandra-2.2.4.error.log > > > So after increasing the streaming_timeout_in_ms value to 3 hours, i was able > to avoid the socketTimeout errors i was getting earlier > (https://issues.apAache.org/jira/browse/CASSANDRA-11826), but now the issue > is repair just stays stuck. > current status :- > [2016-05-19 05:52:50,835] Repair session a0e590e1-1d99-11e6-9d63-b717b380ffdd > for range (-3309358208555432808,-3279958773585646585] finished (progress: 54%) > [2016-05-19 05:53:09,446] Repair session a0e590e3-1d99-11e6-9d63-b717b380ffdd > for range (8149151263857514385,8181801084802729407] finished (progress: 55%) > [2016-05-19 05:53:13,808] Repair session a0e5b7f1-1d99-11e6-9d63-b717b380ffdd > for range (3372779397996730299,3381236471688156773] finished (progress: 55%) > [2016-05-19 05:53:27,543] Repair session a0e5b7f3-1d99-11e6-9d63-b717b380ffdd > for range (-4182952858113330342,-4157904914928848809] finished (progress: 55%) > [2016-05-19 05:53:41,128] Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd > for range (6499366179019889198,6523760493740195344] finished (progress: 55%) > And its 10:46:25 Now, almost 5 hours since it has been stuck right there. > Earlier i could see repair session going on in system.log but there are no > logs coming in right now, all i get in logs is regular index summary > redistribution logs. > Last logs for repair i saw in logs :- > INFO [RepairJobTask:5] 2016-05-19 05:53:41,125 RepairJob.java:152 - [repair > #a0e5df00-1d99-11e6-9d63-b717b380ffdd] TABLE_NAME is fully synced > INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairSession.java:279 - > [repair #a0e5df00-1d99-11e6-9d63-b717b380ffdd] Session completed successfully > INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairRunnable.java:232 - > Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd for range > (6499366179019889198,6523760493740195344] finished > Its an incremental repair, and in "nodetool netstats" output i can see logs > like :- > Repair e3055fb0-1d9d-11e6-9d63-b717b380ffdd > /Node-2 > Receiving 8 files, 1093461 bytes total. Already received 8 files, > 1093461 bytes total > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80872-big-Data.db > 399475/399475 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80879-big-Data.db > 53809/53809 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80878-big-Data.db > 89955/89955 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80881-big-Data.db > 168790/168790 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80886-big-Data.db > 107785/107785 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80880-big-Data.db > 52889/52889 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80884-big-Data.db > 148882/148882 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80883-big-Data.db > 71876/71876 bytes(100%) received from idx:0/Node-2 > Sending 5 files, 863321 bytes total. Already sent 5 files, 863321 > bytes total > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73168-big-Data.db > 161895/161895 bytes(100%) sent to idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-72604-big-Data.db > 399865/399865 bytes(100%) sent to
[jira] [Updated] (CASSANDRA-11845) Hanging repair in cassandra 2.2.4
[ https://issues.apache.org/jira/browse/CASSANDRA-11845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vin01 updated CASSANDRA-11845: -- Attachment: cassandra-2.2.4.error.log > Hanging repair in cassandra 2.2.4 > - > > Key: CASSANDRA-11845 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11845 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging > Environment: Centos 6 >Reporter: vin01 >Priority: Minor > Attachments: cassandra-2.2.4.error.log > > > So after increasing the streaming_timeout_in_ms value to 3 hours, i was able > to avoid the socketTimeout errors i was getting earlier > (https://issues.apAache.org/jira/browse/CASSANDRA-11826), but now the issue > is repair just stays stuck. > current status :- > [2016-05-19 05:52:50,835] Repair session a0e590e1-1d99-11e6-9d63-b717b380ffdd > for range (-3309358208555432808,-3279958773585646585] finished (progress: 54%) > [2016-05-19 05:53:09,446] Repair session a0e590e3-1d99-11e6-9d63-b717b380ffdd > for range (8149151263857514385,8181801084802729407] finished (progress: 55%) > [2016-05-19 05:53:13,808] Repair session a0e5b7f1-1d99-11e6-9d63-b717b380ffdd > for range (3372779397996730299,3381236471688156773] finished (progress: 55%) > [2016-05-19 05:53:27,543] Repair session a0e5b7f3-1d99-11e6-9d63-b717b380ffdd > for range (-4182952858113330342,-4157904914928848809] finished (progress: 55%) > [2016-05-19 05:53:41,128] Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd > for range (6499366179019889198,6523760493740195344] finished (progress: 55%) > And its 10:46:25 Now, almost 5 hours since it has been stuck right there. > Earlier i could see repair session going on in system.log but there are no > logs coming in right now, all i get in logs is regular index summary > redistribution logs. > Last logs for repair i saw in logs :- > INFO [RepairJobTask:5] 2016-05-19 05:53:41,125 RepairJob.java:152 - [repair > #a0e5df00-1d99-11e6-9d63-b717b380ffdd] TABLE_NAME is fully synced > INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairSession.java:279 - > [repair #a0e5df00-1d99-11e6-9d63-b717b380ffdd] Session completed successfully > INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairRunnable.java:232 - > Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd for range > (6499366179019889198,6523760493740195344] finished > Its an incremental repair, and in "nodetool netstats" output i can see logs > like :- > Repair e3055fb0-1d9d-11e6-9d63-b717b380ffdd > /Node-2 > Receiving 8 files, 1093461 bytes total. Already received 8 files, > 1093461 bytes total > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80872-big-Data.db > 399475/399475 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80879-big-Data.db > 53809/53809 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80878-big-Data.db > 89955/89955 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80881-big-Data.db > 168790/168790 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80886-big-Data.db > 107785/107785 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80880-big-Data.db > 52889/52889 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80884-big-Data.db > 148882/148882 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80883-big-Data.db > 71876/71876 bytes(100%) received from idx:0/Node-2 > Sending 5 files, 863321 bytes total. Already sent 5 files, 863321 > bytes total > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73168-big-Data.db > 161895/161895 bytes(100%) sent to idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-72604-big-Data.db > 399865/399865 bytes(100%) sent to idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73147-big-Data.db > 149066/149066 bytes(100%) sent to idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-72682-big-Data.db > 126000/126000 bytes(100%) sent to idx:0/Node-2 > >
[jira] [Comment Edited] (CASSANDRA-11845) Hanging repair in cassandra 2.2.4
[ https://issues.apache.org/jira/browse/CASSANDRA-11845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291939#comment-15291939 ] vin01 edited comment on CASSANDRA-11845 at 5/19/16 7:13 PM: Yeah, its still stuck at 55 % . No new streams are getting created, netstats shows the same output again n again. Only thing that changes in its output is :- Small messages n/a 0 14760878 Gossip messages n/a 0 151698 Here is a longer snippet of netstats output which shows the repair session as well, it has been the same for last 8 or so hrs :- Repair c0c8af20-1d9c-11e6-9d63-b717b380ffdd /Node-3 Receiving 11 files, 13896288 bytes total. Already received 11 files, 13896288 bytes total /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79186-big-Data.db 1598874/1598874 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79196-big-Data.db 736365/736365 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79197-big-Data.db 326558/326558 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79187-big-Data.db 1484827/1484827 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79180-big-Data.db 393636/393636 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79184-big-Data.db 825459/825459 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79188-big-Data.db 3568782/3568782 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79182-big-Data.db 271222/271222 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79193-big-Data.db 4315497/4315497 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79183-big-Data.db 19775/19775 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79192-big-Data.db 355293/355293 bytes(100%) received from idx:0/Node-3 Sending 5 files, 9444101 bytes total. Already sent 5 files, 9444101 bytes total /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73168-big-Data.db 1796825/1796825 bytes(100%) sent to idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-72604-big-Data.db 4549996/4549996 bytes(100%) sent to idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73147-big-Data.db 1658881/1658881 bytes(100%) sent to idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-72682-big-Data.db 1418335/1418335 bytes(100%) sent to idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73173-big-Data.db 20064/20064 bytes(100%) sent to idx:0/Node-3 Read Repair Statistics: Attempted: 1142 Mismatch (Blocking): 0 Mismatch (Background): 0 Pool NameActive Pending Completed Large messages n/a 0779 Small messages n/a 0 14760878 Gossip messages n/a 0 151698 Snippet for system.log using grep - iE "repair|valid|sync" system.log :- INFO [StreamReceiveTask:479] 2016-05-19 05:53:27,539 LocalSyncTask.java:114 - [repair #a0e5b7f3-1d99-11e6-9d63-b717b380ffdd] Sync complete using session a0e5b7f3-1d99-11e6-9d63-b717b380ffdd between /Node-2 and /Node-1 on TABLE_NAME INFO [RepairJobTask:5] 2016-05-19 05:53:27,540 RepairJob.java:152 - [repair #a0e5b7f3-1d99-11e6-9d63-b717b380ffdd] TABLE_NAME is fully synced INFO [RepairJobTask:5] 2016-05-19 05:53:27,541 RepairSession.java:279 - [repair #a0e5b7f3-1d99-11e6-9d63-b717b380ffdd] Session completed successfully INFO [RepairJobTask:5] 2016-05-19 05:53:27,542 RepairRunnable.java:232 - Repair session a0e5b7f3-1d99-11e6-9d63-b717b380ffdd for range (-4182952858113330342,-4157904914928848809] finished INFO [StreamReceiveTask:59] 2016-05-19 05:53:41,124 LocalSyncTask.java:114 - [repair #a0e5df00-1d99-11e6-9d63-b717b380ffdd] Sync complete using session a0e5df00-1d99-11e6-9d63-b717b380ffdd between /Node-2 a nd /Node-1 on
[jira] [Comment Edited] (CASSANDRA-11845) Hanging repair in cassandra 2.2.4
[ https://issues.apache.org/jira/browse/CASSANDRA-11845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291939#comment-15291939 ] vin01 edited comment on CASSANDRA-11845 at 5/19/16 7:12 PM: Yeah, its still stuck at 55 % . No new streams are getting created, netstats shows the same output again n again. Only thing that changes in its output is :- Small messages n/a 0 14760878 Gossip messages n/a 0 151698 Here is a longer snippet of netstats output which shows the repair session as well, it has been the same for last 8 or so hrs :- Repair c0c8af20-1d9c-11e6-9d63-b717b380ffdd /Node-3 Receiving 11 files, 13896288 bytes total. Already received 11 files, 13896288 bytes total /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79186-big-Data.db 1598874/1598874 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79196-big-Data.db 736365/736365 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79197-big-Data.db 326558/326558 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79187-big-Data.db 1484827/1484827 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79180-big-Data.db 393636/393636 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79184-big-Data.db 825459/825459 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79188-big-Data.db 3568782/3568782 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79182-big-Data.db 271222/271222 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79193-big-Data.db 4315497/4315497 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79183-big-Data.db 19775/19775 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79192-big-Data.db 355293/355293 bytes(100%) received from idx:0/Node-3 Sending 5 files, 9444101 bytes total. Already sent 5 files, 9444101 bytes total /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73168-big-Data.db 1796825/1796825 bytes(100%) sent to idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-72604-big-Data.db 4549996/4549996 bytes(100%) sent to idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73147-big-Data.db 1658881/1658881 bytes(100%) sent to idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-72682-big-Data.db 1418335/1418335 bytes(100%) sent to idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73173-big-Data.db 20064/20064 bytes(100%) sent to idx:0/Node-3 Read Repair Statistics: Attempted: 1142 Mismatch (Blocking): 0 Mismatch (Background): 0 Pool NameActive Pending Completed Large messages n/a 0779 Small messages n/a 0 14760878 Gossip messages n/a 0 151698 Snippet for system.log using grep - iE "repair|valid|sync" system.log :- INFO [StreamReceiveTask:479] 2016-05-19 05:53:27,539 LocalSyncTask.java:114 - [repair #a0e5b7f3-1d99-11e6-9d63-b717b380ffdd] Sync complete using session a0e5b7f3-1d99-11e6-9d63-b717b380ffdd between /Node-2 and /Node-1 on TABLE_NAME INFO [RepairJobTask:5] 2016-05-19 05:53:27,540 RepairJob.java:152 - [repair #a0e5b7f3-1d99-11e6-9d63-b717b380ffdd] TABLE_NAME is fully synced INFO [RepairJobTask:5] 2016-05-19 05:53:27,541 RepairSession.java:279 - [repair #a0e5b7f3-1d99-11e6-9d63-b717b380ffdd] Session completed successfully INFO [RepairJobTask:5] 2016-05-19 05:53:27,542 RepairRunnable.java:232 - Repair session a0e5b7f3-1d99-11e6-9d63-b717b380ffdd for range (-4182952858113330342,-4157904914928848809] finished INFO [StreamReceiveTask:59] 2016-05-19 05:53:41,124 LocalSyncTask.java:114 - [repair #a0e5df00-1d99-11e6-9d63-b717b380ffdd] Sync complete using session a0e5df00-1d99-11e6-9d63-b717b380ffdd between /Node-2 a nd /Node-1 on
[jira] [Comment Edited] (CASSANDRA-11845) Hanging repair in cassandra 2.2.4
[ https://issues.apache.org/jira/browse/CASSANDRA-11845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291939#comment-15291939 ] vin01 edited comment on CASSANDRA-11845 at 5/19/16 7:12 PM: Yeah, its still stuck at 55 % . No new streams are getting created, netstats shows the same output again n again. Only thing that changes in its output is :- Small messages n/a 0 14760878 Gossip messages n/a 0 151698 Here is a longer snippet of netstats output which shows the repair session as well, it has been the same for last 8 or so hrs :- Repair c0c8af20-1d9c-11e6-9d63-b717b380ffdd /Node-3 Receiving 11 files, 13896288 bytes total. Already received 11 files, 13896288 bytes total /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79186-big-Data.db 1598874/1598874 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79196-big-Data.db 736365/736365 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79197-big-Data.db 326558/326558 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79187-big-Data.db 1484827/1484827 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79180-big-Data.db 393636/393636 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79184-big-Data.db 825459/825459 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79188-big-Data.db 3568782/3568782 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79182-big-Data.db 271222/271222 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79193-big-Data.db 4315497/4315497 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79183-big-Data.db 19775/19775 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79192-big-Data.db 355293/355293 bytes(100%) received from idx:0/Node-3 Sending 5 files, 9444101 bytes total. Already sent 5 files, 9444101 bytes total /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73168-big-Data.db 1796825/1796825 bytes(100%) sent to idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-72604-big-Data.db 4549996/4549996 bytes(100%) sent to idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73147-big-Data.db 1658881/1658881 bytes(100%) sent to idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-72682-big-Data.db 1418335/1418335 bytes(100%) sent to idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73173-big-Data.db 20064/20064 bytes(100%) sent to idx:0/Node-3 Read Repair Statistics: Attempted: 1142 Mismatch (Blocking): 0 Mismatch (Background): 0 Pool NameActive Pending Completed Large messages n/a 0779 Small messages n/a 0 14760878 Gossip messages n/a 0 151698 Snippet for system.log using grep - iE "repair|valid|sync" system.log :- INFO [StreamReceiveTask:479] 2016-05-19 05:53:27,539 LocalSyncTask.java:114 - [repair #a0e5b7f3-1d99-11e6-9d63-b717b380ffdd] Sync complete using session a0e5b7f3-1d99-11e6-9d63-b717b380ffdd between /Node-2 and /Node-1 on TABLE_NAME INFO [RepairJobTask:5] 2016-05-19 05:53:27,540 RepairJob.java:152 - [repair #a0e5b7f3-1d99-11e6-9d63-b717b380ffdd] TABLE_NAME is fully synced INFO [RepairJobTask:5] 2016-05-19 05:53:27,541 RepairSession.java:279 - [repair #a0e5b7f3-1d99-11e6-9d63-b717b380ffdd] Session completed successfully INFO [RepairJobTask:5] 2016-05-19 05:53:27,542 RepairRunnable.java:232 - Repair session a0e5b7f3-1d99-11e6-9d63-b717b380ffdd for range (-4182952858113330342,-4157904914928848809] finished INFO [StreamReceiveTask:59] 2016-05-19 05:53:41,124 LocalSyncTask.java:114 - [repair #a0e5df00-1d99-11e6-9d63-b717b380ffdd] Sync complete using session a0e5df00-1d99-11e6-9d63-b717b380ffdd between /Node-2 a nd /Node-1 on
[jira] [Comment Edited] (CASSANDRA-11845) Hanging repair in cassandra 2.2.4
[ https://issues.apache.org/jira/browse/CASSANDRA-11845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291939#comment-15291939 ] vin01 edited comment on CASSANDRA-11845 at 5/19/16 7:11 PM: Yeah, its still stuck at 55 % . No new streams are getting created, netstats shows the same output again n again. Only thing that changes in its output is :- Small messages n/a 0 14760878 Gossip messages n/a 0 151698 Here is a longer snippet of netstats output which shows the repair session as well, it has been the same for last 8 or so hrs :- Repair c0c8af20-1d9c-11e6-9d63-b717b380ffdd /Node-1 Receiving 11 files, 13896288 bytes total. Already received 11 files, 13896288 bytes total /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79186-big-Data.db 1598874/1598874 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79196-big-Data.db 736365/736365 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79197-big-Data.db 326558/326558 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79187-big-Data.db 1484827/1484827 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79180-big-Data.db 393636/393636 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79184-big-Data.db 825459/825459 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79188-big-Data.db 3568782/3568782 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79182-big-Data.db 271222/271222 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79193-big-Data.db 4315497/4315497 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79183-big-Data.db 19775/19775 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79192-big-Data.db 355293/355293 bytes(100%) received from idx:0/Node-1 Sending 5 files, 9444101 bytes total. Already sent 5 files, 9444101 bytes total /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73168-big-Data.db 1796825/1796825 bytes(100%) sent to idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-72604-big-Data.db 4549996/4549996 bytes(100%) sent to idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73147-big-Data.db 1658881/1658881 bytes(100%) sent to idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-72682-big-Data.db 1418335/1418335 bytes(100%) sent to idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73173-big-Data.db 20064/20064 bytes(100%) sent to idx:0/Node-1 Read Repair Statistics: Attempted: 1142 Mismatch (Blocking): 0 Mismatch (Background): 0 Pool NameActive Pending Completed Large messages n/a 0779 Small messages n/a 0 14760878 Gossip messages n/a 0 151698 Snippet for system.log using grep - iE "repair|valid|sync" system.log :- INFO [StreamReceiveTask:479] 2016-05-19 05:53:27,539 LocalSyncTask.java:114 - [repair #a0e5b7f3-1d99-11e6-9d63-b717b380ffdd] Sync complete using session a0e5b7f3-1d99-11e6-9d63-b717b380ffdd between /Node-2 and /192.168.200.151 on TABLE_NAME INFO [RepairJobTask:5] 2016-05-19 05:53:27,540 RepairJob.java:152 - [repair #a0e5b7f3-1d99-11e6-9d63-b717b380ffdd] TABLE_NAME is fully synced INFO [RepairJobTask:5] 2016-05-19 05:53:27,541 RepairSession.java:279 - [repair #a0e5b7f3-1d99-11e6-9d63-b717b380ffdd] Session completed successfully INFO [RepairJobTask:5] 2016-05-19 05:53:27,542 RepairRunnable.java:232 - Repair session a0e5b7f3-1d99-11e6-9d63-b717b380ffdd for range (-4182952858113330342,-4157904914928848809] finished INFO [StreamReceiveTask:59] 2016-05-19 05:53:41,124 LocalSyncTask.java:114 - [repair #a0e5df00-1d99-11e6-9d63-b717b380ffdd] Sync complete using session a0e5df00-1d99-11e6-9d63-b717b380ffdd between /Node-2 a nd
[jira] [Comment Edited] (CASSANDRA-11845) Hanging repair in cassandra 2.2.4
[ https://issues.apache.org/jira/browse/CASSANDRA-11845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291939#comment-15291939 ] vin01 edited comment on CASSANDRA-11845 at 5/19/16 7:02 PM: Yeah, its still stuck at 55 % . No new streams are getting created, netstats shows the same output again n again. Only thing that changes in its output is :- Small messages n/a 0 14760878 Gossip messages n/a 0 151698 Here is a longer snippet of netstats output which shows the repair session as well, it has been the same for last 8 or so hrs :- Repair c0c8af20-1d9c-11e6-9d63-b717b380ffdd /Node-1 Receiving 11 files, 13896288 bytes total. Already received 11 files, 13896288 bytes total /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79186-big-Data.db 1598874/1598874 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79196-big-Data.db 736365/736365 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79197-big-Data.db 326558/326558 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79187-big-Data.db 1484827/1484827 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79180-big-Data.db 393636/393636 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79184-big-Data.db 825459/825459 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79188-big-Data.db 3568782/3568782 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79182-big-Data.db 271222/271222 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79193-big-Data.db 4315497/4315497 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79183-big-Data.db 19775/19775 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79192-big-Data.db 355293/355293 bytes(100%) received from idx:0/Node-1 Sending 5 files, 9444101 bytes total. Already sent 5 files, 9444101 bytes total /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73168-big-Data.db 1796825/1796825 bytes(100%) sent to idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-72604-big-Data.db 4549996/4549996 bytes(100%) sent to idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73147-big-Data.db 1658881/1658881 bytes(100%) sent to idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-72682-big-Data.db 1418335/1418335 bytes(100%) sent to idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73173-big-Data.db 20064/20064 bytes(100%) sent to idx:0/Node-1 Read Repair Statistics: Attempted: 1142 Mismatch (Blocking): 0 Mismatch (Background): 0 Pool NameActive Pending Completed Large messages n/a 0779 Small messages n/a 0 14760878 Gossip messages n/a 0 151698 Snippet for system.log using grep - iE "repair|valid|sync" system.log :- INFO [StreamReceiveTask:479] 2016-05-19 05:53:27,539 LocalSyncTask.java:114 - [repair #a0e5b7f3-1d99-11e6-9d63-b717b380ffdd] Sync complete using session a0e5b7f3-1d99-11e6-9d63-b717b380ffdd between /192.168.100.138 and /192.168.200.151 on TABLE_NAME INFO [RepairJobTask:5] 2016-05-19 05:53:27,540 RepairJob.java:152 - [repair #a0e5b7f3-1d99-11e6-9d63-b717b380ffdd] TABLE_NAME is fully synced INFO [RepairJobTask:5] 2016-05-19 05:53:27,541 RepairSession.java:279 - [repair #a0e5b7f3-1d99-11e6-9d63-b717b380ffdd] Session completed successfully INFO [RepairJobTask:5] 2016-05-19 05:53:27,542 RepairRunnable.java:232 - Repair session a0e5b7f3-1d99-11e6-9d63-b717b380ffdd for range (-4182952858113330342,-4157904914928848809] finished INFO [StreamReceiveTask:59] 2016-05-19 05:53:41,124 LocalSyncTask.java:114 - [repair #a0e5df00-1d99-11e6-9d63-b717b380ffdd] Sync complete using session a0e5df00-1d99-11e6-9d63-b717b380ffdd between
[jira] [Comment Edited] (CASSANDRA-11845) Hanging repair in cassandra 2.2.4
[ https://issues.apache.org/jira/browse/CASSANDRA-11845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291939#comment-15291939 ] vin01 edited comment on CASSANDRA-11845 at 5/19/16 6:59 PM: Yeah, its still stuck at 55 % . No new streams are getting created, netstats shows the same output again n again. Only thing that changes in its output is :- Small messages n/a 0 14760878 Gossip messages n/a 0 151698 Here is a longer snippet of netstats output which shows the repair session as well, it has been the same for last 8 or so hrs :- Repair c0c8af20-1d9c-11e6-9d63-b717b380ffdd /Node-1 Receiving 11 files, 13896288 bytes total. Already received 11 files, 13896288 bytes total /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79186-big-Data.db 1598874/1598874 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79196-big-Data.db 736365/736365 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79197-big-Data.db 326558/326558 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79187-big-Data.db 1484827/1484827 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79180-big-Data.db 393636/393636 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79184-big-Data.db 825459/825459 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79188-big-Data.db 3568782/3568782 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79182-big-Data.db 271222/271222 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79193-big-Data.db 4315497/4315497 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79183-big-Data.db 19775/19775 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79192-big-Data.db 355293/355293 bytes(100%) received from idx:0/Node-1 Sending 5 files, 9444101 bytes total. Already sent 5 files, 9444101 bytes total /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73168-big-Data.db 1796825/1796825 bytes(100%) sent to idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-72604-big-Data.db 4549996/4549996 bytes(100%) sent to idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73147-big-Data.db 1658881/1658881 bytes(100%) sent to idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-72682-big-Data.db 1418335/1418335 bytes(100%) sent to idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73173-big-Data.db 20064/20064 bytes(100%) sent to idx:0/Node-1 Read Repair Statistics: Attempted: 1142 Mismatch (Blocking): 0 Mismatch (Background): 0 Pool NameActive Pending Completed Large messages n/a 0779 Small messages n/a 0 14760878 Gossip messages n/a 0 151698 Snippet for system.log using grep - iE "repair|valid|sync" system.log :- INFO [StreamReceiveTask:479] 2016-05-19 05:53:27,539 LocalSyncTask.java:114 - [repair #a0e5b7f3-1d99-11e6-9d63-b717b380ffdd] Sync complete using session a0e5b7f3-1d99-11e6-9d63-b717b380ffdd between /192.168.100.138 and /192.168.200.151 on TABLE_NAME INFO [RepairJobTask:5] 2016-05-19 05:53:27,540 RepairJob.java:152 - [repair #a0e5b7f3-1d99-11e6-9d63-b717b380ffdd] TABLE_NAME is fully synced INFO [RepairJobTask:5] 2016-05-19 05:53:27,541 RepairSession.java:279 - [repair #a0e5b7f3-1d99-11e6-9d63-b717b380ffdd] Session completed successfully INFO [RepairJobTask:5] 2016-05-19 05:53:27,542 RepairRunnable.java:232 - Repair session a0e5b7f3-1d99-11e6-9d63-b717b380ffdd for range (-4182952858113330342,-4157904914928848809] finished INFO [StreamReceiveTask:59] 2016-05-19 05:53:41,124 LocalSyncTask.java:114 - [repair #a0e5df00-1d99-11e6-9d63-b717b380ffdd] Sync complete using session a0e5df00-1d99-11e6-9d63-b717b380ffdd between
[jira] [Commented] (CASSANDRA-11845) Hanging repair in cassandra 2.2.4
[ https://issues.apache.org/jira/browse/CASSANDRA-11845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291939#comment-15291939 ] vin01 commented on CASSANDRA-11845: --- Yeah, its still stuck at 55 % . No new streams are getting created, netstats shows the same output again n again. Only thing that changes in its output is :- Small messages n/a 0 14760878 Gossip messages n/a 0 151698 Here is a longer snippet of netstats output which shows the repair session as well, it has been the same for last 8 or so hrs :- Repair c0c8af20-1d9c-11e6-9d63-b717b380ffdd /Node-1 Receiving 11 files, 13896288 bytes total. Already received 11 files, 13896288 bytes total /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79186-big-Data.db 1598874/1598874 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79196-big-Data.db 736365/736365 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79197-big-Data.db 326558/326558 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79187-big-Data.db 1484827/1484827 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79180-big-Data.db 393636/393636 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79184-big-Data.db 825459/825459 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79188-big-Data.db 3568782/3568782 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79182-big-Data.db 271222/271222 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79193-big-Data.db 4315497/4315497 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79183-big-Data.db 19775/19775 bytes(100%) received from idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79192-big-Data.db 355293/355293 bytes(100%) received from idx:0/Node-1 Sending 5 files, 9444101 bytes total. Already sent 5 files, 9444101 bytes total /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73168-big-Data.db 1796825/1796825 bytes(100%) sent to idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-72604-big-Data.db 4549996/4549996 bytes(100%) sent to idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73147-big-Data.db 1658881/1658881 bytes(100%) sent to idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-72682-big-Data.db 1418335/1418335 bytes(100%) sent to idx:0/Node-1 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73173-big-Data.db 20064/20064 bytes(100%) sent to idx:0/Node-1 Read Repair Statistics: Attempted: 1142 Mismatch (Blocking): 0 Mismatch (Background): 0 Pool NameActive Pending Completed Large messages n/a 0779 Small messages n/a 0 14760878 Gossip messages n/a 0 151698 Snippet for system.log using grep -iE "repair|valid|sync" system.log :- INFO [StreamReceiveTask:479] 2016-05-19 05:53:27,539 LocalSyncTask.java:114 - [repair #a0e5b7f3-1d99-11e6-9d63-b717b380ffdd] Sync complete using session a0e5b7f3-1d99-11e6-9d63-b717b380ffdd between /192.168.100.138 and /192.168.200.151 on TABLE_NAME INFO [RepairJobTask:5] 2016-05-19 05:53:27,540 RepairJob.java:152 - [repair #a0e5b7f3-1d99-11e6-9d63-b717b380ffdd] TABLE_NAME is fully synced INFO [RepairJobTask:5] 2016-05-19 05:53:27,541 RepairSession.java:279 - [repair #a0e5b7f3-1d99-11e6-9d63-b717b380ffdd] Session completed successfully INFO [RepairJobTask:5] 2016-05-19 05:53:27,542 RepairRunnable.java:232 - Repair session a0e5b7f3-1d99-11e6-9d63-b717b380ffdd for range (-4182952858113330342,-4157904914928848809] finished INFO [StreamReceiveTask:59] 2016-05-19 05:53:41,124 LocalSyncTask.java:114 - [repair #a0e5df00-1d99-11e6-9d63-b717b380ffdd] Sync complete using session a0e5df00-1d99-11e6-9d63-b717b380ffdd between /192.168.100.138 a nd /192.168.200.151 on TABLE_NAME INFO
[jira] [Comment Edited] (CASSANDRA-11845) Hanging repair in cassandra 2.2.4
[ https://issues.apache.org/jira/browse/CASSANDRA-11845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291564#comment-15291564 ] vin01 edited comment on CASSANDRA-11845 at 5/19/16 5:24 PM: [-]$ /mydir/apache-cassandra-2.2.4/bin/nodetool compactionstats pending tasks: 0 (output of compactionstats is same on all 3 nodes) Its still stuck at same point. nodetool netstats output summary :- Read Repair Statistics: Attempted: 1142 Mismatch (Blocking): 0 Mismatch (Background): 0 Pool NameActive Pending Completed Large messages n/a 0779 Small messages n/a 0 14758741 Gossip messages n/a 0 135056 was (Author: vin01): [-]$ /mydir/apache-cassandra-2.2.4/bin/nodetool compactionstats pending tasks: 0 Its still stuck at same point. nodetool netstats output summary :- Read Repair Statistics: Attempted: 1142 Mismatch (Blocking): 0 Mismatch (Background): 0 Pool NameActive Pending Completed Large messages n/a 0779 Small messages n/a 0 14758741 Gossip messages n/a 0 135056 > Hanging repair in cassandra 2.2.4 > - > > Key: CASSANDRA-11845 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11845 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging > Environment: Centos 6 >Reporter: vin01 >Priority: Minor > > So after increasing the streaming_timeout_in_ms value to 3 hours, i was able > to avoid the socketTimeout errors i was getting earlier > (https://issues.apAache.org/jira/browse/CASSANDRA-11826), but now the issue > is repair just stays stuck. > current status :- > [2016-05-19 05:52:50,835] Repair session a0e590e1-1d99-11e6-9d63-b717b380ffdd > for range (-3309358208555432808,-3279958773585646585] finished (progress: 54%) > [2016-05-19 05:53:09,446] Repair session a0e590e3-1d99-11e6-9d63-b717b380ffdd > for range (8149151263857514385,8181801084802729407] finished (progress: 55%) > [2016-05-19 05:53:13,808] Repair session a0e5b7f1-1d99-11e6-9d63-b717b380ffdd > for range (3372779397996730299,3381236471688156773] finished (progress: 55%) > [2016-05-19 05:53:27,543] Repair session a0e5b7f3-1d99-11e6-9d63-b717b380ffdd > for range (-4182952858113330342,-4157904914928848809] finished (progress: 55%) > [2016-05-19 05:53:41,128] Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd > for range (6499366179019889198,6523760493740195344] finished (progress: 55%) > And its 10:46:25 Now, almost 5 hours since it has been stuck right there. > Earlier i could see repair session going on in system.log but there are no > logs coming in right now, all i get in logs is regular index summary > redistribution logs. > Last logs for repair i saw in logs :- > INFO [RepairJobTask:5] 2016-05-19 05:53:41,125 RepairJob.java:152 - [repair > #a0e5df00-1d99-11e6-9d63-b717b380ffdd] TABLE_NAME is fully synced > INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairSession.java:279 - > [repair #a0e5df00-1d99-11e6-9d63-b717b380ffdd] Session completed successfully > INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairRunnable.java:232 - > Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd for range > (6499366179019889198,6523760493740195344] finished > Its an incremental repair, and in "nodetool netstats" output i can see logs > like :- > Repair e3055fb0-1d9d-11e6-9d63-b717b380ffdd > /Node-2 > Receiving 8 files, 1093461 bytes total. Already received 8 files, > 1093461 bytes total > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80872-big-Data.db > 399475/399475 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80879-big-Data.db > 53809/53809 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80878-big-Data.db > 89955/89955 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80881-big-Data.db > 168790/168790 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80886-big-Data.db > 107785/107785 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80880-big-Data.db > 52889/52889 bytes(100%) received from idx:0/Node-2 > >
[jira] [Commented] (CASSANDRA-11845) Hanging repair in cassandra 2.2.4
[ https://issues.apache.org/jira/browse/CASSANDRA-11845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291564#comment-15291564 ] vin01 commented on CASSANDRA-11845: --- [-]$ /mydir/apache-cassandra-2.2.4/bin/nodetool compactionstats pending tasks: 0 Its still stuck at same point. nodetool netstats output summary :- Read Repair Statistics: Attempted: 1142 Mismatch (Blocking): 0 Mismatch (Background): 0 Pool NameActive Pending Completed Large messages n/a 0779 Small messages n/a 0 14758741 Gossip messages n/a 0 135056 > Hanging repair in cassandra 2.2.4 > - > > Key: CASSANDRA-11845 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11845 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging > Environment: Centos 6 >Reporter: vin01 >Priority: Minor > > So after increasing the streaming_timeout_in_ms value to 3 hours, i was able > to avoid the socketTimeout errors i was getting earlier > (https://issues.apAache.org/jira/browse/CASSANDRA-11826), but now the issue > is repair just stays stuck. > current status :- > [2016-05-19 05:52:50,835] Repair session a0e590e1-1d99-11e6-9d63-b717b380ffdd > for range (-3309358208555432808,-3279958773585646585] finished (progress: 54%) > [2016-05-19 05:53:09,446] Repair session a0e590e3-1d99-11e6-9d63-b717b380ffdd > for range (8149151263857514385,8181801084802729407] finished (progress: 55%) > [2016-05-19 05:53:13,808] Repair session a0e5b7f1-1d99-11e6-9d63-b717b380ffdd > for range (3372779397996730299,3381236471688156773] finished (progress: 55%) > [2016-05-19 05:53:27,543] Repair session a0e5b7f3-1d99-11e6-9d63-b717b380ffdd > for range (-4182952858113330342,-4157904914928848809] finished (progress: 55%) > [2016-05-19 05:53:41,128] Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd > for range (6499366179019889198,6523760493740195344] finished (progress: 55%) > And its 10:46:25 Now, almost 5 hours since it has been stuck right there. > Earlier i could see repair session going on in system.log but there are no > logs coming in right now, all i get in logs is regular index summary > redistribution logs. > Last logs for repair i saw in logs :- > INFO [RepairJobTask:5] 2016-05-19 05:53:41,125 RepairJob.java:152 - [repair > #a0e5df00-1d99-11e6-9d63-b717b380ffdd] TABLE_NAME is fully synced > INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairSession.java:279 - > [repair #a0e5df00-1d99-11e6-9d63-b717b380ffdd] Session completed successfully > INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairRunnable.java:232 - > Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd for range > (6499366179019889198,6523760493740195344] finished > Its an incremental repair, and in "nodetool netstats" output i can see logs > like :- > Repair e3055fb0-1d9d-11e6-9d63-b717b380ffdd > /Node-2 > Receiving 8 files, 1093461 bytes total. Already received 8 files, > 1093461 bytes total > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80872-big-Data.db > 399475/399475 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80879-big-Data.db > 53809/53809 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80878-big-Data.db > 89955/89955 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80881-big-Data.db > 168790/168790 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80886-big-Data.db > 107785/107785 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80880-big-Data.db > 52889/52889 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80884-big-Data.db > 148882/148882 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80883-big-Data.db > 71876/71876 bytes(100%) received from idx:0/Node-2 > Sending 5 files, 863321 bytes total. Already sent 5 files, 863321 > bytes total > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73168-big-Data.db > 161895/161895 bytes(100%) sent to idx:0/Node-2 > >
[jira] [Commented] (CASSANDRA-11845) Hanging repair in cassandra 2.2.4
[ https://issues.apache.org/jira/browse/CASSANDRA-11845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291457#comment-15291457 ] vin01 commented on CASSANDRA-11845: --- Because of '-XX:+PerfDisableSharedMem' its not possible to use jstack or any similar tools i guess. Also debug logging is not enabled.. so nothing in debug.log, i don't think log level can be changed at runtime.. And yes there are secondary indices in that table. > Hanging repair in cassandra 2.2.4 > - > > Key: CASSANDRA-11845 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11845 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging > Environment: Centos 6 >Reporter: vin01 >Priority: Minor > > So after increasing the streaming_timeout_in_ms value to 3 hours, i was able > to avoid the socketTimeout errors i was getting earlier > (https://issues.apAache.org/jira/browse/CASSANDRA-11826), but now the issue > is repair just stays stuck. > current status :- > [2016-05-19 05:52:50,835] Repair session a0e590e1-1d99-11e6-9d63-b717b380ffdd > for range (-3309358208555432808,-3279958773585646585] finished (progress: 54%) > [2016-05-19 05:53:09,446] Repair session a0e590e3-1d99-11e6-9d63-b717b380ffdd > for range (8149151263857514385,8181801084802729407] finished (progress: 55%) > [2016-05-19 05:53:13,808] Repair session a0e5b7f1-1d99-11e6-9d63-b717b380ffdd > for range (3372779397996730299,3381236471688156773] finished (progress: 55%) > [2016-05-19 05:53:27,543] Repair session a0e5b7f3-1d99-11e6-9d63-b717b380ffdd > for range (-4182952858113330342,-4157904914928848809] finished (progress: 55%) > [2016-05-19 05:53:41,128] Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd > for range (6499366179019889198,6523760493740195344] finished (progress: 55%) > And its 10:46:25 Now, almost 5 hours since it has been stuck right there. > Earlier i could see repair session going on in system.log but there are no > logs coming in right now, all i get in logs is regular index summary > redistribution logs. > Last logs for repair i saw in logs :- > INFO [RepairJobTask:5] 2016-05-19 05:53:41,125 RepairJob.java:152 - [repair > #a0e5df00-1d99-11e6-9d63-b717b380ffdd] TABLE_NAME is fully synced > INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairSession.java:279 - > [repair #a0e5df00-1d99-11e6-9d63-b717b380ffdd] Session completed successfully > INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairRunnable.java:232 - > Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd for range > (6499366179019889198,6523760493740195344] finished > Its an incremental repair, and in "nodetool netstats" output i can see logs > like :- > Repair e3055fb0-1d9d-11e6-9d63-b717b380ffdd > /Node-2 > Receiving 8 files, 1093461 bytes total. Already received 8 files, > 1093461 bytes total > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80872-big-Data.db > 399475/399475 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80879-big-Data.db > 53809/53809 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80878-big-Data.db > 89955/89955 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80881-big-Data.db > 168790/168790 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80886-big-Data.db > 107785/107785 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80880-big-Data.db > 52889/52889 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80884-big-Data.db > 148882/148882 bytes(100%) received from idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80883-big-Data.db > 71876/71876 bytes(100%) received from idx:0/Node-2 > Sending 5 files, 863321 bytes total. Already sent 5 files, 863321 > bytes total > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73168-big-Data.db > 161895/161895 bytes(100%) sent to idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-72604-big-Data.db > 399865/399865 bytes(100%) sent to idx:0/Node-2 > > /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73147-big-Data.db > 149066/149066 bytes(100%) sent to
[jira] [Updated] (CASSANDRA-11845) Hanging repair in cassandra 2.2.4
[ https://issues.apache.org/jira/browse/CASSANDRA-11845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vin01 updated CASSANDRA-11845: -- Description: So after increasing the streaming_timeout_in_ms value to 3 hours, i was able to avoid the socketTimeout errors i was getting earlier (https://issues.apAache.org/jira/browse/CASSANDRA-11826), but now the issue is repair just stays stuck. current status :- [2016-05-19 05:52:50,835] Repair session a0e590e1-1d99-11e6-9d63-b717b380ffdd for range (-3309358208555432808,-3279958773585646585] finished (progress: 54%) [2016-05-19 05:53:09,446] Repair session a0e590e3-1d99-11e6-9d63-b717b380ffdd for range (8149151263857514385,8181801084802729407] finished (progress: 55%) [2016-05-19 05:53:13,808] Repair session a0e5b7f1-1d99-11e6-9d63-b717b380ffdd for range (3372779397996730299,3381236471688156773] finished (progress: 55%) [2016-05-19 05:53:27,543] Repair session a0e5b7f3-1d99-11e6-9d63-b717b380ffdd for range (-4182952858113330342,-4157904914928848809] finished (progress: 55%) [2016-05-19 05:53:41,128] Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd for range (6499366179019889198,6523760493740195344] finished (progress: 55%) And its 10:46:25 Now, almost 5 hours since it has been stuck right there. Earlier i could see repair session going on in system.log but there are no logs coming in right now, all i get in logs is regular index summary redistribution logs. Last logs for repair i saw in logs :- INFO [RepairJobTask:5] 2016-05-19 05:53:41,125 RepairJob.java:152 - [repair #a0e5df00-1d99-11e6-9d63-b717b380ffdd] TABLE_NAME is fully synced INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairSession.java:279 - [repair #a0e5df00-1d99-11e6-9d63-b717b380ffdd] Session completed successfully INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairRunnable.java:232 - Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd for range (6499366179019889198,6523760493740195344] finished Its an incremental repair, and in "nodetool netstats" output i can see logs like :- Repair e3055fb0-1d9d-11e6-9d63-b717b380ffdd /Node-2 Receiving 8 files, 1093461 bytes total. Already received 8 files, 1093461 bytes total /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80872-big-Data.db 399475/399475 bytes(100%) received from idx:0/Node-2 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80879-big-Data.db 53809/53809 bytes(100%) received from idx:0/Node-2 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80878-big-Data.db 89955/89955 bytes(100%) received from idx:0/Node-2 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80881-big-Data.db 168790/168790 bytes(100%) received from idx:0/Node-2 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80886-big-Data.db 107785/107785 bytes(100%) received from idx:0/Node-2 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80880-big-Data.db 52889/52889 bytes(100%) received from idx:0/Node-2 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80884-big-Data.db 148882/148882 bytes(100%) received from idx:0/Node-2 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80883-big-Data.db 71876/71876 bytes(100%) received from idx:0/Node-2 Sending 5 files, 863321 bytes total. Already sent 5 files, 863321 bytes total /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73168-big-Data.db 161895/161895 bytes(100%) sent to idx:0/Node-2 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-72604-big-Data.db 399865/399865 bytes(100%) sent to idx:0/Node-2 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73147-big-Data.db 149066/149066 bytes(100%) sent to idx:0/Node-2 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-72682-big-Data.db 126000/126000 bytes(100%) sent to idx:0/Node-2 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73173-big-Data.db 26495/26495 bytes(100%) sent to idx:0/Node-2 Repair c0c8af20-1d9c-11e6-9d63-b717b380ffdd /Node-3 Receiving 11 files, 13896288 bytes total. Already received 11 files, 13896288 bytes total /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79186-big-Data.db 1598874/1598874 bytes(100%) received from idx:0/Node-3 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79196-big-Data.db 736365/736365 bytes(100%) received
[jira] [Updated] (CASSANDRA-11845) Hanging repair in cassandra 2.2.4
[ https://issues.apache.org/jira/browse/CASSANDRA-11845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vin01 updated CASSANDRA-11845: -- Description: So after increasing the streaming_timeout_in_ms value to 3 hours, i was able to avoid the socketTimeout errors i was getting earlier (https://issues.apache.org/jira/browse/CASSANDRA-11826), but now the issue is repair just stays stuck. current status :- [2016-05-19 05:52:50,835] Repair session a0e590e1-1d99-11e6-9d63-b717b380ffdd for range (-3309358208555432808,-3279958773585646585] finished (progress: 54%) [2016-05-19 05:53:09,446] Repair session a0e590e3-1d99-11e6-9d63-b717b380ffdd for range (8149151263857514385,8181801084802729407] finished (progress: 55%) [2016-05-19 05:53:13,808] Repair session a0e5b7f1-1d99-11e6-9d63-b717b380ffdd for range (3372779397996730299,3381236471688156773] finished (progress: 55%) [2016-05-19 05:53:27,543] Repair session a0e5b7f3-1d99-11e6-9d63-b717b380ffdd for range (-4182952858113330342,-4157904914928848809] finished (progress: 55%) [2016-05-19 05:53:41,128] Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd for range (6499366179019889198,6523760493740195344] finished (progress: 55%) And its 10:46:25 Now, almost 5 hours since it has been stuck right there. Earlier i could see repair session going on in system.log but there are no logs coming in right now, all i get in logs is regular index summary redistribution logs. Last logs for repair i saw in logs :- INFO [RepairJobTask:5] 2016-05-19 05:53:41,125 RepairJob.java:152 - [repair #a0e5df00-1d99-11e6-9d63-b717b380ffdd] TABLE_NAME is fully synced INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairSession.java:279 - [repair #a0e5df00-1d99-11e6-9d63-b717b380ffdd] Session completed successfully INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairRunnable.java:232 - Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd for range (6499366179019889198,6523760493740195344] finished Its an incremental repair, and in "nodetool netstats" output i can see logs like :- Repair e3055fb0-1d9d-11e6-9d63-b717b380ffdd /192.168.100.138 Receiving 8 files, 1093461 bytes total. Already received 8 files, 1093461 bytes total /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80872-big-Data.db 399475/399475 bytes(100%) received from idx:0/192.168.100.138 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80879-big-Data.db 53809/53809 bytes(100%) received from idx:0/192.168.100.138 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80878-big-Data.db 89955/89955 bytes(100%) received from idx:0/192.168.100.138 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80881-big-Data.db 168790/168790 bytes(100%) received from idx:0/192.168.100.138 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80886-big-Data.db 107785/107785 bytes(100%) received from idx:0/192.168.100.138 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80880-big-Data.db 52889/52889 bytes(100%) received from idx:0/192.168.100.138 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80884-big-Data.db 148882/148882 bytes(100%) received from idx:0/192.168.100.138 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80883-big-Data.db 71876/71876 bytes(100%) received from idx:0/192.168.100.138 Sending 5 files, 863321 bytes total. Already sent 5 files, 863321 bytes total /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73168-big-Data.db 161895/161895 bytes(100%) sent to idx:0/192.168.100.138 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-72604-big-Data.db 399865/399865 bytes(100%) sent to idx:0/192.168.100.138 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73147-big-Data.db 149066/149066 bytes(100%) sent to idx:0/192.168.100.138 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-72682-big-Data.db 126000/126000 bytes(100%) sent to idx:0/192.168.100.138 /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/la-73173-big-Data.db 26495/26495 bytes(100%) sent to idx:0/192.168.100.138 Repair c0c8af20-1d9c-11e6-9d63-b717b380ffdd /192.168.100.147 Receiving 11 files, 13896288 bytes total. Already received 11 files, 13896288 bytes total /data/cassandra/data/KEYSPACE_NAME/TABLE_NAME-01ad9750723e11e4bfe0d3887930a87c/tmp-la-79186-big-Data.db 1598874/1598874 bytes(100%) received from idx:0/192.168.100.147
[jira] [Created] (CASSANDRA-11845) Hanging repair in cassandra 2.2.4
vin01 created CASSANDRA-11845: - Summary: Hanging repair in cassandra 2.2.4 Key: CASSANDRA-11845 URL: https://issues.apache.org/jira/browse/CASSANDRA-11845 Project: Cassandra Issue Type: Bug Components: Streaming and Messaging Environment: Centos 6 Reporter: vin01 Priority: Minor So after increasing the streaming_timeout_in_ms value to 3 hours, i was able to avoid the socketTimeout errors i was getting earlier (https://issues.apache.org/jira/browse/CASSANDRA-11826), but now the issue is repair just stays stuck. current status :- [2016-05-19 05:52:50,835] Repair session a0e590e1-1d99-11e6-9d63-b717b380ffdd for range (-3309358208555432808,-3279958773585646585] finished (progress: 54%) [2016-05-19 05:53:09,446] Repair session a0e590e3-1d99-11e6-9d63-b717b380ffdd for range (8149151263857514385,8181801084802729407] finished (progress: 55%) [2016-05-19 05:53:13,808] Repair session a0e5b7f1-1d99-11e6-9d63-b717b380ffdd for range (3372779397996730299,3381236471688156773] finished (progress: 55%) [2016-05-19 05:53:27,543] Repair session a0e5b7f3-1d99-11e6-9d63-b717b380ffdd for range (-4182952858113330342,-4157904914928848809] finished (progress: 55%) [2016-05-19 05:53:41,128] Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd for range (6499366179019889198,6523760493740195344] finished (progress: 55%) And its 10:46:25 Now, almost 5 hours since it has been stuck right there. Earlier i could see repair session going on in system.log but there are no logs coming in right now, all i get in logs is regular index summary redistribution logs. Last logs for repair i saw in logs :- INFO [RepairJobTask:5] 2016-05-19 05:53:41,125 RepairJob.java:152 - [repair #a0e5df00-1d99-11e6-9d63-b717b380ffdd] TABLE_NAME is fully synced INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairSession.java:279 - [repair #a0e5df00-1d99-11e6-9d63-b717b380ffdd] Session completed successfully INFO [RepairJobTask:5] 2016-05-19 05:53:41,126 RepairRunnable.java:232 - Repair session a0e5df00-1d99-11e6-9d63-b717b380ffdd for range (6499366179019889198,6523760493740195344] finished Its an incremental repair, and in "nodetool netstats" output i can see logs like :- Repair e3055fb0-1d9d-11e6-9d63-b717b380ffdd /192.168.100.138 Receiving 8 files, 1093461 bytes total. Already received 8 files, 1093461 bytes total /data/cassandra/data/eviveportal/memberinfo-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80872-big-Data.db 399475/399475 bytes(100%) received from idx:0/192.168.100.138 /data/cassandra/data/eviveportal/memberinfo-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80879-big-Data.db 53809/53809 bytes(100%) received from idx:0/192.168.100.138 /data/cassandra/data/eviveportal/memberinfo-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80878-big-Data.db 89955/89955 bytes(100%) received from idx:0/192.168.100.138 /data/cassandra/data/eviveportal/memberinfo-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80881-big-Data.db 168790/168790 bytes(100%) received from idx:0/192.168.100.138 /data/cassandra/data/eviveportal/memberinfo-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80886-big-Data.db 107785/107785 bytes(100%) received from idx:0/192.168.100.138 /data/cassandra/data/eviveportal/memberinfo-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80880-big-Data.db 52889/52889 bytes(100%) received from idx:0/192.168.100.138 /data/cassandra/data/eviveportal/memberinfo-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80884-big-Data.db 148882/148882 bytes(100%) received from idx:0/192.168.100.138 /data/cassandra/data/eviveportal/memberinfo-01ad9750723e11e4bfe0d3887930a87c/tmp-la-80883-big-Data.db 71876/71876 bytes(100%) received from idx:0/192.168.100.138 Sending 5 files, 863321 bytes total. Already sent 5 files, 863321 bytes total /data/cassandra/data/eviveportal/memberinfo-01ad9750723e11e4bfe0d3887930a87c/la-73168-big-Data.db 161895/161895 bytes(100%) sent to idx:0/192.168.100.138 /data/cassandra/data/eviveportal/memberinfo-01ad9750723e11e4bfe0d3887930a87c/la-72604-big-Data.db 399865/399865 bytes(100%) sent to idx:0/192.168.100.138 /data/cassandra/data/eviveportal/memberinfo-01ad9750723e11e4bfe0d3887930a87c/la-73147-big-Data.db 149066/149066 bytes(100%) sent to idx:0/192.168.100.138 /data/cassandra/data/eviveportal/memberinfo-01ad9750723e11e4bfe0d3887930a87c/la-72682-big-Data.db 126000/126000 bytes(100%) sent to idx:0/192.168.100.138 /data/cassandra/data/eviveportal/memberinfo-01ad9750723e11e4bfe0d3887930a87c/la-73173-big-Data.db 26495/26495 bytes(100%) sent to idx:0/192.168.100.138 Repair c0c8af20-1d9c-11e6-9d63-b717b380ffdd /192.168.100.147 Receiving 11 files, 13896288 bytes total. Already received 11 files, 13896288 bytes total
[jira] [Updated] (CASSANDRA-11826) Repairs failing for some of bigger tables
[ https://issues.apache.org/jira/browse/CASSANDRA-11826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vin01 updated CASSANDRA-11826: -- Description: Architecture for the same :- Cassandra version 2.2.4 3 Nodes - Node-1, Node-2, Node-3, repair started on Node-1 and following logs are also from Node-1 itself. Node-2 and Node-3 are in same data-center, and Node-1 is in a different data-center. Value of streaming_socket_timeout_in_ms is set to 1 Hour which is default for version 2.2.4. Repair succeeded for some of table but failed for one of tables which is quite huge, i looked at sizes of .db files in data directory and there are files up to 1.5 GB for that table. 1.4G la-59543-big-Data.db Logs for the same when repair failed :- WARN [RepairJobTask:5] 2016-05-18 02:25:31,137 RepairJob.java:162 - [repair #5a855825-1cb7-11e6-9f5f-b717b380ffdd] TABLE_NAME sync failed ERROR [RepairJobTask:5] 2016-05-18 02:25:31,138 RepairSession.java:290 - [repair #5a855825-1cb7-11e6-9f5f-b717b380ffdd] Session completed with the following error org.apache.cassandra.exceptions.RepairException: [repair #5a855825-1cb7-11e6-9f5f-b717b380ffdd on KEYSPACE_NAME/TABLE_NAME, (-7892648649079895028,-7818964903881740266]] Sync failed between /Node-3 and /Node-2 at org.apache.cassandra.repair.RemoteSyncTask.syncComplete(RemoteSyncTask.java:67) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:211) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:415) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:163) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] ERROR [RepairJobTask:5] 2016-05-18 02:25:31,138 RepairRunnable.java:243 - Repair session 5a855825-1cb7-11e6-9f5f-b717b380ffdd for range (-7892648649079895028,-7818964903881740266] failed with error [repair #5a855825-1cb7-11e6-9f5f-b717b380ffdd on KEYSPACE_NA$ org.apache.cassandra.exceptions.RepairException: [repair #5a855825-1cb7-11e6-9f5f-b717b380ffdd on KEYSPACE_NAME/TABLE_NAME, (-7892648649079895028,-7818964903881740266]] Sync failed between /Node-3 and /Node-2 at org.apache.cassandra.repair.RemoteSyncTask.syncComplete(RemoteSyncTask.java:67) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:211) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:415) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:163) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] ERROR [STREAM-IN-/Node-2] 2016-05-18 02:25:33,296 StreamSession.java:524 - [Stream #efe521b0-1cb8-11e6-9f5f-b717b380ffdd] Streaming error occurred java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.8.0_72] at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[na:1.8.0_72] at java.net.SocketInputStream.read(SocketInputStream.java:170) ~[na:1.8.0_72] at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[na:1.8.0_72] at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) ~[na:1.8.0_72] at sun.security.ssl.InputRecord.read(InputRecord.java:503) ~[na:1.8.0_72] at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) ~[na:1.8.0_72] at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930) ~[na:1.8.0_72] at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) ~[na:1.8.0_72] at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) ~[na:1.8.0_72] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:53) ~[apache-cassandra-2.2.4.jar:2.2.4] at
[jira] [Updated] (CASSANDRA-11826) Repairs failing for some of bigger tables
[ https://issues.apache.org/jira/browse/CASSANDRA-11826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vin01 updated CASSANDRA-11826: -- Description: Architecture for the same :- Cassandra version 2.2.4 3 Nodes - Node-1, Node-2, Node-3, repair started on Node-1 and following logs are also from Node-1 itself. Node-2 and Node-3 are in same data-center, and Node-1 is in a different data-center. Value of streaming_socket_timeout_in_ms is set to 1 Hour which is default for version 2.2.4 Logs for the same :- WARN [RepairJobTask:5] 2016-05-18 02:25:31,137 RepairJob.java:162 - [repair #5a855825-1cb7-11e6-9f5f-b717b380ffdd] TABLE_NAME sync failed ERROR [RepairJobTask:5] 2016-05-18 02:25:31,138 RepairSession.java:290 - [repair #5a855825-1cb7-11e6-9f5f-b717b380ffdd] Session completed with the following error org.apache.cassandra.exceptions.RepairException: [repair #5a855825-1cb7-11e6-9f5f-b717b380ffdd on KEYSPACE_NAME/TABLE_NAME, (-7892648649079895028,-7818964903881740266]] Sync failed between /Node-3 and /Node-2 at org.apache.cassandra.repair.RemoteSyncTask.syncComplete(RemoteSyncTask.java:67) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:211) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:415) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:163) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] ERROR [RepairJobTask:5] 2016-05-18 02:25:31,138 RepairRunnable.java:243 - Repair session 5a855825-1cb7-11e6-9f5f-b717b380ffdd for range (-7892648649079895028,-7818964903881740266] failed with error [repair #5a855825-1cb7-11e6-9f5f-b717b380ffdd on KEYSPACE_NA$ org.apache.cassandra.exceptions.RepairException: [repair #5a855825-1cb7-11e6-9f5f-b717b380ffdd on KEYSPACE_NAME/TABLE_NAME, (-7892648649079895028,-7818964903881740266]] Sync failed between /Node-3 and /Node-2 at org.apache.cassandra.repair.RemoteSyncTask.syncComplete(RemoteSyncTask.java:67) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:211) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:415) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:163) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] ERROR [STREAM-IN-/Node-2] 2016-05-18 02:25:33,296 StreamSession.java:524 - [Stream #efe521b0-1cb8-11e6-9f5f-b717b380ffdd] Streaming error occurred java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.8.0_72] at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[na:1.8.0_72] at java.net.SocketInputStream.read(SocketInputStream.java:170) ~[na:1.8.0_72] at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[na:1.8.0_72] at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) ~[na:1.8.0_72] at sun.security.ssl.InputRecord.read(InputRecord.java:503) ~[na:1.8.0_72] at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) ~[na:1.8.0_72] at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930) ~[na:1.8.0_72] at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) ~[na:1.8.0_72] at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) ~[na:1.8.0_72] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:53) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] INFO [STREAM-IN-/Node-2] 2016-05-18 02:25:33,296 StreamResultFuture.java:182 -
[jira] [Issue Comment Deleted] (CASSANDRA-11826) Repairs failing for some of bigger tables
[ https://issues.apache.org/jira/browse/CASSANDRA-11826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vin01 updated CASSANDRA-11826: -- Comment: was deleted (was: Architecture for the same :- Cassandra version 2.2.4 3 Nodes - Node-1, Node-2, Node-3, repair started on Node-1 and following logs are also from Node-1 itself. Node-2 and Node-3 in same data-center, and Node-1 is in a different data-center. Value of streaming_socket_timeout_in_ms is set to 1 Hour which is default for version 2.2.4 Logs for the same :- WARN [RepairJobTask:5] 2016-05-18 02:25:31,137 RepairJob.java:162 - [repair #5a855825-1cb7-11e6-9f5f-b717b380ffdd] TABLE_NAME sync failed ERROR [RepairJobTask:5] 2016-05-18 02:25:31,138 RepairSession.java:290 - [repair #5a855825-1cb7-11e6-9f5f-b717b380ffdd] Session completed with the following error org.apache.cassandra.exceptions.RepairException: [repair #5a855825-1cb7-11e6-9f5f-b717b380ffdd on KEYSPACE_NAME/TABLE_NAME, (-7892648649079895028,-7818964903881740266]] Sync failed between /Node-3 and /Node-2 at org.apache.cassandra.repair.RemoteSyncTask.syncComplete(RemoteSyncTask.java:67) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:211) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:415) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:163) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] ERROR [RepairJobTask:5] 2016-05-18 02:25:31,138 RepairRunnable.java:243 - Repair session 5a855825-1cb7-11e6-9f5f-b717b380ffdd for range (-7892648649079895028,-7818964903881740266] failed with error [repair #5a855825-1cb7-11e6-9f5f-b717b380ffdd on KEYSPACE_NA$ org.apache.cassandra.exceptions.RepairException: [repair #5a855825-1cb7-11e6-9f5f-b717b380ffdd on KEYSPACE_NAME/TABLE_NAME, (-7892648649079895028,-7818964903881740266]] Sync failed between /Node-3 and /Node-2 at org.apache.cassandra.repair.RemoteSyncTask.syncComplete(RemoteSyncTask.java:67) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:211) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:415) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:163) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] ERROR [STREAM-IN-/Node-2] 2016-05-18 02:25:33,296 StreamSession.java:524 - [Stream #efe521b0-1cb8-11e6-9f5f-b717b380ffdd] Streaming error occurred java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.8.0_72] at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[na:1.8.0_72] at java.net.SocketInputStream.read(SocketInputStream.java:170) ~[na:1.8.0_72] at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[na:1.8.0_72] at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) ~[na:1.8.0_72] at sun.security.ssl.InputRecord.read(InputRecord.java:503) ~[na:1.8.0_72] at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) ~[na:1.8.0_72] at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930) ~[na:1.8.0_72] at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) ~[na:1.8.0_72] at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) ~[na:1.8.0_72] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:53) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] INFO [STREAM-IN-/Node-2] 2016-05-18 02:25:33,296
[jira] [Commented] (CASSANDRA-11826) Repairs failing for some of bigger tables
[ https://issues.apache.org/jira/browse/CASSANDRA-11826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288688#comment-15288688 ] vin01 commented on CASSANDRA-11826: --- Architecture for the same :- Cassandra version 2.2.4 3 Nodes - Node-1, Node-2, Node-3, repair started on Node-1 and following logs are also from Node-1 itself. Node-2 and Node-3 in same data-center, and Node-1 is in a different data-center. Value of streaming_socket_timeout_in_ms is set to 1 Hour which is default for version 2.2.4 Logs for the same :- WARN [RepairJobTask:5] 2016-05-18 02:25:31,137 RepairJob.java:162 - [repair #5a855825-1cb7-11e6-9f5f-b717b380ffdd] TABLE_NAME sync failed ERROR [RepairJobTask:5] 2016-05-18 02:25:31,138 RepairSession.java:290 - [repair #5a855825-1cb7-11e6-9f5f-b717b380ffdd] Session completed with the following error org.apache.cassandra.exceptions.RepairException: [repair #5a855825-1cb7-11e6-9f5f-b717b380ffdd on KEYSPACE_NAME/TABLE_NAME, (-7892648649079895028,-7818964903881740266]] Sync failed between /Node-3 and /Node-2 at org.apache.cassandra.repair.RemoteSyncTask.syncComplete(RemoteSyncTask.java:67) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:211) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:415) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:163) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] ERROR [RepairJobTask:5] 2016-05-18 02:25:31,138 RepairRunnable.java:243 - Repair session 5a855825-1cb7-11e6-9f5f-b717b380ffdd for range (-7892648649079895028,-7818964903881740266] failed with error [repair #5a855825-1cb7-11e6-9f5f-b717b380ffdd on KEYSPACE_NA$ org.apache.cassandra.exceptions.RepairException: [repair #5a855825-1cb7-11e6-9f5f-b717b380ffdd on KEYSPACE_NAME/TABLE_NAME, (-7892648649079895028,-7818964903881740266]] Sync failed between /Node-3 and /Node-2 at org.apache.cassandra.repair.RemoteSyncTask.syncComplete(RemoteSyncTask.java:67) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:211) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:415) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:163) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] ERROR [STREAM-IN-/Node-2] 2016-05-18 02:25:33,296 StreamSession.java:524 - [Stream #efe521b0-1cb8-11e6-9f5f-b717b380ffdd] Streaming error occurred java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.8.0_72] at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[na:1.8.0_72] at java.net.SocketInputStream.read(SocketInputStream.java:170) ~[na:1.8.0_72] at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[na:1.8.0_72] at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) ~[na:1.8.0_72] at sun.security.ssl.InputRecord.read(InputRecord.java:503) ~[na:1.8.0_72] at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) ~[na:1.8.0_72] at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930) ~[na:1.8.0_72] at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) ~[na:1.8.0_72] at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) ~[na:1.8.0_72] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:53) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] INFO [STREAM-IN-/Node-2] 2016-05-18 02:25:33,296
[jira] [Created] (CASSANDRA-11826) Repairs failing for some of bigger tables
vin01 created CASSANDRA-11826: - Summary: Repairs failing for some of bigger tables Key: CASSANDRA-11826 URL: https://issues.apache.org/jira/browse/CASSANDRA-11826 Project: Cassandra Issue Type: Bug Components: Streaming and Messaging Environment: Centos 6 Reporter: vin01 Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
