Re: New node got stuck joining the cluster after a while
Thanks very much for answering!
Do you think that after failing the joining task of a node to the cluster
should I run some repairs and cleanups?
Thanks!
On Tue, Apr 28, 2015 at 5:13 AM, Carlos Rolo r...@pythian.com wrote:
Hi,
The 2.1.x series is not recommeded for use, especially the first versions.
I would downgrade to 2.0.14 or if must stay on 2.1 upgrade your cluster to
2.1.4 or the imminent release of 2.1.5.
This mailing list as a few tips how to deal with the 2.1.x releases, but
the best way is indeed a downgrade or wait for the 2.1.5.
Regards,
Carlos Juzarte Rolo
Cassandra Consultant
Pythian - Love your data
rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
http://linkedin.com/in/carlosjuzarterolo*
Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
www.pythian.com
On Tue, Apr 28, 2015 at 3:30 AM, Analia Lorenzatto
analialorenza...@gmail.com wrote:
Hello guys,
I have a cluster comprised of 2 nodes, configured with vnodes. Using
2.1.0-2 version of cassandra.
And I am facing an issue when I want to joing a new node to the cluster.
At first starting joining but then it got stuck:
UN 1x.x.x.x 348.11 GB 256 100.0% 1c
UN 1x.x.x.x 342.74 GB 256 100.0% 1c
UJ 1x.x.x.x 26.86 GB 256 ? 1c
I can see some errors on the already working nodes:
*WARN [SharedPool-Worker-7] 2015-04-27 17:41:16,060
SliceQueryFilter.java:236 - Read 5001 live and 66548 tombstoned cells in
usmc.userpixel (see tombstone_warn_threshol*
*d). 5000 columns was requested, slices=[-],
delInfo={deletedAt=-9223372036854775808, localDeletion=2147483647
2147483647}*
*WARN [SharedPool-Worker-32] 2015-04-27 17:41:16,668
SliceQueryFilter.java:236 - Read 2012 live and 30440 tombstoned cells in
usmc.userpixel (see tombstone_warn_thresho*
*ld). 5001 columns was requested,
slices=[b6d051df-0a8f-4c13-b93c-1b4ff0d82b8d:date-],
delInfo={deletedAt=-9223372036854775808, localDeletion=2147483647}*
*ERROR [CompactionExecutor:35638] 2015-04-27 19:06:07,613
CassandraDaemon.java:166 - Exception in thread
Thread[CompactionExecutor:35638,1,main]*
*java.lang.AssertionError: Memory was freed*
*at
org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:281)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at org.apache.cassandra.io.util.Memory.getInt(Memory.java:233)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.io.sstable.IndexSummary.getPositionInSummary(IndexSummary.java:118)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.io.sstable.IndexSummary.getKey(IndexSummary.java:123)
~[apache-cassandra-2.1.0.jar:2.1.0]*
* at
org.apache.cassandra.io.sstable.IndexSummary.binarySearch(IndexSummary.java:92)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.io.sstable.SSTableReader.getSampleIndexesForRanges(SSTableReader.java:1209)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.io.sstable.SSTableReader.estimatedKeysForRanges(SSTableReader.java:1165)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy.worthDroppingTombstones(AbstractCompactionStrategy.java:328)
~[apache-cassandra-2.1.0.jar:2.1.0*
*]*
*at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.findDroppableSSTable(LeveledCompactionStrategy.java:365)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getMaximalTask(LeveledCompactionStrategy.java:127)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:112)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:229)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_51]*
*at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_51]*
*at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_51]*
*at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_51]*
*at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]*
But I do not see any warning or error message in logs of the joining
nodes. I just see an exception there when I run: nodetool info:
root@:~# nodetool info
ID : f5e49647-59fa-474f-b6af-9f65abc43581
Gossip active: true
Thrift active: false
Native Transport active: false
Load : 26.86 GB
Generation No: 1430163258
Uptime (seconds) : 18799
Heap Memory (MB) : 4185.15 / 7566.00
error: null
-- StackTrace --
java.lang.AssertionError
at
Re: New node got stuck joining the cluster after a while
Hi,
The 2.1.x series is not recommeded for use, especially the first versions.
I would downgrade to 2.0.14 or if must stay on 2.1 upgrade your cluster to
2.1.4 or the imminent release of 2.1.5.
This mailing list as a few tips how to deal with the 2.1.x releases, but
the best way is indeed a downgrade or wait for the 2.1.5.
Regards,
Carlos Juzarte Rolo
Cassandra Consultant
Pythian - Love your data
rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
http://linkedin.com/in/carlosjuzarterolo*
Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
www.pythian.com
On Tue, Apr 28, 2015 at 3:30 AM, Analia Lorenzatto
analialorenza...@gmail.com wrote:
Hello guys,
I have a cluster comprised of 2 nodes, configured with vnodes. Using
2.1.0-2 version of cassandra.
And I am facing an issue when I want to joing a new node to the cluster.
At first starting joining but then it got stuck:
UN 1x.x.x.x 348.11 GB 256 100.0% 1c
UN 1x.x.x.x 342.74 GB 256 100.0% 1c
UJ 1x.x.x.x 26.86 GB 256 ? 1c
I can see some errors on the already working nodes:
*WARN [SharedPool-Worker-7] 2015-04-27 17:41:16,060
SliceQueryFilter.java:236 - Read 5001 live and 66548 tombstoned cells in
usmc.userpixel (see tombstone_warn_threshol*
*d). 5000 columns was requested, slices=[-],
delInfo={deletedAt=-9223372036854775808, localDeletion=2147483647
2147483647}*
*WARN [SharedPool-Worker-32] 2015-04-27 17:41:16,668
SliceQueryFilter.java:236 - Read 2012 live and 30440 tombstoned cells in
usmc.userpixel (see tombstone_warn_thresho*
*ld). 5001 columns was requested,
slices=[b6d051df-0a8f-4c13-b93c-1b4ff0d82b8d:date-],
delInfo={deletedAt=-9223372036854775808, localDeletion=2147483647}*
*ERROR [CompactionExecutor:35638] 2015-04-27 19:06:07,613
CassandraDaemon.java:166 - Exception in thread
Thread[CompactionExecutor:35638,1,main]*
*java.lang.AssertionError: Memory was freed*
*at
org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:281)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at org.apache.cassandra.io.util.Memory.getInt(Memory.java:233)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.io.sstable.IndexSummary.getPositionInSummary(IndexSummary.java:118)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.io.sstable.IndexSummary.getKey(IndexSummary.java:123)
~[apache-cassandra-2.1.0.jar:2.1.0]*
* at
org.apache.cassandra.io.sstable.IndexSummary.binarySearch(IndexSummary.java:92)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.io.sstable.SSTableReader.getSampleIndexesForRanges(SSTableReader.java:1209)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.io.sstable.SSTableReader.estimatedKeysForRanges(SSTableReader.java:1165)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy.worthDroppingTombstones(AbstractCompactionStrategy.java:328)
~[apache-cassandra-2.1.0.jar:2.1.0*
*]*
*at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.findDroppableSSTable(LeveledCompactionStrategy.java:365)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getMaximalTask(LeveledCompactionStrategy.java:127)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:112)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:229)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_51]*
*at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_51]*
*at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_51]*
*at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_51]*
*at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]*
But I do not see any warning or error message in logs of the joining
nodes. I just see an exception there when I run: nodetool info:
root@:~# nodetool info
ID : f5e49647-59fa-474f-b6af-9f65abc43581
Gossip active: true
Thrift active: false
Native Transport active: false
Load : 26.86 GB
Generation No: 1430163258
Uptime (seconds) : 18799
Heap Memory (MB) : 4185.15 / 7566.00
error: null
-- StackTrace --
java.lang.AssertionError
at
org.apache.cassandra.locator.TokenMetadata.getTokens(TokenMetadata.java:440)
at
org.apache.cassandra.service.StorageService.getTokens(StorageService.java:2079)
at
org.apache.cassandra.service.StorageService.getTokens(StorageService.java:2068)
New node got stuck joining the cluster after a while
Hello guys,
I have a cluster comprised of 2 nodes, configured with vnodes. Using
2.1.0-2 version of cassandra.
And I am facing an issue when I want to joing a new node to the cluster.
At first starting joining but then it got stuck:
UN 1x.x.x.x 348.11 GB 256 100.0% 1c
UN 1x.x.x.x 342.74 GB 256 100.0% 1c
UJ 1x.x.x.x 26.86 GB 256 ? 1c
I can see some errors on the already working nodes:
*WARN [SharedPool-Worker-7] 2015-04-27 17:41:16,060
SliceQueryFilter.java:236 - Read 5001 live and 66548 tombstoned cells in
usmc.userpixel (see tombstone_warn_threshol*
*d). 5000 columns was requested, slices=[-],
delInfo={deletedAt=-9223372036854775808, localDeletion=2147483647
2147483647}*
*WARN [SharedPool-Worker-32] 2015-04-27 17:41:16,668
SliceQueryFilter.java:236 - Read 2012 live and 30440 tombstoned cells in
usmc.userpixel (see tombstone_warn_thresho*
*ld). 5001 columns was requested,
slices=[b6d051df-0a8f-4c13-b93c-1b4ff0d82b8d:date-],
delInfo={deletedAt=-9223372036854775808, localDeletion=2147483647}*
*ERROR [CompactionExecutor:35638] 2015-04-27 19:06:07,613
CassandraDaemon.java:166 - Exception in thread
Thread[CompactionExecutor:35638,1,main]*
*java.lang.AssertionError: Memory was freed*
*at
org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:281)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at org.apache.cassandra.io.util.Memory.getInt(Memory.java:233)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.io.sstable.IndexSummary.getPositionInSummary(IndexSummary.java:118)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.io.sstable.IndexSummary.getKey(IndexSummary.java:123)
~[apache-cassandra-2.1.0.jar:2.1.0]*
* at
org.apache.cassandra.io.sstable.IndexSummary.binarySearch(IndexSummary.java:92)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.io.sstable.SSTableReader.getSampleIndexesForRanges(SSTableReader.java:1209)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.io.sstable.SSTableReader.estimatedKeysForRanges(SSTableReader.java:1165)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy.worthDroppingTombstones(AbstractCompactionStrategy.java:328)
~[apache-cassandra-2.1.0.jar:2.1.0*
*]*
*at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.findDroppableSSTable(LeveledCompactionStrategy.java:365)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getMaximalTask(LeveledCompactionStrategy.java:127)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:112)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:229)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_51]*
*at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_51]*
*at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_51]*
*at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_51]*
*at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]*
But I do not see any warning or error message in logs of the joining
nodes. I just see an exception there when I run: nodetool info:
root@:~# nodetool info
ID : f5e49647-59fa-474f-b6af-9f65abc43581
Gossip active: true
Thrift active: false
Native Transport active: false
Load : 26.86 GB
Generation No: 1430163258
Uptime (seconds) : 18799
Heap Memory (MB) : 4185.15 / 7566.00
error: null
-- StackTrace --
java.lang.AssertionError
at
org.apache.cassandra.locator.TokenMetadata.getTokens(TokenMetadata.java:440)
at
org.apache.cassandra.service.StorageService.getTokens(StorageService.java:2079)
at
org.apache.cassandra.service.StorageService.getTokens(StorageService.java:2068)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
at
