[
https://issues.apache.org/jira/browse/CASSANDRA-13546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Petrov updated CASSANDRA-13546:
------------------------------------
Description:
Getting error after adding a node in existing cluster, error are coming after 1
hour after started service on seed node and due to this load got increased
vastly on seed node not on new node.
Below errors from system.log on seed node:
{code}
ERROR [NonPeriodicTasks:1] 2017-05-18 10:03:01,819 CassandraDaemon.java:153 -
Exception in thread Thread[NonPeriodicTasks:1,5,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.free(Memory.java:300)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.utils.obs.OffHeapBitSet.close(OffHeapBitSet.java:143)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.utils.BloomFilter.close(BloomFilter.java:116)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.SSTableReader$6.run(SSTableReader.java:645)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
~[na:1.7.0_71]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_71]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown
Source) ~[na:1.7.0_71]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source) ~[na:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[na:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[na:1.7.0_71]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
INFO [CompactionExecutor:162] 2017-05-18 10:03:01,820
ColumnFamilyStore.java:840 - Enqueuing flush of compactions_in_progress: 164
(0%) on-heap, 0 (0%) off-heap
INFO [MemtableFlushWriter:152] 2017-05-18 10:03:01,821 Memtable.java:325 -
Writing Memtable-compactions_in_progress@731506998(0 serialized bytes, 1 ops,
0%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:152] 2017-05-18 10:03:01,825 Memtable.java:364 -
Completed flushing
/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-52434-Data.db
(42 bytes) for commitlog position ReplayPosition(segmentId=1495106933696,
position=9156004)
ERROR [CompactionExecutor:162] 2017-05-18 10:03:01,829 CassandraDaemon.java:153
-
Exception in thread Thread[CompactionExecutor:162,1,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.free(Memory.java:300)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.utils.obs.OffHeapBitSet.close(OffHeapBitSet.java:143)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.utils.BloomFilter.close(BloomFilter.java:116)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:345)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.SSTableRewriter.abort(SSTableRewriter.java:198)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:204)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:75)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:232)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
~[na:1.7.0_71]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[na:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[na:1.7.0_71]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
INFO [main] 2017-05-18 11:06:56,032 CassandraDaemon.java:89 - Hostname:
csdb2-dl.intermesh.net
INFO [main] 2017-05-18 11:06:56,061 YamlConfigurationLoader.java:92 - Loading
settings from file:/etc/cassandra/default.conf/cassandra.yaml
{code}
was:
Getting error after adding a node in existing cluster, error are coming after 1
hour after started service on seed node and due to this load got increased
vastly on seed node not on new node.
Below errors from system.log on seed node:
ERROR [NonPeriodicTasks:1] 2017-05-18 10:03:01,819 CassandraDaemon.java:153 -
Exception in thread Thread[NonPeriodicTasks:1,5,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.free(Memory.java:300)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.utils.obs.OffHeapBitSet.close(OffHeapBitSet.java:143)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.utils.BloomFilter.close(BloomFilter.java:116)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.SSTableReader$6.run(SSTableReader.java:645)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
~[na:1.7.0_71]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_71]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown
Source) ~[na:1.7.0_71]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source) ~[na:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[na:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[na:1.7.0_71]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
INFO [CompactionExecutor:162] 2017-05-18 10:03:01,820
ColumnFamilyStore.java:840 - Enqueuing flush of compactions_in_progress: 164
(0%) on-heap, 0 (0%) off-heap
INFO [MemtableFlushWriter:152] 2017-05-18 10:03:01,821 Memtable.java:325 -
Writing Memtable-compactions_in_progress@731506998(0 serialized bytes, 1 ops,
0%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:152] 2017-05-18 10:03:01,825 Memtable.java:364 -
Completed flushing
/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-52434-Data.db
(42 bytes) for commitlog position ReplayPosition(segmentId=1495106933696,
position=9156004)
ERROR [CompactionExecutor:162] 2017-05-18 10:03:01,829 CassandraDaemon.java:153
-
Exception in thread Thread[CompactionExecutor:162,1,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.free(Memory.java:300)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.utils.obs.OffHeapBitSet.close(OffHeapBitSet.java:143)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.utils.BloomFilter.close(BloomFilter.java:116)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:345)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.SSTableRewriter.abort(SSTableRewriter.java:198)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:204)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:75)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:232)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
~[na:1.7.0_71]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[na:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[na:1.7.0_71]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
INFO [main] 2017-05-18 11:06:56,032 CassandraDaemon.java:89 - Hostname:
csdb2-dl.intermesh.net
INFO [main] 2017-05-18 11:06:56,061 YamlConfigurationLoader.java:92 - Loading
settings from file:/etc/cassandra/default.conf/cassandra.yaml
@
> Getting error while adding a node in existing cluster
> -----------------------------------------------------
>
> Key: CASSANDRA-13546
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13546
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: Unix
> Reporter: Sonu Gupta
> Priority: Blocker
> Fix For: 2.1.2
>
>
> Getting error after adding a node in existing cluster, error are coming after
> 1 hour after started service on seed node and due to this load got increased
> vastly on seed node not on new node.
> Below errors from system.log on seed node:
> {code}
> ERROR [NonPeriodicTasks:1] 2017-05-18 10:03:01,819 CassandraDaemon.java:153 -
> Exception in thread Thread[NonPeriodicTasks:1,5,main]
> java.lang.AssertionError: null
> at org.apache.cassandra.io.util.Memory.free(Memory.java:300)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.utils.obs.OffHeapBitSet.close(OffHeapBitSet.java:143)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.utils.BloomFilter.close(BloomFilter.java:116)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.io.sstable.SSTableReader$6.run(SSTableReader.java:645)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> Source) ~[na:1.7.0_71]
> at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_71]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown
> Source) ~[na:1.7.0_71]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source) ~[na:1.7.0_71]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> [na:1.7.0_71]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> [na:1.7.0_71]
> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
> INFO [CompactionExecutor:162] 2017-05-18 10:03:01,820
> ColumnFamilyStore.java:840 - Enqueuing flush of compactions_in_progress: 164
> (0%) on-heap, 0 (0%) off-heap
> INFO [MemtableFlushWriter:152] 2017-05-18 10:03:01,821 Memtable.java:325 -
> Writing Memtable-compactions_in_progress@731506998(0 serialized bytes, 1 ops,
> 0%/0% of on/off-heap limit)
> INFO [MemtableFlushWriter:152] 2017-05-18 10:03:01,825 Memtable.java:364 -
> Completed flushing
> /var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-52434-Data.db
> (42 bytes) for commitlog position ReplayPosition(segmentId=1495106933696,
> position=9156004)
> ERROR [CompactionExecutor:162] 2017-05-18 10:03:01,829
> CassandraDaemon.java:153 -
> Exception in thread Thread[CompactionExecutor:162,1,main]
> java.lang.AssertionError: null
> at org.apache.cassandra.io.util.Memory.free(Memory.java:300)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.utils.obs.OffHeapBitSet.close(OffHeapBitSet.java:143)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.utils.BloomFilter.close(BloomFilter.java:116)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:345)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.abort(SSTableRewriter.java:198)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:204)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:75)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:232)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> Source) ~[na:1.7.0_71]
> at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_71]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> [na:1.7.0_71]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> [na:1.7.0_71]
> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
> INFO [main] 2017-05-18 11:06:56,032 CassandraDaemon.java:89 - Hostname:
> csdb2-dl.intermesh.net
> INFO [main] 2017-05-18 11:06:56,061 YamlConfigurationLoader.java:92 -
> Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
