Re: Too many compactions, maybe keyspace system?

[Shuo Chen]

Hi, Robert,

I think I found the cause of the too many compactions. I used jmap to dump
the heap and used Eclipse memory analyzer plugin to extract the heap.

In previous reply, It shows the there are too many pending jobs in the
Blocking queue. I checked the cf of the compaction task object. There are
many cfs concerning some empty cfs I created before.

I created 5 keyspaces and about 100 cfs months by cassandra-cli ago and
didnot put any data yet. In  fact, there is only 1 keypaces I created
containing data and the other 5 keyspaces are empty.

When I droped these 5 keyspaces and restarted the high compaction node, It
runs normally with normal mount of compactions.

So maybe there are some bugs of compaction for empty columnfamily?

On Wed, Jan 13, 2016 at 2:39 AM, Robert Coli  wrote:

> On Mon, Jan 11, 2016 at 9:12 PM, Shuo Chen  wrote:
>
>> I have a assumption that, lots of pending compaction tasks jam the memory
>> and raise full gc. The full chokes the process and slows down compaction.
>> And this causes more pending compaction tasks and more pressure on memory.
>>
>
> The question is why there are so many pending compactions, because your
> log doesn't show that much compaction is happening. What keyspaces /
> columnfamilies do you expect to be compacting, and how many SSTables do
> they contain?
>
>
>> Is there a method to list the concrete details of pending compaction
>> tasks?
>>
>
> Nope.
>
> For the record, this type of extended operational debugging is often best
> carried out interactively on #cassandra on freenode IRC.. :)
>
> =Rob
>



-- 
*陈硕* *Shuo Chen*
chenatu2...@gmail.com
chens...@whaty.com

Re: Too many compactions, maybe keyspace system?

[DuyHai Doan]

Interesting, maybe it worths filing a JIRA. Empty tables should not slow
down compaction of other tables

On Sat, Jan 16, 2016 at 10:33 AM, Shuo Chen  wrote:

> Hi, Robert,
>
> I think I found the cause of the too many compactions. I used jmap to dump
> the heap and used Eclipse memory analyzer plugin to extract the heap.
>
> In previous reply, It shows the there are too many pending jobs in the
> Blocking queue. I checked the cf of the compaction task object. There are
> many cfs concerning some empty cfs I created before.
>
> I created 5 keyspaces and about 100 cfs months by cassandra-cli ago and
> didnot put any data yet. In  fact, there is only 1 keypaces I created
> containing data and the other 5 keyspaces are empty.
>
> When I droped these 5 keyspaces and restarted the high compaction node, It
> runs normally with normal mount of compactions.
>
> So maybe there are some bugs of compaction for empty columnfamily?
>
> On Wed, Jan 13, 2016 at 2:39 AM, Robert Coli  wrote:
>
>> On Mon, Jan 11, 2016 at 9:12 PM, Shuo Chen  wrote:
>>
>>> I have a assumption that, lots of pending compaction tasks jam the
>>> memory and raise full gc. The full chokes the process and slows down
>>> compaction. And this causes more pending compaction tasks and more pressure
>>> on memory.
>>>
>>
>> The question is why there are so many pending compactions, because your
>> log doesn't show that much compaction is happening. What keyspaces /
>> columnfamilies do you expect to be compacting, and how many SSTables do
>> they contain?
>>
>>
>>> Is there a method to list the concrete details of pending compaction
>>> tasks?
>>>
>>
>> Nope.
>>
>> For the record, this type of extended operational debugging is often best
>> carried out interactively on #cassandra on freenode IRC.. :)
>>
>> =Rob
>>
>
>
>
> --
> *陈硕* *Shuo Chen*
> chenatu2...@gmail.com
> chens...@whaty.com
>

endless full gc on one node

[Kai Wang]

Hi,

Recently I saw some strange behavior on one of the nodes of a 3-node
cluster. A while ago I created a table and put some data (about 150M) in it
for testing. A few days ago I started to import full data into that table
using normal cql INSERT statements. As soon as inserting started, one node
started non-stop full GC. The other two nodes were totally fine. I stopped
the inserting process, restarted C* on all the nodes. All nodes are fine.
But once I started inserting again, full GC kicked in on that node within a
minute.The insertion speed is moderate. Again, the other two nodes were
fine. I tried this process a couple of times. Every time the same node
jumped into full GC. I even rebooted all the boxes. I checked system.log
but found no errors or warnings before full GC started.

Finally I deleted and recreated the table. All of sudden the problem went
away. The only thing I can think of is that table was created using STCS.
After I inserted 150M data into it, I switched it to LCS. Then I ran
incremental repair a couple of times. I saw validation and normal
compaction on that table as expected. When I recreated the table, I created
it with LCS.

I don't have the problem any more but just want to share the experience.
Maybe someone has an theory on this? BTW I am running C* 2.2.4 with CentOS
7 and Java 8. All boxes have the identical configurations.

Thanks.

Re: endless full gc on one node

[DuyHai Doan]

"As soon as inserting started, one node started non-stop full GC. The other
two nodes were totally fine"

Just a guest, how did you insert data ? Did you use Batch statements ?

On Sat, Jan 16, 2016 at 10:12 PM, Kai Wang  wrote:

> Hi,
>
> Recently I saw some strange behavior on one of the nodes of a 3-node
> cluster. A while ago I created a table and put some data (about 150M) in it
> for testing. A few days ago I started to import full data into that table
> using normal cql INSERT statements. As soon as inserting started, one node
> started non-stop full GC. The other two nodes were totally fine. I stopped
> the inserting process, restarted C* on all the nodes. All nodes are fine.
> But once I started inserting again, full GC kicked in on that node within a
> minute.The insertion speed is moderate. Again, the other two nodes were
> fine. I tried this process a couple of times. Every time the same node
> jumped into full GC. I even rebooted all the boxes. I checked system.log
> but found no errors or warnings before full GC started.
>
> Finally I deleted and recreated the table. All of sudden the problem went
> away. The only thing I can think of is that table was created using STCS.
> After I inserted 150M data into it, I switched it to LCS. Then I ran
> incremental repair a couple of times. I saw validation and normal
> compaction on that table as expected. When I recreated the table, I created
> it with LCS.
>
> I don't have the problem any more but just want to share the experience.
> Maybe someone has an theory on this? BTW I am running C* 2.2.4 with CentOS
> 7 and Java 8. All boxes have the identical configurations.
>
> Thanks.
>

Re: In UJ status for over a week trying to rejoin cluster in Cassandra 3.0.1

[Carlos Fernando Scheidecker Antunes]

DuyHai,

Nothing wrong on the logs either.



> nodetool tpstats
> Pool NameActive   Pending  Completed   Blocked  All 
> time blocked
> MutationStage 0 0  11464 0
>  0
> ViewMutationStage 0 0  0 0
>  0
> ReadStage 0 0  0 0
>  0
> RequestResponseStage  0 0 10 0
>  0
> ReadRepairStage   0 0  0 0
>  0
> CounterMutationStage  0 0  0 0
>  0
> MiscStage 0 0  0 0
>  0
> CompactionExecutor0 0683 0
>  0
> MemtableReclaimMemory 0 0357 0
>  0
> PendingRangeCalculator0 0  5 0
>  0
> GossipStage   0 02682208 0
>  0
> SecondaryIndexManagement  0 0  0 0
>  0
> HintsDispatcher   0 0  0 0
>  0
> MigrationStage0 0  0 0
>  0
> MemtablePostFlush 0 0375 0
>  0
> ValidationExecutor0 0  0 0
>  0
> Sampler   0 0  0 0
>  0
> MemtableFlushWriter   0 0357 0
>  0
> InternalResponseStage 0 0  0 0
>  0
> AntiEntropyStage  0 0  0 0
>  0
> CacheCleanupExecutor  0 0  0 0
>  0
> 
> Message type   Dropped
> READ 0
> RANGE_SLICE  0
> _TRACE   0
> HINT 0
> MUTATION 0
> COUNTER_MUTATION 0
> BATCH_STORE  0
> BATCH_REMOVE 0
> REQUEST_RESPONSE 0
> PAGED_RANGE  0
> READ_REPAIR  0


On Tue, 2016-01-12 at 13:05 +0100, DuyHai Doan wrote:
> Oh, sorry, did not notice the version in the title. Did you check the
> system.log to verify if there isn't any Exception related to data
> streaming ? What is the output of "nodetool tpstats" ?
> 
> 
> On Tue, Jan 12, 2016 at 1:00 PM, DuyHai Doan 
> wrote:
> 
> What is your Cassandra version ? In earlier versions there was
> some issues with streaming that can make the joining process
> stuck.
> 
> 
> On Mon, Jan 11, 2016 at 6:57 AM, Carlos A
>  wrote:
> 
> Hello all,
> 
> 
> 
> I have a small dev environment with 4 machines. One of
> them, I had it removed (.33) from the cluster because
> I wanted to upgrade its HD to a SSD. I then
> reinstalled it and tried to join. It is on UJ status
> for a week now and no changes.
> 
> 
> I had tried node-repair etc but nothing.
> 
> 
> nodetool status output
> 
> 
> Datacenter: DC1
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address   Load   Tokens   OwnsHost
> ID   Rack
> UN  192.168.1.30  16.13 MB   256  ?
> 0e524b1c-b254-45d0-98ee-63b8f34a8531  RAC1
> UN  192.168.1.31  20.12 MB   256  ?
> 1f8000f5-026c-42c7-8189-cf19fbede566  RAC1
> UN  192.168.1.32  17.73 MB   256  ?
> 7b06f9e9-7c41-4364-ab18-f6976fd359e4  RAC1
> UJ  192.168.1.33  877.6 KB   256  ?
> 7a1507b5-198e-4a3a-a9fd-7af9e588fde2  RAC1
> 
> 
> Note: Non-system keyspaces don't have the same
> replication settings, effective ownership information
> is meaningless
> 
> 
> Any tips on fixing this?
> 
> 
> Thanks,
> 
> 
> C.
> 
> 

Run Repairs when a Node is Down

[Anuj Wadehra]

Hi 
We are on 2.0.14,RF=3 in a 3 node cluster. We use repair -pr . Recently, we 
observed that repair -pr for all nodes fails if a node is down. Then I found 
the JIRA 
https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-2290
where an intentional decision was taken to abort the repair if a replica is 
down.
I need to understand the reasoning behind aborting the repair instead of 
proceeding with available replicas.
I have following concerns with the approach:
We say that we have a fault tolerant Cassandra system such that we can afford 
single node failure because RF=3 and we read/write at QUORUM.But when a node 
goes down and we are not sure how much time will be needed to restore the node, 
entire system health is in question as gc_grace_period is approaching and we 
are not able to run repair -pr on any of the nodes.
Then there is a dilemma:
Whether to remove the faulty node well before gc grace period so that we get 
enough time to save data by repairing other two nodes?
This may cause massive streaming which may be unnecessary if we are able to 
bring back the faulty node up before gc grace period.
OR
Wait and hope that the issue will be resolved before gc grace time and we will 
have some buffer to run repair -pr on all nodes.
OR
Increase the gc grace period temporarily. Then we should have capacity planning 
to accomodate the extra storage needed for extra gc grace that may be needed in 
case of node failure scenarios.

Besides knowing the reasoning behind the decision taken in CASSANDRA-2290, I 
need to understand the recommeded approach for maintaing a fault tolerant 
system which can handle node failures such that repair can be run smoothly and 
system health is maintained at all times.

ThanksAnuj Sent from Yahoo Mail on Android