Re: Pending compactions not going down on some nodes of the cluster

[Gianluca Borello]

Thank you for your reply. To address your points:

- We are not running repairs

- Yes, we are storing timeseries-like binary blobs where data is heavily
TTLed (essentially the entire column family is incrementally refreshed with
completely new data every few days)

- I have tried with increasing compactors and even putting the compaction
throughput to unlimited, and nothing changes. Again, the key here to
remember is that if I drain the node, the compactions completely stop after
a few minutes (like they would normally do on another "healthy" node), it's
just the "pending tasks" counter that stays high, and messes up my metrics
in OpsCenter

- As another data point: even if I increase the number of resources
allocated to compactions, I can pretty much measure that the disk I/O
generated by Cassandra is essentially the same as the other nodes that have
no pending compactions. In other words, it really seems like that number of
estimated pending compactions is somehow bogus

Thanks

On Mon, Mar 21, 2016 at 9:45 PM, Fabrice Facorat 
wrote:

> Are you running repairs ?
>
> You may try:
> - increase concurrentçcompaction to 8 (max in 2.1.x)
> - increase compaction_throupghput to more than 16MB/s (48 may be a good
> start)
>
>
> What kind of data are you storing in theses tables ? timeseries ?
>
>
>
> 2016-03-21 23:37 GMT+01:00 Gianluca Borello :
> > Thank you for your reply, definitely appreciate the tip on the compressed
> > size.
> >
> > I understand your point, in fact whenever we bootstrap a new node we see
> a
> > huge number of pending compactions (in the order of thousands), and they
> > usually decrease steadily until they reach 0 in just a few hours. With
> this
> > node, however, we are way beyond that point, it has been 3 days since the
> > number of pending compaction started fluctuating around ~150 without any
> > sign of going down (I can see from Opscenter it's almost a straight line
> > starting a few hours after the bootstrap). In particular, to reply to
> your
> > point:
> >
> > - The number of sstables for this CF on this node is around 250, which
> is in
> > the same range of all the other nodes in the cluster (I counted the
> number
> > on each one of them, and every node is in the 200-400 range)
> >
> > - This theory doesn't seem to explain why, when doing "nodetool drain",
> the
> > compactions completely stop after a few minutes and I get something such
> as:
> >
> > $ nodetool compactionstats
> > pending tasks: 128
> >
> > So no compactions being executed (since there is no more write activity),
> > but the pending number is still high.
> >
> > Thanks again
> >
> >
> > On Mon, Mar 21, 2016 at 3:19 PM, Jeff Jirsa 
> > wrote:
> >>
> >> > We added a bunch of new nodes to a cluster (2.1.13) and everything
> went
> >> > fine, except for the number of pending compactions that is staying
> quite
> >> > high on a subset of the new nodes. Over the past 3 days, the pending
> >> > compactions have never been less than ~130 on such nodes, with peaks
> of
> >> > ~200.
> >>
> >> When you bootstrap with Vnodes, you end up with thousands (or tens of
> >> thousands) of sstables – with 256 Vnodes (default) * 20 sstables per
> node,
> >> your resulting node will have 5k sstables. It takes quite a while for
> >> compaction to chew through that. If you added a bunch of nodes in
> sequence,
> >> you’d have 5k on the first node, then potentailly 10k on the next, and
> could
> >> potentially keep increasing as you start streaming from nodes that have
> way
> >> too many sstables.  This is one of the reasons that many people who
> have to
> >> grow their clusters frequently try not to use vnodes.
> >>
> >> From your other email:
> >>
> >> > Also related to this point, now I'm seeing something even more odd:
> some
> >> > compactions are way bigger than the size of the column family itself,
> such
> >> > as:
> >>
> >> The size reported by compactionstats is the uncompressed size – if
> you’re
> >> using compression, it’s perfectly reasonable for 30G of data to show up
> as
> >> 118G of data during compaction.
> >>
> >> - Jeff
> >>
> >> From: Gianluca Borello
> >> Reply-To: "user@cassandra.apache.org"
> >> Date: Monday, March 21, 2016 at 12:50 PM
> >> To: "user@cassandra.apache.org"
> >> Subject: Pending compactions not going down on some nodes of the cluster
> >>
> >> Hi,
> >>
> >> We added a bunch of new nodes to a cluster (2.1.13) and everything went
> >> fine, except for the number of pending compactions that is staying quite
> >> high on a subset of the new nodes. Over the past 3 days, the pending
> >> compactions have never been less than ~130 on such nodes, with peaks of
> >> ~200. On the other nodes, they correctly fluctuate between 0 and ~20,
> which
> >> has been our norm for a long time.
> >>
> >> We are quite paranoid about pending compactions because in the past such
> >> high number caused a lot of data being brought in 

Re: Pending compactions not going down on some nodes of the cluster

[Fabrice Facorat]

Are you running repairs ?

You may try:
- increase concurrentçcompaction to 8 (max in 2.1.x)
- increase compaction_throupghput to more than 16MB/s (48 may be a good start)


What kind of data are you storing in theses tables ? timeseries ?



2016-03-21 23:37 GMT+01:00 Gianluca Borello :
> Thank you for your reply, definitely appreciate the tip on the compressed
> size.
>
> I understand your point, in fact whenever we bootstrap a new node we see a
> huge number of pending compactions (in the order of thousands), and they
> usually decrease steadily until they reach 0 in just a few hours. With this
> node, however, we are way beyond that point, it has been 3 days since the
> number of pending compaction started fluctuating around ~150 without any
> sign of going down (I can see from Opscenter it's almost a straight line
> starting a few hours after the bootstrap). In particular, to reply to your
> point:
>
> - The number of sstables for this CF on this node is around 250, which is in
> the same range of all the other nodes in the cluster (I counted the number
> on each one of them, and every node is in the 200-400 range)
>
> - This theory doesn't seem to explain why, when doing "nodetool drain", the
> compactions completely stop after a few minutes and I get something such as:
>
> $ nodetool compactionstats
> pending tasks: 128
>
> So no compactions being executed (since there is no more write activity),
> but the pending number is still high.
>
> Thanks again
>
>
> On Mon, Mar 21, 2016 at 3:19 PM, Jeff Jirsa 
> wrote:
>>
>> > We added a bunch of new nodes to a cluster (2.1.13) and everything went
>> > fine, except for the number of pending compactions that is staying quite
>> > high on a subset of the new nodes. Over the past 3 days, the pending
>> > compactions have never been less than ~130 on such nodes, with peaks of
>> > ~200.
>>
>> When you bootstrap with Vnodes, you end up with thousands (or tens of
>> thousands) of sstables – with 256 Vnodes (default) * 20 sstables per node,
>> your resulting node will have 5k sstables. It takes quite a while for
>> compaction to chew through that. If you added a bunch of nodes in sequence,
>> you’d have 5k on the first node, then potentailly 10k on the next, and could
>> potentially keep increasing as you start streaming from nodes that have way
>> too many sstables.  This is one of the reasons that many people who have to
>> grow their clusters frequently try not to use vnodes.
>>
>> From your other email:
>>
>> > Also related to this point, now I'm seeing something even more odd: some
>> > compactions are way bigger than the size of the column family itself, such
>> > as:
>>
>> The size reported by compactionstats is the uncompressed size – if you’re
>> using compression, it’s perfectly reasonable for 30G of data to show up as
>> 118G of data during compaction.
>>
>> - Jeff
>>
>> From: Gianluca Borello
>> Reply-To: "user@cassandra.apache.org"
>> Date: Monday, March 21, 2016 at 12:50 PM
>> To: "user@cassandra.apache.org"
>> Subject: Pending compactions not going down on some nodes of the cluster
>>
>> Hi,
>>
>> We added a bunch of new nodes to a cluster (2.1.13) and everything went
>> fine, except for the number of pending compactions that is staying quite
>> high on a subset of the new nodes. Over the past 3 days, the pending
>> compactions have never been less than ~130 on such nodes, with peaks of
>> ~200. On the other nodes, they correctly fluctuate between 0 and ~20, which
>> has been our norm for a long time.
>>
>> We are quite paranoid about pending compactions because in the past such
>> high number caused a lot of data being brought in memory during some reads
>> and that triggered a chain reaction of full GCs that brought down our
>> cluster, so we try to monitor them closely.
>>
>> Some data points that should let the situation speak for itself:
>>
>> - We use LCS for all our column families
>>
>> - The cluster is operating absolutely fine and seems healthy, and every
>> node is handling pretty much the same load in terms of reads and writes.
>> Also, these nodes with higher pending compactions don't seem in any way
>> performing worse than the others
>>
>> - The pending compactions don't go down even when setting the compaction
>> throughput to unlimited for a very long time
>>
>> - This is the typical output of compactionstats and tpstats:
>>
>> $ nodetool compactionstats
>> pending tasks: 137
>>compaction type   keyspacetable completed total
>> unit   progress
>> Compaction draios   message_data6061112083946939536890
>> bytes 88.06%
>> Compaction draiosmessage_data1   26473390790   37243294809
>> bytes 71.08%
>> Active compaction remaining time :n/a
>>
>> $ nodetool tpstats
>> Pool NameActive   Pending  Completed   Blocked
>> All time blocked
>> CounterMutationStage  0 0 

com.datastax.driver.core.Connection "This should not happen and is likely a bug, please report."

[Thunder Stumpges]

Hi guys,

So, quick background, we are using Outworkers (previously WebSudos) Phantom
v 1.22.0 Which appears to be using DataStax driver 3.0.0. We are running
scala 2.10 inside Samza on Yarn (CDH 5.4.4) with Oracle JDK 8.

This is all pointing at a 3 node dev cluster of DataStax Community v 2.1.13

After running for a short time, the following error is thrown by the
Datastax driver in the Samza container:

Logger: com.datastax.driver.core.Connection
Message:
"Forcing termination of Connection[/:9042-3,
inFlight=0, closed=true]. This should not happen and is likely a bug,
please report."

There aren't really any other logs around this time, so not sure what else
I can provide. It said to "please report" so I guess I am. If what I have
provided is enough information to actually file a bug report, please let me
know and I will. Otherwise, what additional information can I collect for
this?

Thanks,
Thunder

Effectiveness of Scrub Operation vs SSTable previously marked in blacklist

[Michael Fong]

Hi, all,

We recently encountered a scenario under Cassandra 2.0 deployment. Cassandra 
detected a corrupted sstable, and when we attempt to scrub the sstable (with 
all the associated sstables), the corrupted sstable was not included in the 
sstable list. This continues until we restart Cassandra and perform sstable 
again.

After we traced the Cassandra source code, we are a bit confused with the 
effectiveness of scrubbing and SStable being marked in blacklist in Cassandra 
2.0+

It seems from previous version (Cassandra 1.2), the scrub operation would 
operate on a sstable regardless of it being previously marked. However, in 
Cassandra 2.0, the function flows seems changed.

Here is function flow that we traced in Cassandra 2.0 source code:


>From org.apache.cassandra.db.compaction.CompactionManager

...
public void performScrub(ColumnFamilyStore cfStore, final boolean 
skipCorrupted, final boolean checkData) throws InterruptedException, 
ExecutionException

{

performAllSSTableOperation(cfStore, new AllSSTablesOperation()

{
...
private void performAllSSTableOperation(final ColumnFamilyStore cfs, final 
AllSSTablesOperation operation) throws InterruptedException, ExecutionException
{
final Iterable sstables = cfs.markAllCompacting();
...

org.apache.cassandra.db. ColumnFamilyStore
...

public Iterable markAllCompacting()

{

Callable callable = new 
Callable()

{

public Iterable call() throws Exception

{

assert data.getCompacting().isEmpty() : data.getCompacting();

Iterable sstables = 
Lists.newArrayList(AbstractCompactionStrategy.filterSuspectSSTables(getSSTables()));

if (Iterables.isEmpty(sstables))

return null;
...

If it is true, would this flow - marking corrupted sstable in blacklist, defeat 
the original purpose of scrub operation?  Thanks in advanced!


Sincerely,

Michael Fong

Re: Pending compactions not going down on some nodes of the cluster

[Gianluca Borello]

Thank you for your reply, definitely appreciate the tip on the compressed
size.

I understand your point, in fact whenever we bootstrap a new node we see a
huge number of pending compactions (in the order of thousands), and they
usually decrease steadily until they reach 0 in just a few hours. With this
node, however, we are way beyond that point, it has been 3 days since the
number of pending compaction started fluctuating around ~150 without any
sign of going down (I can see from Opscenter it's almost a straight line
starting a few hours after the bootstrap). In particular, to reply to your
point:

- The number of sstables for this CF on this node is around 250, which is
in the same range of all the other nodes in the cluster (I counted the
number on each one of them, and every node is in the 200-400 range)

- This theory doesn't seem to explain why, when doing "nodetool drain", the
compactions completely stop after a few minutes and I get something such as:

$ nodetool compactionstats
pending tasks: 128

So no compactions being executed (since there is no more write activity),
but the pending number is still high.

Thanks again


On Mon, Mar 21, 2016 at 3:19 PM, Jeff Jirsa 
wrote:

> > We added a bunch of new nodes to a cluster (2.1.13) and everything went
> fine, except for the number of pending compactions that is staying quite
> high on a subset of the new nodes. Over the past 3 days, the pending
> compactions have never been less than ~130 on such nodes, with peaks of
> ~200.
>
> When you bootstrap with Vnodes, you end up with thousands (or tens of
> thousands) of sstables – with 256 Vnodes (default) * 20 sstables per node,
> your resulting node will have 5k sstables. It takes quite a while for
> compaction to chew through that. If you added a bunch of nodes in sequence,
> you’d have 5k on the first node, then potentailly 10k on the next, and
> could potentially keep increasing as you start streaming from nodes that
> have way too many sstables.  This is one of the reasons that many people
> who have to grow their clusters frequently try not to use vnodes.
>
> From your other email:
>
> > Also related to this point, now I'm seeing something even more odd:
> some compactions are way bigger than the size of the column family itself,
> such as:
>
> The size reported by compactionstats is the uncompressed size – if you’re
> using compression, it’s perfectly reasonable for 30G of data to show up as
> 118G of data during compaction.
>
> - Jeff
>
> From: Gianluca Borello
> Reply-To: "user@cassandra.apache.org"
> Date: Monday, March 21, 2016 at 12:50 PM
> To: "user@cassandra.apache.org"
> Subject: Pending compactions not going down on some nodes of the cluster
>
> Hi,
>
> We added a bunch of new nodes to a cluster (2.1.13) and everything went
> fine, except for the number of pending compactions that is staying quite
> high on a subset of the new nodes. Over the past 3 days, the pending
> compactions have never been less than ~130 on such nodes, with peaks of
> ~200. On the other nodes, they correctly fluctuate between 0 and ~20, which
> has been our norm for a long time.
>
> We are quite paranoid about pending compactions because in the past such
> high number caused a lot of data being brought in memory during some reads
> and that triggered a chain reaction of full GCs that brought down our
> cluster, so we try to monitor them closely.
>
> Some data points that should let the situation speak for itself:
>
> - We use LCS for all our column families
>
> - The cluster is operating absolutely fine and seems healthy, and every
> node is handling pretty much the same load in terms of reads and writes.
> Also, these nodes with higher pending compactions don't seem in any way
> performing worse than the others
>
> - The pending compactions don't go down even when setting the compaction
> throughput to unlimited for a very long time
>
> - This is the typical output of compactionstats and tpstats:
>
> $ nodetool compactionstats
> pending tasks: 137
>compaction type   keyspacetable completed total
>unit   progress
> Compaction draios   message_data6061112083946939536890
>   bytes 88.06%
> Compaction draiosmessage_data1   26473390790   37243294809
>   bytes 71.08%
> Active compaction remaining time :n/a
>
> $ nodetool tpstats
> Pool NameActive   Pending  Completed   Blocked
>  All time blocked
> CounterMutationStage  0 0  0 0
> 0
> ReadStage 1 0  111766844 0
> 0
> RequestResponseStage  0 0  244259493 0
> 0
> MutationStage 0 0  163268653 0
> 0
> ReadRepairStage   0 08933323 0
> 0
> GossipStage   0 0   

Re: Pending compactions not going down on some nodes of the cluster

[Jeff Jirsa]

> We added a bunch of new nodes to a cluster (2.1.13) and everything went fine, 
> except for the number of pending compactions that is staying quite high on a 
> subset of the new nodes. Over the past 3 days, the pending compactions have 
> never been less than ~130 on such nodes, with peaks of ~200.

When you bootstrap with Vnodes, you end up with thousands (or tens of 
thousands) of sstables – with 256 Vnodes (default) * 20 sstables per node, your 
resulting node will have 5k sstables. It takes quite a while for compaction to 
chew through that. If you added a bunch of nodes in sequence, you’d have 5k on 
the first node, then potentailly 10k on the next, and could potentially keep 
increasing as you start streaming from nodes that have way too many sstables.  
This is one of the reasons that many people who have to grow their clusters 
frequently try not to use vnodes.

>From your other email:

> Also related to this point, now I'm seeing something even more odd: some 
> compactions are way bigger than the size of the column family itself, such as:

The size reported by compactionstats is the uncompressed size – if you’re using 
compression, it’s perfectly reasonable for 30G of data to show up as 118G of 
data during compaction.

- Jeff

From:  Gianluca Borello
Reply-To:  "user@cassandra.apache.org"
Date:  Monday, March 21, 2016 at 12:50 PM
To:  "user@cassandra.apache.org"
Subject:  Pending compactions not going down on some nodes of the cluster

Hi,

We added a bunch of new nodes to a cluster (2.1.13) and everything went fine, 
except for the number of pending compactions that is staying quite high on a 
subset of the new nodes. Over the past 3 days, the pending compactions have 
never been less than ~130 on such nodes, with peaks of ~200. On the other 
nodes, they correctly fluctuate between 0 and ~20, which has been our norm for 
a long time.

We are quite paranoid about pending compactions because in the past such high 
number caused a lot of data being brought in memory during some reads and that 
triggered a chain reaction of full GCs that brought down our cluster, so we try 
to monitor them closely.

Some data points that should let the situation speak for itself:

- We use LCS for all our column families

- The cluster is operating absolutely fine and seems healthy, and every node is 
handling pretty much the same load in terms of reads and writes. Also, these 
nodes with higher pending compactions don't seem in any way performing worse 
than the others

- The pending compactions don't go down even when setting the compaction 
throughput to unlimited for a very long time

- This is the typical output of compactionstats and tpstats:

$ nodetool compactionstats
pending tasks: 137
   compaction type   keyspacetable completed total
unit   progress
Compaction draios   message_data6061112083946939536890   
bytes 88.06%
Compaction draiosmessage_data1   26473390790   37243294809   
bytes 71.08%
Active compaction remaining time :n/a

$ nodetool tpstats
Pool NameActive   Pending  Completed   Blocked  All 
time blocked
CounterMutationStage  0 0  0 0  
   0
ReadStage 1 0  111766844 0  
   0
RequestResponseStage  0 0  244259493 0  
   0
MutationStage 0 0  163268653 0  
   0
ReadRepairStage   0 08933323 0  
   0
GossipStage   0 0 363003 0  
   0
CacheCleanupExecutor  0 0  0 0  
   0
AntiEntropyStage  0 0  0 0  
   0
MigrationStage0 0  2 0  
   0
Sampler   0 0  0 0  
   0
ValidationExecutor0 0  0 0  
   0
CommitLogArchiver 0 0  0 0  
   0
MiscStage 0 0  0 0  
   0
MemtableFlushWriter   0 0  32644 0  
   0
MemtableReclaimMemory 0 0  32644 0  
   0
PendingRangeCalculator0 0527 0  
   0
MemtablePostFlush 0 0  36565 0  
   0
CompactionExecutor270 108621 0  
   0
InternalResponseStage 0 0  0 0  
   0
HintedHandoff 0 0 10 0  
   0

Re: Pending compactions not going down on some nodes of the cluster

[Gianluca Borello]

On Mon, Mar 21, 2016 at 12:50 PM, Gianluca Borello 
wrote:

>
> - It's also interesting to notice how the compaction in the previous
> example is trying to compact ~37 GB, which is essentially the whole size of
> the column family message_data1 as reported by cfstats:
>

Also related to this point, now I'm seeing something even more odd: some
compactions are way bigger than the size of the column family itself, such
as:

$ nodetool compactionstats -H
pending tasks: 110
   compaction type   keyspace   table   completed   total
 unit   progress
Compaction draios   message_data126.28 GB   118.73 GB
bytes 22.13%
Active compaction remaining time :   0h04m30s

It says the total is 118.73 GB, but that column family never got anywhere
close to that size, it has always stayed around ~30GB:

$ du -hs /raid0/cassandra/data/draios/message_data1-*
35G
/raid0/cassandra/data/draios/message_data1-ad87f550ea2b11e5a528dde586fa678e

Re: Pending compactions not going down on some nodes of the cluster

[Gianluca Borello]

On Mon, Mar 21, 2016 at 2:15 PM, Alain RODRIGUEZ  wrote:

>
> What hardware do you use? Can you see it running at the limits (CPU /
> disks IO)? Is there any error on system logs, are disks doing fine?
>
>
Nodes are c3.2xlarge instances on AWS. The nodes are relatively idle, and,
as said in the original email, the other nodes are handling the same load
and are doing just fine (no warnings or errors in the system and cassandra
logs). I mean, even this node is doing very fine, it seems as if there is
some internal bug in the way pending tasks are counted.


> How is configured the concurrent_compactors setting? It looks like you are
> using 2 from 'nodetool compactionstats' output. Could you increase that
> or is it the number of cores of the machine?
>
>
It's 2. I can certainly increase it, but, as I proved earlier, it doesn't
do anything, because if I do a "nodetool drain" and wait a while, the
compactions completely stop (and the node goes totally idle), despite the
pending compactions remaining high, so this really doesn't feel like a
problem of Cassandra not being able to keep up with the compactions.

Re: Pending compactions not going down on some nodes of the cluster

[Alain RODRIGUEZ]

Hi, thanks for the detailed information, it is useful.

SSTables in each level: [43/4, 92/10, 125/100, 0, 0, 0, 0, 0, 0]


Looks like compaction is not doing so hot indeed.

What hardware do you use? Can you see it running at the limits (CPU / disks
IO)? Is there any error on system logs, are disks doing fine?

How is configured the concurrent_compactors setting? It looks like you are
using 2 from 'nodetool compactionstats' output. Could you increase that or
is it the number of cores of the machine?

What size do you use for LCS sstables? Using a small size might produce too
many sstables.

Have those nodes more sstables than the other ones?

Are compactions progressing or getting stuck at some point?

It's also interesting to notice how the compaction in the previous example
> is trying to compact ~37 GB, which is essentially the whole size of the
> column family message_data1 as reported by cfstats


Weird indeed, using LCS. Not a clue about that but a small tip: you can use
'nodetool compactionstats *-H*' to have values in human readable format
since 2.1 :-).

Good luck,
---
Alain Rodriguez - al...@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2016-03-21 20:50 GMT+01:00 Gianluca Borello :

> Hi,
>
> We added a bunch of new nodes to a cluster (2.1.13) and everything went
> fine, except for the number of pending compactions that is staying quite
> high on a subset of the new nodes. Over the past 3 days, the pending
> compactions have never been less than ~130 on such nodes, with peaks of
> ~200. On the other nodes, they correctly fluctuate between 0 and ~20, which
> has been our norm for a long time.
>
> We are quite paranoid about pending compactions because in the past such
> high number caused a lot of data being brought in memory during some reads
> and that triggered a chain reaction of full GCs that brought down our
> cluster, so we try to monitor them closely.
>
> Some data points that should let the situation speak for itself:
>
> - We use LCS for all our column families
>
> - The cluster is operating absolutely fine and seems healthy, and every
> node is handling pretty much the same load in terms of reads and writes.
> Also, these nodes with higher pending compactions don't seem in any way
> performing worse than the others
>
> - The pending compactions don't go down even when setting the compaction
> throughput to unlimited for a very long time
>
> - This is the typical output of compactionstats and tpstats:
>
> $ nodetool compactionstats
> pending tasks: 137
>compaction type   keyspacetable completed total
>unit   progress
> Compaction draios   message_data6061112083946939536890
>   bytes 88.06%
> Compaction draiosmessage_data1   26473390790   37243294809
>   bytes 71.08%
> Active compaction remaining time :n/a
>
> $ nodetool tpstats
> Pool NameActive   Pending  Completed   Blocked
>  All time blocked
> CounterMutationStage  0 0  0 0
> 0
> ReadStage 1 0  111766844 0
> 0
> RequestResponseStage  0 0  244259493 0
> 0
> MutationStage 0 0  163268653 0
> 0
> ReadRepairStage   0 08933323 0
> 0
> GossipStage   0 0 363003 0
> 0
> CacheCleanupExecutor  0 0  0 0
> 0
> AntiEntropyStage  0 0  0 0
> 0
> MigrationStage0 0  2 0
> 0
> Sampler   0 0  0 0
> 0
> ValidationExecutor0 0  0 0
> 0
> CommitLogArchiver 0 0  0 0
> 0
> MiscStage 0 0  0 0
> 0
> MemtableFlushWriter   0 0  32644 0
> 0
> MemtableReclaimMemory 0 0  32644 0
> 0
> PendingRangeCalculator0 0527 0
> 0
> MemtablePostFlush 0 0  36565 0
> 0
> CompactionExecutor270 108621 0
> 0
> InternalResponseStage 0 0  0 0
> 0
> HintedHandoff 0 0 10 0
> 0
> Native-Transport-Requests 6 0  188996929 0
> 79122
>
> Message type   Dropped
> RANGE_SLICE   

Pending compactions not going down on some nodes of the cluster

[Gianluca Borello]

Hi,

We added a bunch of new nodes to a cluster (2.1.13) and everything went
fine, except for the number of pending compactions that is staying quite
high on a subset of the new nodes. Over the past 3 days, the pending
compactions have never been less than ~130 on such nodes, with peaks of
~200. On the other nodes, they correctly fluctuate between 0 and ~20, which
has been our norm for a long time.

We are quite paranoid about pending compactions because in the past such
high number caused a lot of data being brought in memory during some reads
and that triggered a chain reaction of full GCs that brought down our
cluster, so we try to monitor them closely.

Some data points that should let the situation speak for itself:

- We use LCS for all our column families

- The cluster is operating absolutely fine and seems healthy, and every
node is handling pretty much the same load in terms of reads and writes.
Also, these nodes with higher pending compactions don't seem in any way
performing worse than the others

- The pending compactions don't go down even when setting the compaction
throughput to unlimited for a very long time

- This is the typical output of compactionstats and tpstats:

$ nodetool compactionstats
pending tasks: 137
   compaction type   keyspacetable completed total
   unit   progress
Compaction draios   message_data6061112083946939536890
  bytes 88.06%
Compaction draiosmessage_data1   26473390790   37243294809
  bytes 71.08%
Active compaction remaining time :n/a

$ nodetool tpstats
Pool NameActive   Pending  Completed   Blocked  All
time blocked
CounterMutationStage  0 0  0 0
0
ReadStage 1 0  111766844 0
0
RequestResponseStage  0 0  244259493 0
0
MutationStage 0 0  163268653 0
0
ReadRepairStage   0 08933323 0
0
GossipStage   0 0 363003 0
0
CacheCleanupExecutor  0 0  0 0
0
AntiEntropyStage  0 0  0 0
0
MigrationStage0 0  2 0
0
Sampler   0 0  0 0
0
ValidationExecutor0 0  0 0
0
CommitLogArchiver 0 0  0 0
0
MiscStage 0 0  0 0
0
MemtableFlushWriter   0 0  32644 0
0
MemtableReclaimMemory 0 0  32644 0
0
PendingRangeCalculator0 0527 0
0
MemtablePostFlush 0 0  36565 0
0
CompactionExecutor270 108621 0
0
InternalResponseStage 0 0  0 0
0
HintedHandoff 0 0 10 0
0
Native-Transport-Requests 6 0  188996929 0
79122

Message type   Dropped
RANGE_SLICE  0
READ_REPAIR  0
PAGED_RANGE  0
BINARY   0
READ 0
MUTATION 0
_TRACE   0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0

- If I do a nodetool drain on such nodes, and then wait for a while, the
number of pending compactions stays high even if there are no compactions
being executed anymore and the node is completely idle:

$ nodetool compactionstats
pending tasks: 128

- It's also interesting to notice how the compaction in the previous
example is trying to compact ~37 GB, which is essentially the whole size of
the column family message_data1 as reported by cfstats:

$ nodetool cfstats -H draios.message_data1
Keyspace: draios
Read Count: 208168
Read Latency: 2.4791508685292647 ms.
Write Count: 502529
Write Latency: 0.20701542000561163 ms.
Pending Flushes: 0
Table: message_data1
SSTable count: 261
SSTables in each level: [43/4, 92/10, 125/100, 0, 0, 0, 0, 0, 0]
Space used (live): 36.98 GB
Space used (total): 36.98 GB
Space used by snapshots (total): 0 bytes
Off heap memory used (total): 36.21 MB
SSTable Compression Ratio: 0.15461126176169512
Number of keys (estimate): 101025
Memtable cell count: 229344
Memtable data size: 82.4 MB
Memtable off heap memory used: 0 bytes
Memtable switch count: 83
Local read count: 208225
Local read latency: 2.479 ms
Local write count: 502581
Local write latency: 0.208 ms
Pending 

Re: Cassandra 2.2.4: FailureDetector: "Ignoring interval time of ..."

[Paulo Motta]

This will only make failure detector calculations less accurate, so if
you're not having nodes flapping UP/DOWN or being incorrectly marked as
down, this shouldn't be a big problem. You'll probably want to tune
cassandra.fd_max_interval_ms for your cluster environment but I'm not
really familiar with these settings so tuning suggestions are welcome.

If you upgraded from a previous version, this started being logged on
debug.log only on 2.2 so there`s a chance it has been like this for a while
but was never noticed. See CASSANDRA-11388, CASSANDRA-7307 and
CASSANDRA-4375 for more background information.

See CASSANDRA-11388, CASSANDRA-7307 and CASSANDRA-4375 for more background
information.

2016-03-21 14:08 GMT-03:00 Herbert Fischer :

> Hi,
>
> I'm using Cassandra 2.2.4 and I we are getting ~4k messages per hour, per
> node.
>
> It doesn't look good to me. Is it normal? If not, any idea on what might
> be wrong?
>
> Thanks
>
> Herbert
>

RE: Expiring Columns

[Anubhav Kale]

I think the answer is no. There are explicit checks in Read code path to ignore 
anything that’s past the TTL (based on local time of the node under question).

From: Anuj Wadehra [mailto:anujw_2...@yahoo.co.in]
Sent: Monday, March 21, 2016 5:19 AM
To: User 
Subject: Expiring Columns

Hi,

I want to understand how Expiring columns work in Cassandra.


Query:
Documentation says that once TTL of a column expires, tombstones are created/ 
marked when the sstable gets compacted. Is there a possibility that a query 
(range scan/ row query) returns expired column data just because the sstable 
never participated in a compaction after TTL of the column expired?

For Example:
  10 AM Data inserted with ttl=60 seconds
  10:05 AM A query is run on inserted data
  10:07 AM sstable is compacted and column is marked tombstone.

Will the query return expired data in above scenario? If yes/no, why?


Thanks
Anuj






Sent from Yahoo Mail on 
Android

Cassandra 2.2.4: FailureDetector: "Ignoring interval time of ..."

[Herbert Fischer]

Hi,

I'm using Cassandra 2.2.4 and I we are getting ~4k messages per hour, per
node.

It doesn't look good to me. Is it normal? If not, any idea on what might be
wrong?

Thanks

Herbert

Re: Expiring Columns

[Nirmallya Mukherjee]

Anuj, there are a couple of aspects that are important -1. Cassandra will not 
return data with expired TTLs even if compaction has not run2. How is 1 
possible? The read path will eliminate the expired TTLs from the result3. All 
tombstones (explicit delete as well as expired TTLs) hang around for some time 
(gc_grace_seconds)4. Once the gc_grace_seconds (default 10 days) are over the 
compaction will cleanup the tombstones in SSTables that are considered for 
compaction
Thanks,Nirmallya
 

On Monday, 21 March 2016 5:49 PM, Anuj Wadehra  
wrote:
 

 Hi,
I want to understand how Expiring columns work in Cassandra.

Query:Documentation says that once TTL of a column expires, tombstones are 
created/ marked when the sstable gets compacted. Is there a possibility that a 
query (range scan/ row query) returns expired column data just because the 
sstable never participated in a compaction after TTL of the column expired?
For Example:  10 AM Data inserted with ttl=60 seconds  10:05 AM A query is run 
on inserted data  10:07 AM sstable is compacted and column     is marked 
tombstone.
Will the query return expired data in above scenario? If yes/no, why?

ThanksAnuj





Sent from Yahoo Mail on Android

  

Re: Getting Issue while setting up Cassandra in Windows 8.1

[Paulo Motta]

It seems you have problems with your PATH, so system commands are not being
found. You need to recover your original PATH first, you may find on stack
overflow how to do this.

Another problem that seems to be happening is that path names with spaces
are not being interpreted correctly, so if fixing your PATH does not fix
this you might try renaming the "Learning Office Work" folder to some name
without spaces. And if that fixes your problem then it's probably a bug on
the cassandra win init script so you should file an issue on
https://issues.apache.org/jira/browse/cassandra.

2016-03-21 6:20 GMT-03:00 Bhupendra Baraiya :

> HI,
>
>
>   I enabled Powershell for Cassandra installation but then I started
> getting error as failed to run in 64 bit mode rerunning in 32 bit si I
> uninstalled Java and installed a 64 bit version but then I started getting
> below error
>
>
> Can someone please guide me how I can I solve it
>
>
>
>
>
> --
> *From:* Bhupendra Baraiya 
> *Sent:* 18 March 2016 20:11
> *To:* user@cassandra.apache.org
> *Subject:* Getting Issue while setting up Cassandra in Windows 8.1
>
>
> Hi ,
>
>
>
>   I tried install Cassandra using above attached  steps
>
> But then I got below error
>
>
>
>
>
>
>
>
>
> When I have send  this to 'd...@cassandra.apache.org' I got below reply
>
>
>
> You should use the user@cassandra.apache.org list for cassandra-related
> questions, not this (d...@cassandra.apache.org) which is exclusive for
> internal Cassandra development. You can register to the user list by
> sending an e-mail to: user-subscr...@cassandra.apache.org
>
>
>
> Answering your question, it seems your %PATH% variable is broken, for some
> reason you removed the default
> "C:\Windows\System32\WindowsPowerShell\v1.0\" entry, so powershell is not
> found to run properly configured version of Cassandra, so you must fix
> that. If you still want to run with legacy settings (not recommended), you
> must adjust your heap settings (-Xms, -Xmx) on the cassandra.bat script to
> fit your system's memory.
>
>
>
> So I gone to Cassandra.bat and changed Xms and Xmn to 1G and it worked but
> from last 30 minutes its showing me below progress is it natural or
> something wrong has happend
>
>
>
>
>
>
>
>
>
> Thanks and regards,
>
>
>
> *Bhupendra Baraiya*
>
> Continuum Managed Services, LLC.
>
> p: 902-933-0019
>
> e: bhupendra.bara...@continuum.net
>
> w: continuum.net
>
> [image:
> http://cdn2.hubspot.net/hub/281750/file-393087232-png/img/logos/email-continuum-logo-151x26.png]
> 
>
>
>

Expiring Columns

[Anuj Wadehra]

Hi,
I want to understand how Expiring columns work in Cassandra.

Query:Documentation says that once TTL of a column expires, tombstones are 
created/ marked when the sstable gets compacted. Is there a possibility that a 
query (range scan/ row query) returns expired column data just because the 
sstable never participated in a compaction after TTL of the column expired?
For Example:  10 AM Data inserted with ttl=60 seconds  10:05 AM A query is run 
on inserted data  10:07 AM sstable is compacted and column     is marked 
tombstone.
Will the query return expired data in above scenario? If yes/no, why?

ThanksAnuj





Sent from Yahoo Mail on Android

Re: Multi DC setup for analytics

[Anishek Agarwal]

Hey Clint,

we have two separate rings which don't talk to each other but both having
the same DC name "DCX".

@Raja,

We had already gone towards the path you suggested.

thanks all
anishek

On Fri, Mar 18, 2016 at 8:01 AM, Reddy Raja  wrote:

> Yes. Here are the steps.
> You will have to change the DC Names first.
> DC1 and DC2 would be independent clusters.
>
> Create a new DC, DC3 and include these two DC's on DC3.
>
> This should work well.
>
>
> On Thu, Mar 17, 2016 at 11:03 PM, Clint Martin <
> clintlmar...@coolfiretechnologies.com> wrote:
>
>> When you say you have two logical DC both with the same name are you
>> saying that you have two clusters of servers both with the same DC name,
>> nether of which currently talk to each other? IE they are two separate
>> rings?
>>
>> Or do you mean that you have two keyspaces in one cluster?
>>
>> Or?
>>
>> Clint
>> On Mar 14, 2016 2:11 AM, "Anishek Agarwal"  wrote:
>>
>>> Hello,
>>>
>>> We are using cassandra 2.0.17 and have two logical DC having different
>>> Keyspaces but both having same logical name DC1.
>>>
>>> we want to setup another cassandra cluster for analytics which should
>>> get data from both the above DC.
>>>
>>> if we setup the new DC with name DC2 and follow the steps
>>> https://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_add_dc_to_cluster_t.html
>>> will it work ?
>>>
>>> I would think we would have to first change the names of existing
>>> clusters to have to different names and then go with adding another dc
>>> getting data from these?
>>>
>>> Also as soon as we add the node the data starts moving... this will all
>>> be only real time changes done to the cluster right ? we still have to do
>>> the rebuild to get the data for tokens for node in new cluster ?
>>>
>>> Thanks
>>> Anishek
>>>
>>
>
>
> --
> "In this world, you either have an excuse or a story. I preferred to have
> a story"
>

Re: cqlsh problem

[joseph gao]

cqlsh version 5.0.1. nodetool tpstats looks good, log looks good. And I
used specified port 9042. And it immediately returns fail (less than 3
seconds). By the way where should I use '--connect-timeout', cqlsh seems
don't have such parameters.

2016-03-18 17:29 GMT+08:00 Alain RODRIGUEZ :

> Is the node fully healthy or rejecting some requests ?
>
> What are the outputs for "grep -i "ERROR" /var/log/cassandra/system.log"
> and "nodetool tpstats"?
>
> Any error? Any pending / blocked or dropped messages?
>
> Also did you try using distinct ports (9160 for thrift, 9042 for native) -
> out of curiosity, not sure this will help.
>
> What is your version of cqlsh "cqlsh --version" ?
>
> doesn't work most times. But some time it just work fine
>>
>
> Do you fill like this is due to a timeout (query being too big, cluster
> being to busy)? Try setting this higher:
>
> --connect-timeout=CONNECT_TIMEOUT
>
> Specify the connection timeout in seconds
> (default: 5 seconds).
>
>   --request-timeout=REQUEST_TIMEOUT
>
> Specify the default request timeout in seconds 
> (default:
> 10 seconds).
>
> C*heers,
> ---
> Alain Rodriguez - al...@thelastpickle.com
> France
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> 2016-03-18 4:49 GMT+01:00 joseph gao :
>
>> Of course yes.
>>
>> 2016-03-17 22:35 GMT+08:00 Vishwas Gupta :
>>
>>> Have you started the Cassandra service?
>>>
>>> sh cassandra
>>> On 17-Mar-2016 7:59 pm, "Alain RODRIGUEZ"  wrote:
>>>
 Hi, did you try with the address of the node rather than 127.0.0.1

 Is the transport protocol used by cqlsh (not sure if it is thrift or
 binary - native in 2.1)  active ? What is the "nodetool info" output ?

 C*heers,
 ---
 Alain Rodriguez - al...@thelastpickle.com
 France

 The Last Pickle - Apache Cassandra Consulting
 http://www.thelastpickle.com

 2016-03-17 14:26 GMT+01:00 joseph gao :

> hi, all
> cassandra version 2.1.7
> When I use cqlsh to connect cassandra, something is wrong
>
> Connection error: ( Unable to connect to any servers', {'127.0.0.1':
> OperationTimedOut('errors=None, last_host=None,)})
>
> This happens lots of times, but sometime it works just fine. Anybody
> knows why?
>
> --
> --
> Joseph Gao
> PhoneNum:15210513582
> QQ: 409343351
>


>>
>>
>> --
>> --
>> Joseph Gao
>> PhoneNum:15210513582
>> QQ: 409343351
>>
>
>


-- 
--
Joseph Gao
PhoneNum:15210513582
QQ: 409343351