Re: Multi-DC replication and hinted handoff

[Jens Fischer]

Hi,

an update: I am pretty sure it is a problem with insufficient bandwidth. I 
can’t be sure because Cassandra does not seem to provide debug information on 
hint creation (only when replaying hints). When the bandwidth issue is solved I 
will try to reproduce the accumulation of hints by artificially limiting the 
bandwidth.

BG
Jens

On 3. Apr 2019, at 01:48, Stefan Miklosovic 
mailto:stefan.mikloso...@instaclustr.com>> 
wrote:

Hi Jens,

I am reading Cassandra The definitive guide and there is a chapter 9 - Reading 
and Writing Data and section The Cassandra Write Path and this sentence in it:

If a replica does not respond within the timeout, it is presumed to be down and 
a hint is stored for the write.

So your node might be actually fine eventually but it just can not cope with 
the load and it will reply too late after a coordinator has sufficient replies 
from other replicas. So it makes a hint for that write and for that node. I am 
not sure how is this related to turning off handoffs completely. I can do some 
tests locally if time allows to investigate various scenarios. There might be 
some subtle differences 

On Wed, 3 Apr 2019 at 07:19, Jens Fischer 
mailto:j.fisc...@sonnen.de>> wrote:
Yes, Apache Cassandra 3.11.2 (no DSE).

On 2. Apr 2019, at 19:40, sankalp kohli 
mailto:kohlisank...@gmail.com>> wrote:

Are you using OSS C*?

On Fri, Mar 29, 2019 at 1:49 AM Jens Fischer 
mailto:j.fisc...@sonnen.de>> wrote:
Hi,

I have a Cassandra setup with multiple data centres. The vast majority of 
writes are LOCAL_ONE writes to data center DC-A. One node (lets call this node 
A1) in DC-A has accumulated large amounts of hint files (~100 GB). In the logs 
of this node I see lots of messages like the following:

INFO  [HintsDispatcher:26] 2019-03-28 01:49:25,217 
HintsDispatchExecutor.java:289 - Finished hinted handoff of file 
db485ac6-8acd-4241-9e21-7a2b540459de-1553419324363-1.hints to endpoint 
/10.10.2.55<http://10.10.2.55/>: db485ac6-8acd-4241-9e21-7a2b540459de

The node 10.10.2.55 is in DC-B, lets call this node B1. There is no indication 
whatsoever that B1 was down: Nothing in our monitoring, nothing in the logs of 
B1, nothing in the logs of A1. Are there any other situations where hints to B1 
are stored at A1? Other than A1's failure detection detecting B1 as down I 
mean. For example could the reason for the hints be that B1 is overloaded and 
can not handle the intake from the A1? Or that the network connection between 
DC-A and DC-B is to slow?

While researching this I also found the following information on Stack Overflow 
from Ben Slater regarding hints and multi-dc replication:

Another factor here is the consistency level you are using - a LOCAL_* 
consistency level will only require writes to be written to the local DC for 
the operation to be considered a success (and hints will be stored for 
replication to the other DC).
(…)
The hints are the records of writes that have been made in one DC that are not 
yet replicated to the other DC (or even nodes within a DC). I think your 
options to avoid them are: (1) write with ALL or QUOROM (not LOCAL_*) 
consistency - this will slow down your writes but will ensure writes go into 
both DCs before the op completes (2) Don't replicate the data to the second DC 
(by setting the replication factor to 0 for the second DC in the keyspace 
definition) (3) Increase the capacity of the second DC so it can keep up with 
the writes (4) Slow down your writes so the second DC can keep up.

Source: https://stackoverflow.com/a/37382726

This reads like hints are used for “normal” (async) replication between data 
centres, i.e. hints could show up without any nodes being down whatsoever. This 
could explain what I am seeing. Does anyone now more about this? Does that mean 
I will see hints even if I disable hinted handoff?

Any pointers or help are greatly appreciated!

Thanks in advance
Jens


[https://img.sonnen.de/TSEE2019_Banner_sonnenGmbH_de_1.jpg]

Geschäftsführer: Christoph Ostermann (CEO), Oliver Koch, Steffen Schneider, 
Hermann Schweizer.
Amtsgericht Kempten/Allgäu, Registernummer: 10655, Steuernummer 127/137/50792, 
USt.-IdNr. DE272208908


[https://img.sonnen.de/TSEE2019_Banner_sonnenGmbH_de_1.jpg]

Geschäftsführer: Christoph Ostermann (CEO), Oliver Koch, Steffen Schneider, 
Hermann Schweizer.
Amtsgericht Kempten/Allgäu, Registernummer: 10655, Steuernummer 127/137/50792, 
USt.-IdNr. DE272208908


[https://img.sonnen.de/TSEE2019_Banner_sonnenGmbH_de_1.jpg]

Geschäftsführer: Christoph Ostermann (CEO), Oliver Koch, Steffen Schneider, 
Hermann Schweizer.
Amtsgericht Kempten/Allgäu, Registernummer: 10655, Steuernummer 127/137/50792, 
USt.-IdNr. DE272208908

Re: Multi-DC replication and hinted handoff

[Stefan Miklosovic]

Hi Jens,

I am reading Cassandra The definitive guide and there is a chapter 9 -
Reading and Writing Data and section The Cassandra Write Path and this
sentence in it:

If a replica does not respond within the timeout, it is presumed to be down
and a hint is stored for the write.

So your node might be actually fine eventually but it just can not cope
with the load and it will reply too late after a coordinator has sufficient
replies from other replicas. So it makes a hint for that write and for that
node. I am not sure how is this related to turning off handoffs completely.
I can do some tests locally if time allows to investigate various
scenarios. There might be some subtle differences 

On Wed, 3 Apr 2019 at 07:19, Jens Fischer  wrote:

> Yes, Apache Cassandra 3.11.2 (no DSE).
>
> On 2. Apr 2019, at 19:40, sankalp kohli  wrote:
>
> Are you using OSS C*?
>
> On Fri, Mar 29, 2019 at 1:49 AM Jens Fischer  wrote:
>
>> Hi,
>>
>> I have a Cassandra setup with multiple data centres. The vast majority of
>> writes are LOCAL_ONE writes to data center DC-A. One node (lets call this
>> node A1) in DC-A has accumulated large amounts of hint files (~100 GB). In
>> the logs of this node I see lots of messages like the following:
>>
>> INFO  [HintsDispatcher:26] 2019-03-28 01:49:25,217
>> HintsDispatchExecutor.java:289 - Finished hinted handoff of file
>> db485ac6-8acd-4241-9e21-7a2b540459de-1553419324363-1.hints to endpoint /
>> 10.10.2.55: db485ac6-8acd-4241-9e21-7a2b540459de
>>
>> The node 10.10.2.55 is in DC-B, lets call this node B1. There is no
>> indication whatsoever that B1 was down: Nothing in our monitoring, nothing
>> in the logs of B1, nothing in the logs of A1. Are there any other
>> situations where hints to B1 are stored at A1? Other than A1's failure
>> detection detecting B1 as down I mean. For example could the reason for the
>> hints be that B1 is overloaded and can not handle the intake from the A1?
>> Or that the network connection between DC-A and DC-B is to slow?
>>
>> While researching this I also found the following information on Stack
>> Overflow from Ben Slater regarding hints and multi-dc replication:
>>
>> Another factor here is the consistency level you are using - a LOCAL_*
>> consistency level will only require writes to be written to the local DC
>> for the operation to be considered a success (and hints will be stored for
>> replication to the other DC).
>> (…)
>> The hints are the records of writes that have been made in one DC that
>> are not yet replicated to the other DC (or even nodes within a DC). I think
>> your options to avoid them are: (1) write with ALL or QUOROM (not LOCAL_*)
>> consistency - this will slow down your writes but will ensure writes go
>> into both DCs before the op completes (2) Don't replicate the data to the
>> second DC (by setting the replication factor to 0 for the second DC in the
>> keyspace definition) (3) Increase the capacity of the second DC so it can
>> keep up with the writes (4) Slow down your writes so the second DC can keep
>> up.
>>
>>
>> Source: https://stackoverflow.com/a/37382726
>>
>> This reads like hints are used for “normal” (async) replication between
>> data centres, i.e. hints could show up without any nodes being down
>> whatsoever. This could explain what I am seeing. Does anyone now more about
>> this? Does that mean I will see hints even if I disable hinted handoff?
>>
>> Any pointers or help are greatly appreciated!
>>
>> Thanks in advance
>> Jens
>>
>> Geschäftsführer: Christoph Ostermann (CEO), Oliver Koch, Steffen
>> Schneider, Hermann Schweizer.
>> Amtsgericht Kempten/Allgäu, Registernummer: 10655, Steuernummer
>> 127/137/50792, USt.-IdNr. DE272208908
>>
>
> Geschäftsführer: Christoph Ostermann (CEO), Oliver Koch, Steffen
> Schneider, Hermann Schweizer.
> Amtsgericht Kempten/Allgäu, Registernummer: 10655, Steuernummer
> 127/137/50792, USt.-IdNr. DE272208908
>

Re: Multi-DC replication and hinted handoff

[Jens Fischer]

Yes, Apache Cassandra 3.11.2 (no DSE).

On 2. Apr 2019, at 19:40, sankalp kohli 
mailto:kohlisank...@gmail.com>> wrote:

Are you using OSS C*?

On Fri, Mar 29, 2019 at 1:49 AM Jens Fischer 
mailto:j.fisc...@sonnen.de>> wrote:
Hi,

I have a Cassandra setup with multiple data centres. The vast majority of 
writes are LOCAL_ONE writes to data center DC-A. One node (lets call this node 
A1) in DC-A has accumulated large amounts of hint files (~100 GB). In the logs 
of this node I see lots of messages like the following:

INFO  [HintsDispatcher:26] 2019-03-28 01:49:25,217 
HintsDispatchExecutor.java:289 - Finished hinted handoff of file 
db485ac6-8acd-4241-9e21-7a2b540459de-1553419324363-1.hints to endpoint 
/10.10.2.55<http://10.10.2.55/>: db485ac6-8acd-4241-9e21-7a2b540459de

The node 10.10.2.55 is in DC-B, lets call this node B1. There is no indication 
whatsoever that B1 was down: Nothing in our monitoring, nothing in the logs of 
B1, nothing in the logs of A1. Are there any other situations where hints to B1 
are stored at A1? Other than A1's failure detection detecting B1 as down I 
mean. For example could the reason for the hints be that B1 is overloaded and 
can not handle the intake from the A1? Or that the network connection between 
DC-A and DC-B is to slow?

While researching this I also found the following information on Stack Overflow 
from Ben Slater regarding hints and multi-dc replication:

Another factor here is the consistency level you are using - a LOCAL_* 
consistency level will only require writes to be written to the local DC for 
the operation to be considered a success (and hints will be stored for 
replication to the other DC).
(…)
The hints are the records of writes that have been made in one DC that are not 
yet replicated to the other DC (or even nodes within a DC). I think your 
options to avoid them are: (1) write with ALL or QUOROM (not LOCAL_*) 
consistency - this will slow down your writes but will ensure writes go into 
both DCs before the op completes (2) Don't replicate the data to the second DC 
(by setting the replication factor to 0 for the second DC in the keyspace 
definition) (3) Increase the capacity of the second DC so it can keep up with 
the writes (4) Slow down your writes so the second DC can keep up.

Source: https://stackoverflow.com/a/37382726

This reads like hints are used for “normal” (async) replication between data 
centres, i.e. hints could show up without any nodes being down whatsoever. This 
could explain what I am seeing. Does anyone now more about this? Does that mean 
I will see hints even if I disable hinted handoff?

Any pointers or help are greatly appreciated!

Thanks in advance
Jens


[https://img.sonnen.de/TSEE2019_Banner_sonnenGmbH_de_1.jpg]

Geschäftsführer: Christoph Ostermann (CEO), Oliver Koch, Steffen Schneider, 
Hermann Schweizer.
Amtsgericht Kempten/Allgäu, Registernummer: 10655, Steuernummer 127/137/50792, 
USt.-IdNr. DE272208908


[https://img.sonnen.de/TSEE2019_Banner_sonnenGmbH_de_1.jpg]

Geschäftsführer: Christoph Ostermann (CEO), Oliver Koch, Steffen Schneider, 
Hermann Schweizer.
Amtsgericht Kempten/Allgäu, Registernummer: 10655, Steuernummer 127/137/50792, 
USt.-IdNr. DE272208908

Re: Multi-DC replication and hinted handoff

[sankalp kohli]

Are you using OSS C*?

On Fri, Mar 29, 2019 at 1:49 AM Jens Fischer  wrote:

> Hi,
>
> I have a Cassandra setup with multiple data centres. The vast majority of
> writes are LOCAL_ONE writes to data center DC-A. One node (lets call this
> node A1) in DC-A has accumulated large amounts of hint files (~100 GB). In
> the logs of this node I see lots of messages like the following:
>
> INFO  [HintsDispatcher:26] 2019-03-28 01:49:25,217
> HintsDispatchExecutor.java:289 - Finished hinted handoff of file
> db485ac6-8acd-4241-9e21-7a2b540459de-1553419324363-1.hints to endpoint /
> 10.10.2.55: db485ac6-8acd-4241-9e21-7a2b540459de
>
> The node 10.10.2.55 is in DC-B, lets call this node B1. There is no
> indication whatsoever that B1 was down: Nothing in our monitoring, nothing
> in the logs of B1, nothing in the logs of A1. Are there any other
> situations where hints to B1 are stored at A1? Other than A1's failure
> detection detecting B1 as down I mean. For example could the reason for the
> hints be that B1 is overloaded and can not handle the intake from the A1?
> Or that the network connection between DC-A and DC-B is to slow?
>
> While researching this I also found the following information on Stack
> Overflow from Ben Slater regarding hints and multi-dc replication:
>
> Another factor here is the consistency level you are using - a LOCAL_*
> consistency level will only require writes to be written to the local DC
> for the operation to be considered a success (and hints will be stored for
> replication to the other DC).
> (…)
> The hints are the records of writes that have been made in one DC that are
> not yet replicated to the other DC (or even nodes within a DC). I think
> your options to avoid them are: (1) write with ALL or QUOROM (not LOCAL_*)
> consistency - this will slow down your writes but will ensure writes go
> into both DCs before the op completes (2) Don't replicate the data to the
> second DC (by setting the replication factor to 0 for the second DC in the
> keyspace definition) (3) Increase the capacity of the second DC so it can
> keep up with the writes (4) Slow down your writes so the second DC can keep
> up.
>
>
> Source: https://stackoverflow.com/a/37382726
>
> This reads like hints are used for “normal” (async) replication between
> data centres, i.e. hints could show up without any nodes being down
> whatsoever. This could explain what I am seeing. Does anyone now more about
> this? Does that mean I will see hints even if I disable hinted handoff?
>
> Any pointers or help are greatly appreciated!
>
> Thanks in advance
> Jens
>
> Geschäftsführer: Christoph Ostermann (CEO), Oliver Koch, Steffen
> Schneider, Hermann Schweizer.
> Amtsgericht Kempten/Allgäu, Registernummer: 10655, Steuernummer
> 127/137/50792, USt.-IdNr. DE272208908
>

Multi-DC replication and hinted handoff

[Jens Fischer]

Hi,

I have a Cassandra setup with multiple data centres. The vast majority of 
writes are LOCAL_ONE writes to data center DC-A. One node (lets call this node 
A1) in DC-A has accumulated large amounts of hint files (~100 GB). In the logs 
of this node I see lots of messages like the following:

INFO  [HintsDispatcher:26] 2019-03-28 01:49:25,217 
HintsDispatchExecutor.java:289 - Finished hinted handoff of file 
db485ac6-8acd-4241-9e21-7a2b540459de-1553419324363-1.hints to endpoint 
/10.10.2.55: db485ac6-8acd-4241-9e21-7a2b540459de

The node 10.10.2.55 is in DC-B, lets call this node B1. There is no indication 
whatsoever that B1 was down: Nothing in our monitoring, nothing in the logs of 
B1, nothing in the logs of A1. Are there any other situations where hints to B1 
are stored at A1? Other than A1's failure detection detecting B1 as down I 
mean. For example could the reason for the hints be that B1 is overloaded and 
can not handle the intake from the A1? Or that the network connection between 
DC-A and DC-B is to slow?

While researching this I also found the following information on Stack Overflow 
from Ben Slater regarding hints and multi-dc replication:

Another factor here is the consistency level you are using - a LOCAL_* 
consistency level will only require writes to be written to the local DC for 
the operation to be considered a success (and hints will be stored for 
replication to the other DC).
(…)
The hints are the records of writes that have been made in one DC that are not 
yet replicated to the other DC (or even nodes within a DC). I think your 
options to avoid them are: (1) write with ALL or QUOROM (not LOCAL_*) 
consistency - this will slow down your writes but will ensure writes go into 
both DCs before the op completes (2) Don't replicate the data to the second DC 
(by setting the replication factor to 0 for the second DC in the keyspace 
definition) (3) Increase the capacity of the second DC so it can keep up with 
the writes (4) Slow down your writes so the second DC can keep up.

Source: https://stackoverflow.com/a/37382726

This reads like hints are used for “normal” (async) replication between data 
centres, i.e. hints could show up without any nodes being down whatsoever. This 
could explain what I am seeing. Does anyone now more about this? Does that mean 
I will see hints even if I disable hinted handoff?

Any pointers or help are greatly appreciated!

Thanks in advance
Jens


[https://img.sonnen.de/TSEE2019_Banner_sonnenGmbH_de_1.jpg]

Geschäftsführer: Christoph Ostermann (CEO), Oliver Koch, Steffen Schneider, 
Hermann Schweizer.
Amtsgericht Kempten/Allgäu, Registernummer: 10655, Steuernummer 127/137/50792, 
USt.-IdNr. DE272208908

Re: Hinted Handoff

[Agrawal, Pratik]

Please find my comments inline.

From: kurt greaves 
Reply-To: "user@cassandra.apache.org" 
Date: Tuesday, August 7, 2018 at 1:20 AM
To: User 
Subject: Re: Hinted Handoff

Does Cassandra TTL out the hints after max_hint_window_in_ms? From my 
understanding, Cassandra only stops collecting hints after 
max_hint_window_in_ms but can still keep replaying the hints if the node comes 
back again. Is this correct? Is there a way to TTL out hints?

No, but it won't send hints that have passed HH window. Also, this shouldn't be 
caused by HH as the hints maintain the original timestamp with which they were 
written.

  *   We actually saw data resurrecting after HH window. One interesting thing 
to notice is that, the data was resurrecting in intervals (after ~1Hr).
  *   Original timestamp doesn’t help since the other copies of the data are 
actually deleted and tombstones are wiped out after 15 minutes.
  *   The Cassandra version we are using is 2.2.8

Honestly, this sounds more like a use case for a distributed cache rather than 
Cassandra. Keeping data for 30 minutes and then deleting it is going to be a 
nightmare to manage in Cassandra.

  *   Agreed, we are looking into other databases (like Redis, Aerospike). We 
have a write heavy use case and also need optimistic locking + columnar updates.

Thanks,
Pratik

On 7 August 2018 at 07:20, Agrawal, Pratik 
mailto:paagr...@amazon.com.invalid>> wrote:
Does Cassandra TTL out the hints after max_hint_window_in_ms? From my 
understanding, Cassandra only stops collecting hints after 
max_hint_window_in_ms but can still keep replaying the hints if the node comes 
back again. Is this correct? Is there a way to TTL out hints?

Thanks,
Pratik

From: Kyrylo Lebediev 
mailto:kyrylo_lebed...@epam.com>.INVALID>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
mailto:user@cassandra.apache.org>>
Date: Monday, August 6, 2018 at 4:10 PM
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
mailto:user@cassandra.apache.org>>
Subject: Re: Hinted Handoff


Small gc_grace_seconds value lowers max allowed node downtime, which is 15 
minutes in your case. After 15 minutes of downtime you'll need to replace the 
node, as you described. This interval looks too short to be able to do planned 
maintenance. So, in case you set larger value for gc_grace_seconds (lets say, 
hours or a day) will you get visible read amplification / waste a lot of disk 
space / issues with compactions?



Hinted handoff may be the reason in case hinted handoff window is longer than 
gc_grace_seconds. To me it looks like hinted handoff window 
(max_hint_window_in_ms in cassandra.yaml, which defaults to 3h) must always be 
set to a value less than gc_grace_seconds.



Regards,

Kyrill


From: Agrawal, Pratik 
Sent: Monday, August 6, 2018 8:22:27 PM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Hinted Handoff


Hello all,

We use Cassandra in non-conventional way, where our data is short termed (life 
cycle of about 20-30 minutes) where each record is updated ~5 times and then 
deleted. We have GC grace of 15 minutes.

We are seeing 2 problems

1.) A certain number of Cassandra nodes goes down and then we remove it from 
the cluster using Cassandra removenode command and replace the dead nodes with 
new nodes. While new nodes are joining in, we see more nodes down (which are 
not actually down) but we see following errors in the log

“Gossip not settled after 321 polls. Gossip Stage active/pending/completed: 
1/816/0”



To fix the issue, I restarted the server and the nodes now appear to be up and 
the problem is solved



Can this problem be related to 
https://issues.apache.org/jira/browse/CASSANDRA-6590 ?



2.) Meanwhile, after restarting the nodes mentioned above, we see that some old 
deleted data is resurrected (because of short lifecycle of our data). My guess 
at the moment is that these data is resurrected due to hinted handoff. 
Interesting point to note here is that data keeps resurrecting at periodic 
intervals (like an hour) and then finally stops. Could this be caused by hinted 
handoff? if so is there any setting which we can set to specify that 
“invalidate, hinted handoff data after 5-10 minutes”.



Thanks,
Pratik

Re: Hinted Handoff

[Rahul Singh]

What is the data size that you are talking about ? What is your compaction 
strategy?

I wouldn’t recommend having such an aggressive TTL. Why not put a clustering 
key that allows you to get the data fairly quickly but have a longer TTL?



Cassandra can still be used if the there is a legitimate need for multi-dc 
global replication and redundancy not quite available at the same level of 
uptime as in dist. Coaches like REDIS.


Rahul
On Aug 7, 2018, 1:19 AM -0400, kurt greaves , wrote:
> > Does Cassandra TTL out the hints after max_hint_window_in_ms? From my 
> > understanding, Cassandra only stops collecting hints after 
> > max_hint_window_in_ms but can still keep replaying the hints if the node 
> > comes back again. Is this correct? Is there a way to TTL out hints?
>
> No, but it won't send hints that have passed HH window. Also, this shouldn't 
> be caused by HH as the hints maintain the original timestamp with which they 
> were written.
>
> Honestly, this sounds more like a use case for a distributed cache rather 
> than Cassandra. Keeping data for 30 minutes and then deleting it is going to 
> be a nightmare to manage in Cassandra.
>
> > On 7 August 2018 at 07:20, Agrawal, Pratik  
> > wrote:
> > > Does Cassandra TTL out the hints after max_hint_window_in_ms? From my 
> > > understanding, Cassandra only stops collecting hints after 
> > > max_hint_window_in_ms but can still keep replaying the hints if the node 
> > > comes back again. Is this correct? Is there a way to TTL out hints?
> > >
> > > Thanks,
> > > Pratik
> > >
> > > From: Kyrylo Lebediev 
> > > Reply-To: "user@cassandra.apache.org" 
> > > Date: Monday, August 6, 2018 at 4:10 PM
> > > To: "user@cassandra.apache.org" 
> > > Subject: Re: Hinted Handoff
> > >
> > > Small gc_grace_seconds value lowers max allowed node downtime, which is 
> > > 15 minutes in your case. After 15 minutes of downtime you'll need to 
> > > replace the node, as you described. This interval looks too short to be 
> > > able to do planned maintenance. So, in case you set larger value for 
> > > gc_grace_seconds (lets say, hours or a day) will you get visible read 
> > > amplification / waste a lot of disk space / issues with compactions?
> > >
> > > Hinted handoff may be the reason in case hinted handoff window is longer 
> > > than gc_grace_seconds. To me it looks like hinted handoff window 
> > > (max_hint_window_in_ms in cassandra.yaml, which defaults to 3h) must 
> > > always be set to a value less than gc_grace_seconds.
> > >
> > > Regards,
> > > Kyrill
> > > From: Agrawal, Pratik 
> > > Sent: Monday, August 6, 2018 8:22:27 PM
> > > To: user@cassandra.apache.org
> > > Subject: Hinted Handoff
> > >
> > > Hello all,
> > > We use Cassandra in non-conventional way, where our data is short termed 
> > > (life cycle of about 20-30 minutes) where each record is updated ~5 times 
> > > and then deleted. We have GC grace of 15 minutes.
> > > We are seeing 2 problems
> > > 1.) A certain number of Cassandra nodes goes down and then we remove it 
> > > from the cluster using Cassandra removenode command and replace the dead 
> > > nodes with new nodes. While new nodes are joining in, we see more nodes 
> > > down (which are not actually down) but we see following errors in the log
> > > “Gossip not settled after 321 polls. Gossip Stage 
> > > active/pending/completed: 1/816/0”
> > >
> > > To fix the issue, I restarted the server and the nodes now appear to be 
> > > up and the problem is solved
> > >
> > > Can this problem be related to 
> > > https://issues.apache.org/jira/browse/CASSANDRA-6590 ?
> > >
> > > 2.) Meanwhile, after restarting the nodes mentioned above, we see that 
> > > some old deleted data is resurrected (because of short lifecycle of our 
> > > data). My guess at the moment is that these data is resurrected due to 
> > > hinted handoff. Interesting point to note here is that data keeps 
> > > resurrecting at periodic intervals (like an hour) and then finally stops. 
> > > Could this be caused by hinted handoff? if so is there any setting which 
> > > we can set to specify that “invalidate, hinted handoff data after 5-10 
> > > minutes”.
> > >
> > > Thanks,
> > > Pratik
>

Re: Hinted Handoff

[kurt greaves]

>
> Does Cassandra TTL out the hints after max_hint_window_in_ms? From my
> understanding, Cassandra only stops collecting hints after
> max_hint_window_in_ms but can still keep replaying the hints if the node
> comes back again. Is this correct? Is there a way to TTL out hints?


No, but it won't send hints that have passed HH window. Also, this
shouldn't be caused by HH as the hints maintain the original timestamp with
which they were written.

Honestly, this sounds more like a use case for a distributed cache rather
than Cassandra. Keeping data for 30 minutes and then deleting it is going
to be a nightmare to manage in Cassandra.

On 7 August 2018 at 07:20, Agrawal, Pratik 
wrote:

> Does Cassandra TTL out the hints after max_hint_window_in_ms? From my
> understanding, Cassandra only stops collecting hints after
> max_hint_window_in_ms but can still keep replaying the hints if the node
> comes back again. Is this correct? Is there a way to TTL out hints?
>
>
>
> Thanks,
>
> Pratik
>
>
>
> *From: *Kyrylo Lebediev 
> *Reply-To: *"user@cassandra.apache.org" 
> *Date: *Monday, August 6, 2018 at 4:10 PM
> *To: *"user@cassandra.apache.org" 
> *Subject: *Re: Hinted Handoff
>
>
>
> Small gc_grace_seconds value lowers max allowed node downtime, which is 15
> minutes in your case. After 15 minutes of downtime you'll need to replace
> the node, as you described. This interval looks too short to be able to do
> planned maintenance. So, in case you set larger value for gc_grace_seconds
> (lets say, hours or a day) will you get visible read amplification / waste
> a lot of disk space / issues with compactions?
>
>
>
> Hinted handoff may be the reason in case hinted handoff window is longer
> than gc_grace_seconds. To me it looks like hinted handoff window
> (max_hint_window_in_ms in cassandra.yaml, which defaults to 3h) must always
> be set to a value less than gc_grace_seconds.
>
>
>
> Regards,
>
> Kyrill
> ------
>
> *From:* Agrawal, Pratik 
> *Sent:* Monday, August 6, 2018 8:22:27 PM
> *To:* user@cassandra.apache.org
> *Subject:* Hinted Handoff
>
>
>
> Hello all,
>
> We use Cassandra in non-conventional way, where our data is short termed
> (life cycle of about 20-30 minutes) where each record is updated ~5 times
> and then deleted. We have GC grace of 15 minutes.
>
> We are seeing 2 problems
>
> 1.) A certain number of Cassandra nodes goes down and then we remove it
> from the cluster using Cassandra removenode command and replace the dead
> nodes with new nodes. While new nodes are joining in, we see more nodes
> down (which are not actually down) but we see following errors in the log
>
> “Gossip not settled after 321 polls. Gossip Stage
> active/pending/completed: 1/816/0”
>
>
>
> To fix the issue, I restarted the server and the nodes now appear to be up
> and the problem is solved
>
>
>
> Can this problem be related to https://issues.apache.org/
> jira/browse/CASSANDRA-6590 ?
>
>
>
> 2.) Meanwhile, after restarting the nodes mentioned above, we see that
> some old deleted data is resurrected (because of short lifecycle of our
> data). My guess at the moment is that these data is resurrected due to
> hinted handoff. Interesting point to note here is that data keeps
> resurrecting at periodic intervals (like an hour) and then finally stops.
> Could this be caused by hinted handoff? if so is there any setting which we
> can set to specify that “invalidate, hinted handoff data after 5-10
> minutes”.
>
>
>
> Thanks,
> Pratik
>

Re: Hinted Handoff

[Agrawal, Pratik]

Does Cassandra TTL out the hints after max_hint_window_in_ms? From my 
understanding, Cassandra only stops collecting hints after 
max_hint_window_in_ms but can still keep replaying the hints if the node comes 
back again. Is this correct? Is there a way to TTL out hints?

Thanks,
Pratik

From: Kyrylo Lebediev 
Reply-To: "user@cassandra.apache.org" 
Date: Monday, August 6, 2018 at 4:10 PM
To: "user@cassandra.apache.org" 
Subject: Re: Hinted Handoff


Small gc_grace_seconds value lowers max allowed node downtime, which is 15 
minutes in your case. After 15 minutes of downtime you'll need to replace the 
node, as you described. This interval looks too short to be able to do planned 
maintenance. So, in case you set larger value for gc_grace_seconds (lets say, 
hours or a day) will you get visible read amplification / waste a lot of disk 
space / issues with compactions?



Hinted handoff may be the reason in case hinted handoff window is longer than 
gc_grace_seconds. To me it looks like hinted handoff window 
(max_hint_window_in_ms in cassandra.yaml, which defaults to 3h) must always be 
set to a value less than gc_grace_seconds.



Regards,

Kyrill


From: Agrawal, Pratik 
Sent: Monday, August 6, 2018 8:22:27 PM
To: user@cassandra.apache.org
Subject: Hinted Handoff


Hello all,

We use Cassandra in non-conventional way, where our data is short termed (life 
cycle of about 20-30 minutes) where each record is updated ~5 times and then 
deleted. We have GC grace of 15 minutes.

We are seeing 2 problems

1.) A certain number of Cassandra nodes goes down and then we remove it from 
the cluster using Cassandra removenode command and replace the dead nodes with 
new nodes. While new nodes are joining in, we see more nodes down (which are 
not actually down) but we see following errors in the log

“Gossip not settled after 321 polls. Gossip Stage active/pending/completed: 
1/816/0”



To fix the issue, I restarted the server and the nodes now appear to be up and 
the problem is solved



Can this problem be related to 
https://issues.apache.org/jira/browse/CASSANDRA-6590 ?



2.) Meanwhile, after restarting the nodes mentioned above, we see that some old 
deleted data is resurrected (because of short lifecycle of our data). My guess 
at the moment is that these data is resurrected due to hinted handoff. 
Interesting point to note here is that data keeps resurrecting at periodic 
intervals (like an hour) and then finally stops. Could this be caused by hinted 
handoff? if so is there any setting which we can set to specify that 
“invalidate, hinted handoff data after 5-10 minutes”.



Thanks,
Pratik

Re: Hinted Handoff

[Kyrylo Lebediev]

Small gc_grace_seconds value lowers max allowed node downtime, which is 15 
minutes in your case. After 15 minutes of downtime you'll need to replace the 
node, as you described. This interval looks too short to be able to do planned 
maintenance. So, in case you set larger value for gc_grace_seconds (lets say, 
hours or a day) will you get visible read amplification / waste a lot of disk 
space / issues with compactions?


Hinted handoff may be the reason in case hinted handoff window is longer than 
gc_grace_seconds. To me it looks like hinted handoff window 
(max_hint_window_in_ms in cassandra.yaml, which defaults to 3h) must always be 
set to a value less than gc_grace_seconds.


Regards,

Kyrill


From: Agrawal, Pratik 
Sent: Monday, August 6, 2018 8:22:27 PM
To: user@cassandra.apache.org
Subject: Hinted Handoff


Hello all,

We use Cassandra in non-conventional way, where our data is short termed (life 
cycle of about 20-30 minutes) where each record is updated ~5 times and then 
deleted. We have GC grace of 15 minutes.

We are seeing 2 problems

1.) A certain number of Cassandra nodes goes down and then we remove it from 
the cluster using Cassandra removenode command and replace the dead nodes with 
new nodes. While new nodes are joining in, we see more nodes down (which are 
not actually down) but we see following errors in the log

“Gossip not settled after 321 polls. Gossip Stage active/pending/completed: 
1/816/0”



To fix the issue, I restarted the server and the nodes now appear to be up and 
the problem is solved



Can this problem be related to 
https://issues.apache.org/jira/browse/CASSANDRA-6590 ?



2.) Meanwhile, after restarting the nodes mentioned above, we see that some old 
deleted data is resurrected (because of short lifecycle of our data). My guess 
at the moment is that these data is resurrected due to hinted handoff. 
Interesting point to note here is that data keeps resurrecting at periodic 
intervals (like an hour) and then finally stops. Could this be caused by hinted 
handoff? if so is there any setting which we can set to specify that 
“invalidate, hinted handoff data after 5-10 minutes”.



Thanks,
Pratik

Hinted Handoff

[Agrawal, Pratik]

Hello all,

We use Cassandra in non-conventional way, where our data is short termed (life 
cycle of about 20-30 minutes) where each record is updated ~5 times and then 
deleted. We have GC grace of 15 minutes.

We are seeing 2 problems

1.) A certain number of Cassandra nodes goes down and then we remove it from 
the cluster using Cassandra removenode command and replace the dead nodes with 
new nodes. While new nodes are joining in, we see more nodes down (which are 
not actually down) but we see following errors in the log

“Gossip not settled after 321 polls. Gossip Stage active/pending/completed: 
1/816/0”



To fix the issue, I restarted the server and the nodes now appear to be up and 
the problem is solved



Can this problem be related to 
https://issues.apache.org/jira/browse/CASSANDRA-6590 ?



2.) Meanwhile, after restarting the nodes mentioned above, we see that some old 
deleted data is resurrected (because of short lifecycle of our data). My guess 
at the moment is that these data is resurrected due to hinted handoff. 
Interesting point to note here is that data keeps resurrecting at periodic 
intervals (like an hour) and then finally stops. Could this be caused by hinted 
handoff? if so is there any setting which we can set to specify that 
“invalidate, hinted handoff data after 5-10 minutes”.



Thanks,
Pratik

Re: Limitations of Hinted Handoff OverloadedException exception

[kurt greaves]

The coordinator will refuse to send writes/hints to a node if it has a
large backlog of hints (128 * #cores) already and the destination replica
is one of the nodes with hints destined to it.
It will still send writes to any "healthy" node (a node with no outstanding
hints).

The idea is to not further overload already overloaded nodes. If you see
OverloadedExceptions you'll have to repair after all nodes become stable.

See StorageProxy.java#L1327

 called from StorageProxy.java::sendToHintedEndpoints()


On 13 July 2018 at 05:38, Karthick V  wrote:

> Refs : https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/
> opsRepairNodesHintedHandoff.html
>
> On Thu, Jul 12, 2018 at 7:46 PM Karthick V  wrote:
>
>> Hi everyone,
>>
>>  If several nodes experience brief outages simultaneously, substantial
>>> memory pressure can build up on the coordinator.* The coordinator
>>> tracks how many hints it is currently writing, and if the number increases
>>> too much, the coordinator refuses writes and throws the *
>>> *OverloadedException exception.*
>>
>>
>>  In the above statement, it is been said that after some extent(of
>> hints) the* coordinator *will refuse to writes. can someone explain the
>> depth of this limitations and its dependency if any (like disk size or any)?
>>
>> Regards
>> Karthick V
>>
>>
>>

Re: Limitations of Hinted Handoff OverloadedException exception

[Karthick V]

Refs :
https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsRepairNodesHintedHandoff.html

On Thu, Jul 12, 2018 at 7:46 PM Karthick V  wrote:

> Hi everyone,
>
>  If several nodes experience brief outages simultaneously, substantial
>> memory pressure can build up on the coordinator.* The coordinator tracks
>> how many hints it is currently writing, and if the number increases too
>> much, the coordinator refuses writes and throws the *
>> *OverloadedException exception.*
>
>
>  In the above statement, it is been said that after some extent(of
> hints) the* coordinator *will refuse to writes. can someone explain the
> depth of this limitations and its dependency if any (like disk size or any)?
>
> Regards
> Karthick V
>
>
>

Limitations of Hinted Handoff OverloadedException exception

[Karthick V]

Hi everyone,

 If several nodes experience brief outages simultaneously, substantial
> memory pressure can build up on the coordinator.* The coordinator tracks
> how many hints it is currently writing, and if the number increases too
> much, the coordinator refuses writes and throws the *
> *OverloadedException exception.*


 In the above statement, it is been said that after some extent(of
hints) the* coordinator *will refuse to writes. can someone explain the
depth of this limitations and its dependency if any (like disk size or any)?

Regards
Karthick V

Re: about the tombstone and hinted handoff

[kurt greaves]

I don't think that's true/maybe that comment is misleading. Tombstones
AFAIK will be propagated by hints, and the hint system doesn't do anything
to check if a particular row has been tombstoned. To the node receiving the
hints it just looks like it's receiving a bunch of writes, it doesn't know
they are hints.

On 12 April 2018 at 13:51, Jinhua Luo <luajit...@gmail.com> wrote:

> Hi All,
>
> In the doc:
> https://docs.datastax.com/en/cassandra/3.0/cassandra/dml/dml
> AboutDeletes.html
>
> It said "When an unresponsive node recovers, Cassandra uses hinted
> handoff to replay the database mutationsthe node missed while it was
> down. Cassandra does not replay a mutation for a tombstoned record
> during its grace period.".
>
> The tombstone here is on the recovered node or coordinator?
> The tombstone is a special write record, so it must have writetime.
> We could compare the writetime between the version in the hint and the
> version of the tombstone, which is enough to make choice, so why we
> need to wait for gc_grace_seconds here?
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

about the tombstone and hinted handoff

[Jinhua Luo]

Hi All,

In the doc:
https://docs.datastax.com/en/cassandra/3.0/cassandra/dml/dmlAboutDeletes.html

It said "When an unresponsive node recovers, Cassandra uses hinted
handoff to replay the database mutationsthe node missed while it was
down. Cassandra does not replay a mutation for a tombstoned record
during its grace period.".

The tombstone here is on the recovered node or coordinator?
The tombstone is a special write record, so it must have writetime.
We could compare the writetime between the version in the hint and the
version of the tombstone, which is enough to make choice, so why we
need to wait for gc_grace_seconds here?

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

hinted handoff replay for a tombstoned record

[Karthick V]

Hi,
 Recently I Came across this line

Cassandra uses hinted handoff
> <https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsRepairNodesHintedHandoff.html>
>  to replay the database mutations
> <https://docs.datastax.com/en/glossary/doc/glossary/gloss_mutation.html> the
> node missed while it was down. *Cassandra does not replay a mutation for
> a tombstoned record during its grace period*
>

Can any explain why we don't do so?

Ref :
https://docs.datastax.com/en/cassandra/3.0/cassandra/dml/dmlAboutDeletes.html

Re: Hinted handoff throttled even after "nodetool sethintedhandoffthrottlekb 0"

[Andrew Bialecki]

Bit more information. Using jmxterm and inspecting the state of a node when
it's "slow" playing hints, I can see the following from the node that has
hints to play:

$>get MaxHintsInProgress
#mbean = org.apache.cassandra.db:type=StorageProxy:
MaxHintsInProgress = 2048;

$>get HintsInProgress
#mbean = org.apache.cassandra.db:type=StorageProxy:
HintsInProgress = 0;

$>get TotalHints
#mbean = org.apache.cassandra.db:type=StorageProxy:
TotalHints = 129687;

Is there some throttling that would cause hints to not be played at all if,
for instance, the cluster has enough load or something related to a timeout
setting?

On Fri, Oct 27, 2017 at 1:49 AM, Andrew Bialecki <
andrew.biale...@klaviyo.com> wrote:

> We have a 96 node cluster running 3.11 with 256 vnodes each. We're running
> a rolling restart. As we restart nodes, we notice that each node takes a
> while to have all other nodes be marked as up and this corresponds to nodes
> that haven't finished playing hints.
>
> We looked at the hinted handoff throttling, noticed it was still the
> default of 1024, so we tried to turn it off by setting it to zero. Reading
> the source, it looks like that rate limiting won't take affect until the
> current set of hints have finished. So we made that change cluster wide and
> then restarted the next node. However, we still saw the same issue.
>
> Looking at iftop and network throughput, it's very low (~10kB/s) and
> therefore the few 100k of hints that accumulate while the node is restart
> end up take several minutes to get sent.
>
> Any other knobs we should be tuning to increase hinted handoff throughput?
> Or other reasons why hinted handoff runs so slowly?
>
> --
> Andrew Bialecki
>



-- 
Andrew Bialecki

<https://www.klaviyo.com/>

Hinted handoff throttled even after "nodetool sethintedhandoffthrottlekb 0"

[Andrew Bialecki]

We have a 96 node cluster running 3.11 with 256 vnodes each. We're running
a rolling restart. As we restart nodes, we notice that each node takes a
while to have all other nodes be marked as up and this corresponds to nodes
that haven't finished playing hints.

We looked at the hinted handoff throttling, noticed it was still the
default of 1024, so we tried to turn it off by setting it to zero. Reading
the source, it looks like that rate limiting won't take affect until the
current set of hints have finished. So we made that change cluster wide and
then restarted the next node. However, we still saw the same issue.

Looking at iftop and network throughput, it's very low (~10kB/s) and
therefore the few 100k of hints that accumulate while the node is restart
end up take several minutes to get sent.

Any other knobs we should be tuning to increase hinted handoff throughput?
Or other reasons why hinted handoff runs so slowly?

-- 
Andrew Bialecki

Re: Hinted handoff not working

[Robert Wille]

Nope. I added millions of records and several GB to the cluster while one node 
was down, and then ran nodetool flush system hints on a couple of nodes that 
were up, and system/hints has less than 200K in it.

Here’s the relevant part of nodetool cfstats system.hints:

Keyspace: system
Read Count: 28572
Read Latency: 0.01806502869942601 ms.
Write Count: 351
Write Latency: 0.04547008547008547 ms.
Pending Tasks: 0
Table: hints
SSTable count: 1
Space used (live), bytes: 7446
Space used (total), bytes: 80062
SSTable Compression Ratio: 0.2651441528992549
Number of keys (estimate): 128
Memtable cell count: 1
Memtable data size, bytes: 1740

The hints are definitely not being stored.

Robert

On Dec 14, 2014, at 11:44 PM, Jens Rantil 
jens.ran...@tink.semailto:jens.ran...@tink.se wrote:

Hi Robert ,

Maybe you need to flush your memtables to actually see the disk usage increase? 
This applies to both hosts.

Cheers,
Jens




On Sun, Dec 14, 2014 at 3:52 PM, Robert Wille 
rwi...@fold3.commailto:rwi...@fold3.com wrote:

I have a cluster with RF=3. If I shut down one node, add a bunch of data to the 
cluster, I don’t see a bunch of records added to system.hints. Also, du of 
/var/lib/cassandra/data/system/hints of the nodes that are up shows that hints 
aren’t being stored. When I start the down node, its data doesn’t grow until I 
run repair, which then takes a really long time because it is significantly out 
of date. Is there some magic setting I cannot find in the documentation to 
enable hinted handoff? I’m running 2.0.11. Any insights would be greatly 
appreciated.

Thanks

Robert

Hinted handoff not working

[Robert Wille]

I have a cluster with RF=3. If I shut down one node, add a bunch of data to the 
cluster, I don’t see a bunch of records added to system.hints. Also, du of 
/var/lib/cassandra/data/system/hints of the nodes that are up shows that hints 
aren’t being stored. When I start the down node, its data doesn’t grow until I 
run repair, which then takes a really long time because it is significantly out 
of date. Is there some magic setting I cannot find in the documentation to 
enable hinted handoff? I’m running 2.0.11. Any insights would be greatly 
appreciated. 

Thanks

Robert

Re: Hinted handoff not working

[Rahul Neelakantan]

http://www.datastax.com/documentation/cassandra/2.0/cassandra/configuration/configCassandra_yaml_r.html?scroll=reference_ds_qfg_n1r_1k__hinted_handoff_enabled

Rahul

 On Dec 14, 2014, at 9:46 AM, Robert Wille rwi...@fold3.com wrote:
 
 I have a cluster with RF=3. If I shut down one node, add a bunch of data to 
 the cluster, I don’t see a bunch of records added to system.hints. Also, du 
 of /var/lib/cassandra/data/system/hints of the nodes that are up shows that 
 hints aren’t being stored. When I start the down node, its data doesn’t grow 
 until I run repair, which then takes a really long time because it is 
 significantly out of date. Is there some magic setting I cannot find in the 
 documentation to enable hinted handoff? I’m running 2.0.11. Any insights 
 would be greatly appreciated. 
 
 Thanks
 
 Robert
 

Re: Hinted handoff not working

[Robert Wille]

I’ve got hinted_handoff_enabled: true in cassandra.yaml. My settings are all 
default except for the DC, listen addresses and snitch. I should have mentioned 
this in my original post.

On Dec 14, 2014, at 8:02 AM, Rahul Neelakantan ra...@rahul.be wrote:

 http://www.datastax.com/documentation/cassandra/2.0/cassandra/configuration/configCassandra_yaml_r.html?scroll=reference_ds_qfg_n1r_1k__hinted_handoff_enabled
 
 Rahul
 
 On Dec 14, 2014, at 9:46 AM, Robert Wille rwi...@fold3.com wrote:
 
 I have a cluster with RF=3. If I shut down one node, add a bunch of data to 
 the cluster, I don’t see a bunch of records added to system.hints. Also, du 
 of /var/lib/cassandra/data/system/hints of the nodes that are up shows that 
 hints aren’t being stored. When I start the down node, its data doesn’t grow 
 until I run repair, which then takes a really long time because it is 
 significantly out of date. Is there some magic setting I cannot find in the 
 documentation to enable hinted handoff? I’m running 2.0.11. Any insights 
 would be greatly appreciated. 
 
 Thanks
 
 Robert
 

Re: Hinted handoff not working

[Jens Rantil]

Hi Robert ,

Maybe you need to flush your memtables to actually see the disk usage increase? 
This applies to both hosts.

Cheers,
Jens

On Sun, Dec 14, 2014 at 3:52 PM, Robert Wille rwi...@fold3.com wrote:

 I have a cluster with RF=3. If I shut down one node, add a bunch of data to 
 the cluster, I don’t see a bunch of records added to system.hints. Also, du 
 of /var/lib/cassandra/data/system/hints of the nodes that are up shows that 
 hints aren’t being stored. When I start the down node, its data doesn’t grow 
 until I run repair, which then takes a really long time because it is 
 significantly out of date. Is there some magic setting I cannot find in the 
 documentation to enable hinted handoff? I’m running 2.0.11. Any insights 
 would be greatly appreciated. 
 Thanks
 Robert

Observations/concerns with repair and hinted handoff

[Robert Wille]

I have spent a lot of time working with single-node, RF=1 clusters in my 
development. Before I deploy a cluster to our live environment, I have spent 
some time learning how to work with a multi-node cluster with RF=3. There were 
some surprises. I’m wondering if people here can enlighten me. I don’t exactly 
have that warm, fuzzy feeling.

I created a three-node cluster with RF=3. I then wrote to the cluster pretty 
heavily to cause some dropped mutation messages. The dropped messages didn’t 
trickle in, but came in a burst. I suspect full GC is the culprit, but I don’t 
really know. Anyway, I ended up with 17197 dropped mutation messages on node 1, 
6422 on node 2, and none on node 3. In order to learn about repair, I waited 
for compaction to finish doing its thing, recorded the size and estimated 
number of keys for each table, started up repair (nodetool repair keyspace) 
on all three nodes, and waited for it to complete before doing anything else 
(even reads). When repair and compaction were done, I checked the size and 
estimated number of keys for each table. All tables on all nodes grew in size 
and estimated number of keys. The estimated number of keys for each node grew 
by 65k, 272k and 247k (.2%, .7% and .6%) for nodes 1, 2 and 3 respectively. I 
expected some growth, but that’s significantly more new keys than I had dropped 
mutation messages. I also expected the most new data on node 1, and none on 
node 3, which didn’t come close to what actually happened. Perhaps a mutation 
message contains more than one record? Perhaps the dropped mutation message 
counter is incremented on the coordinator, not the node that was overloaded?

I repeated repair, and the second time around the tables remained unchanged, as 
expected. I would hope that repair wouldn’t do anything to the tables if they 
were in sync. 

Just to be clear, I’m not overly concerned about the unexpected increase in 
number of keys. I’m pretty sure that repair did the needful thing and did bring 
the nodes in sync. The unexpected results more likely indicates that I’m 
ignorant, and it really bothers me when I don’t understand something. If you 
have any insights, I’d appreciate them.

One of the dismaying things about repair was that the first time around it took 
about 4 hours, with a completely idle cluster (except for repairs, of course), 
and only 6 GB of data on each node. I can bootstrap a node with 6 GB of data in 
a couple of minutes. That makes repair something like 50 to 100 times more 
expensive than bootstrapping. I know I should run repair on one node at a time, 
but even if you divide by three, that’s still a horrifically long time for such 
a small amount of data. The second time around, repair only took 30 minutes. 
That’s much better, but best-case is still about 10x longer than bootstrapping. 
Should repair really be taking this long? When I have 300 GB of data, is a 
best-case repair going to take 25 hours, and a repair with a modest amount of 
work more than 100 hours? My records are quite small. Those 6 GB contain almost 
40 million partitions. 

Following my repair experiment, I added a fourth node, and then tried killing a 
node and importing a bunch of data while the node was down. As far as repair is 
concerned, this seems to work fine (although again, glacially). However, I 
noticed that hinted handoff doesn’t seem to be working. I added several million 
records (with consistency=one), and nothing appeared in system.hints (du -hs 
showed a few dozen K bytes), nor did I get any pending Hinted Handoff tasks in 
the Thread Pool Stats. When I started up the down node (less than 3 hours 
later), the missed data didn’t appear to get sent to it. The tables did not 
grow, compaction events didn’t schedule, and there wasn’t any appreciable CPU 
utilization by the cluster. With millions of records that were missed while it 
was down, I should have noticed something if it actually was replaying the 
hints. Is there some magic setting to turn on hinted handoffs? Were there too 
many hints and so it just deleted them? My assumption is that if hinted handoff 
is working, then my need for repair should be much less, which given my 
experience so far, would be a really good thing.

Given the horrifically long time it takes to repair a node, and hinted handoff 
apparently not working, if a node goes down, is it better to bootstrap a new 
one than to repair the node that went down? I would expect that even if I chose 
to bootstrap a new node, it would need to be repaired anyway, since it would 
probably miss writes while bootstrapping.

Thanks in advance

Robert

Re: Side effects of hinted handoff lead to consistency problem

[Jason Tang]

After check the log and configuration, I found it caused by two reason.

 1. GC grace seconds
I using hector client to connect cassandra, and the default value of GC
grace seconds for each column family is **Zero** ! So when hinted handoff
replay the temporary value, the tombstone on other two node is deleted by
compaction. And then client will get the temporary value.

 2. Secondary index
Even after fix the first problem, I can still get temporary result from
cassandra client. And I use the command like get my_cf where
column_one='value'  to query the data, then the temporary value show
again. But when I using the raw key to query the record again, it
disappeared.
And from client, we always using row key to get the data, and in this
way, I didn't get the temporary value.

So it seems the secondary index is not restricted by the consistency
configuration.

And when I change GC grace seconds to 10 days. our problem solved, but
it is still a strange behavior when using index query.


2013/10/8 Jason Tang ares.t...@gmail.com

 I have a 3 nodes cluster, replicate_factor is 3 also. Consistency level is
 Write quorum, Read quorum.
 Traffic has three major steps
 Create:
 Rowkey: 
 Column: status=new, requests=x
 Update:
  Rowkey: 
  Column: status=executing, requests=x
 Delete:
  Rowkey: 

 When one node down, it can work according to consistency configuration,
 and the final status is all requests are finished and delete.

 So if running cassandra client to list the result (also set consistency
 quorum). It shows empty (only rowkey left)， which is correct.

 But if we start the dead node, the hinted handoff model will write back
 the data to this node. So there are lots of create, update, delete.

 I don't know due to GC or compaction, the delete records on other two
 nodes seems not work, and if using cassandra client to list the data (also
 consistency quorum), the deleted row show again with column value.

 And if using client to check the data several times, you can find the data
 is changed, seems hinted handoff replay operation, the deleted data show up
 and then disappear.

 So the hinted handoff mechanism will faster the repair, but the temporary
 data will be seen from external (if data is deleted).

 Is there a way to have this procedure invisible from external, until the
 hinted handoff finished?

 What I want is final status synchronization, the temporary status is out
 of date and also incorrect, should never been seen from external.

 Is it due to row delete instead of column delete? Or compaction?

Side effects of hinted handoff lead to consistency problem

[Jason Tang]

I have a 3 nodes cluster, replicate_factor is 3 also. Consistency level is
Write quorum, Read quorum.
Traffic has three major steps
Create:
Rowkey: 
Column: status=new, requests=x
Update:
 Rowkey: 
 Column: status=executing, requests=x
Delete:
 Rowkey: 

When one node down, it can work according to consistency configuration, and
the final status is all requests are finished and delete.

So if running cassandra client to list the result (also set consistency
quorum). It shows empty (only rowkey left)， which is correct.

But if we start the dead node, the hinted handoff model will write back the
data to this node. So there are lots of create, update, delete.

I don't know due to GC or compaction, the delete records on other two nodes
seems not work, and if using cassandra client to list the data (also
consistency quorum), the deleted row show again with column value.

And if using client to check the data several times, you can find the data
is changed, seems hinted handoff replay operation, the deleted data show up
and then disappear.

So the hinted handoff mechanism will faster the repair, but the temporary
data will be seen from external (if data is deleted).

Is there a way to have this procedure invisible from external, until the
hinted handoff finished?

What I want is final status synchronization, the temporary status is out of
date and also incorrect, should never been seen from external.

Is it due to row delete instead of column delete? Or compaction?

Re: Decommissioned nodes not leaving and Hinted Handoff flood

[aaron morton]

Thanks for sharing, here is some more information…

 1 - At first, one of my node came down 5 min and when it came back it get 
 flooded by Hinted Handoff so hard that it could not handle the real time 
 queries properly. I haven't find a way to prioritize app queries rather than 
 Hinted Handoff.
You can disable hint delivery with nodetool pausehandoff or reduce the hint 
throughput 
https://github.com/apache/cassandra/blob/cassandra-1.2/conf/cassandra.yaml#L50
 
 2 - Nodes keep hints for a node that has been removed.
The hints are stored with a TTL that is the gc_grace_seconds for the CF a the 
time the hint is written, so they will eventually be purged by compaction. 

You can also delete the hints using the Hinted Handoff bean 
https://github.com/apache/cassandra/blob/cassandra-1.2/src/java/org/apache/cassandra/db/HintedHandOffManagerMBean.java#L30

 3 - Nodes with 500MB to 3GB hints stored for a removed node can't be 
 decommissioned, they stuck after streaming their data.
The hint KS is defined using the LocalStrategy and so it not replicated. They 
should not be involved in streaming. 

Cheers

-
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 10/07/2013, at 12:47 AM, Alain RODRIGUEZ arodr...@gmail.com wrote:

 Hi,
 
 C*1.2.2.
 
 I have removed 4 nodes with nodetool decommission. 2 of them have left with 
 no issue, while the 2 others nodes remained leaving even after streaming 
 their data.
 
 The only specific thing of these 2 nodes is that they had a lot of hints 
 pending. Hints from a node that couldn't come back and that I removed earlier 
 (because of the heavy load induced by Hinted Handoff while coming back, which 
 induced a lot of latencies in our app. This node didn't manage to come back 
 after 10 minutes, I removed it).
 
 So there I faced 3 bugs (or problems) :
 
 1 - At first, one of my node came down 5 min and when it came back it get 
 flooded by Hinted Handoff so hard that it could not handle the real time 
 queries properly. I haven't find a way to prioritize app queries rather than 
 Hinted Handoff.
 2 - Nodes keep hints for a node that has been removed.
 3 - Nodes with 500MB to 3GB hints stored for a removed node can't be 
 decommissioned, they stuck after streaming their data.
 
 
 As solutions for this 3 issues I did the following:
 
 Solution to 1 - I removed this down node (nodetool removenode)
 Solution to 2 - Stop the node remove system hints
 Solution to 3 - Stop the node and removenode instead of decommission
 
 Now I have no more issue, yet I felt I had to report this. Maybe my 
 experience can help users to get out of tricky situations and commiters to 
 detect some issues,  specially about hinted handoff.
 
 Alain
 
 

Decommissioned nodes not leaving and Hinted Handoff flood

[Alain RODRIGUEZ]

Hi,

C*1.2.2.

I have removed 4 nodes with nodetool decommission. 2 of them have left
with no issue, while the 2 others nodes remained leaving even after
streaming their data.

The only specific thing of these 2 nodes is that they had a lot of hints
pending. Hints from a node that couldn't come back and that I removed
earlier (because of the heavy load induced by Hinted Handoff while coming
back, which induced a lot of latencies in our app. This node didn't manage
to come back after 10 minutes, I removed it).

So there I faced 3 bugs (or problems) :

1 - At first, one of my node came down 5 min and when it came back it get
flooded by Hinted Handoff so hard that it could not handle the real time
queries properly. I haven't find a way to prioritize app queries rather
than Hinted Handoff.
2 - Nodes keep hints for a node that has been removed.
3 - Nodes with 500MB to 3GB hints stored for a removed node can't be
decommissioned, they stuck after streaming their data.


As solutions for this 3 issues I did the following:

Solution to 1 - I removed this down node (nodetool removenode)
Solution to 2 - Stop the node remove system hints
Solution to 3 - Stop the node and removenode instead of decommission

Now I have no more issue, yet I felt I had to report this. Maybe my
experience can help users to get out of tricky situations and commiters to
detect some issues,  specially about hinted handoff.

Alain

Re: Restart node = hinted handoff flood

[Michał Michalski]

My blind guess is: https://issues.apache.org/jira/browse/CASSANDRA-5179

In our case the only sensible solution was to pause hints delivery and 
disable storing them (both done with a nodetool: pausehandoff and 
disablehandoff). Once they TTL'd (3 hours by default I believe?) I 
turned HH on again and started to repair. However, problem has returned 
on the next day, so I had to do a quick C* upgrade with the version 
having this patch applied (we use a self-built 1.2.1 with a few 
additional patches applied).


M.

W dniu 04.07.2013 18:41, Alain RODRIGUEZ pisze:

The point is that there is no way, afaik, to limit the speed of these
Hinted Handoff since it's not a stream like repair or bootstrap, no way
either to keep the node out of the ring during the time it is receiving
hints since hints and normal traffic both go through gossip protocol on
port 7000.

How to avoid this Hinted Handoff flood on returning nodes ?

Alain


2013/7/4 Alain RODRIGUEZ arodr...@gmail.com


Hi,

Using C*1.2.2 12 EC2 xLarge cluster.

When I restart a node, if it spend a few minutes down, when I bring it up,
all the cpu are blocked at 100%, even once compactions are disabled,
inducing a very big and intolerable latency in my app. I suspect Hinted
Handoff to be the cause of this. disabling gossip fix the problem, enabling
it again brings the latency back (with a lot of gc, dropped messages...).

Is there a way to disable HH ? Are they responsible for this issue ?

I currently have this node down, any fast insight would be appreciated.

Alain

Restart node = hinted handoff flood

[Alain RODRIGUEZ]

Hi,

Using C*1.2.2 12 EC2 xLarge cluster.

When I restart a node, if it spend a few minutes down, when I bring it up,
all the cpu are blocked at 100%, even once compactions are disabled,
inducing a very big and intolerable latency in my app. I suspect Hinted
Handoff to be the cause of this. disabling gossip fix the problem, enabling
it again brings the latency back (with a lot of gc, dropped messages...).

Is there a way to disable HH ? Are they responsible for this issue ?

I currently have this node down, any fast insight would be appreciated.

Alain

Re: Restart node = hinted handoff flood

[Alain RODRIGUEZ]

The point is that there is no way, afaik, to limit the speed of these
Hinted Handoff since it's not a stream like repair or bootstrap, no way
either to keep the node out of the ring during the time it is receiving
hints since hints and normal traffic both go through gossip protocol on
port 7000.

How to avoid this Hinted Handoff flood on returning nodes ?

Alain


2013/7/4 Alain RODRIGUEZ arodr...@gmail.com

 Hi,

 Using C*1.2.2 12 EC2 xLarge cluster.

 When I restart a node, if it spend a few minutes down, when I bring it up,
 all the cpu are blocked at 100%, even once compactions are disabled,
 inducing a very big and intolerable latency in my app. I suspect Hinted
 Handoff to be the cause of this. disabling gossip fix the problem, enabling
 it again brings the latency back (with a lot of gc, dropped messages...).

 Is there a way to disable HH ? Are they responsible for this issue ?

 I currently have this node down, any fast insight would be appreciated.

 Alain

Re: unexplained hinted handoff

[Dane Miller]

On Sun, Apr 14, 2013 at 11:28 AM, aaron morton aa...@thelastpickle.com wrote:
  If hints are being stored, doesn't that imply DOWN nodes, and why don't I
 see that in the logs?

 Hints are stored for two reasons. First if the node is down when the write
 request starts, second if the node does not reply to the coordinator before
 rpc_timeout. If you are not seeing dropped write messages it may indicate
 network issues between the nodes.

Very helpful! I increased the timeouts for *_request_timeout settings
in cassandra.yaml and no longer see HH in the logs.  Still not sure
why I saw logs about dropped reads, and not dropped mutations.

Anyhow, big timeouts helps in this case.  Thanks :)

Dane

Re: unexplained hinted handoff

[aaron morton]

 Do slow reads trigger hint storage?
No. 
But dropped read messages is often an indicator that the node is overwhelmed.

  If hints are being stored, doesn't that imply DOWN nodes, and why don't I 
 see that in the logs?
Hints are stored for two reasons. First if the node is down when the write 
request starts, second if the node does not reply to the coordinator before 
rpc_timeout. If you are not seeing dropped write messages it may indicate 
network issues between the nodes. 

 I'm seeing hinted handoff kick in on all our nodes during periods of
 high activity,
Are you seeing log messages about hints been sent to nodes?

Cheers

  
-
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 13/04/2013, at 8:23 AM, Dane Miller d...@optimalsocial.com wrote:

 On Fri, Apr 12, 2013 at 1:12 PM, Dane Miller d...@optimalsocial.com wrote:
 I'm seeing hinted handoff kick in on all our nodes during periods of
 high activity, but all the nodes seem to be up (according to the logs
 and nodetool status).  The pattern in the logs is something like this:
 
 18:10:45 194 READ messages dropped in last 5000ms
 18:11:10 Started hinted handoff for host:
 7668c813-41a9-4d42-b362-5420528fefa0 with IP: /10
 18:11:11 Finished hinted handoff of 13 rows to endpoint /10
 
 This happens on all the nodes every 10 min, and with a different
 endpoint each time.  tpstats shows thousands of dropped reads, but no
 other types of messages are dropped.
 
 Do slow reads trigger hint storage?  If hints are being stored,
 doesn't that imply DOWN nodes, and why don't I see that in the logs?
 
 Sorry, meant to add: Cassandra 1.2.3, Ubuntu 12.04 x64

unexplained hinted handoff

[Dane Miller]

I'm seeing hinted handoff kick in on all our nodes during periods of
high activity, but all the nodes seem to be up (according to the logs
and nodetool status).  The pattern in the logs is something like this:

18:10:45 194 READ messages dropped in last 5000ms
18:11:10 Started hinted handoff for host:
7668c813-41a9-4d42-b362-5420528fefa0 with IP: /10
18:11:11 Finished hinted handoff of 13 rows to endpoint /10

This happens on all the nodes every 10 min, and with a different
endpoint each time.  tpstats shows thousands of dropped reads, but no
other types of messages are dropped.

Do slow reads trigger hint storage?  If hints are being stored,
doesn't that imply DOWN nodes, and why don't I see that in the logs?

Dane

Re: unexplained hinted handoff

[Dane Miller]

On Fri, Apr 12, 2013 at 1:12 PM, Dane Miller d...@optimalsocial.com wrote:
 I'm seeing hinted handoff kick in on all our nodes during periods of
 high activity, but all the nodes seem to be up (according to the logs
 and nodetool status).  The pattern in the logs is something like this:

 18:10:45 194 READ messages dropped in last 5000ms
 18:11:10 Started hinted handoff for host:
 7668c813-41a9-4d42-b362-5420528fefa0 with IP: /10
 18:11:11 Finished hinted handoff of 13 rows to endpoint /10

 This happens on all the nodes every 10 min, and with a different
 endpoint each time.  tpstats shows thousands of dropped reads, but no
 other types of messages are dropped.

 Do slow reads trigger hint storage?  If hints are being stored,
 doesn't that imply DOWN nodes, and why don't I see that in the logs?

Sorry, meant to add: Cassandra 1.2.3, Ubuntu 12.04 x64

Re: Hinted Handoff

[Michal Michalski]

It contains mutation (data) that is to be sent to proper endpoint.

M.

W dniu 25.03.2013 20:15, Kanwar Sangha pisze:

Hi - Quick question. Do hints contain the actual data or the data is read from 
the SStables and then sent to the other node when it comes up ?

Thanks,
Kanwar

Hinted Handoff

[Kanwar Sangha]

Hi - Quick question. Do hints contain the actual data or the data is read from 
the SStables and then sent to the other node when it comes up ?

Thanks,
Kanwar

Re: hinted handoff disabling trade-offs

[Matt Kap]

Thanks Aaron, appreciate the advice.

On Tue, Mar 19, 2013 at 3:14 AM, aaron morton aa...@thelastpickle.com wrote:
  I think I understand what it means for
 application-level data, but the part I'm not entirely sure about is
 what it could mean for Cassandra internals.

 Internally it means the write will not be retries to nodes that were either
 down or did not ack before rpc_timeout. That's all.

 If you are doing thing with read_repair_chance == 0 and CL ONE you are in a
 very eventually consistent world. The only thing that will guarantee
 consistency for you now is running nodetool repair.




 My cluster is under heavy write load. I'm considering disabling Hinted
 Handoffs so the nodes recover quicker in case compactions begin to
 back up.

 If the node cluster is approaching capacity, then ultimately the thing to do
 is add more nodes. The only things to do are disable the commit log and use
 a lower CL.

 If it's approaching capacity you will start to see pending mutations back
 up, maybe some dropped mutations and the maybe an increase in the difference
 between the latency reported in the proxyhistograms and the cfhistograms or
 cfstats.

 Cheers

 -
 Aaron Morton
 Freelance Cassandra Consultant
 New Zealand

 @aaronmorton
 http://www.thelastpickle.com

 On 16/03/2013, at 4:50 PM, Matt Kap matvey1...@gmail.com wrote:

 Thanks Aaron.

 I am using CL=ONE. read_repair_chance=0. The part which I'm wondering
 about is what happens to the internal Cassandra writes if Hinted
 Handoffs are disabled. I think I understand what it means for
 application-level data, but the part I'm not entirely sure about is
 what it could mean for Cassandra internals.

 My cluster is under heavy write load. I'm considering disabling Hinted
 Handoffs so the nodes recover quicker in case compactions begin to
 back up.

 On Wed, Mar 6, 2013 at 2:06 AM, aaron morton aa...@thelastpickle.com
 wrote:

 The advantage of HH is that it reduces the probability of a DigestMismatch
 when using a CL  ONE. A DigestMismatch means the read has to run a second
 time before returning to the client.

 - No risk of hinted-handoffs building up
 - No risk of hinted-handoffs flooding a node that just came up


 See the yaml config settings for the max hint window and the throttling.

 Can anyone suggest any other factors that I'm missing here. Specifically
 reasons
 not to do this.


 If you are doing this for performance first make sure your data model is
 efficient, that you are doing the most efficient reads (see my presentation
 here http://www.datastax.com/events/cassandrasummit2012/presentations), and
 your caching is bang on. Then consider if you can tune the CL, and if your
 client is token aware so it directs traffic to a node that has it.

 Cheers

 -
 Aaron Morton
 Freelance Cassandra Developer
 New Zealand

 @aaronmorton
 http://www.thelastpickle.com

 On 4/03/2013, at 9:19 PM, Michael Kjellman mkjell...@barracuda.com wrote:

 Also, if you have enough hints being created that its significantly
 impacting your heap I have a feeling things are going to get out of sync
 very quickly.

 On Mar 4, 2013, at 9:17 PM, Wz1975 wz1...@yahoo.com wrote:

 Why do you think disabling hinted handoff will improve memory usage?


 Thanks.
 -Wei

 Sent from my Samsung smartphone on ATT


  Original message 
 Subject: Re: hinted handoff disabling trade-offs
 From: Michael Kjellman mkjell...@barracuda.com
 To: user@cassandra.apache.org user@cassandra.apache.org
 CC:


 Repair is slow.

 On Mar 4, 2013, at 8:07 PM, Matt Kap matvey1...@gmail.com wrote:

 I am looking to get a second opinion about disabling hinted-handoffs. I
 have an application that can tolerate a fair amount of inconsistency
 (advertising domain), and so I'm weighting the pros and cons of hinted
 handoffs. I'm running Cassandra 1.0, looking to upgrade to 1.1 soon.

 Pros of disabling hinted handoffs:
 - Reduces heap
 - Improves GC performance
 - No risk of hinted-handoffs building up
 - No risk of hinted-handoffs flooding a node that just came up

 Cons
 - Some writes can be lost, at least until repair runs

 Can anyone suggest any other factors that I'm missing here. Specifically
 reasons
 not to do this.

 Cheers!
 -Matt


 Copy, by Barracuda, helps you store, protect, and share all your amazing
 things. Start today: www.copy.com.


 --
 Copy, by Barracuda, helps you store, protect, and share all your amazing
 things. Start today: www.copy.com.
  ­­





 --
 www.calcmachine.com - easy online calculator.





-- 
www.calcmachine.com - easy online calculator.

Re: hinted handoff disabling trade-offs

[Matt Kap]

Thanks Aaron.

I am using CL=ONE. read_repair_chance=0. The part which I'm wondering
about is what happens to the internal Cassandra writes if Hinted
Handoffs are disabled. I think I understand what it means for
application-level data, but the part I'm not entirely sure about is
what it could mean for Cassandra internals.

My cluster is under heavy write load. I'm considering disabling Hinted
Handoffs so the nodes recover quicker in case compactions begin to
back up.

On Wed, Mar 6, 2013 at 2:06 AM, aaron morton aa...@thelastpickle.com wrote:
 The advantage of HH is that it reduces the probability of a DigestMismatch
 when using a CL  ONE. A DigestMismatch means the read has to run a second
 time before returning to the client.

 - No risk of hinted-handoffs building up
 - No risk of hinted-handoffs flooding a node that just came up

 See the yaml config settings for the max hint window and the throttling.

 Can anyone suggest any other factors that I'm missing here. Specifically
 reasons
 not to do this.

 If you are doing this for performance first make sure your data model is
 efficient, that you are doing the most efficient reads (see my presentation
 here http://www.datastax.com/events/cassandrasummit2012/presentations), and
 your caching is bang on. Then consider if you can tune the CL, and if your
 client is token aware so it directs traffic to a node that has it.

 Cheers

 -
 Aaron Morton
 Freelance Cassandra Developer
 New Zealand

 @aaronmorton
 http://www.thelastpickle.com

 On 4/03/2013, at 9:19 PM, Michael Kjellman mkjell...@barracuda.com wrote:

 Also, if you have enough hints being created that its significantly
 impacting your heap I have a feeling things are going to get out of sync
 very quickly.

 On Mar 4, 2013, at 9:17 PM, Wz1975 wz1...@yahoo.com wrote:

 Why do you think disabling hinted handoff will improve memory usage?


 Thanks.
 -Wei

 Sent from my Samsung smartphone on ATT


  Original message 
 Subject: Re: hinted handoff disabling trade-offs
 From: Michael Kjellman mkjell...@barracuda.com
 To: user@cassandra.apache.org user@cassandra.apache.org
 CC:


 Repair is slow.

 On Mar 4, 2013, at 8:07 PM, Matt Kap matvey1...@gmail.com wrote:

 I am looking to get a second opinion about disabling hinted-handoffs. I
 have an application that can tolerate a fair amount of inconsistency
 (advertising domain), and so I'm weighting the pros and cons of hinted
 handoffs. I'm running Cassandra 1.0, looking to upgrade to 1.1 soon.

 Pros of disabling hinted handoffs:
 - Reduces heap
 - Improves GC performance
 - No risk of hinted-handoffs building up
 - No risk of hinted-handoffs flooding a node that just came up

 Cons
 - Some writes can be lost, at least until repair runs

 Can anyone suggest any other factors that I'm missing here. Specifically
 reasons
 not to do this.

 Cheers!
 -Matt

 Copy, by Barracuda, helps you store, protect, and share all your amazing
 things. Start today: www.copy.com.


 --
 Copy, by Barracuda, helps you store, protect, and share all your amazing
 things. Start today: www.copy.com.
   ­­





-- 
www.calcmachine.com - easy online calculator.

Re: Hinted handoff

[aaron morton]

 For example in my case running 'upgradesstables' on ~300GB node takes about 
 30+ hours.
+1
And repair.

  1 TB at 1Gb/s will take roughly 2.2hrs assume we stream from say 100 nodes…
All of these suggestions on node size boil down to: make sure compaction, 
repair, node replacement and node moves complete in a time that is acceptable 
to you. So would suggest you run some tests. 

Cheers

-
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 7/03/2013, at 5:06 AM, Michal Michalski mich...@opera.com wrote:

 I think it's still true, but not because of network-related issues, but 
 because of the maintenance problems it will cause during per-node operations. 
 For example in my case running 'upgradesstables' on ~300GB node takes about 
 30+ hours. The other IO-intensive operations will probably be a pain too.
 
 M.
 
 W dniu 07.03.2013 13:59, Kanwar Sangha pisze:
 Do you have 9.5TB on a single node ?
 In the normal case it's best to have around 300 to 500GB per node. With that 
 much data is will take a week to run repair or replace a failed node.
 
 Hi Aaron - This was true for pre 1.2 but with 1.2 and virtual nodes, does 
 this still hold ? 1 TB at 1Gb/s will take roughly 2.2hrs assume we stream 
 from say 100 nodes...
 
 
 From: aaron morton [mailto:aa...@thelastpickle.com]
 Sent: 06 March 2013 23:47
 To: user@cassandra.apache.org
 Subject: Re: Hinted handoff
 
 Check the IO utilisation using iostat
 
 You *really* should not need to make HH run faster, if you do there is some 
 thing bad going on. I would consider dropping the hints and running repair.
 
 Data is ~9.5 TB
 Do you have 9.5TB on a single node ?
 In the normal case it's best to have around 300 to 500GB per node. With that 
 much data is will take a week to run repair or replace a failed node.
 
 Cheers
 
 -
 Aaron Morton
 Freelance Cassandra Developer
 New Zealand
 
 @aaronmorton
 http://www.thelastpickle.com
 
 On 6/03/2013, at 1:22 PM, Kanwar Sangha 
 kan...@mavenir.commailto:kan...@mavenir.com wrote:
 
 
 Is this correct ?
 
 I have Raid 0 setup for 16 TB across 8 disks. Each disk is 7.2kRPM with IOPS 
 of 80 per disk. Data is ~9.5 TB
 
 So 4K * 80 * 9.5 = 3040 KB ~  23.75 Mb/s.
 
 So basically I am limited at the disk rather than the n/w
 
 From: Kanwar Sangha [mailto:kan...@mavenir.comhttp://mavenir.com]
 Sent: 06 March 2013 15:11
 To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
 Subject: RE: Hinted handoff
 
 After trying to bump up the hinted_handoff_throttle_in_kb to 1G/b per sec, 
 It still does not go above 25Mb/s.  Is there a limitation ?
 
 
 
 From: Kanwar Sangha [mailto:kan...@mavenir.com]
 Sent: 06 March 2013 14:41
 To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
 Subject: RE: Hinted handoff
 
 Got the param. thanks
 
 From: Kanwar Sangha [mailto:kan...@mavenir.com]
 Sent: 06 March 2013 13:50
 To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
 Subject: Hinted handoff
 
 Hi - Is there a way to increase the hinted handoff throughput ? I am seeing 
 around 8Mb/s (bits).
 
 Thanks,
 Kanwar
 
 
 

RE: Hinted handoff

[Kanwar Sangha]

Do you have 9.5TB on a single node ?
In the normal case it's best to have around 300 to 500GB per node. With that 
much data is will take a week to run repair or replace a failed node.

Hi Aaron - This was true for pre 1.2 but with 1.2 and virtual nodes, does this 
still hold ? 1 TB at 1Gb/s will take roughly 2.2hrs assume we stream from say 
100 nodes...


From: aaron morton [mailto:aa...@thelastpickle.com]
Sent: 06 March 2013 23:47
To: user@cassandra.apache.org
Subject: Re: Hinted handoff

Check the IO utilisation using iostat

You *really* should not need to make HH run faster, if you do there is some 
thing bad going on. I would consider dropping the hints and running repair.

Data is ~9.5 TB
Do you have 9.5TB on a single node ?
In the normal case it's best to have around 300 to 500GB per node. With that 
much data is will take a week to run repair or replace a failed node.

Cheers

-
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 6/03/2013, at 1:22 PM, Kanwar Sangha 
kan...@mavenir.commailto:kan...@mavenir.com wrote:


Is this correct ?

I have Raid 0 setup for 16 TB across 8 disks. Each disk is 7.2kRPM with IOPS of 
80 per disk. Data is ~9.5 TB

So 4K * 80 * 9.5 = 3040 KB ~  23.75 Mb/s.

So basically I am limited at the disk rather than the n/w

From: Kanwar Sangha [mailto:kan...@mavenir.comhttp://mavenir.com]
Sent: 06 March 2013 15:11
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
Subject: RE: Hinted handoff

After trying to bump up the hinted_handoff_throttle_in_kb to 1G/b per sec, It 
still does not go above 25Mb/s.  Is there a limitation ?



From: Kanwar Sangha [mailto:kan...@mavenir.com]
Sent: 06 March 2013 14:41
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
Subject: RE: Hinted handoff

Got the param. thanks

From: Kanwar Sangha [mailto:kan...@mavenir.com]
Sent: 06 March 2013 13:50
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
Subject: Hinted handoff

Hi - Is there a way to increase the hinted handoff throughput ? I am seeing 
around 8Mb/s (bits).

Thanks,
Kanwar

Re: Hinted handoff

[Michal Michalski]

I think it's still true, but not because of network-related issues, but 
because of the maintenance problems it will cause during per-node 
operations. For example in my case running 'upgradesstables' on ~300GB 
node takes about 30+ hours. The other IO-intensive operations will 
probably be a pain too.


M.

W dniu 07.03.2013 13:59, Kanwar Sangha pisze:

Do you have 9.5TB on a single node ?
In the normal case it's best to have around 300 to 500GB per node. With that 
much data is will take a week to run repair or replace a failed node.

Hi Aaron - This was true for pre 1.2 but with 1.2 and virtual nodes, does this 
still hold ? 1 TB at 1Gb/s will take roughly 2.2hrs assume we stream from say 
100 nodes...


From: aaron morton [mailto:aa...@thelastpickle.com]
Sent: 06 March 2013 23:47
To: user@cassandra.apache.org
Subject: Re: Hinted handoff

Check the IO utilisation using iostat

You *really* should not need to make HH run faster, if you do there is some 
thing bad going on. I would consider dropping the hints and running repair.

Data is ~9.5 TB
Do you have 9.5TB on a single node ?
In the normal case it's best to have around 300 to 500GB per node. With that 
much data is will take a week to run repair or replace a failed node.

Cheers

-
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 6/03/2013, at 1:22 PM, Kanwar Sangha 
kan...@mavenir.commailto:kan...@mavenir.com wrote:


Is this correct ?

I have Raid 0 setup for 16 TB across 8 disks. Each disk is 7.2kRPM with IOPS of 
80 per disk. Data is ~9.5 TB

So 4K * 80 * 9.5 = 3040 KB ~  23.75 Mb/s.

So basically I am limited at the disk rather than the n/w

From: Kanwar Sangha [mailto:kan...@mavenir.comhttp://mavenir.com]
Sent: 06 March 2013 15:11
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
Subject: RE: Hinted handoff

After trying to bump up the hinted_handoff_throttle_in_kb to 1G/b per sec, It 
still does not go above 25Mb/s.  Is there a limitation ?



From: Kanwar Sangha [mailto:kan...@mavenir.com]
Sent: 06 March 2013 14:41
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
Subject: RE: Hinted handoff

Got the param. thanks

From: Kanwar Sangha [mailto:kan...@mavenir.com]
Sent: 06 March 2013 13:50
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
Subject: Hinted handoff

Hi - Is there a way to increase the hinted handoff throughput ? I am seeing 
around 8Mb/s (bits).

Thanks,
Kanwar

Hinted handoff

[Kanwar Sangha]

Hi - Is there a way to increase the hinted handoff throughput ? I am seeing 
around 8Mb/s (bits).

Thanks,
Kanwar

RE: Hinted handoff

[Kanwar Sangha]

Got the param. thanks

From: Kanwar Sangha [mailto:kan...@mavenir.com]
Sent: 06 March 2013 13:50
To: user@cassandra.apache.org
Subject: Hinted handoff

Hi - Is there a way to increase the hinted handoff throughput ? I am seeing 
around 8Mb/s (bits).

Thanks,
Kanwar

RE: Hinted handoff

[Kanwar Sangha]

After trying to bump up the hinted_handoff_throttle_in_kb to 1G/b per sec, It 
still does not go above 25Mb/s.  Is there a limitation ?



From: Kanwar Sangha [mailto:kan...@mavenir.com]
Sent: 06 March 2013 14:41
To: user@cassandra.apache.org
Subject: RE: Hinted handoff

Got the param. thanks

From: Kanwar Sangha [mailto:kan...@mavenir.com]
Sent: 06 March 2013 13:50
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
Subject: Hinted handoff

Hi - Is there a way to increase the hinted handoff throughput ? I am seeing 
around 8Mb/s (bits).

Thanks,
Kanwar

RE: Hinted handoff

[Kanwar Sangha]

Is this correct ?

I have Raid 0 setup for 16 TB across 8 disks. Each disk is 7.2kRPM with IOPS of 
80 per disk. Data is ~9.5 TB

So 4K * 80 * 9.5 = 3040 KB ~  23.75 Mb/s.

So basically I am limited at the disk rather than the n/w

From: Kanwar Sangha [mailto:kan...@mavenir.com]
Sent: 06 March 2013 15:11
To: user@cassandra.apache.org
Subject: RE: Hinted handoff

After trying to bump up the hinted_handoff_throttle_in_kb to 1G/b per sec, It 
still does not go above 25Mb/s.  Is there a limitation ?



From: Kanwar Sangha [mailto:kan...@mavenir.com]
Sent: 06 March 2013 14:41
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
Subject: RE: Hinted handoff

Got the param. thanks

From: Kanwar Sangha [mailto:kan...@mavenir.com]
Sent: 06 March 2013 13:50
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
Subject: Hinted handoff

Hi - Is there a way to increase the hinted handoff throughput ? I am seeing 
around 8Mb/s (bits).

Thanks,
Kanwar

Re: Hinted handoff

[aaron morton]

Check the IO utilisation using iostat

You *really* should not need to make HH run faster, if you do there is some 
thing bad going on. I would consider dropping the hints and running repair. 

 Data is ~9.5 TB
Do you have 9.5TB on a single node ? 
In the normal case it's best to have around 300 to 500GB per node. With that 
much data is will take a week to run repair or replace a failed node. 

Cheers
 
-
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 6/03/2013, at 1:22 PM, Kanwar Sangha kan...@mavenir.com wrote:

 Is this correct ?
  
 I have Raid 0 setup for 16 TB across 8 disks. Each disk is 7.2kRPM with IOPS 
 of 80 per disk. Data is ~9.5 TB
  
 So 4K * 80 * 9.5 = 3040 KB ~  23.75 Mb/s.
  
 So basically I am limited at the disk rather than the n/w
  
 From: Kanwar Sangha [mailto:kan...@mavenir.com] 
 Sent: 06 March 2013 15:11
 To: user@cassandra.apache.org
 Subject: RE: Hinted handoff
  
 After trying to bump up the “hinted_handoff_throttle_in_kb” to 1G/b per sec, 
 It still does not go above 25Mb/s.  Is there a limitation ?
  
  
  
 From: Kanwar Sangha [mailto:kan...@mavenir.com] 
 Sent: 06 March 2013 14:41
 To: user@cassandra.apache.org
 Subject: RE: Hinted handoff
  
 Got the param. thanks
  
 From: Kanwar Sangha [mailto:kan...@mavenir.com] 
 Sent: 06 March 2013 13:50
 To: user@cassandra.apache.org
 Subject: Hinted handoff
  
 Hi – Is there a way to increase the hinted handoff throughput ? I am seeing 
 around 8Mb/s (bits).
  
 Thanks,
 Kanwar

Re: hinted handoff disabling trade-offs

[aaron morton]

The advantage of HH is that it reduces the probability of a DigestMismatch when 
using a CL  ONE. A DigestMismatch means the read has to run a second time 
before returning to the client. 

  - No risk of hinted-handoffs building up
  - No risk of hinted-handoffs flooding a node that just came up
See the yaml config settings for the max hint window and the throttling. 

  Can anyone suggest any other factors that I'm missing here. Specifically
  reasons
  not to do this.
If you are doing this for performance first make sure your data model is 
efficient, that you are doing the most efficient reads (see my presentation 
here http://www.datastax.com/events/cassandrasummit2012/presentations), and 
your caching is bang on. Then consider if you can tune the CL, and if your 
client is token aware so it directs traffic to a node that has it. 

Cheers

-
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 4/03/2013, at 9:19 PM, Michael Kjellman mkjell...@barracuda.com wrote:

 Also, if you have enough hints being created that its significantly impacting 
 your heap I have a feeling things are going to get out of sync very quickly.
 
 On Mar 4, 2013, at 9:17 PM, Wz1975 wz1...@yahoo.com wrote:
 
 Why do you think disabling hinted handoff will improve memory usage? 
 
 
 Thanks.
 -Wei
 
 Sent from my Samsung smartphone on ATT 
 
 
  Original message 
 Subject: Re: hinted handoff disabling trade-offs 
 From: Michael Kjellman mkjell...@barracuda.com 
 To: user@cassandra.apache.org user@cassandra.apache.org 
 CC: 
 
 
 Repair is slow.
 
 On Mar 4, 2013, at 8:07 PM, Matt Kap matvey1...@gmail.com wrote:
 
  I am looking to get a second opinion about disabling hinted-handoffs. I
  have an application that can tolerate a fair amount of inconsistency
  (advertising domain), and so I'm weighting the pros and cons of hinted
  handoffs. I'm running Cassandra 1.0, looking to upgrade to 1.1 soon.
  
  Pros of disabling hinted handoffs:
  - Reduces heap
  - Improves GC performance
  - No risk of hinted-handoffs building up
  - No risk of hinted-handoffs flooding a node that just came up
  
  Cons
  - Some writes can be lost, at least until repair runs
  
  Can anyone suggest any other factors that I'm missing here. Specifically
  reasons
  not to do this.
  
  Cheers!
  -Matt
 
 Copy, by Barracuda, helps you store, protect, and share all your amazing 
 things. Start today: www.copy.com.
 
 -- 
 Copy, by Barracuda, helps you store, protect, and share all your amazing 
 things. Start today: www.copy.com.
   ­­  

hinted handoff disabling trade-offs

[Matt Kap]

I am looking to get a second opinion about disabling hinted-handoffs. I
have an application that can tolerate a fair amount of inconsistency
(advertising domain), and so I'm weighting the pros and cons of hinted
handoffs. I'm running Cassandra 1.0, looking to upgrade to 1.1 soon.

Pros of disabling hinted handoffs:
- Reduces heap
- Improves GC performance
- No risk of hinted-handoffs building up
- No risk of hinted-handoffs flooding a node that just came up

Cons
- Some writes can be lost, at least until repair runs

Can anyone suggest any other factors that I'm missing here. Specifically
reasons
not to do this.

Cheers!
-Matt

Re: hinted handoff disabling trade-offs

[Michael Kjellman]

Repair is slow.

On Mar 4, 2013, at 8:07 PM, Matt Kap matvey1...@gmail.com wrote:

 I am looking to get a second opinion about disabling hinted-handoffs. I
 have an application that can tolerate a fair amount of inconsistency
 (advertising domain), and so I'm weighting the pros and cons of hinted
 handoffs. I'm running Cassandra 1.0, looking to upgrade to 1.1 soon.
 
 Pros of disabling hinted handoffs:
 - Reduces heap
 - Improves GC performance
 - No risk of hinted-handoffs building up
 - No risk of hinted-handoffs flooding a node that just came up
 
 Cons
 - Some writes can be lost, at least until repair runs
 
 Can anyone suggest any other factors that I'm missing here. Specifically
 reasons
 not to do this.
 
 Cheers!
 -Matt

Copy, by Barracuda, helps you store, protect, and share all your amazing
things. Start today: www.copy.com.

Re: hinted handoff disabling trade-offs

[Wz1975]

Why do you think disabling hinted handoff will improve memory usage? 


Thanks.
-Wei

Sent from my Samsung smartphone on ATT

 Original message 
Subject: Re: hinted handoff disabling trade-offs 
From: Michael Kjellman mkjell...@barracuda.com 
To: user@cassandra.apache.org user@cassandra.apache.org 
CC:  

Repair is slow.

On Mar 4, 2013, at 8:07 PM, Matt Kap matvey1...@gmail.com wrote:

 I am looking to get a second opinion about disabling hinted-handoffs. I
 have an application that can tolerate a fair amount of inconsistency
 (advertising domain), and so I'm weighting the pros and cons of hinted
 handoffs. I'm running Cassandra 1.0, looking to upgrade to 1.1 soon.
 
 Pros of disabling hinted handoffs:
 - Reduces heap
 - Improves GC performance
 - No risk of hinted-handoffs building up
 - No risk of hinted-handoffs flooding a node that just came up
 
 Cons
 - Some writes can be lost, at least until repair runs
 
 Can anyone suggest any other factors that I'm missing here. Specifically
 reasons
 not to do this.
 
 Cheers!
 -Matt

Copy, by Barracuda, helps you store, protect, and share all your amazing 
things. Start today: www.copy.com.

Re: hinted handoff disabling trade-offs

[Michael Kjellman]

Also, if you have enough hints being created that its significantly impacting 
your heap I have a feeling things are going to get out of sync very quickly.

On Mar 4, 2013, at 9:17 PM, Wz1975 
wz1...@yahoo.commailto:wz1...@yahoo.com wrote:

Why do you think disabling hinted handoff will improve memory usage?


Thanks.
-Wei

Sent from my Samsung smartphone on ATT


 Original message 
Subject: Re: hinted handoff disabling trade-offs
From: Michael Kjellman mkjell...@barracuda.commailto:mkjell...@barracuda.com
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org 
user@cassandra.apache.orgmailto:user@cassandra.apache.org
CC:


Repair is slow.

On Mar 4, 2013, at 8:07 PM, Matt Kap 
matvey1...@gmail.commailto:matvey1...@gmail.com wrote:

 I am looking to get a second opinion about disabling hinted-handoffs. I
 have an application that can tolerate a fair amount of inconsistency
 (advertising domain), and so I'm weighting the pros and cons of hinted
 handoffs. I'm running Cassandra 1.0, looking to upgrade to 1.1 soon.

 Pros of disabling hinted handoffs:
 - Reduces heap
 - Improves GC performance
 - No risk of hinted-handoffs building up
 - No risk of hinted-handoffs flooding a node that just came up

 Cons
 - Some writes can be lost, at least until repair runs

 Can anyone suggest any other factors that I'm missing here. Specifically
 reasons
 not to do this.

 Cheers!
 -Matt

Copy, by Barracuda, helps you store, protect, and share all your amazing
things. Start today: www.copy.comhttp://www.copy.com.

Copy, by Barracuda, helps you store, protect, and share all your amazing
things. Start today: www.copy.com.

Re: continue seeing Finished hinted handoff of 0 rows to endpoint

[Mina Naguib]

On 2012-11-24, at 10:37 AM, Chuan-Heng Hsiao hsiao.chuanh...@gmail.com wrote:

 However, I continue seeing the following in /var/log/cassandra/system.log
 
 INFO [HintedHandoff:1] 2012-11-24 22:58:28,088
 HintedHandOffManager.java (line 296) Started hinted handoff for token:
 27949589543905115548813332729343195104 with IP: /192.168.0.10
 INFO [HintedHandoff:1] 2012-11-24 22:58:28,089
 HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows
 to endpoint /192.168.0.10
 
 every ten mins.


See if https://issues.apache.org/jira/browse/CASSANDRA-4740 is relevant in your 
case.

Re: continue seeing Finished hinted handoff of 0 rows to endpoint

[Chuan-Heng Hsiao]

Hi Mina,

Thank you very much for replying.
My machines are running Oracle java 1.7.09.
so that might be the issue. (My machines are running
Oracle java 1.7.09 from the beginning.)

I cleaned up the specific empty CF again, and the
hinted handoff of 0 rows showed up again, and
the scrub system HintsColumnFamily trick worked this time.

I'll try to reproduce the situation and check the phantom TCP connections.

Sincerely,
Hsiao


On Mon, Nov 26, 2012 at 8:50 AM, Mina Naguib
mina.nag...@bloomdigital.com wrote:


 On 2012-11-24, at 10:37 AM, Chuan-Heng Hsiao hsiao.chuanh...@gmail.com 
 wrote:

 However, I continue seeing the following in /var/log/cassandra/system.log

 INFO [HintedHandoff:1] 2012-11-24 22:58:28,088
 HintedHandOffManager.java (line 296) Started hinted handoff for token:
 27949589543905115548813332729343195104 with IP: /192.168.0.10
 INFO [HintedHandoff:1] 2012-11-24 22:58:28,089
 HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows
 to endpoint /192.168.0.10

 every ten mins.


 See if https://issues.apache.org/jira/browse/CASSANDRA-4740 is relevant in 
 your case.

continue seeing Finished hinted handoff of 0 rows to endpoint

[Chuan-Heng Hsiao]

Hi Cassandra Devs,

I intended to reduce the size of the db by the following steps:

1. removing all keys from one cf (somehow I can get all keys from the cf).
2. run nodetool cleanup on that cf one-node-by-one-node.

the size of the cf on one node is about 150 G,
I've made another cf with the same data of that cf,
and the new cf is only about 70G. It's the main reason
why I want to remove the data in the original cf.

The reason that I did not directly remove the cf is because it seems
not good to
change the schema after inserting a lot of data.

However, I continue seeing the following in /var/log/cassandra/system.log

INFO [HintedHandoff:1] 2012-11-24 22:58:28,088
HintedHandOffManager.java (line 296) Started hinted handoff for token:
27949589543905115548813332729343195104 with IP: /192.168.0.10
INFO [HintedHandoff:1] 2012-11-24 22:58:28,089
HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows
to endpoint /192.168.0.10

every ten mins.
(It seems fall into some infinite loop.)

I am wondering whether it is really some infinite loop?
Or it is just dealing with a lot of tombstone data?

Sincerely,
Hsiao

Re: continue seeing Finished hinted handoff of 0 rows to endpoint

[Michael Kjellman]

Some people (myself included) have seen issues when upgrading from 1.1.2
to 1.1.6 with tombstoned rows in the HintsColumnFamily

Some (mysql included) have fixed this by doing a

nodetool scrub system HintsColumnFamily
-mike

On 11/24/12 7:37 AM, Chuan-Heng Hsiao hsiao.chuanh...@gmail.com wrote:

Hi Cassandra Devs,

I intended to reduce the size of the db by the following steps:

1. removing all keys from one cf (somehow I can get all keys from the cf).
2. run nodetool cleanup on that cf one-node-by-one-node.

the size of the cf on one node is about 150 G,
I've made another cf with the same data of that cf,
and the new cf is only about 70G. It's the main reason
why I want to remove the data in the original cf.

The reason that I did not directly remove the cf is because it seems
not good to
change the schema after inserting a lot of data.

However, I continue seeing the following in /var/log/cassandra/system.log

INFO [HintedHandoff:1] 2012-11-24 22:58:28,088
HintedHandOffManager.java (line 296) Started hinted handoff for token:
27949589543905115548813332729343195104 with IP: /192.168.0.10
INFO [HintedHandoff:1] 2012-11-24 22:58:28,089
HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows
to endpoint /192.168.0.10

every ten mins.
(It seems fall into some infinite loop.)

I am wondering whether it is really some infinite loop?
Or it is just dealing with a lot of tombstone data?

Sincerely,
Hsiao


'Like' us on Facebook for exclusive content and other resources on all 
Barracuda Networks solutions.
Visit http://barracudanetworks.com/facebook

Re: continue seeing Finished hinted handoff of 0 rows to endpoint

[Michael Kjellman]

Sorry, auto correct changed 'myself' to 'mysql'. Sorry for the confusion...

On Nov 24, 2012, at 9:29 AM, Michael Kjellman mkjell...@barracuda.com wrote:

 Some people (myself included) have seen issues when upgrading from 1.1.2
 to 1.1.6 with tombstoned rows in the HintsColumnFamily
 
 Some (mysql included) have fixed this by doing a
 
 nodetool scrub system HintsColumnFamily
 -mike
 
 On 11/24/12 7:37 AM, Chuan-Heng Hsiao hsiao.chuanh...@gmail.com wrote:
 
 Hi Cassandra Devs,
 
 I intended to reduce the size of the db by the following steps:
 
 1. removing all keys from one cf (somehow I can get all keys from the cf).
 2. run nodetool cleanup on that cf one-node-by-one-node.
 
 the size of the cf on one node is about 150 G,
 I've made another cf with the same data of that cf,
 and the new cf is only about 70G. It's the main reason
 why I want to remove the data in the original cf.
 
 The reason that I did not directly remove the cf is because it seems
 not good to
 change the schema after inserting a lot of data.
 
 However, I continue seeing the following in /var/log/cassandra/system.log
 
 INFO [HintedHandoff:1] 2012-11-24 22:58:28,088
 HintedHandOffManager.java (line 296) Started hinted handoff for token:
 27949589543905115548813332729343195104 with IP: /192.168.0.10
 INFO [HintedHandoff:1] 2012-11-24 22:58:28,089
 HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows
 to endpoint /192.168.0.10
 
 every ten mins.
 (It seems fall into some infinite loop.)
 
 I am wondering whether it is really some infinite loop?
 Or it is just dealing with a lot of tombstone data?
 
 Sincerely,
 Hsiao
 
 
 'Like' us on Facebook for exclusive content and other resources on all 
 Barracuda Networks solutions.
 Visit http://barracudanetworks.com/facebook
 
 

'Like' us on Facebook for exclusive content and other resources on all 
Barracuda Networks solutions.
Visit http://barracudanetworks.com/facebook

Re: continue seeing Finished hinted handoff of 0 rows to endpoint

[Chuan-Heng Hsiao]

Hi Mike and Cassandra Devs,

It seems like the reason was because I was trying to
cleanup 2 CFs at the same time.

The scrub command successfully snapshoot HintsColumnFamily,
but seems hanged also (maybe because I did not stop cleanup.)

I ended up restarting the node, and redoing the cleanup one-CF-by-one-CF.
Now everything seems fine.

Sincerely,
Hsiao


On Sun, Nov 25, 2012 at 1:55 AM, Michael Kjellman
mkjell...@barracuda.com wrote:
 Sorry, auto correct changed 'myself' to 'mysql'. Sorry for the confusion...

 On Nov 24, 2012, at 9:29 AM, Michael Kjellman mkjell...@barracuda.com 
 wrote:

 Some people (myself included) have seen issues when upgrading from 1.1.2
 to 1.1.6 with tombstoned rows in the HintsColumnFamily

 Some (mysql included) have fixed this by doing a

 nodetool scrub system HintsColumnFamily
 -mike

 On 11/24/12 7:37 AM, Chuan-Heng Hsiao hsiao.chuanh...@gmail.com wrote:

 Hi Cassandra Devs,

 I intended to reduce the size of the db by the following steps:

 1. removing all keys from one cf (somehow I can get all keys from the cf).
 2. run nodetool cleanup on that cf one-node-by-one-node.

 the size of the cf on one node is about 150 G,
 I've made another cf with the same data of that cf,
 and the new cf is only about 70G. It's the main reason
 why I want to remove the data in the original cf.

 The reason that I did not directly remove the cf is because it seems
 not good to
 change the schema after inserting a lot of data.

 However, I continue seeing the following in /var/log/cassandra/system.log

 INFO [HintedHandoff:1] 2012-11-24 22:58:28,088
 HintedHandOffManager.java (line 296) Started hinted handoff for token:
 27949589543905115548813332729343195104 with IP: /192.168.0.10
 INFO [HintedHandoff:1] 2012-11-24 22:58:28,089
 HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows
 to endpoint /192.168.0.10

 every ten mins.
 (It seems fall into some infinite loop.)

 I am wondering whether it is really some infinite loop?
 Or it is just dealing with a lot of tombstone data?

 Sincerely,
 Hsiao


 'Like' us on Facebook for exclusive content and other resources on all 
 Barracuda Networks solutions.
 Visit http://barracudanetworks.com/facebook



 'Like' us on Facebook for exclusive content and other resources on all 
 Barracuda Networks solutions.

 Visit http://barracudanetworks.com/facebook

Re: Hinted Handoff runs every ten minutes

[Vegard Berget]

 Hi,
HintsColumnFamily directory on Node 1 (the first to be upgraded):1.8K
Oct 27 11:27 system-HintsColumnFamily-hf-2-Data.db79 Oct 27 11:27
system-HintsColumnFamily-hf-2-Digest.sha1496 Oct 27 11:27
system-HintsColumnFamily-hf-2-Filter.db26 Oct 27 11:27
system-HintsColumnFamily-hf-2-Index.db4.3K Oct 27 11:27
system-HintsColumnFamily-hf-2-Statistics.db

HintsColumnFamily-directory on Node 2 (the second to be upgraded):12K
Oct 27 11:37 system-HintsColumnFamily-hf-3-Data.db79 Oct 27 11:37
system-HintsColumnFamily-hf-3-Digest.sha1976 Oct 27 11:37
system-HintsColumnFamily-hf-3-Filter.db11 Oct 27 11:37
system-HintsColumnFamily-hf-3-Index.db4.3K Oct 27 11:37
system-HintsColumnFamily-hf-3-Statistics.db
11:37 is exactly ten minutes after the node was started after
upgrading, according to the logs, which means that the first node's
sstables is generated when the second node starts.

The sstable2json is more or less equal for both nodes, but node 1
starts with 4000 and node 2 with
00.The rest looks like this:00: {f7:
{deletedAt: -9223372036854775808, subColumns: []},
f7: {deletedAt:
-9223372036854775808, subColumns: []},(The f7X is
repeated with multiple different keys, but all are above
f7X.  The deletedAt is always equal - on both
nodes)
Of course, this could be caused by me doing a mistake while upgrading.
 I have not tried to compact, but repair runs every night.  It's a
low traffic cluster to test cassandra, if that matters.
.vegard,

- Original Message -
From: user@cassandra.apache.org
To:
Cc:
Sent:Sun, 11 Nov 2012 13:16:01 -0600
Subject:Re: Hinted Handoff runs every ten minutes

 How many hint sstables are there? What does sstable2json show?

 On Thu, Nov 8, 2012 at 3:23 PM, Mike Heffner  wrote:
  Is there a ticket open for this for 1.1.6?
 
  We also noticed this after upgrading from 1.1.3 to 1.1.6. Every
node runs a
  0 row hinted handoff every 10 minutes. N-1 nodes hint to the same
node,
  while that node hints to another node.
 
 
  On Tue, Oct 30, 2012 at 1:35 PM, Vegard Berget  wrote:
 
  Hi,
 
  I have the exact same problem with 1.16. HintsColumnFamily
consists of
  one row (Rowkey 00, nothing more). The problem started after
upgrading
  from 1.1.4 to 1.1.6. Every ten minutes HintedHandoffManager starts
and
  finishes after sending 0 rows.
 
  .vegard,
 
 
 
  - Original Message -
  From:
  user@cassandra.apache.org
 
  To:
  
  Cc:
 
  Sent:
  Mon, 29 Oct 2012 23:45:30 +0100
 
  Subject:
  Re: Hinted Handoff runs every ten minutes
 
 
  Dne 29.10.2012 23:24, Stephen Pierce napsal(a):
   I'm running 1.1.5; the bug says it's fixed in 1.0.9/1.1.0.
  
   How can I check to see why it keeps running HintedHandoff?
  you have tombstone is system.HintsColumnFamily use list command in
  cassandra-cli to check
 
 
 
 
  --
 
  Mike Heffner 
  Librato, Inc.
 
 

 -- 
 Jonathan Ellis
 Project Chair, Apache Cassandra
 co-founder of DataStax, the source for professional Cassandra support
 http://www.datastax.com

Re: Hinted Handoff runs every ten minutes

[Jonathan Ellis]

How many hint sstables are there?  What does sstable2json show?

On Thu, Nov 8, 2012 at 3:23 PM, Mike Heffner m...@librato.com wrote:
 Is there a ticket open for this for 1.1.6?

 We also noticed this after upgrading from 1.1.3 to 1.1.6. Every node runs a
 0 row hinted handoff every 10 minutes. N-1 nodes hint to the same node,
 while that node hints to another node.


 On Tue, Oct 30, 2012 at 1:35 PM, Vegard Berget p...@fantasista.no wrote:

 Hi,

 I have the exact same problem with 1.1.6.  HintsColumnFamily consists of
 one row (Rowkey 00, nothing more).   The problem started after upgrading
 from 1.1.4 to 1.1.6.  Every ten minutes HintedHandoffManager starts and
 finishes  after sending 0 rows.

 .vegard,



 - Original Message -
 From:
 user@cassandra.apache.org

 To:
 user@cassandra.apache.org
 Cc:

 Sent:
 Mon, 29 Oct 2012 23:45:30 +0100

 Subject:
 Re: Hinted Handoff runs every ten minutes


 Dne 29.10.2012 23:24, Stephen Pierce napsal(a):
  I'm running 1.1.5; the bug says it's fixed in 1.0.9/1.1.0.
 
  How can I check to see why it keeps running HintedHandoff?
 you have tombstone is system.HintsColumnFamily use list command in
 cassandra-cli to check




 --

   Mike Heffner m...@librato.com
   Librato, Inc.





-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: Hinted Handoff runs every ten minutes

[Mike Heffner]

Is there a ticket open for this for 1.1.6?

We also noticed this after upgrading from 1.1.3 to 1.1.6. Every node runs a
0 row hinted handoff every 10 minutes. N-1 nodes hint to the same node,
while that node hints to another node.


On Tue, Oct 30, 2012 at 1:35 PM, Vegard Berget p...@fantasista.no wrote:

 Hi,

 I have the exact same problem with 1.1.6.  HintsColumnFamily consists of
 one row (Rowkey 00, nothing more).   The problem started after upgrading
 from 1.1.4 to 1.1.6.  Every ten minutes HintedHandoffManager starts and
 finishes  after sending 0 rows.

 .vegard,



 - Original Message -
 From:
 user@cassandra.apache.org

 To:
 user@cassandra.apache.org
 Cc:

 Sent:
 Mon, 29 Oct 2012 23:45:30 +0100

 Subject:
 Re: Hinted Handoff runs every ten minutes


 Dne 29.10.2012 23:24, Stephen Pierce napsal(a):
  I'm running 1.1.5; the bug says it's fixed in 1.0.9/1.1.0.
 
  How can I check to see why it keeps running HintedHandoff?
 you have tombstone is system.HintsColumnFamily use list command in
 cassandra-cli to check




-- 

  Mike Heffner m...@librato.com
  Librato, Inc.

Re[2]: Hinted Handoff runs every ten minutes

[Marcus Both]

When I did the upgrade from 1.0.9 to 1.1.6, I had this same issue.
And then I fixed it with the following steps below, in each of the nodes.

[default@unknown] use system;
Authenticated to keyspace: system
[default@system] list HintsColumnFamily;
Using default limit of 100
Using default column limit of 100
---
RowKey: 5554

1 Row Returned.
Elapsed time: 84 msec(s).
[default@system] del HintsColumnFamily[5554];
system keyspace is not user-modifiable.
InvalidRequestException(why:system keyspace is not user-modifiable.)
at 
org.apache.cassandra.thrift.Cassandra$remove_result.read(Cassandra.java:18213)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at 
org.apache.cassandra.thrift.Cassandra$Client.recv_remove(Cassandra.java:861)
at 
org.apache.cassandra.thrift.Cassandra$Client.remove(Cassandra.java:845)
at org.apache.cassandra.cli.CliClient.executeDelete(CliClient.java:455)
at 
org.apache.cassandra.cli.CliClient.executeCLIStatement(CliClient.java:225)
at 
org.apache.cassandra.cli.CliMain.processStatementInteractive(CliMain.java:219)
at org.apache.cassandra.cli.CliMain.main(CliMain.java:346)
[default@system] 
[root@ ~]# /usr/local/apache-cassandra/bin/nodetool -h 127.0.0.1 scrub system 
HintsColumnFamily
[root@ ~]# /usr/local/apache-cassandra/bin/cassandra-cli -h 127.0.0.1 -p 9160
Connected to: nosql-world on 127.0.0.1/9160
Welcome to Cassandra CLI version 1.1.6

Type 'help;' or '?' for help.
Type 'quit;' or 'exit;' to quit.

[default@unknown] use system; 
Authenticated to keyspace: system
[default@system] list HintsColumnFamily; 
Using default limit of 100
Using default column limit of 100

0 Row Returned.
Elapsed time: 73 msec(s).
[default@system] 


tail /var/log/cassandra/system.log 
 INFO [CompactionExecutor:193] 2012-11-06 12:03:12,233 OutputHandler.java (line 
42) Scrubbing 
SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hf-11-Data.db')
 INFO [CompactionExecutor:193] 2012-11-06 12:03:12,236 OutputHandler.java (line 
42) Scrub of 
SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hf-11-Data.db')
 complete; looks like all 1 rows were tombstoned
 
--
Marcus Both

On Mon, 5 Nov 2012 19:21:40 +
Stephen Pierce spie...@verifyle.com wrote:

 I did have a tombstone. I couldn't seem to delete it, so I truncated the 
 column family. 
 
 Thanks for the help.
 
 Steve
 
 [default@system] list HintsColumnFamily;
 Using default limit of 100
 Using default column limit of 100
 ---
 RowKey: 5554
 
 I wasn't able to delete the row:
 
 [default@system] del HintsColumnFamily[5554];
 row removed.
 [default@system] list HintsColumnFamily;
 Using default limit of 100
 Using default column limit of 100
 ---
 RowKey: 5554
 
 But I finally truncated the column family:
 
 [default@system] truncate HintsColumnFamily;
 HintsColumnFamily truncated.
 
 
 -Original Message-
 From: Radim Kolar [mailto:h...@filez.com] 
 Sent: Monday, October 29, 2012 3:46 PM
 To: user@cassandra.apache.org
 Subject: Re: Hinted Handoff runs every ten minutes
 
 Dne 29.10.2012 23:24, Stephen Pierce napsal(a):
  I'm running 1.1.5; the bug says it's fixed in 1.0.9/1.1.0.
 
  How can I check to see why it keeps running HintedHandoff?
 you have tombstone is system.HintsColumnFamily use list command in 
 cassandra-cli to check
 
-- 
Marcus Both 

RE: Hinted Handoff runs every ten minutes

[Stephen Pierce]

I did have a tombstone. I couldn't seem to delete it, so I truncated the column 
family. 

Thanks for the help.

Steve

[default@system] list HintsColumnFamily;
Using default limit of 100
Using default column limit of 100
---
RowKey: 5554

I wasn't able to delete the row:

[default@system] del HintsColumnFamily[5554];
row removed.
[default@system] list HintsColumnFamily;
Using default limit of 100
Using default column limit of 100
---
RowKey: 5554

But I finally truncated the column family:

[default@system] truncate HintsColumnFamily;
HintsColumnFamily truncated.


-Original Message-
From: Radim Kolar [mailto:h...@filez.com] 
Sent: Monday, October 29, 2012 3:46 PM
To: user@cassandra.apache.org
Subject: Re: Hinted Handoff runs every ten minutes

Dne 29.10.2012 23:24, Stephen Pierce napsal(a):
 I'm running 1.1.5; the bug says it's fixed in 1.0.9/1.1.0.

 How can I check to see why it keeps running HintedHandoff?
you have tombstone is system.HintsColumnFamily use list command in 
cassandra-cli to check

Re: Hinted Handoff runs every ten minutes

[Vegard Berget]

Hi,
I have the exact same problem with 1.1.6.  HintsColumnFamily consists
of one row (Rowkey 00, nothing more).   The problem started after
upgrading from 1.1.4 to 1.1.6.  Every ten minutes
HintedHandoffManager starts and finishes  after sending 0 rows.  
.vegard,

- Original Message -
From: user@cassandra.apache.org
To:
Cc:
Sent:Mon, 29 Oct 2012 23:45:30 +0100
Subject:Re: Hinted Handoff runs every ten minutes

 Dne 29.10.2012 23:24, Stephen Pierce napsal(a):
  I'm running 1.1.5; the bug says it's fixed in 1.0.9/1.1.0.
 
  How can I check to see why it keeps running HintedHandoff?
 you have tombstone is system.HintsColumnFamily use list command in 
 cassandra-cli to check

Re: Hinted Handoff runs every ten minutes

[Vegard Berget]

Hi,
I have the exact same problem with 1.1.6.  HintsColumnFamily consists
of one row (Rowkey 00, nothing more).   The problem started after
upgrading from 1.1.4 to 1.1.6.  Every ten minutes
HintedHandoffManager starts and finishes  after sending 0 rows.  
.vegard,

- Original Message -
From: user@cassandra.apache.org
To:
Cc:
Sent:Mon, 29 Oct 2012 23:45:30 +0100
Subject:Re: Hinted Handoff runs every ten minutes

 Dne 29.10.2012 23:24, Stephen Pierce napsal(a):
  I'm running 1.1.5; the bug says it's fixed in 1.0.9/1.1.0.
 
  How can I check to see why it keeps running HintedHandoff?
 you have tombstone is system.HintsColumnFamily use list command in 
 cassandra-cli to check

Re: Hinted Handoff storage inflation

[aaron morton]

 With both data centers functional, the test takes just a few minutes to run, 
 with one data center down, 15x the amount of time.
Could you provide the numbers, it's easier to get a feel for how the throughput 
is dropping. Does latency reported by nodetool cf stats change ? 
I'm also interested to know how long hints were collected for. 

Each coordinator will be writing three hints, which will be slowing down the 
other writes it needs to do. 

 but I found that the storage overhead was the same regardless of the size of 
 the batch mutation (i.e., 5 vs 25 mutations made no difference).
Batch size makes no difference. Each row mutation is treated as an individual 
command, the batch is simply a way to reduce network calls. 

 Each write is new data only (no overwrites). Each mutation adds a row to one 
 column family with a column containing about ~100 bytes of data and a new row 
 to another column family with a SuperColumn containing 2x17KiB payloads.
I cannot remember anyone raising this sort of issue about HH before. It may be 
that no one has looked at how that level of hints is handled. 
Could you reproduce the problem with a smaller test case ? 

Cheers

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 27/10/2012, at 7:56 AM, Mattias Larsson mlars...@yahoo-inc.com wrote:

 
 On Oct 24, 2012, at 6:05 PM, aaron morton wrote:
 
 Hints store the columns, row key, KS name and CF id(s) for each mutation to 
 each node. Where an executed mutation will store the most recent columns 
 collated with others under the same row key. So depending on the type of 
 mutation hints will take up more space. 
 
 The worse case would be lots of overwrites. After that writing a small 
 amount of data to many rows would result in a lot of the serialised space 
 being devoted to row keys, KS name and CF id.
 
 16Gb is a lot though. What was the write workload like ?
 
 Each write is new data only (no overwrites). Each mutation adds a row to one 
 column family with a column containing about ~100 bytes of data and a new row 
 to another column family with a SuperColumn containing 2x17KiB payloads. 
 These are sent in batches with several in them, but I found that the storage 
 overhead was the same regardless of the size of the batch mutation (i.e., 5 
 vs 25 mutations made no difference). A total of 1,000,000 mutations like 
 these are sent over the duration of the test.
 
 
 You can get an estimate on the number of keys in the Hints CF using nodetool 
 cfstats. Also some metrics in the JMX will tell you how many hints are 
 stored. 
 
 This has a huge impact on write performance as well.
 Yup. Hints are added to the same Mutation thread pool as normal mutations. 
 They are processed async to the mutation request but they still take 
 resources to store. 
 
 You can adjust how long hints a collected for with max_hint_window_in_ms in 
 the yaml file. 
 
 How long did the test run for ? 
 
 
 With both data centers functional, the test takes just a few minutes to run, 
 with one data center down, 15x the amount of time.
 
 /dml
 
 

RE: Hinted Handoff runs every ten minutes

[Stephen Pierce]

I'm running 1.1.5; the bug says it's fixed in 1.0.9/1.1.0. 

How can I check to see why it keeps running HintedHandoff?

Steve
 

-Original Message-
From: Brandon Williams [mailto:dri...@gmail.com] 
Sent: Wednesday, October 24, 2012 4:56 AM
To: user@cassandra.apache.org
Subject: Re: Hinted Handoff runs every ten minutes

On Sun, Oct 21, 2012 at 6:44 PM, aaron morton aa...@thelastpickle.com wrote:
 I *think* this may be ghost rows which have not being compacted.

You would be correct in the case of 1.0.8:
https://issues.apache.org/jira/browse/CASSANDRA-3955

-Brandon

Re: Hinted Handoff runs every ten minutes

[Radim Kolar]

Dne 29.10.2012 23:24, Stephen Pierce napsal(a):

I'm running 1.1.5; the bug says it's fixed in 1.0.9/1.1.0.

How can I check to see why it keeps running HintedHandoff?
you have tombstone is system.HintsColumnFamily use list command in 
cassandra-cli to check

Re: how to stop hinted handoff

[Tamar Fraenkel]

Thanks, that did the trick!

*Tamar Fraenkel *
Senior Software Engineer, TOK Media

[image: Inline image 1]

ta...@tok-media.com
Tel:   +972 2 6409736
Mob:  +972 54 8356490
Fax:   +972 2 5612956





On Thu, Oct 11, 2012 at 3:42 AM, Roshan codeva...@gmail.com wrote:

 Hello

 You can delete the hints from JConsole by using HintedHadOffManager MBean.

 Thanks.






 --
 View this message in context:
 http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-to-stop-hinted-handoff-tp7583060p7583086.html
 Sent from the cassandra-u...@incubator.apache.org mailing list archive at
 Nabble.com.

tokLogo.png

Re: Hinted Handoff runs every ten minutes

[Brandon Williams]

On Sun, Oct 21, 2012 at 6:44 PM, aaron morton aa...@thelastpickle.com wrote:
 I *think* this may be ghost rows which have not being compacted.

You would be correct in the case of 1.0.8:
https://issues.apache.org/jira/browse/CASSANDRA-3955

-Brandon

Re: Hinted Handoff runs every ten minutes

[Tamar Fraenkel]

Is there a walk around other than upgrade?
Thanks,
*Tamar Fraenkel *
Senior Software Engineer, TOK Media

[image: Inline image 1]

ta...@tok-media.com
Tel:   +972 2 6409736
Mob:  +972 54 8356490
Fax:   +972 2 5612956





On Wed, Oct 24, 2012 at 1:56 PM, Brandon Williams dri...@gmail.com wrote:

 On Sun, Oct 21, 2012 at 6:44 PM, aaron morton aa...@thelastpickle.com
 wrote:
  I *think* this may be ghost rows which have not being compacted.

 You would be correct in the case of 1.0.8:
 https://issues.apache.org/jira/browse/CASSANDRA-3955

 -Brandon

tokLogo.png

Hinted Handoff storage inflation

[Mattias Larsson]

I'm testing various scenarios in a multi data center configuration. The setup 
is 10 Cassandra 1.1.5 nodes configured into two data centers, 5 nodes in each 
DC (RF DC1:3,DC2:3, write consistency LOCAL_QUORUM). I have a synthetic random 
data generator that I can run, and each run adds roughly 1GiB of data to each 
node per run,

DC  RackStatus State   LoadEffective-Ownership
  
DC1 RAC1Up Normal  1010.71 MB  60.00% 
DC2 RAC1Up Normal  1009.08 MB  60.00% 
DC1 RAC1Up Normal  1.01 GB 60.00% 
DC2 RAC1Up Normal  1 GB60.00% 
DC1 RAC1Up Normal  1.01 GB 60.00% 
DC2 RAC1Up Normal  1014.45 MB  60.00% 
DC1 RAC1Up Normal  1.01 GB 60.00% 
DC2 RAC1Up Normal  1.01 GB 60.00% 
DC1 RAC1Up Normal  1.01 GB 60.00% 
DC2 RAC1Up Normal  1.01 GB 60.00% 

Now, if I kill all the nodes in DC2, and run the data generator again, I would 
expect roughly 2GiB to be added to each node in DC1 (local replicas + hints to 
other data center), instead I get this:

DC  RackStatus State   LoadEffective-Ownership
  
DC1 RAC1Up Normal  17.56 GB60.00% 
DC2 RAC1Down   Normal  1009.08 MB  60.00% 
DC1 RAC1Up Normal  17.47 GB60.00% 
DC2 RAC1Down   Normal  1 GB60.00% 
DC1 RAC1Up Normal  17.22 GB60.00% 
DC2 RAC1Down   Normal  1014.45 MB  60.00% 
DC1 RAC1Up Normal  16.94 GB60.00% 
DC2 RAC1Down   Normal  1.01 GB 60.00% 
DC1 RAC1Up Normal  17.26 GB60.00% 
DC2 RAC1Down   Normal  1.01 GB 60.00% 

Checking the sstables on a node reveals this,

-bash-3.2$ du -hs HintsColumnFamily/
16G HintsColumnFamily/
-bash-3.2$

So it seems that what I would have expected to be 1GiB of hints is much larger 
in reality, a 15x-16x inflation. This has a huge impact on write performance as 
well.

If I bring DC2 up again, eventually the load will drop down and even out to 
2GiB across the entire cluster.

I'm wondering if this inflation is intended or if it is possibly a bug or 
something I'm doing wrong? Assuming this inflation is correct, what is the best 
way to deal with temporary connectivity issues with a second data center? Write 
performance is paramount in my use case. A 2x-3x overhead is doable, but not 
15x-16x.

Thanks,
/dml

Re: Hinted Handoff runs every ten minutes

[aaron morton]

Thanks. 
I thought it had been addressed so before but couldn't find the ticket. 

Cheers

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 25/10/2012, at 12:56 AM, Brandon Williams dri...@gmail.com wrote:

 On Sun, Oct 21, 2012 at 6:44 PM, aaron morton aa...@thelastpickle.com wrote:
 I *think* this may be ghost rows which have not being compacted.
 
 You would be correct in the case of 1.0.8:
 https://issues.apache.org/jira/browse/CASSANDRA-3955
 
 -Brandon

Re: Hinted Handoff storage inflation

[aaron morton]

Hints store the columns, row key, KS name and CF id(s) for each mutation to 
each node. Where an executed mutation will store the most recent columns 
collated with others under the same row key. So depending on the type of 
mutation hints will take up more space. 

The worse case would be lots of overwrites. After that writing a small amount 
of data to many rows would result in a lot of the serialised space being 
devoted to row keys, KS name and CF id.

16Gb is a lot though. What was the write workload like ?
You can get an estimate on the number of keys in the Hints CF using nodetool 
cfstats. Also some metrics in the JMX will tell you how many hints are stored. 

 This has a huge impact on write performance as well.
Yup. Hints are added to the same Mutation thread pool as normal mutations. They 
are processed async to the mutation request but they still take resources to 
store. 

You can adjust how long hints a collected for with max_hint_window_in_ms in the 
yaml file. 

How long did the test run for ? 


Cheers

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 25/10/2012, at 11:26 AM, Mattias Larsson mlars...@yahoo-inc.com wrote:

 
 I'm testing various scenarios in a multi data center configuration. The setup 
 is 10 Cassandra 1.1.5 nodes configured into two data centers, 5 nodes in each 
 DC (RF DC1:3,DC2:3, write consistency LOCAL_QUORUM). I have a synthetic 
 random data generator that I can run, and each run adds roughly 1GiB of data 
 to each node per run,
 
 DC  RackStatus State   LoadEffective-Ownership
 
 DC1 RAC1Up Normal  1010.71 MB  60.00% 
 DC2 RAC1Up Normal  1009.08 MB  60.00% 
 DC1 RAC1Up Normal  1.01 GB 60.00% 
 DC2 RAC1Up Normal  1 GB60.00% 
 DC1 RAC1Up Normal  1.01 GB 60.00% 
 DC2 RAC1Up Normal  1014.45 MB  60.00% 
 DC1 RAC1Up Normal  1.01 GB 60.00% 
 DC2 RAC1Up Normal  1.01 GB 60.00% 
 DC1 RAC1Up Normal  1.01 GB 60.00% 
 DC2 RAC1Up Normal  1.01 GB 60.00% 
 
 Now, if I kill all the nodes in DC2, and run the data generator again, I 
 would expect roughly 2GiB to be added to each node in DC1 (local replicas + 
 hints to other data center), instead I get this:
 
 DC  RackStatus State   LoadEffective-Ownership
 
 DC1 RAC1Up Normal  17.56 GB60.00% 
 DC2 RAC1Down   Normal  1009.08 MB  60.00% 
 DC1 RAC1Up Normal  17.47 GB60.00% 
 DC2 RAC1Down   Normal  1 GB60.00% 
 DC1 RAC1Up Normal  17.22 GB60.00% 
 DC2 RAC1Down   Normal  1014.45 MB  60.00% 
 DC1 RAC1Up Normal  16.94 GB60.00% 
 DC2 RAC1Down   Normal  1.01 GB 60.00% 
 DC1 RAC1Up Normal  17.26 GB60.00% 
 DC2 RAC1Down   Normal  1.01 GB 60.00% 
 
 Checking the sstables on a node reveals this,
 
 -bash-3.2$ du -hs HintsColumnFamily/
 16G   HintsColumnFamily/
 -bash-3.2$
 
 So it seems that what I would have expected to be 1GiB of hints is much 
 larger in reality, a 15x-16x inflation. This has a huge impact on write 
 performance as well.
 
 If I bring DC2 up again, eventually the load will drop down and even out to 
 2GiB across the entire cluster.
 
 I'm wondering if this inflation is intended or if it is possibly a bug or 
 something I'm doing wrong? Assuming this inflation is correct, what is the 
 best way to deal with temporary connectivity issues with a second data 
 center? Write performance is paramount in my use case. A 2x-3x overhead is 
 doable, but not 15x-16x.
 
 Thanks,
 /dml
 
 

Re: Hinted Handoff runs every ten minutes

[Tamar Fraenkel]

Hi!
I am having the same issue on 1.0.8.
Checked number of SSTables, on two nodes I have 1 (on each) and on 1 node I
have none.
Thanks,

*Tamar Fraenkel *
Senior Software Engineer, TOK Media

[image: Inline image 1]

ta...@tok-media.com
Tel:   +972 2 6409736
Mob:  +972 54 8356490
Fax:   +972 2 5612956





On Mon, Oct 22, 2012 at 1:44 AM, aaron morton aa...@thelastpickle.comwrote:

 I *think* this may be ghost rows which have not being compacted.

 How many SSTables are on disk for the HintedHandoff CF ?

 Cheers

   -
 Aaron Morton
 Freelance Developer
 @aaronmorton
 http://www.thelastpickle.com

 On 19/10/2012, at 7:16 AM, David Daeschler david.daesch...@gmail.com
 wrote:

 Hi Steve,

 Also confirming this. After having a node go down on Cassandra 1.0.8
 there seems to be hinted handoff between two of our 4 nodes every 10
 minutes. Our setup also shows 0 rows. It does not appear to have any
 effect on the operation of the ring, just fills up the log files.

 - David



 On Thu, Oct 18, 2012 at 2:10 PM, Stephen Pierce spie...@verifyle.com
 wrote:

 I installed Cassandra on three nodes. I then ran a test suite against them
 to generate load. The test suite is designed to generate the same type of
 load that we plan to have in production. As one of many tests, I reset one
 of the nodes to check the failure/recovery modes.  Cassandra worked just
 fine.



 I stopped the load generation, and got distracted with some other
 project/problem. A few days later, I noticed something strange on one of
 the
 nodes. On this node hinted handoff starts every ten minutes, and while it
 seems to finish without any errors, it will be started again in ten
 minutes.
 None of the nodes has any traffic, and hasn’t for several days. I checked
 the logs, and this goes back to the initial failure/recovery testing:



 INFO [HintedHandoff:1] 2012-10-18 10:19:26,618 HintedHandOffManager.java
 (line 294) Started hinted handoff for token:
 113427455640312821154458202477256070484 with IP: /192.168.128.136

 INFO [HintedHandoff:1] 2012-10-18 10:19:26,779 HintedHandOffManager.java
 (line 390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136

 INFO [HintedHandoff:1] 2012-10-18 10:29:26,622 HintedHandOffManager.java
 (line 294) Started hinted handoff for token:
 113427455640312821154458202477256070484 with IP: /192.168.128.136

 INFO [HintedHandoff:1] 2012-10-18 10:29:26,735 HintedHandOffManager.java
 (line 390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136

 INFO [HintedHandoff:1] 2012-10-18 10:39:26,624 HintedHandOffManager.java
 (line 294) Started hinted handoff for token:
 113427455640312821154458202477256070484 with IP: /192.168.128.136

 INFO [HintedHandoff:1] 2012-10-18 10:39:26,751 HintedHandOffManager.java
 (line 390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136



 The other nodes are happy and don’t show this behavior. All the test data
 is
 readable, and everything is fine, but I’m curious why hinted handoff is
 running on one node all the time.



 I searched the bug database, and I found a bug that seems to have the same
 symptoms:

 https://issues.apache.org/jira/browse/CASSANDRA-3733

 Although it’s been marked fixed in 0.6, this describes my problem exactly.



 I’m running Cassandra 1.1.5 from Datastax on Centos 6.0:


 http://rpm.datastax.com/community/noarch/apache-cassandra11-1.1.5-1.noarch.rpm



 Is anyone else seeing this behavior? What can I do to provide more
 information?



 Steve





 --
 David Daeschler



tokLogo.png

Re: Hinted Handoff runs every ten minutes

[aaron morton]

I *think* this may be ghost rows which have not being compacted.

How many SSTables are on disk for the HintedHandoff CF ?

Cheers

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 19/10/2012, at 7:16 AM, David Daeschler david.daesch...@gmail.com wrote:

 Hi Steve,
 
 Also confirming this. After having a node go down on Cassandra 1.0.8
 there seems to be hinted handoff between two of our 4 nodes every 10
 minutes. Our setup also shows 0 rows. It does not appear to have any
 effect on the operation of the ring, just fills up the log files.
 
 - David
 
 
 
 On Thu, Oct 18, 2012 at 2:10 PM, Stephen Pierce spie...@verifyle.com wrote:
 I installed Cassandra on three nodes. I then ran a test suite against them
 to generate load. The test suite is designed to generate the same type of
 load that we plan to have in production. As one of many tests, I reset one
 of the nodes to check the failure/recovery modes.  Cassandra worked just
 fine.
 
 
 
 I stopped the load generation, and got distracted with some other
 project/problem. A few days later, I noticed something strange on one of the
 nodes. On this node hinted handoff starts every ten minutes, and while it
 seems to finish without any errors, it will be started again in ten minutes.
 None of the nodes has any traffic, and hasn’t for several days. I checked
 the logs, and this goes back to the initial failure/recovery testing:
 
 
 
 INFO [HintedHandoff:1] 2012-10-18 10:19:26,618 HintedHandOffManager.java
 (line 294) Started hinted handoff for token:
 113427455640312821154458202477256070484 with IP: /192.168.128.136
 
 INFO [HintedHandoff:1] 2012-10-18 10:19:26,779 HintedHandOffManager.java
 (line 390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136
 
 INFO [HintedHandoff:1] 2012-10-18 10:29:26,622 HintedHandOffManager.java
 (line 294) Started hinted handoff for token:
 113427455640312821154458202477256070484 with IP: /192.168.128.136
 
 INFO [HintedHandoff:1] 2012-10-18 10:29:26,735 HintedHandOffManager.java
 (line 390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136
 
 INFO [HintedHandoff:1] 2012-10-18 10:39:26,624 HintedHandOffManager.java
 (line 294) Started hinted handoff for token:
 113427455640312821154458202477256070484 with IP: /192.168.128.136
 
 INFO [HintedHandoff:1] 2012-10-18 10:39:26,751 HintedHandOffManager.java
 (line 390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136
 
 
 
 The other nodes are happy and don’t show this behavior. All the test data is
 readable, and everything is fine, but I’m curious why hinted handoff is
 running on one node all the time.
 
 
 
 I searched the bug database, and I found a bug that seems to have the same
 symptoms:
 
 https://issues.apache.org/jira/browse/CASSANDRA-3733
 
 Although it’s been marked fixed in 0.6, this describes my problem exactly.
 
 
 
 I’m running Cassandra 1.1.5 from Datastax on Centos 6.0:
 
 http://rpm.datastax.com/community/noarch/apache-cassandra11-1.1.5-1.noarch.rpm
 
 
 
 Is anyone else seeing this behavior? What can I do to provide more
 information?
 
 
 
 Steve
 
 
 
 
 
 -- 
 David Daeschler

Hinted Handoff runs every ten minutes

[Stephen Pierce]

I installed Cassandra on three nodes. I then ran a test suite against them to 
generate load. The test suite is designed to generate the same type of load 
that we plan to have in production. As one of many tests, I reset one of the 
nodes to check the failure/recovery modes.  Cassandra worked just fine.

I stopped the load generation, and got distracted with some other 
project/problem. A few days later, I noticed something strange on one of the 
nodes. On this node hinted handoff starts every ten minutes, and while it seems 
to finish without any errors, it will be started again in ten minutes. None of 
the nodes has any traffic, and hasn't for several days. I checked the logs, and 
this goes back to the initial failure/recovery testing:

INFO [HintedHandoff:1] 2012-10-18 10:19:26,618 HintedHandOffManager.java (line 
294) Started hinted handoff for token: 113427455640312821154458202477256070484 
with IP: /192.168.128.136
INFO [HintedHandoff:1] 2012-10-18 10:19:26,779 HintedHandOffManager.java (line 
390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136
INFO [HintedHandoff:1] 2012-10-18 10:29:26,622 HintedHandOffManager.java (line 
294) Started hinted handoff for token: 113427455640312821154458202477256070484 
with IP: /192.168.128.136
INFO [HintedHandoff:1] 2012-10-18 10:29:26,735 HintedHandOffManager.java (line 
390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136
INFO [HintedHandoff:1] 2012-10-18 10:39:26,624 HintedHandOffManager.java (line 
294) Started hinted handoff for token: 113427455640312821154458202477256070484 
with IP: /192.168.128.136
INFO [HintedHandoff:1] 2012-10-18 10:39:26,751 HintedHandOffManager.java (line 
390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136

The other nodes are happy and don't show this behavior. All the test data is 
readable, and everything is fine, but I'm curious why hinted handoff is running 
on one node all the time.

I searched the bug database, and I found a bug that seems to have the same 
symptoms:
https://issues.apache.org/jira/browse/CASSANDRA-3733
Although it's been marked fixed in 0.6, this describes my problem exactly.

I'm running Cassandra 1.1.5 from Datastax on Centos 6.0:
http://rpm.datastax.com/community/noarch/apache-cassandra11-1.1.5-1.noarch.rpm

Is anyone else seeing this behavior? What can I do to provide more information?

Steve

Re: Hinted Handoff runs every ten minutes

[David Daeschler]

Hi Steve,

Also confirming this. After having a node go down on Cassandra 1.0.8
there seems to be hinted handoff between two of our 4 nodes every 10
minutes. Our setup also shows 0 rows. It does not appear to have any
effect on the operation of the ring, just fills up the log files.

- David



On Thu, Oct 18, 2012 at 2:10 PM, Stephen Pierce spie...@verifyle.com wrote:
 I installed Cassandra on three nodes. I then ran a test suite against them
 to generate load. The test suite is designed to generate the same type of
 load that we plan to have in production. As one of many tests, I reset one
 of the nodes to check the failure/recovery modes.  Cassandra worked just
 fine.



 I stopped the load generation, and got distracted with some other
 project/problem. A few days later, I noticed something strange on one of the
 nodes. On this node hinted handoff starts every ten minutes, and while it
 seems to finish without any errors, it will be started again in ten minutes.
 None of the nodes has any traffic, and hasn’t for several days. I checked
 the logs, and this goes back to the initial failure/recovery testing:



 INFO [HintedHandoff:1] 2012-10-18 10:19:26,618 HintedHandOffManager.java
 (line 294) Started hinted handoff for token:
 113427455640312821154458202477256070484 with IP: /192.168.128.136

 INFO [HintedHandoff:1] 2012-10-18 10:19:26,779 HintedHandOffManager.java
 (line 390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136

 INFO [HintedHandoff:1] 2012-10-18 10:29:26,622 HintedHandOffManager.java
 (line 294) Started hinted handoff for token:
 113427455640312821154458202477256070484 with IP: /192.168.128.136

 INFO [HintedHandoff:1] 2012-10-18 10:29:26,735 HintedHandOffManager.java
 (line 390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136

 INFO [HintedHandoff:1] 2012-10-18 10:39:26,624 HintedHandOffManager.java
 (line 294) Started hinted handoff for token:
 113427455640312821154458202477256070484 with IP: /192.168.128.136

 INFO [HintedHandoff:1] 2012-10-18 10:39:26,751 HintedHandOffManager.java
 (line 390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136



 The other nodes are happy and don’t show this behavior. All the test data is
 readable, and everything is fine, but I’m curious why hinted handoff is
 running on one node all the time.



 I searched the bug database, and I found a bug that seems to have the same
 symptoms:

 https://issues.apache.org/jira/browse/CASSANDRA-3733

 Although it’s been marked fixed in 0.6, this describes my problem exactly.



 I’m running Cassandra 1.1.5 from Datastax on Centos 6.0:

 http://rpm.datastax.com/community/noarch/apache-cassandra11-1.1.5-1.noarch.rpm



 Is anyone else seeing this behavior? What can I do to provide more
 information?



 Steve





-- 
David Daeschler

Re: how to stop hinted handoff

[CharSyam]

if you don't use Write Level ANY, It will be automactically turned off.

Just use Write Level One.

2012/10/9 Manu Zhang owenzhang1...@gmail.com

 Hi, all

 I tried out the client_only example as another local node 127.0.0.2 and
 then it went down. Now the node (127.0.0.1) started hinted handoff to
 iteself.  How to stop that?

 Thanks!

Re: how to stop hinted handoff

[Manu Zhang]

The example used ONE but the hinted hand off is still performed for every
10 minutes

On Wed, Oct 10, 2012 at 10:25 AM, CharSyam chars...@gmail.com wrote:

 if you don't use Write Level ANY, It will be automactically turned off.

 Just use Write Level One.


 2012/10/9 Manu Zhang owenzhang1...@gmail.com

 Hi, all

 I tried out the client_only example as another local node 127.0.0.2 and
 then it went down. Now the node (127.0.0.1) started hinted handoff to
 iteself.  How to stop that?

 Thanks!

Re: endless hinted handoff with 1.1

[aaron morton]

kinds of like  https://issues.apache.org/jira/browse/CASSANDRA-3733 but maybe 
different. 

Have you recently dropped as CF ? it looks like the hints CF is only compacted 
if hints are replayed. If they are dropped because the CF no longer exists 
compaction is not forced ( 
https://github.com/apache/cassandra/blob/cassandra-1.1/src/java/org/apache/cassandra/db/HintedHandOffManager.java#L370
 ) 

Can you create a ticket on https://issues.apache.org/jira/browse/CASSANDRA with 
the details and update the thread. 

Cheers

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 21/05/2012, at 8:36 PM, Arend-Jan Wijtzes wrote:

 Hi,
 
 We are running a small test cluster and recently installed Cassandra 1.1 and
 started with a new clean database. We keep seeing these messages in the log 
 on just one of our nodes:
 
 INFO [HintedHandoff:1] 2012-05-21 09:49:56,757 HintedHandOffManager.java 
 (line 294) Started hinted handoff for token: 
 85070591730234615865843651857942052864 with IP: /10.0.0.73
 INFO [HintedHandoff:1] 2012-05-21 09:49:56,775 HintedHandOffManager.java 
 (line 382) Finished hinted handoff of 0 rows to endpoint /10.0.0.73
 INFO [HintedHandoff:1] 2012-05-21 09:59:56,756 HintedHandOffManager.java 
 (line 294) Started hinted handoff for token: 
 42535295865117307932921825928971026432 with IP: /10.0.0.69
 INFO [HintedHandoff:1] 2012-05-21 09:59:56,757 HintedHandOffManager.java 
 (line 382) Finished hinted handoff of 0 rows to endpoint /10.0.0.69
 INFO [HintedHandoff:1] 2012-05-21 09:59:56,757 HintedHandOffManager.java 
 (line 294) Started hinted handoff for token: 
 85070591730234615865843651857942052864 with IP: /10.0.0.73
 INFO [HintedHandoff:1] 2012-05-21 09:59:56,775 HintedHandOffManager.java 
 (line 382) Finished hinted handoff of 0 rows to endpoint /10.0.0.73
 INFO [HintedHandoff:1] 2012-05-21 10:09:56,757 HintedHandOffManager.java 
 (line 294) Started hinted handoff for token: 
 42535295865117307932921825928971026432 with IP: /10.0.0.69
 INFO [HintedHandoff:1] 2012-05-21 10:09:56,758 HintedHandOffManager.java 
 (line 382) Finished hinted handoff of 0 rows to endpoint /10.0.0.69
 INFO [HintedHandoff:1] 2012-05-21 10:09:56,758 HintedHandOffManager.java 
 (line 294) Started hinted handoff for token: 
 85070591730234615865843651857942052864 with IP: /10.0.0.73
 INFO [HintedHandoff:1] 2012-05-21 10:09:56,879 HintedHandOffManager.java 
 (line 382) Finished hinted handoff of 0 rows to endpoint /10.0.0.73
 
 
 All four nodes are up:
 
 -bash-4.1$ nodetool ring -h localhost
 Note: Ownership information does not include topology, please specify a 
 keyspace. 
 Address DC  RackStatus State   LoadOwns   
  Token   
   
 127605887595351923798765477786913079296 
 10.0.0.65   datacenter1 rack1   Up Normal  244.41 MB   25.00% 
  0   
 10.0.0.69   datacenter1 rack1   Up Normal  155.39 MB   25.00% 
  42535295865117307932921825928971026432  
 10.0.0.73   datacenter1 rack1   Up Normal  220.42 MB   25.00% 
  85070591730234615865843651857942052864  
 10.0.0.77   datacenter1 rack1   Up Normal  296.14 MB   25.00% 
  127605887595351923798765477786913079296 
 
 
 This has been going on for days. Note that it's just two key's in the log
 that keep repeating. No recent messages about HintedHandOff in the logs 
 on the other nodes.
 
 Let me know if you need more info.
 
 Arend-Jan
 
 -- 
 Arend-Jan Wijtzes -- Wiseguys -- www.wise-guys.nl

endless hinted handoff with 1.1

[Arend-Jan Wijtzes]

Hi,

We are running a small test cluster and recently installed Cassandra 1.1 and
started with a new clean database. We keep seeing these messages in the log 
on just one of our nodes:

INFO [HintedHandoff:1] 2012-05-21 09:49:56,757 HintedHandOffManager.java (line 
294) Started hinted handoff for token: 85070591730234615865843651857942052864 
with IP: /10.0.0.73
 INFO [HintedHandoff:1] 2012-05-21 09:49:56,775 HintedHandOffManager.java (line 
382) Finished hinted handoff of 0 rows to endpoint /10.0.0.73
 INFO [HintedHandoff:1] 2012-05-21 09:59:56,756 HintedHandOffManager.java (line 
294) Started hinted handoff for token: 42535295865117307932921825928971026432 
with IP: /10.0.0.69
 INFO [HintedHandoff:1] 2012-05-21 09:59:56,757 HintedHandOffManager.java (line 
382) Finished hinted handoff of 0 rows to endpoint /10.0.0.69
 INFO [HintedHandoff:1] 2012-05-21 09:59:56,757 HintedHandOffManager.java (line 
294) Started hinted handoff for token: 85070591730234615865843651857942052864 
with IP: /10.0.0.73
 INFO [HintedHandoff:1] 2012-05-21 09:59:56,775 HintedHandOffManager.java (line 
382) Finished hinted handoff of 0 rows to endpoint /10.0.0.73
 INFO [HintedHandoff:1] 2012-05-21 10:09:56,757 HintedHandOffManager.java (line 
294) Started hinted handoff for token: 42535295865117307932921825928971026432 
with IP: /10.0.0.69
 INFO [HintedHandoff:1] 2012-05-21 10:09:56,758 HintedHandOffManager.java (line 
382) Finished hinted handoff of 0 rows to endpoint /10.0.0.69
 INFO [HintedHandoff:1] 2012-05-21 10:09:56,758 HintedHandOffManager.java (line 
294) Started hinted handoff for token: 85070591730234615865843651857942052864 
with IP: /10.0.0.73
 INFO [HintedHandoff:1] 2012-05-21 10:09:56,879 HintedHandOffManager.java (line 
382) Finished hinted handoff of 0 rows to endpoint /10.0.0.73


All four nodes are up:

-bash-4.1$ nodetool ring -h localhost
Note: Ownership information does not include topology, please specify a 
keyspace. 
Address DC  RackStatus State   LoadOwns 
   Token   

   127605887595351923798765477786913079296 
10.0.0.65   datacenter1 rack1   Up Normal  244.41 MB   25.00%   
   0   
10.0.0.69   datacenter1 rack1   Up Normal  155.39 MB   25.00%   
   42535295865117307932921825928971026432  
10.0.0.73   datacenter1 rack1   Up Normal  220.42 MB   25.00%   
   85070591730234615865843651857942052864  
10.0.0.77   datacenter1 rack1   Up Normal  296.14 MB   25.00%   
   127605887595351923798765477786913079296 


This has been going on for days. Note that it's just two key's in the log
that keep repeating. No recent messages about HintedHandOff in the logs 
on the other nodes.

Let me know if you need more info.

Arend-Jan

-- 
Arend-Jan Wijtzes -- Wiseguys -- www.wise-guys.nl

Re: Several times hinted handoff for the same node with Cassandra 1.0.8

[Brandon Williams]

Just ignore it: https://issues.apache.org/jira/browse/CASSANDRA-3955

On Mon, Mar 12, 2012 at 9:31 PM, Roshan codeva...@gmail.com wrote:
 Hi

 I have upgrade our development Cassandra cluster (2 nodes) from 1.0.6 to
 1.0.8 version.

 After upgrade to 1.0.8 version, one node keep trying to send hints every 10
 minutes (seems). There is no any network issues between two nodes and can do
 ping with using server name and ip address. Also the nodetool ring is
 working fine from both end. Could someone please help on this?

 Here is the Cassandra stack of the problem node:

 2012-03-13 13:12:56,138 INFO  [StorageService] Cassandra version: 1.0.8
 2012-03-13 13:12:56,138 INFO  [StorageService] Thrift API version: 19.20.0
 2012-03-13 13:12:56,138 INFO  [StorageService] Loading persisted ring state
 2012-03-13 13:12:56,149 INFO  [StorageService] Starting up server gossip
 2012-03-13 13:12:56,150 INFO  [ColumnFamilyStore] Enqueuing flush of
 Memtable-LocationInfo@1899843147(29/36 serialized/live bytes, 1 ops)
 2012-03-13 13:12:56,152 INFO  [Memtable] Writing
 Memtable-LocationInfo@1899843147(29/36 serialized/live bytes, 1 ops)
 2012-03-13 13:12:56,238 INFO  [Memtable] Completed flushing
 /data/cassandradb/data/system/LocationInfo-hc-50-Data.db (80 bytes)
 2012-03-13 13:12:56,253 INFO  [MessagingService] Starting Messaging Service
 on port 7000
 2012-03-13 13:12:56,259 INFO  [StorageService] Using saved token
 144939581272443147669723010154540982565
 2012-03-13 13:12:56,261 INFO  [ColumnFamilyStore] Enqueuing flush of
 Memtable-LocationInfo@1915784320(53/66 serialized/live bytes, 2 ops)
 2012-03-13 13:12:56,261 INFO  [Memtable] Writing
 Memtable-LocationInfo@1915784320(53/66 serialized/live bytes, 2 ops)
 2012-03-13 13:12:56,350 INFO  [Gossiper] Node /10.1.161.67 has restarted,
 now UP
 2012-03-13 13:12:56,350 INFO  [Gossiper] InetAddress /10.1.161.67 is now UP
 2012-03-13 13:12:56,351 INFO  [StorageService] Node /10.1.161.67 state jump
 to normal
 2012-03-13 13:12:56,378 INFO  [Memtable] Completed flushing
 /data/cassandradb/data/system/LocationInfo-hc-51-Data.db (163 bytes)
 2012-03-13 13:12:56,389 INFO  [CompactionTask] Compacting
 [SSTableReader(path='/data/cassandradb/data/system/LocationInfo-hc-51-Data.db'),
 SSTableReader(path='/data/cassandradb/data/system/LocationInfo-hc-50-Data.db'),
 SSTableReader(path='/data/cassandradb/data/system/LocationInfo-hc-48-Data.db'),
 SSTableReader(path='/data/cassandradb/data/system/LocationInfo-hc-49-Data.db')]
 2012-03-13 13:12:56,402 INFO  [StorageService] Node
 app8.dev1.net/10.1.161.68 state jump to normal
 2012-03-13 13:12:56,403 INFO  [StorageService] Bootstrap/Replace/Move
 completed! Now serving reads.
 2012-03-13 13:12:56,403 INFO  [Mx4jTool] Will not load MX4J, mx4j-tools.jar
 is not in the classpath
 2012-03-13 13:12:56,438 INFO  [CassandraDaemon] Binding thrift service to
 app8.dev1.net/10.1.161.68:9160
 2012-03-13 13:12:56,445 INFO  [CassandraDaemon] Using TFastFramedTransport
 with a max frame size of 62914560 bytes.
 2012-03-13 13:12:56,448 INFO  [CassandraDaemon] Using synchronous/threadpool
 thrift server on app8.dev1.net/10.1.161.68 : 9160
 2012-03-13 13:12:56,448 INFO  [CassandraDaemon] Listening for thrift
 clients...
 2012-03-13 13:12:56,498 INFO  [CompactionTask] Compacted to
 [/data/cassandradb/data/system/LocationInfo-hc-52-Data.db,].  844 to 438
 (~51% of original) bytes for 4 keys at 0.003941MB/s.  Time: 106ms.
 2012-03-13 13:13:42,410 INFO  [HintedHandOffManager] Started hinted handoff
 for token: 59868989542208531803879358296598929701 with IP: /10.1.161.67
 2012-03-13 13:13:42,490 INFO  [HintedHandOffManager] Finished hinted handoff
 of 0 rows to endpoint /10.1.161.67
 2012-03-13 13:23:28,946 INFO  [HintedHandOffManager] Started hinted handoff
 for token: 59868989542208531803879358296598929701 with IP: /10.1.161.67
 2012-03-13 13:23:28,948 INFO  [HintedHandOffManager] Finished hinted handoff
 of 0 rows to endpoint /10.1.161.67
 Waiting for data... (interrupt to abort)

 --
 View this message in context: 
 http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Several-times-hinted-handoff-for-the-same-node-with-Cassandra-1-0-8-tp7367386p7367386.html
 Sent from the cassandra-u...@incubator.apache.org mailing list archive at 
 Nabble.com.

Cassandra keeps on logging Finished hinted handoff of 0 rows to endpoint

[Manoj Mainali]

Hi,

I have been running Cassandra 1.0.7 and in the log file I see the log saying

 Finished hinted handoff of 0 rows to endpoint /{ipaddress}

The above issue can be reproduced by the following steps,

1. Start a cluster with 2 node, suppose node1 and node2
2. Create a keyspace with rf=2, create column family
3. Stop node2
4. Insert some rows, suppose 100, to cluster with consistency level 1
5. Restart node2

When the node2 is restarted, node1 sends the hints to the node2 and from
the log I see that 100 rows are sent. But, after that in the interval of
approximately 10 mins, Cassandra logs Finished hinted handoff of 0 rows to
the endpoint ..

When I do the list hintscolumnfamily from the cassandra-cli, it shows a
result of 1 row, but no columns data.

There seems to be a issue raised before,
https://issues.apache.org/jira/browse/CASSANDRA-3733, and it says it is
fixed in 1.0.7. However, I keep seeing the above log.

It seems that the Cassandra is trying to send hints message even when all
the hints are delivered and there are no more hints left. Is there a way to
solve the above issue?
Recently, another issue was also raised
https://issues.apache.org/jira/browse/CASSANDRA-3935 and they are similar,
but I am not sure if they are caused by the same reason.

Does anyone know how to solve the issue?

Thanks,
Manoj

Re: Cassandra keeps on logging Finished hinted handoff of 0 rows to endpoint

[Brandon Williams]

It's a special case of a single sstable existing for hints:
https://issues.apache.org/jira/browse/CASSANDRA-3955

On Fri, Feb 24, 2012 at 5:43 AM, Manoj Mainali mainalima...@gmail.com wrote:
 Hi,

 I have been running Cassandra 1.0.7 and in the log file I see the log saying

  Finished hinted handoff of 0 rows to endpoint /{ipaddress}

 The above issue can be reproduced by the following steps,

 1. Start a cluster with 2 node, suppose node1 and node2
 2. Create a keyspace with rf=2, create column family
 3. Stop node2
 4. Insert some rows, suppose 100, to cluster with consistency level 1
 5. Restart node2

 When the node2 is restarted, node1 sends the hints to the node2 and from the
 log I see that 100 rows are sent. But, after that in the interval of
 approximately 10 mins, Cassandra logs Finished hinted handoff of 0 rows to
 the endpoint ..

 When I do the list hintscolumnfamily from the cassandra-cli, it shows a
 result of 1 row, but no columns data.

 There seems to be a issue raised
 before, https://issues.apache.org/jira/browse/CASSANDRA-3733, and it says it
 is fixed in 1.0.7. However, I keep seeing the above log.

 It seems that the Cassandra is trying to send hints message even when all
 the hints are delivered and there are no more hints left. Is there a way to
 solve the above issue?
 Recently, another issue was also
 raised https://issues.apache.org/jira/browse/CASSANDRA-3935 and they are
 similar, but I am not sure if they are caused by the same reason.

 Does anyone know how to solve the issue?

 Thanks,
 Manoj

Re: hinted handoff 16 s delay

[Jonathan Ellis]

Right.

See also https://issues.apache.org/jira/browse/CASSANDRA-3958.

On Thu, Feb 23, 2012 at 5:43 PM, Todd Burruss bburr...@expedia.com wrote:
 if I remember correctly, cassandra has a random delay in it so hint
 deliver is staggered and does not overwhelm the just restarted node.

 On 2/23/12 1:46 PM, Hontvári József Levente hontv...@flyordie.com
 wrote:

I have played with a test cluster, stopping cassandra on one node and
updating a row on another. I noticed a delay in delivering hinted
handoffs for which I don't know the rationale. After the node which
originally received the update noticed that the other server is up, it
waited 16 s before it started pushing the hints.

Here is the log:

  INFO [GossipStage:1] 2012-02-23 20:05:32,516 StorageService.java (line
988) Node /192.0.2.1 state jump to normal
  INFO [HintedHandoff:1] 2012-02-23 20:05:49,766
HintedHandOffManager.java (line 296) Started hinted handoff for token: 1
with IP: /192.0.2.1
  INFO [HintedHandoff:1] 2012-02-23 20:05:50,048 ColumnFamilyStore.java
(line 704) Enqueuing flush of
Memtable-HintsColumnFamily@1352140719(205/1639 serialized/live bytes, 2
ops)
  INFO [FlushWriter:31] 2012-02-23 20:05:50,049 Memtable.java (line 246)
Writing Memtable-HintsColumnFamily@1352140719(205/1639 serialized/live
bytes, 2 ops)
  INFO [FlushWriter:31] 2012-02-23 20:05:50,192 Memtable.java (line 283)
Completed flushing
/media/data/cassandra/data/system/HintsColumnFamily-hc-10-Data.db (290
bytes)
  INFO [CompactionExecutor:70] 2012-02-23 20:05:50,193
CompactionTask.java (line 113) Compacting
[SSTableReader(path='/media/data/cassandra/data/system/HintsColumnFamily-h
c-10-Data.db'),
SSTableReader(path='/media/data/cassandra/data/system/HintsColumnFamily-hc
-9-Data.db')]
  INFO [HintedHandoff:1] 2012-02-23 20:05:50,195
HintedHandOffManager.java (line 373) Finished hinted handoff of 1 rows
to endpoint /192.0.2.1





-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: Cassandra keeps on logging Finished hinted handoff of 0 rows to endpoint

[Manoj Mainali]

Thanks.

On Saturday, February 25, 2012, Brandon Williams dri...@gmail.com wrote:
 It's a special case of a single sstable existing for hints:
 https://issues.apache.org/jira/browse/CASSANDRA-3955

 On Fri, Feb 24, 2012 at 5:43 AM, Manoj Mainali mainalima...@gmail.com
wrote:
 Hi,

 I have been running Cassandra 1.0.7 and in the log file I see the log
saying

  Finished hinted handoff of 0 rows to endpoint /{ipaddress}

 The above issue can be reproduced by the following steps,

 1. Start a cluster with 2 node, suppose node1 and node2
 2. Create a keyspace with rf=2, create column family
 3. Stop node2
 4. Insert some rows, suppose 100, to cluster with consistency level 1
 5. Restart node2

 When the node2 is restarted, node1 sends the hints to the node2 and from
the
 log I see that 100 rows are sent. But, after that in the interval of
 approximately 10 mins, Cassandra logs Finished hinted handoff of 0 rows
to
 the endpoint ..

 When I do the list hintscolumnfamily from the cassandra-cli, it shows a
 result of 1 row, but no columns data.

 There seems to be a issue raised
 before, https://issues.apache.org/jira/browse/CASSANDRA-3733, and it
says it
 is fixed in 1.0.7. However, I keep seeing the above log.

 It seems that the Cassandra is trying to send hints message even when all
 the hints are delivered and there are no more hints left. Is there a way
to
 solve the above issue?
 Recently, another issue was also
 raised https://issues.apache.org/jira/browse/CASSANDRA-3935 and they are
 similar, but I am not sure if they are caused by the same reason.

 Does anyone know how to solve the issue?

 Thanks,
 Manoj

hinted handoff 16 s delay

[Hontvári József Levente]

I have played with a test cluster, stopping cassandra on one node and 
updating a row on another. I noticed a delay in delivering hinted 
handoffs for which I don't know the rationale. After the node which 
originally received the update noticed that the other server is up, it 
waited 16 s before it started pushing the hints.


Here is the log:

 INFO [GossipStage:1] 2012-02-23 20:05:32,516 StorageService.java (line 
988) Node /192.0.2.1 state jump to normal
 INFO [HintedHandoff:1] 2012-02-23 20:05:49,766 
HintedHandOffManager.java (line 296) Started hinted handoff for token: 1 
with IP: /192.0.2.1
 INFO [HintedHandoff:1] 2012-02-23 20:05:50,048 ColumnFamilyStore.java 
(line 704) Enqueuing flush of 
Memtable-HintsColumnFamily@1352140719(205/1639 serialized/live bytes, 2 ops)
 INFO [FlushWriter:31] 2012-02-23 20:05:50,049 Memtable.java (line 246) 
Writing Memtable-HintsColumnFamily@1352140719(205/1639 serialized/live 
bytes, 2 ops)
 INFO [FlushWriter:31] 2012-02-23 20:05:50,192 Memtable.java (line 283) 
Completed flushing 
/media/data/cassandra/data/system/HintsColumnFamily-hc-10-Data.db (290 
bytes)
 INFO [CompactionExecutor:70] 2012-02-23 20:05:50,193 
CompactionTask.java (line 113) Compacting 
[SSTableReader(path='/media/data/cassandra/data/system/HintsColumnFamily-hc-10-Data.db'), 
SSTableReader(path='/media/data/cassandra/data/system/HintsColumnFamily-hc-9-Data.db')]
 INFO [HintedHandoff:1] 2012-02-23 20:05:50,195 
HintedHandOffManager.java (line 373) Finished hinted handoff of 1 rows 
to endpoint /192.0.2.1

Re: hinted handoff 16 s delay

[Todd Burruss]

if I remember correctly, cassandra has a random delay in it so hint
deliver is staggered and does not overwhelm the just restarted node.

On 2/23/12 1:46 PM, Hontvári József Levente hontv...@flyordie.com
wrote:

I have played with a test cluster, stopping cassandra on one node and
updating a row on another. I noticed a delay in delivering hinted
handoffs for which I don't know the rationale. After the node which
originally received the update noticed that the other server is up, it
waited 16 s before it started pushing the hints.

Here is the log:

  INFO [GossipStage:1] 2012-02-23 20:05:32,516 StorageService.java (line
988) Node /192.0.2.1 state jump to normal
  INFO [HintedHandoff:1] 2012-02-23 20:05:49,766
HintedHandOffManager.java (line 296) Started hinted handoff for token: 1
with IP: /192.0.2.1
  INFO [HintedHandoff:1] 2012-02-23 20:05:50,048 ColumnFamilyStore.java
(line 704) Enqueuing flush of
Memtable-HintsColumnFamily@1352140719(205/1639 serialized/live bytes, 2
ops)
  INFO [FlushWriter:31] 2012-02-23 20:05:50,049 Memtable.java (line 246)
Writing Memtable-HintsColumnFamily@1352140719(205/1639 serialized/live
bytes, 2 ops)
  INFO [FlushWriter:31] 2012-02-23 20:05:50,192 Memtable.java (line 283)
Completed flushing
/media/data/cassandra/data/system/HintsColumnFamily-hc-10-Data.db (290
bytes)
  INFO [CompactionExecutor:70] 2012-02-23 20:05:50,193
CompactionTask.java (line 113) Compacting
[SSTableReader(path='/media/data/cassandra/data/system/HintsColumnFamily-h
c-10-Data.db'), 
SSTableReader(path='/media/data/cassandra/data/system/HintsColumnFamily-hc
-9-Data.db')]
  INFO [HintedHandoff:1] 2012-02-23 20:05:50,195
HintedHandOffManager.java (line 373) Finished hinted handoff of 1 rows
to endpoint /192.0.2.1

Re: hinted handoff 16 s delay

[Maki Watanabe]

I've verified it in the source: deliverHintsToEndpointInternal in
HintedHandOffManager.java
Yes it add random delay before HH delivery.

2012/2/24 Todd Burruss bburr...@expedia.com:
 if I remember correctly, cassandra has a random delay in it so hint
 deliver is staggered and does not overwhelm the just restarted node.

 On 2/23/12 1:46 PM, Hontvári József Levente hontv...@flyordie.com
 wrote:

I have played with a test cluster, stopping cassandra on one node and
updating a row on another. I noticed a delay in delivering hinted
handoffs for which I don't know the rationale. After the node which
originally received the update noticed that the other server is up, it
waited 16 s before it started pushing the hints.

Here is the log:

  INFO [GossipStage:1] 2012-02-23 20:05:32,516 StorageService.java (line
988) Node /192.0.2.1 state jump to normal
  INFO [HintedHandoff:1] 2012-02-23 20:05:49,766
HintedHandOffManager.java (line 296) Started hinted handoff for token: 1
with IP: /192.0.2.1
  INFO [HintedHandoff:1] 2012-02-23 20:05:50,048 ColumnFamilyStore.java
(line 704) Enqueuing flush of
Memtable-HintsColumnFamily@1352140719(205/1639 serialized/live bytes, 2
ops)
  INFO [FlushWriter:31] 2012-02-23 20:05:50,049 Memtable.java (line 246)
Writing Memtable-HintsColumnFamily@1352140719(205/1639 serialized/live
bytes, 2 ops)
  INFO [FlushWriter:31] 2012-02-23 20:05:50,192 Memtable.java (line 283)
Completed flushing
/media/data/cassandra/data/system/HintsColumnFamily-hc-10-Data.db (290
bytes)
  INFO [CompactionExecutor:70] 2012-02-23 20:05:50,193
CompactionTask.java (line 113) Compacting
[SSTableReader(path='/media/data/cassandra/data/system/HintsColumnFamily-h
c-10-Data.db'),
SSTableReader(path='/media/data/cassandra/data/system/HintsColumnFamily-hc
-9-Data.db')]
  INFO [HintedHandoff:1] 2012-02-23 20:05:50,195
HintedHandOffManager.java (line 373) Finished hinted handoff of 1 rows
to endpoint /192.0.2.1





-- 
w3m

Hinted handoff bug?

[Fredrik L Stigbäck]

Hi,
We,re running cassandra 1.0.3.
I've done some testing with 2 nodes (node A, node B), replication factor 2.
I take node A down, writing some data to node B and then take node A up.
Sometimes hints aren't delivered when node A comes up.

I've done some debugging in org.apache.cassandra.db.HintedHandOffManager 
and sometimes node B ends up in a strange state in method
org.apache.cassandra.db.HintedHandOffManager.deliverHints(final 
InetAddress to), where 
org.apache.cassandra.db.HintedHandOffManager.queuedDeliveries already 
has node A in it's Set and therefore no hints will ever be delivered to 
node A.
The only reason for this that I can see is that in 
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(InetAddress 
endpoint) the hintStore.isEmpty() check returns true and the endpoint 
(node A)  isn't removed from 
org.apache.cassandra.db.HintedHandOffManager.queuedDeliveries. Then no 
hints will ever be delivered again until node B is restarted.

During what conditions will hintStore.isEmpty() return true?
Shouldn't the hintStore.isEmpty() check be inside the try {} finally{} 
clause, removing the endpoint from queuedDeliveries in the finally block?


public void deliverHints(final InetAddress to)
{
logger_.debug(deliverHints to {}, to);
*if (!queuedDeliveries.add(to))*
return;
...
}

private void deliverHintsToEndpoint(InetAddress endpoint) throws 
IOException, DigestMismatchException, InvalidRequestException, 
TimeoutException,

{
ColumnFamilyStore hintStore = 
Table.open(Table.SYSTEM_TABLE).getColumnFamilyStore(HINTS_CF);

*   if (hintStore.isEmpty())*
return; // nothing to do, don't confuse users by logging a 
no-op handoff

try
{
..
}
finally
{
*queuedDeliveries.remove(endpoint);*
}
}

Regards
/Fredrik

Re: Hinted handoff bug?

[Sylvain Lebresne]

You're right, good catch.
Do you mind opening a ticket on jira
(https://issues.apache.org/jira/browse/CASSANDRA)?

--
Sylvain

On Thu, Dec 1, 2011 at 10:03 AM, Fredrik L Stigbäck
fredrik.l.stigb...@sitevision.se wrote:
 Hi,
 We,re running cassandra 1.0.3.
 I've done some testing with 2 nodes (node A, node B), replication factor 2.
 I take node A down, writing some data to node B and then take node A up.
 Sometimes hints aren't delivered when node A comes up.

 I've done some debugging in org.apache.cassandra.db.HintedHandOffManager and
 sometimes node B ends up in a strange state in method
 org.apache.cassandra.db.HintedHandOffManager.deliverHints(final InetAddress
 to), where org.apache.cassandra.db.HintedHandOffManager.queuedDeliveries
 already has node A in it's Set and therefore no hints will ever be delivered
 to node A.
 The only reason for this that I can see is that in
 org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(InetAddress
 endpoint) the hintStore.isEmpty() check returns true and the endpoint (node
 A)  isn't removed from
 org.apache.cassandra.db.HintedHandOffManager.queuedDeliveries. Then no hints
 will ever be delivered again until node B is restarted.
 During what conditions will hintStore.isEmpty() return true?
 Shouldn't the hintStore.isEmpty() check be inside the try {} finally{}
 clause, removing the endpoint from queuedDeliveries in the finally block?

 public void deliverHints(final InetAddress to)
 {
     logger_.debug(deliverHints to {}, to);
     if (!queuedDeliveries.add(to))
     return;
         ...
 }

 private void deliverHintsToEndpoint(InetAddress endpoint) throws
 IOException, DigestMismatchException, InvalidRequestException,
 TimeoutException,
 {
     ColumnFamilyStore hintStore =
 Table.open(Table.SYSTEM_TABLE).getColumnFamilyStore(HINTS_CF);
     if (hintStore.isEmpty())
     return; // nothing to do, don't confuse users by logging a no-op
 handoff
     try
     {
     ..
     }
     finally
     {
     queuedDeliveries.remove(endpoint);
     }
 }

 Regards
 /Fredrik

Re: Hinted handoff bug?

[Fredrik L Stigbäck]

Yes, I'll do that.

/Fredrik
Sylvain Lebresne skrev 2011-12-01 11:10:

You're right, good catch.
Do you mind opening a ticket on jira
(https://issues.apache.org/jira/browse/CASSANDRA)?

--
Sylvain

On Thu, Dec 1, 2011 at 10:03 AM, Fredrik L Stigbäck
fredrik.l.stigb...@sitevision.se  wrote:

Hi,
We,re running cassandra 1.0.3.
I've done some testing with 2 nodes (node A, node B), replication factor 2.
I take node A down, writing some data to node B and then take node A up.
Sometimes hints aren't delivered when node A comes up.

I've done some debugging in org.apache.cassandra.db.HintedHandOffManager and
sometimes node B ends up in a strange state in method
org.apache.cassandra.db.HintedHandOffManager.deliverHints(final InetAddress
to), where org.apache.cassandra.db.HintedHandOffManager.queuedDeliveries
already has node A in it's Set and therefore no hints will ever be delivered
to node A.
The only reason for this that I can see is that in
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(InetAddress
endpoint) the hintStore.isEmpty() check returns true and the endpoint (node
A)  isn't removed from
org.apache.cassandra.db.HintedHandOffManager.queuedDeliveries. Then no hints
will ever be delivered again until node B is restarted.
During what conditions will hintStore.isEmpty() return true?
Shouldn't the hintStore.isEmpty() check be inside the try {} finally{}
clause, removing the endpoint from queuedDeliveries in the finally block?

public void deliverHints(final InetAddress to)
{
 logger_.debug(deliverHints to {}, to);
 if (!queuedDeliveries.add(to))
 return;
 ...
}

private void deliverHintsToEndpoint(InetAddress endpoint) throws
IOException, DigestMismatchException, InvalidRequestException,
TimeoutException,
{
 ColumnFamilyStore hintStore =
Table.open(Table.SYSTEM_TABLE).getColumnFamilyStore(HINTS_CF);
 if (hintStore.isEmpty())
 return; // nothing to do, don't confuse users by logging a no-op
handoff
 try
 {
 ..
 }
 finally
 {
 queuedDeliveries.remove(endpoint);
 }
}

Regards
/Fredrik

Re: Hinted handoff bug?

[Terje Marthinussen]

Sorry for not checking source to see if things have changed but i just 
remembered an issue I have forgotten to make jira for.

In old days, nodes would periodically try to deliver queues.

However, this was at some stage changed so it only deliver if a node is being 
marked up.

However, you can definitely have a scenario where  A fails to deliver to B so 
it send the hint to C instead.

However, B is not really down, it just could not accept that packet at that 
time and C always (correctly in this case) thinks B is up and it never tries to 
deliver the hints to B.

Will this change fix this, or do we need to get back the thread that 
periodically tried to deliver hints regardless of node status changes?

Regards,
Terje

On 1 Dec 2011, at 19:10, Sylvain Lebresne sylv...@datastax.com wrote:

 You're right, good catch.
 Do you mind opening a ticket on jira
 (https://issues.apache.org/jira/browse/CASSANDRA)?
 
 --
 Sylvain
 
 On Thu, Dec 1, 2011 at 10:03 AM, Fredrik L Stigbäck
 fredrik.l.stigb...@sitevision.se wrote:
 Hi,
 We,re running cassandra 1.0.3.
 I've done some testing with 2 nodes (node A, node B), replication factor 2.
 I take node A down, writing some data to node B and then take node A up.
 Sometimes hints aren't delivered when node A comes up.
 
 I've done some debugging in org.apache.cassandra.db.HintedHandOffManager and
 sometimes node B ends up in a strange state in method
 org.apache.cassandra.db.HintedHandOffManager.deliverHints(final InetAddress
 to), where org.apache.cassandra.db.HintedHandOffManager.queuedDeliveries
 already has node A in it's Set and therefore no hints will ever be delivered
 to node A.
 The only reason for this that I can see is that in
 org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(InetAddress
 endpoint) the hintStore.isEmpty() check returns true and the endpoint (node
 A)  isn't removed from
 org.apache.cassandra.db.HintedHandOffManager.queuedDeliveries. Then no hints
 will ever be delivered again until node B is restarted.
 During what conditions will hintStore.isEmpty() return true?
 Shouldn't the hintStore.isEmpty() check be inside the try {} finally{}
 clause, removing the endpoint from queuedDeliveries in the finally block?
 
 public void deliverHints(final InetAddress to)
 {
 logger_.debug(deliverHints to {}, to);
 if (!queuedDeliveries.add(to))
 return;
 ...
 }
 
 private void deliverHintsToEndpoint(InetAddress endpoint) throws
 IOException, DigestMismatchException, InvalidRequestException,
 TimeoutException,
 {
 ColumnFamilyStore hintStore =
 Table.open(Table.SYSTEM_TABLE).getColumnFamilyStore(HINTS_CF);
 if (hintStore.isEmpty())
 return; // nothing to do, don't confuse users by logging a no-op
 handoff
 try
 {
 ..
 }
 finally
 {
 queuedDeliveries.remove(endpoint);
 }
 }
 
 Regards
 /Fredrik

Re: Hinted handoff bug?

[Jonathan Ellis]

Nope, that's a separate issue.
https://issues.apache.org/jira/browse/CASSANDRA-3554

On Thu, Dec 1, 2011 at 5:59 PM, Terje Marthinussen
tmarthinus...@gmail.com wrote:
 Sorry for not checking source to see if things have changed but i just 
 remembered an issue I have forgotten to make jira for.

 In old days, nodes would periodically try to deliver queues.

 However, this was at some stage changed so it only deliver if a node is being 
 marked up.

 However, you can definitely have a scenario where  A fails to deliver to B so 
 it send the hint to C instead.

 However, B is not really down, it just could not accept that packet at that 
 time and C always (correctly in this case) thinks B is up and it never tries 
 to deliver the hints to B.

 Will this change fix this, or do we need to get back the thread that 
 periodically tried to deliver hints regardless of node status changes?

 Regards,
 Terje

 On 1 Dec 2011, at 19:10, Sylvain Lebresne sylv...@datastax.com wrote:

 You're right, good catch.
 Do you mind opening a ticket on jira
 (https://issues.apache.org/jira/browse/CASSANDRA)?

 --
 Sylvain

 On Thu, Dec 1, 2011 at 10:03 AM, Fredrik L Stigbäck
 fredrik.l.stigb...@sitevision.se wrote:
 Hi,
 We,re running cassandra 1.0.3.
 I've done some testing with 2 nodes (node A, node B), replication factor 2.
 I take node A down, writing some data to node B and then take node A up.
 Sometimes hints aren't delivered when node A comes up.

 I've done some debugging in org.apache.cassandra.db.HintedHandOffManager and
 sometimes node B ends up in a strange state in method
 org.apache.cassandra.db.HintedHandOffManager.deliverHints(final InetAddress
 to), where org.apache.cassandra.db.HintedHandOffManager.queuedDeliveries
 already has node A in it's Set and therefore no hints will ever be delivered
 to node A.
 The only reason for this that I can see is that in
 org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(InetAddress
 endpoint) the hintStore.isEmpty() check returns true and the endpoint (node
 A)  isn't removed from
 org.apache.cassandra.db.HintedHandOffManager.queuedDeliveries. Then no hints
 will ever be delivered again until node B is restarted.
 During what conditions will hintStore.isEmpty() return true?
 Shouldn't the hintStore.isEmpty() check be inside the try {} finally{}
 clause, removing the endpoint from queuedDeliveries in the finally block?

 public void deliverHints(final InetAddress to)
 {
         logger_.debug(deliverHints to {}, to);
         if (!queuedDeliveries.add(to))
             return;
         ...
 }

 private void deliverHintsToEndpoint(InetAddress endpoint) throws
 IOException, DigestMismatchException, InvalidRequestException,
 TimeoutException,
 {
         ColumnFamilyStore hintStore =
 Table.open(Table.SYSTEM_TABLE).getColumnFamilyStore(HINTS_CF);
         if (hintStore.isEmpty())
             return; // nothing to do, don't confuse users by logging a no-op
 handoff
     try
     {
         ..
     }
     finally
     {
             queuedDeliveries.remove(endpoint);
     }
 }

 Regards
 /Fredrik



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

1.0.2. hinted handoff assert

[Radim Kolar]

If you are still hunting 1.0 assert during hintedhandoff, i can 
reproduce it anytime. both nodes are 1.0.2. And just hint, we are using 
super column families.


 INFO [HintedHandoff:1] 2011-11-08 21:33:54,118 
HintedHandOffManager.java (line 268) Started hinted handoff for token: 0 
with IP: /***.104.18
ERROR [HintedHandoff:1] 2011-11-08 21:33:54,257 
AbstractCassandraDaemon.java (line 133) Fatal exception in thread 
Thread[HintedHandoff:1,1,main]

java.lang.AssertionError
at 
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:301)
at 
org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:81)
at 
org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:353)
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

at java.lang.Thread.run(Thread.java:679)
ERROR [HintedHandoff:1] 2011-11-08 21:33:54,274 
AbstractCassandraDaemon.java (line 133) Fatal exception in thread 
Thread[HintedHandoff:1,1,main]

java.lang.AssertionError
at 
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:301)
at 
org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:81)
at 
org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:353)
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

at java.lang.Thread.run(Thread.java:679)

Re: 1.0.2. hinted handoff assert

[Jonathan Ellis]

If you can post a log at DEBUG level to
https://issues.apache.org/jira/browse/CASSANDRA-3440 that would be
helpful.

2011/11/8 Radim Kolar h...@sendmail.cz:
 If you are still hunting 1.0 assert during hintedhandoff, i can reproduce it
 anytime. both nodes are 1.0.2. And just hint, we are using super column
 families.

  INFO [HintedHandoff:1] 2011-11-08 21:33:54,118 HintedHandOffManager.java
 (line 268) Started hinted handoff for token: 0 with IP: /***.104.18
 ERROR [HintedHandoff:1] 2011-11-08 21:33:54,257 AbstractCassandraDaemon.java
 (line 133) Fatal exception in thread Thread[HintedHandoff:1,1,main]
 java.lang.AssertionError
        at
 org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:301)
        at
 org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:81)
        at
 org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:353)
        at
 org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
        at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:679)
 ERROR [HintedHandoff:1] 2011-11-08 21:33:54,274 AbstractCassandraDaemon.java
 (line 133) Fatal exception in thread Thread[HintedHandoff:1,1,main]
 java.lang.AssertionError
        at
 org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:301)
        at
 org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:81)
        at
 org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:353)
        at
 org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
        at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:679)





-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: 1.0.2. hinted handoff assert

[Radim Kolar]

mine issue has different stacktrace

Re: 1.0.2. hinted handoff assert

[Jonathan Ellis]

You're right.  https://issues.apache.org/jira/browse/CASSANDRA-3466

On Tue, Nov 8, 2011 at 3:14 PM, Radim Kolar h...@sendmail.cz wrote:
 mine issue has different stacktrace




-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com