Re: When are hints written?

[Jan]

HI Bo; 

you raised 2 questions: 
20% system utilization
Hints 

20% system utilization:  For a node or a cluster to have 20% utilization is 
Normal during peak write operation.  
Hints:   hints are written when a node is unreachable;C* 3.0  has a 
complete over haul in the way hints have been implemented. 

Recommend reading up this blog article: 
http://www.datastax.com/dev/blog/whats-coming-to-cassandra-in-3-0-improved-hint-storage-and-delivery

hope this helps
Jan/



On Thu, 4/21/16, Jens Rantil <jens.ran...@tink.se> wrote:

 Subject: Re: When are hints written?
 To: user@cassandra.apache.org
 Date: Thursday, April 21, 2016, 8:57 AM
 
 Hi again
 Bo,
 I assume this is the piece of
 documentation you are referring to? 
http://docs.datastax.com/en/cassandra/2.0/cassandra/dml/dml_about_hh_c.html?scroll=concept_ds_ifg_jqx_zj__performance
 
 > If a
 replica node is overloaded or unavailable, and the failure
 detector has not yet marked it down, then expect most or all
 writes to that node to fail after the timeout triggered by
 write_request_timeout_in_ms,
 which defaults to 10 seconds. During that time, Cassandra
 writes the hint when the timeout is reached.
 I'm not an expert on this, but
 the way I've seen is that hints are written stored as
 soon as there is _any_ issues writing a mutation
 (insert/update/delete) to a node. By "issue", that
 essentially means that a node hasn't acknowledged back
 to the coordinator that the write succeeded within
 write_request_timeout_in_ms. This includes TCP/socket
 timeouts, connection issues or that the node is down. The
 hints are stored for a maximum timespan defaulting to 3
 hours.
 
 Cheers,
 Jens
 On Thu, Apr
 21, 2016 at 8:06 AM Bo Finnerup Madsen <bo.gunder...@gmail.com>
 wrote:
 Hi Jens,
 Thank you
 for the tip!ALL would definitely cure our hints
 issue, but as you note, it is not optimal as we are unable
 to take down nodes without clients failing.
 I am most probably overlooking
 something in the documentation, but I cannot see any
 description of when hints are written other than when a node
 is marked as being down. And since none of our nodes have
 been marked as being down (at least according to the logs),
 I suspect that there is some timeout that governs when hints
 are written?
 Regarding
 your other post: Yes, 3.0.3 is pretty new. But we are new to
 this cassandra game, and our schema-fu is not strong enough
 for us to create a schema without using materialized views
 :)
 
 ons. 20. apr. 2016 kl. 17.09 skrev Jens Rantil
 <jens.ran...@tink.se>:
 Hi Bo,
 > In our case, I would like for the
 cluster to wait for the write to be persisted on the
 relevant nodes before returning an ok to the client. But I don't know which
 knobs to turn to accomplish this? or if it is even possible
 :)
 This is what write consistency
 option is for. Have a look at 
https://docs.datastax.com/en/cassandra/2.0/cassandra/dml/dml_config_consistency_c.html.
 Note, however that if you use ALL, your clients will fail
 (throw exception, depending on language) as soon as a single
 partition can't be written. This means you can't do
 online maintenance of a Cassandra node (such as upgrading it
 etc.) without experiencing write issues.
 Cheers,Jens
 On Wed, Apr 20, 2016 at 3:39 PM Bo Finnerup Madsen
 <bo.gunder...@gmail.com>
 wrote:
 Hi,
 We have a
 small 5 node cluster of m4.xlarge clients that receives
 writes from ~20 clients. The clients will write as fast as
 they can, and the whole process is limited by the write
 performance of the cassandra cluster.After we have tweaked our schema to
 avoid large partitions, the load is going ok and we
 don't see any warnings or errors in the cassandra logs.
 But we do see quite a lot of hint handoff activity. During
 the load, the cassandra nodes are quite loaded, with linux
 reporting a load as high as 20.
 I have read the available
 documentation on how hints works, and to my understanding
 hints should only be written if a node is down. But as far
 as I can see, none of the nodes are marked as down during
 the load. So I suspect I am missing something
 :)We have configured the servers
 with write_request_timeout_in_ms: 12 and the clients
 with a timeout of 13, but still get hints
 stored.
 In our case, I
 would like for the cluster to wait for the write to be
 persisted on the relevant nodes before returning an ok to
 the client. But I don't know which knobs to turn to
 accomplish this? or if it is even possible :)
 We are running cassandra 3.0.3, with
 8Gb heap and a replication factor of 3.
 Thank you in advance!
 Yours sincerely,  Bo
 Madsen
 -- 
 
 
 
 
 
 
 
 
 Jens Rantil
 Backend Developer @ Tink
 Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
 For urgent matters you can reach me at
 +46-708-84 18 32.
 -- 
 
 
 
 
 
 
 
 
 Jens Rantil
 Backend Developer @ Tink
 Tink AB, Wallingatan 5, 111 60
 Stockholm, Sweden
 For urgent matters you can
 reach me at +46-708-84 18 32.

Re: When are hints written?

[Jens Rantil]

Hi again Bo,

I assume this is the piece of documentation you are referring to?
http://docs.datastax.com/en/cassandra/2.0/cassandra/dml/dml_about_hh_c.html?scroll=concept_ds_ifg_jqx_zj__performance

> If a replica node is overloaded or unavailable, and the failure detector
has not yet marked it down, then expect most or all writes to that node to
fail after the timeout triggered by write_request_timeout_in_ms,
<http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configCassandra_yaml_r.html#reference_ds_qfg_n1r_1k__write_request_timeout_in_ms>
which defaults to 10 seconds. During that time, Cassandra writes the hint
when the timeout is reached.

I'm not an expert on this, but the way I've seen is that hints are written
stored as soon as there is _any_ issues writing a mutation
(insert/update/delete) to a node. By "issue", that essentially means that a
node hasn't acknowledged back to the coordinator that the write succeeded
within write_request_timeout_in_ms. This includes TCP/socket timeouts,
connection issues or that the node is down. The hints are stored for a
maximum timespan defaulting to 3 hours.

Cheers,
Jens

On Thu, Apr 21, 2016 at 8:06 AM Bo Finnerup Madsen <bo.gunder...@gmail.com>
wrote:

> Hi Jens,
>
> Thank you for the tip!
> ALL would definitely cure our hints issue, but as you note, it is not
> optimal as we are unable to take down nodes without clients failing.
>
> I am most probably overlooking something in the documentation, but I
> cannot see any description of when hints are written other than when a node
> is marked as being down. And since none of our nodes have been marked as
> being down (at least according to the logs), I suspect that there is some
> timeout that governs when hints are written?
>
> Regarding your other post: Yes, 3.0.3 is pretty new. But we are new to
> this cassandra game, and our schema-fu is not strong enough for us to
> create a schema without using materialized views :)
>
>
> ons. 20. apr. 2016 kl. 17.09 skrev Jens Rantil <jens.ran...@tink.se>:
>
>> Hi Bo,
>>
>> > In our case, I would like for the cluster to wait for the write to be
>> persisted on the relevant nodes before returning an ok to the client.
>> But I don't know which knobs to turn to accomplish this? or if it is even
>> possible :)
>>
>> This is what write consistency option is for. Have a look at
>> https://docs.datastax.com/en/cassandra/2.0/cassandra/dml/dml_config_consistency_c.html.
>> Note, however that if you use ALL, your clients will fail (throw exception,
>> depending on language) as soon as a single partition can't be written. This
>> means you can't do online maintenance of a Cassandra node (such as
>> upgrading it etc.) without experiencing write issues.
>>
>> Cheers,
>> Jens
>>
>> On Wed, Apr 20, 2016 at 3:39 PM Bo Finnerup Madsen <
>> bo.gunder...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> We have a small 5 node cluster of m4.xlarge clients that receives writes
>>> from ~20 clients. The clients will write as fast as they can, and the whole
>>> process is limited by the write performance of the cassandra cluster.
>>> After we have tweaked our schema to avoid large partitions, the load is
>>> going ok and we don't see any warnings or errors in the cassandra logs. But
>>> we do see quite a lot of hint handoff activity. During the load, the
>>> cassandra nodes are quite loaded, with linux reporting a load as high as 20.
>>>
>>> I have read the available documentation on how hints works, and to my
>>> understanding hints should only be written if a node is down. But as far as
>>> I can see, none of the nodes are marked as down during the load. So I
>>> suspect I am missing something :)
>>> We have configured the servers with write_request_timeout_in_ms: 12
>>> and the clients with a timeout of 13, but still get hints stored.
>>>
>>> In our case, I would like for the cluster to wait for the write to be
>>> persisted on the relevant nodes before returning an ok to the client. But I
>>> don't know which knobs to turn to accomplish this? or if it is even
>>> possible :)
>>>
>>> We are running cassandra 3.0.3, with 8Gb heap and a replication factor
>>> of 3.
>>>
>>> Thank you in advance!
>>>
>>> Yours sincerely,
>>>   Bo Madsen
>>>
>> --
>>
>> Jens Rantil
>> Backend Developer @ Tink
>>
>> Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
>> For urgent matters you can reach me at +46-708-84 18 32.
>>
> --

Jens Rantil
Backend Developer @ Tink

Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
For urgent matters you can reach me at +46-708-84 18 32.

Re: When are hints written?

[Bo Finnerup Madsen]

Hi Jens,

Thank you for the tip!
ALL would definitely cure our hints issue, but as you note, it is not
optimal as we are unable to take down nodes without clients failing.

I am most probably overlooking something in the documentation, but I cannot
see any description of when hints are written other than when a node is
marked as being down. And since none of our nodes have been marked as being
down (at least according to the logs), I suspect that there is some timeout
that governs when hints are written?

Regarding your other post: Yes, 3.0.3 is pretty new. But we are new to this
cassandra game, and our schema-fu is not strong enough for us to create a
schema without using materialized views :)

ons. 20. apr. 2016 kl. 17.09 skrev Jens Rantil <jens.ran...@tink.se>:

> Hi Bo,
>
> > In our case, I would like for the cluster to wait for the write to be
> persisted on the relevant nodes before returning an ok to the client. But
> I don't know which knobs to turn to accomplish this? or if it is even
> possible :)
>
> This is what write consistency option is for. Have a look at
> https://docs.datastax.com/en/cassandra/2.0/cassandra/dml/dml_config_consistency_c.html.
> Note, however that if you use ALL, your clients will fail (throw exception,
> depending on language) as soon as a single partition can't be written. This
> means you can't do online maintenance of a Cassandra node (such as
> upgrading it etc.) without experiencing write issues.
>
> Cheers,
> Jens
>
> On Wed, Apr 20, 2016 at 3:39 PM Bo Finnerup Madsen <bo.gunder...@gmail.com>
> wrote:
>
>> Hi,
>>
>> We have a small 5 node cluster of m4.xlarge clients that receives writes
>> from ~20 clients. The clients will write as fast as they can, and the whole
>> process is limited by the write performance of the cassandra cluster.
>> After we have tweaked our schema to avoid large partitions, the load is
>> going ok and we don't see any warnings or errors in the cassandra logs. But
>> we do see quite a lot of hint handoff activity. During the load, the
>> cassandra nodes are quite loaded, with linux reporting a load as high as 20.
>>
>> I have read the available documentation on how hints works, and to my
>> understanding hints should only be written if a node is down. But as far as
>> I can see, none of the nodes are marked as down during the load. So I
>> suspect I am missing something :)
>> We have configured the servers with write_request_timeout_in_ms: 12
>> and the clients with a timeout of 13, but still get hints stored.
>>
>> In our case, I would like for the cluster to wait for the write to be
>> persisted on the relevant nodes before returning an ok to the client. But I
>> don't know which knobs to turn to accomplish this? or if it is even
>> possible :)
>>
>> We are running cassandra 3.0.3, with 8Gb heap and a replication factor of
>> 3.
>>
>> Thank you in advance!
>>
>> Yours sincerely,
>>   Bo Madsen
>>
> --
>
> Jens Rantil
> Backend Developer @ Tink
>
> Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
> For urgent matters you can reach me at +46-708-84 18 32.
>

Re: When are hints written?

[Vinu Thomas]

how do you unsubscribe



From: Bo Finnerup Madsen <bo.gunder...@gmail.com>
Sent: Wednesday, April 20, 2016 9:38 AM
To: user@cassandra.apache.org
Subject: When are hints written?

Hi,

We have a small 5 node cluster of m4.xlarge clients that receives writes from 
~20 clients. The clients will write as fast as they can, and the whole process 
is limited by the write performance of the cassandra cluster.
After we have tweaked our schema to avoid large partitions, the load is going 
ok and we don't see any warnings or errors in the cassandra logs. But we do see 
quite a lot of hint handoff activity. During the load, the cassandra nodes are 
quite loaded, with linux reporting a load as high as 20.

I have read the available documentation on how hints works, and to my 
understanding hints should only be written if a node is down. But as far as I 
can see, none of the nodes are marked as down during the load. So I suspect I 
am missing something :)
We have configured the servers with write_request_timeout_in_ms: 12 and the 
clients with a timeout of 13, but still get hints stored.

In our case, I would like for the cluster to wait for the write to be persisted 
on the relevant nodes before returning an ok to the client. But I don't know 
which knobs to turn to accomplish this? or if it is even possible :)

We are running cassandra 3.0.3, with 8Gb heap and a replication factor of 3.

Thank you in advance!

Yours sincerely,
  Bo Madsen

Re: When are hints written?

[Jens Rantil]

Hi Bo,

> In our case, I would like for the cluster to wait for the write to be
persisted on the relevant nodes before returning an ok to the client. But I
don't know which knobs to turn to accomplish this? or if it is even
possible :)

This is what write consistency option is for. Have a look at
https://docs.datastax.com/en/cassandra/2.0/cassandra/dml/dml_config_consistency_c.html.
Note, however that if you use ALL, your clients will fail (throw exception,
depending on language) as soon as a single partition can't be written. This
means you can't do online maintenance of a Cassandra node (such as
upgrading it etc.) without experiencing write issues.

Cheers,
Jens

On Wed, Apr 20, 2016 at 3:39 PM Bo Finnerup Madsen 
wrote:

> Hi,
>
> We have a small 5 node cluster of m4.xlarge clients that receives writes
> from ~20 clients. The clients will write as fast as they can, and the whole
> process is limited by the write performance of the cassandra cluster.
> After we have tweaked our schema to avoid large partitions, the load is
> going ok and we don't see any warnings or errors in the cassandra logs. But
> we do see quite a lot of hint handoff activity. During the load, the
> cassandra nodes are quite loaded, with linux reporting a load as high as 20.
>
> I have read the available documentation on how hints works, and to my
> understanding hints should only be written if a node is down. But as far as
> I can see, none of the nodes are marked as down during the load. So I
> suspect I am missing something :)
> We have configured the servers with write_request_timeout_in_ms: 12
> and the clients with a timeout of 13, but still get hints stored.
>
> In our case, I would like for the cluster to wait for the write to be
> persisted on the relevant nodes before returning an ok to the client. But I
> don't know which knobs to turn to accomplish this? or if it is even
> possible :)
>
> We are running cassandra 3.0.3, with 8Gb heap and a replication factor of
> 3.
>
> Thank you in advance!
>
> Yours sincerely,
>   Bo Madsen
>
-- 

Jens Rantil
Backend Developer @ Tink

Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
For urgent matters you can reach me at +46-708-84 18 32.

When are hints written?

[Bo Finnerup Madsen]

Hi,

We have a small 5 node cluster of m4.xlarge clients that receives writes
from ~20 clients. The clients will write as fast as they can, and the whole
process is limited by the write performance of the cassandra cluster.
After we have tweaked our schema to avoid large partitions, the load is
going ok and we don't see any warnings or errors in the cassandra logs. But
we do see quite a lot of hint handoff activity. During the load, the
cassandra nodes are quite loaded, with linux reporting a load as high as 20.

I have read the available documentation on how hints works, and to my
understanding hints should only be written if a node is down. But as far as
I can see, none of the nodes are marked as down during the load. So I
suspect I am missing something :)
We have configured the servers with write_request_timeout_in_ms: 12 and
the clients with a timeout of 13, but still get hints stored.

In our case, I would like for the cluster to wait for the write to be
persisted on the relevant nodes before returning an ok to the client. But I
don't know which knobs to turn to accomplish this? or if it is even
possible :)

We are running cassandra 3.0.3, with 8Gb heap and a replication factor of 3.

Thank you in advance!

Yours sincerely,
  Bo Madsen