RE: Not serving read requests while running nodetool repair

2017-10-18 Thread Steinmaurer, Thomas 
Hi Jeff,

thanks for the info. Hoped that nodetool provides an option for that. We will 
go with the temporary QUORUM approach for now.

Thanks again.

Thomas

From: Jeff Jirsa [mailto:jji...@gmail.com]
Sent: Mittwoch, 18. Oktober 2017 15:46
To: user@cassandra.apache.org
Subject: Re: Not serving read requests while running nodetool repair

You can accomplish this by manually tweaking the values in the dynamic snitch 
mbean so other nodes won’t select it for reads
--
Jeff Jirsa


On Oct 18, 2017, at 3:24 AM, Steinmaurer, Thomas 
<thomas.steinmau...@dynatrace.com<mailto:thomas.steinmau...@dynatrace.com>> 
wrote:
Hi Nicolas,

newly bootstrapping is not really an option, I think.

I hoped there might be a mode (perhaps there is, but haven’t found it yet) 
where a node might not actively participate in read requests but still writing 
newly arriving data during a repair or whatever maintenance task one thinks 
that some sort of write-only is appropriate.

Thanks,
Thomas

From: Nicolas Guyomar [mailto:nicolas.guyo...@gmail.com]
Sent: Mittwoch, 18. Oktober 2017 09:58
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Re: Not serving read requests while running nodetool repair

Hi Thomas,

AFAIK  temporarily reading at LOCAL_QUORUM/QUORUM until nodetool repair is 
finished is the way to go. You can still disable binary/thrift on the node to 
"protect" it from acting as a coordinator, and complete its repair quietly, but 
I'm not sure that would make such a huge difference in recovery time.

If you disable gossip the node will be see as down, thus disabling repair if 
I'm correct

If repair is taking too much time, or switching to Read Quorum not feasible 
within your application, is it OK for you to shutdown the node, wipe its data 
and launch a repair on it at an appropriate time windows ?



On 18 October 2017 at 08:04, Steinmaurer, Thomas 
<thomas.steinmau...@dynatrace.com<mailto:thomas.steinmau...@dynatrace.com>> 
wrote:
Hello,

due to performance/latency reasons, we are currently reading and writing time 
series data at consistency level  ONE/ANY.

In case of a node being down and recovering after the default hinted handoff 
window of 3 hrs, we may potentially read stale data from the recovering node. 
Of course, from an operational POV, we will trigger a nodetool repair after the 
recovered node has start up, but to my understanding, this still may cause 
reading stale data from this particular node until nodetool repair is finished, 
which may take several hours. Is this correct?

Is there a way (e.g. in newer releases 3.0/3.11) to tell a node not being part 
of read requests? Something like a counterpart of nodetool drain for writes?

Other options:


-  Disabling binary/thrift: Although the node won’t act as coordinator 
then for client requests, as it won’t be available from a client perspective, 
inter-node, the recovering node is being contacted by other nodes for serving 
requests, right?

-  Disabling gossip: Basically the node is marked as DOWN, so treated 
like it is not available, thus not an option?

-  Temporarily reading at LOCAL_QUORUM/QUORUM until nodetool repair is 
finished?

Did I miss something?

Thanks,
Thomas

The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freist<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313=gmail=g>ädterstra<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313=gmail=g>ße
 
313<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313=gmail=g>

The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 313
The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 313

Re: Not serving read requests while running nodetool repair

2017-10-18 Thread Jeff Jirsa 
You can accomplish this by manually tweaking the values in the dynamic snitch 
mbean so other nodes won’t select it for reads

-- 
Jeff Jirsa


> On Oct 18, 2017, at 3:24 AM, Steinmaurer, Thomas 
> <thomas.steinmau...@dynatrace.com> wrote:
> 
> Hi Nicolas,
>  
> newly bootstrapping is not really an option, I think.
>  
> I hoped there might be a mode (perhaps there is, but haven’t found it yet) 
> where a node might not actively participate in read requests but still 
> writing newly arriving data during a repair or whatever maintenance task one 
> thinks that some sort of write-only is appropriate.
>  
> Thanks,
> Thomas
>  
> From: Nicolas Guyomar [mailto:nicolas.guyo...@gmail.com] 
> Sent: Mittwoch, 18. Oktober 2017 09:58
> To: user@cassandra.apache.org
> Subject: Re: Not serving read requests while running nodetool repair
>  
> Hi Thomas,
>  
> AFAIK  temporarily reading at LOCAL_QUORUM/QUORUM until nodetool repair is 
> finished is the way to go. You can still disable binary/thrift on the node to 
> "protect" it from acting as a coordinator, and complete its repair quietly, 
> but I'm not sure that would make such a huge difference in recovery time.
>  
> If you disable gossip the node will be see as down, thus disabling repair if 
> I'm correct
>  
> If repair is taking too much time, or switching to Read Quorum not feasible 
> within your application, is it OK for you to shutdown the node, wipe its data 
> and launch a repair on it at an appropriate time windows ? 
>  
>  
>  
> On 18 October 2017 at 08:04, Steinmaurer, Thomas 
> <thomas.steinmau...@dynatrace.com> wrote:
> Hello,
>  
> due to performance/latency reasons, we are currently reading and writing time 
> series data at consistency level  ONE/ANY.
>  
> In case of a node being down and recovering after the default hinted handoff 
> window of 3 hrs, we may potentially read stale data from the recovering node. 
> Of course, from an operational POV, we will trigger a nodetool repair after 
> the recovered node has start up, but to my understanding, this still may 
> cause reading stale data from this particular node until nodetool repair is 
> finished, which may take several hours. Is this correct?
>  
> Is there a way (e.g. in newer releases 3.0/3.11) to tell a node not being 
> part of read requests? Something like a counterpart of nodetool drain for 
> writes?
>  
> Other options:
>  
> -  Disabling binary/thrift: Although the node won’t act as 
> coordinator then for client requests, as it won’t be available from a client 
> perspective, inter-node, the recovering node is being contacted by other 
> nodes for serving requests, right?
> 
> -  Disabling gossip: Basically the node is marked as DOWN, so treated 
> like it is not available, thus not an option?
> 
> -  Temporarily reading at LOCAL_QUORUM/QUORUM until nodetool repair 
> is finished?
> 
>  
> Did I miss something?
>  
> Thanks,
> Thomas
>  
> The contents of this e-mail are intended for the named addressee only. It 
> contains information that may be confidential. Unless you are the named 
> addressee or an authorized designee, you may not copy or use it, or disclose 
> it to anyone else. If you received it in error please notify us immediately 
> and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) 
> is a company registered in Linz whose registered office is at 4040 Linz, 
> Austria, Freistädterstraße 313
>  
> The contents of this e-mail are intended for the named addressee only. It 
> contains information that may be confidential. Unless you are the named 
> addressee or an authorized designee, you may not copy or use it, or disclose 
> it to anyone else. If you received it in error please notify us immediately 
> and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) 
> is a company registered in Linz whose registered office is at 4040 Linz, 
> Austria, Freistädterstraße 313

RE: Not serving read requests while running nodetool repair

2017-10-18 Thread Steinmaurer, Thomas 
Hi Nicolas,

newly bootstrapping is not really an option, I think.

I hoped there might be a mode (perhaps there is, but haven’t found it yet) 
where a node might not actively participate in read requests but still writing 
newly arriving data during a repair or whatever maintenance task one thinks 
that some sort of write-only is appropriate.

Thanks,
Thomas

From: Nicolas Guyomar [mailto:nicolas.guyo...@gmail.com]
Sent: Mittwoch, 18. Oktober 2017 09:58
To: user@cassandra.apache.org
Subject: Re: Not serving read requests while running nodetool repair

Hi Thomas,

AFAIK  temporarily reading at LOCAL_QUORUM/QUORUM until nodetool repair is 
finished is the way to go. You can still disable binary/thrift on the node to 
"protect" it from acting as a coordinator, and complete its repair quietly, but 
I'm not sure that would make such a huge difference in recovery time.

If you disable gossip the node will be see as down, thus disabling repair if 
I'm correct

If repair is taking too much time, or switching to Read Quorum not feasible 
within your application, is it OK for you to shutdown the node, wipe its data 
and launch a repair on it at an appropriate time windows ?



On 18 October 2017 at 08:04, Steinmaurer, Thomas 
<thomas.steinmau...@dynatrace.com<mailto:thomas.steinmau...@dynatrace.com>> 
wrote:
Hello,

due to performance/latency reasons, we are currently reading and writing time 
series data at consistency level  ONE/ANY.

In case of a node being down and recovering after the default hinted handoff 
window of 3 hrs, we may potentially read stale data from the recovering node. 
Of course, from an operational POV, we will trigger a nodetool repair after the 
recovered node has start up, but to my understanding, this still may cause 
reading stale data from this particular node until nodetool repair is finished, 
which may take several hours. Is this correct?

Is there a way (e.g. in newer releases 3.0/3.11) to tell a node not being part 
of read requests? Something like a counterpart of nodetool drain for writes?

Other options:


-  Disabling binary/thrift: Although the node won’t act as coordinator 
then for client requests, as it won’t be available from a client perspective, 
inter-node, the recovering node is being contacted by other nodes for serving 
requests, right?

-  Disabling gossip: Basically the node is marked as DOWN, so treated 
like it is not available, thus not an option?

-  Temporarily reading at LOCAL_QUORUM/QUORUM until nodetool repair is 
finished?

Did I miss something?

Thanks,
Thomas

The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freist<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313=gmail=g>ädterstra<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313=gmail=g>ße
 
313<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313=gmail=g>

The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 313

Re: Not serving read requests while running nodetool repair

2017-10-18 Thread Nicolas Guyomar 
Hi Thomas,

AFAIK  temporarily reading at LOCAL_QUORUM/QUORUM until nodetool repair is
finished is the way to go. You can still disable binary/thrift on the node
to "protect" it from acting as a coordinator, and complete its repair
quietly, but I'm not sure that would make such a huge difference in
recovery time.

If you disable gossip the node will be see as down, thus disabling repair
if I'm correct

If repair is taking too much time, or switching to Read Quorum not feasible
within your application, is it OK for you to shutdown the node, wipe its
data and launch a repair on it at an appropriate time windows ?



On 18 October 2017 at 08:04, Steinmaurer, Thomas <
thomas.steinmau...@dynatrace.com> wrote:

> Hello,
>
>
>
> due to performance/latency reasons, we are currently reading and writing
> time series data at consistency level  ONE/ANY.
>
>
>
> In case of a node being down and recovering after the default hinted
> handoff window of 3 hrs, we may potentially read stale data from the
> recovering node. Of course, from an operational POV, we will trigger a
> nodetool repair after the recovered node has start up, but to my
> understanding, this still may cause reading stale data from this particular
> node until nodetool repair is finished, which may take several hours. Is
> this correct?
>
>
>
> Is there a way (e.g. in newer releases 3.0/3.11) to tell a node not being
> part of read requests? Something like a counterpart of nodetool drain for
> writes?
>
>
>
> Other options:
>
>
>
> -  Disabling binary/thrift: Although the node won’t act as
> coordinator then for client requests, as it won’t be available from a
> client perspective, inter-node, the recovering node is being contacted by
> other nodes for serving requests, right?
>
> -  Disabling gossip: Basically the node is marked as DOWN, so
> treated like it is not available, thus not an option?
>
> -  Temporarily reading at LOCAL_QUORUM/QUORUM until nodetool
> repair is finished?
>
>
>
> Did I miss something?
>
>
>
> Thanks,
>
> Thomas
>
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freist
> 
> ädterstra
> 
> ße 313
> 
>

Not serving read requests while running nodetool repair

2017-10-18 Thread Steinmaurer, Thomas 
Hello,

due to performance/latency reasons, we are currently reading and writing time 
series data at consistency level  ONE/ANY.

In case of a node being down and recovering after the default hinted handoff 
window of 3 hrs, we may potentially read stale data from the recovering node. 
Of course, from an operational POV, we will trigger a nodetool repair after the 
recovered node has start up, but to my understanding, this still may cause 
reading stale data from this particular node until nodetool repair is finished, 
which may take several hours. Is this correct?

Is there a way (e.g. in newer releases 3.0/3.11) to tell a node not being part 
of read requests? Something like a counterpart of nodetool drain for writes?

Other options:


-  Disabling binary/thrift: Although the node won't act as coordinator 
then for client requests, as it won't be available from a client perspective, 
inter-node, the recovering node is being contacted by other nodes for serving 
requests, right?

-  Disabling gossip: Basically the node is marked as DOWN, so treated 
like it is not available, thus not an option?

-  Temporarily reading at LOCAL_QUORUM/QUORUM until nodetool repair is 
finished?

Did I miss something?

Thanks,
Thomas

The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freist?dterstra?e 313