Re: New token allocation and adding a new DC

[Dikang Gu]

I fixed the new allocation algorithm in non bootstrap case,
https://issues.apache.org/jira/browse/CASSANDRA-13080?filter=-2, the fix is
in 3.12+, but not in 3.0.


On Wed, Jan 24, 2018 at 9:32 AM, Oleksandr Shulgin <
oleksandr.shul...@zalando.de> wrote:

> On Thu, Jan 18, 2018 at 5:19 AM, kurt greaves 
> wrote:
>
>> Didn't know that about auto_bootstrap and the algorithm. We should
>> probably fix that. Can you create a JIRA for that issue?
>>
>
> Will do.
>
>
>> Workaround for #2 would be to truncate system.available_ranges after
>> "bootstrap".
>>
>
> Thanks, that seems to help.
>
> Initially we cannot login with cqlsh to run the truncate command on such a
> "bootstrapped" node.  But with the help of yet another workaround, namely
> pulling in the roles data by means of repairing system_auth keyspace only,
> it seems to be possible.  At least we see that netstats reports the ongoing
> streaming operations this time.
>
> --
> Alex
>
>


-- 
Dikang

RE: Cassandra Repair Duration.

[King, Marshall]

You should run “primary range” repairs on all nodes. That will go a lot faster 
than full repair. In C* 2x you can do this in parallel.. you can determine how 
many you can run at the same time.. basically how much pressure you can put on 
ur system.

Marshall

From: Karthick V [mailto:karthick...@gmail.com]
Sent: Wednesday, January 24, 2018 1:58 AM
To: user@cassandra.apache.org
Subject: Re: Cassandra Repair Duration.

Periodically I have been running Full repair process befor GC Grace period as 
mentioned in the best practices.Initially, all went well but as the data size 
increases Repair duration has increased drastically and we are also facing 
Query timeouts during that time and we have tried incremental repair facing 
some OOM issues.

After running a repair process for more than 80 Hours we have ended up with the 
question

why can't we run a repair process if and only if a Cassandra node got a 
downtime?

Say if there is no downtime during a GC grace period Do we still face 
Inconsistency among nodes? if yes, then doesn't Hinted Handoff handle those?

Cluster Info: Having two DataCenter with 8 machines each with a disk size of 
1TB, C* v_2.1.13  and having around 420GB data each.

On Wed, Jan 24, 2018 at 2:46 PM, Karthick V 
> wrote:
Hi,







The information contained in this electronic mail transmission may be 
privileged and confidential, and therefore, protected from disclosure. If you 
have received this communication in error, please notify us immediately by 
replying to this message and deleting the email and its attachments from all 
computers without copying or disclosing it.

Re: New token allocation and adding a new DC

[Oleksandr Shulgin]

On Thu, Jan 18, 2018 at 5:19 AM, kurt greaves  wrote:

> Didn't know that about auto_bootstrap and the algorithm. We should
> probably fix that. Can you create a JIRA for that issue?
>

Will do.


> Workaround for #2 would be to truncate system.available_ranges after
> "bootstrap".
>

Thanks, that seems to help.

Initially we cannot login with cqlsh to run the truncate command on such a
"bootstrapped" node.  But with the help of yet another workaround, namely
pulling in the roles data by means of repairing system_auth keyspace only,
it seems to be possible.  At least we see that netstats reports the ongoing
streaming operations this time.

--
Alex

Re: Cassandra Repair Duration.

[brian . spindler]

Hi Karthick, repairs can be tricky.  

You can (and probably should) run repairs as apart of routine maintenance.  And 
of course absolutely if you lose a node in a bad way.  If you decommission a 
node for example, no “extra” repair needed. 

If you are using TWCS you should probably not run repairs on those cf.

We have a combination of scripts and locks to run repairs across an 18 node 
cluster 1 node at a time, typically takes around 2-3days and so we run it once 
a week.  

The great folks at tlp have put together http://cassandra-reaper.io/ which 
makes managing repairs even easier and probably more performant since as I 
understand, it used range repairs. 

Good luck,
-B

> On Jan 24, 2018, at 4:57 AM, Karthick V  wrote:
> 
> Periodically I have been running Full repair process befor GC Grace period as 
> mentioned in the best practices.Initially, all went well but as the data size 
> increases Repair duration has increased drastically and we are also facing 
> Query timeouts during that time and we have tried incremental repair facing 
> some OOM issues.
> 
> After running a repair process for more than 80 Hours we have ended up with 
> the question
> 
> why can't we run a repair process if and only if a Cassandra node got a 
> downtime? 
> 
> Say if there is no downtime during a GC grace period Do we still face 
> Inconsistency among nodes? if yes, then doesn't Hinted Handoff handle those? 
> 
> Cluster Info: Having two DataCenter with 8 machines each with a disk size of 
> 1TB, C* v_2.1.13  and having around 420GB data each.
> 
>> On Wed, Jan 24, 2018 at 2:46 PM, Karthick V  wrote:
>> Hi,
>> 
>> 
>> 
>> 
>> 
>> 
> 

RequestResponseStage Threadpool

[Robert Emery]

Hiya,

We have our Cassandra 2.1 cluster being monitored by checking the JMX
on the org.apache.cassandra.metrics type=ThreadPools bean

We've got a threshold of 15 for warnings as per
https://blog.pythian.com/guide-to-cassandra-thread-pools/ however we
don't know if this is a sensible warning threshold as this does
occasionally warn at the moment however there is no real obvious cause
in the Cassandra logs.

So 2 questions:

- What does this thread-pool actually do? All the examples are so
abstract i.e. "executes callbacks"; my understanding is that if a
client queries the cluster and then never reads the result it would be
in the RequestResponseStage Pending until it times out?

- Is there any tracing or similar that can be used to inspect the
content of the thread-pool so we can see what it actually is that's
pending?

-- 
Robert Emery

-- 
 


Codeweavers 
January
 Newsletter 
  
l  *Codeweavers 2017 Review Infographic 
*


*What updates would you like to receive from Codeweavers via email? 
*



*Phone:* 0800 021 0888  * Email: *contac...@codeweavers.net
*Codeweavers Ltd* | Barn 4 | Dunston Business Village | Dunston | ST18 9AB
Registered in England and Wales No. 04092394 | VAT registration no. 974 
9705 63 

   
  [image: 
https://plus.google.com/b/105942302039373248738/+CodeweaversNet] 
  [image: 
https://twitter.com/CodeweaversTeam] 

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Cassandra Repair Duration.

[Karthick V]

Periodically I have been running Full repair process befor GC Grace period
as mentioned in the best practices.Initially, all went well but as the data
size increases Repair duration has increased drastically and we are also
facing Query timeouts during that time and we have tried incremental repair
facing some OOM issues.

After running a repair process for more than 80 Hours we have ended up with
the question

why can't we run a repair process if and only if a Cassandra node got a
downtime?

Say if there is no downtime during a GC grace period Do we still face
Inconsistency among nodes? if yes, then doesn't Hinted Handoff handle
those?

Cluster Info: Having two DataCenter with 8 machines each with a disk size
of 1TB, C* v_2.1.13  and having around 420GB data each.

On Wed, Jan 24, 2018 at 2:46 PM, Karthick V  wrote:

> Hi,
>
>
>
>
>
>
>

Upgrading sstables not using all available compaction slots on version 2.2

[Oleksandr Shulgin]

Hello,

In the process of upgrading our cluster from 2.1 to 2.2 we have triggered
the SSTable rewriting process like this:

$ nodetool upgradesstables -j 4  # concurrent_compactors=5

Then if we would immediately check the compactionstats, we see that 4
compactions of type 'Upgrade sstables' are running for the same
keyspace/table.  Fine.

What is surprising, though, is that when any of the tasks which are smaller
in size complete, no new tasks are added and we are left waiting for a
smaller number of long running tasks to complete, e.g:

pending tasks: 4
 idcompaction type   keyspace
   table   completed  totalunit   progress
   d535e962-00de-11e8-9c2c-ade7b0b922e3   Upgrade sstables   cel_data
 event_history_unbounded32.87 GB   84.12 GB   bytes 39.08%
   d535e960-00de-11e8-9c2c-ade7b0b922e3   Upgrade sstables   cel_data
 event_history_unbounded33.15 GB   84.79 GB   bytes 39.09%

Periodically, an ordinary compaction would be listed, but no new Upgrade
tasks appear.
Only after all of the "current" tasks are finished, the new 4 tasks are
scheduled and the process repeats.

We would expect that new tasks would be scheduled as soon as a compaction
slot is freed up, but this doesn't seem to happen.  Is this by design?  I
do not see any open issue about this in Jira.

Cheers,
-- 
Oleksandr "Alex" Shulgin | Database Engineer | Zalando SE | Tel: +49 176
127-59-707

Cassandra Repair Duration.

[Karthick V]

Hi,