Re: Cluster CPU usage limit

[Elliott Sims]

It's interesting and a bit surprising that 256 write threads isn't enough.
Even with a lot of cores, I'd expect you to be able to saturate CPU with
that many threads.  I'd make sure you don't have other bottlenecks, like
GC, IOPs, network, or "microbursts" where your load is actually fluctuating
between 20-100% CPU.
Admittedly, I actually did get best results with 256 threads (and haven't
tested higher, but lower is definitely not enough), but every advice I've
seen is for a lower write thread count being optimal for most cases.

On Thu, Sep 6, 2018 at 5:51 AM, onmstester onmstester 
wrote:

> IMHO, Cassandra write is more of a CPU bound task, so while determining
> cluster write throughput, what CPU usage percent (avg among all cluster
> nodes) should be determined as limit?
> Rephrase: what's the normal CPU usage in Cassandra cluster (while no
> compaction, streaming or heavy-read running) ?
> For a cluster with 10 nodes, i got 700K write per seconds for my data
> model, average cpu load is about 40%, i'm going to increase number of
> native threads (now is 256) and native queue (1024) to increase throughput
> (and CPU usage subsequently).
>
> Sent using Zoho Mail 
>
>
>

impact/incompatibility of patch backport on Cassandra 3.11.2

[Ahmed Eljami]

Hi,

We are testing Cassandra 3.11.2  and we sawed that it contains a critcal
bug wich was fixed in 3.11.3 (
https://issues.apache.org/jira/browse/CASSANDRA-13929).

After about 1 months of testing, we haven't encountered this bug in our
environnement, but to be sure before going in production, we would like to
know if it's possible to backport this patch in version 3.11.2 ?

Do you think that this patch could be backported wihtout
impact/incompatibility on 3.11.2 ? Or it be  safer to migrate to Cassandra
3.11.3 ?

Our fears that 3.11.3 also not stable due to this bug:
https://issues.apache.org/jira/browse/CASSANDRA-14672

Best regards.

Re: Refering to non-existing node

[Vlad]

Hi, this node isn't in system.peers on both nodes. 

On Thursday, September 6, 2018 10:02 PM, Michael Shuler 
 wrote:
 

 On 09/06/2018 01:48 PM, Vlad wrote:
> Hi,
> 3 node cluster, Cassandra 3.9, GossipingPropertyFileSnitch, one DC
> 
> I removed dead node with `nodetool assassinate`. It was also seed node,
> so I removed it from seeds list on two other nodes and restarted them.
> 
> But I still see in log
> `DEBUG [GossipTasks:1] 2018-09-06 18:32:05,149 Gossiper.java:337 -
> Convicting /10.100.129.116 with status LEFT - alive false`
> 
> 
> What does this log mean? How can I get rid of this?

https://twitter.com/faltering/status/559845791741657088

If you *only* used assassinate, without moving through the removal
steps, then the node is still in the system.peers (I think) table on
some or all of the nodes.

-- 
Michael

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org



   

Re: URGENT: disable reads from node

[Vlad]

Hi,
this node isn't in system.peers on both nodes. 

On Wednesday, August 29, 2018 4:22 PM, Vlad  
wrote:
 

 Hi,
>You'll need to disable the native transportWell, this is what I did already, 
>it seems repair is running

I'm not sure whether repair will finish within 3 hours, but I can run it again 
(as it's incremental repair by default, right?)


I'm not sure about RF=3 and QUORUM reads because of load/disk space constrains 
we have, but we'll definitely consider this.

Thanks to all for help!
 

On Wednesday, August 29, 2018 4:13 PM, Alexander Dejanovski 
 wrote:
 

 Kurt is right. 
So here are the options I can think of : - use the join_ring false technique 
and rely on hints. You'll need to disable the native transport on the node as 
well to prevent direct connections to be made to it. Hopefully, you can run 
repair in less than 3 hours which is the hint window (hints will be collected 
while the node hasn't joined the ring). Otherwise you'll have more consistency 
issues after the node joins the ring again. Maybe incremental repair could help 
fixing this quickly afterwards if you've been running full repairs that 
involved anticompaction (if you're running at least Cassandra 2.2).- Fully 
re-bootstrap the node by replacing itself, using the replace_address_first_boot 
technique (but since you have RF=2, that would most probably mean some data 
loss since you read/write at ONE)- Try to cheat the dynamic snitch to take the 
node out of reads. You would then have the node join the ring normally, disable 
native transport and raise Severity (in 
org.apache.cassandra.db:type=DynamicEndpointSnitch) to something like 50 so the 
node won't be selected by the dynamic snitch. I guess the value will reset 
itself over time so you may need to set it to 50 on a regular basis while 
repair is happening.
I would then strongly consider moving to RF=3 because RF=2 will lead you to 
this type of situation again in the future and does not allow quorum reads with 
fault tolerance. Good luck,
On Wed, Aug 29, 2018 at 1:56 PM Vlad  wrote:

I restarted with cassandra.join_ring=falsenodetool status on other nodes shows 
this node as DN, while it see itself as UN.


>I'd say best to just query at QUORUM until you can finish repairs.We have RH 
>2, so I guess QUORUM queries will fail. Also different application should be 
>changed for this. 

On Wednesday, August 29, 2018 2:41 PM, kurt greaves  
wrote:
 

 Note that you'll miss incoming writes if you do that, so you'll be 
inconsistent even after the repair. I'd say best to just query at QUORUM until 
you can finish repairs.
On 29 August 2018 at 21:22, Alexander Dejanovski  wrote:

Hi Vlad, you must restart the node but first disable joining the cluster, as 
described in the second part of this blog post : 
http://thelastpickle.com/blog/ 2018/08/02/Re-Bootstrapping- 
Without-Bootstrapping.html
Once repaired, you'll have to run "nodetool join" to start serving reads.

Le mer. 29 août 2018 à 12:40, Vlad  a écrit :

Will it help to set read_repair_chance to 1 (compaction is 
SizeTieredCompactionStrategy)? 

On Wednesday, August 29, 2018 1:34 PM, Vlad  
wrote:
 

 Hi,
quite urgent questions:due to disk and C* start problem we were forced to 
delete commit logs from one of nodes.
Now repair is running, but meanwhile some reads bring no data (RF=2)

Can this node be excluded from reads queries? And that  all reads will be 
redirected to other node in the ring?

Thanks to All for help.


   
-- 
-Alexander DejanovskiFrance@alexanderdeja
ConsultantApache Cassandra Consultinghttp://www.thelastpickle.com



   
-- 
-Alexander DejanovskiFrance@alexanderdeja
ConsultantApache Cassandra Consultinghttp://www.thelastpickle.com

   

   

Re: Refering to non-existing node

[Michael Shuler]

On 09/06/2018 01:48 PM, Vlad wrote:
> Hi,
> 3 node cluster, Cassandra 3.9, GossipingPropertyFileSnitch, one DC
> 
> I removed dead node with `nodetool assassinate`. It was also seed node,
> so I removed it from seeds list on two other nodes and restarted them.
> 
> But I still see in log
> `DEBUG [GossipTasks:1] 2018-09-06 18:32:05,149 Gossiper.java:337 -
> Convicting /10.100.129.116 with status LEFT - alive false`
> 
> 
> What does this log mean? How can I get rid of this?

https://twitter.com/faltering/status/559845791741657088

If you *only* used assassinate, without moving through the removal
steps, then the node is still in the system.peers (I think) table on
some or all of the nodes.

-- 
Michael

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Refering to non-existing node

[Vlad]

Hi,3 node cluster, Cassandra 3.9, GossipingPropertyFileSnitch, one DC

I removed dead node with `nodetool assassinate`. It was also seed node, so I 
removed it from seeds list on two other nodes and restarted them.
But I still see in log 
`DEBUG [GossipTasks:1] 2018-09-06 18:32:05,149 Gossiper.java:337 - Convicting 
/10.100.129.116 with status LEFT - alive false`

What does this log mean? How can I get rid of this?
Thanks.

Stress Test

[rajasekhar kommineni]

Hello Folks,

Does any body refer good documentation on Cassandra stress test. 

I have below questions.

1) Which server is good to start the test, Cassandra server or Application 
server.
2) I am using Datastax Java driver, is any good documentation for stress test 
specific to this driver.
3) How to analyze the stress test output.

Thanks,


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Using CDC Feature to Stream C* to Kafka (Design Proposal)

[Joy Gao]

Hi all,

We are fairly new to Cassandra. We began looking into the CDC feature
introduced in 3.0. As we spent more time looking into it, the complexity
began to add up (i.e. duplicated mutation based on RF, out of order
mutation, mutation does not contain full row of data, etc). These
limitations have already been mentioned in the discussion thread in
CASSANDRA-8844, so we understand the design decisions around this. However,
we do not want to push solving this complexity to every downstream
consumers, where they each have to handle
deduping/ordering/read-before-write to get full row; instead we want to
solve them earlier in the pipeline, so the change message are
deduped/ordered/complete by the time they arrive in Kafka. Dedupe can be
solved with a cache, and ordering can be solved since mutations have
timestamps, but the one we have the most trouble with is not having the
full row of data.

We had a couple discussions with some folks in other companies who are
working on applying CDC feature for their real-time data pipelines. On a
high-level, the common feedback we gathered is to use a stateful processing
approach to maintain a separate db which mutations are applied to, which
then allows them to construct the "before" and "after" data without having
to query the original Cassandra db on each mutation. The downside of this
is the operational overhead of having to maintain this intermediary db for
CDC.

We have an unconventional idea (inspired by DSE Advanced Replication) that
eliminates some of the operational overhead, but with tradeoff of
increasing code complexity and memory pressure. The high level idea is a
stateless processing approach where we have a process in each C* node that
parse mutation from CDC logs and query local node to get the "after" data,
which avoid network hops and thus making reading full-row of data more
efficient. We essentially treat the mutations in CDC log as change
notifications. To solve dedupe/ordering, only the primary node for each
token range will send the data to Kafka, but data are reconciled with peer
nodes to prevent data loss.

We have a* WIP design doc
* that goes
over this idea in details.

We haven't sort out all the edge cases yet, but would love to get some
feedback from the community on the general feasibility of this approach.
Any ideas/concerns/questions would be helpful to us. Thanks!

Joy

RES: Large sstables

[Versátil]

Remove my email please

 

De: Vitali Dyachuk [mailto:vdjat...@gmail.com] 
Enviada em: quinta-feira, 6 de setembro de 2018 08:00
Para: user@cassandra.apache.org
Assunto: Re: Large sstables

 

What i have done is:
1) added more disks, so the compaction will carry on
2) when i've switched to LCS from STCS the STCS queues for the processing big 
sstables have remained, so i've stopped these queues with nodetool stop -id 
queue_id
and LCS compaction has started to process sstables , i'm using 3.0.17 C* 
with RF3

However the question remains if i use sstablesplit on 200Gb sstables to split 
it to 200Mb files, will it help the LCS compaction?
Will LCS just take some data from that big sstable and try to merge with other 
sstable on L0 adn other levels so i just have to wait until the LCS compaction 
will finish?

 

 

On Sun, Sep 2, 2018 at 9:55 AM shalom sagges mailto:shalomsag...@gmail.com> > wrote:

If there are a lot of droppable tombstones, you could also run User Defined 
Compaction on that (and on other) SSTable(s). 

 

This blog post explains it well:

http://thelastpickle.com/blog/2016/10/18/user-defined-compaction.html

 

On Fri, Aug 31, 2018 at 12:04 AM Mohamadreza Rostami 
mailto:mohamadrezarosta...@gmail.com> > wrote:

Hi,Dear Vitali
The best option for you is migrating data to the new table and change portion 
key patterns to a better distribution of data and you sstables become smaller 
but if your data already have good distribution and your data is really big you 
must add new server to your datacenter, if you change compassion strategy it 
has some risk.

> On Shahrivar 8, 1397 AP, at 19:54, Jeff Jirsa   > wrote:
> 
> Either of those are options, but there’s also sstablesplit to break it up a 
> bit
> 
> Switching to LCS can be a problem depending on how many sstables /overlaps 
> you have 
> 
> -- 
> Jeff Jirsa
> 
> 
>> On Aug 30, 2018, at 8:05 AM, Vitali Dyachuk >  > wrote:
>> 
>> Hi,
>> Some of the sstables got too big 100gb and more so they are not compactiong 
>> any more so some of the disks are running out of space. I'm running C* 
>> 3.0.17, RF3 with 10 disks/jbod with STCS.
>> What are my options? Completely delete all data on this node and rejoin it 
>> to the cluster, change CS to LCS then run repair?
>> Vitali.
>> 
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
>  
> For additional commands, e-mail: user-h...@cassandra.apache.org 
>  
> 


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
 
For additional commands, e-mail: user-h...@cassandra.apache.org 
 

Re: Bootstrap streaming issues

[Jai Bheemsen Rao Dhanwada]

Here is the stacktrace from the failure, it looks like it's trying to
gather all the columfamily metrics and going OOM. Is this just for the JMX
metrics?

https://github.com/apache/cassandra/blob/cassandra-2.1.16/src/java/org/apache/cassandra/metrics/ColumnFamilyMetrics.java

ERROR [MessagingService-Incoming-/10.133.33.57] 2018-09-06 15:43:19,280
CassandraDaemon.java:231 - Exception in thread
Thread[MessagingService-Incoming-/x.x.x.x,5,main]
java.lang.OutOfMemoryError: Java heap space
at java.io.DataInputStream.(DataInputStream.java:58)
~[na:1.8.0_151]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:139)
~[apache-cassandra-2.1.16.jar:2.1.16]
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:88)
~[apache-cassandra-2.1.16.jar:2.1.16]
ERROR [InternalResponseStage:1] 2018-09-06 15:43:19,281
CassandraDaemon.java:231 - Exception in thread
Thread[InternalResponseStage:1,5,main]
java.lang.OutOfMemoryError: Java heap space
at
org.apache.cassandra.metrics.ColumnFamilyMetrics$AllColumnFamilyMetricNameFactory.createMetricName(
*ColumnFamilyMetrics.java:784*) ~[apache-cassandra-2.1.16.jar:2.1.16]
at
org.apache.cassandra.metrics.ColumnFamilyMetrics.createColumnFamilyHistogram(ColumnFamilyMetrics.java:716)
~[apache-cassandra-2.1.16.jar:2.1.16]
at
org.apache.cassandra.metrics.ColumnFamilyMetrics.(ColumnFamilyMetrics.java:597)
~[apache-cassandra-2.1.16.jar:2.1.16]
at
org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:361)
~[apache-cassandra-2.1.16.jar:2.1.16]
at
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:527)
~[apache-cassandra-2.1.16.jar:2.1.16]
at
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:498)
~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:335)
~[apache-cassandra-2.1.16.jar:2.1.16]
at
org.apache.cassandra.db.DefsTables.addColumnFamily(DefsTables.java:385)
~[apache-cassandra-2.1.16.jar:2.1.16]
at
org.apache.cassandra.db.DefsTables.mergeColumnFamilies(DefsTables.java:293)
~[apache-cassandra-2.1.16.jar:2.1.16]
at
org.apache.cassandra.db.DefsTables.mergeSchemaInternal(DefsTables.java:194)
~[apache-cassandra-2.1.16.jar:2.1.16]
at
org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java:166)
~[apache-cassandra-2.1.16.jar:2.1.16]
at
org.apache.cassandra.service.MigrationTask$1.response(MigrationTask.java:75)
~[apache-cassandra-2.1.16.jar:2.1.16]
at
org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:54)
~[apache-cassandra-2.1.16.jar:2.1.16]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
~[apache-cassandra-2.1.16.jar:2.1.16]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[na:1.8.0_151]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[na:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_151]

On Thu, Aug 30, 2018 at 12:51 PM Jai Bheemsen Rao Dhanwada <
jaibheem...@gmail.com> wrote:

> thank you
>
> On Thu, Aug 30, 2018 at 11:58 AM Jeff Jirsa  wrote:
>
>> This is the closest JIRA that comes to mind (from memory, I didn't
>> search, there may be others):
>> https://issues.apache.org/jira/browse/CASSANDRA-8150
>>
>> The best blog that's all in one place on tuning GC in cassandra is
>> actually Amy's 2.1 tuning guide:
>> https://tobert.github.io/pages/als-cassandra-21-tuning-guide.html - it's
>> somewhat out of date as it's for 2.1, but since that's what you're running,
>> that works out in your favor.
>>
>>
>>
>>
>>
>> On Thu, Aug 30, 2018 at 10:53 AM Jai Bheemsen Rao Dhanwada <
>> jaibheem...@gmail.com> wrote:
>>
>>> Hi Jeff,
>>>
>>> Is there any JIRA that talks about increasing the HEAP will help?
>>> Also, any other alternatives than increasing the HEAP Size? last time
>>> when I tried increasing the heap, longer GC Pauses caused more damage in
>>> terms of latencies while gc pause.
>>>
>>> On Wed, Aug 29, 2018 at 11:07 PM Jai Bheemsen Rao Dhanwada <
>>> jaibheem...@gmail.com> wrote:
>>>
 okay, thank you

 On Wed, Aug 29, 2018 at 11:04 PM Jeff Jirsa  wrote:

> You’re seeing an OOM, not a socket error / timeout.
>
> --
> Jeff Jirsa
>
>
> On Aug 29, 2018, at 10:56 PM, Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
>
> Jeff,
>
> any idea if this is somehow related to :
> https://issues.apache.org/jira/browse/CASSANDRA-11840?
> does increasing the value of streaming_socket_timeout_in_ms to a
> higher value helps?
>
> On Wed, Aug 29, 2018 at 10:52 PM Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
>
>> I have 72 nodes in the cluster, across 8 datacenters.. the moment I
>> try to 

Mutation of bytes is too large for the maxiumum size of

[Saladi Naidu]

We are receiving following error
9140-    at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) 
[apache-cassandra-3.0.10.jar:3.0.10]
9141-    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]

9142:WARN  [SharedPool-Worker-1] 2018-09-06 14:29:46,071 
AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread 
Thread[SharedPool-Worker-1,5,main]: {}

9143-java.lang.IllegalArgumentException: Mutation of 16777251 bytes is too 
large for the maximum size of 16777216

9144-    at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:256) 
~[apache-cassandra-3.0.10.jar:3.0.10]


I found following link that explained the cause 
By design intent the maximum allowed segment size is 50% of the configured 
commit_log_segment_size_in_mb. This is so Cassandra avoids writing segments 
with large amounts of empty space.To elaborate; up to two 32MB segments will 
fit into 64MB, however 40MB will only fit once leaving a larger amount of 
unused space. 
"I would like to find what table/column family this write/mutation is causing 
this error so that I can reach out to right application team, log does not 
provide any details regarding the mutation at all, is there a way to find that 
out
Mutation of bytes is too large for the maxiumum size of
  
|  
|   
|   
|   ||

   |

  |
|  
|   |  
Mutation of bytes is too large for the maxiumum size of
 Summary Apache Cassandra will discard mutations larger than a predetermined 
size. This note addresses why this h...  |   |

  |

  |

 


 Naidu Saladi 

Re: Regarding migrating data from Oracle to Cassandra.migrate data from Oracle to Cassandra.

[sha p]

Thank you Jeff.
While migration , how can test/validate against Cassandra particularly i am
going for "parallel run". Any sample strategy?


Regards,
Shyam

On Thu, 6 Sep 2018, 09:48 Jeff Jirsa,  wrote:

> It very much depends on your application. You'll PROBABLY want to double
> write for some period of time -  start writes to both Cassandra and Oracle,
> and then ensure they're both in sync. Once you're sure they're both in
> sync, move your reads from Oracle to Cassandra.
>
>
>
> On Wed, Sep 5, 2018 at 8:58 PM sha p  wrote:
>
>> Hi all,
>> Sir how should I keep track of the data which is moved to Cassandra ,
>> what are the best strategies available?
>>
>> Regards,
>> Shyam
>>
>> On Wed, 5 Sep 2018, 18:51 sha p,  wrote:
>>
>>>
>>> Hi all ,
 Me new to Cassandra , i was asked to migrate data from Oracle to
 Cassandra.
 Please help me giving your valuable guidance.
 1) Can it be done using open source Cassandra.
 2) Where should I start data model from?
 3) I should use java, what kind of  jar/libs/tools I need use ?
 4) How I decide the size of cluster , please provide some sample
 guidelines.
 5) this should be in production , so what kind of things i should take
 care for better support or debugging tomorrow?
 6) Please provide some good books /links which can help me in this task.


 Thanks in advance.
 Highly appreciated your every amal help.

 Regards,
 Shyam

>>>

Re: Large sstables

[Vitali Dyachuk]

What i have done is:
1) added more disks, so the compaction will carry on
2) when i've switched to LCS from STCS the STCS queues for the processing
big sstables have remained, so i've stopped these queues with nodetool stop
-id queue_id
and LCS compaction has started to process sstables , i'm using 3.0.17
C* with RF3

However the question remains if i use sstablesplit on 200Gb sstables to
split it to 200Mb files, will it help the LCS compaction?
Will LCS just take some data from that big sstable and try to merge with
other sstable on L0 adn other levels so i just have to wait until the LCS
compaction will finish?


On Sun, Sep 2, 2018 at 9:55 AM shalom sagges  wrote:

> If there are a lot of droppable tombstones, you could also run User
> Defined Compaction on that (and on other) SSTable(s).
>
> This blog post explains it well:
> http://thelastpickle.com/blog/2016/10/18/user-defined-compaction.html
>
> On Fri, Aug 31, 2018 at 12:04 AM Mohamadreza Rostami <
> mohamadrezarosta...@gmail.com> wrote:
>
>> Hi,Dear Vitali
>> The best option for you is migrating data to the new table and change
>> portion key patterns to a better distribution of data and you sstables
>> become smaller but if your data already have good distribution and your
>> data is really big you must add new server to your datacenter, if you
>> change compassion strategy it has some risk.
>>
>> > On Shahrivar 8, 1397 AP, at 19:54, Jeff Jirsa  wrote:
>> >
>> > Either of those are options, but there’s also sstablesplit to break it
>> up a bit
>> >
>> > Switching to LCS can be a problem depending on how many sstables
>> /overlaps you have
>> >
>> > --
>> > Jeff Jirsa
>> >
>> >
>> >> On Aug 30, 2018, at 8:05 AM, Vitali Dyachuk 
>> wrote:
>> >>
>> >> Hi,
>> >> Some of the sstables got too big 100gb and more so they are not
>> compactiong any more so some of the disks are running out of space. I'm
>> running C* 3.0.17, RF3 with 10 disks/jbod with STCS.
>> >> What are my options? Completely delete all data on this node and
>> rejoin it to the cluster, change CS to LCS then run repair?
>> >> Vitali.
>> >>
>> >
>> > -
>> > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> > For additional commands, e-mail: user-h...@cassandra.apache.org
>> >
>>
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>

RE: nodetool cleanup - compaction remaining time

[Steinmaurer, Thomas]

Alain,

compaction throughput is set to 32.

Regards,
Thomas

From: Alain RODRIGUEZ 
Sent: Donnerstag, 06. September 2018 11:50
To: user cassandra.apache.org 
Subject: Re: nodetool cleanup - compaction remaining time

Hello Thomas.

Be aware that this behavior happens when the compaction throughput is set to 0 
(unthrottled/unlimited). I believe the estimate uses the speed limit for 
calculation (which is often very much wrong anyway).

I just meant to say, you might want to make sure that it's due to cleanup type 
of compaction indeed and not due to some changes you could have made in the 
compaction throughput threshold.

C*heers,
---
Alain Rodriguez - @arodream - 
al...@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

Le jeu. 6 sept. 2018 à 06:51, Jeff Jirsa 
mailto:jji...@gmail.com>> a écrit :
Probably worth a JIRA (especially if you can repro in 3.0 or higher, since 2.1 
is critical fixes only)

On Wed, Sep 5, 2018 at 10:46 PM Steinmaurer, Thomas 
mailto:thomas.steinmau...@dynatrace.com>> 
wrote:
Hello,

is it a known issue / limitation that cleanup compactions aren’t counted in the 
compaction remaining time?

nodetool compactionstats -H

pending tasks: 1
   compaction type   keyspace   table   completed totalunit   
progress
   CleanupXXX YYY   908.16 GB   1.13 TB   bytes 
78.63%
Active compaction remaining time :   0h00m00s


This is with 2.1.18.


Thanks,
Thomas

The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 313
The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 313

Re: nodetool cleanup - compaction remaining time

[Alain RODRIGUEZ]

>
> As far as I can remember, if you have unthrottled compaction, then the
> message is different: it says "n/a".


Ah right!

I am now completely convinced this needs a JIRA as well (indeed, if it's
not fixed in C*3+, as Jeff mentioned).
Thanks for the feedback Alex.

Le jeu. 6 sept. 2018 à 11:06, Oleksandr Shulgin <
oleksandr.shul...@zalando.de> a écrit :

> On Thu, Sep 6, 2018 at 11:50 AM Alain RODRIGUEZ 
> wrote:
>
>>
>> Be aware that this behavior happens when the compaction throughput is set
>> to *0 *(unthrottled/unlimited). I believe the estimate uses the speed
>> limit for calculation (which is often very much wrong anyway).
>>
>
> As far as I can remember, if you have unthrottled compaction, then the
> message is different: it says "n/a".  The all zeroes you usually see when
> you only have Validation compactions, and apparently Cleanup work the same
> way, at least in the 2.1 version.
>
>
> https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/tools/nodetool/CompactionStats.java#L102
>
> Actually, if you look closely, it's obvious that only real Compaction
> tasks count toward remainingBytes, so all Validation/Clenaup/Upgrade don't
> count.  The reason must be that only actual compaction is affected by the
> throttling parameter.  Is that assumption correct?
>
> In any case it would make more sense to measure the actual throughput to
> provide an accurate estimate.  Not sure if there is JIRA issue for that
> already.
>
> --
> Alex
>
>

Re: nodetool cleanup - compaction remaining time

[Oleksandr Shulgin]

On Thu, Sep 6, 2018 at 11:50 AM Alain RODRIGUEZ  wrote:

>
> Be aware that this behavior happens when the compaction throughput is set
> to *0 *(unthrottled/unlimited). I believe the estimate uses the speed
> limit for calculation (which is often very much wrong anyway).
>

As far as I can remember, if you have unthrottled compaction, then the
message is different: it says "n/a".  The all zeroes you usually see when
you only have Validation compactions, and apparently Cleanup work the same
way, at least in the 2.1 version.

https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/tools/nodetool/CompactionStats.java#L102

Actually, if you look closely, it's obvious that only real Compaction tasks
count toward remainingBytes, so all Validation/Clenaup/Upgrade don't
count.  The reason must be that only actual compaction is affected by the
throttling parameter.  Is that assumption correct?

In any case it would make more sense to measure the actual throughput to
provide an accurate estimate.  Not sure if there is JIRA issue for that
already.

--
Alex

Cluster CPU usage limit

[onmstester onmstester]

IMHO, Cassandra write is more of a CPU bound task, so while determining cluster 
write throughput, what CPU usage percent (avg among all cluster nodes) should 
be determined as limit?   Rephrase: what's the normal CPU usage in Cassandra 
cluster (while no compaction, streaming or heavy-read running) ? For a cluster 
with 10 nodes, i got 700K write per seconds for my data model, average cpu load 
is about 40%, i'm going to increase number of native threads (now is 256) and 
native queue (1024) to increase throughput (and CPU usage subsequently). Sent 
using Zoho Mail

Re: nodetool cleanup - compaction remaining time

[Alain RODRIGUEZ]

Hello Thomas.

Be aware that this behavior happens when the compaction throughput is set
to *0 *(unthrottled/unlimited). I believe the estimate uses the speed limit
for calculation (which is often very much wrong anyway).

I just meant to say, you might want to make sure that it's due to cleanup
type of compaction indeed and not due to some changes you could have made
in the compaction throughput threshold.

C*heers,
---
Alain Rodriguez - @arodream - al...@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

Le jeu. 6 sept. 2018 à 06:51, Jeff Jirsa  a écrit :

> Probably worth a JIRA (especially if you can repro in 3.0 or higher, since
> 2.1 is critical fixes only)
>
> On Wed, Sep 5, 2018 at 10:46 PM Steinmaurer, Thomas <
> thomas.steinmau...@dynatrace.com> wrote:
>
>> Hello,
>>
>>
>>
>> is it a known issue / limitation that cleanup compactions aren’t counted
>> in the compaction remaining time?
>>
>>
>>
>> nodetool compactionstats -H
>>
>>
>>
>> pending tasks: 1
>>
>>compaction type   keyspace   table   completed total
>> unit   progress
>>
>>CleanupXXX YYY   908.16 GB   1.13 TB
>> bytes 78.63%
>>
>> Active compaction remaining time :   0h00m00s
>>
>>
>>
>>
>>
>> This is with 2.1.18.
>>
>>
>>
>>
>>
>> Thanks,
>>
>> Thomas
>>
>>
>> The contents of this e-mail are intended for the named addressee only. It
>> contains information that may be confidential. Unless you are the named
>> addressee or an authorized designee, you may not copy or use it, or
>> disclose it to anyone else. If you received it in error please notify us
>> immediately and then destroy it. Dynatrace Austria GmbH (registration
>> number FN 91482h) is a company registered in Linz whose registered office
>> is at 4040 Linz, Austria, Freistädterstraße 313
>>
>

Re:

[sha p]

Thank you so much Andrew.  I will start reading it.

On Thu, 6 Sep 2018, 10:26 Andrew Baker,  wrote:

> Hi Shyam,
>
>   Those are big questions! The book *Cassandra: The Definitive Guide *is
> a good place to start, it will walk you through a little bit of each of
> those questions. It should be a challenging project. Look around at
> http://cassandra.apache.org/ and Datastax has some good tutorials and
> videos too as I recall.
>
> -Andrew
>
> On Wed, Sep 5, 2018 at 6:19 AM sha p  wrote:
>
>> Hi all ,
>> Me new to Cassandra , i was asked to migrate data from Oracle to
>> Cassandra.
>> Please help me giving your valuable guidance.
>> 1) Can it be done using open source Cassandra.
>> 2) Where should I start data model from?
>> 3) I should use java, what kind of  jar/libs/tools I need use ?
>> 4) How I decide the size of cluster , please provide some sample
>> guidelines.
>> 5) this should be in production , so what kind of things i should take
>> care for better support or debugging tomorrow?
>> 6) Please provide some good books /links which can help me in this task.
>>
>>
>> Thanks in advance.
>> Highly appreciated your every amal help.
>>
>> Regards,
>> Shyam
>>
>