Re: Improve data load performance

[Elliott Sims]

For write threads, check "nodetool tpstats"

Are you loading the data serially?  That is, one query at a time?  If so
(and if you have no clear resource bottlenecks) you're probably going to
want to add some concurrency into the process.  Break the data up into
smaller chunks and have several threads inserting at once.

On Wed, Aug 15, 2018 at 1:35 PM, Abdul Patel  wrote:

> User in dev env with 4 node cluster , 50k records with inserts of 70k
> characters (json in text)
> This will happen daily in some intervals not yet defined on a single table.
> Its within 1 data center
>
>
> On Wednesday, August 15, 2018, Durity, Sean R 
> wrote:
>
>> Might also help to know:
>>
>> Size of cluster
>>
>> How much data is being loaded (# of inserts/actual data size)
>>
>> Single table or multiple tables?
>>
>> Is this a one-time or occasional load or more frequently?
>>
>> Is the data located in the same physical data center as the cluster? (any
>> network latency?)
>>
>>
>>
>> On the client side, prepared statements and ExecuteAsync can really speed
>> things up.
>>
>>
>>
>>
>>
>> Sean Durity
>>
>>
>>
>> *From:* Elliott Sims 
>> *Sent:* Wednesday, August 15, 2018 1:13 PM
>> *To:* user@cassandra.apache.org
>> *Subject:* [EXTERNAL] Re: Improve data load performance
>>
>>
>>
>> Step one is always to measure your bottlenecks.  Are you spending a lot
>> of time compacting?  Garbage collecting?  Are you saturating CPU?  Or just
>> a few cores?  Or I/O?  Are repairs using all your I/O?  Are you just
>> running out of write threads?
>>
>>
>>
>> On Wed, Aug 15, 2018 at 5:48 AM, Abdul Patel  wrote:
>>
>> Application team is trying to load data with leveled compaction and its
>> taking 1hr to load , what are  best options to load data faster ?
>>
>>
>>
>> On Tuesday, August 14, 2018, @Nandan@ 
>> wrote:
>>
>> Bro, Please explain your question as much as possible.
>> This is not a single line Q session where we will able to understand
>> your in-depth queries in a single line.
>> For better and suitable reply, Please ask a question and elaborate what
>> steps you took for your question and what issue are you getting and all..
>>
>> I hope I am making it clear. Don't take it personally.
>>
>>
>>
>> Thanks
>>
>>
>>
>> On Wed, Aug 15, 2018 at 8:25 AM Abdul Patel  wrote:
>>
>> How can we improve data load performance?
>>
>>
>>
>> --
>>
>> The information in this Internet Email is confidential and may be legally
>> privileged. It is intended solely for the addressee. Access to this Email
>> by anyone else is unauthorized. If you are not the intended recipient, any
>> disclosure, copying, distribution or any action taken or omitted to be
>> taken in reliance on it, is prohibited and may be unlawful. When addressed
>> to our clients any opinions or advice contained in this Email are subject
>> to the terms and conditions expressed in any applicable governing The Home
>> Depot terms of business or client engagement letter. The Home Depot
>> disclaims all responsibility and liability for the accuracy and content of
>> this attachment and for any damages or losses arising from any
>> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
>> items of a destructive nature, which may be contained in this attachment
>> and shall not be liable for direct, indirect, consequential or special
>> damages in connection with this e-mail message or its attachment.
>>
>

Re: "minimum backup" in vnodes

[Elliott Sims]

Assuming this isn't an existing cluster, the easiest method is probably to
use logical "racks" to explicitly control which hosts have a full replica
of the data.  with RF3 and 3 "racks", each "rack" has one complete replica.
If you're not using the logical racks, I think the replicas are spread
randomly and you can't generally reduce your backup count safely even with
a lot of work.

On Wed, Aug 15, 2018 at 1:39 PM, Carl Mueller <
carl.muel...@smartthings.com.invalid> wrote:

> Goal: backup a cluster with the minimum amount of data. Restore to be done
> with sstableloader
>
> Let's start with a basic case:
> - six node cluster
> - one datacenter
> - RF3
> - data is perfectly replicated/repaired
> - Manual tokens (no vnodes)
> - simplest strategy
>
> In this case, it is (theoretically) possible to get an perfect backup of
> data by storing the snapshots of two of the six nodes in the cluster due to
> replication factor.
>
> I once tried to parse the ring output with vnodes (256) and came to the
> conclusion that it was not possible with vnodes, maybe you could avoid one
> or two nodes of the six... tops. But I may have had an incorrect
> understanding of how ranges are replicated in vnodes.
>
> Would it be possible to pick only two nodes out of a six node cluster with
> vnodes and RF-3 that will backup the cluster?
>
>

"minimum backup" in vnodes

[Carl Mueller]

Goal: backup a cluster with the minimum amount of data. Restore to be done
with sstableloader

Let's start with a basic case:
- six node cluster
- one datacenter
- RF3
- data is perfectly replicated/repaired
- Manual tokens (no vnodes)
- simplest strategy

In this case, it is (theoretically) possible to get an perfect backup of
data by storing the snapshots of two of the six nodes in the cluster due to
replication factor.

I once tried to parse the ring output with vnodes (256) and came to the
conclusion that it was not possible with vnodes, maybe you could avoid one
or two nodes of the six... tops. But I may have had an incorrect
understanding of how ranges are replicated in vnodes.

Would it be possible to pick only two nodes out of a six node cluster with
vnodes and RF-3 that will backup the cluster?

Re: Improve data load performance

[Abdul Patel]

User in dev env with 4 node cluster , 50k records with inserts of 70k
characters (json in text)
This will happen daily in some intervals not yet defined on a single table.
Its within 1 data center

On Wednesday, August 15, 2018, Durity, Sean R 
wrote:

> Might also help to know:
>
> Size of cluster
>
> How much data is being loaded (# of inserts/actual data size)
>
> Single table or multiple tables?
>
> Is this a one-time or occasional load or more frequently?
>
> Is the data located in the same physical data center as the cluster? (any
> network latency?)
>
>
>
> On the client side, prepared statements and ExecuteAsync can really speed
> things up.
>
>
>
>
>
> Sean Durity
>
>
>
> *From:* Elliott Sims 
> *Sent:* Wednesday, August 15, 2018 1:13 PM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] Re: Improve data load performance
>
>
>
> Step one is always to measure your bottlenecks.  Are you spending a lot of
> time compacting?  Garbage collecting?  Are you saturating CPU?  Or just a
> few cores?  Or I/O?  Are repairs using all your I/O?  Are you just running
> out of write threads?
>
>
>
> On Wed, Aug 15, 2018 at 5:48 AM, Abdul Patel  wrote:
>
> Application team is trying to load data with leveled compaction and its
> taking 1hr to load , what are  best options to load data faster ?
>
>
>
> On Tuesday, August 14, 2018, @Nandan@ 
> wrote:
>
> Bro, Please explain your question as much as possible.
> This is not a single line Q session where we will able to understand
> your in-depth queries in a single line.
> For better and suitable reply, Please ask a question and elaborate what
> steps you took for your question and what issue are you getting and all..
>
> I hope I am making it clear. Don't take it personally.
>
>
>
> Thanks
>
>
>
> On Wed, Aug 15, 2018 at 8:25 AM Abdul Patel  wrote:
>
> How can we improve data load performance?
>
>
>
> --
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions or advice contained in this Email are subject
> to the terms and conditions expressed in any applicable governing The Home
> Depot terms of business or client engagement letter. The Home Depot
> disclaims all responsibility and liability for the accuracy and content of
> this attachment and for any damages or losses arising from any
> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
> items of a destructive nature, which may be contained in this attachment
> and shall not be liable for direct, indirect, consequential or special
> damages in connection with this e-mail message or its attachment.
>

RE: [EXTERNAL] Re: Improve data load performance

[Durity, Sean R]

Might also help to know:
Size of cluster
How much data is being loaded (# of inserts/actual data size)
Single table or multiple tables?
Is this a one-time or occasional load or more frequently?
Is the data located in the same physical data center as the cluster? (any 
network latency?)

On the client side, prepared statements and ExecuteAsync can really speed 
things up.


Sean Durity

From: Elliott Sims 
Sent: Wednesday, August 15, 2018 1:13 PM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Re: Improve data load performance

Step one is always to measure your bottlenecks.  Are you spending a lot of time 
compacting?  Garbage collecting?  Are you saturating CPU?  Or just a few cores? 
 Or I/O?  Are repairs using all your I/O?  Are you just running out of write 
threads?

On Wed, Aug 15, 2018 at 5:48 AM, Abdul Patel 
mailto:abd786...@gmail.com>> wrote:
Application team is trying to load data with leveled compaction and its taking 
1hr to load , what are  best options to load data faster ?


On Tuesday, August 14, 2018, @Nandan@ 
mailto:nandanpriyadarshi...@gmail.com>> wrote:
Bro, Please explain your question as much as possible.
This is not a single line Q session where we will able to understand your 
in-depth queries in a single line.
For better and suitable reply, Please ask a question and elaborate what steps 
you took for your question and what issue are you getting and all..
I hope I am making it clear. Don't take it personally.

Thanks

On Wed, Aug 15, 2018 at 8:25 AM Abdul Patel 
mailto:abd786...@gmail.com>> wrote:
How can we improve data load performance?




The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

Re: Improve data load performance

[Abdul Patel]

I didnt see any such bottlenecks , they are testing to write json file as
an text in cassandra which is slow ..rest of performance looks good?
Regarding write threads where i can chexk how many configured and if there
is bittleneck?

On Wednesday, August 15, 2018, Elliott Sims  wrote:

> Step one is always to measure your bottlenecks.  Are you spending a lot of
> time compacting?  Garbage collecting?  Are you saturating CPU?  Or just a
> few cores?  Or I/O?  Are repairs using all your I/O?  Are you just running
> out of write threads?
>
> On Wed, Aug 15, 2018 at 5:48 AM, Abdul Patel  wrote:
>
>> Application team is trying to load data with leveled compaction and its
>> taking 1hr to load , what are  best options to load data faster ?
>>
>>
>> On Tuesday, August 14, 2018, @Nandan@ 
>> wrote:
>>
>>> Bro, Please explain your question as much as possible.
>>> This is not a single line Q session where we will able to understand
>>> your in-depth queries in a single line.
>>> For better and suitable reply, Please ask a question and elaborate what
>>> steps you took for your question and what issue are you getting and all..
>>>
>>> I hope I am making it clear. Don't take it personally.
>>>
>>> Thanks
>>>
>>> On Wed, Aug 15, 2018 at 8:25 AM Abdul Patel  wrote:
>>>
 How can we improve data load performance?
>>>
>>>
>

Re: Improve data load performance

[Elliott Sims]

Step one is always to measure your bottlenecks.  Are you spending a lot of
time compacting?  Garbage collecting?  Are you saturating CPU?  Or just a
few cores?  Or I/O?  Are repairs using all your I/O?  Are you just running
out of write threads?

On Wed, Aug 15, 2018 at 5:48 AM, Abdul Patel  wrote:

> Application team is trying to load data with leveled compaction and its
> taking 1hr to load , what are  best options to load data faster ?
>
>
> On Tuesday, August 14, 2018, @Nandan@ 
> wrote:
>
>> Bro, Please explain your question as much as possible.
>> This is not a single line Q session where we will able to understand
>> your in-depth queries in a single line.
>> For better and suitable reply, Please ask a question and elaborate what
>> steps you took for your question and what issue are you getting and all..
>>
>> I hope I am making it clear. Don't take it personally.
>>
>> Thanks
>>
>> On Wed, Aug 15, 2018 at 8:25 AM Abdul Patel  wrote:
>>
>>> How can we improve data load performance?
>>
>>

Re: Setting up rerouting java/python driver read requests from unresponsive nodes to good ones

[Chris Lohfink]

That’s what the retry handler does (see Horia’s response). You can also use the 
speculative retry to possibly send requests to multiple coordinators a little 
earlier as well to reduce the impact of the slow requests (ie a GC). 
https://docs.datastax.com/en/developer/java-driver/3.1/manual/speculative_execution/

Chris

Sent from my iPhone

> On Aug 15, 2018, at 6:57 AM, Horia Mocioi  wrote:
> 
> Hello,
> 
> I believe that this is what you are looking for - 
> https://docs.datastax.com/en/developer/java-driver/3.5/manual/retries/
> 
> In particular, tryNextHost().
> 
> Regards,
> Horia
> 
>> On ons, 2018-08-15 at 14:16 +0300, Vsevolod Filaretov wrote:
>> Hello Cassandra community!
>> 
>> Unfortunately, I cannot find the corresponding info via load balancing 
>> manuals, so the question is:
>> 
>> Is it possible to set up java/python cassandra driver to redirect 
>> unsuccessful read requests from the coordinator node, which came to be 
>> unresponsive during the session, to the up and running one (dynamically 
>> switch to other coordinator node from the dead one)?
>> 
>> If the answer is no, what could be my alternatives?
>> 
>> Thank you all in advance,
>> Vsevolod Filaretov.

Re: Setting up rerouting java/python driver read requests from unresponsive nodes to good ones

[Horia Mocioi]

Hello,

I believe that this is what you are looking for - 
https://docs.datastax.com/en/developer/java-driver/3.5/manual/retries/

In particular, 
tryNextHost().

Regards,
Horia

On ons, 2018-08-15 at 14:16 +0300, Vsevolod Filaretov wrote:
Hello Cassandra community!

Unfortunately, I cannot find the corresponding info via load balancing manuals, 
so the question is:

Is it possible to set up java/python cassandra driver to redirect unsuccessful 
read requests from the coordinator node, which came to be unresponsive during 
the session, to the up and running one (dynamically switch to other coordinator 
node from the dead one)?

If the answer is no, what could be my alternatives?

Thank you all in advance,
Vsevolod Filaretov.

Setting up rerouting java/python driver read requests from unresponsive nodes to good ones

[Vsevolod Filaretov]

Hello Cassandra community!

Unfortunately, I cannot find the corresponding info via load balancing
manuals, so the question is:

Is it possible to set up java/python cassandra driver to redirect
unsuccessful read requests from the coordinator node, which came to be
unresponsive during the session, to the up and running one (dynamically
switch to other coordinator node from the dead one)?

If the answer is no, what could be my alternatives?

Thank you all in advance,
Vsevolod Filaretov.

Re: Improve data load performance

[Abdul Patel]

Application team is trying to load data with leveled compaction and its
taking 1hr to load , what are  best options to load data faster ?


On Tuesday, August 14, 2018, @Nandan@ 
wrote:

> Bro, Please explain your question as much as possible.
> This is not a single line Q session where we will able to understand
> your in-depth queries in a single line.
> For better and suitable reply, Please ask a question and elaborate what
> steps you took for your question and what issue are you getting and all..
>
> I hope I am making it clear. Don't take it personally.
>
> Thanks
>
> On Wed, Aug 15, 2018 at 8:25 AM Abdul Patel  wrote:
>
>> How can we improve data load performance?
>
>

Re: JBOD disk failure

[kurt greaves]

Yep. It might require a full node replace depending on what data is lost
from the system tables. In some cases you might be able to recover from
partially lost system info, but it's not a sure thing.

On Wed., 15 Aug. 2018, 17:55 Christian Lorenz, <
christian.lor...@webtrekk.com> wrote:

> Thank you for the answers. We are using the current version 3.11.3 So this
> one includes CASSANDRA-6696.
>
> So if I get this right, losing system tables will need a full node
> rebuild. Otherwise repair will get the node consistent again.
>
>
>
> Regards,
>
> Christian
>
>
>
>
>
> *Von: *kurt greaves 
> *Antworten an: *"user@cassandra.apache.org" 
> *Datum: *Mittwoch, 15. August 2018 um 04:53
> *An: *User 
> *Betreff: *Re: JBOD disk failure
>
>
>
> If that disk had important data in the system tables however you might
> have some trouble and need to replace the entire instance anyway.
>
>
>
> On 15 August 2018 at 12:20, Jeff Jirsa  wrote:
>
> Depends on version
>
>
>
> For versions without the fix from Cassandra-6696, the only safe option on
> single disk failure is to stop and replace the whole instance - this is
> important because in older versions of Cassandra, you could have data in
> one sstable, a tombstone shadowing it in another disk, and it could be very
> far behind gc_grace_seconds. On disk failure in this scenario, if the disk
> holding the tombstone is lost, repair will propagate the
> (deleted/resurrected) data to the other replicas, which probably isn’t what
> you want to happen.
>
>
>
> With 6696, you should be safe to replace the disk and run repair - 6696
> will keep data for a given token range all on the same disks, so the
> resurrection problem is solved.
>
>
>
>
>
> --
>
> Jeff Jirsa
>
>
>
>
> On Aug 14, 2018, at 6:10 AM, Christian Lorenz <
> christian.lor...@webtrekk.com> wrote:
>
> Hi,
>
>
>
> given a cluster with RF=3 and CL=LOCAL_ONE and application is deleting
> data, what happens if the nodes are setup with JBOD and one disk fails? Do
> I get consistent results while the broken drive is replaced and a nodetool
> repair is running on the node with the replaced drive?
>
>
>
> Kind regards,
>
> Christian
>
>
>

Re: Reading cardinality from Statistics.db failed

[Vitali Dyachuk]

I've upgraded to 3.0.17 and the issue is still there, Is there a jira
ticket for that bug  or should i create one?

On Wed, Jul 25, 2018 at 2:57 PM Vitali Dyachuk  wrote:

> I'm using 3.0.15. I see that there is some fix for sstable metadata in
> 3.0.16 https://issues.apache.org/jira/browse/CASSANDRA-14217 - is that a
> fix for "reading cardinalyti from statistics.db" ?
>
>
> On Wed, Jul 25, 2018 at 1:02 PM Hannu Kröger  wrote:
>
>> What version of Cassandra are you running? There is a bug in 3.10.0 and
>> certain 3.0.x that occurs in certain conditions and corrupts that file.
>>
>> Hannu
>>
>> Vitali Dyachuk  kirjoitti 25.7.2018 kello 10.48:
>>
>> Hi,
>> I have noticed in the cassandra system.log that there is some issue with
>> sstable metadata, the messages says:
>> WARN  [Thread-6] 2018-07-25 07:12:47,928 SSTableReader.java:249 - Reading
>> cardinality from Statistics.db failed for
>> /opt/data/disk5/data/keyspace/table/mc-big-Data.db
>> Although there is no such file. The message has appeared after i've
>> changed the compaction strategy from SizeTiered to Leveled.
>> Currently i'm running nodetool scrub to rebuilt the sstable, and it takes
>> a lot of time to scrub all sstables.
>> Reading the code it is said that if this metada is broken, then
>> estimating the keys will be done using index summary. How expensive it is ?
>>
>> https://github.com/apache/cassandra/blob/cassandra-3.0.15/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L245
>>
>> The main question is why has this happened?
>>
>> Thanks,
>> Vitali Djatsuk.
>>
>>

Re: JBOD disk failure

[Christian Lorenz]

Thank you for the answers. We are using the current version 3.11.3 So this one 
includes CASSANDRA-6696.
So if I get this right, losing system tables will need a full node rebuild. 
Otherwise repair will get the node consistent again.

Regards,
Christian


Von: kurt greaves 
Antworten an: "user@cassandra.apache.org" 
Datum: Mittwoch, 15. August 2018 um 04:53
An: User 
Betreff: Re: JBOD disk failure

If that disk had important data in the system tables however you might have 
some trouble and need to replace the entire instance anyway.

On 15 August 2018 at 12:20, Jeff Jirsa 
mailto:jji...@gmail.com>> wrote:
Depends on version

For versions without the fix from Cassandra-6696, the only safe option on 
single disk failure is to stop and replace the whole instance - this is 
important because in older versions of Cassandra, you could have data in one 
sstable, a tombstone shadowing it in another disk, and it could be very far 
behind gc_grace_seconds. On disk failure in this scenario, if the disk holding 
the tombstone is lost, repair will propagate the (deleted/resurrected) data to 
the other replicas, which probably isn’t what you want to happen.

With 6696, you should be safe to replace the disk and run repair - 6696 will 
keep data for a given token range all on the same disks, so the resurrection 
problem is solved.


--
Jeff Jirsa


On Aug 14, 2018, at 6:10 AM, Christian Lorenz 
mailto:christian.lor...@webtrekk.com>> wrote:
Hi,

given a cluster with RF=3 and CL=LOCAL_ONE and application is deleting data, 
what happens if the nodes are setup with JBOD and one disk fails? Do I get 
consistent results while the broken drive is replaced and a nodetool repair is 
running on the node with the replaced drive?

Kind regards,
Christian