Fwd: Cassandra Query

[Hareesh Veduraj]

Hi Team,

I have a new requirement , where I need to copy all the rows from one table
to another table in Cassandra, where the second table contains one extra
column.

I have written a python script, which reads each row and inserts . But the
problem is in stage environment I'm observing the select count query is
returning different result each time I execute, so the row count is
varying.

The base  table contains 6 lakhs rows.The stage cluster is of 5 instances
with a replication factor of 3.

I'm able to successfully run the query in dev cluster with the same data,
where 3 instances are there and replication factor is 3.

Can anyone help me out what is the best way to achieve this?Also, what is
the reason for the variation in select count query?

Below is the Python script snippet:

cql='select key, column1, column2, column3, column4, value from table1
limit 100';
insert_table2_query = session.prepare('INSERT INTO table2 (key, column1,
column2, column3, column4, column5, value) VALUES (?, ?, ?, ?, ?, ?, ?)');

def insert_into_table2():
counter =1;
for row in session.execute(cql):
try:
val1=row.column4;
val2 = row.value;
if(len(val1)==0 or val1 == None) :
 val1="";
session.execute(insert_table2_query,(row.key,row.column1,
row.column2,row.column3,val1,val2,val2))
print("processed row count : "+str(counter));
counter = counter + 1;
except Exception as e:
counter = counter + 1;
print 'failed to insert the row ';


Appreciating any help.

Thanks,
Hareesh.A.V
Mob: +7022643519

Warning for large batch sizes with a small number of statements

[Tim Moore]

Hi,

I'm trying to understand some of the details of the
batch_size_warn_threshold_in_kb/batch_size_fail_threshold_in_kb settings.

Specifically, why are the thresholds measured in kb rather than the number
of partitions affected?

We have run into the limit in a situation where there is a batch with
writes to two tables (because we wanted to ensure atomicity of the writes
in the case of a failure). In some situations, the data inserted into one
of these tables can be large enough to push the total batch size over the
limit.

In this specific case, we were able to rewrite things so that it could be
split into separate statement executions with the application handling
retry on failure so atomicity is not needed. It left us wondering, however,
whether executing this as a batch was really problematic, or if the warning
from Cassandra was spurious in this case.

Is it really the total data size that matters, or just the number of
affected partitions?

Thanks,
Tim

-- 
Tim Moore
*Senior Engineer, Lagom, Lightbend, Inc.*
tim.mo...@lightbend.com
+61 420 981 589
Skype: timothy.m.moore

Re: Node Failure Scenario

[Erick Ramirez]

Use the replace_address method with its own IP address. Make sure you
delete the contents of the following directories:
- data/
- commitlog/
- saved_caches/

Forget rejoining with repair -- it will just cause more problems. Cheers!

On Mon, Nov 13, 2017 at 2:54 PM, Anshu Vajpayee 
wrote:

> Hi All ,
>
> There was a node failure in one of production cluster due to disk
> failure.  After h/w recovery that node is noew ready be part of cluster,
> but it doesn't has any data due to disk crash.
>
>
>
> I can think of following option :
>
>
>
> 1. replace the node with same. using replace_address
>
> 2. Set bootstrap=false , start the node and run the repair to stream the
> data.
>
>
>
> Please suggest if both option are good and which is  best as per your
> experience. This is live production cluster.
>
>
> Thanks,
>
>
> --
> *C*heers,*
> *Anshu V*
>
>
>

Node Failure Scenario

[Anshu Vajpayee]

Hi All ,

There was a node failure in one of production cluster due to disk failure.
After h/w recovery that node is noew ready be part of cluster, but it
doesn't has any data due to disk crash.



I can think of following option :



1. replace the node with same. using replace_address

2. Set bootstrap=false , start the node and run the repair to stream the
data.



Please suggest if both option are good and which is  best as per your
experience. This is live production cluster.


Thanks,


-- 
*C*heers,*
*Anshu V*

Re: How to know if bootstrap is still running

[Erick Ramirez]

+1 and run nodetool compactionstats so you can see 2Is in progress.

On Mon, Nov 13, 2017 at 7:00 AM, kurt greaves  wrote:

> bootstrap will wait for secondary indexes and MV's to build before
> completing. if either are still shown in compactions then it will wait for
> them to complete before finishing joining. If not you can try nodetool
> bootstrap resume if it's available on your version.
>
> On 12 Nov. 2017 19:19, "Joel Samuelsson" 
> wrote:
>
> I'm trying to add a new node to a small existing cluster. During the
> bootstrap one of the nodes went down. I'm not sure at what point in the
> process the node went down, all files may have been sent before that
> happened. Currently:
> nodetool netstats says that all files are received 100%
> nodetool status says that the new node is still joining
>
> How can I know if bootstrap has hung?
>
>
>

Re: unrecognized column family in logs

[Erick Ramirez]

nodetool describecluster will show (a) if there are multiple schema
versions, and/or (b) unreachable nodes. Cheers!

On Fri, Nov 10, 2017 at 6:39 PM, Romain Hardouin <
romainh...@yahoo.fr.invalid> wrote:

> Does "nodetool describecluster" shows an actual schema disagreement?
> You can try "nodetool resetlocalschema" to fix the issue on the node
> experiencing disagreement.
>
> Romain
>
> Le jeudi 9 novembre 2017 à 02:55:22 UTC+1, Erick Ramirez <
> flightc...@gmail.com> a écrit :
>
>
> It looks like you have a schema disagreement in your cluster which you
> need to look into.
>
> And you're right since that column family ID is equivalent to Friday, June
> 24, 2016 10:14:49 AM PDT.
>
> Have a look at the table IDs in system.schema_columnfamilies for clues.
> Cheers!
>
> On Thu, Nov 9, 2017 at 4:50 AM, Anubhav Kale  invalid> wrote:
>
> Hello,
>
>
>
> We run Cassandra 2.1.13 and since last few days we’re seeing below in logs
> occasionally. The node then becomes unresponsive to cqlsh.
>
>
>
> ERROR [SharedPool-Worker-2] 2017-11-08 17:02:32,362
> CommitLogSegment.java:441 - Attempted to write commit log entry for
> unrecognized column family: 2854d160-3a2f-11e6-925c- b143135bdc80
>
>
>
> https://github.com/mariusae/ cassandra/blob/master/src/
> java/org/apache/cassandra/db/ commitlog/CommitLogSegment. java#L95
> 
>
>
>
> The column family has heavy writes, but it hasn’t changed schema / load
> wise recently. How can this be troubleshooted / fixed ?
>
>
>
> Thanks !
>
>
>
>
>
>
>
>
>

Re: running C* on AWS EFS storage ...

[kurt greaves]

What's wrong with just detaching the EBS volume and then attaching it to
the new node?​ Assuming you have a separate mount for your C* data (which
you probably should).

Re: running C* on AWS EFS storage ...

[Subroto Barua]

 when we replace node, we can dynamically reassign the IPs to the new node, 
reducing downtime. But we are not sure about the performance in terms of 
latency, throughput etc (we are heavy on reads).
On Sunday, November 12, 2017, 2:03:14 PM PST, Jeff Jirsa  
wrote:  
 
 Any reason you think EFS would be better than EBS?

-- Jeff Jirsa

On Nov 12, 2017, at 1:38 PM, Subroto Barua  wrote:


Hi,
We have a unique requirement to replace C* (3.0.x on RHEL) nodeswith a new AWS 
AMI image periodically. The current process (add node/decommissionetc) is a 
very manual and time consuming process. We currently use EBS andexploring EFS 
as an option to speed up the process.
Does anybody have any experience running C* on EFS storage?Any 
comments/pros/cons going with this combination in Prod?

Thanks,
Subroto

  

Re: running C* on AWS EFS storage ...

[Jeff Jirsa]

Any reason you think EFS would be better than EBS?

-- 
Jeff Jirsa


> On Nov 12, 2017, at 1:38 PM, Subroto Barua  
> wrote:
> 
> Hi,
> 
> We have a unique requirement to replace C* (3.0.x on RHEL) nodes with a new 
> AWS AMI image periodically. The current process (add node/decommission etc) 
> is a very manual and time consuming process. We currently use EBS and 
> exploring EFS as an option to speed up the process.
> Does anybody have any experience running C* on EFS storage? Any 
> comments/pros/cons going with this combination in Prod?
> 
> 
> Thanks,
> 
> Subroto
> 

running C* on AWS EFS storage ...

[Subroto Barua]

Hi,
We have a unique requirement to replace C* (3.0.x on RHEL) nodeswith a new AWS 
AMI image periodically. The current process (add node/decommissionetc) is a 
very manual and time consuming process. We currently use EBS andexploring EFS 
as an option to speed up the process.
Does anybody have any experience running C* on EFS storage?Any 
comments/pros/cons going with this combination in Prod?

Thanks,
Subroto

Re: Safe to run cleanup before repair?

[Joel Samuelsson]

Great, thanks for your replies.

2017-11-12 21:44 GMT+01:00 Jeff Jirsa :

> That is: bootstrap will maintain whatever consistency guarantees you had
> when you started.
>
> --
> Jeff Jirsa
>
>
> On Nov 12, 2017, at 12:41 PM, kurt greaves  wrote:
>
> By default, bootstrap will stream from the primary replica of the range it
> is taking ownership of. So Node 3 would have to stream from Node 2 if it
> was taking ownership of Node 2's tokens.
> On 13 Nov. 2017 05:00, "Joel Samuelsson" 
> wrote:
>
>> Yeah, sounds right. What I'm worried about is the following:
>> I used to have only 2 nodes with RF 2 so both nodes had a copy of all
>> data. There were incosistencies since I was unable to run repair, so some
>> parts of the data may only exist on one node. I have now added two nodes,
>> thus changing which nodes own what parts of the data. My concern is if a
>> piece of data is now owned by say Node 1 and Node 3 but before the addition
>> of new nodes only existed on Node 2 and a cleanup would then delete it
>> permanently since Node 2 no longer owns it. Could this ever happen?
>>
>

Re: Safe to run cleanup before repair?

[Jeff Jirsa]

That is: bootstrap will maintain whatever consistency guarantees you had when 
you started.

-- 
Jeff Jirsa


> On Nov 12, 2017, at 12:41 PM, kurt greaves  wrote:
> 
> By default, bootstrap will stream from the primary replica of the range it is 
> taking ownership of. So Node 3 would have to stream from Node 2 if it was 
> taking ownership of Node 2's tokens.
>> On 13 Nov. 2017 05:00, "Joel Samuelsson"  wrote:
>> Yeah, sounds right. What I'm worried about is the following:
>> I used to have only 2 nodes with RF 2 so both nodes had a copy of all data. 
>> There were incosistencies since I was unable to run repair, so some parts of 
>> the data may only exist on one node. I have now added two nodes, thus 
>> changing which nodes own what parts of the data. My concern is if a piece of 
>> data is now owned by say Node 1 and Node 3 but before the addition of new 
>> nodes only existed on Node 2 and a cleanup would then delete it permanently 
>> since Node 2 no longer owns it. Could this ever happen?

Re: Safe to run cleanup before repair?

[kurt greaves]

By default, bootstrap will stream from the primary replica of the range it
is taking ownership of. So Node 3 would have to stream from Node 2 if it
was taking ownership of Node 2's tokens.
On 13 Nov. 2017 05:00, "Joel Samuelsson"  wrote:

> Yeah, sounds right. What I'm worried about is the following:
> I used to have only 2 nodes with RF 2 so both nodes had a copy of all
> data. There were incosistencies since I was unable to run repair, so some
> parts of the data may only exist on one node. I have now added two nodes,
> thus changing which nodes own what parts of the data. My concern is if a
> piece of data is now owned by say Node 1 and Node 3 but before the addition
> of new nodes only existed on Node 2 and a cleanup would then delete it
> permanently since Node 2 no longer owns it. Could this ever happen?
>

Re: How to know if bootstrap is still running

[kurt greaves]

bootstrap will wait for secondary indexes and MV's to build before
completing. if either are still shown in compactions then it will wait for
them to complete before finishing joining. If not you can try nodetool
bootstrap resume if it's available on your version.

On 12 Nov. 2017 19:19, "Joel Samuelsson"  wrote:

I'm trying to add a new node to a small existing cluster. During the
bootstrap one of the nodes went down. I'm not sure at what point in the
process the node went down, all files may have been sent before that
happened. Currently:
nodetool netstats says that all files are received 100%
nodetool status says that the new node is still joining

How can I know if bootstrap has hung?

Re: Safe to run cleanup before repair?

[Joel Samuelsson]

Yeah, sounds right. What I'm worried about is the following:
I used to have only 2 nodes with RF 2 so both nodes had a copy of all data.
There were incosistencies since I was unable to run repair, so some parts
of the data may only exist on one node. I have now added two nodes, thus
changing which nodes own what parts of the data. My concern is if a piece
of data is now owned by say Node 1 and Node 3 but before the addition of
new nodes only existed on Node 2 and a cleanup would then delete it
permanently since Node 2 no longer owns it. Could this ever happen?

Re: Safe to run cleanup before repair?

[Jeff Jirsa]

Cleanup, very simply, throws away data no longer owned by the instance
because of range movements.

Repair only repairs data owned by the instance (it ignores data that would
be cleared by cleanup).

I don't see any reason why you can't run cleanup before repair.


On Sun, Nov 12, 2017 at 9:35 AM, Joel Samuelsson 
wrote:

> So, I have a cluster which grew too large data-wise so that compactions no
> longer worked (because of full disk). I have now added new nodes so that
> data is spread more thin. However, I know there are incosistencies in the
> cluster and I need to run a repair but those also fail because of out of
> disk errors. Is it safe to run cleanup before I run the repair or might I
> lose data because of said incosistencies?
>

Safe to run cleanup before repair?

[Joel Samuelsson]

So, I have a cluster which grew too large data-wise so that compactions no
longer worked (because of full disk). I have now added new nodes so that
data is spread more thin. However, I know there are incosistencies in the
cluster and I need to run a repair but those also fail because of out of
disk errors. Is it safe to run cleanup before I run the repair or might I
lose data because of said incosistencies?

How to know if bootstrap is still running

[Joel Samuelsson]

I'm trying to add a new node to a small existing cluster. During the
bootstrap one of the nodes went down. I'm not sure at what point in the
process the node went down, all files may have been sent before that
happened. Currently:
nodetool netstats says that all files are received 100%
nodetool status says that the new node is still joining

How can I know if bootstrap has hung?