Re: a solution of getting cassandra cross-datacenter latency at a certain time

[Stone Fang]

thanks for you response.

@Ryan Svihla,yeah you are right,there will be a ttl.only store most recent
data.

@Chris
good idea on getting the  instantaneous  cross-datacenter latency.but seems
still have issues.
1.based on application insert/update record,not a regular generate
record.once the network between application server and cassandra is bad,
cannot capture the latency value,because casssandra dont know it is the
reason that application have not sent record.or request fail because of the
large latency.

2.as you suggested,it seems that cannot get the latency at a certain time
in recent.
network latency always stable,but when workload are heavy and replication
cross datacenter need to wait.and it will increase cross-datacenter latency.
so may be there is a large difference between this minute and next minute

stone

On Mon, Aug 8, 2016 at 9:10 PM, Chris Lohfink  wrote:

> If you invoke the values operation on the mbean every minute (or whatever
> period) you can get a histogram of the cross dc the latencies. Just keep
> track of the values of each bin in histogram and look at the delta from
> previous time to the current time to find how many latencies occurred in
> each bins range during the period.
>
> Also can wait for CASSANDRA-11752
>  for the a
> "recent" histogram (although would need to apply it to this histogram as
> well).
>
> Chris Lohfink
>
> On Mon, Aug 8, 2016 at 8:50 AM, Ryan Svihla  wrote:
>
>> The first issue I can think of is the Latency table, if I understand you
>> correctly, has an unbounded size for the partition key of DC and will over
>> time just get larger as more measurements are recorded.
>>
>> Regards,
>>
>> Ryan Svihla
>>
>> On Aug 8, 2016, at 2:58 AM, Stone Fang  wrote:
>>
>> *objective*:get cassandra cross-datacenter latency in time
>>
>> *existing ticket:*
>>
>> there is a ticket [track cross-datacenter latency](https://issues.apache
>> .org/jira/browse/CASSANDRA-11569)
>> but it is a statistics value from node starting,i want to get the
>> instantaneous value in a certain time.
>>
>> *thought*
>>
>> want to write a message into **MESSAGE TABLE** in 1s timer task(the
>> period is similar to most of cross datacenter latency )
>> ,and replicate to other datacenter,there will be a delay.and I capture
>> it,and write to **LATENCY TABLE**.i can query the latency value from this
>> table with the condition of certain time.
>>
>> *schema*
>>
>> message table for replicating data cross datacenter
>>
>>
>> create keyspace heartbeat with replication=
>> {'class':'NetworkTopologyStrategy','dc1':1, 'dc2':1...};
>>
>>
>>
>>  CREATE TABLE HEARTBEAT.MESSAGE{
>> CREATED TIMESTAMP,
>> FROMDC VARCHAR,
>> PRIMARY KEY(CREATED,FROMDC)
>> }
>>
>> latency Table for querying latency value
>>
>>  CREATE TABLE SYSTEM.LATENCY{
>>  FROMDC VARCHAR,
>>  ARRIVED TIMESTAMP,
>>  CREATED TIMESTAMP,
>>  LANTENCY BIGINT
>>  PRIMARY KEY(FROMDC,ARRIVED)
>> }WITH CLUSTERING ORDER BY(ARRIVED DESC);
>>
>> problems
>>
>> 1.can this solution work to get the cross-datacenter latency?
>>
>>
>> 2.create heartbeat keyspace in cassandra bootstrap process,i need to load
>> Heartbeat keyspace in Scheam.java.and save this keyspace into SystemSchema.
>> also need to check if this keyspace has exist after first node start.so i
>> think this is not a good solution.
>>
>> 3.compared to 1,try another solution.generate heartbeat message in a
>> standalone jar.but always i need to capture heartbeat message mutation in
>> cassandra.so i need to check if the mutation is about heartbeat message.and
>> it seems strange that check the heartbeat keyspace which is not defined in
>> cassandra,but third-party.
>>
>> hope to see your thought on this.
>> thanks
>> stone
>>
>>
>

Re: Adding Materialized View triggers "Mutation Too Large" error.

[Jason J. W. Williams]

Hi Paulo,

Thanks for bug link! I've subscribed to it for updates. Thankfully, these
tables for something still in beta so we can wipe the data. But it'll be
great when its fixed.

-J

On Mon, Aug 8, 2016 at 4:26 PM, Paulo Motta 
wrote:

> What happens is that when trying to rebuild the MV, the rebuilder tries to
> create a very large batch that exceeds commitlog_segment_size_in_mb. This
> limitation is currently being addressed on CASSANDRA-11670. Two options I
> can see to workaround this for now:
> 1) increase commitlog_segment_size_in_mb as you suggested
> 2) Re-create the base table and views and reinsert all data again to the
> base able.
>
> 2016-08-08 19:52 GMT-03:00 Jason J. W. Williams  >:
>
>> HI Guys,
>>
>> We're running Cassandra 3.0.8, and needed to add a field to a table and
>> it's materialized views. We dropped the MVs, added the fields to the
>> underlying table, and recreated MVs with the new fields. However, the MV
>> creation is failing with:
>>
>>
>> WARN  [CompactionExecutor:6933] 2016-08-08 22:48:11,075
>> ViewBuilder.java:178 - Materialized View failed to complete, sleeping 5
>> minutes before restarting
>> java.lang.IllegalArgumentException: Mutation of 17709054 bytes is too
>> large for the maximum size of 16777216
>>
>> https://gist.github.com/williamsjj/49e4c6a0b3324343205386d040560e1e
>>
>> I'm considering bumping commitlog_segment_size_in_mb up to 64MB to get
>> around this, but it seems like the MV code should be smart enough to stay
>> under the limit.
>>
>> Any advice is greatly appreciated.
>>
>> -J
>>
>
>

Re: Adding Materialized View triggers "Mutation Too Large" error.

[Paulo Motta]

What happens is that when trying to rebuild the MV, the rebuilder tries to
create a very large batch that exceeds commitlog_segment_size_in_mb. This
limitation is currently being addressed on CASSANDRA-11670. Two options I
can see to workaround this for now:
1) increase commitlog_segment_size_in_mb as you suggested
2) Re-create the base table and views and reinsert all data again to the
base able.

2016-08-08 19:52 GMT-03:00 Jason J. W. Williams :

> HI Guys,
>
> We're running Cassandra 3.0.8, and needed to add a field to a table and
> it's materialized views. We dropped the MVs, added the fields to the
> underlying table, and recreated MVs with the new fields. However, the MV
> creation is failing with:
>
>
> WARN  [CompactionExecutor:6933] 2016-08-08 22:48:11,075
> ViewBuilder.java:178 - Materialized View failed to complete, sleeping 5
> minutes before restarting
> java.lang.IllegalArgumentException: Mutation of 17709054 bytes is too
> large for the maximum size of 16777216
>
> https://gist.github.com/williamsjj/49e4c6a0b3324343205386d040560e1e
>
> I'm considering bumping commitlog_segment_size_in_mb up to 64MB to get
> around this, but it seems like the MV code should be smart enough to stay
> under the limit.
>
> Any advice is greatly appreciated.
>
> -J
>

Adding Materialized View triggers "Mutation Too Large" error.

[Jason J. W. Williams]

HI Guys,

We're running Cassandra 3.0.8, and needed to add a field to a table and
it's materialized views. We dropped the MVs, added the fields to the
underlying table, and recreated MVs with the new fields. However, the MV
creation is failing with:


WARN  [CompactionExecutor:6933] 2016-08-08 22:48:11,075
ViewBuilder.java:178 - Materialized View failed to complete, sleeping 5
minutes before restarting
java.lang.IllegalArgumentException: Mutation of 17709054 bytes is too large
for the maximum size of 16777216

https://gist.github.com/williamsjj/49e4c6a0b3324343205386d040560e1e

I'm considering bumping commitlog_segment_size_in_mb up to 64MB to get
around this, but it seems like the MV code should be smart enough to stay
under the limit.

Any advice is greatly appreciated.

-J

Re: compactions stuck and restart doesn't kill it

[Surbhi Gupta]

I have faced similar kind of issue while we were using LCS ...
We ran the scrub and it was helpful to clear the corruption and compaction
caught up well after that ...


On 8 August 2016 at 13:23, John Wong  wrote:

> Hi.
>
> I don't see any error logs indicate corrupted sstable error. Is it safe to
> run this?
>
> Thanks.
>
> On Mon, Aug 8, 2016 at 1:18 PM, Surbhi Gupta 
> wrote:
>
>> Can you see if any of the sstable is corrupt ?
>> I have seen with my past experience  , if any of the sstable which is
>> part of the compaction is corrupt then compaction kind of hangs.
>> Try to run nodetool scrub ...
>>
>> On 8 August 2016 at 09:34, John Wong  wrote:
>>
>>> On Mon, Aug 8, 2016 at 11:16 AM, Surbhi Gupta 
>>> wrote:
>>>
 Once you restart a node compaction will start automatically, if u dont
 want to do so then do
 nodetool disableautocompaction as soon as node is started .


>>> Thanks. I certainly can give that a try for the specific column family,
>>> but even so this would be a temporary solution, as the root issue is the
>>> compaction won't finish. I see nothing in the log. FYI, gathering more info
>>> from the developer, this CF stores large rows.
>>>
>>> If anyone ever encountered similar issue, please let me know.
>>>
>>> John
>>>
>>
>>
>

Re: compactions stuck and restart doesn't kill it

[John Wong]

Hi.

I don't see any error logs indicate corrupted sstable error. Is it safe to
run this?

Thanks.

On Mon, Aug 8, 2016 at 1:18 PM, Surbhi Gupta 
wrote:

> Can you see if any of the sstable is corrupt ?
> I have seen with my past experience  , if any of the sstable which is part
> of the compaction is corrupt then compaction kind of hangs.
> Try to run nodetool scrub ...
>
> On 8 August 2016 at 09:34, John Wong  wrote:
>
>> On Mon, Aug 8, 2016 at 11:16 AM, Surbhi Gupta 
>> wrote:
>>
>>> Once you restart a node compaction will start automatically, if u dont
>>> want to do so then do
>>> nodetool disableautocompaction as soon as node is started .
>>>
>>>
>> Thanks. I certainly can give that a try for the specific column family,
>> but even so this would be a temporary solution, as the root issue is the
>> compaction won't finish. I see nothing in the log. FYI, gathering more info
>> from the developer, this CF stores large rows.
>>
>> If anyone ever encountered similar issue, please let me know.
>>
>> John
>>
>
>

Re: Doing rolling upgrades on two DCs

[Erik Forkalsud]

The low-risk approach would be to change one thing at a time, but I 
don't think there's
a technical reason you couldn't change the commit log and data directory 
locations

while updating to a new version.

- EF

On 08/08/2016 07:58 AM, Timon Wong wrote:

Thank you!
We would do a patch level upgrade, from 2.2.1 to 2.2.7 (the latest 
stable). Besides, we want to adjust both `commitlog_directory` and 
`data_file_directories` settings (separate them into two better IOPS 
volumes), and we assumed they should be done before or after upgrade?


On Sun, Aug 7, 2016 at 5:38 AM, Forkalsrud, Erik > wrote:



You didn't mention which version you're upgrading to, but
generally you shouldn't add or
remove nodes while the nodes your cluster is not all on on the
same version. So if you're
going to add nodes, you should do that before you upgrade the
first node, or after you have
upgraded all of them.

Shifting traffic away from DC1 may not be necessary, but shouldn't
hurt.


- EF

_
From: Timon Wong [timon86.w...@gmail.com
]
Sent: Saturday, August 6, 2016 02:03
To: user@cassandra.apache.org 
Subject: Doing rolling upgrades on two DCs

Hello, newbie here.

Currently we are using Cassandra v2.2.1 (two DCs, naming them DC1,
DC2, respectively), and recently we have two issues:

1. Cannot adding new node due to CASSANDRA-10961
2. Cannot repair node, due to CASSANDRA-10501

We learned that we can do rolling upgrade, since we are only using
one DC1 for now (DC2 is just for replica), we have following steps
in mind:

1. Backup;
2. Doing upgrade on DC2. (Follows this guide:

https://docs.datastax.com/en/latest-upgrade/upgrade/cassandra/upgrdCassandraDetails.html

);
3. Do some checks (client compatibility, etc);
4. Adding nodes to DC2 (using new Cassandra);
5. Traffic our loads to DC2;
6. Upgrading DC1.

Is that possible, or do we miss something?

Thanks in advance!

Timon

Re: Verify cassandra backup and restore in C * 2.1

[Rajath Subramanyam]

Hi Indranil,

One approach is to do a row count on the original source table and the
table that is restored from backup. How are you backing up data ? I am
assuming you are issuing snapshot commands (either incremental or
otherwise).

I hope this helps.

- Rajath


Rajath Subramanyam


On Mon, Aug 8, 2016 at 4:29 AM, INDRANIL BASU  wrote:

> Hello,
>
>How can we verify that cassandra data is backed up and restored
> correctly?
>
> *-- IB*
>
>
>
>

Re: compactions stuck and restart doesn't kill it

[Surbhi Gupta]

Can you see if any of the sstable is corrupt ?
I have seen with my past experience  , if any of the sstable which is part
of the compaction is corrupt then compaction kind of hangs.
Try to run nodetool scrub ...

On 8 August 2016 at 09:34, John Wong  wrote:

> On Mon, Aug 8, 2016 at 11:16 AM, Surbhi Gupta 
> wrote:
>
>> Once you restart a node compaction will start automatically, if u dont
>> want to do so then do
>> nodetool disableautocompaction as soon as node is started .
>>
>>
> Thanks. I certainly can give that a try for the specific column family,
> but even so this would be a temporary solution, as the root issue is the
> compaction won't finish. I see nothing in the log. FYI, gathering more info
> from the developer, this CF stores large rows.
>
> If anyone ever encountered similar issue, please let me know.
>
> John
>

Re: setting ttl using timestamp field of record during insertion in cassandra

[Sujeet Kumar]

Thanks.Actually i want to start ttl from the record timestamp field, not
from insertion time(ttl starts from insertion time).

On Mon, Aug 8, 2016 at 9:36 PM,  wrote:

> Perhaps you can set the default TTL when you create the table instead of
> on every insert:
>
> http://docs.datastax.com/en/cql/3.1/cql/cql_reference/tabProp.html
>
> (see default_time_to_live property)
>
>
>
> Then no need for UDF.
>
>
>
>
>
> Sean Durity
>
> lord of the (C*) rings (Lead Cassandra Admin)
>
> Big DATA Team
>
> MTC 2250
>
> #cassandra - for the latest news and updates
>
>
>
> *From:* Sujeet Kumar [mailto:genioussuj...@gmail.com]
> *Sent:* Monday, August 08, 2016 11:18 AM
> *To:* user@cassandra.apache.org
> *Subject:* setting ttl using timestamp field of record during insertion
> in cassandra
>
>
>
> Hi
>
>
>
> I want to set TTL 30 days from the time field of table
> record(id,name,time) during insertion.For this I am creating a User Defined
> Function(UDF) bigint fun(rTime,cTime) as
>
>
>
> CREATE FUNCTION fun(rtime timestamp,ctime timestamp) CALLED ON NULL INPUT
> RETURNS bigint LANGUAGE java as 'return 2592000-((ctime.toTime()
> -rtime.toTime())/1000);';
>
>
>
> here,function fun is calculating the time in seconds this data should
> live.
>
> 2592000 is the time in seconds for 30 days.
>
>
>
> I am trying to use above function for setting TTL as
>
>
>
> INSERT INTO record(id,name,time) VALUES (123,'data123','2016-08-08
> 06:06:00') USING TTL fun('2016-08-08 06:06:00',totimestamp(now()));
>
>
>
> getting error as
>
> Syntax Exception: ErrorMessage code=2000 
>
>
>
> Is there any other way to set ttl based on record time field. What is
> problem with above approach.
>
>
>
> Thanks.
>
>
>
> Sujeet
>
>
>
> --
>
>
>
> --
>
> Sujeet Kumar
>
> ME(CSA)
>
> IISc Banglore
>
> sujeet.ku...@csa.iisc.ernet.in
>
> sujeet.cs...@mnnit.ac.in
>
> --
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions or advice contained in this Email are subject
> to the terms and conditions expressed in any applicable governing The Home
> Depot terms of business or client engagement letter. The Home Depot
> disclaims all responsibility and liability for the accuracy and content of
> this attachment and for any damages or losses arising from any
> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
> items of a destructive nature, which may be contained in this attachment
> and shall not be liable for direct, indirect, consequential or special
> damages in connection with this e-mail message or its attachment.
>



-- 

-- 
Sujeet Kumar
ME(CSA)
IISc Banglore
sujeet.ku...@csa.iisc.ernet.in
sujeet.cs...@mnnit.ac.in

Re: compactions stuck and restart doesn't kill it

[John Wong]

On Mon, Aug 8, 2016 at 11:16 AM, Surbhi Gupta 
wrote:

> Once you restart a node compaction will start automatically, if u dont
> want to do so then do
> nodetool disableautocompaction as soon as node is started .
>
>
Thanks. I certainly can give that a try for the specific column family, but
even so this would be a temporary solution, as the root issue is the
compaction won't finish. I see nothing in the log. FYI, gathering more info
from the developer, this CF stores large rows.

If anyone ever encountered similar issue, please let me know.

John

RE: setting ttl using timestamp field of record during insertion in cassandra

[SEAN_R_DURITY]

Perhaps you can set the default TTL when you create the table instead of on 
every insert:
http://docs.datastax.com/en/cql/3.1/cql/cql_reference/tabProp.html
(see default_time_to_live property)

Then no need for UDF.


Sean Durity
lord of the (C*) rings (Lead Cassandra Admin)
Big DATA Team
MTC 2250
#cassandra - for the latest news and updates

From: Sujeet Kumar [mailto:genioussuj...@gmail.com]
Sent: Monday, August 08, 2016 11:18 AM
To: user@cassandra.apache.org
Subject: setting ttl using timestamp field of record during insertion in 
cassandra

Hi

I want to set TTL 30 days from the time field of table record(id,name,time) 
during insertion.For this I am creating a User Defined Function(UDF) bigint 
fun(rTime,cTime) as

CREATE FUNCTION fun(rtime timestamp,ctime timestamp) CALLED ON NULL INPUT 
RETURNS bigint LANGUAGE java as 'return 2592000-((ctime.toTime() 
-rtime.toTime())/1000);';

here,function fun is calculating the time in seconds this data should live.
2592000 is the time in seconds for 30 days.

I am trying to use above function for setting TTL as

INSERT INTO record(id,name,time) VALUES (123,'data123','2016-08-08 06:06:00') 
USING TTL fun('2016-08-08 06:06:00',totimestamp(now()));

getting error as
Syntax Exception: ErrorMessage code=2000 

Is there any other way to set ttl based on record time field. What is problem 
with above approach.

Thanks.

Sujeet

--

--
Sujeet Kumar
ME(CSA)
IISc Banglore
sujeet.ku...@csa.iisc.ernet.in
sujeet.cs...@mnnit.ac.in



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

Re: compactions stuck and restart doesn't kill it

[Surbhi Gupta]

Once you restart a node compaction will start automatically, if u dont want
to do so then do
nodetool disableautocompaction as soon as node is started .

On 8 August 2016 at 07:22, John Wong  wrote:

> Hi
>
> We have a compaction stuck. No progress ever made.
>
> nodetool compactionstats
> pending tasks: 2
>   compaction typekeyspace   column family
> completed   total  unit  progress
>Compactionmy_columnfamily   0
> 410383527 bytes 0.00%
> Active compaction remaining time :   0h00m06s
>
> I tried nodetool stop compactions --COMPACTION but that didn't do anything.
>
> We also tried multiple node restarts, but a few mins later compactions
> will come back. What can we do?
>
> We are still at version 1.2.19 unfortunately (we have plan to move
> forward...)
>
> Thanks.
>
> John
>

setting ttl using timestamp field of record during insertion in cassandra

[Sujeet Kumar]

Hi


I want to set TTL 30 days from the time field of table record(id,name,time)
during insertion.For this I am creating a User Defined Function(UDF) bigint
fun(rTime,cTime) as



CREATE FUNCTION fun(rtime timestamp,ctime timestamp) CALLED ON NULL INPUT
RETURNS bigint LANGUAGE java as 'return 2592000-((ctime.toTime()
-rtime.toTime())/1000);';



here,function fun is calculating the time in seconds this data should live.

2592000 is the time in seconds for 30 days.



I am trying to use above function for setting TTL as



INSERT INTO record(id,name,time) VALUES (123,'data123','2016-08-08
06:06:00') USING TTL fun('2016-08-08 06:06:00',totimestamp(now()));



getting error as

Syntax Exception: ErrorMessage code=2000 



Is there any other way to set ttl based on record time field. What is
problem with above approach.


Thanks.


Sujeet

-- 

-- 
Sujeet Kumar
ME(CSA)
IISc Banglore
sujeet.ku...@csa.iisc.ernet.in
sujeet.cs...@mnnit.ac.in

Re: Doing rolling upgrades on two DCs

[Timon Wong]

Thank you!
We would do a patch level upgrade, from 2.2.1 to 2.2.7 (the latest stable).
Besides, we want to adjust both `commitlog_directory` and
`data_file_directories` settings (separate them into two better IOPS
volumes), and we assumed they should be done before or after upgrade?

On Sun, Aug 7, 2016 at 5:38 AM, Forkalsrud, Erik  wrote:

>
> You didn't mention which version you're upgrading to, but generally you
> shouldn't add or
> remove nodes while the nodes your cluster is not all on on the same
> version. So if you're
> going to add nodes, you should do that before you upgrade the first node,
> or after you have
> upgraded all of them.
>
> Shifting traffic away from DC1 may not be necessary, but shouldn't hurt.
>
>
> - EF
>
> _
> From: Timon Wong [timon86.w...@gmail.com]
> Sent: Saturday, August 6, 2016 02:03
> To: user@cassandra.apache.org
> Subject: Doing rolling upgrades on two DCs
>
> Hello, newbie here.
>
> Currently we are using Cassandra v2.2.1 (two DCs, naming them DC1, DC2,
> respectively), and recently we have two issues:
>
> 1. Cannot adding new node due to CASSANDRA-10961
> 2. Cannot repair node, due to CASSANDRA-10501
>
> We learned that we can do rolling upgrade, since we are only using one DC1
> for now (DC2 is just for replica), we have following steps in mind:
>
> 1. Backup;
> 2. Doing upgrade on DC2. (Follows this guide:
> https://docs.datastax.com/en/latest-upgrade/upgrade/cassandra/
> upgrdCassandraDetails.html);
> 3. Do some checks (client compatibility, etc);
> 4. Adding nodes to DC2 (using new Cassandra);
> 5. Traffic our loads to DC2;
> 6. Upgrading DC1.
>
> Is that possible, or do we miss something?
>
> Thanks in advance!
>
> Timon
>

compactions stuck and restart doesn't kill it

[John Wong]

Hi

We have a compaction stuck. No progress ever made.

nodetool compactionstats
pending tasks: 2
  compaction typekeyspace   column family
completed   total  unit  progress
   Compactionmy_columnfamily   0
410383527 bytes 0.00%
Active compaction remaining time :   0h00m06s

I tried nodetool stop compactions --COMPACTION but that didn't do anything.

We also tried multiple node restarts, but a few mins later compactions will
come back. What can we do?

We are still at version 1.2.19 unfortunately (we have plan to move
forward...)

Thanks.

John

Re: a solution of getting cassandra cross-datacenter latency at a certain time

[Chris Lohfink]

If you invoke the values operation on the mbean every minute (or whatever
period) you can get a histogram of the cross dc the latencies. Just keep
track of the values of each bin in histogram and look at the delta from
previous time to the current time to find how many latencies occurred in
each bins range during the period.

Also can wait for CASSANDRA-11752
 for the a "recent"
histogram (although would need to apply it to this histogram as well).

Chris Lohfink

On Mon, Aug 8, 2016 at 8:50 AM, Ryan Svihla  wrote:

> The first issue I can think of is the Latency table, if I understand you
> correctly, has an unbounded size for the partition key of DC and will over
> time just get larger as more measurements are recorded.
>
> Regards,
>
> Ryan Svihla
>
> On Aug 8, 2016, at 2:58 AM, Stone Fang  wrote:
>
> *objective*:get cassandra cross-datacenter latency in time
>
> *existing ticket:*
>
> there is a ticket [track cross-datacenter latency](https://issues.
> apache.org/jira/browse/CASSANDRA-11569)
> but it is a statistics value from node starting,i want to get the
> instantaneous value in a certain time.
>
> *thought*
>
> want to write a message into **MESSAGE TABLE** in 1s timer task(the period
> is similar to most of cross datacenter latency )
> ,and replicate to other datacenter,there will be a delay.and I capture
> it,and write to **LATENCY TABLE**.i can query the latency value from this
> table with the condition of certain time.
>
> *schema*
>
> message table for replicating data cross datacenter
>
>
> create keyspace heartbeat with replication=
> {'class':'NetworkTopologyStrategy','dc1':1, 'dc2':1...};
>
>
>
>  CREATE TABLE HEARTBEAT.MESSAGE{
> CREATED TIMESTAMP,
> FROMDC VARCHAR,
> PRIMARY KEY(CREATED,FROMDC)
> }
>
> latency Table for querying latency value
>
>  CREATE TABLE SYSTEM.LATENCY{
>  FROMDC VARCHAR,
>  ARRIVED TIMESTAMP,
>  CREATED TIMESTAMP,
>  LANTENCY BIGINT
>  PRIMARY KEY(FROMDC,ARRIVED)
> }WITH CLUSTERING ORDER BY(ARRIVED DESC);
>
> problems
>
> 1.can this solution work to get the cross-datacenter latency?
>
>
> 2.create heartbeat keyspace in cassandra bootstrap process,i need to load
> Heartbeat keyspace in Scheam.java.and save this keyspace into SystemSchema.
> also need to check if this keyspace has exist after first node start.so i
> think this is not a good solution.
>
> 3.compared to 1,try another solution.generate heartbeat message in a
> standalone jar.but always i need to capture heartbeat message mutation in
> cassandra.so i need to check if the mutation is about heartbeat message.and
> it seems strange that check the heartbeat keyspace which is not defined in
> cassandra,but third-party.
>
> hope to see your thought on this.
> thanks
> stone
>
>

Re: a solution of getting cassandra cross-datacenter latency at a certain time

[Ryan Svihla]

The first issue I can think of is the Latency table, if I understand you 
correctly, has an unbounded size for the partition key of DC and will over time 
just get larger as more measurements are recorded.

Regards,

Ryan Svihla

> On Aug 8, 2016, at 2:58 AM, Stone Fang  wrote:
> 
> objective:get cassandra cross-datacenter latency in time
> 
> existing ticket:
> 
> there is a ticket [track cross-datacenter 
> latency](https://issues.apache.org/jira/browse/CASSANDRA-11569)
> but it is a statistics value from node starting,i want to get the 
> instantaneous value in a certain time.
> 
> thought 
> 
> want to write a message into **MESSAGE TABLE** in 1s timer task(the period is 
> similar to most of cross datacenter latency ) 
> ,and replicate to other datacenter,there will be a delay.and I capture it,and 
> write to **LATENCY TABLE**.i can query the latency value from this table with 
> the condition of certain time.
> 
> schema
> 
> message table for replicating data cross datacenter
>
> 
> create keyspace heartbeat with replication=
> {'class':'NetworkTopologyStrategy','dc1':1, 'dc2':1...};
> 
>
> 
>  CREATE TABLE HEARTBEAT.MESSAGE{
> CREATED TIMESTAMP,
> FROMDC VARCHAR,
> PRIMARY KEY(CREATED,FROMDC)
> }
> 
> latency Table for querying latency value  
> 
>  CREATE TABLE SYSTEM.LATENCY{
>  FROMDC VARCHAR,
>  ARRIVED TIMESTAMP,
>  CREATED TIMESTAMP,
>  LANTENCY BIGINT 
>  PRIMARY KEY(FROMDC,ARRIVED)
> }WITH CLUSTERING ORDER BY(ARRIVED DESC);
> 
> problems
> 
> 1.can this solution work to get the cross-datacenter latency?
> 
> 
> 2.create heartbeat keyspace in cassandra bootstrap process,i need to load 
> Heartbeat keyspace in Scheam.java.and save this keyspace into SystemSchema.
> also need to check if this keyspace has exist after first node start.so i
> think this is not a good solution.
> 
> 3.compared to 1,try another solution.generate heartbeat message in a 
> standalone jar.but always i need to capture heartbeat message mutation in 
> cassandra.so i need to check if the mutation is about heartbeat message.and 
> it seems strange that check the heartbeat keyspace which is not defined in 
> cassandra,but third-party.
> 
> hope to see your thought on this.
> thanks
> stone
> 

Verify cassandra backup and restore in C * 2.1

[INDRANIL BASU]

Hello,
   How can we verify that cassandra data is backed up and restored correctly?

-- IB


  

Re: Stale value appears after consecutive TRUNCATE

[horschi]

Hi Yuji,

can you reproduce the behaviour with a single node?

The reason I ask is because I probably have the same issue with my
automated tests (which run truncate between every test), which run on my
local laptop.

Maybe around 5 tests randomly fail out of my 1800. I can see that the
failed tests sometimes show data from other tests, which I think must be
because of a failed truncate. This behaviour is seems to be CQL only, or at
least has gotten worse with CQL. I did not experience this with Thrift.

regards,
Christian



On Mon, Aug 8, 2016 at 7:34 AM, Yuji Ito  wrote:

> Hi all,
>
> I have a question about clearing table and commit log replay.
> After some tables were truncated consecutively, I got some stale values.
> This problem doesn't occur when I clear keyspaces with DROP (and CREATE).
>
> I'm testing the following test with node failure.
> Some stale values appear at checking phase.
>
> Test iteration:
> 1. initialize tables as below
> 2. request a lot of read/write concurrently
> 3. check all records
> 4. repeat from the beginning
>
> I use C* 2.2.6. There are 3 nodes (replication_factor: 3).
> Each node kills cassandra process at random intervals and restarts it
> immediately.
>
> My initialization:
> 1. clear tables with TRUNCATE
> 2. INSERT initial records
> 3. check if all values are correct
>
> If any phase fails (because of node failure), the initialization starts
> all over again.
> So, tables are sometimes truncated consecutively.
> Though the check in the initialization is OK, stale data appears when I
> execute "SELECT * FROM mykeyspace.mytable;" after a lot of requests are
> completed.
>
> The problem is likely to occur when the ReplayPosition's value in
> "truncated_at" is initialized as below after an empty table is truncated.
>
> Column Family ID: truncated_at
> ----: 0x
> 0156597cd4c7
> (this value was acquired just after phase 1 in my initialization)
>
> I guess some unexpected replays occur.
> Does anyone know the behavior?
>
> Thanks,
> Yuji
>

Spark/Cassandra/Zeppelin integration: How to

[DuyHai Doan]

Hello all

For those who are struggling to integrate those 3 stacks (and I know there
are quite a few, given the number of questions I got on this topic), I have
written a blog post to explain the steps to make it work:
http://www.doanduyhai.com/blog/?p=2325


Optionally I have pre-built some Zeppelin binaries for different versions
of Spark/Datastax Enterprise to make your life easier, just pick the right
version for you:
http://www.doanduyhai.com/blog/?p=2325#zeppelin_custom_builds


Duy Hai DOAN

a solution of getting cassandra cross-datacenter latency at a certain time

[Stone Fang]

*objective*:get cassandra cross-datacenter latency in time

*existing ticket:*

there is a ticket [track cross-datacenter latency](
https://issues.apache.org/jira/browse/CASSANDRA-11569)
but it is a statistics value from node starting,i want to get the
instantaneous value in a certain time.

*thought*

want to write a message into **MESSAGE TABLE** in 1s timer task(the period
is similar to most of cross datacenter latency )
,and replicate to other datacenter,there will be a delay.and I capture
it,and write to **LATENCY TABLE**.i can query the latency value from this
table with the condition of certain time.

*schema*

message table for replicating data cross datacenter


create keyspace heartbeat with replication=
{'class':'NetworkTopologyStrategy','dc1':1, 'dc2':1...};



 CREATE TABLE HEARTBEAT.MESSAGE{
CREATED TIMESTAMP,
FROMDC VARCHAR,
PRIMARY KEY(CREATED,FROMDC)
}

latency Table for querying latency value

 CREATE TABLE SYSTEM.LATENCY{
 FROMDC VARCHAR,
 ARRIVED TIMESTAMP,
 CREATED TIMESTAMP,
 LANTENCY BIGINT
 PRIMARY KEY(FROMDC,ARRIVED)
}WITH CLUSTERING ORDER BY(ARRIVED DESC);

problems

1.can this solution work to get the cross-datacenter latency?


2.create heartbeat keyspace in cassandra bootstrap process,i need to load
Heartbeat keyspace in Scheam.java.and save this keyspace into SystemSchema.
also need to check if this keyspace has exist after first node start.so i
think this is not a good solution.

3.compared to 1,try another solution.generate heartbeat message in a
standalone jar.but always i need to capture heartbeat message mutation in
cassandra.so i need to check if the mutation is about heartbeat message.and
it seems strange that check the heartbeat keyspace which is not defined in
cassandra,but third-party.

hope to see your thought on this.
thanks
stone