Reason for Trace Message Drop

[Varun Barala]

Hi all,

Can anyone tell me that what are all possible reasons for below log:-


*"INFO  [ScheduledTasks:1] 2016-06-14 06:27:39,498
MessagingService.java:929 - _TRACE messages were dropped in last 5000 ms:
928 for internal timeout and 0 for cross node timeout".*
I searched online for the same and found some reasons like:-

* Disk is not able to keep up with your ingest
* Resources are not able to support all parallel running tasks
* If other nodes are down then due to large hint replay
* Heavy workload

But in this case other kind of messages (mutation, read, write etc)  should
be dropped by *C** but It doesn't happen.

-
Cluster Specifications
--
number of nodes = 1
total number of CF = 2000

-
Machine Specifications
--
RAM 30 GB
hard disk SSD
ubuntu 14.04


Thanks in advance!!

Regards,
Varun Barala

Re: Reason for Trace Message Drop

[Varun Barala]

Thanks!! Eric Stevens for your reply.

We have following JVM settings :-
-
*memtable_offheap_space_in_mb: 15360  (*found in casandra.yaml
*)*
*MAX_HEAP_SIZE="16G"  (*found in cassandra-env.sh
*)*-

And I also found big GC in log. But messages and big GC were logged at
different-different time in system.log. I was expecting to happen them at
same time after reading your reply. I also manually triggered GC but
messages were not dropped.

is *TRACE message drop *harmful or it's okay we can neglect them?

Thank you!!


On Wed, Jun 15, 2016 at 8:45 PM, Eric Stevens <migh...@gmail.com> wrote:

> This is better kept to the User groups.
>
> What are your JVM memory settings for Cassandra, and have you seen big
> GC's in your logs?
>
> The reason I ask is because that's a large number of column families,
> which produces memory pressure, and at first blush that strikes me as a
> likely cause.
>
>
> On Wed, Jun 15, 2016 at 3:23 AM Varun Barala <varunbaral...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> Can anyone tell me that what are all possible reasons for below log:-
>>
>>
>> *"INFO  [ScheduledTasks:1] 2016-06-14 06:27:39,498
>> MessagingService.java:929 - _TRACE messages were dropped in last 5000 ms:
>> 928 for internal timeout and 0 for cross node timeout".*
>> I searched online for the same and found some reasons like:-
>>
>> * Disk is not able to keep up with your ingest
>> * Resources are not able to support all parallel running tasks
>> * If other nodes are down then due to large hint replay
>> * Heavy workload
>>
>> But in this case other kind of messages (mutation, read, write etc)
>>  should be dropped by *C** but It doesn't happen.
>>
>> -
>> Cluster Specifications
>> --
>> number of nodes = 1
>> total number of CF = 2000
>>
>> -
>> Machine Specifications
>> --
>> RAM 30 GB
>> hard disk SSD
>> ubuntu 14.04
>>
>>
>> Thanks in advance!!
>>
>> Regards,
>> Varun Barala
>>
>

Re: Reason for Trace Message Drop

[Varun Barala]

Thanks Eric Stevens for your detailed reply!!

I got your points.


On Thu, Jun 16, 2016 at 11:49 PM, Eric Stevens <migh...@gmail.com> wrote:

> Are you executing all queries with tracing enabled?  If so that introduces
> overhead you probably don't want.  Most people probably don't see this log
> very often because it's the exception to query with tracing enabled, and
> not the rule (it's a diagnostic thing usually turned on only when
> troubleshooting another problem).
>
> The trace messages being dropped aren't in-and-of-themselves a problem, it
> just means that your traces would be incomplete.  However this indicates a
> cluster that's either unhealthy or getting close to it.
>
> Since you have a single node, run nodetool tpstats (if you had a cluster
> you'd want to do this on each node).  Look at the bottom section, ideally
> it looks like this:
> Message type   Dropped
> READ 0
> RANGE_SLICE  0
> _TRACE   0
> MUTATION 0
> COUNTER_MUTATION 0
> BINARY   0
> REQUEST_RESPONSE 0
> PAGED_RANGE  0
> READ_REPAIR  0
>
> If you're seeing MUTATIONs or COUNTER_MUTATIONs dropped, your data
> integrity is at risk, you need to reduce pressure in some way, as your
> cluster is falling behind.  That might be expanding your node count
> *above* your RF, or it might be tuning your application to reduce read or
> write pressure (note that expanding a cluster involves read pressure on the
> existing nodes, so to successfully expand your cluster you probably *also* 
> need
> to reduce pressure from the application, because the act of growing your
> cluster will cause you to get even further behind).
>
> On a single node, like yours, mutations dropped is data lost.  On a
> cluster with RF > 1, there's just a *chance* that you have lost data (the
> same mutation needs to be dropped by all replicas for it to be
> irrecoverably lost).  Dropped mutations are always bad.
>
> In this scenario you probably also want to double check nodetool
> compactionstats, and ideally you have < 4 or 5 compactions.  Lower is
> always better, 4 or 5 is getting worrying if it remains in that range.  If
> you're above 10, you're falling behind on compaction too, and your read
> performance will be suffering.
>
> Finally RE your JVM settings, with a 30GB node, I think you could turn
> that up a bit if you convert to G1GC.  JVM tuning is definitely not my
> strong point, there are others on this list who will be able to help you do
> a better job of it.  If you're seeing big GC pauses, then you do need to
> work on this, even if they're not the direct cause of the dropped traces.
> With that column family count you'll be under more GC pressure than you
> would be with a lower CF count (there is a fixed memory cost per CF).
> Reconsider your data model, usually this many column families suggests
> dynamically creating CF's (eg to solve multi-tenancy).  If your CF count
> will grow steadily over time at any appreciable rate, that's an
> anti-pattern.
>
> On Thu, Jun 16, 2016 at 2:40 AM Varun Barala <varunbaral...@gmail.com>
> wrote:
>
>> Thanks!! Eric Stevens for your reply.
>>
>> We have following JVM settings :-
>> -
>> *memtable_offheap_space_in_mb: 15360  (*found in casandra.yaml
>> *)*
>> *MAX_HEAP_SIZE="16G"  (*found in cassandra-env.sh
>> *)*-
>>
>> And I also found big GC in log. But messages and big GC were logged at
>> different-different time in system.log. I was expecting to happen them at
>> same time after reading your reply. I also manually triggered GC but
>> messages were not dropped.
>>
>> is *TRACE message drop *harmful or it's okay we can neglect them?
>>
>> Thank you!!
>>
>>
>> On Wed, Jun 15, 2016 at 8:45 PM, Eric Stevens <migh...@gmail.com> wrote:
>>
>>> This is better kept to the User groups.
>>>
>>> What are your JVM memory settings for Cassandra, and have you seen big
>>> GC's in your logs?
>>>
>>> The reason I ask is because that's a large number of column families,
>>> which produces memory pressure, and at first blush that strikes me as a
>>> likely cause.
>>>
>>>
>>> On Wed, Jun 15, 2016 at 3:23 AM Varun Barala <varunbaral...@gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> Can anyone tell me that what are all possible reasons for below log:-
>>>>
>>>&g

Re: Read data from specific node in cassandra

[Varun Barala]

Hi Siddharth Verma,

You can define consistency level LOCAL_ONE.

and you can applyh consistency level during statement creation.

like this -> statement.setConsistencyLevel(ConsistencyLevel.LOCAL_ONE);

On Thu, May 5, 2016 at 3:55 PM, Siddharth Verma <
verma.siddha...@snapdeal.com> wrote:

> Hi,
> We have a 3 node cluster in DC1, where replication factor of keyspace is 3.
> How can i read data only from one particular node in java driver?
>
> Thanks,
> Siddharth Verma
>

regarding drain process

[Varun Barala]

Hi all,


Recently I'm facing a problem with cassandra nodes. Nodes go down very
frequently.
I went through system.log and found the reason that somehow c*
triggers *draining
process.*

I know the purpose of *nodetool drain *but it should not trigger
automatically, right ?

or is there any specific settings for the same ?


*we are using C*-2.1.13.*

please let me know if you need more info.

Thanking you!!

Regards,
Varun Barala

(C)* stable version after 3.5

[Varun Barala]

Hi all users,


Currently we are using cassandra-2.1.13 but we want to upgrade *2.1.13 to
3.x* in production.

Could anyone please tell me which is the most stable *cassandra version
after 3.5*.

Thanking You!!


Regards,
Varun Barala

Re: (C)* stable version after 3.5

[Varun Barala]

Thanks everyone for your detailed explanation!!

We are going to upgrade from 2.1 to 3.0.8 as many people agreed that It's
safe skipping measure version 2.2.

Could anyone please tell me which *cassandra java driver* will be
compatible with 3.0.8 ?
Currently we are using *2.1.7.1 driver version* So If we upgrade to newer
version then will there be any backward-compatibility issues ?

Thanking You!!

On Thu, Jul 14, 2016 at 6:04 PM, Romain Hardouin 
wrote:

> DSE 4.8 uses C* 2.1 and DSE 5.0 uses C* 3.0. So I would say that 2.1->3.0
> is more tested by DataStax than 2.2->3.0.
>
>
> Le Jeudi 14 juillet 2016 11h37, Stefano Ortolani  a
> écrit :
>
>
> FWIW, I've recently upgraded from 2.1 to 3.0 without issues of any sort,
> but admittedly I haven't been using anything too fancy.
>
> Cheers,
> Stefano
>
> On Wed, Jul 13, 2016 at 10:28 PM, Alain RODRIGUEZ 
> wrote:
>
> Hi Anuj
>
> From
> https://docs.datastax.com/en/latest-upgrade/upgrade/cassandra/upgrdBestPractCassandra.html
> :
>
>
>- Employ a continual upgrade strategy for each year. Upgrades are
>impacted by the version you are upgrading from and the version you are
>upgrading to. The greater the gap between the current version and the
>target version, the more complex the upgrade.
>
>
> And I could not find it but historically I am quite sure it was explicitly
> recommended not to skip a major update (for a rolling upgrade), even if I
> could not find it. Anyway it is clear that the bigger the gap is, the more
> careful we need to be.
>
> On the other hand, I see 2.2 as a 2.1 + some feature but no real breaking
> changes (as 3.0 was already on the pipe) and doing a 2.2 was decided
> because 3.0 was taking a long time to be released and some feature were
> ready for a while.
>
> I might be wrong on some stuff above, but one can only speak with his
> knowledge and from his point of view. So I ended up saying:
>
> Also I am not sure if the 2.2 major version is something you can skip
> while upgrading through a rolling restart. I believe you can, but it is not
> what is recommended.
>
>
> Note that "I am not sure", "I believe you can"... So it was more a
> thought, something to explore for Varun :-).
>
> And I actually encouraged him to move forward. Now that Tyler Hobbs
> confirmed it works, you can put a lot more trust on the fact that this
> upgrade will work :-). I would still encourage people to test it (for
> client compatibility, corner cases due to models, ...).
>
> I hope I am more clear now,
>
> C*heers,
> ---
> Alain Rodriguez - al...@thelastpickle.com
> France
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> 2016-07-13 18:39 GMT+02:00 Tyler Hobbs :
>
>
> On Wed, Jul 13, 2016 at 11:32 AM, Anuj Wadehra 
> wrote:
>
> Why do you think that skipping 2.2 is not recommended when NEWS.txt
> suggests otherwise? Can you elaborate?
>
>
> We test upgrading from 2.1 -> 3.x and upgrading from 2.2 -> 3.x
> equivalently.  There should not be a difference in terms of how well the
> upgrade is supported.
>
>
> --
> Tyler Hobbs
> DataStax 
>
>
>
>
>
>

Fwd: Counter global logical clock

[Varun Barala]

Hi all Users,

Recently I was restoring some sstables by changing their timestamp.
But in case of counter It's different there are few things I'd like to ask:-

* What is the maximum value of global logical clock? *Long.MAX_VALUE* ?
** what will happen If updates hit this MAX_VALUE value ?*

I read rule of merging of counter columns:-
- global + global = keep the shard with the highest logical clock
- global + local  = keep the global one
- global + remote = keep the global one
- local  + local  = sum counts (and logical clocks)
- local  + remote = keep the local one
- remote + remote = keep the shard with the highest logical clock

So in my case ::



** Can I change to 'set as global and set logical clock to maximum to make
sure it always win when merge' ? * How can we modify a counter column so it
will always win when merge?*

I'm using *ka* sstable version.

Thanking You!!

Regards,
Varun Barala

Re: regarding drain process

[Varun Barala]

Jeff Jirsa thanks for your reply!!

We are not using any chef/puppet and It happens only at one node other
nodes are working fine.
And all machines are using same AMI image.

Did anybody face such situation or have any suggestions ?

Thanking you.

On Wed, Jul 27, 2016 at 10:27 PM, Jeff Jirsa <jeff.ji...@crowdstrike.com>
wrote:

> Are you running chef/puppet or similar?
>
>
>
> *From: *Varun Barala <varunbaral...@gmail.com>
> *Reply-To: *"user@cassandra.apache.org" <user@cassandra.apache.org>
> *Date: *Tuesday, July 26, 2016 at 10:15 PM
> *To: *"user@cassandra.apache.org" <user@cassandra.apache.org>
> *Subject: *regarding drain process
>
>
>
> Hi all,
>
>
> Recently I'm facing a problem with cassandra nodes. Nodes go down very
> frequently.
> I went through system.log and found the reason that somehow c* triggers 
> *draining
> process.*
>
> I know the purpose of *nodetool drain *but it should not trigger
> automatically, right ?
>
> or is there any specific settings for the same ?
>
>
> *we are using C*-2.1.13.*
>
> please let me know if you need more info.
>
> Thanking you!!
>
> Regards,
> Varun Barala
>

Re: CQL datatype for long?

[Varun Barala]

 use `bigint` for long.


Regards,
Varun Barala

On Thu, Dec 8, 2016 at 10:32 AM, Check Peck <comptechge...@gmail.com> wrote:

> What is the CQL data type I should use for long? I have to create a column
> with long data type. Cassandra version is 2.0.10.
>
> CREATE TABLE storage (
>   key text,
>   clientid int,
>   deviceid long, // this is wrong I guess as I don't see long in CQL?
>   PRIMARY KEY (topic, partition)
> );
>
> I need to have "deviceid" as long data type. Bcoz I am getting deviceid as
> long and that's how I want to store it.
>

Re: CQL datatype for long?

[Varun Barala]

yes!!

On Thu, Dec 8, 2016 at 1:21 PM, Check Peck <comptechge...@gmail.com> wrote:

> And then from datastax java driver, I can use. Am I right?
>
> To Read:
> row.getLong();
>
> To write
> boundStatement.setLong()
>
>
> On Wed, Dec 7, 2016 at 6:50 PM, Varun Barala <varunbaral...@gmail.com>
> wrote:
>
>>  use `bigint` for long.
>>
>>
>> Regards,
>> Varun Barala
>>
>> On Thu, Dec 8, 2016 at 10:32 AM, Check Peck <comptechge...@gmail.com>
>> wrote:
>>
>>> What is the CQL data type I should use for long? I have to create a
>>> column with long data type. Cassandra version is 2.0.10.
>>>
>>> CREATE TABLE storage (
>>>   key text,
>>>   clientid int,
>>>   deviceid long, // this is wrong I guess as I don't see long in CQL?
>>>   PRIMARY KEY (topic, partition)
>>> );
>>>
>>> I need to have "deviceid" as long data type. Bcoz I am getting deviceid
>>> as long and that's how I want to store it.
>>>
>>
>>
>

Re: Cassandra crashed with OOM, and the system.log and debug.log doesn't match.

[Varun Barala]

Hi,


*How many column families are there? What is the heap size?*

You can turn off logs for statusLogger.java and gc to optimize heap usage.

Can you also monitor cpu usage and memory usage? IMO, in your case memory
is the bottle-neck.

Thanks!!

On Mon, Jul 10, 2017 at 5:07 PM, 张强  wrote:

> Hi experts, I've a single cassandra 3.11.0 node working with kairosdb (a
> time series database), after running 4 days with stable workload, the
> database client start to get "request errors", but there are not a lot of
> error or warning messages in the cassandra log file, the client start to
> receive error message at about 7-7 21:03:00, and kairosdb keep retrying
> after that time, but there isn't much logs in the cassandra log file.
> I've noticed the abnormal status at about 7-8 16:00:00, then I've typed a
> "nodetool tablestats" command to get some information, the command got an
> error, and while that time, the cassandra process start to crash, and
> generated a dump file.
> After C* shutdown, I take the logs to see what happened, and I found
> something strange inside the logs.
>
> 1. In the system.log, there are two lines shows that no logs between
> 2017-07-07 21:03:50 to 2017-07-08 16:07:33, I think that is a pretty long
> period without any logs, and in gc.log file, there are a lot of logs shows
> long time GC, that should be logged in system.log.
> INFO  [ReadStage-1] 2017-07-07 21:03:50,824 NoSpamLogger.java:91 - Maximum
> memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB
> WARN  [PERIODIC-COMMIT-LOG-SYNCER] 2017-07-08 16:07:33,347
> NoSpamLogger.java:94 - Out of 1 commit log syncs over the past 0.00s with
> average duration of 60367.73ms, 1 have exceeded the configured commit
> interval by an average of 50367.73ms
>
> 2. In the system.log, there is a log shows very long time GC, and then the
> C* start to close.
> WARN  [ScheduledTasks:1] 2017-07-08 16:07:46,846 NoSpamLogger.java:94 -
> Some operations timed out, details available at debug level (debug.log)
> WARN  [Service Thread] 2017-07-08 16:10:36,114 GCInspector.java:282 -
> ConcurrentMarkSweep GC in 688850ms.  CMS Old Gen: 2114938312 -> 469583832;
> Par Eden Space: 837584 -> 305319752; Par Survivor Space: 41943040 ->
> 25784008
> ..
> ERROR [Thrift:22] 2017-07-08 16:10:56,322 CassandraDaemon.java:228 -
> Exception in thread Thread[Thrift:22,5,main]
> java.lang.OutOfMemoryError: Java heap space
>
> 3. In the debug.log, the last INFO level log is at 2017-07-07 14:43:59,
> the log is:
> INFO  [IndexSummaryManager:1] 2017-07-07 14:43:59,967
> IndexSummaryRedistribution.java:75 - Redistributing index summaries
> After that, there are DEBUG level logs until 2017-07-07 21:11:34, but no
> more INFO level or other level logs in that log file, while there are still
> many logs in the system.log after 2017-07-07 14:43:59. Why doesn't these
> two log files match?
>
> My hardware is 4 core cpu and 12G ram, and I'm using windows server 2012
> r2.
> Do you know what's going on here? and is there anything I can do to
> prevent that situation?
>
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>

Re: Help in c* Data modelling

[Varun Barala]

Hi,

You can create pseudo index table.

IMO, structure can be:-


CREATE TABLE IF NOT EXISTS test.user ( account_id bigint, pid bigint,
disp_name text, status int, PRIMARY KEY (account_id, pid) ) WITH
CLUSTERING ORDER BY (pid ASC);
CREATE TABLE IF NOT EXISTS test.user_index ( account_id bigint, pid
bigint, disp_name text, status int, PRIMARY KEY ((account_id, status),
disp_name) ) WITH CLUSTERING ORDER BY (disp_name ASC);


to support query *:-  select * from site24x7.wm_current_status where uid=1
order by dispName asc;*
You can use *in condition* on last partition key *status *in table

*test.user_index.*



*It depends on your use case and amount of data as well. It can be
optimized more...*
Thanks!!

On Sun, Jul 23, 2017 at 2:48 AM, techpyaasa .  wrote:

> Hi ,
>
> We have a table like below :
>
> CREATE TABLE ks.cf ( accountId bigint, pid bigint, dispName text, status
>> int, PRIMARY KEY (accountId, pid) ) WITH CLUSTERING ORDER BY (pid ASC);
>
>
>
> We would like to have following queries possible on the above table:
>
> select * from site24x7.wm_current_status where uid=1 and mid=1;
> select * from site24x7.wm_current_status where uid=1 order by dispName asc;
> select * from site24x7.wm_current_status where uid=1 and status=0 order by
> dispName asc;
>
> I know first query is possible by default , but I want the last 2 queries
> also to work.
>
> So can some one please let me know how can I achieve the same in
> cassandra(c*-2.1.17). I'm ok with applying indexes etc,
>
> Thanks
> TechPyaasa
>

Materialize View in production

[Varun Barala]

Hi everyone,


We were thinking to use cassandra's materialize view. We went through the
performance benchmark which seems reasonable to us.
http://www.datastax.com/dev/blog/materialized-view-performance-in-cassandra-3-x

But we'd like to hear from user community about their opinions.




** About stability* About bugs* About performance (cluster size or DCs)*
Could you please help us to make the decision.

Thanking you!!


Regards,
Varun Barala

Re: Alter table gc_grace_seconds

[Varun Barala]

* Which C* version are you using?
* How many nodes are there in this cluster?

These tombstones will not be deleted if they are not older than
gc_grace_seconds.


On Sun, Oct 1, 2017 at 10:14 PM, Gábor Auth <auth.ga...@gmail.com> wrote:

> Hi,
>
> On Sun, Oct 1, 2017 at 3:44 PM Varun Barala <varunbaral...@gmail.com>
> wrote:
>
>> This is the property of table and It's not written in sstables. If you
>> change gc_grace, It'll get applied for all the data.
>>
>
> Hm... I've migrated lot of data from `number_item` to `measurement` CF
> because of scheme upgrade. During the migration, the script created rows in
> the `measurement` CF and deleted the migrated rows in the `number_item` CF,
> one-by-one.
>
> I've just take a look on the sstables of `number_item` and it is full of
> deleted rows:
> {
>   "type" : "row",
>   "position" : 146160,
>   "clustering" : [ "humidity", "97781fd0-9dab-11e7-a3d5-7f6ef9a844c7" ],
>   "deletion_info" : { "marked_deleted" : "2017-09-25T11:51:19.165276Z",
> "local_delete_time" : "2017-09-25T11:51:19Z" },
>   "cells" : [ ]
> }
>
> How can I purge these old rows? :)
>
> I've tried: compact, scrub, cleanup, clearsnapshot, flush and full repair.
>
> Bye,
> Gábor Auth
>
>

Re: Alter table gc_grace_seconds

[Varun Barala]

Hi,
This is the property of table and It's not written in sstables. If you
change gc_grace, It'll get applied for all the data. Thanks!!

C* stores this info inside schema_columnfamilies

Regards,
Varun Barala

On Sun, Oct 1, 2017 at 9:23 PM, Gábor Auth <auth.ga...@gmail.com> wrote:

> Hi,
>
> The `alter table number_item with gc_grace_seconds = 3600;` is sets the
> grace seconds of tombstones of the future modification of number_item
> column family or affects all existing data?
>
> Bye,
> Gábor Auth
>
>

Re: Alter table gc_grace_seconds

[Varun Barala]

* You should not try on real clusters directly.

Did you change gc_grace for all column families?

> But not in the `number_item` CF... :(
Could you please explain?

Thanks!!

On Mon, Oct 2, 2017 at 2:24 AM, Gábor Auth <auth.ga...@gmail.com> wrote:

> Hi,
>
> On Sun, Oct 1, 2017 at 7:44 PM Varun Barala <varunbaral...@gmail.com>
> wrote:
>
>> Sorry If I misunderstood the situation.
>>
>
> Ok, I'm confused... :/
>
> I've just tested it on the same cluster and the compact removed the
> marked_deleted rows. But not in the `number_item` CF... :(
>
> Cassandra 3.11.0, two DC (with 4-4 nodes).
>
> Bye,
> Gábor Auth
>

Re: Alter table gc_grace_seconds

[Varun Barala]

Sorry If I misunderstood the situation.


{
  "type" : "row",
  "position" : 146160,
  "clustering" : [ "humidity", "97781fd0-9dab-11e7-a3d5-7f6ef9a844c7" ],
  "deletion_info" : { "marked_deleted" : "2017-09-25T11:51:19.165276Z",
"local_delete_time" : "2017-09-25T11:51:19Z" },
  "cells" : [ ]
}


How did we come to know that TTL is applied by looking at this data?


I performed a basic test in my local:-


*# C* version* 3.0.14,

*# Table schema:-*
CREATE TABLE test.table1 (
id int,
ck1 text,
ck2 text,
nk1 text,
PRIMARY KEY (id,ck1, ck2)
);


*# insert statement:-*
insert into test.table1 (id, ck1, ck2, nk1) VALUES (1,'1','1','1');


*# performed flush to generate the sstable(json format):-*
  {
"partition" : {
  "key" : [ "1" ],
  "position" : 0
},
"rows" : [
  {
"type" : "row",
"position" : 30,
"clustering" : [ "1", "1" ],
"liveness_info" : { "tstamp" : "2017-10-01T17:28:56.889Z" },
"cells" : [
  { "name" : "nk1", "value" : "1" }
]
  }
]
  }

*# delete statement:-*
delete from test.table1 where id=1 and ck1='1' and ck2='1';

*# performed flush to generate new sstable(json format):-*
 {
"partition" : {
  "key" : [ "1" ],
  "position" : 0
},
"rows" : [
  {
"type" : "row",
"position" : 32,
"clustering" : [ "1", "1" ],
"deletion_info" : { "marked_deleted" : "2017-10-01T17:30:15.397Z",
"local_delete_time" : "2017-10-01T17:30:15Z" },
"cells" : [ ]
  }
]
  }


*# performed compaction to compact above two sstables (latest won,
tombstone has not been purged):-*  {
"partition" : {
  "key" : [ "1" ],
  "position" : 0
},
"rows" : [
  {
"type" : "row",
"position" : 32,
"clustering" : [ "1", "1" ],
"deletion_info" : { "marked_deleted" : "2017-10-01T17:30:15.397Z",
"local_delete_time" : "2017-10-01T17:30:15Z" },
"cells" : [ ]
  }
]
  }

*# changed gc_grace:-*
alter TABLE  test.table1 WITH gc_grace_seconds = 0;

*# performed compaction again(no sstable exists, since it has purged the
tombstones older than gc_grace):-*
no sstable exist in data dir


Please let me know If I'm missing some fact.

Thanks!!

On Mon, Oct 2, 2017 at 1:11 AM, Gábor Auth  wrote:

> Hi,
>
> On Sun, Oct 1, 2017 at 6:53 PM Jonathan Haddad  wrote:
>
>> The TTL is applied to the cells on insert. Changing it doesn't change the
>> TTL on data that was inserted previously.
>>
>
> Is there any way to purge out these tombstoned data?
>
> Bye,
> Gábor Auth
>

Re: Alter table gc_grace_seconds

[Varun Barala]

+1 Justing,

Might be possible C* is not compacting the sstables [
https://stackoverflow.com/questions/28437301/cassandra-not-compacting-sstables
].

You can this fact by doing below procedure:-

*Run this before compaction:-*
ls /var/lib/cassandra/data/mat/number_item-*/

Store result to some file [just to keep track of sstables]


*Run compaction:-*nodetool compact mat  number_item

*List directories again:-*
ls /var/lib/cassandra/data/mat/number_item-*/

Now check both the list results. If they have some common sstables then we
can say that C* is not compacting sstables.

Thanks!!

On Mon, Oct 2, 2017 at 2:32 PM, Justin Cameron <jus...@instaclustr.com>
wrote:

> >> * You should not try on real clusters directly.
> >Why not? :)
>
> It's highly recommended that you complete a full repair before the GC
> grace period expires, otherwise it's possible you could experience zombie
> data (i.e. data that was previously deleted coming back to life)
>
> See http://thelastpickle.com/blog/2016/07/27/about-deletes-
> and-tombstones.html for a good overview of the problem
>
>
> On Mon, 2 Oct 2017 at 16:51 Gábor Auth <auth.ga...@gmail.com> wrote:
>
>> Hi,
>>
>> On Mon, Oct 2, 2017 at 5:55 AM Varun Barala <varunbaral...@gmail.com>
>> wrote:
>>
>>> *select gc_grace_seconds from system_schema.tables where keyspace_name =
>>> 'keyspace' and table_name = 'number_item;*
>>>
>>
>> cassandra@cqlsh:mat> DESCRIBE TABLE mat.number_item;
>>
>>
>> CREATE TABLE mat.number_item (
>>nodeid uuid,
>>type text,
>>created timeuuid,
>>value float,
>>PRIMARY KEY (nodeid, type, created)
>> ) WITH CLUSTERING ORDER BY (type ASC, created ASC)
>>AND bloom_filter_fp_chance = 0.01
>>AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>>AND cdc = false
>>AND comment = ''
>>AND compaction = {'class': 
>> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
>> 'max_threshold': '32', 'min_threshold': '4'}
>>AND compression = {'chunk_length_in_kb': '64', 'class': '
>> org.apache.cassandra.io.compress.LZ4Compressor'}
>>AND crc_check_chance = 1.0
>>AND dclocal_read_repair_chance = 0.1
>>AND default_time_to_live = 0
>>AND gc_grace_seconds = 3600
>>AND max_index_interval = 2048
>>AND memtable_flush_period_in_ms = 0
>>AND min_index_interval = 128
>>AND read_repair_chance = 0.0
>>AND speculative_retry = '99PERCENTILE';
>>
>> cassandra@cqlsh:mat> select gc_grace_seconds from system_schema.tables
>> where keyspace_name = 'mat' and table_name = 'number_item';
>>
>> gc_grace_seconds
>> --
>> 3600
>>
>> (1 rows)
>>
>> Bye,
>> Gábor Auth
>>
> --
>
>
> *Justin Cameron*Senior Software Engineer
>
>
> <https://www.instaclustr.com/>
>
>
> This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
> and Instaclustr Inc (USA).
>
> This email and any attachments may contain confidential and legally
> privileged information.  If you are not the intended recipient, do not copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender and then immediately delete the message.
>

Re: Alter table gc_grace_seconds

[Varun Barala]

Can you share result of query:-

*select gc_grace_seconds from system_schema.tables where keyspace_name =
'keyspace' and table_name = 'number_item;*

Thanks!!

On Mon, Oct 2, 2017 at 3:42 AM, Gábor Auth <auth.ga...@gmail.com> wrote:

> Hi,
>
> On Sun, Oct 1, 2017 at 9:36 PM Varun Barala <varunbaral...@gmail.com>
> wrote:
>
>> * You should not try on real clusters directly.
>>
>
> Why not? :)
>
> Did you change gc_grace for all column families?
>>
>
> Not, only on the `number_item` CF.
>
> > But not in the `number_item` CF... :(
>> Could you please explain?
>>
>
> I've tried the test case that you described and it is works (the compact
> removed the marked_deleted rows) on a newly created CF. But the same
> gc_grace_seconds settings has no effect in the `number_item` CF (millions
> of rows has been deleted during a last week migration).
>
> Bye,
> Gábor Auth
>
>

Re: Alter table gc_grace_seconds

[Varun Barala]

https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsConfigureCompaction.html

As pe STCS:
*This strategy triggers a minor compaction when there are a number of
similar sized SSTables on disk as configured by the table subproperty,
min_threshold.*
In your case you have *min_threshold=4 and 3 Big sstable and one small
sstable.*
Either you can change min_threshold to three in your case or you can change
compaction strategy for this table.

Thanks!!

On Mon, Oct 2, 2017 at 4:48 PM, Steinmaurer, Thomas <
thomas.steinmau...@dynatrace.com> wrote:

> Hello Justin,
>
>
>
> yes, but in real-world, hard to accomplish for high volume column families
> >= 3-digit GB. Even with the default of 10 days grace period, this may get
> a real challenge, to complete a full repair. J
>
>
>
> Possibly back again to the discussion that incremental repair has some
> flaws, full repair (-full option) in 3.0+ (2.2+?) isn’t behaving the same
> way as in 2.1 anymore, due to troubles when kicking off with –pr on several
> nodes at once.
>
>
>
> Regards,
>
> Thomas
>
>
>
> *From:* Justin Cameron [mailto:jus...@instaclustr.com]
> *Sent:* Montag, 02. Oktober 2017 08:32
> *To:* user@cassandra.apache.org
> *Subject:* Re: Alter table gc_grace_seconds
>
>
>
> >> * You should not try on real clusters directly.
>
> >Why not? :)
>
> It's highly recommended that you complete a full repair before the GC
> grace period expires, otherwise it's possible you could experience zombie
> data (i.e. data that was previously deleted coming back to life)
>
> See http://thelastpickle.com/blog/2016/07/27/about-deletes-
> and-tombstones.html for a good overview of the problem
>
>
>
>
>
> On Mon, 2 Oct 2017 at 16:51 Gábor Auth <auth.ga...@gmail.com> wrote:
>
> Hi,
>
> On Mon, Oct 2, 2017 at 5:55 AM Varun Barala <varunbaral...@gmail.com>
> wrote:
>
> *select gc_grace_seconds from system_schema.tables where keyspace_name =
> 'keyspace' and table_name = 'number_item;*
>
>
>
> cassandra@cqlsh:mat> DESCRIBE TABLE mat.number_item;
>
>
> CREATE TABLE mat.number_item (
>nodeid uuid,
>type text,
>created timeuuid,
>value float,
>PRIMARY KEY (nodeid, type, created)
> ) WITH CLUSTERING ORDER BY (type ASC, created ASC)
>AND bloom_filter_fp_chance = 0.01
>AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>AND cdc = false
>AND comment = ''
>AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
>AND compression = {'chunk_length_in_kb': '64', 'class': '
> org.apache.cassandra.io.compress.LZ4Compressor'}
>AND crc_check_chance = 1.0
>AND dclocal_read_repair_chance = 0.1
>AND default_time_to_live = 0
>AND gc_grace_seconds = 3600
>AND max_index_interval = 2048
>AND memtable_flush_period_in_ms = 0
>AND min_index_interval = 128
>AND read_repair_chance = 0.0
>AND speculative_retry = '99PERCENTILE';
>
> cassandra@cqlsh:mat> select gc_grace_seconds from system_schema.tables
> where keyspace_name = 'mat' and table_name = 'number_item';
>
> *gc_grace_seconds*
> --
> *3600*
>
> (1 rows)
>
>
> Bye,
> Gábor Auth
>
> --
>
> *Justin Cameron*
> Senior Software Engineer
>
>
>
> [image: Image removed by sender.] <https://www.instaclustr.com/>
>
>
> This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
> and Instaclustr Inc (USA).
>
> This email and any attachments may contain confidential and legally
> privileged information.  If you are not the intended recipient, do not copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender and then immediately delete the message.
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
> <https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313=gmail=g>
>

Re: Cassandra stress tool - data generation

[Varun Barala]

Hi,

https://www.instaclustr.com/deep-diving-into-cassandra-stress-part-1/

In the blog, They covered many things in detail.

Thanks!!

On Thu, Nov 2, 2017 at 12:38 AM, Lucas Benevides <
lu...@maurobenevides.com.br> wrote:

> Dear community,
>
> I am using Cassandra Stress Tool and trying to simulate IoT generated data.
> So I created a column family with the device_id as the partition key.
>
> But in every different operation (the parameter received in the -n option)
> the generated values are the same. For instance, I have a column called
> observation_time which is supposed to be the time measured by the sensor.
> But in every partition the values are equal.
>
> Is there a way to make those values be randomically generated with
> different seeds? I need this way so that if the same device_id occurs
> again, it makes an INSERT instead of an UPSERT.
>
> To clarify: What is happening now (fictional data):
>
> operation 1
> device 1
> ts1: 01/01/1970
> ts2: 02/01/1980
> ts3: 03/01/1990
>
> operation 2
> device 2
> ts1: 01/01/1970
> ts2: 02/01/1980
> ts3: 03/01/1990
>
> What I want:
> operation1
> device 1
> ts1: 01/01/1970
> ts2: 02/01/1980
> ts3: 03/01/1990
>
> operation2
> device 2
> ts1: 02/01/1971  #Different values here.
> ts2: 05/01/1982
> ts3: 08/01/1993
>
> Thanks in advance,
> Lucas Benevides
>
>

Re: Cassandra stress tool - data generation

[Varun Barala]

https://www.instaclustr.com/deep-diving-cassandra-stress-part-3-using-yaml-profiles/
In this particular blog, they mentioned your case.

Changed uniform() distribution to seq() distribution
https://issues.apache.org/jira/browse/CASSANDRA-12490

Thanks!!


On Thu, Nov 2, 2017 at 12:54 AM, Varun Barala <varunbaral...@gmail.com>
wrote:

> Hi,
>
> https://www.instaclustr.com/deep-diving-into-cassandra-stress-part-1/
>
> In the blog, They covered many things in detail.
>
> Thanks!!
>
> On Thu, Nov 2, 2017 at 12:38 AM, Lucas Benevides <
> lu...@maurobenevides.com.br> wrote:
>
>> Dear community,
>>
>> I am using Cassandra Stress Tool and trying to simulate IoT generated
>> data.
>> So I created a column family with the device_id as the partition key.
>>
>> But in every different operation (the parameter received in the -n
>> option) the generated values are the same. For instance, I have a column
>> called observation_time which is supposed to be the time measured by the
>> sensor. But in every partition the values are equal.
>>
>> Is there a way to make those values be randomically generated with
>> different seeds? I need this way so that if the same device_id occurs
>> again, it makes an INSERT instead of an UPSERT.
>>
>> To clarify: What is happening now (fictional data):
>>
>> operation 1
>> device 1
>> ts1: 01/01/1970
>> ts2: 02/01/1980
>> ts3: 03/01/1990
>>
>> operation 2
>> device 2
>> ts1: 01/01/1970
>> ts2: 02/01/1980
>> ts3: 03/01/1990
>>
>> What I want:
>> operation1
>> device 1
>> ts1: 01/01/1970
>> ts2: 02/01/1980
>> ts3: 03/01/1990
>>
>> operation2
>> device 2
>> ts1: 02/01/1971  #Different values here.
>> ts2: 05/01/1982
>> ts3: 08/01/1993
>>
>> Thanks in advance,
>> Lucas Benevides
>>
>>
>

Re: Reg :- Multiple Node Cluster set up in Virtual Box

[Varun Barala]

you can try *CCM*
https://academy.datastax.com/planet-cassandra/getting-started-with-ccm-cassandra-cluster-manager

Thanks

On Mon, Nov 6, 2017 at 10:12 PM, @Nandan@ 
wrote:

> Hi Users ,
>  Just seeking some perfect guidelines to set up multi-node cluster  single
> Data Center in single host machine.
> I am currently using windows 10 as host machine and installed Oracle
> virtual box in which I created 4 virtual machines and all had Ubuntu 16.04
> I would like to configure a flexible robust no SPOF  data center.
> So please let me know how do I start and what steps, I have to follow to
> configure this multi node cluster?
> My goal is to create 4 node cluster now and later based on learning
> experiences I will remove 1 node and add 2 more nodes to check everything
> should be working perfectly.
>
> Just hope to get some step by step guidelines from all of you.
>
> Thanks in advance and best regards,
> Nandan Priyadarshi
>

Re: Re: Re: how to configure the Token Allocation Algorithm

[Varun Barala]

Hi,

Managing `initial_token` by yourself will give you more control over
scale-in and scale-out.
Let's say you have three node cluster with `num_token: 1`

And your initial range looks like:-

Datacenter: datacenter1
==
AddressRackStatus State   LoadOwns
 Token

 3074457345618258602
127.0.0.1  rack1   Up Normal  98.96 KiB   66.67%
 -9223372036854775808
127.0.0.2  rack1   Up Normal  98.96 KiB   66.67%
 -3074457345618258603
127.0.0.3  rack1   Up Normal  98.96 KiB   66.67%
 3074457345618258602

Now let's say you want to scale out the cluster to twice the current
throughput(means you are adding 3 more nodes)

If you are using AWS EBS volumes then you can use the same volumes and spin
three more nodes by selecting midpoints of existing ranges which means your
new nodes are already having data.
Once you have mounted volumes on your new nodes:-
* You need to delete every system table except schema related tables.
* You need to generate system/local table by yourself which has `Bootstrap
state` as completed and schema-version same as other existing nodes.
* You need to remove extra data on all the machines using cleanup commands

This is how you can scale out Cassandra cluster in the minutes. In case you
want to add nodes one by one then you need to write some small tool which
will always figure out the bigger range in the existing cluster and will
split it into the half.

However, I never tested it thoroughly but this should work conceptually. So
here we are taking advantage of the fact that we have volumes(data) for
the new node beforehand so we no need to bootstrap them.

Thanks & Regards,
Varun Barala

On Tue, Oct 2, 2018 at 2:31 PM onmstester onmstester 
wrote:

>
>
> Sent using Zoho Mail <https://www.zoho.com/mail/>
>
>
>  On Mon, 01 Oct 2018 18:36:03 +0330 *Alain RODRIGUEZ
> >* wrote 
>
> Hello again :),
>
> I thought a little bit more about this question, and I was actually
> wondering if something like this would work:
>
> Imagine 3 node cluster, and create them using:
> For the 3 nodes: `num_token: 4`
> Node 1: `intial_token: -9223372036854775808, -4611686018427387905, -2,
> 4611686018427387901`
> Node 2: `intial_token: -7686143364045646507, -3074457345618258604,
> 1537228672809129299, 6148914691236517202`
> Node 3: `intial_token: -6148914691236517206, -1537228672809129303,
> 3074457345618258600, 7686143364045646503`
>
>  If you know the initial size of your cluster, you can calculate the total
> number of tokens: number of nodes * vnodes and use the formula/python
> code above to get the tokens. Then use the first token for the first node,
> move to the second node, use the second token and repeat. In my case there
> is a total of 12 tokens (3 nodes, 4 tokens each)
> ```
> >>> number_of_tokens = 12
> >>> [str(((2**64 / number_of_tokens) * i) - 2**63) for i in
> range(number_of_tokens)]
> ['-9223372036854775808', '-7686143364045646507', '-6148914691236517206',
> '-4611686018427387905', '-3074457345618258604', '-1537228672809129303',
> '-2', '1537228672809129299', '3074457345618258600', '4611686018427387901',
> '6148914691236517202', '7686143364045646503']
> ```
>
>
> Using manual initial_token (your idea), how could i add a new node to a
> long running cluster (the procedure)?
>
>

Re: ***UNCHECKED*** Query regarding cassandra column write time set by client Timestamp Generator

[Varun Barala]

I'm not sure about the use cases. But other approaches can also be
considered:-

* Every mutation will have the timestamp in the commitlog [So taking backup
of the commitlogs will give you this functionality]
* At client side, you fetch the existing writetime for those columns from
the db and also log the actual timestamp which is associated with the
current update/insert statements
https://docs.datastax.com/en/drivers/java/3.6/com/datastax/driver/core/Statement.html#getDefaultTimestamp--
(though this should only be used for debugging purposes!)


Thanks!

On Tue, Apr 9, 2019 at 5:31 PM Ben Slater 
wrote:

> Maybe stabledump can help you?
> https://cassandra.apache.org/doc/4.0/tools/sstable/sstabledump.html
>
> ---
>
>
> *Ben Slater*
> *Chief Product Officer*
>
>
>    
>
>
> Read our latest technical blog posts here
> .
>
> This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
> and Instaclustr Inc (USA).
>
> This email and any attachments may contain confidential and legally
> privileged information.  If you are not the intended recipient, do not copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender and then immediately delete the message.
>
>
> On Tue, 9 Apr 2019 at 19:26, Mahesh Daksha  wrote:
>
>> Thanks Ben for your response.
>> WRITETIME  gives the information of about the column value already
>> residing int the table. We intend to know  the timestamp of the record
>> which is about to apply/update.
>> This is needed to understand the timestamp difference of the data
>> residing in table with the one going to overwite the same.
>>
>> This all information is needed as out update statements going silent (not
>> reflecting any changes) in database. Not even returning any error or
>> exception.
>>
>> Thanks,
>> Mahesh Daksha
>>
>> On Tue, Apr 9, 2019 at 2:46 PM Ben Slater 
>> wrote:
>>
>>> Not in the logs but I think you should be able to use the WRITETIME
>>> function to view via CQL (see
>>> https://cassandra.apache.org/doc/latest/cql/dml.html#select)
>>>
>>> Cheers
>>> Ben
>>>
>>> ---
>>>
>>>
>>> *Ben Slater*
>>> *Chief Product Officer*
>>>
>>>
>>> 
>>> 
>>> 
>>>
>>> Read our latest technical blog posts here
>>> .
>>>
>>> This email has been sent on behalf of Instaclustr Pty. Limited
>>> (Australia) and Instaclustr Inc (USA).
>>>
>>> This email and any attachments may contain confidential and legally
>>> privileged information.  If you are not the intended recipient, do not copy
>>> or disclose its content, but please reply to this email immediately and
>>> highlight the error to the sender and then immediately delete the message.
>>>
>>>
>>> On Tue, 9 Apr 2019 at 16:51, Mahesh Daksha  wrote:
>>>
 Hello,

 I have configured the timestamp generator at cassandra client as below:

 cluster.setTimestampGenerator(new AtomicMonotonicTimestampGenerator());


 My cassandra client inserting and updating few of the rows in a table.
 My query is where in the cassandra debug logs I can see the query write
 time associated by with updated columns in the update query (sent by
 cient). Or if there is any other way I can log the same at client
 itself.

 Basically I want to see the write time sent by client to cassandra
 cluster.

 Thanks,
 Mahesh Daksha

>>>