Re: owns (effective)? Cassandra 4 b4

[Shalom Sagges]

Ah, missed the keyspace name there.
My bad :-)

On Wed, Feb 10, 2021 at 4:13 PM Bowen Song  wrote:

> In your example, the nodetool output depends on whether the keyspace name
> is present in the "nodetool status" command line parameters. "nodetool
> status" command without a keyspace name may not show the effective
> ownerships, but it should always show the effective ownership information
> when you use "nodetool status keyspace_name".
> On 10/02/2021 13:12, Shalom Sagges wrote:
>
> I don't think it's related specifically to 4.0
> I can see this issue on 3.11 and even on previous versions as well.
> It occurs when the replication factor is not similar on all keyspaces.
> For example, if you have a cluster with 2 DCs and one of the keyspaces has
> a RF of DC1: 3 whie other keyspaces have DC1: 3, DC2 : 3, you will see the
> reported behaviour.
>
> Hope this helps!
>
> On Wed, Feb 10, 2021 at 2:24 PM Erick Ramirez 
> wrote:
>
>> This appears to be the same issue reported in CASSANDRA-16273
>> <https://issues.apache.org/jira/browse/CASSANDRA-16273> affecting 4.0
>> and is currently open/unresolved. Cheers!
>>
>>>

Re: owns (effective)? Cassandra 4 b4

[Shalom Sagges]

I don't think it's related specifically to 4.0
I can see this issue on 3.11 and even on previous versions as well.
It occurs when the replication factor is not similar on all keyspaces.
For example, if you have a cluster with 2 DCs and one of the keyspaces has
a RF of DC1: 3 whie other keyspaces have DC1: 3, DC2 : 3, you will see the
reported behaviour.

Hope this helps!

On Wed, Feb 10, 2021 at 2:24 PM Erick Ramirez 
wrote:

> This appears to be the same issue reported in CASSANDRA-16273
>  affecting 4.0 and
> is currently open/unresolved. Cheers!
>
>>

Re: Upgrading to 3.11.8 Caused Map Failures

[Shalom Sagges]

You are right Yakir.
How did I miss that?? It was a misconfiguration on my end.

Thanks a lot!

On Sat, Dec 12, 2020 at 9:28 PM Yakir Gibraltar  wrote:

> See also:
> https://support.datastax.com/hc/en-us/articles/360027838911
>
>
> On Sat, Dec 12, 2020 at 9:11 PM Yakir Gibraltar  wrote:
>
>> Hi Shalom,
>> See bug: https://issues.apache.org/jira/browse/CASSANDRA-14978
>> Try to disable mmap:
>> disk_access_mode=standard
>> or
>> disk_access_mode=mmap_index_only
>> Yakir Gibraltar.
>>
>
>
> --
> *בברכה,*
> *יקיר גיברלטר*
>

Re: Upgrading to 3.11.8 Caused Map Failures

[Shalom Sagges]

Forgot to mention that there were also LEAK DETECTED errors:

ERROR [Reference-Reaper] 2020-12-11 03:25:42,172 Ref.java:229 - LEAK
DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@451030de) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1272432140:Memory@[7f623780..7f623aa0)
was not released before the reference was garbage collected
ERROR [Reference-Reaper] 2020-12-11 03:25:42,172 Ref.java:229 - LEAK
DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@4fe85bae) to class
org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@183159863
:[Memory@[0..f060), Memory@[0..10e6c0)] was not released before the
reference was garbage collected
ERROR [Reference-Reaper] 2020-12-11 03:25:42,173 Ref.java:229 - LEAK
DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@4eb88b74) to class
org.apache.cassandra.io.util.MmappedRegions$Tidier@992658185:/data_path/md-1105027-big-Data.db
was not released before the reference was garbage collected
ERROR [Reference-Reaper] 2020-12-11 03:25:42,176 Ref.java:229 - LEAK
DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@3692dae9) to class
org.apache.cassandra.io.util.FileHandle$Cleanup@1791308664:/data_path/md-1105027-big-Index.db
was not released before the reference was garbage collected



On Fri, Dec 11, 2020 at 6:50 PM Shalom Sagges 
wrote:

> Hi All,
>
> I upgraded Cassandra from v3.11.4 to v3.11.8.
> The upgrade went smoothly, however, after a few hours, a node crashed on
> OOM and a few hours later, another one crashed.
>
> Seems like they crashed from excessive GC behaviour (CMS). The logs show
> Map failures on CompactionExecutor:
>
> ERROR *[CompactionExecutor:744] *2020-12-11 03:25:42,169
> JVMStabilityInspector.java:94 - OutOfMemory error letting the JVM handle
> the error:
> ERROR [CompactionExecutor:744] 2020-12-11 03:25:37,765
> CassandraDaemon.java:235 - Exception in thread
> Thread[CompactionExecutor:744,1,main]
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
> at
> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157)
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:170)
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:73)
> ...
> ...
> Caused by: java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940)
> at
> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153)
> ... 23 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
> ... 24 common frames omitted
>
>
> *[CompactionExecutor:744] did the following before the crash:*
> INFO  [CompactionExecutor:744] 2020-12-11 03:00:29,985
> NoSpamLogger.java:91 - Maximum memory usage reached (536870912), cannot
> allocate chunk of 1048576
> WARN  [CompactionExecutor:744] 2020-12-11 03:10:57,437
> BigTableWriter.java:211 - Writing large partition  (108.963MiB)
> WARN  [CompactionExecutor:744] 2020-12-11 03:10:57,437
> BigTableWriter.java:211 - Writing large partition  (151.155MiB)
> WARN  [CompactionExecutor:744] 2020-12-11 03:11:16,445
> BigTableWriter.java:211 - Writing large partition  (253.149MiB)
>
>
> *Some more info:*
> The *max_map_count* is set to 1048575, so all is well there.
> Hugepages are enabled by default (I know I should disable them), but I
> don't think it can cause this behaviour.
> This never happened on v3.11.4, only on v3.11.8.
>
>
> I'd really appreciate your help on this one.
> Thanks!
>
>
>
>
>
>

Upgrading to 3.11.8 Caused Map Failures

[Shalom Sagges]

Hi All,

I upgraded Cassandra from v3.11.4 to v3.11.8.
The upgrade went smoothly, however, after a few hours, a node crashed on
OOM and a few hours later, another one crashed.

Seems like they crashed from excessive GC behaviour (CMS). The logs show
Map failures on CompactionExecutor:

ERROR *[CompactionExecutor:744] *2020-12-11 03:25:42,169
JVMStabilityInspector.java:94 - OutOfMemory error letting the JVM handle
the error:
ERROR [CompactionExecutor:744] 2020-12-11 03:25:37,765
CassandraDaemon.java:235 - Exception in thread
Thread[CompactionExecutor:744,1,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
at
org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157)
at
org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
at
org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
at
org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:170)
at
org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:73)
...
...
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940)
at
org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153)
... 23 common frames omitted
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
... 24 common frames omitted


*[CompactionExecutor:744] did the following before the crash:*
INFO  [CompactionExecutor:744] 2020-12-11 03:00:29,985 NoSpamLogger.java:91
- Maximum memory usage reached (536870912), cannot allocate chunk of 1048576
WARN  [CompactionExecutor:744] 2020-12-11 03:10:57,437
BigTableWriter.java:211 - Writing large partition  (108.963MiB)
WARN  [CompactionExecutor:744] 2020-12-11 03:10:57,437
BigTableWriter.java:211 - Writing large partition  (151.155MiB)
WARN  [CompactionExecutor:744] 2020-12-11 03:11:16,445
BigTableWriter.java:211 - Writing large partition  (253.149MiB)


*Some more info:*
The *max_map_count* is set to 1048575, so all is well there.
Hugepages are enabled by default (I know I should disable them), but I
don't think it can cause this behaviour.
This never happened on v3.11.4, only on v3.11.8.


I'd really appreciate your help on this one.
Thanks!

Re: Enable Ttracing

[Shalom Sagges]

Thanks a lot guys!

I have a feeling that this tool will give me hell. 
I'll just have to wait till they implement it and monitor the clusters, but
at least I know what to expect.


Thanks again



On Tue, Nov 17, 2020 at 1:33 AM Jeff Jirsa  wrote:

> (Just to put this in perspective, it's probably a 10-20x increase in
> writes, depending on how many replicas you're using, and your read:write
> mix. Most people who use tracing do so probabilistically, with p < 1)
>
> On Mon, Nov 16, 2020 at 2:09 PM Erick Ramirez 
> wrote:
>
>> I wouldn't do it. It will significantly affect the performance of your
>> cluster so your monitoring ends up killing the very thing it's monitoring.
>>
>> Perhaps you could tell us what outcome you're after so we'd be able to
>> offer a better solution. Like audit logging, for example. Cheers!
>>
>>>

Enable Ttracing

[Shalom Sagges]

Hi Guys,

Our Service team would like to add a 3rd party tool (AppDynamics) that will
monitor Cassandra.
This tool will get read permissions on the system_traces keyspace and also
needs to enable TRACING.
tracetype_query_ttl in the yaml file will be reduced from 24 hours to 5
minutes.
I feel and fear that using TRACING ON constantly will add more pressure on
the cluster.
Am I correct with this feeling or is it safe to use tracing on a regular
basis?

Thanks!

Re: Some nodes has excessive GC Count compared to others

[Shalom Sagges]

I agree with Erick and believe it's most likely a hot partitions issue.

I'd check "Compacted partition maximum bytes" in nodetool tablestats on
those "affected" nodes and compare the result with the other nodes.
I'd also check how the cpu_load is affected. From my experience, during
excessive GC the cpu_load is first to show stress, which correlates with
dropped mutations/reads that may occur (you can search the logs for such
drops). If you run repairs and use local_quorum you might not feel it, but
still worth to avoid this.
Another thing that comes to mind is a considerable amount of tombstones or
updated rows for some partitions that may create unnecessary overhead and
increased GC pressure.


Regards,

On Thu, Sep 3, 2020 at 1:54 AM Erick Ramirez 
wrote:

> That would have been my first response too -- hot partitions. If you know
> the partition keys, you can quickly confirm it with nodetool getendpoints.
> Cheers!
>
>>

A Bootstrapped Node Doesn't Receive All Data

[Shalom Sagges]

Hi All,

Apologies for the long email, so TL;DR:
A node bootstrapped successfully but only got the data which it was the
owner of, and didn't get the data as a replica.

I'm experiencing a really odd situation during node bootstrap.

Cassandra 3.11.4.
Background:
Due to a capacity issue on one of our clusters, we decided to increase the
storage space on each node. Since the nodes are set with RAID 0, the only
valid option was to decommission the node, add the disks and rebuild, and
once done, bootstrap it back to the cluster.

Everything was going well until one of our system guys accidentally handled
the disks on an active server instead of a decommissioned one. The node
immediately crashed since the data directory was destroyed.
I had to forcefully remove it from the cluster, but when I added it back,
it only got 1/3 of the data it actually had to get.
I've tried several times, even after a rolling restart of the whole
cluster, and always got the same result. I've tried to repair the node
using -pr but it practically didn't repair anything. I've tried to fully
repair it (without -pr) and the repair did fix the problem.
So I concluded that the node only got the data which it was the owner of,
but didn't get the data as a replica. I got the same outcome on both the
node that crashed and the node that was gracefully decommissioned.

Any idea why it's acting like this?

Thanks!

Re: disable debug message on read repair

[Shalom Sagges]

Hi Gil,

You can run a full repair on your cluster. But if these messages come back
again, you need to check what's causing these data inconsistencies.


On Sun, Mar 8, 2020 at 10:11 AM Gil Ganz  wrote:

> Hey all
> I have a lot of debug message about read repairs in my debug log :
>
> DEBUG [ReadRepairStage:346] 2020-03-08 08:09:12,959 ReadCallback.java:242
> - Digest mismatch:
> org.apache.cassandra.service.DigestMismatchException: Mismatch for key
> DecoratedKey(-28476014476640,
> 000400871130303a33613a37643a33613a62643a383000)
> (38d3509295a283326a71887113fc vs 033cbba98c43e7ba6f15c0ba462f5fcc)
> at
> org.apache.cassandra.service.DigestResolver.compareResponses(DigestResolver.java:92)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:233)
> ~[apache-cassandra-3.11.5.jar:3.11.5]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [na:1.8.0_201]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [na:1.8.0_201]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
> [apache-cassandra-3.11.5.jar:3.11.5]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_201]
>
> Version is 3.11.5, anyone knows how to disable just these warnings?
> Thanks
> Gil
>
>

Re: KeyCache Harmless Error on Startup

[Shalom Sagges]

Thanks Erick!

I will check with the owners of this keyspace, hoping to find the culprit.
If they won't come up with anything, is there a way to read the key cache
file? (as I understand it's a binary file)
On another note, there's actually another keyspace I missed to point out on
which I found a weird behavior (not necessarily related though).

CREATE KEYSPACE ks3 WITH replication = {'class': 'NetworkTopologyStrategy',
'DC1': '3', 'DC2': '3'}  AND durable_writes = true;

CREATE TABLE ks3.tbl4 (
account_id text,
consumer_phone_number text,
channel text,
event_time_stamp timestamp,
brand_phone_number text,
campaign_id bigint,
engagement_id bigint,
event_type text,
PRIMARY KEY ((account_id, consumer_phone_number), channel,
event_time_stamp)
);

When I select from this table, I get the following warning:
*cqlsh.py:395: DateOverFlowWarning: Some timestamps are larger than Python
datetime can represent. Timestamps are displayed in milliseconds from
epoch.*

I don't know if it's related but worth pointing out.

account_id   | consumer_phone_number  | channel
| event_time_stamp | brand_phone_number   | campaign_id |
engagement_id | event_type
+--+-+--+--+-+---+
   12345678 | OIs1HXovJ9W/AJZI+Tm8CSCbAavdVI06qt0c | sms |
*1580305508799000* | PY0yHHItI9BibOtNis8hDuLwN91prPa+ |null |
   null |opt-out


Thanks!



On Thu, Jan 30, 2020 at 1:43 AM Erick Ramirez  wrote:

> Specifically for the NegativeArraySizeException, what's happening is that
>> the keyLength is so huge that it blows up MAX_UNSIGNED_SHORT so it looks
>> like it's a negative value. Someone will correct me if I got that wrong but
>> the "Key length longer than max" error confirms that.
>>
>
> Is it possible that you have a rogue metric_name value that's impossibly
> long? I'm a bit more convinced now that's what's happening because you said
> it happens on multiple servers which rules out local file corruption at the
> filesystem level. Cheers!
>

Re: KeyCache Harmless Error on Startup

[Shalom Sagges]

Hi again,

Does anyone perhaps have an idea on what could've gone wrong here?
Could it be just a calculation error on startup?

Thanks!

On Sun, Jan 26, 2020 at 5:57 PM Shalom Sagges 
wrote:

> Hi Jeff,
>
> It is happening on multiple servers and even on different DCs.
> The schema contains two keyspaces as follows:
>
> CREATE KEYSPACE ks1 WITH replication = {'class':
> 'NetworkTopologyStrategy', 'DC1': '3', 'DC2': '3'}  AND durable_writes =
> true;
>
> CREATE TABLE ks1.tbl1 (
> account_id text,
> metric_name text,
> day_of_month int,
> hour_of_day int,
> insert_time timestamp,
> metric_value text,
> PRIMARY KEY ((account_id, metric_name, day_of_month, hour_of_day),
> insert_time)
> ) WITH CLUSTERING ORDER BY (insert_time DESC)
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND default_time_to_live = 2764800
> AND gc_grace_seconds = 864000;
>
> CREATE TABLE ks1.tbl2 (
> account_id text,
> lambda_uuid uuid,
> metric_name text,
> day_of_month int,
> insert_time timestamp,
> hour_of_day int,
> metric_value text,
> PRIMARY KEY ((account_id, lambda_uuid, metric_name, day_of_month),
> insert_time)
> ) WITH CLUSTERING ORDER BY (insert_time DESC)
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND default_time_to_live = 2764800
> AND gc_grace_seconds = 864000;
>
>
> CREATE KEYSPACE ks2 WITH replication = {'class':
> 'NetworkTopologyStrategy', 'DC1': '3', 'DC2': '3'}  AND durable_writes =
> true;
>
> CREATE TABLE ks2.tbl3 (
> account_id text,
> lambda_uuid uuid,
> insert_time timestamp,
> data text,
> error_count int,
> info_count int,
> request_id text,
> warn_count int,
> PRIMARY KEY ((account_id, lambda_uuid), insert_time)
> ) WITH CLUSTERING ORDER BY (insert_time DESC)
> AND default_time_to_live = 2764800
> AND gc_grace_seconds = 864000;
>
>
> I'm afraid I am unable to provide the actual key cache file, but for what
> it's worth, here's a data example from the above tables:
>
>
> *ks1.tbl1:*
>  account_id | metric_name   |
> day_of_month | hour_of_day | insert_time | metric_value
>
> +---+--+-+-+---
>12345678 | com.selfbuilds.faas.system.invocation-finished-successful |
>1 |  10 | 2020-01-01 10:59:03.588000+ |
> {"invocationIdentifier":"51176dd4-aa34-4834-bcf5-7e3d5b5494d1","lambdaUUID":"ADD3A5B4-7497-499A-8364-A97108C7400B","event":"","invocationState":"SUCCEEDED","executionTimeInMs":3392}
>
>
> *ks1.tbl2:*
>  account_id | lambda_uuid  | metric_name
>   | day_of_month | insert_time
> | hour_of_day | metric_value
>
> +--+-+--+-+-+--
>12345678 | ed7f0629-8d5c-47dc-a66d-4a1197aa01bc |
> com.selfbuilds.faas.system.invocation-triggered |9 | 2020-01-09
> 13:03:06.653000+ |  13 |
> {"invocationIdentifier":"3631e9ea-b230-4c4d-a051-c7b4241ee231","lambdaUUID":"ED7F0629-8D5C-47DC-A66D-4A1197AA01BC","event":"","lambdaName":"1578574875000","lastModifierOfLambda":"FaaSAll"}
>
>
> *ks2.tbl3:*
>  account_id | lambda_uuid  | insert_time
>   | data
> | error_count | info_count | request_id
>   | warn_count
>
> +--+-++-++--+
>12345678 | 9f4eaa4f-42de-4c6e-9c96-02a4749281f3 | 2020-01-01
> 01:30:22.302000+ |
> [{"timestamp":1577842222458,"level":"Info","message":"Best Logs in
> town","extras":[]}] |  

Re: KeyCache Harmless Error on Startup

[Shalom Sagges]

Hi Jeff,

It is happening on multiple servers and even on different DCs.
The schema contains two keyspaces as follows:

CREATE KEYSPACE ks1 WITH replication = {'class': 'NetworkTopologyStrategy',
'DC1': '3', 'DC2': '3'}  AND durable_writes = true;

CREATE TABLE ks1.tbl1 (
account_id text,
metric_name text,
day_of_month int,
hour_of_day int,
insert_time timestamp,
metric_value text,
PRIMARY KEY ((account_id, metric_name, day_of_month, hour_of_day),
insert_time)
) WITH CLUSTERING ORDER BY (insert_time DESC)
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND default_time_to_live = 2764800
AND gc_grace_seconds = 864000;

CREATE TABLE ks1.tbl2 (
account_id text,
lambda_uuid uuid,
metric_name text,
day_of_month int,
insert_time timestamp,
hour_of_day int,
metric_value text,
PRIMARY KEY ((account_id, lambda_uuid, metric_name, day_of_month),
insert_time)
) WITH CLUSTERING ORDER BY (insert_time DESC)
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND default_time_to_live = 2764800
AND gc_grace_seconds = 864000;


CREATE KEYSPACE ks2 WITH replication = {'class': 'NetworkTopologyStrategy',
'DC1': '3', 'DC2': '3'}  AND durable_writes = true;

CREATE TABLE ks2.tbl3 (
account_id text,
lambda_uuid uuid,
insert_time timestamp,
data text,
error_count int,
info_count int,
request_id text,
warn_count int,
PRIMARY KEY ((account_id, lambda_uuid), insert_time)
) WITH CLUSTERING ORDER BY (insert_time DESC)
AND default_time_to_live = 2764800
AND gc_grace_seconds = 864000;


I'm afraid I am unable to provide the actual key cache file, but for what
it's worth, here's a data example from the above tables:


*ks1.tbl1:*
 account_id | metric_name   |
day_of_month | hour_of_day | insert_time | metric_value
+---+--+-+-+---
   12345678 | com.selfbuilds.faas.system.invocation-finished-successful |
 1 |  10 | 2020-01-01 10:59:03.588000+ |
{"invocationIdentifier":"51176dd4-aa34-4834-bcf5-7e3d5b5494d1","lambdaUUID":"ADD3A5B4-7497-499A-8364-A97108C7400B","event":"","invocationState":"SUCCEEDED","executionTimeInMs":3392}


*ks1.tbl2:*
 account_id | lambda_uuid  | metric_name
  | day_of_month | insert_time
| hour_of_day | metric_value
+--+-+--+-+-+--
   12345678 | ed7f0629-8d5c-47dc-a66d-4a1197aa01bc |
com.selfbuilds.faas.system.invocation-triggered |9 | 2020-01-09
13:03:06.653000+ |  13 |
{"invocationIdentifier":"3631e9ea-b230-4c4d-a051-c7b4241ee231","lambdaUUID":"ED7F0629-8D5C-47DC-A66D-4A1197AA01BC","event":"","lambdaName":"1578574875000","lastModifierOfLambda":"FaaSAll"}


*ks2.tbl3:*
 account_id | lambda_uuid  | insert_time
  | data
| error_count | info_count | request_id
  | warn_count
+--+-++-++--+
   12345678 | 9f4eaa4f-42de-4c6e-9c96-02a4749281f3 | 2020-01-01
01:30:22.302000+ |
[{"timestamp":157784458,"level":"Info","message":"Best Logs in
town","extras":[]}] |   0 |  1 |
2966683f-cf37-4ea3-9d82-1de46207d51e |  0



Thanks for your help on this one!



On Thu, Jan 23, 2020 at 5:40 PM Jeff Jirsa  wrote:

> 489MB key seems huge. Unlikely it’s really that large.
>
> If it’s happening on multiple servers it’d be interesting to see the
> schema and/or saved cache file if it’s not incredibly sensitive
>
> On Jan 23, 2020, at 3:48 AM, Shalom Sagges  wrote:
>
> 
> Hi All,
>
> Cassandra 3.11.4.
> On one of our clusters, during startup, I see two types of  "Harmless
> error" notification regarding the keycache:
>
&

KeyCache Harmless Error on Startup

[Shalom Sagges]

Hi All,

Cassandra 3.11.4.
On one of our clusters, during startup, I see two types of  "Harmless
error" notification regarding the keycache:

*Server 1:*

INFO  [pool-3-thread-1] 2020-01-23 04:34:46,167
AutoSavingCache.java:263 - *Harmless
error reading saved cache* /path/to/saved_caches/KeyCache-e.db
*java.lang.NegativeArraySizeException: null*
at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:401)
at
org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:472)
at
org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:220)
at
org.apache.cassandra.cache.AutoSavingCache$3.call(AutoSavingCache.java:165)
at
org.apache.cassandra.cache.AutoSavingCache$3.call(AutoSavingCache.java:161)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
INFO  [pool-3-thread-1] 2020-01-23 04:34:46,168 AutoSavingCache.java:174 -
Completed loading (188 ms; 10916 keys) KeyCache cache


*Server 2:*

INFO  [pool-3-thread-1] 2020-01-15 11:18:35,797 AutoSavingCache.java:263 -
Harmless error reading saved cache /path/to/saved_caches//KeyCache-e.db
*java.io.IOException: Corrupted key cache. Key length of 489735494 is
longer than maximum of 65535*
at
org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:469)
at
org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:220)
at
org.apache.cassandra.cache.AutoSavingCache$3.call(AutoSavingCache.java:165)
at
org.apache.cassandra.cache.AutoSavingCache$3.call(AutoSavingCache.java:161)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
INFO  [pool-3-thread-1] 2020-01-15 11:18:35,799 AutoSavingCache.java:174 -
Completed loading (285 ms; 20681 keys) KeyCache cache

The KeyCache size is a lot less than max default (100MB). So does this mean
that the key itself is the culprit?

I'd really appreciate some guidance on what this message actually means in
the logs during startup and what problems may arise from it.
I see that one of the nodes experiences high CPU load and even gets kernel
panic from time to time, so I want to understand if the above may cause
such issues.

Thanks a lot!

Re: RE:_Why_suddenly_cassandra_3.11.4_and_3.11.3_“can_not_find_or_load_main_class_-ea”?

[Shalom Sagges]

Hi Georgelin,

Do you have jdk 1.8 installed?
Is JAVA_HOME set in your cassandra-env.sh file?
Also, try to check the /var/log/cassandra/startup.log for additional
information.

Hope this helps.

On Tue, Dec 24, 2019 at 10:39 AM gloCalHelp.com 
wrote:

> To Dimo：
>
>  Thank you for your reply and guiding , but not only my customized
> soure, but also the original cassandra 3.11.3/4 suddenly cann't run as
> several years ago and throw the -ea errors, and I have check the original
> running bin/cassandra source and centos's environment, no any -ea let.
> And the cassandra's running JVM's parameters are shown as below, there
>  is no any -ea JVM variable left. I will check Centos JVM's environment,
> in addition to the suddenly deleted lib folder too, it seems that some
> hackers play a prank on me to stop my developing and using cassandra.
>
>   Is there any genius can help me?
>
> Sincerely yours,
> Georgelin
> www_8ems_...@sina.com
> mobile:0086 180 5986 1565
>
>
> - 原始邮件 -
> 发件人：Dimo Velev 
> 收件人：user@cassandra.apache.org, dev-help ,
> www_8ems_com 
>
> 主题：RE:_Why_suddenly_cassandra_3.11.4_and_3.11.3_“can_not_find_or_load_main_class_-ea”?
> 日期：2019年12月24日 16点13分
>
> -ea is a jvm parameter to enable assertions. Maybe you are passing it at
> the wrong place and the jvm interprets it as class name that you want to
> start.
>
>  Original message 
> From: "gloCalHelp.com" 
> Date: 24/12/2019 08:03 (GMT+01:00)
> To: user , dev-help <
> dev-h...@cassandra.apache.org>, www_8ems_com 
> Subject: Why suddenly cassandra 3.11.4 and 3.11.3 “can not find or load
> main class -ea”?
>
> Dear cassandra pioneers and users：
> Yesterday afternoon, I can run cassandra 3.11.4's git-clone-src by using
> "bin/cassandra", but after I changed some codes by Eclipse,
>  and compile without any error, not only the runnable source version, but
> also I redownload 3.11.4-bin.tar.gz and 3.11.3 from official website,
> they can't run by command of "./bin/cassandra", the steps and logs are as
> below:
>
> [gloCalHelp.com@gloCalHelp5 apache-cassandra-3.11.4]$ ./bin/cassandra &
> [1] 5872 [gloCalHelp.com@gloCalHelp5 apache-cassandra-3.11.4]$ classname
> is+ org.apache.cassandra.service.CassandraDaemon +CLASSPATH
> is+./bin/../conf:./bin/../build/classes/main:./bin/../build/classes/thrift:./bin/../lib/airline-0.6.jar:./bin/../lib/antlr-runtime-3.5.2.jar
> *:./bin/../lib/apache-cassandra-3.11.4.jar:*
> ./bin/../lib/apache-cassandra-thrift-3.11.4.jar:./bin/../lib/asm-5.0.4.jar:./bin/../lib/caffeine-2.2.6.jar:./bin/../lib/cassandra-driver-core-3.0.1-shaded.jar:./bin/../lib/commons-cli-1.1.jar:./bin/../lib/commons-codec-1.9.jar:./bin/../lib/commons-lang3-3.1.jar:./bin/../lib/commons-math3-3.2.jar:./bin/../lib/compress-lzf-0.8.4.jar:./bin/../lib/concurrentlinkedhashmap-lru-1.4.jar:./bin/../lib/concurrent-trees-2.4.0.jar:./bin/../lib/disruptor-3.0.1.jar:./bin/../lib/ecj-4.4.2.jar:./bin/../lib/guava-18.0.jar:./bin/../lib/HdrHistogram-2.1.9.jar:./bin/../lib/high-scale-lib-1.0.6.jar:./bin/../lib/hppc-0.5.4.jar:./bin/../lib/jackson-core-asl-1.9.13.jar:./bin/../lib/jackson-mapper-asl-1.9.13.jar:./bin/../lib/jamm-0.3.0.jar:./bin/../lib/javax.inject.jar:./bin/../lib/jbcrypt-0.3m.jar:./bin/../lib/jcl-over-slf4j-1.7.7.jar:./bin/../lib/jctools-core-1.2.1.jar:./bin/../lib/jflex-1.6.0.jar:./bin/../lib/jna-4.2.2.jar:./bin/../lib/joda-time-2.4.jar:./bin/../lib/json-simple-1.1.jar:./bin/../lib/jstackjunit-0.0.1.jar:./bin/../lib/libthrift-0.9.2.jar:./bin/../lib/log4j-over-slf4j-1.7.7.jar:./bin/../lib/logback-classic-1.1.3.jar:./bin/../lib/logback-core-1.1.3.jar:./bin/../lib/lz4-1.3.0.jar:./bin/../lib/metrics-core-3.1.5.jar:./bin/../lib/metrics-jvm-3.1.5.jar:./bin/../lib/metrics-logback-3.1.5.jar:./bin/../lib/netty-all-4.0.44.Final.jar:./bin/../lib/ohc-core-0.4.4.jar:./bin/../lib/ohc-core-j8-0.4.4.jar:./bin/../lib/reporter-config3-3.0.3.jar:./bin/../lib/reporter-config-base-3.0.3.jar:./bin/../lib/sigar-1.6.4.jar:./bin/../lib/slf4j-api-1.7.7.jar:./bin/../lib/snakeyaml-1.11.jar:./bin/../lib/snappy-java-1.1.1.7.jar:./bin/../lib/snowball-stemmer-1.3.0.581.1.jar:./bin/../lib/ST4-4.0.8.jar:./bin/../lib/stream-2.5.2.jar:./bin/../lib/thrift-server-0.3.7.jar:./bin/../lib/jsr223/
> */*.jar
>
> Error: Could not find or load main class -ea
> [1]+ Done ./bin/cassandra
> [gloCalHelp.com@gloCalHelp5 apache-cassandra-3.11.4]$ free -m
>  total used free shared buffers cached Mem:
> 4567 801 *3766* 5 20 190 -/+ buffers/cache: 590 3977 Swap: 1031 0 1031
> and the main class CassandraDaemon and classpath are there as
> ":./bin/../lib/apache-cassandra-3.11.4.jar:",
> why Centos 6.9 cann't run in win10's vmware now?
>
> I would very appreciate your guiding. Thank you ahead.
>
>
>
> Sincerely yours,
> Georgelin
> www_8ems_...@sina.com
> mobile:0086 180 5986 1565
>
>

Re: Cassandra is not showing a node up hours after restart

[Shalom Sagges]

Sorry, disregard the schema ID. It's too early in the morning here ;)

On Tue, Nov 26, 2019 at 7:58 AM Shalom Sagges 
wrote:

> Hi Paul,
>
> From the gossipinfo output, it looks like the node's IP address and
> rpc_address are different.
> /192.168.*187*.121 vs RPC_ADDRESS:192.168.*185*.121
> You can also see that there's a schema disagreement between nodes, e.g.
> schema_id on node001 is fd2dcb4b-ca62-30df-b8f2-d3fd774f2801 and on node002
> it is fd2dcb4b-ca62-30df-b8f2-d3fd774f2801.
> You can run nodetool describecluster to see it as well.
> So I suggest to change the rpc_address to the ip_address of the node or
> set it to 0.0.0.0 and it should resolve the issue.
>
> Hope this helps!
>
>
> On Tue, Nov 26, 2019 at 4:05 AM Inquistive allen 
> wrote:
>
>> Hello ,
>>
>> Check and compare everything parameters
>>
>> 1. Java version should ideally match across all nodes in the cluster
>> 2. Check if port 7000 is open between the nodes. Use telnet or nc commands
>> 3. You must see some clues in system logs, why the gossip is failing.
>>
>> Do confirm on the above things.
>>
>> Thanks
>>
>>
>> On Tue, 26 Nov, 2019, 2:50 AM Paul Mena,  wrote:
>>
>>> NTP was restarted on the Cassandra nodes, but unfortunately I’m still
>>> getting the same result: the restarted node does not appear to be rejoining
>>> the cluster.
>>>
>>>
>>>
>>> Here’s another data point: “nodetool gossipinfo”, when run from the
>>> restarted node (“node001”) shows a status of “normal”:
>>>
>>>
>>>
>>> user@node001=> nodetool -u gossipinfo
>>>
>>> /192.168.187.121
>>>
>>>   generation:1574364410
>>>
>>>   heartbeat:209150
>>>
>>>   NET_VERSION:8
>>>
>>>   RACK:rack1
>>>
>>>   STATUS:NORMAL,-104847506331695918
>>>
>>>   RELEASE_VERSION:2.1.9
>>>
>>>   SEVERITY:0.0
>>>
>>>   LOAD:5.78684155614E11
>>>
>>>   HOST_ID:c99cf581-f4ae-4aa9-ab37-1a114ab2429b
>>>
>>>   SCHEMA:fd2dcb4b-ca62-30df-b8f2-d3fd774f2801
>>>
>>>   DC:datacenter1
>>>
>>>   RPC_ADDRESS:192.168.185.121
>>>
>>>
>>>
>>> When run from one of the other nodes, however, node001’s status is shown
>>> as “shutdown”:
>>>
>>>
>>>
>>> user@node002=> nodetool gossipinfo
>>>
>>> /192.168.187.121
>>>
>>>   generation:1491825076
>>>
>>>   heartbeat:2147483647
>>>
>>>   STATUS:shutdown,true
>>>
>>>   RACK:rack1
>>>
>>>   NET_VERSION:8
>>>
>>>   LOAD:5.78679987693E11
>>>
>>>   RELEASE_VERSION:2.1.9
>>>
>>>   DC:datacenter1
>>>
>>>   SCHEMA:fd2dcb4b-ca62-30df-b8f2-d3fd774f2801
>>>
>>>   HOST_ID:c99cf581-f4ae-4aa9-ab37-1a114ab2429b
>>>
>>>   RPC_ADDRESS:192.168.185.121
>>>
>>>   SEVERITY:0.0
>>>
>>>
>>>
>>>
>>>
>>> *Paul Mena*
>>>
>>> Senior Application Administrator
>>>
>>> WHOI - Information Services
>>>
>>> 508-289-3539
>>>
>>>
>>>
>>> *From:* Paul Mena
>>> *Sent:* Monday, November 25, 2019 9:29 AM
>>> *To:* user@cassandra.apache.org
>>> *Subject:* RE: Cassandra is not showing a node up hours after restart
>>>
>>>
>>>
>>> I’ve just discovered that NTP is not running on any of these Cassandra
>>> nodes, and that the timestamps are all over the map. Could this be causing
>>> my issue?
>>>
>>>
>>>
>>> user@remote=> ansible pre-prod-cassandra -a date
>>>
>>> node001.intra.myorg.org | CHANGED | rc=0 >>
>>>
>>> Mon Nov 25 13:58:17 UTC 2019
>>>
>>>
>>>
>>> node004.intra.myorg.org | CHANGED | rc=0 >>
>>>
>>> Mon Nov 25 14:07:20 UTC 2019
>>>
>>>
>>>
>>> node003.intra.myorg.org | CHANGED | rc=0 >>
>>>
>>> Mon Nov 25 13:57:06 UTC 2019
>>>
>>>
>>>
>>> node001.intra.myorg.org | CHANGED | rc=0 >>
>>>
>>> Mon Nov 25 14:07:22 UTC 2019
>>>
>>>
>>>
>>> *Paul Mena*
>>>
>>> Senior Application Administrator
>>>
>>> WHOI - Information Services
>>>
>>> 508-289-

Re: Cassandra is not showing a node up hours after restart

[Shalom Sagges]

Hi Paul,

>From the gossipinfo output, it looks like the node's IP address and
rpc_address are different.
/192.168.*187*.121 vs RPC_ADDRESS:192.168.*185*.121
You can also see that there's a schema disagreement between nodes, e.g.
schema_id on node001 is fd2dcb4b-ca62-30df-b8f2-d3fd774f2801 and on node002
it is fd2dcb4b-ca62-30df-b8f2-d3fd774f2801.
You can run nodetool describecluster to see it as well.
So I suggest to change the rpc_address to the ip_address of the node or set
it to 0.0.0.0 and it should resolve the issue.

Hope this helps!


On Tue, Nov 26, 2019 at 4:05 AM Inquistive allen 
wrote:

> Hello ,
>
> Check and compare everything parameters
>
> 1. Java version should ideally match across all nodes in the cluster
> 2. Check if port 7000 is open between the nodes. Use telnet or nc commands
> 3. You must see some clues in system logs, why the gossip is failing.
>
> Do confirm on the above things.
>
> Thanks
>
>
> On Tue, 26 Nov, 2019, 2:50 AM Paul Mena,  wrote:
>
>> NTP was restarted on the Cassandra nodes, but unfortunately I’m still
>> getting the same result: the restarted node does not appear to be rejoining
>> the cluster.
>>
>>
>>
>> Here’s another data point: “nodetool gossipinfo”, when run from the
>> restarted node (“node001”) shows a status of “normal”:
>>
>>
>>
>> user@node001=> nodetool -u gossipinfo
>>
>> /192.168.187.121
>>
>>   generation:1574364410
>>
>>   heartbeat:209150
>>
>>   NET_VERSION:8
>>
>>   RACK:rack1
>>
>>   STATUS:NORMAL,-104847506331695918
>>
>>   RELEASE_VERSION:2.1.9
>>
>>   SEVERITY:0.0
>>
>>   LOAD:5.78684155614E11
>>
>>   HOST_ID:c99cf581-f4ae-4aa9-ab37-1a114ab2429b
>>
>>   SCHEMA:fd2dcb4b-ca62-30df-b8f2-d3fd774f2801
>>
>>   DC:datacenter1
>>
>>   RPC_ADDRESS:192.168.185.121
>>
>>
>>
>> When run from one of the other nodes, however, node001’s status is shown
>> as “shutdown”:
>>
>>
>>
>> user@node002=> nodetool gossipinfo
>>
>> /192.168.187.121
>>
>>   generation:1491825076
>>
>>   heartbeat:2147483647
>>
>>   STATUS:shutdown,true
>>
>>   RACK:rack1
>>
>>   NET_VERSION:8
>>
>>   LOAD:5.78679987693E11
>>
>>   RELEASE_VERSION:2.1.9
>>
>>   DC:datacenter1
>>
>>   SCHEMA:fd2dcb4b-ca62-30df-b8f2-d3fd774f2801
>>
>>   HOST_ID:c99cf581-f4ae-4aa9-ab37-1a114ab2429b
>>
>>   RPC_ADDRESS:192.168.185.121
>>
>>   SEVERITY:0.0
>>
>>
>>
>>
>>
>> *Paul Mena*
>>
>> Senior Application Administrator
>>
>> WHOI - Information Services
>>
>> 508-289-3539
>>
>>
>>
>> *From:* Paul Mena
>> *Sent:* Monday, November 25, 2019 9:29 AM
>> *To:* user@cassandra.apache.org
>> *Subject:* RE: Cassandra is not showing a node up hours after restart
>>
>>
>>
>> I’ve just discovered that NTP is not running on any of these Cassandra
>> nodes, and that the timestamps are all over the map. Could this be causing
>> my issue?
>>
>>
>>
>> user@remote=> ansible pre-prod-cassandra -a date
>>
>> node001.intra.myorg.org | CHANGED | rc=0 >>
>>
>> Mon Nov 25 13:58:17 UTC 2019
>>
>>
>>
>> node004.intra.myorg.org | CHANGED | rc=0 >>
>>
>> Mon Nov 25 14:07:20 UTC 2019
>>
>>
>>
>> node003.intra.myorg.org | CHANGED | rc=0 >>
>>
>> Mon Nov 25 13:57:06 UTC 2019
>>
>>
>>
>> node001.intra.myorg.org | CHANGED | rc=0 >>
>>
>> Mon Nov 25 14:07:22 UTC 2019
>>
>>
>>
>> *Paul Mena*
>>
>> Senior Application Administrator
>>
>> WHOI - Information Services
>>
>> 508-289-3539
>>
>>
>>
>> *From:* Inquistive allen 
>> *Sent:* Monday, November 25, 2019 2:46 AM
>> *To:* user@cassandra.apache.org
>> *Subject:* Re: Cassandra is not showing a node up hours after restart
>>
>>
>>
>> Hello team,
>>
>>
>>
>> Just to add on to the discussion, one may run,
>>
>> Nodetool disablebinary followed by a nodetool disablethrift followed by
>> nodetool drain.
>>
>> Nodetool drain also does the work of nodetool flush+ declaring in the
>> cluster that I'm down and not accepting traffic.
>>
>>
>>
>> Thanks
>>
>>
>>
>>
>>
>> On Mon, 25 Nov, 2019, 12:55 AM Surbhi Gupta, 
>> wrote:
>>
>> Before Cassandra shutdown, nodetool drain should be executed first. As
>> soon as you do nodetool drain, others node will see this node down and no
>> new traffic will come to this node.
>>
>> I generally gives 10 seconds gap between nodetool drain and Cassandra
>> stop.
>>
>>
>>
>> On Sun, Nov 24, 2019 at 9:52 AM Paul Mena  wrote:
>>
>> Thank you for the replies. I had made no changes to the config before the
>> rolling restart.
>>
>>
>>
>> I can try another restart but was wondering if I should do it
>> differently. I had simply done "service cassandra stop" followed by
>> "service cassandra start".  Since then I've seen some suggestions to
>> proceed the shutdown with "nodetool disablegossip" and/or "nodetool drain".
>> Are these commands advisable? Are any other commands recommended either
>> before the shutdown or after the startup?
>>
>>
>>
>> Thanks again!
>>
>>
>>
>> Paul
>> --
>>
>> *From:* Naman Gupta 
>> *Sent:* Sunday, November 24, 2019 11:18:14 AM
>> *To:* user@cassandra.apache.org
>> *Subject:* Re: Cassandra is not showing a node up 

Re: Understanding TRACE logging

[shalom sagges]

Thanks Laxmikant and Paul.

@Laxmikant, Unfortunately, this cluster is still on 2.1 so ECAudit won't
support it, but will check it out once it's upgraded to 3.x (should happen
pretty soon).
@Paul, I will definitely try the Wireshark method.

Thanks a lot guys for your help!

On Thu, Sep 26, 2019 at 11:05 PM Paul Chandler  wrote:

> Hi Shalom,
>
> When tracking down specific queries I have used ngrep and fed the results
> into Wireshark, this will allow you to find out everything about the
> requests coming into the node from the client, as long as the connection is
> not encrypted.
>
> I wrote this up here a few months ago:
> http://www.redshots.com/finding-rogue-cassandra-queries/
>
> I hope this helps.
>
> Paul
>
>
>
>
>
> On 26 Sep 2019, at 10:21, Laxmikant Upadhyay 
> wrote:
>
> One of the way to figure out  what queries have run is to use audit
> logging  plugin supported in 3.x, 2.2
> https://github.com/Ericsson/ecaudit
>
> On Thu, Sep 26, 2019 at 2:19 PM shalom sagges 
> wrote:
>
>> Thanks for the quick response Jeff!
>>
>> The EXECUTE lines are a prepared statement with the specified number of
>> parameters.
>> Is it possible to find out on which keyspace/table these prepared
>> statements run?
>> Can I get additional information from the prepared statement's ID? e.g.
>> EXECUTE *d67e6a07c24b675f492686078b46c9**97*
>>
>> Thanks!
>>
>> On Thu, Sep 26, 2019 at 11:14 AM Jeff Jirsa  wrote:
>>
>>> The EXECUTE lines are a prepared statement with the specified number of
>>> parameters.
>>>
>>>
>>> On Wed, Sep 25, 2019 at 11:38 PM shalom sagges 
>>> wrote:
>>>
>>>> Hi All,
>>>>
>>>> I've been trying to find which queries are run on a Cassandra node.
>>>> I've enabled DEBUG and ran *nodetool setlogginglevel
>>>> org.apache.cassandra.transport TRACE*
>>>>
>>>> I did get some queries, but it's definitely not all the queries that
>>>> are run on this database.
>>>> I've also found a lot of DEBUG [SharedPool-Worker-72] 2019-09-25
>>>> 06:29:16,674 Message.java:437 - Received: EXECUTE
>>>> 2a6022010ffaf55229262de917657d0f with 6 values at consistency LOCAL_QUORUM,
>>>> v=3 but I don't understand what information I can gain from that and why it
>>>> appears many times (a lot more then the queries I wish to track).
>>>>
>>>> Can someone help me understand this type of logging?
>>>> Thanks!
>>>> DEBUG [SharedPool-Worker-88] 2019-09-25 06:29:16,793 Message.java:437 -
>>>> Received: EXECUTE 2a6022010ffaf55229262de917657d0f with 6 values at
>>>> consistency LOCAL_QUORUM, v=3
>>>> DEBUG [SharedPool-Worker-87] 2019-09-25 06:29:16,780 Message.java:437 -
>>>> Received: EXECUTE 447fdb9c8dfae53fafd78c7583aeb0f1 with 3 values at
>>>> consistency LOCAL_QUORUM, v=3
>>>> DEBUG [SharedPool-Worker-86] 2019-09-25 06:29:16,770 Message.java:437 -
>>>> Received: EXECUTE db812ac40b66c326f728452350eb0ab2 with 3 values at
>>>> consistency LOCAL_QUORUM, v=3
>>>> DEBUG [SharedPool-Worker-84] 2019-09-25 06:29:16,761 Message.java:437 -
>>>> Received: EXECUTE 7119db57e0a2041206f62c6d48fb4329 with 3 values at
>>>> consistency LOCAL_QUORUM, v=3
>>>> DEBUG [SharedPool-Worker-82] 2019-09-25 06:29:16,759 Message.java:437 -
>>>> Received: QUERY UPDATE tbl1 SET col6=?,col7=?,col8=?,col9=? WHERE col1=?
>>>> AND col2=? AND col3=? AND col4=? AND col5=?;, v=3
>>>> DEBUG [SharedPool-Worker-85] 2019-09-25 06:29:16,751 Message.java:437 -
>>>> Received: EXECUTE 2cddc1f6af3c6efbeaf435f9b7ec1c8a with 4 values at
>>>> consistency LOCAL_ONE, v=3
>>>> DEBUG [SharedPool-Worker-83] 2019-09-25 06:29:16,745 Message.java:437 -
>>>> Received: EXECUTE db812ac40b66c326f728452350eb0ab2 with 3 values at
>>>> consistency LOCAL_QUORUM, v=3
>>>> DEBUG [SharedPool-Worker-81] 2019-09-25 06:29:16,734 Message.java:437 -
>>>> Received: EXECUTE 7119db57e0a2041206f62c6d48fb4329 with 3 values at
>>>> consistency LOCAL_QUORUM, v=3
>>>> DEBUG [SharedPool-Worker-79] 2019-09-25 06:29:16,732 Message.java:437 -
>>>> Received: EXECUTE e779e97bc0de5e5e121db71c5cb2b727 with 11 values at
>>>> consistency LOCAL_QUORUM, v=3
>>>> DEBUG [SharedPool-Worker-80] 2019-09-25 06:29:16,731 Message.java:437 -
>>>> Received: EXECUTE 91af551f94a4394b96ef9afff71dfcc1 with 2 values at
>>>> consistency LOCAL_QUORUM, v=3
>>>> DEB

Re: Understanding TRACE logging

[shalom sagges]

Thanks for the quick response Jeff!

The EXECUTE lines are a prepared statement with the specified number of
parameters.
Is it possible to find out on which keyspace/table these prepared
statements run?
Can I get additional information from the prepared statement's ID? e.g.
EXECUTE *d67e6a07c24b675f492686078b46c9**97*

Thanks!

On Thu, Sep 26, 2019 at 11:14 AM Jeff Jirsa  wrote:

> The EXECUTE lines are a prepared statement with the specified number of
> parameters.
>
>
> On Wed, Sep 25, 2019 at 11:38 PM shalom sagges 
> wrote:
>
>> Hi All,
>>
>> I've been trying to find which queries are run on a Cassandra node.
>> I've enabled DEBUG and ran *nodetool setlogginglevel
>> org.apache.cassandra.transport TRACE*
>>
>> I did get some queries, but it's definitely not all the queries that are
>> run on this database.
>> I've also found a lot of DEBUG [SharedPool-Worker-72] 2019-09-25
>> 06:29:16,674 Message.java:437 - Received: EXECUTE
>> 2a6022010ffaf55229262de917657d0f with 6 values at consistency LOCAL_QUORUM,
>> v=3 but I don't understand what information I can gain from that and why it
>> appears many times (a lot more then the queries I wish to track).
>>
>> Can someone help me understand this type of logging?
>> Thanks!
>> DEBUG [SharedPool-Worker-88] 2019-09-25 06:29:16,793 Message.java:437 -
>> Received: EXECUTE 2a6022010ffaf55229262de917657d0f with 6 values at
>> consistency LOCAL_QUORUM, v=3
>> DEBUG [SharedPool-Worker-87] 2019-09-25 06:29:16,780 Message.java:437 -
>> Received: EXECUTE 447fdb9c8dfae53fafd78c7583aeb0f1 with 3 values at
>> consistency LOCAL_QUORUM, v=3
>> DEBUG [SharedPool-Worker-86] 2019-09-25 06:29:16,770 Message.java:437 -
>> Received: EXECUTE db812ac40b66c326f728452350eb0ab2 with 3 values at
>> consistency LOCAL_QUORUM, v=3
>> DEBUG [SharedPool-Worker-84] 2019-09-25 06:29:16,761 Message.java:437 -
>> Received: EXECUTE 7119db57e0a2041206f62c6d48fb4329 with 3 values at
>> consistency LOCAL_QUORUM, v=3
>> DEBUG [SharedPool-Worker-82] 2019-09-25 06:29:16,759 Message.java:437 -
>> Received: QUERY UPDATE tbl1 SET col6=?,col7=?,col8=?,col9=? WHERE col1=?
>> AND col2=? AND col3=? AND col4=? AND col5=?;, v=3
>> DEBUG [SharedPool-Worker-85] 2019-09-25 06:29:16,751 Message.java:437 -
>> Received: EXECUTE 2cddc1f6af3c6efbeaf435f9b7ec1c8a with 4 values at
>> consistency LOCAL_ONE, v=3
>> DEBUG [SharedPool-Worker-83] 2019-09-25 06:29:16,745 Message.java:437 -
>> Received: EXECUTE db812ac40b66c326f728452350eb0ab2 with 3 values at
>> consistency LOCAL_QUORUM, v=3
>> DEBUG [SharedPool-Worker-81] 2019-09-25 06:29:16,734 Message.java:437 -
>> Received: EXECUTE 7119db57e0a2041206f62c6d48fb4329 with 3 values at
>> consistency LOCAL_QUORUM, v=3
>> DEBUG [SharedPool-Worker-79] 2019-09-25 06:29:16,732 Message.java:437 -
>> Received: EXECUTE e779e97bc0de5e5e121db71c5cb2b727 with 11 values at
>> consistency LOCAL_QUORUM, v=3
>> DEBUG [SharedPool-Worker-80] 2019-09-25 06:29:16,731 Message.java:437 -
>> Received: EXECUTE 91af551f94a4394b96ef9afff71dfcc1 with 2 values at
>> consistency LOCAL_QUORUM, v=3
>> DEBUG [SharedPool-Worker-78] 2019-09-25 06:29:16,731 Message.java:437 -
>> Received: EXECUTE 2a6022010ffaf55229262de917657d0f with 6 values at
>> consistency LOCAL_QUORUM, v=3
>> DEBUG [SharedPool-Worker-75] 2019-09-25 06:29:16,720 Message.java:437 -
>> Received: EXECUTE b665e5f576dfe70845269d63b485c8ee with 2 values at
>> consistency LOCAL_QUORUM, v=3
>> DEBUG [SharedPool-Worker-77] 2019-09-25 06:29:16,715 Message.java:437 -
>> Received: EXECUTE ce545d85a7ee7c8ad58875afa72d9cf6 with 3 values at
>> consistency LOCAL_QUORUM, v=3
>> DEBUG [SharedPool-Worker-74] 2019-09-25 06:29:16,703 Message.java:437 -
>> Received: EXECUTE 7119db57e0a2041206f62c6d48fb4329 with 3 values at
>> consistency LOCAL_QUORUM, v=3
>> DEBUG [SharedPool-Worker-76] 2019-09-25 06:29:16,686 Message.java:437 -
>> Received: EXECUTE b665e5f576dfe70845269d63b485c8ee with 2 values at
>> consistency LOCAL_QUORUM, v=3
>> DEBUG [SharedPool-Worker-71] 2019-09-25 06:29:16,682 Message.java:437 -
>> Received: EXECUTE 2a6022010ffaf55229262de917657d0f with 6 values at
>> consistency LOCAL_QUORUM, v=3
>> DEBUG [SharedPool-Worker-73] 2019-09-25 06:29:16,675 Message.java:437 -
>> Received: EXECUTE b665e5f576dfe70845269d63b485c8ee with 2 values at
>> consistency LOCAL_QUORUM, v=3
>> DEBUG [SharedPool-Worker-72] 2019-09-25 06:29:16,674 Message.java:437 -
>> Received: EXECUTE 2a6022010ffaf55229262de917657d0f with 6 values at
>> consistency LOCAL_QUORUM, v=3
>> DEBUG [SharedPool-Worker-69] 2019-09-25 06:29:16

Understanding TRACE logging

[shalom sagges]

Hi All,

I've been trying to find which queries are run on a Cassandra node.
I've enabled DEBUG and ran *nodetool setlogginglevel
org.apache.cassandra.transport TRACE*

I did get some queries, but it's definitely not all the queries that are
run on this database.
I've also found a lot of DEBUG [SharedPool-Worker-72] 2019-09-25
06:29:16,674 Message.java:437 - Received: EXECUTE
2a6022010ffaf55229262de917657d0f with 6 values at consistency LOCAL_QUORUM,
v=3 but I don't understand what information I can gain from that and why it
appears many times (a lot more then the queries I wish to track).

Can someone help me understand this type of logging?
Thanks!
DEBUG [SharedPool-Worker-88] 2019-09-25 06:29:16,793 Message.java:437 -
Received: EXECUTE 2a6022010ffaf55229262de917657d0f with 6 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-87] 2019-09-25 06:29:16,780 Message.java:437 -
Received: EXECUTE 447fdb9c8dfae53fafd78c7583aeb0f1 with 3 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-86] 2019-09-25 06:29:16,770 Message.java:437 -
Received: EXECUTE db812ac40b66c326f728452350eb0ab2 with 3 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-84] 2019-09-25 06:29:16,761 Message.java:437 -
Received: EXECUTE 7119db57e0a2041206f62c6d48fb4329 with 3 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-82] 2019-09-25 06:29:16,759 Message.java:437 -
Received: QUERY UPDATE tbl1 SET col6=?,col7=?,col8=?,col9=? WHERE col1=?
AND col2=? AND col3=? AND col4=? AND col5=?;, v=3
DEBUG [SharedPool-Worker-85] 2019-09-25 06:29:16,751 Message.java:437 -
Received: EXECUTE 2cddc1f6af3c6efbeaf435f9b7ec1c8a with 4 values at
consistency LOCAL_ONE, v=3
DEBUG [SharedPool-Worker-83] 2019-09-25 06:29:16,745 Message.java:437 -
Received: EXECUTE db812ac40b66c326f728452350eb0ab2 with 3 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-81] 2019-09-25 06:29:16,734 Message.java:437 -
Received: EXECUTE 7119db57e0a2041206f62c6d48fb4329 with 3 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-79] 2019-09-25 06:29:16,732 Message.java:437 -
Received: EXECUTE e779e97bc0de5e5e121db71c5cb2b727 with 11 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-80] 2019-09-25 06:29:16,731 Message.java:437 -
Received: EXECUTE 91af551f94a4394b96ef9afff71dfcc1 with 2 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-78] 2019-09-25 06:29:16,731 Message.java:437 -
Received: EXECUTE 2a6022010ffaf55229262de917657d0f with 6 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-75] 2019-09-25 06:29:16,720 Message.java:437 -
Received: EXECUTE b665e5f576dfe70845269d63b485c8ee with 2 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-77] 2019-09-25 06:29:16,715 Message.java:437 -
Received: EXECUTE ce545d85a7ee7c8ad58875afa72d9cf6 with 3 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-74] 2019-09-25 06:29:16,703 Message.java:437 -
Received: EXECUTE 7119db57e0a2041206f62c6d48fb4329 with 3 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-76] 2019-09-25 06:29:16,686 Message.java:437 -
Received: EXECUTE b665e5f576dfe70845269d63b485c8ee with 2 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-71] 2019-09-25 06:29:16,682 Message.java:437 -
Received: EXECUTE 2a6022010ffaf55229262de917657d0f with 6 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-73] 2019-09-25 06:29:16,675 Message.java:437 -
Received: EXECUTE b665e5f576dfe70845269d63b485c8ee with 2 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-72] 2019-09-25 06:29:16,674 Message.java:437 -
Received: EXECUTE 2a6022010ffaf55229262de917657d0f with 6 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-69] 2019-09-25 06:29:16,644 Message.java:437 -
Received: EXECUTE 2cddc1f6af3c6efbeaf435f9b7ec1c8a with 4 values at
consistency LOCAL_ONE, v=3
DEBUG [SharedPool-Worker-68] 2019-09-25 06:29:16,635 Message.java:437 -
Received: EXECUTE b665e5f576dfe70845269d63b485c8ee with 2 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-53] 2019-09-25 06:29:16,635 Message.java:437 -
Received: EXECUTE e779e97bc0de5e5e121db71c5cb2b727 with 11 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-66] 2019-09-25 06:29:16,635 Message.java:437 -
Received: EXECUTE 447fdb9c8dfae53fafd78c7583aeb0f1 with 3 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-65] 2019-09-25 06:29:16,623 Message.java:437 -
Received: EXECUTE d67e6a07c24b675f492686078b46c997 with 3 values at
consistency LOCAL_ONE, v=3
DEBUG [SharedPool-Worker-61] 2019-09-25 06:29:16,621 Message.java:437 -
Received: QUERY SELECT column4 FROM ks2.tbl2 WHERE column1='' AND
column2='' AND ts1>1569358692193;, v=3
DEBUG [SharedPool-Worker-62] 2019-09-25 06:29:16,618 Message.java:437 -
Received: EXECUTE d67e6a07c24b675f492686078b46c997 with 3 values at
consistency LOCAL_ONE, v=3

Re: AbstractLocalAwareExecutorService Exception During Upgrade

[shalom sagges]

Hi Again,

Trying to push this up as I wasn't able to find the root cause of this
issue.
Perhaps I need to upgrade to 3.0 first?
Will be happy to get some ideas.

Opened https://issues.apache.org/jira/browse/CASSANDRA-15172 with more
details.

Thanks!

On Thu, Jun 6, 2019 at 5:31 AM Jonathan Koppenhofer 
wrote:

> Not sure about why repair is running, but we are also seeing the same
> merkle tree issue in a mixed version cluster in which we have intentionally
> started a repair against 2 upgraded DCs. We are currently researching, and
> can post back if we find the issue, but also would appreciate if someone
> has a suggestion. We have also run a local repair in an upgraded DC in this
> same mixed version cluster without issue.
>
> We are going 2.1.x to 3.0.x... and yes, we know you are not supposed to
> run repairs in mixed version clusters, so don't do it :) this is kind of a
> special circumstances where other things have gone wrong.
>
> Thanks
>
> On Wed, Jun 5, 2019, 5:23 PM shalom sagges  wrote:
>
>> If anyone has any idea on what might cause this issue, it'd be great.
>>
>> I don't understand what could trigger this exception.
>> But what I really can't understand is why repairs started to run suddenly
>> :-\
>> There's no cron job running, no active repair process, no Validation
>> compactions, Reaper is turned off  I see repair running only in the
>> logs.
>>
>> Thanks!
>>
>>
>> On Wed, Jun 5, 2019 at 2:32 PM shalom sagges 
>> wrote:
>>
>>> Hi All,
>>>
>>> I'm having a bad situation where after upgrading 2 nodes (binaries only)
>>> from 2.1.21 to 3.11.4 I'm getting a lot of warnings as follows:
>>>
>>> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on
>>> thread Thread[ReadStage-5,5,main]: {}
>>> java.lang.ArrayIndexOutOfBoundsException: null
>>>
>>>
>>> I also see errors on repairs but no repair is running at all. I verified
>>> this with ps -ef command and nodetool compactionstats. The error I see is:
>>> Failed creating a merkle tree for [repair
>>> #a95498f0-8783-11e9-b065-81cdbc6bee08 on system_auth/users, []], /
>>> 1.2.3.4 (see log for details)
>>>
>>> I saw repair errors on data tables as well.
>>> nodetool status shows all are UN and nodetool describecluster shows two
>>> schema versions as expected.
>>>
>>>
>>> After the warnings appeared, clients started to get timed out read/write
>>> queries.
>>> Restarting the 2 nodes solved the clients' connection issues, but the
>>> warnings are still being generated in the logs.
>>>
>>> Did anyone encounter such an issue and knows what this means?
>>>
>>> Thanks!
>>>
>>>

Re: AbstractLocalAwareExecutorService Exception During Upgrade

[shalom sagges]

If anyone has any idea on what might cause this issue, it'd be great.

I don't understand what could trigger this exception.
But what I really can't understand is why repairs started to run suddenly
:-\
There's no cron job running, no active repair process, no Validation
compactions, Reaper is turned off  I see repair running only in the
logs.

Thanks!


On Wed, Jun 5, 2019 at 2:32 PM shalom sagges  wrote:

> Hi All,
>
> I'm having a bad situation where after upgrading 2 nodes (binaries only)
> from 2.1.21 to 3.11.4 I'm getting a lot of warnings as follows:
>
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[ReadStage-5,5,main]: {}
> java.lang.ArrayIndexOutOfBoundsException: null
>
>
> I also see errors on repairs but no repair is running at all. I verified
> this with ps -ef command and nodetool compactionstats. The error I see is:
> Failed creating a merkle tree for [repair
> #a95498f0-8783-11e9-b065-81cdbc6bee08 on system_auth/users, []], /1.2.3.4
> (see log for details)
>
> I saw repair errors on data tables as well.
> nodetool status shows all are UN and nodetool describecluster shows two
> schema versions as expected.
>
>
> After the warnings appeared, clients started to get timed out read/write
> queries.
> Restarting the 2 nodes solved the clients' connection issues, but the
> warnings are still being generated in the logs.
>
> Did anyone encounter such an issue and knows what this means?
>
> Thanks!
>
>

AbstractLocalAwareExecutorService Exception During Upgrade

[shalom sagges]

Hi All,

I'm having a bad situation where after upgrading 2 nodes (binaries only)
from 2.1.21 to 3.11.4 I'm getting a lot of warnings as follows:

AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-5,5,main]: {}
java.lang.ArrayIndexOutOfBoundsException: null


I also see errors on repairs but no repair is running at all. I verified
this with ps -ef command and nodetool compactionstats. The error I see is:
Failed creating a merkle tree for [repair
#a95498f0-8783-11e9-b065-81cdbc6bee08 on system_auth/users, []], /1.2.3.4
(see log for details)

I saw repair errors on data tables as well.
nodetool status shows all are UN and nodetool describecluster shows two
schema versions as expected.


After the warnings appeared, clients started to get timed out read/write
queries.
Restarting the 2 nodes solved the clients' connection issues, but the
warnings are still being generated in the logs.

Did anyone encounter such an issue and knows what this means?

Thanks!

Re: Collecting Latency Metrics

[shalom sagges]

Thanks a lot for your comments.
This mailing list is truly *the *definitive guide to Cassandra
*. *
The knowledge transferred here is invaluable.
So just wanted to give a big shout out to anyone who is helping out here.

Regards,

On Thu, May 30, 2019 at 6:10 PM Jon Haddad  wrote:

> Yep.  I would *never* use mean when it comes to performance to make any
> sort of decisions.  I prefer to graph all the p99 latencies as well as the
> max.
>
> Some good reading on the topic:
> https://bravenewgeek.com/everything-you-know-about-latency-is-wrong/
>
> On Thu, May 30, 2019 at 7:35 AM Chris Lohfink 
> wrote:
>
>> For what it is worth, generally I would recommend just using the mean vs
>> calculating it yourself. It's a lot easier and averages are meaningless for
>> anything besides trending anyway (which is really what this is useful for,
>> finding issues on the larger scale), especially with high volume clusters
>> so the loss in accuracy kinda moot. Your average for local reads/writes
>> will almost always be sub millisecond but you might end up having 500
>> millisecond requests or worse that the mean will hide.
>>
>> Chris
>>
>> On Thu, May 30, 2019 at 6:30 AM shalom sagges 
>> wrote:
>>
>>> Thanks for your replies guys. I really appreciate it.
>>>
>>> @Alain, I use Graphite for backend on top of Grafana. But the goal is to
>>> move from Graphite to Prometheus eventually.
>>>
>>> I tried to find a direct way of getting a specific Latency metric in
>>> average and as Chris pointed out, then Mean value isn't that accurate.
>>> I do not wish to use the percentile metrics either, but a single latency
>>> metric like the *"Local read latency" *output in nodetool tablestats.
>>> Looking at the code of nodetool tablestats, it seems that C* also
>>> divides *ReadTotalLatency.Count* with *ReadLatency.Count *to get the
>>> latency result.
>>>
>>> So I guess I will have no choice but to run the calculation on my own
>>> via Graphite:
>>>
>>> divideSeries(averageSeries(keepLastValue(nonNegativeDerivative($env.path.to.host.$host.org_apache_cassandra_metrics.Table.$ks.$cf.ReadTotalLatency.Count))),averageSeries(keepLastValue(nonNegativeDerivative($env.path.to.host.$host.org_apache_cassandra_metrics.Table.$ks.$cf.ReadLatency.Count
>>>
>>> Does this seem right to you?
>>>
>>> Thanks!
>>>
>>> On Thu, May 30, 2019 at 12:34 AM Paul Chandler 
>>> wrote:
>>>
>>>> There are various attributes under
>>>> org.apache.cassandra.metrics.ClientRequest.Latency.Read these measure the
>>>> latency in milliseconds
>>>>
>>>> Thanks
>>>>
>>>> Paul
>>>> www.redshots.com
>>>>
>>>> > On 29 May 2019, at 15:31, shalom sagges 
>>>> wrote:
>>>> >
>>>> > Hi All,
>>>> >
>>>> > I'm creating a dashboard that should collect read/write latency
>>>> metrics on C* 3.x.
>>>> > In older versions (e.g. 2.0) I used to divide the total read latency
>>>> in microseconds with the read count.
>>>> >
>>>> > Is there a metric attribute that shows read/write latency without the
>>>> need to do the math, such as in nodetool tablestats "Local read latency"
>>>> output?
>>>> > I saw there's a Mean attribute in
>>>> org.apache.cassandra.metrics.ReadLatency but I'm not sure this is the right
>>>> one.
>>>> >
>>>> > I'd really appreciate your help on this one.
>>>> > Thanks!
>>>> >
>>>> >
>>>>
>>>>
>>>> -
>>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>>>
>>>>

Re: Collecting Latency Metrics

[shalom sagges]

Sorry for the duplicated emails but I just want to make sure I'm doing
it correctly:
To summarize, are both ways accurate or one is better than the other?

divideSeries(averageSeries(keepLastValue(nonNegativeDerivative($env.path.to.host.$host.org_apache_cassandra_metrics.Table.$ks.$cf.ReadTotalLatency.Count))),averageSeries(keepLastValue(nonNegativeDerivative($env.path.to.host.$host.org_apache_cassandra_metrics.Table.$ks.$cf.ReadLatency.Count

OR

alias(scaleToSeconds(averageSeriesWithWildcards(nonNegativeDerivative($env.path.to.host.$host.org_apache_cassandra_metrics.Table.$ks.$cf.ReadTotalLatency.Count),7,8,9),1),'test')

WDYT?


On Thu, May 30, 2019 at 2:29 PM shalom sagges 
wrote:

> Thanks for your replies guys. I really appreciate it.
>
> @Alain, I use Graphite for backend on top of Grafana. But the goal is to
> move from Graphite to Prometheus eventually.
>
> I tried to find a direct way of getting a specific Latency metric in
> average and as Chris pointed out, then Mean value isn't that accurate.
> I do not wish to use the percentile metrics either, but a single latency
> metric like the *"Local read latency" *output in nodetool tablestats.
> Looking at the code of nodetool tablestats, it seems that C* also divides
> *ReadTotalLatency.Count* with *ReadLatency.Count *to get the latency
> result.
>
> So I guess I will have no choice but to run the calculation on my own via
> Graphite:
>
> divideSeries(averageSeries(keepLastValue(nonNegativeDerivative($env.path.to.host.$host.org_apache_cassandra_metrics.Table.$ks.$cf.ReadTotalLatency.Count))),averageSeries(keepLastValue(nonNegativeDerivative($env.path.to.host.$host.org_apache_cassandra_metrics.Table.$ks.$cf.ReadLatency.Count
>
> Does this seem right to you?
>
> Thanks!
>
> On Thu, May 30, 2019 at 12:34 AM Paul Chandler  wrote:
>
>> There are various attributes under
>> org.apache.cassandra.metrics.ClientRequest.Latency.Read these measure the
>> latency in milliseconds
>>
>> Thanks
>>
>> Paul
>> www.redshots.com
>>
>> > On 29 May 2019, at 15:31, shalom sagges  wrote:
>> >
>> > Hi All,
>> >
>> > I'm creating a dashboard that should collect read/write latency metrics
>> on C* 3.x.
>> > In older versions (e.g. 2.0) I used to divide the total read latency in
>> microseconds with the read count.
>> >
>> > Is there a metric attribute that shows read/write latency without the
>> need to do the math, such as in nodetool tablestats "Local read latency"
>> output?
>> > I saw there's a Mean attribute in
>> org.apache.cassandra.metrics.ReadLatency but I'm not sure this is the right
>> one.
>> >
>> > I'd really appreciate your help on this one.
>> > Thanks!
>> >
>> >
>>
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>

Re: Collecting Latency Metrics

[shalom sagges]

Thanks for your replies guys. I really appreciate it.

@Alain, I use Graphite for backend on top of Grafana. But the goal is to
move from Graphite to Prometheus eventually.

I tried to find a direct way of getting a specific Latency metric in
average and as Chris pointed out, then Mean value isn't that accurate.
I do not wish to use the percentile metrics either, but a single latency
metric like the *"Local read latency" *output in nodetool tablestats.
Looking at the code of nodetool tablestats, it seems that C* also divides
*ReadTotalLatency.Count* with *ReadLatency.Count *to get the latency
result.

So I guess I will have no choice but to run the calculation on my own via
Graphite:
divideSeries(averageSeries(keepLastValue(nonNegativeDerivative($env.path.to.host.$host.org_apache_cassandra_metrics.Table.$ks.$cf.ReadTotalLatency.Count))),averageSeries(keepLastValue(nonNegativeDerivative($env.path.to.host.$host.org_apache_cassandra_metrics.Table.$ks.$cf.ReadLatency.Count

Does this seem right to you?

Thanks!

On Thu, May 30, 2019 at 12:34 AM Paul Chandler  wrote:

> There are various attributes under
> org.apache.cassandra.metrics.ClientRequest.Latency.Read these measure the
> latency in milliseconds
>
> Thanks
>
> Paul
> www.redshots.com
>
> > On 29 May 2019, at 15:31, shalom sagges  wrote:
> >
> > Hi All,
> >
> > I'm creating a dashboard that should collect read/write latency metrics
> on C* 3.x.
> > In older versions (e.g. 2.0) I used to divide the total read latency in
> microseconds with the read count.
> >
> > Is there a metric attribute that shows read/write latency without the
> need to do the math, such as in nodetool tablestats "Local read latency"
> output?
> > I saw there's a Mean attribute in
> org.apache.cassandra.metrics.ReadLatency but I'm not sure this is the right
> one.
> >
> > I'd really appreciate your help on this one.
> > Thanks!
> >
> >
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: Collecting Latency Metrics

[shalom sagges]

If I only send ReadTotalLatency to Graphite/Grafana, can I run an average
on it and use "scale to seconds=1" ?
Will that do the trick?

Thanks!

On Wed, May 29, 2019 at 5:31 PM shalom sagges 
wrote:

> Hi All,
>
> I'm creating a dashboard that should collect read/write latency metrics on
> C* 3.x.
> In older versions (e.g. 2.0) I used to divide the total read latency in
> microseconds with the read count.
>
> Is there a metric attribute that shows read/write latency without the need
> to do the math, such as in nodetool tablestats "Local read latency" output?
> I saw there's a Mean attribute in org.apache.cassandra.metrics.ReadLatency
> but I'm not sure this is the right one.
>
> I'd really appreciate your help on this one.
> Thanks!
>
>
>

Collecting Latency Metrics

[shalom sagges]

Hi All,

I'm creating a dashboard that should collect read/write latency metrics on
C* 3.x.
In older versions (e.g. 2.0) I used to divide the total read latency in
microseconds with the read count.

Is there a metric attribute that shows read/write latency without the need
to do the math, such as in nodetool tablestats "Local read latency" output?
I saw there's a Mean attribute in org.apache.cassandra.metrics.ReadLatency
but I'm not sure this is the right one.

I'd really appreciate your help on this one.
Thanks!

Re: Select in allow filtering stalls whole cluster. How to prevent such behavior?

[shalom sagges]

Hi Attila,

I'm definitely no guru, but I've experienced several cases where people at
my company used allow filtering and caused major performance issues.
As data size increases, the impact will be stronger. If you have large
partitions, performance will decrease.
GC can be affected. And if GC stops the world too long for too many times,
you will feel it.

I sincerely believe the best way would be to educate the users and remodel
the data. Perhaps you need to denormalize your tables or at least use
secondary indices (I prefer to keep it as simple as possible and
denormalize).
If it's a cluster for analytics, perhaps you need to build a designated
cluster only for that so if something does break or get too pressured,
normal activities wouldn't be affected, but there are pros and cons for
that idea too.

Hope this helps.

Regards,


On Tue, May 28, 2019 at 9:43 AM Attila Wind  wrote:

> Hi Gurus,
>
> Looks we stopped this thread. However I would be very much curious answers
> regarding b) ...
>
> Anyone any comments on that?
> I do see this as a potential production outage risk now... Especially as
> we are planning to run analysis queries by hand exactly like that over the
> cluster...
>
> thanks!
> Attila Wind
>
> http://www.linkedin.com/in/attilaw
> Mobile: +36 31 7811355
>
>
> On 2019. 05. 23. 11:42, shalom sagges wrote:
>
> a) Interesting... But only in case you do not provide partitioning key
> right? (so IN() is for partitioning key?)
>
> I think you should ask yourself a different question. Why am I using ALLOW
> FILTERING in the first place? What happens if I remove it from the query?
> I prefer to denormalize the data to multiple tables or at least create an
> index on the requested column (preferably queried together with a known
> partition key).
>
> b) Still does not explain or justify "all 8 nodes to halt and
> unresponsiveness to external requests" behavior... Even if servers are busy
> with the request seriously becoming non-responsive...?
>
> I think it can justify the unresponsiveness. When using ALLOW FILTERING,
> you are doing something like a full table scan in a relational database.
>
> There is a lot of information on the internet regarding this subject such
> as
> https://www.instaclustr.com/apache-cassandra-scalability-allow-filtering-partition-keys/
>
> Hope this helps.
>
> Regards,
>
> On Thu, May 23, 2019 at 7:33 AM Attila Wind 
>  wrote:
>
>> Hi,
>>
>> "When you run a query with allow filtering, Cassandra doesn't know where
>> the data is located, so it has to go node by node, searching for the
>> requested data."
>>
>> a) Interesting... But only in case you do not provide partitioning key
>> right? (so IN() is for partitioning key?)
>>
>> b) Still does not explain or justify "all 8 nodes to halt and
>> unresponsiveness to external requests" behavior... Even if servers are busy
>> with the request seriously becoming non-responsive...?
>>
>> cheers
>> Attila Wind
>>
>> http://www.linkedin.com/in/attilaw
>> Mobile: +36 31 7811355
>>
>>
>> On 2019. 05. 23. 0:37, shalom sagges wrote:
>>
>> Hi Vsevolod,
>>
>> 1) Why such behavior? I thought any given SELECT request is handled by a
>> limited subset of C* nodes and not by all of them, as per connection
>> consistency/table replication settings, in case.
>> When you run a query with allow filtering, Cassandra doesn't know where
>> the data is located, so it has to go node by node, searching for the
>> requested data.
>>
>> 2) Is it possible to forbid ALLOW FILTERING flag for given users/groups?
>> I'm not familiar with such a flag. In my case, I just try to educate the
>> R teams.
>>
>> Regards,
>>
>> On Wed, May 22, 2019 at 5:01 PM Vsevolod Filaretov 
>> wrote:
>>
>>> Hello everyone,
>>>
>>> We have an 8 node C* cluster with large volume of unbalanced data. Usual
>>> per-partition selects work somewhat fine, and are processed by limited
>>> number of nodes, but if user issues SELECT WHERE IN () ALLOW FILTERING,
>>> such command stalls all 8 nodes to halt and unresponsiveness to external
>>> requests while disk IO jumps to 100% across whole cluster. In several
>>> minutes all nodes seem to finish ptocessing the request and cluster goes
>>> back to being responsive. Replication level across whole data is 3.
>>>
>>> 1) Why such behavior? I thought any given SELECT request is handled by a
>>> limited subset of C* nodes and not by all of them, as per connection
>>> consistency/table replication settings, in case.
>>>
>>> 2) Is it possible to forbid ALLOW FILTERING flag for given users/groups?
>>>
>>> Thank you all very much in advance,
>>> Vsevolod Filaretov.
>>>
>>

Re: Select in allow filtering stalls whole cluster. How to prevent such behavior?

[shalom sagges]

a) Interesting... But only in case you do not provide partitioning key
right? (so IN() is for partitioning key?)

I think you should ask yourself a different question. Why am I using ALLOW
FILTERING in the first place? What happens if I remove it from the query?
I prefer to denormalize the data to multiple tables or at least create an
index on the requested column (preferably queried together with a known
partition key).

b) Still does not explain or justify "all 8 nodes to halt and
unresponsiveness to external requests" behavior... Even if servers are busy
with the request seriously becoming non-responsive...?

I think it can justify the unresponsiveness. When using ALLOW FILTERING,
you are doing something like a full table scan in a relational database.

There is a lot of information on the internet regarding this subject such
as
https://www.instaclustr.com/apache-cassandra-scalability-allow-filtering-partition-keys/

Hope this helps.

Regards,

On Thu, May 23, 2019 at 7:33 AM Attila Wind  wrote:

> Hi,
>
> "When you run a query with allow filtering, Cassandra doesn't know where
> the data is located, so it has to go node by node, searching for the
> requested data."
>
> a) Interesting... But only in case you do not provide partitioning key
> right? (so IN() is for partitioning key?)
>
> b) Still does not explain or justify "all 8 nodes to halt and
> unresponsiveness to external requests" behavior... Even if servers are busy
> with the request seriously becoming non-responsive...?
>
> cheers
> Attila Wind
>
> http://www.linkedin.com/in/attilaw
> Mobile: +36 31 7811355
>
>
> On 2019. 05. 23. 0:37, shalom sagges wrote:
>
> Hi Vsevolod,
>
> 1) Why such behavior? I thought any given SELECT request is handled by a
> limited subset of C* nodes and not by all of them, as per connection
> consistency/table replication settings, in case.
> When you run a query with allow filtering, Cassandra doesn't know where
> the data is located, so it has to go node by node, searching for the
> requested data.
>
> 2) Is it possible to forbid ALLOW FILTERING flag for given users/groups?
> I'm not familiar with such a flag. In my case, I just try to educate the
> R teams.
>
> Regards,
>
> On Wed, May 22, 2019 at 5:01 PM Vsevolod Filaretov 
> wrote:
>
>> Hello everyone,
>>
>> We have an 8 node C* cluster with large volume of unbalanced data. Usual
>> per-partition selects work somewhat fine, and are processed by limited
>> number of nodes, but if user issues SELECT WHERE IN () ALLOW FILTERING,
>> such command stalls all 8 nodes to halt and unresponsiveness to external
>> requests while disk IO jumps to 100% across whole cluster. In several
>> minutes all nodes seem to finish ptocessing the request and cluster goes
>> back to being responsive. Replication level across whole data is 3.
>>
>> 1) Why such behavior? I thought any given SELECT request is handled by a
>> limited subset of C* nodes and not by all of them, as per connection
>> consistency/table replication settings, in case.
>>
>> 2) Is it possible to forbid ALLOW FILTERING flag for given users/groups?
>>
>> Thank you all very much in advance,
>> Vsevolod Filaretov.
>>
>

Re: Select in allow filtering stalls whole cluster. How to prevent such behavior?

[shalom sagges]

Hi Vsevolod,

1) Why such behavior? I thought any given SELECT request is handled by a
limited subset of C* nodes and not by all of them, as per connection
consistency/table replication settings, in case.
When you run a query with allow filtering, Cassandra doesn't know where the
data is located, so it has to go node by node, searching for the requested
data.

2) Is it possible to forbid ALLOW FILTERING flag for given users/groups?
I'm not familiar with such a flag. In my case, I just try to educate the
R teams.

Regards,

On Wed, May 22, 2019 at 5:01 PM Vsevolod Filaretov 
wrote:

> Hello everyone,
>
> We have an 8 node C* cluster with large volume of unbalanced data. Usual
> per-partition selects work somewhat fine, and are processed by limited
> number of nodes, but if user issues SELECT WHERE IN () ALLOW FILTERING,
> such command stalls all 8 nodes to halt and unresponsiveness to external
> requests while disk IO jumps to 100% across whole cluster. In several
> minutes all nodes seem to finish ptocessing the request and cluster goes
> back to being responsive. Replication level across whole data is 3.
>
> 1) Why such behavior? I thought any given SELECT request is handled by a
> limited subset of C* nodes and not by all of them, as per connection
> consistency/table replication settings, in case.
>
> 2) Is it possible to forbid ALLOW FILTERING flag for given users/groups?
>
> Thank you all very much in advance,
> Vsevolod Filaretov.
>

Re: Python driver concistency problem

[shalom sagges]

In a lot of cases, the issue is with the data model.
Can you describe the table?
Can you provide the query you use to retrieve the data?
What's the load on your cluster?
Are there lots of tombstones?

You can set the consistency level to ONE, just to check if you get
responses. Although normally I would never use ALL unless I run a DDL
command.
I prefer local_quorum if I want my consistency to be strong while keeping
Cassandra's high availability.

Regards,

Re: nodetool repair failing with "Validation failed in /X.X.X.X

[shalom sagges]

Hi Rhys,

I encountered this error after adding new SSTables to a cluster and running
nodetool refresh (v3.0.12).
The refresh worked, but after starting repairs on the cluster, I got the
"Validation failed in /X.X.X.X" error on the remote DC.
A rolling restart solved the issue for me.

Hope this helps!



On Sat, May 4, 2019 at 3:58 PM Rhys Campbell
 wrote:

>
> > Hello,
> >
> > I’m having issues running repair on an Apache Cassandra Cluster. I’m
> getting "Failed creating a merkle tree“ errors on the replication partner
> nodes. Anyone have any experience of this? I am running 2.2.13.
> >
> > Further details here…
> https://issues.apache.org/jira/projects/CASSANDRA/issues/CASSANDRA-15109?filter=allopenissues
> >
> > Best,
> >
> > Rhys
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: Accidentaly removed SSTables of unneeded data

[shalom sagges]

Hi Simon,

If you haven't did that already, try to drain and restart the node you
deleted the data from.
Then run the repair again.

Regards,

On Thu, May 2, 2019 at 5:53 PM Simon ELBAZ  wrote:

> Hi,
>
> I am running Cassandra v2.1 on a 3 node cluster.
>
> *# yum list installed | grep cassa*
> *cassandra21.noarch2.1.12-1
> @datastax*
> *cassandra21-tools.noarch  2.1.12-1
> @datastax   *
>
> Unfortunately, I accidentally removed the SSTables (using rm) (older than
> 10 days) of a table on the 3 nodes.
>
> Running 'nodetool repair' on one of the 3 nodes returns error. Whereas, it
> does not on another.
>
> I don't need to recover the lost data but I would like 'nodetool repair'
> not returning an error.
>
> Thanks for any advice.
>
> Simon
>

Re: Decommissioning a new node when the state is JOINING

[shalom sagges]

I would just stop the service of the joining node and then delete the data,
commit logs and saved caches.
After stopping the node while joining, the cluster will remove it from the
list (i.e. nodetool status) without the need to decommission.



On Tue, Apr 30, 2019 at 2:44 PM Akshay Bhardwaj <
akshay.bhardwaj1...@gmail.com> wrote:

> Hi Experts,
>
> I have a cassandra cluster running with 5 nodes. For some reason, I was
> creating a new cassandra cluster, but one of the nodes intended for new
> cluster had the same cassandra.yml file as the existing cluster. This
> resulted in the new node joining the existing cluster, making total no. of
> nodes as 6.
>
> As of now in "nodetool status" command, I see that the state of the new
> node is JOINING, and also rebalancing data with other nodes.
> What is the best way to decommission the node?
>
>1. Can I execute "nodetool decommission" immediately for the new node?
>2. Should I wait for the new node to finish sync, and decommission
>only after that?
>3. Any other quick approach without data loss for existing cluster?
>
>
> Thanks in advance!
>
> Akshay Bhardwaj
> +91-97111-33849
>

Re: A Question About Hints

[shalom sagges]

Everyone really should move off of the 2.x versions just like you are doing.

Tell me about it... But since there are a lot of groups involved, these
things take time unfortunately.


Thanks for your assistance Kenneth


On Mon, Mar 4, 2019 at 11:04 PM Kenneth Brotman
 wrote:

> Since you are in the process of upgrading, I’d do nothing on the settings
> right now.  But if you wanted to do something on the settings in the
> meantime, based on my read of the information available, I’d maybe double
> the default settings. The upgrade will help a lot of things as you know.
>
>
>
> Everyone really should move off of the 2.x versions just like you are
> doing.
>
>
>
> *From:* shalom sagges [mailto:shalomsag...@gmail.com]
> *Sent:* Monday, March 04, 2019 12:34 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: A Question About Hints
>
>
>
> See my comments inline.
>
>
>
> Do the 8 nodes clusters have the problem too?
>
> Yes
>
>
>
> To the same extent?
>
> It depends on the throughput, but basically the smaller clusters get low
> throughput, so the problem is naturally smaller.
>
>
>
> Is it any cluster across multi-DC’s?
>
> Yes
>
>
>
> Do all the clusters use nodes with similar specs?
>
> All nodes have similar specs within a cluster but different specs on
> different clusters.
>
>
>
> The version of Cassandra you are on can make a difference.  What version
> are you on?
>
> Currently I'm on various versions, 2.0.14, 2.1.15 and 3.0.12. In the
> process of upgrading to 3.11.4
>
>
>
> Did you see Edward Capriolo’s presentation at 26:19 into the YouTube video
> at: https://www.youtube.com/watch?v=uN4FtAjYmLU where he briefly mentions
> you can get into trouble if you go to fast or two slow?
>
> I guess you can say it about almost any parameter you change :)
>
>
>
> BTW, I thought the comments at the end of the article you mentioned were
> really good.
>
> The entire article is very good, but I wonder if it's still valid since it
> was created around 4 years ago.
>
>
>
> Thanks!
>
>
>
>
>
>
>
>
>
> On Mon, Mar 4, 2019 at 9:37 PM Kenneth Brotman 
> wrote:
>
> Makes sense  If you have time and don’t mind, could you answer the
> following:
>
> Do the 8 nodes clusters have the problem too?
>
> To the same extent?
>
> Is it just the clusters with the large node count?
>
> Is it any cluster across multi-DC’s?
>
> Do all the clusters use nodes with similar specs?
>
>
>
> The version of Cassandra you are on can make a difference.  What version
> are you on?
>
>
>
> Did you see Edward Capriolo’s presentation at 26:19 into the YouTube video
> at: https://www.youtube.com/watch?v=uN4FtAjYmLU where he briefly mentions
> you can get into trouble if you go to fast or two slow?
>
> BTW, I thought the comments at the end of the article you mentioned were
> really good.
>
>
>
>
>
>
>
> *From:* shalom sagges [mailto:shalomsag...@gmail.com]
> *Sent:* Monday, March 04, 2019 11:04 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: A Question About Hints
>
>
>
> It varies...
>
> Some clusters have 48 nodes, others 24 nodes and some 8 nodes.
>
> Both settings are on default.
>
>
>
> I’d try making a single conservative change to one or the other, measure
> and reassess.  Then do same to other setting.
>
> That's the plan, but I thought I might first get some valuable information
> from someone in the community that has already experienced in this type of
> change.
>
>
>
> Thanks!
>
>
>
> On Mon, Mar 4, 2019 at 8:27 PM Kenneth Brotman
>  wrote:
>
> It sounds like your use case might be appropriate for tuning those two
> settings some.
>
>
>
> How many nodes are in the cluster?
>
> Are both settings definitely on the default values currently?
>
>
>
> I’d try making a single conservative change to one or the other, measure
> and reassess.  Then do same to other setting.
>
>
>
> Then of course share your results with us.
>
>
>
> *From:* shalom sagges [mailto:shalomsag...@gmail.com]
> *Sent:* Monday, March 04, 2019 9:54 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: A Question About Hints
>
>
>
> Hi Kenneth,
>
>
>
> The concern is that in some cases, hints accumulate on nodes, and it takes
> a while until they are delivered (multi DCs).
>
> I see that whenever there are  a lot of hints in play,like after a rolling
> restart, the cluster works harder. That's why I want to decrease the hints
> delivery time.
>
> I didn't want to change the configuration blindly and thought th

Re: A Question About Hints

[shalom sagges]

See my comments inline.

Do the 8 nodes clusters have the problem too?
Yes

To the same extent?

It depends on the throughput, but basically the smaller clusters get low
throughput, so the problem is naturally smaller.


Is it any cluster across multi-DC’s?

Yes


Do all the clusters use nodes with similar specs?

All nodes have similar specs within a cluster but different specs on
different clusters.


The version of Cassandra you are on can make a difference.  What version
are you on?

Currently I'm on various versions, 2.0.14, 2.1.15 and 3.0.12. In the
process of upgrading to 3.11.4


Did you see Edward Capriolo’s presentation at 26:19 into the YouTube video
at: https://www.youtube.com/watch?v=uN4FtAjYmLU where he briefly mentions
you can get into trouble if you go to fast or two slow?

I guess you can say it about almost any parameter you change :)


BTW, I thought the comments at the end of the article you mentioned were
really good.

The entire article is very good, but I wonder if it's still valid since it
was created around 4 years ago.


Thanks!





On Mon, Mar 4, 2019 at 9:37 PM Kenneth Brotman 
wrote:

> Makes sense.  If you have time and don’t mind, could you answer the
> following:
>
> Do the 8 nodes clusters have the problem too?
>
> To the same extent?
>
> Is it just the clusters with the large node count?
>
> Is it any cluster across multi-DC’s?
>
> Do all the clusters use nodes with similar specs?
>
>
>
> The version of Cassandra you are on can make a difference.  What version
> are you on?
>
>
>
> Did you see Edward Capriolo’s presentation at 26:19 into the YouTube video
> at: https://www.youtube.com/watch?v=uN4FtAjYmLU where he briefly mentions
> you can get into trouble if you go to fast or two slow?
>
> BTW, I thought the comments at the end of the article you mentioned were
> really good.
>
>
>
>
>
>
>
> *From:* shalom sagges [mailto:shalomsag...@gmail.com]
> *Sent:* Monday, March 04, 2019 11:04 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: A Question About Hints
>
>
>
> It varies...
>
> Some clusters have 48 nodes, others 24 nodes and some 8 nodes.
>
> Both settings are on default.
>
>
>
> I’d try making a single conservative change to one or the other, measure
> and reassess.  Then do same to other setting.
>
> That's the plan, but I thought I might first get some valuable information
> from someone in the community that has already experienced in this type of
> change.
>
>
>
> Thanks!
>
>
>
> On Mon, Mar 4, 2019 at 8:27 PM Kenneth Brotman
>  wrote:
>
> It sounds like your use case might be appropriate for tuning those two
> settings some.
>
>
>
> How many nodes are in the cluster?
>
> Are both settings definitely on the default values currently?
>
>
>
> I’d try making a single conservative change to one or the other, measure
> and reassess.  Then do same to other setting.
>
>
>
> Then of course share your results with us.
>
>
>
> *From:* shalom sagges [mailto:shalomsag...@gmail.com]
> *Sent:* Monday, March 04, 2019 9:54 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: A Question About Hints
>
>
>
> Hi Kenneth,
>
>
>
> The concern is that in some cases, hints accumulate on nodes, and it takes
> a while until they are delivered (multi DCs).
>
> I see that whenever there are  a lot of hints in play,like after a rolling
> restart, the cluster works harder. That's why I want to decrease the hints
> delivery time.
>
> I didn't want to change the configuration blindly and thought the
> community might have some experience on this subject.
>
>
>
> I went over the cassandra.yaml file but didn't find any information on
> optimizing these attributes, just that the max_throttle is divided between
> nodes in the cluster and that I should increase the
> max_hints_delivery_threads because I have multi-dc deployments.
>
>
>
> # Maximum throttle in KBs per second, per delivery thread.  This will be
> # reduced proportionally to the number of nodes in the cluster.  (If there
> # are two nodes in the cluster, each delivery thread will use the maximum
> # rate; if there are three, each will throttle to half of the maximum,
> # since we expect two nodes to be delivering hints simultaneously)
> hinted_handoff_throttle_in_kb: 1024
>
> # Number of threads with which to deliver hints;
> # Consider increasing this number when you have multi-dc deployments, since
> # cross-dc handoff tends to be slower
> max_hints_delivery_threads: 2
>
>
>
>
>
> Thanks for your help!
>
>
>
>
>
> On Mon, Mar 4, 2019 at 6:44 PM Kenneth Brotman
>  wrote:
>
> What is the concern

Re: A Question About Hints

[shalom sagges]

It varies...
Some clusters have 48 nodes, others 24 nodes and some 8 nodes.
Both settings are on default.

I’d try making a single conservative change to one or the other, measure
and reassess.  Then do same to other setting.

That's the plan, but I thought I might first get some valuable information
from someone in the community that has already experienced in this type of
change.


Thanks!


On Mon, Mar 4, 2019 at 8:27 PM Kenneth Brotman 
wrote:

> It sounds like your use case might be appropriate for tuning those two
> settings some.
>
>
>
> How many nodes are in the cluster?
>
> Are both settings definitely on the default values currently?
>
>
>
> I’d try making a single conservative change to one or the other, measure
> and reassess.  Then do same to other setting.
>
>
>
> Then of course share your results with us.
>
>
>
> *From:* shalom sagges [mailto:shalomsag...@gmail.com]
> *Sent:* Monday, March 04, 2019 9:54 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: A Question About Hints
>
>
>
> Hi Kenneth,
>
>
>
> The concern is that in some cases, hints accumulate on nodes, and it takes
> a while until they are delivered (multi DCs).
>
> I see that whenever there are  a lot of hints in play,like after a rolling
> restart, the cluster works harder. That's why I want to decrease the hints
> delivery time.
>
> I didn't want to change the configuration blindly and thought the
> community might have some experience on this subject.
>
>
>
> I went over the cassandra.yaml file but didn't find any information on
> optimizing these attributes, just that the max_throttle is divided between
> nodes in the cluster and that I should increase the
> max_hints_delivery_threads because I have multi-dc deployments.
>
>
>
> # Maximum throttle in KBs per second, per delivery thread.  This will be
> # reduced proportionally to the number of nodes in the cluster.  (If there
> # are two nodes in the cluster, each delivery thread will use the maximum
> # rate; if there are three, each will throttle to half of the maximum,
> # since we expect two nodes to be delivering hints simultaneously.)
> hinted_handoff_throttle_in_kb: 1024
>
> # Number of threads with which to deliver hints;
> # Consider increasing this number when you have multi-dc deployments, since
> # cross-dc handoff tends to be slower
> max_hints_delivery_threads: 2
>
>
>
>
>
> Thanks for your help!
>
>
>
>
>
> On Mon, Mar 4, 2019 at 6:44 PM Kenneth Brotman
>  wrote:
>
> What is the concern?  Why are you looking there?  The casssandra.yml file
> has some notes about it.  Did you read them?
>
>
>
> *From:* shalom sagges [mailto:shalomsag...@gmail.com]
> *Sent:* Monday, March 04, 2019 7:22 AM
> *To:* user@cassandra.apache.org
> *Subject:* A Question About Hints
>
>
>
> Hi All,
>
>
>
> Does anyone know what is the most optimal hints configuration (multiple
> DCs) in terms of
>
> max_hints_delivery_threads and hinted_handoff_throttle_in_kb?
>
> If it's different for various use cases, is there a rule of thumb I can
> work with?
>
>
>
> I found this post but it's quite old:
>
> http://www.uberobert.com/bandwidth-cassandra-hinted-handoff/
>
>
>
> Thanks!
>
>

Re: A Question About Hints

[shalom sagges]

Hi Kenneth,

The concern is that in some cases, hints accumulate on nodes, and it takes
a while until they are delivered (multi DCs).
I see that whenever there are  a lot of hints in play,like after a rolling
restart, the cluster works harder. That's why I want to decrease the hints
delivery time.
I didn't want to change the configuration blindly and thought the community
might have some experience on this subject.

I went over the cassandra.yaml file but didn't find any information on
optimizing these attributes, just that the max_throttle is divided between
nodes in the cluster and that I should increase the
max_hints_delivery_threads because I have multi-dc deployments.

# Maximum throttle in KBs per second, per delivery thread.  This will be
# reduced proportionally to the number of nodes in the cluster.  (If there
# are two nodes in the cluster, each delivery thread will use the maximum
# rate; if there are three, each will throttle to half of the maximum,
# since we expect two nodes to be delivering hints simultaneously.)
hinted_handoff_throttle_in_kb: 1024

# Number of threads with which to deliver hints;
# Consider increasing this number when you have multi-dc deployments, since
# cross-dc handoff tends to be slower
max_hints_delivery_threads: 2


Thanks for your help!


On Mon, Mar 4, 2019 at 6:44 PM Kenneth Brotman 
wrote:

> What is the concern?  Why are you looking there?  The casssandra.yml file
> has some notes about it.  Did you read them?
>
>
>
> *From:* shalom sagges [mailto:shalomsag...@gmail.com]
> *Sent:* Monday, March 04, 2019 7:22 AM
> *To:* user@cassandra.apache.org
> *Subject:* A Question About Hints
>
>
>
> Hi All,
>
>
>
> Does anyone know what is the most optimal hints configuration (multiple
> DCs) in terms of
>
> max_hints_delivery_threads and hinted_handoff_throttle_in_kb?
>
> If it's different for various use cases, is there a rule of thumb I can
> work with?
>
>
>
> I found this post but it's quite old:
>
> http://www.uberobert.com/bandwidth-cassandra-hinted-handoff/
>
>
>
> Thanks!
>

A Question About Hints

[shalom sagges]

Hi All,

Does anyone know what is the most optimal hints configuration (multiple
DCs) in terms of
max_hints_delivery_threads and hinted_handoff_throttle_in_kb?
If it's different for various use cases, is there a rule of thumb I can
work with?

I found this post but it's quite old:
http://www.uberobert.com/bandwidth-cassandra-hinted-handoff/

Thanks!

Re: Question on changing node IP address

[shalom sagges]

Thanks for the info Alex!

I read
https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsSwitchSnitch.html
but still have a few questions:

Our clusters are comprised of 2 DCs with no rack configuration, RF=3 on
each DC.
In this scenario, if I wish to seamlessly change the snitch with 0
downtime, do I need to add the cassandra-rackdc.properties file, change the
snitch in cassandra.yaml and restart one by one?
Will this method cause problems?

Thanks!


On Wed, Feb 27, 2019 at 12:18 PM Alexander Dejanovski <
a...@thelastpickle.com> wrote:

> You'll be fine with the SimpleSnitch (which shouldn't be used either
> because it doesn't allow a cluster to use multiple datacenters or racks).
> Just change the IP and upon restart the node will redeclare itself in the
> ring. If your node is a seed node, you'll need to update your seed list
> across the cluster.
>
> On Wed, Feb 27, 2019 at 10:52 AM wxn...@zjqunshuo.com <
> wxn...@zjqunshuo.com> wrote:
>
>> I'm using SimpleSnitch. I have only one DC. Is there any problem to
>> follow the below procedure?
>>
>> -Simon
>>
>> *From:* Alexander Dejanovski 
>> *Date:* 2019-02-27 16:07
>> *To:* user 
>> *Subject:* Re: Question on changing node IP address
>>
>> I confirm what Oleksandr said.
>> Just stop Cassandra, change the IP, and restart Cassandra.
>> If you're using the GossipingPropertyFileSnitch, the node will redeclare
>> its new IP through Gossip and that's it.
>> If you're using the PropertyFileSnitch, well... you shouldn't as it's a
>> rather dangerous and tedious snitch to use. But if you are, it'll require
>> to change the file containing all the IP addresses across the cluster.
>>
>> I've been changing IPs on a whole cluster back in 2.1 this way and it
>> went through seamlessly.
>>
>> Cheers,
>>
>> On Wed, Feb 27, 2019 at 8:54 AM Oleksandr Shulgin <
>> oleksandr.shul...@zalando.de> wrote:
>>
>>> On Wed, Feb 27, 2019 at 4:15 AM wxn...@zjqunshuo.com <
>>> wxn...@zjqunshuo.com> wrote:
>>>
 >After restart with the new address the server will notice it and log a
 warning, but it will keep token ownership as long as it keeps the old host
 id (meaning it must use the same data directory as before restart).

 Based on my understanding, token range is binded to host id. As long as
 host id doesn't change, everything is ok. Besides data directory, any other
 thing can lead to host id change? And how host id is caculated? For
 example, if I upgrade Cassandra binary to a new version, after restart,
 will host id change?

>>>
>>> I believe host id is calculated once the new node is initialized and
>>> never changes afterwards, even through major upgrades.  It is stored in
>>> system keyspace in data directory, and is stable across restarts.
>>>
>>> --
>>> Alex
>>>
>>> --
>> -
>> Alexander Dejanovski
>> France
>> @alexanderdeja
>>
>> Consultant
>> Apache Cassandra Consulting
>> http://www.thelastpickle.com
>>
>> --
> -
> Alexander Dejanovski
> France
> @alexanderdeja
>
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>

Re: Question on changing node IP address

[shalom sagges]

If you're using the PropertyFileSnitch, well... you shouldn't as it's a
rather dangerous and tedious snitch to use

I inherited Cassandra clusters that use the PropertyFileSnitch. It's been
working fine, but you've kinda scared me :-)
Why is it dangerous to use?
If I decide to change the snitch, is it seamless or is there a specific
procedure one must follow?

Thanks!


On Wed, Feb 27, 2019 at 10:08 AM Alexander Dejanovski <
a...@thelastpickle.com> wrote:

> I confirm what Oleksandr said.
> Just stop Cassandra, change the IP, and restart Cassandra.
> If you're using the GossipingPropertyFileSnitch, the node will redeclare
> its new IP through Gossip and that's it.
> If you're using the PropertyFileSnitch, well... you shouldn't as it's a
> rather dangerous and tedious snitch to use. But if you are, it'll require
> to change the file containing all the IP addresses across the cluster.
>
> I've been changing IPs on a whole cluster back in 2.1 this way and it went
> through seamlessly.
>
> Cheers,
>
> On Wed, Feb 27, 2019 at 8:54 AM Oleksandr Shulgin <
> oleksandr.shul...@zalando.de> wrote:
>
>> On Wed, Feb 27, 2019 at 4:15 AM wxn...@zjqunshuo.com <
>> wxn...@zjqunshuo.com> wrote:
>>
>>> >After restart with the new address the server will notice it and log a
>>> warning, but it will keep token ownership as long as it keeps the old host
>>> id (meaning it must use the same data directory as before restart).
>>>
>>> Based on my understanding, token range is binded to host id. As long as
>>> host id doesn't change, everything is ok. Besides data directory, any other
>>> thing can lead to host id change? And how host id is caculated? For
>>> example, if I upgrade Cassandra binary to a new version, after restart,
>>> will host id change?
>>>
>>
>> I believe host id is calculated once the new node is initialized and
>> never changes afterwards, even through major upgrades.  It is stored in
>> system keyspace in data directory, and is stable across restarts.
>>
>> --
>> Alex
>>
>> --
> -
> Alexander Dejanovski
> France
> @alexanderdeja
>
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>

Re: forgot to run nodetool cleanup

[shalom sagges]

Cleanup is a great way to free up disk space.

Just note you might run into
https://issues.apache.org/jira/browse/CASSANDRA-9036 if you use a version
older than 2.0.15.



On Thu, Feb 14, 2019 at 10:20 AM Oleksandr Shulgin <
oleksandr.shul...@zalando.de> wrote:

> On Wed, Feb 13, 2019 at 6:47 PM Jeff Jirsa  wrote:
>
>> Depending on how bad data resurrection is, you should run it for any host
>> that loses a range. In vnodes, that's usually all hosts.
>>
>> Cleanup with LCS is very cheap. Cleanup with STCS/TWCS is a bit more work.
>>
>
> Wait, doesn't cleanup just rewrite every SSTable one by one?  Why would
> compaction strategy matter?  Do you mean that after cleanup STCS may pick
> some resulting tables to re-compact them due to the min/max size
> difference, which would not be the case with LCS?
>
>
>> If you're just TTL'ing all data, it may not be worth the effort.
>>
>
> Indeed, but in our case the main reason to scale out is that the nodes are
> running out of disk space, so we really want to get rid of the extra copies.
>
> --
> Alex
>
>

Re: Upgrade From 2.0 to 2.1

[shalom sagges]

Very soon. If not today, it will be up tomorrow. :)
Yayyy, just saw the release of 3.11.4.  :-)

You'll need to go to v3 for 3.11. Congratulations on being aware enough to
do this - advanced upgrade coordination, it's absolutely the right thing to
do, but most people don't know it's possible or useful.
Thanks a lot Jeff for clarifying this.
I really hoped the answer would be different. Now I need to nag our R
teams again :-)

Thanks!

On Mon, Feb 11, 2019 at 8:21 PM Michael Shuler 
wrote:

> On 2/11/19 9:24 AM, shalom sagges wrote:
> > I've successfully upgraded a 2.0 cluster to 2.1 on the way to upgrade to
> > 3.11 (hopefully 3.11.4 if it'd be released very soon).
>
> Very soon. If not today, it will be up tomorrow. :)
>
> --
> Michael
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Upgrade From 2.0 to 2.1

[shalom sagges]

Hi All,

I've successfully upgraded a 2.0 cluster to 2.1 on the way to upgrade to
3.11 (hopefully 3.11.4 if it'd be released very soon).

I have 2 small questions:

   1. Currently the Datastax clients are enforcing Protocol Version 2 to
   prevent mixed cluster issues. Do I need now to enforce Protocol Version 3
   while upgrading from 2.1 to 3.11 or can I still use Protocol Version 2?

   2. After the upgrade, I found that system table NodeIdInfo has not been
   upgraded, i.e. I still see it in *-jb-* convention. Does this mean that
   this table is obsolete and can be removed?


Thanks!

Re: Upgrade to v3.11.3

[shalom sagges]

Thanks a lot Anuj!



On Wed, Jan 16, 2019 at 4:56 PM Anuj Wadehra  wrote:

> Hi Shalom,
>
> Just a suggestion. Before upgrading to 3.11.3 make sure you are not
> impacted by any open crtitical defects especially related to RT which may
> cause data loss e.g.14861.
>
> Please find my response below:
>
> The upgrade process that I know of is from 2.0.14 to 2.1.x (higher than
> 2.1.9 I think) and then from 2.1.x to 3.x. Do I need to upgrade first to
> 3.0.x or can I upgraded directly from 2.1.x to 3.11.3?
>
> Response: Yes, you can upgrade from 2.0.14 to some latest stable version
> of 2.1.x (only 2.1.9+)  and then upgrade to 3.11.3.
>
> Can I run upgradesstables on several nodes in parallel? Is it crucial to
> run it one node at a time?
>
> Response: Yes, you can run in parallel.
>
>
> When running upgradesstables on a node, does that node still serves writes
> and reads?
>
> Response: Yes.
>
>
> Can I use open JDK 8 (instead of Oracle JDK) with C* 3.11.3?
>
> Response: We have not tried but it should be okay. See
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-13916
> .
>
>
> Is there a way to speed up the upgradesstables process? (besides
> compaction_throughput)
>
>
> Response: If clearing pending compactions caused by rewriting sstable is a
> concern,probably you can also try increasing concurrent compactors.
>
>
>
> Disclaimer: The information provided in above response is my personal
> opinion based on the best of my knowledge and experience. We do
> not take any responsibility and we are not liable for any damage caused by
> actions taken based on above information.
> Thanks
> Anuj
>
>
> On Wed, 16 Jan 2019 at 19:15, shalom sagges
>  wrote:
> Hi All,
>
> I'm about to start a rolling upgrade process from version 2.0.14 to
> version 3.11.3.
> I have a few small questions:
>
>1. The upgrade process that I know of is from 2.0.14 to 2.1.x (higher
>than 2.1.9 I think) and then from 2.1.x to 3.x. Do I need to upgrade first
>to 3.0.x or can I upgraded directly from 2.1.x to 3.11.3?
>
>2. Can I run upgradesstables on several nodes in parallel? Is it
>crucial to run it one node at a time?
>
>3. When running upgradesstables on a node, does that node still serves
>writes and reads?
>
>4. Can I use open JDK 8 (instead of Oracle JDK) with C* 3.11.3?
>
>5. Is there a way to speed up the upgradesstables process? (besides
>compaction_throughput)
>
>
> Thanks!
>
>

Upgrade to v3.11.3

[shalom sagges]

Hi All,

I'm about to start a rolling upgrade process from version 2.0.14 to version
3.11.3.
I have a few small questions:

   1. The upgrade process that I know of is from 2.0.14 to 2.1.x (higher
   than 2.1.9 I think) and then from 2.1.x to 3.x. Do I need to upgrade first
   to 3.0.x or can I upgraded directly from 2.1.x to 3.11.3?

   2. Can I run upgradesstables on several nodes in parallel? Is it crucial
   to run it one node at a time?

   3. When running upgradesstables on a node, does that node still serves
   writes and reads?

   4. Can I use open JDK 8 (instead of Oracle JDK) with C* 3.11.3?

   5. Is there a way to speed up the upgradesstables process? (besides
   compaction_throughput)


Thanks!

Re: Query With Limit Clause

[shalom sagges]

Thanks a lot for the info :)

On Tue, Nov 6, 2018 at 11:11 AM DuyHai Doan  wrote:

> Cassandra will execute such request using a Partition Range Scan.
>
> See more details here http://www.doanduyhai.com/blog/?p=13191, chapter E
> Cluster Read Path (look at the formula of Concurrency Factor)
>
>
>
> On Tue, Nov 6, 2018 at 8:21 AM shalom sagges 
> wrote:
>
>> Hi All,
>>
>> If I run for example:
>> select * from myTable limit 3;
>>
>> Does Cassandra do a full table scan regardless of the limit?
>>
>> Thanks!
>>
>

Query With Limit Clause

[shalom sagges]

Hi All,

If I run for example:
select * from myTable limit 3;

Does Cassandra do a full table scan regardless of the limit?

Thanks!

Re: Re: High CPU usage on some of the nodes due to message coalesce

[shalom sagges]

I guess the code experts could shed more light on
org.apache.cassandra.util.coalesceInternal and SepWorker.run.
I'll just add anything I can think of

Any cron or other scheduler running on those nodes?
Lots of Java processes running simultaneously?
Heavy repair continuously running?
Lots of pending compactions?
Is the number of CPU cores the same in all the nodes?
Did you try rebooting one of the nodes?


On Sun, Oct 21, 2018 at 4:55 PM onmstester onmstester
 wrote:

>
> What takes the most CPU? System or User?
>
>
>  most of it is used by org.apache.cassandra.util.coalesceInternal and
> SepWorker.run
>
> Did you try removing a problematic node and installing a brand new one
> (instead of re-adding)?
>
> I did not install a new node, but did remove the problematic node and CPU
> load in all the cluster became normal again
>
> When you decommissioned these nodes, did the high CPU "move" to other
> nodes (probably data model/query issues) or was it completely gone? (server
> issues)
>
> it was completely gone
>
>

Re: Re: High CPU usage on some of the nodes due to message coalesce

[shalom sagges]

What takes the most CPU? System or User?
Did you try removing a problematic node and installing a brand new one
(instead of re-adding)?
When you decommissioned these nodes, did the high CPU "move" to other nodes
(probably data model/query issues) or was it completely gone? (server
issues)


On Sun, Oct 21, 2018 at 3:52 PM onmstester onmstester
 wrote:

> I don't think that root cause is related to Cassandra config, because the
> nodes are homogeneous and config for all of them are the same (16GB heap
> with default gc), also mutation counter and Native Transport counter is the
> same in all of the nodes, but only these 3 nodes experiencing 100% CPU
> usage (others have less than 20% CPU usage)
> I even decommissioned these 3 nodes from cluster and re-add them, but
> still the same
> The cluster is OK without these 3 nodes (in a state that these nodes are
> decommissioned)
>
> Sent using Zoho Mail 
>
>
>  Forwarded message 
> From : Chris Lohfink 
> To : 
> Date : Sat, 20 Oct 2018 23:24:03 +0330
> Subject : Re: High CPU usage on some of the nodes due to message coalesce
>  Forwarded message 
>
> 1s young gcs are horrible and likely cause of *some* of your bad metrics.
> How large are your mutations/query results and what gc/heap settings are
> you using?
>
> You can use https://github.com/aragozin/jvm-tools to see the threads
> generating allocation pressure and using the cpu (ttop) and what garbage is
> being created (hh --dead-young).
>
> Just a shot in the dark, I would *guess* you have rather large mutations
> putting pressure on commitlog and heap. G1 with a larger heap might help in
> that scenario to reduce fragmentation and adjust its eden and survivor
> regions to the allocation rate better (but give it a bigger reserve space)
> but theres limits to what can help if you cant change your workload.
> Without more info on schema etc its hard to tell but maybe that can help
> give you some ideas on places to look. It could just as likely be repair
> coordination, wide partition reads, or compactions so need to look more at
> what within the app is causing the pressure to know if its possible to
> improve with settings or if the load your application is producing exceeds
> what your cluster can handle (needs more nodes).
>
> Chris
>
> On Oct 20, 2018, at 5:18 AM, onmstester onmstester <
> onmstes...@zoho.com.INVALID> wrote:
>
> 3 nodes in my cluster have 100% cpu usage and most of it is used by
> org.apache.cassandra.util.coalesceInternal and SepWorker.run?
> The most active threads are the messaging-service-incomming.
> Other nodes are normal, having 30 nodes, using Rack Aware strategy. with
> 10 rack each having 3 nodes. The problematic nodes are configured for one
> rack, on normal write load, system.log reports too many hint message
> dropped (cross node). also there are alot of parNewGc with about 700-1000ms
> and commit log isolated disk, is utilized about 80-90%. on startup of these
> 3 nodes, there are alot of "updateing topology" logs (1000s of them
> pending).
> Using iperf, i'm sure that network is OK
> checking NTPs and mutations on each node, load is balanced among the nodes.
> using apache cassandra 3.11.2
> I can not not figure out the root cause of the problem, although there are
> some obvious symptoms.
>
> Best Regards
>
> Sent using Zoho Mail 
>
>
>
>

Re: Read timeouts when performing rolling restart

[shalom sagges]

Hi Riccardo,

Does this issue occur when performing a single restart or after several
restarts during a rolling restart (as mentioned in your original post)?
We have a cluster that when performing a rolling restart, we prefer to wait
~10-15 minutes between each restart because we see an increase of GC for a
few minutes.
If we keep restarting the nodes quickly one after the other, the
applications experience timeouts (probably due to GC and hints).

Hope this helps!

On Thu, Sep 13, 2018 at 2:20 AM Riccardo Ferrari  wrote:

> A little update on the progress.
>
> First:
> Thank you Thomas. I checked the code in the patch and briefly skimmed
> through the 3.0.6 code. Yup it should be fixed.
> Thank you Surbhi. At the moment we don't need authentication as the
> instances are locked down.
>
> Now:
> - Unfortunately the start_transport_native trick does not always work. On
> some nodes works on other don't. What do I mean? I still experience
> timeouts and dropped messages during startup.
> - I realized that cutting the concurrent_compactors to 1 was not really a
> good idea, minimum vlaue should be 2, currently testing 4 (that is the
> min(n_cores, n_disks))
> - After rising the compactors to 4 I still see some dropped messages for
> HINT and MUTATIONS. This happens during startup. Reason is "for internal
> timeout". Maybe too many compactors?
>
> Thanks!
>
>
> On Wed, Sep 12, 2018 at 7:09 PM, Surbhi Gupta 
> wrote:
>
>> Another thing to notice is :
>>
>> system_auth WITH replication = {'class': 'SimpleStrategy',
>> 'replication_factor': '1'}
>>
>> system_auth has a replication factor of 1 and even if one node is down it
>> may impact the system because of the replication factor.
>>
>>
>>
>> On Wed, 12 Sep 2018 at 09:46, Steinmaurer, Thomas <
>> thomas.steinmau...@dynatrace.com> wrote:
>>
>>> Hi,
>>>
>>>
>>>
>>> I remember something that a client using the native protocol gets
>>> notified too early by Cassandra being ready due to the following issue:
>>>
>>> https://issues.apache.org/jira/browse/CASSANDRA-8236
>>>
>>>
>>>
>>> which looks similar, but above was marked as fixed in 2.2.
>>>
>>>
>>>
>>> Thomas
>>>
>>>
>>>
>>> *From:* Riccardo Ferrari 
>>> *Sent:* Mittwoch, 12. September 2018 18:25
>>> *To:* user@cassandra.apache.org
>>> *Subject:* Re: Read timeouts when performing rolling restart
>>>
>>>
>>>
>>> Hi Alain,
>>>
>>>
>>>
>>> Thank you for chiming in!
>>>
>>>
>>>
>>> I was thinking to perform the 'start_native_transport=false' test as
>>> well and indeed the issue is not showing up. Starting the/a node with
>>> native transport disabled and letting it cool down lead to no timeout
>>> exceptions no dropped messages, simply a crystal clean startup. Agreed it
>>> is a workaround
>>>
>>>
>>>
>>> # About upgrading:
>>>
>>> Yes, I desperately want to upgrade despite is a long and slow task. Just
>>> reviewing all the changes from 3.0.6 to 3.0.17
>>> is going to be a huge pain, top of your head, any breaking change I
>>> should absolutely take care of reviewing ?
>>>
>>>
>>>
>>> # describecluster output: YES they agree on the same schema version
>>>
>>>
>>>
>>> # keyspaces:
>>>
>>> system WITH replication = {'class': 'LocalStrategy'}
>>>
>>> system_schema WITH replication = {'class': 'LocalStrategy'}
>>>
>>> system_auth WITH replication = {'class': 'SimpleStrategy',
>>> 'replication_factor': '1'}
>>>
>>> system_distributed WITH replication = {'class': 'SimpleStrategy',
>>> 'replication_factor': '3'}
>>>
>>> system_traces WITH replication = {'class': 'SimpleStrategy',
>>> 'replication_factor': '2'}
>>>
>>>
>>>
>>>  WITH replication = {'class': 'SimpleStrategy',
>>> 'replication_factor': '3'}
>>>
>>>   WITH replication = {'class': 'SimpleStrategy',
>>> 'replication_factor': '3'}
>>>
>>>
>>>
>>> # Snitch
>>>
>>> Ec2Snitch
>>>
>>>
>>>
>>> ## About Snitch and replication:
>>>
>>> - We have the default DC and all nodes are in the same RACK
>>>
>>> - We are planning to move to GossipingPropertyFileSnitch configuring the
>>> cassandra-rackdc accortingly.
>>>
>>> -- This should be a transparent change, correct?
>>>
>>>
>>>
>>> - Once switched to GPFS, we plan to move to 'NetworkTopologyStrategy'
>>> with 'us-' DC and replica counts as before
>>>
>>> - Then adding a new DC inside the VPC, but this is another story...
>>>
>>>
>>>
>>> Any concerns here ?
>>>
>>>
>>>
>>> # nodetool status 
>>>
>>> --  Address Load   Tokens   Owns (effective)  Host
>>> ID   Rack
>>> UN  10.x.x.a  177 GB 256  50.3%
>>> d8bfe4ad-8138-41fe-89a4-ee9a043095b5  rr
>>> UN  10.x.x.b152.46 GB  256  51.8%
>>> 7888c077-346b-4e09-96b0-9f6376b8594f  rr
>>> UN  10.x.x.c   159.59 GB  256  49.0%
>>> 329b288e-c5b5-4b55-b75e-fbe9243e75fa  rr
>>> UN  10.x.x.d  162.44 GB  256  49.3%
>>> 07038c11-d200-46a0-9f6a-6e2465580fb1  rr
>>> UN  10.x.x.e174.9 GB   256  50.5%
>>> c35b5d51-2d14-4334-9ffc-726f9dd8a214  rr
>>> UN  10.x.x.f  194.71 GB  256  49.2%
>>> 

Re: Large sstables

[shalom sagges]

If there are a lot of droppable tombstones, you could also run User Defined
Compaction on that (and on other) SSTable(s).

This blog post explains it well:
http://thelastpickle.com/blog/2016/10/18/user-defined-compaction.html

On Fri, Aug 31, 2018 at 12:04 AM Mohamadreza Rostami <
mohamadrezarosta...@gmail.com> wrote:

> Hi,Dear Vitali
> The best option for you is migrating data to the new table and change
> portion key patterns to a better distribution of data and you sstables
> become smaller but if your data already have good distribution and your
> data is really big you must add new server to your datacenter, if you
> change compassion strategy it has some risk.
>
> > On Shahrivar 8, 1397 AP, at 19:54, Jeff Jirsa  wrote:
> >
> > Either of those are options, but there’s also sstablesplit to break it
> up a bit
> >
> > Switching to LCS can be a problem depending on how many sstables
> /overlaps you have
> >
> > --
> > Jeff Jirsa
> >
> >
> >> On Aug 30, 2018, at 8:05 AM, Vitali Dyachuk  wrote:
> >>
> >> Hi,
> >> Some of the sstables got too big 100gb and more so they are not
> compactiong any more so some of the disks are running out of space. I'm
> running C* 3.0.17, RF3 with 10 disks/jbod with STCS.
> >> What are my options? Completely delete all data on this node and rejoin
> it to the cluster, change CS to LCS then run repair?
> >> Vitali.
> >>
> >
> > -
> > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: user-h...@cassandra.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: User Defined Types?

[shalom sagges]

Thanks a lot Anup! :-)



On Mon, Aug 6, 2018 at 5:45 AM, Anup Shirolkar <
anup.shirol...@instaclustr.com> wrote:

> Hi,
>
> Few of the caveats can be found here:
> https://issues.apache.org/jira/browse/CASSANDRA-7423
>
> The JIRA is implemented in version *3.6* and you are on 3.0,
> So you are affected by UDT behaviour (stored as BLOB) mentioned in the
> JIRA.
>
> Cheers,
> Anup
>
> On 5 August 2018 at 23:29, shalom sagges  wrote:
>
>> Hi All,
>>
>> Are there any known caveats for User Defined Types in Cassandra (version
>> 3.0)?
>> One of our teams wants to start using them. I wish to assess it and see
>> if it'd be wise (or not) to refrain from using UDTs.
>>
>>
>> Thanks!
>>
>
>
>
> --
>
> Anup Shirolkar
>
> Consultant
>
> +61 420 602 338
>
> <https://www.instaclustr.com/solutions/managed-apache-kafka/>
>
> <https://www.facebook.com/instaclustr>   <https://twitter.com/instaclustr>
><https://www.linkedin.com/company/instaclustr>
>
> Read our latest technical blog posts here
> <https://www.instaclustr.com/blog/>.
>

User Defined Types?

[shalom sagges]

Hi All,

Are there any known caveats for User Defined Types in Cassandra (version
3.0)?
One of our teams wants to start using them. I wish to assess it and see if
it'd be wise (or not) to refrain from using UDTs.


Thanks!

Re: Stumped By Cassandra delays

[shalom sagges]

Hi Gareth,

If you're using batches for multiple partitions, this may be the root cause
you've been looking for.

https://inoio.de/blog/2016/01/13/cassandra-to-batch-or-not-to-batch/

If batches are optimally used and only one node is misbehaving, check if
NTP on the node is properly synced.

Hope this helps!


On Sat, Jul 21, 2018 at 9:31 PM, Gareth Collins 
wrote:

> Hello,
>
> We are running Cassandra 2.1.14 in AWS, with c5.4xlarge machines
> (initially these were m4.xlarge) for our cassandra servers and
> m4.xlarge for our application servers. On one of our clusters having
> problems we have 6 C* nodes and 6 AS nodes (two nodes for C*/AS in
> each availability zone).
>
> In the deployed application it seems to be a common use-case to one of
> the following. These use cases are having periodic errors:
> (1) Copy one Cassandra table to another table using the application server.
> (2) Export from a Cassandra table to file using the application server.
>
> The application server is reading from the table via token range, the
> token range queries being calculated to ensure the whole token range
> for a query falls on the same node. i.e. the query looks like this:
>
> select * from  where token(key) > ? and token(key) <= ?
>
> This was probably initially done on the assumption that the driver
> would be able to figure out which nodes contained the data. As we
> realized now the driver only supports routing to the right node if the
> partition key is defined in the where clause.
>
> When we do the read we are doing a lot of queries in parallel to
> maximize performance. I believe when the copy is being run there are
> currently 5 threads per machine doing the copy for a max of 30
> concurrent read requests across the cluster.
>
> Specifically these tasks been periodically having a few of these errors:
>
> INFO  [ScheduledTasks:1] 2018-07-13 20:03:20,124
> MessagingService.java:929 - REQUEST_RESPONSE messages were dropped in
> last 5000 ms: 1 for internal timeout and 0 for cross node timeout
>
> Which are causing errors in the read by token range queries.
>
> Running "nodetool settraceprobability 1" and running the test when
> failing we could see that this timeout would occur when using a
> coordinator on the read query (i.e. the co-ordinator sent the message
> but didn't get a response to the query from the other node within the
> time limit). We were seeing these timeouts periodically even if we set
> the timeouts to 60 seconds.
>
> As I mentioned at the beginning we had initially been using m4.xlarge
> for our Cassandra servers. After discussion with AWS it was suggested
> that we could be hitting performance limits (i.e. either network or
> disk - I believe more likely network as I didn't see the disk getting
> hit very hard) so we upgraded the Cassandra servers and everything was
> fine for a while.
>
> But then the problems started to re-occur recently...pretty
> consistently failing on these copy or export jobs running overnight.
> Having looked at resource usage statistics graphs it appeared that the
> C* servers were not heavily loaded at all (the app servers were being
> maxed out) and I did not see any significant garbage collections in
> the logs that could explain the delays.
>
> As a last resort I decided to turn up the logging on the server and
> client, datastax client set to debug and server set to the following
> logs via nodetool...the goal being to maximize logging while cutting
> out the very verbose stuff (e.g. Message.java appears to print out the
> whole message in 2.1.14 when put into debug -> it looks like that was
> moved to trace in a later 2.1.x release):
>
> bin/nodetool setlogginglevel org.apache.cassandra.tracing.Tracing INFO
> bin/nodetool setlogginglevel org.apache.cassandra.transport.Message INFO
> bin/nodetool setlogginglevel org.apache.cassandra.db.ColumnFamilyStore
> DEBUG
> bin/nodetool setlogginglevel org.apache.cassandra.gms.Gossiper DEBUG
> bin/nodetool setlogginglevel
> org.apache.cassandra.db.filter.SliceQueryFilter DEBUG
> bin/nodetool setlogginglevel
> org.apache.cassandra.service.pager.AbstractQueryPager INFO
> bin/nodetool setlogginglevel org.apache.cassandra TRACE
>
> Of course when we did this (as part of turning on the logging the
> application servers were restarted) the problematic export to file
> jobs which had failed every time for the last week succeeded and ran
> much faster than they had run usually (47 minutes vs 1 1/2 hours) so I
> decided to look for the biggest delay (which turned out to be ~9
> seconds and see what I could find in the log - outside of this time,
> the response times were up to perhaps 20ms). Here is what I found:
>
> (1) Only one Cassandra node had delays at a time.
>
> (2) On the Cassandra node that did had delays there was no significant
> information from the GCInspector (the system stopped processing client
> requests between 05:32:33 - 05:32:43). If anything it confirmed my
> belief that the system was lightly loaded
>
> 

Re: Cassandra didn't order data according to clustering order

[shalom sagges]

The clustering column is ordered per partition key.

So if for example I create the following table:
create table desc_test (
   id text,
   name text,
   PRIMARY KEY (id,name)
) WITH CLUSTERING ORDER BY (name DESC );


I insert a few rows:

insert into desc_test (id , name ) VALUES ( 'abc', 'abc');
insert into desc_test (id , name ) VALUES ( 'abc', 'bcd');
insert into desc_test (id , name ) VALUES ( 'abc', 'aaa');
insert into desc_test (id , name ) VALUES ( 'fgh', 'aaa');
insert into desc_test (id , name ) VALUES ( 'fgh', 'bcd');
insert into desc_test (id , name ) VALUES ( 'fgh', 'abc');


And then read:
select * from desc_test;

 id  | name
-+--
  fgh |  bcd
  fgh |  abc
  fgh |  aaa
 abc |  bcd
 abc |  abc
 abc |  aaa

(6 rows)


You can see that the data is properly ordered in descending mode, BUT
*for each partition key. *
So in order to achieve what you want, you will have to add the relevant
partition key for each select query.

Hope this helps


On Sun, Jul 15, 2018 at 2:16 PM, Soheil Pourbafrani 
wrote:

> I created table using the command:
> CREATE TABLE correlated_data (
> processing_timestamp bigint,
> generating_timestamp bigint,
> data text,
> PRIMARY KEY (processing_timestamp, generating_timestamp)
> ) WITH CLUSTERING ORDER BY (generating_timestamp DESC);
>
>
> When I get data using the command :
> SELECT * FROM correlated_data LIMIT 1 ;
>
> I expect it return the row with the biggest field "generating_timestamp",
> but I got the same row every time I run the query, while row with bigger "
> generating_timestamp" exists. What's the problem?
>

Re: saving distinct data in cassandra result in many tombstones

[shalom sagges]

 1. How to use sharding partition key in a way that partitions end up in
different nodes?
You could, for example, create a table with a bucket column added to the
partition key:
Table distinct(
hourNumber int,
bucket int, //could be a 5 minute bucket for example
key text,
distinctValue long
primary key ((hourNumber,bucket))
)

2. if i set gc_grace_seconds to 0, would it replace the row at memtable
(not saving repeated rows in sstables) or it would be done at first
compaction?
Overlapping rows in the memtables are merged regardless of the
gc_grace_seconds period. Setting gc_grace_seconds to 0 will immediately
evict tombstones during compaction but will disable hints delivery. You
should set gc_grace_seconds>max_hint_window_in_ms



On Tue, Jun 19, 2018 at 7:23 AM, onmstester onmstester 
wrote:

> Two other questions:
> 1. How to use sharding partition key in a way that partitions end up in
> different nodes?
> 2. if i set gc_grace_seconds to 0, would it replace the row at memtable
> (not saving repeated rows in sstables) or it would be done at first
> compaction?
>
> Sent using Zoho Mail 
>
>
>  On Tue, 19 Jun 2018 08:16:28 +0430 *onmstester onmstester
> >* wrote 
>
> Can i set gc_grace_seconds to 0 in this case? because reappearing deleted
> data has no impact on my Business Logic, i'm just either creating a new row
> or replacing the exactly same row.
>
> Sent using Zoho Mail 
>
>
>  On Wed, 13 Jun 2018 03:41:51 +0430 *Elliott Sims
> >* wrote 
>
>
>
> If this is data that expires after a certain amount of time, you probably
> want to look into using TWCS and TTLs to minimize the number of tombstones.
> Decreasing gc_grace_seconds then compacting will reduce the number of
> tombstones, but at the cost of potentially resurrecting deleted data if the
> table hasn't been repaired during the grace interval.  You can also just
> increase the tombstone thresholds, but the queries will be pretty
> expensive/wasteful.
>
> On Tue, Jun 12, 2018 at 2:02 AM, onmstester onmstester <
> onmstes...@zoho.com> wrote:
>
>
> Hi,
>
> I needed to save a distinct value for a key in each hour, the problem with
> saving everything and computing distincts in memory is that there
> are too many repeated data.
> Table schema:
> Table distinct(
> hourNumber int,
> key text,
> distinctValue long
> primary key (hourNumber)
> )
>
> I want to retrieve distinct count of all keys in a specific hour and using
> this data model it would be achieved by reading a single partition.
> The problem : i can't read from this table, system.log indicates that more
> than 100K tombstones read and no live data in it. The gc_grace time is
> the default (10 days), so i thought decreasing it to 1 hour and run
> compaction, but is this a right approach at all? i mean the whole idea of
> replacing
> some millions of rows. each  10 times in a partition again and again that
> creates alot of tombstones just to achieve distinct behavior?
>
> Thanks in advance
>
> Sent using Zoho Mail 
>
>
>
>

Re: Does LOCAL_ONE still replicate data?

[shalom sagges]

 It's advisable to set the RF to 3 regardless of the consistency level.

If using RF=1, Read CL=LOCAL_ONE and a node goes down in the local DC, you
will not be able to read data related to this node until it goes back up.
For writes and CL=LOCAL_ONE, the write will fail (if it falls on the token
ranges of the downed node).



On Tue, May 8, 2018 at 3:05 PM, Lucas Benevides  wrote:

> Yes, but remind that there is Write Consistency and Read Consistency.
> To prevent the reads from reaching the other DC, you should set the Read
> Consistency LOCAL_ONE.
> As Hannu Kroger said, the LOCAL_ONE may be enough to you but maybe not if
> you want to be sure that your data was written also in another DC.
>
> Lucas B. Dias
>
>
> 2018-05-08 7:26 GMT-03:00 Hannu Kröger :
>
>> Writes are always replicated to all nodes (if they are online).
>>
>> LOCAL_ONE in writes just means that client will get an “OK” for the write
>> only after at least node in local datacenter has acknowledged that the
>> write is done.
>>
>> If all local replicas are offline, then the write will fail even if it
>> gets written in your other DC.
>>
>> Hannu
>>
>>
>> On 8 May 2018, at 13:24, Jakub Lida  wrote:
>>
>> Hi,
>>
>> I want to add a new DC to an existing cluster (RF=1 per DC).
>> Will setting consistency to LOCAL_ONE on all machines make it still
>> replicate write requests sent to online DCs to all DCs (including the new
>> one being rebuilt) and only isolate read requests from reaching the new DC?
>> That is basically want I want to accomplish.
>>
>> Thanks in advance, Jakub
>>
>>
>>
>

Re: Dropped Mutations

[Shalom Sagges]

Thanks a lot Hitesh!

I'll try to re-tune the heap to a lower level


Shalom Sagges
DBA
T: +972-74-700-4035
<http://www.linkedin.com/company/164748> <http://twitter.com/liveperson>
<http://www.facebook.com/LivePersonInc> We Create Meaningful Connections
<https://liveperson.docsend.com/view/8iiswfp>


On Thu, Apr 19, 2018 at 12:42 AM, hitesh dua <hiteshd...@gmail.com> wrote:

> Hi ,
>
> I'll recommend tuning you heap size further( preferably lower) as large
> Heap size can lead to Large Garbage collection pauses also known as also
> known as a stop-the-world event. A pause occurs when a region of memory is
> full and the JVM needs to make space to continue. During a pause all
> operations are suspended. Because a pause affects networking, the node can
> appear as down to other nodes in the cluster. Additionally, any Select and
> Insert statements will wait, which increases read and write latencies.
>
> Any pause of more than a second, or multiple pauses within a second that
> add to a large fraction of that second, should be avoided. The basic cause
> of the problem is the rate of data stored in memory outpaces the rate at
> which data can be removed
>
> MUTATION : If a write message is processed after its timeout
> (write_request_timeout_in_ms) it either sent a failure to the client or it
> met its requested consistency level and will relay on hinted handoff and
> read repairs to do the mutation if it succeeded.
>
> Another possible cause of the Issue could be you HDDs as that could too
> be a bottleneck.
>
> *MAX_HEAP_SIZE*
> The recommended maximum heap size depends on which GC is used:
> Hardware setupRecommended MAX_HEAP_SIZE
> Older computers Typically 8 GB.
> CMS for newer computers (8+ cores) with up to 256 GB RAM No more 14 GB.
>
>
> Thanks,
> Hitesh dua
> hiteshd...@gmail.com
>
> On Wed, Apr 18, 2018 at 10:07 PM, shalom sagges <shalomsag...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> I have a 44 node cluster (22 nodes on each DC).
>> Each node has 24 cores and 130 GB RAM, 3 TB HDDs.
>> Version 2.0.14 (soon to be upgraded)
>> ~10K writes per second per node.
>> Heap size: 8 GB max, 2.4 GB newgen
>>
>> I deployed Reaper and GC started to increase rapidly. I'm not sure if
>> it's because there was a lot of inconsistency in the data, but I decided to
>> increase the heap to 16 GB and new gen to 6 GB. I increased the max tenure
>> from 1 to 5.
>>
>> I tested on a canary node and everything was fine but when I changed the
>> entire DC, I suddenly saw a lot of dropped mutations in the logs on most of
>> the nodes. (Reaper was not running on the cluster yet but a manual repair
>> was running).
>>
>> Can the heap increment cause lots of dropped mutations?
>> When is a mutation considered as dropped? Is it during flush? Is it
>> during the write to the commit log or memtable?
>>
>> Thanks!
>>
>>
>>
>>
>

-- 
This message may contain confidential and/or privileged information. 
If 
you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in 
error, please advise the sender immediately by reply email and delete this 
message. Thank you.

Dropped Mutations

[shalom sagges]

Hi All,

I have a 44 node cluster (22 nodes on each DC).
Each node has 24 cores and 130 GB RAM, 3 TB HDDs.
Version 2.0.14 (soon to be upgraded)
~10K writes per second per node.
Heap size: 8 GB max, 2.4 GB newgen

I deployed Reaper and GC started to increase rapidly. I'm not sure if it's
because there was a lot of inconsistency in the data, but I decided to
increase the heap to 16 GB and new gen to 6 GB. I increased the max tenure
from 1 to 5.

I tested on a canary node and everything was fine but when I changed the
entire DC, I suddenly saw a lot of dropped mutations in the logs on most of
the nodes. (Reaper was not running on the cluster yet but a manual repair
was running).

Can the heap increment cause lots of dropped mutations?
When is a mutation considered as dropped? Is it during flush? Is it during
the write to the commit log or memtable?

Thanks!

Re: Text or....

[shalom sagges]

Thanks DuyHai!

I'm using the default table compression. Is there anything else I should
look into?
Regarding the table compression, I understand that for write heavy tables,
it's best to keep the default and not compress it further. Have I
understood correctly?

On Wed, Apr 4, 2018 at 3:28 PM, DuyHai Doan <doanduy...@gmail.com> wrote:

> Compress it and stores it as a blob.
> Unless you ever need to index it but I guess even with SASI indexing a so
> huge text block is not a good idea
>
> On Wed, Apr 4, 2018 at 2:25 PM, shalom sagges <shalomsag...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> A certain application is writing ~55,000 characters for a single row.
>> Most of these characters are entered to one column with "text" data type.
>>
>> This looks insanely large for one row.
>> Would you suggest to change the data type from "text" to BLOB or any
>> other option that might fit this scenario?
>>
>> Thanks!
>>
>
>

Text or....

[shalom sagges]

Hi All,

A certain application is writing ~55,000 characters for a single row. Most
of these characters are entered to one column with "text" data type.

This looks insanely large for one row.
Would you suggest to change the data type from "text" to BLOB or any other
option that might fit this scenario?

Thanks!

Re: Large Partitions

[shalom sagges]

Thanks Ali!

I use a 13 months TTL on this table. I guess I need to remodel this table.
And I'll definitely try this tool.



On Tue, Apr 3, 2018 at 1:28 AM, Ali Hubail <ali.hub...@petrolink.com> wrote:

> system.log should show you some warnings about wide rows. Do a grep on
> system.log for 'Writing large partition' The message could be different for
> the c* version you're using though. Plus, this doesn't show you all of the
> large partitions.
>
> There is a nice tool that analyzes sstables and can show the large
> partitions:
> https://github.com/tolbertam/sstable-tools
>
>
> By "how to deal with them?" it depends. If you don't need those
> partitions then you can delete them. You can also use TTL if it fits you or
> remodel your table to only hold upto 100k rows or 100mb per partition
> (whichever comes first). If you're going to remodel the table, aim for much
> less than 100k/100mb per partition.
>
> *Ali Hubail*
>
> Confidentiality warning: This message and any attachments are intended
> only for the persons to whom this message is addressed, are confidential,
> and may be privileged. If you are not the intended recipient, you are
> hereby notified that any review, retransmission, conversion to hard copy,
> copying, modification, circulation or other use of this message and any
> attachments is strictly prohibited. If you receive this message in error,
> please notify the sender immediately by return email, and delete this
> message and any attachments from your system. Petrolink International
> Limited its subsidiaries, holding companies and affiliates disclaims all
> responsibility from and accepts no liability whatsoever for the
> consequences of any unauthorized person acting, or refraining from acting,
> on any information contained in this message. For security purposes, staff
> training, to assist in resolving complaints and to improve our customer
> service, email communications may be monitored and telephone calls may be
> recorded.
>
>
> *shalom sagges <shalomsag...@gmail.com <shalomsag...@gmail.com>>*
>
> 04/02/2018 03:57 PM
> Please respond to
> user@cassandra.apache.org
>
> To
> user@cassandra.apache.org,
>
> cc
> Subject
> Large Partitions
>
>
>
>
> Hi All,
>
> I ran nodetool cfstats (v2.0.14) on a keyspace and found that there are a
> few large partitions. I assume that since "Compacted partition maximum
> bytes": 802187438 (~800 MB) and since
> "Compacted partition mean bytes": 100465 (~100 KB), it means that most
> partitions are in okay size and only a few are large. Am I assuming
> correctly?
>
> If so, can anyone suggest how to find those large partitions and how to
> deal with them? (cfstats output below)
>
> Thanks!
>
>
> nodetool cfstats keyspace1;
>
> Table: table1
> SSTable count: 16
> Space used (live), bytes: 453844035587
> Space used (total), bytes: 453844035587
> Off heap memory used (total), bytes: 440787635
> SSTable Compression Ratio: 0.17417149031966575
> Number of keys (estimate): 33651200
> Memtable cell count: 27966
> Memtable data size, bytes: 41698140
> Memtable switch count: 199727
> Local read count: 86494530
> Local read latency: 2.646 ms
> Local write count: 247712138
> Local write latency: 0.030 ms
> Pending tasks: 0
> Bloom filter false positives: 2182242
> Bloom filter false ratio: 0.02251
> Bloom filter space used, bytes: 53135136
> Bloom filter off heap memory used, bytes: 53135008
> Index summary off heap memory used, bytes: 11560419
> Compression metadata off heap memory used, bytes: 376092208
> Compacted partition minimum bytes: 373
>
> * Compacted partition maximum bytes: 802187438 Compacted partition mean
> bytes: 100465*
> Average live cells per slice (last five minutes): 37.0
> Average tombstones per slice (last five minutes): 0.0
>
>

Large Partitions

[shalom sagges]

Hi All,

I ran nodetool cfstats (v2.0.14) on a keyspace and found that there are a
few large partitions. I assume that since "Compacted partition maximum
bytes": 802187438 (~800 MB) and since
"Compacted partition mean bytes": 100465 (~100 KB), it means that most
partitions are in okay size and only a few are large. Am I assuming
correctly?

If so, can anyone suggest how to find those large partitions and how to
deal with them? (cfstats output below)

Thanks!


nodetool cfstats keyspace1;

Table: table1
SSTable count: 16
Space used (live), bytes: 453844035587
Space used (total), bytes: 453844035587
Off heap memory used (total), bytes: 440787635
SSTable Compression Ratio: 0.17417149031966575
Number of keys (estimate): 33651200
Memtable cell count: 27966
Memtable data size, bytes: 41698140
Memtable switch count: 199727
Local read count: 86494530
Local read latency: 2.646 ms
Local write count: 247712138
Local write latency: 0.030 ms
Pending tasks: 0
Bloom filter false positives: 2182242
Bloom filter false ratio: 0.02251
Bloom filter space used, bytes: 53135136
Bloom filter off heap memory used, bytes: 53135008
Index summary off heap memory used, bytes: 11560419
Compression metadata off heap memory used, bytes: 376092208
Compacted partition minimum bytes: 373

*Compacted partition maximum bytes: 802187438Compacted partition mean
bytes: 100465*
Average live cells per slice (last five minutes): 37.0
Average tombstones per slice (last five minutes): 0.0

Re: How to Protect Tracing Requests From Client Side

[shalom sagges]

Thanks Guys!

This really helps!



On Fri, Mar 23, 2018 at 7:10 AM, Mick Semb Wever 
wrote:

> Is there a way to protect C* on the server side from tracing commands that
>> are executed from clients?
>>
>
>
> If you really needed a way to completely disable all and any possibility
> of tracing you could start each C* node with tracing switched to a noop
> implementation.
>
> eg, add to the jvm.options file the line
>
> -Dcassandra.custom_tracing_class=somepackage.NoOpTracing
>
>
> while also putting into each $CASSANDRA_HOME/lib/ a jar file containing
> this NoOpTracing class…
>
> ```
> package somepackage;
>
> import java.net.InetAddress;
> import java.nio.ByteBuffer;
> import java.util.Map;
> import java.util.UUID;
> import org.apache.cassandra.tracing.*;
> import org.apache.cassandra.utils.FBUtilities;
>
> /** Starting Cassandra with '-Dcassandra.custom_tracing_
> class=org.apache.cassandra.tracing.NoOpTracing'
>  * will forcibly disable all tracing.
>  *
>  * This can be useful in defensive environments.
>  */
> public final class NoOpTracing extends Tracing {
>
> @Override
> protected void stopSessionImpl() {}
>
> @Override
> public TraceState begin(String request, InetAddress client,
> Map parameters) {
> return NoOpTraceState.INSTANCE;
> }
>
> @Override
> protected TraceState newTraceState(InetAddress coordinator, UUID
> sessionId, TraceType traceType) {
> return NoOpTraceState.INSTANCE;
> }
>
> @Override
> public void trace(ByteBuffer sessionId, String message, int ttl) {}
>
> private static class NoOpTraceState extends TraceState {
> private static final NoOpTraceState INSTANCE = new
> NoOpTraceState();
> private NoOpTraceState() {
> super(FBUtilities.getBroadcastAddress(), UUID.randomUUID(),
> TraceType.NONE);
> }
> @Override
> protected void traceImpl(String message) {}
> }
> }
> ```
>
> regards,
> Mick
>
>
> --
> Mick Semb Wever
> Australia
>
> The Last Pickle
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>

Re: How to Protect Tracing Requests From Client Side

[shalom sagges]

Thanks a lot Rahul! :-)

On Thu, Mar 22, 2018 at 8:03 PM, Rahul Singh <rahul.xavier.si...@gmail.com>
wrote:

> Execute ‘nodetool settraceprobability 0’ on all nodes. It does zero
> percentage of he tracing.
>
> --
> Rahul Singh
> rahul.si...@anant.us
>
> Anant Corporation
>
> On Mar 22, 2018, 11:10 AM -0500, shalom sagges <shalomsag...@gmail.com>,
> wrote:
>
> Hi All,
>
> Is there a way to protect C* on the server side from tracing commands that
> are executed from clients?
>
> Thanks!
>
>

How to Protect Tracing Requests From Client Side

[shalom sagges]

Hi All,

Is there a way to protect C* on the server side from tracing commands that
are executed from clients?

Thanks!

Re: compaction stuck at 99.99%

[shalom sagges]

If the problem is recurring, then you might have a corrupted SSTable.
Check the system log. If a certain file is corrupted, you'll find it.

grep -i corrupt /system.log*


On Wed, Mar 21, 2018 at 2:18 PM, Jerome Basa  wrote:

> hi,
>
> when i run `nodetool compactionstats` there’s this one compaction
> that’s stuck at 99.99% and the CPU load on the node is high compared
> to other nodes. i tried stopping the compaction but nothing happens.
> aside from restarting cassandra; what else can be done with this
> issue? thanks
>
>
> $ nodetool version
> ReleaseVersion: 3.0.14
>
>
> $ nodetool compactionstats -H
> pending tasks: 1
>  id   compaction type
> keyspacetable   completed  totalunit
> progress
>6fb294d0-264a-11e8-ad75-b98b064c302bCompaction
> some_keyspace   some_table90.42 MB   90.43 MB   bytes
>99.99%
>
>
>
> regards,
> -jerome
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: Delete System_Traces Table

[shalom sagges]

Thanks a lot Chris and Rahul!

On Mon, Mar 19, 2018 at 5:54 PM, Chris Lohfink <clohf...@apple.com> wrote:

> traces and auth in that version have a whitelist of tables that can be
> dropped (legacy auth tables).
>
> https://github.com/apache/cassandra/blob/cassandra-3.0.
> 12/src/java/org/apache/cassandra/service/ClientState.java#L367
>
> It does make sense to allowing CREATEs in the distributed tables, mostly
> because of auth. That way if the auth tables are changed in later version
> you can pre-prime them before an upgrade. Might be a bit of overstep in
> protecting users from themselves but it doesnt hurt anything to have the
> table there.  Just ignore it and its existence will not cause any issues.
>
> Chris
>
>
> On Mar 19, 2018, at 10:27 AM, shalom sagges <shalomsag...@gmail.com>
> wrote:
>
> That's weird... I'm using 3.0.12, so I should've still been able to drop
> it, no?
>
> Also, if I intend to upgrade to version 3.11.2, will the existence of the
> table cause any issues?
>
> Thanks!
>
> On Mon, Mar 19, 2018 at 4:30 PM, Chris Lohfink <clohf...@apple.com> wrote:
>
>> Oh I misread original, I see.
>>
>> With https://issues.apache.org/jira/browse/CASSANDRA-13813 you wont be
>> able to drop the table, but would be worth a ticket to prevent creation in
>> those keyspaces or allow some sort of override if allowing create.
>>
>> Chris
>>
>>
>> On Mar 19, 2018, at 9:15 AM, shalom sagges <shalomsag...@gmail.com>
>> wrote:
>>
>> Yes, that's correct.
>>
>> I'd definitely like to keep the default tables.
>>
>> On Mon, Mar 19, 2018 at 4:10 PM, Rahul Singh <
>> rahul.xavier.si...@gmail.com> wrote:
>>
>>> I think he just wants to delete the test table not the whole keyspace.
>>> Is that correct?
>>>
>>> --
>>> Rahul Singh
>>> rahul.si...@anant.us
>>>
>>> Anant Corporation
>>>
>>> On Mar 19, 2018, 9:08 AM -0500, Chris Lohfink <clohf...@apple.com>,
>>> wrote:
>>>
>>> No.
>>>
>>> Why do you want to? If you don't use tracing they will be empty, and if
>>> were able to drop them you will no longer be able to use tracing in
>>> debugging.
>>>
>>> Chris
>>>
>>> On Mar 19, 2018, at 7:52 AM, shalom sagges <shalomsag...@gmail.com>
>>> wrote:
>>>
>>> Hi All,
>>>
>>> I accidentally created a test table on the system_traces keyspace.
>>>
>>> When I tried to drop the table with the Cassandra user, I got the
>>> following error:
>>> *Unauthorized: Error from server: code=2100 [Unauthorized]
>>> message="Cannot DROP "*
>>>
>>> Is there a way to drop this table permanently?
>>>
>>> Thanks!
>>>
>>>
>>>
>>
>>
>
>

Re: Delete System_Traces Table

[shalom sagges]

That's weird... I'm using 3.0.12, so I should've still been able to drop
it, no?

Also, if I intend to upgrade to version 3.11.2, will the existence of the
table cause any issues?

Thanks!

On Mon, Mar 19, 2018 at 4:30 PM, Chris Lohfink <clohf...@apple.com> wrote:

> Oh I misread original, I see.
>
> With https://issues.apache.org/jira/browse/CASSANDRA-13813 you wont be
> able to drop the table, but would be worth a ticket to prevent creation in
> those keyspaces or allow some sort of override if allowing create.
>
> Chris
>
>
> On Mar 19, 2018, at 9:15 AM, shalom sagges <shalomsag...@gmail.com> wrote:
>
> Yes, that's correct.
>
> I'd definitely like to keep the default tables.
>
> On Mon, Mar 19, 2018 at 4:10 PM, Rahul Singh <rahul.xavier.si...@gmail.com
> > wrote:
>
>> I think he just wants to delete the test table not the whole keyspace. Is
>> that correct?
>>
>> --
>> Rahul Singh
>> rahul.si...@anant.us
>>
>> Anant Corporation
>>
>> On Mar 19, 2018, 9:08 AM -0500, Chris Lohfink <clohf...@apple.com>,
>> wrote:
>>
>> No.
>>
>> Why do you want to? If you don't use tracing they will be empty, and if
>> were able to drop them you will no longer be able to use tracing in
>> debugging.
>>
>> Chris
>>
>> On Mar 19, 2018, at 7:52 AM, shalom sagges <shalomsag...@gmail.com>
>> wrote:
>>
>> Hi All,
>>
>> I accidentally created a test table on the system_traces keyspace.
>>
>> When I tried to drop the table with the Cassandra user, I got the
>> following error:
>> *Unauthorized: Error from server: code=2100 [Unauthorized]
>> message="Cannot DROP "*
>>
>> Is there a way to drop this table permanently?
>>
>> Thanks!
>>
>>
>>
>
>

Re: Delete System_Traces Table

[shalom sagges]

Yes, that's correct.

I'd definitely like to keep the default tables.

On Mon, Mar 19, 2018 at 4:10 PM, Rahul Singh <rahul.xavier.si...@gmail.com>
wrote:

> I think he just wants to delete the test table not the whole keyspace. Is
> that correct?
>
> --
> Rahul Singh
> rahul.si...@anant.us
>
> Anant Corporation
>
> On Mar 19, 2018, 9:08 AM -0500, Chris Lohfink <clohf...@apple.com>, wrote:
>
> No.
>
> Why do you want to? If you don't use tracing they will be empty, and if
> were able to drop them you will no longer be able to use tracing in
> debugging.
>
> Chris
>
> On Mar 19, 2018, at 7:52 AM, shalom sagges <shalomsag...@gmail.com> wrote:
>
> Hi All,
>
> I accidentally created a test table on the system_traces keyspace.
>
> When I tried to drop the table with the Cassandra user, I got the
> following error:
> *Unauthorized: Error from server: code=2100 [Unauthorized] message="Cannot
> DROP "*
>
> Is there a way to drop this table permanently?
>
> Thanks!
>
>
>

Delete System_Traces Table

[shalom sagges]

Hi All,

I accidentally created a test table on the system_traces keyspace.

When I tried to drop the table with the Cassandra user, I got the following
error:
*Unauthorized: Error from server: code=2100 [Unauthorized] message="Cannot
DROP "*

Is there a way to drop this table permanently?

Thanks!

Re: Version Rollback

[shalom sagges]

These are really good directions. Thanks a lot everyone!

@Kenneth - The cluster is comprised of 44 nodes, version 2.0.14, ~2.5TB of
data per node. It's gonna be a major version upgrade (or upgrades to be
exact... version 3.x is the target).

@Jeff, I have a passive DC. What if I upgrade the passive DC and if all
goes well, move the applications to work with the passive DC and then
upgrade the active DC. Is this doable?
Also, Would you suggest to upgrade one node (binaries), upgrade the
SSTables and move to the second node, and then third etc, or first upgrade
binaries to all nodes, and only then start with the SSTables upgrade?

Thanks!



On Tue, Feb 27, 2018 at 7:47 PM, Jeff Jirsa <jji...@gmail.com> wrote:

> MOST minor versions support rollback - the exceptions are those where
> internode protocol changes (3.0.14 being the only one in recent memory), or
> where sstable format changes (again rare). No major versions support
> rollback - the only way to do it is to upgrade in a way that you can
> effectively reinstall the old version without data loss.
>
> The steps usually look like:
>
> Test in a lab
> Test in a lab again
> Test in a lab a few more times
> Snapshot everything
>
> If you have a passive data center:
> - upgrade one instance
> - check to see if it’s happy
> - upgrade another
> - check to see if it’s happy
> - continue until the passive dc is done
> - if at any point they’re unhappy rebuild (wipe and restream the old
> version) the dc from the active dc
>
> On the active DCs, you’ll want to canary it one replica at a time so you
> can treat a failed upgrade like a bad disk:
> - upgrade one instance
> - check if it’s happy; if it’s not treat it like a failed disk and replace
> it with the old version
> - if you’re using single token, do another instance in a different replica
> set, repeat until you’re out of different replicas.
> - if you’re using vnodes but a rack aware snitch and have more racks than
> your RF, do another instance in the same rack as the canary, repeat until
> you’re out of instances in that rack
>
> This is typically your point of no return - as soon as you have two
> replicas in the new version there’s no more rollback practical.
>
>
>
> --
> Jeff Jirsa
>
>
> On Feb 27, 2018, at 9:22 AM, Carl Mueller <carl.muel...@smartthings.com>
> wrote:
>
> My speculation is that IF (bigif) the sstable formats are compatible
> between the versions, which probably isn't the case for major versions,
> then you could drop back.
>
> If the sstables changed format, then you'll probably need to figure out
> how to rewrite the sstables in the older format and then sstableloader them
> in the older-version cluster if need be. Alas, while there is an sstable
> upgrader, there isn't a downgrader AFAIK.
>
> And I don't have an intimate view of version-by-version sstable format
> changes and compatibilities. You'd probably need to check the upgrade
> instructions (which you presumably did if you're upgrading versions) to
> tell.
>
> Basically, version rollback is pretty unlikely to be done.
>
> The OTHER option:
>
> 1) build a new cluster with the new version, no new data.
>
> 2) code your driver interfaces to interface with both clusters. Write to
> both, but read preferentially from the new, then fall through to the old.
> Yes, that gets hairy on multiple row queries. Port your data with sstable
> loading from the old to the new gradually.
>
> When you've done a full load of all the data from old to new, and you're
> satisfied with the new cluster stability, retire the old cluster.
>
> For merging two multirow sets you'll probably need your multirow queries
> to return the partition hash value (or extract the code that generates the
> hash), and have two simulaneous java-driver ResultSets going, and merge
> their results, providing the illusion of a single database query. You'll
> need to pay attention to both the row key ordering and column key ordering
> to ensure the combined results are properly ordered.
>
> Writes will be slowed by the double-writes, reads you'll be bound by the
> worse performing cluster.
>
> On Tue, Feb 27, 2018 at 8:23 AM, Kenneth Brotman <
> kenbrot...@yahoo.com.invalid> wrote:
>
>> Could you tell us the size and configuration of your Cassandra cluster?
>>
>>
>>
>> Kenneth Brotman
>>
>>
>>
>> *From:* shalom sagges [mailto:shalomsag...@gmail.com]
>> *Sent:* Tuesday, February 27, 2018 6:19 AM
>> *To:* user@cassandra.apache.org
>> *Subject:* Version Rollback
>>
>>
>>
>> Hi All,
>>
>> I'm planning to upgrade my C* cluster to version 3.x and was wondering
>> what's the best way to 

Version Rollback

[shalom sagges]

Hi All,

I'm planning to upgrade my C* cluster to version 3.x and was wondering
what's the best way to perform a rollback if need be.

If I used snapshot restoration, I would be facing data loss, depends when I
took the snapshot (i.e. a rollback might be required after upgrading half
the cluster for example).

If I add another DC to the cluster with the old version, then I could point
the apps to talk to that DC if anything bad happens, but building it is
really time consuming and requires a lot of resources.

Can anyone provide recommendations on this matter? Any ideas on how to make
the upgrade foolproof, or at least "really really safe"?

Thanks!

Re: SSTableLoader Question

[shalom sagges]

Sounds good.

Thanks for the explanation!

On Sun, Feb 18, 2018 at 5:15 PM, Rahul Singh <rahul.xavier.si...@gmail.com>
wrote:

> If you don’t have access to the file you don’t have access to the file.
> I’ve seen this issue several times. It’s he easiest low hanging fruit to
> resolve. So figure it out and make sure that it’s Cassandra.Cassandra from
> root to he Data folder and either run as root or sudo it.
>
> If it’s compacted it won’t be there so you won’t have the file. I’m not
> aware of this event being communicated to Sstableloader via SEDA. Besides,
> the sstable that you are loading SHOULD not be live. If you at streaming a
> life sstable, it means you are using sstableloader not as it is designed to
> be used - which is with static files.
>
> --
> Rahul Singh
> rahul.si...@anant.us
>
> Anant Corporation
>
> On Feb 18, 2018, 9:22 AM -0500, shalom sagges <shalomsag...@gmail.com>,
> wrote:
>
> Not really sure with which user I ran it (root or cassandra), although I
> don't understand why a permission issue will generate a File not Found
> exception?
>
> And in general, what if a file is being streamed and got compacted before
> the streaming ended. Does Cassandra know how to handle this?
>
> Thanks!
>
> On Sun, Feb 18, 2018 at 3:58 PM, Rahul Singh <rahul.xavier.si...@gmail.com
> > wrote:
>
>> Check permissions maybe? Who owns the files vs. who is running
>> sstableloader.
>>
>> --
>> Rahul Singh
>> rahul.si...@anant.us
>>
>> Anant Corporation
>>
>> On Feb 18, 2018, 4:26 AM -0500, shalom sagges <shalomsag...@gmail.com>,
>> wrote:
>>
>> Hi All,
>>
>> C* version 2.0.14.
>>
>> I was loading some data to another cluster using SSTableLoader. The
>> streaming failed with the following error:
>>
>>
>> Streaming error occurred
>> java.lang.RuntimeException: java.io.*FileNotFoundException*:
>> /data1/keyspace1/table1/keyspace1-table1-jb-65174-Data.db (No such file
>> or directory)
>> at org.apache.cassandra.io.compress.CompressedRandomAccessReade
>> r.open(CompressedRandomAccessReader.java:59)
>> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader
>> (SSTableReader.java:1409)
>> at org.apache.cassandra.streaming.compress.CompressedStreamWrit
>> er.write(CompressedStreamWriter.java:55)
>> at org.apache.cassandra.streaming.messages.OutgoingFileMessage$
>> 1.serialize(OutgoingFileMessage.java:59)
>> at org.apache.cassandra.streaming.messages.OutgoingFileMessage$
>> 1.serialize(OutgoingFileMessage.java:42)
>> at org.apache.cassandra.streaming.messages.StreamMessage.serial
>> ize(StreamMessage.java:45)
>> at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMes
>> sageHandler.sendMessage(ConnectionHandler.java:339)
>> at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMes
>> sageHandler.run(ConnectionHandler.java:311)
>> at java.lang.Thread.run(Thread.java:722)
>> Caused by: java.io.*FileNotFoundException*:
>> /data1/keyspace1/table1/keyspace1-table1-jb-65174-Data.db (No such file
>> or directory)
>> at java.io.RandomAccessFile.open(Native Method)
>> at java.io.RandomAccessFile.(RandomAccessFile.java:233)
>> at org.apache.cassandra.io.util.RandomAccessReader.(Rando
>> mAccessReader.java:58)
>> at org.apache.cassandra.io.compress.CompressedRandomAccessReade
>> r.(CompressedRandomAccessReader.java:76)
>> at org.apache.cassandra.io.compress.CompressedRandomAccessReade
>> r.open(CompressedRandomAccessReader.java:55)
>> ... 8 more
>>  WARN 18:31:35,938 [Stream #7243efb0-1262-11e8-8562-d19d5fe7829c] Stream
>> failed
>>
>>
>>
>> Did I miss something when running the load? Was the file suddenly missing
>> due to compaction?
>> If so, did I need to disable auto compaction or stop the service
>> beforehand? (didn't find any reference to compaction in the docs)
>>
>> I know it's an old version, but I didn't find any related bugs on "File
>> not found" exceptions.
>>
>> Thanks!
>>
>>
>>
>

Re: SSTableLoader Question

[shalom sagges]

Not really sure with which user I ran it (root or cassandra), although I
don't understand why a permission issue will generate a File not Found
exception?

And in general, what if a file is being streamed and got compacted before
the streaming ended. Does Cassandra know how to handle this?

Thanks!

On Sun, Feb 18, 2018 at 3:58 PM, Rahul Singh <rahul.xavier.si...@gmail.com>
wrote:

> Check permissions maybe? Who owns the files vs. who is running
> sstableloader.
>
> --
> Rahul Singh
> rahul.si...@anant.us
>
> Anant Corporation
>
> On Feb 18, 2018, 4:26 AM -0500, shalom sagges <shalomsag...@gmail.com>,
> wrote:
>
> Hi All,
>
> C* version 2.0.14.
>
> I was loading some data to another cluster using SSTableLoader. The
> streaming failed with the following error:
>
>
> Streaming error occurred
> java.lang.RuntimeException: java.io.*FileNotFoundException*:
> /data1/keyspace1/table1/keyspace1-table1-jb-65174-Data.db (No such file
> or directory)
> at org.apache.cassandra.io.compress.CompressedRandomAccessReade
> r.open(CompressedRandomAccessReader.java:59)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader
> (SSTableReader.java:1409)
> at org.apache.cassandra.streaming.compress.CompressedStreamWrit
> er.write(CompressedStreamWriter.java:55)
> at org.apache.cassandra.streaming.messages.OutgoingFileMessage$
> 1.serialize(OutgoingFileMessage.java:59)
> at org.apache.cassandra.streaming.messages.OutgoingFileMessage$
> 1.serialize(OutgoingFileMessage.java:42)
> at org.apache.cassandra.streaming.messages.StreamMessage.
> serialize(StreamMessage.java:45)
> at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMes
> sageHandler.sendMessage(ConnectionHandler.java:339)
> at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMes
> sageHandler.run(ConnectionHandler.java:311)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.*FileNotFoundException*: /data1/keyspace1/table1/
> keyspace1-table1-jb-65174-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.(Rando
> mAccessReader.java:58)
> at org.apache.cassandra.io.compress.CompressedRandomAccessReade
> r.(CompressedRandomAccessReader.java:76)
> at org.apache.cassandra.io.compress.CompressedRandomAccessReade
> r.open(CompressedRandomAccessReader.java:55)
> ... 8 more
>  WARN 18:31:35,938 [Stream #7243efb0-1262-11e8-8562-d19d5fe7829c] Stream
> failed
>
>
>
> Did I miss something when running the load? Was the file suddenly missing
> due to compaction?
> If so, did I need to disable auto compaction or stop the service
> beforehand? (didn't find any reference to compaction in the docs)
>
> I know it's an old version, but I didn't find any related bugs on "File
> not found" exceptions.
>
> Thanks!
>
>
>

SSTableLoader Question

[shalom sagges]

Hi All,

C* version 2.0.14.

I was loading some data to another cluster using SSTableLoader. The
streaming failed with the following error:


Streaming error occurred
java.lang.RuntimeException: java.io.*FileNotFoundException*:
/data1/keyspace1/table1/keyspace1-table1-jb-65174-Data.db (No such file or
directory)
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.
open(CompressedRandomAccessReader.java:59)
at org.apache.cassandra.io.sstable.SSTableReader.
openDataReader(SSTableReader.java:1409)
at org.apache.cassandra.streaming.compress.
CompressedStreamWriter.write(CompressedStreamWriter.java:55)
at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.
serialize(OutgoingFileMessage.java:59)
at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.
serialize(OutgoingFileMessage.java:42)
at org.apache.cassandra.streaming.messages.StreamMessage.serialize(
StreamMessage.java:45)
at org.apache.cassandra.streaming.ConnectionHandler$
OutgoingMessageHandler.sendMessage(ConnectionHandler.java:339)
at org.apache.cassandra.streaming.ConnectionHandler$
OutgoingMessageHandler.run(ConnectionHandler.java:311)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.*FileNotFoundException*:
/data1/keyspace1/table1/keyspace1-table1-jb-65174-Data.db (No such file or
directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at org.apache.cassandra.io.util.RandomAccessReader.(
RandomAccessReader.java:58)
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<
init>(CompressedRandomAccessReader.java:76)
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.
open(CompressedRandomAccessReader.java:55)
... 8 more
 WARN 18:31:35,938 [Stream #7243efb0-1262-11e8-8562-d19d5fe7829c] Stream
failed



Did I miss something when running the load? Was the file suddenly missing
due to compaction?
If so, did I need to disable auto compaction or stop the service
beforehand? (didn't find any reference to compaction in the docs)

I know it's an old version, but I didn't find any related bugs on "File not
found" exceptions.

Thanks!

C* Logs to Kibana

[shalom sagges]

Hi All,

I want to push the Cassandra logs (version 3.x) to Kibana.
Is there a way to configure the Cassandra logs to be in json format?

If modifying the logs to json is not an option, I came across this blog
post from about a year ago regarding that matter:
https://medium.com/@alain.rastoul/pushing-cassandra-logs-into-elasticsearch-9be3b52af754

Is that a good way of accomplishing that?

Thanks!

Re: 3.0.15 or 3.11.1

[shalom sagges]

Thanks a lot for the info!
Much appreciated.

On Tue, Jan 9, 2018 at 2:33 AM, Mick Semb Wever 
wrote:

>
>
>> Can you please provide dome JIRAs for superior fixes and performance
>> improvements which are present in 3.11.1 but are missing in 3.0.15.
>>
>
>
> Some that come to mind…
>
> Cassandra Storage Engine: CASSANDRA-12269, CASSANDRA-12731
>
> Streaming and Compaction: CASSANDRA-11206, CASSANDRA-
> 9766, CASSANDRA-11623,
>
> Reintroduce off heap memtables –  CASSANDRA-9472
>
>

Re: 3.0.15 or 3.11.1

[shalom sagges]

Thanks Guys!

Sorry for the late reply.
I'm interested in TWCS where I understand is more stable in 3.11.1 than in
3.0.15, tombstone compaction and slow logs.

I don't plan to use MVs and SASI in the near future, as I understand are
not Production ready.

Is it okay to use the above features?





On Thu, Jan 4, 2018 at 1:07 AM, Mick Semb Wever 
wrote:

>
>
>> I want to upgrade from 2.x to 3.x.
>>
>> I can definitely use the features in 3.11.1 but it's not a must.
>> So my question is, is 3.11.1 stable and suitable for Production compared
>> to 3.0.15?
>>
>
>
> Use 3.11.1 and don't use any 3.0.x or 3.x features.
> 3.11.1 is effectively three sequential patch releases, and the tick-tock
> releases offered a number of superior fixes and performance improvements
> over what was done in 3.0.x.
>
> Introduce the use of new features later on, one at a time, after thorough
> testing and staging.
>
> regards,
> Mick
>

3.0.15 or 3.11.1

[shalom sagges]

Hi All,

I want to upgrade from 2.x to 3.x.

I can definitely use the features in 3.11.1 but it's not a must.
So my question is, is 3.11.1 stable and suitable for Production compared to
3.0.15?

Thanks!

Re: Adding a New Node

[shalom sagges]

Thanks Kurt!

That sorted things in my head. Much appreciated!



On Tue, Oct 24, 2017 at 12:29 PM, kurt greaves <k...@instaclustr.com> wrote:

> Your node shouldn't show up in DC1 in nodetool status from the other
> nodes, this implies a configuration problem. Sounds like you haven't added
> the new node to all the existing nodes cassandra-topology.properties file.
> You don't need to do a rolling restart with PropertyFileSnitch, it should
> reload the cassandra-topology.properties file automatically every 5 seconds.
>
> With GPFS each node only needs to know about its own topology settings in
> cassandra-rackdc.properties, so the problem you point out in 2 goes away,
> as when adding a node you only need to specify its configuration and that
> will be propagated to the rest of the cluster through gossip.
>
> On 24 October 2017 at 07:13, shalom sagges <shalomsag...@gmail.com> wrote:
>
>> Hi Everyone,
>>
>> I have 2 DCs (v2.0.14) with the following topology.properties:
>>
>> DC1:
>> xxx11=DC1:RAC1
>> xxx12=DC1:RAC1
>> xxx13=DC1:RAC1
>> xxx14=DC1:RAC1
>> xxx15=DC1:RAC1
>>
>>
>> DC2:
>> yyy11=DC2:RAC1
>> yyy12=DC2:RAC1
>> yyy13=DC2:RAC1
>> yyy14=DC2:RAC1
>> yyy15=DC2:RAC1
>>
>>
>> # default for unknown nodes
>> default=DC1:RAC1
>>
>> Now let's say that I want to add a new node yyy16 to DC2, and I've added
>> yyy16 to the topology properties file only on that specific node.
>>
>> What I saw is that during bootstrap, the new node is receiving data only
>> from DC2 nodes (which is what I want), but nodetool status on other nodes
>> shows that it was joining to DC1 (which is the default DC for unknown
>> nodes).
>>
>> So I have a few questions on this matter:
>>
>> 1) What are the implications of such a bootstrap, where the joining node
>> actually gets data from nodes in the right DC, but all nodes see it in the
>> default DC when running nodetool status?
>>
>> 2) I know that I must change the topology.properties file on all nodes to
>> be the same. If I do that, do I need to perform a rolling restart on all of
>> the cluster before each bootstrap (which is a real pain for large clusters)?
>>
>> 3) Regarding the Snitch, the docs say that the recommended snitch in
>> Production is the GossipingPropertyFileSnitch with
>> cassandra-rackdc.properties file.
>> What's the difference between the GossipingPropertyFileSnitchand and the
>> PropertyFileSnitch?
>> I currently use PropertyFileSnitch and cassandra-topology.properties.
>>
>>
>> Thanks!
>>
>>
>>
>>
>>
>>
>

Adding a New Node

[shalom sagges]

Hi Everyone,

I have 2 DCs (v2.0.14) with the following topology.properties:

DC1:
xxx11=DC1:RAC1
xxx12=DC1:RAC1
xxx13=DC1:RAC1
xxx14=DC1:RAC1
xxx15=DC1:RAC1


DC2:
yyy11=DC2:RAC1
yyy12=DC2:RAC1
yyy13=DC2:RAC1
yyy14=DC2:RAC1
yyy15=DC2:RAC1


# default for unknown nodes
default=DC1:RAC1

Now let's say that I want to add a new node yyy16 to DC2, and I've added
yyy16 to the topology properties file only on that specific node.

What I saw is that during bootstrap, the new node is receiving data only
from DC2 nodes (which is what I want), but nodetool status on other nodes
shows that it was joining to DC1 (which is the default DC for unknown
nodes).

So I have a few questions on this matter:

1) What are the implications of such a bootstrap, where the joining node
actually gets data from nodes in the right DC, but all nodes see it in the
default DC when running nodetool status?

2) I know that I must change the topology.properties file on all nodes to
be the same. If I do that, do I need to perform a rolling restart on all of
the cluster before each bootstrap (which is a real pain for large clusters)?

3) Regarding the Snitch, the docs say that the recommended snitch in
Production is the GossipingPropertyFileSnitch with
cassandra-rackdc.properties file.
What's the difference between the GossipingPropertyFileSnitchand and the
PropertyFileSnitch?
I currently use PropertyFileSnitch and cassandra-topology.properties.


Thanks!

Re: User Defined Compaction Issue

[shalom sagges]

Awesome explanation :-)

Thanks a lot!

On Tue, Sep 26, 2017 at 3:40 PM, Jeff Jirsa <jji...@gmail.com> wrote:

> Write row A, flush into sstable 1
> Delete row A, flush the tombstone into sstable 2
>
> The tombstone in sstable 2 can’t be removed until row A in sstable 1 gets
> removed. If you just keep recompacting sstable 2 by itself, the row in
> sstable A remains on disk.
>
>
>
> --
> Jeff Jirsa
>
>
> On Sep 26, 2017, at 2:01 AM, shalom sagges <shalomsag...@gmail.com> wrote:
>
> Thanks Jeff!
>
> I'll try that.
> I'm not sure I understand how the tombstones are covering data in another
> file. Do you have a small example perhaps?
>
> Thanks again!
>
> On Tue, Sep 26, 2017 at 1:38 AM, Jeff Jirsa <jji...@gmail.com> wrote:
>
>> The problem is likely that your sstables overlap - your 91% droppable
>> tombstones are probably covering data in another file. Until that data is
>> removed, those tombstones cant be dropped.
>>
>> This is why a full major compaction often works better for simply
>> reclaiming disk space (though it takes a lot more disk as it runs and has
>> other drawbacks) - it joins the shadowed data with the tombstones into a
>> single file. You can sorta fake that by using user-defined compaction with
>> multiple sstables to try to compact away the shadowed data - pick 2 or more
>> -Data files and compact them together. Works best if you can be sure which
>> data is overlapping, but short of that you'll probably want to pick data
>> with approximately the same (or older) calendar timestamps.
>>
>>
>>
>> On Mon, Sep 25, 2017 at 11:10 AM, shalom sagges <shalomsag...@gmail.com>
>> wrote:
>>
>>> Hi Everyone,
>>>
>>> I'm running into an issue I can't seem to Solve.
>>>
>>> I execute force compaction in order to reclaim back storage.
>>> Everything was working fine for a time, but after a while I found that
>>> tombstones aren't being removed any longer.
>>>
>>> For example, I've compacted the following SSTable:
>>> *21G *Sep 19 10:18 file-jb-69103-Data.db
>>> Estimated droppable tombstones:
>>>
>>> *0.9115601492568154*
>>> I ran it with jmxterm and the compaction started and completed
>>> successfully.
>>>  $>run -b org.apache.cassandra.db:type=CompactionManager
>>> forceUserDefinedCompaction file-jb-69103-Data.db
>>>  #calling operation forceUserDefinedCompaction of mbean
>>> org.apache.cassandra.db:type=CompactionManager
>>>  #operation returns:
>>> null
>>>
>>>
>>> A new SStable was created, but no tombstones were removed.
>>>  *21G *Sep 25 12:16 file-jb-71319-Data.db
>>>  Estimated droppable tombstones:
>>>
>>> * 0.9115601492568154*
>>> I've tried running it from jconsole as well, but the outcome was the
>>> same.
>>>
>>> Is anyone familiar with this issue?
>>> Btw, this cluster is on version 2.0.14 .
>>>
>>>
>>> Thanks!
>>> Shalom
>>>
>>>
>>
>

Re: User Defined Compaction Issue

[shalom sagges]

Thanks Jeff!

I'll try that.
I'm not sure I understand how the tombstones are covering data in another
file. Do you have a small example perhaps?

Thanks again!

On Tue, Sep 26, 2017 at 1:38 AM, Jeff Jirsa <jji...@gmail.com> wrote:

> The problem is likely that your sstables overlap - your 91% droppable
> tombstones are probably covering data in another file. Until that data is
> removed, those tombstones cant be dropped.
>
> This is why a full major compaction often works better for simply
> reclaiming disk space (though it takes a lot more disk as it runs and has
> other drawbacks) - it joins the shadowed data with the tombstones into a
> single file. You can sorta fake that by using user-defined compaction with
> multiple sstables to try to compact away the shadowed data - pick 2 or more
> -Data files and compact them together. Works best if you can be sure which
> data is overlapping, but short of that you'll probably want to pick data
> with approximately the same (or older) calendar timestamps.
>
>
>
> On Mon, Sep 25, 2017 at 11:10 AM, shalom sagges <shalomsag...@gmail.com>
> wrote:
>
>> Hi Everyone,
>>
>> I'm running into an issue I can't seem to Solve.
>>
>> I execute force compaction in order to reclaim back storage.
>> Everything was working fine for a time, but after a while I found that
>> tombstones aren't being removed any longer.
>>
>> For example, I've compacted the following SSTable:
>> *21G *Sep 19 10:18 file-jb-69103-Data.db
>> Estimated droppable tombstones:
>>
>> *0.9115601492568154*
>> I ran it with jmxterm and the compaction started and completed
>> successfully.
>>  $>run -b org.apache.cassandra.db:type=CompactionManager
>> forceUserDefinedCompaction file-jb-69103-Data.db
>>  #calling operation forceUserDefinedCompaction of mbean
>> org.apache.cassandra.db:type=CompactionManager
>>  #operation returns:
>> null
>>
>>
>> A new SStable was created, but no tombstones were removed.
>>  *21G *Sep 25 12:16 file-jb-71319-Data.db
>>  Estimated droppable tombstones:
>>
>> * 0.9115601492568154*
>> I've tried running it from jconsole as well, but the outcome was the
>> same.
>>
>> Is anyone familiar with this issue?
>> Btw, this cluster is on version 2.0.14 .
>>
>>
>> Thanks!
>> Shalom
>>
>>
>

User Defined Compaction Issue

[shalom sagges]

Hi Everyone,

I'm running into an issue I can't seem to Solve.

I execute force compaction in order to reclaim back storage.
Everything was working fine for a time, but after a while I found that
tombstones aren't being removed any longer.

For example, I've compacted the following SSTable:
*21G *Sep 19 10:18 file-jb-69103-Data.db
Estimated droppable tombstones:

*0.9115601492568154*
I ran it with jmxterm and the compaction started and completed
successfully.
 $>run -b org.apache.cassandra.db:type=CompactionManager
forceUserDefinedCompaction file-jb-69103-Data.db
 #calling operation forceUserDefinedCompaction of mbean
org.apache.cassandra.db:type=CompactionManager
 #operation returns:
null


A new SStable was created, but no tombstones were removed.
 *21G *Sep 25 12:16 file-jb-71319-Data.db
 Estimated droppable tombstones:

* 0.9115601492568154*
I've tried running it from jconsole as well, but the outcome was the same.

Is anyone familiar with this issue?
Btw, this cluster is on version 2.0.14 .


Thanks!
Shalom

Re: old big tombstone data file occupy much disk space

[Shalom Sagges]

Thanks! :-)



On Mon, Sep 4, 2017 at 2:56 PM, Nicolas Guyomar <nicolas.guyo...@gmail.com>
wrote:

> Wrong copy/paste !
>
> Looking at the code, it should do nothing :
>
>  // look up the sstables now that we're on the compaction executor, so we
> don't try to re-compact
>  // something that was already being compacted earlier.
>
> On 4 September 2017 at 13:54, Nicolas Guyomar <nicolas.guyo...@gmail.com>
> wrote:
>
>> You'll get the WARN "Will not compact {}: it is not an active sstable"
>>  :)
>>
>> On 4 September 2017 at 12:07, Shalom Sagges <shal...@liveperson.com>
>> wrote:
>>
>>> By the way, does anyone know what happens if I run a user defined
>>> compaction on an sstable that's already in compaction?
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Sun, Sep 3, 2017 at 2:55 PM, Shalom Sagges <shal...@liveperson.com>
>>> wrote:
>>>
>>>> Try this blog by The Last Pickle:
>>>>
>>>> http://thelastpickle.com/blog/2016/10/18/user-defined-compaction.html
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Shalom Sagges
>>>> DBA
>>>> <http://www.linkedin.com/company/164748>
>>>> <http://twitter.com/liveperson> <http://www.facebook.com/LivePersonInc> We
>>>> Create Meaningful Connections
>>>>
>>>>
>>>>
>>>> On Sat, Sep 2, 2017 at 8:34 PM, Jeff Jirsa <jji...@gmail.com> wrote:
>>>>
>>>>> If you're on 3.0 (3.0.6 or 3.0.8 or newer I don't remember which),
>>>>> TWCS was designed for ttl-only time series use cases
>>>>>
>>>>> Alternatively, if you have IO to spare, you may find LCS works as well
>>>>> (it'll cause quite a bit more compaction, but a much higher chance to
>>>>> compact away tombstones)
>>>>>
>>>>> There are also tombstone focused sub properties to more aggressively
>>>>> compact sstables that have a lot of tombstones - check the docs for
>>>>> "unchecked tombstone compaction" and "tombstone threshold" - enabling 
>>>>> those
>>>>> will enable more aggressive automatic single-sstable compactions
>>>>>
>>>>> --
>>>>> Jeff Jirsa
>>>>>
>>>>>
>>>>> On Sep 2, 2017, at 7:10 AM, qf zhou <zhouqf2...@gmail.com> wrote:
>>>>>
>>>>>
>>>>> Yes, your are right. I am using STCS compaction strategy with some
>>>>> kind of timeseries model. Too much disk space has been occupied.
>>>>>
>>>>> What should I  do to stop  the  disk full ?
>>>>>
>>>>>  I only want to keep 100 days data most recently,  so I set 
>>>>> default_time_to_live
>>>>> = 864(100 days ).
>>>>>
>>>>> I know I need to do something to stop the disk space cost, but I
>>>>> really don’t know how to do it.
>>>>>
>>>>>
>>>>> Here is the strategy of the big data table :
>>>>>
>>>>> AND compaction = {'class': 'org.apache.cassandra.db.compa
>>>>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32',
>>>>> 'min_threshold': '12', 'tombstone_threshold': '0.1',
>>>>> 'unchecked_tombstone_compaction': 'true'}
>>>>> AND compression = {'chunk_length_in_kb': '64', 'class': '
>>>>> org.apache.cassandra.io.compress.LZ4Compressor'}
>>>>> AND crc_check_chance = 1.0
>>>>> AND dclocal_read_repair_chance = 0.1
>>>>> AND default_time_to_live = 864
>>>>> AND gc_grace_seconds = 432000
>>>>>
>>>>>
>>>>>
>>>>> 在 2017年9月2日，下午7:34，Nicolas Guyomar <nicolas.guyo...@gmail.com> 写道：
>>>>>
>>>>> your are using STCS compaction strategy with some kind of timeseries
>>>>> model, and you are going to end up with yor disk full!
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>> This message may contain confidential and/or privileged information.
>>> If you are not the addressee or authorized to receive this on behalf of
>>> the addressee you must not use, copy, disclose or take action based on this
>>> message or any information herein.
>>> If you have received this message in error, please advise the sender
>>> immediately by reply email and delete this message. Thank you.
>>>
>>
>>
>

-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.

Re: old big tombstone data file occupy much disk space

[Shalom Sagges]

By the way, does anyone know what happens if I run a user defined
compaction on an sstable that's already in compaction?





On Sun, Sep 3, 2017 at 2:55 PM, Shalom Sagges <shal...@liveperson.com>
wrote:

> Try this blog by The Last Pickle:
>
> http://thelastpickle.com/blog/2016/10/18/user-defined-compaction.html
>
>
>
>
>
>
> Shalom Sagges
> DBA
> <http://www.linkedin.com/company/164748> <http://twitter.com/liveperson>
> <http://www.facebook.com/LivePersonInc> We Create Meaningful Connections
>
>
>
> On Sat, Sep 2, 2017 at 8:34 PM, Jeff Jirsa <jji...@gmail.com> wrote:
>
>> If you're on 3.0 (3.0.6 or 3.0.8 or newer I don't remember which), TWCS
>> was designed for ttl-only time series use cases
>>
>> Alternatively, if you have IO to spare, you may find LCS works as well
>> (it'll cause quite a bit more compaction, but a much higher chance to
>> compact away tombstones)
>>
>> There are also tombstone focused sub properties to more aggressively
>> compact sstables that have a lot of tombstones - check the docs for
>> "unchecked tombstone compaction" and "tombstone threshold" - enabling those
>> will enable more aggressive automatic single-sstable compactions
>>
>> --
>> Jeff Jirsa
>>
>>
>> On Sep 2, 2017, at 7:10 AM, qf zhou <zhouqf2...@gmail.com> wrote:
>>
>>
>> Yes, your are right. I am using STCS compaction strategy with some kind
>> of timeseries model. Too much disk space has been occupied.
>>
>> What should I  do to stop  the  disk full ?
>>
>>  I only want to keep 100 days data most recently,  so I set 
>> default_time_to_live
>> = 864(100 days ).
>>
>> I know I need to do something to stop the disk space cost, but I really
>> don’t know how to do it.
>>
>>
>> Here is the strategy of the big data table :
>>
>> AND compaction = {'class': 'org.apache.cassandra.db.compa
>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32',
>> 'min_threshold': '12', 'tombstone_threshold': '0.1',
>> 'unchecked_tombstone_compaction': 'true'}
>> AND compression = {'chunk_length_in_kb': '64', 'class': '
>> org.apache.cassandra.io.compress.LZ4Compressor'}
>> AND crc_check_chance = 1.0
>> AND dclocal_read_repair_chance = 0.1
>> AND default_time_to_live = 864
>> AND gc_grace_seconds = 432000
>>
>>
>>
>> 在 2017年9月2日，下午7:34，Nicolas Guyomar <nicolas.guyo...@gmail.com> 写道：
>>
>> your are using STCS compaction strategy with some kind of timeseries
>> model, and you are going to end up with yor disk full!
>>
>>
>>
>

-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.

Re: old big tombstone data file occupy much disk space

[Shalom Sagges]

Try this blog by The Last Pickle:

http://thelastpickle.com/blog/2016/10/18/user-defined-compaction.html






Shalom Sagges
DBA
<http://www.linkedin.com/company/164748> <http://twitter.com/liveperson>
<http://www.facebook.com/LivePersonInc> We Create Meaningful Connections



On Sat, Sep 2, 2017 at 8:34 PM, Jeff Jirsa <jji...@gmail.com> wrote:

> If you're on 3.0 (3.0.6 or 3.0.8 or newer I don't remember which), TWCS
> was designed for ttl-only time series use cases
>
> Alternatively, if you have IO to spare, you may find LCS works as well
> (it'll cause quite a bit more compaction, but a much higher chance to
> compact away tombstones)
>
> There are also tombstone focused sub properties to more aggressively
> compact sstables that have a lot of tombstones - check the docs for
> "unchecked tombstone compaction" and "tombstone threshold" - enabling those
> will enable more aggressive automatic single-sstable compactions
>
> --
> Jeff Jirsa
>
>
> On Sep 2, 2017, at 7:10 AM, qf zhou <zhouqf2...@gmail.com> wrote:
>
>
> Yes, your are right. I am using STCS compaction strategy with some kind of
> timeseries model. Too much disk space has been occupied.
>
> What should I  do to stop  the  disk full ?
>
>  I only want to keep 100 days data most recently,  so I set 
> default_time_to_live
> = 864(100 days ).
>
> I know I need to do something to stop the disk space cost, but I really
> don’t know how to do it.
>
>
> Here is the strategy of the big data table :
>
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '12', 'tombstone_threshold': '0.1',
> 'unchecked_tombstone_compaction': 'true'}
> AND compression = {'chunk_length_in_kb': '64', 'class': '
> org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 864
> AND gc_grace_seconds = 432000
>
>
>
> 在 2017年9月2日，下午7:34，Nicolas Guyomar <nicolas.guyo...@gmail.com> 写道：
>
> your are using STCS compaction strategy with some kind of timeseries
> model, and you are going to end up with yor disk full!
>
>
>

-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.

Re: Working With Prepared Statements

[Shalom Sagges]

Thanks guys for all the info!




Shalom Sagges
DBA
T: +972-74-700-4035
<http://www.linkedin.com/company/164748> <http://twitter.com/liveperson>
<http://www.facebook.com/LivePersonInc> We Create Meaningful Connections
<https://liveperson.docsend.com/view/8iiswfp>


On Wed, Aug 30, 2017 at 10:54 AM, Oleksandr Shulgin <
oleksandr.shul...@zalando.de> wrote:

> On Tue, Aug 29, 2017 at 12:33 PM, Shalom Sagges <shal...@liveperson.com>
> wrote:
>
>> Insights, anyone?
>>
>
> There were reports of Cassandra failing to start due to trying to load the
> prepared statements from a cached table.  This can only affect you if you
> have a lot (tens of thousands, IIRC) of prepared statements.  A fix seems
> to have made it into 3.11.1:
>
> https://issues.apache.org/jira/browse/CASSANDRA-13641
>
> Search the the users list for "prepared statement preload" and you'll find
> some recent threads.
>
> Cheers,
> --
> Alex
>
>

-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.

Re: Working With Prepared Statements

[Shalom Sagges]

Thanks a lot!
I'll make sure it'll be prepared once.


Shalom Sagges
DBA
T: +972-74-700-4035
<http://www.linkedin.com/company/164748> <http://twitter.com/liveperson>
<http://www.facebook.com/LivePersonInc> We Create Meaningful Connections
<https://liveperson.docsend.com/view/8iiswfp>


On Tue, Aug 29, 2017 at 6:10 PM, I PVP <i...@hotmail.com> wrote:

> Hi Shalom,
>
> If that helps..
> From the documentation : "You should prepare only once, and cache the
> PreparedStatement in your application (it is thread-safe). If you call
> prepare multiple times with the same query string, the driver will log a
> warning”
>
>
> On August 29, 2017 at 12:04:34 PM, Shalom Sagges (shal...@liveperson.com)
> wrote:
>
> That's a good to know post.
>
> Thanks for the info Nicolas!
>
>
> Shalom Sagges
> DBA
> T: +972-74-700-4035 <+972%2074-700-4035>
> <http://www.linkedin.com/company/164748> <http://twitter.com/liveperson>
> <http://www.facebook.com/LivePersonInc> We Create Meaningful Connections
> <https://liveperson.docsend.com/view/8iiswfp>
>
>
> On Tue, Aug 29, 2017 at 3:04 PM, Nicolas Guyomar <
> nicolas.guyo...@gmail.com> wrote:
>
>> I would suggest to read this post by the last pickle:
>> http://thelastpickle.com/blog/2016/09/15/Null-bindin
>> gs-on-prepared-statements-and-undesired-tombstone-creation.html  and
>> make sure you are not concerned by the mentioned behavior, because some
>> people still choose to use C* v2.X because of some bugs in v3 branch
>>
>> Have fun !
>>
>> On 29 August 2017 at 13:43, Shalom Sagges <shal...@liveperson.com> wrote:
>>
>>> Sounds great then.
>>>
>>> Thanks a lot guys! 
>>>
>>>
>>> Shalom Sagges
>>> DBA
>>> <http://www.linkedin.com/company/164748> <http://twitter.com/liveperson>
>>> <http://www.facebook.com/LivePersonInc> We Create Meaningful Connections
>>>
>>>
>>>
>>> On Tue, Aug 29, 2017 at 2:41 PM, Nicolas Guyomar <
>>> nicolas.guyo...@gmail.com> wrote:
>>>
>>>> Hi Shalom,
>>>>
>>>> AFAIK, you are completely safe with prepared statement, there are no
>>>> caveats using them, and you will have better performance.
>>>>
>>>> Make sure to only prepare them once ;)
>>>>
>>>> On 29 August 2017 at 13:41, Matija Gobec <matija0...@gmail.com> wrote:
>>>>
>>>>> I don't see any disadvantages or warning signs. You will see a
>>>>> performance increase on moderate request rate frequency.
>>>>>
>>>>> On Tue, Aug 29, 2017 at 1:28 PM, Shalom Sagges <shal...@liveperson.com
>>>>> > wrote:
>>>>>
>>>>>> Hi Matija,
>>>>>>
>>>>>> I just wish to know if there are any disadvantages when using
>>>>>> prepared statement or any warning signs I should look for. Queries will 
>>>>>> run
>>>>>> multiple times so it fits the use case.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>>
>>>>>> Shalom Sagges
>>>>>> DBA
>>>>>> <http://www.linkedin.com/company/164748>
>>>>>> <http://twitter.com/liveperson>
>>>>>> <http://www.facebook.com/LivePersonInc> We Create Meaningful
>>>>>> Connections
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 29, 2017 at 2:18 PM, Matija Gobec <matija0...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Do you have any concrete questions re prepared statements?
>>>>>>>
>>>>>>> They are faster to execute since the statement is already parsed and
>>>>>>> in C* and you just pass the parameters. No additional statement 
>>>>>>> processing
>>>>>>> is needed.
>>>>>>>
>>>>>>> Matija
>>>>>>>
>>>>>>> On Tue, Aug 29, 2017 at 12:33 PM, Shalom Sagges <
>>>>>>> shal...@liveperson.com> wrote:
>>>>>>>
>>>>>>>> Insights, anyone?
>>>>>>>>
>>>>>>>>
>>>>>>>> Shalom Sagges
>>>>>>>> DBA
>>>>>>>> <http://www.linkedin.com/company/16474

Re: Working With Prepared Statements

[Shalom Sagges]

That's a good to know post.

Thanks for the info Nicolas!


Shalom Sagges
DBA
T: +972-74-700-4035
<http://www.linkedin.com/company/164748> <http://twitter.com/liveperson>
<http://www.facebook.com/LivePersonInc> We Create Meaningful Connections
<https://liveperson.docsend.com/view/8iiswfp>


On Tue, Aug 29, 2017 at 3:04 PM, Nicolas Guyomar <nicolas.guyo...@gmail.com>
wrote:

> I would suggest to read this post by the last pickle:
> http://thelastpickle.com/blog/2016/09/15/Null-bindings-on-prepared-
> statements-and-undesired-tombstone-creation.html  and make sure you are
> not concerned by the mentioned behavior, because some people still choose
> to use C* v2.X because of some bugs in v3 branch
>
> Have fun !
>
> On 29 August 2017 at 13:43, Shalom Sagges <shal...@liveperson.com> wrote:
>
>> Sounds great then.
>>
>> Thanks a lot guys! 
>>
>>
>> Shalom Sagges
>> DBA
>> <http://www.linkedin.com/company/164748> <http://twitter.com/liveperson>
>> <http://www.facebook.com/LivePersonInc> We Create Meaningful Connections
>>
>>
>>
>> On Tue, Aug 29, 2017 at 2:41 PM, Nicolas Guyomar <
>> nicolas.guyo...@gmail.com> wrote:
>>
>>> Hi Shalom,
>>>
>>> AFAIK, you are completely safe with prepared statement, there are no
>>> caveats using them, and you will have better performance.
>>>
>>> Make sure to only prepare them once ;)
>>>
>>> On 29 August 2017 at 13:41, Matija Gobec <matija0...@gmail.com> wrote:
>>>
>>>> I don't see any disadvantages or warning signs. You will see a
>>>> performance increase on moderate request rate frequency.
>>>>
>>>> On Tue, Aug 29, 2017 at 1:28 PM, Shalom Sagges <shal...@liveperson.com>
>>>> wrote:
>>>>
>>>>> Hi Matija,
>>>>>
>>>>> I just wish to know if there are any disadvantages when using prepared
>>>>> statement or any warning signs I should look for. Queries will run 
>>>>> multiple
>>>>> times so it fits the use case.
>>>>>
>>>>> Thanks!
>>>>>
>>>>>
>>>>> Shalom Sagges
>>>>> DBA
>>>>> <http://www.linkedin.com/company/164748>
>>>>> <http://twitter.com/liveperson>
>>>>> <http://www.facebook.com/LivePersonInc> We Create Meaningful
>>>>> Connections
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 29, 2017 at 2:18 PM, Matija Gobec <matija0...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Do you have any concrete questions re prepared statements?
>>>>>>
>>>>>> They are faster to execute since the statement is already parsed and
>>>>>> in C* and you just pass the parameters. No additional statement 
>>>>>> processing
>>>>>> is needed.
>>>>>>
>>>>>> Matija
>>>>>>
>>>>>> On Tue, Aug 29, 2017 at 12:33 PM, Shalom Sagges <
>>>>>> shal...@liveperson.com> wrote:
>>>>>>
>>>>>>> Insights, anyone?
>>>>>>>
>>>>>>>
>>>>>>> Shalom Sagges
>>>>>>> DBA
>>>>>>> <http://www.linkedin.com/company/164748>
>>>>>>> <http://twitter.com/liveperson>
>>>>>>> <http://www.facebook.com/LivePersonInc> We Create Meaningful
>>>>>>> Connections
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Aug 28, 2017 at 10:43 AM, Shalom Sagges <
>>>>>>> shal...@liveperson.com> wrote:
>>>>>>>
>>>>>>>> Hi Everyone,
>>>>>>>>
>>>>>>>> I want to start working with Prepared Statements.
>>>>>>>>
>>>>>>>> I've read https://docs.datastax.com/en/developer/java-driver/3.1/
>>>>>>>> manual/statements/prepared/ and just wanted to know if there are
>>>>>>>> any other considerations I need to take into account when deciding to 
>>>>>>>> use
>>>>>>>> Prepared Statements.
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>>
&

Re: Working With Prepared Statements

[Shalom Sagges]

Sounds great then.

Thanks a lot guys! 


Shalom Sagges
DBA
<http://www.linkedin.com/company/164748> <http://twitter.com/liveperson>
<http://www.facebook.com/LivePersonInc> We Create Meaningful Connections



On Tue, Aug 29, 2017 at 2:41 PM, Nicolas Guyomar <nicolas.guyo...@gmail.com>
wrote:

> Hi Shalom,
>
> AFAIK, you are completely safe with prepared statement, there are no
> caveats using them, and you will have better performance.
>
> Make sure to only prepare them once ;)
>
> On 29 August 2017 at 13:41, Matija Gobec <matija0...@gmail.com> wrote:
>
>> I don't see any disadvantages or warning signs. You will see a
>> performance increase on moderate request rate frequency.
>>
>> On Tue, Aug 29, 2017 at 1:28 PM, Shalom Sagges <shal...@liveperson.com>
>> wrote:
>>
>>> Hi Matija,
>>>
>>> I just wish to know if there are any disadvantages when using prepared
>>> statement or any warning signs I should look for. Queries will run multiple
>>> times so it fits the use case.
>>>
>>> Thanks!
>>>
>>>
>>> Shalom Sagges
>>> DBA
>>> <http://www.linkedin.com/company/164748> <http://twitter.com/liveperson>
>>> <http://www.facebook.com/LivePersonInc> We Create Meaningful Connections
>>>
>>>
>>>
>>> On Tue, Aug 29, 2017 at 2:18 PM, Matija Gobec <matija0...@gmail.com>
>>> wrote:
>>>
>>>> Do you have any concrete questions re prepared statements?
>>>>
>>>> They are faster to execute since the statement is already parsed and in
>>>> C* and you just pass the parameters. No additional statement processing is
>>>> needed.
>>>>
>>>> Matija
>>>>
>>>> On Tue, Aug 29, 2017 at 12:33 PM, Shalom Sagges <shal...@liveperson.com
>>>> > wrote:
>>>>
>>>>> Insights, anyone?
>>>>>
>>>>>
>>>>> Shalom Sagges
>>>>> DBA
>>>>> <http://www.linkedin.com/company/164748>
>>>>> <http://twitter.com/liveperson>
>>>>> <http://www.facebook.com/LivePersonInc> We Create Meaningful
>>>>> Connections
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Aug 28, 2017 at 10:43 AM, Shalom Sagges <
>>>>> shal...@liveperson.com> wrote:
>>>>>
>>>>>> Hi Everyone,
>>>>>>
>>>>>> I want to start working with Prepared Statements.
>>>>>>
>>>>>> I've read https://docs.datastax.com/en/developer/java-driver/3.1/
>>>>>> manual/statements/prepared/ and just wanted to know if there are any
>>>>>> other considerations I need to take into account when deciding to use
>>>>>> Prepared Statements.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>>
>>>>>> Shalom Sagges
>>>>>> DBA
>>>>>> <http://www.linkedin.com/company/164748>
>>>>>> <http://twitter.com/liveperson>
>>>>>> <http://www.facebook.com/LivePersonInc> We Create Meaningful
>>>>>> Connections
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> This message may contain confidential and/or privileged information.
>>>>> If you are not the addressee or authorized to receive this on behalf
>>>>> of the addressee you must not use, copy, disclose or take action based on
>>>>> this message or any information herein.
>>>>> If you have received this message in error, please advise the sender
>>>>> immediately by reply email and delete this message. Thank you.
>>>>>
>>>>
>>>>
>>>
>>> This message may contain confidential and/or privileged information.
>>> If you are not the addressee or authorized to receive this on behalf of
>>> the addressee you must not use, copy, disclose or take action based on this
>>> message or any information herein.
>>> If you have received this message in error, please advise the sender
>>> immediately by reply email and delete this message. Thank you.
>>>
>>
>>
>

-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.

Re: Working With Prepared Statements

[Shalom Sagges]

Hi Matija,

I just wish to know if there are any disadvantages when using prepared
statement or any warning signs I should look for. Queries will run multiple
times so it fits the use case.

Thanks!


Shalom Sagges
DBA
<http://www.linkedin.com/company/164748> <http://twitter.com/liveperson>
<http://www.facebook.com/LivePersonInc> We Create Meaningful Connections



On Tue, Aug 29, 2017 at 2:18 PM, Matija Gobec <matija0...@gmail.com> wrote:

> Do you have any concrete questions re prepared statements?
>
> They are faster to execute since the statement is already parsed and in C*
> and you just pass the parameters. No additional statement processing is
> needed.
>
> Matija
>
> On Tue, Aug 29, 2017 at 12:33 PM, Shalom Sagges <shal...@liveperson.com>
> wrote:
>
>> Insights, anyone?
>>
>>
>> Shalom Sagges
>> DBA
>> <http://www.linkedin.com/company/164748> <http://twitter.com/liveperson>
>> <http://www.facebook.com/LivePersonInc> We Create Meaningful Connections
>>
>>
>>
>> On Mon, Aug 28, 2017 at 10:43 AM, Shalom Sagges <shal...@liveperson.com>
>> wrote:
>>
>>> Hi Everyone,
>>>
>>> I want to start working with Prepared Statements.
>>>
>>> I've read https://docs.datastax.com/en/developer/java-driver/3.1/
>>> manual/statements/prepared/ and just wanted to know if there are any
>>> other considerations I need to take into account when deciding to use
>>> Prepared Statements.
>>>
>>> Thanks!
>>>
>>>
>>> Shalom Sagges
>>> DBA
>>> <http://www.linkedin.com/company/164748> <http://twitter.com/liveperson>
>>> <http://www.facebook.com/LivePersonInc> We Create Meaningful Connections
>>>
>>>
>>>
>>
>>
>> This message may contain confidential and/or privileged information.
>> If you are not the addressee or authorized to receive this on behalf of
>> the addressee you must not use, copy, disclose or take action based on this
>> message or any information herein.
>> If you have received this message in error, please advise the sender
>> immediately by reply email and delete this message. Thank you.
>>
>
>

-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.

Re: Working With Prepared Statements

[Shalom Sagges]

Insights, anyone?


Shalom Sagges
DBA
<http://www.linkedin.com/company/164748> <http://twitter.com/liveperson>
<http://www.facebook.com/LivePersonInc> We Create Meaningful Connections



On Mon, Aug 28, 2017 at 10:43 AM, Shalom Sagges <shal...@liveperson.com>
wrote:

> Hi Everyone,
>
> I want to start working with Prepared Statements.
>
> I've read https://docs.datastax.com/en/developer/java-driver/
> 3.1/manual/statements/prepared/ and just wanted to know if there are any
> other considerations I need to take into account when deciding to use
> Prepared Statements.
>
> Thanks!
>
>
> Shalom Sagges
> DBA
> <http://www.linkedin.com/company/164748> <http://twitter.com/liveperson>
> <http://www.facebook.com/LivePersonInc> We Create Meaningful Connections
>
>
>

-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.

Working With Prepared Statements

[Shalom Sagges]

Hi Everyone,

I want to start working with Prepared Statements.

I've read
https://docs.datastax.com/en/developer/java-driver/3.1/manual/statements/prepared/
and
just wanted to know if there are any other considerations I need to take
into account when deciding to use Prepared Statements.

Thanks!


Shalom Sagges
DBA
<http://www.linkedin.com/company/164748> <http://twitter.com/liveperson>
<http://www.facebook.com/LivePersonInc> We Create Meaningful Connections

-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.

Re: Adding a New Node With The Same IP of an Old Node

[Shalom Sagges]

Thanks Nitan!

Eventually it was a firewall issue related to the Centos7 node.

Once fixed, the rolling restart resolved the issue completely.

Thanks again!



Shalom Sagges
DBA
T: +972-74-700-4035
<http://www.linkedin.com/company/164748> <http://twitter.com/liveperson>
<http://www.facebook.com/LivePersonInc> We Create Meaningful Connections
<https://liveperson.docsend.com/view/8iiswfp>


On Tue, Jul 4, 2017 at 7:52 PM, Nitan Kainth <ni...@bamlabs.com> wrote:

> Try rolling restart of cluster as solution for schema version mismatch.
>
>
> Sent from my iPhone
>
> On Jul 4, 2017, at 8:31 AM, Shalom Sagges <shal...@liveperson.com> wrote:
>
> Hi Experts,
>
> My plan is to upgrade the C* nodes' OS from Centos6 to Centos7.
> Since an upgrade wasn't recommended I needed to install new machines with
> Centos7 and join them to the cluster.
> I didn't want to decommission/bootstrap dozens of nodes, so I decided to
> do the following:
>
>- Create a new machine
>- Copy all data, schema data and commit logs using rsync from the old
>node to the new one
>- Stop Cassandra on the old node, copy the delta and shut the node down
>- Change the hostname and IP address of the new node to the hostname
>and IP of the old node.
>- Start Cassandra on the new node.
>
> My thought was that the "new" node will join the cluster as the "old" node
> since it has all of its data and metadata, however, after Cassandra started
> up, it saw all its peers in DN state. The same goes for the other nodes,
> that saw the new node is DN state.
>
> I didn't find any errors or warnings in the logs, but I did see that the 
> *schema
> version* on the new node was different from the others( I'm assuming
> that's the issue here?)
>
>
> *Functioning node*
> nodetool describecluster
> Cluster Information:
> Name: MyCluster
> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Schema versions:
> 503bf151-c59c-35a0-8350-ca73e82098f5: [x.x.x.32,
> x.x.x.126, x.x.x.35, x.x.x.1, x.x.x.28, x.x.x.15, x.x.x.252, x.x.x.253,
> x.x.x.31]
>
> UNREACHABLE: [x.x.x.2]
>
>
> *New node*
> nodetool describecluster
> Cluster Information:
> Name: MyCluster
> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Schema versions:
> dab67163-65d4-3895-82e6-ffc07bf5c17a: [x.x.x.2]
>
> UNREACHABLE: [x.x.x.31, x.x.x.1, x.x.x.28, x.x.x.252,
> x.x.x.253, x.x.x.15, x.x.x.126, x.x.x.35, x.x.x.32]
>
>
> I'd really REALLY appreciate some guidance. Did I do something wrong? Is
> there a way to fix this?
>
>
> Thanks a lot!
>
> Shalom Sagges
> DBA
> <http://www.linkedin.com/company/164748> <http://twitter.com/liveperson>
> <http://www.facebook.com/LivePersonInc> We Create Meaningful Connections
>
>
>
> This message may contain confidential and/or privileged information.
> If you are not the addressee or authorized to receive this on behalf of
> the addressee you must not use, copy, disclose or take action based on this
> message or any information herein.
> If you have received this message in error, please advise the sender
> immediately by reply email and delete this message. Thank you.
>
>

-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.

Adding a New Node With The Same IP of an Old Node

[Shalom Sagges]

Hi Experts,

My plan is to upgrade the C* nodes' OS from Centos6 to Centos7.
Since an upgrade wasn't recommended I needed to install new machines with
Centos7 and join them to the cluster.
I didn't want to decommission/bootstrap dozens of nodes, so I decided to do
the following:

   - Create a new machine
   - Copy all data, schema data and commit logs using rsync from the old
   node to the new one
   - Stop Cassandra on the old node, copy the delta and shut the node down
   - Change the hostname and IP address of the new node to the hostname and
   IP of the old node.
   - Start Cassandra on the new node.

My thought was that the "new" node will join the cluster as the "old" node
since it has all of its data and metadata, however, after Cassandra started
up, it saw all its peers in DN state. The same goes for the other nodes,
that saw the new node is DN state.

I didn't find any errors or warnings in the logs, but I did see that
the *schema
version* on the new node was different from the others( I'm assuming that's
the issue here?)


*Functioning node*
nodetool describecluster
Cluster Information:
Name: MyCluster
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
503bf151-c59c-35a0-8350-ca73e82098f5: [x.x.x.32, x.x.x.126,
x.x.x.35, x.x.x.1, x.x.x.28, x.x.x.15, x.x.x.252, x.x.x.253, x.x.x.31]

UNREACHABLE: [x.x.x.2]


*New node*
nodetool describecluster
Cluster Information:
Name: MyCluster
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
dab67163-65d4-3895-82e6-ffc07bf5c17a: [x.x.x.2]

UNREACHABLE: [x.x.x.31, x.x.x.1, x.x.x.28, x.x.x.252,
x.x.x.253, x.x.x.15, x.x.x.126, x.x.x.35, x.x.x.32]


I'd really REALLY appreciate some guidance. Did I do something wrong? Is
there a way to fix this?


Thanks a lot!

Shalom Sagges
DBA
<http://www.linkedin.com/company/164748> <http://twitter.com/liveperson>
<http://www.facebook.com/LivePersonInc> We Create Meaningful Connections

-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.

Re: Reaper v0.6.1 released

[Shalom Sagges]

That's awesome!! Thanks for contributing! 


Shalom Sagges
DBA
T: +972-74-700-4035
<http://www.linkedin.com/company/164748> <http://twitter.com/liveperson>
<http://www.facebook.com/LivePersonInc> We Create Meaningful Connections



On Thu, Jun 15, 2017 at 2:32 AM, Jonathan Haddad <j...@jonhaddad.com> wrote:

> Hey folks!
>
> I'm proud to announce the 0.6.1 release of the Reaper project, the open
> source repair management tool for Apache Cassandra.
>
> This release improves the Cassandra backend significantly, making it a
> first class citizen for storing repair schedules and managing repair
> progress.  It's no longer necessary to manage a PostgreSQL DB in addition
> to your Cassandra DB.
>
> We've been very active since we forked the original Spotify repo.  Since
> this time we've added:
>
> * A native Cassandra backend
> * Support for versions > 2.0
> * Merged in the WebUI, maintained by Stefan Podkowinski (
> https://github.com/spodkowinski/cassandra-reaper-ui)
> * Support for incremental repair (probably best to avoid till Cassandra
> 4.0, see CASSANDRA-9143)
>
> We're excited to continue making improvements past the original intent of
> the project.  With the lack of Cassandra 3.0 support in OpsCenter, there's
> a gap that needs to be filled for tools that help with managing a cluster.
> Alex Dejanovski showed me a prototype he recently put together for a really
> nice view into cluster health.  We're also looking to add in support common
> cluster operations like snapshots, upgradesstables, cleanup, and setting
> options at runtime.
>
> Grab it here: https://github.com/thelastpickle/cassandra-reaper
>
> Feedback / bug reports / ideas are very much appreciated.
>
> We have a dedicated, low traffic ML here: https://groups.google.com/
> forum/#!forum/tlp-apache-cassandra-reaper-users
>
> Jon Haddad
> Principal Consultant, The Last Pickle
> http://thelastpickle.com/
>

-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.