Fwd: Re: Cassandra uneven data repartition

2023-01-06 Thread onmstester onmstester via user
Isn't there a very big (>40GB) sstable in /volumes/cassandra/data/data1? If there is you could split it or change your data model to prevent such sstables. Sent using https://www.zoho.com/mail/ Forwarded message From: Loïc CHANEL via user To: Date: Fri, 06

RE: Best compaction strategy for rarely used data

2023-01-06 Thread onmstester onmstester via user
Another solution: distribute data in more tables, for example you could create multiple tables based on value or hash_bucket of one of the columns, by doing this current data volume  and compaction overhead would be divided to the number of underlying tables. Although there is a limitation for

Re: Fwd: Re: Problem on setup Cassandra v4.0.1 cluster

2022-10-08 Thread onmstester onmstester via user
timestamp) are common due to manual config or no one thought such problems could prevent a Cassandra node from joining the cluster! Sent using https://www.zoho.com/mail/ On Mon, 31 Jan 2022 16:35:50 +0330 onmstester onmstester wrote --- Once again it was related to hostname

Re: Using zstd compression on Cassandra 3.x

2022-09-13 Thread onmstester onmstester via user
I patched this on 3.11.2 easily: 1. build jar file from src and put in cassandra/lib directory 2. restart cassandra service 3. alter table using compression zstd and rebuild sstables But it was in a time when 4.0 was not available yet and after that i upgraded to 4.0 immidiately. Sent

Re: Compaction task priority

2022-09-06 Thread onmstester onmstester via user
PM Jim Shaw <mailto:jxys...@gmail.com> wrote: if capacity allowed,  increase compaction_throughput_mb_per_sec as 1st tuning,  and if still behind, increase concurrent_compactors as 2nd tuning. Regards, Jim On Fri, Sep 2, 2022 at 3:05 AM onmstester onmstester via user <ma

Re: Compaction task priority

2022-09-02 Thread onmstester onmstester via user
onmstester onmstester via user wrote --- I was there too! and found nothing to work around it except stopping big/unnecessary compactions manually (using nodetool stop) whenever they appears by some shell scrips (using crontab) Sent using https://www.zoho.com/mail/ On Fri, 02 Sep 2022

Re: Compaction task priority

2022-09-02 Thread onmstester onmstester via user
I was there too! and found nothing to work around it except stopping big/unnecessary compactions manually (using nodetool stop) whenever they appears by some shell scrips (using crontab) Sent using https://www.zoho.com/mail/ On Fri, 02 Sep 2022 10:59:22 +0430 Gil Ganz wrote ---

Re: slow compactions

2022-03-06 Thread onmstester onmstester
Forgot to mention that i'm using default STCS for all tables On Sun, 06 Mar 2022 12:29:52 +0330 onmstester onmstester wrote Hi, Sometimes compactions getting so slow (a few KBs per second for each compaction) on a few nodes which would be fixed temporarily by restarting

slow compactions

2022-03-06 Thread onmstester onmstester
Hi, Sometimes compactions getting so slow (a few KBs per second for each compaction) on a few nodes which would be fixed temporarily by restarting  restarting cassandra (although would coming back a few hours later). Copied sstables related to slow compactions to a isolated/single node

Re: TLS/SSL overhead

2022-02-07 Thread onmstester onmstester
persistent connections therefore the visible impact is on connection establishment time (TLS handshake is expensive). Encryption will make thundering herd problems worse. You should watch out for those two issues. Dinesh On Feb 5, 2022, at 3:53 AM, onmstester onmstester <mailto:onmstes..

TLS/SSL overhead

2022-02-05 Thread onmstester onmstester
Hi, Anyone measured impact of wire encryption using TLS (client_encryption/server_encryption) on cluster latency/throughput? It may be dependent on Hardware or even data model but I already did some sort of measurements and got to 2% for client encryption and 3-5% for client + server

Fwd: Re: Cassandra internal bottleneck

2022-02-05 Thread onmstester onmstester
Thanks, I've got only one client, 10 threads and 1K async writes, This single client was able to send 110K insert/seconds to single node cluster but its only sending 90K insert/seconds to the cluster with 2 nodes(client CPU/network usage is less than 20%) Sent using

Cassandra internal bottleneck

2022-02-05 Thread onmstester onmstester
Hi, I'm trying to evaluate performance of Apache Cassandra V4.0.1 for write-only workloads using on-premise physical servers. On a single node cluster, doing some optimizations i was able to make CPU of node >90%, throughput is high enough and CPU is the bottleneck as i expected. Then

Fwd: Re: Problem on setup Cassandra v4.0.1 cluster

2022-01-31 Thread onmstester onmstester
Once again it was related to hostname configuration (I remember had problem with this multiple times before even on different applications), this time the root cause was a typo in one of multiple config files for hostname (different name on /etc/hostname with /etc/hosts)! I fixed that and now

Problem on setup Cassandra v4.0.1 cluster

2022-01-31 Thread onmstester onmstester
Hi, I'm trying to setup a Cluster of  apache Cassandra version 4.0.1 with 2 nodes: 1. on node1 (192.168.1.1), extracted tar.gz and config these on yml: - seeds: "192.168.1.1" listen_address: 192.168.1.1 rpc_address: 192.168.1.1 2. started node1 and a few seconds later it is UN

Re: gc throughput

2021-11-17 Thread onmstester onmstester
to GC, regardless what is the GC metric you are looking at saying, you will need to address the issue and that probably will involve some GC tunings. On 15/11/2021 06:00, onmstester onmstester wrote: Hi, We are using Apache Cassandra 3.11.2 with its default gc

Re: Separating storage and processing

2021-11-15 Thread onmstester onmstester
Thank You Sent using https://www.zoho.com/mail/ On Tue, 16 Nov 2021 10:00:19 +0330 wrote > I can, but i thought with 5TB per node already violated best practices (1-2 >TB per node) and won't be a good idea to 2X or 3X that? The main downside of larger disks is that it takes

Re: Separating storage and processing

2021-11-15 Thread onmstester onmstester
I can, but i thought with 5TB per node already violated best practices (1-2 TB per node) and won't be a good idea to 2X or 3X that? Sent using https://www.zoho.com/mail/ On Mon, 15 Nov 2021 20:55:53 +0330 wrote It sounds like you can downsize your cluster but increase your

gc throughput

2021-11-14 Thread onmstester onmstester
Hi, We are using Apache Cassandra 3.11.2 with its default gc configuration (CMS and ...) on a 16GB heap, i inspected gc logs using gcviewer and it reported 92% of throughput, is that means not necessary to do any further tuning for gc? and everything is ok with gc of Cassandra? Sent

Separating storage and processing

2021-11-14 Thread onmstester onmstester
Hi, In our Cassandra cluster, because of big rows in input data/data model with TTL of several months, we ended up using almost 80% of storage (5TB per node), but having less than 20% of CPU usage which almost all of it would be writing rows to memtables and compacting sstables, so a lot of

Re: New Servers - Cassandra 4

2021-08-11 Thread onmstester onmstester
Hi, What about this type of blades, which gives you about 12 (commodity) servers  in 3U: https://www.supermicro.com/en/products/microcloud Sent using https://www.zoho.com/mail/ On Tue, 03 Aug 2021 02:01:13 +0430 Joe Obernberger wrote Thank you Max.  That is a solid choice. 

Re: Question about the num_tokens

2021-04-28 Thread onmstester onmstester
Some posts/papers discusses this in more detail. for example the one from thelastpickle: https://thelastpickle.com/blog/2019/02/21/set-up-a-cluster-with-even-token-distribution.html Which says: Using statistical computation, the point where all clusters of any size always had a good token

Re: What Happened To Alternate Storage And Rocksandra?

2021-03-12 Thread onmstester onmstester
Beside the enhancements at storage layer, i think there are couple of good ideas in Rocksdb that could be used in Cassandra, like the one with disabling sort at memtable-insert part (write data fast like commitlig) and only sort the data when flushing/creating sst files. Sent using

Fwd: Re: using zstd cause high memtable switch count

2021-02-28 Thread onmstester onmstester
, Feb 28, 2021 at 9:22 PM onmstester onmstester <mailto:onmstes...@zoho.com.invalid> wrote: Hi, I'm using 3.11.2, just add the patch for zstd and changed table compression from default (LZ4) to zstd with level 1 and chunk 64kb, everything is fine (disk usage decreased by 40% and CPU

using zstd cause high memtable switch count

2021-02-28 Thread onmstester onmstester
Hi, I'm using 3.11.2, just add the patch for zstd and changed table compression from default (LZ4) to zstd with level 1 and chunk 64kb, everything is fine (disk usage decreased by 40% and CPU usage is almost the same as before), only the memtable switch count was changed dramatically; with

number of racks in a deployment with VMs

2021-02-14 Thread onmstester onmstester
Hi, In a article by thelastpickle [1], i noticed: The key here is to configure the cluster so that for a given datacenter the number of racks is the same as the replication factor. When using virtual machines as Cassandra nodes we have to set up the cluster in a way that number of racks

Fwd: Re: local read from coordinator

2020-11-14 Thread onmstester onmstester
https://www.zoho.com/mail/ Forwarded message From: onmstester onmstester To: "user" Date: Sat, 14 Nov 2020 08:24:14 +0330 Subject: Re: local read from coordinator Forwarded message Thank you Jeff, I disabled dynamic_sni

Re: local read from coordinator

2020-11-13 Thread onmstester onmstester
is chosen in practice) On Nov 11, 2020, at 3:46 AM, Alex Ott <mailto:alex...@gmail.com> wrote: if you force routing key, then the replica that owns the data will be selected as coordinator On Wed, Nov 11, 2020 at 12:35 PM onmstester onmstester <mailto:onmstes...@zoho.com.inval

Fwd: Re: local read from coordinator

2020-11-11 Thread onmstester onmstester
Thanx, But i'm OK with coordinator part, actually i was looking for kind of read CL to force to read from the coordinator only with no other connections to other nodes! Sent using https://www.zoho.com/mail/ Forwarded message From: Alex Ott To: "user" Date:

local read from coordinator

2020-11-10 Thread onmstester onmstester
Hi, I'm going to read all the data in the cluster as fast as possible, i'm aware that spark could do such things out of the box but just wanted to do it at low level to see how fast it could be. So: 1. retrieved partition keys on each node using nodetool ring token ranges and getting distinct

OOM on ccm with large cluster on a single node

2020-10-27 Thread onmstester onmstester
Hi, I'm using ccm to create a cluster of 80 nodes on a physical server with 10 cores and 64GB of ram, but always the 43th node could not start with error: java.lang.OutOfMemoryError: unable to create new native thread apache cassandra 3.11.2 cassandra xmx600M 30GB of memory is still

reducing RF wen using token allocation algorithm

2020-10-26 Thread onmstester onmstester
Hi, I've set up cluster with: 3.11.2 30 nodes RF=3,single dc, NetworkStrategy Now i'm going to reduce rf to 2, but i've setup cluster with vnode=16 and allocation algorithm(allocate_tokens_for_keyspace) for the main keyspace (which i'm reducing its RF), so is the procedure still be 1.

Re: dropped mutations cross node

2020-10-05 Thread onmstester onmstester
Thanks, I've done a lot of conf changes to fix  the problem but nothing worked (last one was disabling hints) and after a few days problem gone!! The source of droppedCrossNode was changing every half an hour and it was not always the new nodes No difference between new nodes and old ones in

dropped mutations cross node

2020-09-21 Thread onmstester onmstester
Hi,  I've extended a cluster by 10% and after that each hour, on some of the nodes (which changes randomly each time),  "dropped mutations cross node" appears on logs (each time 1 or 2 drops and some times some thousands with cross node latency from 3000ms to 9ms or 90seconds!) and insert

Re: Node is UNREACHABLE after decommission

2020-09-19 Thread onmstester onmstester
Another workaround that i used for UNREACHABLE nodes problem, is to restart the whole cluster and it would be fixed, but i don't know if it cause any problem or not Sent using https://www.zoho.com/mail/ On Fri, 18 Sep 2020 01:19:35 +0430 Paulo Motta wrote Oh, if you're adding

Re: data modeling qu: use a Map datatype, or just simple rows... ?

2020-09-19 Thread onmstester onmstester
I used Cassandra Set (no experience with map ), and one thing for sure is that with Cassandra collections you are limited to a few thousands entry per row (less than 10K for better performance) Sent using https://www.zoho.com/mail/ On Fri, 18 Sep 2020 20:33:21 +0430 Attila Wind

Re: Re: streaming stuck on joining a node with TBs of data

2020-08-05 Thread onmstester onmstester
by sending bigger sstables at sending side or by merging sstables in memtable at receiving side) (Just fixed a wrong word in my previous question) On Wed, 05 Aug 2020 10:02:51 +0430 onmstester onmstester <mailto:onmstes...@zoho.com.INVALID> wrote OK. Thanks I'm using STCS. Anyway

Re: Re: streaming stuck on joining a node with TBs of data

2020-08-04 Thread onmstester onmstester
LCS, change sstable size from 160M to something higher), but there's no magic to join / compact those data files on the sending side before sending. On Mon, Aug 3, 2020 at 4:15 AM onmstester onmstester <mailto:onmstes...@zoho.com.invalid> wrote: IMHO (reading system.log) each streamed-i

Fwd: Re: streaming stuck on joining a node with TBs of data

2020-08-03 Thread onmstester onmstester
configuration in cassandra to force streamed-in to pass memtable-sstable cycle, to have bigger sstables at first place? Sent using https://www.zoho.com/mail/ Forwarded message From: onmstester onmstester To: "user" Date: Sun, 02 Aug 2020 08:35:30 +04

Re: streaming stuck on joining a node with TBs of data

2020-08-01 Thread onmstester onmstester
own risk). On Jul 31, 2020, at 11:46 PM, onmstester onmstester <mailto:onmstes...@zoho.com.invalid> wrote: No Secondary index, No SASI, No materialized view Sent using https://www.zoho.com/mail/ On Sat, 01 Aug 2020 11:02:54 +0430 Jeff Jirsa <mailto:jji...@

Re: streaming stuck on joining a node with TBs of data

2020-08-01 Thread onmstester onmstester
No Secondary index, No SASI, No materialized view Sent using https://www.zoho.com/mail/ On Sat, 01 Aug 2020 11:02:54 +0430 Jeff Jirsa wrote Are there secondary indices involved?  On Jul 31, 2020, at 10:51 PM, onmstester onmstester <mailto:onmstes...@zoho.com.inva

streaming stuck on joining a node with TBs of data

2020-07-31 Thread onmstester onmstester
Hi, I'm going to join multiple new nodes to already existed and running cluster. Each node should stream in >2TB of data, and it took a few days (with 500Mb streaming) to almost get finished. But it stuck on streaming-in from one final node, but i can not see any bottleneck on any side

Re: Multi DCs vs Single DC performance

2020-07-28 Thread onmstester onmstester
 in a way I havent personally figured out yet (maybe if you had a very high replica count per DC, then using forwarding and EACH_QUORUM may get fun, but you'd be better off dropping the replica count than coming up with stuff like this). On Tue, Jul 28, 2020 at 8:27 PM onmstester onmstester

Multi DCs vs Single DC performance

2020-07-28 Thread onmstester onmstester
Hi, Logically, i do not need to use multiple DCs(cluster is not geographically separated), but i wonder if splitting the cluster to two half (two separate dc) would decrease overhead of node ack/communication and result in better (write) performance? Sent using https://www.zoho.com/mail/

Re: design principle to manage roll back

2020-07-14 Thread onmstester onmstester
Hi, I think that Cassandra alone is not suitable for your use case. You can use a mix of Distributed/NoSQL (to storing single records of whatever makes your input the big data) & Relational/Single Database (for transactional non-big data part) Sent using https://www.zoho.com/mail/

Relation between num_tokens and cluster extend limitations

2020-07-13 Thread onmstester onmstester
Hi, I'm using allocate_tokens_for_keyspace and num_tokens=32 and i wan't to extend the size of some clusters. I read in articles that for num_tokens=4, one should add more 25% of cluster size for the cluster to become balanced again. 1. For example, with num_tokens=4 and already have 16

Re: Running Large Clusters in Production

2020-07-10 Thread onmstester onmstester
Yes, you should handle the routing logic at app level I wish there was another level of sharding (above dc, rack) as cluster to distribute data on multiple cluster! but i don't think there is any other database that does such a thing for you. Another problem with big cluster is for huge amount

Cassandra crashes when using offheap_objects for memtable_allocation_type

2020-06-02 Thread onmstester onmstester
I just changed these properties to increase flushed file size (decrease number of compactions): memtable_allocation_type from heap_buffers to offheap_objects memtable_offheap_space_in_mb: from default (2048) to 8192 Using default value for other memtable/compaction/commitlog configurations .

Fwd: Re: [Discuss] num_tokens default in Cassandra 4.0

2020-02-03 Thread onmstester onmstester
out node 1 & 4, then ranges B & L would no longer meet CL=quorum;  but you can do that in the top diagram, since there are no ranges shared between node 1 & 4. Hope that helps. - Max On Feb 3, 2020, at 8:39 pm, onmstester onmstester <mailto:onmstes...@zoho.com.INVALI

Fwd: Re: [Discuss] num_tokens default in Cassandra 4.0

2020-02-03 Thread onmstester onmstester
Sorry if its trivial, but i do not understand how num_tokens affects availability, with RF=3, CLW,CLR=quorum, the cluster could tolerate to lost at most one node and all of the tokens assigned to that node would be also assigned to two other nodes no matter what num_tokens is, right? Sent

Re: bug in cluster key push down

2020-01-12 Thread onmstester onmstester
rows, so i suppose that clustering key restrictions been pushed down to storage engine. Thanks Jeff Sent using https://www.zoho.com/mail/ On Mon, 13 Jan 2020 08:38:44 +0330 onmstester onmstester <mailto:onmstes...@zoho.com.INVALID> wrote Done. https://issues.apache

Re: bug in cluster key push down

2020-01-12 Thread onmstester onmstester
  Sent from my iPhone On Jan 12, 2020, at 6:04 AM, onmstester onmstester <mailto:onmstes...@zoho.com.invalid> wrote: Using Apache Cassandra 3.11.2, defined a table like this: create table my_table(                    partition text,            clusterin

bug in cluster key push down

2020-01-12 Thread onmstester onmstester
Using Apache Cassandra 3.11.2, defined a table like this: create table my_table(                    partition text,            clustering1 int,       clustering2 text,       data set,     primary key (partition, clustering1, clustering2)) and

cassandra collection best practices and performance

2020-01-07 Thread onmstester onmstester
Sweet spot for set and list items count (in datastax's documents, the max is 2billions)? Write and read performance of Set vs List vs simple partition row? Thanks in advance

Cluster of small clusters

2019-11-16 Thread onmstester onmstester
Each cassandra node creates 6 seperate threads for incomming and outgoing streams to other nodes in the cluster. So with big clusters for example 100 nodes, it would be more than 600 threads running in each Cassandra app, that would cause performance problems, so better have multiple small

Re: Cassandra.Link Knowledge Base - v. 0.4

2019-07-21 Thread onmstester onmstester
Thank you all! Sent using https://www.zoho.com/mail/ On Sat, 20 Jul 2019 16:13:29 +0430 Rahul Singh wrote Hey Cassandra community , Thanks for all the feedback in the past on my cassandra knowledge base project. Without the feedback cycle it’s not really for the community. 

Re: How to set up a cluster with allocate_tokens_for_keyspace?

2019-05-05 Thread onmstester onmstester
r of token per node and the number of racks. Regards, Anthony On Sat, 4 May 2019 at 19:14, onmstester onmstester <mailto:onmstes...@zoho.com.invalid> wrote: I just read this article by tlp: https://thelastpickle.com/blog/2019/02/21/set-up-a-cluster-with-even-token-dis

Fwd: Re: How to set up a cluster with allocate_tokens_for_keyspace?

2019-05-04 Thread onmstester onmstester
to specify tokens anymore, you can just use allocate_tokens_for_keyspace. On Sat, May 4, 2019 at 2:14 AM onmstester onmstester <mailto:onmstes...@zoho.com.invalid> wrote: > > I just read this article by tlp: > https://thelastpickle.com/blog/2019/02/21/set-up-a-cluster-

How to set up a cluster with allocate_tokens_for_keyspace?

2019-05-04 Thread onmstester onmstester
I just read this article by tlp: https://thelastpickle.com/blog/2019/02/21/set-up-a-cluster-with-even-token-distribution.html   Noticed that: >>We will need to set the tokens for the seed nodes in each rack manually. This >>is to prevent each node from randomly calculating its own token

Re: when the "delete statement" would be deleted?

2019-04-24 Thread onmstester onmstester
Found the answer: it would be deleted after gc_grace Just decreased the gc_grace, run compact, and the "marked_deleted" partitions purged from sstable Sent using https://www.zoho.com/mail/ On Wed, 24 Apr 2019 14:15:33 +0430 onmstester onmstester wrote Just delete

when the "delete statement" would be deleted?

2019-04-24 Thread onmstester onmstester
Just deleted multiple partitions from one of my tables, dumping sstables shows that the data successfully deleted, but the 'marked_deleted' rows for each of partitions still exists on sstable and allocates storage. Is there any way to get rid of these delete statements storage overhead

Re: gc_grace config for time serie database

2019-04-17 Thread onmstester onmstester
I do not use table default ttl (every row has its own TTL) and also no update occurs to the rows. I suppose that (because of immutable nature of everything in cassandra) cassandra would keep only the insertion timestamp + the original ttl and  computes ttl of a row using these two and current

can i delete a sstable with Estimated droppable tombstones > 1, manually?

2019-03-19 Thread onmstester onmstester
Running: SSTablemetadata /THE_KEYSPACE_DIR/mc-1421-big-Data.db result was: Estimated droppable tombstones: 1.2 Having STCS and data disk usage of 80% (do not have enough free space for normal compaction), Is it OK to just: 1. stop Cassandra, 2. delete mc-1421* and then 3. start Cassandra?

Re: removenode force vs assasinate

2019-03-11 Thread onmstester onmstester
The only option to stream decommissioned node's data is to run "nodetool decommission" on the decommissioned node (while cassandra is running on the node) removenode only streams data from node's relpica, so any data that only stored on decommissioned node would be lost. You should monitoring

Re: removenode force vs assasinate

2019-03-11 Thread onmstester onmstester
You should first try with removenode which triggers cluster streaming, if removenode failes or stuck, Assassinate is the last solution. Sent using https://www.zoho.com/mail/ On Mon, 11 Mar 2019 14:27:13 +0330 Ahmed Eljami wrote Hello, Can someone explain me the

forgot to run nodetool cleanup

2019-02-12 Thread onmstester onmstester
Hi, I should have run cleanup after adding a few nodes to my cluster, about 2 months ago, the ttl is 6 month, What happens now? Should i worry about any  catastrophics? Should i run the cleanup now? Thanks in advance Sent using https://www.zoho.com/mail/

Fwd: Question about allocate_tokens_for_keyspace

2019-01-28 Thread onmstester onmstester
You could only have one keyspace for the value of allocate_tokens_for_keyspace   to specify a keyspace from which the algorithm can find the replication to optimize for. So as far as your keyspaces are using similar replication strategies and replication factor you should not worry about this.

slow commitlog sync

2018-12-23 Thread onmstester onmstester
Hi, I'm seeing a lot of logs like this in all of my nodes (every 5 minutes): WARN  [PERIODIC-COMMIT-LOG-SYNCER] 2018-05-23 08:59:19,075 NoSpamLogger.java:94 - Out of 50 commit log syncs over the past 300s with average duration of 300.00ms, 30 have exceeded the configured commit interval by an

Fwd: Cassandra does launch since computer was accidentally unplugged

2018-12-08 Thread onmstester onmstester
Delete the file: C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1542650688953.log and restart Cassandra. Its possible that you lose a bit of data that just existed on this log (not matter if you have replica or could re-insert data again) Sent using Zoho Mail Forwarded

Fwd: Re: How to gracefully decommission a highly loaded node?

2018-12-06 Thread onmstester onmstester
wrong with decommissioning while someones writing to Cluster? Using Apache Cassandra 3.11.2 Sent using Zoho Mail Forwarded message From : onmstester onmstester To : "user" Date : Wed, 05 Dec 2018 09:00:34 +0330 Subject : Fwd: Re: How to gracefully de

Fwd: Re: How to gracefully decommission a highly loaded node?

2018-12-04 Thread onmstester onmstester
ts you can also disablebinary, disablethrift and disablehandoff to stop serving client requests.  -- SIMON FONTANA OSCARSSON Software Developer Ericsson Ölandsgatan 1 37133 Karlskrona, Sweden simon.fontana.oscars...@ericsson.com www.ericsson.com On tis, 2018-12-04 at 14:21 +0330, onmstester onmstester

How to gracefully decommission a highly loaded node?

2018-12-04 Thread onmstester onmstester
One node suddenly uses 100% CPU, i suspect hardware problems and do not have time to trace that, so decided to just remove the node from the cluster, but although the node state changed to UL, but no sign of Leaving: the node is still compacting and flushing memtables, writing mutations and CPU

Fwd: RE : issue while connecting to apache-cassandra-3.11.1 hosted on a remote VM.

2018-11-16 Thread onmstester onmstester
Also set rpc_address to your remote ip address and restart cassandra. Run nodetool status on Cassandra node to be sure that its running properly. The port you should look for and connect to is 9042, 7199 is the JMX port Sent using Zoho Mail Forwarded message From :

Fwd: Re: Multiple cluster for a single application

2018-11-08 Thread onmstester onmstester
Thank you all, Actually, "the documents" i mentioned in my question, was a talk in youtube seen long time ago and could not find it. Also noticing that a lot of companies like Netflix built hundreds of Clusters each having 10s of nodes and saying that its much stable, i just concluded that big

Multiple cluster for a single application

2018-11-05 Thread onmstester onmstester
Hi, One of my applications requires to create a cluster with more than 100 nodes, I've read documents recommended to use clusters with less than 50 or 100 nodes (Netflix got hundreds of clusters with less 100 nodes on each). Is it a good idea to use multiple clusters for a single application,

Fwd: Re: A quick question on unlogged batch

2018-11-02 Thread onmstester onmstester
unlogged batch meaningfully outperforms parallel execution of individual statements, especially at scale, and creates lower memory pressure on both the clients and cluster.  They do outperform parallel individuals, but in cost of higher pressure on coordinators which leads to more blocked

Fwd: Re: Re: How to set num tokens on live node

2018-11-02 Thread onmstester onmstester
I am facing. Any comments. Thanks and Regards, Goutham On Fri, Nov 2, 2018 at 1:08 AM onmstester onmstester wrote: -- Regards Goutham Reddy IMHO, the best option with two datacenters is to config replication strategy to stream data from dc with wrong num_token to correct one, and then a repair

Fwd: Re: How to set num tokens on live node

2018-11-02 Thread onmstester onmstester
IMHO, the best option with two datacenters is to config replication strategy to stream data from dc with wrong num_token to correct one, and then a repair on each node would move your data to the other dc Sent using Zoho Mail Forwarded message From : Goutham reddy To

Fwd: A quick question on unlogged batch

2018-11-01 Thread onmstester onmstester
Read this: https://docs.datastax.com/en/cql/3.3/cql/cql_reference/batch_r.html Please use batch (any type of batch) for statements that only concerns a single partition, otherwise it cause a lot of performance degradation on your cluster and after a while throughput would be alot less than

Fwd: Re: Re: High CPU usage on some of the nodes due to message coalesce

2018-10-21 Thread onmstester onmstester
Any cron or other scheduler running on those nodes? no Lots of Java processes running simultaneously? no, just Apache Cassandra Heavy repair continuously running? none Lots of pending compactions? none, the cpu goes to 100% on first seconds of insert (write load) so no memtable flushed yet,  Is

Re: Re: High CPU usage on some of the nodes due to message coalesce

2018-10-21 Thread onmstester onmstester
What takes the most CPU? System or User?  most of it is used by  org.apache.cassandra.util.coalesceInternal and SepWorker.run Did you try removing a problematic node and installing a brand new one (instead of re-adding)? I did not install a new node, but did remove the problematic node and CPU

Fwd: Re: High CPU usage on some of the nodes due to message coalesce

2018-10-21 Thread onmstester onmstester
or if the load your application is producing exceeds what your cluster can handle (needs more nodes). Chris On Oct 20, 2018, at 5:18 AM, onmstester onmstester wrote: 3 nodes in my cluster have 100% cpu usage and most of it is used by org.apache.cassandra.util.coalesceInternal and SepWorker.run

How to validate if network infrastructure is efficient for Cassandra cluster?

2018-10-21 Thread onmstester onmstester
Currently, before launching the production cluster, i run 'iperf -s' on half of the cluster and then run 'iperf -c $nextIP' on the other half using parallel ssh, So simultaneously all cluster's nodes are connecting together (paired) and then examining the result of iperfs, doing the math that

High CPU usage on some of the nodes due to message coalesce

2018-10-20 Thread onmstester onmstester
3 nodes in my cluster have 100% cpu usage and most of it is used by  org.apache.cassandra.util.coalesceInternal and SepWorker.run? The most active threads are the messaging-service-incomming. Other nodes are normal, having 30 nodes, using Rack Aware strategy. with 10 rack each having 3 nodes.

Re: Re: Re: how to configure the Token Allocation Algorithm

2018-10-02 Thread onmstester onmstester
Sent using Zoho Mail On Mon, 01 Oct 2018 18:36:03 +0330 Alain RODRIGUEZ wrote Hello again :), I thought a little bit more about this question, and I was actually wondering if something like this would work: Imagine 3 node cluster, and create them using: For the 3 nodes: `num_token:

Fwd: Re: Re: how to configure the Token Allocation Algorithm

2018-10-01 Thread onmstester onmstester
ed message On Mon, Oct 1, 2018 at 12:18 PM onmstester onmstester wrote: What if instead of running that python and having one node with non-vnode config, i remove the first seed node and re-add it after cluster was fully up ? so the token ranges of first seed node would also b

Fwd: Re: how to configure the Token Allocation Algorithm

2018-10-01 Thread onmstester onmstester
Thanks Alain, What if instead of running that python and having one node with non-vnode config, i remove the first seed node and re-add it after cluster was fully up ? so the token ranges of first seed node would also be assigned by Allocation Alg Forwarded message

how to configure the Token Allocation Algorithm

2018-09-30 Thread onmstester onmstester
Since i failed to find a document on how to configure and use the Token Allocation Algorithm (to replace the random Algorithm), just wanted to be sure about the procedure i've done: 1. Using Apache Cassandra 3.11.2 2. Configured one of seed nodes with num_tokens=8 and started it. 3. Using Cqlsh

High CPU usage on writer application

2018-09-24 Thread onmstester onmstester
Hi,  My app writes 100K rows per seconds to a C* cluster (including 30 nodes and using version 3.11.2). There are 20 threads, each writing 10K (list size in below code is 100K) statements using async API: for (Statement s:list) { ResultSetFuture future = session.executeAsync(s);

Re: node replacement failed

2018-09-22 Thread onmstester onmstester
Another question, Is there a management tool to do nodetool cleanup one by one (wait until finish of cleaning up one node then start clean up for the next node in cluster)? On Sat, 22 Sep 2018 16:02:17 +0330 onmstester onmstester wrote I have a cunning plan (Baldrick wise) to solve

Re: node replacement failed

2018-09-22 Thread onmstester onmstester
problem? On Mon, 10 Sep 2018 17:12:48 +0430 onmstester onmstester wrote Thanks Alain, First here it is more detail about my cluster: 10 racks + 3 nodes on each rack nodetool status: shows 27 nodes UN and 3 nodes all related to single rack as DN version 3.11.2 Option 1: (Change schema

Re: stuck with num_tokens 256

2018-09-22 Thread onmstester onmstester
If you have problems with balance you can add new nodes using the algorithm and it'll balance out the cluster. You probably want to stick to 256 tokens though. I read somewhere (don't remember the ref) that all nodes of the cluster should use the same algorithm, so if my cluster suffer from

Re: stuck with num_tokens 256

2018-09-22 Thread onmstester onmstester
the old DC and hopefully end up with a balanced cluster. Definitely test beforehand though because that was just me theorising... I'll note though that if your existing clusters don't have any major issues it's probably not worth the migration at this point. On Sat, 22 Sep 2018 at 17:40, onmstester

stuck with num_tokens 256

2018-09-22 Thread onmstester onmstester
I noticed that currently there is a discussion in ML with subject: changing default token behavior for 4.0. Any recommendation to guys like me who already have multiple clusters ( > 30 nodes in each cluster) with random partitioner and num_tokens = 256? I should also add some nodes to existing

Scale SASI index

2018-09-17 Thread onmstester onmstester
By adding new nodes to cluster, should i rebuild SASI indexes on all nodes ?

Re: node replacement failed

2018-09-14 Thread onmstester onmstester
Thanks, I am still thinking about it, but before going deeper, is this still an issue for you at the moment? Yes, It is.

Re: node replacement failed

2018-09-10 Thread onmstester onmstester
*heers, --- Alain Rodriguez - @arodream - al...@thelastpickle.com France / Spain The Last Pickle - Apache Cassandra Consulting http://www.thelastpickle.com Le lun. 10 sept. 2018 à  09:09, onmstester onmstester a écrit : Any idea? Sent using Zoho Mail On Sun, 09 Sep 2018 11:23:17

Re: node replacement failed

2018-09-10 Thread onmstester onmstester
Any idea? Sent using Zoho Mail On Sun, 09 Sep 2018 11:23:17 +0430  onmstester onmstester wrote Hi, Cluster Spec: 30 nodes RF = 2 NetworkTopologyStrategy GossipingPropertyFileSnitch + rack aware Suddenly i lost all disks of cassandar-data on one of my racks, after replacing the disks

node replacement failed

2018-09-09 Thread onmstester onmstester
Hi, Cluster Spec: 30 nodes RF = 2 NetworkTopologyStrategy GossipingPropertyFileSnitch + rack aware Suddenly i lost all disks of cassandar-data on one of my racks, after replacing the disks, tried to replace the nodes with same ip using this:

Re: [EXTERNAL] Re: adding multiple node to a cluster, cleanup and num_tokens

2018-09-08 Thread onmstester onmstester
Thanks Jeff, You mean that with RF=2, num_tokens = 256 and having less than 256 nodes i should not worry about data distribution? Sent using Zoho Mail On Sat, 08 Sep 2018 21:30:28 +0430 Jeff Jirsa wrote Virtual nodes accomplish two primary goals 1) it makes it easier to gradually

  1   2   >