Best practices while designing backup storage system for big Cassandra cluster

2019-03-28 Thread manish khandelwal
Hi I would like to know is there any guideline for selecting storage device (disk type) for Cassandra backups. As per my current observation, NearLine (NL) disk on SAN slows down significantly while copying backup files (taking full backup) from all node simultaneously. Will using SSD disk

Changing existing Cassandra cluster from single rack configuration to multi racks configuration

2019-03-06 Thread manish khandelwal
We have a 6 node Cassandra cluster in which all the nodes are in same rack in a dc. We want to take advantage of "multi rack" cluster (example: parallel upgrade on all the nodes in same rack without downtime). I would like to know what is the recommended process to change an existing cluster with

Question regarding bloom filters

2019-10-28 Thread manish khandelwal
Hi All I was reading about the read path. It says bloom filters are read from memory. But bloom filters can become very large so they may occupy lot of memory. So are all bloom filters loaded in memory on startup or are loaded with read requests. Regards Manish

Re: Cassandra Repair question

2019-10-18 Thread manish khandelwal
No it will only cover primary ranges of nodes on single rac. Repair with -pr option is to be run on all nodes in a rolling manner. Regards Manish On 19 Oct 2019 10:03, "Krish Donald" wrote: > Hi Cassandra experts, > > > We are on Cassandra 3.11.1. > > We have to run repairs for a big cluster.

Corrupt SSTable Cassandra 3.11.2

2020-02-13 Thread manish khandelwal
Hi I see a corrupt SSTable in one of my keyspace table on one node. Cluster is 3 nodes with replication 3. Cassandra version is 3.11.2. I am thinking on following lines to resolve the corrupt SSTable issue. 1. Run nodetool scrub. 2. If step 1 fails, run offline sstabablescrub. 3. If step 2 fails,

Re: Corrupt SSTable Cassandra 3.11.2

2020-02-25 Thread manish khandelwal
Thanks all for your support. I executed the discussed process (barring repair, as table was read for reporting only) and it worked fine in production. Regards Manish >

Re: [EXTERNAL] Cassandra 3.11.X upgrades

2020-03-03 Thread manish khandelwal
Should upgradesstables not be run after every node is upgraded? If we need to rollback then we will not be able to downgrade sstables to older version. Regards Manish On Tue, Mar 3, 2020 at 11:26 PM Hossein Ghiyasi Mehr wrote: > It's more safe to upgrade one node before upgrading another node

Re: sstableloader: How much does it actually need?

2020-02-06 Thread manish khandelwal
Yes you will have all the data in two nodes provided there is no mutation drop at node level or data is repaired For example if you data A,B,C and D. with RF=3 and 4 nodes (node1, node2, node3 and node4) Data A is in node1, node2 and node3 Data B is in node2, node3, and node4 Data C is in node3,

Re: Corrupt SSTable Cassandra 3.11.2

2020-02-13 Thread manish khandelwal
ompany/datastax> > <https://www.facebook.com/datastax> <https://twitter.com/datastax> > <http://feeds.feedburner.com/datastax> <https://github.com/datastax/> > > <https://www.datastax.com/accelerate> > > > > On Fri, 14 Feb 2020 at 04:39, manish

Re: Corrupt SSTable Cassandra 3.11.2

2020-02-13 Thread manish khandelwal
Hi Erick Thanks for your quick response. I have attached the full stacktrace which show exception during validation phase of table repair. I would like to know what will be "ordinary hammer" in this case. Do you want to suggest that deleting only corrupt sstable file ( in this case

Re: Corrupt SSTable Cassandra 3.11.2

2020-02-13 Thread manish khandelwal
Thanks Erick I would like to explain how data resurrection can take place with single SSTable deletion. Consider this case of table with Levelled Compaction Strategy 1. Data A written a long time back. 2. Data A is deleted and tombstone is created. 3. After GC grace tombstone is purgeable. 4.

Re: Corrupt SSTable Cassandra 3.11.2

2020-02-13 Thread manish khandelwal
rictly correct to do is treat every corrupt sstable > exception as a failed host, and replace it just like you would a failed > host. > > > On Thu, Feb 13, 2020 at 10:55 PM manish khandelwal < > manishkhandelwa...@gmail.com> wrote: > >> Thanks Erick >> >> I w

Re: Cassandra2.0.14 : Obsolete files not being deleted after compaction

2020-01-21 Thread manish khandelwal
ll the files for any given sequence number ie data, index, toc > etc > > Regards, > > Nitan > > Cell: 510 449 9629 > > On Jan 21, 2020, at 5:36 AM, manish khandelwal < > manishkhandelwa...@gmail.com> wrote: > >  > Hi Team > > I am observing some obs

Re: Cassandra2.0.14 : Obsolete files not being deleted after compaction

2020-01-22 Thread manish khandelwal
dead: session failed* Does anyone remember any JIRA ticket related to obsolete sstables not being deleted after compaction? Regards Manish On Wed, Jan 22, 2020 at 11:37 AM Jeff Jirsa wrote: > > > On Tue, Jan 21, 2020 at 8:58 PM manish khandelwal < > manishkhandelwa...@

Cassandra2.0.14 : Obsolete files not being deleted after compaction

2020-01-21 Thread manish khandelwal
Hi Team I am observing some obsolete files in Cassandra 2.0.14 which are already compacted but not removed from the system after compaction. As per CASSANDRA-7872 , after GC grace period has passed the sstables are open for read again and can

Impact of setting low value for flag -XX:MaxDirectMemorySize

2020-04-17 Thread manish khandelwal
What will be the impact of setting the value of XX:MaxDirectMemorySize to some low value. Currently default value for off heap is equal to heap memory. I saw this open ticket discussing this but could not infer much from it. https://issues.apache.org/jira/browse/CASSANDRA-10930 Regards Manish

Consistency with Datacenter switch

2020-03-15 Thread manish khandelwal
While switching over datacenters, there is a chance of mutation drop because of which inconsistency may occur. To avoid inconsistency we can do following : Monitor and if require then run repair 1. Monitor tpstats in all nodes. If dropped message count is 0, it can be inferred no mutation

Re: Consistency with Datacenter switch

2020-03-15 Thread manish khandelwal
one dc to local quorum in the other dc > without losing any writes? > > The easiest way to do this strictly correctly is to take the latency hit > and do quorum while you run repair, then you can switch to local quorum on > the other side. > > A few more notes inline > > >

Re: High value of "All Time Blocked" for "Native-Transport-Requests"

2020-03-30 Thread manish khandelwal
Thanks for the response. With continuous runs I am observing my off heap memory increasing and *memory usage remains high even if no traffic is running*. Sometimes a node is hung with following error: ERROR [ReadStage-5] 2020-03-12 20:47:46,406 JVMStabilityInspector.java:74 - OutOfMemory error

Re: High value of "All Time Blocked" for "Native-Transport-Requests"

2020-03-30 Thread manish khandelwal
I have already updated that property to mmap_index_only. And in pmap I could see only index files are loaded into memory not the SSTables. I feel this is some to do with direct memory buffer. There is a opened JIRA ticket https://issues.apache.org/jira/browse/CASSANDRA-10930 as well regarding

Re: High value of "All Time Blocked" for "Native-Transport-Requests"

2020-03-30 Thread manish khandelwal
To add on to my previous email, I think we get this error when actual off heap memory breaches "off heap memory limit" as mentioned in this JIRA <https://issues.apache.org/jira/browse/CASSANDRA-10930>. On Mon, Mar 30, 2020 at 3:10 PM manish khandelwal < manishkhandelwa...@gma

Re: Impact of setting low value for flag -XX:MaxDirectMemorySize

2020-04-22 Thread manish khandelwal
I am running spark (max heap 4G) and a java application (4G) with my Cassandra server (8G). After heavy loading, if I run a spark process some main memory is pushed into swap. But if a restart Cassandra and execute the spark process memory is not pushed into the swap. Idea behind asking the

Re: Add a new node of 3.11.5 in a 3.11.0 Cassandra Cluster

2020-05-09 Thread manish khandelwal
You should not bootstrap with mixed cluster. First you should upgrade and then bootstrap a new node. If you are not able to upgrade due to disk space constraints while running upgradesstable, then 1. Reduce the number of threads for upgradesstable (--jobs parameter). By default it is 2. You can

Re: Can repair run when rebuild command is running on one of the node

2020-05-11 Thread manish khandelwal
Thanks Erick for quick response On Tue, May 12, 2020 at 9:11 AM Erick Ramirez wrote: > It isn't ideal to be running repairs while you're adding a new DC. When > you add nodes with auto_bootstrap: false, they immediately join the ring > but without data on them (which is by design). If repairs

Can repair run when rebuild command is running on one of the node

2020-05-11 Thread manish khandelwal
Hi Team I have a 5 node data center. If I create another datacenter, and run rebuild on nodes of node datacenter. While the rebuild command is running on one of the node of new datacenter can our schedule repair run along with it. I am using repair -pr Regards Manish

High value of "All Time Blocked" for "Native-Transport-Requests"

2020-03-23 Thread manish khandelwal
I am working with Cassandra 3.11.2. In nodetool tpstats output I am observing high number in "All Time Blocked" column for "Native-Transport-Requests" pool. I have two questions 1. Does a high number of "ALL Time Blocked" count impact client latency? 2. Does it increase off head memory overtime?

Re: TEST Cluster corrupt after removenode. how to restore

2020-05-19 Thread manish khandelwal
Even when one table is corrupt your repair will fail. To handle this case without data loss you could replace the complete node ( safest and sureshot option) On 19 May 2020 20:20, "Leena Ghatpande" wrote: > One of the tables SStable got corrupted on all nodes. But repairs were > failing for

Rollback Cassandra after 1 node upgrade

2020-09-04 Thread manish khandelwal
Hi We have been forced into rolling back our Cassandra after 1 node upgrade. The node was upgraded 10 days ago. We have the backup of the old data. Strategy one which we are thinking : 1. Rollback to old binaries and configuration. 2. Restore the old data from backup. 3. Run Repair. Another

Re: Rollback Cassandra after 1 node upgrade

2020-09-04 Thread manish khandelwal
Around 1TB On Sat, Sep 5, 2020 at 9:45 AM Aakash Pandhi wrote: > How much data to restore and repair on that node? > > Sincerely, > > Aakash Pandhi > > > On Friday, September 4, 2020, 11:08:56 PM CDT, manish khandelwal < > manishkhandelwa...@gmail.com> wrote:

Re: Rollback Cassandra after 1 node upgrade

2020-09-05 Thread manish khandelwal
Thanks Alex for the reply. On Sat, Sep 5, 2020 at 3:09 PM Oleksandr Shulgin < oleksandr.shul...@zalando.de> wrote: > On Sat, Sep 5, 2020 at 5:55 AM manish khandelwal < > manishkhandelwa...@gmail.com> wrote: > >> Hi >> >> We have been forced into rolling

Re: Rollback Cassandra after 1 node upgrade

2020-09-04 Thread manish khandelwal
3.11.2 to 2.1.16 On Sat, Sep 5, 2020 at 9:27 AM Surbhi Gupta wrote: > Hi Manish, > > Please provide both versions. > > Thanks > Surbhi > > On Fri, Sep 4, 2020 at 8:55 PM manish khandelwal < > manishkhandelwa...@gmail.com> wrote: > >> Hi >> >&g

Re: efficient delete over a "wide" table?

2020-09-04 Thread manish khandelwal
> > Would there be a difference (in terms of generated tombstones) if I would > > a) issue delete one-by-one like > DELETE FROM ... WHERE ... uuid = 'a' > DELETE FROM ... WHERE ... uuid = 'b' > ... > DELETE FROM ... WHERE ... uuid = 'z' > > or > > b) issue delete in a group fashion like > DELETE

Re: CVE-2020-13946 Apache Cassandra RMI Rebind Vulnerability

2020-09-01 Thread manish khandelwal
Hi Sam Is there any alternative to avoid this vulnerability? Like upgrade to specific JVM version. Regards Manish On Tue, Sep 1, 2020 at 8:03 PM Sam Tunnicliffe wrote: > CVE-2020-13946 Apache Cassandra RMI Rebind Vulnerability > > Versions Affected: > All versions prior to: 2.1.22, 2.2.18,

Re: Cassandra scale-out with no traffic on newly joined nodes

2020-09-08 Thread manish khandelwal
Can you add new host as contact points and see if traffic lands on them or not? Also you can verify new nodes are added in system.peers of host name which you are giving as contact points On Tue, Sep 8, 2020 at 11:27 AM Sandeep Nethi wrote: > Yes, all nodes are UN and no issues identified.

Re: Getting Invalid Yaml exception after we upgraded java version

2020-08-21 Thread manish khandelwal
Pushpendra You can check this Datastax post . Looks like a similar error. You can run cassandra-stress utility to bypass the yaml parser. Regards Manish On Fri, Aug 21, 2020 at 11:29 AM

Re: Anti-entropy repair with a 4 node cluster replication factor 4

2020-10-27 Thread manish khandelwal
If you run full repair then it should be fine, since all the replicas are present on all the nodes. If you are using -pr option then you need to run on all the nodes. On Tue, Oct 27, 2020 at 4:11 PM Fred Al wrote: > Hello! > Running Cassandra 2.2.9 with a 4 node cluster with replication factor

Re: Cassandra on K8S

2020-08-03 Thread manish khandelwal
ondition of which container starts first. The k8s scheduler and > kubelet daemon assign IPs to pods. > > On Mon, Aug 3, 2020 at 11:14 PM manish khandelwal < > manishkhandelwa...@gmail.com> wrote: > >> I have started reading about how to deploy Cassandra with K8S. But a

Re: Cassandra on K8S

2020-08-03 Thread manish khandelwal
en and how this IP swap can occur?* Regards Manish On Mon, Jul 6, 2020 at 10:40 PM Jeff Jirsa wrote: > > > On Mon, Jul 6, 2020 at 10:01 AM manish khandelwal < > manishkhandelwa...@gmail.com> wrote: > >> I have started reading about how to deploy Cassandra with K8S. But

Re: Cassandra on K8S

2020-08-03 Thread manish khandelwal
other nodes IP* Regards Manish On Tue, Aug 4, 2020 at 10:07 AM manish khandelwal < manishkhandelwa...@gmail.com> wrote: > But again if > Some Cassandra node (pod) with particular IP X is down, > Second Cassandra node (pod) tries to take the IP X of first Cassandra node, > Sec

Re: Connect java application to Cassandra in Kubernetes

2020-08-11 Thread manish khandelwal
Erick, even if we give service as a contact point datastax driver will calculate IPs of topology to connect the cluster. So I think as John proposed in answer to similar question, after driver initialization in case of total outage of Cassandra cluster, bouncing of application is the only option.

Re: Why a READ REPAIR ?

2020-08-11 Thread manish khandelwal
Hi Tobias READ2 will not be blocked by READ repair of READ1. Regards Manish On Tue, Aug 11, 2020 at 6:02 PM Tobias Eriksson wrote: > Thanx Erick, > > Perhaps this is super obvious but I need a confirmation as you say “…not > subsequent reads for other data unrelated to the read being

Cassandra on K8S

2020-07-06 Thread manish khandelwal
I have started reading about how to deploy Cassandra with K8S. But as I read more I feel there are a lot of challenges in running Cassandra on K8s. Some of the challenges which I feel are 1. POD IPs identification - If the pods go down and when they come up their IPs change, how is it handled as

Re: Implication of having similar private IPs in two datacenters

2020-07-03 Thread manish khandelwal
I did a small poc for this *(similar rpc_address for both data centers) . *My observations: 1. rpc_address is used by a client( datastax java driver) in my case to connect to cassandra nodes. 2. Driver is keeping a list of contact points which it is updating using system.peers table 3. Having two

Re: Implication of having similar private IPs in two datacenters

2020-06-24 Thread manish khandelwal
The nodes within one datacenter will talk on private IPs and for communication across data center public ips are used. Does that answer your question? or you are asking in terms of network layer. As per this link

Implication of having similar private IPs in two datacenters

2020-06-24 Thread manish khandelwal
Are there any implication if I have similar set of private IPs in two data centers. For example if I have A,B and C as private IPs in one data center and I define A,B and C as my private IPs in other data center, does it impact the cluster? Regards

Re: Cassandra upgrade from 3.11.3 -> 3.11.6

2020-06-24 Thread manish khandelwal
Rightly said by Surbhi, it is not good to scale with mixed versions as debugging issues will be very difficult. Better to upgrade first and then scale. Regards On Wed, Jun 24, 2020 at 11:20 AM Surbhi Gupta wrote: > In case of any issue, it gets very difficult to debug when we have > multiple

Re: Cassandra upgrade from 3.11.3 -> 3.11.6

2020-06-26 Thread manish khandelwal
r/bootstrap) with different file versions is >>>> usually a problem. Running a mixed version cluster is fine – for the time >>>> you are doing the upgrade. I would not stay on mixed versions for any >>>> longer than that. It takes more time, but I separa

Re: Do I understand the working of RF and CL correctly?

2020-06-12 Thread manish khandelwal
Yes Manu you are correct in your understanding. Consistency level defines minimum number of nodes that must acknowledge a read or write operation before the operation can be considered as successful. In your case, since two nodes were up and you queried with CL 1, your request was successful. That

Re: Consistency level shows as null in Java driver

2020-06-12 Thread manish khandelwal
This is how getConsistencyLevel method is implemented. This method returns consistencylevel of the query or null if no consistency level has been set using setConsistencyLevel. Regards Manish On Fri, Jun 12, 2020 at 3:43 PM Manu Chadha wrote: > Hi > > In my Cassandra Java driver code, I am

Re: Replacing a Cassandra node in K8S

2020-07-28 Thread manish khandelwal
e correct that the volume will get remounted. > If however you are using a local or hostPath volume, then it will be lost > and you will want to restart Cassandra with the -Dcassandra.replace_address > option. > > On Mon, Jul 27, 2020 at 5:17 AM manish khandelwal < > manishkhandelwa...@

Replacing a Cassandra node in K8S

2020-07-27 Thread manish khandelwal
Hi Team I was wondering how Cassandra node is replaced if one of the worker node fails in k8s. My understanding is that since PVCs are remounted to their volume mounts, no matter where the pods are rescheduled (any node), so replacing a node will not be a issue only ip will get changed.

Re: Corrupt sstables_activity

2020-07-06 Thread manish khandelwal
I did a POC on CCM, removed sstable_activity sstables files from directory after stopping the node. Restarted the node, sstable_activity table was generated again. You can verify it in your test environment and see if node is working fine without any issues. Important thing is to validate any step

Re: Issue with anti-compaction while running full repair with -pr option

2020-11-09 Thread manish khandelwal
Pushpendra, Probably you can read all the data using spark with Consistency level ALL for repairing the data. Regards Manish On Mon, Nov 9, 2020 at 11:31 AM Alexander DEJANOVSKI wrote: > Hi, > > You have two options to disable anticompaction when running full repair: > > - add the list of DCs

Re: Anti Compactions while running repair

2020-11-08 Thread manish khandelwal
cremental repair. > > > Le ven. 6 nov. 2020 à 04:37, manish khandelwal < > manishkhandelwa...@gmail.com> a écrit : > >> In documentation it is given that while running incremental repairs, anti >> compaction is done which results in repaired and unrepaired ss

Re: Anti Compactions while running repair

2020-11-09 Thread manish khandelwal
Thanks Alex On Mon, Nov 9, 2020 at 12:36 PM Alexander DEJANOVSKI wrote: > Only sstables at unrepaired state go through anticompaction. > > Le lun. 9 nov. 2020 à 07:01, manish khandelwal < > manishkhandelwa...@gmail.com> a écrit : > >> Thanks Alex. >> &g

Anti Compactions while running repair

2020-11-05 Thread manish khandelwal
In documentation it is given that while running incremental repairs, anti compaction is done which results in repaired and unrepaired sstables. Since anti compaction also runs with full repair and primary range repairs, I have the following question: Is anti compaction different in case of full

Re: Suggestions on Running UpgradeSSTables

2021-05-19 Thread manish khandelwal
One should avoid doing things (apart from normal traffic) in mixed mode. The best approach is to upgrade the nodes as fast as possible and then do other activities. Regards On Thu, May 20, 2021 at 10:44 AM Jai Bheemsen Rao Dhanwada < jaibheem...@gmail.com> wrote: > Thanks for the suggestions.

How to remove tombstones in a levelled compaction table in Cassandra 2.1.16?

2021-07-05 Thread manish khandelwal
and queries are getting timed out. Am I right in my thinking? What is the possible way to get out of this? I thought of using major compaction but for LCS that was introduced in Cassandra 2.2. Also user defined compactions dont work on LCS tables. Regards Manish Khandelwal

Re: How to remove tombstones in a levelled compaction table in Cassandra 2.1.16?

2021-07-05 Thread manish khandelwal
ht of using major compaction but for LCS that was introduced in >> Cassandra 2.2. Also user defined compactions dont work on LCS tables. >> >> >> >> Regards >> >> Manish Khandelwal >> > > If it's tombstones specifically you'll be able to

Re: How to remove tombstones in a levelled compaction table in Cassandra 2.1.16?

2021-07-06 Thread manish khandelwal
note: that when you > first run this IF any sstables are above the tombstone_ratio setting for > that table they will be compacted, this may cause extra load on the cluster. > > Vytenis > ... always do your own research and verify what people say. :) > > On Mon, Jul 5, 2021 at 10:11 PM manish k

Re: Best strategy to run repair

2021-03-22 Thread manish khandelwal
Also try to use Cassandra reaper (as Kane also mentioned) for subrange repair. Doing subrange repair yourself may lead to a lot of trouble as calculating correct subranges is not an easy task. On Tue, Mar 23, 2021 at 3:38 AM Kane Wilson wrote: > -pr on all nodes takes much longer as you'll do

Re: Best strategy to run repair

2021-03-22 Thread manish khandelwal
n Mon, 22 Mar 2021 at 20:28, manish khandelwal < > manishkhandelwa...@gmail.com> wrote: > >> Also try to use Cassandra reaper (as Kane also mentioned) for subrange >> repair. Doing subrange repair yourself may lead to a lot of trouble as >> calculating correct subranges is

Setting read_ahead_kb in Kubernetes environment

2021-04-06 Thread manish khandelwal
Hi I am struggling to find way to set read_ahead_kb for Cassandra in our Kubernetes environment. Any way by which we can achieve this? My issue is that since PV is mounted in a container and it can have any name depending on partition number. Also wanted to know where it will be set, inside the

Re: [UPGRADATION] Apache Cassandra from version 3.0.9 to 4.0.0

2021-09-06 Thread manish khandelwal
Totally agree with Jeff and Bowen there. Don't try to achieve something faster by cutting corners. Migration to GCP from physical DC should be done on the same versions. On Mon, Sep 6, 2021 at 2:11 PM Bowen Song wrote: > Hello Ashish, > > > I'm slightly worried about this: > > *Since I won't be

Re: Number of DCs in Cassandra

2021-07-14 Thread manish khandelwal
I don't think there is any restriction on the number of data centers. So technically you can add as many data centers you want. Performance depends on how you use your cluster. For example, one of your data centers could be read only, or is there traffic on all the data centers. On Wed, Jul 14,

Re: High memory usage during nodetool repair

2021-08-02 Thread manish khandelwal
mmap is used for faster reads and as you guessed right you might see read performance degradation. If you are seeing high memory usage after repairs due to mmaped files, the only way to reduce the memory usage is to trigger some other process which requires memory. *mmapped* files use buffer/cache

Re: High memory usage during nodetool repair

2021-08-02 Thread manish khandelwal
Missed the heap part, not sure why is that happening On Tue, Aug 3, 2021 at 8:59 AM manish khandelwal < manishkhandelwa...@gmail.com> wrote: > mmap is used for faster reads and as you guessed right you might see read > performance degradation. If you are seeing high memory usage a

Long GC pauses during repair

2021-08-03 Thread manish khandelwal
Long GC (1 seconds /2 seconds) pauses seen during repair on the coordinator. Running full repair with partition range option. GC collector is CMS and heap is 14G. Cluster is 7+7. Cassandra version is 3.11.2. Not much traffic when repair is running. What could be the probable cause of long gc

Re: Reduce num_tokens on single node cluster

2021-07-30 Thread manish khandelwal
Single node does not make any sense in Cassandra. It should not make any difference with a single node with whatever number of tokens you start with. You can change your tokens on the test server and see what happens. On Fri, Jul 30, 2021 at 5:53 PM Maxim Parkachov wrote: > Hi everyone, > > I

Re: Issue with native protocol

2021-07-28 Thread manish khandelwal
Have you tried restarting your application? It should renegotiate the protoco during handshake and should resolve the issue. On Wed, Jul 28, 2021 at 3:06 PM Srinivas Polamarasetty < srinivas.polamarase...@logmein.com> wrote: > Query shows 4 but App team also getting below error. They were not

Re: Long GC pauses during repair

2021-08-04 Thread manish khandelwal
eap on 1 node > to see. or go GC1 if it is easy way. > > Thanks, > Jim > > On Tue, Aug 3, 2021 at 3:33 AM manish khandelwal < > manishkhandelwa...@gmail.com> wrote: > >> Long GC (1 seconds /2 seconds) pauses seen during repair on the >> coordinator. Runnin

Re: How does a node decide where each of its vnodes will be replicated to?

2021-11-08 Thread manish khandelwal
Just to add on to your response: *num_tokens* define the number of vnodes a node can have. Default is 256. *Initial token* range is predefined (For murmur -2**63 to 2**63-1) So if you have one node in (does not make sense) cluster with num_tokens as 256 then you will have 256vnodes. Scaling up

Re: Storing user activity logs

2021-07-19 Thread manish khandelwal
I concur with Eliot view. Only way you can reduce partition size is by tweaking your partition key. Here with user_id as partition key, partition size depends on the activity of the user. For a superactive user it can become large in no time. After changing the key migration of old data to the

Re: Number of DCs in Cassandra

2021-07-14 Thread manish khandelwal
We are planning to go with 5 DCs with RF of 3 in each. All DCs will have > reads and writes. Most queries are done at LOCAL_QUORUM. > A very few Simple and CAS queries (<0.1%) will be done at QUORUM > consistency. > > On Wed, Jul 14, 2021 at 12:19 PM manish khandelwal < > m

Re: Query timed out after PT2M

2022-02-03 Thread manish khandelwal
It maybe the case you have lots of tombstones in this table which is making reads slow and timeouts during bulk reads. On Fri, Feb 4, 2022, 03:23 Joe Obernberger wrote: > So it turns out that number after PT is increments of 60 seconds. I > changed the timeout to 96, and now I get PT16M

Re: Cassandra 4.0 upgrade from Cassandra 3x

2022-02-10 Thread manish khandelwal
You can see upgrading instructions here https://github.com/apache/cassandra/blob/cassandra-4.0.2/NEWS.txt. On Fri, Feb 11, 2022 at 2:52 AM Abdul Patel wrote: > Hi > apart from standard upgrade process any thing specific needs ti be > handled separately for this upgrade process? > > Any changes

Re: Hanging repairs in Cassandra

2022-01-18 Thread manish khandelwal
ue is easily reproducible, I would re-run the repair with a > reduce scope (such as table and token range) to get less logs related to > the repair session. Less logs means less time spend on reading and > analysing them. > > Hope this helps. > On 18/01/2022 10:03, manish khand

Hanging repairs in Cassandra

2022-01-18 Thread manish khandelwal
I have a Cassandra 3.11.2 cluster with two DCs. While running repair , I am observing the following behavior. I am seeing that node is not able to receive merkle tree from one or two nodes. Also I am able to see that the missing nodes did send the merkle tree but it was not received. This make

Re: Hanging repairs in Cassandra

2022-01-18 Thread manish khandelwal
f you > accidentally ran "nodetool repair" on two nodes and both involve the same > table, or if you cancelled and then restarted a "nodetool repair" on a node > without waiting or killing the remannings of the first repair session on > other nodes. > On 18/01/2022 11

Re: Hanging repairs in Cassandra

2022-01-18 Thread manish khandelwal
since then. For > that reason, I would rather not compare the issue you have with some 9 > years old issues someone else had. > > > On 18/01/2022 16:11, manish khandelwal wrote: > > I am not sure what is happening but it has happened thrice. It is > happening that merkle tre

Re: Hanging repairs in Cassandra

2022-01-21 Thread manish khandelwal
me behavior. >> >> Regards >> Manish >> >> On Tue, Jan 18, 2022, 22:50 Bowen Song wrote: >> >>> The link was related to Cassandra 1.2, and it was 9 years ago. Cassandra >>> was full of bugs at that time, and it has improved a l

Handshaking version by Cassandra nodes

2022-01-22 Thread manish khandelwal
When a node does handshaking? I see log of handshaking messages in my logs, what do they signify? Regards Manish

Re: Hanging repairs in Cassandra

2022-01-19 Thread manish khandelwal
repair, I had to either keep retrying or break down the token ranges > into smaller subsets and use the "-st" and "-et" parameters. Since then > I've switched to use Cassandra Reaper and have never had similar issues. > > > On 19/01/2022 02:22, manish khandelw

Re: Hanging repairs in Cassandra

2022-01-24 Thread manish khandelwal
e to draw upon the community’s knowledge of a current release of the > database. > > — Scott > > On Jan 21, 2022, at 8:51 PM, manish khandelwal < > manishkhandelwa...@gmail.com> wrote: > >  > Hi All > > After going through the system.logs, I still see sometimes t

Re: Hanging repairs in Cassandra

2022-01-24 Thread manish khandelwal
a TCP connection in such short time, unless the number of > connections is very large and generally short lived (which often see in web > servers). > On 24/01/2022 13:03, manish khandelwal wrote: > > Hi All > > Thanks for the suggestions. The issue was *tcp_keepalive_time* ha

Re: Query around Data Modelling

2022-06-22 Thread manish khandelwal
Table1 should be fine if some column values are not entered than Cassandra will not create entry for them so partiton will almost be same in both cases. On Thu, Jun 23, 2022, 07:08 MyWorld wrote: > Hi all, > > Just a small query around data Modelling. > Suppose we have to design the data model

Re: Wrong Consistency level seems to be used

2022-07-21 Thread manish khandelwal
Consistency can also be set on a statement basis. So please check in your code that you might be setting consistency 'ALL' for some queries. On Thu, Jul 21, 2022 at 7:23 PM pwozniak wrote: > Hi, > > we have the following code (java driver): > > cluster =

Table with 'compact storage' is not shown in "describe table" output in cqlsh

2022-09-27 Thread manish khandelwal
Hi All As I understand there was a plan to drop *Compact Storage* support with *Cassandra 4* but later few issues were identified which resulted in continued support for Compact Storage in Cassandra 4. My cluster with a few old "compact storage" tables was able to come up with Cassandra 4.0.5.

Re: Table with 'compact storage' is not shown in "describe table" output in cqlsh

2022-09-28 Thread manish khandelwal
he warning. Regards Manish On Tue, Sep 27, 2022 at 3:22 PM manish khandelwal < manishkhandelwa...@gmail.com> wrote: > Hi All > > As I understand there was a plan to drop *Compact Storage* support with > *Cassandra > 4* but later few issues were identified which resulted in

Local read request going across DC

2022-09-20 Thread manish khandelwal
Hi All In one of our cluster, read request with consistency "LOCAL_QUORUM" is going across DC. When we run query setting CONSISTECY to LOCAL_QUORUM in cqlsh, with tracing on, we see READ and digest request sent across to nodes on other DC. I have checked gossipinfo, peers table, nodetool status.

Re: Local read request going across DC

2022-09-21 Thread manish khandelwal
Thanks Erick for the response. read_repair_chance is 0. Can speculative_retry cause this? We have that set at 99 percentile. Regards Manish On Wed, Sep 21, 2022 at 11:17 AM Erick Ramirez wrote: > It sounds like read-repair chance is enabled on the table. Check the table > schema for a

Re: Local read request going across DC

2022-09-21 Thread manish khandelwal
Thanks a lot Erick for your help and support. Regards Manish On Wed, Sep 21, 2022, 19:12 Erick Ramirez wrote: > Just to circle back here, I've reviewed the trace output and it shows > multiple requests fired off for lookups on the `roles` table which > indicated to me that the default

Re: Query for Cassandra Driver

2022-12-22 Thread manish khandelwal
Hi Deepti I think you can reach out to https://groups.google.com/a/lists.datastax.com/g/cpp-driver-user. Regards Manish On Fri, Dec 23, 2022 at 12:52 PM Deepti Sharma S via user < user@cassandra.apache.org> wrote: > Hello Team, > > > > Could you please help in answering below query. > > > > >

Re: Query regarding EOS for Cassandra version 3.11.13

2022-12-15 Thread manish khandelwal
3.11.x versions will be maintained till May July 2023. Please refer https://cassandra.apache.org/_/download.html On Thu, Dec 15, 2022, 20:55 Pranav Kumar (EXT) via user < user@cassandra.apache.org> wrote: > Hi Team, > > > > Could you please help us to know when version 3.11.13 is going to be

Re: Cleanup

2023-02-16 Thread manish khandelwal
There is no advantage of running cleanup if no new nodes are introduced. So cleanup time should remain same when adding new nodes. Cleanup is a local to node so network bandwidth should have no effect on reducing cleanup time. Dont ignore cleanup as it can cause you disks occupied without any

JIRA account creation request

2023-02-15 Thread manish khandelwal
Hi I would like to get my JIRA account created as I would like to contribute. Here are my details email address : manishkhandelwa...@gmail.com If any other information is needed please let me know. Regards Manish

Re: Impact of column names on storage

2023-06-12 Thread manish khandelwal
I think it is verbose to make it readable in sstableoutput. Column names should not have any impact of final space usage. On Mon, Jun 12, 2023 at 6:25 PM Dimpal Gurabani wrote: > Hi all, > > We have a table with 15 columns and ~1M rows. Looking at the output of the > sstabledump tool, it seems

Backporting CASSANDRA-18560 to Cassandra 4.0.10

2023-07-17 Thread manish khandelwal
I see a critical bug https://issues.apache.org/jira/browse/CASSANDRA-18507 fixed in Cassandra 4.0.10. But also see that one critical bug https://issues.apache.org/jira/browse/CASSANDRA-18560 introduced and that is going to be fixed in 4.0.11. Can fix of

Re: Backporting CASSANDRA-18560 to Cassandra 4.0.10

2023-07-17 Thread manish khandelwal
hat is possible. 4.0.10 was already released. We can not > backport anything to what is already released. I believe you need to update > to 4.0.11. > > Regards > > ________ > From: manish khandelwal > Sent: Monday, July 17, 2023 11:40 &g

Re: 4.0 upgrade

2023-07-07 Thread manish khandelwal
Yes repairs are prohibited in mixed version cluster. If you want to monitor please disable repairs till complete upgrade is finished On Sat, Jul 8, 2023, 01:21 Runtian Liu wrote: > Hi, > > We are upgrading our Cassandra clusters from 3.0.27 to 4.0.6 and we > observed some error related to

  1   2   >