RE: system.size_estimates - safe to remove sstables?

2018-03-12 Thread Kunal Gangakhedkar
No, this is a different cluster. Kunal On 13-Mar-2018 6:27 AM, "Kenneth Brotman" <kenbrot...@yahoo.com.invalid> wrote: Kunal, Is this the GCE cluster you are speaking of in the “Adding new DC?” thread? Kenneth Brotman *From:* Kunal Gangakhedkar [mailto:kgangakhed...@

Re: [EXTERNAL] RE: Adding new DC?

2018-03-12 Thread Kunal Gangakhedkar
ogle and one on Amazon? > > > > Kenneth Brotman > > > > *From:* Kunal Gangakhedkar [mailto:kgangakhed...@gmail.com] > *Sent:* Monday, March 12, 2018 4:24 PM > *To:* user@cassandra.apache.org > *Cc:* Nikhil Soman > *Subject:* Re: [EXTERNAL] RE: Adding new DC? > &

Re: [EXTERNAL] RE: Adding new DC?

2018-03-12 Thread Kunal Gangakhedkar
rap: false' to the google cloud instances (not restarted the service as per the doc). In the AWS instance, I had added 'auto_bootstrap: false' - the doc says we need to do "nodetool rebuild" and hence no automatic bootstrapping. But, haven't gotten to that step yet. Thanks, Kunal &g

Re: [EXTERNAL] RE: Adding new DC?

2018-03-12 Thread Kunal Gangakhedkar
> > > How large is the cluster to migrate (# of nodes and size of data). The > preferred method might depend on how much data needs to move. Is any > application outage acceptable? > > > > Sean Durity > > lord of the (C*) rings (Staff Systems Engineer – Cassandra) >

Re: [EXTERNAL] RE: Adding new DC?

2018-03-12 Thread Kunal Gangakhedkar
> > > Sean Durity > > lord of the (C*) rings (Staff Systems Engineer – Cassandra) > > *From:* Kunal Gangakhedkar [mailto:kgangakhed...@gmail.com] > *Sent:* Sunday, March 11, 2018 10:20 PM > *To:* user@cassandra.apache.org > *Subject:* [EXTERNAL] RE: Adding new DC? > >

RE: Adding new DC?

2018-03-11 Thread Kunal Gangakhedkar
le for us to see? No, they are in different regions - GCE setup is in us-east while AWS setup is in Asia-south (Mumbai) Thanks, Kunal Kenneth Brotman *From:* Kunal Gangakhedkar [mailto:kgangakhed...@gmail.com] *Sent:* Sunday, March 11, 2018 2:32 PM *To:* user@cassandra.apache.org *Subject:* Adding n

Adding new DC?

2018-03-11 Thread Kunal Gangakhedkar
Hi all, We currently have a cluster in GCE for one of the customers. They want it to be migrated to AWS. I have setup one node in AWS to join into the cluster by following: https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_add_dc_to_cluster_t.html Will add more nodes once the

Re: system.size_estimates - safe to remove sstables?

2018-03-11 Thread Kunal Gangakhedkar
Finally, got a chance to work on it over the weekend. It worked as advertised. :) Thanks a lot, Chris. Kunal On 8 March 2018 at 10:47, Kunal Gangakhedkar <kgangakhed...@gmail.com> wrote: > Thanks a lot, Chris. > > Will try it today/tomorrow and update here. > > Thanks, &

Re: system.size_estimates - safe to remove sstables?

2018-03-07 Thread Kunal Gangakhedkar
Thanks a lot, Chris. Will try it today/tomorrow and update here. Thanks, Kunal On 7 March 2018 at 00:25, Chris Lohfink <clohf...@apple.com> wrote: > While its off you can delete the files in the directory yeah > > Chris > > > On Mar 6, 2018, at 2:35 AM, Kunal

Re: system.size_estimates - safe to remove sstables?

2018-03-06 Thread Kunal Gangakhedkar
ny chance space used by snapshots? What files exist there that are taking > up space? > > > On Mar 5, 2018, at 1:02 AM, Kunal Gangakhedkar <kgangakhed...@gmail.com> > wrote: > > > > Hi all, > > > > I have a 2-node cluster running cassandra 2.1.18. > >

system.size_estimates - safe to remove sstables?

2018-03-04 Thread Kunal Gangakhedkar
Hi all, I have a 2-node cluster running cassandra 2.1.18. One of the nodes has run out of disk space and died - almost all of it shows up as occupied by size_estimates CF. Out of 296GiB, 288GiB shows up as consumed by size_estimates in 'du -sh' output. This is while the other node is chugging

Re: TRUNCATE on a disk almost full - possible?

2017-04-22 Thread Kunal Gangakhedkar
n should be rather fast as it is mostly a file-deletion >> process with some metadata updates. >> >> 2017-04-21 11:21 GMT+02:00 Kunal Gangakhedkar <kgangakhed...@gmail.com>: >> >>> Hi all, >>> >>> We have a CF that's grown too large - it's not

TRUNCATE on a disk almost full - possible?

2017-04-21 Thread Kunal Gangakhedkar
Hi all, We have a CF that's grown too large - it's not getting actively used in the app right now. The on-disk size of the . directory is ~407GB and I have only ~40GB free left on the disk. I understand that if I trigger a TRUNCATE on this CF, cassandra will try to take snapshot. My question: Is

Re: Backups eating up disk space

2017-02-27 Thread Kunal Gangakhedkar
nal On 13 January 2017 at 18:30, Kunal Gangakhedkar <kgangakhed...@gmail.com> wrote: > Great, thanks a lot to all for the help :) > > I finally took the dive and went with Razi's suggestions. > In summary, this is what I did: > >- turn off incremental backups on each of t

Re: Backups eating up disk space

2017-01-13 Thread Kunal Gangakhedkar
arsnapshot* command is provided so that you don’t accidentally delete > actively used files. Last I used *clearsnapshot*, (a very long time > ago), I thought it left behind the directory, but this could have been > fixed in newer versions (so you might want to check that). > >

Re: Backups eating up disk space

2017-01-11 Thread Kunal Gangakhedkar
same > files that are in your data directory, and are created when an sstable is > written to disk. At the time, they take up (almost) no space, so they > aren't a big deal, but when the sstable gets compacted, they stick around, > so they end up not freeing space up. > > > &

Re: Backups eating up disk space

2017-01-10 Thread Kunal Gangakhedkar
the backups? This is my first production deployment - so, still trying to learn. Thanks, Kunal On 10 January 2017 at 21:36, Jonathan Haddad <j...@jonhaddad.com> wrote: > You can just delete them off the filesystem (rm) > > On Tue, Jan 10, 2017 at 8:02 AM Kunal Gangakhedkar < > kg

Backups eating up disk space

2017-01-10 Thread Kunal Gangakhedkar
Hi all, We have a 3-node cassandra cluster with incremental backup set to true. Each node has 1TB data volume that stores cassandra data. The load in the output of 'nodetool status' comes up at around 260GB each node. All our keyspaces use replication factor = 3. However, the df output shows

Re: Cassandra OOM on joining existing ring

2015-07-12 Thread Kunal Gangakhedkar
:44 PM, Kunal Gangakhedkar kgangakhed...@gmail.com wrote: And here is my cassandra-env.sh https://gist.github.com/kunalg/2c092cb2450c62be9a20 Kunal On 11 July 2015 at 00:04, Kunal Gangakhedkar kgangakhed...@gmail.com wrote: From jhat output, top 10 entries for Instance Count for All

Re: Cassandra OOM on joining existing ring

2015-07-10 Thread Kunal Gangakhedkar
Attaching the stack dump captured from the last OOM. Kunal On 10 July 2015 at 13:32, Kunal Gangakhedkar kgangakhed...@gmail.com wrote: Forgot to mention: the data size is not that big - it's barely 10GB in all. Kunal On 10 July 2015 at 13:29, Kunal Gangakhedkar kgangakhed...@gmail.com

Re: Cassandra OOM on joining existing ring

2015-07-10 Thread Kunal Gangakhedkar
Forgot to mention: the data size is not that big - it's barely 10GB in all. Kunal On 10 July 2015 at 13:29, Kunal Gangakhedkar kgangakhed...@gmail.com wrote: Hi, I have a 2 node setup on Azure (east us region) running Ubuntu server 14.04LTS. Both nodes have 8GB RAM. One of the nodes

Cassandra OOM on joining existing ring

2015-07-10 Thread Kunal Gangakhedkar
Hi, I have a 2 node setup on Azure (east us region) running Ubuntu server 14.04LTS. Both nodes have 8GB RAM. One of the nodes (seed node) died with OOM - so, I am trying to add a replacement node with same configuration. The problem is this new node also keeps dying with OOM - I've restarted

Re: Cassandra OOM on joining existing ring

2015-07-10 Thread Kunal Gangakhedkar
. Any large partitions could blow you away. -- Jack Krupansky On Fri, Jul 10, 2015 at 4:22 AM, Kunal Gangakhedkar kgangakhed...@gmail.com wrote: Attaching the stack dump captured from the last OOM. Kunal On 10 July 2015 at 13:32, Kunal Gangakhedkar kgangakhed...@gmail.com wrote: Forgot

Re: Cassandra OOM on joining existing ring

2015-07-10 Thread Kunal Gangakhedkar
, and eBay. On Fri, Jul 10, 2015 at 1:42 PM, Kunal Gangakhedkar kgangakhed...@gmail.com wrote: Thanks, Sebastian. Couple of questions (I'm really new to cassandra): 1. How do I interpret the output of 'nodetool cfstats' to figure out the issues? Any documentation pointer on that would be helpful

Re: Cassandra OOM on joining existing ring

2015-07-10 Thread Kunal Gangakhedkar
And here is my cassandra-env.sh https://gist.github.com/kunalg/2c092cb2450c62be9a20 Kunal On 11 July 2015 at 00:04, Kunal Gangakhedkar kgangakhed...@gmail.com wrote: From jhat output, top 10 entries for Instance Count for All Classes (excluding platform) shows: 2088223 instances of class

Re: Cassandra OOM on joining existing ring

2015-07-10 Thread Kunal Gangakhedkar
customers in 45 countries, DataStax is the database technology and transactional backbone of choice for the worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay. On Fri, Jul 10, 2015 at 12:55 PM, Kunal Gangakhedkar kgangakhed...@gmail.com wrote: I upgraded my instance from 8GB

Re: Cassandra OOM on joining existing ring

2015-07-10 Thread Kunal Gangakhedkar
occupying 193607512 bytes. JFYI. Kunal On 10 July 2015 at 23:49, Kunal Gangakhedkar kgangakhed...@gmail.com wrote: Thanks for quick reply. 1. I don't know what are the thresholds that I should look for. So, to save this back-and-forth, I'm attaching the cfstats output for the keyspace

Re: Cassandra OOM on joining existing ring

2015-07-10 Thread Kunal Gangakhedkar
if your rows and partitions are reasonably small. Any large partitions could blow you away. -- Jack Krupansky On Fri, Jul 10, 2015 at 4:22 AM, Kunal Gangakhedkar kgangakhed...@gmail.com wrote: Attaching the stack dump captured from the last OOM. Kunal On 10 July 2015 at 13:32, Kunal