Run Mixed Workload using two instances on one node

Hi, We are trying to Decouple our Reporting DB from OLTP. Need urgent help on the feasibility of proposed solution for PRODUCTION. Use Case: Currently, our OLTP and Reporting application and DB are same. Some CF are used for both OLTP and Reporting while others are solely used for

Re: Run Mixed Workload using two instances on one node

something like elasticsearch for the reports? Its designed for that sort of thing.On Mar 17, 2015 8:07 AM, Anuj Wadehra anujw_2...@yahoo.co.in wrote: Hi, We are trying to Decouple our Reporting DB from OLTP. Need urgent help on the feasibility of proposed solution for PRODUCTION. Use Case

Re: Run Mixed Workload using two instances on one node

use cases? OLTP clients use the OLTP DC; reporting clients use the Reporting DC. The reporting CFs could have smaller replication factor on the OLTP DC, if needed, to keep writes and data size minimized there.     Sean Durity – Cassandra Admin, Big Data Team From: Anuj Wadehra [mailto:anujw_2

Re: error deleting messages

. Please look at your logs to make sure that there are no exception. Thanks Anuj Wadehra From:joss Earl j...@rareformnewmedia.com Date:Tue, 24 Mar, 2015 at 6:17 pm Subject:Re: error deleting messages It inserts 100,000 messages, I then start deleting the messages by grabbing chunks of 100 at a time

Re: Impact of removing compactions_in_progress folder

Any comments on exceptions related to unfinished compactions on Cassandra start up? Best way to deal with them? Side effects of deleting compactions_in_progress folder to resolve the issue? Thanks Anuj Wadehra Sent from Yahoo Mail on Android From:Anuj Wadehra anujw_2...@yahoo.co.in Date:Mon

Re: Drawbacks of Major Compaction now that Automatic Tombstone Compaction Exists

are there any disadvantages? Thanks Anuj Wadehra Sent from Yahoo Mail on Android From:Anuj Wadehra anujw_2...@yahoo.co.in Date:Mon, 13 Apr, 2015 at 12:33 am Subject:Re: Drawbacks of Major Compaction now that Automatic Tombstone Compaction Exists No. Anuj Wadehra On Monday, 13 April 2015

Re: Handle Write Heavy Loads in Cassandra 2.0.3

in_memory_compaction_limit_in_mb to 125.Though still getting logs saying compacting large row. We are planning to upgrade to 2.0.14 as 2.1 is not yet production ready. I would appreciate if you could answer the queries posted in initial mail. Thanks Anuj Wadehra Sent from Yahoo Mail on Android

Re: Cassandra tombstones being created by updating rows with TTL's

to that. Thanks Anuj Wadehra Sent from Yahoo Mail on Android From:Laing, Michael michael.la...@nytimes.com Date:Tue, 21 Apr, 2015 at 10:21 pm Subject:Re: Cassandra tombstones being created by updating rows with TTL's Hmm - we read/write with Local Quorum always - I'd recommend that as that is your

Re: Drawbacks of Major Compaction now that Automatic Tombstone Compaction Exists

Great !!! Thanks Andrei !!! Thats the answer I was looking for :) Thanks Anuj Wadehra Sent from Yahoo Mail on Android From:Andrei Ivanov aiva...@iponweb.net Date:Thu, 23 Apr, 2015 at 11:57 pm Subject:Re: Drawbacks of Major Compaction now that Automatic Tombstone Compaction Exists Just

Handle Write Heavy Loads in Cassandra 2.0.3

Hi,   Recently, we discovered that  millions of mutations were getting dropped on our cluster. Eventually, we solved this problem by increasing the value of memtable_flush_writers from 1 to 3. We usually write 3 CFs simultaneously an one of them has 4 Secondary Indexes.   New changes also

Re: Handle Write Heavy Loads in Cassandra 2.0.3

one at high speeds.  Thanks Anuj Wadehra Sent from Yahoo Mail on Android From:Anuj Wadehra anujw_2...@yahoo.co.in Date:Mon, 20 Apr, 2015 at 7:53 pm Subject:Handle Write Heavy Loads in Cassandra 2.0.3 Hi,   Recently, we discovered that  millions of mutations were getting dropped on our

Re: Handle Write Heavy Loads in Cassandra 2.0.3

Any other suggestions on the JVM Tuning and Cassandra config we did to solve the promotion failures during gc? I would appreciate if someone can try to answer our queries mentioned in initial mail? Thanks Anuj Wadehra Sent from Yahoo Mail on Android From:Anuj Wadehra anujw_2

Re: Handle Write Heavy Loads in Cassandra 2.0.3

PM, Anuj Wadehra anujw_2...@yahoo.co.in wrote: Thanks Brice!! We are using Red Hat Linux 6.4..24 cores...64Gb Ram..SSDs in RAID5..CPU are not overloaded even in peak load..I dont think IO is an issue as iostat shows await17 all times..util attrbute in iostat usually increases from 0 to 100

Re: Drawbacks of Major Compaction now that Automatic Tombstone Compaction Exists

. Do you see any problems with Cassandra processing such a huge file?  Is there any Max sstable size beyond which performance etc degrades? What are the implications? Thanks Anuj Wadehra Sent from Yahoo Mail on Android From:Robert Coli rc...@eventbrite.com Date:Fri, 17 Apr, 2015 at 10:55 pm

Re: RE: Cassandra tombstones being created by updating rows with TTL's

major compactions frequently. Tombstone threshold in yaml may be increased keeping in mind read latency needs. Thanks Anuj Wadehra Sent from Yahoo Mail on Android From:Walsh, Stephen stephen.wa...@aspect.com Date:Wed, 22 Apr, 2015 at 7:56 pm Subject:RE: Cassandra tombstones being created

Re: Getting ParNew GC in ... CMS Old Gen ... in logs

then please increase range timeout in yaml.  Thanks Anuj Wadehra Sent from Yahoo Mail on Android From:shahab shahab.mok...@gmail.com Date:Mon, 20 Apr, 2015 at 9:59 pm Subject:Getting ParNew GC in ... CMS Old Gen ... in logs Hi, I am keep getting following line in the cassandra logs, apparently

Re: Handle Write Heavy Loads in Cassandra 2.0.3

Small correction: we are making writes in 5 cf an reading frm one at high speeds.  Thanks Anuj Wadehra Sent from Yahoo Mail on Android From:Anuj Wadehra anujw_2...@yahoo.co.in Date:Mon, 20 Apr, 2015 at 7:53 pm Subject:Handle Write Heavy Loads in Cassandra 2.0.3 Hi,   Recently, we

Re: Getting ParNew GC in ... CMS Old Gen ... in logs

. This is quite normal with CMS unless it happens too frequenltly several times in a sec. I think query time has more to do with read timeout in yaml. Try increasing it. If its a range query then please increase range timeout in yaml.  Thanks Anuj Wadehra Sent from Yahoo Mail on Android

Re: Multiple Cluster vs Multiple DC

Multiple DC is usually useful in case u need Geo Redundancy or have distributed workload. Do u have these clusters at same physical location? If 3 nodes are at diff locations dont u need geo red for disaster ? Whats ur RF for cf?  Thanks Anuj Sent from Yahoo Mail on Android From:Alain

Re: Drawbacks of Major Compaction now that Automatic Tombstone Compaction Exists

Hi Robert, Any comments or suggestions ? Thanks Anuj Wadehra Sent from Yahoo Mail on Android From:Anuj Wadehra anujw_2...@yahoo.co.in Date:Wed, 15 Apr, 2015 at 8:59 am Subject:Re: Drawbacks of Major Compaction now that Automatic Tombstone Compaction Exists Hi Robert, By automatic

Re: Drawbacks of Major Compaction now that Automatic Tombstone Compaction Exists

No. Anuj Wadehra On Monday, 13 April 2015 12:23 AM, Sebastian Estevez sebastian.este...@datastax.com wrote: Have you tried user defined compactions via JMX?On Apr 12, 2015 1:40 PM, Anuj Wadehra anujw_2...@yahoo.co.in wrote: Recently we faced an issue where every repair operation

Drawbacks of Major Compaction now that Automatic Tombstone Compaction Exists

Recently we faced an issue where every repair operation caused addition of hundreds of sstables (CASSANDRA-9146). In order to bring situation under control and make sure reads are not impacted, we were left with no option but to run major compaction to ensure that thousands of tiny sstables are

Impact of removing compactions_in_progress folder

Often we face errors on Cassandra start regarding unfinished compactions particularly when cassandra was abrupty shut down . Problem gets resolved when we delete /var/lib/cassandra/data/system/compactions_in_progress folder. Does deletion of the folder has any impact on  integrity of data or

Re: Drawbacks of Major Compaction now that Automatic Tombstone Compaction Exists

such issue in prod we can go ahead with one time major compaction. Thanks Anuj Wadehra From:Robert Coli rc...@eventbrite.com Date:Tue, 14 Apr, 2015 at 12:54 am Subject:Re: Drawbacks of Major Compaction now that Automatic Tombstone Compaction Exists On Mon, Apr 13, 2015 at 10:52 AM, Anuj

Re: [Cassandra 2.0] truncate table

You can try doing it from cassandra cli. Set consistency level to All and then truncate. Anuj Wadehra Sent from Yahoo Mail on Android From:Parth Setya setya.pa...@gmail.com Date:Thu, 9 Apr, 2015 at 7:31 pm Subject:Re: [Cassandra 2.0] truncate table As per this thread http

Re: Huge number of sstables after adding server to existing cluster

We faced compaction issue with SCTS in 2.0.3. Till we upgrade, we added a dummy read every 1000 writes as workaround . Compaction started happenning in Write only heavy loads. Anuj Wadehra From:graham sanderson gra...@vast.com Date:Sun, 5 Apr, 2015 at 9:35 am Subject:Re: Huge number

Re: replace_address vs add+removenode

themselves so that ranges for which dead node was responsible is taken care. Anuj Wadehra From:Ulrich Geilmann ulrich.geilm...@freiheit.com Date:Wed, 1 Apr, 2015 at 9:58 pm Subject:replace_address vs add+removenode Hi. The documentation suggests to use the replace_address startup parameter

Re: Frequent timeout issues

indexes in cf ..make sure that memtable flush que size is set at least equal to no of indexes.. monitoring iostat and gc logs may help.. Thanks Anuj Wadehra From:Amlan Roy amlan@cleartrip.com Date:Wed, 1 Apr, 2015 at 9:27 pm Subject:Re: Frequent timeout issues Did not see any exception

Re: Bookstrapping new node isn't pulling schema from cluster

some additional streaming. By the way , what problems did you face in Hector when u upgrade to 2.0.14? We are also planning to do that very soon. Thanks Anuj Wadehra Sent from Yahoo Mail on Android From:Eric Stevens migh...@gmail.com Date:Sun, 19 Apr, 2015 at 6:47 pm Subject:Re

Throttle Heavy Read / Write Loads

could please share your suggestions on the above mentioned approach or share any alternatives to the approach. Thanks Anuj Wadehra

Re: Add new DC to cluster

of doing this. Any suggestions against this..please feel free to discuss... Thanks Anuj Wadehra Sent from Yahoo Mail on Android From:Prem Yadav ipremya...@gmail.com Date:Sun, 7 Jun, 2015 at 8:19 pm Subject:Add new DC to cluster Hi, We have an existing cluster consisting of 3 DCs

Hundreds of sstables after every Repair

Hi, We are using 2.0.3 and vnodes. After every repair -pr operation  50+ tiny sstables( 10K) get created. And these sstables never get compacted due to coldness issue. I have raised https://issues.apache.org/jira/browse/CASSANDRA-9146 for this issue but I have been told to upgrade. Till we

Re: Minor Compactions Not Triggered

2015 11:37 PM, Robert Coli rc...@eventbrite.com wrote: On Mon, Jun 1, 2015 at 11:25 AM, Anuj Wadehra anujw_2...@yahoo.co.in wrote: | As per the algorithm shared in the CASSANDRA 6654, I understand that tombstone_threshold property only comes into picture if you have expirying columns

Re: Hundreds of sstables after every Repair

Any suggestions or comments on this one? Thanks Anuj Wadehra Sent from Yahoo Mail on Android From:Anuj Wadehra anujw_2...@yahoo.co.in Date:Sun, 7 Jun, 2015 at 1:54 am Subject:Hundreds of sstables after every Repair Hi, We are using 2.0.3 and vnodes. After every repair -pr operation  50

Re: Hundreds of sstables after every Repair

during repair. When everything is in sync , why repair creates tiny sstables to repair data? Thanks Anuj Wadehra Sent from Yahoo Mail on Android From:Ken Hancock ken.hanc...@schange.com Date:Tue, 9 Jun, 2015 at 8:24 pm Subject:Re: Hundreds of sstables after every Repair I think this came up

Re: Hundreds of sstables after every Repair

: linkedin.com/in/carlosjuzarteroloMobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649www.pythian.com On Tue, Jun 9, 2015 at 5:11 PM, Anuj Wadehra anujw_2...@yahoo.co.in wrote: | We were facing dropped mutations earlier and we increased flush writers. Now there are no dropped mutations

Re: Read Consistency

, 2015 at 2:16 PM, Anuj Wadehra anujw_2...@yahoo.co.in wrote: Why would it fail and with what Thrift error? What if the data didnt exist on any of the nodes..query wont fail if doesnt find data.. Not convinced.. Sent from Yahoo Mail on Android From:arun sirimalla arunsi...@gmail.com Date:Tue, 23

Re: Read Consistency

at 11:21 AM, Philip Thompson philip.thomp...@datastax.com wrote: Anuj, In the first scenario, the data from the single node holding data is returned. The query will not fail if the consistency level is met, even if the read was inconsistent. On Tue, Jun 23, 2015 at 2:16 PM, Anuj Wadehra anujw_2

Read Consistency

Hi, Need to validate my understanding.. RF=3 , Read CL = Quorum What would be returned to the client in following scenarios: Scenario 1: Read query is fired for a key, data is found on one node and not found on other two nodes who are responsible for the token corresponding to key.

Re: Read Consistency

different data with different timestamps. Read query will return the data with most recent timestamp and trigger a read repair in the backend . On Tue, Jun 23, 2015 at 10:57 AM, Anuj Wadehra anujw_2...@yahoo.co.in wrote: Hi, Need to validate my understanding.. RF=3 , Read CL = Quorum What

Adding Nodes With Inconsistent Data

Hi, We faced a scenario where we lost little data after adding 2 nodes in the cluster. There were intermittent dropped mutations in the cluster. Need to verify my understanding how this may have happened to do Root Cause Analysis: Scenario: 3 nodes, RF=3, Read / Write CL= Quorum 1. Due to

Re: Hundreds of sstables after every Repair

NTP output attached. Any other comments on the two queries ? Thanks Anuj Wadehra Sent from Yahoo Mail on Android From:Anuj Wadehra anujw_2...@yahoo.co.in Date:Tue, 9 Jun, 2015 at 10:59 pm Subject:Re: Hundreds of sstables after every Repair Yes. We use NTP. We also thought that drift

Re: Hundreds of sstables after every Repair

NTP output attached. Any other comments on the two queries ? Thanks Anuj Wadehra Sent from Yahoo Mail on Android From:Anuj Wadehra anujw_2...@yahoo.co.in Date:Tue, 9 Jun, 2015 at 10:59 pm Subject:Re: Hundreds of sstables after every Repair Yes. We use NTP. We also thought that drift

Re: Hundreds of sstables after every Repair

NTP output attached. Any other comments on the two queries ? Thanks Anuj Wadehra Sent from Yahoo Mail on Android From:Anuj Wadehra anujw_2...@yahoo.co.in Date:Tue, 9 Jun, 2015 at 10:59 pm Subject:Re: Hundreds of sstables after every Repair Yes. We use NTP. We also thought that drift

Re: Dropped mutation messages

U said RF=1...missed that..so not sure eventual consistency is creating issues.. Thanks Anuj Wadehra Sent from Yahoo Mail on Android From:Anuj Wadehra anujw_2...@yahoo.co.in Date:Sat, 13 Jun, 2015 at 11:31 pm Subject:Re: Dropped mutation messages I think the messages dropped

Re: Dropped mutation messages

memtable_flush_writers to 3 from 1 ( we were writing to multiple cf simultaneously). We also fixed GC issues and reduced total_memtable_size_in_mb to ensure that most memtables are flushed early in heavy write loads. Thanks Anuj Wadehra Sent from Yahoo Mail on Android From:Robert Wille rwi

Minor Compactions Not Triggered

Hi, I am using Cassandra 2.0.3 and we use STCS for all CFs. We have recently faced an issue where sstable count of certain CFs went into THOUSANDS. We realized that every week, when repair -pr ran on each node, it created 50+ tiny sstables of around 1kb. These tables were never compacted during

Re: Minor Compactions Not Triggered

any other sstable Thanks Anuj Sent from Yahoo Mail on Android From:Robert Coli rc...@eventbrite.com Date:Mon, 1 Jun, 2015 at 10:56 pm Subject:Re: Minor Compactions Not Triggered On Sun, May 31, 2015 at 11:37 AM, Anuj Wadehra anujw_2...@yahoo.co.in wrote: 2. We thought that CQL compaction

Re: RE: Cassandra datacenters replication advanced usage

can add Run no and run count with each record. When rows read for a Run matches the count , polling consumer knows that run is fully replicated. Not sure its the best solution. Thanks Anuj Wadehra Sent from Yahoo Mail on Android From:Fabrice Douchant fdouch...@gfproducts.ch Date:Tue, 2 Jun

Re: Different number of records from COPY command

I have never exported data myself but can u just try setting 'consistency ALL' on cqlsh before executing command? Thanks Anuj Wadehra Sent from Yahoo Mail on Android From:Saurabh Chandolia s.chando...@gmail.com Date:Tue, 2 Jun, 2015 at 8:47 pm Subject:Different number of records from COPY

Re: Adding Nodes With Inconsistent Data

, whatever you do, yet following best practices or understanding the internals is the key imho. I would say it is a good question though. Alain. 2015-06-24 19:43 GMT+02:00 Anuj Wadehra anujw_2...@yahoo.co.in: | Hi, We faced a scenario where we lost little data after adding 2 nodes in the cluster

Re: Read Consistency

to generally assume you are not writing data at consistently at QUORUM CL so therefore your reads may be inconsistent. On Tuesday, June 23, 2015, Anuj Wadehra anujw_2...@yahoo.co.in wrote: Thanks..So all of us agree that in scenario 1, data would be returned and that was my initial understanding

Re: Read Consistency

the node with that data may not be queried at all (the other two may). Keep in mind, these scenarios seem to generally assume you are not writing data at consistently at QUORUM CL so therefore your reads may be inconsistent. On Tuesday, June 23, 2015, Anuj Wadehra anujw_2...@yahoo.co.in wrote

Re: Periodic Anti-Entropy repair

You should use nodetool repair -pr on every node to make sure that each range is repaired only once. Thanks Anuj Wadehra Sent from Yahoo Mail on Android From:Brice Argenson bargen...@gmail.com Date:Sat, 23 May, 2015 at 12:31 am Subject:Periodic Anti-Entropy repair Hi everyone, We

Re: Read Consistency

do cleanup or rollback on one node so you need to do it yourself to make sure that integrity of data is maintained in case strong consistency is a requirement. Right? We use Hector by the way and plannning to switch to CQL driver.. Thanks Anuj Wadehra Sent from Yahoo Mail on Android

Re: RE: Manual Indexing With Buckets

Data Team To engage the team, create a request   From: Anuj Wadehra [mailto:anujw_2...@yahoo.co.in] Sent: Friday, July 24, 2015 3:57 AM To: user@cassandra.apache.org Subject: Re: Manual Indexing With Buckets   Can anyone take this one?   Thanks Anuj Sent from Yahoo Mail on Android From:Anuj

Re: Manual Indexing With Buckets

Can anyone take this one? Thanks Anuj Sent from Yahoo Mail on Android From:Anuj Wadehra anujw_2...@yahoo.co.in Date:Thu, 23 Jul, 2015 at 10:57 pm Subject:Manual Indexing With Buckets We have a primary table and we need search capability by batchid column. So we are creating a manual index

Re: if seed is diff on diff nodes, any problem ?

As per my understanding, 2 same seed nodes per dc is the way to go.. If u r not creating two isolated set of nodes in ur cluster, there may be nodes referring each other in a way that everyone is able to know everyone else.. Anuj Sent from Yahoo Mail on Android From:Chris Mawata

Manual Indexing With Buckets

We have a primary table and we need search capability by batchid column. So we are creating a manual index for search by batch id. We are using buckets to restrict a row size in batch id index table to 50mb. As batch size may vary drastically ( ie one batch id may be associated to 100k row keys

Best Practise for Updating Index and Reporting Tables

We have a transaction table,3 manually created index tables and few tables for reporting.  One option is to go for atomic batch mutations so that for each transaction every index table and other reporting tables are updated synchronously.  Other option is to update other tables async, there

Re: Best Practise for Updating Index and Reporting Tables

consistency. The only thing an atomic batch guarantees is that all of the statements in the batch will eventually be executed. Both approaches are eventually consistent, so you have to deal with inconsistency either way. On Jul 23, 2015, at 11:46 AM, Anuj Wadehra anujw_2...@yahoo.co.in wrote: We

Re: RE: Manual Indexing With Buckets

)? Is a row in the primary only associated with one batch?     Sean Durity – Cassandra Admin, Big Data Team To engage the team, create a request   From: Anuj Wadehra [mailto:anujw_2...@yahoo.co.in] Sent: Friday, July 24, 2015 3:57 AM To: user@cassandra.apache.org Subject: Re: Manual Indexing

Re: Unbalanced disk load

Moreover, if you are using SSDs keeping data directories and commitlog on separate disks wont provide much benefit. As Nate said, relying on RAID with RF=1 is not good design. Cassandra replicas provide greater fault tolerance and HA as they are on different nodes.  Thanks Anuj Sent

Re: Cassandra OOM on joining existing ring

We faced similar issue where we had 60k sstables due to coldness bug in 2.0.3. We solved it by following Datastax recommendation for Production at http://docs.datastax.com/en/cassandra/1.2/cassandra/install/installRecommendSettings.html : Step 1 : Add the following line to /etc/sysctl.conf :

Re: Repair time comparison for Cassandra 2.1.11

Ok. I dont have much experience with 2.1 as we are on 2.0.x. Are you using sequential repair? If yes, parallel repair can be faster but you need to make sure that your application has sufficient room to run when cluster is running repair. Are you observing any WARN or ERROR messages in logs

Re: Repair time comparison for Cassandra 2.1.11

Repair can take long time if you have lota of inconaistent data. If you havent restarted nodes yet, you can  run nodetool tpstats command on all nodes to make sure that there no mutation drops. Thanks Anuj Sent from Yahoo Mail on Android From:"badr...@tuta.io" Date:Sun, 15

Re: Repair time comparison for Cassandra 2.1.11

For the error, you can see  http://www.scriptscoop.net/t/3bac9a3307ac/cassandra-lost-notification-from-nodetool-repair.html Lost notification should not be a problem.please see  https://issues.apache.org/jira/browse/CASSANDRA-7909 Infact, we are also currently facing an issue where merkle

Re: handling down node cassandra 2.0.15

Sis you set the JVM_OPTS to replace address? That is usually the error I get when I forget to set the replace_address on Cassandra-env.   JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=address_of_dead_node     From: Anishek Agarwal [mailto:anis...@gmail.com] Sent: Monday, November 16, 2015

Re: handling down node cassandra 2.0.15

Hi Abhishek, In my opinion, you already have data and bootstrapping is not needed here. You can set auto_bootstrap to false in Cassandra.yaml and once the cassandra is rebooted, you should run repair to fix the inconsistent data. ThanksAnuj On Monday, 16 November 2015 10:34 PM, Josh

Re: Repair Hangs while requesting Merkle Trees

different TCP stack tuning for cross-DC, notably your buffer sizes, window params, cassandra-specific stuff like otc_coalescing_strategy, inter_dc_tcp_nodelay, etc. On Sat, Nov 14, 2015 at 10:35 AM, Anuj Wadehra <anujw_2...@yahoo.co.in> wrote: One more observation.We observed that there are

Re: Repair Hangs while requesting Merkle Trees

? In OutboundTcpConnection.java,  when isTimeOut method always returns false for non-droppable verb such as Merkle Tree Request(verb=REPAIR_MESSAGE),why increasing request timeout solved problem on one occasion ? Thanks Anuj Wadehra On Thursday, 12 November 2015 2:35 AM, Anuj Wadehra <anujw_2...@yahoo.co.in>

Repair Hangs while requesting Merkle Trees

method always returns false for non-droppable verb such as Merkle Tree Request(verb=REPAIR_MESSAGE),why increasing request timeout solved problem on one occasion ? Thanks Anuj Wadehra

Re: handling down node cassandra 2.0.15

ectories and set auto_bootstrap = false and then restart the node, followed by repair on this machine right ? thanks anishek On Mon, Nov 16, 2015 at 11:40 PM, Anuj Wadehra <anujw_2...@yahoo.co.in> wrote: Hi Abhishek, In my opinion, you already have data and bootstrapping is not nee

Re: Replication of data over 2 Datacentre's, when one node fails we get replica issues

Hi Walsh, My comments: 1. Keeping RF at 2 and CL at LOCAL_QUORUM would not give you any additional fault tolerance. You wont be able to afford a single node failure with RF=2. I would suggest keeping it at 3 so that you can tolerate a single node failure. Your query failed because RF=2 and

Re: Repair Hangs while requesting Merkle Trees

ow much data can be in-flight between acknowledgements, and the default size is pitiful for any decent   network size. Google around for TCP tuning/buffer tuning and you should find some good resources. On Mon, Nov 16, 2015 at 5:23 PM, Anuj Wadehra <anujw_2...@yahoo.co.in> wrote: Hi Bryan, Than

Re: Repair Hangs while requesting Merkle Trees

+44) (0) 20 8144 9872 On Wed, Nov 11, 2015 at 1:06 PM, Anuj Wadehra <anujw_2...@yahoo.co.in> wrote: Hi, we are using 2.0.14. We have 2 DCs at remote locations with 10GBps connectivity.We are able to complete repair (-par -pr) on 5 nodes. On only one node in DC2, we are unable to comple

Re: Repair Hangs while requesting Merkle Trees

roid From:"Anuj Wadehra" <anujw_2...@yahoo.co.in> Date:Sat, 14 Nov, 2015 at 11:59 pm Subject:Re: Repair Hangs while requesting Merkle Trees Thanks Daemeon !! I wil capture the output of netstats and share in next few days. We were thinking of taking tcp dumps also. If its a ne

Re: memtable and sstables

Memtables are for storing writes in memory till they are flushed to disk as sstables and once flushed, space gets released from commit logs too.. If your are updating some columns only that data would be there in memtables not entire row..Dont think of memtables as row cache.. This is my

Re: How to prevent queries being routed to new DC?

Hi Tom, While reading data ( even at CL LOCAL_QUORUM), if data in different nodes required to meet CL in your local cluster doesnt match, data will be read from remote dc for read repair if read_repair_chance is not 0. Imp points: 1.If you are reading and writing at local_quorum you can set

Re: Throttling Cassandra Load

Hi, Any suggestions/comments on approach ? What you guys are doing to keep check on misbehaved clients and restrict Cassandra load. Note: We will be moving to CQL driver but that will take months.  Anuj Sent from Yahoo Mail on Android From:"Anuj Wadehra" <anujw_2...@yahoo.co

Throttling Cassandra Load

Hi, We are using Cassandra 2.0.14 with Hector 1.1.4. Each node in cluster has an application using Hector and a Cassandra instance. I want suggestions on the approach we are taking for throttling Cassandra load. Problem Statement: Misbehaved clients can bring down Cassandra clusters by

Re: Throttling Cassandra Load

:"Robert Coli" <rc...@eventbrite.com> Date:Wed, 23 Sep, 2015 at 2:43 am Subject:Re: Throttling Cassandra Load On Tue, Sep 22, 2015 at 1:06 PM, Anuj Wadehra <anujw_2...@yahoo.co.in> wrote: We are using Cassandra 2.0.14 with Hector 1.1.4. Each node in cluster has an ap

Re: Throttling Cassandra Load

we can think of applying similar throttling with native protocol. Yes CQL driver may provide us some advanced properties for tuning connection pooling and timeout idle connections. Thanks Anuj Sent from Yahoo Mail on Android From:"Anuj Wadehra" <anujw_2...@yahoo.co.in>

Re: Handle Node Failure with Repair -pr

Hi All !!! Any comments on the repair -pr scenarios..please share how you deal with such scenarios.. Thanks Anuj Sent from Yahoo Mail on Android From:"Anuj Wadehra" <anujw_2...@yahoo.co.in> Date:Sat, 5 Dec, 2015 at 12:57 am Subject:Handle Node Failure with Repair -pr Hi

Re: Re: Re: Cassandra Tuning Issue

Hi Jerry, Its great that you got performance improvement. Moreover, I agree with what Graham said. I think that you are using extremely large Heaps with CMS and that too in very odd ratio..Having 40G for new gen and leaving only 20G old gen seems unreasonable..Its hard to believe that you are

Re: Repair Hangs while requesting Merkle Trees

rred) 3. Hinted handoff started for 3rd node (10.X.14.115 ) but hint replay timed-out. If it's a network issue then why the issue is only in DC2 and mostly observed on one node. ThanksAnuj On Sunday, 29 November 2015 10:44 PM, Anuj Wadehra <anujw_2...@yahoo.co.in> wrote: Yes. I think

Re: Repair Hangs while requesting Merkle Trees

via its public IP. Thanks Anuj On Tue, 24/11/15, Paulo Motta <pauloricard...@gmail.com> wrote: Subject: Re: Repair Hangs while requesting Merkle Trees To: "user@cassandra.apache.org" <user@cassandra.apache.org>, &q

Re: Repair Hangs while requesting Merkle Trees

Please find attached netstat -t -as output for the node on which repair hung and the node which never got Merkle Tree Request. ThanksAnuj On Sunday, 29 November 2015 11:13 PM, Anuj Wadehra <anujw_2...@yahoo.co.in> wrote: Hi All, I am summarizing the setup, problem & key ob

Re: Re:Re: Re: Re: Cassandra Tuning Issue

Anuj, Thanks! I will retry now! By the way, how to " inform the C* email list as well so that others know" as Jack said? I am sorry I have not do that yet. Thanks jerry At 2015-12-09 01:09:07, "Anuj Wadehra" <anujw_2...@yahoo.co.in> wrote: Hi Jerry, Its great that you got

Handle Node Failure with Repair -pr

Hi Guys !! I need comments on my understanding of repair -pr ..If you are using repair -pr in your cluster then following statements hold true: 1. If a node goes down for long time and your not sure when will it return, you must ensure that subrange repair for the defected node range is done

Re: Thousands of pending compactions using STCS

There was a JIRA that cold sstables are not compacted leading to thousands of sstables. Issue got fixed in 2.0.4. Which version of Cassandra are you using? Anuj Sent from Yahoo Mail on Android From:"Jeff Jirsa" Date:Fri, 11 Dec, 2015 at 10:42 pm Subject:Re:

Re: Thousands of pending compactions using STCS

Sorry I missed the version in your mail..you are on 2.0.16..so it cant be coldness issue.. Anuj  Sent from Yahoo Mail on Android From:"Anuj Wadehra" <anujw_2...@yahoo.co.in> Date:Fri, 11 Dec, 2015 at 10:48 pm Subject:Re: Thousands of pending compactions using STCS There was

Revisit Cassandra EOL Policy

Hi, As per my understanding, a Cassandra version n is implicitly declared EOL when two major versions are released after the version n i.e. when version n + 2 is released. I think the EOL policy must be revisted in interest of the expanding Cassandra user base.  Concerns with current EOL

Formal EOL for 2.0.17 and 2.1.12

Hi, Can someone help me by providing formal dates of EOL for Cassandra 2.0.17 and 2.1.12? ThanksAnuj Sent from Yahoo Mail on Android

Re: New node has high network and disk usage.

Hi Vickrum, I would have proceeded with diagnosis as follows: 1. Analysis of sar report to check system health -cpu memory swap disk etc.  System seems to be overloaded. This is evident from mutation drops. 2. Make sure that  all recommended Cassandra production settings available at Datastax

Re: Revisit Cassandra EOL Policy

how various users are dealing with the situation. Are you upgrading Cassandra every 3-6 mths? How do you cut short your planning,test and release cycles for Cassandra upgrades in your application/products? ThanksAnuj On Tue, 5 Jan, 2016 at 8:04 pm, Anuj Wadehra<anujw_2...@yahoo.co

Re: Revisit Cassandra EOL Policy

Thanks Jack !! "Unfortunately how to get firm agreement  on what criteria should be used to judge "Production Ready" is unclear." The most reliable way of determining production ready is to go to Apache Cassandra website. At any point of time, it clearly states the most stable/production ready

Re: Revisit Cassandra EOL Policy

Yes. I am planning to raise one JIRA for that. Thanks for the suggestion. Anuj On Friday, 8 January 2016 11:06 PM, Michael Shuler wrote: On 01/08/2016 11:27 AM, Jonathan Haddad wrote: > Why wouldn't you keep a bug free version of something in production?  If >

Re: Revisit Cassandra EOL Policy

ersion of something in production?  If I found a version of *anything* that was bug free I don't think I'd ever upgrade again. On Fri, Jan 8, 2016 at 9:18 AM Anuj Wadehra <anujw_2...@yahoo.co.in> wrote: Thanks Robert !!! "I don't run X.Y.Z versions where Z is under 6, so in general this does

Re: Repair Hangs while requesting Merkle Trees

triggered for cross DC nodes..and hints replay being timed-out..Is that an indication of a network issue? I am getting in tough with network team to capture netstats and tcpdump too.. Thanks Anuj On Wed, 18/11/15, Anuj Wadehra <anuj

  1   2   3   >