batchstatement
TL/DR:
- only 1 out of 14 statements in a batch do not mutate the partition..
- no error is logged in the application layer, or Cassandra system.log or
Cassandra debug.log
- using C#, and Datastax latest driver
- cluster is a 1-node, dev setup.
- datastax driver configured with LOCAL_QUORUM at the session, and
statement level.
- using preparedstatements.. 1,000% sure there's no typo.. (but I've been
wrong before)
I have about 14 statements that get batched up together. They're updating
at most 2, maybe 3 denormalized tables.. all the same user object, just
different lookup keys.
To help visualize, the tables look a little like these.. abbreviated..
User table..
CREATE TABLE user (u_id uuid, act_id uuid, ext_id text, dt_created
timeuuid, dt_mod timeuuid, is_group Boolean, first_name text)
Users By Account (or plan)
CREATE TABLE user_by_act (act_id uuid, u_id uuid, first_name text)
User By external identifier
CREATE TABLE user_by_ext (ext_id text, u_id uuid, act_id uuid, first_name
text)
I create a batch that updates all the tables.. various updates are broken
out into separate statements, so for example, there's a statement that
updates the external ID in the 'user' table.
UPDATE user_by_ext SET ext_id = :ext_id WHERE u_id = :u_id
This particular batch has 14 statements total, across all 3 tables. They
are only updating at most 3 partitions.. a single partition may have 4 or
more statements to update various parts of the partition. e.g. first name
and last name are a single statement added to the batch.
Here's the problem... of those 14 statements.. across the 3 partitions...
ONE and ONLY ONE update doesn't work.. Absolutely every other discreet
update in the whole batch works.
List boundStatements = new
List();
// *
// user table
boundStatements.Add(SessionManager.UserInsertStatement.Bind(new
{ u_id= user.UserId, act_id = user.ActId, dt_created = nowId, dt_mod =
nowId, is_everyone = user.IsEveryone, is_group = user.IsGroup }));
if (!string.IsNullOrWhiteSpace(user.ExtId))
//
// this statement gets added to the list.. it is part of the batch
// but it NEVER updates the actual field in the databse.
// I have moved it around.. up, down... the only thing that works
// is if I call execute on the first binding above, and then add the rest
// of these as a separate batch.
boundStatements.Add(SessionManager.UserUpdateExtIdStatement.Bind(new { u_id
= user.UserId, ext_id = user.ExtId, dt_mod = nowId }));
//
if (!string.IsNullOrWhiteSpace(user.Email))
boundStatements.Add(SessionManager.UserUpdateEmailStatement.Bind(new { u_id
= user.UserId, email = user.Email, dt_mod = nowId }));
BoundStatement userProfile =
CreateUserProfileBoundStatement(nowId, user);
if (userProfile != null)
boundStatements.Add(userProfile);
// *
// user_by_act table
CreateUserAccountInsertBoundStatements(boundStatements, user,
nowId);
// *
// user_by_ext table
if (!string.IsNullOrWhiteSpace(user.ExtId))
{
boundStatements.Add(SessionManager.UserExtInsertStatement.Bind(new { ext_id
= user.ExtId, act_id = user.ActId, dt_created = nowId, dt_mod = nowId,
is_group = user.IsGroup, u_id = user.UserId }));
BoundStatement userByExtProfile =
CreateUserByExtProfileBoundStatement(nowId, user);
if (userByExtProfile != null)
boundStatements.Add(userByExtProfile);
if (!string.IsNullOrWhiteSpace(user.Email))
boundStatements.Add(SessionManager.UserExtUpdateEmailStatement.Bind(new {
ext_id = user.ExtId, email = user.Email, dt_mod = nowId }));
}
--
Randy Lynn
rl...@getavail.com
office:
859.963.1616 <+1-859-963-1616> ext 202
163 East Main Street - Lexington, KY 40507 - USA
<https://www.getavail.com/> getavail.com <https://www.getavail.com/>
Re: [EXTERNAL] Re: JVM Heap erratic
> > NEW_HEAP: 25 to 50 % - nothing to do with CPU core contrary to > documentation/comments in the file imho > > MaxTenuringThreshold: 15 - From 1 all the way up to 15, that's what gave > me the best results in the past, it reduces major GC and makes the most of > New Gen/minor GC, that are less impacting, but still "stop the world GC". > Default is 1, which is often way to short to expire objects... > > SurvarorRatio: 2 to 8 - controls survivor spaces size. It will be: > 'Survivor total space = New Gen Size / (SurvarorRatio + 2)'. Dividing by 2 > you have the size of each survivor. Here it will depend how fast the Eden > space is allocated. Increasing the survivor space will disminuish the Eden > space (where new objects are allocated) and there is a tradeoff here as > well and a balance to find. > > I would try with these settings on a canary node: > > HEAP - 16 GB (if read heavy, if not probably between 8 and 12 GB is > better). > NEW_HEAP - 50% of the heap (4 - 8GB) > > MaxTenuringThreshold: 15 > > SurvarorRatio: 4, > > > > When testing GC, there is not a better way than using a canary node, pick > one rack and node(s) you want in this rack to test. This should not impact > availability or consistency. If you're able to reproduce the workload > perfectly in the staging cluster, it's perfect but I don't know much > companies/people able to do this and the use of a canary node should be > safe :). > > I could probably share some thoughts on what the cluster really needs, > rather than making guesses and suggesting a somewhat arbitrary tuning, as I > did above if you would share a gc.log file with us from one of the nodes. > Garbage Collection tuning is a bit tricky, but a good tuning can divide > latency while cutting in the number of host. I have seen impressives > changes in the past really. > There is a lot of details in this log file about where is the biggest > pressure, the allocation rate, the GC duration distribution for each type > of GC, etc. With this, I could see where the pressure is and suggest how to > work on it. > > Be aware that extra GC is also sometimes the consequence (and not a cause) > of an issue. Due to pending requests, wide partitions, ongoing compactions, > repairs or an intensive workload, GC can pressure can increase and mask > another underlying, and root issue. You might want to check that the > cluster is healthy other than GC, as a lot of distinct internal parts of > Cassandra have an impact on the GC. > > > > Hope that helps, > > > > C*heers, > > --- > > Alain Rodriguez - @arodream - al...@thelastpickle.com > > France / Spain > > > > The Last Pickle - Apache Cassandra Consulting > > http://www.thelastpickle.com > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.thelastpickle.com&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=_N23kKi9SY0Tt1hzPDfk1n9Mg6eqAQOI_sIwbUN_L88&s=pcPsSYYtEJTtrRfRCwYCziiHKlKMtY-Zn4x_ZQmHDZQ&e=> > > > > > > 2018-06-28 23:19 GMT+01:00 Elliott Sims : > > Odd. Your "post-GC" heap level seems a lot lower than your max, which > implies that you should be OK with ~10GB. I'm guessing either you're > genuinely getting a huge surge in needed heap and running out, or it's > falling behind and garbage is building up. If the latter, there might be > some tweaking you can do. Probably worth turning on GC logging and digging > through exactly what's happening. > > > > CMS is kind of hard to tune and can have problems with heap fragmentation > since it doesn't compact, but if it's working for you I'd say stick with it. > > > > On Thu, Jun 28, 2018 at 3:14 PM, Randy Lynn wrote: > > Thanks for the feedback.. > > > > Getting tons of OOM lately.. > > > > You mentioned overprovisioned heap size... well... > > tried 8GB = OOM > > tried 12GB = OOM > > tried 20GB w/ G1 = OOM (and long GC pauses usually over 2 secs) > > tried 20GB w/ CMS = running > > > > we're java 8 update 151. > > 3.11.1. > > > > We've got one table that's got a 400MB partition.. that's the max.. the > 99th is < 100MB, and 95th < 30MB.. > > So I'm not sure that I'm overprovisioned, I'm just not quite yet to the > heap size based on our partition sizes. > > All queries use cluster key, so I'm not accidentally reading a whole > partition. > > The last place I'm looking - which maybe should be the first - is > tombstones. > > > > sorry for the afternoon rant! thanks for your
rebuild on running node
Anyone ever make stupid mistakes? :) TL/DR: I ran rebuild on a node that is already up and running in an existing data center.. what happens? This is what I did... Assume I have DC_syndey and adding DC_sydney_new But also have a DC_us.. from a node in DC_sydney_new I intended to type "rebuild -- DC_sydney" What I actually did was from one of the nodes in DC_us I typed "rebuild -- DC_sydney I cancelled the process The node in US is showing a increased "load" right now.. What are the consequences? Should I run rebuild again? should I run repair? Or is all OK? -- Randy Lynn rl...@getavail.com office: 859.963.1616 <+1-859-963-1616> ext 202 163 East Main Street - Lexington, KY 40507 - USA <https://www.getavail.com/> getavail.com <https://www.getavail.com/>
Re: C* in multiple AWS AZ's
Thanks Alain, Wanted to just circle back on all the above.. Thanks everyone for your help, and input. I'm glad to hear someone else did a site-to-site tunnel with Cassandra between regions. When originally setting up all the docs and information all preached public IP's. I can totally understand that it's probably a throughput / latency problem that the tunnel introduces, but so far it seems to be working well. The tunnel is stable too, which originally I was concerned with hiccups and maintenance on the tunnel. I did go with GPFS. The move went great. The new i3 nodes are pretty awesome - especially the NVMe! I did go with ubuntu 16.04 LTS versus the AWS Linux AMI. So far no stability problems with the NVMe or the i3 in general with the Ubuntu 16.04. On Tue, Jul 3, 2018 at 7:12 AM, Alain RODRIGUEZ wrote: > Hi, > > Nice that you solved the issue. I had some thoughts while reading: > > >> My original thought was a new DC parallel to the current, and then >> decommission the other DC. >> > > I also think this is the best way to go when possible. It can be reverted > any time in the process, respect distributions, you can switch app per app, > and some other advantages. > > Also, I am a bit late on this topic, but using this: > > AP-SYDNEY, and US-EAST.. I'm using Ec2Snitch over a site-to-site tunnel.. >> I'm wanting to move the current US-EAST from AZ 1a to 1e. > > > You could have kept Ec2Snitch and use the 'dc_suffix' option in > 'cassandra-rackdc.properties' > file to create a non-conflicting logical datacenter in the same region. For > example with a 'dc_suffix' set to '-WHATEVER' the new data center created > would have been called 'US-EAST-WHATEVER' :-). > > I know all docs say use ec2multiregion for multi-DC. >> > > Using EC2RegionSnitch brought nothing but troubles to me in the past. You > have to open security groups for both public and private IPs, the number of > rules is limited (or was for me when I did it) and after some headaches, we > did the same, a tunnel between VPCs and used ec2_snitch. > > Yet, now I would keep GPFS if that works fully for you now. It will even > allow you to hybrid the cluster with non AWS hardware or transition out of > AWS if you need at some point in the future. > > As a side note, using i3, you might have to use a recent operating system > (such as Ubuntu 16.04) to have the latest drivers for NVMe. NVMe support in > Ubuntu 14.04 AMI is not reliable. It might be absent or lead to data > corruption. Make sure the OS in use works well with this hardware. > > https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ssd- > instance-store.html > > C*heers, > --- > Alain Rodriguez - @arodream - al...@thelastpickle.com > France / Spain > > The Last Pickle - Apache Cassandra Consulting > http://www.thelastpickle.com > > 2018-06-29 16:35 GMT+01:00 Pradeep Chhetri : > >> Ohh i see now. It makes sense. Thanks a lot. >> >> On Fri, Jun 29, 2018 at 9:17 PM, Randy Lynn wrote: >> >>> data is only lost if you stop the node. between restarts the storage is >>> fine. >>> >>> On Fri, Jun 29, 2018 at 10:39 AM, Pradeep Chhetri >> > wrote: >>> >>>> Isnt NVMe storage an instance storage ie. the data will be lost in case >>>> the instance restarts. How are you going to make sure that there is no data >>>> loss in case instance gets rebooted? >>>> >>>> On Fri, 29 Jun 2018 at 7:00 PM, Randy Lynn wrote: >>>> >>>>> GPFS - Rahul FTW! Thank you for your help! >>>>> >>>>> Yes, Pradeep - migrating to i3 from r3. moving for NVMe storage, I did >>>>> not have the benefit of doing benchmarks.. but we're moving from 1,500 >>>>> IOPS >>>>> so I intrinsically know we'll get better throughput. >>>>> >>>>> On Fri, Jun 29, 2018 at 7:21 AM, Rahul Singh < >>>>> rahul.xavier.si...@gmail.com> wrote: >>>>> >>>>>> Totally agree. GPFS for the win. EC2 multi region snitch is an >>>>>> automation tool like Ansible or Puppet. Unless you have two orders of >>>>>> magnitude more servers than you do now, you don’t need it. >>>>>> >>>>>> Rahul >>>>>> On Jun 29, 2018, 6:18 AM -0400, kurt greaves , >>>>>> wrote: >>>>>> >>>>>> Yes. You would just end up with a rack named differently to the AZ. >>>>>> This is not a problem as racks are just logical. I would reco
Re: [EXTERNAL] Re: consultant recommendations
great feedback everyone!! thanks for the help!! On Fri, Jun 29, 2018 at 7:17 PM, Joe Schwartz wrote: > Aaron Morton at the Last Pickle is solid; he knows his stuff. > > Also, like Sean said, nothing against Instacluster; they are good folks > too.! > > Joe > > > Joseph B. Schwartz > Western Region Sales Director > Mobile: 408-316-0289 > > On Fri, Jun 29, 2018 at 11:46 AM, Durity, Sean R < > sean_r_dur...@homedepot.com> wrote: > >> I haven’t ever hired a Cassandra consultant, but the company named The >> Last Pickle (yes, an odd name) has some outstanding Cassandra experts. Not >> sure how they work, but worth a mention here. >> >> >> >> Nothing against Instacluster. There are great folks there, too. >> >> >> >> >> >> Sean Durity >> >> >> >> *From:* Evelyn Smith >> *Sent:* Friday, June 29, 2018 1:54 PM >> *To:* user@cassandra.apache.org >> *Subject:* [EXTERNAL] Re: consultant recommendations >> >> >> >> Hey Randy >> >> >> >> Instaclustr provides consulting services for Cassandra as well as managed >> services if you are looking to offload the admin burden. >> >> >> >> https://www.instaclustr.com/services/cassandra-consulting/ >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.instaclustr.com_services_cassandra-2Dconsulting_&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=RwebTlJAmTPu7McYCKfiVJi1fMdZkv1UCyDM_9xSuN0&s=96Keil0PP3CAqd0mOG7PdP0wEaXkIcHrVSqd3BfzJQQ&e=> >> >> >> >> Alternatively, send me an email at evelyn.ba...@instaclustr.com and I’d >> be happy to chase this up on Monday with the head of consulting (it’s >> Friday night my time). >> >> >> >> Cheers, >> >> Evelyn. >> >> >> >> On 30 Jun 2018, at 2:26 am, Randy Lynn wrote: >> >> >> >> Having some OOM issues. Would love to get feedback from the group on what >> companies/consultants you might use? >> >> -- >> >> *Randy Lynn * >> rl...@getavail.com >> >> office: >> >> 859.963.1616 <+1-859-963-1616>ext 202 >> >> 163 East Main Street - Lexington, KY 40507 - USA >> <https://maps.google.com/?q=163+East+Main+Street+-+Lexington,+KY+40507+-+USA&entry=gmail&source=g> >> >> [image: Image removed by sender.] >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.getavail.com_&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=RwebTlJAmTPu7McYCKfiVJi1fMdZkv1UCyDM_9xSuN0&s=9C4Bt-5C4wzgMJVP1YFDulbknSRBqLRn7C7ngwRBoUA&e=> >> >> getavail.com >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.getavail.com_&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=RwebTlJAmTPu7McYCKfiVJi1fMdZkv1UCyDM_9xSuN0&s=9C4Bt-5C4wzgMJVP1YFDulbknSRBqLRn7C7ngwRBoUA&e=> >> >> >> > > -- Randy Lynn rl...@getavail.com office: 859.963.1616 <+1-859-963-1616> ext 202 163 East Main Street - Lexington, KY 40507 - USA <https://www.getavail.com/> getavail.com <https://www.getavail.com/>
consultant recommendations
Having some OOM issues. Would love to get feedback from the group on what companies/consultants you might use? -- Randy Lynn rl...@getavail.com office: 859.963.1616 <+1-859-963-1616> ext 202 163 East Main Street - Lexington, KY 40507 - USA <https://www.getavail.com/> getavail.com <https://www.getavail.com/>
Re: C* in multiple AWS AZ's
data is only lost if you stop the node. between restarts the storage is fine. On Fri, Jun 29, 2018 at 10:39 AM, Pradeep Chhetri wrote: > Isnt NVMe storage an instance storage ie. the data will be lost in case > the instance restarts. How are you going to make sure that there is no data > loss in case instance gets rebooted? > > On Fri, 29 Jun 2018 at 7:00 PM, Randy Lynn wrote: > >> GPFS - Rahul FTW! Thank you for your help! >> >> Yes, Pradeep - migrating to i3 from r3. moving for NVMe storage, I did >> not have the benefit of doing benchmarks.. but we're moving from 1,500 IOPS >> so I intrinsically know we'll get better throughput. >> >> On Fri, Jun 29, 2018 at 7:21 AM, Rahul Singh < >> rahul.xavier.si...@gmail.com> wrote: >> >>> Totally agree. GPFS for the win. EC2 multi region snitch is an >>> automation tool like Ansible or Puppet. Unless you have two orders of >>> magnitude more servers than you do now, you don’t need it. >>> >>> Rahul >>> On Jun 29, 2018, 6:18 AM -0400, kurt greaves , >>> wrote: >>> >>> Yes. You would just end up with a rack named differently to the AZ. This >>> is not a problem as racks are just logical. I would recommend migrating all >>> your DCs to GPFS though for consistency. >>> >>> On Fri., 29 Jun. 2018, 09:04 Randy Lynn, wrote: >>> >>>> So we have two data centers already running.. >>>> >>>> AP-SYDNEY, and US-EAST.. I'm using Ec2Snitch over a site-to-site >>>> tunnel.. I'm wanting to move the current US-EAST from AZ 1a to 1e.. >>>> I know all docs say use ec2multiregion for multi-DC. >>>> >>>> I like the GPFS idea. would that work with the multi-DC too? >>>> What's the downside? status would report rack of 1a, even though in 1e? >>>> >>>> Thanks in advance for the help/thoughts!! >>>> >>>> >>>> On Thu, Jun 28, 2018 at 6:20 PM, kurt greaves >>>> wrote: >>>> >>>>> There is a need for a repair with both DCs as rebuild will not stream >>>>> all replicas, so unless you can guarantee you were perfectly consistent at >>>>> time of rebuild you'll want to do a repair after rebuild. >>>>> >>>>> On another note you could just replace the nodes but use GPFS instead >>>>> of EC2 snitch, using the same rack name. >>>>> >>>>> On Fri., 29 Jun. 2018, 00:19 Rahul Singh, < >>>>> rahul.xavier.si...@gmail.com> wrote: >>>>> >>>>>> Parallel load is the best approach and then switch your Data access >>>>>> code to only access the new hardware. After you verify that there are no >>>>>> local read / writes on the OLD dc and that the updates are only via >>>>>> Gossip, >>>>>> then go ahead and change the replication factor on the key space to have >>>>>> zero replicas in the old DC. Then you can decommissioned. >>>>>> >>>>>> This way you are hundred percent sure that you aren’t missing any new >>>>>> data. No need for a DC to DC repair but a repair is always healthy. >>>>>> >>>>>> Rahul >>>>>> On Jun 28, 2018, 9:15 AM -0500, Randy Lynn , >>>>>> wrote: >>>>>> >>>>>> Already running with Ec2. >>>>>> >>>>>> My original thought was a new DC parallel to the current, and then >>>>>> decommission the other DC. >>>>>> >>>>>> Also my data load is small right now.. I know small is relative >>>>>> term.. each node is carrying about 6GB.. >>>>>> >>>>>> So given the data size, would you go with parallel DC or let the new >>>>>> AZ carry a heavy load until the others are migrated over? >>>>>> and then I think "repair" to cleanup the replications? >>>>>> >>>>>> >>>>>> On Thu, Jun 28, 2018 at 10:09 AM, Rahul Singh < >>>>>> rahul.xavier.si...@gmail.com> wrote: >>>>>> >>>>>>> You don’t have to use EC2 snitch on AWS but if you have already >>>>>>> started with it , it may put a node in a different DC. >>>>>>> >>>>>>> If your data density won’t be ridiculous You could add 3 to >>>>>
Re: C* in multiple AWS AZ's
GPFS - Rahul FTW! Thank you for your help! Yes, Pradeep - migrating to i3 from r3. moving for NVMe storage, I did not have the benefit of doing benchmarks.. but we're moving from 1,500 IOPS so I intrinsically know we'll get better throughput. On Fri, Jun 29, 2018 at 7:21 AM, Rahul Singh wrote: > Totally agree. GPFS for the win. EC2 multi region snitch is an automation > tool like Ansible or Puppet. Unless you have two orders of magnitude more > servers than you do now, you don’t need it. > > Rahul > On Jun 29, 2018, 6:18 AM -0400, kurt greaves , > wrote: > > Yes. You would just end up with a rack named differently to the AZ. This > is not a problem as racks are just logical. I would recommend migrating all > your DCs to GPFS though for consistency. > > On Fri., 29 Jun. 2018, 09:04 Randy Lynn, wrote: > >> So we have two data centers already running.. >> >> AP-SYDNEY, and US-EAST.. I'm using Ec2Snitch over a site-to-site tunnel.. >> I'm wanting to move the current US-EAST from AZ 1a to 1e.. >> I know all docs say use ec2multiregion for multi-DC. >> >> I like the GPFS idea. would that work with the multi-DC too? >> What's the downside? status would report rack of 1a, even though in 1e? >> >> Thanks in advance for the help/thoughts!! >> >> >> On Thu, Jun 28, 2018 at 6:20 PM, kurt greaves >> wrote: >> >>> There is a need for a repair with both DCs as rebuild will not stream >>> all replicas, so unless you can guarantee you were perfectly consistent at >>> time of rebuild you'll want to do a repair after rebuild. >>> >>> On another note you could just replace the nodes but use GPFS instead of >>> EC2 snitch, using the same rack name. >>> >>> On Fri., 29 Jun. 2018, 00:19 Rahul Singh, >>> wrote: >>> >>>> Parallel load is the best approach and then switch your Data access >>>> code to only access the new hardware. After you verify that there are no >>>> local read / writes on the OLD dc and that the updates are only via Gossip, >>>> then go ahead and change the replication factor on the key space to have >>>> zero replicas in the old DC. Then you can decommissioned. >>>> >>>> This way you are hundred percent sure that you aren’t missing any new >>>> data. No need for a DC to DC repair but a repair is always healthy. >>>> >>>> Rahul >>>> On Jun 28, 2018, 9:15 AM -0500, Randy Lynn , wrote: >>>> >>>> Already running with Ec2. >>>> >>>> My original thought was a new DC parallel to the current, and then >>>> decommission the other DC. >>>> >>>> Also my data load is small right now.. I know small is relative term.. >>>> each node is carrying about 6GB.. >>>> >>>> So given the data size, would you go with parallel DC or let the new AZ >>>> carry a heavy load until the others are migrated over? >>>> and then I think "repair" to cleanup the replications? >>>> >>>> >>>> On Thu, Jun 28, 2018 at 10:09 AM, Rahul Singh < >>>> rahul.xavier.si...@gmail.com> wrote: >>>> >>>>> You don’t have to use EC2 snitch on AWS but if you have already >>>>> started with it , it may put a node in a different DC. >>>>> >>>>> If your data density won’t be ridiculous You could add 3 to different >>>>> DC/ Region and then sync up. After the new DC is operational you can >>>>> remove >>>>> one at a time on the old DC and at the same time add to the new one. >>>>> >>>>> Rahul >>>>> On Jun 28, 2018, 9:03 AM -0500, Randy Lynn , >>>>> wrote: >>>>> >>>>> I have a 6-node cluster I'm migrating to the new i3 types. >>>>> But at the same time I want to migrate to a different AZ. >>>>> >>>>> What happens if I do the "running node replace method" with 1 node at >>>>> a time moving to the new AZ. Meaning, I'll have temporarily; >>>>> >>>>> 5 nodes in AZ 1c >>>>> 1 new node in AZ 1e. >>>>> >>>>> I'll wash-rinse-repeat till all 6 are on the new machine type and in >>>>> the new AZ. >>>>> >>>>> Any thoughts about whether this gets weird with the Ec2Snitch and a RF >>>>> 3? >>>>> >>>>> -- >>>>> Randy Lynn >
Re: Cassandra read/sec and write/sec
Abdul - I use DataDog I track the "latency one minute rate" for both read/writes. But I'm interested to see what others say and if I got that right? On Thu, Jun 28, 2018 at 6:19 PM, Abdul Patel wrote: > Hi all > > We use prometheus to monitor cassandra and then put it on graphana for > dashboard. > Whats the parameter to m3asure throughput of cassandra? > -- Randy Lynn rl...@getavail.com office: 859.963.1616 <+1-859-963-1616> ext 202 163 East Main Street - Lexington, KY 40507 - USA <https://www.getavail.com/> getavail.com <https://www.getavail.com/>
Re: C* in multiple AWS AZ's
So we have two data centers already running.. AP-SYDNEY, and US-EAST.. I'm using Ec2Snitch over a site-to-site tunnel.. I'm wanting to move the current US-EAST from AZ 1a to 1e.. I know all docs say use ec2multiregion for multi-DC. I like the GPFS idea. would that work with the multi-DC too? What's the downside? status would report rack of 1a, even though in 1e? Thanks in advance for the help/thoughts!! On Thu, Jun 28, 2018 at 6:20 PM, kurt greaves wrote: > There is a need for a repair with both DCs as rebuild will not stream all > replicas, so unless you can guarantee you were perfectly consistent at time > of rebuild you'll want to do a repair after rebuild. > > On another note you could just replace the nodes but use GPFS instead of > EC2 snitch, using the same rack name. > > On Fri., 29 Jun. 2018, 00:19 Rahul Singh, > wrote: > >> Parallel load is the best approach and then switch your Data access code >> to only access the new hardware. After you verify that there are no local >> read / writes on the OLD dc and that the updates are only via Gossip, then >> go ahead and change the replication factor on the key space to have zero >> replicas in the old DC. Then you can decommissioned. >> >> This way you are hundred percent sure that you aren’t missing any new >> data. No need for a DC to DC repair but a repair is always healthy. >> >> Rahul >> On Jun 28, 2018, 9:15 AM -0500, Randy Lynn , wrote: >> >> Already running with Ec2. >> >> My original thought was a new DC parallel to the current, and then >> decommission the other DC. >> >> Also my data load is small right now.. I know small is relative term.. >> each node is carrying about 6GB.. >> >> So given the data size, would you go with parallel DC or let the new AZ >> carry a heavy load until the others are migrated over? >> and then I think "repair" to cleanup the replications? >> >> >> On Thu, Jun 28, 2018 at 10:09 AM, Rahul Singh < >> rahul.xavier.si...@gmail.com> wrote: >> >>> You don’t have to use EC2 snitch on AWS but if you have already started >>> with it , it may put a node in a different DC. >>> >>> If your data density won’t be ridiculous You could add 3 to different >>> DC/ Region and then sync up. After the new DC is operational you can remove >>> one at a time on the old DC and at the same time add to the new one. >>> >>> Rahul >>> On Jun 28, 2018, 9:03 AM -0500, Randy Lynn , wrote: >>> >>> I have a 6-node cluster I'm migrating to the new i3 types. >>> But at the same time I want to migrate to a different AZ. >>> >>> What happens if I do the "running node replace method" with 1 node at a >>> time moving to the new AZ. Meaning, I'll have temporarily; >>> >>> 5 nodes in AZ 1c >>> 1 new node in AZ 1e. >>> >>> I'll wash-rinse-repeat till all 6 are on the new machine type and in the >>> new AZ. >>> >>> Any thoughts about whether this gets weird with the Ec2Snitch and a RF 3? >>> >>> -- >>> Randy Lynn >>> rl...@getavail.com >>> >>> office: >>> 859.963.1616 <+1-859-963-1616> ext 202 >>> 163 East Main Street - Lexington, KY 40507 - USA >>> <https://maps.google.com/?q=163+East+Main+Street+-+Lexington,+KY+40507+-+USA&entry=gmail&source=g> >>> >>> <https://www.getavail.com/> getavail.com <https://www.getavail.com/> >>> >>> >> >> >> -- >> Randy Lynn >> rl...@getavail.com >> >> office: >> 859.963.1616 <+1-859-963-1616> ext 202 >> 163 East Main Street - Lexington, KY 40507 - USA >> <https://maps.google.com/?q=163+East+Main+Street+-+Lexington,+KY+40507+-+USA&entry=gmail&source=g> >> >> <https://www.getavail.com/> getavail.com <https://www.getavail.com/> >> >> -- Randy Lynn rl...@getavail.com office: 859.963.1616 <+1-859-963-1616> ext 202 163 East Main Street - Lexington, KY 40507 - USA <https://www.getavail.com/> getavail.com <https://www.getavail.com/>
Re: JVM Heap erratic
Thanks for the feedback.. Getting tons of OOM lately.. You mentioned overprovisioned heap size... well... tried 8GB = OOM tried 12GB = OOM tried 20GB w/ G1 = OOM (and long GC pauses usually over 2 secs) tried 20GB w/ CMS = running we're java 8 update 151. 3.11.1. We've got one table that's got a 400MB partition.. that's the max.. the 99th is < 100MB, and 95th < 30MB.. So I'm not sure that I'm overprovisioned, I'm just not quite yet to the heap size based on our partition sizes. All queries use cluster key, so I'm not accidentally reading a whole partition. The last place I'm looking - which maybe should be the first - is tombstones. sorry for the afternoon rant! thanks for your eyes! On Thu, Jun 28, 2018 at 5:54 PM, Elliott Sims wrote: > It depends a bit on which collector you're using, but fairly normal. Heap > grows for a while, then the JVM decides via a variety of metrics that it's > time to run a collection. G1GC is usually a bit steadier and less sawtooth > than the Parallel Mark Sweep , but if your heap's a lot bigger than needed > I could see it producing that pattern. > > On Thu, Jun 28, 2018 at 9:23 AM, Randy Lynn wrote: > >> I have datadog monitoring JVM heap. >> >> Running 3.11.1. >> 20GB heap >> G1 for GC.. all the G1GC settings are out-of-the-box >> >> Does this look normal? >> >> https://drive.google.com/file/d/1hLMbG53DWv5zNKSY88BmI3Wd0ic >> _KQ07/view?usp=sharing >> >> I'm a C# .NET guy, so I have no idea if this is normal Java behavior. >> >> >> >> -- >> Randy Lynn >> rl...@getavail.com >> >> office: >> 859.963.1616 <+1-859-963-1616> ext 202 >> 163 East Main Street - Lexington, KY 40507 - USA >> <https://maps.google.com/?q=163+East+Main+Street+-+Lexington,+KY+40507+-+USA&entry=gmail&source=g> >> >> <https://www.getavail.com/> getavail.com <https://www.getavail.com/> >> > > -- Randy Lynn rl...@getavail.com office: 859.963.1616 <+1-859-963-1616> ext 202 163 East Main Street - Lexington, KY 40507 - USA <https://www.getavail.com/> getavail.com <https://www.getavail.com/>
JVM Heap erratic
I have datadog monitoring JVM heap. Running 3.11.1. 20GB heap G1 for GC.. all the G1GC settings are out-of-the-box Does this look normal? https://drive.google.com/file/d/1hLMbG53DWv5zNKSY88BmI3Wd0ic_KQ07/view?usp=sharing I'm a C# .NET guy, so I have no idea if this is normal Java behavior. -- Randy Lynn rl...@getavail.com office: 859.963.1616 <+1-859-963-1616> ext 202 163 East Main Street - Lexington, KY 40507 - USA <https://www.getavail.com/> getavail.com <https://www.getavail.com/>
Re: C* in multiple AWS AZ's
Already running with Ec2. My original thought was a new DC parallel to the current, and then decommission the other DC. Also my data load is small right now.. I know small is relative term.. each node is carrying about 6GB.. So given the data size, would you go with parallel DC or let the new AZ carry a heavy load until the others are migrated over? and then I think "repair" to cleanup the replications? On Thu, Jun 28, 2018 at 10:09 AM, Rahul Singh wrote: > You don’t have to use EC2 snitch on AWS but if you have already started > with it , it may put a node in a different DC. > > If your data density won’t be ridiculous You could add 3 to different DC/ > Region and then sync up. After the new DC is operational you can remove one > at a time on the old DC and at the same time add to the new one. > > Rahul > On Jun 28, 2018, 9:03 AM -0500, Randy Lynn , wrote: > > I have a 6-node cluster I'm migrating to the new i3 types. > But at the same time I want to migrate to a different AZ. > > What happens if I do the "running node replace method" with 1 node at a > time moving to the new AZ. Meaning, I'll have temporarily; > > 5 nodes in AZ 1c > 1 new node in AZ 1e. > > I'll wash-rinse-repeat till all 6 are on the new machine type and in the > new AZ. > > Any thoughts about whether this gets weird with the Ec2Snitch and a RF 3? > > -- > Randy Lynn > rl...@getavail.com > > office: > 859.963.1616 <+1-859-963-1616> ext 202 > 163 East Main Street - Lexington, KY 40507 - USA > <https://maps.google.com/?q=163+East+Main+Street+-+Lexington,+KY+40507+-+USA&entry=gmail&source=g> > > <https://www.getavail.com/> getavail.com <https://www.getavail.com/> > > -- Randy Lynn rl...@getavail.com office: 859.963.1616 <+1-859-963-1616> ext 202 163 East Main Street - Lexington, KY 40507 - USA <https://www.getavail.com/> getavail.com <https://www.getavail.com/>
C* in multiple AWS AZ's
I have a 6-node cluster I'm migrating to the new i3 types. But at the same time I want to migrate to a different AZ. What happens if I do the "running node replace method" with 1 node at a time moving to the new AZ. Meaning, I'll have temporarily; 5 nodes in AZ 1c 1 new node in AZ 1e. I'll wash-rinse-repeat till all 6 are on the new machine type and in the new AZ. Any thoughts about whether this gets weird with the Ec2Snitch and a RF 3? -- Randy Lynn rl...@getavail.com office: 859.963.1616 <+1-859-963-1616> ext 202 163 East Main Street - Lexington, KY 40507 - USA <https://www.getavail.com/> getavail.com <https://www.getavail.com/>
