Re: Expensive to run nodetool status often?
Have a look in org.apache.cassandra.net:type=FailureDetector *Brooke Jensen* VP Technical Operations & Customer Services www.instaclustr.com | support.instaclustr.com <https://support.instaclustr.com/hc/en-us> This email has been sent on behalf of Instaclustr Limited (Australia) and Instaclustr Inc (USA). This email and any attachments may contain confidential and legally privileged information. If you are not the intended recipient, do not copy or disclose its content, but please reply to this email immediately and highlight the error to the sender and then immediately delete the message. On 26 January 2017 at 04:20, Xiaolei Li <xiao...@sumologic.com> wrote: > Thanks for the advice! > > I do export a lot via JMX already. But I couldn't find the equivalent of > the Status column (Up/Down + Normal/Leaving/Joining/Moving) from the > status output. Does anyone know if those are available via JMX? > > Thank you. > > Best, > x. > > On Wed, Jan 25, 2017 at 8:15 AM, Jonathan Haddad <j...@jonhaddad.com> > wrote: > >> You're about to walk down an unfortunate path. I strongly recommend >> getting the information you need for monitoring using JMX. That's actually >> how nodetool gets all it's information. Instead of parsing output, if you >> use JMX, you'll have access to a *ton* of useful (and some not so useful) >> information. >> >> If you aren't familiar, run Cassandra locally and type "jconsole" in your >> terminal. That'll bring up a decent GUI that you can use to browse all the >> available metrics. >> >> You can export metrics a bunch of ways. Jolokia, mx4j, jmx_exporter (for >> prometheus), and I know there's a collectd plugin but I haven't used it, >> might be worth checking out or maybe someone else can weigh in. >> >> Jon >> >> On Wed, Jan 25, 2017 at 7:48 AM Xiaolei Li <xiao...@sumologic.com> wrote: >> >>> I'm planning to run "nodetool status -r" on every node every minute, >>> storing the output in a file, and aggregating it somewhere else for >>> monitoring. >>> >>> Is that a good idea? How expensive is it to be running status every >>> minute. >>> >>> Best, >>> x. >>> >> >
Re: Troubleshooting random node latency spikes
Hi Ted. How long are the latency spikes when they occur? Have you investigated compactions (nodetool compactionstats) during the spike? Are you also seeing large latency spikes in the p95 (95th percentile) metrics? p99 catches outliers, which aren't necessarily always cause for alarm. Are the nodes showing any other signs of stress? CPU, GC, etc? Is there anything pending in nodetool tpstats? Regarding the read repairs, have you tested writing at a higher consistency to see if that changes the number of RR occurring? *Brooke Jensen* VP Technical Operations & Customer Services www.instaclustr.com | support.instaclustr.com <https://support.instaclustr.com/hc/en-us> This email has been sent on behalf of Instaclustr Limited (Australia) and Instaclustr Inc (USA). This email and any attachments may contain confidential and legally privileged information. If you are not the intended recipient, do not copy or disclose its content, but please reply to this email immediately and highlight the error to the sender and then immediately delete the message. On 18 January 2017 at 02:11, <sean_r_dur...@homedepot.com> wrote: > Is this Java 8 with the G1 garbage collector or CMS? With Java 7 and CMS, > garbage collection can cause delays like you are seeing. I haven’t seen > that problem with G1, but garbage collection is where I would start looking. > > > > > > Sean Durity > > *From:* Ted Pearson [mailto:t...@tedpearson.com] > *Sent:* Thursday, January 05, 2017 2:34 PM > *To:* user@cassandra.apache.org > *Subject:* Troubleshooting random node latency spikes > > > > Greetings! > > I'm working on setting up a new cassandra cluster with a write-heavy > workload (50% writes), and I've run into a strange spiky latency problem. > My application metrics showed random latency spikes. I tracked the latency > back to spikes on individual cassandra nodes. > ClientRequest.Latency.Read/Write.p99 > is occasionally jumping on one node at a time to several seconds, instead > of its normal value of around 1000 microseconds. I also noticed > that ReadRepair.RepairedBackground.m1_rate goes from zero to a non-zero > (around 1-2/sec) during the spike on that node. I'm lost as to why these > spikes are happening, hope someone can give me ideas. > > I attempted to test if the ReadRepair metric is causally linked to the > latency spikes, but even when I changed dclocal_read_repair_chance to 0 on > my tables, even though the metrics showed no ReadRepair.Attempted, the > ReadRepair.RepairedBackground metric still went up during latency spikes. > Am I misunderstanding what this metric tracks? I don't understand why it > went up if I turned off read repair. > > I'm currently running 2.2.6 in a dual-datacenter setup. It's patched to > allow metrics to be recency-biased instead of tracking latency over the > entire running of the java process. I'm using STCS. There is a large amount > of data per node, about 500GB currently. I expect each row to be less than > 10KB. It's currently running on way overpowered hardware - 512GB/raid 0 on > nvme/44 cores on 2 sockets. All of my queries (reads and writes) are > LOCAL_ONE and I'm using r=3. > > > > Thanks, > > Ted > > -- > > The information in this Internet Email is confidential and may be legally > privileged. It is intended solely for the addressee. Access to this Email > by anyone else is unauthorized. If you are not the intended recipient, any > disclosure, copying, distribution or any action taken or omitted to be > taken in reliance on it, is prohibited and may be unlawful. When addressed > to our clients any opinions or advice contained in this Email are subject > to the terms and conditions expressed in any applicable governing The Home > Depot terms of business or client engagement letter. The Home Depot > disclaims all responsibility and liability for the accuracy and content of > this attachment and for any damages or losses arising from any > inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other > items of a destructive nature, which may be contained in this attachment > and shall not be liable for direct, indirect, consequential or special > damages in connection with this e-mail message or its attachment. > > -- > > The information in this Internet Email is confidential and may be legally > privileged. It is intended solely for the addressee. Access to this Email > by anyone else is unauthorized. If you are not the intended recipient, any > disclosure, copying, distribution or any action taken or omitted to be > taken in reliance on it, is prohibited and may be unlawful. When addressed > to our clients any opinions or advice contained in this Email are subject > to t
Re: Which version is stable enough for production environment?
Like I said, test in a lower environment first with your data model to be sure. *Brooke Jensen* VP Technical Operations & Customer Services www.instaclustr.com | support.instaclustr.com <https://support.instaclustr.com/hc/en-us> This email has been sent on behalf of Instaclustr Limited (Australia) and Instaclustr Inc (USA). This email and any attachments may contain confidential and legally privileged information. If you are not the intended recipient, do not copy or disclose its content, but please reply to this email immediately and highlight the error to the sender and then immediately delete the message. On 30 November 2016 at 19:58, Benjamin Roth <benjamin.r...@jaumo.com> wrote: > I didn't mean to criticise you. It was meant as a notice for all of those > who are planning to use MVs. > > I already made proposals to solve these issues on the dev list and plan to > test them on our own cluster during the next days. I am currently working > heavily on this as we have big trouble bootstrapping new nodes on our 3.10 > cluster - due to these known issues. > > JFYI. > > 2016-11-30 9:49 GMT+01:00 kurt Greaves <k...@instaclustr.com>: > >> Yes Benjamin, no one said it wouldn't. We're actively backporting things >> as we get time, if you find something you'd like backported raise an issue >> and let us know. We're well aware of the issues affecting MVs, but they >> haven't really been solved anywhere yet. >> >> On 30 November 2016 at 07:54, Benjamin Roth <benjamin.r...@jaumo.com> >> wrote: >> >>> Hi Brooke, >>> >>> Just had a quick look on your code and I will promise that your LTS >>> version will have the same issues with MVs as any other version. >>> For details check CASSANDRA-12905 or CASSANDRA-12888. >>> >>> 2016-11-30 8:35 GMT+01:00 Brooke Jensen <bro...@instaclustr.com>: >>> >>>> 2.1 will be end of life soon. >>>> >>>> We have a number of customers running 3.7 in production and it's quite >>>> stable. However you should always test in a lower environment first with >>>> your data model to be sure. >>>> >>>> If you're interested, we have made available a patched version of 3.7 >>>> <https://www.instaclustr.com/blog/2016/10/19/patched-cassandra-3-7/> >>>> which backports some key patches from 3.9. >>>> https://github.com/instaclustr/cassandra >>>> >>>> >>>> *Brooke Jensen* >>>> VP Technical Operations & Customer Services >>>> www.instaclustr.com | support.instaclustr.com >>>> <https://support.instaclustr.com/hc/en-us> >>>> >>>> This email has been sent on behalf of Instaclustr Limited (Australia) >>>> and Instaclustr Inc (USA). This email and any attachments may contain >>>> confidential and legally privileged information. If you are not the >>>> intended recipient, do not copy or disclose its content, but please reply >>>> to this email immediately and highlight the error to the sender and then >>>> immediately delete the message. >>>> >>>> On 30 November 2016 at 18:20, Benjamin Roth <benjamin.r...@jaumo.com> >>>> wrote: >>>> >>>>> What are the compaction issues / hint corruprions you encountered? Are >>>>> there JIRA tickets for it? >>>>> I am curios cause I use 3.10 (trunk) in production. >>>>> >>>>> For anyone who is planning to use MVs: >>>>> They basically work. We use them in production since some months, BUT >>>>> (it's a quite big one) maintainance is a pain. Bootstrapping and repairs >>>>> may be - depending on the model, config, amount of data - really, really >>>>> painful. I'm currently investigating intensively. >>>>> >>>>> 2016-11-30 3:11 GMT+01:00 Harikrishnan Pillai <hpil...@walmartlabs.com >>>>> >: >>>>> >>>>>> 3.0 has "off the heap memtable" impl removed and if you have a >>>>>> requirement for this,its not available.If you don't have the requirement >>>>>> 3.0.9 can be tried out. 3.9 version we did some testing and find lot >>>>>> issues >>>>>> in compaction,hint corruption etc. >>>>>> >>>>>> Regards >>>>>> >>>>>> Hari >>>>>> >>>>>> >>>>>> -- >>>>>> *From:* Discovery <
Re: Which version is stable enough for production environment?
2.1 will be end of life soon. We have a number of customers running 3.7 in production and it's quite stable. However you should always test in a lower environment first with your data model to be sure. If you're interested, we have made available a patched version of 3.7 <https://www.instaclustr.com/blog/2016/10/19/patched-cassandra-3-7/> which backports some key patches from 3.9. https://github.com/instaclustr/cassandra *Brooke Jensen* VP Technical Operations & Customer Services www.instaclustr.com | support.instaclustr.com <https://support.instaclustr.com/hc/en-us> This email has been sent on behalf of Instaclustr Limited (Australia) and Instaclustr Inc (USA). This email and any attachments may contain confidential and legally privileged information. If you are not the intended recipient, do not copy or disclose its content, but please reply to this email immediately and highlight the error to the sender and then immediately delete the message. On 30 November 2016 at 18:20, Benjamin Roth <benjamin.r...@jaumo.com> wrote: > What are the compaction issues / hint corruprions you encountered? Are > there JIRA tickets for it? > I am curios cause I use 3.10 (trunk) in production. > > For anyone who is planning to use MVs: > They basically work. We use them in production since some months, BUT > (it's a quite big one) maintainance is a pain. Bootstrapping and repairs > may be - depending on the model, config, amount of data - really, really > painful. I'm currently investigating intensively. > > 2016-11-30 3:11 GMT+01:00 Harikrishnan Pillai <hpil...@walmartlabs.com>: > >> 3.0 has "off the heap memtable" impl removed and if you have a >> requirement for this,its not available.If you don't have the requirement >> 3.0.9 can be tried out. 3.9 version we did some testing and find lot issues >> in compaction,hint corruption etc. >> >> Regards >> >> Hari >> >> >> -- >> *From:* Discovery <wl_...@qq.com> >> *Sent:* Tuesday, November 29, 2016 5:59 PM >> *To:* user >> *Subject:* Re: Which version is stable enough for production environment? >> >> Why version 3.x is not recommended? Thanks. >> >> >> -- Original -- >> *From: * "Harikrishnan Pillai";<hpil...@walmartlabs.com>; >> *Date: * Wed, Nov 30, 2016 09:57 AM >> *To: * "user"<user@cassandra.apache.org>; >> *Subject: * Re: Which version is stable enough for production >> environment? >> >> Cassandra 2.1.16 >> >> >> -- >> *From:* Discovery <wl_...@qq.com> >> *Sent:* Tuesday, November 29, 2016 5:42 PM >> *To:* user >> *Subject:* Which version is stable enough for production environment? >> >> Hi Cassandra Experts, >> >> We prepare to deploy Cassandra in production env, but >> we can not confirm which version is stable and recommended, could someone >> in this mail list give the suggestion? Thanks in advance! >> >> >> Best Regards >> Discovery >> 11/30/2016 >> > > > > -- > Benjamin Roth > Prokurist > > Jaumo GmbH · www.jaumo.com > Wehrstraße 46 · 73035 Göppingen · Germany > Phone +49 7161 304880-6 <+49%207161%203048806> · Fax +49 7161 304880-1 > <+49%207161%203048801> > AG Ulm · HRB 731058 · Managing Director: Jens Kammerer >
Re: Cassandra Upgrade
Hi Shalom. That seems like the long way around doing it. If you clear the data from the node then add it back in then you will have to restream and recompact the data again for each node. Is there a particular reason why you would need to do it this way? The way we do it is to update Cassandra on each node as per the steps Ben linked to. Once all nodes are on the newer version you can run upgradesstables. If you have a large cluster and are using racks you can do the upgrade one rack at a time to speed things up. Either way, this should enable you to do the upgrade fairly quickly with no downtime. Regards, *Brooke Jensen* VP Technical Operations & Customer Services www.instaclustr.com | support.instaclustr.com <https://support.instaclustr.com/hc/en-us> This email has been sent on behalf of Instaclustr Limited (Australia) and Instaclustr Inc (USA). This email and any attachments may contain confidential and legally privileged information. If you are not the intended recipient, do not copy or disclose its content, but please reply to this email immediately and highlight the error to the sender and then immediately delete the message. On 29 November 2016 at 21:12, Ben Dalling <b.dall...@locp.co.uk> wrote: > Hi Shalom, > > There is a pretty good write up of the procedure written up here ( > https://docs.datastax.com/en/upgrade/doc/upgrade/cassandra/ > upgrdCassandraDetails.html). Things to highlight are: > > >- Don't have a repair running while carrying out the upgrade (so that >does timebox your upgrade). >- When the upgrade is complete. Run "nodetool upgradesstables" on all >the nodes. > > > Pretty much what you suggested. > > Best wishes, > > Ben > > On 29 November 2016 at 09:52, Shalom Sagges <shal...@liveperson.com> > wrote: > >> Hi Everyone, >> >> Hypothetically speaking, can I add a new node with version 2.2.8 to a >> 2.0.14 cluster? >> Meaning, instead of upgrading the cluster, I'd like to remove a node, >> clear all its data, install 2.2.8 and add it back to the cluster, with the >> process eventually performed on all nodes one by one. >> >> Is this possible? >> >> Thanks! >> >> >> Shalom Sagges >> DBA >> T: +972-74-700-4035 >> <http://www.linkedin.com/company/164748> <http://twitter.com/liveperson> >> <http://www.facebook.com/LivePersonInc> We Create Meaningful Connections >> >> <https://engage.liveperson.com/idc-mobile-first-consumer/?utm_medium=email_source=mkto_campaign=idcsig> >> >> >> This message may contain confidential and/or privileged information. >> If you are not the addressee or authorized to receive this on behalf of >> the addressee you must not use, copy, disclose or take action based on this >> message or any information herein. >> If you have received this message in error, please advise the sender >> immediately by reply email and delete this message. Thank you. >> > > > > -- > *Ben Dalling** MSc, CEng, MBCS CITP* > League of Crafty Programmers Ltd > Mobile: +44 (0) 776 981-1900 > email: b.dall...@locp.co.uk > www: http://www.locp.co.uk > http://www.linkedin.com/in/bendalling >
Re: Cassandra installation best practices
Hi Mehdi, In addition, give some thought to your cluster topology. For maximum fault tolerance and availability I would recommend using at least three nodes with a replication factor of three. Ideally, you should also use Cassandra logical racks. This will reduce the risk of outage and make ongoing management of the cluster a lot easier. *Brooke Jensen* VP Technical Operations & Customer Services www.instaclustr.com | support.instaclustr.com <https://support.instaclustr.com/hc/en-us> This email has been sent on behalf of Instaclustr Limited (Australia) and Instaclustr Inc (USA). This email and any attachments may contain confidential and legally privileged information. If you are not the intended recipient, do not copy or disclose its content, but please reply to this email immediately and highlight the error to the sender and then immediately delete the message. On 18 October 2016 at 04:02, Anuj Wadehra <anujw_2...@yahoo.co.in> wrote: > Hi Mehdi, > > You can refer https://docs.datastax.com/en/landing_page/doc/landing_page/ > recommendedSettings.html . > > Thanks > Anuj > > On Mon, 17 Oct, 2016 at 10:20 PM, Mehdi Bada > > <mehdi.b...@dbi-services.com> wrote: > Hi all, > > It is exist some best practices when installing Cassandra on production > environment? Some standard to follow? For instance, the file system type > etc.. > >
Re: 1) select collection_type whith keywork 'in' . 2)how to using UDF in update?
Hi. Can you please show me the specific error message you are receiving, as well as the schema for this table (you can use the cql command "describe table"). *Brooke Jensen* VP Technical Operations & Customer Services www.instaclustr.com | support.instaclustr.com <https://support.instaclustr.com/hc/en-us> This email has been sent on behalf of Instaclustr Limited (Australia) and Instaclustr Inc (USA). This email and any attachments may contain confidential and legally privileged information. If you are not the intended recipient, do not copy or disclose its content, but please reply to this email immediately and highlight the error to the sender and then immediately delete the message. On 26 June 2016 at 17:25, lowping <lowp...@163.com> wrote: > Hi All: > > > question 1: > > I got a error about this cql， does it fixed already ？？？ > select collection_type where id in (‘a’,’b’) > > question 2: > > I want use UDF in update, but this cql can’t execute. have some advise？？？ > > update table_name set field=my_function(field) where … > > > tnk u so much >
