Re: CassKop : a Cassandra operator for Kubernetes developped by Orange
This all looks really awesome, while i wish there wasn't as much duplicated effort across the different projects it's great to see how different teams are solving this very real problem! Also Attila we do have our cloud offering as you mentioned, but we also have our own k8s operator which is Apache licensed. So awesome work to the teams from Orange, Datastax and Sky and welcome to the fun of building on a hugely popular and constantly changing project :) Most folk don't realize you can't just run a stateful set! On Sat, May 25, 2019, 05:54 Attila Wind wrote: > Maybe my understanding is wrong and I am not really a "deployment guru" > but it looks like to me that > > Orange (https://github.com/Orange-OpenSource/cassandra-k8s-operator, 1 > contributor and 1 commit for now on 2019-05-24) > and sky-uk/cassandra-operator ( > https://github.com/sky-uk/cassandra-operator , it's in alpha phase and > not recommended in production, 3 contributors, 24 commits btw > 2019.03.25-2019.05.21, 32 Issues) > are developing something I could use in my OWN(!) Kubernetes based > solution (even on premise if I want or whatever) > They are both open source. Right? > > While > Datastax and Instaclustr are commercial players and offer the solution in > a tightly-coupled way with Cloud only > (I just took a quick look on Instaclustr but could not even figure out > pricing info for this service... probably I am lame... or not? :-)) > > So this looks to me a nice competition... > What do I miss? > > ps.: maybe Orange and sky-uk/cassandra-operator guys should cooperate..?? > Others are clearly building business around it > > cheers > Attila Wind > > http://www.linkedin.com/in/attilaw > Mobile: +36 31 7811355 > > > On 2019. 05. 24. 20:36, John Sanda wrote: > > There is also > https://github.com/sky-uk/cassandra-operator > > On Fri, May 24, 2019 at 2:34 PM Rahul Singh > wrote: > >> Fantastic! Now there are three teams making k8s operators for C*: >> Datastax, Instaclustr, and now Orange. >> >> rahul.xavier.si...@gmail.com >> >> http://cassandra.link >> >> I'm speaking at #DataStaxAccelerate, the world’s premiere >> #ApacheCassandra conference, and I want to see you there! Use my code >> Singh50 for 50% off your registration. www.datastax.com/accelerate >> >> >> On Fri, May 24, 2019 at 9:07 AM Jean-Armel Luce >> wrote: >> >>> Hi folks, >>> >>> We are excited to announce that CassKop, a Cassandra operator for >>> Kubernetes developped by Orange teams, is now ready for Beta testing. >>> >>> CassKop works as a usual K8S controller (reconcile the real state with a >>> desired state) and automates the Cassandra operations through JMX. All the >>> operations are launched by calling standard K8S APIs (kubectl apply …) or >>> by using a K8S plugin (kubectl casskop …). >>> >>> CassKop is developed in GO, based on CoreOS operator-sdk framework. >>> Main features already available : >>> - deploying a rack aware cluster (or AZ aware cluster) >>> - scaling up & down (including cleanups) >>> - setting and modifying configuration parameters (C* and JVM parameters) >>> - adding / removing a datacenter in Cassandra (all datacenters must be >>> in the same region) >>> - rebuilding nodes >>> - removing node or replacing node (in case of hardware failure) >>> - upgrading C* or Java versions (including upgradesstables) >>> - monitoring (using Prometheus/Grafana) >>> - ... >>> >>> By using local and persistent volumes, it is possible to handle failures >>> or stop/start nodes for maintenance operations with no transfer of data >>> between nodes. >>> Moreover, we can deploy cassandra-reaper in K8S and use it for >>> scheduling repair sessions. >>> For now, we can deploy a C* cluster only as a mono-region cluster. We >>> will work during the next weeks to be able to deploy a C* cluster as a >>> multi regions cluster. >>> >>> Still in the roadmap : >>> - Network encryption >>> - Monitoring (exporting logs and metrics) >>> - backup & restore >>> - multi-regions support >>> >>> We'd be interested to hear you try this and let us know what you think! >>> >>> Please read the description and installation instructions on >>> https://github.com/Orange-OpenSource/cassandra-k8s-operator. >>> For a quick start, you can also follow this step by step guide : >>> https://orange-opensource.github.io/cassandra-k8s-operator/index.html?slides=Slides-CassKop-demo.md#1 >>> >>> >>> The CassKop Team >>> >> -- > > - John > >
Re: Cassandra DCOS | monitoring connection and user activity
To dig a little deeper. It sounds like JMX is just enabled for loopback, if JMX was disabled/unreachable entirely, nodetool wouldn't work at all. You will likely need to customize your Cassandra config to allow remote JMX connections (though this does increase your attack surface and risk from a security perspective). Alternatively, whatever monitoring solution you land on will likely need a local agent to connect to JMX via loopback. To answer your specific monitoring questions, iirc JMX will expose the number of active connections per node, however, it does not show what queries are being executed. Cassandra does support slow query logging (later versions), or you can use query tracing (probabilistic and at the session level) to see what is happening in real time. Though query tracing is fairly heavy from a performance perspective. I generally don't recommend thinking about tracking bad query issues in Cassandra the same way you would with a relational DB. Cassandra does not have long running queries that get stuck, hold resources, hold locks etc. Most bad performance smells from bad queries (as opposed to just an increase in load) will show up in the logs e.g. large batches, queries timing out, slow queries (when slow query logger is enabled), excessive/long GC pauses, etc However, you mention you get timeouts when you increase the load on Cassandra. If you had some monitoring in place to graph reads and writes per second against timeouts you might discover you simply need to add more nodes or look at trying to squeeze more performance out of what you have. Refer to Anups previous thoughts on where to get started with getting operational data out of Cassandra On Wed, Oct 31, 2018 at 8:21 PM Anup Shirolkar < anup.shirol...@instaclustr.com> wrote: > Hi, > > Looks like you need a monitoring for Cassandra but without using JMX. > It is possible to use metric reporting libraries in Cassandra: > https://wiki.apache.org/cassandra/Metrics#Reporting > > I do not have specific experience with using Cassandra on DCOS but, > monitoring with libraries and tools should not be any different. > > There are various options available to establish good monitoring > (Graphite, Prometheus, Grafana) > > Helpful links: > > > https://blog.pythian.com/monitoring-apache-cassandra-metrics-graphite-grafana/ > https://github.com/instaclustr/cassandra-exporter > https://prometheus.io/docs/instrumenting/exporters/ > https://grafana.com/dashboards/5408 > > http://thelastpickle.com/blog/2017/04/05/cassandra-graphite-measurements-filtered.html > > Regards, > > Anup Shirolkar > > > > > On Wed, 31 Oct 2018 at 18:41, Caesar, Maik wrote: > >> Hello All, >> >> have someone experience with monitoring cassandra in DCOS? >> >> If we increase the load to the Casandra in DCOS, the application get >> timeouts and loose the connection and I do not have any information about >> what happened. >> >> Is there a way to get information about the amount of current connection >> and which queries are executed? Cassandra in DCOS has disabled the JMX >> interface and I think the noodetool do not provide such information. >> >> >> >> Regards >> >> Maik >> >> >> >> DXC Technology Company -- This message is transmitted to you by or on >> behalf of DXC Technology Company or one of its affiliates. It is intended >> exclusively for the addressee. The substance of this message, along with >> any attachments, may contain proprietary, confidential or privileged >> information or information that is otherwise legally exempt from >> disclosure. Any unauthorized review, use, disclosure or distribution is >> prohibited. If you are not the intended recipient of this message, you are >> not authorized to read, print, retain, copy or disseminate any part of this >> message. If you have received this message in error, please destroy and >> delete all copies and notify the sender by return e-mail. Regardless of >> content, this e-mail shall not operate to bind DXC Technology Company or >> any of its affiliates to any order or other contract unless pursuant to >> explicit written agreement or government initiative expressly permitting >> the use of e-mail for such purpose. --. >> > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Reliability at Scale Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer
Re: Advantage over Cassandra in Kubernetes
This is a fairly high-level question which could end up going quite deep, but below is a quick summary off the top of my head. You can get a few advantages when running Cassandra in Kubernetes, particularly: - Easy discovery and network connectivity with other services running on K8s - Reproducible, repeatable operations and deployments - A cloud-independent approach to container orchestration, that is supported by all major cloud providers. - Easy backups, deployments, scaling etc via statefulsets or an operator (see https://github.com/instaclustr/cassandra-operator). There are also some rough edges with running Cassandra on Kubernetes: - Failure domain placement with statefulsets is still challenging (v1.12 goes a long way to fixing this - https://kubernetes.io/blog/2018/10/11/topology-aware-volume-provisioning-in-kubernetes/ ) - Getting resource constraints correct and working out scheduling in constrained environments can be maddening. - Only a few, small deployments (that I'm aware of) are running Cassandra in Kubernetes in production. So you will be breaking new ground and encounter problems that haven't been solved before. - The Cassandra examples in the official Kubernetes documentation is not something you want to take into production. Cheers Ben On Thu, Oct 11, 2018 at 6:50 PM Goutham reddy wrote: > Hi, > We are in the process of setting up Cassandra cluster with high > availability. So the debate is installing Cassandra in Kubernetes cluster. > Can someone throw some light, what advantages can I get when created > Cassandra cluster inside Kubernetes cluster. Any comments are highly > appreciated:) > > Thanks and Regards, > Goutham Reddy Aenugu. > -- > Regards > Goutham Reddy > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Reliability at Scale Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer
Re: Jmx_exporter CPU spike
Hi Rajpal I'd invite you to have a look at https://github.com/zegelin/cassandra-exporter Significantly faster (bypasses JMX rpc stuff, 10ms to collect metrics for 300 tables vs 2-3 seconds via JMX), plus the naming/tagging fits far better into the Prometheus world. Still missing a few stats like GC etc, but feel free to submit a PR! Ben On Mon, Jul 9, 2018 at 12:03 AM Rahul Singh wrote: > How often are you polling the JMX? How much of a spike are you seeing in > CPU? > > -- > Rahul Singh > rahul.si...@anant.us > > Anant Corporation > On Jul 5, 2018, 2:45 PM -0500, rajpal reddy , > wrote: > > > we have Qualys security scan running causing the cpu spike. We are seeing > the CPU spike only when Jmx metrics are exposed using Jmx_exporter. tried > setting up imx authentication still see cpu spike. if i stop using jmx > exporter we don’t see any cpu spike. is there any thing we have to tune to > make work with Jmx_exporter? > > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Reliability at Scale Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer
Re: Using K8s to Manage Cassandra in Production
Here is the expectations around compatibility levels https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/csi-new-client-library-procedure.md#client-capabilities Though references to gold, silver, bronze etc seem to have largely gone away... not sure what's going on there? For a full reference just browser through the repo, https://github.com/kubernetes-client/java/blob/master/kubernetes/README.md is a good place to start as is https://github.com/kubernetes-client/java/tree/master/examples The Java driver doesn't have as much of the nice things in https://github.com/kubernetes/client-go/tree/master/tools but it does have some good helper classes in the util package so I guess we spent a little more time wiring things together? Code Generation is done via the jsonschema2pojo maven plugin and we also just have the raw CRD definition in a resource directory. On Wed, May 23, 2018 at 11:23 AM vincent gromakowski < vincent.gromakow...@gmail.com> wrote: > Thanks ! Do you have some pointers on the available features ? I am more > afraid of the lack of custom controller integration, for instance the code > generator... > > 2018-05-23 17:17 GMT+02:00 Ben Bromhead <b...@instaclustr.com>: > >> The official Kubernetes Java driver is actually pretty feature complete, >> if not exactly idiomatic Java... it's only missing full examples to get it >> to GOLD compatibility levels iirc. >> >> A few reasons we went down the Java path: >> >>- Cassandra community engagement was the primary concern. If you are >>a developer in the Cassandra community you have a base level of Java >>knowledge, so it means if you want to work on the Kubernetes operator you >>only have to learn 1 thing, Kubernetes. If the operator was in Go, >>you would then have two things to learn, Go and Kubernetes :) >>- We actually wrote an initial PoC in Go (based off the etcd >>operator, you can find it here >>https://github.com/benbromhead/cassandra-operator-old ), but because >>it was in Go we ended up making architectural decisions simply because >>Go doesn't do JMX, so it felt like we were just fighting different >>ecosystems just to be part of the cool group. >> >> Some other less important points weighed the decision in Java's favour: >> >>- The folk at Instaclustr all know Java, and are productive in it >>from day 1. Go is fun and relatively simple, but not our forte. >>- Mature package management, Generics/inability to write DRY >>code, a million if err statements (: >>- Some other awesome operators/controllers are written in JVM based >>languages. The sparkKubernetes resource manager (which is a k8s >> controller) >>is written in Scala. >> >> >> On Wed, May 23, 2018 at 10:04 AM vincent gromakowski < >> vincent.gromakow...@gmail.com> wrote: >> >>> Why did you choose java for the operator implementation when everybody >>> seems to use the go client (probably for greater functionalities) ? >>> >>> 2018-05-23 15:39 GMT+02:00 Ben Bromhead <b...@instaclustr.com>: >>> >>>> You can get a good way with StatefulSets, but as Tom mentioned there >>>> are still some issues with this, particularly around scaling up and down. >>>> >>>> We are working on an Operator for Apache Cassandra, you can find it >>>> here https://github.com/instaclustr/cassandra-operator. This is a >>>> joint project between Instaclustr, Pivotal and a few other folk. >>>> >>>> Currently it's a work in progress, but we would love any or all early >>>> feedback/PRs/issues etc. Our first GA release will target the following >>>> capabilities: >>>> >>>>- Safe scaling up and down (including decommissioning) >>>>- Backup/restore workflow (snapshots only initially) >>>>- Built in prometheus integration and discovery >>>> >>>> Other features like repair, better PV support, maybe even a nice >>>> dashboard will be on the way. >>>> >>>> >>>> On Wed, May 23, 2018 at 7:35 AM Tom Petracca <tpetra...@palantir.com> >>>> wrote: >>>> >>>>> Using a statefulset should get you pretty far, though will likely be >>>>> less effective than a coreos-style “operator”. Some random points: >>>>> >>>>>- For scale-up: a node shouldn’t report “ready” until it’s in the >>>>>NORMAL state; this will prevent multiple nodes from bootstrapping at >>>>
Re: Using K8s to Manage Cassandra in Production
The official Kubernetes Java driver is actually pretty feature complete, if not exactly idiomatic Java... it's only missing full examples to get it to GOLD compatibility levels iirc. A few reasons we went down the Java path: - Cassandra community engagement was the primary concern. If you are a developer in the Cassandra community you have a base level of Java knowledge, so it means if you want to work on the Kubernetes operator you only have to learn 1 thing, Kubernetes. If the operator was in Go, you would then have two things to learn, Go and Kubernetes :) - We actually wrote an initial PoC in Go (based off the etcd operator, you can find it here https://github.com/benbromhead/cassandra-operator-old ), but because it was in Go we ended up making architectural decisions simply because Go doesn't do JMX, so it felt like we were just fighting different ecosystems just to be part of the cool group. Some other less important points weighed the decision in Java's favour: - The folk at Instaclustr all know Java, and are productive in it from day 1. Go is fun and relatively simple, but not our forte. - Mature package management, Generics/inability to write DRY code, a million if err statements (: - Some other awesome operators/controllers are written in JVM based languages. The sparkKubernetes resource manager (which is a k8s controller) is written in Scala. On Wed, May 23, 2018 at 10:04 AM vincent gromakowski < vincent.gromakow...@gmail.com> wrote: > Why did you choose java for the operator implementation when everybody > seems to use the go client (probably for greater functionalities) ? > > 2018-05-23 15:39 GMT+02:00 Ben Bromhead <b...@instaclustr.com>: > >> You can get a good way with StatefulSets, but as Tom mentioned there are >> still some issues with this, particularly around scaling up and down. >> >> We are working on an Operator for Apache Cassandra, you can find it here >> https://github.com/instaclustr/cassandra-operator. This is a joint >> project between Instaclustr, Pivotal and a few other folk. >> >> Currently it's a work in progress, but we would love any or all early >> feedback/PRs/issues etc. Our first GA release will target the following >> capabilities: >> >>- Safe scaling up and down (including decommissioning) >>- Backup/restore workflow (snapshots only initially) >>- Built in prometheus integration and discovery >> >> Other features like repair, better PV support, maybe even a nice >> dashboard will be on the way. >> >> >> On Wed, May 23, 2018 at 7:35 AM Tom Petracca <tpetra...@palantir.com> >> wrote: >> >>> Using a statefulset should get you pretty far, though will likely be >>> less effective than a coreos-style “operator”. Some random points: >>> >>>- For scale-up: a node shouldn’t report “ready” until it’s in the >>>NORMAL state; this will prevent multiple nodes from bootstrapping at >>> once. >>>- For scale-down: as of now there isn’t a mechanism to know if a pod >>>is getting decommissioned because you’ve permanently lowered replica >>> count, >>>or because it’s just getting bounced/re-scheduled, thus knowing whether >>> or >>>not to decommission is basically impossible. Relevant issue: >>>kubernetes/kubernetes#1462 >>><https://github.com/kubernetes/kubernetes/issues/1462> >>> >>> >>> >>> *From: *Pradeep Chhetri <prad...@stashaway.com> >>> *Reply-To: *"user@cassandra.apache.org" <user@cassandra.apache.org> >>> *Date: *Friday, May 18, 2018 at 10:20 AM >>> *To: *"user@cassandra.apache.org" <user@cassandra.apache.org> >>> *Subject: *Re: Using K8s to Manage Cassandra in Production >>> >>> >>> >>> Hello Hassaan, >>> >>> >>> >>> We use cassandra helm chart[0] for deploying cassandra over kubernetes >>> in production. We have around 200GB cas data. It works really well. You can >>> scale up nodes easily (I haven't tested scaling down). >>> >>> >>> >>> I would say that if you are worried about running cassandra over k8s in >>> production, maybe you should first try setting it for your >>> staging/preproduction and gain confidence over time. >>> >>> >>> >>> I have tested situations where i have killed the host running cassandra >>> container and have seen that container moves to a different node and joins >>> cluster properly. So from my experience its pretty good. No issues till yet.
Re: Using K8s to Manage Cassandra in Production
You can get a good way with StatefulSets, but as Tom mentioned there are still some issues with this, particularly around scaling up and down. We are working on an Operator for Apache Cassandra, you can find it here https://github.com/instaclustr/cassandra-operator. This is a joint project between Instaclustr, Pivotal and a few other folk. Currently it's a work in progress, but we would love any or all early feedback/PRs/issues etc. Our first GA release will target the following capabilities: - Safe scaling up and down (including decommissioning) - Backup/restore workflow (snapshots only initially) - Built in prometheus integration and discovery Other features like repair, better PV support, maybe even a nice dashboard will be on the way. On Wed, May 23, 2018 at 7:35 AM Tom Petracca <tpetra...@palantir.com> wrote: > Using a statefulset should get you pretty far, though will likely be less > effective than a coreos-style “operator”. Some random points: > >- For scale-up: a node shouldn’t report “ready” until it’s in the >NORMAL state; this will prevent multiple nodes from bootstrapping at once. >- For scale-down: as of now there isn’t a mechanism to know if a pod >is getting decommissioned because you’ve permanently lowered replica count, >or because it’s just getting bounced/re-scheduled, thus knowing whether or >not to decommission is basically impossible. Relevant issue: >kubernetes/kubernetes#1462 ><https://github.com/kubernetes/kubernetes/issues/1462> > > > > *From: *Pradeep Chhetri <prad...@stashaway.com> > *Reply-To: *"user@cassandra.apache.org" <user@cassandra.apache.org> > *Date: *Friday, May 18, 2018 at 10:20 AM > *To: *"user@cassandra.apache.org" <user@cassandra.apache.org> > *Subject: *Re: Using K8s to Manage Cassandra in Production > > > > Hello Hassaan, > > > > We use cassandra helm chart[0] for deploying cassandra over kubernetes in > production. We have around 200GB cas data. It works really well. You can > scale up nodes easily (I haven't tested scaling down). > > > > I would say that if you are worried about running cassandra over k8s in > production, maybe you should first try setting it for your > staging/preproduction and gain confidence over time. > > > > I have tested situations where i have killed the host running cassandra > container and have seen that container moves to a different node and joins > cluster properly. So from my experience its pretty good. No issues till yet. > > > > [0]: https://github.com/kubernetes/charts/tree/master/incubator/cassandra > [github.com] > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_kubernetes_charts_tree_master_incubator_cassandra=DwMFaQ=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8=1oh1YI8i5eJD1DFTwooO7U92fFi2fjan6lqP61yAiyo=dupKDpZi0lkjFkqaSd6XaEj5nuY1T5UObgAcXCBqo7A=0WTYStEM1zvh2BQKvnVLRpukxgr0aDLyGffyE1V2xik=> > > > > > > Regards, > > Pradeep > > > > On Fri, May 18, 2018 at 1:01 PM, Павел Сапежко <amelius0...@gmail.com> > wrote: > > Hi, Hassaan! For example we are using C* in k8s in production for our > video surveillance system. Moreover, we are using Ceph RBD as our storage > for cassandra. Today we have 8 C* nodes each manages 2Tb of data. > > > > On Fri, May 18, 2018 at 9:27 AM Hassaan Pasha <hpa...@an10.io> wrote: > > Hi, > > > > I am trying to craft a deployment strategy for deploying and maintaining a > C* cluster. I was wondering if there are actual production deployments of > C* using K8s as the orchestration layer. > > > > I have been given the impression that K8s managing a C* cluster can be a > recipe for disaster, especially if you aren't well versed with the > intricacies of a scale-up/down event. I know use cases where people are > using Mesos or a custom tool built with terraform/chef etc to run their > production clusters but have yet to find a real K8s use case. > > > > *Questions?* > > Is K8s a reasonable choice for managing a production C* cluster? > > Are there documented use cases for this? > > > > Any help would be greatly appreciated. > > > > -- > > Regards, > > > > *Hassaan Pasha* > > -- > > Regrads, > > Pavel Sapezhko > > > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Reliability at Scale Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer
Re: Cassandra Summit 2019 / Cassandra Summit 2018
Distributed Data Day is shaping up to look like a Cassandra related summit. See http://distributeddatasummit.com/2018-sf/speakers You'll most likely see all the same familiar faces and familiar vendors :) However note it is not the official Apache Cassandra Summit and it is being run by a for profit group. On Thu, May 3, 2018 at 5:33 AM Horia Mocioi <horia.moc...@ericsson.com> wrote: > Hello, > > Are there any updates on this event? > > Regards, > Horia > > On tis, 2018-02-27 at 11:42 +, Carlos Rolo wrote: > > Hello all, > > I'm interested planning/organizing a small kinda of NGCC in Lisbon, > Portugal in late May early June. Just waiting for the venue to confirm > possible dates. > > Would be a 1day event kinda last year, is this something people would be > interested? I can push a google form for accessing the interest today. > > > Regards, > > Carlos Juzarte Rolo > Cassandra Consultant / Datastax Certified Architect / Cassandra MVP > > Pythian - Love your data > > rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin: > *linkedin.com/in/carlosjuzarterolo > <http://linkedin.com/in/carlosjuzarterolo>* > Mobile: +351 918 918 100 <+351%20918%20918%20100> > www.pythian.com > > On Tue, Feb 27, 2018 at 11:39 AM, Kenneth Brotman < > kenbrot...@yahoo.com.invalid> wrote: > > Event planning is fun as long as you can pace it out properly. Once you > set a firm date for an event the pressure on you to keep everything on > track is nerve racking. To do something on the order of Cassandra Summit > 2016, I think we are should plan for 2020. It’s too late for 2018 and even > trying to meet the timeline for everything that would have to come together > makes 2019 too nerve racking a target date. The steps should be: > > Form a planning committee > > Bring potential sponsors into the planning early > > Select an event planning vendor to guide us and to do the > heavy lifting for us > > > > In the meantime, we could have a World-wide Distributed Asynchronous > Cassandra Convention which offers four benefits: > > It allows us to address the fact that we are a world-wide > group that needs a way to reach everyone in a way where no one is > geographically disadvantaged > > No travel time, no travel expenses and no ticket fees > makes it accessible to a lot of people that otherwise would have to miss out > > The lower production costs and simpler administrative workload allows us > to reach implementation sooner > > It’s cutting edge, world class innovation like Cassandra > > > > Kenneth Brotman > > > > *From:* Jeff Jirsa [mailto:jji...@gmail.com] > *Sent:* Monday, February 26, 2018 9:38 PM > *To:* cassandra > *Subject:* Re: Cassandra Summit 2019 / Cassandra Summit 2018 > > > > Instaclustr sponsored the 2017 NGCC (Next Gen Cassandra Conference), which > was developer/development focused (vs user focused). > > > > For 2018, we're looking at options for both a developer conference and a > user conference. There's a lot of logistics involved, and I think it's > fairly obvious that most of the PMC members aren't professional event > planners, so it's possible that either/both conferences may not happen, but > we're doing our best to try to put something together. > > > > > > On Mon, Feb 26, 2018 at 3:00 PM, Rahul Singh <rahul.xavier.si...@gmail.com> > wrote: > > I think some of the Instaclustr folks had done one last year which I > really wanted to go to.. Distributed / Async both would be easier to get > people to write papers, make slides, do youtube videos with.. and then we > could do a virtual web conf of the best submissions. > > > On Feb 26, 2018, 1:04 PM -0600, Kenneth Brotman < > kenbrot...@yahoo.com.invalid>, wrote: > > Is there any planning yet for a Cassandra Summit 2019 or Cassandra Summit > 2018 (probably too late)? > > > > Is there a planning committee? > > > > Who wants there to be a Cassandra Summit 2019 and who thinks there is a > better way? > > > > We could try a Cassandra Distributed Summit 2019 where we meet virtually > and perhaps asynchronously, but there would be a lot more energy and > bonding if it’s not virtual. I’m up for any of these. > > > > Kenneth Brotman > > > > > > > > > > -- > > > > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Reliability at Scale Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer
Re: Cassandra datastax cerrification
Certification is only as good as the organizations that recognize it. Identify what you want to get out of certification, whether that certificate will get what you want and make a decision based on that. The delivered content can be the best in the world, but if no one recognizes it then the certification element becomes meaningless. I would be asking any certification provider which companies actually accept it or recognize it as a demonstration of skill. Especially if you are going to pay money for it. On the other hand if you are just looking to build knowledge / get some classroom experience then you can look at the various options with a more subjective approach. This is just my 2c based on a past life in the IT Security world where "certifications" are a massive business and acceptance of certifications can vary place to place and not a commentary on any of the certification providers you directly mentioned :) On Sat, Apr 14, 2018 at 11:09 AM Abdul Patel <abd786...@gmail.com> wrote: > Hi All, > > I am preparing for cassandra certification(dba) orielly has stopped the > cassandra cerrification so the best bet is datastax now ..as per my > knwledge ds201 and 220 should be enough for cerrification and also i am > reading definitive guide on cassandra ..any other material required ? Any > practise test websites? As certification is costly and wanna clear in one > go ... > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Reliability at Scale Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer
Re: Sorl/DSE Spark
Thanks Jeff. On Thu, Apr 12, 2018, 21:37 Jeff Jirsa <jji...@gmail.com> wrote: > Pretty sure Ben meant that datastax produces DSE, not Cassandra, and since > the questions specifically mentions DSE in the subject (implying that the > user is going to be running either solr or spark within DSE to talk to > cassandra), Ben’s recommendation seems quite reasonable to me. > > > > -- > Jeff Jirsa > > > On Apr 12, 2018, at 6:23 PM, Niclas Hedhman <nic...@apache.org> wrote: > > Ben, > > 1. I don't see anything in this thread that is DSE specific, so I think it > belongs here. > > 2. Careful when you say that Datastax produces Cassandra. Cassandra is a > product of Apache Software Foundation, and no one else. You, Ben, should be > very well aware of this, to avoid further trademark issues between Datastax > and ASF. > > Cheers > Niclas Hedhman > Member of ASF > > On Thu, Apr 12, 2018 at 9:57 PM, Ben Bromhead <b...@instaclustr.com> wrote: > >> Folks this is the user list for Apache Cassandra. I would suggest >> redirecting the question to Datastax the commercial entity that produces >> the software. >> >> On Thu, Apr 12, 2018 at 9:51 AM vincent gromakowski < >> vincent.gromakow...@gmail.com> wrote: >> >>> Best practise is to use a dedicated DC for analytics separated from the >>> hot DC. >>> >>> Le jeu. 12 avr. 2018 à 15:45, sha p <shatestt...@gmail.com> a écrit : >>> >>>> Got it. >>>> Thank you so for your detailed explanation. >>>> >>>> Regards, >>>> Shyam >>>> >>>> On Thu, 12 Apr 2018, 17:37 Evelyn Smith, <u5015...@gmail.com> wrote: >>>> >>>>> Cassandra tends to be used in a lot of web applications. It’s loads >>>>> are more natural and evenly distributed. Like people logging on throughout >>>>> the day. And people operating it tend to be latency sensitive. >>>>> >>>>> Spark on the other hand will try and complete it’s tasks as quickly as >>>>> possible. This might mean bulk reading from the Cassandra at 10 times the >>>>> usual operations load, but for only say 5 minutes every half hour (however >>>>> long it takes to read in the data for a job and whenever that job is run). >>>>> In this case during that 5 minutes your normal operations work (customers) >>>>> are going to experience a lot of latency. >>>>> >>>>> This even happens with streaming jobs, every time spark goes to >>>>> interact with Cassandra it does so very quickly, hammers it for reads and >>>>> then does it’s own stuff until it needs to write things out. This might >>>>> equate to intermittent latency spikes. >>>>> >>>>> In theory, you can throttle your reads and writes but I don’t know >>>>> much about this and don’t see people actually doing it. >>>>> >>>>> Regards, >>>>> Evelyn. >>>>> >>>>> On 12 Apr 2018, at 4:30 pm, sha p <shatestt...@gmail.com> wrote: >>>>> >>>>> Evelyn, >>>>> Can you please elaborate on below >>>>> Spark is notorious for causing latency spikes in Cassandra which is >>>>> not great if you are are sensitive to that. >>>>> >>>>> >>>>> On Thu, 12 Apr 2018, 10:46 Evelyn Smith, <u5015...@gmail.com> wrote: >>>>> >>>>>> Are you building a search engine -> Solr >>>>>> Are you building an analytics function -> Spark >>>>>> >>>>>> I feel they are used in significantly different use cases, what are >>>>>> you trying to build? >>>>>> >>>>>> If it’s an analytics functionality that’s seperate from your >>>>>> operations functionality I’d build it in it’s own DC. Spark is notorious >>>>>> for causing latency spikes in Cassandra which is not great if you are are >>>>>> sensitive to that. >>>>>> >>>>>> Regards, >>>>>> Evelyn. >>>>>> >>>>>> On 12 Apr 2018, at 6:55 am, kooljava2 <koolja...@yahoo.com.INVALID> >>>>>> wrote: >>>>>> >>>>>> Hello, >>>>>> >>>>>> We are exploring on configuring Sorl/Spark. Wanted to get input on >>>>>> this. >>>>>> 1) How do we decide which one to use? >>>>>> 2) Do we run this on a DC where there is less workload? >>>>>> >>>>>> Any other suggestion or comments are appreciated. >>>>>> >>>>>> Thank you. >>>>>> >>>>>> >>>>>> >>>>> -- >> Ben Bromhead >> CTO | Instaclustr <https://www.instaclustr.com/> >> +1 650 284 9692 >> Reliability at Scale >> Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer >> > > > > -- > Niclas Hedhman, Software Developer > http://zest.apache.org - New Energy for Java > > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Reliability at Scale Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer
Re: Sorl/DSE Spark
On Thu, Apr 12, 2018, 21:23 Niclas Hedhman <nic...@apache.org> wrote: > Ben, > > 1. I don't see anything in this thread that is DSE specific, so I think it > belongs here. > DSE is literally in the title. > 2. Careful when you say that Datastax produces Cassandra. Cassandra is a > product of Apache Software Foundation, and no one else. You, Ben, should be > very well aware of this, to avoid further trademark issues between Datastax > and ASF. > Given the context and subject, the software I was referring to is DSE. Mind you, it would be hilarious if this email caused more trademark issues with Datastax. > Cheers > Niclas Hedhman > Member of ASF > > On Thu, Apr 12, 2018 at 9:57 PM, Ben Bromhead <b...@instaclustr.com> wrote: > >> Folks this is the user list for Apache Cassandra. I would suggest >> redirecting the question to Datastax the commercial entity that produces >> the software. >> >> On Thu, Apr 12, 2018 at 9:51 AM vincent gromakowski < >> vincent.gromakow...@gmail.com> wrote: >> >>> Best practise is to use a dedicated DC for analytics separated from the >>> hot DC. >>> >>> Le jeu. 12 avr. 2018 à 15:45, sha p <shatestt...@gmail.com> a écrit : >>> >>>> Got it. >>>> Thank you so for your detailed explanation. >>>> >>>> Regards, >>>> Shyam >>>> >>>> On Thu, 12 Apr 2018, 17:37 Evelyn Smith, <u5015...@gmail.com> wrote: >>>> >>>>> Cassandra tends to be used in a lot of web applications. It’s loads >>>>> are more natural and evenly distributed. Like people logging on throughout >>>>> the day. And people operating it tend to be latency sensitive. >>>>> >>>>> Spark on the other hand will try and complete it’s tasks as quickly as >>>>> possible. This might mean bulk reading from the Cassandra at 10 times the >>>>> usual operations load, but for only say 5 minutes every half hour (however >>>>> long it takes to read in the data for a job and whenever that job is run). >>>>> In this case during that 5 minutes your normal operations work (customers) >>>>> are going to experience a lot of latency. >>>>> >>>>> This even happens with streaming jobs, every time spark goes to >>>>> interact with Cassandra it does so very quickly, hammers it for reads and >>>>> then does it’s own stuff until it needs to write things out. This might >>>>> equate to intermittent latency spikes. >>>>> >>>>> In theory, you can throttle your reads and writes but I don’t know >>>>> much about this and don’t see people actually doing it. >>>>> >>>>> Regards, >>>>> Evelyn. >>>>> >>>>> On 12 Apr 2018, at 4:30 pm, sha p <shatestt...@gmail.com> wrote: >>>>> >>>>> Evelyn, >>>>> Can you please elaborate on below >>>>> Spark is notorious for causing latency spikes in Cassandra which is >>>>> not great if you are are sensitive to that. >>>>> >>>>> >>>>> On Thu, 12 Apr 2018, 10:46 Evelyn Smith, <u5015...@gmail.com> wrote: >>>>> >>>>>> Are you building a search engine -> Solr >>>>>> Are you building an analytics function -> Spark >>>>>> >>>>>> I feel they are used in significantly different use cases, what are >>>>>> you trying to build? >>>>>> >>>>>> If it’s an analytics functionality that’s seperate from your >>>>>> operations functionality I’d build it in it’s own DC. Spark is notorious >>>>>> for causing latency spikes in Cassandra which is not great if you are are >>>>>> sensitive to that. >>>>>> >>>>>> Regards, >>>>>> Evelyn. >>>>>> >>>>>> On 12 Apr 2018, at 6:55 am, kooljava2 <koolja...@yahoo.com.INVALID> >>>>>> wrote: >>>>>> >>>>>> Hello, >>>>>> >>>>>> We are exploring on configuring Sorl/Spark. Wanted to get input on >>>>>> this. >>>>>> 1) How do we decide which one to use? >>>>>> 2) Do we run this on a DC where there is less workload? >>>>>> >>>>>> Any other suggestion or comments are appreciated. >>>>>> >>>>>> Thank you. >>>>>> >>>>>> >>>>>> >>>>> -- >> Ben Bromhead >> CTO | Instaclustr <https://www.instaclustr.com/> >> +1 650 284 9692 >> Reliability at Scale >> Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer >> > > > > -- > Niclas Hedhman, Software Developer > http://zest.apache.org - New Energy for Java > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Reliability at Scale Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer
Re: Sorl/DSE Spark
Folks this is the user list for Apache Cassandra. I would suggest redirecting the question to Datastax the commercial entity that produces the software. On Thu, Apr 12, 2018 at 9:51 AM vincent gromakowski < vincent.gromakow...@gmail.com> wrote: > Best practise is to use a dedicated DC for analytics separated from the > hot DC. > > Le jeu. 12 avr. 2018 à 15:45, sha p <shatestt...@gmail.com> a écrit : > >> Got it. >> Thank you so for your detailed explanation. >> >> Regards, >> Shyam >> >> On Thu, 12 Apr 2018, 17:37 Evelyn Smith, <u5015...@gmail.com> wrote: >> >>> Cassandra tends to be used in a lot of web applications. It’s loads are >>> more natural and evenly distributed. Like people logging on throughout the >>> day. And people operating it tend to be latency sensitive. >>> >>> Spark on the other hand will try and complete it’s tasks as quickly as >>> possible. This might mean bulk reading from the Cassandra at 10 times the >>> usual operations load, but for only say 5 minutes every half hour (however >>> long it takes to read in the data for a job and whenever that job is run). >>> In this case during that 5 minutes your normal operations work (customers) >>> are going to experience a lot of latency. >>> >>> This even happens with streaming jobs, every time spark goes to interact >>> with Cassandra it does so very quickly, hammers it for reads and then does >>> it’s own stuff until it needs to write things out. This might equate to >>> intermittent latency spikes. >>> >>> In theory, you can throttle your reads and writes but I don’t know much >>> about this and don’t see people actually doing it. >>> >>> Regards, >>> Evelyn. >>> >>> On 12 Apr 2018, at 4:30 pm, sha p <shatestt...@gmail.com> wrote: >>> >>> Evelyn, >>> Can you please elaborate on below >>> Spark is notorious for causing latency spikes in Cassandra which is not >>> great if you are are sensitive to that. >>> >>> >>> On Thu, 12 Apr 2018, 10:46 Evelyn Smith, <u5015...@gmail.com> wrote: >>> >>>> Are you building a search engine -> Solr >>>> Are you building an analytics function -> Spark >>>> >>>> I feel they are used in significantly different use cases, what are you >>>> trying to build? >>>> >>>> If it’s an analytics functionality that’s seperate from your operations >>>> functionality I’d build it in it’s own DC. Spark is notorious for causing >>>> latency spikes in Cassandra which is not great if you are are sensitive to >>>> that. >>>> >>>> Regards, >>>> Evelyn. >>>> >>>> On 12 Apr 2018, at 6:55 am, kooljava2 <koolja...@yahoo.com.INVALID> >>>> wrote: >>>> >>>> Hello, >>>> >>>> We are exploring on configuring Sorl/Spark. Wanted to get input on >>>> this. >>>> 1) How do we decide which one to use? >>>> 2) Do we run this on a DC where there is less workload? >>>> >>>> Any other suggestion or comments are appreciated. >>>> >>>> Thank you. >>>> >>>> >>>> >>> -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Reliability at Scale Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer
Re: datastax cassandra minimum hardware recommendation
Also, DS charge by core ;) Anecdotally, we run a large fleet of Apache C* nodes on AWS with a good portion of supported instances that run with 16GB of RAM and 4 cores, which is fine for those workloads. On Wed, Apr 4, 2018 at 11:08 AM sujeet jog <sujeet@gmail.com> wrote: > Thanks Alain > > On Wed, Apr 4, 2018 at 3:12 PM, Alain RODRIGUEZ <arodr...@gmail.com> > wrote: > >> Hello. >> >> For questions to Datastax, I recommend you to ask them directly. I often >> had a quick answer and they probably can answer this better than we do :). >> >> Apache Cassandra (and probably DSE-Cassandra) can work with 8 CPU (and >> less!). I would not go much lower though. I believe the memory amount and >> good disk throughputs are more important. It also depends on the >> workload type and intensity, encryption, compression etc. >> >> 8 CPUs is probably just fine if well tuned, and here in the mailing list, >> we 'support' any fancy configuration settings, but with no guarantee on the >> response time and without taking the responsibility for your cluster :). >> >> It reminds me of my own start with Apache Cassandra. I started with >> t1.micro back then on AWS, and people were still helping me here, of course >> after a couple of jokes such as 'you should rather try to play a >> PlayStation 4 game in your Gameboy', that's fair enough I guess :). Well it >> was working in prod and I learned how to tune Apache Cassandra, I had no >> other options to have this working. >> >> Having more CPU probably improves resiliency to some problems and reduces >> the importance of having a cluster perfectly tuned. >> >> Benchmark your workload, test it. This would be the most accurate answer >> here given the details we have. >> >> C*heers, >> --- >> Alain Rodriguez - @arodream - al...@thelastpickle.com >> France / Spain >> >> The Last Pickle - Apache Cassandra Consulting >> http://www.thelastpickle.com >> >> 2018-04-04 9:44 GMT+01:00 sujeet jog <sujeet@gmail.com>: >> >>> the datastax site has a hardware recommendation of 16CPU / 32G RAM for >>> DSE Enterprise, Any idea what is the minimum hardware recommendation >>> supported, can each node be 8CPU and the support covering it ?.. >>> >> >> > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Reliability at Scale Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer
Re: Roadmap for 4.0
ease. I suppose keep this time frame in mind when choosing your tickets. >>> >>> We can aim for an earlier date (midyear?) but I figure the >>> testing/validation/bugfixing period prior to release might drag on a bit so >>> being a bit conservative here. >>> The main goal would be to not let list 1 grow unless we're well ahead, >>> and only cull from it if we're heavily over-committed or we decide the >>> improvement can wait. I assume this all sounds like common sense but >>> figured it's better to spell it out now. >>> >>> >>> NEXT STEPS >>> After 2 weeks/whenever the discussion dies off I'll consolidate all the >>> tickets, relevant comments and follow up with a summary, where we can >>> discuss/nitpick issues and come up with a final list to go ahead with. >>> >>> On a side note, in conjunction with this effort we'll obviously have to >>> do something about validation and testing. I'll keep that out of this email >>> for now, but there will be a follow up so that those of us willing to help >>> validate/test trunk can avoid duplicating effort. >>> >>> REVIEW >>> This is the list of "huge/breaking" tickets that got mentioned in the >>> last roadmap discussion and their statuses. This is not terribly important >>> but just so we can keep in mind what we previously talked about. I think we >>> leave it up to the relevant contributors to decide whether they want to get >>> the still open tickets into 4.0. >>> >>> CASSANDRA-9425 Immutable node-local schema >>> <https://issues.apache.org/jira/browse/CASSANDRA-9425> - Committed >>> CASSANDRA-10699 Strongly consistent schema alterations >>> <https://issues.apache.org/jira/browse/CASSANDRA-10699> - Open, no >>> discussion in quite some time. >>> CASSANDRA-12229 NIO streaming >>> <https://issues.apache.org/jira/browse/CASSANDRA-12229> - Committed >>> CASSANDRA-8457 NIO messaging >>> <https://issues.apache.org/jira/browse/CASSANDRA-8457> - Committed >>> CASSANDRA-12345 Gossip 2.0 >>> <https://issues.apache.org/jira/browse/CASSANDRA-12345> - Open, no sign >>> of any action. >>> CASSANDRA-9754 Make index info heap friendly for large CQL partitions >>> <https://issues.apache.org/jira/browse/CASSANDRA-9754> - In progress >>> but no update in a long time. >>> CASSANDRA-11559 enhanced node representation >>> <https://issues.apache.org/jira/browse/CASSANDRA-11559> - Open, no >>> change since early 2016. >>> CASSANDRA-6246 epaxos >>> <https://issues.apache.org/jira/browse/CASSANDRA-6246> - In progress >>> but no update since Feb 2017. >>> CASSANDRA-7544 storage port configurable per node >>> <https://issues.apache.org/jira/browse/CASSANDRA-7544> - Committed >>> CASSANDRA-5 remove thrift support >>> <https://issues.apache.org/jira/browse/CASSANDRA-5> - Committed >>> CASSANDRA-10857 dropping compact storage >>> <https://issues.apache.org/jira/browse/CASSANDRA-10857> - Committed >>> >>> To start us off... >>> And here are my lists to get us started. >>> 1. >>> CASSANDRA-8460 - Tiered/Cold storage for TWCS >>> <https://issues.apache.org/jira/browse/CASSANDRA-8460> >>> CASSANDRA-12783 - Batchlog redesign >>> <https://issues.apache.org/jira/browse/CASSANDRA-12783> >>> CASSANDRA-11559 - Enchance node representation >>> <https://issues.apache.org/jira/browse/CASSANDRA-11559> >>> CASSANDRA-12344 - Forward writes to replacement node with same >>> address <https://issues.apache.org/jira/browse/CASSANDRA-12344> >>> CASSANDRA-8119 - More expressive Consistency Levels >>> <https://issues.apache.org/jira/browse/CASSANDRA-8119> >>> CASSANDRA-14210 - Optimise SSTables upgrade task scheduling >>> <https://issues.apache.org/jira/browse/CASSANDRA-14210> >>> CASSANDRA-10540 - RangeAwareCompaction >>> <https://issues.apache.org/jira/browse/CASSANDRA-10540> >>> >>> >>> 2: >>> CASSANDRA-10726 - Read repair inserts should not be blocking >>> <https://issues.apache.org/jira/browse/CASSANDRA-10726> >>> CASSANDRA-9754 - Make index info heap friendly for large CQL partitions >>> <https://issues.apache.org/jira/browse/CASSANDRA-9754> >>> CASSANDRA-12294 - LDAP auth >>> <https://issues.apache.org/jira/browse/CASSANDRA-12294> >>> CASSANDRA-12151 - Audit logging >>> <https://issues.apache.org/jira/browse/CASSANDRA-12151> >>> CASSANDRA-10495 - Fix streaming with vnodes >>> <https://issues.apache.org/jira/browse/CASSANDRA-10495> >>> >>> Also, here's some handy JQL to start you off: >>> project = CASSANDRA AND fixVersion in (4.x, 4.0) AND issue in >>> watchedIssues() AND status != Resolved >>> >>> >> > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Reliability at Scale Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer
Re: Amazon Time Sync Service + ntpd vs chrony
I wouldn't 100% rely on your clients to generate timestamps (actually don't 100% rely on timestamps at all!) . Clients tend to be stateless, scaled up and down, stopped, started, ntp takes time to skew a clock and they are more likely to be moved between hypervisor's in cloud environments etc. All these combine to give you more scenarios where clients have unreliable clocks that are not roughly in sync with each other. By far and large the worst time related bugs I've experienced are due to Cassandra clients having the wrong timestamp set for it's writetime. Of course this depends on what you are prioritising... relative accuracy of any given writetime on one row to any other given row or just respecting what the client thinks is right. On Thu, Mar 8, 2018, 21:24 Jeff Jirsa <jji...@gmail.com> wrote: > Clients can race (and go backward), so the more computer answer tends to > be to use LWT/CAS to guarantee state if you have a data model where it > matters. > > > -- > Jeff Jirsa > > > On Mar 8, 2018, at 6:18 PM, Dor Laor <d...@scylladb.com> wrote: > > While NTP on the servers is important, make sure that you use client > timestamps and > not server. Since the last write wins, the data generator should be the > one setting its timestamp. > > On Thu, Mar 8, 2018 at 2:12 PM, Ben Slater <ben.sla...@instaclustr.com> > wrote: > >> It is important to make sure you are using the same NTP servers across >> your cluster - we used to see relatively frequent NTP issues across our >> fleet using default/public NTP servers until (back in 2015) we implemented >> our own NTP pool (see >> https://www.instaclustr.com/apache-cassandra-synchronization/ which >> references some really good and detailed posts from logentries.com on >> the potential issues). >> >> Cheers >> Ben >> >> On Fri, 9 Mar 2018 at 02:07 Michael Shuler <mich...@pbandjelly.org> >> wrote: >> >>> As long as your nodes are syncing time using the same method, that >>> should be good. Don't mix daemons, however, since they may sync from >>> different sources. Whether you use ntpd, openntp, ntpsec, chrony isn't >>> really important, since they are all just background daemons to sync the >>> system clock. There is nothing Cassandra-specific. >>> >>> -- >>> Kind regards, >>> Michael >>> >>> On 03/08/2018 04:15 AM, Kyrylo Lebediev wrote: >>> > Hi! >>> > >>> > Recently Amazon announced launch of Amazon Time Sync Service >>> > ( >>> https://aws.amazon.com/blogs/aws/keeping-time-with-amazon-time-sync-service/ >>> ) >>> > and now it's AWS-recommended way for time sync on EC2 instances >>> > (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html). >>> > It's stated there that chrony is faster / more precise than ntpd. >>> > >>> > Nothing to say correct time sync configuration is very important for >>> any >>> > C* setup. >>> > >>> > Does anybody have positive experience using crony, Amazon Time Sync >>> > Service with Cassandra and/or combination of them? >>> > Any concerns regarding chrony + Amazon Time Sync Service + Cassandra? >>> > Are there any chrony best-practices/custom settings for C* setups? >>> > >>> > Thanks, >>> > Kyrill >>> > >>> >>> >>> - >>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >>> For additional commands, e-mail: user-h...@cassandra.apache.org >>> >>> -- >> >> >> *Ben Slater* >> >> *Chief Product Officer <https://www.instaclustr.com/>* >> >> <https://www.facebook.com/instaclustr> >> <https://twitter.com/instaclustr> >> <https://www.linkedin.com/company/instaclustr> >> >> Read our latest technical blog posts here >> <https://www.instaclustr.com/blog/>. >> >> This email has been sent on behalf of Instaclustr Pty. Limited >> (Australia) and Instaclustr Inc (USA). >> >> This email and any attachments may contain confidential and legally >> privileged information. If you are not the intended recipient, do not copy >> or disclose its content, but please reply to this email immediately and >> highlight the error to the sender and then immediately delete the message. >> > > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Reliability at Scale Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer
Re: C* 3 node issue -Urgent
Just to clarify that behaviour. QUORUM only applies to the default
superuser, subsequent superusers you create later on are still only queried
at LOCAL_ONE. E.g.
protected static ConsistencyLevel consistencyForRole(String role)
{
if (role.equals(DEFAULT_SUPERUSER_NAME))
return ConsistencyLevel.QUORUM;
else
return ConsistencyLevel.LOCAL_ONE;
}
Despite the fact it suggests consistency for a given role... the function
actually gets passed the username not the role (role lookup happens after
authentication iirc).
Best practices would suggest you change the default superuser password to
some long random password and throw it away and use other superuser
accounts. The Cassandra user is only their to bootstrap auth and nothing
else.
If your RF for the system auth table is very high it will not make it
difficult to login, just to change your password :)
On Wed, 6 Sep 2017 at 11:43 Jeff Jirsa <jji...@gmail.com> wrote:
> More explicitly - if you have 60 nodes, setting rf=60 will likely make it
> very difficult for you to log in as a superuser.
>
> --
> Jeff Jirsa
>
>
> > On Sep 6, 2017, at 11:40 AM, Jon Haddad <jonathan.had...@gmail.com>
> wrote:
> >
> > I wouldn’t worry about being meticulous about keeping RF = N as the
> cluster grows. If you had 60 nodes and your auth data was only on 9 you’d
> be completely fine.
> >
> >> On Sep 6, 2017, at 11:36 AM, Cogumelos Maravilha <
> cogumelosmaravi...@sapo.pt> wrote:
> >>
> >> After insert a new node we should:
> >>
> >> ALTER KEYSPACE system_auth WITH REPLICATION = { 'class' : ...
> >> 'replication_factor' : x };
> >>
> >> x = number of nodes in dc
> >>
> >> The default user and password should work:
> >> -u cassandra -p cassandra
> >>
> >> Cheers.
> >>
> >>> On 23-08-2017 11:14, kurt greaves wrote:
> >>> The cassandra user requires QUORUM consistency to be achieved for
> >>> authentication. Normal users only require ONE. I suspect your
> >>> system_auth keyspace has an RF of 1, and the node that owns the
> >>> cassandra users data is down.
> >>>
> >>> Steps to recover:
> >>> 1. Turn off authentication on all the nodes
> >>> 2. Restart the nodes and make sure they are UN
> >>> 3. Alter system_auth to have a higher RF than 1 (3 is probably
> >>> appropriate)
> >>> 4. Turn auth back on and restart
> >>> 5. Create a new user and use that from now on.
> >>>
> >>>
> >>
> >>
> >> -
> >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> >> For additional commands, e-mail: user-h...@cassandra.apache.org
> >>
> >
> >
> > -
> > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: user-h...@cassandra.apache.org
> >
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: Full table scan with cassandra
Apache Cassandra is not great in terms of performance at the moment for batch analytics workloads that require a full table scan. I would look at FiloDB for all the benefits and familiarity of Cassandra with better streaming and analytics performance: https://github.com/filodb/FiloDB There are also some outstanding tickets around improving bulk reads in Cassandra (see https://issues.apache.org/jira/browse/CASSANDRA-9259 for the full gory details), but it appears to be abandonded by the initial set of contributors. On Wed, 16 Aug 2017 at 09:51 Alex Kotelnikov <alex.kotelni...@diginetica.com> wrote: > Hey, > > we are trying Cassandra as an alternative for storage huge stream of data > coming from our customers. > > Storing works quite fine, and I started to validate how retrieval does. We > have two types of that: fetching specific records and bulk retrieval for > general analysis. > Fetching single record works like charm. But it is not so with bulk fetch. > > With a moderately small table of ~2 million records, ~10Gb raw data I > observed very slow operation (using token(partition key) ranges). It takes > minutes to perform full retrieval. We tried a couple of configurations > using virtual machines, real hardware and overall looks like it is not > possible to all table data in a reasonable time (by reasonable I mean that > since we have 1Gbit network 10Gb can be transferred in a couple of minutes > from one server to another and when we have 10+ cassandra servers and 10+ > spark executors total time should be even smaller). > > I tried datastax spark connector. Also I wrote a simple test case using > datastax java driver and see how fetch of 10k records takes ~10s so I > assume that "sequential" scan will take 200x more time, equals ~30 minutes. > > May be we are totally wrong trying to use Cassandra this way? > > -- > > Best Regards, > > > *Alexander Kotelnikov* > > *Team Lead* > > DIGINETICA > Retail Technology Company > > m: +7.921.915.06.28 <+7%20921%20915-06-28> > > *www.diginetica.com <http://www.diginetica.com/>* > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: Answering the question of Cassandra Summit 2017
Thanks for explanation Patrick, I think it helps the community to hear direct from the horses mouth. I also wanted to thank you, your team and Datastax for being the organising force behind the Cassandra Summit for all these years. Politics aside it has been an awesome place for the community to meet, bond and make some great (and sometimes hazy) memories! In terms of the future, I'm pretty positive there will be a place and format for the Apache Cassandra community to meet in person in the new year and the fact that Datastax would be an enthusiastic participant is wonderful. <3 The Instaclustr team On Fri, 11 Aug 2017 at 17:27 Patrick McFadin <pmcfa...@gmail.com> wrote: > Hello Cassandra Community, > > I know this is a hot topic so let me TL;DR for those of you looking for > the short answer. Will there be a Cassandra Summit in 2017? DataStax will > not be holding a Cassandra Summit in 2017, but instead multiple DataStax > Summits in 2018. > > More details. Last year was pretty chaotic for the Cassandra community and > where DataStax fit with the project. I don’t need to re-cap all the drama. > You can go look at the dev and user lists around this time last year if you > want to re-live it. It’s safe to say 2016 is a year I wouldn’t want to do > again for a lot of reasons. Those of us at the Cassandra Summit 2016 knew > it was the end of something, now the question is what’s next and what will > it be? > > Having a place for people together and talk about what we as a community > do is really important. Those of you that know me, know how much I live > that. When we started talking summit inside DataStax, we realized it would > be a hot button issue. When I started talking to people in the community, > it was even more of a hot button issue. Having DataStax run the Cassandra > Summit was going to cause a lot of heartache and would further divide the > community with questions and accusations. It’s just much easier to hold a > DataStax Summit so we are just out there plainly and move forward. > > What DataStax will be doing different. > > We will be moving back to a more regional format instead of the big bang > single event in San Jose starting in 2018. Fun fact. Almost 80% of > attendees of Cassandra Summit were from the Bay Area. That means we have > developers and operators from a lot of other places being excluded which > isn’t cool. We will also be inviting talks from the Cassandra Community. > You don’t have to be a DataStax customer or partner to get on the speaking > list. > > If there is some new group or company that launches a Cassandra Summit, > DataStax will happily be a sponsor. There are some for-profit, professional > conference companies like the Linux Foundation out there that just may and > if so, I’ll see you there. After being involved in making the Cassandra > Summit happen for years, I can say it’s no small effort. > > There it is. Fire away with your questions, comments. All I ask is keep it > respectful because this is a community of amazing people. You have changed > the world over these years and I know it won’t stop. You know I got a hug > for you wherever we just happen to meet. > > Patrick > > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: DSE 5.0 Upgrade
Hi Cass Savy I would suggest directing your question to Datastax, or you could look at an open source alternative such as Apache Cassandra :) Ben On Wed, 10 May 2017 at 22:50 cass savy <casss...@gmail.com> wrote: > Team, > > 1. What is the stable version for DSE 5.0 to upgrade from DSE 4.8.x? > > 2. Is anybody switched to using DSE Unified auth model which enfores to > use one auth policy as primary and other secondary? > > 3. Do I need to use multi-auth/DSE Unified auth for me upgrade to DSE 5.0 > or higher? > Our old clusters are using internal authentication and we learnt that > we have to go with DSE Unifiedauth model as part of upgrade? > > 4. Does anybody migrate from Java driver to Datastax enterprise Java > driver 1.1 or 1.2 recently which will be only driver that will support DSE > 5.0 and above and new auth policies added by DSE? > > Or > > Has anybody made Java driver version 3.2 work with DSE 5.0 or 5.1? > > 4. For AWS, what is prod recommended AMI with CentOS and for DSE 5.x > versions? > > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: Cassandra 2.1.13: Using JOIN_RING=False
A rough guess on the best way to do this with cql, without looking too deeply into it: You would likely have to implement a custom load balancing policy within the driver that only uses coordinator nodes and/or modify refreshNodeListAndTokenMap() behaviour in ControlConnection.java (there should also be an equivalent class in the python driver). On Thu, 11 May 2017 at 08:19 Dikang Gu <dikan...@gmail.com> wrote: > 1. The coordinator still store system data and hints, but they should not > store any user data since they are not part of ring. > 2. We are using coordinator for thrift client. For cql based drivers, they > needs to talk to nodes in the ring, so I think coordinator mode won't work > for them. > > -Dikang > > On Tue, May 9, 2017 at 2:01 PM, Anubhav Kale < > anubhav.k...@microsoft.com.invalid> wrote: > >> Hello, >> >> >> >> With some inspiration from the Cassandra Summit talk from last year, we >> are trying to setup a cluster with coordinator-only nodes. We setup >> join_ring=false in env.sh, disabled auth in YAML and the nodes are able to >> start just fine. However, we’re running into a few problems >> >> >> >> 1] The nodes marked with join_ring=false continue to store data. Why ? >> >> 2] We tried Python driver’s whitelistedpolicy. But we notice message like >> below, so we are not able to send queries to all nodes marked as >> coordinators. We also changed the Scala driver to support whitelisting, but >> see the same thing. What are we missing ? >> >> 3] Is there any way to concretely tell that only coordinator nodes are >> getting requests from clients ? We don’t have OpsCenter. >> >> >> >> Thanks ! >> >> >> >> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection] >> Removing host not found in peers metadata: >> >> 2017-05-09 20:45:25,060 [INFO] cassandra.cluster: Cassandra host >> 10.80.10.128 removed >> >> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: Removing host >> 10.80.10.128 >> >> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection] >> Removing host not found in peers metadata: >> >> 2017-05-09 20:45:25,060 [INFO] cassandra.cluster: Cassandra host >> 10.80.10.127 removed >> >> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: Removing host >> 10.80.10.127 >> >> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection] >> Removing host not found in peers metadata: >> >> 2017-05-09 20:45:25,060 [INFO] cassandra.cluster: Cassandra host >> 10.80.10.129 removed >> >> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: Removing host >> 10.80.10.129 >> >> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection] >> Finished fetching ring info >> >> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection] >> Rebuilding token map due to topology changes >> >> 2017-05-09 20:45:25,081 [DEBUG] cassandra.metadata: user functions table >> not found >> >> 2017-05-09 20:45:25,081 [DEBUG] cassandra.metadata: user aggregates table >> not found >> >> 2017-05-09 20:45:25,098 [DEBUG] cassandra.cluster: Control connection >> created >> >> 2017-05-09 20:45:25,099 [DEBUG] cassandra.pool: Initializing connection >> for host 10.80.10.125 >> >> 2017-05-09 20:45:25,099 [DEBUG] cassandra.pool: Initializing connection >> for host 10.80.10.126 >> >> >> > > > > -- > Dikang > > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: Reg:- Apache Solr with DSE Query
Hi Nandan I would suggest referring your question to Datastax, otherwise you could look at some open source alternatives such as: https://github.com/strapdata/elassandra https://github.com/Stratio/cassandra-lucene-index Cheers Ben On Thu, 11 May 2017 at 03:06 @Nandan@ <nandanpriyadarshi...@gmail.com> wrote: > Hi, > > Details are as below:- > 1) Have table:- video_info > 2) PRIMARY KEY:- video_id UUID > 3) having records around 5. > 4) Table is having around 30-35 columns > 5) Using DSE 4.8 > > Need clarifications and suggestions:- > 1) I need to search by few 3-4 columns like Video_title, video_actor etc.. > 2) If I am implementing Solr indexing on this single table, then we can > able to do a query from other columns and much more.. but is it going to > effect my READ and WRITE speed. > 3) is it will be a good idea or not to implement SOLR directly. > > Please suggest on above. > Thanks. > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: [Question] Automated cluster cleanup
Hi Luigi Under the hood, nodetool is actually just a command line wrapper around certain JMX calls. If you are looking to automate some more of commonplace nodetool actions, have a look at the nodetool source and it will show exactly what JMX calls (and parameters) are being passed. One thing to keep in mind with JMX, is it does allow a remote user to do some scary things to Cassandra and it has included remote code execution vulns. So ensure you lock down JMX thoroughly (password/username auth, certification auth, fw rules etc). For the other most common management, repairs, check out Cassandra reaper https://github.com/thelastpickle/cassandra-reaper. Ben On Thu, 27 Apr 2017 at 16:37 Luigi Tagliamonte <lu...@sysdig.com> wrote: > Hello Cassandra users, > my cluster is getting bigger and I was looking into automating some > tedious operations like the node cleanup after adding a new node to the > cluster. > > I gave a quick search and I didn't find any good available option, so I > decided to look into the JMX interface (In the storage service, I found the > method: forceKeyspaceCleanup that seems a good candidate), before going > hardcore with SSH+nodetool sessions. > > I was wondering if somebody here wants to share his experiences about this > task, and what do you think about JMX approach instead of the SSH one. > > Thank you. > > -- > Luigi > --- > “The only way to get smarter is by playing a smarter opponent.” > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: compaction falling behind
You can do so in two ways: 1) direct observation: You can keep an eye on the number of pending compactions. This will fluctuate with load, compaction strategy, ongoing repairs and nodes bootstrapping but generally the pattern is it should trend towards 0. There have been a number of bugs in past versions of Cassandra whereby the number of pending compactions is not reported correctly, so depending on what version of Cassandra you run this could impact you. 2) Indirect observation You can keep an eye on metrics that healthy compaction will directly contribute to. These include the number of sstables per read histogram, estimated droppable tombstones, tombstones per read etc. You should keep an eye on these things anyway as they can often show you areas where you can fine tune compaction or your data model. Everything exposed by nodetool is consumable via JMX which is great to plug into your metrics/monitoring/observability system :) On Mon, 13 Feb 2017 at 13:23 John Sanda <john.sa...@gmail.com> wrote: > What is a good way to determine whether or not compaction is falling > behind? I read a couple things earlier that suggest nodetool > compactionstats might not be the most reliable thing to use. > > > > - John > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: Authentication with Java driver
If the processes are launched separately or you fork before setting up the
cluster object it won't share credentials.
On Wed, Feb 8, 2017, 02:33 Yuji Ito <y...@imagine-orb.com> wrote:
> Thanks Ben,
>
> Do you mean lots of instances of the process or lots of instances of the
> cluster/session object?
>
>
> Lots of instances of the process are generated.
> I wanted to confirm that `other` doesn't authenticate.
>
> If I want to avoid that, my application has to create new cluster/session
> objects per instance.
> But it is inefficient and uncommon.
> So, we aren't sure that the application works when a lot of
> cluster/session objects are created.
> Is it correct?
>
> Thank you,
> Yuji
>
>
>
> On Wed, Feb 8, 2017 at 12:01 PM, Ben Bromhead <b...@instaclustr.com> wrote:
>
> On Tue, 7 Feb 2017 at 17:52 Yuji Ito <y...@imagine-orb.com> wrote:
>
> Thanks Andrew, Ben,
>
> My application creates a lot of instances connecting to Cassandra with
> basically the same set of credentials.
>
> Do you mean lots of instances of the process or lots of instances of the
> cluster/session object?
>
>
> After an instance connects to Cassandra with the credentials, can any
> instance connect to Cassandra without credentials?
>
> As long as you don't share the session or cluster objects. Each new
> cluster/session will need to reauthenticate.
>
>
> == example ==
> A first = new A("database", "user", "password"); // proper credentials
> r = first.get();
> ...
> A other = new A("database", "user", "pass"); // wrong password
> r = other.get();
> == example ==
>
> I want to refuse the `other` instance with improper credentials.
>
>
> This looks like you are creating new cluster/session objects (filling in
> the blanks for your pseudocode here). So "other" will not authenticate to
> Cassandra.
>
> This brings up a wider point of why you are doing this? Generally most
> applications will create a single longed lived session object that lasts
> the life of the application process.
>
> I would not rely on Cassandra auth to authenticate downstream actors, not
> because it's bad, just its generally inefficient to create lots of session
> objects. The session object maintains a connection pool, pipelines
> requests, is thread safe and generally pretty solid.
>
>
>
>
> Yuji
>
>
> On Wed, Feb 8, 2017 at 4:11 AM, Ben Bromhead <b...@instaclustr.com> wrote:
>
> What are you specifically trying to achieve? Are you trying to
> authenticate multiple Cassandra users from a single application instance?
> Or will your have lot's of application instances connecting to Cassandra
> using the same set of credentials? Or a combination of both? Multiple
> application instances with different credentials?
>
> On Tue, 7 Feb 2017 at 06:19 Andrew Tolbert <andrew.tolb...@datastax.com>
> wrote:
>
> Hello,
>
> The API seems kind of not correct because credentials should be
> usually set with a session but actually they are set with a cluster.
>
>
> With the datastax driver, Session is what manages connection pools to
> each node. Cluster manages configuration and a separate connection
> ('control connection') to subscribe to state changes (schema changes, node
> topology changes, node up/down events).
>
>
> So, if there are 1000 clients, then with this API it has to create
> 1000 cluster instances ?
>
>
> I'm unsure how common it is for per-user authentication to be done when
> connecting to the database. I think an application would normally
> authenticate with one set of credentials instead of multiple. The protocol
> Cassandra uses does authentication at the connection level instead of at
> the request level, so that is currently a limitation to support something
> like reusing Sessions for authenticating multiple users.
>
> Thanks,
> Andy
>
>
> On Tue, Feb 7, 2017 at 7:19 AM Hiroyuki Yamada <mogwa...@gmail.com> wrote:
>
> Hi,
>
> The API seems kind of not correct because credentials should be
> usually set with a session but actually they are set with a cluster.
>
> So, if there are 1000 clients, then with this API it has to create
> 1000 cluster instances ?
> 1000 clients seems usual if there are many nodes (say 20) and each
> node has some concurrency (say 50),
> but 1000 cluster instances seems too many.
>
> Is this an expected way to do this ? or
> Is there any way to authenticate per session ?
>
> Thanks,
> Hiro
>
> On Tue, Feb 7, 2017 at 11:38 AM, Yuji Ito <y...@imagine-orb.com> wrote:
> > Hi all,
> >
> > I want to know how to authenticate
Instaclustr Masters scholarship
As part of our commitment to contributing back to the Apache Cassandra open source project and the wider community we are always looking for ways we can foster knowledge sharing and improve usability of Cassandra itself. One of the ways we have done so previously was to open up our internal builds and versions of Cassandra (https://github.com/instaclustr/cassandra). We have also been looking at a few novel or outside the box ways we can further contribute back to the community. As such, we are sponsoring a masters project in conjunction with the Australian based University of Canberra. Instaclustr’s staff will be available to provide advice and feedback to the successful candidate. *Scope* Distributed database systems are relatively new technology compared to traditional relational databases. Distributed advantages provide significant advantages in terms of reliability and scalability but often at a cost of increased complexity. This complexity presents challenges for testing of these systems to prove correct operation across all possible system states. The scope of this masters scholarship is to use the Apache Cassandra repair process as an example to consider and improve available approaches to distributed database systems testing. The repair process in Cassandra is a scheduled process that runs to ensure the multiple copies of each piece of data that is maintained by Cassandra are kept synchronised. Correct operation of repairs has been an ongoing challenge for the Cassandra project partly due to the difficulty in designing and developing comprehensive automated tests for this functionality. The expected scope of this project is to: - survey and understand the existing testing framework available as part of the Cassandra project, particularly as it pertains to testing repairs - consider, research and develop enhanced approaches to testing of repairs - submit any successful approaches to the Apache Cassandra project for feedback and inclusion in the project code base Australia is a pretty great place to advance your education and is welcoming of foreign students. We are also open to sponsoring a PhD project with a more in depth focus for the right candidate. For more details please don't hesitate to get in touch with myself or reach out to i...@instaclustr.com. Cheers Ben -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: Authentication with Java driver
On Tue, 7 Feb 2017 at 17:52 Yuji Ito <y...@imagine-orb.com> wrote:
Thanks Andrew, Ben,
My application creates a lot of instances connecting to Cassandra with
basically the same set of credentials.
Do you mean lots of instances of the process or lots of instances of the
cluster/session object?
After an instance connects to Cassandra with the credentials, can any
instance connect to Cassandra without credentials?
As long as you don't share the session or cluster objects. Each new
cluster/session will need to reauthenticate.
== example ==
A first = new A("database", "user", "password"); // proper credentials
r = first.get();
...
A other = new A("database", "user", "pass"); // wrong password
r = other.get();
== example ==
I want to refuse the `other` instance with improper credentials.
This looks like you are creating new cluster/session objects (filling in
the blanks for your pseudocode here). So "other" will not authenticate to
Cassandra.
This brings up a wider point of why you are doing this? Generally most
applications will create a single longed lived session object that lasts
the life of the application process.
I would not rely on Cassandra auth to authenticate downstream actors, not
because it's bad, just its generally inefficient to create lots of session
objects. The session object maintains a connection pool, pipelines
requests, is thread safe and generally pretty solid.
Yuji
On Wed, Feb 8, 2017 at 4:11 AM, Ben Bromhead <b...@instaclustr.com> wrote:
What are you specifically trying to achieve? Are you trying to authenticate
multiple Cassandra users from a single application instance? Or will your
have lot's of application instances connecting to Cassandra using the same
set of credentials? Or a combination of both? Multiple application
instances with different credentials?
On Tue, 7 Feb 2017 at 06:19 Andrew Tolbert <andrew.tolb...@datastax.com>
wrote:
Hello,
The API seems kind of not correct because credentials should be
usually set with a session but actually they are set with a cluster.
With the datastax driver, Session is what manages connection pools to each
node. Cluster manages configuration and a separate connection ('control
connection') to subscribe to state changes (schema changes, node topology
changes, node up/down events).
So, if there are 1000 clients, then with this API it has to create
1000 cluster instances ?
I'm unsure how common it is for per-user authentication to be done when
connecting to the database. I think an application would normally
authenticate with one set of credentials instead of multiple. The protocol
Cassandra uses does authentication at the connection level instead of at
the request level, so that is currently a limitation to support something
like reusing Sessions for authenticating multiple users.
Thanks,
Andy
On Tue, Feb 7, 2017 at 7:19 AM Hiroyuki Yamada <mogwa...@gmail.com> wrote:
Hi,
The API seems kind of not correct because credentials should be
usually set with a session but actually they are set with a cluster.
So, if there are 1000 clients, then with this API it has to create
1000 cluster instances ?
1000 clients seems usual if there are many nodes (say 20) and each
node has some concurrency (say 50),
but 1000 cluster instances seems too many.
Is this an expected way to do this ? or
Is there any way to authenticate per session ?
Thanks,
Hiro
On Tue, Feb 7, 2017 at 11:38 AM, Yuji Ito <y...@imagine-orb.com> wrote:
> Hi all,
>
> I want to know how to authenticate Cassandra users for multiple instances
> with Java driver.
> For instance, each thread creates a instance to access Cassandra with
> authentication.
>
> As the implementation example, only the first constructor builds a cluster
> and a session.
> Other constructors use them.
> This example is implemented according to the datastax document: "Basically
> you will want to share the same cluster and session instances across your
> application".
>
http://www.datastax.com/dev/blog/4-simple-rules-when-using-the-datastax-drivers-for-cassandra
>
> However, other constructors don't authenticate the user and the password.
> That's because they don't need to build a cluster and a session.
>
> So, should I create a cluster and a session per instance for the
> authentication?
> If yes, can I create a lot of instances(clusters and sessions) to access
C*
> concurrently?
>
> == example ==
> public class A {
> private static Cluster cluster = null;
> private static Map<String, Session> sessions = null;
> private Session session;
>
> public A (String keyspace, String user, String password) {
> if (cluster == null) {
> builder = Cluster.builder();
> ...
> builder = builder.withCredentials(use
Re: Authentication with Java driver
What are you specifically trying to achieve? Are you trying to authenticate
multiple Cassandra users from a single application instance? Or will your
have lot's of application instances connecting to Cassandra using the same
set of credentials? Or a combination of both? Multiple application
instances with different credentials?
On Tue, 7 Feb 2017 at 06:19 Andrew Tolbert <andrew.tolb...@datastax.com>
wrote:
> Hello,
>
> The API seems kind of not correct because credentials should be
> usually set with a session but actually they are set with a cluster.
>
>
> With the datastax driver, Session is what manages connection pools to
> each node. Cluster manages configuration and a separate connection
> ('control connection') to subscribe to state changes (schema changes, node
> topology changes, node up/down events).
>
>
> So, if there are 1000 clients, then with this API it has to create
> 1000 cluster instances ?
>
>
> I'm unsure how common it is for per-user authentication to be done when
> connecting to the database. I think an application would normally
> authenticate with one set of credentials instead of multiple. The protocol
> Cassandra uses does authentication at the connection level instead of at
> the request level, so that is currently a limitation to support something
> like reusing Sessions for authenticating multiple users.
>
> Thanks,
> Andy
>
>
> On Tue, Feb 7, 2017 at 7:19 AM Hiroyuki Yamada <mogwa...@gmail.com> wrote:
>
> Hi,
>
> The API seems kind of not correct because credentials should be
> usually set with a session but actually they are set with a cluster.
>
> So, if there are 1000 clients, then with this API it has to create
> 1000 cluster instances ?
> 1000 clients seems usual if there are many nodes (say 20) and each
> node has some concurrency (say 50),
> but 1000 cluster instances seems too many.
>
> Is this an expected way to do this ? or
> Is there any way to authenticate per session ?
>
> Thanks,
> Hiro
>
> On Tue, Feb 7, 2017 at 11:38 AM, Yuji Ito <y...@imagine-orb.com> wrote:
> > Hi all,
> >
> > I want to know how to authenticate Cassandra users for multiple instances
> > with Java driver.
> > For instance, each thread creates a instance to access Cassandra with
> > authentication.
> >
> > As the implementation example, only the first constructor builds a
> cluster
> > and a session.
> > Other constructors use them.
> > This example is implemented according to the datastax document:
> "Basically
> > you will want to share the same cluster and session instances across your
> > application".
> >
> http://www.datastax.com/dev/blog/4-simple-rules-when-using-the-datastax-drivers-for-cassandra
> >
> > However, other constructors don't authenticate the user and the password.
> > That's because they don't need to build a cluster and a session.
> >
> > So, should I create a cluster and a session per instance for the
> > authentication?
> > If yes, can I create a lot of instances(clusters and sessions) to access
> C*
> > concurrently?
> >
> > == example ==
> > public class A {
> > private static Cluster cluster = null;
> > private static Map<String, Session> sessions = null;
> > private Session session;
> >
> > public A (String keyspace, String user, String password) {
> > if (cluster == null) {
> > builder = Cluster.builder();
> > ...
> > builder = builder.withCredentials(user, password);
> > cluster = builder.build();
> > }
> > session = sessions.get(keyspace);
> > if (session == null) {
> > session = cluster.connection(keyspace);
> > sessions.put(keyspace, session)
> > }
> > ...
> > }
> > ...
> > public ResultSet update(...) {
> > ...
> > public ResultSet get(...) {
> > ...
> > }
> > == example ==
> >
> > Thanks,
> > Yuji
>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: Cassandra Authentication
the volume of data is pretty low + you still want to be able to authenticate even if you have more nodes down than the RF for other keyspaces. Essentially you don't want auth to be the thing that stops you serving requests. On Wed, 18 Jan 2017 at 14:57 Jai Bheemsen Rao Dhanwada < jaibheem...@gmail.com> wrote: > Thanks Ben, > > RF 3 isn't sufficient for system_auth? as we are using 3 RF for other > production KS, do you see any challenges? > > On Wed, Jan 18, 2017 at 2:39 PM, Ben Bromhead <b...@instaclustr.com> wrote: > > We have a process that syncs and manages RF==N and we also control and > manage users, however that entails it's own set of challenges and > maintenance. > > For most users I would suggest 3 < RF <=5 is sufficient. Also make sure > you don't use the user "Cassandra" in production as authentication queries > are done at QUORUM. > > On Wed, 18 Jan 2017 at 13:41 Jai Bheemsen Rao Dhanwada < > jaibheem...@gmail.com> wrote: > > Hello, > > When enabling Authentication on cassandra, is it required to set the RF > same as the no.of nodes( > https://docs.datastax.com/en/cql/3.1/cql/cql_using/update_ks_rf_t.html)? > or can I live with RF of 3 in each DC (other KS are using 3) > > If it has to be equal to the number of nodes then, every time adding or > removing a node requires update of RF. > > Thanks in advance. > > -- > Ben Bromhead > CTO | Instaclustr <https://www.instaclustr.com/> > +1 650 284 9692 <+1%20650-284-9692> > Managed Cassandra / Spark on AWS, Azure and Softlayer > > > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: Cassandra Authentication
We have a process that syncs and manages RF==N and we also control and manage users, however that entails it's own set of challenges and maintenance. For most users I would suggest 3 < RF <=5 is sufficient. Also make sure you don't use the user "Cassandra" in production as authentication queries are done at QUORUM. On Wed, 18 Jan 2017 at 13:41 Jai Bheemsen Rao Dhanwada < jaibheem...@gmail.com> wrote: > Hello, > > When enabling Authentication on cassandra, is it required to set the RF > same as the no.of nodes( > https://docs.datastax.com/en/cql/3.1/cql/cql_using/update_ks_rf_t.html)? > or can I live with RF of 3 in each DC (other KS are using 3) > > If it has to be equal to the number of nodes then, every time adding or > removing a node requires update of RF. > > Thanks in advance. > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: [Announcement] BigGraphite
This looks super cool On Tue, 6 Dec 2016 at 01:49 DuyHai Doan <doanduy...@gmail.com> wrote: > Hello Cassandra users > > On behalf of Corentin Chary from Criteo, I'm happy to announce the release > of BigGraphite, an OSS Graphite plugin dedicated for time series based on > Apache Cassandra. BigGraphite is leveraging latest features as SASI index & > Back Pressure. > > More info here: > > https://github.com/criteo/biggraphite/wiki/BigGraphite-Announcement > http://labs.criteo.com/blog/ > > Regards > > Duy Hai DOAN > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: Why does `now()` produce different times within the same query?
> > > > I will note that Ben seems to suggest keeping the return of now() unique > across > call while keeping the time component equals, thus varying the rest of the > uuid > bytes. However: > - I'm starting to wonder what this would buy us. Why would someone be > super >confused by the time changing across calls (in a single > statement/batch), but >be totally not confused by the actual full return to not be equal? > Given that a common way of interacting with timeuuids is with toTimestamp I can see the confusion and assumptions around behaviour. And how is >that actually useful: you're having different result anyway and you're >letting the server pick the timestamp in the first place, so you're > probably >not caring about milliseconds precision of that timestamp in the first > place. > If you want consistency of timestamps within your query as OP did I can see how this is useful. Postgres claims this is a "feature". - This would basically be a violation of the timeuuid spec > Not quite... Type 1 uuids let you swap out the low 47 bits of the node component with other randomly generated bits ( https://www.ietf.org/rfc/rfc4122.txt) - This would be a big pain in the code and make of now() a special case > among functions. I'm unconvinced special cases are making things easier > in general. > On reflection, I have to agree here, now() has been around for ever and this is the first anecdote I've seen of someone getting caught out. However with my user advocate hat on I think it would be worth investigating further beyond a documentation update if others found it a sticking point in Cassandra adoption. > So I'm all for improving the documentation if this confuses users due to > expectations (mistakenly) carried from prior experiences, and please > feel free to open a JIRA for that. I'm a lot less in agreement that there > is > something wrong with the way the function behave in principle. > > > I can see why this issue has been largely ignored and hasn't had a > chance for > > the behaviour to be formally defined > > Don't make too much assumptions. The behavior is perfectly well defined: > now() > is a "normal" function and is evaluated whenever it's called according to > the > timeuuid spec (or as close to it as we can make it). > Maybe formally defined is the wrong term... Formally documented? > > On Thu, Dec 1, 2016 at 7:25 AM, Benjamin Roth <benjamin.r...@jaumo.com> > wrote: > > Great comment. +1 > > Am 01.12.2016 06:29 schrieb "Ben Bromhead" <b...@instaclustr.com>: > > tl;dr +1 yup raise a jira to discuss how now() should behave in a single > statement (and possible extend to batch statements). > > The values of now should be the same if you assume that now() works like > it does in relational databases such as postgres or mysql, however at the > moment it instead works like sysdate() in mysql. Given that CQL is supposed > to be SQL like, I think the assumption around the behaviour of now() was a > fair one to make. > > I definitely agree that raising a jira ticket would be a great place to > discuss what the behaviour of now() should be for Cassandra. Personally I > would be in favour of seeing the deterministic component (the actual time > part) being the same across multiple calls in the one statement or multiple > statements in a batch. > > Cassandra documentation does not make any claims as to how now() works > within a single statement and reading the code it shows the intent is to > work like sysdate() from MySQL rather than now(). One of the identified > dangers of making cql similar to sql is that, while yes it aids adoption, > users will find that SQL like things don't behave as expected. Of course as > a user, one shouldn't have to read the source code to determine correct > behaviour. > > Given that a timeuuid is made up of deterministic and (pseudo) > non-deterministic components I can see why this issue has been largely > ignored and hasn't had a chance for the behaviour to be formally defined > (you would expect now to return the same time in the one statement despite > multiple calls, but you wouldn't expect the same behaviour for say a call > to rand()). > > > > > > > > On Wed, 30 Nov 2016 at 19:54 Cody Yancey <yan...@uber.com> wrote: > > This is not a bug, and in fact changing it would be a serious bug. > > False. Absolutely no consumer would be broken by a change to guarantee an > identical time component that isn't broken already, for the simple reason > your code already has to handle that case, as it is in fact the majority > case RIGHT NOW. Users can hit this bug, in production, because unit tests > might
Re: Why does `now()` produce different times within the same query?
ster. We most definitely CANNOT control clock skew on the thousands of > sensors that write to our cluster. > > Thanks, > Cody > > On Wed, Nov 30, 2016 at 5:33 PM, Robert Wille <rwi...@fold3.com> wrote: > > In my opinion, this is not broken and “fixing” it would break existing > code. Consider a batch that includes multiple inserts, each of which > inserts the value returned by now(). Getting the same UUID for each insert > would be a major problem. > > Cheers > > Robert > > > On Nov 30, 2016, at 4:46 PM, Todd Fast <t...@digitalexistence.com> wrote: > > FWIW I'd suggest opening a bug--this behavior is certainly quite > unexpected and more than just a documentation issue. In general I can't > imagine any desirable properties of the current implementation, and there > are likely a bunch of latent bugs sitting out there, so it should be fixed. > > Todd > > On Wed, Nov 30, 2016 at 12:37 PM Terry Liu <t...@turnitin.com> wrote: > > Sorry for my typo. Obviously, I meant: > "It appears that a single query that calls Cassandra's`now()` time > function *multiple times *may actually cause a query to write or return > different times." > > Less of a surprise now that I realize more about the implementation, but I > agree that more explicit documentation around when exactly the "execution" > of each now() statement happens and what implications it has for the > resulting timestamps would be helpful when running into this. > > Thanks for the quick responses! > > -Terry > > > > On Tue, Nov 29, 2016 at 2:45 PM, Marko Švaljek <msval...@gmail.com> wrote: > > every now() call in statement is under the hood "replaced" with newly > generated uuid. > > It can happen that they belong to different milliseconds in time. > > If you need to have same timestamps you need to set them on the client > side. > > > @msvaljek <https://twitter.com/msvaljek> > > 2016-11-29 22:49 GMT+01:00 Terry Liu <t...@turnitin.com>: > > It appears that a single query that calls Cassandra's `now()` time > function may actually cause a query to write or return different times. > > Is this the expected or defined behavior, and if so, why does it behave > like this rather than evaluating `now()` once across an entire statement? > > This really affects UPDATE statements but to test it more easily, you > could try something like: > > SELECT toTimestamp(now()) as a, toTimestamp(now()) as b > FROM keyspace.table > LIMIT 100; > > If you run that a few times, you should eventually see that the timestamp > returned moves onto the next millisecond mid-query. > > -- > *Software Engineer* > Turnitin - http://www.turnitin.com > t...@turnitin.com > > > > > > -- > *Software Engineer* > Turnitin - http://www.turnitin.com > t...@turnitin.com > > > > > > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: Clarify Support for 2.2 on Download Page
Hi Derek You should subscribe and post this question to the Dev list, they will be able to get you sorted quickly! Normally you can edit documentation directly via github (e.g. https://github.com/apache/cassandra/tree/trunk/doc/source), however the download source appears to be outside the Cassandra repo. Ben On Wed, 16 Nov 2016 at 13:08 Derek Burdick <derek.burd...@gmail.com> wrote: > Hi, is it possible to update the language on the Apache Cassandra Download > page to reflect that version 2.2 will enter Critical Fix Only support after > November 21st? > > The current language creates quite a bit of confusion in the community > with how long 2.2 and 2.1 will receive fixes from the community. > > http://cassandra.apache.org/download/ > > Specifically these three lines: > >- Apache Cassandra 3.0 is supported until May 2017. The latest release >is 3.0.9 > > <http://www.apache.org/dyn/closer.lua/cassandra/3.0.9/apache-cassandra-3.0.9-bin.tar.gz> > (pgp > > <http://www.apache.org/dist/cassandra/3.0.9/apache-cassandra-3.0.9-bin.tar.gz.asc> >, md5 > > <http://www.apache.org/dist/cassandra/3.0.9/apache-cassandra-3.0.9-bin.tar.gz.md5> > and sha1 > > <http://www.apache.org/dist/cassandra/3.0.9/apache-cassandra-3.0.9-bin.tar.gz.sha1>), >released on 2016-09-20. >- Apache Cassandra 2.2 is supported until November 2016. The latest >release is 2.2.8 > > <http://www.apache.org/dyn/closer.lua/cassandra/2.2.8/apache-cassandra-2.2.8-bin.tar.gz> > (pgp > > <http://www.apache.org/dist/cassandra/2.2.8/apache-cassandra-2.2.8-bin.tar.gz.asc> >, md5 > > <http://www.apache.org/dist/cassandra/2.2.8/apache-cassandra-2.2.8-bin.tar.gz.md5> > and sha1 > > <http://www.apache.org/dist/cassandra/2.2.8/apache-cassandra-2.2.8-bin.tar.gz.sha1>), >released on 2016-09-28. >- Apache Cassandra 2.1 is supported until November 2016 with critical >fixes only. The latest release is 2.1.16 > > <http://www.apache.org/dyn/closer.lua/cassandra/2.1.16/apache-cassandra-2.1.16-bin.tar.gz> > (pgp > > <http://www.apache.org/dist/cassandra/2.1.16/apache-cassandra-2.1.16-bin.tar.gz.asc> >, md5 > > <http://www.apache.org/dist/cassandra/2.1.16/apache-cassandra-2.1.16-bin.tar.gz.md5> > and sha1 > > <http://www.apache.org/dist/cassandra/2.1.16/apache-cassandra-2.1.16-bin.tar.gz.sha1>), >released on 2016-10-10. > > > What would be the best approach to help get this changed? > > -Derek > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: Is it *safe* to issue multiple replace-node at the same time?
Same rack and no range movements, my first instinct is to say yes it is safe (I like to treat racks as one giant meta node). However I would want to have a read through the replace code. On Mon, Nov 21, 2016, 07:22 Dikang Gu <dikan...@gmail.com> wrote: > Hi guys, > > Sometimes we need to replace multiple hosts in the same rack, is it safe > to replace them in parallel, using the replace-node command? > > Will it cause any data inconsistency if we do so? > > Thanks > Dikang. > > -- > Dikang > > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: Any Bulk Load on Large Data Set Advice?
+1 on parquet and S3. Combined with spark running on spot instances your grant money will go much further! On Thu, 17 Nov 2016 at 07:21 Jonathan Haddad <j...@jonhaddad.com> wrote: > If you're only doing this for spark, you'll be much better off using > parquet and HDFS or S3. While you *can* do analytics with cassandra, it's > not all that great at it. > On Thu, Nov 17, 2016 at 6:05 AM Joe Olson <technol...@nododos.com> wrote: > > I received a grant to do some analysis on netflow data (Local IP address, > Local Port, Remote IP address, Remote Port, time, # of packets, etc) using > Cassandra and Spark. The de-normalized data set is about 13TB out the door. > I plan on using 9 Cassandra nodes (replication factor=3) to store the data, > with Spark doing the aggregation. > > Data set will be immutable once loaded, and am using the replication > factor = 3 to somewhat simulate the real world. Most of the analysis will > be of the sort "Give me all the remote ip addresses for source IP 'X' > between time t1 and t2" > > I built and tested a bulk loader following this example in GitHub: > https://github.com/yukim/cassandra-bulkload-example to generate the > SSTables, but I have not executed it on the entire data set yet. > > Any advice on how to execute the bulk load under this configuration? Can > I generate the SSTables in parallel? Once generated, can I write the > SSTables to all nodes simultaneously? Should I be doing any kind of sorting > by the partition key? > > This is a lot of data, so I figured I'd ask before I pulled the trigger. > Thanks in advance! > > > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: High system CPU during high write workload
Hi Abhishek
The article with the futex bug description lists the solution, which is to
upgrade to a version of RHEL or CentOS that have the specified patch.
What help do you specifically need? If you need help upgrading the OS I
would look at the documentation for RHEL or CentOS.
Ben
On Mon, 14 Nov 2016 at 22:48 Abhishek Gupta <gupta.abhis...@snapdeal.com>
wrote:
Hi,
We are seeing an issue where the system CPU is shooting off to a figure or
> 90% when the cluster is subjected to a relatively high write workload i.e
4k wreq/secs.
2016-11-14T13:27:47.900+0530 Process summary
process cpu=695.61%
application cpu=676.11% (*user=200.63% sys=475.49%) **<== Very High
System CPU *
other: cpu=19.49%
heap allocation rate *403mb*/s
[000533] user= 1.43% sys= 6.91% alloc= 2216kb/s - SharedPool-Worker-129
[000274] user= 0.38% sys= 7.78% alloc= 2415kb/s - SharedPool-Worker-34
[000292] user= 1.24% sys= 6.77% alloc= 2196kb/s - SharedPool-Worker-56
[000487] user= 1.24% sys= 6.69% alloc= 2260kb/s - SharedPool-Worker-79
[000488] user= 1.24% sys= 6.56% alloc= 2064kb/s - SharedPool-Worker-78
[000258] user= 1.05% sys= 6.66% alloc= 2250kb/s - SharedPool-Worker-41
On doing strace it was found that the following system call is consuming
all the system CPU
timeout 10s strace -f -p 5954 -c -q
% time seconds usecs/call callserrors syscall
-- --- --- - -
*88.33 1712.798399 16674102723 22191 futex* 3.98 77.098730
4356 17700 read
3.27 63.474795 394253 16129 restart_syscall
3.23 62.601530 29768 2103 epoll_wait
On searching we found the following bug with the RHEL 6.6, CentOS 6.6
kernel seems to be a probable cause for the issue:
https://docs.datastax.com/en/landing_page/doc/landing_page/troubleshooting/cassandra/fetuxWaitBug.html
The patch fix mentioned in the doc is also not present in our kernel.
sudo rpm -q --changelog kernel-`uname -r` | grep futex | grep ref
- [kernel] futex_lock_pi() key refcnt fix (Danny Feng) [566347]
{CVE-2010-0623}
Can some who has faced and resolved this issue help us here.
Thanks,
Abhishek
--
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: Priority for cassandra nodes in cluster
+1 w/ Benjamin. However if you wish to make use of spare hardware capacity, look to something like mesos DC/OS or kubernetes. You can run multiple services across a fleet of hardware, but provision equal resources to Cassandra and have somewhat reliable hardware sharing mechanisms. On Sat, 12 Nov 2016 at 14:12 Jon Haddad <jonathan.had...@gmail.com> wrote: > Agreed w/ Benjamin. Trying to diagnose issues in prod will be a > nightmare. Keep your DB servers homogeneous. > > On Nov 12, 2016, at 1:52 PM, Benjamin Roth <benjamin.r...@jaumo.com> > wrote: > > 1. From a 15 year experience of running distributed Services: dont Mix > Services on machines if you don't have to. Dedicate each server to a single > task if you can afford it. It is easier to manage and reduces risks in case > of overload or failure > 2. You can assign a different number of tokens for each node by setting > this in Cassandra.yaml before you bootstrap that node > > Am 12.11.2016 22:48 schrieb "sat" <sathish.al...@gmail.com>: > > Hi, > > We are planning to install 3 node cluster in production environment. Is it > possible to provide weightage or priority to the nodes in cluster. > > Eg., We want more more records to be written to first 2 nodes and less to > the 3rd node. We are thinking of this approach because we want to install > other IO intensive messaging server in the 3rd node, in order to reduce the > load we are requesting for this approach. > > > Thanks and Regards > A.SathishKumar > > > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: Are Cassandra writes are faster than reads?
Awesome! For a full explanation of what you are seeing (we call it micro batching) check out Adam Zegelins talk on it https://www.youtube.com/watch?v=wF3Ec1rdWgc On Tue, 8 Nov 2016 at 02:21 Rajesh Radhakrishnan < rajesh.radhakrish...@phe.gov.uk> wrote: > > Hi, > > Just found that reducing the batch size below 20 also increases the > writing speed and reduction in memory usage(especially for Python driver). > > Kind regards, > Rajesh R > > ------ > *From:* Ben Bromhead [b...@instaclustr.com] > *Sent:* 07 November 2016 05:44 > *To:* user@cassandra.apache.org > *Subject:* Re: Are Cassandra writes are faster than reads? > > They can be and it depends on your compaction strategy :) > > On Sun, 6 Nov 2016 at 21:24 Ali Akhtar <ali.rac...@gmail.com > <http://redir.aspx?REF=KvuN_F91CkILmAKkPOD8RLOkpaObm4vWZ4CTx2PNAjG8Cvd6wAfUCAFtYWlsdG86YWxpLnJhYzIwMEBnbWFpbC5jb20.>> > wrote: > > tl;dr? I just want to know if updates are bad for performance, and if so, > for how long. > > On Mon, Nov 7, 2016 at 10:23 AM, Ben Bromhead <b...@instaclustr.com > <http://redir.aspx?REF=bOLz-2Z_cjZ-R5mW4ySFRmRgIvYoWF43pRrpxxUsOOC8Cvd6wAfUCAFtYWlsdG86YmVuQGluc3RhY2x1c3RyLmNvbQ..> > > wrote: > > Check out https://wiki.apache.org/cassandra/WritePathForUsers > <http://redir.aspx?REF=z6gebtTM9Bi4b1ZEZqnpcgJOwnifCWloccEOX28F8UC8Cvd6wAfUCAFodHRwczovL3dpa2kuYXBhY2hlLm9yZy9jYXNzYW5kcmEvV3JpdGVQYXRoRm9yVXNlcnM.> > for > the full gory details. > > On Sun, 6 Nov 2016 at 21:09 Ali Akhtar <ali.rac...@gmail.com > <http://redir.aspx?REF=KvuN_F91CkILmAKkPOD8RLOkpaObm4vWZ4CTx2PNAjG8Cvd6wAfUCAFtYWlsdG86YWxpLnJhYzIwMEBnbWFpbC5jb20.>> > wrote: > > How long does it take for updates to get merged / compacted into the main > data file? > > On Mon, Nov 7, 2016 at 5:31 AM, Ben Bromhead <b...@instaclustr.com > <http://redir.aspx?REF=bOLz-2Z_cjZ-R5mW4ySFRmRgIvYoWF43pRrpxxUsOOC8Cvd6wAfUCAFtYWlsdG86YmVuQGluc3RhY2x1c3RyLmNvbQ..> > > wrote: > > To add some flavor as to how the commitlog implementation is so quick. > > It only flushes to disk every 10s by default. So writes are effectively > done to memory and then to disk asynchronously later on. This is generally > accepted to be OK, as the write is also going to other nodes. > > You can of course change this behavior to flush on each write or to skip > the commitlog altogether (danger!). This however will change how "safe" > things are from a durability perspective. > > On Sun, Nov 6, 2016, 12:51 Jeff Jirsa <jeff.ji...@crowdstrike.com > <http://redir.aspx?REF=CSJmlUdwjTSoe3NQdZNlO6pFPeaI_KxNpZweB-GbDYO8Cvd6wAfUCAFtYWlsdG86amVmZi5qaXJzYUBjcm93ZHN0cmlrZS5jb20.>> > wrote: > > Cassandra writes are particularly fast, for a few reasons: > > > > 1) Most writes go to a commitlog (append-only file, written > linearly, so particularly fast in terms of disk operations) and then pushed > to the memTable. Memtable is flushed in batches to the permanent data > files, so it buffers many mutations and then does a sequential write to > persist that data to disk. > > 2) Reads may have to merge data from many data tables on disk. > Because the writes (described very briefly in step 1) write to immutable > files, updates/deletes have to be merged on read – this is extra effort for > the read path. > > > > If you don’t do much in terms of overwrites/deletes, and your partitions > are particularly small, and your data fits in RAM (probably mmap/page cache > of data files, unless you’re using the row cache), reads may be very fast > for you. Certainly individual reads on low-merge workloads can be < 0.1ms. > > > > - Jeff > > > > *From: *Vikas Jaiman <er.vikasjai...@gmail.com > <http://redir.aspx?REF=VgqqnBUEzP6sLWofnDxFp3iyHQ4TGCTJL8MbqH0NOUK8Cvd6wAfUCAFtYWlsdG86ZXIudmlrYXNqYWltYW5AZ21haWwuY29t> > > > *Reply-To: *"user@cassandra.apache.org > <http://redir.aspx?REF=yxCMb2E-WgRKlJCeCUpFf-0-Th-NE4pZJyZdWo0SRMS8Cvd6wAfUCAFtYWlsdG86dXNlckBjYXNzYW5kcmEuYXBhY2hlLm9yZw..>" > <user@cassandra.apache.org > <http://redir.aspx?REF=yxCMb2E-WgRKlJCeCUpFf-0-Th-NE4pZJyZdWo0SRMS8Cvd6wAfUCAFtYWlsdG86dXNlckBjYXNzYW5kcmEuYXBhY2hlLm9yZw..> > > > *Date: *Sunday, November 6, 2016 at 12:42 PM > *To: *"user@cassandra.apache.org > <http://redir.aspx?REF=yxCMb2E-WgRKlJCeCUpFf-0-Th-NE4pZJyZdWo0SRMS8Cvd6wAfUCAFtYWlsdG86dXNlckBjYXNzYW5kcmEuYXBhY2hlLm9yZw..>" > <user@cassandra.apache.org > <http://redir.aspx?REF=yxCMb2E-WgRKlJCeCUpFf-0-Th-NE4pZJyZdWo0SRMS8Cvd6wAfUCAFtYWlsdG86dXNlckBjYXNzYW5kcmEuYXBhY2hlLm9yZw..> > > > *Subject: *Are Cassandra writes are faster than reads? &
Re: Are Cassandra writes are faster than reads?
They can be and it depends on your compaction strategy :) On Sun, 6 Nov 2016 at 21:24 Ali Akhtar <ali.rac...@gmail.com> wrote: > tl;dr? I just want to know if updates are bad for performance, and if so, > for how long. > > On Mon, Nov 7, 2016 at 10:23 AM, Ben Bromhead <b...@instaclustr.com> wrote: > > Check out https://wiki.apache.org/cassandra/WritePathForUsers for the > full gory details. > > On Sun, 6 Nov 2016 at 21:09 Ali Akhtar <ali.rac...@gmail.com> wrote: > > How long does it take for updates to get merged / compacted into the main > data file? > > On Mon, Nov 7, 2016 at 5:31 AM, Ben Bromhead <b...@instaclustr.com> wrote: > > To add some flavor as to how the commitlog implementation is so quick. > > It only flushes to disk every 10s by default. So writes are effectively > done to memory and then to disk asynchronously later on. This is generally > accepted to be OK, as the write is also going to other nodes. > > You can of course change this behavior to flush on each write or to skip > the commitlog altogether (danger!). This however will change how "safe" > things are from a durability perspective. > > On Sun, Nov 6, 2016, 12:51 Jeff Jirsa <jeff.ji...@crowdstrike.com> wrote: > > Cassandra writes are particularly fast, for a few reasons: > > > > 1) Most writes go to a commitlog (append-only file, written > linearly, so particularly fast in terms of disk operations) and then pushed > to the memTable. Memtable is flushed in batches to the permanent data > files, so it buffers many mutations and then does a sequential write to > persist that data to disk. > > 2) Reads may have to merge data from many data tables on disk. > Because the writes (described very briefly in step 1) write to immutable > files, updates/deletes have to be merged on read – this is extra effort for > the read path. > > > > If you don’t do much in terms of overwrites/deletes, and your partitions > are particularly small, and your data fits in RAM (probably mmap/page cache > of data files, unless you’re using the row cache), reads may be very fast > for you. Certainly individual reads on low-merge workloads can be < 0.1ms. > > > > - Jeff > > > > *From: *Vikas Jaiman <er.vikasjai...@gmail.com> > *Reply-To: *"user@cassandra.apache.org" <user@cassandra.apache.org> > *Date: *Sunday, November 6, 2016 at 12:42 PM > *To: *"user@cassandra.apache.org" <user@cassandra.apache.org> > *Subject: *Are Cassandra writes are faster than reads? > > > > Hi all, > > > > Are Cassandra writes are faster than reads ?? If yes, why is this so? I am > using consistency 1 and data is in memory. > > > > Vikas > > -- > Ben Bromhead > CTO | Instaclustr <https://www.instaclustr.com/> > +1 650 284 9692 > Managed Cassandra / Spark on AWS, Azure and Softlayer > > > -- > Ben Bromhead > CTO | Instaclustr <https://www.instaclustr.com/> > +1 650 284 9692 > Managed Cassandra / Spark on AWS, Azure and Softlayer > > > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: Are Cassandra writes are faster than reads?
Check out https://wiki.apache.org/cassandra/WritePathForUsers for the full gory details. On Sun, 6 Nov 2016 at 21:09 Ali Akhtar <ali.rac...@gmail.com> wrote: > How long does it take for updates to get merged / compacted into the main > data file? > > On Mon, Nov 7, 2016 at 5:31 AM, Ben Bromhead <b...@instaclustr.com> wrote: > > To add some flavor as to how the commitlog implementation is so quick. > > It only flushes to disk every 10s by default. So writes are effectively > done to memory and then to disk asynchronously later on. This is generally > accepted to be OK, as the write is also going to other nodes. > > You can of course change this behavior to flush on each write or to skip > the commitlog altogether (danger!). This however will change how "safe" > things are from a durability perspective. > > On Sun, Nov 6, 2016, 12:51 Jeff Jirsa <jeff.ji...@crowdstrike.com> wrote: > > Cassandra writes are particularly fast, for a few reasons: > > > > 1) Most writes go to a commitlog (append-only file, written > linearly, so particularly fast in terms of disk operations) and then pushed > to the memTable. Memtable is flushed in batches to the permanent data > files, so it buffers many mutations and then does a sequential write to > persist that data to disk. > > 2) Reads may have to merge data from many data tables on disk. > Because the writes (described very briefly in step 1) write to immutable > files, updates/deletes have to be merged on read – this is extra effort for > the read path. > > > > If you don’t do much in terms of overwrites/deletes, and your partitions > are particularly small, and your data fits in RAM (probably mmap/page cache > of data files, unless you’re using the row cache), reads may be very fast > for you. Certainly individual reads on low-merge workloads can be < 0.1ms. > > > > - Jeff > > > > *From: *Vikas Jaiman <er.vikasjai...@gmail.com> > *Reply-To: *"user@cassandra.apache.org" <user@cassandra.apache.org> > *Date: *Sunday, November 6, 2016 at 12:42 PM > *To: *"user@cassandra.apache.org" <user@cassandra.apache.org> > *Subject: *Are Cassandra writes are faster than reads? > > > > Hi all, > > > > Are Cassandra writes are faster than reads ?? If yes, why is this so? I am > using consistency 1 and data is in memory. > > > > Vikas > > -- > Ben Bromhead > CTO | Instaclustr <https://www.instaclustr.com/> > +1 650 284 9692 > Managed Cassandra / Spark on AWS, Azure and Softlayer > > > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: Are Cassandra writes are faster than reads?
To add some flavor as to how the commitlog implementation is so quick. It only flushes to disk every 10s by default. So writes are effectively done to memory and then to disk asynchronously later on. This is generally accepted to be OK, as the write is also going to other nodes. You can of course change this behavior to flush on each write or to skip the commitlog altogether (danger!). This however will change how "safe" things are from a durability perspective. On Sun, Nov 6, 2016, 12:51 Jeff Jirsa <jeff.ji...@crowdstrike.com> wrote: > Cassandra writes are particularly fast, for a few reasons: > > > > 1) Most writes go to a commitlog (append-only file, written > linearly, so particularly fast in terms of disk operations) and then pushed > to the memTable. Memtable is flushed in batches to the permanent data > files, so it buffers many mutations and then does a sequential write to > persist that data to disk. > > 2) Reads may have to merge data from many data tables on disk. > Because the writes (described very briefly in step 1) write to immutable > files, updates/deletes have to be merged on read – this is extra effort for > the read path. > > > > If you don’t do much in terms of overwrites/deletes, and your partitions > are particularly small, and your data fits in RAM (probably mmap/page cache > of data files, unless you’re using the row cache), reads may be very fast > for you. Certainly individual reads on low-merge workloads can be < 0.1ms. > > > > - Jeff > > > > *From: *Vikas Jaiman <er.vikasjai...@gmail.com> > *Reply-To: *"user@cassandra.apache.org" <user@cassandra.apache.org> > *Date: *Sunday, November 6, 2016 at 12:42 PM > *To: *"user@cassandra.apache.org" <user@cassandra.apache.org> > *Subject: *Are Cassandra writes are faster than reads? > > > > Hi all, > > > > Are Cassandra writes are faster than reads ?? If yes, why is this so? I am > using consistency 1 and data is in memory. > > > > Vikas > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: Handle Leap Seconds with Cassandra
Based on most of what I've said previously pretty much most ways of avoiding your ordering issue of the leap second is going to be a "hack" and there will be some amount of hope involved. If the updates occur more than 300ms apart and you are confident your nodes have clocks that are within 150ms of each other, then I'd close my eyes and hope they all leap second at the same time within that 150ms. If they are less then 300ms (I'm guessing you meant less 300ms), then I would look to figure out what the smallest gap is between those two updates and make sure your nodes clocks are close enough in that gap that the leap second will occur on all nodes within that gap. If that's not good enough, you could just halt those scenarios for 2 seconds over the leap second and then resume them once you've confirmed all clocks have skipped. On Wed, 2 Nov 2016 at 18:13 Anuj Wadehra <anujw_2...@yahoo.co.in> wrote: > Thanks Ben for taking out time for the detailed reply !! > > We dont need strict ordering for all operations but we are looking for > scenarios where 2 quick updates to same column of same row are possible. By > quick updates, I mean >300 ms. Configuring NTP properly (as mentioned in > some blogs in your link) should give fair relative accuracy between the > Cassandra nodes. But leap second takes the clock back for an ENTIRE one > sec (huge) and the probability of old write overwriting the new one > increases drastically. So, we want to be proactive with things. > > I agree that you should avoid such scebaruos with design (if possible). > > Good to know that you guys have setup your own NTP servers as per the > recommendation. Curious..Do you also do some monitoring around NTP? > > > > Thanks > Anuj > > On Fri, 28 Oct, 2016 at 12:25 AM, Ben Bromhead > > <b...@instaclustr.com> wrote: > If you need guaranteed strict ordering in a distributed system, I would > not use Cassandra, Cassandra does not provide this out of the box. I would > look to a system that uses lamport or vector clocks. Based on your > description of how your systems runs at the moment (and how close your > updates are together), you have either already experienced out of order > updates or there is a real possibility you will in the future. > > Sorry to be so dire, but if you do require causal consistency / strict > ordering, you are not getting it at the moment. Distributed systems theory > is really tricky, even for people that are "experts" on distributed systems > over unreliable networks (I would certainly not put myself in that > category). People have made a very good name for themselves by showing that > the vast majority of distributed databases have had bugs when it comes to > their various consistency models and the claims these databases make. > > So make sure you really do need guaranteed causal consistency/strict > ordering or if you can design around it (e.g. using conflict free > replicated data types) or choose a system that is designed to provide it. > > Having said that... here are some hacky things you could do in Cassandra > to try and get this behaviour, which I in no way endorse doing :) > >- Cassandra counters do leverage a logical clock per shard and you >could hack something together with counters and lightweight transactions, >but you would want to do your homework on counters accuracy during before >diving into it... as I don't know if the implementation is safe in the >context of your question. Also this would probably require a significant >rework of your application plus a significant performance hit. I would >invite a counter guru to jump in here... > > >- You can leverage the fact that timestamps are monotonic if you >isolate writes to a single node for a single shared... but you then loose >Cassandra's availability guarantees, e.g. a keyspace with an RF of 1 and a >CL of > ONE will get monotonic timestamps (if generated on the server >side). > > >- Continuing down the path of isolating writes to a single node for a >given shard you could also isolate writes to the primary replica using your >client driver during the leap second (make it a minute either side of the >leap), but again you lose out on availability and you are probably already >experiencing out of ordered writes given how close your writes and updates >are. > > > A note on NTP: NTP is generally fine if you use it to keep the clocks > synced between the Cassandra nodes. If you are interested in how we have > implemented NTP at Instaclustr, see our blogpost on it > https://www.instaclustr.com/blog/2015/11/05/apache-cassandra-synchronization/ > . > > > > Ben > > > On Thu, 27 Oct 2016 at 10:18 Anuj W
Re: Introducing Cassandra 3.7 LTS
We are not publishing the build artefacts for our LTS at the moment as we don't test them on the different distros (debian/ubuntu, centos etc). If anyone wishes to do so feel free to create a PR and submit them! On Wed, 2 Nov 2016 at 11:37 Jesse Hodges <hodges.je...@gmail.com> wrote: > awesome, thanks for the tip! > > -Jesse > > On Wed, Nov 2, 2016 at 12:39 PM, Benjamin Roth <benjamin.r...@jaumo.com> > wrote: > > You can build one on your own very easily. Just check out the desired git > repo and do this: > > > http://stackoverflow.com/questions/8989192/how-to-package-the-cassandra-source-code-into-debian-package > > 2016-11-02 17:35 GMT+01:00 Jesse Hodges <hodges.je...@gmail.com>: > > Just curious, has anybody created a debian package for this? > > Thanks, Jesse > > On Sat, Oct 22, 2016 at 7:45 PM, Kai Wang <dep...@gmail.com> wrote: > > This is awesome! Stability is the king. > > Thank you so much! > > On Oct 19, 2016 2:56 PM, "Ben Bromhead" <b...@instaclustr.com> wrote: > > Hi All > > I am proud to announce we are making available our production build of > Cassandra 3.7 that we run at Instaclustr (both for ourselves and our > customers). Our release of Cassandra 3.7 includes a number of backported > patches from later versions of Cassandra e.g. 3.8 and 3.9 but doesn't > include the new features of these releases. > > You can find our release of Cassandra 3.7 LTS on github here ( > https://github.com/instaclustr/cassandra). You can read more of our > thinking and how this applies to our managed service here ( > https://www.instaclustr.com/blog/2016/10/19/patched-cassandra-3-7/). > > We also have an expanded FAQ about why and how we are approaching 3.x in > this manner (https://github.com/instaclustr/cassandra#cassandra-37-lts), > however I've included the top few question and answers below: > > *Is this a fork?* > No, This is just Cassandra with a different release cadence for those who > want 3.x features but are slightly more risk averse than the current > schedule allows. > > *Why not just use the official release?* > With the 3.x tick-tock branch we have encountered more instability than > with the previous release cadence. We feel that releasing new features > every other release makes it very hard for operators to stabilize their > production environment without bringing in brand new features that are not > battle tested. With the release of Cassandra 3.8 and 3.9 simultaneously the > bug fix branch included new and real-world untested features, specifically > CDC. We have decided to stick with Cassandra 3.7 and instead backport > critical issues and maintain it ourselves rather than trying to stick with > the current Apache Cassandra release cadence. > > *Why backport?* > At Instaclustr we support and run a number of different versions of Apache > Cassandra on behalf of our customers. Over the course of managing Cassandra > for our customers we often encounter bugs. There are existing patches for > some of them, others we patch ourselves. Generally, if we can, we try to > wait for the next official Apache Cassandra release, however in the need to > ensure our customers remain stable and running we will sometimes backport > bugs and write our own hotfixes (which are also submitted back to the > community). > > *Why release it?* > A number of our customers and people in the community have asked if we > would make this available, which we are more than happy to do so. This > repository represents what Instaclustr runs in production for Cassandra 3.7 > and this is our way of helping the community get a similar level of > stability as what you would get from our managed service. > > Cheers > > Ben > > > > -- > Ben Bromhead > CTO | Instaclustr <https://www.instaclustr.com/> > +1 650 284 9692 > Managed Cassandra / Spark on AWS, Azure and Softlayer > > > > > > -- > Benjamin Roth > Prokurist > > Jaumo GmbH · www.jaumo.com > Wehrstraße 46 · 73035 Göppingen · Germany > Phone +49 7161 304880-6 · Fax +49 7161 304880-1 > AG Ulm · HRB 731058 · Managing Director: Jens Kammerer > > > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: Handle Leap Seconds with Cassandra
If you need guaranteed strict ordering in a distributed system, I would not use Cassandra, Cassandra does not provide this out of the box. I would look to a system that uses lamport or vector clocks. Based on your description of how your systems runs at the moment (and how close your updates are together), you have either already experienced out of order updates or there is a real possibility you will in the future. Sorry to be so dire, but if you do require causal consistency / strict ordering, you are not getting it at the moment. Distributed systems theory is really tricky, even for people that are "experts" on distributed systems over unreliable networks (I would certainly not put myself in that category). People have made a very good name for themselves by showing that the vast majority of distributed databases have had bugs when it comes to their various consistency models and the claims these databases make. So make sure you really do need guaranteed causal consistency/strict ordering or if you can design around it (e.g. using conflict free replicated data types) or choose a system that is designed to provide it. Having said that... here are some hacky things you could do in Cassandra to try and get this behaviour, which I in no way endorse doing :) - Cassandra counters do leverage a logical clock per shard and you could hack something together with counters and lightweight transactions, but you would want to do your homework on counters accuracy during before diving into it... as I don't know if the implementation is safe in the context of your question. Also this would probably require a significant rework of your application plus a significant performance hit. I would invite a counter guru to jump in here... - You can leverage the fact that timestamps are monotonic if you isolate writes to a single node for a single shared... but you then loose Cassandra's availability guarantees, e.g. a keyspace with an RF of 1 and a CL of > ONE will get monotonic timestamps (if generated on the server side). - Continuing down the path of isolating writes to a single node for a given shard you could also isolate writes to the primary replica using your client driver during the leap second (make it a minute either side of the leap), but again you lose out on availability and you are probably already experiencing out of ordered writes given how close your writes and updates are. A note on NTP: NTP is generally fine if you use it to keep the clocks synced between the Cassandra nodes. If you are interested in how we have implemented NTP at Instaclustr, see our blogpost on it https://www.instaclustr.com/blog/2015/11/05/apache-cassandra-synchronization/ . Ben On Thu, 27 Oct 2016 at 10:18 Anuj Wadehra <anujw_2...@yahoo.co.in> wrote: > Hi Ben, > > Thanks for your reply. We dont use timestamps in primary key. We rely on > server side timestamps generated by coordinator. So, no functions at > client side would help. > > Yes, drifts can create problems too. But even if you ensure that nodes are > perfectly synced with NTP, you will surely mess up the order of updates > during the leap second(interleaving). Some applications update same column > of same row quickly (within a second ) and reversing the order would > corrupt the data. > > I am interested in learning how people relying on strict order of updates > handle leap second scenario when clock goes back one second(same second is > repeated). What kind of tricks people use to ensure that server side > timestamps are monotonic ? > > As per my understanding NTP slew mode may not be suitable for Cassandra as > it may cause unpredictable drift amongst the Cassandra nodes. Ideas ?? > > > Thanks > Anuj > > > > Sent from Yahoo Mail on Android > <https://overview.mail.yahoo.com/mobile/?.src=Android> > > On Thu, 20 Oct, 2016 at 11:25 PM, Ben Bromhead > > <b...@instaclustr.com> wrote: > http://www.datastax.com/dev/blog/preparing-for-the-leap-second gives a > pretty good overview > > If you are using a timestamp as part of your primary key, this is the > situation where you could end up overwriting data. I would suggest using > timeuuid instead which will ensure that you get different primary keys even > for data inserted at the exact same timestamp. > > The blog post also suggests using certain monotonic timestamp classes in > Java however these will not help you if you have multiple clients that may > overwrite data. > > As for the interleaving or out of order problem, this is hard to address > in Cassandra without resorting to external coordination or LWTs. If you are > relying on a wall clock to guarantee order in a distributed system you will > get yourself into trouble even without leap seconds (clock drift, NTP > inaccur
Re: Introducing Cassandra 3.7 LTS
Thanks Sankalp, we are also reviewing our internal 2.1 list against what you published (though we are trying to upgrade everyone to later versions e.g. 2.2). It's great to compare notes. On Thu, 20 Oct 2016 at 16:19 sankalp kohli <kohlisank...@gmail.com> wrote: > This is awesome. I have send out the patches which we back ported into 2.1 > on the dev list. > > On Wed, Oct 19, 2016 at 4:33 PM, kurt Greaves <k...@instaclustr.com> > wrote: > > > On 19 October 2016 at 21:07, sfesc...@gmail.com <sfesc...@gmail.com> > wrote: > > Wow, thank you for doing this. This sentiment regarding stability seems to > be widespread. Is the team reconsidering the whole tick-tock cadence? If > not, I would add my voice to those asking that it is revisited. > > > There has certainly been discussion regarding the tick-tock cadence, and > it seems safe to say it will change. There hasn't been any official > announcement yet, however. > > Kurt Greaves > k...@instaclustr.com > www.instaclustr.com > > > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: Handle Leap Seconds with Cassandra
http://www.datastax.com/dev/blog/preparing-for-the-leap-second gives a pretty good overview If you are using a timestamp as part of your primary key, this is the situation where you could end up overwriting data. I would suggest using timeuuid instead which will ensure that you get different primary keys even for data inserted at the exact same timestamp. The blog post also suggests using certain monotonic timestamp classes in Java however these will not help you if you have multiple clients that may overwrite data. As for the interleaving or out of order problem, this is hard to address in Cassandra without resorting to external coordination or LWTs. If you are relying on a wall clock to guarantee order in a distributed system you will get yourself into trouble even without leap seconds (clock drift, NTP inaccuracy etc). On Thu, 20 Oct 2016 at 10:30 Anuj Wadehra <anujw_2...@yahoo.co.in> wrote: > Hi, > > I would like to know how you guys handle leap seconds with Cassandra. > > I am not bothered about the livelock issue as we are using appropriate > versions of Linux and Java. I am more interested in finding an optimum > answer for the following question: > > How do you handle wrong ordering of multiple writes (on same row and > column) during the leap second? You may overwrite the new value with old > one (disaster). > > And Downtime is no option :) > > I can see that CASSANDRA-9131 is still open.. > > FYI..we are on 2.0.14 .. > > > Thanks > Anuj > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Introducing Cassandra 3.7 LTS
Hi All I am proud to announce we are making available our production build of Cassandra 3.7 that we run at Instaclustr (both for ourselves and our customers). Our release of Cassandra 3.7 includes a number of backported patches from later versions of Cassandra e.g. 3.8 and 3.9 but doesn't include the new features of these releases. You can find our release of Cassandra 3.7 LTS on github here ( https://github.com/instaclustr/cassandra). You can read more of our thinking and how this applies to our managed service here ( https://www.instaclustr.com/blog/2016/10/19/patched-cassandra-3-7/). We also have an expanded FAQ about why and how we are approaching 3.x in this manner (https://github.com/instaclustr/cassandra#cassandra-37-lts), however I've included the top few question and answers below: *Is this a fork?* No, This is just Cassandra with a different release cadence for those who want 3.x features but are slightly more risk averse than the current schedule allows. *Why not just use the official release?* With the 3.x tick-tock branch we have encountered more instability than with the previous release cadence. We feel that releasing new features every other release makes it very hard for operators to stabilize their production environment without bringing in brand new features that are not battle tested. With the release of Cassandra 3.8 and 3.9 simultaneously the bug fix branch included new and real-world untested features, specifically CDC. We have decided to stick with Cassandra 3.7 and instead backport critical issues and maintain it ourselves rather than trying to stick with the current Apache Cassandra release cadence. *Why backport?* At Instaclustr we support and run a number of different versions of Apache Cassandra on behalf of our customers. Over the course of managing Cassandra for our customers we often encounter bugs. There are existing patches for some of them, others we patch ourselves. Generally, if we can, we try to wait for the next official Apache Cassandra release, however in the need to ensure our customers remain stable and running we will sometimes backport bugs and write our own hotfixes (which are also submitted back to the community). *Why release it?* A number of our customers and people in the community have asked if we would make this available, which we are more than happy to do so. This repository represents what Instaclustr runs in production for Cassandra 3.7 and this is our way of helping the community get a similar level of stability as what you would get from our managed service. Cheers Ben -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: Adding disk capacity to a running node
yup you would need to copy the files across to the new volume from the dir you wanted to give additional space to. Rough steps would look like: 1. Create EBS volume (make it big... like 3TB) 2. Attach to instance 3. Mount/format EBS volume 4. Stop C* 5. Copy full/troublesome directory to the EBS volume 6. Remove copied files (using rsync for the copy / remove step can be a good idea) 7. bind mount EBS volume with the same path as the troublesome directory 8. Start C* back up 9. Let it finish compacting / streaming etc 10. Stop C* 11. remove bind mount 12. copy files back to ephemeral 13. start C* back up 14. repeat on other nodes 15. run repair You can use this process if you somehow end up in a full disk situation. If you end up in a low disk situation you'll have other issues (like corrupt / half written SSTable components), but it's better than nothing Also to maintain your read throughput during this whole thing, double check the EBS volumes read_ahead_kb setting on the block volume and reduce it to something sane like 0 or 16. On Mon, 17 Oct 2016 at 13:42 Seth Edwards <s...@pubnub.com> wrote: > @Ben > > Interesting idea, is this also an option for situations where the disk is > completely full and Cassandra has stopped? (Not that I want to go there). > > If this was the route taken, and we did > > mount --bind /mnt/path/to/large/sstable /mnt/newebs > > We would still need to do some manual copying of files? such as > > mv /mnt/path/to/large/sstable.sd /mnt/newebs ? > > Thanks! > > On Mon, Oct 17, 2016 at 12:59 PM, Ben Bromhead <b...@instaclustr.com> > wrote: > > Yup as everyone has mentioned ephemeral are fine if you run in multiple > AZs... which is pretty much mandatory for any production deployment in AWS > (and other cloud providers) . i2.2xls are generally your best bet for high > read throughput applications on AWS. > > Also on AWS ephemeral storage will generally survive a user initiated > restart. For the times that AWS retires an instance, you get plenty of > notice and it's generally pretty rare. We run over 1000 instances on AWS > and see one forced retirement a month if that. We've never had an instance > pulled from under our feet without warning. > > To add another option for the original question, one thing you can do is > to attach a large EBS drive to the instance and bind mount it to the > directory for the table that has the very large SSTables. You will need to > copy data across to the EBS volume. Let everything compact and then copy > everything back and detach EBS. Latency may be higher than normal on the > node you are doing this on (especially if you are used to i2.2xl > performance). > > This is something we often have to do, when we encounter pathological > compaction situations associated with bootstrapping, adding new DCs or STCS > with a dominant table or people ignore high disk usage warnings :) > > On Mon, 17 Oct 2016 at 12:43 Jeff Jirsa <jeff.ji...@crowdstrike.com> > wrote: > > Ephemeral is fine, you just need to have enough replicas (in enough AZs > and enough regions) to tolerate instances being terminated. > > > > > > > > *From: *Vladimir Yudovin <vla...@winguzone.com> > *Reply-To: *"user@cassandra.apache.org" <user@cassandra.apache.org> > *Date: *Monday, October 17, 2016 at 11:48 AM > *To: *user <user@cassandra.apache.org> > > > *Subject: *Re: Adding disk capacity to a running node > > > > It's extremely unreliable to use ephemeral (local) disks. Even if you > don't stop instance by yourself, it can be restarted on different server in > case of some hardware failure or AWS initiated update. So all node data > will be lost. > > > > Best regards, Vladimir Yudovin, > > > *Winguzone > <https://urldefense.proofpoint.com/v2/url?u=https-3A__winguzone.com-3Ffrom-3Dlist=DQMFaQ=08AGY6txKsvMOP6lYkHQpPMRA1U6kqhAwGa8-0QCg3M=yfYEBHVkX6l0zImlOIBID0gmhluYPD5Jje-3CtaT3ow=ixOxpX-xpw1dJZNpaMT3mepToWX8gzmsVaXFizQLzoU=4q7P9fddEYpXwPR-h9yA_tk5JwR8l6c7cKJ-LQTVcGM=> > - Hosted Cloud Cassandra on Azure and SoftLayer.Launch your cluster in > minutes.* > > > > > > On Mon, 17 Oct 2016 14:45:00 -0400*Seth Edwards <s...@pubnub.com > <s...@pubnub.com>>* wrote > > > > These are i2.2xlarge instances so the disks currently configured as > ephemeral dedicated disks. > > > > On Mon, Oct 17, 2016 at 11:34 AM, Laing, Michael < > michael.la...@nytimes.com> wrote: > > > > You could just expand the size of your ebs volume and extend the file > system. No data is lost - assuming you are running Linux. > > > > > > On Monday, October 17, 2016, Seth Edwards
Re: Adding disk capacity to a running node
ne.com> > wrote: > > > > Yes, Cassandra should keep percent of disk usage equal for all disk. > Compaction process and SSTable flushes will use new disk to distribute both > new and existing data. > > > > Best regards, Vladimir Yudovin, > > > *Winguzone > <https://urldefense.proofpoint.com/v2/url?u=https-3A__winguzone.com-3Ffrom-3Dlist=DQMFaQ=08AGY6txKsvMOP6lYkHQpPMRA1U6kqhAwGa8-0QCg3M=yfYEBHVkX6l0zImlOIBID0gmhluYPD5Jje-3CtaT3ow=ixOxpX-xpw1dJZNpaMT3mepToWX8gzmsVaXFizQLzoU=4q7P9fddEYpXwPR-h9yA_tk5JwR8l6c7cKJ-LQTVcGM=> > - Hosted Cloud Cassandra on Azure and SoftLayer.Launch your cluster in > minutes.* > > > > > > On Mon, 17 Oct 2016 11:43:27 -0400*Seth Edwards <s...@pubnub.com > <s...@pubnub.com>>* wrote > > > > We have a few nodes that are running out of disk capacity at the moment > and instead of adding more nodes to the cluster, we would like to add > another disk to the server and add it to the list of data directories. My > question, is, will Cassandra use the new disk for compactions on sstables > that already exist in the primary directory? > > > > > > > > Thanks! > > > > > > > > CONFIDENTIALITY NOTE: This e-mail and any attachments are confidential and > may be legally privileged. If you are not the intended recipient, do not > disclose, copy, distribute, or use this email or any attachments. If you > have received this in error please let the sender know and then delete the > email and all attachments. > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: Does increment/decrement by 0 generate any commits ?
According to https://issues.apache.org/jira/browse/CASSANDRA-7304 unset values in a prepared statement for a counter does not change the value of the counter. This applies for versions of Cassandra 2.2 and above. I would also look to verify the claimed behaviour myself. On Tue, 11 Oct 2016 at 09:49 Dorian Hoxha <dorian.ho...@gmail.com> wrote: > I just have a bunch of counters in 1 row, and I want to selectively update > them. And I want to keep prepared queries. But I don't want to keep 30 > prepared queries (1 for each counter column, but keep only 1). So in most > cases, I will increment 1 column by positive integer and the others by 0. > > Makes sense ? > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: Is there any way to throttle the memtable flushing throughput?
A few thoughts on the larger problem at hand. The AWS instance type you are using is not appropriate for a production workload. Also with memtable flushes that cause spiky write throughput it sounds like your commitlog is on the same disk as your data directory, combined with the use of non-SSD EBS I'm not surprised this is happening. The small amount of memory on the node could also mean your flush writers are getting backed up (blocked), possibly causing JVM heap pressure and other fun things (you can check this with nodetool tpstats). Before you get into tuning memtable flushing I would do the following: - Reset your commitlog_sync settings back to default - Use an EC2 instance type with at least 15GB of memory, 4 cores and is EBS optimized (dedicated EBS bandwidth) - Use gp2 or io2 EBS volumes - Put your commitlog on a separate EBS volume. - Make sure your memtable_flush_writers are not being blocked, if so increase the number of flush writers (no more than # of cores) - Optimize your read_ahead_kb size and compression_chunk_length to keep those EBS reads as small as possible. Once you have fixed the above, memtable flushing should not be an issue. Even if you can't/don't want to upgrade the instance type, the other steps will help things. Ben On Tue, 11 Oct 2016 at 10:23 Satoshi Hikida <sahik...@gmail.com> wrote: > Hi, > > I'm investigating the read/write performance of the C* (Ver. 2.2.8). > However, I have an issue about memtable flushing which forces the spiky > write throughput. And then it affects the latency of the client's requests. > > So I want to know the answers for the following questions. > > 1. Is there any way that throttling the write throughput of the memtable > flushing? If it exists, how can I do that? > 2. Is there any way to reduce the spike of the write bandwidth during the > memtable flushing? >(I'm in trouble because the delay of the request increases when the > spike of the write bandwidth occurred) > > I'm using one C* node for this investigation. And C* runs on an EC2 > instance (2vCPU, 4GB memory), In addition, I attach two magnetic disks to > the instance, one stores system data(root file system.(/)), the other > stores C* data (data files and commit logs). > > I also changed a few configurations. > - commitlog_sync: batch > - commitlog_sync_batch_window_in_ms: 2 > (Using default value for the other configurations) > > > Regards, > Satoshi > > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: How Fast Does Information Spread With Gossip?
Gossip propagation is generally best modelled by epidemic algorithms. Luckily for us Cassandra's gossip protocol is fairly simply. Cassandra will perform one Gossip Task every second. Within each gossip task it will randomly gossip with another available node in the cluster, it will also possibly attempt to gossip with a down node (based on a random chance that increases as the number of down nodes increases) and if it hasn't gossiped with seed that round it may also attempt to gossip with a defined seed. So Cassandra can do up to 3 rounds per second, however these extra rounds are supposed to be optimizations for improving average case convergence and recovering from split brain scenarios quicker than would normally occur. Assuming just one gossip round per second, for a new piece of information to spread to all members of the cluster via gossip, you would see a worst case performance of O(n) gossip rounds where n is the number of nodes in the cluster. This is because each Cassandra node can gossip to any other node irrespective of topology (a fully connected mesh). There is some ongoing discussion about expanding gossip to utilise partial views of the cluster and exchanging those, or using spanning/broadcast trees to speed up convergence and reduce workload in large clusters (1000+) nodes, see https://issues.apache.org/jira/browse/CASSANDRA-12345 for details. On Fri, 16 Sep 2016 at 01:01 Jens Rantil <jens.ran...@tink.se> wrote: > > Is a minute a reasonable upper bound for most clusters? > > I have no numbers and I'm sure this differs depending on how large your > cluster is. We have a small cluster of around 12 nodes and I statuses > generally propagate in under 5 seconds for sure. So, it will definitely be > less than 1 minute. > > Cheers, > Jens > > On Wed, Sep 14, 2016 at 8:49 PM jerome <jeromefroel...@hotmail.com> wrote: > >> Hi, >> >> >> I was curious if anyone had any kind of statistics or ballpark figures on >> how long it takes information to propagate through a cluster with Gossip? >> I'm particularly interested in how fast information about the liveness of a >> node spreads. For example, in an n-node cluster the median amount of time >> it takes for all nodes to learn that a node went down is f(n) seconds. Is a >> minute a reasonable upper bound for most clusters? Too high, too low? >> >> >> Thanks, >> >> Jerome >> > -- > > Jens Rantil > Backend Developer @ Tink > > Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden > For urgent matters you can reach me at +46-708-84 18 32. > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: SS Tables Files Streaming
Note that incremental repair strategies (2.1+) run anti-compaction against sstables in the range being repaired, so this will prevent overstreaming based on the ranges in the repair session. On Mon, 9 May 2016 at 10:31 Ben Bromhead <b...@instaclustr.com> wrote: > Yup, with repair and particularly bootstrap is there is a decent amount of > "over streaming" of data due to the fact it's just sending an sstable. > > On Fri, 6 May 2016 at 14:49 Anubhav Kale <anubhav.k...@microsoft.com> > wrote: > >> Does repair really send SS Table files as is ? Wouldn’t data for tokens >> be distributed across SS Tables ? >> >> >> >> *From:* Jeff Jirsa [mailto:jeff.ji...@crowdstrike.com] >> *Sent:* Friday, May 6, 2016 2:12 PM >> >> >> *To:* user@cassandra.apache.org >> *Subject:* Re: SS Tables Files Streaming >> >> >> >> Also probably sstableloader / bulk loading interface >> >> >> >> >> >> >> >> >> >> (I don’t think any of these necessarily stream “as-is”, but that’s a >> different conversation I suspect) >> >> >> >> >> >> *From: *Jonathan Haddad >> *Reply-To: *"user@cassandra.apache.org" >> *Date: *Friday, May 6, 2016 at 1:52 PM >> *To: *"user@cassandra.apache.org" >> *Subject: *Re: SS Tables Files Streaming >> >> >> >> Repairs, bootstamp, decommission. >> >> >> >> On Fri, May 6, 2016 at 1:16 PM Anubhav Kale <anubhav.k...@microsoft.com> >> wrote: >> >> Hello, >> >> >> >> In what scenarios can SS Table files on disk from Node 1 go to Node 2 as >> is ? I’m aware this happens in *nodetool rebuild* and I am assuming >> this does *not* happen in repairs. Can someone confirm ? >> >> >> >> The reason I ask is I am working on a solution for backup / restore and I >> need to be sure if I boot a node, start copying over backed up files then >> those files won’t get overwritten by something coming from other nodes. >> >> >> >> Thanks ! >> >> -- > Ben Bromhead > CTO | Instaclustr <https://www.instaclustr.com/> > +1 650 284 9692 > Managed Cassandra / Spark on AWS, Azure and Softlayer > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: SS Tables Files Streaming
Yup, with repair and particularly bootstrap is there is a decent amount of "over streaming" of data due to the fact it's just sending an sstable. On Fri, 6 May 2016 at 14:49 Anubhav Kale <anubhav.k...@microsoft.com> wrote: > Does repair really send SS Table files as is ? Wouldn’t data for tokens be > distributed across SS Tables ? > > > > *From:* Jeff Jirsa [mailto:jeff.ji...@crowdstrike.com] > *Sent:* Friday, May 6, 2016 2:12 PM > > > *To:* user@cassandra.apache.org > *Subject:* Re: SS Tables Files Streaming > > > > Also probably sstableloader / bulk loading interface > > > > > > > > > > (I don’t think any of these necessarily stream “as-is”, but that’s a > different conversation I suspect) > > > > > > *From: *Jonathan Haddad > *Reply-To: *"user@cassandra.apache.org" > *Date: *Friday, May 6, 2016 at 1:52 PM > *To: *"user@cassandra.apache.org" > *Subject: *Re: SS Tables Files Streaming > > > > Repairs, bootstamp, decommission. > > > > On Fri, May 6, 2016 at 1:16 PM Anubhav Kale <anubhav.k...@microsoft.com> > wrote: > > Hello, > > > > In what scenarios can SS Table files on disk from Node 1 go to Node 2 as > is ? I’m aware this happens in *nodetool rebuild* and I am assuming this > does *not* happen in repairs. Can someone confirm ? > > > > The reason I ask is I am working on a solution for backup / restore and I > need to be sure if I boot a node, start copying over backed up files then > those files won’t get overwritten by something coming from other nodes. > > > > Thanks ! > > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: Changing Racks of Nodes
If the rack as defined in Cassandra stays the same (e.g. cassandra-rackdc.properties), things will keep working as expected... except when the actual rack (or fault domain) goes down and you are likely to lose more nodes than expected. If you change the rack as defined in Cassandra, the node will start handling queries it does not have data for. The best way to change the move racks is to decommission the node, then bootstrap it with the new rack settings. On Wed, 20 Apr 2016 at 15:49 Anubhav Kale <anubhav.k...@microsoft.com> wrote: > Hello, > > > > If a running node moves around and changes its rack in the process, when > its back in the cluster (through ignore-rack property), is it a correct > statement that queries will not see some data residing on this node until a > repair is run ? > > > > Or, is it more like the node may get requests for the data it does not own > (meaning data will never “disappear”) ? > > > > I’d appreciate some details on this topic from experts ! > > > > Thanks ! > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: Optional TLS CQL Encryption
Hi Jason If you enable encryption it will be always on. Optional encryption is generally a bad idea (tm). Also always creating a new session every query is also a bad idea (tm) even without the minimal overhead of encryption. If you are really hell bent on doing this you could have a node that is part of the cluster but has -Dcassandra.join_ring=false set in jvm options in cassandra-env.sh so it does not get any data and configure that to have no encryption enabled. This is known as a fat client. Then connect to that specific node whenever you want to do terrible non encrypted things. Having said all that, please don't do this. Cheers On Tue, 19 Apr 2016 at 15:32 Jason J. W. Williams <jasonjwwilli...@gmail.com> wrote: > Hey Guys, > > Is there a way to make TLS encryption optional for the CQL listener? We'd > like to be able to use for remote management connections but not for same > datacenter usage (since the build/up tear down cost is too high for things > that don't use pools). > > Right now it appears if we enable encryption it requires it for all > connections, which definitely is not what we want. > > -J > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: How can I make Cassandra stable in a 2GB RAM node environment ?
+1 for http://opensourceconnections.com/blog/2013/08/31/building- the-perfect-cassandra-test-environment/ <http://opensourceconnections.com/blog/2013/08/31/building-the-perfect-cassandra-test-environment/> We also run Cassandra on t2.mediums for our Developer clusters. You can force Cassandra to do most "memory" things by hitting the disk instead (on disk compaction passes, flush immediately to disk) and by throttling client connections. In fact on the t2 series memory is not the biggest concern, but rather the CPU credit issue. On Mon, 7 Mar 2016 at 11:53 Robert Coli <rc...@eventbrite.com> wrote: > On Fri, Mar 4, 2016 at 8:27 PM, Jack Krupansky <jack.krupan...@gmail.com> > wrote: > >> Please review the minimum hardware requirements as clearly documented: >> >> http://docs.datastax.com/en/cassandra/3.x/cassandra/planning/planPlanningHardware.html >> > > That is a document for Datastax Cassandra, not Apache Cassandra. It's > wonderful that Datastax provides docs, but Datastax Cassandra is a superset > of Apache Cassandra. Presuming that the requirements of one are exactly > equivalent to the requirements of the other is not necessarily reasonable. > > Please adjust your hardware usage to at least meet the clearly documented >> minimum requirements. If you continue to encounter problems once you have >> corrected your configuration error, please resubmit the details with >> updated hardware configuration details. >> > > Disagree. OP specifically stated that they knew this was not a recommended > practice. It does not seem unlikely that they are constrained to use this > hardware for reasons outside of their control. > > >> Just to be clear, development on less than 4 GB is not supported and >> production on less than 8 GB is not supported. Those are not suggestions or >> guidelines or recommendations, they are absolute requirements. >> > > What does "supported" mean here? That Datastax will not provide support if > you do not follow the above recommendations? Because it certainly is > "supported" in the sense of "it can be made to work" ... ? > > The premise of a minimum RAM level seems meaningless without context. How > much data are you serving from your 2GB RAM node? What is the rate of > client requests? > > To be clear, I don't recommend trying to run production Cassandra with > under 8GB of RAM on your node, but "absolute requirement" is a serious > overstatement. > > > http://opensourceconnections.com/blog/2013/08/31/building-the-perfect-cassandra-test-environment/ > > Has some good discussion of how to run Cassandra in a low memory > environment. Maybe someone should tell John that his 64MB of JVM heap for a > test node is 62x too small to be "supported"? :D > > =Rob > > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: „Using Timestamp“ Feature
When using client supplied timestamps you need to ensure the clock on the client is in sync with the nodes in the cluster otherwise behaviour will be unpredictable. On Thu, 18 Feb 2016 at 08:50 Tyler Hobbs <ty...@datastax.com> wrote: > 2016-02-18 2:00 GMT-06:00 Matthias Niehoff < > matthias.nieh...@codecentric.de>: > >> >> * is the 'using timestamp' feature (and providing statement timestamps) >> sufficiently robust and mature to build an application on? >> > > Yes. It's been there since the start of CQL3. > > >> * In a BatchedStatement, can different statements have different >> (explicitly provided) timestamps, or is the BatchedStatement's timestamp >> used for them all? Is this specified / stable behaviour? >> > > Yes, you can separate timestamps per statement. And, in fact, if you > potentially mix inserts and deletes on the same rows, you *should *use > explicit timestamps with different values. See the timestamp notes here: > http://cassandra.apache.org/doc/cql3/CQL.html#batchStmt > > >> * cqhsh reports a syntax error when I use 'using timestamp' with an >> update statement (works with 'insert'). Is there a good reason for this, or >> is it a bug? >> > > The "USING TIMESTAMP" goes in a different place in update statements. It > should be something like: > > UPDATE mytable USING TIMESTAMP ? SET col = ? WHERE key = ? > > > -- > Tyler Hobbs > DataStax <http://datastax.com/> > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: Cassandra nodes reduce disks per node
you can do this in a "rolling" fashion (one node at a time). On Wed, 17 Feb 2016 at 14:03 Branton Davis <branton.da...@spanning.com> wrote: > We're about to do the same thing. It shouldn't be necessary to shut down > the entire cluster, right? > > On Wed, Feb 17, 2016 at 12:45 PM, Robert Coli <rc...@eventbrite.com> > wrote: > >> >> >> On Tue, Feb 16, 2016 at 11:29 PM, Anishek Agarwal <anis...@gmail.com> >> wrote: >>> >>> To accomplish this can I just copy the data from disk1 to disk2 with in >>> the relevant cassandra home location folders, change the cassanda.yaml >>> configuration and restart the node. before starting i will shutdown the >>> cluster. >>> >> >> Yes. >> >> =Rob >> >> > > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: Sudden disk usage
+1 to checking for snapshots. Cassandra by default will automatically snapshot tables before destructive actions like drop or truncate. Some general advice regarding cleanup. Cleanup will result in a temporary increase in both disk I/O load and disk space usage (especially with STCS). It should only be used as part of a planned increase in capacity when you still have plenty of disk space left on your existing nodes. If you are running Cassandra in the cloud (AWS, Azure etc) you can add an EBS volume, copy your sstables to it then bind mount it to the troubled CF directory. This will give you some emergency disk space to let compaction and cleanup do its thing safely. On Tue, 16 Feb 2016 at 10:57 Robert Coli <rc...@eventbrite.com> wrote: > On Sat, Feb 13, 2016 at 4:30 PM, Branton Davis <branton.da...@spanning.com > > wrote: > >> We use SizeTieredCompaction. The nodes were about 67% full and we were >> planning on adding new nodes (doubling the cluster to 6) soon. >> > > Be sure to add those new nodes one at a time. > > Have you checked for, and cleared, old snapshots? Snapshots are > automatically taken at various times and have the unusual property of > growing larger over time. This is because they are hard links of data files > and do not take up disk space of their own until the files they link to are > compacted into new files. > > =Rob > > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: Back to the futex()? :(
TfRa> are my JVM >> args. I realized I neglected to adjust memtable_flush_writers as I was >> writing this--so I'll get on that. Aside from that, I'm not sure what to >> do. (Thanks, again, for reading.) >> >> * They were batched for consistency--I'm hoping to return to using them >> when I'm back at normal load, which is tiny compared to backloading, but >> the impact on performance was eye-opening. >> ___ >> Will Hayworth >> Developer, Engagement Engine >> Atlassian >> >> My pronoun is "they". <http://pronoun.is/they> >> >> >> > -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: Re : Possibility of using 2 different snitches in the Multi_DC cluster
Also you may want to run multiple data centres in the one AWS region (load segmentation, spark etc). +1 GPFS for everything On Wed, 3 Feb 2016 at 07:42 sai krishnam raju potturi <pskraj...@gmail.com> wrote: > thanks a lot Robert. Greatly appreciate it. > > thanks > Sai > > On Tue, Feb 2, 2016 at 6:19 PM, Robert Coli <rc...@eventbrite.com> wrote: > >> On Tue, Feb 2, 2016 at 1:23 PM, sai krishnam raju potturi < >> pskraj...@gmail.com> wrote: >> >>> What is the possibility of using GossipingPropertFileSnitch on >>> datacenters in our private cloud, and Ec2MultiRegionSnitch in AWS? >>> >> >> You should just use GPFS everywhere. >> >> This is also the reason why you should not use EC2MRS if you might ever >> have a DC that is outside of AWS. Just use GPFS. >> >> =Rob >> PS - To answer your actual question... one "can" use different snitches >> on a per node basis, but ONE REALLY REALLY SHOULDN'T CONSIDER THIS A VALID >> APPROACH AND IF ONE TRIES AND FAILS I WILL POINT AND LAUGH AND NOT HELP >> THEM :D >> > > -- Ben Bromhead CTO | Instaclustr +1 650 284 9692
Re: Any tips on how to track down why Cassandra won't cluster?
Check network connectivity. If you are using public addresses as the broadcast, make sure you can telnet from one node to the other nodes public address using the internode port. Last time I looked into something like this, for some reason if you only add a security group id to the allowed traffic in a security group you still need to add public IP addresses for each node in a security groups allowed inbound traffic as well. On Wed, 3 Feb 2016 at 11:49 Richard L. Burton III <mrbur...@gmail.com> wrote: > I'm deploying 2 nodes at the moment using cassandra-dse on Amazon. I > configured it to use EC2Snitch and configured rackdc to use us-east with > rack "1". > > The second node points to the first node as the seed e.g., "seeds": > ["54.*.*.*"] and all of the ports are open. > > Any suggestions on how to track down what might trigger this problem? I'm > not receiving any exceptions. > > > -- > -Richard L. Burton III > @rburton > -- Ben Bromhead CTO | Instaclustr +1 650 284 9692
Re: EC2 storage options for C*
gt;>>>>>> Thank you all for the suggestions. I'm torn between GP2 vs >>>>>>> Ephemeral. GP2 after testing is a viable contender for our workload. The >>>>>>> only worry I have is EBS outages, which have happened. >>>>>>> >>>>>>> On Sunday, January 31, 2016, Jeff Jirsa <jeff.ji...@crowdstrike.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Also in that video - it's long but worth watching >>>>>>>> >>>>>>>> We tested up to 1M reads/second as well, blowing out page cache to >>>>>>>> ensure we weren't "just" reading from memory >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Jeff Jirsa >>>>>>>> >>>>>>>> >>>>>>>> On Jan 31, 2016, at 9:52 AM, Jack Krupansky < >>>>>>>> jack.krupan...@gmail.com> wrote: >>>>>>>> >>>>>>>> How about reads? Any differences between read-intensive and >>>>>>>> write-intensive workloads? >>>>>>>> >>>>>>>> -- Jack Krupansky >>>>>>>> >>>>>>>> On Sun, Jan 31, 2016 at 3:13 AM, Jeff Jirsa < >>>>>>>> jeff.ji...@crowdstrike.com> wrote: >>>>>>>> >>>>>>>>> Hi John, >>>>>>>>> >>>>>>>>> We run using 4T GP2 volumes, which guarantee 10k iops. Even at 1M >>>>>>>>> writes per second on 60 nodes, we didn’t come close to hitting even >>>>>>>>> 50% >>>>>>>>> utilization (10k is more than enough for most workloads). PIOPS is not >>>>>>>>> necessary. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> From: John Wong >>>>>>>>> Reply-To: "user@cassandra.apache.org" >>>>>>>>> Date: Saturday, January 30, 2016 at 3:07 PM >>>>>>>>> To: "user@cassandra.apache.org" >>>>>>>>> Subject: Re: EC2 storage options for C* >>>>>>>>> >>>>>>>>> For production I'd stick with ephemeral disks (aka instance >>>>>>>>> storage) if you have running a lot of transaction. >>>>>>>>> However, for regular small testing/qa cluster, or something you >>>>>>>>> know you want to reload often, EBS is definitely good enough and we >>>>>>>>> haven't >>>>>>>>> had issues 99%. The 1% is kind of anomaly where we have flush blocked. >>>>>>>>> >>>>>>>>> But Jeff, kudo that you are able to use EBS. I didn't go through >>>>>>>>> the video, do you actually use PIOPS or just standard GP2 in your >>>>>>>>> production cluster? >>>>>>>>> >>>>>>>>> On Sat, Jan 30, 2016 at 1:28 PM, Bryan Cheng < >>>>>>>>> br...@blockcypher.com> wrote: >>>>>>>>> >>>>>>>>>> Yep, that motivated my question "Do you have any idea what kind >>>>>>>>>> of disk performance you need?". If you need the performance, its >>>>>>>>>> hard to >>>>>>>>>> beat ephemeral SSD in RAID 0 on EC2, and its a solid, battle tested >>>>>>>>>> configuration. If you don't, though, EBS GP2 will save a _lot_ of >>>>>>>>>> headache. >>>>>>>>>> >>>>>>>>>> Personally, on small clusters like ours (12 nodes), we've found >>>>>>>>>> our choice of instance dictated much more by the balance of price, >>>>>>>>>> CPU, and >>>>>>>>>> memory. We're using GP2 SSD and we find that for our patterns the >>>>>>>>>> disk is >>>>>>>>>> rarely the bottleneck. YMMV, of course. >>>>>>>>>> >>>>>>>>>> On Fri, Jan 29, 2016 at 7:32 PM, Jeff Jirsa < >>>>>>>>>> jeff.ji...@crowdstrike.com> wrote: >>>>>>>>>> >>>>>>>>>>> If you have to ask that question, I strongly recommend m4 or c4 >>>>>>>>>>> instances with GP2 EBS. When you don’t care about replacing a node >>>>>>>>>>> because >>>>>>>>>>> of an instance failure, go with i2+ephemerals. Until then, GP2 EBS >>>>>>>>>>> is >>>>>>>>>>> capable of amazing things, and greatly simplifies life. >>>>>>>>>>> >>>>>>>>>>> We gave a talk on this topic at both Cassandra Summit and AWS >>>>>>>>>>> re:Invent: https://www.youtube.com/watch?v=1R-mgOcOSd4 It’s >>>>>>>>>>> very much a viable option, despite any old documents online that say >>>>>>>>>>> otherwise. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> From: Eric Plowe >>>>>>>>>>> Reply-To: "user@cassandra.apache.org" >>>>>>>>>>> Date: Friday, January 29, 2016 at 4:33 PM >>>>>>>>>>> To: "user@cassandra.apache.org" >>>>>>>>>>> Subject: EC2 storage options for C* >>>>>>>>>>> >>>>>>>>>>> My company is planning on rolling out a C* cluster in EC2. We >>>>>>>>>>> are thinking about going with ephemeral SSDs. The question is this: >>>>>>>>>>> Should >>>>>>>>>>> we put two in RAID 0 or just go with one? We currently run a >>>>>>>>>>> cluster in our >>>>>>>>>>> data center with 2 250gig Samsung 850 EVO's in RAID 0 and we are >>>>>>>>>>> happy with >>>>>>>>>>> the performance we are seeing thus far. >>>>>>>>>>> >>>>>>>>>>> Thanks! >>>>>>>>>>> >>>>>>>>>>> Eric >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>> >>>> >>> >>> >>> -- >>> Steve Robenalt >>> Software Architect >>> sroben...@highwire.org <bza...@highwire.org> >>> (office/cell): 916-505-1785 >>> >>> HighWire Press, Inc. >>> 425 Broadway St, Redwood City, CA 94063 >>> www.highwire.org >>> >>> Technology for Scholarly Communication >>> >> >> > -- Ben Bromhead CTO | Instaclustr +1 650 284 9692
Re: Deploying OpsCenter behind a HTTP(S) proxy
OpsCenter is a little bit tricky to simply just rewrite urls, the lhr requests and rest endpoints it hits are all specified a little differently in the javascript app it loads. We ended up monkey patching a buttload of the js files to get all the requests working properly with our proxy. Everytime a new release of OpsCenter comes out we have to rework it. If you are a DSE customer I would raise it as a support issue :) On 18 June 2015 at 02:29, Spencer Brown lilspe...@gmail.com wrote: First, your firewall should really be your frontend There operational frontend is apache, which is common. You want every url with opscenter in it handled elsewhere. You could also set up proxies for /. cluster-configs, etc... Then there is mod_rewrite, which provides a lot more granularity about when you want what gets handled where.I set up the architectural infrastructure for Orbitz and some major banks, and I'd be happpy to help you out on this. I charge $30/hr., but what you need isn't very complex so we're really just talking $100. On Thu, Jun 18, 2015 at 5:13 AM, Jonathan Ballet jbal...@gfproducts.ch wrote: Hi, I'm looking for information on how to correctly deploy an OpsCenter instance behind a HTTP(S) proxy. I have a running instance of OpsCenter 5.1 reachable at http://opscenter:/opscenter/ but I would like to be able to serve this kind of tool under a single hostname on HTTPS along with other tools of this kind, for easier convenience. I'm currently using Apache as my HTTP front-end and I tried this naive configuration: VirtualHost *:80 ServerName tools ... ProxyPreserveHost On # Proxy to OpsCenter # ProxyPass /opscenter/ http://opscenter:/opscenter/ ProxyPassReverse/opscenter/ http://opscenter:/opscenter/ /VirtualHost This doesn't quite work, as OpsCenter seem to also serve specific endpoints from / directly Of course, it doesn't correctly work, as OpsCenter seem to also serve specific data from / directly, such as: /cluster-configs /TestCluster /meta /rc /tcp Is there something I can configure in OpsCenter so that it serves these URLs from somewhere else, or a list of known URLs that I can remap on the proxy, or better yet, a known proxy configuration to put in front of OpsCenter? Regards, Jonathan -- Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr http://twitter.com/instaclustr | (650) 284 9692
Re: Lucene index plugin for Apache Cassandra
Looks awesome, do you have any examples/benchmarks of using these indexes for various cluster sizes e.g. 20 nodes, 60 nodes, 100s+? On 10 June 2015 at 09:08, Andres de la Peña adelap...@stratio.com wrote: Hi all, With the release of Cassandra 2.1.6, Stratio is glad to present its open source Lucene-based implementation of C* secondary indexes https://github.com/Stratio/cassandra-lucene-index as a plugin that can be attached to Apache Cassandra. Before the above changes, Lucene index was distributed inside a fork of Apache Cassandra, with all the difficulties implied. As of now, the fork is discontinued and new users should use the recently created plugin, which maintains all the features of Stratio Cassandra https://github.com/Stratio/stratio-cassandra. Stratio's Lucene index extends Cassandra’s functionality to provide near real-time distributed search engine capabilities such as with ElasticSearch or Solr, including full text search capabilities, free multivariable search, relevance queries and field-based sorting. Each node indexes its own data, so high availability and scalability is guaranteed. We hope this will be useful to the Apache Cassandra community. Regards, -- Andrés de la Peña http://www.stratio.com/ Avenida de Europa, 26. Ática 5. 3ª Planta 28224 Pozuelo de Alarcón, Madrid Tel: +34 91 352 59 42 // *@stratiobd https://twitter.com/StratioBD* -- Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr http://twitter.com/instaclustr | (650) 284 9692
Re: Spark SQL JDBC Server + DSE
at the email above and delete this email and any attachments and destroy any copies thereof. Any review, retransmission, dissemination, copying or other use of, or taking any action in reliance upon, this information by persons or entities other than the intended recipient is strictly prohibited. *From: *Mohammed Guller moham...@glassbeam.com *Reply-To: *user@cassandra.apache.org *Date: *Thursday, May 28, 2015 at 8:26 PM *To: *user@cassandra.apache.org user@cassandra.apache.org *Subject: *RE: Spark SQL JDBC Server + DSE Anybody out there using DSE + Spark SQL JDBC server? Mohammed *From:* Mohammed Guller [mailto:moham...@glassbeam.com moham...@glassbeam.com] *Sent:* Tuesday, May 26, 2015 6:17 PM *To:* user@cassandra.apache.org *Subject:* Spark SQL JDBC Server + DSE Hi – As I understand, the Spark SQL Thrift/JDBC server cannot be used with the open source C*. Only DSE supports the Spark SQL JDBC server. We would like to find out whether how many organizations are using this combination. If you do use DSE + Spark SQL JDBC server, it would be great if you could share your experience. For example, what kind of issues you have run into? How is the performance? What reporting tools you are using? Thank you! Mohammed -- Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr http://twitter.com/instaclustr | (650) 284 9692
Re: check active queries on cluster
A warning on enabling debug and trace logging on the write path. You will be writing information about every query to disk. If you have any significant volume of requests going through the nodes things will get slow pretty quickly. At least with C* 2.1 and using the default logging config. On 1 June 2015 at 07:34, Sebastian Martinka sebastian.marti...@mercateo.com wrote: You could enable DEBUG logging for org.apache.cassandra.transport.Message and TRACE logging for org.apache.cassandra.cql3.QueryProcessor in the log4j-server.properties file: log4j.logger.org.apache.cassandra.transport.Message=DEBUG log4j.logger.org.apache.cassandra.cql3.QueryProcessor=TRACE Afterwards you get the following output from all PreparedStatements in the system.log file: DEBUG [Native-Transport-Requests:167] 2015-06-01 15:56:15,186 Message.java (line 302) Received: PREPARE INSERT INTO dba_test.cust_view (leid, vid, geoarea, ver) VALUES (?, ?, ?, ?);, v=2 TRACE [Native-Transport-Requests:167] 2015-06-01 15:56:15,187 QueryProcessor.java (line 283) Stored prepared statement 61956319a6d7c84c25414c96edf6e38c with 4 bind markers DEBUG [Native-Transport-Requests:167] 2015-06-01 15:56:15,187 Tracing.java (line 159) request complete DEBUG [Native-Transport-Requests:167] 2015-06-01 15:56:15,187 Message.java (line 309) Responding: RESULT PREPARED 61956319a6d7c84c25414c96edf6e38c [leid(dba_test, cust_view), org.apache.cassandra.db.marshal.UTF8Type][vid(dba_test, cust_view), org.apache.cassandra.db.marshal.UTF8Type][geoarea(dba_test, cust_view), org.apache.cassandra.db.marshal.UTF8Type][ver(dba_test, cust_view), org.apache.cassandra.db.marshal.LongType] (resultMetadata=[0 columns]), v=2 *Von:* Robert Coli [mailto:rc...@eventbrite.com] *Gesendet:* Freitag, 17. April 2015 19:23 *An:* user@cassandra.apache.org *Betreff:* Re: check active queries on cluster On Thu, Apr 16, 2015 at 11:10 PM, Rahul Bhardwaj rahul.bhard...@indiamart.com wrote: We want to track active queries on cassandra cluster. Is there any tool or way to find all active queries on cassandra ? You can get a count of them with : https://issues.apache.org/jira/browse/CASSANDRA-5084 =Rob -- Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr http://twitter.com/instaclustr | (650) 284 9692
Re: Multiple cassandra instances per physical node
@Sean - You can manually change the ports used by Datastax agent using the address.yaml file in the agent install directory. +1 on using racks to separate it out... but it will increase operational complexity somewhat On 26 May 2015 at 08:11, Nate McCall n...@thelastpickle.com wrote: If you're running multiple nodes on a single server, vnodes give you no control over which instance has which key (whereas you can assign initial tokens). Therefore you could have two of your three replicas on the same physical server which, if it goes down, you can't read or write at quorum. Yep. You *will* have overlapping ranges on each physical server so long as Vnodes 'number of nodes in the cluster'. However, can't you use the topology snitch to put both nodes in the same rack? Won't that prevent the issue and still allow you to maintain quorum if a single server goes down? If I have a 20-node cluster with 2 nodes on each physical server, can I use 10 racks to properly segment my partitions? That's a good point, yes. I'd still personally prefer the operational simplicity of simply spacing out token assignments though, but YMMV. -- - Nate McCall Austin, TX @zznate Co-Founder Sr. Technical Consultant Apache Cassandra Consulting http://www.thelastpickle.com -- Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr http://twitter.com/instaclustr | (650) 284 9692
Re: Java 8
DSE 4.6.5 supports Java 8 ( http://docs.datastax.com/en/datastax_enterprise/4.6/datastax_enterprise/RNdse46.html?scroll=RNdse46__rel465) and DSE 4.6.5 is Cassandra 2.0.14 under the hood. I would go with 8 On 7 May 2015 at 04:51, Paulo Motta pauloricard...@gmail.com wrote: First link was broken (sorry), here is the correct link: http://docs.datastax.com/en/cassandra/2.0/cassandra/install/installJREJNAabout_c.html 2015-05-07 8:49 GMT-03:00 Paulo Motta pauloricard...@gmail.com: The official recommendation is to run with Java7 ( http://docs.datastax.com/en/cassandra/2.0/cassandra/install/installJREabout_c.html), mostly to play it safe I guess, however you can probably already run C* with Java8, since it has been stable for a while. We've been running with Java8 for several months now without any noticeable problem. Regarding source compatibility, the official plan is compile with Java8 starting from version 3.0. You may find more information on this ticket: https://issues.apache.org/jira/browse/CASSANDRA-8168 https://issues.apache.org/jira/browse/CASSANDRA-8168 2015-05-07 8:32 GMT-03:00 Stefan Podkowinski stefan.podkowin...@1und1.de : Hi Are there any plans to support Java 8 for Cassandra 2.0, now that Java 7 is EOL? Currently Java 7 is also recommended for 2.1. Are there any reasons not to recommend Java 8 for 2.1? Thanks, Stefan -- Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr http://twitter.com/instaclustr | (650) 284 9692
Re: Data migration
Use SSTableloader it comes with Cassandra and is designed for moving data between clusters and is far simpler than sqoop. it should even work with a schema change like you described (changing columns). It would probably/definitely break if you were dropping tables. Mind you I've never tried sstableloader while schema changes were occurring so happy to be wrong. On 14 April 2015 at 05:40, Prem Yadav ipremya...@gmail.com wrote: Look into sqoop. I believe using sqoop you can transfer data between C* clusters. I haven't tested it though. other option is to write a program to read from one cluster and write the required data to another. On Tue, Apr 14, 2015 at 12:27 PM, skrynnikov_m skrinniko...@epsysoft.com.ua wrote: Hello!!! Need to migrate data from one C* cluster to another periodically. During migration schema can change(add or remove one, two fields). Could you please suggest some tool? -- Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr http://twitter.com/instaclustr | (650) 284 9692
Re: High latencies for simple queries
cqlsh runs on the internal cassandra python drivers: cassandra-pylib and cqlshlib. I would not recommend using them at all (nothing wrong with them, they are just not built with external users in mind). I have never used python-driver in anger so I can't comment on whether it is genuinely slower than the internal C* python driver, but this might be a question for python-driver folk. On 28 March 2015 at 00:34, Artur Siekielski a...@vhex.net wrote: On 03/28/2015 12:13 AM, Ben Bromhead wrote: One other thing to keep in mind / check is that doing these tests locally the cassandra driver will connect using the network stack, whereas postgres supports local connections over a unix domain socket (this is also enabled by default). Unix domain sockets are significantly faster than tcp as you don't have a network stack to traverse. I think any driver using libpq will attempt to use the domain socket when connecting locally. Good catch. I assured that psycopg2 connects through a TCP socket and the numbers increased by about 20%, but it still is an order of magnitude faster than Cassandra. But I'm going to hazard a guess something else is going on with the Cassandra connection as I'm able to get 0.5ms queries locally and that's even with trace turned on. Using python-driver? -- Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr http://twitter.com/instaclustr | (650) 284 9692
Re: High latencies for simple queries
Latency can be so variable even when testing things locally. I quickly
fired up postgres and did the following with psql:
ben=# CREATE TABLE foo(i int, j text, PRIMARY KEY(i));
CREATE TABLE
ben=# \timing
Timing is on.
ben=# INSERT INTO foo VALUES(2, 'yay');
INSERT 0 1
Time: 1.162 ms
ben=# INSERT INTO foo VALUES(3, 'yay');
INSERT 0 1
Time: 1.108 ms
I then fired up a local copy of Cassandra (2.0.12)
cqlsh CREATE KEYSPACE foo WITH replication = { 'class' : 'SimpleStrategy',
'replication_factor' : 1 };
cqlsh USE foo;
cqlsh:foo CREATE TABLE foo(i int PRIMARY KEY, j text);
cqlsh:foo TRACING ON;
Now tracing requests.
cqlsh:foo INSERT INTO foo (i, j) VALUES (1, 'yay');
Tracing session: 7a7dced0-d4b2-11e4-b950-85c3c9bd91a0
activity | timestamp| source
| source_elapsed
---+--+---+
execute_cql3_query | 11:52:55,229 |
127.0.0.1 | 0
Parsing INSERT INTO foo (i, j) VALUES (1, 'yay'); | 11:52:55,229 |
127.0.0.1 | 43
Preparing statement | 11:52:55,229 |
127.0.0.1 |141
Determining replicas for mutation | 11:52:55,229 |
127.0.0.1 |291
Acquiring switchLock read lock | 11:52:55,229 |
127.0.0.1 |403
Appending to commitlog | 11:52:55,229 |
127.0.0.1 |413
Adding to foo memtable | 11:52:55,229 |
127.0.0.1 |432
Request complete | 11:52:55,229 |
127.0.0.1 |541
All this on a mac book pro with 16gb of memory and an SSD
So ymmv?
On 27 March 2015 at 08:28, Tyler Hobbs ty...@datastax.com wrote:
Just to check, are you concerned about minimizing that latency or
maximizing throughput?
I'll that latency is what you're actually concerned about. A fair amount
of that latency is probably happening in the python driver. Although it
can easily execute ~8k operations per second (using cpython), in some
scenarios it can be difficult to guarantee sub-ms latency for an individual
query due to how some of the internals work. In particular, it uses
python's Conditions for cross-thread signalling (from the event loop thread
to the application thread). Unfortunately, python's Condition
implementation includes a loop with a minimum sleep of 1ms if the Condition
isn't already set when you start the wait() call. This is why, with a
single application thread, you will typically see a minimum of 1ms latency.
Another source of similar latencies for the python driver is the Asyncore
event loop, which is used when libev isn't available. I would make sure
that you can use the LibevConnection class with the driver to avoid this.
On Fri, Mar 27, 2015 at 6:24 AM, Artur Siekielski a...@vhex.net wrote:
I'm running Cassandra locally and I see that the execution time for the
simplest queries is 1-2 milliseconds. By a simple query I mean either
INSERT or SELECT from a small table with short keys.
While this number is not high, it's about 10-20 times slower than
Postgresql (even if INSERTs are wrapped in transactions). I know that the
nature of Cassandra compared to Postgresql is different, but for some
scenarios this difference can matter.
The question is: is it normal for Cassandra to have a minimum latency of
1 millisecond?
I'm using Cassandra 2.1.2, python-driver.
--
Tyler Hobbs
DataStax http://datastax.com/
--
Ben Bromhead
Instaclustr | www.instaclustr.com | @instaclustr
http://twitter.com/instaclustr | (650) 284 9692
Re: Arbitrary nested tree hierarchy data model
+1 would love to see how you do it On 27 March 2015 at 07:18, Jonathan Haddad j...@jonhaddad.com wrote: I'd be interested to see that data model. I think the entire list would benefit! On Thu, Mar 26, 2015 at 8:16 PM Robert Wille rwi...@fold3.com wrote: I have a cluster which stores tree structures. I keep several hundred unrelated trees. The largest has about 180 million nodes, and the smallest has 1 node. The largest fanout is almost 400K. Depth is arbitrary, but in practice is probably less than 10. I am able to page through children and siblings. It works really well. Doesn’t sound like its exactly like what you’re looking for, but if you want any pointers on how I went about implementing mine, I’d be happy to share. On Mar 26, 2015, at 3:05 PM, List l...@airstreamcomm.net wrote: Not sure if this is the right place to ask, but we are trying to model a user-generated tree hierarchy in which they create child objects of a root node, and can create an arbitrary number of children (and children of children, and on and on). So far we have looked at storing each tree structure as a single document in JSON format and reading/writing it out in it's entirety, doing materialized paths where we store the root id with every child and the tree structure above the child as a map, and some form of an adjacency list (which does not appear to be very viable as looking up the entire tree would be ridiculous). The hope is to end up with a data model that allows us to display the entire tree quickly, as well as see the entire path to a leaf when selecting that leaf. If anyone has some suggestions/experience on how to model such a tree heirarchy we would greatly appreciate your input. -- Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr http://twitter.com/instaclustr | (650) 284 9692
Re: Really high read latency
wrote:
Also, two control questions:
- Are you using EBS for data storage? It might introduce
additional latencies.
- Are you doing proper paging when querying the keyspace?
Cheers,
Jens
On Mon, Mar 23, 2015 at 5:56 AM, Dave Galbraith
david92galbra...@gmail.com wrote:
Hi! So I've got a table like this:
CREATE TABLE default.metrics (row_time int,attrs varchar,offset
int,value double, PRIMARY KEY(row_time, attrs, offset)) WITH COMPACT
STORAGE AND bloom_filter_fp_chance=0.01 AND caching='KEYS_ONLY' AND
comment='' AND dclocal_read_repair_chance=0 AND gc_grace_seconds=864000
AND
index_interval=128 AND read_repair_chance=1 AND replicate_on_write='true'
AND populate_io_cache_on_flush='false' AND default_time_to_live=0 AND
speculative_retry='NONE' AND memtable_flush_period_in_ms=0 AND
compaction={'class':'DateTieredCompactionStrategy','timestamp_resolution':'MILLISECONDS'}
AND compression={'sstable_compression':'LZ4Compressor'};
and I'm running Cassandra on an EC2 m3.2xlarge out in the cloud, with
4 GB of heap space. So it's timeseries data that I'm doing so I increment
row_time each day, attrs is additional identifying information about
each series, and offset is the number of milliseconds into the day for
each data point. So for the past 5 days, I've been inserting 3k
points/second distributed across 100k distinct attrses. And now when I
try to run queries on this data that look like
SELECT * FROM default.metrics WHERE row_time = 5 AND attrs =
'potatoes_and_jam'
it takes an absurdly long time and sometimes just times out. I did
nodetool cftsats default and here's what I get:
Keyspace: default
Read Count: 59
Read Latency: 397.12523728813557 ms.
Write Count: 155128
Write Latency: 0.3675690719921613 ms.
Pending Flushes: 0
Table: metrics
SSTable count: 26
Space used (live): 35146349027
Space used (total): 35146349027
Space used by snapshots (total): 0
SSTable Compression Ratio: 0.10386468749216264
Memtable cell count: 141800
Memtable data size: 31071290
Memtable switch count: 41
Local read count: 59
Local read latency: 397.126 ms
Local write count: 155128
Local write latency: 0.368 ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.0
Bloom filter space used: 2856
Compacted partition minimum bytes: 104
Compacted partition maximum bytes: 36904729268
Compacted partition mean bytes: 986530969
Average live cells per slice (last five minutes):
501.66101694915255
Maximum live cells per slice (last five minutes): 502.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
Ouch! 400ms of read latency, orders of magnitude higher than it has
any right to be. How could this have happened? Is there something
fundamentally broken about my data model? Thanks!
--
Jens Rantil
Backend engineer
Tink AB
Email: jens.ran...@tink.se
Phone: +46 708 84 18 32
Web: www.tink.se
Facebook https://www.facebook.com/#!/tink.se Linkedin
http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_phototrkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary
Twitter https://twitter.com/tink
--
Ben Bromhead
Instaclustr | www.instaclustr.com | @instaclustr
http://twitter.com/instaclustr | (650) 284 9692
Re: Deleted snapshot files filling up /var/lib/cassandra
Sorry for the late reply. To immediately solve the problem you can restart Cassandra and all the open file descriptors to the deleted snapshots should disappear. As for why it happened I would first address the disk space issue and see if the snapshot errors + open file descriptors issue still occurs (I am unclear as to whether you got the snapshot exception after the disk filled up or before), if you still have issues with repair not letting go of snapshotted files even with free disk space I would look to raise a ticket in Jira. On 17 March 2015 at 12:46, David Wahler dwah...@indeed.com wrote: On Mon, Mar 16, 2015 at 6:51 PM, Ben Bromhead b...@instaclustr.com wrote: If you are running a sequential repair (or have previously run a sequential repair that is still running) Cassandra will still have the file descriptors open for files in the snapshot it is using for the repair operation. Yeah, that aligns with my understanding of how the repair process works. But the cluster has no repair sessions active (I think; when I run nodetool tpstats, the AntiEntropyStage and AntiEntropySessions values are zero on all nodes) and the space still hasn't been freed. -- Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr http://twitter.com/instaclustr | (650) 284 9692
Re: Deleted snapshot files filling up /var/lib/cassandra
If you are running a sequential repair (or have previously run a sequential repair that is still running) Cassandra will still have the file descriptors open for files in the snapshot it is using for the repair operation. From the http://www.datastax.com/dev/blog/repair-in-cassandra *Cassandra 1.2 introduced a new option to repair to help manage the problems caused by the nodes all repairing with each other at the same time, it is call a snapshot repair, or sequential repair. As of Cassandra 2.1, sequential repair is the default, and the old parallel repair an option. Sequential repair has all of the nodes involved take a snapshot, the snapshot lives until the repair finishes, and then is removed. By taking a snapshot, repair can procede in a serial fashion, such that only two nodes are ever comparing with each other at a time. This makes the overall repair process slower, but decreases the burden placed on the nodes, and means you have less impact on reads/writes to the system.* On 16 March 2015 at 16:33, David Wahler dwah...@indeed.com wrote: On Mon, Mar 16, 2015 at 6:12 PM, Ben Bromhead b...@instaclustr.com wrote: Cassandra will by default snapshot your data directory on the following events: TRUNCATE and DROP schema events when you run nodetool repair when you run nodetool snapshot Snapshots are just hardlinks to existing SSTables so the only disk space they take up is for files that have since been compacted away. Disk space for snapshots will be freed when the last link to the files are removed. You can remove all snapshots in a cluster using nodetool clearsnapshot Snapshots will fail if you are out of disk space (this is counterintuitive to the above, but it is true), if you have not increased the number of available file descriptors or if there are permissions issues. Out of curiosity, how often are you running repair? Thanks for the information. We're running repair once per week, as recommended by the Datastax documentation. The repair is staggered to run on one machine at a time with the --partitioner-range option in order to spread out the load. Running nodetool clearsnapshot doesn't free up any space. I'm guessing that because the snapshot files have been deleted from the filesystem, Cassandra thinks the snapshots are already gone. But because it still has the file descriptors open, the disk space hasn't actually been reclaimed. -- Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr http://twitter.com/instaclustr | (650) 284 9692
Re: Deleted snapshot files filling up /var/lib/cassandra
Cassandra will by default snapshot your data directory on the following events: - TRUNCATE and DROP schema events - when you run nodetool repair - when you run nodetool snapshot Snapshots are just hardlinks to existing SSTables so the only disk space they take up is for files that have since been compacted away. Disk space for snapshots will be freed when the last link to the files are removed. You can remove all snapshots in a cluster using nodetool clearsnapshot Snapshots will fail if you are out of disk space (this is counterintuitive to the above, but it is true), if you have not increased the number of available file descriptors or if there are permissions issues. Out of curiosity, how often are you running repair? On 16 March 2015 at 15:52, David Wahler dwah...@indeed.com wrote: On Mon, Mar 16, 2015 at 5:28 PM, Jan cne...@yahoo.com wrote: David; all the packaged installations use the /var/lib/cassandra directory. Could you check your yaml config files and see if you are using this default directory for backups May want to change it to a location with more disk space. We're using the default /var/lib/cassandra as our data directory, mounted as its own LVM volume. I don't see anything in cassandra.yaml about a backup directory. There is an incremental_backups option which is set to false. Increasing the available disk space doesn't really seem like a solution. We have only about 450MB of live data on the most heavily-loaded server, and the space taken up by these deleted files is growing by several GB per day. For now we can work around the problem by periodically restarting servers to close the file handles, but that hurts our availability and seems like a hack. -- Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr http://twitter.com/instaclustr | (650) 284 9692
Re: how to clear data from disk
To clarify on why this behaviour occurs, by default Cassandra will snapshot a table when you perform any destructive action (TRUNCATE, DROP etc) see http://www.datastax.com/documentation/cql/3.0/cql/cql_reference/truncate_r.html To free disk space after such an operation you will always need to clear the snapshots (using either of above suggested methods). Unfortunately this can be a bit painful if you are rotating your tables, say by month, and want to remove the oldest one from disk as your client will need to speak JMX as well. You can disable this behaviour through the use of auto_snapshot in cassandra.yaml. Though I would strongly recommend leaving this feature enabled in any sane production environment and cleaning up snapshots as an independent task!! On 10 March 2015 at 20:43, Patrick McFadin pmcfa...@gmail.com wrote: Or just manually delete the files. The directories are broken down by keyspace and table. Patrick On Mon, Mar 9, 2015 at 7:50 PM, 曹志富 cao.zh...@gmail.com wrote: nodetool clearsnapshot -- Ranger Tsao 2015-03-10 10:47 GMT+08:00 鄢来琼 laiqiong@gtafe.com: Hi ALL, After drop table, I found the data is not removed from disk, I should reduce the gc_grace_seconds before the drop operation. I have to wait for 10 days, but there is not enough disk. Could you tell me there is method to clear the data from disk quickly? Thank you very much! Peter -- Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr http://twitter.com/instaclustr | (650) 284 9692
Re: run cassandra on a small instance
-- GPG me!! gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B -- Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr http://twitter.com/instaclustr | (650) 284 9692
Re: bootstrapping manually when auto_bootstrap=false ?
- In Cassandra yaml set auto_bootstrap = false - Boot node - nodetool rebuild Very similar to http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_add_dc_to_cluster_t.html On 18 December 2014 at 14:04, Kevin Burton bur...@spinn3r.com wrote: I’m trying to figure out the best way to bootstrap our nodes. I *think* I want our nodes to be manually bootstrapped. This way an admin has to explicitly bring up the node in the cluster and I don’t have to worry about a script accidentally provisioning new nodes. The problem is HOW do you do it? I couldn’t find any reference anywhere in the documentation. I *think* I run nodetool repair? but it’s unclear.. -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com -- Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr http://twitter.com/instaclustr | +61 415 936 359
Re: simple data movement ?
Just copy the data directory from each prod node to your test node (and relevant configuration files etc). If your IP addresses are different between test and prod, follow https://engineering.eventbrite.com/changing-the-ip-address-of-a-cassandra-node-with-auto_bootstrapfalse/ On 18 December 2014 at 09:10, Langston, Jim jim.langs...@dynatrace.com wrote: Hi all, I have set up a test environment with C* 2.1.2, wanting to test our applications against it. I currently have C* 1.2.9 in production and want to use that data for testing. What would be a good approach for simply taking a copy of the production data and moving it into the test env and having the test env C* use that data ? The test env. is identical is size, with the difference being the versions of C*. Thanks, Jim The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it -- Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr http://twitter.com/instaclustr | +61 415 936 359
Re: Repair/Compaction Completion Confirmation
https://github.com/BrianGallew/cassandra_range_repair This breaks down the repair operation into very small portions of the ring as a way to try and work around the current fragile nature of repair. Leveraging range repair should go some way towards automating repair (this is how the automatic repair service in DataStax opscenter works, this is how we perform repairs). We have had a lot of success running repairs in a similar manner against vnode enabled clusters. Not 100% bullet proof, but way better than nodetool repair On 28 October 2014 08:32, Tim Heckman t...@pagerduty.com wrote: On Mon, Oct 27, 2014 at 1:44 PM, Robert Coli rc...@eventbrite.com wrote: On Mon, Oct 27, 2014 at 1:33 PM, Tim Heckman t...@pagerduty.com wrote: I know that when issuing some operations via nodetool, the command blocks until the operation is finished. However, is there a way to reliably determine whether or not the operation has finished without monitoring that invocation of nodetool? In other words, when I run 'nodetool repair' what is the best way to reliably determine that the repair is finished without running something equivalent to a 'pgrep' against the command I invoked? I am curious about trying to do the same for major compactions too. This is beyond a FAQ at this point, unfortunately; non-incremental repair is awkward to deal with and probably impossible to automate. In The Future [1] the correct solution will be to use incremental repair, which mitigates but does not solve this challenge entirely. As brief meta commentary, it would have been nice if the project had spent more time optimizing the operability of the critically important thing you must do once a week [2]. https://issues.apache.org/jira/browse/CASSANDRA-5483 =Rob [1] http://www.datastax.com/dev/blog/anticompaction-in-cassandra-2-1 [2] Or, more sensibly, once a month with gc_grace_seconds set to 34 days. Thank you for getting back to me so quickly. Not the answer that I was secretly hoping for, but it is nice to have confirmation. :) Cheers! -Tim -- Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr http://twitter.com/instaclustr | +61 415 936 359
Re: Dependency Hell: STORM 0.9.2 and Cassandra 2.0
I haven't had to deal with this problem specifically and don't know if there is a storm specific solution, but the general Java way of dealing with projects who have conflicting dependencies would be to either exclude one of the conflicting dependencies using maven and see if it works, otherwise rename the conflicting dependency using http://maven.apache.org/plugins/maven-dependency-plugin/usage.html so both projects can use there own versions of guava without the package names conflicting (and the jvm will load the correct classes for each dep). On 25 October 2014 06:13, Gary Zhao garyz...@gmail.com wrote: Hello Anyone encountered the following issue and any workaround? Our Storm topology was written in Clojure. Our team is upgrading one of our storm topology from using cassandra 1.2 to cassandra 2.0, and we have found one problem that is difficult to tackle. Cassandra 2.0Java driver requires google guava 1.6. Unfortuanately, storm 0.9.2 provides a lower version. Because of that, a topology will not be able to contact Cassandra databases. Thanks Gary -- Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr http://twitter.com/instaclustr | +61 415 936 359
Re: Error: No module named cql
It looks like easy_install is using python2.6 and installing cql in the 2.6 packages directory: /usr/lib/python2.6/site-packages/ cqlsh is using the python executable for you environment (which looks like 2.7) and thus is looking for cql in the site packages dir (amongst others). To quickly install the cql module explicitly for python 2.7 run: python2.7 -m easy_install cql Though you might also want to sort out your easy_install so it matches the version of python that is used by default. On 15 October 2014 11:48, Tim Dunphy bluethu...@gmail.com wrote: Hey all, I'm using cassandra 2.1.0 on CentOS 6.5 And when I try to run cqlsh on the command line I get this error: root@beta-new:~] #cqlsh Python CQL driver not installed, or not on PYTHONPATH. You might try easy_install cql. Python: /usr/local/bin/python Module load path: ['/usr/local/apache-cassandra-2.1.0/bin', '/usr/local/lib/python27.zip', '/usr/local/lib/python2.7', '/usr/local/lib/python2.7/plat-linux2', '/usr/local/lib/python2.7/lib-tk', '/usr/local/lib/python2.7/lib-old', '/usr/local/lib/python2.7/lib-dynload', '/root/.local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/site-packages'] Error: No module named cql I tried following the advice from the error and ran that command: [root@beta-new:~] #easy_install cql Searching for cql Best match: cql 1.4.0 Processing cql-1.4.0-py2.6.egg cql 1.4.0 is already the active version in easy-install.pth Using /usr/lib/python2.6/site-packages/cql-1.4.0-py2.6.egg Processing dependencies for cql Finished processing dependencies for cql And that seems to go ok! However when I try to run it again: [root@beta-new:~] #cqlsh Python CQL driver not installed, or not on PYTHONPATH. You might try easy_install cql. Python: /usr/local/bin/python Module load path: ['/usr/local/apache-cassandra-2.1.0/bin', '/usr/local/lib/python27.zip', '/usr/local/lib/python2.7', '/usr/local/lib/python2.7/plat-linux2', '/usr/local/lib/python2.7/lib-tk', '/usr/local/lib/python2.7/lib-old', '/usr/local/lib/python2.7/lib-dynload', '/root/.local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/site-packages'] Error: No module named cql I get the same exact error. How on earth do I break out of this feeback loop? Thanks! Tim -- GPG me!! gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B -- Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr | +61 415 936 359
Re: How to enable client-to-node encrypt communication with Astyanax cassandra client
Haven't personally followed this but give it a go: http://lyubent.github.io/security/planetcassandra/2013/05/31/ssl-for-astyanax.html On 8 October 2014 20:46, Lu, Boying boying...@emc.com wrote: Hi, All, I’m trying to enable client-to-node encrypt communication in Cassandra (2.0.7) with Astyanax client library (version=1.56.48) I found the links about how to enable this feature: http://www.datastax.com/documentation/cassandra/2.0/cassandra/security/secureSSLClientToNode_t.html But this only says how to set up in the server side, but not the client side. Here is my configuration on the server side (in yaml): client_encryption_options: enabled: true keystore: full-path-to-keystore-file *#same file used by Cassandra server* keystore_password: some-password truststore: fullpath-to-truststore-file *#same file used by Cassandra server* truststore_password: some-password # More advanced defaults below: # protocol: TLS # algorithm: SunX509 # store_type: JKS cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA] require_client_auth: true http://www.datastax.com/dev/blog/accessing-secure-dse-clusters-with-cql-native-protocol This link says something about client side, but not how to do it with the Astyanax client library. Searching the Astyanax source codes, I found the class SSLConnectionContext maybe useful And here is my code snippet: AstyanaxContextCluster clusterContext = new AstyanaxContext.Builder() .forCluster(clusterName) .forKeyspace(keyspaceName) .withAstyanaxConfiguration(new AstyanaxConfigurationImpl() .setRetryPolicy(new QueryRetryPolicy(10, 1000))) .withConnectionPoolConfiguration(new ConnectionPoolConfigurationImpl(_clusterName) .setMaxConnsPerHost(1) .setAuthenticationCredentials(credentials) *.setSSLConnectionContext(sslContext)* .setSeeds(String.format(%1$s:%2$d, uri.getHost(), uri.getPort())) ) .buildCluster(ThriftFamilyFactory.getInstance()); But when I tried to connect to the Cassandra server, I got following error: Caused by: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161) at org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:158) at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65) at org.apache.cassandra.thrift.Cassandra$Client.send_login(Cassandra.java:567) at org.apache.cassandra.thrift.Cassandra$Client.login(Cassandra.java:559) at com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.open(ThriftSyncConnectionFactoryImpl.java:203) ... 6 more It looks like that my SSL settings are incorrect. Does anyone know how to resolve this issue? Thanks Boying -- Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr http://twitter.com/instaclustr | +61 415 936 359
Re: Repair taking long time
use https://github.com/BrianGallew/cassandra_range_repair On 30 September 2014 05:24, Ken Hancock ken.hanc...@schange.com wrote: On Mon, Sep 29, 2014 at 2:29 PM, Robert Coli rc...@eventbrite.com wrote: As an aside, you just lose with vnodes and clusters of the size. I presume you plan to grow over appx 9 nodes per DC, in which case you probably do want vnodes enabled. I typically only see discussion on vnodes vs. non-vnodes, but it seems to me that might be more important to discuss the number of vnodes per node. A small cluster having 256 vnodes/node is unwise given some of the sequential operations that are still done. Even if operations were done in parallel, having a 256x increase in parallelization seems an equally bad choice. I've never seen any discussion on how many vnodes per node might be an appropriate answer based a planned cluster size -- does such a thing exist? Ken -- Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr http://twitter.com/instaclustr | +61 415 936 359
Re: best practice for waiting for schema changes to propagate
The system.peers table which is a copy of some gossip info the node has stored, including the schema version. You should query this and wait until all schema versions have converged. http://www.datastax.com/documentation/cql/3.0/cql/cql_using/use_sys_tab_cluster_t.html http://www.datastax.com/dev/blog/the-data-dictionary-in-cassandra-1-2 As ensuring that the driver keeps talking to the node you made the schema change on I would ask the drivers specific mailing list / IRC: - MAILING LIST: https://groups.google.com/a/lists.datastax.com/forum/#!forum/java-driver-user - IRC: #datastax-drivers on irc.freenode.net http://freenode.net/ On 30 September 2014 10:16, Clint Kelly clint.ke...@gmail.com wrote: Hi all, I often have problems with code that I write that uses the DataStax Java driver to create / modify a keyspace or table and then soon after reads the metadata for the keyspace to verify that whatever changes I made the keyspace or table are complete. As an example, I may create a table called `myTableName` and then very soon after do something like: assert(session .getCluster() .getMetaData() .getKeyspace(myKeyspaceName) .getTable(myTableName) != null) I assume this fails sometimes because the default round-robin load balancing policy for the Java driver will send my create-table request to one node and the metadata read to another, and because it takes some time for the table creation to propagate across all of the nodes in my cluster. What is the best way to deal with this problem? Is there a standard way to wait for schema changes to propagate? Best regards, Clint -- Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr http://twitter.com/instaclustr | +61 415 936 359
Re: Experience with multihoming cassandra?
I'm guessing your talking about multi-homing because you want to have multiple tenants (different apps/ teams etc) to make better use of resources ? As Jared mentioned running multiple Cassandra processes on the same hardware that participate in the same cluster doesn't make much sense from a failure domain point of view (it could mess up how C* replicates with replicas for a key being potentially on the same physical server). As for splitting up a server for multi-tenancy purposes this then becomes a question of virtualisation as while there is some multi-tenant support in C* (auth, throttling per keyspace), it is fairly limited at best. There a whole range of options out there ranging from xen, vmware etc through to lightweight virtualisation like linux namespaces with cgroups, etc. I think Spotify run C* in production using namespaces with cgroup iirc and you could using something like docker to help manage this for you. Docker will also help with managing network addressing etc (the multi homed aspect). We've also had a lot of success running C* with docker (and previously SmartOS and solaris zones). Though you will be treading new / undocumented ground and thus expect to have to solve a few issues along the way. On 26 September 2014 04:32, Jared Biel jared.b...@bolderthinking.com wrote: Doing this seems counter-productive to Cassandra's design/use-cases. It's best at home running on a large number of smaller servers rather than a small number of large servers. Also, as you said, you won't get any of the high availability benefits that it offers if you run multiple copies of Cassandra on the same box. On 25 September 2014 16:58, Donald Smith donald.sm...@audiencescience.com wrote: We have large boxes with 256G of RAM and SSDs. From iostat, top, and sar we think the system has excess capacity. Anyone have recommendations about multihoming http://en.wikipedia.org/wiki/Multihoming cassandra on such a node (connecting it to multiple IPs and running multiple cassandras simultaneously)? I’m skeptical, since Cassandra already has built-in multi-threading and since if the node went down multiple nodes would disappear. We’re using C* version 2.0.9. A google/bing search for multihoming cassandra doesn’t turn much up. *Donald A. Smith* | Senior Software Engineer P: 425.201.3900 x 3866 C: (206) 819-5965 F: (646) 443-2333 dona...@audiencescience.com [image: AudienceScience] -- Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr http://twitter.com/instaclustr | +61 415 936 359
Re: DSE install interfering with apache Cassandra 2.1.0
check your cqlshrc file (sometimes in ~/.cassandra) ?
I've been caught out before when playing with a RC of 2.1
On 30 September 2014 01:25, Andrew Cobley a.e.cob...@dundee.ac.uk wrote:
Without the apache cassandra running I ran jps -l on this machine ,the
only result was
338 sun.tool.jps.Jps
The Mac didn’t like the netstat command so I ran
netstat -atp tcp | grep 9160
no result
Also for the native port:
netstat-atp tcp | grep 9042
gave no result (command may be wrong)
So I ran port scan using the network utility (between 0 and 1).
Results as shown:
Port Scan has started…
Port Scanning host: 127.0.0.1
Open TCP Port: 631ipp
Port Scan has completed…
Hope this helps.
Andy
On 29 Sep 2014, at 15:09, Sumod Pawgi spa...@gmail.com wrote:
Please run jps to check which Java services are still running and to
make sure if c* is running. Then please check if 9160 port is in use.
netstat -nltp | grep 9160
This will confirm what is happening in your case.
Sent from my iPhone
On 29-Sep-2014, at 7:15 pm, Andrew Cobley a.e.cob...@dundee.ac.uk wrote:
Hi All,
Just come across this one, I’m at a bit of a loss on how to fix it.
A user here did the following steps
On a MAC
Install Datastax Enterprise (DSE) using the dmg file
test he can connect using the DSE cqlsh window
Unistall DSE (full uninstall which stops the services)
download apache cassandra 2.1.0
unzip
change to the non directory run sudo ./cassandra
Now when he tries to connect using cqlsh from apache cassandra 2.1.0 bin
he gets
Connection error: ('Unable to connect to any servers', {'127.0.0.1':
ConnectionShutdown('Connection AsyncoreConnection(4528514448)
127.0.0.1:9160 (closed) is already closed',)})
This is probably related to
http://mail-archives.apache.org/mod_mbox/incubator-cassandra-user/201409.mbox/%3CCALHCZd7RGSahJUbK32WoTr9JRoA+4K=mrfocmxuk0nbzoqq...@mail.gmail.com%3E
but I can’t see why the uninstall of DSE is leaving the apache cassandra
release cqlsh unable to attach to the apache cassandra runtime.
Ta
Andy
The University of Dundee is a registered Scottish Charity, No: SC015096
The University of Dundee is a registered Scottish Charity, No: SC015096
--
Ben Bromhead
Instaclustr | www.instaclustr.com | @instaclustr
http://twitter.com/instaclustr | +61 415 936 359
Re: DSE install interfering with apache Cassandra 2.1.0
Only recently! Moving off list (c* users bcc'd).
On 30 September 2014 19:20, Andrew Cobley a.e.cob...@dundee.ac.uk wrote:
HI Ben,
yeah, that was it, recovered from the Cassandra summit ?
Andy
On 30 Sep 2014, at 08:19, Ben Bromhead b...@instaclustr.com wrote:
check your cqlshrc file (sometimes in ~/.cassandra) ?
I've been caught out before when playing with a RC of 2.1
On 30 September 2014 01:25, Andrew Cobley a.e.cob...@dundee.ac.uk wrote:
Without the apache cassandra running I ran jps -l on this machine ,the
only result was
338 sun.tool.jps.Jps
The Mac didn’t like the netstat command so I ran
netstat -atp tcp | grep 9160
no result
Also for the native port:
netstat-atp tcp | grep 9042
gave no result (command may be wrong)
So I ran port scan using the network utility (between 0 and 1).
Results as shown:
Port Scan has started…
Port Scanning host: 127.0.0.1
Open TCP Port: 631ipp
Port Scan has completed…
Hope this helps.
Andy
On 29 Sep 2014, at 15:09, Sumod Pawgi spa...@gmail.com wrote:
Please run jps to check which Java services are still running and to
make sure if c* is running. Then please check if 9160 port is in use.
netstat -nltp | grep 9160
This will confirm what is happening in your case.
Sent from my iPhone
On 29-Sep-2014, at 7:15 pm, Andrew Cobley a.e.cob...@dundee.ac.uk
wrote:
Hi All,
Just come across this one, I’m at a bit of a loss on how to fix it.
A user here did the following steps
On a MAC
Install Datastax Enterprise (DSE) using the dmg file
test he can connect using the DSE cqlsh window
Unistall DSE (full uninstall which stops the services)
download apache cassandra 2.1.0
unzip
change to the non directory run sudo ./cassandra
Now when he tries to connect using cqlsh from apache cassandra 2.1.0
bin he gets
Connection error: ('Unable to connect to any servers', {'127.0.0.1':
ConnectionShutdown('Connection AsyncoreConnection(4528514448)
127.0.0.1:9160 (closed) is already closed',)})
This is probably related to
http://mail-archives.apache.org/mod_mbox/incubator-cassandra-user/201409.mbox/%3CCALHCZd7RGSahJUbK32WoTr9JRoA+4K=mrfocmxuk0nbzoqq...@mail.gmail.com%3E
but I can’t see why the uninstall of DSE is leaving the apache
cassandra release cqlsh unable to attach to the apache cassandra runtime.
Ta
Andy
The University of Dundee is a registered Scottish Charity, No: SC015096
The University of Dundee is a registered Scottish Charity, No: SC015096
--
Ben Bromhead
Instaclustr | www.instaclustr.com | @instaclustr
http://twitter.com/instaclustr | +61 415 936 359
The University of Dundee is a registered Scottish Charity, No: SC015096
--
Ben Bromhead
Instaclustr | www.instaclustr.com | @instaclustr
http://twitter.com/instaclustr | +61 415 936 359
Re: no change observed in read latency after switching from EBS to SSD storage
EBS vs local SSD in terms of latency you are using ms as your unit of measurement. If your query runs for 10s you will not notice anything. What is a few less ms for the life of a 10 second query. To reiterate what Rob said. The query is probably slow because of your use case / data model, not the underlying disk. On 17 September 2014 14:21, Tony Anecito adanec...@yahoo.com wrote: If you cached your tables or the database you may not see any difference at all. Regards, -Tony On Tuesday, September 16, 2014 6:36 PM, Mohammed Guller moham...@glassbeam.com wrote: Hi - We are running Cassandra 2.0.5 on AWS on m3.large instances. These instances were using EBS for storage (I know it is not recommended). We replaced the EBS storage with SSDs. However, we didn't see any change in read latency. A query that took 10 seconds when data was stored on EBS still takes 10 seconds even after we moved the data directory to SSD. It is a large query returning 200,000 CQL rows from a single partition. We are reading 3 columns from each row and the combined data in these three columns for each row is around 100 bytes. In other words, the raw data returned by the query is approximately 20MB. I was expecting at least 5-10 times reduction in read latency going from EBS to SSD, so I am puzzled why we are not seeing any change in performance. Does anyone have insight as to why we don't see any performance impact on the reads going from EBS to SSD? Thanks, Mohammed -- Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr http://twitter.com/instaclustr | +61 415 936 359
Re: Moving Cassandra from EC2 Classic into VPC
On 8 Sep 2014, at 12:34 pm, Oleg Dulin oleg.du...@gmail.com wrote: Another idea I had was taking the ec2-snitch configuration and converting it into a Property file snitch. But I still don't understand how to perform this move since I need my newly created VPC instances to have public IPs -- something I would like to avoid. Off the top of my head something like this might work if you want a no downtime approach: Use the gossiping property file snitch in the VPC data centre. Use a public elastic ip for each node. Have the instances in the VPC join your existing cluster. Decommission old cluster. Change the advertised endpoint addresses afterwards to the private addresses for nodes in the VPC using the following: https://engineering.eventbrite.com/changing-the-ip-address-of-a-cassandra-node-with-auto_bootstrapfalse/ Once that is done, remove the elastic IPs from the instances.
Re: Question about EC2 and SSDs
On 5 Sep 2014, at 10:05 am, Steve Robenalt sroben...@highwire.org wrote: We are migrating a small cluster on AWS from instances based on spinning disks (using instance store) to SSD-backed instances and we're trying to pick the proper instance type. Some of the recommendations for spinning disks say to use different drives for log vs data partitions to avoid issues with seek delays and contention for the disk heads. Since SSDs don't have the same seek delays, is it still recommended to use 2 SSD drives? Or is one sufficient? As a side note, splitting the commit log and data dirs into different volumes doesn’t do a whole lot of good on AWS irrespective of whether you are on spinning disks or SSDs. Simply because the volumes presented to the vm may be on the same disk. Just raid the available volumes and be done with it. Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr | +61 415 936 359
Re: Heterogenous cluster and vnodes
Hey, I have a few of VM host (bare metal) machines with varying amounts of free hard drive space on them. For simplicity let’s say I have three machine like so: * Machine 1 - Harddrive 1: 150 GB available. * Machine 2: - Harddrive 1: 150 GB available. - Harddrive 2: 150 GB available. * Machine 3. - Harddrive 1: 150 GB available. I am setting up a Cassandra cluster between them and as I see it I have two options: 1. I set up one Cassandra node/VM per bare metal machine. I assign all free hard drive space to each Cassandra node and I balance the cluster using vnodes proportionally to the amount of free hard drive space (CPU/RAM is not going to be a bottle neck here). 2. I set up four VMs, each running a Cassandra node with equal amount of hard drive space and equal amount of vnodes. Machine 2 runs two VMs. This setup will potentially create a situation where if Machine 2 goes down you may lose two replicas. As the two VMs on Machine 2 might be replicas for the same key. General question: Is any of these preferable to the other? I understand 1) yields lower high-availability (since nodes are on the same hardware). Other way around (2 would be potentially lower availability)… Cassandra thinks two of the vm's are separate when they in fact rely on the same underlying machine. Question about alternative 1: With varying vnodes, can I always be sure that replicas are never put on the same virtual machine? Yes… mostly https://issues.apache.org/jira/browse/CASSANDRA-4123 Or is varying vnodes really only useful/recommended when migrating from machines with varying hardware (like mentioned in [1])? Changing the number of vnodes changes the portion of the ring a node is responsible for. You can use it to account for different types of hardware, you can also use it for creating awesome situations like hotspots if you aren't careful… ymmv. At the end of the day I would throw out the extra hard drive / not use it / put more hard drives in the other machines. Why? Hard drives are cheap and your time as an admin for the cluster isn't. If you do add more hard drives you can also split out the commit log etc onto different disks. I would take less problems over trying to draw every last scrap of performance out of the available hardware any day of the year. Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr | +61 415 936 359
Re: Can't Add AWS Node due to /mnt/cassandra/data directory
Make sure you have also setup the ephemeral drives as a raid device (use mdadm) and mounted it under /mnt/cassandra otherwise your data dir is the os partition which is usually very small. Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr | +61 415 936 359 On 27 Aug 2014, at 8:21 pm, Stephen Portanova sport...@gmail.com wrote: Worked great! Thanks Mark! On Wed, Aug 27, 2014 at 2:00 AM, Mark Reddy mark.l.re...@gmail.com wrote: Hi stephen, I have never added a node via OpsCenter, so this may be a short coming of that process. However in non OpsCenter installs you would have to create the data directories first: sudo mkdir -p /mnt/cassandra/commitlog sudo mkdir -p /mnt/cassandra/data sudo mkdir -p /mnt/cassandra/saved_caches And then give the cassandra user ownership of those directories: sudo chown -R cassandra:cassandra /mnt/cassandra Once this is done Cassandra will have the correct directories and permission to start up. Mark On 27 August 2014 09:50, Stephen Portanova sport...@gmail.com wrote: I already have a 3node m3.large DSE cluster, but I can't seem to add another m3.large node. I'm using the ubuntu-trusty-14.04-amd64-server-20140607.1 (ami-a7fdfee2) AMI (instance-store backed, PV) on AWS, I install java 7 and the JNA, then I go into opscenter to add a node. Things look good for 3 or 4 green circles, until I either get this error: Start Errored: Timed out waiting for Cassandra to start. or this error: Agent Connection Errored: Timed out waiting for agent to connect. I check the system.log and output.log, and they both say: INFO [main] 2014-08-27 08:17:24,642 CLibrary.java (line 121) JNA mlockall successful ERROR [main] 2014-08-27 08:17:24,644 CassandraDaemon.java (line 235) Directory /mnt/cassandra/data doesn't exist ERROR [main] 2014-08-27 08:17:24,645 CassandraDaemon.java (line 239) Has no permission to create /mnt/cassandra/data directory INFO [Thread-1] 2014-08-27 08:17:24,646 DseDaemon.java (line 477) DSE shutting down... ERROR [Thread-1] 2014-08-27 08:17:24,725 CassandraDaemon.java (line 199) Exception in thread Thread[Thread-1,5,main] java.lang.AssertionError at org.apache.cassandra.gms.Gossiper.addLocalApplicationState(Gossiper.java:1263) at com.datastax.bdp.gms.DseState.setActiveStatus(DseState.java:171) at com.datastax.bdp.server.DseDaemon.stop(DseDaemon.java:478) at com.datastax.bdp.server.DseDaemon$1.run(DseDaemon.java:384) My agent.log file says: Node is still provisioning, not attempting to determine ip. INFO [Initialization] 2014-08-27 08:40:57,848 Sleeping for 20s before trying to determine IP over JMX again INFO [Initialization] 2014-08-27 08:41:17,849 Node is still provisioning, not attempting to determine ip. INFO [Initialization] 2014-08-27 08:41:17,849 Sleeping for 20s before trying to determine IP over JMX again INFO [Initialization] 2014-08-27 08:41:37,849 Node is still provisioning, not attempting to determine ip. INFO [Initialization] 2014-08-27 08:41:37,850 Sleeping for 20s before trying to determine IP over JMX again INFO [Initialization] 2014-08-27 08:41:57,850 Node is still provisioning, not attempting to determine ip. I feel like I'm missing something easy with the mount, so if you could point me in the right direction, I would really appreciate it! -- Stephen Portanova (480) 495-2634 -- Stephen Portanova (480) 495-2634
Re: stalled nodetool repair?
https://github.com/mstump/cassandra_range_repair Also very useful. Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr | +61 415 936 359 On 22/08/2014, at 6:12 AM, Robert Coli rc...@eventbrite.com wrote: On Thu, Aug 21, 2014 at 12:32 PM, Kevin Burton bur...@spinn3r.com wrote: How do I watch the progress of nodetool repair. This is a very longstanding operational problem in Cassandra. Repair barely works and is opaque, yet one is expected to run it once a week in the default configuration. An unreasonably-hostile-in-tone-but-otherwise-accurate description of the status quo before the re-write of streaming in 2.0 : https://issues.apache.org/jira/browse/CASSANDRA-5396 A proposal to change the default for gc_grace_seconds to 34 days, so that this fragile and heavyweight operation only has to be done once a month : https://issues.apache.org/jira/browse/CASSANDRA-5850 granted , this is a lot of data, but it would be nice to at least see some progress. Here's the rewrite of streaming, where progress indication improves dramatically over the prior status quo : https://issues.apache.org/jira/browse/CASSANDRA-5286 And here's two open tickets on making repair less opaque (thx yukim@#cassandra) : https://issues.apache.org/jira/browse/CASSANDRA-5483 https://issues.apache.org/jira/browse/CASSANDRA-5839 =Rob
Re: stalled nodetool repair?
Ah sorry that is the original repo, see https://github.com/BrianGallew/cassandra_range_repair for the updated version of the script with vnode support Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr | +61 415 936 359 On 22 Aug 2014, at 2:19 pm, DuyHai Doan doanduy...@gmail.com wrote: Thanks Ben for the link. Still this script does not work with vnodes, which exclude a wide range of C* config On Thu, Aug 21, 2014 at 5:51 PM, Ben Bromhead b...@instaclustr.com wrote: https://github.com/mstump/cassandra_range_repair Also very useful. Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr | +61 415 936 359 On 22/08/2014, at 6:12 AM, Robert Coli rc...@eventbrite.com wrote: On Thu, Aug 21, 2014 at 12:32 PM, Kevin Burton bur...@spinn3r.com wrote: How do I watch the progress of nodetool repair. This is a very longstanding operational problem in Cassandra. Repair barely works and is opaque, yet one is expected to run it once a week in the default configuration. An unreasonably-hostile-in-tone-but-otherwise-accurate description of the status quo before the re-write of streaming in 2.0 : https://issues.apache.org/jira/browse/CASSANDRA-5396 A proposal to change the default for gc_grace_seconds to 34 days, so that this fragile and heavyweight operation only has to be done once a month : https://issues.apache.org/jira/browse/CASSANDRA-5850 granted , this is a lot of data, but it would be nice to at least see some progress. Here's the rewrite of streaming, where progress indication improves dramatically over the prior status quo : https://issues.apache.org/jira/browse/CASSANDRA-5286 And here's two open tickets on making repair less opaque (thx yukim@#cassandra) : https://issues.apache.org/jira/browse/CASSANDRA-5483 https://issues.apache.org/jira/browse/CASSANDRA-5839 =Rob
Re: any plans for coprocessors?
http://www.datastax.com/dev/blog/whats-new-in-cassandra-2-0-prototype-triggers-support http://www.datastax.com/documentation/cql/3.1/cql/cql_reference/trigger_r.html Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr | +61 415 936 359 On 26 Jul 2014, at 11:32 am, Kevin Burton bur...@spinn3r.com wrote: Are there any plans to add coprocessors to cassandra? Embedding logic directly in a cassandra daemon would be nice. -- Founder/CEO Spinn3r.com Location: San Francisco, CA blog: http://burtonator.wordpress.com … or check out my Google+ profile
Re: possible to have TTL on individual collection values?
Create a table with a set as one of the columns using cqlsh, populate with a few records. Connect using the cassandra-cli, run list on your table/cf and you'll see how the sets work. Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr | +61 415 936 359 On 13/07/2014, at 11:19 AM, Kevin Burton bur...@spinn3r.com wrote: On Sat, Jul 12, 2014 at 6:05 PM, Keith Wright kwri...@nanigans.com wrote: Yes each item in the set can have a different TTL so long as they are upserted with commands having differing TTLs. Ah… ok. So you can just insert them with unique UPDATE/INSERT commands with different USING TTLs and it will work. That makes sense. You should read about how collections/maps work in CQL3 in terms of their CQL2 structure. Definitely. I tried but the documentation is all over the map. This is one of the problems with Cassandra IMO. It's evolving so fast that it's difficult to find the correct documentation. -- Founder/CEO Spinn3r.com Location: San Francisco, CA blog: http://burtonator.wordpress.com … or check out my Google+ profile
