Re: monitor and alert tool for open source cassandra
Hello James, This project has provides a Docker Compose environment for using Prometheus and Grafana along with preconfigured files for Prometheus and Graphite consumption. I would, however, prefer Prometheus over Graphite as Grafana's backend. https://github.com/thelastpickle/docker-cassandra-bootstrap Cheers, Joaquin Casares Consultant Austin, TX Apache Cassandra Consulting http://www.thelastpickle.com On Mon, Sep 24, 2018 at 8:44 PM James Shaw wrote: > Adam: >Thanks! > Very helpful. I will take a look. > > James > > On Mon, Sep 24, 2018 at 6:59 PM Adam Zegelin wrote: > >> Hi James, >> >> Prometheus is the most common monitoring solution for K8s-managed >> applications. >> >> There are a number of options to get Cassandra metrics into Prometheus. >> One of which, shameless plug, is something I've been working on for the >> past few months -- cassandra-exporter, a JVM agent that aims to be the >> easiest & fastest way to get Cassandra metrics into Prometheus. >> >> Check it out on GitHub: https://github.com/zegelin/cassandra-exporter >> >> cassandra-exporter is part of Instaclustrs current work-in-progress >> cassandra-operator for Kubernetes. >> If you're interested in running Cassandra on a K8s cluster and would like >> easy scale up/scale down, automatic Prometheus integration, automatic >> backups, etc, then an operator makes it a lot easier to manage. >> Check it out on GitHub: https://github.com/instaclustr/cassandra-operator >> The project is approaching MVP status, and we would certainly appreciate >> any feedback or contributions. >> >> Regards, >> Adam >> >> >> >> On Mon, 24 Sep 2018 at 14:32, James Shaw wrote: >> >>> Hi, there: >>>What are latest good tools for monitoring open source cassandra >>> ? >>> I was used to Datastax opscenter tool, felt all tasks quite easy. Now on >>> new project, open source cassandra, on Kubernetes container/docker, logs in >>> Splunk, feel very challenge. >>> Most wanted metrics are read / write latency, read / write time out. >>> >>> Any advice is welcome. I appreciate your help! >>> >>> Thanks, >>> >>> James >>> >>
Re: EXT: Cassandra Monitoring tool
Hello Aneesh, While this doesn't provide a GUI, tablesnap is a community tool that does a great job at handling backups: https://github.com/JeremyGrosser/tablesnap Cheers, Joaquin Joaquin Casares Consultant Austin, TX Apache Cassandra Consulting http://www.thelastpickle.com On Fri, May 25, 2018 at 11:36 AM, ANEESH KUMAR K.M <anee...@gmail.com> wrote: > Thank you Hari for the details. > > One more question, please suggest me some cluster managing too for > Cassandra cluster. Looking for opensource tools that will support to take > snapshot and restore via GUI. > > Regards, > Aneesh > > On Fri, May 25, 2018 at 10:00 PM, Harikrishnan Pillai < > hpil...@walmartlabs.com> wrote: > >> I assume you are using open source cassandra and you can look at >> Prometheus grafana for cassandra monitoring and lot information available >> in internet regarding how to setup the Prometheus monitoring for cassandra . >> >> >> Sent from my iPhone >> >> > On May 25, 2018, at 9:23 AM, ANEESH KUMAR K.M <anee...@gmail.com> >> wrote: >> > >> > Please suggest me some good cluster monitoring tool for cassandra multi >> region cluster. >> > >> >> - >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: user-h...@cassandra.apache.org >> >> >
Re: Cassandra Monitoring tool
Hello Aneesh, Alain and I also worked on this project that you might find useful: https://github.com/thelastpickle/docker-cassandra-bootstrap It is a Docker Compose environment to showcase the Prometheus graphs that we worked on. There's still a missing update, so do please check back soon, but the majority of what we've worked on is housed in that repo. If you decide to move forward with Prometheus, you can use the curl scripts provided in the repo to upload the charts and all preset alerts into your Grafana environment. If you use the GUI to upload the graphs, you'll lose all preset visual thresholds and alerts, but still keep all completed tooltips/descriptions. Now that I remembered it was missing an update, we should push the new features out sooner rather than later. The new update includes better descriptive text, alert text, and alert fixes. Hope it helps! Joaquin Joaquin Casares Consultant Austin, TX Apache Cassandra Consulting http://www.thelastpickle.com On Fri, May 25, 2018 at 12:54 PM, Alain RODRIGUEZ <arodr...@gmail.com> wrote: > Hello, > > With more details on your criteria, it would be easier. > > I used: > > - SPM from Sematext (Commercial - dashboards out of the box) > - Datastax OpsCenter (Commercial, With DSE only nowadays) > - Grafana/Graphite or Prometheus (Open source - 'do it yourself' - but > some templates exist) > - Datadog (Commercial - dashboards out of the box) > > It depends what you want really. All those worked for me at some point in > time :). > > Note: We built the Dashboards that are out of the box for Datadog. I > believe it should put anyone starting with Cassandra in good conditions to > operate and troubleshot Apache Cassandra. My opinion here is biased most > definitely :). I liked the interfaces of Datadog (using regularly) and > Sematext (old experience) the most. > > C*heers, > --- > Alain Rodriguez - @arodream - al...@thelastpickle.com > France / Spain > > The Last Pickle - Apache Cassandra Consulting > http://www.thelastpickle.com > > 2018-05-25 17:23 GMT+01:00 ANEESH KUMAR K.M <anee...@gmail.com>: > >> Please suggest me some good cluster monitoring tool for cassandra multi >> region cluster. >> >> >
Re: Cassandra reaper
Sure thing Abdul, That's great to hear! Unfortunately, the JMX authentication needs to be in the config file currently. And even if the JMX authentication was stored within Cassandra, we would still need to store connection details within the yaml and storing the JMX authentication credentials within Cassandra may not be ideal from a security standpoint. The UI keeps logs of all the previous repairs, to the best of my knowledge. If you want to completely uninstall Reaper, you can perform a DROP KEYSPACE reaper_db; from within cqlsh, but that would remove all schedules as well. Cheers, Joaquin Joaquin Casares Consultant Austin, TX Apache Cassandra Consulting http://www.thelastpickle.com On Tue, Apr 24, 2018 at 7:49 PM, Abdul Patel <abd786...@gmail.com> wrote: > Thanks Joaquin, > > Yes i used the same and worked fine ..only thing is i had to add userid > password ..which is somewhat annoyoing to keep in comfig file ..can i get > reed of it and still store on reaper_db keyspace? > Also how to clean reaper_db by deleting completed reaper information from > gui? Or any other cleanup is required? > > > On Tuesday, April 24, 2018, Joaquin Casares <joaq...@thelastpickle.com> > wrote: > >> Hello Abdul, >> >> Depending on what you want your backend to be stored on, you'll want to >> use a different file. >> >> So if you want your Reaper state to be stored within a Cassandra cluster, >> which I would recommend, use this file as your base file: >> >> https://github.com/thelastpickle/cassandra-reaper/blob/ >> master/src/packaging/resource/cassandra-reaper-cassandra.yaml >> >> Make a copy of the yaml and include your system-specific settings. Then >> symlink it to the following location: >> >> /etc/cassandra-reaper/cassandra-reaper.yaml >> >> For completeness, this file is an example of how to use a Postgres server >> to store the Reaper state: >> >> https://github.com/thelastpickle/cassandra-reaper/blob/ >> master/src/packaging/resource/cassandra-reaper.yaml >> >> >> Hope that helped! >> >> Joaquin Casares >> Consultant >> Austin, TX >> >> Apache Cassandra Consulting >> http://www.thelastpickle.com >> >> On Tue, Apr 24, 2018 at 7:07 PM, Abdul Patel <abd786...@gmail.com> wrote: >> >>> Thanks >>> >>> But the differnce here is cassandra-reaper-caasandra has more paramters >>> than the cassandra-reaper.yaml >>> Can i just use the 1 file with all details or it looks for one specific >>> file? >>> >>> >>> On Tuesday, April 24, 2018, Joaquin Casares <joaq...@thelastpickle.com> >>> wrote: >>> >>>> Hello Abdul, >>>> >>>> You'll only want one: >>>> >>>> The yaml file used by the service is located at >>>> /etc/cassandra-reaper/cassandra-reaper.yaml and alternate config >>>> templates can be found under /etc/cassandra-reaper/configs. It is >>>> recommended to create a new file with your specific configuration and >>>> symlink it as /etc/cassandra-reaper/cassandra-reaper.yaml to avoid >>>> your configuration from being overwritten during upgrades. >>>> >>>> Adapt the config file to suit your setup and then run `sudo service >>>> cassandra-reaper start`. >>>> >>>> >>>> Source: http://cassandra-reaper.io/docs/download/install/#service-co >>>> nfiguration >>>> >>>> >>>> Hope that helps! >>>> >>>> Joaquin Casares >>>> Consultant >>>> Austin, TX >>>> >>>> Apache Cassandra Consulting >>>> http://www.thelastpickle.com >>>> >>>> On Tue, Apr 24, 2018 at 6:51 PM, Abdul Patel <abd786...@gmail.com> >>>> wrote: >>>> >>>>> Hi All, >>>>> >>>>> For reaper do we need both file or only one? >>>>> >>>>> Cassandra.reaper.yaml >>>>> Cassandra-reaper-cassandra.yaml >>>>> >>>> >>>> >>
Re: Cassandra reaper
Hello Abdul, Depending on what you want your backend to be stored on, you'll want to use a different file. So if you want your Reaper state to be stored within a Cassandra cluster, which I would recommend, use this file as your base file: https://github.com/thelastpickle/cassandra-reaper/blob/master/src/packaging/resource/cassandra-reaper-cassandra.yaml Make a copy of the yaml and include your system-specific settings. Then symlink it to the following location: /etc/cassandra-reaper/cassandra-reaper.yaml For completeness, this file is an example of how to use a Postgres server to store the Reaper state: https://github.com/thelastpickle/cassandra-reaper/blob/master/src/packaging/resource/cassandra-reaper.yaml Hope that helped! Joaquin Casares Consultant Austin, TX Apache Cassandra Consulting http://www.thelastpickle.com On Tue, Apr 24, 2018 at 7:07 PM, Abdul Patel <abd786...@gmail.com> wrote: > Thanks > > But the differnce here is cassandra-reaper-caasandra has more paramters > than the cassandra-reaper.yaml > Can i just use the 1 file with all details or it looks for one specific > file? > > > On Tuesday, April 24, 2018, Joaquin Casares <joaq...@thelastpickle.com> > wrote: > >> Hello Abdul, >> >> You'll only want one: >> >> The yaml file used by the service is located at >> /etc/cassandra-reaper/cassandra-reaper.yaml and alternate config >> templates can be found under /etc/cassandra-reaper/configs. It is >> recommended to create a new file with your specific configuration and >> symlink it as /etc/cassandra-reaper/cassandra-reaper.yaml to avoid your >> configuration from being overwritten during upgrades. >> >> Adapt the config file to suit your setup and then run `sudo service >> cassandra-reaper start`. >> >> >> Source: http://cassandra-reaper.io/docs/download/install/#service- >> configuration >> >> >> Hope that helps! >> >> Joaquin Casares >> Consultant >> Austin, TX >> >> Apache Cassandra Consulting >> http://www.thelastpickle.com >> >> On Tue, Apr 24, 2018 at 6:51 PM, Abdul Patel <abd786...@gmail.com> wrote: >> >>> Hi All, >>> >>> For reaper do we need both file or only one? >>> >>> Cassandra.reaper.yaml >>> Cassandra-reaper-cassandra.yaml >>> >> >>
Re: Cassandra reaper
Hello Abdul, You'll only want one: The yaml file used by the service is located at /etc/cassandra-reaper/cassandra-reaper.yaml and alternate config templates can be found under /etc/cassandra-reaper/configs. It is recommended to create a new file with your specific configuration and symlink it as /etc/cassandra-reaper/cassandra-reaper.yaml to avoid your configuration from being overwritten during upgrades. Adapt the config file to suit your setup and then run `sudo service cassandra-reaper start`. Source: http://cassandra-reaper.io/docs/download/install/#service-configuration Hope that helps! Joaquin Casares Consultant Austin, TX Apache Cassandra Consulting http://www.thelastpickle.com On Tue, Apr 24, 2018 at 6:51 PM, Abdul Patel <abd786...@gmail.com> wrote: > Hi All, > > For reaper do we need both file or only one? > > Cassandra.reaper.yaml > Cassandra-reaper-cassandra.yaml >
Re: [EXTERNAL] Cassandra vs MySQL
Hello Oliver, The first thing that I check when seeing if a workload will work well within Cassandra is by looking at it's read patterns. Once the read patterns can be written down on paper, we need to figure out how the write patterns will populate the required tables. Since you know enough about CQL, it's mainly about checking to see how denormalization is going to work out for on-disk read access requests. Once the read and write patterns are known, we can see if Cassandra will be a good fit for denormalizing your workflow and thereby benefiting from a datastore that can scale out horizontally. If your datastore can scale out horizontally then Cassandra should be faster than a single node MySQL cluster. If your datastore has too many relational requirements, is built in for a queue-like purpose, or other edge cases, then it doesn't matter how fast Cassandra is if it's not the correct tool for the job. I hope that helps align your discovery/investigation process. :) Cheers, Joaquin Casares Consultant Austin, TX Apache Cassandra Consulting http://www.thelastpickle.com On Tue, Mar 20, 2018 at 1:44 PM, Oliver Ruebenacker <cur...@gmail.com> wrote: > > Hello, > > Thanks for all the responses. > > I do know some SQL and CQL, so I know the main differences. You can do > joins in MySQL, but the bigger your data, the less likely you want to do > that. > > If you are a team that wants to consider migrating from MySQL to > Cassandra, you need some reason to believe that it is going to be faster. > What evidence is there? > > Even the Cassandra home page has references to benchmarks to make the > case for Cassandra. Unfortunately, they seem to be about five to six years > old. It doesn't make sense to keep them there if you just can't compare. > > Best, Oliver > > On Tue, Mar 20, 2018 at 1:13 PM, Durity, Sean R < > sean_r_dur...@homedepot.com> wrote: > >> I’m not sure there is a fair comparison. MySQL and Cassandra have >> different ways of solving related (but not necessarily the same) problems >> of storing and retrieving data. >> >> >> >> The data model between MySQL and Cassandra is likely to be very >> different. The key for Cassandra is that you need to model for the queries >> that will be executed. If you cannot know the queries ahead of time, >> Cassandra is not the best choice. If table scans are typically required, >> Cassandra is not a good choice. If you need more than a few hundred tables >> in a cluster, Cassandra is not a good choice. >> >> >> >> If multi-datacenter replication is required, Cassandra is an awesome >> choice. If you are going to always query by a partition key (or primary >> key), Cassandra is a great choice. The nice thing is that the performance >> scales linearly, so additional data is fine (as long as you add nodes) – >> again, if your data model is designed for Cassandra. If you like >> no-downtime upgrades and extreme reliability and availability, Cassandra is >> a great choice. >> >> >> >> Personally, I hope to never have to use/support MySQL again, and I love >> working with Cassandra. But, Cassandra is not the choice for all data >> problems. >> >> >> >> >> >> Sean Durity >> >> >> >> *From:* Oliver Ruebenacker [mailto:cur...@gmail.com] >> *Sent:* Monday, March 12, 2018 3:58 PM >> *To:* user@cassandra.apache.org >> *Subject:* [EXTERNAL] Cassandra vs MySQL >> >> >> >> >> >> Hello, >> >> We have a project currently using MySQL single-node with 5-6TB of data >> and some performance issues, and we plan to add data up to a total size of >> maybe 25-30TB. >> >> We are thinking of migrating to Cassandra. I have been trying to find >> benchmarks or other guidelines to compare MySQL and Cassandra, but most of >> them seem to be five years old or older. >> >> Is there some good more recent material? >> >> Thanks! >> >> Best, Oliver >> >> >> -- >> >> Oliver Ruebenacker >> >> Senior Software Engineer, Diabetes Portal >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.type2diabetesgenetics.org_=DwMFaQ=MtgQEAMQGqekjTjiAhkudQ=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ=j3Lz6pcGNV-FgBKxSeA0Lj6Jh2PC7f53PrXNjGYOPiU=1qS6jO1gSrBpPz6yc33IUcVUA-Q0jKm6jmjJr1u89Tc=>, >> Broad Institute >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.broadinstitute.org_=DwMFaQ=MtgQEAMQGqekjTjiAhkudQ=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ=j3Lz6pcGNV-FgBKxSeA0Lj6Jh2PC7f53PrXNjGYOPiU=bzHFcavS9i7dzp6ahF4aLzSmH_LukAHXbiiLk03LeD8=> >&g
Re: Whch version is the best version to run now?
Hello Kenneth, We've been recommending 3.11.2 for new projects since it comes with quite a few performance improvements and bug fixes not available in 3.0.16. Upgrading to newer versions is taken on a case-by-case basis, but if 2.x is pretty important to the customer, 2.2.12 is our go-to recommendation. Cheers, Joaquin Casares Consultant Austin, TX Apache Cassandra Consulting http://www.thelastpickle.com On Fri, Mar 2, 2018 at 4:37 PM, Kenneth Brotman < kenbrot...@yahoo.com.invalid> wrote: > Seems like a lot of people are running old versions of Cassandra. What is > the best version, most reliable stable version to use now? > > > > Kenneth Brotman >
Re: Indexing for lots of numbers
Hello Oliver, I don't see this being a particularly good fit for Cassandra, but I hope someone confirms this. However, your use case did look interesting for another project I've interacted with indirectly, Pilosa, which used to have a Cassandra backend before a complete Golang rewrite: https://github.com/pilosa/pilosa. It seems as though the intersect PQL command may work for your use case: https://www.pilosa.com/docs/latest/query-language/#intersect. Cheers, Joaquin Joaquin Casares Consultant Austin, TX Apache Cassandra Consulting http://www.thelastpickle.com On Fri, Jan 5, 2018 at 5:15 PM, Oliver Ruebenacker <cur...@gmail.com> wrote: > > Hello, > > Let's say I have a table that has one column with a unique id as a > primary key, and then hundreds of columns of floats, although a large > fraction of cells are empty. > > I want to create an application that allows users to pick one or more > number columns, specify a condition (equality or inequality) for each > number column picked, and then get all rows that satisfy all the conditions > provided. An empty cell would mean condition is not fulfilled. > > (for example, find all rows for which column35 < 0.5 AND column72 > 1.5 > AND column236 < 0.1) > > What's the best way to enable this? A secondary index for each number > column? Would I have to obtain all columns that satisfy one condition and > filter them down, or is there a better way? Does Cassandra support finding > out which index best to start with? > > Thanks! > > Best, Oliver > > -- > Oliver Ruebenacker > Senior Software Engineer, Diabetes Portal > <http://www.type2diabetesgenetics.org/>, Broad Institute > <http://www.broadinstitute.org/> > >
Re: Difference between yum and git
Hi Yuji, Were both tests run from the same machine at close the same time? If not, noisy neighbors may be affecting your performance on different AWS instances. You should verify that you're using the same version of Java during both tests. Also, ensure that you're using the same test instance (that is not running Cassandra) to connect to both Cassandra clusters. Cheers, Joaquin Joaquin Casares Consultant Austin, TX Apache Cassandra Consulting http://www.thelastpickle.com On Wed, May 10, 2017 at 5:01 AM, Yuji Ito <y...@imagine-orb.com> wrote: > Hi all, > > I'm trying a simple performance test. > The test requests select operations (CL.SERIAL or CL.QUORUM) by increasing > the number of threads. > There is the difference of the performance between C* installed by yum and > C* which I built by myself. > What causes the difference? > > I use C* 2.2.8. > One of them was installed by yum (# yum install cassandra22). > Another was acquired by git from https://github.com/apache/cass > andra/tree/cassandra-2.2.8 and built it by myself. > I changed cassandra.yaml to set `commitlog_sync: batch` and > `commitlog_sync_batch_window_in_ms: 2`. > > My environment: > - a cluster has 3 nodes > - node: AWS EC2 m4.large with 200 IOPS EBS volume > - Replication Factor: 3 > - 1 rows > > Result: > ** yum > select (CL.SERIAL) > threads operations/sec > 1 188 > 2 156 > 4 434 > 8 396 > 16 837 > 32 1176 > 64 2206 > 128 4115 > 256 7272 > > ** git > select (CL.SERIAL) > threads operations/sec > 1 192 > 2 162 > 4 264 > 8 446 > 16 733 > 32 1114 > 64 1715 > 128 2776 > 256 3920 > > ** yum > select (CL.QUORUM) > threads operations/sec > 1 434 > 2 909 > 4 1481 > 8 1904 > 16 2666 > 32 3106 > 64 3555 > 128 5000 > 256 9014 > > ** git > select (CL.QUORUM) > threads operations/sec > 1 666 > 2 1538 > 4 2500 > 8 > 16 4210 > 32 5333 > 64 6597 > 128 7356 > 256 8075 > > Thanks, > Yuji > >
Re: OOM on Apache Cassandra on 30 Plus node at the same time
Hello Shravan, Typically asynchronous requests are recommended over batch statements since batch statements will cause more work on the coordinator node while individual requests, when using a TokenAwarePolicy, will hit a specific coordinator, perform a local disk seek, and return the requested information. The only times that using batch statements are ideal is if writing to the same partition key, even if it's across multiple tables when using the same hashing algorithm (like murmur3). Could you provide a bit of insight into what the batch statement was trying to accomplish and how many child statements were bundled up within that batch? Cheers, Joaquin Joaquin Casares Consultant Austin, TX Apache Cassandra Consulting http://www.thelastpickle.com On Fri, Mar 3, 2017 at 11:18 AM, Shravan Ch <chall...@outlook.com> wrote: > Hello, > > More than 30 plus Cassandra servers in the primary DC went down OOM > exception below. What puzzles me is the scale at which it happened (at the > same minute). I will share some more details below. > > System Log: http://pastebin.com/iPeYrWVR > GC Log: http://pastebin.com/CzNNGs0r > > <http://pastebin.com/CzNNGs0r>During the OOM I saw lot of WARNings like > the below (these were there for quite sometime may be weeks) > *WARN [SharedPool-Worker-81] 2017-03-01 19:55:41,209 > BatchStatement.java:252 - Batch of prepared statements for [keyspace.table] > is of size 225455, exceeding specified threshold of 65536 by 159919.* > > * Environment:* > We are using ApacheCassandra-2.1.9 on Multi DC cluster. Primary DC (more > C* nodes on SSD and apps run here) and secondary DC (geographically remote > and more like a DR to primary) on SAS drives. > Cassandra config: > > Java 1.8.0_65 > Garbage Collector: G1GC > memtable_allocation_type: offheap_objects > > Post this OOM I am seeing huge hints pile up on majority of the nodes and > the pending hints keep going up. I have increased HintedHandoff > CoreThreads to 6 but that did not help (I admit that I tried this on one > node to try). > > nodetool compactionstats -H > pending tasks: 3 > compaction typekeyspace table > completed totalunit progress > Compaction system hints > 28.5 GB 92.38 GB bytes 30.85% > > > Appreciate your inputs here. > > Thanks, > Shravan >
Re: Cassandra data model right definition
Hi Mehdi, I can help clarify a few things. As Carlos said, Cassandra is a Wide Column Store. Theoretically a row can have 2 billion columns, but in practice it shouldn't have more than 100 million columns. Cassandra partitions data to certain nodes based on the partition key(s), but does provide the option of setting zero or more clustering keys. Together, the partition key(s) and clustering key(s) form the primary key. When writing to Cassandra, you will need to provide the full primary key, however, when reading from Cassandra, you only need to provide the full partition key. When you only provide the partition key for a read operation, you're able to return all columns that exist on that partition with low latency. These columns are displayed as "CQL rows" to make it easier to reason about. Consider the schema: CREATE TABLE foo ( bar uuid, boz uuid, baz timeuuid, data1 text, data2 text, PRIMARY KEY ((bar, boz), baz) ); When you write to Cassandra you will need to send bar, boz, and baz and optionally data*, if it's relevant for that CQL row. If you chose not to define a data* field for a particular CQL row, then nothing is stored nor allocated on disk. But I wouldn't consider that caveat to be "schema-less". However, all writes to the same bar/boz will end up on the same Cassandra replica set (a configurable number of nodes) and be stored on the same place(s) on disk within the SSTable(s). And on disk, each field that's not a partition key is stored as a column, including clustering keys (this is optimized in Cassandra 3+, but now we're getting deep into internals). In this way you can get fast responses for all activity for bar/boz either over time, or for a specific time, with roughly the same number of disk seeks, with varying lengths on the disk scans. Hope that helps! Joaquin Casares Consultant Austin, TX Apache Cassandra Consulting http://www.thelastpickle.com On Fri, Sep 30, 2016 at 11:40 AM, Carlos Alonso <i...@mrcalonso.com> wrote: > Cassandra is a Wide Column Store http://db-engines.com/en/system/Cassandra > > Carlos Alonso | Software Engineer | @calonso <https://twitter.com/calonso> > > On 30 September 2016 at 18:24, Mehdi Bada <mehdi.b...@dbi-services.com> > wrote: > >> Hi all, >> >> I have a theoritical question: >> - Is Apache Cassandra really a column store? >> Column store mean storing the data as column rather than as a rows. >> >> In fact C* store the data as row, and data is partionned with row key. >> >> Finally, for me, Cassandra is a row oriented schema less DBMS Is it >> true for you also??? >> >> Many thanks in advance for your reply >> >> Best Regards >> Mehdi Bada >> >> >> *Mehdi Bada* | Consultant >> Phone: +41 32 422 96 00 | Mobile: +41 79 928 75 48 | Fax: +41 32 422 96 >> 15 >> dbi services, Rue de la Jeunesse 2, CH-2800 Delémont >> mehdi.b...@dbi-services.com >> www.dbi-services.com >> >> >> >> >> *⇒ dbi services is recruiting Oracle & SQL Server experts ! – Join the >> team >> <http://www.dbi-services.com/fr/dbi-services-et-ses-collaborateurs/offres-emplois-opportunites-carrieres/>* >> > >
Re: TRUNCATE throws OperationTimedOut randomly
Hi George,
Try grepping for WARN and ERROR on the system.logs across all nodes when
you run the command. Could you post any of the recent stacktraces that you
see?
Cheers,
Joaquin Casares
Consultant
Austin, TX
Apache Cassandra Consulting
http://www.thelastpickle.com
On Wed, Sep 28, 2016 at 12:43 PM, George Sigletos <sigle...@textkernel.nl>
wrote:
> Thanks a lot for your reply.
>
> I understand that truncate is an expensive operation. But throwing a
> timeout while truncating a table that is already empty?
>
> A workaround is to set a high --request-timeout when connecting. Even 20
> seconds is not always enough
>
> Kind regards,
> George
>
>
> On Wed, Sep 28, 2016 at 6:59 PM, Edward Capriolo <edlinuxg...@gmail.com>
> wrote:
>
>> Truncate does a few things (based on version)
>> truncate takes snapshots
>> truncate causes a flush
>> in very old versions truncate causes a schema migration.
>>
>> In newer versions like cassandra 3.4 you have this knob.
>>
>> # How long the coordinator should wait for truncates to complete
>> # (This can be much longer, because unless auto_snapshot is disabled
>> # we need to flush first so we can snapshot before removing the data.)
>> truncate_request_timeout_in_ms: 6
>>
>>
>> In older versions you can not control when this call will timeout, it is
>> fairly normal that it does!
>>
>>
>> On Wed, Sep 28, 2016 at 12:50 PM, George Sigletos <sigle...@textkernel.nl
>> > wrote:
>>
>>> Hello,
>>>
>>> I keep executing a TRUNCATE command on an empty table and it throws
>>> OperationTimedOut randomly:
>>>
>>> cassandra@cqlsh> truncate test.mytable;
>>> OperationTimedOut: errors={}, last_host=cassiebeta-01
>>> cassandra@cqlsh> truncate test.mytable;
>>> OperationTimedOut: errors={}, last_host=cassiebeta-01
>>>
>>> Having a 3 node cluster running 2.1.14. No connectivity problems. Has
>>> anybody come across the same error?
>>>
>>> Thanks,
>>> George
>>>
>>>
>>
>
Re: How long/how many days 'nodetool gossipinfo' will have decommissioned nodes info
Hello Techpyassa, Sometimes old gossip information tends to echo around for quite a bit longer than intended. I'm unsure how long the LEFT messages are supposed to be echoed for, but if you want to force the removal of a removed node from gossip, you can use the Assassinate Endpoint JMX command. On larger clusters, running this command synchronously across all machines may be required. Instructions on Assassinate Endpoint can be found here: https://gist.github.com/justenwalker/8338334 If you're planning on recommissioning the same node, upon bootstrapping the gossiped message should change to a JOINING message overwriting the LEFT message. I've personally never checked `nodetool gossipinfo` before recommissioning a node and typically only ensure the node does not appear in `nodetool status`. Hope that helps, Joaquin Casares Consultant Austin, TX Apache Cassandra Consulting http://www.thelastpickle.com On Sun, Sep 25, 2016 at 2:17 PM, Laxmikanth S <techpya...@gmail.com> wrote: > Hi, > > Recently we have decommissioned nodes from Cassandra cluster , but even > after nearly 48 hours 'nodetool gossipinfo' still shows the removed nodes( > as LEFT). > > I just wanted to recommission the same node again. So just wanted to know > , will it create a problem if I recommission the same node(same IP) again > while its state is maintained as LEFT in 'nodetool gossipnfo'. > > > Thanks, > Techpyaasa >
Re: Upgrading from Cassandra 2.1.12 to 3.0.9
Hello Razi, Since you were using a highly stable version of 2.1.x, you may want to stick with using 3.0.9. 3.7 has introduced many great features, but has not been as heavily tested in production as 3.0.9. Running heterogenous clusters, even when using the same major version (e.g. 3.0.8 and 3.0.9), is never recommended. Running a cluster that spans major releases, for longer than the timespan of a routine upgrade, is strongly not advised. Hope that helps! Joaquin Casares Consultant Austin, TX Apache Cassandra Consulting http://www.thelastpickle.com On Fri, Sep 23, 2016 at 8:17 AM, Khaja, Raziuddin (NIH/NLM/NCBI) [C] < raziuddin.kh...@nih.gov> wrote: > Hello all, > > > > I would like to upgrade my Cassandra cluster from 2.1.12 to 3.7. > > > > I have read the following documentation: > > · http://docs.datastax.com/en/latest-upgrade/upgrade/ > cassandra/upgrdCassandra.html > > · https://github.com/apache/cassandra/blob/cassandra-3.7/NEWS.txt > > > > but still had some questions: > > · The upgrdCassandra.html page says : “Upgrade from Cassandra 2.1 > versions later or equal to 2.1.9 directly to Cassandra 3.0.x”, and I am not > sure if this includes 3.x, my question is: Can I upgrade directly to 3.7? > > · Can I run a heterogeneous cluster, with one DC running 2.1.12 > and another DC running 3.7? > > > > Thanks, > > -Razi >
Re: datastax community ami -- broken? --- datastax-agent conflicts with opscenter-agent
Hello again John, The AMI has been patched and tested for both DSE and C* and works for the standard 3 node test. The new code has been pushed to the 2.4 branch so launching a new set of instances will give you an updated AMI. You should now have the newest version of OpsCenter installed, along with the new DataStax Agents (that replace the OpsCenter Agents). Also, I've patched the two bugs for the motd and for allowing the other nodes to join. The issue came from a new release of nodetool that contained some unexpected text that choked up the AMI code as it waited for nodes to come online. Let me know if you see any further issues. Thanks, Joaquin Casares DataStax Software Engineer in Test http://www.datastax.com/what-we-offer/products-services/training/virtual-training On Fri, Dec 6, 2013 at 2:02 PM, Joaquin Casares joaq...@datastax.comwrote: Hey John, Thanks for letting us know. I'm also seeing that the motd gets stuck, but if I ctrl-c during the message and try a `nodetool status` there doesn't appear to be an issue. I'm currently investigating why it's getting stuck. Are you seeing something similar? What happens if you try to run a `sudo service cassandra restart`? Could you send me your /var/log/cassandra/system.log if this still fails? Also, I realize now that the package name for the newest version of OpsCenter changed from opscenter-free to opscenter. I committed that change to our dev AMI and am testing it now. Once this change is made you will no longer have to install agents via OpsCenter since they should already be on the system. That being said, you won't hit the current OpsCenter/DataStax Agent version mismatch you've been hitting. Also, we currently only have one AMI. Each time an instance is launched the newest version of the code is pulled down from https://github.com/riptano/ComboAMI to ensure the code never gets stale and can easily keep up with DSE/C* releases as well as AMI code fixes. I'll reply again as soon as I figure out and patch this motd issue. Thanks, Joaquin Casares DataStax Software Engineer in Test http://www.datastax.com/what-we-offer/products-services/training/virtual-training On Fri, Dec 6, 2013 at 7:16 AM, John R. Frank j...@mit.edu wrote: Hi C* experts, In the last 18hrs or so, I have been having trouble getting cassandra instances to launch using the datastax community AMI. Has anyone else seen this? The instance comes up but then cassandra fails to run. The most informative error message that I've seen so far is in the obscenter agent install log (below) --- see especially this line: datastax-agent conflicts with opscenter-agent I have also attached the ami.log I run into these issues when launching with either the browser page: https://aws.amazon.com/amis/datastax-auto-clustering-ami-2-2 Or command line, like this: ec2-run-instances ami-814ec2e8 -t m1.large --region us-east-1 --key buildbot -n $num_instances -d --clustername Foo --totalnodes $num_instances --version community --opscenter yes Is there a newer AMI? Any advice? jrf Some agent installations failed: - 10.6.133.241: Failure installing agent on 10.6.133.241. Error output: Unable to install the opscenter-agent package. Please check your apt-get configuration as well as the agent install log (/var/log/opscenter-agent/ installer.log). Standard output: Removing old opscenter-agent files. opscenter-agent: unrecognized service Reading package lists... Building dependency tree... Reading state information... 0 upgraded, 0 newly installed, 0 to remove and 171 not upgraded. Starting agent installation process for version 3.2.2 Reading package lists... Building dependency tree... Reading state information... sysstat is already the newest version. 0 upgraded, 0 newly installed, 0 to remove and 171 not upgraded. Selecting previously unselected package opscenter-agent. dpkg: regarding .../opscenter-agent.deb containing opscenter-agent: datastax-agent conflicts with opscenter-agent opscenter-agent (version 3.2.2) is to be installed. opscenter-agent provides opscenter-agent and is to be installed. dpkg: error processing opscenter_agent_setup.vYRzL0Tevn/opscenter-agent.deb (--install): conflicting packages - not installing opscenter-agent Errors were encountered while processing: opscenter_agent_setup.vYRzL0Tevn/opscenter-agent.deb FAILURE: Unable to install the opscenter-agent package. Please check your apt-get configuration as well as the agent install log (/var/log/opscenter-agent/installer.log). Exit code: 1
Re: Questions regarding DataStax AMI
Oh you're right, sorry about that. The concept of keeping older packages was recently implemented and while using --version community, you would need --release 1.0 in order to get 1.0.10. If you are using --version enterprise, you can use --release 2.0 to get DataStax Enterprise 2.0 which comes bundled with 1.0.8. As long as you don't include --analyticsnodes or --searchnodes, you will get vanilla Cassandra. As of now, those are the only options available. Thanks for pointing that out and sorry about the confusion, Joaquin Casares DataStax Software Engineer/Support On Fri, Jul 27, 2012 at 3:50 AM, Tamar Fraenkel ta...@tok-media.com wrote: HI! I tried starting a cluster with Cluster started with these options: --clustername Name --totalnodes 3 --version community --release 1.0.8 But Cassandra's version is 1.1.2 Thanks *Tamar Fraenkel * Senior Software Engineer, TOK Media [image: Inline image 1] ta...@tok-media.com Tel: +972 2 6409736 Mob: +972 54 8356490 Fax: +972 2 5612956 On Thu, Jul 26, 2012 at 9:56 PM, Tamar Fraenkel ta...@tok-media.comwrote: What should be the value to create it with Cassandra 1.0.8 Tamar Sent from my iPod On Jul 26, 2012, at 7:06 PM, Joaquin Casares joaq...@datastax.com wrote: Yes, you can easily do this by using the --release version switch as found here: http://www.datastax.com/docs/1.0/install/install_ami Thanks, Joaquin Casares DataStax Software Engineer/Support On Thu, Jul 26, 2012 at 12:44 AM, Tamar Fraenkel ta...@tok-media.comwrote: Hi! Is there a way to launch EC2 cluster from DataStax latest community AMI that will run Cassandra 1.0.8 and not 1.1.2? Thanks *Tamar Fraenkel * Senior Software Engineer, TOK Media tokLogo.png ta...@tok-media.com Tel: +972 2 6409736 Mob: +972 54 8356490 Fax: +972 2 5612956 tokLogo.png
Re: Questions regarding DataStax AMI
Yes, you can easily do this by using the --release version switch as found here: http://www.datastax.com/docs/1.0/install/install_ami Thanks, Joaquin Casares DataStax Software Engineer/Support On Thu, Jul 26, 2012 at 12:44 AM, Tamar Fraenkel ta...@tok-media.comwrote: Hi! Is there a way to launch EC2 cluster from DataStax latest community AMI that will run Cassandra 1.0.8 and not 1.1.2? Thanks *Tamar Fraenkel * Senior Software Engineer, TOK Media [image: Inline image 1] ta...@tok-media.com Tel: +972 2 6409736 Mob: +972 54 8356490 Fax: +972 2 5612956 tokLogo.png
Re: Snapshot failing on JSON files in 1.1.0
Hello Alain, Yes, the AMI is geared to working out of the box for most dev purposes. We recently spotted an issue with JNA 3.2.7 on Ubuntu 10.10 not being picked up. You can try running `apt-get install libjna-java` but in order for the change to be activated, you must restart your Cassandra service. Could you try removing the older version in the /usr/share/java/ folder and restarting your Cassandra service? You should see a 'JNA mlockall successful' message in your system.log. If not, then remove the remaining jna.jar and replace it with the newest jna.jar as found here: https://github.com/twall/jna. Upon another restart of your node, you should see the 'JNA mlockall successful' message as well as the ability to run the snapshots. I was unable to replicate this on a new instance so all new launches should include the patched code. Do let me know if anyone else sees this issue. Thanks, Joaquin Casares DataStax Software Engineer/Support On Wed, Jun 13, 2012 at 10:28 AM, Alain RODRIGUEZ arodr...@gmail.comwrote: if I do : locate jna /opt/java/64/jdk1.6.0_31/db/docs/html/ref/rrefsqljnaturaljoin.html /root/.m2/repository/net/java/dev/jna /root/.m2/repository/net/java/dev/jna/jna /root/.m2/repository/net/java/dev/jna/jna/3.2.7 /root/.m2/repository/net/java/dev/jna/jna/3.2.7/jna-3.2.7-sources.jar /root/.m2/repository/net/java/dev/jna/jna/3.2.7/jna-3.2.7-sources.jar.sha1 /root/.m2/repository/net/java/dev/jna/jna/3.2.7/jna-3.2.7.jar /root/.m2/repository/net/java/dev/jna/jna/3.2.7/jna-3.2.7.jar.sha1 /root/.m2/repository/net/java/dev/jna/jna/3.2.7/jna-3.2.7.pom /root/.m2/repository/net/java/dev/jna/jna/3.2.7/jna-3.2.7.pom.sha1 /usr/share/doc/libjna-java /usr/share/doc/libjna-java/README.Debian /usr/share/doc/libjna-java/changelog.Debian.gz /usr/share/doc/libjna-java/copyright /usr/share/java/jna-3.2.4.jar /usr/share/java/jna.jar /usr/share/maven-repo/net/java/dev/jna /usr/share/maven-repo/net/java/dev/jna/jna /usr/share/maven-repo/net/java/dev/jna/jna/3.2.4 /usr/share/maven-repo/net/java/dev/jna/jna/debian /usr/share/maven-repo/net/java/dev/jna/jna/3.2.4/jna-3.2.4.jar /usr/share/maven-repo/net/java/dev/jna/jna/3.2.4/jna-3.2.4.pom /usr/share/maven-repo/net/java/dev/jna/jna/debian/jna-debian.jar /usr/share/maven-repo/net/java/dev/jna/jna/debian/jna-debian.pom /var/cache/apt/archives/libjna-java_3.2.4-2_amd64.deb /var/lib/dpkg/info/libjna-java.list /var/lib/dpkg/info/libjna-java.md5sums So what version am I using (jna 3.2.7 or 3.2.4 ?) Should I do an apt-get install libjna-java and does this need a restart ? Alain 2012/6/13 Alain RODRIGUEZ arodr...@gmail.com: Hi Aaron, We are using Ubuntu (AMI Datastax 1.0.9 as I said). Release:10.10 Codename: maverick ERROR [RMI TCP Connection(37732)-10.248.10.94] 2012-06-13 15:00:17,157 CLibrary.java (line 153) Unable to create hard link com.sun.jna.LastErrorException: errno was 1 at org.apache.cassandra.utils.CLibrary.link(Native Method) at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:145) at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:857) at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1412) at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1462) at org.apache.cassandra.db.Table.snapshot(Table.java:210) at org.apache.cassandra.service.StorageService.takeSnapshot(StorageService.java:1710) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427) at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360
Re: EC2 raid0 disks ?
Correct. Not with ephemeral storage. Here's a complete list of the drives that are attached: http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/index.html?InstanceStorage.html#StorageOnInstanceTypes Joaquin Casares DataStax Software Engineer/Support On Tue, Oct 4, 2011 at 4:01 PM, Yang tedd...@gmail.com wrote: it seems that how many virtual disks you can have is fixed: on m2.4xlarge you have 2 disks, while on m2.2xlarge you have only 1, so I can't setup a raid0 on m2.2xlarge am I correct? Thanks Yang
Re: EC2 raid0 disks ?
Hello again, Also, EBS volumes can be attached, but the performance issues cause other issues when running a healthy cluster. From experience running clusters on EBS volumes bring their own set of unique problems and are harder to debug. Here's a quick link that provides a bit more background information on why it's not the best fit for Cassandra. http://www.mail-archive.com/user@cassandra.apache.org/msg11022.html Thanks, Joaquin Casares DataStax Software Engineer/Support 2011/10/4 Yi Yang i...@iyyang.com AFAIK it's around 450G per ephemeral disk. BTW randomly you can get high performance EBS drives as well. Performance are good for DB but are random in IOps. --Original Message-- From: Yang To: user@cassandra.apache.org ReplyTo: user@cassandra.apache.org Subject: EC2 raid0 disks ? Sent: Oct 5, 2011 5:01 AM it seems that how many virtual disks you can have is fixed: on m2.4xlarge you have 2 disks, while on m2.2xlarge you have only 1, so I can't setup a raid0 on m2.2xlarge am I correct? Thanks Yang 從我的 BlackBerry(R) 無線裝置
Re: EC2 raid0 disks ?
Not a problem! We ran a test a few months back and know it's better to use RAID0 vs. just one mount directory. It slips my mind whether the test was also run for multiple directories vs. RAID0 as well. We chose the RAID0 method for the AMI as to avoid confusion and allow for all the sstables to be on one drive and easier to find. This is also in line with many setups that we see in the wild. Thanks, Joaquin Casares DataStax Software Engineer/Support 2011/10/4 Yang tedd...@gmail.com Thanks guys. btw, what is the performance difference between doing a raid0 on the multiple ephemeral drives available, and then assign it to cassandra data directory, vs creating a mount on each of these drives, and then specify all of these to cassandra's data directory list? since these drives are all virtual, would there be any benefit at all in doing a raid0 ? Yang 2011/10/4 Joaquin Casares joaq...@datastax.com: Hello again, Also, EBS volumes can be attached, but the performance issues cause other issues when running a healthy cluster. From experience running clusters on EBS volumes bring their own set of unique problems and are harder to debug. Here's a quick link that provides a bit more background information on why it's not the best fit for Cassandra. http://www.mail-archive.com/user@cassandra.apache.org/msg11022.html Thanks, Joaquin Casares DataStax Software Engineer/Support 2011/10/4 Yi Yang i...@iyyang.com AFAIK it's around 450G per ephemeral disk. BTW randomly you can get high performance EBS drives as well. Performance are good for DB but are random in IOps. --Original Message-- From: Yang To: user@cassandra.apache.org ReplyTo: user@cassandra.apache.org Subject: EC2 raid0 disks ? Sent: Oct 5, 2011 5:01 AM it seems that how many virtual disks you can have is fixed: on m2.4xlarge you have 2 disks, while on m2.2xlarge you have only 1, so I can't setup a raid0 on m2.2xlarge am I correct? Thanks Yang 從我的 BlackBerry(R) 無線裝置
Re: Cassandra start/stop scripts
Did you install via a package or tarball binaries? Packages allow you to run cassandra as a service with sudo service cassandra start|stop But if you are running via tarballs, then yes, running a kill command against Cassandra is the way to do it since Cassandra runs in crash-only mode. Kill pid would work however. Thanks, Joaquin Casares DataStax Software Engineer/Support On Tue, Jul 26, 2011 at 12:19 PM, Priyanka priya...@gmail.com wrote: I do the same way... On Tue, Jul 26, 2011 at 1:07 PM, mcasandra [via [hidden email]http://user/SendEmail.jtp?type=nodenode=6622997i=0] [hidden email] http://user/SendEmail.jtp?type=nodenode=6622997i=1wrote: I need to write cassandra start/stop script. Currently I run cassandra to start and kill -9 to stop. Is this the best way? kill -9 doesn't sound right :) Wondering how others do it. -- If you reply to this email, your message will be added to the discussion below: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cassandra-start-stop-scripts-tp6622977p6622977.html To start a new topic under [hidden email]http://user/SendEmail.jtp?type=nodenode=6622997i=2, email [hidden email]http://user/SendEmail.jtp?type=nodenode=6622997i=3 To unsubscribe from [hidden email]http://user/SendEmail.jtp?type=nodenode=6622997i=4, click here. -- View this message in context: Re: Cassandra start/stop scriptshttp://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cassandra-start-stop-scripts-tp6622977p6622997.html Sent from the cassandra-u...@incubator.apache.org mailing list archivehttp://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/at Nabble.com.
Re: Possible bug in Cassandra 0.8.1?
For the Too many open files error see: http://www.datastax.com/docs/0.8/troubleshooting/index#java-reports-an-error-saying-there-are-too-many-open-files Restart the node and see if the node is able to complete the pending repair this time. Your node may have just been stuck on this error that caused everything else to halt. Joaquin Casares DataStax Software Engineer/Support On Mon, Jul 18, 2011 at 3:12 PM, Sameer Farooqui cassandral...@gmail.comwrote: I'm running into a quirky issue with Brisk 1.0 Beta 2 (w/ Cassandra 0.8.1). I think the last node in our cluster is having problems (10.201.x.x). OpsCenter and nodetool ring (run from that node) show the node as down, but the rest of the cluster sees it as up. If I run nodetool ring from one of the first 11 nodes, I get this output... everything is up: ubuntu@ip-10-85-x-x:~/brisk/resources/cassandra$ bin/nodetool -h localhost ring Address DC RackStatus State Load OwnsToken 148873535527910577765226390751398592512 10.2.x.xDC1 RAC1Up Normal 901.57 GB 12.50% 0 10.116.x.xDC2 RAC1Up Normal 258.22 GB 6.25% 10633823966279326983230456482242756608 10.110.x.xDC1 RAC1Up Normal 129.07 GB 6.25% 21267647932558653966460912964485513216 10.2.x.x DC1 RAC1Up Normal 128.5 GB 12.50% 42535295865117307932921825928971026432 10.114.x.x DC2 RAC1Up Normal 257.31 GB 6.25% 53169119831396634916152282411213783040 10.210.x.x DC1 RAC1Up Normal 128.66 GB 6.25% 63802943797675961899382738893456539648 10.207.x.x DC1 RAC2Up Normal 643.12 GB 12.50% 85070591730234615865843651857942052864 10.85.x.xDC2 RAC1Up Normal 256.76 GB 6.25% 95704415696513942849074108340184809472 10.2.x.xDC1 RAC2Up Normal 128.95 GB 6.25% 106338239662793269832304564822427566080 10.96.x.xDC1 RAC2Up Normal 128.29 GB 12.50% 127605887595351923798765477786913079296 10.194.x.xDC2 RAC1Up Normal 257.14 GB 6.25% 138239711561631250781995934269155835904 10.201.x.xDC1 RAC2Up Normal 129.45 GB 6.25% 148873535527910577765226390751398592512 However, OpsCenter shows the last node (10.201.x.x) as unresponsive: http://blueplastic.com/accenture/unresponsive.PNG And if I try to run nodetool ring from the 10.201.x.x node, I get connection errors like this: ubuntu@ip-10-194-x-x:~/brisk/resources/cassandra$ bin/nodetool -h localhost ring Error connection to remote JMX agent! java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: java.net.SocketTimeoutException: Read timed out] at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:338) at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248) at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:141) at org.apache.cassandra.tools.NodeProbe.init(NodeProbe.java:111) at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:559) Caused by: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: java.net.SocketTimeoutException: Read timed out] at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:101) tpstats command also didn't work: ubuntu@ip-10-194-x-x:~/brisk/resources/cassandra$ bin/nodetool -h localhost tpstats Error connection to remote JMX agent! java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.Co nnectIOException: error during JRMP connection establishment; nested exception is: java.net.SocketTimeoutException: Read timed out] at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:338) Looking at what's listening on 7199 on the node shows a bunch of results: ubuntu@ip-10-194-x-x:~/brisk/resources/cassandra$ sudo netstat -anp | grep 7199 tcp0 0 0.0.0.0:71990.0.0.0:* LISTEN 1459/java tcp8 0 10.194.x.x:7199 10.2.x.x:40135 CLOSE_WAIT - tcp8 0 127.0.0.1:7199 127.0.0.1:49835 CLOSE_WAIT - tcp8 0 127.0.0.1:7199 127.0.0.1:55087 CLOSE_WAIT - tcp8 0 127.0.0.1:7199 127.0.0.1:49837 CLOSE_WAIT - tcp8 0 127.0.0.1:7199 127.0.0.1:55647 CLOSE_WAIT - tcp8 0 127.0.0.1:7199 127.0.0.1:49833 CLOSE_WAIT - tcp8 0 127.0.0.1:7199 127.0.0.1:52935 CLOSE_WAIT - tcp8 0
Re: Problem with PropertyFileSnitch in Amazon EC2
Could you verify any security settings that may come into play with Elastic IPs? You should make sure the appropriate ports are open. See: http://www.datastax.com/docs/0.8/brisk/install_brisk_ami for a list of ports in the first chart. Joaquin Casares DataStax Software Engineer/Support On Mon, Jun 20, 2011 at 7:41 PM, Sameer Farooqui cassandral...@gmail.comwrote: Quick update... I'm trying to get a 3-node cluster defined the following way in the topology.properties file to work first: 10.68.x.x=DC1:RAC1 10.198.x.x=DC1:RAC2 10.204.x.x=DC1:RAC3 I'll split up the 3rd node into a separate data center later. Also, ignore that comment I made about the $BRISK_HOME/lib/ folder not existing. When you run ANT, I believe it populates correctly, but I'll have to confirm/test later. Based on Joaquin @ DataStax's suggestion, I tried changing the Seed IP in all 3 nodes' YAML file to the Amazon Private IP, instead of the Elastic IP. After this change, all three nodes joined the ring correctly: ubuntu@ip-10-68-x-x:~/brisk-1.0~beta1.2/resources/cassandra/conf$ ../bin/nodetool -h localhost ring Address Status State LoadOwnsToken 113427455640312821154458202477256070485 10.68.x.x Up Normal 10.9 KB 33.33% 0 10.198.x.x Up Normal 15.21 KB33.33% 56713727820156410577229101238628035242 10.204.x.x Up Normal 6.55 KB 33.33% 113427455640312821154458202477256070485 PasteBin is down and is showing me a diligent cat typing on a keyboard, so I uploaded some relevant DEBUG level log files here: http://blueplastic.com/accenture/N1-system-seed_is_ElasticIP.log (problem exists) http://blueplastic.com/accenture/N2-system-seed_is_ElasticIP.log (problem exists) http://blueplastic.com/accenture/N1-system-seed_is_privateIP.log(everything works) http://blueplastic.com/accenture/N2-system-seed_is_privateIP.log(everything works) But if I want to set up the Brisk cluster across Amazon regions, I have to be able to use the Elastic IP for the seed. Also, using v 0.7.4 of Cassandra in Amazon, we successfully set up a 30+ node cluster using 3 seed nodes which were declared in the YAML file using Elastic IPs. All 30 nodes were in the same region and availability zone. So, in an older version of Cassandra, providing the Seeds as Elastic IP used to work. In my current setup, even though nodes 1 2 are in the same region availability zone, I can't seem to get them to join the same ring correctly. Here is what the system log file shows when I declare the Seed using Elastic IP: INFO [Thread-4] 2011-06-21 00:10:30,849 BriskDaemon.java (line 187) Listening for thrift clients... DEBUG [GossipTasks:1] 2011-06-21 00:10:31,608 Gossiper.java (line 201) Assuming current protocol version for /50.17.x.x DEBUG [WRITE-/50.17.212.84] 2011-06-21 00:10:31,610 OutboundTcpConnection.java (line 161) attempting to connect to /50.17.x.x DEBUG [GossipTasks:1] 2011-06-21 00:10:32,610 Gossiper.java (line 201) Assuming current protocol version for /50.17.x.x DEBUG [ScheduledTasks:1] 2011-06-21 00:10:32,613 StorageLoadBalancer.java (line 334) Disseminating load info ... DEBUG [GossipTasks:1] 2011-06-21 00:10:33,611 Gossiper.java (line 201) Assuming current protocol version for /50.17.x.x DEBUG [GossipTasks:1] 2011-06-21 00:10:34,612 Gossiper.java (line 201) Assuming current protocol version for /50.17.x.x But when I use private IP, the log shows: INFO [Thread-4] 2011-06-21 00:19:47,993 BriskDaemon.java (line 187) Listening for thrift clients... DEBUG [ScheduledTasks:1] 2011-06-21 00:19:49,769 StorageLoadBalancer.java (line 334) Disseminating load info ... DEBUG [WRITE-/10.198.126.193] 2011-06-21 00:20:09,658 OutboundTcpConnection.java (line 161) attempting to connect to /10.198.x.x INFO [GossipStage:1] 2011-06-21 00:20:09,690 Gossiper.java (line 637) Node /10.198.x.x is now part of the cluster DEBUG [GossipStage:1] 2011-06-21 00:20:09,691 MessagingService.java (line 158) Resetting pool for /10.198.x.x INFO [GossipStage:1] 2011-06-21 00:20:09,691 Gossiper.java (line 605) InetAddress /10.198.x.x is now UP DEBUG [HintedHandoff:1] 2011-06-21 00:20:09,692 HintedHandOffManager.java (line 282) Checking remote schema before delivering hints DEBUG [HintedHandoff:1] 2011-06-21 00:20:09,692 HintedHandOffManager.java (line 274) schema for /10.198.x.x matches local schema DEBUG [HintedHandoff:1] 2011-06-21 00:20:09,692 HintedHandOffManager.java (line 288) Sleeping 11662ms to stagger hint delivery - Sameer On Mon, Jun 20, 2011 at 2:28 PM, Sameer Farooqui cassandral...@gmail.comwrote: Hi, I'm setting up a 3 node test cluster in multiple Amazon Availability Zones to test cross-zone internode communication (and eventually cross-region communications). But I wanted to start with a cross-zone setup and am having trouble getting the nodes to connect to each other and join one 3-node ring. All nodes just seem
Re: Ec2 Stress Results
Did the images have EBS storage or Instance Store storage? Typically EBS volumes aren't the best to be benchmarking against: http://www.mail-archive.com/user@cassandra.apache.org/msg11022.html Joaquin Casares DataStax Software Engineer/Support On Wed, Apr 20, 2011 at 5:12 PM, Jonathan Ellis jbel...@gmail.com wrote: A few months ago I was seeing 12k writes/s on a single EC2 XL. So something is wrong. My first suspicion is that your client node may be the bottleneck. On Wed, Apr 20, 2011 at 2:56 PM, Alex Araujo cassandra-us...@alex.otherinbox.com wrote: Does anyone have any Ec2 benchmarks/experiences they can share? I am trying to get a sense for what to expect from a production cluster on Ec2 so that I can compare my application's performance against a sane baseline. What I have done so far is: 1. Lunched a 4 node cluster of m1.xlarge instances in the same availability zone using PyStratus (https://github.com/digitalreasoning/PyStratus). Each node has the following specs (according to Amazon): 15 GB memory 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each) 1,690 GB instance storage 64-bit platform 2. Changed the default PyStratus directories in order to have commit logs on the root partition and data files on ephemeral storage: commitlog_directory: /var/cassandra-logs data_file_directories: [/mnt/cassandra-data] 2. Gave each node 10GB of MAX_HEAP; 1GB HEAP_NEWSIZE in conf/cassandra-env.sh 3. Ran `contrib/stress/bin/stress -d node1,..,node4 -n 1000 -t 100` on a separate m1.large instance: total,interval_op_rate,interval_key_rate,avg_latency,elapsed_time ... 9832712,7120,7120,0.004948514851485148,842 9907616,7490,7490,0.0043189949802413755,852 9978357,7074,7074,0.004560353967289125,863 1000,2164,2164,0.004065933558194335,867 4. Truncated Keyspace1.Standard1: # /usr/local/apache-cassandra/bin/cassandra-cli -host localhost -port 9160 Connected to: Test Cluster on x.x.x.x/9160 Welcome to cassandra CLI. Type 'help;' or '?' for help. Type 'quit;' or 'exit;' to quit. [default@unknown] use Keyspace1; Authenticated to keyspace: Keyspace1 [default@Keyspace1] truncate Standard1; null 5. Expanded the cluster to 8 nodes using PyStratus and sanity checked using nodetool: # /usr/local/apache-cassandra/bin/nodetool -h localhost ring Address Status State LoadOwns Token x.x.x.x Up Normal 1.3 GB 12.50% 21267647932558653966460912964485513216 x.x.x.x Up Normal 3.06 GB 12.50% 42535295865117307932921825928971026432 x.x.x.x Up Normal 1.16 GB 12.50% 63802943797675961899382738893456539648 x.x.x.x Up Normal 2.43 GB 12.50% 85070591730234615865843651857942052864 x.x.x.x Up Normal 1.22 GB 12.50% 106338239662793269832304564822427566080 x.x.x.xUp Normal 2.74 GB 12.50% 127605887595351923798765477786913079296 x.x.x.xUp Normal 1.22 GB 12.50% 148873535527910577765226390751398592512 x.x.x.x Up Normal 2.57 GB 12.50% 170141183460469231731687303715884105728 6. Ran `contrib/stress/bin/stress -d node1,..,node8 -n 1000 -t 100` on a separate m1.large instance again: total,interval_op_rate,interval_key_rate,avg_latency,elapsed_time ... 9880360,9649,9649,0.003210443956226165,720 9942718,6235,6235,0.003206934154398794,731 9997035,5431,5431,0.0032615939761032457,741 1000,296,296,0.002660033726812816,742 In a nutshell, 4 nodes inserted at 11,534 writes/sec and 8 nodes inserted at 13,477 writes/sec. Those numbers seem a little low to me, but I don't have anything to compare to. I'd like to hear others' opinions before I spin my wheels with with number of nodes, threads, memtable, memory, and/or GC settings. Cheers, Alex. -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: help
http://wiki.apache.org/cassandra/FAQ#unsubscribe http://wiki.apache.org/cassandra/FAQ#unsubscribeIs this what you're looking for? Joaquin Casares DataStax Software Engineer/Support On Tue, Apr 12, 2011 at 2:03 PM, Denis Kirpichenkov den.doki.kirpichen...@gmail.com wrote:
