Documentation about TTL and tombstones
Hello community, by reading the documentation about TTL https://cassandra.apache.org/doc/4.1/cassandra/operating/compaction/index.html#ttl It mention that it creates a tombstone when data expired, how does it possible without writing to the tombstone on the table ? I thought TTL doesn't create tombstones since the ttl is present together with the write time timestmap at the row level Greetings Jean Carlo "The best way to predict the future is to invent it" Alan Kay
Re: Gossips pending task increasing, nodes are DOWN
Yes, I can see. Looks like cluster got overcharged and full repair was running, we got to restart all nodes and it seems to fix the problem. We will upgrade version soon Saludos Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Thu, Mar 17, 2022 at 5:12 PM Jeff Jirsa wrote: > This release is from Sep 2016 (5.5 years ago) and has no fixes applied to > it since. There are likely MANY issues with that version. > > On Thu, Mar 17, 2022 at 9:07 AM Jean Carlo > wrote: > >> Hello, >> >> After some restart, we go a list of nodes unreachable. These nodes are >> being seen as DOWN for the rest of the peers but they are running and keep >> accumulating gossip pending taks. Restarting the node does not solve the >> problem. >> >> The cassandra version is the 3.9 and the cluster has 80 nodes . >> >> Is there an issue with this version ? >> >> Jean Carlo >> >> "The best way to predict the future is to invent it" Alan Kay >> >
Gossips pending task increasing, nodes are DOWN
Hello, After some restart, we go a list of nodes unreachable. These nodes are being seen as DOWN for the rest of the peers but they are running and keep accumulating gossip pending taks. Restarting the node does not solve the problem. The cassandra version is the 3.9 and the cluster has 80 nodes . Is there an issue with this version ? Jean Carlo "The best way to predict the future is to invent it" Alan Kay
Error in bootstrapping node
) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) ERROR [main] 2021-12-15 19:39:40,756 StorageService.java:1889 - Error while waiting on bootstrap to complete. Bootstrap will have to be restarted. java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455) at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1882) at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1115) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:689) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:613) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:402) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:527) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:702) After using nodetool bootstrap resume, I got the same error with INFO [OptionalTasks:1] 2021-12-16 04:41:34,078 CassandraRoleManager.java:432 - Setup task failed with error, rescheduling org.apache.cassandra.exceptions.IsBootstrappingException: Cannot read from a bootstrapping node Thanks Jean Carlo "The best way to predict the future is to invent it" Alan Kay
Re: sstableloader & num_tokens change
Hello Concerning the original question, I agreed with @eric_ramirez, sstableloader is transparent for token allocation number. just for info @voytek, check this post out https://thelastpickle.com/blog/2019/02/21/set-up-a-cluster-with-even-token-distribution.html You lay be interested to now if you have your cluster well balanced with 32 tokens. 32 tokens seems to be the future default value, but changing the default vnodes token numbers seems not to be so straight forward cheers Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Sat, Jan 25, 2020 at 5:05 AM Erick Ramirez wrote: > On the subject of DSBulk, sstableloader is the tool of choice for this > scenario. > > +1 to Sergio and I'm confirming that DSBulk is designed as a bulk loader > for CSV/JSON formats. Cheers! >
Re: [EXTERNAL] Re: COPY command with where condition
Hello
Nobody has mentioned but you can use spark cassandra connector also.
Preferably if your data set is so big that a simple copy to csv cannot
handle it
Saludos
Jean Carlo
"The best way to predict the future is to invent it" Alan Kay
On Fri, Jan 17, 2020 at 8:11 PM Durity, Sean R
wrote:
> sstablekeys (in the tools directory?) can extract the actual keys from
> your sstables. You have to run it on each node and then combine and de-dupe
> the final results, but I have used this technique with a query generator to
> extract data more efficiently.
>
>
>
>
>
> Sean Durity
>
>
>
> *From:* Chris Splinter
> *Sent:* Friday, January 17, 2020 1:47 PM
> *To:* adrien ruffie
> *Cc:* user@cassandra.apache.org; Erick Ramirez
> *Subject:* [EXTERNAL] Re: COPY command with where condition
>
>
>
> Do you know your partition keys?
>
>
>
> One option could be to enumerate that list of partition keys in separate
> cmds to make the individual operations less expensive for the cluster.
>
>
>
> For example:
>
> Say your partition key column is called id and the ids in your database
> are [1,2,3]
>
>
>
> You could do
>
> ./dsbulk unload --dsbulk.schema.keyspace 'dev_keyspace' -query "SELECT *
> FROM probe_sensors WHERE id = 1 AND localisation_id = 208812" -url
> /home/dump
>
> ./dsbulk unload --dsbulk.schema.keyspace 'dev_keyspace' -query "SELECT *
> FROM probe_sensors WHERE id = 2 AND localisation_id = 208812" -url
> /home/dump
>
> ./dsbulk unload --dsbulk.schema.keyspace 'dev_keyspace' -query "SELECT *
> FROM probe_sensors WHERE id = 3 AND localisation_id = 208812" -url
> /home/dump
>
>
>
>
>
> Does that option work for you?
>
>
>
>
>
>
>
> On Fri, Jan 17, 2020 at 12:17 PM adrien ruffie
> wrote:
>
> I don't really know for the moment in production environment, but for
> developpment environment the table contains more than 10.000.000 rows.
>
> But we need just a sub dataset of this table not the entirety ...
> --
>
> *De :* Chris Splinter
> *Envoyé :* vendredi 17 janvier 2020 17:40
> *À :* adrien ruffie
> *Cc :* user@cassandra.apache.org ; Erick
> Ramirez
> *Objet :* Re: COPY command with where condition
>
>
>
> What you are seeing there is a standard read timeout, how many rows do you
> expect back from that query?
>
>
>
> On Fri, Jan 17, 2020 at 9:50 AM adrien ruffie
> wrote:
>
> Thank you very much,
>
>
>
> so I do this request with for example -->
>
>
>
> ./dsbulk unload --dsbulk.schema.keyspace 'dev_keyspace' -query "SELECT *
> FROM probe_sensors WHERE localisation_id = 208812 ALLOW FILTERING" -url
> /home/dump
>
>
>
>
>
> But I get the following error
>
> com.datastax.dsbulk.executor.api.exception.BulkExecutionException:
> Statement execution failed: SELECT * FROM crt_sensors WHERE site_id =
> 208812 ALLOW FILTERING (Cassandra timeout during read query at consistency
> LOCAL_ONE (1 responses were required but only 0 replica responded))
>
>
>
> but I configured my driver with following driver.conf, but nothing work
> correctly. Do you know what is the problem ?
>
>
>
> datastax-java-driver {
>
> basic {
>
>
>
>
>
> contact-points = ["data1com:9042","data2.com:9042 [data2.com]
> <https://urldefense.com/v3/__http:/data2.com:9042__;!!M-nmYVHPHQ!aPA4KExKulLx_PrHwhUQwPy881v1sjBkj35R1lAx2EUxSkRCLwmtNon0SMW0XbLKH7jCV5U$>
> "]
>
>
>
> request {
>
> timeout = "200"
>
> consistency = "LOCAL_ONE"
>
>
>
> }
>
> }
>
> advanced {
>
>
>
> auth-provider {
>
> class = PlainTextAuthProvider
>
> username = "superuser"
>
> password = "mypass"
>
>
>
> }
>
> }
>
> }
> --
>
> *De :* Chris Splinter
> *Envoyé :* vendredi 17 janvier 2020 16:17
> *À :* user@cassandra.apache.org
> *Cc :* Erick Ramirez
> *Objet :* Re: COPY command with where condition
>
>
>
> DSBulk has an option that lets you specify the query ( including a WHERE
> clause )
>
>
>
> See Example 19 in this blog post for details:
> https://www.datastax.com/blog/2019/06/datastax-bulk-loader-unloading
> [datastax.com]
> <https://urldefense.com/v3/__https:/www.datastax.com/blog/2019/06/datastax-bulk-loader-unloading__;!!M-nmYVHPHQ!aPA4KExKulLx_PrHwhUQwPy881v1sjBkj35R1lAx2EUxSkRCLwmtNon0SMW0XbLKBUuw2Cc$>
>
>
>
> On Fri, Jan
Re: How to assure data consistency in switch over to standby dc
Hello Laxmiant, your application has to deal with eventually consistency if you are using cassandra. Ensure to have R + W > RF And have the repairs runing periodically. This is the best way to be the most cosistent and coherent Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Thu, Jan 16, 2020 at 3:18 PM Laxmikant Upadhyay wrote: > Hi Alex, > > You are right, that will solve the problem. but unfortunately i won't be > able to meet my sla with write each quorum . I am using local quorum for > both read and write. > Any other way ? > > > On Thu, Jan 16, 2020, 5:45 PM Oleksandr Shulgin < > oleksandr.shul...@zalando.de> wrote: > >> On Thu, Jan 16, 2020 at 1:04 PM Laxmikant Upadhyay < >> laxmikant@gmail.com> wrote: >> >>> Hi, >>> What I meant fromActive/standby model is that even though data is being >>> replicated (asynchronously) to standby DC , client will only access the >>> data from active DC (let's say using local_quorum). >>> >>> you have "to switch" your clients without any issues since your writes >>> are replicated on all DC. >>> --> that is not true because there is a chance of mutation drop. (Hints, >>> read repair may help to some extent but data consistency is not guaranteed >>> unless you run anti- entropy repair ) >>> >> >> What are the consistency levels used by your application(s)? >> >> E.g. for strong consistency across multiple DCs you could use EACH_QUORUM >> for the write requests and LOCAL_QUORUM for reads, with a replication >> factor >= 3 per DC. >> >> -- >> Alex >> >>
Re: Log output when Cassandra is "up"?
Hello Using nodetool status can be slow meanwhile your cluster is bigger. Not sure about nodetoo info but I prefer to rely on the message "Starting listening for CQL clients". In my company we use a script that checks the cqlsh command. if it succeed then that node is ready and UP. Concerning to "Starting listening for CQL clients" message, once you get it, you will be able to do cqlsh, so both of them are the same Cheers Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Thu, Jan 9, 2020 at 9:14 AM wrote: > I've always used the wait_for task on port 9042 for Ansible > > > > https://docs.ansible.com/ansible/latest/modules/wait_for_module.html > > > > > > > > > > *From:* Voytek Jarnot > *Sent:* 08 January 2020 21:37 > *To:* user@cassandra.apache.org > *Subject:* Log output when Cassandra is "up"? > > > > Needing to know when Cassandra is finished initializing and is up & > running. > > > > Had some scripts which were looking through system.log for "No gossip > backlog; proceeding", but that turns out not to be 100% reliable. > > > > Is looking for "Starting listening for CQL clients" considered definitive? > I.E., always gets output on success, and not on failure? > > > > Thanks >
Re: Cassandra copy command
Hello Rahul, To ensure the consistency among the DCs, it is enough to run a repair command. You can do it using http://cassandra-reaper.io/ or runing the commande *nodetool repair* with the respectively options in every node. You do not need to count the rows in every DC to ensure cassandra is sync amongs DC after you have run the repair. But if you still want to do it, use Spark for it. Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Wed, Aug 21, 2019 at 1:51 PM Rahul Reddy wrote: > Yep I did run rebuild on each new node > > On Wed, Aug 21, 2019, 7:25 AM Stefan Miklosovic < > stefan.mikloso...@instaclustr.com> wrote: > >> Hi Rahul, >> >> how did you add that dc3 to cluster? The rule of thumb here is to do >> rebuild from each node, for example like here >> >> https://docs.datastax.com/en/archived/cassandra/3.0/cassandra/operations/opsAddDCToCluster.html >> >> On Wed, 21 Aug 2019 at 12:57, Rahul Reddy >> wrote: >> > >> > Hi sefan, >> > >> > I'm adding new DC3 to exiting cluster and see discripencies couple of >> millions in Nodetool cfstats in new DC. >> > >> > My table size is 50gb >> > I'm trying to run copy entire table. >> > >> > Copy table to 'full_tablr.csv' with delimiter ','; >> > >> > If I run above command from dc3. Does it get the data only from dc3? >> > >> > >> > >> > On Wed, Aug 21, 2019, 6:46 AM Stefan Miklosovic < >> stefan.mikloso...@instaclustr.com> wrote: >> >> >> >> Hi Rahul, >> >> >> >> what is your motivation behind this? Why do you want to make sure the >> >> count is same? What is the purpose of that? All you should care about >> >> is that Cassandra will return you right results. It was designed from >> >> the very bottom to do that for you, you should not be bothered too >> >> much about such discrepancies, they will be always there in general. >> >> But the important fact is that once queried, you can rest assured it >> >> is returned (and consequentially repaired if data not match) as they >> >> should. >> >> >> >> What copy command you are talking about precisely, why you cant use >> just repair? >> >> >> >> On Wed, 21 Aug 2019 at 12:14, Rahul Reddy >> wrote: >> >> > >> >> > Hello, >> >> > >> >> > I have 3 datacenters . Want to make sure record count is same in all >> dc's . If I run copy command node1 in dc1 does it get the data from only >> dc1? Nodetool cfstats I'm seeing discrepancies in partitions count is it >> because we didn't run cleanup after adding few nodes and remove them?. To >> rule out any discripencies I want to run copy command from 3 DC's and >> compare. Please let me know if copy command extracts data from the DC only >> I ran it from? >> >> >> >> - >> >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >> >> For additional commands, e-mail: user-h...@cassandra.apache.org >> >> >> >> - >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: user-h...@cassandra.apache.org >> >>
Re: Differing snitches in different datacenters
Hello Voytek, In my opinion, It would be better for you to continue using GossipingPropertyFileSnitch in AWS as well. I would do it to avoid surprises. I've set up datacenters in AWS using GossipingPropertyFileSnitch with zero problems. Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Wed, Jul 31, 2019 at 9:06 PM Voytek Jarnot wrote: > Thanks Paul. Yes - finding a definitive answer is where I'm failing as > well. I think we're probably going to try it and see what happens, but > that's a bit worrisome. > > On Mon, Jul 29, 2019 at 3:35 PM Paul Chandler wrote: > >> Hi Voytek, >> >> I looked into this a little while ago, and couldn’t really find a >> definitive answer. We ended up keeping the GossipingPropertyFileSnitch in >> our GCP Datacenter, the only downside that I could see is that you have to >> manually specify the rack and DC. But doing it that way does allow you to >> create a multi vendor cluster if you wished in the future. >> >> I would also be interested if anyone has the definitive answer one this. >> >> Thanks >> >> Paul >> www.redshots.com >> >> On 29 Jul 2019, at 17:06, Voytek Jarnot wrote: >> >> Just a quick bump - hoping someone can shed some light on whether running >> different snitches in different datacenters is a terrible idea or no. It'd >> be fairly temporary, once the new DC is stood up and nodes are rebuilt, the >> old DC will be decomissioned. >> >> On Thu, Jul 25, 2019 at 12:36 PM Voytek Jarnot >> wrote: >> >>> Quick and hopefully easy question for the list. Background is existing >>> cluster (1 DC) will be migrated to AWS-hosted cluster via standing up a >>> second datacenter, existing cluster will be subsequently decommissioned. >>> >>> We currently use GossipingPropertyFileSnitch and are thinking about >>> using Ec2MultiRegionSnitch in the new AWS DC - that'd position us nicely if >>> in the future we want to run a multi-DC cluster in AWS. My question is: are >>> there any issues with one DC using GossipingPropertyFileSnitch and the >>> other using Ec2MultiRegionSnitch? This setup would be temporary, existing >>> until the new DC nodes have rebuilt and the old DC is decommissioned. >>> >>> Thanks, >>> Voytek Jarnot >>> >> >>
Re: Re: Re: how to configure the Token Allocation Algorithm
Hello Anthony, Effectively I did not start the seed of every rack firsts. Thank you for the post. I believe this is something important to have as official documentation in cassandra.apache.org. This issues as many others are not documented properly. Of course I find the blog of last pickle very useful in this matters, but having a properly documentation of how to start a fresh new cluster cassandra is basic. I have one question about your post, when you mention "*However, therein lies the problem, for existing clusters updating this setting is easy, as a keyspace already exists*" What is the interest to use allocate_tokens_for_keyspace in a cluster with data if there tokens are already distributed? in the worst case scenario, the cluster is already unbalanced Cheers Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Mon, Apr 29, 2019 at 2:45 AM Anthony Grasso wrote: > Hi Jean, > > It sounds like there are no nodes in one of the racks for the eu-west-3 > datacenter. What does the output of nodetool status look like currently? > > Note, you will need to start a node in each rack before creating the > keyspace. I wrote a blog post with the procedure to set up a new cluster > using the predictive token allocation algorithm: > http://thelastpickle.com/blog/2019/02/21/set-up-a-cluster-with-even-token-distribution.html > > Regards, > Anthony > > On Fri, 26 Apr 2019 at 19:53, Jean Carlo > wrote: > >> Creating a fresh new cluster in aws using this procedure, I got this >> problem once I am bootstrapping the second rack of the cluster of 6 >> machines with 3 racks and a keyspace of rf 3 >> >> WARN [main] 2019-04-26 11:37:43,845 TokenAllocation.java:63 - Selected >> tokens [-5106267594614944625, 623001446449719390, 7048665031315327212, >> 3265006217757525070, 5054577454645148534, 314677103601736696, >> 7660890915606146375, -5329427405842523680] >> ERROR [main] 2019-04-26 11:37:43,860 CassandraDaemon.java:749 - Fatal >> configuration error >> org.apache.cassandra.exceptions.ConfigurationException: Token allocation >> failed: the number of racks 2 in datacenter eu-west-3 is lower than its >> replication factor 3. >> >> Someone got this problem ? >> >> I am not quite sure why I have this, since my cluster has 3 racks. >> >> Cluster Information: >> Name: test >> Snitch: org.apache.cassandra.locator.GossipingPropertyFileSnitch >> DynamicEndPointSnitch: enabled >> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner >> Schema versions: >> 3bf63440-fad7-3371-9c14-4855ad11ee83: [192.0.0.1, 192.0.0.2] >> >> >> >> Jean Carlo >> >> "The best way to predict the future is to invent it" Alan Kay >> >> >> On Thu, Jan 24, 2019 at 10:32 AM Ahmed Eljami >> wrote: >> >>> Hi folks, >>> >>> What about adding new keyspaces in the existing cluster, test_2 with >>> the same RF. >>> >>> It will use the same logic as the existing kesypace test ? Or I should >>> restart nodes and add the new keyspace to the cassandra.yaml ? >>> >>> Thanks. >>> >>> Le mar. 2 oct. 2018 à 10:28, Varun Barala a >>> écrit : >>> >>>> Hi, >>>> >>>> Managing `initial_token` by yourself will give you more control over >>>> scale-in and scale-out. >>>> Let's say you have three node cluster with `num_token: 1` >>>> >>>> And your initial range looks like:- >>>> >>>> Datacenter: datacenter1 >>>> == >>>> AddressRackStatus State LoadOwns >>>>Token >>>> >>>> 3074457345618258602 >>>> >>>> 127.0.0.1 rack1 Up Normal 98.96 KiB 66.67% >>>>-9223372036854775808 >>>> 127.0.0.2 rack1 Up Normal 98.96 KiB 66.67% >>>>-3074457345618258603 >>>> 127.0.0.3 rack1 Up Normal 98.96 KiB 66.67% >>>>3074457345618258602 >>>> >>>> Now let's say you want to scale out the cluster to twice the current >>>> throughput(means you are adding 3 more nodes) >>>> >>>> If you are using AWS EBS volumes then you can use the same volumes and >>>> spin three more nodes by selecting midpoints of existing ranges which means >>>> your new nodes are already having data. >>>> Once you have mounted volumes on your new nodes:- >>>> * You need to delete
Re: Re: Re: how to configure the Token Allocation Algorithm
Creating a fresh new cluster in aws using this procedure, I got this problem once I am bootstrapping the second rack of the cluster of 6 machines with 3 racks and a keyspace of rf 3 WARN [main] 2019-04-26 11:37:43,845 TokenAllocation.java:63 - Selected tokens [-5106267594614944625, 623001446449719390, 7048665031315327212, 3265006217757525070, 5054577454645148534, 314677103601736696, 7660890915606146375, -5329427405842523680] ERROR [main] 2019-04-26 11:37:43,860 CassandraDaemon.java:749 - Fatal configuration error org.apache.cassandra.exceptions.ConfigurationException: Token allocation failed: the number of racks 2 in datacenter eu-west-3 is lower than its replication factor 3. Someone got this problem ? I am not quite sure why I have this, since my cluster has 3 racks. Cluster Information: Name: test Snitch: org.apache.cassandra.locator.GossipingPropertyFileSnitch DynamicEndPointSnitch: enabled Partitioner: org.apache.cassandra.dht.Murmur3Partitioner Schema versions: 3bf63440-fad7-3371-9c14-4855ad11ee83: [192.0.0.1, 192.0.0.2] Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Thu, Jan 24, 2019 at 10:32 AM Ahmed Eljami wrote: > Hi folks, > > What about adding new keyspaces in the existing cluster, test_2 with the > same RF. > > It will use the same logic as the existing kesypace test ? Or I should > restart nodes and add the new keyspace to the cassandra.yaml ? > > Thanks. > > Le mar. 2 oct. 2018 à 10:28, Varun Barala a > écrit : > >> Hi, >> >> Managing `initial_token` by yourself will give you more control over >> scale-in and scale-out. >> Let's say you have three node cluster with `num_token: 1` >> >> And your initial range looks like:- >> >> Datacenter: datacenter1 >> == >> AddressRackStatus State LoadOwns >> Token >> >>3074457345618258602 >> 127.0.0.1 rack1 Up Normal 98.96 KiB 66.67% >> -9223372036854775808 >> 127.0.0.2 rack1 Up Normal 98.96 KiB 66.67% >> -3074457345618258603 >> 127.0.0.3 rack1 Up Normal 98.96 KiB 66.67% >> 3074457345618258602 >> >> Now let's say you want to scale out the cluster to twice the current >> throughput(means you are adding 3 more nodes) >> >> If you are using AWS EBS volumes then you can use the same volumes and >> spin three more nodes by selecting midpoints of existing ranges which means >> your new nodes are already having data. >> Once you have mounted volumes on your new nodes:- >> * You need to delete every system table except schema related tables. >> * You need to generate system/local table by yourself which has >> `Bootstrap state` as completed and schema-version same as other existing >> nodes. >> * You need to remove extra data on all the machines using cleanup commands >> >> This is how you can scale out Cassandra cluster in the minutes. In case >> you want to add nodes one by one then you need to write some small tool >> which will always figure out the bigger range in the existing cluster and >> will split it into the half. >> >> However, I never tested it thoroughly but this should work conceptually. >> So here we are taking advantage of the fact that we have volumes(data) for >> the new node beforehand so we no need to bootstrap them. >> >> Thanks & Regards, >> Varun Barala >> >> On Tue, Oct 2, 2018 at 2:31 PM onmstester onmstester >> wrote: >> >>> >>> >>> Sent using Zoho Mail <https://www.zoho.com/mail/> >>> >>> >>> On Mon, 01 Oct 2018 18:36:03 +0330 *Alain RODRIGUEZ >>> >* wrote >>> >>> Hello again :), >>> >>> I thought a little bit more about this question, and I was actually >>> wondering if something like this would work: >>> >>> Imagine 3 node cluster, and create them using: >>> For the 3 nodes: `num_token: 4` >>> Node 1: `intial_token: -9223372036854775808, -4611686018427387905, -2, >>> 4611686018427387901` >>> Node 2: `intial_token: -7686143364045646507, -3074457345618258604, >>> 1537228672809129299, 6148914691236517202` >>> Node 3: `intial_token: -6148914691236517206, -1537228672809129303, >>> 3074457345618258600, 7686143364045646503` >>> >>> If you know the initial size of your cluster, you can calculate the >>> total number of tokens: number of nodes * vnodes and use the >>> formula/python code above to get the tokens. Then use the first token for >>> the first nod
Re: [EXTERNAL] Re: Getting Consistency level TWO when it is requested LOCAL_ONE
I think this jira https://issues.apache.org/jira/browse/CASSANDRA-9895 Answer my question Saludos Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Fri, Apr 12, 2019 at 10:04 AM Jean Carlo wrote: > Hello Sean > > Well this is a little bit confusing. After digging into the doc, I found > this old documentation of Datastax that says > "First, we can dynamically adjust behavior depending on the cluster size > and arrangement. Cassandra prefers to perform batchlog writes to two > different replicas in the same datacenter as the coordinator." > > https://www.datastax.com/dev/blog/atomic-batches-in-cassandra-1-2 > > Which may explain the message in the timeout. However I do not know if > this information is still true. Someone know if this is still true? > > Reading the comments in > https://issues.apache.org/jira/browse/CASSANDRA-9620 that says ' *Writing > the batch log will always be done using CL ONE.*' Contradict what I > understood from datastax's doc > > Yes I understood batches are not for speed. Still we are using it for a > consistency need. > > @Mahesh Yes we do set the consistency like that > > Thank you > > Jean Carlo > > "The best way to predict the future is to invent it" Alan Kay > > > On Thu, Apr 11, 2019 at 3:39 PM Durity, Sean R < > sean_r_dur...@homedepot.com> wrote: > >> https://issues.apache.org/jira/browse/CASSANDRA-9620 has something >> similar that was determined to be a driver error. I would start with >> looking at the driver version and also the RetryPolicy that is in effect >> for the Cluster. Secondly, I would look at whether a batch is really needed >> for the statements. Cassandra batches are for atomicity – not speed. >> >> >> >> Sean Durity >> >> Staff Systems Engineer – Cassandra >> >> MTC 2250 >> >> #cassandra - for the latest news and updates >> >> >> >> >> >> *From:* Mahesh Daksha >> *Sent:* Thursday, April 11, 2019 5:21 AM >> *To:* user@cassandra.apache.org >> *Subject:* [EXTERNAL] Re: Getting Consistency level TWO when it is >> requested LOCAL_ONE >> >> >> >> Hi Jean, >> >> >> >> I want to understand how you are setting the write consistency level as >> LOCAL ONE. That is with every query you mentioning consistency level or you >> have set the spring cassandra config with provided consistency level. >> >> Like this: >> >> cluster.setQueryOptions(new >> QueryOptions().setConsistencyLevel(ConsistencyLevel.valueOf(cassandraConsistencyLevel))); >> >> >> >> The only possibility i see of such behavior is its getting overridden >> from some where. >> >> >> >> Thanks, >> >> Mahesh Daksha >> >> >> >> On Thu, Apr 11, 2019 at 1:43 PM Jean Carlo >> wrote: >> >> Hello everyone, >> >> >> >> I have a case where the developers are using spring data framework for >> Cassandra. We are writing batches setting consistency level at LOCAL_ONE >> but we got a timeout like this >> >> >> >> *Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: >> Cassandra timeout during BATCH_LOG write query at consistency TWO (2 >> replica were required but only 1 acknowledged the write)* >> >> >> >> Is it Cassandra that somehow writes to the *system.batchlog* using >> consistency TWO or is it spring data that makes some dirty things behind >> the scenes ? >> >> (I want to believe it is the second one) >> >> >> >> Cheers >> >> >> >> Jean Carlo >> >> >> "The best way to predict the future is to invent it" Alan Kay >> >> >> -- >> >> The information in this Internet Email is confidential and may be legally >> privileged. It is intended solely for the addressee. Access to this Email >> by anyone else is unauthorized. If you are not the intended recipient, any >> disclosure, copying, distribution or any action taken or omitted to be >> taken in reliance on it, is prohibited and may be unlawful. When addressed >> to our clients any opinions or advice contained in this Email are subject >> to the terms and conditions expressed in any applicable governing The Home >> Depot terms of business or client engagement letter. The Home Depot >> disclaims all responsibility and liability for the accuracy and content of >> this attachment and for any damages or losses arising from any >> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other >> items of a destructive nature, which may be contained in this attachment >> and shall not be liable for direct, indirect, consequential or special >> damages in connection with this e-mail message or its attachment. >> >
Re: [EXTERNAL] Re: Getting Consistency level TWO when it is requested LOCAL_ONE
Hello Sean Well this is a little bit confusing. After digging into the doc, I found this old documentation of Datastax that says "First, we can dynamically adjust behavior depending on the cluster size and arrangement. Cassandra prefers to perform batchlog writes to two different replicas in the same datacenter as the coordinator." https://www.datastax.com/dev/blog/atomic-batches-in-cassandra-1-2 Which may explain the message in the timeout. However I do not know if this information is still true. Someone know if this is still true? Reading the comments in https://issues.apache.org/jira/browse/CASSANDRA-9620 that says ' *Writing the batch log will always be done using CL ONE.*' Contradict what I understood from datastax's doc Yes I understood batches are not for speed. Still we are using it for a consistency need. @Mahesh Yes we do set the consistency like that Thank you Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Thu, Apr 11, 2019 at 3:39 PM Durity, Sean R wrote: > https://issues.apache.org/jira/browse/CASSANDRA-9620 has something > similar that was determined to be a driver error. I would start with > looking at the driver version and also the RetryPolicy that is in effect > for the Cluster. Secondly, I would look at whether a batch is really needed > for the statements. Cassandra batches are for atomicity – not speed. > > > > Sean Durity > > Staff Systems Engineer – Cassandra > > MTC 2250 > > #cassandra - for the latest news and updates > > > > > > *From:* Mahesh Daksha > *Sent:* Thursday, April 11, 2019 5:21 AM > *To:* user@cassandra.apache.org > *Subject:* [EXTERNAL] Re: Getting Consistency level TWO when it is > requested LOCAL_ONE > > > > Hi Jean, > > > > I want to understand how you are setting the write consistency level as > LOCAL ONE. That is with every query you mentioning consistency level or you > have set the spring cassandra config with provided consistency level. > > Like this: > > cluster.setQueryOptions(new > QueryOptions().setConsistencyLevel(ConsistencyLevel.valueOf(cassandraConsistencyLevel))); > > > > The only possibility i see of such behavior is its getting overridden from > some where. > > > > Thanks, > > Mahesh Daksha > > > > On Thu, Apr 11, 2019 at 1:43 PM Jean Carlo > wrote: > > Hello everyone, > > > > I have a case where the developers are using spring data framework for > Cassandra. We are writing batches setting consistency level at LOCAL_ONE > but we got a timeout like this > > > > *Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: > Cassandra timeout during BATCH_LOG write query at consistency TWO (2 > replica were required but only 1 acknowledged the write)* > > > > Is it Cassandra that somehow writes to the *system.batchlog* using > consistency TWO or is it spring data that makes some dirty things behind > the scenes ? > > (I want to believe it is the second one) > > > > Cheers > > > > Jean Carlo > > > "The best way to predict the future is to invent it" Alan Kay > > > -- > > The information in this Internet Email is confidential and may be legally > privileged. It is intended solely for the addressee. Access to this Email > by anyone else is unauthorized. If you are not the intended recipient, any > disclosure, copying, distribution or any action taken or omitted to be > taken in reliance on it, is prohibited and may be unlawful. When addressed > to our clients any opinions or advice contained in this Email are subject > to the terms and conditions expressed in any applicable governing The Home > Depot terms of business or client engagement letter. The Home Depot > disclaims all responsibility and liability for the accuracy and content of > this attachment and for any damages or losses arising from any > inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other > items of a destructive nature, which may be contained in this attachment > and shall not be liable for direct, indirect, consequential or special > damages in connection with this e-mail message or its attachment. >
Getting Consistency level TWO when it is requested LOCAL_ONE
Hello everyone, I have a case where the developers are using spring data framework for Cassandra. We are writing batches setting consistency level at LOCAL_ONE but we got a timeout like this *Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during BATCH_LOG write query at consistency TWO (2 replica were required but only 1 acknowledged the write)* Is it Cassandra that somehow writes to the *system.batchlog* using consistency TWO or is it spring data that makes some dirty things behind the scenes ? (I want to believe it is the second one) Cheers Jean Carlo "The best way to predict the future is to invent it" Alan Kay
Re: All time blocked in nodetool tpstats
In my cluster, I have it at 4096. I think you can start with 1024 and check if you have no native requested blocked. I believe this parameter depends on the cluster traffic Cheers Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Tue, Apr 9, 2019 at 7:59 PM Abdul Patel wrote: > Hi, > > My nodetool tpstats arw showing all time blocked high numbers a d also > read dropped messages as 400 . > Client is expeirince high timeouts. > Checked few online forums they recommend to increase, > native_transport_max_threads. > As of jow its commented with 128 .. > Is it adviabke to increase this and also can this fix timeout issue? > >
Re: About using Ec2MultiRegionSnitch
Awesome Thank you very much Cheers Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Tue, Mar 5, 2019 at 2:47 PM Jeff Jirsa wrote: > > > > > On Mar 5, 2019, at 5:32 AM, Jean Carlo > wrote: > > > > Hello Jeff, thank you for the answer. But what will be the advantage of > GossipingPropertyFileSnitch over Ec2MultiRegionSnitch exactly ? The > possibility to name the DCs ? > > > Yes > > And if you ever move out of aws you won’t have any problems > > > > > And another question, are you telling me that > GossipingPropertyFileSnitch works fine in AWS and there is no need of > Ec2Snitch? > > Yea, just set the dc name to the region and the rack to the AZ in the > property file > > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > >
Re: About using Ec2MultiRegionSnitch
Hello Jeff, thank you for the answer. But what will be the advantage of GossipingPropertyFileSnitch over Ec2MultiRegionSnitch exactly ? The possibility to name the DCs ? And another question, are you telling me that GossipingPropertyFileSnitch works fine in AWS and there is no need of Ec2Snitch? Thank you ? Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Tue, Mar 5, 2019 at 2:24 PM Jeff Jirsa wrote: > Ec2 multi should work fine in one region, but consider using > GossipingPropertyFileSnitch if there’s even a chance you’ll want something > other than AWS regions as dc names - multicloud, hybrid, analytics DCs, etc > > > > -- > Jeff Jirsa > > > On Mar 5, 2019, at 5:12 AM, Jean Carlo wrote: > > Hello everyone, I am preparing a cluster single dc in AWS. For the moment > there is not need to have a Multi DC cluster but I would like to avoid a > Snitch migration in the future. I would like to know if > Ec2MultiRegionSnitch works also for cluster single DC. Or The migration > Ec2Snitch to Ec2MultiRegionSnitch is inevitable > > Thank you > > Jean Carlo > > "The best way to predict the future is to invent it" Alan Kay > >
About using Ec2MultiRegionSnitch
Hello everyone, I am preparing a cluster single dc in AWS. For the moment there is not need to have a Multi DC cluster but I would like to avoid a Snitch migration in the future. I would like to know if Ec2MultiRegionSnitch works also for cluster single DC. Or The migration Ec2Snitch to Ec2MultiRegionSnitch is inevitable Thank you Jean Carlo "The best way to predict the future is to invent it" Alan Kay
Re: Usage of allocate_tokens_for_keyspace for a new cluster
Hello, I find this discussion very useful. I did not see information about this topic in cassandra apache web page. Shouldn't we have a instruction page to create a new fresh cluster following the right instructions to use correctly the option allocate_tokens_for_keyspace ? Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Thu, Feb 14, 2019 at 8:52 PM DuyHai Doan wrote: > Ok thanks John > > On Thu, Feb 14, 2019 at 8:51 PM Jonathan Haddad wrote: > >> Create the first node, setting the tokens manually. >> Create the keyspace. >> Add the rest of the nodes with the allocate tokens uncommented. >> >> On Thu, Feb 14, 2019 at 11:43 AM DuyHai Doan >> wrote: >> >>> Hello users >>> >>> By looking at the mailing list archive, there was already some questions >>> about the flag "allocate_tokens_for_keyspace" from cassandra.yaml >>> >>> I'm starting a fresh new cluster (with 0 data). >>> >>> The keyspace used by the project is raw_data so I >>> set allocate_tokens_for_keyspace = raw_data in the cassandra.yaml >>> >>> However the cluster fails to start, the keyspace does not exist yet (of >>> course, it is not yet created). >>> >>> So to me it is like chicken and egg issue: >>> >>> 1. You create a fresh new cluster with the option >>> "allocate_tokens_for_keyspace" commented out, in this case you cannot >>> optimize the token allocations >>> 2. You create a fresh new cluster with option >>> "allocate_tokens_for_keyspace" pointing to a not-yet created keyspace, it >>> fails (logically) >>> >>> The third option is: >>> >>> a. create a new cluster with "allocate_tokens_for_keyspace" commented >>> out >>> b. create the keyspace "raw_data" >>> c. set allocate_tokens_for_keyspace = raw_data >>> >>> My question is, since after step a. the token allocation is *already* >>> done, what's the point setting the flag in step c. >>> >>> Regards >>> >>> Duy Hai DOAN >>> >> -- >> Jon Haddad >> http://www.rustyrazorblade.com >> twitter: rustyrazorblade >> >
Re: question about the gain of increasing the number of vnode
Hi Victor, Take a look to this jira https://issues.apache.org/jira/browse/CASSANDRA-13701 I may answer your questions Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Mon, Jan 21, 2019 at 12:08 PM VICTOR IBARRA wrote: > > Good morning every one, > > I would like have a contact with the cassandra community for the questions > of cluster configuration > > Today i have many questions and differents projets about the configuration > of cluster cassandra and with the general problems of configuration > migration and for the use of vnodes. > > and the principal question is what about the gain to use 256 vnodes vs 16 > vnodes for example > > Best regards > -- > L'integrité de ce message n'étant pas assurée sur internet, VICTOR IBARRA > ne peut être tenue responsable de son contenu en ce compris les pièces > jointes. Toute utilisation ou diffusion non autorisée est interdite. Si > vous n'êtes pas destinataire de ce message, merci de le détruire et > d'avertir l'expéditeur. > > The integrity of this message cannot be guaranteed on the Internet. > VICTOR IBARRA can not therefore be considered liable for the contents > including its attachments. Any unauthorized use or dissemination is > prohibited. If you are not the intended recipient of this message, then > please delete it and notify the sender. >
Re: Repairs are slow after upgrade to 3.11.3
Hello, Can I ask you why did you upgrade from 3.11.2 ? did you experience some java heap problems ? Unfortunately I cannot answer your question :( I am in the 2.1 and about to upgrade to 3.11 Best greatings Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Wed, Aug 29, 2018 at 3:06 AM Maxim Parkachov wrote: > Hi everyone, > > couple of days ago I have upgraded Cassandra from 3.11.2 to 3.11.3 and I > see that repair time is practically doubled. Does someone else experience > the same regression ? > > Regards, > Maxim. >
Cassandra 3.11 and subrange repairs
Hello everyone, I am just wondering if someone is using this tool to make repairs in cassandra 3.11 https://github.com/BrianGallew/cassandra_range_repair Or everybody is using cassandra-reaper ? :) I am willing to use cassandra-reaper soon but meanwhile I will just need to cron the repairs in cluster. Actually, I want to know if cassandra_range_repair works properly in 3.11 because its repository is not active so far Best greetings Jean Carlo "The best way to predict the future is to invent it" Alan Kay
restoring with commitlogs in C* 3.11
Hello everyone, I am testing the procedure of restoring a table using the commitlogs without success. I am following this doc (even if this is for DSE ) https://support.datastax.com/hc/en-us/articles/115001593706-Manual-Backup-and-Restore-with-Point-in-time-and-table-level-restore- I am probably missing something. Is there anyone who is using the commitlog to make point in time restoring ? Is there a procedure to make it works ? This is my procedure. 1- Add this line to cassandra-env.sh JVM_OPTS="$JVM_OPTS -Dcassandra.replayList=pns_nonreg_bench.cf1" 2- I edit the file /etc/cassandra/commitlog_archiving.properties like this archive_command=/bin/ln %path /disk2/cassandra/commit_log_backup/%name restore_command=cp -f %from %to restore_directories=/disk1/cassandra/commitlog/ restore_point_in_time= precision=MICROSECONDS 3- I restart all the nodes 4- I made a delete on cassandra and the modify the restore_point_in_time to set it before the delete restore_point_in_time=2018:07:30 09:54:00 # before the delete 5- then I restart cassandra and I just see in the logs this INFO [main] 2018-07-30 10:40:43,652 CommitLog.java:157 - Replaying /var/opt/hosting/db/cassandra/commitlog/CommitLog-6-1532697162576.log INFO [main] 2018-07-30 10:40:43,705 CommitLog.java:159 - Log replay complete, 0 replayed mutations I did not get the data back, (re write it ). For infor I made the procedure using snapshots and the replaying the commitlogs without success Thank you very much Jean Carlo "The best way to predict the future is to invent it" Alan Kay
Re: Problem to activate mode DEBUG to see the slow queries
Thank you ahmed! Saludos Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Thu, Jun 28, 2018 at 11:21 AM, Ahmed Eljami wrote: > Hello Jean Carlo, > > To activate Debug mode, you should edit "logback.xml " not > "log4j-server.properties" > > > Ahmed. >
Problem to activate mode DEBUG to see the slow queries
hello, concerning to slow queries https://issues.apache.org/jira/browse/CASSANDRA-12403 how do I activate the logging debug for the slow queries ? I tried with nodetool setlogging org.apache.cassandra.db.monitoring DEBUG and also with log4j-server.properties and I did not get the expected result Thx Jean Carlo "The best way to predict the future is to invent it" Alan Kay
Re: repair in C* 3.11.2 and anticompactions
Thanks alain and Lerh, It is clear now. In order to avoid problems and charge in the cluster doing anticompactions, I am going to use repair by sub ranges. I studied this tool called cassandra-list-subranges <https://github.com/pauloricardomg/cassandra-list-subranges> it seems it still works for the versions 3.11.2. And I will take a look also to cassandra_range_repair <https://github.com/BrianGallew/cassandra_range_repair> which it is more recent Do you have any remarks for cassandra-list-subranges <https://github.com/pauloricardomg/cassandra-list-subranges> ? Saludos Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Thu, May 24, 2018 at 11:12 AM, Alain RODRIGUEZ <arodr...@gmail.com> wrote: > Hi Jean, > > Here is what Alexander wrote about it, a few months ago, in the comments > of the article mentioned above: > > "A full repair is an incremental one that doesn't skip repaired data. >> Performing anticompaction in that case too (regardless it is a valid >> approach or not) allows to mark as repaired SSTables that weren't before >> the full repair was started. >> >> It was clearly added with the intent of making full repair part of a >> routine where incremental repairs are also executed, leaving only subrange >> for people who do not want to use incremental. >> >> One major drawback is that by doing so, the project increased the >> operational complexity of running full repairs as it does not allow >> repairing the same keyspace from 2 nodes concurrently without risking some >> failures during validation compaction (when an SSTable is being >> anticompacted, it cannot go through validation compaction)." >> >> > I hope this helps, > > C*heers, > --- > Alain Rodriguez - @arodream - al...@thelastpickle.com > France / Spain > > The Last Pickle - Apache Cassandra Consulting > http://www.thelastpickle.com > > 2018-05-23 21:48 GMT+01:00 Lerh Chuan Low <l...@instaclustr.com>: > >> Hey Jean, >> >> I think it still does anticompaction by default regardless, it will not >> do so only if you do subrange repair. TLP wrote a pretty good article on >> that: http://thelastpickle.com/blog/2017/12/14/should-you- >> use-incremental-repair.html >> >> On 24 May 2018 at 00:42, Jean Carlo <jean.jeancar...@gmail.com> wrote: >> >>> Hello >>> >>> I just want to understand why, if I run a repair non incremental like >>> this >>> >>> nodetool -h 127.0.0.1 -p 7100 repair -full -pr keyspace1 standard1 >>> >>> Cassandra does anticompaction as the logs show >>> >>> INFO [CompactionExecutor:20] 2018-05-23 16:36:27,598 >>> CompactionManager.java:1545 - Anticompacting [BigTableReader(path='/home/jr >>> iveraura/.ccm/test/node1/data0/keyspace1/standard1-36a6ec405 >>> e9411e8b1d1b38a73559799/mc-2-big-Data.db')] >>> >>> As far as I understood the anticompactions are used to make the repair >>> incremantals possible, so I was expecting no having anticompactions making >>> repairs with the options -pr -full >>> >>> Anyone knows why does cassandra make those anticompactions ? >>> >>> Thanks >>> >>> Jean Carlo >>> >>> "The best way to predict the future is to invent it" Alan Kay >>> >> >> >
repair in C* 3.11.2 and anticompactions
Hello I just want to understand why, if I run a repair non incremental like this nodetool -h 127.0.0.1 -p 7100 repair -full -pr keyspace1 standard1 Cassandra does anticompaction as the logs show INFO [CompactionExecutor:20] 2018-05-23 16:36:27,598 CompactionManager.java:1545 - Anticompacting [BigTableReader(path='/home/jriveraura/.ccm/test/node1/data0/keyspace1/standard1-36a6ec405e9411e8b1d1b38a73559799/mc-2-big-Data.db')] As far as I understood the anticompactions are used to make the repair incremantals possible, so I was expecting no having anticompactions making repairs with the options -pr -full Anyone knows why does cassandra make those anticompactions ? Thanks Jean Carlo "The best way to predict the future is to invent it" Alan Kay
Re: copy from one table to another
You can use the same procedure to restore a table from snapshot from datastax webpage https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_backup_snapshot_restore_t.html Just two modifications. after step 5, modify the name of the sstables to add the name of the table you want to copy to. and in the step 6 copy the sstables to the right directory corresponding to the tale you want to copy to. Be sure you have an snapshot of the table source and ignore step 4 of course Saludos Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Sun, Apr 8, 2018 at 6:33 PM, Dmitry Saprykin <saprykin.dmi...@gmail.com> wrote: > You can copy hardlinks to ALL SSTables from old to new table and then > delete part of data you do not need in a new one. > > On Sun, Apr 8, 2018 at 10:20 AM, Nitan Kainth <nitankai...@gmail.com> > wrote: > >> If it for testing and you don’t need any specific data, just copy a set >> of sstables with all files of that sequence and move to target tables >> directory and rename it. >> >> Restart target node or run nodetool refresh >> >> Sent from my iPhone >> >> On Apr 8, 2018, at 4:15 AM, onmstester onmstester <onmstes...@zoho.com> >> wrote: >> >> Is there any way to copy some part of a table to another table in >> cassandra? A large amount of data should be copied so i don't want to fetch >> data to client and stream it back to cassandra using cql. >> >> Sent using Zoho Mail <https://www.zoho.com/mail/> >> >> >> >
select fields and heap usage
Hello, Talking about heap pressure, is there any difference between make a '*select **' and a '*select field *'? Is cassandra taking into the heap the whole partition to make the merge doesn't matters if we make a select of few rows ? Best greatings Jean Carlo "The best way to predict the future is to invent it" Alan Kay
Re: Initializing a multiple node cluster (multiple datacenters)
Hi Jonathan Yes I do think this is a good idea about the doc. About the clarification, this is still true for the 2.1 ? We are planing upgrading to the 3.1 but not in the next months. We will stick for few more months on the 2.1. I believe this is true also for the 2.1 but I would like to confirm I am missing something Saludos Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Thu, Feb 22, 2018 at 10:28 PM, Kenneth Brotman < kenbrot...@yahoo.com.invalid> wrote: > I will heavy lift the docs for a while, do my Slender Cassandra reference > project and then I’ll try to find one or two areas where I can contribute > code to get going on that. I have read the section on contributing before > I start. I’ll self-assign the JIRA right now. > > > > Kenneth Brotman > > > > *From:* Jonathan Haddad [mailto:j...@jonhaddad.com] > *Sent:* Thursday, February 22, 2018 1:21 PM > *To:* user@cassandra.apache.org > *Subject:* Re: Initializing a multiple node cluster (multiple datacenters) > > > > Kenneth, if you want to take the JIRA, feel free to self-assign it to > yourself and put up a pull request or patch, and I'll review. I'd be very > happy to get more people involved in the docs. > > > > On Thu, Feb 22, 2018 at 12:56 PM Kenneth Brotman < > kenbrot...@yahoo.com.invalid> wrote: > > That information would have saved me time too. Thanks for making a JIRA > for it Jon. Perhaps this is a good JIRA for me to begin with. > > > > Kenneth Brotman > > > > *From:* Jon Haddad [mailto:jonathan.had...@gmail.com] *On Behalf Of *Jon > Haddad > *Sent:* Thursday, February 22, 2018 11:11 AM > *To:* user > *Subject:* Re: Initializing a multiple node cluster (multiple datacenters) > > > > Great question. Unfortunately, our OSS docs lack a step by step process > on how to add a DC, I’ve created a JIRA to do that: https://issues.apache. > org/jira/browse/CASSANDRA-14254 > > > > The datastax docs are pretty good for this though: https://docs.datastax. > com/en/cassandra/latest/cassandra/operations/opsAddDCToCluster.html > > > > Regarding token allocation, it was random prior to 3.0. In 3.0 and up, it > is calculated a little more intelligently. in 3.11.2, which was just > released, CASSANDRA-13080 was backported which will help out when you add > your second DC. If you go this route, you can drop your token count down > to 16 and get all the benefits with no drawbacks. > > > > At this point I would go straight to 3.11.2 and skip 3.0 as there were > quite a few improvements that make it worthwhile along the way, in my > opinion. We work with several customers that are running 3.11 and are > pretty happy with it > > > > Yes, if there’s no data, you can initialize the cluster with > auto_boostrap: true. Be sure to change any key spaces using simple > strategy to NTS first, and replica them to the new DC as well. > > > > Jon > > > > > > On Feb 22, 2018, at 10:53 AM, Jean Carlo <jean.jeancar...@gmail.com> > wrote: > > > > Hi jonathan > > > > Thank you for the answer. Do you know where to look to understand why this > works. As i understood all the node then will chose ramdoms tokens. How can > i assure the correctness of the ring? > > > > So as you said. Under the condition that there.is no data in the cluster. > I can initialize a cluster multi dc without disable auto bootstrap.? > > > > On Feb 22, 2018 5:43 PM, "Jonathan Haddad" <j...@jonhaddad.com> wrote: > > If it's a new cluster, there's no need to disable auto_bootstrap. That > setting prevents the first node in the second DC from being a replica for > all the data in the first DC. If there's no data in the first DC, you can > skip a couple steps and just leave it on. > > > > Leave it on, and enjoy your afternoon. > > > > Seeds don't bootstrap by the way, changing the setting on those nodes > doesn't do anything. > > > > On Thu, Feb 22, 2018 at 8:36 AM Jean Carlo <jeanjeancar...@gmail.com > <jean.jeancar...@gmail.com>> wrote: > > Hello > > I would like to clarify this, > > > > In order to initialize a cassandra multi dc cluster, without data. If I > follow the documentation datastax > > > https://docs.datastax.com/en/cassandra/2.1/cassandra/initialize/ > initializeMultipleDS.html > > It says > >- auto_bootstrap: false (Add this setting *only* when initializing a >clean node with no data.) > > But I dont understand the way this works regarding to the auto_bootstraps. > > If all the machines make their own tokens in a ramdon way using > murmur3partitioner and vnodes , it isn't probable that two nodes will have > the tokens in common ? > > It is not better to bootstrap first the seeds with auto_bootstrap: false > and then the rest of the nodes with auto_bootstrap: true ? > > > > Thank you for the help > > > > Jean Carlo > > > "The best way to predict the future is to invent it" Alan Kay > > > > > >
Re: Initializing a multiple node cluster (multiple datacenters)
Hi jonathan Thank you for the answer. Do you know where to look to understand why this works. As i understood all the node then will chose ramdoms tokens. How can i assure the correctness of the ring? So as you said. Under the condition that there.is no data in the cluster. I can initialize a cluster multi dc without disable auto bootstrap.? On Feb 22, 2018 5:43 PM, "Jonathan Haddad" <j...@jonhaddad.com> wrote: If it's a new cluster, there's no need to disable auto_bootstrap. That setting prevents the first node in the second DC from being a replica for all the data in the first DC. If there's no data in the first DC, you can skip a couple steps and just leave it on. Leave it on, and enjoy your afternoon. Seeds don't bootstrap by the way, changing the setting on those nodes doesn't do anything. On Thu, Feb 22, 2018 at 8:36 AM Jean Carlo <jean.jeancar...@gmail.com> wrote: > Hello > > I would like to clarify this, > > In order to initialize a cassandra multi dc cluster, without data. If I > follow the documentation datastax > > https://docs.datastax.com/en/cassandra/2.1/cassandra/initialize/ > initializeMultipleDS.html > > > It says > >- auto_bootstrap: false (Add this setting *only* when initializing a >clean node with no data.) > > But I dont understand the way this works regarding to the auto_bootstraps. > > If all the machines make their own tokens in a ramdon way using > murmur3partitioner and vnodes , it isn't probable that two nodes will have > the tokens in common ? > > It is not better to bootstrap first the seeds with auto_bootstrap: false > and then the rest of the nodes with auto_bootstrap: true ? > > Thank you for the help > > Jean Carlo > > "The best way to predict the future is to invent it" Alan Kay >
Initializing a multiple node cluster (multiple datacenters)
Hello I would like to clarify this, In order to initialize a cassandra multi dc cluster, without data. If I follow the documentation datastax https://docs.datastax.com/en/cassandra/2.1/cassandra/initialize/initializeMultipleDS.html It says - auto_bootstrap: false (Add this setting *only* when initializing a clean node with no data.) But I dont understand the way this works regarding to the auto_bootstraps. If all the machines make their own tokens in a ramdon way using murmur3partitioner and vnodes , it isn't probable that two nodes will have the tokens in common ? It is not better to bootstrap first the seeds with auto_bootstrap: false and then the rest of the nodes with auto_bootstrap: true ? Thank you for the help Jean Carlo "The best way to predict the future is to invent it" Alan Kay
Re: Remote host closed connection during handshake SSL Cassandra 3.0.9
Hi Marcus >java version ? We use oracle-java8-jre the version 8u92. >TLSv1 disabled ? No it is not disable. We tried also with the version 1.2 Saludos Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Wed, Jan 31, 2018 at 12:11 PM, Marcus Haarmann <marcus.haarm...@midoco.de > wrote: > java version ? TLSv1 disabled ? (TLSv1 should not be used any more, since > it is outdated, but should work internally) > > Marcus Haarmann > > ------ > *Von: *"Jean Carlo" <jean.jeancar...@gmail.com> > *An: *user@cassandra.apache.org > *Gesendet: *Mittwoch, 31. Januar 2018 11:53:37 > *Betreff: *Remote host closed connection during handshake SSL Cassandra > 3.0.9 > > Hello! > > I have a problem enabling inter-node encryption in cassandra 3.0.9 > > After I set my conf like that: > > server_encryption_options: > internode_encryption: all > keystore: /etc/certs/node1.keystore > keystore_password: cassandra > truststore: /etc/certs/node1.truststore > truststore_password: cassandra > # More advanced defaults below: > protocol: *TLSv1* > # algorithm: SunX509 > # store_type: JKS > cipher_suites: [*TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA*] > # require_client_auth: false > > I got this error all the time > > ERROR [ACCEPT-/node1] 2018-01-31 11:29:20,358 MessagingService.java:1081 - > SSL handshake error for inbound connection from > a8265dd[SSL_NULL_WITH_NULL_NULL: > Socket[addr=/node2,port=40352,localport=7001]] > javax.net.ssl.SSLHandshakeException: Remote host closed connection during > handshake > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:992) > ~[na:1.8.0_92] > at > sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) > ~[na:1.8.0_92] > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:928) > ~[na:1.8.0_92] > at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) > ~[na:1.8.0_92] > at sun.security.ssl.AppInputStream.read(AppInputStream.java:71) > ~[na:1.8.0_92] > at java.io.DataInputStream.readInt(DataInputStream.java:387) > ~[na:1.8.0_92] > at org.apache.cassandra.net.MessagingService$SocketThread. > run(MessagingService.java:1055) ~[apache-cassandra-3.0.9.jar:3.0.9] > Caused by: java.io.EOFException: SSL peer shut down incorrectly > at sun.security.ssl.InputRecord.read(InputRecord.java:505) > ~[na:1.8.0_92] > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) > ~[na:1.8.0_92] > ... 6 common frames omitted > > > I think I tested the correctness of my certs using the command openssl > s_client ( or at least I think I did) > > user@node1 /home/user $ openssl s_client -connect node2:7001 -tls1 > CONNECTED(0003) > ... > SSL-Session: > Protocol : *TLSv1* > Cipher: *ECDHE-RSA-AES256-SHA* > ... > > So it seems I am using the right configuration but still having the 'SSL > peer shut down incorrectly' error. Anyone have had this error before? > > best greetings > > Jean Carlo > > "The best way to predict the future is to invent it" Alan Kay > >
Remote host closed connection during handshake SSL Cassandra 3.0.9
Hello! I have a problem enabling inter-node encryption in cassandra 3.0.9 After I set my conf like that: server_encryption_options: internode_encryption: all keystore: /etc/certs/node1.keystore keystore_password: cassandra truststore: /etc/certs/node1.truststore truststore_password: cassandra # More advanced defaults below: protocol: *TLSv1* # algorithm: SunX509 # store_type: JKS cipher_suites: [*TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA*] # require_client_auth: false I got this error all the time ERROR [ACCEPT-/node1] 2018-01-31 11:29:20,358 MessagingService.java:1081 - SSL handshake error for inbound connection from a8265dd[SSL_NULL_WITH_NULL_NULL: Socket[addr=/node2,port=40352,localport=7001]] javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:992) ~[na:1.8.0_92] at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) ~[na:1.8.0_92] at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:928) ~[na:1.8.0_92] at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) ~[na:1.8.0_92] at sun.security.ssl.AppInputStream.read(AppInputStream.java:71) ~[na:1.8.0_92] at java.io.DataInputStream.readInt(DataInputStream.java:387) ~[na:1.8.0_92] at org.apache.cassandra.net.MessagingService$SocketThread.run(MessagingService.java:1055) ~[apache-cassandra-3.0.9.jar:3.0.9] Caused by: java.io.EOFException: SSL peer shut down incorrectly at sun.security.ssl.InputRecord.read(InputRecord.java:505) ~[na:1.8.0_92] at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) ~[na:1.8.0_92] ... 6 common frames omitted I think I tested the correctness of my certs using the command openssl s_client ( or at least I think I did) user@node1 /home/user $ openssl s_client -connect node2:7001 -tls1 CONNECTED(0003) ... SSL-Session: Protocol : *TLSv1* Cipher: *ECDHE-RSA-AES256-SHA* ... So it seems I am using the right configuration but still having the 'SSL peer shut down incorrectly' error. Anyone have had this error before? best greetings Jean Carlo "The best way to predict the future is to invent it" Alan Kay
Re: Restoring fresh new cluster from existing snapshot
Hello, Basically, every node has to have the same token range. So yes you have to play with initial_token having the same numbers of tokens per node like the cluster source. To save time and if you dont have any constraints about the name of the cluster etc. you can just copy and paste the complete keyspace system node by node. So you will have the same cluster( cluster name, confs, etc) Saludos Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Thu, Jan 11, 2018 at 10:28 AM, Pradeep Chhetri <prad...@stashaway.com> wrote: > Hello Jean, > > I am running cassandra 3.11.1. > > Since i dont have much cassandra operations experience yet, I have a > follow-up question - how can i ensure the same token ranges distribution ? > Do i need to set initial_token configuration for each cassandra node ? > > Thank you for the quick response. > > > > > > On Thu, Jan 11, 2018 at 3:04 PM, Jean Carlo <jean.jeancar...@gmail.com> > wrote: > >> Hello Pradeep, >> >> Actually the key here is to know if your cluster has the same token >> ranges distribution. So it is not only the same size but also the same >> tokens match node by node, from cluster source to cluster destination. In >> that case, you can use nodetool refresh.So after copy all your sstable node >> by node, it would be enough to make nodetool refresh in every node to >> restore your data. You can also restart casandra instead of doing nodetool >> refresh. It will help you to avoid the compactions after refreshing. >> >> >> Saludos >> >> Jean Carlo >> >> "The best way to predict the future is to invent it" Alan Kay >> >> On Thu, Jan 11, 2018 at 9:58 AM, Pradeep Chhetri <prad...@stashaway.com> >> wrote: >> >>> Hello everyone, >>> >>> We are running cassandra cluster inside containers over Kubernetes. We >>> have a requirement where we need to restore a fresh new cluster with >>> existing snapshot on weekly basis. >>> >>> Currently, while doing it manually. i need to copy the snapshot folder >>> inside container and then run sstableloader utility to load those tables. >>> >>> Since the source and destination cluster size is equal, I was thinking >>> if there are some easy way to just copy and paste the complete data >>> directory by mapping the nodes one to one. >>> >>> Since i wasn't able to find documentation around other backup >>> restoration methods apart from nodetool snapshot and sstableloader, I >>> haven't explored much. I recently came across this project - >>> https://github.com/Netflix/Priam but tried it yet. >>> >>> Would be very happy if i can get some ideas around various ways of >>> backup/restoration while running inside containers. >>> >>> Thank you >>> >> >> >
Re: Restoring fresh new cluster from existing snapshot
Hello Pradeep, Actually the key here is to know if your cluster has the same token ranges distribution. So it is not only the same size but also the same tokens match node by node, from cluster source to cluster destination. In that case, you can use nodetool refresh.So after copy all your sstable node by node, it would be enough to make nodetool refresh in every node to restore your data. You can also restart casandra instead of doing nodetool refresh. It will help you to avoid the compactions after refreshing. Saludos Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Thu, Jan 11, 2018 at 9:58 AM, Pradeep Chhetri <prad...@stashaway.com> wrote: > Hello everyone, > > We are running cassandra cluster inside containers over Kubernetes. We > have a requirement where we need to restore a fresh new cluster with > existing snapshot on weekly basis. > > Currently, while doing it manually. i need to copy the snapshot folder > inside container and then run sstableloader utility to load those tables. > > Since the source and destination cluster size is equal, I was thinking if > there are some easy way to just copy and paste the complete data directory > by mapping the nodes one to one. > > Since i wasn't able to find documentation around other backup restoration > methods apart from nodetool snapshot and sstableloader, I haven't explored > much. I recently came across this project - https://github.com/Netflix/ > Priam but tried it yet. > > Would be very happy if i can get some ideas around various ways of > backup/restoration while running inside containers. > > Thank you >
Re: Restore cassandra snapshots
Hi Pradeep, Because you use sstableloader, you don't need to restore de system keyspace. Your procedure is correct to me. Best regards On Oct 18, 2017 4:22 AM, "Pradeep Chhetri" <prad...@stashaway.com> wrote: Hi Anthony I did the following steps to restore. Please let me know if I missed something. - Took snapshots on the 3 nodes of the existing cluster simultaneously - copied that snapshots respectively on the 3 nodes of the freshly created cluster - ran sstableloader on each of the application table. ( I didn't restore the system related tables ) on all of three node. I was assuming that since I ran from all the three snapshots, all the tokens should be there so thought that this will not cause any data loss. Do you see that I might have data loss. I am not sure how to verify the data loss although I did ran count of few table to verify the row count. Thank you the help. On Wed, 18 Oct 2017 at 5:39 AM, Anthony Grasso <anthony.gra...@gmail.com> wrote: > Hi Pradeep, > > If you are going to copy N snapshots to N nodes you will need to make sure > you have the System keyspace as part of that snapshot. The System keyspace > that is local to each node, contains the token allocations for that > particular node. This allows the node to work out what data it is > responsible for. Further to that, if you are restoring the System keyspace > from snapshots, make sure that the cluster name of the new cluster is > exactly the same as the cluster which generated the System keyspace > snapshots. > > Regards, > Anthony > > On 16 October 2017 at 23:28, Jean Carlo <jean.jeancar...@gmail.com> wrote: > >> HI, >> >> Yes of course, you can use sstableloader from every sstable to your new >> cluster. Actually this is the common procedure. Just check the log of >> cassandra, you shouldn't see any errors of streaming. >> >> >> However, because the fact you are migrating from on cluster of N nodes to >> another of N nodes, I believe you can just copy and paste your data node >> per node and make a nodetool refresh. Checking obviously the correct names >> of your sstables. >> You can check the tokens of your node using nodetool info -T >> >> But I think sstableloader is the easy way :) >> >> >> >> >> Saludos >> >> Jean Carlo >> >> "The best way to predict the future is to invent it" Alan Kay >> >> On Mon, Oct 16, 2017 at 1:55 PM, Pradeep Chhetri <prad...@stashaway.com> >> wrote: >> >>> Hi Jean, >>> >>> Thank you for the quick response. I am not sure how to achieve that. Can >>> i set the tokens for a node via cqlsh ? >>> >>> I know that i can check the nodetool rings to get the tokens allocated >>> to a node. >>> >>> I was thinking to basically run sstableloader for each of the snapshots >>> and was assuming it will load the complete data properly. Isn't that the >>> case. >>> >>> Thank you. >>> >>> On Mon, Oct 16, 2017 at 5:21 PM, Jean Carlo <jean.jeancar...@gmail.com> >>> wrote: >>> >>>> Hi, >>>> >>>> Be sure that you have the same tokens distribution than your original >>>> cluster. So if you are going to restore from old node 1 to new node 1, make >>>> sure that the new node and the old node have the same tokens. >>>> >>>> >>>> Saludos >>>> >>>> Jean Carlo >>>> >>>> "The best way to predict the future is to invent it" Alan Kay >>>> >>>> On Mon, Oct 16, 2017 at 1:40 PM, Pradeep Chhetri <prad...@stashaway.com >>>> > wrote: >>>> >>>>> Hi, >>>>> >>>>> I am trying to restore an empty 3-node cluster with the three >>>>> snapshots taken on another 3-node cluster. >>>>> >>>>> What is the best approach to achieve it without loosing any data >>>>> present in the snapshot. >>>>> >>>>> Thank you. >>>>> Pradeep >>>>> >>>> >>>> >>> >> >
Re: Restore cassandra snapshots
HI, Yes of course, you can use sstableloader from every sstable to your new cluster. Actually this is the common procedure. Just check the log of cassandra, you shouldn't see any errors of streaming. However, because the fact you are migrating from on cluster of N nodes to another of N nodes, I believe you can just copy and paste your data node per node and make a nodetool refresh. Checking obviously the correct names of your sstables. You can check the tokens of your node using nodetool info -T But I think sstableloader is the easy way :) Saludos Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Mon, Oct 16, 2017 at 1:55 PM, Pradeep Chhetri <prad...@stashaway.com> wrote: > Hi Jean, > > Thank you for the quick response. I am not sure how to achieve that. Can i > set the tokens for a node via cqlsh ? > > I know that i can check the nodetool rings to get the tokens allocated to > a node. > > I was thinking to basically run sstableloader for each of the snapshots > and was assuming it will load the complete data properly. Isn't that the > case. > > Thank you. > > On Mon, Oct 16, 2017 at 5:21 PM, Jean Carlo <jean.jeancar...@gmail.com> > wrote: > >> Hi, >> >> Be sure that you have the same tokens distribution than your original >> cluster. So if you are going to restore from old node 1 to new node 1, make >> sure that the new node and the old node have the same tokens. >> >> >> Saludos >> >> Jean Carlo >> >> "The best way to predict the future is to invent it" Alan Kay >> >> On Mon, Oct 16, 2017 at 1:40 PM, Pradeep Chhetri <prad...@stashaway.com> >> wrote: >> >>> Hi, >>> >>> I am trying to restore an empty 3-node cluster with the three snapshots >>> taken on another 3-node cluster. >>> >>> What is the best approach to achieve it without loosing any data present >>> in the snapshot. >>> >>> Thank you. >>> Pradeep >>> >> >> >
Re: Migrate from one cluster of N nodes to another cluster of M nodes where N>M
Hi Kurt, Thanks for your answer. I was analysing this migration you said, and if I am not wrong, I wont be able to have two cluster independents. I wil try to explain this, as I can see, after the splits of DC's, I wont be able to change the name of the cluster, Am I right ? Even if I change the topology file (cassandra-topology.properties). Or it is possible ? That's why you mentioned (firewall rules are helpful here) isn't it? Saludos Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Mon, Oct 16, 2017 at 5:29 AM, kurt greaves <k...@instaclustr.com> wrote: > If you create a new cluster and mimic the tokens across less nodes you > will still have downtime/missing data between the point when you copy all > the SSTables across and any new writes to the old cluster after you take > the copy. > Only way to really do this effectively is to do a DC migration. Brief run > through would be to: > >1. Bring up a new DC (ensure you are using NetworkTopologyStrategy >beforehand) >2. Change your clients to only query the old DC (load balancing policy >+ local consistency levels) >3. Replicate the keyspace across to the new DC >4. Rebuild the nodes in the new DC from the old DC >5. Run a repair >6. Point clients at new DC >7. Remove replication from old DC >8. Split the DC's into two clusters (firewall rules are helpful here). > >
Re: Restore cassandra snapshots
Hi, Be sure that you have the same tokens distribution than your original cluster. So if you are going to restore from old node 1 to new node 1, make sure that the new node and the old node have the same tokens. Saludos Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Mon, Oct 16, 2017 at 1:40 PM, Pradeep Chhetri <prad...@stashaway.com> wrote: > Hi, > > I am trying to restore an empty 3-node cluster with the three snapshots > taken on another 3-node cluster. > > What is the best approach to achieve it without loosing any data present > in the snapshot. > > Thank you. > Pradeep >
Migrate from one cluster of N nodes to another cluster of M nodes where N>M
Hello cassandra folks. So I want to ask how to migrate a keyspace in production to another smaller cluster without have a downtime service. So I was thinking that I dont want to use sstableloader and paid for the compactions and streaming. I want actually to use the procedure when both clusters have the same numbers of nodes so I just need to make sure to have the same token range repartition. However my case is not the same. I will migrate for a cluster smaller than in production and I was thinking that I could have the same token repartition if I combine the tokens of diferents nodes. If I can combine the tokens of differents machines, Do you see any problems if I combine also the sstables of the same machines from the old cluster to the new one ? I think to combine the tokens I can use CCM to recreate my cluster in production and then play with removenode. For info, my keyspace is rf 3 and I have 3 racks Best regards Jean Carlo "The best way to predict the future is to invent it" Alan Kay
Re: Bootstraping a new node
Thx @Jacob, I was reading that article, I dont understand how we can add a node in the cluster using auto_bootstrap equal to false and not having a maldistribution of tokens. I know that cassandra use random token ranges if it doesn't have tokens associated. So it doesn't compute the new tokens ranges from the existing ones. it could lead to some tokens being owned by two nodes, am I right ? Saludos Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Fri, Sep 29, 2017 at 1:21 PM, Jacob Shadix <jacobsha...@gmail.com> wrote: > Are you building out a new DC? > > This is a good article about all things re. bootstrap - > http://thelastpickle.com/blog/2017/05/23/auto-bootstrapping-part1.html > > Refer to the auto_bootstrap: false and see if that meets your use case. > Pay CLOSE attention to the caveats. Bootstrapping w/o streaming risks data > inconsistencies for the cluster as you are allowing reads to be performed > on a node with token ranges assigned, but no data. > > -- Jacob Shadix > > On Fri, Sep 29, 2017 at 4:30 AM, Jean Carlo <jean.jeancar...@gmail.com> > wrote: > >> >> Hello cassandra community >> >> When bootstraping a new node, there is a way to say to cassandra from >> which DC make the streaming ? >> >> Or there is a way to just join the machine to the cluster just >> calculating the new tokens ( I am using vnodes) and not making any >> streaming to later make a rebuild from keyspace ? >> >> Best greetings >> >> Jean Carlo >> >> "The best way to predict the future is to invent it" Alan Kay >> > >
Bootstraping a new node
Hello cassandra community When bootstraping a new node, there is a way to say to cassandra from which DC make the streaming ? Or there is a way to just join the machine to the cluster just calculating the new tokens ( I am using vnodes) and not making any streaming to later make a rebuild from keyspace ? Best greetings Jean Carlo "The best way to predict the future is to invent it" Alan Kay
Analysis of the jvm heap cassandra
Hello community, I am analysing a dump of 11GB and I am using eclipse memory analyser. I would like to know if you guys really need a machine with 32GB to analyse a dump of G1GC of 30GB or there is another smarter way to do that. Greetings Jean Carlo "The best way to predict the future is to invent it" Alan Kay
Understanding gossip and seeds
I have a question. When I change the list of seeds of my cluster, and I activate the log cassandra in mode TRACE nodetool setlogginglevel org.apache.cassandra.gms.Gossiper TRACE I can see that the Gossip Digest does not change at all, and it conserve the previous seed list. I was surprise to see that and I would like to understand. Cassandra only take this change in account when we use the option JVM_OPTS="$JVM_OPTS -Dcassandra.load_ring_state=false" Why doesn't cassandra take in account the new list of seeds in the ossipDigest? Saludos Jean Carlo "The best way to predict the future is to invent it" Alan Kay
Seed gossip version will not connect with that version
Hello I have repairs that hangs because this problem WARN [MessagingService-Outgoing-/10.0.0.143] 2017-07-04 10:29:50,076 OutboundTcpConnection.java:416 - Seed gossip version is -2147483648; will not connect with that version INFO [HANDSHAKE-/10.0.0.143] 2017-07-04 10:29:50,076 OutboundTcpConnection.java:496 - Cannot handshake version with /10.0.0.143 INFO [HANDSHAKE-/10.0.0.143] 2017-07-04 10:29:50,090 OutboundTcpConnection.java:487 - Handshaking version with /10.0.0.143 Is it enough to solve by restarting cassandra both nodes ? Best regards Jean Carlo "The best way to predict the future is to invent it" Alan Kay
Merkle trees requests hanging
Hello. What if a node send a merkle tree to its replica but this one would never received by any network issues. The repair will be hanging eternally ? or Should I modify the parameter # streaming_socket_timeout_in_ms: 0 to avoid this ? Saludos Jean Carlo "The best way to predict the future is to invent it" Alan Kay
Sending merkle trees
Hello In cassandra 2.1. When a replica has sent its merkle tree, Does it via streaming ? which protocol does it use ? Best regards Jean Carlo "The best way to predict the future is to invent it" Alan Kay
False positive increasing
Hello Lately I am observing that the false positives of one of my nodes are increasing in a continous way (1 per 5min) Bloom filter false positives: 532 Bloom filter false ratio: 0.01449 Bloom filter space used: 1.34 MB Bloom filter off heap memory used: 1.33 MB At the same time I can see that the duration of GC has increased also There is a link between the increasment of the GC and the bloom filter ? Jean Carlo
CASSANDRA-12849
Hello the jira 12849 has already a patch dispo. Might someone take a look of this jira ? https://issues.apache.org/jira/browse/CASSANDRA-12849 Saludos Jean Carlo "The best way to predict the future is to invent it" Alan Kay
Restoring a table cassandra - compactions
Hello.
During the restore of a table using its snapshot and nodetool refresh, I
could see that cassandra starts to make a lot of compactions (depending on
the size of the data).
I wanted to know why and I found this in the code of cassandra 2.1.14.
for CASSANDRA-4872
+// force foreign sstables to level 0
+try
+{
+if (new
File(descriptor.filenameFor(Component.STATS)).exists())
+{
+SSTableMetadata oldMetadata =
SSTableMetadata.serializer.deserialize(descriptor);
+LeveledManifest.mutateLevel(oldMetadata, descriptor,
descriptor.filenameFor(Component.STATS), 0);
+}
+}
+catch (IOException e)
This is very interesting and I wanted to know if this was coded taking into
account only the case of a migration from STCS to LCS or if for the case
LCS to LCS this is not pertinent
In my case, I use nodetool refresh not only to restore a table but also to
make an exact copy of any table LCS. So I think the levels do not need to
change.
@Marcus Can you be so kind to clarify this for me please ?
Thenk you very much in advance
Best regards
Jean Carlo
"The best way to predict the future is to invent it" Alan Kay
Re: cassandra OOM
Hello @Durity Would you mind to share information about your cluster? Actually I am interested to know which version of cassandra you use. And how much time do the gc pauses spend. Thank you very much Saludos Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Tue, Apr 25, 2017 at 7:47 PM, Durity, Sean R <sean_r_dur...@homedepot.com > wrote: > We have seen much better stability (and MUCH less GC pauses) from G1 with > a variety of heap sizes. I don’t even consider CMS any more. > > > > > > Sean Durity > > > > *From:* Gopal, Dhruva [mailto:dhruva.go...@aspect.com] > *Sent:* Tuesday, April 04, 2017 5:34 PM > *To:* user@cassandra.apache.org > *Subject:* Re: cassandra OOM > > > > Thanks, that’s interesting – so CMS is a better option for > stability/performance? We’ll try this out in our cluster. > > > > *From: *Alexander Dejanovski <a...@thelastpickle.com> > *Reply-To: *"user@cassandra.apache.org" <user@cassandra.apache.org> > *Date: *Monday, April 3, 2017 at 10:31 PM > *To: *"user@cassandra.apache.org" <user@cassandra.apache.org> > *Subject: *Re: cassandra OOM > > > > Hi, > > > > we've seen G1GC going OOM on production clusters (repeatedly) with a 16GB > heap when the workload is intense, and given you're running on m4.2xl I > wouldn't go over 16GB for the heap. > > > > I'd suggest to revert back to CMS, using a 16GB heap and up to 6GB of new > gen. You can use 5 as MaxTenuringThreshold as an initial value and activate > GC logging to fine tune the settings afterwards. > > > > FYI CMS tends to perform better than G1 even though it's a little bit > harder to tune. > > > > Cheers, > > > > On Mon, Apr 3, 2017 at 10:54 PM Gopal, Dhruva <dhruva.go...@aspect.com> > wrote: > > 16 Gig heap, with G1. Pertinent info from jvm.options below (we’re using > m2.2xlarge instances in AWS): > > > > > > # > > # HEAP SETTINGS # > > # > > > > # Heap size is automatically calculated by cassandra-env based on this > > # formula: max(min(1/2 ram, 1024MB), min(1/4 ram, 8GB)) > > # That is: > > # - calculate 1/2 ram and cap to 1024MB > > # - calculate 1/4 ram and cap to 8192MB > > # - pick the max > > # > > # For production use you may wish to adjust this for your environment. > > # If that's the case, uncomment the -Xmx and Xms options below to override > the > > # automatic calculation of JVM heap memory. > > # > > # It is recommended to set min (-Xms) and max (-Xmx) heap sizes to > > # the same value to avoid stop-the-world GC pauses during resize, and > > # so that we can lock the heap in memory on startup to prevent any > > # of it from being swapped out. > > -Xms16G > > -Xmx16G > > > > # Young generation size is automatically calculated by cassandra-env > > # based on this formula: min(100 * num_cores, 1/4 * heap size) > > # > > # The main trade-off for the young generation is that the larger it > > # is, the longer GC pause times will be. The shorter it is, the more > > # expensive GC will be (usually). > > # > > # It is not recommended to set the young generation size if using the > > # G1 GC, since that will override the target pause-time goal. > > # More info: http://www.oracle.com/technetwork/articles/java/ > g1gc-1984535.html > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.oracle.com_technetwork_articles_java_g1gc-2D1984535.html=DwMGaQ=MtgQEAMQGqekjTjiAhkudQ=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ=sW03C2XjzKcalSLXhtI4w0y-hPFk4-Nmh4BIt46jHxk=xuMqARzoTSasEmAPkP7fVOcPZS050fy1N2_Ac5poOtA=> > > # > > # The example below assumes a modern 8-core+ machine for decent > > # times. If in doubt, and if you do not particularly want to tweak, go > > # 100 MB per physical CPU core. > > #-Xmn800M > > > > # > > # GC SETTINGS # > > # > > > > ### CMS Settings > > > > #-XX:+UseParNewGC > > #-XX:+UseConcMarkSweepGC > > #-XX:+CMSParallelRemarkEnabled > > #-XX:SurvivorRatio=8 > > #-XX:MaxTenuringThreshold=1 > > #-XX:CMSInitiatingOccupancyFraction=75 > > #-XX:+UseCMSInitiatingOccupancyOnly > > #-XX:CMSWaitDuration=1 > > #-XX:+CMSParallelInitialMarkEnabled > > #-XX:+CMSEdenChunksRecordAlways > > # some JVMs will fill up their heap when accessed via JMX, see > CASSANDRA-6541 > > #-XX:+CMSClassUnloadingEnabled > > > > ### G1 Settings (experimental, comment previous section and uncomment > section below to enable)
Re: Adding multiple nodes to a Cassandra cluster
I think it is wise to add them one by one. taking in account that the cluster must re arrange the tokens among the replicas. Saludos Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Tue, Jan 24, 2017 at 12:00 PM, Andreas Fritzler < andreas.fritz...@gmail.com> wrote: > Hi, > > what is the best way to add multiple nodes to an existing Cassandra > cluster. The cluster runs in 1 DC. > > I found this guide on the datastax website [1]. Are there any other best > practices I should consider? > > Regards, > Andreas > > [1] https://docs.datastax.com/en/cassandra/2.1/cassandra/ > operations/ops_add_node_to_cluster_t.html > >
Re: Why compacting process uses more data that is expected
Hi Alexander, It would actually be a great improvement for ops if we could add a switch to compactionstats in order to have the compression ratio applied automatically. Yes actually that can be nice. Thank you guys for you reply. I have 34GB in total size of our sstable @Jonathan Haddad. And the ratio is 0.17053810600081237 SSTable Compression Ratio: 0.17053810600081237 Then my actual size of data uncompressed is 0.17053810600081237 x 34GB ~~ 200GB which makes sense :) Nice to know the subtlenes of cassandra :) Saludos Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Thu, Jan 5, 2017 at 8:53 AM, Alexander Dejanovski <a...@thelastpickle.com > wrote: > Indeed, nodetool compactionstats shows uncompressed sizes. > As Oleksandr suggests, use the table compression ratio to compute the > actual size on disk. > > It would actually be a great improvement for ops if we could add a switch > to compactionstats in order to have the compression ratio applied > automatically. > > On Thu, Jan 5, 2017 at 7:22 AM Oleksandr Shulgin < > oleksandr.shul...@zalando.de> wrote: > >> On Jan 4, 2017 17:58, "Jean Carlo" <jean.jeancar...@gmail.com> wrote: >> >> Hello guys >> >> I have a table with 34Gb of data in sstables (including tmp). And I can >> see cassandra is doing some compactions on it. What surprissed me is that >> nodetool compactionstats says he is compacting 138.66GB >> >> >> root@node001 /root # nodetool compactionstats -H >> pending tasks: 103 >> * compaction typekeyspace table >> completed total unit progress* >> Compaction keyspace1 table_02 112.74 GB >> 138.66 GB bytes 81.31% >> Active compaction remaining time : 0h03m27s >> >> So My question is, from where those 138.66GB come if my table has only >> 34GB of data. >> >> >> Hello, >> >> I believe that output of compactionstats shows you the size of >> *uncompressed* data. Can you check (with nodetool tablestats) your >> compression ratio? >> >> -- >> Alex >> >> -- > - > Alexander Dejanovski > France > @alexanderdeja > > Consultant > Apache Cassandra Consulting > http://www.thelastpickle.com >
Why compacting process uses more data that is expected
Hello guys I have a table with 34Gb of data in sstables (including tmp). And I can see cassandra is doing some compactions on it. What surprissed me is that nodetool compactionstats says he is compacting 138.66GB root@node001 /root # nodetool compactionstats -H pending tasks: 103 * compaction typekeyspace table completed total unit progress* Compaction keyspace1 table_02 112.74 GB 138.66 GB bytes 81.31% Active compaction remaining time : 0h03m27s So My question is, from where those 138.66GB come if my table has only 34GB of data. Best greetings Jean Carlo "The best way to predict the future is to invent it" Alan Kay
Cassandra Ignores path to HeadDumpFile defined by cassandra-env.sh
Hi all, We got recently a OOM error in cassandra, and it happened that cassandra made the dump in the path defined by debian/init. <https://github.com/apache/cassandra/blob/cassandra-2.1/debian/init> However we defined the CASSANDRA_HEAPDUMP_DIR in the file /etc/default/cassandra so cassandra must do the dump in it. Checking the cassandra jvm arguments, I can see that the value of -XX:HeapDumpPath is charge two times. And the last corresponds to the value set by default. I suppose cassandra takes in account the last -XX:HeapDumpPath and not that one set in cassandra-env.sh which isn't the expected behavior. Did anyone get the same behavior ? <https://github.com/apache/cassandra/blob/cassandra-2.1/debian/init> Best regards Jean Carlo "The best way to predict the future is to inven <https://github.com/apache/cassandra/blob/cassandra-2.1/debian/init>t it" Alan Kay
Re: cassandra dump file path
Yes, we did it. So if the parameter in cassandra-env.sh is used only if we have a OOM, what is for the definition of *-XX:HeapDumpPath=/var/lib/cassandra/java_1475461286.hprof *in /etc/init.d/cassandra for? Saludos Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Tue, Oct 4, 2016 at 2:58 AM, Yabin Meng <yabinm...@gmail.com> wrote: > Have you restarted Cassandra after making changes in cassandra-env.sh? > > Yabin > > On Mon, Oct 3, 2016 at 7:44 AM, Jean Carlo <jean.jeancar...@gmail.com> > wrote: > >> OK I got the response to one of my questions. In the script >> /etc/init.d/cassandra we set the path for the heap dump by default in the >> cassandra_home. >> >> Now the thing I don't understand is, why do the dumps are located by the >> file set by /etc/init.d/cassandra and not by the conf file >> cassandra-env.sh? >> >> Anyone any idea? >> >> >> Saludos >> >> Jean Carlo >> >> "The best way to predict the future is to invent it" Alan Kay >> >> On Mon, Oct 3, 2016 at 12:00 PM, Jean Carlo <jean.jeancar...@gmail.com> >> wrote: >> >>> >>> Hi >>> >>> I see in the log of my node cassandra that the parameter >>> -XX:HeapDumpPath is charged two times. >>> >>> INFO [main] 2016-10-03 04:21:29,941 CassandraDaemon.java:205 - JVM >>> Arguments: [-ea, -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar, >>> -XX:+CMSClassUnloadingEnabled, -XX:+UseThreadPriorities, >>> -XX:ThreadPriorityPolicy=42, -Xms6G, -Xmx6G, -Xmn600M, >>> *-XX:+HeapDumpOnOutOfMemoryError, >>> -XX:HeapDumpPath=/cassandra/dumps/cassandra-1475461287-pid34435.hprof*, >>> -Xss256k, -XX:StringTableSize=103, -XX:+UseParNewGC, >>> -XX:+UseConcMarkSweepGC, -XX:+CMSParallelRemarkEnabled, >>> -XX:SurvivorRatio=8, -XX:MaxTenuringThreshold=1, >>> -XX:CMSInitiatingOccupancyFraction=30, -XX:+UseCMSInitiatingOccupancyOnly, >>> -XX:+UseTLAB, -XX:CompileCommandFile=/etc/cassandra/hotspot_compiler, >>> -XX:CMSWaitDuration=1, -XX:+CMSParallelInitialMarkEnabled, >>> -XX:+CMSEdenChunksRecordAlways, -XX:CMSWaitDuration=1, >>> -XX:+UseCondCardMark, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, >>> -XX:+PrintGCApplicationStoppedTime, >>> -Xloggc:/var/opt/hosting/log/cassandra/gc.log, >>> -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=20, >>> -XX:GCLogFileSize=20M, -Djava.net.preferIPv4Stack=true, >>> -Dcom.sun.management.jmxremote.port=7199, >>> -Dcom.sun.management.jmxremote.rmi.port=7199, >>> -Dcom.sun.management.jmxremote.ssl=false, >>> -Dcom.sun.management.jmxremote.authenticate=false, >>> -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password, >>> -Djava.io.tmpdir=/var/opt/hosting/db/cassandra/tmp, >>> -javaagent:/usr/share/cassandra/lib/jolokia-jvm-1.0.6-agent.jar=port=8778,host=0.0.0.0, >>> -Dcassandra.auth_bcrypt_gensalt_log2_rounds=4, >>> -Dlogback.configurationFile=logback.xml, >>> -Dcassandra.logdir=/var/log/cassandra, >>> -Dcassandra.storagedir=, >>> -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid, >>> *-XX:HeapDumpPath=/var/lib/cassandra/java_1475461286.hprof*, >>> -XX:ErrorFile=/var/lib/cassandra/hs_err_1475461286.log] >>> >>> This option is defined in cassandra-env.sh >>> >>> if [ "x$CASSANDRA_HEAPDUMP_DIR" != "x" ]; then >>> JVM_OPTS="$JVM_OPTS >>> -XX:HeapDumpPath=$CASSANDRA_HEAPDUMP_DIR/cassandra-`date >>> +%s`-pid$$.hprof" >>> fi >>> and we defined before the value of CASSANDRA_HEAPDUMP_DIR before to >>> >>> >>> */cassandra/dumps/* >>> It is seems that cassandra does not care about the conf in >>> cassandra-env.sh and he only takes in account the last set for HeapDumpPath >>> >>> */var/lib/cassandra/java_1475461286.hprof* >>> This causes problems when we have to dump the heap because cassandra >>> uses the disk not suitable to do it. >>> >>> Is *XX:HeapDumpPath *set in another place/file that I dont know? >>> >>> Thxs >>> >>> Jean Carlo >>> >>> "The best way to predict the future is to invent it" Alan Kay >>> >> >> >
Re: Incremental repairs in 3.0
Hi Paulo!
Sorry there was something I was doing wrong.
Now I can see that the value of Repaired At changes even if there is no
streaming. I am using cassandra 2.1.14 and the comand was nodetool repair
-inc -par.
Anyway good to know this:
> If you're using subrange repair, please note that this has only partial
support for incremental repair due to CASSANDRA-10422 and should not mark
sstables as repaired, so you could be hitting CASSANDRA-12489.
Thx Paulo :)
Saludos
Jean Carlo
"The best way to predict the future is to invent it" Alan Kay
On Mon, Sep 12, 2016 at 2:06 PM, Paulo Motta <pauloricard...@gmail.com>
wrote:
> > I truncate a table lcs, Then I inserted one line and I used nodetool
> flush to have all the sstables. Using a RF 3 I ran a repair -inc directly
> and I observed that the value of Reaired At was equal 0.
>
> Were you able to troubleshoot this? The value of repairedAt should be
> mutated even when there is not streaming, otherwise there might be
> something going on. What version are you using and what command did you use
> to trigger incremental repair?
>
> If you're using subrange repair, please note that this has only partial
> support for incremental repair due to CASSANDRA-10422 and should not mark
> sstables as repaired, so you could be hitting CASSANDRA-12489.
>
> 2016-09-07 8:15 GMT-03:00 Jean Carlo <jean.jeancar...@gmail.com>:
>
>> Well I did an small test on my cluster and I didn't get the results I was
>> expecting.
>>
>> I truncate a table lcs, Then I inserted one line and I used nodetool
>> flush to have all the sstables. Using a RF 3 I ran a repair -inc directly
>> and I observed that the value of Reaired At was equal 0.
>>
>> So I start to think that if there is no changes ( diff on the merkles
>> trees) the repair will not pass to the streaming phase, and it is there
>> where the sstables are marked as repaired.
>>
>> I did another test to confirm my assomptions and I saw the sstables
>> marked as repaired. ("repaired at" value isn't 0). Well just those sstables
>> not sync.
>>
>> So my quesion is, if we migrate to repair inc in prod and we dont use the
>> migration procedure, for tables that some sstables are never mutated, they
>> will keep in a not repaired state ?
>>
>> Probably there is something I am not able to see
>>
>>
>>
>>
>>
>> Saludos
>>
>> Jean Carlo
>>
>> "The best way to predict the future is to invent it" Alan Kay
>>
>> On Tue, Sep 6, 2016 at 8:19 PM, Bryan Cheng <br...@blockcypher.com>
>> wrote:
>>
>>> HI Jean,
>>>
>>> This blog post is a pretty good resource: http://www.datastax.
>>> com/dev/blog/anticompaction-in-cassandra-2-1
>>>
>>> I believe in 2.1.x you don't need to do the manual migration procedure,
>>> but if you run regular repairs and the data set under LCS is fairly large
>>> (what this means will probably depend on your data model and
>>> hardware/cluster makeup) you can take advantage of a full repair to make
>>> anticompaction a bit easier. What we observed was the anticompaction
>>> procedure taking longer than a standard full repair and with a higher load
>>> on the cluster while running.
>>>
>>> On Tue, Sep 6, 2016 at 2:00 AM, Jean Carlo <jean.jeancar...@gmail.com>
>>> wrote:
>>>
>>>> Hi @Bryan
>>>>
>>>> When you said "sizable amount of data" you meant a huge amount of data
>>>> right? Our big table is in LCS and if we use the migration process we will
>>>> need to run a repair seq over this table for a long time.
>>>>
>>>> We are planning to go to repairs inc using the version 2.1.14
>>>>
>>>>
>>>> Saludos
>>>>
>>>> Jean Carlo
>>>>
>>>> "The best way to predict the future is to invent it" Alan Kay
>>>>
>>>> On Tue, Jun 21, 2016 at 4:34 PM, Vlad <qa23d-...@yahoo.com> wrote:
>>>>
>>>>> Thanks for answer!
>>>>>
>>>>> >It may still be a good idea to manually migrate if you have a
>>>>> sizable amount of data
>>>>> No, it would be brand new ;-) 3.0 cluster
>>>>>
>>>>>
>>>>>
>>>>> On Tuesday, June 21, 2016 1:21 AM, Bryan Cheng <br...@blockcypher.com>
>>>>> wrote:
>>>>>
>>>>>
>>>>> Sorry, meant to say "therefore manual migration procedure should be
>>>
Re: Incremental repairs in 3.0
Well I did an small test on my cluster and I didn't get the results I was
expecting.
I truncate a table lcs, Then I inserted one line and I used nodetool flush
to have all the sstables. Using a RF 3 I ran a repair -inc directly and I
observed that the value of Reaired At was equal 0.
So I start to think that if there is no changes ( diff on the merkles
trees) the repair will not pass to the streaming phase, and it is there
where the sstables are marked as repaired.
I did another test to confirm my assomptions and I saw the sstables marked
as repaired. ("repaired at" value isn't 0). Well just those sstables not
sync.
So my quesion is, if we migrate to repair inc in prod and we dont use the
migration procedure, for tables that some sstables are never mutated, they
will keep in a not repaired state ?
Probably there is something I am not able to see
Saludos
Jean Carlo
"The best way to predict the future is to invent it" Alan Kay
On Tue, Sep 6, 2016 at 8:19 PM, Bryan Cheng <br...@blockcypher.com> wrote:
> HI Jean,
>
> This blog post is a pretty good resource: http://www.datastax.
> com/dev/blog/anticompaction-in-cassandra-2-1
>
> I believe in 2.1.x you don't need to do the manual migration procedure,
> but if you run regular repairs and the data set under LCS is fairly large
> (what this means will probably depend on your data model and
> hardware/cluster makeup) you can take advantage of a full repair to make
> anticompaction a bit easier. What we observed was the anticompaction
> procedure taking longer than a standard full repair and with a higher load
> on the cluster while running.
>
> On Tue, Sep 6, 2016 at 2:00 AM, Jean Carlo <jean.jeancar...@gmail.com>
> wrote:
>
>> Hi @Bryan
>>
>> When you said "sizable amount of data" you meant a huge amount of data
>> right? Our big table is in LCS and if we use the migration process we will
>> need to run a repair seq over this table for a long time.
>>
>> We are planning to go to repairs inc using the version 2.1.14
>>
>>
>> Saludos
>>
>> Jean Carlo
>>
>> "The best way to predict the future is to invent it" Alan Kay
>>
>> On Tue, Jun 21, 2016 at 4:34 PM, Vlad <qa23d-...@yahoo.com> wrote:
>>
>>> Thanks for answer!
>>>
>>> >It may still be a good idea to manually migrate if you have a sizable
>>> amount of data
>>> No, it would be brand new ;-) 3.0 cluster
>>>
>>>
>>>
>>> On Tuesday, June 21, 2016 1:21 AM, Bryan Cheng <br...@blockcypher.com>
>>> wrote:
>>>
>>>
>>> Sorry, meant to say "therefore manual migration procedure should be
>>> UNnecessary"
>>>
>>> On Mon, Jun 20, 2016 at 3:21 PM, Bryan Cheng <br...@blockcypher.com>
>>> wrote:
>>>
>>> I don't use 3.x so hopefully someone with operational experience can
>>> chime in, however my understanding is: 1) Incremental repairs should be the
>>> default in the 3.x release branch and 2) sstable repairedAt is now properly
>>> set in all sstables as of 2.2.x for standard repairs and therefore manual
>>> migration procedure should be necessary. It may still be a good idea to
>>> manually migrate if you have a sizable amount of data and are using LCS as
>>> anticompaction is rather painful.
>>>
>>> On Sun, Jun 19, 2016 at 6:37 AM, Vlad <qa23d-...@yahoo.com> wrote:
>>>
>>> Hi,
>>>
>>> assuming I have new, empty Cassandra cluster, how should I start using
>>> incremental repairs? Is incremental repair is default now (as I don't see
>>> *-inc* option in nodetool) and nothing is needed to use it, or should
>>> we perform migration procedure
>>> <http://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsRepairNodesMigration.html>
>>> anyway? And what happens to new column families?
>>>
>>> Regards.
>>>
>>>
>>>
>>>
>>>
>>>
>>
>
Re: How to start using incremental repairs?
Hi Paulo Can you clarify me please if what you said here 1. Migration procedure is no longer necessary after CASSANDRA-8004, and since you never ran repair before this would not make any difference anyway, so just run repair and by default (CASSANDRA-7250) this will already be incremental. applies for the version 2.1.14. I ask because I see that the jira CASSANDRA-8004 is resolved for the version 2.1.2 and we are considering to migrate to repairs inc before go to the version 3.0.x Thhx :) Saludos Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Fri, Aug 26, 2016 at 9:04 PM, Stefano Ortolani <ostef...@gmail.com> wrote: > An extract of this conversation should definitely be posted somewhere. > Read a lot but never learnt all these bits... > > On Fri, Aug 26, 2016 at 2:53 PM, Paulo Motta <pauloricard...@gmail.com> > wrote: > >> > I must admit that I fail to understand currently how running repair >> with -pr could leave unrepaired data though, even when ran on all nodes in >> all DCs, and how that could be specific to incremental repair (and would >> appreciate if someone shared the explanation). >> >> Anti-compaction, which marks tables as repaired, is disabled for partial >> range repairs (which includes partitioner-range repair) to avoid the extra >> I/O cost of needing to run anti-compaction multiple times in a node to >> repair it completely. For example, there is an optimization which skips >> anti-compaction for sstables fully contained in the repaired range (only >> the repairedAt field is mutated), which is leveraged by full range repair, >> which would not work in many cases for partial range repairs, yielding >> higher I/O. >> >> 2016-08-26 10:17 GMT-03:00 Stefano Ortolani <ostef...@gmail.com>: >> >>> I see. Didn't think about it that way. Thanks for clarifying! >>> >>> >>> On Fri, Aug 26, 2016 at 2:14 PM, Paulo Motta <pauloricard...@gmail.com> >>> wrote: >>> >>>> > What is the underlying reason? >>>> >>>> Basically to minimize the amount of anti-compaction needed, since with >>>> RF=3 you'd need to perform anti-compaction 3 times in a particular node to >>>> get it fully repaired, while without it you can just repair the full node's >>>> range in one run. Assuming you run repair frequent enough this will not be >>>> a big deal, since you will skip already repaired data in the next round so >>>> you will not have the problem of re-doing work as in non-inc non-pr repair. >>>> >>>> 2016-08-26 7:57 GMT-03:00 Stefano Ortolani <ostef...@gmail.com>: >>>> >>>>> Hi Paulo, could you elaborate on 2? >>>>> I didn't know incremental repairs were not compatible with -pr >>>>> What is the underlying reason? >>>>> >>>>> Regards, >>>>> Stefano >>>>> >>>>> >>>>> On Fri, Aug 26, 2016 at 1:25 AM, Paulo Motta <pauloricard...@gmail.com >>>>> > wrote: >>>>> >>>>>> 1. Migration procedure is no longer necessary after CASSANDRA-8004, >>>>>> and since you never ran repair before this would not make any difference >>>>>> anyway, so just run repair and by default (CASSANDRA-7250) this will >>>>>> already be incremental. >>>>>> 2. Incremental repair is not supported with -pr, -local or -st/-et >>>>>> options, so you should run incremental repair in all nodes in all DCs >>>>>> sequentially (you should be aware that this will probably generate >>>>>> inter-DC >>>>>> traffic), no need to disable autocompaction or stopping nodes. >>>>>> >>>>>> 2016-08-25 18:27 GMT-03:00 Aleksandr Ivanov <ale...@gmail.com>: >>>>>> >>>>>>> I’m new in Cassandra and trying to figure out how to _start_ using >>>>>>> incremental repairs. I have seen article about “Migrating to incremental >>>>>>> repairs” but since I didn’t use repairs before at all and I use >>>>>>> Cassandra >>>>>>> version v3.0.8, then maybe not all steps are needed which are mentioned >>>>>>> in >>>>>>> Datastax article. >>>>>>> Should I start with full repair or I can start with executing >>>>>>> “nodetool repair -pr my_keyspace” on all nodes without autocompaction >>>>>>> disabling and node stopping? >>>>>>> >>>>>>> I have 6 datacenters with 6 nodes in each DC. Is it enough to run >>>>>>> “nodetool repair -pr my_keyspace” in one DC only or it should be >>>>>>> executed >>>>>>> on all nodes in _all_ DCs? >>>>>>> >>>>>>> I have tried to perform “nodetool repair -pr my_keyspace” on all >>>>>>> nodes in all datacenters sequentially but I still can see non repaired >>>>>>> SSTables for my_keyspace (Repaired at: 0). Is it expected behavior if >>>>>>> during repair data in my_keyspace wasn’t modified (no writes, no reads)? >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >
Upgrade cassandra 2.1.14 to 3.0.7
Hello guys We are planing to upgrade cassandra soon to the version 3.0.7 from 2.1.14. Our concern is the repair, in 3.0.7 repairs inc are by default. Then it means that once we do the upgrade to 3.0.7 we must follow the migration process of repairs inc for all our data in order to mark the sstables as repaired ? or we can just run directly the repair command without need to mark the sstables previously? My first test with ccm tells me that we don't need to mark the sstables because the repair in the 3.0.7 do it for you, but I want to ask if someone has done this migration and confirm my assumption Best regards. Jean Carlo "The best way to predict the future is to invent it" Alan Kay
Re: Sync failed between in AntiEntropySessions - Repair
Thank you very much Paulo On Aug 5, 2016 17:31, "Paulo Motta" <pauloricard...@gmail.com> wrote: > https://issues.apache.org/jira/browse/CASSANDRA-11840 > > increase streaming_socket_timeout to 8640 or upgrade to > cassandra-2.1.15. > > 2016-08-05 12:28 GMT-03:00 Jean Carlo <jean.jeancar...@gmail.com>: > >> >> Hello Paulo, >> >> Thx for your fast replay. >> >> You are right about that node, I did not see it that fast. In this node >> we have errors of .SocketTimeoutException: null >> >> >> >> ERROR [STREAM-IN-/192.168.0.146] 2016-08-04 19:10:59,456 >> StreamSession.java:505 - [Stream #06c02460-5a5e-11e6-8e9a-a5bf51981ad8] >> Streaming error occurred >> java.net.SocketTimeoutException: null >> at >> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211) >> ~[na:1.8.0_60] >> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) >> ~[na:1.8.0_60] >> at >> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) >> ~[na:1.8.0_60] >> at org.apache.cassandra.streaming.messages.StreamMessage. >> deserialize(StreamMessage.java:51) ~[apache-cassandra-2.1.14.jar:2.1.14] >> at org.apache.cassandra.streaming.ConnectionHandler$IncomingMes >> sageHandler.run(ConnectionHandler.java:257) >> ~[apache-cassandra-2.1.14.jar:2.1.14] >> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] >> >> We have this error with at least 5 more nodes in the same log. But the >> error doesn't say that much >> >> >> Saludos >> >> Jean Carlo >> >> "The best way to predict the future is to invent it" Alan Kay >> >> On Fri, Aug 5, 2016 at 5:16 PM, Paulo Motta <pauloricard...@gmail.com> >> wrote: >> >>> you need to check 192.168.0.36/10.234.86.36 for streaming ERRORS >>> >>> 2016-08-05 12:08 GMT-03:00 Jean Carlo <jean.jeancar...@gmail.com>: >>> >>>> Hi Paulo >>>> >>>> I found the lines, we got an exception "Outgoing stream handler has >>>> been closed" these are >>>> >>>> ERROR [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,772 >>>> StreamSession.java:621 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40] >>>> Remote peer 192.168.0.36 failed stream session. >>>> INFO [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,772 >>>> StreamResultFuture.java:180 - [Stream >>>> #c4e79260-5a46-11e6-9993-e11d93fd5b40] >>>> Session with /192.168.0.36 is complete >>>> WARN [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,773 >>>> StreamResultFuture.java:207 - [Stream >>>> #c4e79260-5a46-11e6-9993-e11d93fd5b40] >>>> Stream failed >>>> ERROR [StreamReceiveTask:107995] 2016-08-04 16:55:53,782 >>>> StreamReceiveTask.java:183 - Error applying streamed data: >>>> java.lang.RuntimeException: Outgoing stream handler has been closed >>>> at >>>> org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:138) >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>>> at >>>> org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:697) >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>>> at >>>> org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:653) >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>>> at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletio >>>> nRunnable.run(StreamReceiveTask.java:179) >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >>>> at >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>> [na:1.8.0_60] >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>> [na:1.8.0_60] >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>> [na:1.8.0_60] >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>> [na:1.8.0_60] >>>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] >>>> >>>> ERROR [StreamReceiveTask:107995] 2016-08-04 16:55:53,782 >>>> StreamSession.java:505 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40] >>>> Streaming error occurred >>>> java.lang.RuntimeExcept
Re: Sync failed between in AntiEntropySessions - Repair
Hello Paulo, Thx for your fast replay. You are right about that node, I did not see it that fast. In this node we have errors of .SocketTimeoutException: null ERROR [STREAM-IN-/192.168.0.146] 2016-08-04 19:10:59,456 StreamSession.java:505 - [Stream #06c02460-5a5e-11e6-8e9a-a5bf51981ad8] Streaming error occurred java.net.SocketTimeoutException: null at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211) ~[na:1.8.0_60] at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[na:1.8.0_60] at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) ~[na:1.8.0_60] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:51) ~[apache-cassandra-2.1.14.jar:2.1.14] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:257) ~[apache-cassandra-2.1.14.jar:2.1.14] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] We have this error with at least 5 more nodes in the same log. But the error doesn't say that much Saludos Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Fri, Aug 5, 2016 at 5:16 PM, Paulo Motta <pauloricard...@gmail.com> wrote: > you need to check 192.168.0.36/10.234.86.36 for streaming ERRORS > > 2016-08-05 12:08 GMT-03:00 Jean Carlo <jean.jeancar...@gmail.com>: > >> Hi Paulo >> >> I found the lines, we got an exception "Outgoing stream handler has been >> closed" these are >> >> ERROR [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,772 >> StreamSession.java:621 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40] >> Remote peer 192.168.0.36 failed stream session. >> INFO [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,772 >> StreamResultFuture.java:180 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40] >> Session with /192.168.0.36 is complete >> WARN [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,773 >> StreamResultFuture.java:207 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40] >> Stream failed >> ERROR [StreamReceiveTask:107995] 2016-08-04 16:55:53,782 >> StreamReceiveTask.java:183 - Error applying streamed data: >> java.lang.RuntimeException: Outgoing stream handler has been closed >> at >> org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:138) >> ~[apache-cassandra-2.1.14.jar:2.1.14] >> at >> org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:697) >> ~[apache-cassandra-2.1.14.jar:2.1.14] >> at >> org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:653) >> ~[apache-cassandra-2.1.14.jar:2.1.14] >> at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletio >> nRunnable.run(StreamReceiveTask.java:179) ~[apache-cassandra-2.1.14.jar: >> 2.1.14] >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> [na:1.8.0_60] >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> [na:1.8.0_60] >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> [na:1.8.0_60] >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> [na:1.8.0_60] >> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] >> >> ERROR [StreamReceiveTask:107995] 2016-08-04 16:55:53,782 >> StreamSession.java:505 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40] >> Streaming error occurred >> java.lang.RuntimeException: Outgoing stream handler has been closed >> at >> org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:138) >> ~[apache-cassandra-2.1.14.jar:2.1.14] >> at >> org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:697) >> ~[apache-cassandra-2.1.14.jar:2.1.14] >> at >> org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:653) >> ~[apache-cassandra-2.1.14.jar:2.1.14] >> at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletio >> nRunnable.run(StreamReceiveTask.java:179) ~[apache-cassandra-2.1.14.jar: >> 2.1.14] >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> [na:1.8.0_60] >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> [na:1.8.0_60] >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> [na:1.8.0_60] >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPool
Re: Sync failed between in AntiEntropySessions - Repair
Hi Paulo I found the lines, we got an exception "Outgoing stream handler has been closed" these are ERROR [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,772 StreamSession.java:621 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40] Remote peer 192.168.0.36 failed stream session. INFO [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,772 StreamResultFuture.java:180 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40] Session with /192.168.0.36 is complete WARN [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,773 StreamResultFuture.java:207 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40] Stream failed ERROR [StreamReceiveTask:107995] 2016-08-04 16:55:53,782 StreamReceiveTask.java:183 - Error applying streamed data: java.lang.RuntimeException: Outgoing stream handler has been closed at org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:138) ~[apache-cassandra-2.1.14.jar:2.1.14] at org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:697) ~[apache-cassandra-2.1.14.jar:2.1.14] at org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:653) ~[apache-cassandra-2.1.14.jar:2.1.14] at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:179) ~[apache-cassandra-2.1.14.jar:2.1.14] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_60] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_60] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_60] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_60] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] ERROR [StreamReceiveTask:107995] 2016-08-04 16:55:53,782 StreamSession.java:505 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40] Streaming error occurred java.lang.RuntimeException: Outgoing stream handler has been closed at org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:138) ~[apache-cassandra-2.1.14.jar:2.1.14] at org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:697) ~[apache-cassandra-2.1.14.jar:2.1.14] at org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:653) ~[apache-cassandra-2.1.14.jar:2.1.14] at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:179) ~[apache-cassandra-2.1.14.jar:2.1.14] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_60] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_60] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_60] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_60] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] I al looking through the changes files to see if it is a bug fixed in the 2.1.15 Saludos Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Fri, Aug 5, 2016 at 4:24 PM, Paulo Motta <pauloricard...@gmail.com> wrote: > It seems you have a streaming error, look for ERROR statement in the > streaming classes before that which may give you a more specific root > cause. In any case, I'd suggest you to upgrade to 2.1.15 as there were a > couple of streaming fixes on this version that might help. > > 2016-08-05 11:15 GMT-03:00 Jean Carlo <jean.jeancar...@gmail.com>: > >> >> Hi guys, Doing a repair I got this error for 2 tokenranges. >> >> ERROR [Thread-2499244] 2016-08-04 20:05:24,288 StorageService.java:3068 - >> Repair session 41e4bab0-5a63-11e6-9993-e11d93fd5b40 for range >> (487410372471205090,492009442088088379] failed with error >> org.apache.cassandra.exceptions.RepairExcep >> tion: [repair #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv, >> (487410372471205090,492009442088088379]] Sync failed between / >> 192.168.0.144 and /192.168.0.37 >> java.util.concurrent.ExecutionException: java.lang.RuntimeException: >> org.apache.cassandra.exceptions.RepairException: [repair >> #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv, >> (487410372471205090,492009442088088379]] Sync failed between / >> 192.168.0.144 and /192.168.0.37 >> at java.util.concurrent.FutureTask.report(FutureTask.java:122) >> [na:1.8.0_60] >> at java.util.concurrent.FutureTask.get(FutureTask.java:192) >> [na:1.8.0_60] >> at >> org.apache.cassandra.service.StorageService$4.runMayThrow(StorageService.java:3059) >> ~[apache-cassandra-2.1.14.jar:2.1.14] >> at >> org.apache.cassandra
Sync failed between in AntiEntropySessions - Repair
Hi guys, Doing a repair I got this error for 2 tokenranges. ERROR [Thread-2499244] 2016-08-04 20:05:24,288 StorageService.java:3068 - Repair session 41e4bab0-5a63-11e6-9993-e11d93fd5b40 for range (487410372471205090,492009442088088379] failed with error org.apache.cassandra.exceptions.RepairExcep tion: [repair #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv, (487410372471205090,492009442088088379]] Sync failed between /192.168.0.144 and /192.168.0.37 java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.RepairException: [repair #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv, (487410372471205090,492009442088088379]] Sync failed between /192.168.0.144 and /192.168.0.37 at java.util.concurrent.FutureTask.report(FutureTask.java:122) [na:1.8.0_60] at java.util.concurrent.FutureTask.get(FutureTask.java:192) [na:1.8.0_60] at org.apache.cassandra.service.StorageService$4.runMayThrow(StorageService.java:3059) ~[apache-cassandra-2.1.14.jar:2.1.14] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) [apache-cassandra-2.1.14.jar:2.1.14] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_60] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_60] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.RepairException: [repair #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv, (487410372471205090,492009442088088379]] Sync failed between /10.234.72.144 and /10.234.86.37 at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) [apache-cassandra-2.1.14.jar:2.1.14] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_60] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_60] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_60] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_60] ... 1 common frames omitted Caused by: org.apache.cassandra.exceptions.RepairException: [repair #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv, (487410372471205090,492009442088088379]] Sync failed between /192.168.0.144 and /192.168.0.37 at org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:223) ~[apache-cassandra-2.1.14.jar:2.1.14] at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:422) ~[apache-cassandra-2.1.14.jar:2.1.14] at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:134) ~[apache-cassandra-2.1.14.jar:2.1.14] at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[apache-cassandra-2.1.14.jar:2.1.14] ... 3 common frames omitted Is it a network problem or we are having a bug ? There is nothing else in the log of cassandra more than Sync failed between. We are having cassandra 2.1.14 Best regards Jean Carlo "The best way to predict the future is to invent it" Alan Kay
Exclude a host from the repair process
If a node is down in my cluster. Is it possible to exclude him from the repair process in order to continue with the repair? If not Is the repair continue reparing the other replicas even if one is down? Best regards Jean Carlo "The best way to predict the future is to invent it" Alan Kay
Re: Question about sequential repair vs parallel
Thank you Stefano. I have problems to understand the implementation of a sequential repair. In the doc of datastax <https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/opsRepairNodesManualRepair.html>they explain that in a sequential repair "The coordinator node constructs the Merkle trees for one replica after the other". But in the logs I can see this lines INFO [AntiEntropyStage:1] 2016-06-24 10:02:09,000 RepairSession.java:171 - [repair #f31281c0-39e1-11e6-96a9-d559db24120b] Received merkle tree for syndic_mail_unsub from /192.0.0.2 For every replica. Then is it the coordinator how construct the merkle tree demanding that to every replica? Why with repair sequential are we having more stream of sstable? Are they sharing the whole snapshot? Thnak you again <https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/opsRepairNodesManualRepair.html> Saludos Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Thu, Jun 23, 2016 at 5:34 PM, Stefano Ortolani <ostef...@gmail.com> wrote: > Yes, because you keep a snapshot in the meanwhile if I remember correctly. > > Regards, > Stefano > > On Thu, Jun 23, 2016 at 4:22 PM, Jean Carlo <jean.jeancar...@gmail.com> > wrote: > >> Cassandra 2.1.12 >> >> In the moment of a repair -pr sequential, we are experimenting an >> exponential increase of number of sstables. For a table lcs. >> >> If someone of you guys kowns if theoritically speaking a sequential >> repair doing more sstable streaming among replicas than a parallel repair? >> >> >> >> Best regards >> >> Jean Carlo >> >> "The best way to predict the future is to invent it" Alan Kay >> > >
Question about sequential repair vs parallel
Cassandra 2.1.12 In the moment of a repair -pr sequential, we are experimenting an exponential increase of number of sstables. For a table lcs. If someone of you guys kowns if theoritically speaking a sequential repair doing more sstable streaming among replicas than a parallel repair? Best regards Jean Carlo "The best way to predict the future is to invent it" Alan Kay
Re: Data repairing on one node questionably affects data on others
Well that may explain why you do have an unbalanced distribution of data. It is not cassandra normal behaivor. If you lose your disk, and you don't have a backup of your data (at least system because it is not replicated), this is what I think should be the normal procedure to recover your node. - stop cassandra on the node down - nodetool removenode id - update de conf of your cluster to remove the ip of the node removed (cassandra.yaml, cassandra-topology.properties). You might want to make a copy of your original conf - restart your cluster - check the node does not appear in nodetool status - restore your previous conf of your cluster. including the node that was down - on the node down, if its ip address is in the list of seeds, remove it in order to do the boobstrap - restart cassandra in every node one by one. It is what we do in our cluster. Saludos Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Tue, Apr 26, 2016 at 2:25 PM, ssiv...@gmail.com <ssiv...@gmail.com> wrote: > No, I didn't. I just want to understand how C* handle such case and what > is a predictable behavior. > > > On 04/26/2016 02:51 PM, Jean Carlo wrote: > > Did you use a backup of the keyspace system? > > If not, you might do removenode of that node and re added to the cluster > to re generate new tokens. > > > Saludos > > Jean Carlo > > "The best way to predict the future is to invent it" Alan Kay > > On Tue, Apr 26, 2016 at 12:06 AM, <ssiv...@gmail.com>ssiv...@gmail.com < > ssiv...@gmail.com> wrote: > >> Hi All, >> >> I have cluster of 7 nodes completely balanced (each node owns ~500GB of >> data). >> And I have one keyspace and one table and three replicas. Than, I just >> failed one node's disk, replace it with a new one and started repairing. >> During that process I noticed that additional two nodes have started >> getting data, and at the end of the repairing three nodes have twice more >> data than at the beginning. >> I'm curious, is it a normal behavior for Cassandra? Why not only one >> node, but three, have gotten data during repairing? May be it's because of >> clocks skew? >> >> Thanks! >> >> -- >> best regards, >> Sergey >> >> > > -- > Thanks, > Serj > >
Re: Data repairing on one node questionably affects data on others
Did you use a backup of the keyspace system? If not, you might do removenode of that node and re added to the cluster to re generate new tokens. Saludos Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Tue, Apr 26, 2016 at 12:06 AM, ssiv...@gmail.com <ssiv...@gmail.com> wrote: > Hi All, > > I have cluster of 7 nodes completely balanced (each node owns ~500GB of > data). > And I have one keyspace and one table and three replicas. Than, I just > failed one node's disk, replace it with a new one and started repairing. > During that process I noticed that additional two nodes have started > getting data, and at the end of the repairing three nodes have twice more > data than at the beginning. > I'm curious, is it a normal behavior for Cassandra? Why not only one node, > but three, have gotten data during repairing? May be it's because of clocks > skew? > > Thanks! > > -- > best regards, > Sergey > >
Re: Migration from 2.0.10 to 2.1.12
@Laing thx for the info.
@Carlos I also check that page and I did not find it. I was asking to know
if someone has done smth to avoid change the column name everytime
cassandra adds new words to the list.
Saludos
Jean Carlo
"The best way to predict the future is to invent it" Alan Kay
On Wed, Mar 30, 2016 at 4:02 PM, Carlos Alonso <i...@mrcalonso.com> wrote:
> Well...
>
> I guess that if full is a reserved word there's nothing you can do to
> change that but then, checking which are the keywords for that version it
> turns out that full is not there!!
> https://docs.datastax.com/en/cql/3.1/cql/cql_reference/keywords_r.html
>
> Maybe a bug?
>
> Regards
>
> Carlos Alonso | Software Engineer | @calonso <https://twitter.com/calonso>
>
> On 30 March 2016 at 15:41, Jean Carlo <jean.jeancar...@gmail.com> wrote:
>
>> Yes we did some reads and writes, the problem is that adding double
>> quotes force us to modify our code to change and insert like that
>>
>> INSERT INTO table1 (bill_id, *full*, name,provider_date ,total) values
>> ('qs','full','name','2015-02-23','toltal');
>>
>> to this
>>
>> INSERT INTO table1 (bill_id, *"full"*, name,provider_date ,total) values
>> ('qs','full','name','2015-02-23','toltal');
>>
>> this last one is ok, but obviously the first one makes an error:
>>
>> cqlsh:pns_fr_2_jean> INSERT INTO table1 (bill_id, full,
>> name,provider_date ,total) values
>> ('qs','full','name','2015-02-23','toltal');
>> SyntaxException: > message="line 1:29 no viable alternative at input '*full*' (INSERT INTO
>> table1 (bill_id, [full]...)">
>>
>> and it is because the name of the column is not longer full, is "full"
>>
>>
>>
>> Best regards
>>
>> Jean Carlo
>>
>> "The best way to predict the future is to invent it" Alan Kay
>>
>> On Wed, Mar 30, 2016 at 3:26 PM, Eric Evans <eev...@wikimedia.org> wrote:
>>
>>> On Wed, Mar 30, 2016 at 8:08 AM, Jean Carlo <jean.jeancar...@gmail.com>
>>> wrote:
>>> > With double quotes it doesn't show error
>>> >
>>> > CREATE TABLE table1 ( bill_id text, "full" text, name text,
>>> > provider_date timestamp, total text, PRIMARY KEY ( bill_id) ) ;
>>> >
>>> > but it changes the name of the column
>>>
>>> I don't think it does though; You're meant to be able to use the
>>> output of DESC TABLE to (re)create these tables, so it's quoted below
>>> for the same reason you did so above.
>>>
>>> Obviously it would be better to have column names that parse without
>>> needing to be double quoted, but it should work. Have you tried some
>>> reads and writes?
>>>
>>> > desc table table1;
>>> >
>>> > CREATE TABLE pns_fr_2_jean.table1 (
>>> > bill_id text PRIMARY KEY,
>>> > "full" text,
>>> > name text,
>>> > provider_date timestamp,
>>> > total text
>>> > )
>>> >
>>> > instead of
>>> >
>>> > CREATE TABLE pns_fr_2_jean.table1 (
>>> > bill_id text PRIMARY KEY,
>>> > full text,
>>> > name text,
>>> > provider_date timestamp,
>>> > total text
>>> > )
>>>
>>>
>>> --
>>> Eric Evans
>>> eev...@wikimedia.org
>>>
>>
>>
>
Re: Migration from 2.0.10 to 2.1.12
Yes we did some reads and writes, the problem is that adding double quotes
force us to modify our code to change and insert like that
INSERT INTO table1 (bill_id, *full*, name,provider_date ,total) values
('qs','full','name','2015-02-23','toltal');
to this
INSERT INTO table1 (bill_id, *"full"*, name,provider_date ,total) values
('qs','full','name','2015-02-23','toltal');
this last one is ok, but obviously the first one makes an error:
cqlsh:pns_fr_2_jean> INSERT INTO table1 (bill_id, full, name,provider_date
,total) values ('qs','full','name','2015-02-23','toltal');
SyntaxException:
and it is because the name of the column is not longer full, is "full"
Best regards
Jean Carlo
"The best way to predict the future is to invent it" Alan Kay
On Wed, Mar 30, 2016 at 3:26 PM, Eric Evans <eev...@wikimedia.org> wrote:
> On Wed, Mar 30, 2016 at 8:08 AM, Jean Carlo <jean.jeancar...@gmail.com>
> wrote:
> > With double quotes it doesn't show error
> >
> > CREATE TABLE table1 ( bill_id text, "full" text, name text,
> > provider_date timestamp, total text, PRIMARY KEY ( bill_id) ) ;
> >
> > but it changes the name of the column
>
> I don't think it does though; You're meant to be able to use the
> output of DESC TABLE to (re)create these tables, so it's quoted below
> for the same reason you did so above.
>
> Obviously it would be better to have column names that parse without
> needing to be double quoted, but it should work. Have you tried some
> reads and writes?
>
> > desc table table1;
> >
> > CREATE TABLE pns_fr_2_jean.table1 (
> > bill_id text PRIMARY KEY,
> > "full" text,
> > name text,
> > provider_date timestamp,
> > total text
> > )
> >
> > instead of
> >
> > CREATE TABLE pns_fr_2_jean.table1 (
> > bill_id text PRIMARY KEY,
> > full text,
> > name text,
> > provider_date timestamp,
> > total text
> > )
>
>
> --
> Eric Evans
> eev...@wikimedia.org
>
Re: Migration from 2.0.10 to 2.1.12
With double quotes it doesn't show error CREATE TABLE table1 ( bill_id text, "full" text, name text, provider_date timestamp, total text, PRIMARY KEY ( bill_id) ) ; but it changes the name of the column desc table table1; CREATE TABLE pns_fr_2_jean.table1 ( bill_id text PRIMARY KEY, *"full" text,* name text, provider_date timestamp, total text ) instead of CREATE TABLE pns_fr_2_jean.table1 ( bill_id text PRIMARY KEY, *full text,* name text, provider_date timestamp, total text ) and with singles quotes the result is an error CREATE TABLE table1 ( bill_id text, * 'full' text*, name text, provider_date timestamp, total text, PRIMARY KEY ( bill_id) ) ; SyntaxException: So far using double quotes doesn't resolve my problem Saludos Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Wed, Mar 30, 2016 at 12:53 PM, Carlos Alonso <i...@mrcalonso.com> wrote: > Try surrounding it with quotes. > > Didn't know about the reserved keyword full though. > > Regards > > Carlos Alonso | Software Engineer | @calonso <https://twitter.com/calonso> > > On 30 March 2016 at 10:36, Jean Carlo <jean.jeancar...@gmail.com> wrote: > >> Hi! >> >> I am creating my new tables on cassandra 2.1.12 and I have this one >> >> CREATE TABLE table1 ( >> bill_id text, >> full text, >> name text, >> provider_date timestamp, >> total text, >> PRIMARY KEY ( bill_id) >> ) >> >> And cassandra tells me this error >> >> SyntaxException: > message="line 1:44 mismatched input 'full' expecting ')' (... table1 ( >> bill_id text, [full] text...)"> >> >> Is the word "full" reserved by cassandra? >> >> In order to not change all the configuration in production, there is a >> way to conserve the name of my column as it was with the version 2.0.10? >> >> >> Saludos >> >> Jean Carlo >> >> "The best way to predict the future is to invent it" Alan Kay >> > >
Migration from 2.0.10 to 2.1.12
Hi! I am creating my new tables on cassandra 2.1.12 and I have this one CREATE TABLE table1 ( bill_id text, full text, name text, provider_date timestamp, total text, PRIMARY KEY ( bill_id) ) And cassandra tells me this error SyntaxException: Is the word "full" reserved by cassandra? In order to not change all the configuration in production, there is a way to conserve the name of my column as it was with the version 2.0.10? Saludos Jean Carlo "The best way to predict the future is to invent it" Alan Kay
Re: Cassandra-stress output
However can it be nice to have the posibility to configurate that with
cassandra options, or when using a file yaml to insert data on any table.
Saludos
Jean Carlo
"The best way to predict the future is to invent it" Alan Kay
On Thu, Mar 10, 2016 at 10:48 AM, Jean Carlo <jean.jeancar...@gmail.com>
wrote:
> Thank you very much S. Alborghetti I will consider that suggestion.
>
>
> Saludos
>
> Jean Carlo
>
> "The best way to predict the future is to invent it" Alan Kay
>
> On Thu, Mar 10, 2016 at 5:47 AM, Stefania Alborghetti <
> stefania.alborghe...@datastax.com> wrote:
>
>> On Tue, Mar 8, 2016 at 8:39 PM, Jean Carlo <jean.jeancar...@gmail.com>
>> wrote:
>>
>> > Hi guys,
>> >
>> > I use cassandra stress to populate the next table
>> >
>> > CREATE TABLE cf1 (
>> > kvalue text,
>> > ktype text,
>> > prov text,
>> > dname text,
>> > dattrib blob,
>> > dvalue text,
>> > PRIMARY KEY (kvalue, ktype, prov, dname)
>> > ) WITH bloom_filter_fp_chance = 0.01
>> > AND caching = '{"keys":"ALL", "rows_per_partition":"60"}'
>> > AND comment = ''
>> > AND compaction = {'class':
>> > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
>> > AND compression = {'sstable_compression':
>> > 'org.apache.cassandra.io.compress.SnappyCompressor'}
>> > AND dclocal_read_repair_chance = 0.02
>> > AND default_time_to_live = 0
>> > AND gc_grace_seconds = 864000
>> > AND max_index_interval = 2048
>> > AND memtable_flush_period_in_ms = 0
>> > AND min_index_interval = 128
>> > AND read_repair_chance = 0.01
>> > AND speculative_retry = '99.0PERCENTILE';
>> >
>> > And cassandra stress create the next string to the field kvalue of type
>> > text:
>> >
>> > "P*d,xY\x03m\x1b\x10\x0b$\x04pt-G\x08\n`7\x1fs\x15kH\x02i1\x16jf%YM"
>> >
>> > what bothers me is that kvalue has control characters like \x03. do you
>> > guys know any way to avoid creating this kind of characters while using
>> > cassandra-stress?
>> >
>> >
>> >
>> > Thank you very much
>> >
>> > Jean Carlo
>> >
>> > "The best way to predict the future is to invent it" Alan Kay
>> >
>>
>>
>> There is no way to avoid the control characters (<32 and ==127), other
>> than
>> modifying the source code, which is located in
>> tools/stress/src/org/apache/cassandra/stress/generate/values/Strings.java.
>>
>> Changing this line:
>>
>> chars[i++] = (char) (((v & 127) + 32) & 127);
>>
>> with this:
>>
>> chars[i++] = (char) (((v & 127) % 95) + 32);
>>
>> should work but I could not avoid the expensive modulo operation. You can
>> rebuild cassandra-stress with ant stress-build.
>>
>> I wonder if the original intention was to avoid control characters
>> however,
>> given the +32 in the original line. For this reason I've copied this
>> message to the dev mailing list.
>>
>>
>> --
>>
>>
>> [image: datastax_logo.png] <http://www.datastax.com/>
>>
>> Stefania Alborghetti
>>
>> Apache Cassandra Software Engineer
>>
>> |+852 6114 9265| stefania.alborghe...@datastax.com
>>
>
>
Re: Cassandra-stress output
Thank you very much S. Alborghetti I will consider that suggestion.
Saludos
Jean Carlo
"The best way to predict the future is to invent it" Alan Kay
On Thu, Mar 10, 2016 at 5:47 AM, Stefania Alborghetti <
stefania.alborghe...@datastax.com> wrote:
> On Tue, Mar 8, 2016 at 8:39 PM, Jean Carlo <jean.jeancar...@gmail.com>
> wrote:
>
> > Hi guys,
> >
> > I use cassandra stress to populate the next table
> >
> > CREATE TABLE cf1 (
> > kvalue text,
> > ktype text,
> > prov text,
> > dname text,
> > dattrib blob,
> > dvalue text,
> > PRIMARY KEY (kvalue, ktype, prov, dname)
> > ) WITH bloom_filter_fp_chance = 0.01
> > AND caching = '{"keys":"ALL", "rows_per_partition":"60"}'
> > AND comment = ''
> > AND compaction = {'class':
> > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> > AND compression = {'sstable_compression':
> > 'org.apache.cassandra.io.compress.SnappyCompressor'}
> > AND dclocal_read_repair_chance = 0.02
> > AND default_time_to_live = 0
> > AND gc_grace_seconds = 864000
> > AND max_index_interval = 2048
> > AND memtable_flush_period_in_ms = 0
> > AND min_index_interval = 128
> > AND read_repair_chance = 0.01
> > AND speculative_retry = '99.0PERCENTILE';
> >
> > And cassandra stress create the next string to the field kvalue of type
> > text:
> >
> > "P*d,xY\x03m\x1b\x10\x0b$\x04pt-G\x08\n`7\x1fs\x15kH\x02i1\x16jf%YM"
> >
> > what bothers me is that kvalue has control characters like \x03. do you
> > guys know any way to avoid creating this kind of characters while using
> > cassandra-stress?
> >
> >
> >
> > Thank you very much
> >
> > Jean Carlo
> >
> > "The best way to predict the future is to invent it" Alan Kay
> >
>
>
> There is no way to avoid the control characters (<32 and ==127), other than
> modifying the source code, which is located in
> tools/stress/src/org/apache/cassandra/stress/generate/values/Strings.java.
>
> Changing this line:
>
> chars[i++] = (char) (((v & 127) + 32) & 127);
>
> with this:
>
> chars[i++] = (char) (((v & 127) % 95) + 32);
>
> should work but I could not avoid the expensive modulo operation. You can
> rebuild cassandra-stress with ant stress-build.
>
> I wonder if the original intention was to avoid control characters however,
> given the +32 in the original line. For this reason I've copied this
> message to the dev mailing list.
>
>
> --
>
>
> [image: datastax_logo.png] <http://www.datastax.com/>
>
> Stefania Alborghetti
>
> Apache Cassandra Software Engineer
>
> |+852 6114 9265| stefania.alborghe...@datastax.com
>
Cassandra-stress output
Hi guys,
I use cassandra stress to populate the next table
CREATE TABLE cf1 (
kvalue text,
ktype text,
prov text,
dname text,
dattrib blob,
dvalue text,
PRIMARY KEY (kvalue, ktype, prov, dname)
) WITH bloom_filter_fp_chance = 0.01
AND caching = '{"keys":"ALL", "rows_per_partition":"60"}'
AND comment = ''
AND compaction = {'class':
'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
AND compression = {'sstable_compression':
'org.apache.cassandra.io.compress.SnappyCompressor'}
AND dclocal_read_repair_chance = 0.02
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.01
AND speculative_retry = '99.0PERCENTILE';
And cassandra stress create the next string to the field kvalue of type
text:
"P*d,xY\x03m\x1b\x10\x0b$\x04pt-G\x08\n`7\x1fs\x15kH\x02i1\x16jf%YM"
what bothers me is that kvalue has control characters like \x03. do you
guys know any way to avoid creating this kind of characters while using
cassandra-stress?
Thank you very much
Jean Carlo
"The best way to predict the future is to invent it" Alan Kay
Re: Do I have to use repair -inc with the option -par forcely?
Hi , Thx @alain for you reply. Yes we have the 2.1.12. We are definitely facing CASSANDRA-10422 <https://issues.apache.org/jira/browse/CASSANDRA-10422>. However I cannot run incremental repairs without add -par @carlos if what you say it's correct, it will be really nice because the process to migrate it's quite tedious if you have many nodes Saludos Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Tue, Feb 16, 2016 at 4:39 PM, Carlos Rolo <r...@pythian.com> wrote: > +1 on what Alain said, but I do think if you are high enough on a 2.1.x > (will look later) version you don't need to follow the documentation. It is > outdated. Run a full repair, the you can start incremental repairs since > the SSTables will have the metadata on them about the last repair. > > Wait someone to confirm this/or confirm the docs are correct. > > Regards, > > Carlos Juzarte Rolo > Cassandra Consultant / Datastax Certified Architect / Cassandra MVP > > Pythian - Love your data > > rolo@pythian | Twitter: @cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo > <http://linkedin.com/in/carlosjuzarterolo>* > Mobile: +351 91 891 81 00 | Tel: +1 613 565 8696 x1649 > www.pythian.com > > On Tue, Feb 16, 2016 at 1:45 PM, Alain RODRIGUEZ <arodr...@gmail.com> > wrote: > >> Hi, >> >> I am testing repairs repairs -inc -par and I can see that in all my nodes >>> the numbers of sstables explode to 5k from 5 sstables. >> >> >> This looks like a known issue, see >> https://issues.apache.org/jira/browse/CASSANDRA-10422 >> Make sure your version is higher than 2.1.12, 2.2.4, 3.0.1, 3.1 to avoid >> this (and you are indeed facing CASSANDRA-10422). >> I am not sure you are facing this though, as you don't seem to be using >> subranges (nodetool repair -st and -et options) >> >> *It is anyway to run repairs incrementals but not -par ?* >>> >>> I know that is it not possible to run sequential repair with incremental >>> repair at the same time. >>> >> >> >> From http://www.datastax.com/dev/blog/more-efficient-repairs >> "Incremental repairs can be opted into via the -inc option to nodetool >> repair. This is compatible with both sequential and parallel (-par) >> repair, e.g., bin/nodetool -par -inc ." >> So you should be able to remove -par. Not sure this will solve your issue >> though. >> >> >> Did you respect this process to migrate to incremental repairs? >> >> https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/opsRepairNodesMigration.html#opsRepairNodesMigration__ol_dxj_gp5_2s >> >> C*heers, >> - >> Alain Rodriguez >> France >> >> The Last Pickle >> http://www.thelastpickle.com >> >> >> >> 2016-02-10 17:45 GMT+01:00 Jean Carlo <jean.jeancar...@gmail.com>: >> >>> Hi guys; The question is on the subject. >>> >>> I am testing repairs repairs -inc -par and I can see that in all my >>> nodes the numbers of sstables explode to 5k from 5 sstables. >>> >>> I cannot permit this behaivor on my cluster in production. >>> >>> *It is anyway to run repairs incrementals but not -par ?* >>> >>> I know that is it not possible to run sequential repair with incremental >>> repair at the same time. >>> >>> Best regards >>> >>> Jean Carlo >>> >>> "The best way to predict the future is to invent it" Alan Kay >>> >> >> > > -- > > > >
Re: 3k sstables during a repair incremental !!
Hi !
@Paulo
.
Yes we are using vnodes, 256 per node and we have 6 nodes in the cluster.
RF=3. The data is inserted using jmeter with consistency=LOCAL_ONE.
Because this is a test, we generate our data and we insert them using
jmeter.
After the repair finished, all the nodes seems to be freezed, no compaction
were runing, even if nodetool tpstats says
CompactionExecutor
283 1424 0 0
After I restart cassandra, the comactions started to run.
Saludos
Jean Carlo
"The best way to predict the future is to invent it" Alan Kay
On Thu, Feb 11, 2016 at 8:42 AM, Marcus Eriksson <krum...@gmail.com> wrote:
> The reason for this is probably
> https://issues.apache.org/jira/browse/CASSANDRA-10831 (which only affects
> 2.1)
>
> So, if you had problems with incremental repair and LCS before, upgrade to
> 2.1.13 and try again
>
> /Marcus
>
> On Wed, Feb 10, 2016 at 2:59 PM, horschi <hors...@gmail.com> wrote:
>
>> Hi Jean,
>>
>> we had the same issue, but on SizeTieredCompaction. During repair the
>> number of SSTables and pending compactions were exploding.
>>
>> It not only affected latencies, at some point Cassandra ran out of heap.
>>
>> After the upgrade to 2.2 things got much better.
>>
>> regards,
>> Christian
>>
>>
>> On Wed, Feb 10, 2016 at 2:46 PM, Jean Carlo <jean.jeancar...@gmail.com>
>> wrote:
>> > Hi Horschi !!!
>> >
>> > I have the 2.1.12. But I think it is something related to Level
>> compaction
>> > strategy. It is impressive that we passed from 6 sstables to 3k sstable.
>> > I think this will affect the latency on production because the number of
>> > compactions going on
>> >
>> >
>> >
>> > Best regards
>> >
>> > Jean Carlo
>> >
>> > "The best way to predict the future is to invent it" Alan Kay
>> >
>> > On Wed, Feb 10, 2016 at 2:37 PM, horschi <hors...@gmail.com> wrote:
>> >>
>> >> Hi Jean,
>> >>
>> >> which Cassandra version do you use?
>> >>
>> >> Incremental repair got much better in 2.2 (for us at least).
>> >>
>> >> kind regards,
>> >> Christian
>> >>
>> >> On Wed, Feb 10, 2016 at 2:33 PM, Jean Carlo <jean.jeancar...@gmail.com
>> >
>> >> wrote:
>> >> > Hello guys!
>> >> >
>> >> > I am testing the repair inc in my custer cassandra. I am doing my
>> test
>> >> > over
>> >> > these tables
>> >> >
>> >> > CREATE TABLE pns_nonreg_bench.cf3 (
>> >> > s text,
>> >> > sp int,
>> >> > d text,
>> >> > dp int,
>> >> > m map<text, text>,
>> >> > t timestamp,
>> >> > PRIMARY KEY (s, sp, d, dp)
>> >> > ) WITH CLUSTERING ORDER BY (sp ASC, d ASC, dp ASC)
>> >> >
>> >> > AND compaction = {'class':
>> >> > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
>> >> > AND compression = {'sstable_compression':
>> >> > 'org.apache.cassandra.io.compress.SnappyCompressor'}
>> >> >
>> >> > CREATE TABLE pns_nonreg_bench.cf1 (
>> >> > ise text PRIMARY KEY,
>> >> > int_col int,
>> >> > text_col text,
>> >> > ts_col timestamp,
>> >> > uuid_col uuid
>> >> > ) WITH bloom_filter_fp_chance = 0.01
>> >> > AND compaction = {'class':
>> >> > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
>> >> > AND compression = {'sstable_compression':
>> >> > 'org.apache.cassandra.io.compress.SnappyCompressor'}
>> >> >
>> >> > table cf1
>> >> > Space used (live): 665.7 MB
>> >> > table cf2
>> >> > Space used (live): 697.03 MB
>> >> >
>> >> > It happens that when I do repair -inc -par on theses tables, cf2 got
>> a
>> >> > pick
>> >> > of 3k sstables. When the repair finish, it takes 30 min or more to
>> >> > finish
>> >> > all the compactations and return to 6 sstable.
>> >> >
>> >> > I am a little concern about if this will happen on production. is it
>> >> > normal?
>> >> >
>> >> > Saludos
>> >> >
>> >> > Jean Carlo
>> >> >
>> >> > "The best way to predict the future is to invent it" Alan Kay
>> >
>> >
>>
>
>
3k sstables during a repair incremental !!
Hello guys!
I am testing the repair inc in my custer cassandra. I am doing my test over
these tables
*CREATE TABLE pns_nonreg_bench.cf3* (
s text,
sp int,
d text,
dp int,
m map<text, text>,
t timestamp,
PRIMARY KEY (s, sp, d, dp)
) WITH CLUSTERING ORDER BY (sp ASC, d ASC, dp ASC)
AND compaction = {'class': 'org.apache.cassandra.db.compaction.
*LeveledCompactionStrategy*'}
AND compression = {'sstable_compression':
'org.apache.cassandra.io.compress.*SnappyCompressor*'}
*CREATE TABLE pns_nonreg_bench.cf1* (
ise text PRIMARY KEY,
int_col int,
text_col text,
ts_col timestamp,
uuid_col uuid
) WITH bloom_filter_fp_chance = 0.01
AND compaction = {'class': 'org.apache.cassandra.db.compaction.
*LeveledCompactionStrategy*'}
AND compression = {'sstable_compression':
'org.apache.cassandra.io.compress.*SnappyCompressor*'}
*table cf1Space used (live): 665.7 MB*
*table cf2*
*Space used (live): 697.03 MB*
It happens that when I do repair -inc -par on theses tables, *cf2 got a
pick of 3k sstables*. When the repair finish, it takes 30 min or more to
finish all the compactations and return to 6 sstable.
I am a little concern about if this will happen on production. is it normal?
Saludos
Jean Carlo
"The best way to predict the future is to invent it" Alan Kay
Re: 3k sstables during a repair incremental !!
Correction:
*table cf3*
*Space used (live): 697.03 MB*
It happens that when I do repair -inc -par on theses tables, *cf3 got a
pick of 3k sstables*. When the repair finish, it takes 30 min or more to
finish all the compactations and return to 6 sstable.
Saludos
Jean Carlo
"The best way to predict the future is to invent it" Alan Kay
On Wed, Feb 10, 2016 at 2:33 PM, Jean Carlo <jean.jeancar...@gmail.com>
wrote:
> Hello guys!
>
> I am testing the repair inc in my custer cassandra. I am doing my test
> over these tables
>
> *CREATE TABLE pns_nonreg_bench.cf3* (
> s text,
> sp int,
> d text,
> dp int,
> m map<text, text>,
> t timestamp,
> PRIMARY KEY (s, sp, d, dp)
> ) WITH CLUSTERING ORDER BY (sp ASC, d ASC, dp ASC)
>
> AND compaction = {'class': 'org.apache.cassandra.db.compaction.
> *LeveledCompactionStrategy*'}
> AND compression = {'sstable_compression':
> 'org.apache.cassandra.io.compress.*SnappyCompressor*'}
>
> *CREATE TABLE pns_nonreg_bench.cf1* (
> ise text PRIMARY KEY,
> int_col int,
> text_col text,
> ts_col timestamp,
> uuid_col uuid
> ) WITH bloom_filter_fp_chance = 0.01
> AND compaction = {'class': 'org.apache.cassandra.db.compaction.
> *LeveledCompactionStrategy*'}
> AND compression = {'sstable_compression':
> 'org.apache.cassandra.io.compress.*SnappyCompressor*'}
>
>
>
> *table cf1Space used (live): 665.7 MB*
>
> *table cf2*
> *Space used (live): 697.03 MB*
>
> It happens that when I do repair -inc -par on theses tables, *cf2 got a
> pick of 3k sstables*. When the repair finish, it takes 30 min or more to
> finish all the compactations and return to 6 sstable.
>
> I am a little concern about if this will happen on production. is it
> normal?
>
> Saludos
>
> Jean Carlo
>
> "The best way to predict the future is to invent it" Alan Kay
>
Re: 3k sstables during a repair incremental !!
Hello Horschi,
Yes I understand. Thx
Best regards
Jean Carlo
"The best way to predict the future is to invent it" Alan Kay
On Wed, Feb 10, 2016 at 3:00 PM, horschi <hors...@gmail.com> wrote:
> btw: I am not saying incremental Repair in 2.1 is broken, but ... ;-)
>
> On Wed, Feb 10, 2016 at 2:59 PM, horschi <hors...@gmail.com> wrote:
> > Hi Jean,
> >
> > we had the same issue, but on SizeTieredCompaction. During repair the
> > number of SSTables and pending compactions were exploding.
> >
> > It not only affected latencies, at some point Cassandra ran out of heap.
> >
> > After the upgrade to 2.2 things got much better.
> >
> > regards,
> > Christian
> >
> >
> > On Wed, Feb 10, 2016 at 2:46 PM, Jean Carlo <jean.jeancar...@gmail.com>
> wrote:
> >> Hi Horschi !!!
> >>
> >> I have the 2.1.12. But I think it is something related to Level
> compaction
> >> strategy. It is impressive that we passed from 6 sstables to 3k sstable.
> >> I think this will affect the latency on production because the number of
> >> compactions going on
> >>
> >>
> >>
> >> Best regards
> >>
> >> Jean Carlo
> >>
> >> "The best way to predict the future is to invent it" Alan Kay
> >>
> >> On Wed, Feb 10, 2016 at 2:37 PM, horschi <hors...@gmail.com> wrote:
> >>>
> >>> Hi Jean,
> >>>
> >>> which Cassandra version do you use?
> >>>
> >>> Incremental repair got much better in 2.2 (for us at least).
> >>>
> >>> kind regards,
> >>> Christian
> >>>
> >>> On Wed, Feb 10, 2016 at 2:33 PM, Jean Carlo <jean.jeancar...@gmail.com
> >
> >>> wrote:
> >>> > Hello guys!
> >>> >
> >>> > I am testing the repair inc in my custer cassandra. I am doing my
> test
> >>> > over
> >>> > these tables
> >>> >
> >>> > CREATE TABLE pns_nonreg_bench.cf3 (
> >>> > s text,
> >>> > sp int,
> >>> > d text,
> >>> > dp int,
> >>> > m map<text, text>,
> >>> > t timestamp,
> >>> > PRIMARY KEY (s, sp, d, dp)
> >>> > ) WITH CLUSTERING ORDER BY (sp ASC, d ASC, dp ASC)
> >>> >
> >>> > AND compaction = {'class':
> >>> > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> >>> > AND compression = {'sstable_compression':
> >>> > 'org.apache.cassandra.io.compress.SnappyCompressor'}
> >>> >
> >>> > CREATE TABLE pns_nonreg_bench.cf1 (
> >>> > ise text PRIMARY KEY,
> >>> > int_col int,
> >>> > text_col text,
> >>> > ts_col timestamp,
> >>> > uuid_col uuid
> >>> > ) WITH bloom_filter_fp_chance = 0.01
> >>> > AND compaction = {'class':
> >>> > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> >>> > AND compression = {'sstable_compression':
> >>> > 'org.apache.cassandra.io.compress.SnappyCompressor'}
> >>> >
> >>> > table cf1
> >>> > Space used (live): 665.7 MB
> >>> > table cf2
> >>> > Space used (live): 697.03 MB
> >>> >
> >>> > It happens that when I do repair -inc -par on theses tables, cf2 got
> a
> >>> > pick
> >>> > of 3k sstables. When the repair finish, it takes 30 min or more to
> >>> > finish
> >>> > all the compactations and return to 6 sstable.
> >>> >
> >>> > I am a little concern about if this will happen on production. is it
> >>> > normal?
> >>> >
> >>> > Saludos
> >>> >
> >>> > Jean Carlo
> >>> >
> >>> > "The best way to predict the future is to invent it" Alan Kay
> >>
> >>
>
Re: 3k sstables during a repair incremental !!
Hi Horschi !!!
I have the 2.1.12. But I think it is something related to Level compaction
strategy. It is impressive that we passed from 6 sstables to 3k sstable.
I think this will affect the latency on production because the number of
compactions going on
Best regards
Jean Carlo
"The best way to predict the future is to invent it" Alan Kay
On Wed, Feb 10, 2016 at 2:37 PM, horschi <hors...@gmail.com> wrote:
> Hi Jean,
>
> which Cassandra version do you use?
>
> Incremental repair got much better in 2.2 (for us at least).
>
> kind regards,
> Christian
>
> On Wed, Feb 10, 2016 at 2:33 PM, Jean Carlo <jean.jeancar...@gmail.com>
> wrote:
> > Hello guys!
> >
> > I am testing the repair inc in my custer cassandra. I am doing my test
> over
> > these tables
> >
> > CREATE TABLE pns_nonreg_bench.cf3 (
> > s text,
> > sp int,
> > d text,
> > dp int,
> > m map<text, text>,
> > t timestamp,
> > PRIMARY KEY (s, sp, d, dp)
> > ) WITH CLUSTERING ORDER BY (sp ASC, d ASC, dp ASC)
> >
> > AND compaction = {'class':
> > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> > AND compression = {'sstable_compression':
> > 'org.apache.cassandra.io.compress.SnappyCompressor'}
> >
> > CREATE TABLE pns_nonreg_bench.cf1 (
> > ise text PRIMARY KEY,
> > int_col int,
> > text_col text,
> > ts_col timestamp,
> > uuid_col uuid
> > ) WITH bloom_filter_fp_chance = 0.01
> > AND compaction = {'class':
> > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> > AND compression = {'sstable_compression':
> > 'org.apache.cassandra.io.compress.SnappyCompressor'}
> >
> > table cf1
> > Space used (live): 665.7 MB
> > table cf2
> > Space used (live): 697.03 MB
> >
> > It happens that when I do repair -inc -par on theses tables, cf2 got a
> pick
> > of 3k sstables. When the repair finish, it takes 30 min or more to finish
> > all the compactations and return to 6 sstable.
> >
> > I am a little concern about if this will happen on production. is it
> normal?
> >
> > Saludos
> >
> > Jean Carlo
> >
> > "The best way to predict the future is to invent it" Alan Kay
>
Re: 3k sstables during a repair incremental !!
Hello Kai
This is for *cf3*
nodetool cfstats pns_nonreg_bench.cf3 -H
Keyspace: pns_nonreg_bench
Read Count: 23594
Read Latency: 1.2980987539204882 ms.
Write Count: 148161
Write Latency: 0.04608940274431193 ms.
Pending Flushes: 0
Table: cf3
SSTable count: 11489
SSTables in each level: [11485/4, 4, 0, 0, 0, 0, 0, 0, 0]
Space used (live): 954.89 MB
Space used (total): 954.89 MB
Space used by snapshots (total): 0 bytes
Off heap memory used (total): 3.74 MB
SSTable Compression Ratio: 0.4245371280396339
Number of keys (estimate): 1051613
Memtable cell count: 0
Memtable data size: 0 bytes
Memtable off heap memory used: 0 bytes
Memtable switch count: 889
Local read count: 23594
Local read latency: 1.299 ms
Local write count: 148161
Local write latency: 0.047 ms
Pending flushes: 0
Bloom filter false positives: 16609
Bloom filter false ratio: 0.0
Bloom filter space used: 1.64 MB
Bloom filter off heap memory used: 1.55 MB
Index summary off heap memory used: 1.88 MB
Compression metadata off heap memory used: 318.56 KB
Compacted partition minimum bytes: 643 bytes
Compacted partition maximum bytes: 924 bytes
Compacted partition mean bytes: 914 bytes
Average live cells per slice (last five minutes): 1.0
Maximum live cells per slice (last five minutes): 1.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
This is for *cf1*
nodetool cfstats pns_nonreg_bench.cf1 -H
Keyspace: pns_nonreg_bench
Read Count: 24207
Read Latency: 0.9072712851654481 ms.
Write Count: 148380
Write Latency: 0.04561746866154468 ms.
Pending Flushes: 0
Table: cf1
SSTable count: 1942
It seems that nodetool does not retrieve the rest of the information, I
think it is due to compactions
nodetool tpstats
CompactionExecutor283 1424
0 0
Saludos
Jean Carlo
"The best way to predict the future is to invent it" Alan Kay
On Wed, Feb 10, 2016 at 6:00 PM, Kai Wang <dep...@gmail.com> wrote:
> Jean,
>
> What does your cfstats look like? Especially "SSTables in each level" line.
>
> On Wed, Feb 10, 2016 at 8:33 AM, Jean Carlo <jean.jeancar...@gmail.com>
> wrote:
>
>> Hello guys!
>>
>> I am testing the repair inc in my custer cassandra. I am doing my test
>> over these tables
>>
>> *CREATE TABLE pns_nonreg_bench.cf3* (
>> s text,
>> sp int,
>> d text,
>> dp int,
>> m map<text, text>,
>> t timestamp,
>> PRIMARY KEY (s, sp, d, dp)
>> ) WITH CLUSTERING ORDER BY (sp ASC, d ASC, dp ASC)
>>
>> AND compaction = {'class': 'org.apache.cassandra.db.compaction.
>> *LeveledCompactionStrategy*'}
>> AND compression = {'sstable_compression':
>> 'org.apache.cassandra.io.compress.*SnappyCompressor*'}
>>
>> *CREATE TABLE pns_nonreg_bench.cf1* (
>> ise text PRIMARY KEY,
>> int_col int,
>> text_col text,
>> ts_col timestamp,
>> uuid_col uuid
>> ) WITH bloom_filter_fp_chance = 0.01
>> AND compaction = {'class': 'org.apache.cassandra.db.compaction.
>> *LeveledCompactionStrategy*'}
>> AND compression = {'sstable_compression':
>> 'org.apache.cassandra.io.compress.*SnappyCompressor*'}
>>
>>
>>
>> *table cf1Space used (live): 665.7 MB*
>>
>> *table cf2*
>> *Space used (live): 697.03 MB*
>>
>> It happens that when I do repair -inc -par on theses tables, *cf2 got a
>> pick of 3k sstables*. When the repair finish, it takes 30 min or more to
>> finish all the compactations and return to 6 sstable.
>>
>> I am a little concern about if this will happen on production. is it
>> normal?
>>
>> Saludos
>>
>> Jean Carlo
>>
>> "The best way to predict the future is to invent it" Alan Kay
>>
>
>
Do I have to use repair -inc with the option -par forcely?
Hi guys; The question is on the subject. I am testing repairs repairs -inc -par and I can see that in all my nodes the numbers of sstables explode to 5k from 5 sstables. I cannot permit this behaivor on my cluster in production. *It is anyway to run repairs incrementals but not -par ?* I know that is it not possible to run sequential repair with incremental repair at the same time. Best regards Jean Carlo "The best way to predict the future is to invent it" Alan Kay
Re: Cassandra's log is full of mesages reset by peers even without traffic
Hello Annuj,, I checked my settings and this what I got. root@node001[SPH][BENCH][PnS3]:~$ sysctl -A | grep net.ipv4 | grep net.ipv4.tcp_keepalive_probes net.ipv4.tcp_keepalive_probes = 9 root@node001[SPH][BENCH][PnS3]:~$ sysctl -A | grep net.ipv4 | grep net.ipv4.tcp_keepalive_intvl net.ipv4.tcp_keepalive_intvl = 75 root@node001[SPH][BENCH][PnS3]:~$ sysctl -A | grep net.ipv4 | grep net.ipv4.tcp_keepalive_time net.ipv4.tcp_keepalive_time = 300The tcp_keepalive_time is quite high in comparation to that written on the doc https://docs.datastax.com/en/cassandra/2.1/cassandra/troubleshooting/trblshootIdleFirewall.html Do you think that is ok? Best regards Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Fri, Jan 29, 2016 at 11:02 AM, Anuj Wadehra <anujw_2...@yahoo.co.in> wrote: > Hi Jean, > > Please make sure that your Firewall is not dropping TCP connections which > are in use. Tcp keep alive on all nodes must be less than the firewall > setting. Please refer to > > https://docs.datastax.com/en/cassandra/2.0/cassandra/troubleshooting/trblshootIdleFirewall.html > for > details on TCP settings. > > > Thanks > Anuj > > Sent from Yahoo Mail on Android > <https://overview.mail.yahoo.com/mobile/?.src=Android> > > On Fri, 29 Jan, 2016 at 3:21 pm, Jean Carlo > <jean.jeancar...@gmail.com> wrote: > Hello guys, > > I have a cluster cassandra 2.1.12 with 6 nodes. All the logs of my nodes > are having this messages marked as INFO > > INFO [SharedPool-Worker-1] 2016-01-29 10:40:57,745 Message.java:532 - > Unexpected exception during request; channel = [id: 0xff15eb8c, / > 172.16.162.4:9042] > java.io.IOException: Error while read(...): Connection reset by peer > at io.netty.channel.epoll.Native.readAddress(Native Method) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.doReadBytes(EpollSocketChannel.java:675) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:714) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] > > This happens either the cluster is stressed or not. Btw it is not > production. The ip marked there (172.16.162.4) belongs to a node of the > cluster, this is not the only node that appears, acctually we are having > all the node's ip having that reset by peer problem. > > Our cluster is having more reads than writes. like 50 reads per second. > > Any one got the same problem? > > > Best regards > > Jean Carlo > > "The best way to predict the future is to invent it" Alan Kay > >
Re: Cassandra's log is full of mesages reset by peers even without traffic
Hi anuj, Thx for your replay, acctually I paste part of the result of the command gre done over one log and I can see only the ip of the machine local grep "Unexpected exception during request" /var/opt/hosting/log/cassandra/system.log INFO [SharedPool-Worker-1] 2016-01-29 10:40:47,744 Message.java:532 - Unexpected exception during request; channel = [id: 0x6ebe93cb, / 172.16.162.4:9042] INFO [SharedPool-Worker-1] 2016-01-29 10:40:57,745 Message.java:532 - Unexpected exception during request; channel = [id: 0xff15eb8c, / 172.16.162.4:9042] INFO [SharedPool-Worker-1] 2016-01-29 10:54:33,721 Message.java:532 - Unexpected exception during request; channel = [id: 0xc42cc7ff, / 172.16.162.2:11436 :> /172.16.162.5:9042] INFO [SharedPool-Worker-2] 2016-01-29 10:45:47,761 Message.java:532 - Unexpected exception during request; channel = [id: 0x603349e4, / 172.16.162.4:9042] INFO [SharedPool-Worker-3] 2016-01-29 10:48:17,766 Message.java:532 - Unexpected exception during request; channel = [id: 0x5bed4eae, / 172.16.162.4:9042] INFO [SharedPool-Worker-2] 2016-01-29 10:48:27,767 Message.java:532 - Unexpected exception during request; channel = [id: 0x6136756b, / 172.16.162.4:9042] INFO [SharedPool-Worker-4] 2016-01-29 10:48:27,767 Message.java:532 - Unexpected exception during request; channel = [id: 0x17c83eb8, / 172.16.162.4:9042] INFO [SharedPool-Worker-4] 2016-01-29 10:52:07,778 Message.java:532 - Unexpected exception during request; channel = [id: 0x1a78b589, / 172.16.162.4:9042] INFO [SharedPool-Worker-2] 2016-01-29 10:52:17,779 Message.java:532 - Unexpected exception during request; channel = [id: 0x017117b3, / 172.16.162.4:9042] INFO [SharedPool-Worker-2] 2016-01-29 11:01:37,813 Message.java:532 - Unexpected exception during request; channel = [id: 0x3efbd844, / 172.16.162.4:9042] Then I don't know if the firewall has something to do on that case, becasue it is a local connection over native protocol Best regards Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Fri, Jan 29, 2016 at 11:02 AM, Anuj Wadehra <anujw_2...@yahoo.co.in> wrote: > Hi Jean, > > Please make sure that your Firewall is not dropping TCP connections which > are in use. Tcp keep alive on all nodes must be less than the firewall > setting. Please refer to > > https://docs.datastax.com/en/cassandra/2.0/cassandra/troubleshooting/trblshootIdleFirewall.html > for > details on TCP settings. > > > Thanks > Anuj > > Sent from Yahoo Mail on Android > <https://overview.mail.yahoo.com/mobile/?.src=Android> > > On Fri, 29 Jan, 2016 at 3:21 pm, Jean Carlo > <jean.jeancar...@gmail.com> wrote: > Hello guys, > > I have a cluster cassandra 2.1.12 with 6 nodes. All the logs of my nodes > are having this messages marked as INFO > > INFO [SharedPool-Worker-1] 2016-01-29 10:40:57,745 Message.java:532 - > Unexpected exception during request; channel = [id: 0xff15eb8c, / > 172.16.162.4:9042] > java.io.IOException: Error while read(...): Connection reset by peer > at io.netty.channel.epoll.Native.readAddress(Native Method) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.doReadBytes(EpollSocketChannel.java:675) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:714) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) > ~[netty-all-4.0.23.Final.jar:4.0.23.Final] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] > > This happens either the cluster is stressed or not. Btw it is not > production. The ip marked there (172.16.162.4) belongs to a node of the > cluster, this is not the only node that appears, acctually we are having > all the node's ip having that reset by peer problem. > > Our cluster is having more reads than writes. like 50 reads per second. > > Any one got the same problem? > > > Best regards > > Jean Carlo > > "The best way to predict the future is to invent it" Alan Kay > >
Cassandra's log is full of mesages reset by peers even without traffic
Hello guys, I have a cluster cassandra 2.1.12 with 6 nodes. All the logs of my nodes are having this messages marked as INFO INFO [SharedPool-Worker-1] 2016-01-29 10:40:57,745 Message.java:532 - Unexpected exception during request; channel = [id: 0xff15eb8c, / 172.16.162.4:9042] java.io.IOException: Error while read(...): Connection reset by peer at io.netty.channel.epoll.Native.readAddress(Native Method) ~[netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.doReadBytes(EpollSocketChannel.java:675) ~[netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:714) ~[netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326) ~[netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264) ~[netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) ~[netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) ~[netty-all-4.0.23.Final.jar:4.0.23.Final] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] This happens either the cluster is stressed or not. Btw it is not production. The ip marked there (172.16.162.4) belongs to a node of the cluster, this is not the only node that appears, acctually we are having all the node's ip having that reset by peer problem. Our cluster is having more reads than writes. like 50 reads per second. Any one got the same problem? Best regards Jean Carlo "The best way to predict the future is to invent it" Alan Kay
Tuning chunk_length_kb in cassandra 2.1.12
Hi guys
I want to set the param chunk_length_kb in order to improve the read
latency of my cassandra_stress's test.
This is the table
CREATE TABLE "Keyspace1".standard1 (
key blob PRIMARY KEY,
"C0" blob,
"C1" blob,
"C2" blob,
"C3" blob,
"C4" blob
) WITH bloom_filter_fp_chance = 0.1
AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
AND comment = ''
AND compaction = {'sstable_size_in_mb': '160', 'class':
'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
AND compression = {'sstable_compression':
'org.apache.cassandra.io.compress.SnappyCompressor'}
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99.0PERCENTILE';
I have 6 columns of type blob. This table is filled by cassandra_stres
admin@cqlsh:Keyspace1> select * from standard1 limit 2;
key|
C0 |
C1 |
C2 |
C3 | C4
+++++
0x4b343050393536353531 |
0xe0e3d68ed1536e4d994aa74860270ac91cf7941acb5eefd925815481298f0d558d4f |
0xa43f78202576f1ccbdf50657792fac06f0ca7c9416ee68a08125c8dce4dfd085131d |
0xab12b06bf64c73e708d1b96fea9badc678303906e3d5f5f96fae7d8092ee0df0c54c |
0x428a157cb598487a1b938bdb6c45b09fad3b6408fddc290a6b332b91426b00ddaeb2 |
0x0583038d881ab25be72155bc3aa5cb9ec3aab8e795601abe63a2b35f48ce1e359f5e
I am having a read latency of ~500 microseconds, I think it takes to much
time comparing to the write latency of ~30 microseconds.
My first clue is to fix the chunk_length_kb to a value close to the size
of the rows in kb
Am I in the right direction? If it is true, how can I compute the size of a
row?
Other question, the value of "Compacted partition" of the command nodetool
cfstats migth give me a value close to the chunk_length_kb ?
Best regards
Jean Carlo
"The best way to predict the future is to invent it" Alan Kay
Re: Slow performance after upgrading from 2.0.9 to 2.1.11
I am having the same issue after upgrade cassandra 2.1.12 from 2.0.10. I am not good on jvm so I would like to know how to do what @CorryOpdenakker propose with cassandra. :) I check concurrent_compactors Saludos Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Fri, Jan 29, 2016 at 9:24 PM, Corry Opdenakker <co...@bestdata.be> wrote: > Hi guys, > Cassandra is still new for me, but I have a lot of java tuning experience. > > For root cause detection of performance degradations its always good to > start with collecting a series of java thread dumps. Take at problem > occurrence using a loopscript for example 60 thread dumps with an interval > of 1 or 2 seconds. > Then load those dumps into IBM thread dump analyzer or in "eclipse mat" or > any similar tool and see which methods appear to be most active or blocking > others. > > Its really very useful > > Same can be be done in a normal situation to compare the difference. > > That should give more insights. > > Cheers, Corry > > > Op vrijdag 29 januari 2016 heeft Peddi, Praveen <pe...@amazon.com> het > volgende geschreven: > >> Hello, >> We have another update on performance on 2.1.11. compression_chunk_size >> didn’t really help much but We changed concurrent_compactors from default >> to 64 in 2.1.11 and read latencies improved significantly. However, 2.1.11 >> read latencies are still 1.5 slower than 2.0.9. One thing we noticed in JMX >> metric that could affect read latencies is that 2.1.11 is running >> ReadRepairedBackground and ReadRepairedBlocking too frequently compared to >> 2.0.9 even though our read_repair_chance is same on both. Could anyone >> shed some light on why 2.1.11 could be running read repair 10 to 50 times >> more in spite of same configuration on both clusters? >> >> dclocal_read_repair_chance=0.10 AND >> read_repair_chance=0.00 AND >> >> Here is the table for read repair metrics for both clusters. >> 2.0.9 2.1.11 >> ReadRepairedBackground 5MinAvg 0.006 0.1 >> 15MinAvg 0.009 0.153 >> ReadRepairedBlocking 5MinAvg 0.002 0.55 >> 15MinAvg 0.007 0.91 >> >> Thanks >> Praveen >> >> From: Jeff Jirsa <jeff.ji...@crowdstrike.com> >> Reply-To: <user@cassandra.apache.org> >> Date: Thursday, January 14, 2016 at 2:58 PM >> To: "user@cassandra.apache.org" <user@cassandra.apache.org> >> Subject: Re: Slow performance after upgrading from 2.0.9 to 2.1.11 >> >> Sorry I wasn’t as explicit as I should have been >> >> The same buffer size is used by compressed reads as well, but tuned with >> compression_chunk_size table property. It’s likely true that if you lower >> compression_chunk_size, you’ll see improved read performance. >> >> This was covered in the AWS re:Invent youtube link I sent in my original >> reply. >> >> >> >> From: "Peddi, Praveen" >> Reply-To: "user@cassandra.apache.org" >> Date: Thursday, January 14, 2016 at 11:36 AM >> To: "user@cassandra.apache.org", Zhiyan Shao >> Cc: "Agrawal, Pratik" >> Subject: Re: Slow performance after upgrading from 2.0.9 to 2.1.11 >> >> Hi, >> We will try with reduced “rar_buffer_size” to 4KB. However >> CASSANDRA-10249 <https://issues.apache.org/jira/browse/CASSANDRA-10249> says >> "this only affects users who have 1. disabled compression, 2. switched to >> buffered i/o from mmap’d”. None of this is true for us I believe. We use >> default disk_access_mode which should be mmap. We also used >> LZ4Compressor when created table. >> >> We will let you know if this property had any effect. We were testing >> with 2.1.11 and this was only fixed in 2.1.12 so we need to play with >> latest version. >> >> Praveen >> >> >> >> From: Jeff Jirsa <jeff.ji...@crowdstrike.com> >> Reply-To: <user@cassandra.apache.org> >> Date: Thursday, January 14, 2016 at 1:29 PM >> To: Zhiyan Shao <zhiyan.s...@gmail.com>, "user@cassandra.apache.org" < >> user@cassandra.apache.org> >> Cc: "Agrawal, Pratik" <paagr...@amazon.com> >> Subject: Re: Slow performance after upgrading from 2.0.9 to 2.1.11 >> >> This may be due to https://issues.apache.org/jira/browse/CASSANDRA-10249 >> / https://issues.apache.org/jira/browse/CASSANDRA-8894 - whether or not >> this is really the case depends on how much of your data is in page cache, >> and whether or not you’re using mmap. Since the original question was asked >> by someo
