Re: Batch size warnings

Right you are, thank you Cody. Wondering if I may reach out again to the list and ask a similar question in a more specific way: Scenario: Cassandra 3.x, small cluster (<10 nodes), 1 DC Is a batch warn threshold of 50kb, and average batch sizes in the 40kb range a recipe for regret? Should we

Re: Batch size warnings

There is a disconnect between write.3 and write.4, but it can only affect performance, not consistency. The presence or absence of a row's txnUUID in the IncompleteTransactions table is the ultimate source of truth, and rows whose txnUUID are not null will be checked against that truth in the read

Re: Batch size warnings

Appreciate the long writeup Cody. Yeah, we're good with temporary inconsistency (thankfully) as well. I'm going to try to ride the batch train and hope it doesn't derail - our load is fairly static (or, more precisely, increase in load is fairly slow and can be projected). Enjoyed your

Re: Batch size warnings

Hi Voytek, I think the way you are using it is definitely the canonical way. Unfortunately, as you learned, there are some gotchas. We tried substantially increasing the batch size and it worked for a while, until we reached new scale, and we increased it again, and so forth. It works, but soon

Re: Batch size warnings

@Ed, what you just said reminded me a lot of RAMP transactions. I did a blog post on it here: http://rustyrazorblade.com/2015/11/ramp-made-easy/ I've been considering doing a follow up on how to do a Cassandra data model enabling RAMP transactions, but that takes time, and I have almost zero of

Re: Batch size warnings

I have been circling around a thought process over batches. Now that Cassandra has aggregating functions, it might be possible write a type of record that has an END_OF_BATCH type marker and the data can be suppressed from view until it was all there. IE you write something like a checksum record

Re: Batch size warnings

Ok thanks. Im investingating a Lot. There will be some improvements coming but cannot promise if it will solve All existing problems. We will see and keep working on it. Am 07.12.2016 17:58 schrieb "Voytek Jarnot" : > Been about a month since I have up on it, but it was

Re: Batch size warnings

Been about a month since I have up on it, but it was very much related to the stuff you're dealing with ... Basically Cassandra just stepping on its own er, tripping over its own feet streaming MVs. On Dec 7, 2016 10:45 AM, "Benjamin Roth" wrote: > I meant the

Re: Batch size warnings

I meant the mv thing Am 07.12.2016 17:27 schrieb "Voytek Jarnot" : > Sure, about which part? > > default batch size warning is 5kb > I've increased it to 30kb, and will need to increase to 40kb (8x default > setting) to avoid WARN log messages about batch sizes. I do

Re: Batch size warnings

Sure, about which part? default batch size warning is 5kb I've increased it to 30kb, and will need to increase to 40kb (8x default setting) to avoid WARN log messages about batch sizes. I do realize it's just a WARNing, but may as well avoid those if I can configure it out. That said, having to

Batch size warnings

The low default value for batch_size_warn_threshold_in_kb is making me wonder if I'm perhaps approaching the problem of atomicity in a non-ideal fashion. With one data set duplicated/denormalized into 5 tables to support queries, we use batches to ensure inserts make it to all or 0 tables. This

Re: Batch size warnings

Could you please be more specific? Am 07.12.2016 17:10 schrieb "Voytek Jarnot" : > Should've mentioned - running 3.9. Also - please do not recommend MVs: I > tried, they're broken, we punted. > > On Wed, Dec 7, 2016 at 10:06 AM, Voytek Jarnot >

Re: Batch size warnings

Should've mentioned - running 3.9. Also - please do not recommend MVs: I tried, they're broken, we punted. On Wed, Dec 7, 2016 at 10:06 AM, Voytek Jarnot wrote: > The low default value for batch_size_warn_threshold_in_kb is making me > wonder if I'm perhaps approaching