Streaming and MVs

2016-11-29 Thread Benjamin Roth
s during that period, not only the base table 3. Rebuild Same like bootstrap, isn't it? Did I forget any cases? What do you think? -- Benjamin Roth Prokurist Jaumo GmbH · www.jaumo.com Wehrstraße 46 · 73035 Göppingen · Germany Phone +49 7161 304880-6 · Fax +49 7161 304880-1 AG Ulm · HRB 731058 · Managing Director: Jens Kammerer

CASSANDRA-12888: Streaming and MVs

2016-12-02 Thread Benjamin Roth
t fixes is immense and maybe will be a huge step to get MVs production ready. Thank you very much, Benjamin -- Forwarded message ------ From: Benjamin Roth Date: 2016-11-29 17:04 GMT+01:00 Subject: Streaming and MVs To: dev@cassandra.apache.org I don't know where else to discuss t

Re: Failed Dtest will block cutting releases

2016-12-02 Thread Benjamin Roth
On Mon, Nov 21, 2016 at 11:31 AM sankalp kohli > > > > wrote: > > > > > > > Hi, > > > > We should not cut a releases if Dtest are not passing. I won't > > block > > > > 3.10 on this since we are just discussing this. > > > > > > > > Please provide feedback on this. > > > > > > > > Thanks, > > > > Sankalp > > > > > > > > > > -- Benjamin Roth Prokurist Jaumo GmbH · www.jaumo.com Wehrstraße 46 · 73035 Göppingen · Germany Phone +49 7161 304880-6 · Fax +49 7161 304880-1 AG Ulm · HRB 731058 · Managing Director: Jens Kammerer

Re: Failed Dtest will block cutting releases

2016-12-04 Thread Benjamin Roth
GMT+01:00 Edward Capriolo : > I think it is fair to run a flakey test again. If it is determine it flaked > out due to a conflict with another test or something ephemeral in a long > process it is not worth blocking a release. > > Just deleting it is probably not a good path. > &

Re: Failed Dtest will block cutting releases

2016-12-04 Thread Benjamin Roth
ix them! > > [0] https://github.com/riptano/cassandra-dtest/pull/1399 > [1] > https://issues.apache.org/jira/issues/?jql=project%20% > 3D%20CASSANDRA%20AND%20labels%20%3D%20test-failure%20AND% > 20resolution%20%3D%20unresolved > > -- > Kind regards, > Michael Shuler >

Re: Collecting slow queries

2016-12-05 Thread Benjamin Roth
Mon, Dec 5, 2016 at 12:15 PM, Jan wrote: > > > > > >> HI Folks; > > >> is there a way for 'Collecting slow queries' in the Apache Cassandra. > > ?I > > >> am aware of the DSE product offering such an option, but need the > > so

Re: CASSANDRA-12888: Streaming and MVs

2016-12-07 Thread Benjamin Roth
urrent behavior for MV streaming originating from > repair. > - Create new ticket to include only the base tables and not MVs in > keyspace-level repair, since repairing the base already repairs the views > to avoid people shooting themselves in the foot. > > Please let me know what

Re: CASSANDRA-12888: Streaming and MVs

2016-12-07 Thread Benjamin Roth
Grmpf! 1000+ consecutive must be wrong. I guess I mixed sth up. But it repaired over and over again for 1 or 2 days. 2016-12-07 9:01 GMT+01:00 Benjamin Roth : > Hi Paolo, > > First of all thanks for your review! > > I had the same concerns as you but I thought it is beeing hand

Re: [VOTE] 3.X branch feature freeze

2017-01-13 Thread Benjamin Roth
+1 also I appreciate any effort on MV stability. It is an official 3.x feature but not production ready for the masses. Am 13.01.2017 18:34 schrieb "Jonathan Ellis" : > +1 > > On Fri, Jan 13, 2017 at 11:21 AM, Aleksey Yeschenko > wrote: > > > Hi all! > > > > It seems like we have a general conse

Re: [VOTE] 3.X branch feature freeze

2017-01-13 Thread Benjamin Roth
ter during bootstrap / etc? Any idea of fixing them is a major > undertaking? > > Jon > > On Fri, Jan 13, 2017 at 9:39 AM Benjamin Roth > wrote: > > +1 also I appreciate any effort on MV stability. It is an official 3.x > feature but not production ready for the masses.

Re: Wrapping up tick-tock

2017-01-14 Thread Benjamin Roth
w. > >> > > > >> > > Therefore, I suggest first that we collectively roll up our > > sleeves to > >> > vet > >> > > 3.10 as the last tick-tock release. Stick a fork in it, it’s > > done. No > >> > &g

Re: Why does CockroachDB github website say Cassandra has no Availability on datacenter failure?

2017-02-07 Thread Benjamin Roth
Btw this isn't the Bronx either. It's not incorrect to be polite. Am 07.02.2017 13:45 schrieb "Bernardo Sanchez" < bernard...@pointclickcare.com>: > guys this isn't twitter. stop your stupid posts > > From: benjamin.le...@datastax.com > Sent: February 7, 2017 7:43 AM > To: dev@cassandra.apache.or

Re: If reading from materialized view with a consistency level of quorum am I guaranteed to have the most recent view?

2017-02-09 Thread Benjamin Roth
Yes it is Am 10.02.2017 00:46 schrieb "Kant Kodali" : > If reading from materialized view with a consistency level of quorum am I > guaranteed to have the most recent view? other words is w + r > n contract > maintained for MV's as well for both reads and writes? > > Thanks! >

Re: If reading from materialized view with a consistency level of quorum am I guaranteed to have the most recent view?

2017-02-10 Thread Benjamin Roth
specify > a Consistency Level on wrote for the Materialized View. So, you cannot > apply the R+W>RF formula. > > >Brian > > > On Feb 10, 2017, at 3:17 AM, Kant Kodali wrote: > > > > thanks! > > > > On Thu, Feb 9, 2017 at 8:51 PM, Be

Re: If reading from materialized view with a consistency level of quorum am I guaranteed to have the most recent view?

2017-02-10 Thread Benjamin Roth
d View. So, you > cannot apply the R+W>RF formula. > > > > >Brian > > > >> On Feb 10, 2017, at 3:17 AM, Kant Kodali wrote: > >> > >> thanks! > >> > >> On Thu, Feb 9, 2017 at 8:51 PM, Benjamin Roth > >> wrote: >

Re: If reading from materialized view with a consistency level of quorum am I guaranteed to have the most recent view?

2017-02-10 Thread Benjamin Roth
Hi Kant Is it clear now? Sorry for the confusion! Have a nice one Am 10.02.2017 09:17 schrieb "Kant Kodali" : thanks! On Thu, Feb 9, 2017 at 8:51 PM, Benjamin Roth wrote: > Yes it is > > Am 10.02.2017 00:46 schrieb "Kant Kodali" : > > > If reading fro

Re: If reading from materialized view with a consistency level of quorum am I guaranteed to have the most recent view?

2017-02-10 Thread Benjamin Roth
through MV but prior to MV getting the update from the base table. > so there isn't any way to make sure to read after MV had been successfully > updated. is that correct? > > On Fri, Feb 10, 2017 at 6:30 AM, Benjamin Roth > wrote: > > > Hi Kant > > > > Is it

Re: If reading from materialized view with a consistency level of quorum am I guaranteed to have the most recent view?

2017-02-10 Thread Benjamin Roth
(which is allowed as of today) > > On Fri, Feb 10, 2017 at 10:23 AM, Benjamin Roth > wrote: > > > It depends on your model. > > If the base table + MV have the same partition key, then the MV mutations > > are applied synchronously, so they are written as soon the write re

Re: If reading from materialized view with a consistency level of quorum am I guaranteed to have the most recent view?

2017-02-10 Thread Benjamin Roth
Kodali : > Okies now I understand what you mean by "same" partition key. I think you > are saying > > PRIMARY KEY(col1, col2, col3) == PRIMARY KEY(col2, col1, col3) // so far I > assumed they are different partition keys. > > On Fri, Feb 10, 2017 at 10:36 AM, Benj

Re: If reading from materialized view with a consistency level of quorum am I guaranteed to have the most recent view?

2017-02-10 Thread Benjamin Roth
rent partition keys according to you right? > > PRIMARY KEY ((a, b), c, d) and > PRIMARY KEY ((a, b), d, c, e) > > On Fri, Feb 10, 2017 at 10:48 AM, Benjamin Roth > wrote: > > > Same partition key: > > > > PRIMARY KEY ((a, b), c, d) and > > PRI

Re: If reading from materialized view with a consistency level of quorum am I guaranteed to have the most recent view?

2017-02-10 Thread Benjamin Roth
mutations. 2017-02-10 19:58 GMT+01:00 DuyHai Doan : > See my blog post to understand how MV is implemented: > http://www.doanduyhai.com/blog/?p=1930 > > On Fri, Feb 10, 2017 at 7:48 PM, Benjamin Roth > wrote: > > > Same partition key: > > > > PRIMARY KEY ((a, b),

Re: If reading from materialized view with a consistency level of quorum am I guaranteed to have the most recent view?

2017-02-11 Thread Benjamin Roth
For MVs regarding this threads question only the partition key matters. Different primary keys can have the same partition key. Which is the case in the example in your last comment. Am 10.02.2017 20:26 schrieb "Kant Kodali" : @Benjamin Roth: How do you say something is a different P

MV improvements

2017-02-15 Thread Benjamin Roth
27;d appreciate any kind of support. Thanks folks! -- Benjamin Roth Prokurist Jaumo GmbH · www.jaumo.com Wehrstraße 46 · 73035 Göppingen · Germany Phone +49 7161 304880-6 · Fax +49 7161 304880-1 AG Ulm · HRB 731058 · Managing Director: Jens Kammerer

Re: Pluggable throttling of read and write queries

2017-02-20 Thread Benjamin Roth
Stupid question: Why do you rate limit a database, especially writes. Wouldn't that cause a lot of new issues like back pressure on the rest of your system or timeouts in case of blocking requests? Also rate limiting has to be based on per coordinator calculations and not cluster wide. It reminds m

Re: Pluggable throttling of read and write queries

2017-02-20 Thread Benjamin Roth
wn 10/20/100. In many > cases, workloads can be throttled and still meet business goals - nightly > analytics jobs, for example, may be fine running over the course of 3 hours > instead of 15 minutes, especially if the slightly-higher-response-latency > over 3 hours is better than mu

Official CS docs

2017-03-01 Thread benjamin roth
Hi guys, Is there a reason that the docs are part of the git repo? In my personal opinion this is very complicated and it puts the hurdle to contribute to docs very high. There are so many questions on userlists that repeat over and over again and that could be put into a knowledge base. But ...

Re: Official CS docs

2017-03-01 Thread benjamin roth
; drawbacks. Do you have any example project with a process that we should > follow instead? Did you have a look at the README file in the docs tree > and actually try to add or change any content? What would hold you back > to work from there and submit a patch? > > > > On 01.0

Need feedback on CASSANDRA-13066

2017-03-01 Thread benjamin roth
Hi guys, I started working on 13066. My intention is to offer a table-setting that allows a operator to optimize MV streaming in some cases or simply "on purpose - i know what i do". MV write path streaming can be ommitted e.g. if: - data is append only - no PK is added to MV so no stale data can

Lots of dtest errors on local machine

2017-03-01 Thread benjamin roth
Hi again, I wanted to run some dtests (e.g. from materialized_views_test.py) to check my changes. A while ago, everything worked fine but today I ran into a lot of errors like this: https://gist.github.com/brstgt/114d76769d97dc72059f9252330c4142 This happened on 2 different machines (macos + linu

Re: Lots of dtest errors on local machine

2017-03-02 Thread benjamin roth
Sorry - it was my fault. I introduced a bug and was blinded by the tons of debug output of dtests. 2017-03-01 17:20 GMT+01:00 benjamin roth : > Hi again, > > I wanted to run some dtests (e.g. from materialized_views_test.py) to > check my changes. A while ago, everything worked fine

Consistent vs inconsistent range movements

2017-03-03 Thread benjamin roth
Hi, Can anyone tell the difference between consistent + inconsistent range movements? What exactly makes them consistent or inconsistent? In what situations can both of them occur? It would be great to get a correct and deep understanding of that for further MV improvments. My intuition tells me

Re: State of triggers

2017-03-05 Thread benjamin roth
While I was reading the MV paragraph in your post, an idea popped up: The problem with MV inconsistencies and inconsistent range movement is that the "MV contract" is broken. This only happens because base data and replica data reside on different hosts. If base data + replicas would stay on the s

Re: State of triggers

2017-03-05 Thread benjamin roth
find the user_id that corresponds to email > 'xxx' so that your MV partitioner idea can work ? > > > > On Sun, Mar 5, 2017 at 9:05 AM, benjamin roth wrote: > > > While I was reading the MV paragraph in your post, an idea popped up: > > > > The problem with

Re: State of triggers

2017-03-05 Thread benjamin roth
Sorry. Answer was to fast. Maybe you are right. Am 05.03.2017 09:21 schrieb "benjamin roth" : > No. You just change the partitioner. That's all > > Am 05.03.2017 09:15 schrieb "DuyHai Doan" : > >> "How can that be achieved? I haven't don

Re: State of triggers

2017-03-05 Thread benjamin roth
Not maybe. You are absolutely right. Bad idea. Hmpf. Am 05.03.2017 09:23 schrieb "benjamin roth" : > Sorry. Answer was to fast. Maybe you are right. > > Am 05.03.2017 09:21 schrieb "benjamin roth" : > >> No. You just change the partitioner. That's al

Re: State of triggers

2017-03-05 Thread benjamin roth
There is a German saying: Sometimes you don't see the woods because of the lots of trees. Am 05.03.2017 09:25 schrieb "DuyHai Doan" : > No problem, distributed systems are hard to reason about, I got caught many > times in the past > > On Sun, Mar 5, 2017 at 9:

Re: Contribute to the Cassandra wiki

2017-03-13 Thread benjamin roth
First: I am positively surprised how many guys would like to contribute to docs. Some days ago I posted to the dev-list about doc-contribution. I think this applies here again. From my point of view "in-tree docs" are a good choice for technical references that go closely with the code versioning.

Re: Contribute to the Cassandra wiki

2017-03-13 Thread benjamin roth
Hm maybe a different theme / CSS could make it look a little bit more "2017" https://moinmo.in/HelpOnThemes 2017-03-13 16:27 GMT+01:00 Jeremy Hanna : > The moinmoin wiki was preferred but because of spam, images couldn’t be > attached. The options were to use confluence or have a moderated list

Re: Contribute to the Cassandra wiki

2017-03-13 Thread benjamin roth
Contribution Guide +1 Github WebUI +1 Pull requests +1 Rest: Inspect + Adapt 2017-03-13 19:38 GMT+01:00 Stefan Podkowinski : > Agreed. Let's not give up on this as quickly. My suggestion is to at > least provide a getting started guide for writing docs, before > complaining about too few contrib

Re: Code quality, principles and rules

2017-03-17 Thread benjamin roth
I think you can refactor any project with little risk and increase test coverage. What is needed: Rules. Discipline. Perseverance. Small iterations. Small iterations. Small iterations. - Refactor in the smallest possible unit - Split large classes into smaller ones. Remove god classes by pul

Re: Documentation contributors guide

2017-03-17 Thread benjamin roth
Isn't there a way to script that with just a few lines of python or whatever? 2017-03-17 21:03 GMT+01:00 Jeff Jirsa : > > > On 2017-03-17 12:33 (-0700), Stefan Podkowinski wrote: > > > As you can see there's a large part about using GitHub for editing on > > the page. I'd like to know what you t

Re: Cassandra on RocksDB experiment result

2017-04-21 Thread benjamin roth
I am not a PMC member or sth but just my 2 cents: As long as it is designed from the ground to improve testability, this can be a big chance. If not - it will be a big risk. Theres a huge difference between running a quick hack as a proof of concept and designing a generic architecture and retain

Re: Integrating vendor-specific code and developing plugins

2017-05-15 Thread benjamin roth
Absolutely + Separate repos for separate modules also separate responsibility. IMHO it makes a heterogenuous structure more manageable. Both in a technical and a human or insitutional way. 2017-05-15 13:54 GMT+02:00 Jonathan Haddad : > There's a handful of issues I can think of with shipping eve

Re: State of Materialized Views

2017-07-17 Thread benjamin roth
Hi Kurt, First of all thanks for this elaborate post. At this moment, I don't want to come up with a solution for all MV issues but I would like to point out, why I was quite active some time ago and why I pulled myself back. As you also mentioned in different words, it seems to me that MVs are

Re: State of Materialized Views

2017-07-24 Thread benjamin roth
Hi Josh, Who is "we" in this case? Best, Ben 2017-07-24 15:41 GMT+02:00 Josh McKenzie : > > > > The initial contributors turned their back on MVs > > > We're working on the following MV-related issues in the 4.0 time-frame: > CASSANDRA-13162 > CASSANDRA-13547 > CASSANDRA-13127 >

Re: Customize Cassandra to execute a query multiple times

2017-07-27 Thread benjamin roth
+1 2017-07-27 9:26 GMT+02:00 Marco Massenzio : > Why not simply have a microservice that does this for you? > > It may expose an API that allows to either store queries and/or conditions > that trigger the queries (maybe time elapsed, an alert generated, > whatever...) and it would then connect t

Re: Repair scheduling tools

2018-04-05 Thread benjamin roth
That would be totally awesome! Not sure if it helps here but for completeness: We completely "dumped" regular repairs - no matter if 'nodetool repair' or reaper - and run our own tool that does simply CL_ALL scraping over the whole cluster. It runs now for over a year in production and the only pr

Re: Repair scheduling tools

2018-04-05 Thread benjamin roth
gt; thousands of nodes. It's a totally different situation today. > > > On Thu, Apr 5, 2018 at 11:17 AM benjamin roth wrote: > > > That would be totally awesome! > > > > Not sure if it helps here but for completeness: > > We completely "dumped" re